Repository: cython/cython

Issue:
[BUG] `not None` semantics is not supported in Pure Python mode for memory views
### Describe the bug

Pure python mode does not support `not None` when used with memory views. See examples below.

### Code to reproduce the behaviour:

```python
import cython
import typing
import numpy as np

def process_buffer(input_view: cython.int[:,:],
                   output_view: typing.Optional[cython.int[:,:]] = None):

    if output_view is None:
        # Creating a default view, e.g.
        output_view = np.empty_like(input_view)
    
    # process 'input_view' into 'output_view'
    return output_view

process_buffer(None, None)
```

> **Note**
> There is additional issue that memoryviews does not support `typing.Optional()`. Compiling the example fails with error: `typing.Optional[...] cannot be applied to non-Python type int[:]`

### Expected behaviour

`TypeError` is raised in the same way as it is raised in cython version:

```cython
import numpy as np

def process_buffer(int[:,:] input_view not None,
                   int[:,:] output_view=None):

   if output_view is None:
       # Creating a default view, e.g.
       output_view = np.empty_like(input_view)

   # process 'input_view' into 'output_view'
   return output_view


process_buffer(None, None)
```

Also `typing.Optional[]` of memoryview succeeds without error.

### OS

any

### Python version

3.9

### Cython version

master

### Additional context

Similar semantics is already present in Extension types: https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#extension-types-and-none


Discussion:
The difficulty is backwards compatibility (i.e. Cython 3 was probably the time to make this change is we wanted to). I think we'd probably have to introduce a `cython.NonOptional` decorator (name to be debated...) rather than change the behaviour of existing annotated memoryviews.

We should definitely make sure `typing.Optional[cython.int[:]]` is accepted (if it isn't already)

Much as it'd be nice to make it consistent.
As much as I'd dislike making incompatible changes at this point, we could still admit a mistake and change it right in 3.0.1, so that users can depend on that onwards. Pure Python mode hasn't been used much in the past, so we're mostly talking about newly written, still maintained code here that doesn't need to work in Cython 0.29.x.

Also, this is the behaviour that most users would expect or at least want, so it's not like we're really talking something away.

I'd certainly prefer using the standard Optional[] mechanism over anything special that we could come up with ourselves. Special cases aren't special enough to break the rules. This one really doesn't feel like it should be special.


Resolving patch:
diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py
+++ b/Cython/Compiler/Nodes.py
@@ -980,7 +980,7 @@ def inject_type_from_annotations(self, env):
             if "typing.Optional" in modifiers:
                 # "x: Optional[...]"  =>  explicitly allow 'None'
                 arg_type = arg_type.resolve()
-                if arg_type and not arg_type.is_pyobject:
+                if arg_type and not arg_type.can_be_optional():
                     # We probably already reported this as "cannot be applied to non-Python type".
                     # error(annotation.pos, "Only Python type arguments can use typing.Optional[...]")
                     pass
@@ -989,14 +989,14 @@ def inject_type_from_annotations(self, env):
             elif arg_type is py_object_type:
                 # exclude ": object" from the None check - None is a generic object.
                 self.or_none = True
-            elif self.default and self.default.is_none and (arg_type.is_pyobject or arg_type.equivalent_type):
+            elif self.default and self.default.is_none and (arg_type.can_be_optional() or arg_type.equivalent_type):
                 # "x: ... = None"  =>  implicitly allow 'None'
-                if not arg_type.is_pyobject:
+                if not arg_type.can_be_optional():
                     arg_type = arg_type.equivalent_type
                 if not self.or_none:
                     warning(self.pos, "PEP-484 recommends 'typing.Optional[...]' for arguments that can be None.")
                     self.or_none = True
-            elif arg_type.is_pyobject and not self.or_none:
+            elif not self.or_none and arg_type.can_be_optional():
                 self.not_none = True
 
         return arg_type
@@ -1226,10 +1226,9 @@ class TemplatedTypeNode(CBaseTypeNode):
     name = None
 
     def _analyse_template_types(self, env, base_type):
-        require_python_types = base_type.python_type_constructor_name in (
-            'typing.Optional',
-            'dataclasses.ClassVar',
-        )
+        require_optional_types = base_type.python_type_constructor_name == 'typing.Optional'
+        require_python_types = base_type.python_type_constructor_name == 'dataclasses.ClassVar'
+
         in_c_type_context = env.in_c_type_context and not require_python_types
 
         template_types = []
@@ -1242,11 +1241,11 @@ def _analyse_template_types(self, env, base_type):
                     error(template_node.pos, "unknown type in template argument")
                     ttype = error_type
                 # For Python generics we can be a bit more flexible and allow None.
-            elif require_python_types and not ttype.is_pyobject:
+            elif require_python_types and not ttype.is_pyobject or require_optional_types and not ttype.can_be_optional():
                 if ttype.equivalent_type and not template_node.as_cython_attribute():
                     ttype = ttype.equivalent_type
                 else:
-                    error(template_node.pos, "%s[...] cannot be applied to non-Python type %s" % (
+                    error(template_node.pos, "%s[...] cannot be applied to type %s" % (
                         base_type.python_type_constructor_name,
                         ttype,
                     ))
diff --git a/Cython/Compiler/PyrexTypes.py b/Cython/Compiler/PyrexTypes.py
--- a/Cython/Compiler/PyrexTypes.py
+++ b/Cython/Compiler/PyrexTypes.py
@@ -334,6 +334,10 @@ def is_complete(self):
     def is_simple_buffer_dtype(self):
         return False
 
+    def can_be_optional(self):
+        """Returns True if type can be used with typing.Optional[]."""
+        return False
+
     def struct_nesting_depth(self):
         # Returns the number levels of nested structs. This is
         # used for constructing a stack for walking the run-time
@@ -721,6 +725,10 @@ def is_complete(self):
         # incomplete since the underlying struct doesn't have a cython.memoryview object.
         return 0
 
+    def can_be_optional(self):
+        """Returns True if type can be used with typing.Optional[]."""
+        return True
+
     def declaration_code(self, entity_code,
             for_display = 0, dll_linkage = None, pyrex = 0):
         # XXX: we put these guards in for now...
@@ -1254,6 +1262,10 @@ def can_coerce_to_pyobject(self, env):
     def can_coerce_from_pyobject(self, env):
         return True
 
+    def can_be_optional(self):
+        """Returns True if type can be used with typing.Optional[]."""
+        return True
+
     def default_coerced_ctype(self):
         """The default C type that this Python type coerces to, or None."""
         return None


Test patch:
diff --git a/tests/errors/e_typing_errors.pyx b/tests/errors/e_typing_errors.pyx
--- a/tests/errors/e_typing_errors.pyx
+++ b/tests/errors/e_typing_errors.pyx
@@ -40,20 +40,17 @@ cdef class Cls(object):
 
 
 _ERRORS = """
-13:42: typing.Optional[...] cannot be applied to non-Python type int
-13:66: typing.Optional[...] cannot be applied to non-Python type double
-13:93: typing.Optional[...] cannot be applied to non-Python type float
-14:42: typing.Optional[...] cannot be applied to non-Python type double complex
-14:70: typing.Optional[...] cannot be applied to non-Python type long
-14:95: typing.Optional[...] cannot be applied to non-Python type long long
-24:30: typing.Optional[...] cannot be applied to non-Python type int
-24:47: typing.Optional[...] cannot be applied to non-Python type float
-24:87: typing.Optional[...] cannot be applied to non-Python type long
-
-20:30: typing.Optional[...] cannot be applied to non-Python type MyStruct
+13:42: typing.Optional[...] cannot be applied to type int
+13:66: typing.Optional[...] cannot be applied to type double
+13:93: typing.Optional[...] cannot be applied to type float
+14:42: typing.Optional[...] cannot be applied to type double complex
+14:70: typing.Optional[...] cannot be applied to type long
+14:95: typing.Optional[...] cannot be applied to type long long
+24:30: typing.Optional[...] cannot be applied to type int
+24:47: typing.Optional[...] cannot be applied to type float
+24:87: typing.Optional[...] cannot be applied to type long
+
+20:30: typing.Optional[...] cannot be applied to type MyStruct
 
 28:20: Modifier 'typing.ClassVar' is not allowed here.
-
-# FIXME: this should be ok :-?
-33:52: typing.Optional[...] cannot be applied to non-Python type double[:]
 """
diff --git a/tests/errors/e_typing_optional.py b/tests/errors/e_typing_optional.py
--- a/tests/errors/e_typing_optional.py
+++ b/tests/errors/e_typing_optional.py
@@ -32,12 +32,12 @@ def optional_memoryview(d: double[:], o: Optional[double[:]]):
 
 
 _ERRORS = """
-13:44: typing.Optional[...] cannot be applied to non-Python type int
-13:69: typing.Optional[...] cannot be applied to non-Python type double
-13:97: typing.Optional[...] cannot be applied to non-Python type float
-14:44: typing.Optional[...] cannot be applied to non-Python type double complex
-14:73: typing.Optional[...] cannot be applied to non-Python type long
-14:100: typing.Optional[...] cannot be applied to non-Python type long long
-
-20:33: typing.Optional[...] cannot be applied to non-Python type MyStruct
+13:44: typing.Optional[...] cannot be applied to type int
+13:69: typing.Optional[...] cannot be applied to type double
+13:97: typing.Optional[...] cannot be applied to type float
+14:44: typing.Optional[...] cannot be applied to type double complex
+14:73: typing.Optional[...] cannot be applied to type long
+14:100: typing.Optional[...] cannot be applied to type long long
+
+20:33: typing.Optional[...] cannot be applied to type MyStruct
 """
diff --git a/tests/memoryview/memoryview_annotation_typing.py b/tests/memoryview/memoryview_annotation_typing.py
--- a/tests/memoryview/memoryview_annotation_typing.py
+++ b/tests/memoryview/memoryview_annotation_typing.py
@@ -5,8 +5,13 @@
 from __future__ import annotations  # object[:] cannot be evaluated
 
 import cython
+try:
+    import typing
+except ImportError:
+    pass  # Workaround for python 2.7
 import numpy
 
+COMPILED = cython.compiled
 
 def one_dim(a: cython.double[:]):
     """
@@ -37,6 +42,38 @@ def two_dim(a: cython.double[:,:]):
     a[0,0] *= 3
     return a[0,0], a[0,1], a.ndim
 
+def slice_none(m: cython.double[:]):
+    """
+    >>> try:
+    ...     a = slice_none(None)
+    ... except TypeError as exc:
+    ...     assert COMPILED
+    ...     if "Argument 'm' must not be None" not in str(exc): raise
+    ... else:
+    ...     assert a == 1
+    ...     assert not COMPILED
+    """
+    return 1 if m is None else 2
+
+
+def slice_optional(m: typing.Optional[cython.double[:]]):
+    """
+    >>> slice_optional(None)
+    1
+    >>> a = numpy.ones((10,), numpy.double)
+    >>> slice_optional(a)
+    2
+
+    # Make sure that we actually evaluate the type and don't just accept everything.
+    >>> try:
+    ...     x = slice_optional(123)
+    ... except TypeError as exc:
+    ...     if not COMPILED: raise
+    ... else:
+    ...     assert not COMPILED
+    """
+    return 1 if m is None else 2
+
 
 @cython.nogil
 @cython.cfunc