Repository: cython/cython

Issue:
[ENH] Builtin type support for `memoryview`
<!--
**Note:**
- Do not use the bug and feature tracker for support requests. Use the `cython-users` mailing list instead.
- Did you search for similar issues already? Please do, it helps to save us precious time that we otherwise could not invest into development.
- Did you try the latest master branch or pre-release? It might already have what you want to report. Also see the [Changelog](https://github.com/cython/cython/blob/master/CHANGES.rst) regarding recent changes.
-->

**Is your feature request related to a problem? Please describe.**

It would be useful to have support for a generic builtin `memoryview`. This can come in handy when the type and dimensionality of the `memoryview` are unknown. In these cases it's tricky to use the Cython `memoryview` as we don't have enough information to type it correctly (or maybe there are tricks I don't know?). The benefit being (much like other builtin types) Cython can translate operations on the `memoryview` into more efficient Python C API calls.

**Describe the solution you'd like**

Here's some sample code showing how this might be used.

```cython
def nbytes(memoryview m):
    cdef Py_ssize_t i, n = m.itemsize
    for i in range(m.ndim):
        n *= m.shape[i]
    return n
```

**Describe alternatives you've considered**

If one knows the type and dimensionality, a Cython `memoryview` can be used. However this doesn't cover the case discussed above (unless I'm missing something). Another alternative is to use the Python C API directly (though this may be less pleasant for other readers and could be buggy if not carefully used).

**Additional context**

NA


Discussion:
What C API functions are you hoping that Cython will generate? There don't look to be a huge number and they're mostly about creating memoryviews: https://docs.python.org/3/c-api/memoryview.html. Similarly [the semi-public struct](https://github.com/python/cpython/blob/8784d3300ec4ffc58bc0e9ab3cff9a24187dbe4c/Include/memoryobject.h#L57) doesn't actually expose a lot of information - all the useful shape/strides/suboffset info is hidden an opaque array at the end.

Looking at in more detail I think you'd get the information by accessing the buffer object so maybe there is something that Cython could do here...
Yeah I’m hoping things like `memoryview(a)` are translated into `PyMemoryView_FromObject(a)`, `isinstance(mv, memoryview)` gets translated into an appropriate low-level type check, it may also be valuable to have a method to call `PyMemoryView_GetContiguous`, etc..

As to the underlying `Py_buffer`, I think we should be able to get access to it pretty easily as `PyMemoryView_GET_BUFFER` is a macro that does just that. Though agree queries around the buffer can routed through this mechanism.
If you wanted to have a go at writing it then https://github.com/cython/cython/blob/be43235ba3f4fca32dceda46424b36afb3cb00c9/Cython/Compiler/Builtin.py#L252 is where all the types and their member functions are declared (a few are also in Optimize.py, but I think memoryviews would make most sense in Builtin.py. I think getting something basic working is likely to be fairly easy (these type declarations are pretty simple and repetitive).

I suspect the hardest part will be providing fallback versions for PyPy, but that's something we can give suggestions on once it's clear what's needed.
Thanks for the pointer. That seems like a good place to start.

Do you think we could add [C inline properties]( https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#c-inline-properties ) for the various properties on `memoryview` as well? If so, would they ago in `Builtin.py` or would they go elsewhere? How does Cython handle this kind of thing usually?
> Do you think we could add C inline properties for the various properties on memoryview as well?

It doesn't look like we currently have any examples of handling this kind of thing for the core types. It'd probably need to go in `Builtin.py` with a new class (`BuiltinProperty`) alongside `BuiltinMethod` and `BuiltinAttribute` but that's easier said than done.
> How does Cython handle this kind of thing usually?

There is no "usually" here because most of the code in `Builtin.py` predates C inline properties by years, if not a decade or more. ;-)

But yes, inline properties seem right for this. I remember seeing a somewhat recent PR (this year?) that included an unrelated refactoring of `Builtin.py` to use a class for the extension type declarations. Can't find it right now, but it might be worth basing the changes on that.

Resolving patch:
diff --git a/Cython/Compiler/Builtin.py b/Cython/Compiler/Builtin.py
--- a/Cython/Compiler/Builtin.py
+++ b/Cython/Compiler/Builtin.py
@@ -6,7 +6,7 @@
 
 from .StringEncoding import EncodedString
 from .Symtab import BuiltinScope, StructOrUnionScope, ModuleScope, Entry
-from .Code import UtilityCode
+from .Code import UtilityCode, TempitaUtilityCode
 from .TypeSlots import Signature
 from . import PyrexTypes
 
@@ -91,6 +91,28 @@ def declare_in_type(self, self_type):
             self.py_name, method_type, self.cname, utility_code=self.utility_code)
 
 
+class BuiltinProperty(object):
+    # read only for now
+    def __init__(self, py_name, property_type, call_cname,
+                 exception_value=None, exception_check=None, utility_code=None):
+        self.py_name = py_name
+        self.property_type = property_type
+        self.call_cname = call_cname
+        self.utility_code = utility_code
+        self.exception_value = exception_value
+        self.exception_check = exception_check
+
+    def declare_in_type(self, self_type):
+        self_type.scope.declare_cproperty(
+            self.py_name,
+            self.property_type,
+            self.call_cname,
+            exception_value=self.exception_value,
+            exception_check=self.exception_check,
+            utility_code=self.utility_code
+        )
+
+
 builtin_function_table = [
     # name,        args,   return,  C API func,           py equiv = "*"
     BuiltinFunction('abs',        "d",    "d",     "fabs",
@@ -338,6 +360,32 @@ def declare_in_type(self, self_type):
     ("frozenset", "PyFrozenSet_Type", []),
     ("Exception", "((PyTypeObject*)PyExc_Exception)[0]", []),
     ("StopAsyncIteration", "((PyTypeObject*)__Pyx_PyExc_StopAsyncIteration)[0]", []),
+    ("memoryview", "PyMemoryView_Type", [
+        # TODO - format would be nice, but hard to get
+        # __len__ can be accessed through a direct lookup of the buffer (but probably in Optimize.c)
+        # error checking would ideally be limited api only
+        BuiltinProperty("ndim", PyrexTypes.c_int_type, '__Pyx_PyMemoryview_Get_ndim',
+                        exception_value="-1", exception_check=True,
+                        utility_code=TempitaUtilityCode.load_cached(
+                            "memoryview_get_from_buffer", "Builtins.c",
+                            context=dict(name="ndim")
+                        )
+        ),
+        BuiltinProperty("readonly", PyrexTypes.c_bint_type, '__Pyx_PyMemoryview_Get_readonly',
+                        exception_value="-1", exception_check=True,
+                        utility_code=TempitaUtilityCode.load_cached(
+                            "memoryview_get_from_buffer", "Builtins.c",
+                            context=dict(name="readonly")
+                        )
+        ),
+        BuiltinProperty("itemsize", PyrexTypes.c_py_ssize_t_type, '__Pyx_PyMemoryview_Get_itemsize',
+                        exception_value="-1", exception_check=True,
+                        utility_code=TempitaUtilityCode.load_cached(
+                            "memoryview_get_from_buffer", "Builtins.c",
+                            context=dict(name="itemsize")
+                        )
+        )]
+    )
 ]
 
 
@@ -349,6 +397,7 @@ def declare_in_type(self, self_type):
     'tuple', 'list', 'dict', 'set', 'frozenset',
     # 'str',             # only in Py3.x
     # 'file',            # only in Py2.x
+    'memoryview'
 })
 
 
@@ -424,10 +473,10 @@ def init_builtins():
         '__debug__', PyrexTypes.c_const_type(PyrexTypes.c_bint_type),
         pos=None, cname='(!Py_OptimizeFlag)', is_cdef=True)
 
-    global list_type, tuple_type, dict_type, set_type, frozenset_type
-    global bytes_type, str_type, unicode_type, basestring_type, slice_type
-    global float_type, long_type, bool_type, type_type, complex_type, bytearray_type
-    global int_type
+    global type_type, list_type, tuple_type, dict_type, set_type, frozenset_type
+    global slice_type, bytes_type, str_type, unicode_type, basestring_type, bytearray_type
+    global float_type, int_type, long_type, bool_type, complex_type
+    global memoryview_type, py_buffer_type
     type_type  = builtin_scope.lookup('type').type
     list_type  = builtin_scope.lookup('list').type
     tuple_type = builtin_scope.lookup('tuple').type
@@ -445,6 +494,7 @@ def init_builtins():
     long_type = builtin_scope.lookup('long').type
     bool_type  = builtin_scope.lookup('bool').type
     complex_type  = builtin_scope.lookup('complex').type
+    memoryview_type = builtin_scope.lookup('memoryview').type
 
     # Set up type inference links between equivalent Python/C types
     bool_type.equivalent_type = PyrexTypes.c_bint_type
@@ -456,6 +506,8 @@ def init_builtins():
     complex_type.equivalent_type = PyrexTypes.c_double_complex_type
     PyrexTypes.c_double_complex_type.equivalent_type = complex_type
 
+    py_buffer_type = builtin_scope.lookup('Py_buffer').type
+
 
 init_builtins()
 
diff --git a/Cython/Compiler/FusedNode.py b/Cython/Compiler/FusedNode.py
--- a/Cython/Compiler/FusedNode.py
+++ b/Cython/Compiler/FusedNode.py
@@ -401,13 +401,28 @@ def _buffer_parse_format_string_check(self, pyx_code, decl_code,
 
         pyx_code.context.update(
             specialized_type_name=specialized_type.specialization_string,
-            sizeof_dtype=self._sizeof_dtype(dtype))
+            sizeof_dtype=self._sizeof_dtype(dtype),
+            ndim_dtype=specialized_type.ndim,
+            dtype_is_struct_obj=int(dtype.is_struct or dtype.is_pyobject))
 
+        # use the memoryview object to check itemsize and ndim.
+        # In principle it could check more, but these are the easiest to do quickly
         pyx_code.put_chunk(
             u"""
                 # try {{dtype}}
-                if itemsize == -1 or itemsize == {{sizeof_dtype}}:
-                    memslice = {{coerce_from_py_func}}(arg, 0)
+                if (((itemsize == -1 and arg_as_memoryview.itemsize == {{sizeof_dtype}})
+                        or itemsize == {{sizeof_dtype}})
+                        and arg_as_memoryview.ndim == {{ndim_dtype}}):
+                    {{if dtype_is_struct_obj}}
+                    if __PYX_IS_PYPY2:
+                        # I wasn't able to diagnose why, but PyPy2 fails to convert a
+                        # memoryview to a Cython memoryview in this case
+                        memslice = {{coerce_from_py_func}}(arg, 0)
+                    else:
+                    {{else}}
+                    if True:
+                    {{endif}}
+                        memslice = {{coerce_from_py_func}}(arg_as_memoryview, 0)
                     if memslice.memview:
                         __PYX_XCLEAR_MEMVIEW(&memslice, 1)
                         # print 'found a match for the buffer through format parsing'
@@ -474,9 +489,20 @@ def _buffer_checks(self, buffer_types, pythran_types, pyx_code, decl_code, env):
         self._buffer_check_numpy_dtype(pyx_code, buffer_types, pythran_types)
         pyx_code.dedent(2)
 
-        for specialized_type in buffer_types:
-            self._buffer_parse_format_string_check(
-                    pyx_code, decl_code, specialized_type, env)
+        # creating a Cython memoryview from a Python memoryview avoids the
+        # need to get the buffer multiple times, and we can
+        # also use it to check itemsizes etc
+        pyx_code.put_chunk(
+            """
+            try:
+                arg_as_memoryview = memoryview(arg)
+            except TypeError:
+                pass
+            """)
+        with pyx_code.indenter("else:"):
+            for specialized_type in buffer_types:
+                self._buffer_parse_format_string_check(
+                        pyx_code, decl_code, specialized_type, env)
 
     def _buffer_declarations(self, pyx_code, decl_code, all_buffer_types, pythran_types):
         """
@@ -490,6 +516,7 @@ def _buffer_declarations(self, pyx_code, decl_code, all_buffer_types, pythran_ty
 
                 void __PYX_XCLEAR_MEMVIEW({{memviewslice_cname}} *, int have_gil)
                 bint __pyx_memoryview_check(object)
+                bint __PYX_IS_PYPY2 "(CYTHON_COMPILING_IN_PYPY && PY_MAJOR_VERSION == 2)"
             """)
 
         pyx_code.local_variable_declarations.put_chunk(
@@ -514,6 +541,12 @@ def _buffer_declarations(self, pyx_code, decl_code, all_buffer_types, pythran_ty
                 ndarray = __Pyx_ImportNumPyArrayTypeIfAvailable()
             """)
 
+        pyx_code.imports.put_chunk(
+            u"""
+                cdef memoryview arg_as_memoryview
+            """
+        )
+
         seen_typedefs = set()
         seen_int_dtypes = set()
         for buffer_type in all_buffer_types:
diff --git a/Cython/Compiler/Optimize.py b/Cython/Compiler/Optimize.py
--- a/Cython/Compiler/Optimize.py
+++ b/Cython/Compiler/Optimize.py
@@ -2709,6 +2709,40 @@ def _handle_simple_function_bool(self, node, function, pos_args):
             # coerce back to Python object as that's the result we are expecting
             return operand.coerce_to_pyobject(self.current_env())
 
+    PyMemoryView_FromObject_func_type = PyrexTypes.CFuncType(
+        Builtin.memoryview_type, [
+            PyrexTypes.CFuncTypeArg("value", PyrexTypes.py_object_type, None)
+            ])
+
+    PyMemoryView_FromBuffer_func_type = PyrexTypes.CFuncType(
+        Builtin.memoryview_type, [
+            PyrexTypes.CFuncTypeArg("value", Builtin.py_buffer_type, None)
+            ])
+
+    def _handle_simple_function_memoryview(self, node, function, pos_args):
+        if len(pos_args) != 1:
+            self._error_wrong_arg_count('memoryview', node, pos_args, '1')
+            return node
+        else:
+            if pos_args[0].type.is_pyobject:
+                return ExprNodes.PythonCapiCallNode(
+                    node.pos, "PyMemoryView_FromObject",
+                    self.PyMemoryView_FromObject_func_type,
+                    args = [pos_args[0]],
+                    is_temp = node.is_temp,
+                    py_name = "memoryview")
+            elif pos_args[0].type.is_ptr and pos_args[0].base_type is Builtin.py_buffer_type:
+                # TODO - this currently doesn't work because the buffer fails a
+                # "can coerce to python object" test earlier. But it'd be nice to support
+                return ExprNodes.PythonCapiCallNode(
+                    node.pos, "PyMemoryView_FromBuffer",
+                    self.PyMemoryView_FromBuffer_func_type,
+                    args = [pos_args[0]],
+                    is_temp = node.is_temp,
+                    py_name = "memoryview")
+        return node
+
+
     ### builtin functions
 
     Pyx_strlen_func_type = PyrexTypes.CFuncType(


Test patch:
diff --git a/tests/run/builtin_memory_view.pyx b/tests/run/builtin_memory_view.pyx
new file mode 100644
--- /dev/null
+++ b/tests/run/builtin_memory_view.pyx
@@ -0,0 +1,40 @@
+# mode: run
+
+# Tests Python's builtin memoryview.
+
+from __future__ import print_function
+
+cimport cython
+#from cpython.memoryview cimport PyMemoryView_GET_BUFFER
+
+@cython.test_fail_if_path_exists("//SimpleCallNode")
+def test_convert_from_obj(o):
+    """
+    >>> abc = b'abc'
+    >>> all(x == y for x, y in zip(test_convert_from_obj(abc), abc))
+    True
+    """
+    return memoryview(o)
+
+# TODO - this currently doesn't work because the buffer fails a
+# "can coerce to python object" test earlier. But it'd be nice to support
+'''
+def test_create_from_buffer():
+    """
+    memoryview from Py_buffer exists and is special-cased
+    >>> mview = test_create_from_buffer()
+    >>> >>> all(x == y for x, y in zip(mview, b'argh!'))
+    True
+    """
+    other_view = memoryview(b'argh!')
+    cdef Py_buffer *buf = PyMemoryView_GET_BUFFER(other_view)
+    return memoryview(buf)
+'''
+
+@cython.test_fail_if_path_exists("//AttributeNode")
+def test_optimized_attributes(memoryview view):
+    """
+    >>> test_optimized_attributes(memoryview(b'zzzzz'))
+    1 1 True
+    """
+    print(view.itemsize, view.ndim, view.readonly)