Repository: pyodide/pyodide

Issue:
Performance issues with buffer conversions from python to javascript
Hello @hoodmane, thank you for the magnificent APIs (deep/shallowCopyToJavaScript) that would eliminate the heavy cost of big array conversion from python to Javascript.  I think the shallowCopy version is designed for memory address reference instead of bulks of content copy. But I find the perfomance not so good as I've imagined, which is displayed as the following:
![image](https://user-images.githubusercontent.com/34263430/107121264-74600900-68cc-11eb-8e92-da3b8b09b5f4.png)

It takes about 1~7 seconds for shallowCopyToJavascript() to complete the memory address reference and maybe some necessary meta data copy I guess. However, it's not  adequate for a realtime computation.  Any suggestions for better conversion performance?

_Originally posted by @daoxian in https://github.com/iodide-project/pyodide/issues/1167#issuecomment-774488338_


Discussion:
Hi @daoxian. Thanks for reporting this bad behavior. 

The handling of memory buffers is currently not good, this is an issue we hope to fix at some point. I think the current behavior is to make a javascript `Array` of 1080 javascript `Array`s of 3-byte `Uint8Array`. The innermost layer is sliced out of the buffer memory so saying `arr[700][500][0] = 255` will adjust the original data, but populating those large javascript `Array`s is very inefficient at all (remember that an `Array` is a hashmap). 

My guess is it would do much better if you transposed the array to have shape `(3, 1080, 1920)`, though of course this isn't very ergonomic if for instance you are intending to use this data to back an image.

Would you look at #1168? That issue is closely related to the poor performance here. If you have an opinion about that discussion it would be helpful. One thing that limits my ability to fix the Buffer handling code is that I don't really know what the use cases look like, so I really appreciate any input you have. Of course if you want to take a stab at improving the buffer conversion code, let me know and I can tell you what I think you'll need to know.
Indeed, it's much faster after the image transposed to (3, 1080, 1920).


see the actual case from #1168  and codes in #1203
@hoodmane 
The `shallowCopyToJavascript` method seems only available in ES6 JS, not in nodejs.
In Node.js, either of the follows will trigger a runtime error:
>  img = img.shallowCopyToJavascript(depth=1);
>  img = img.shallowCopyToJavascript(1);

` JsException: ReferenceError: depth is not defined`


And after I have a look at the code, I can't found the definition of variable `depth`:
>     shallowCopyToJavascript : function(){
>       let idresult = _python2js_with_depth(_getPtr(this), depth);
>       let result = Module.hiwire.get_value(idresult);
>       Module.hiwire.decref(idresult);
>       return result;
>     },

Maybe a param `depth`  should be there like function(depth) ?

I've tested it successfully in local env.  I'll  open a PR and fix it.
However, it takes  around 30~40ms for shallowCopyToJavascript to convert a image of  size (287, 287, 3), a little too long for the realtime computation.  I hope the conversion be accomplished in no more than 10ms.
> However, it takes around 30~40ms for shallowCopyToJavascript to convert a image of size (287, 287, 3)

Again, I'm hoping to fix this soon, there is no technical reason that it can't be much faster.

Resolving patch:
diff --git a/docs/sphinx_pyodide/sphinx_pyodide/jsdoc.py b/docs/sphinx_pyodide/sphinx_pyodide/jsdoc.py
--- a/docs/sphinx_pyodide/sphinx_pyodide/jsdoc.py
+++ b/docs/sphinx_pyodide/sphinx_pyodide/jsdoc.py
@@ -4,6 +4,7 @@
 from docutils.utils import new_document
 
 from collections import OrderedDict
+import re
 
 from sphinx import addnodes
 from sphinx.util import rst
@@ -11,9 +12,13 @@
 from sphinx.ext.autosummary import autosummary_table, extract_summary
 
 from sphinx_js.jsdoc import Analyzer as JsAnalyzer
-from sphinx_js.ir import Function
+from sphinx_js.ir import Class, Function
 from sphinx_js.parsers import path_and_formal_params, PathVisitor
-from sphinx_js.renderers import AutoFunctionRenderer, AutoAttributeRenderer
+from sphinx_js.renderers import (
+    AutoFunctionRenderer,
+    AutoAttributeRenderer,
+    AutoClassRenderer,
+)
 
 
 class PyodideAnalyzer:
@@ -47,7 +52,12 @@ def get_object_from_json(self, json):
         path components which JsAnalyzer.get_object requires.
         """
         path = self.longname_to_path(json["longname"])
-        kind = "function" if json["kind"] == "function" else "attribute"
+        if json["kind"] == "function":
+            kind = "function"
+        elif json["kind"] == "class":
+            kind = "class"
+        else:
+            kind = "attribute"
         obj = self.inner.get_object(path, kind)
         obj.kind = kind
         return obj
@@ -58,12 +68,16 @@ def create_js_doclets(self):
         """
 
         def get_val():
-            return OrderedDict([["attribute", []], ["function", []]])
+            return OrderedDict([["attribute", []], ["function", []], ["class", []]])
 
         self.js_docs = {key: get_val() for key in ["globals", "pyodide", "PyProxy"]}
         items = {"PyProxy": []}
         for (key, group) in self._doclets_by_class.items():
             key = [x for x in key if "/" not in x]
+            if key[-1] == "PyBuffer":
+                # PyBuffer stuff is documented as a class. Would be nice to have
+                # a less ad hoc way to deal with this...
+                continue
             if key[-1] == "globalThis":
                 items["globals"] = group
             if key[0] == "pyodide." and key[-1] == "Module":
@@ -76,7 +90,13 @@ def get_val():
                 if json.get("access", None) == "private":
                     continue
                 obj = self.get_object_from_json(json)
+                if isinstance(obj, Class):
+                    # sphinx-jsdoc messes up array types. Fix them.
+                    for x in obj.members:
+                        if hasattr(x, "type"):
+                            x.type = re.sub("Array\.<([a-zA-Z_0-9]*)>", r"\1[]", x.type)
                 if obj.name[0] == '"' and obj.name[-1] == '"':
+                    # sphinx-jsdoc messes up Symbol attributes. Fix them.
                     obj.name = "[" + obj.name[1:-1] + "]"
                 self.js_docs[key][obj.kind].append(obj)
 
@@ -97,11 +117,13 @@ def get_rst(self, obj):
             JsDoc also has an AutoClassRenderer which may be useful in the future."""
             if isinstance(obj, Function):
                 renderer = AutoFunctionRenderer
+            elif isinstance(obj, Class):
+                renderer = AutoClassRenderer
             else:
                 renderer = AutoAttributeRenderer
-            return renderer(self, app, arguments=["dummy"]).rst(
-                [obj.name], obj, use_short_name=False
-            )
+            return renderer(
+                self, app, arguments=["dummy"], options={"members": ["*"]}
+            ).rst([obj.name], obj, use_short_name=False)
 
         def get_rst_for_group(self, objects):
             return [self.get_rst(obj) for obj in objects]
@@ -144,6 +166,9 @@ def run(self):
             for group_name, group_objects in value.items():
                 if not group_objects:
                     continue
+                if group_name == "class":
+                    # Plural of class is "classes" not "classs"
+                    group_name += "e"
                 result.append(self.format_heading(group_name.title() + "s:"))
                 table_items = self.get_summary_table(module, group_objects)
                 table_markup = self.format_table(table_items)


Test patch:
diff --git a/packages/numpy/test_numpy.py b/packages/numpy/test_numpy.py
--- a/packages/numpy/test_numpy.py
+++ b/packages/numpy/test_numpy.py
@@ -1,3 +1,6 @@
+import pytest
+
+
 def test_numpy(selenium):
     selenium.load_package("numpy")
     selenium.run("import numpy")
@@ -191,3 +194,107 @@ def test_runwebworker_numpy(selenium_standalone):
         """
     )
     assert output == "[0. 0. 0. 0. 0.]"
+
+
+def test_get_buffer(selenium):
+    selenium.run_js(
+        """
+        await pyodide.runPythonAsync(`
+            import numpy as np
+            x = np.arange(24)
+            z1 = x.reshape([8,3])
+            z2 = z1[-1::-1]
+            z3 = z1[::,-1::-1]
+            z4 = z1[-1::-1,-1::-1]
+        `);
+        for(let x of ["z1", "z2", "z3", "z4"]){
+            let z = pyodide.pyimport(x).getBuffer("u32");
+            for(let idx1 = 0; idx1 < 8; idx1++) {
+                for(let idx2 = 0; idx2 < 3; idx2++){
+                    let v1 = z.data[z.offset + z.strides[0] * idx1 + z.strides[1] * idx2];
+                    let v2 = pyodide.runPython(`repr(${x}[${idx1}, ${idx2}])`);
+                    console.log(`${v1}, ${typeof(v1)}, ${v2}, ${typeof(v2)}, ${v1===v2}`);
+                    if(v1.toString() !== v2){
+                        throw new Error(`Discrepancy ${x}[${idx1}, ${idx2}]: ${v1} != ${v2}`);
+                    }
+                }
+            }
+            z.release();
+        }
+        """
+    )
+
+
+@pytest.mark.parametrize(
+    "arg",
+    [
+        "np.arange(6).reshape((2, -1))",
+        "np.arange(12).reshape((3, -1))[::2, ::2]",
+        "np.arange(12).reshape((3, -1))[::-1, ::-1]",
+        "np.arange(12).reshape((3, -1))[::, ::-1]",
+        "np.arange(12).reshape((3, -1))[::-1, ::]",
+        "np.arange(12).reshape((3, -1))[::-2, ::-2]",
+        "np.arange(6).reshape((2, -1)).astype(np.int8, order='C')",
+        "np.arange(6).reshape((2, -1)).astype(np.int8, order='F')",
+        "np.arange(6).reshape((2, -1, 1))",
+        "np.ones((1, 1))[0:0]",  # shape[0] == 0
+        "np.ones(1)",  # ndim == 0
+    ]
+    + [
+        f"np.arange(3).astype(np.{type_})"
+        for type_ in ["int8", "uint8", "int16", "int32", "float32", "float64"]
+    ],
+)
+def test_get_buffer_roundtrip(selenium, arg):
+    selenium.run_js(
+        f"""
+        await pyodide.runPythonAsync(`
+            import numpy as np
+            x = {arg}
+        `);
+        window.x_js_buf = pyodide.pyimport("x").getBuffer();
+        x_js_buf.length = x_js_buf.data.length;
+        """
+    )
+
+    selenium.run_js(
+        """
+        pyodide.runPython(`
+            import itertools
+            from unittest import TestCase
+            from js import x_js_buf
+            assert_equal = TestCase().assertEqual
+
+            assert_equal(x_js_buf.ndim, x.ndim)
+            assert_equal(x_js_buf.shape.to_py(), list(x.shape))
+            assert_equal(x_js_buf.strides.to_py(), [s/x.itemsize for s in x.data.strides])
+            assert_equal(x_js_buf.format, x.data.format)
+            if len(x) == 0:
+                assert x_js_buf.length == 0
+            else:
+                minoffset = 1000
+                maxoffset = 0
+                for tup in itertools.product(*[range(n) for n in x.shape]):
+                    offset = x_js_buf.offset + sum(x*y for (x,y) in zip(tup, x_js_buf.strides))
+                    minoffset = min(offset, minoffset)
+                    maxoffset = max(offset, maxoffset)
+                    assert_equal(x[tup], x_js_buf.data[offset])
+                assert_equal(minoffset, 0)
+                assert_equal(maxoffset + 1, x_js_buf.length)
+            x_js_buf.release()
+        `);
+        """
+    )
+
+
+def test_get_buffer_error_messages(selenium):
+    with pytest.raises(Exception, match="Javascript has no Float16Array"):
+        selenium.run_js(
+            """
+            await pyodide.runPythonAsync(`
+                import numpy as np
+                x = np.ones(2, dtype=np.float16)
+            `);
+            pyodide.pyimport("x").getBuffer();
+            """
+        )
diff --git a/src/tests/test_pyproxy.py b/src/tests/test_pyproxy.py
--- a/src/tests/test_pyproxy.py
+++ b/src/tests/test_pyproxy.py
@@ -198,6 +198,40 @@ def test():
     assert result == result2
 
 
+def test_pyproxy_get_buffer(selenium):
+    selenium.run_js(
+        """
+        await pyodide.runPython(`
+            from sys import getrefcount
+            z1 = memoryview(bytes(range(24))).cast("b", [8,3])
+            z2 = z1[-1::-1]
+        `);
+        for(let x of ["z1", "z2"]){
+            pyodide.runPython(`assert getrefcount(${x}) == 2`);
+            let proxy = pyodide.globals.get(x);
+            pyodide.runPython(`assert getrefcount(${x}) == 3`);
+            let z = proxy.getBuffer();
+            pyodide.runPython(`assert getrefcount(${x}) == 4`);
+            proxy.destroy();
+            pyodide.runPython(`assert getrefcount(${x}) == 3`);
+            for(let idx1 = 0; idx1 < 8; idx1++) {
+                for(let idx2 = 0; idx2 < 3; idx2++){
+                    let v1 = z.data[z.offset + z.strides[0] * idx1 + z.strides[1] * idx2];
+                    let v2 = pyodide.runPython(`repr(${x}[${idx1}, ${idx2}])`);
+                    console.log(`${v1}, ${typeof(v1)}, ${v2}, ${typeof(v2)}, ${v1===v2}`);
+                    if(v1.toString() !== v2){
+                        throw new Error(`Discrepancy ${x}[${idx1}, ${idx2}]: ${v1} != ${v2}`);
+                    }
+                }
+            }
+            z.release();
+            pyodide.runPython(`assert getrefcount(${x}) == 2`);
+            pyodide.runPython(`del ${x}`);
+        }
+        """
+    )
+
+
 def test_pyproxy_mixins(selenium):
     result = selenium.run_js(
         """