Repository: pyodide/pyodide

Issue:
Performance issues in buffer conversions from javascript to python 
since in my case from time to time in python I have to aquire arrays returned from JS methods,  i.e.,  a huge image array returned by some method of opencv.js which is now lacked in python packages.  The cost of automatic copying the array data from JavaScript to Python is too much, and can't be beared in some realtime computation cases.

_Originally posted by @daoxian in https://github.com/iodide-project/pyodide/issues/1192#issuecomment-774496411_
Soundness issues in buffer conversions code
Currently if one converts a 1d ndarray from Python ==> javascript, `_python2js_shareable_buffer_recursive` produces a `TypedArray` object that shares memory with the `ndarray` but without associating the lifetime of the `TypedArray` with the backing memory in any way, so the memory is borrowed. If the Python object is later garbage collected, then the memory backing this `TypedArray` will be freed, but without any warning to someone who stored the `TypedArray`. The memory may then be reallocated to something else, potentially leading to memory corruption. If the `TypedArray` is reimported into Python it yields a Python `memoryview` which also points to the same borrowed memory.

I think the only reason this doesn't happen more often is that we leak a lot of memory and people don't use the buffer conversions that much.

I think in the case that the user attempts to import a `TypedArray` on the Wasm heap into python, we should actually throw an error because that memory has been malloc'd and it may eventually be freed but we have no way of knowing when it is freed. I think we need some extra Javascript type which is a `TypedArray` referencing owned memory with a `release` method to let go of the memory when it is done. If the memory has a static lifetime then the `release` method can be a no-op. Then this pair `(TypedArray, release)` can be safely imported into Python as a Buffer with a `bf_release` implementation that calls the `release` function. 

This obviously will require a more carefully designed API, but the current system simply does not work.

See PR #1215 which partially addresses this. @daoxian


Discussion:
@daoxian If you wouldn't you mind posting a small code example, it would be helpful.
> @daoxian If you wouldn't you mind posting a small code example, it would be helpful.

Sure. 

In python, I want to resize a image in opencv. Currently no opencv present in pyodide packages , so I have to call opencv.js which residents in JS:

>             print(im_patch.size, im_patch.shape)  # m_patch is an numpy ndarray stands for an image
>             im_patch_bytes = im_patch.tobytes()   # conver to bytes in order to avoid mem copy when converting to JS arg of resize_img()
>             scaled_img_raw = js.g_tracker.resize_img(im_patch_bytes, (im_patch.shape[0], im_patch.shape[1]), (model_sz, model_sz))
>             im_patch = np.asarray(scaled_img_raw, dtype=np.uint8).reshape(model_sz, model_sz, 3)


In  JavaScript class method:
>   resize_img (img_bytes /* auto converted to Uint8ClampedArray already */, from_shape, to_shape) {
>     const img=cv.matFromArray(from_shape[0], from_shape[1], cv.CV_8UC3, img_bytes);    
>     const scaledImg = new cv.Mat();
>     cv.resize(img, scaledImg, new cv.Size(to_shape[1], to_shape[0]), 0, 0, cv.INTER_NEAREST);
>     return new Uint8ClampedArray(scaledImg.data); 
>   }

As to why I don't use Pillow to resize an image is due to its poor performance.
The case is that I'll migrate an `Object Tracking Algorithm` written in python to JavaScript so as to run it in the browser. It's a realtime tracking algorithm. With the input of a streaming of images, it should process every image and run as fast as possible.
Thanks! This is a very interesting example.

In your particular use case there is a way to fix the performance. I think roughly the following should work:
```javascript
function alloc_matrix_3d(shape0, shape1, shape2){
   let num_bytes = shape0*shape1*shape2;
   let ptr = pyodide._module._PyMem_Malloc(num_bytes);
   let bytes = pyodie._module.HEAP8.subarray(ptr, ptr + num_bytes);
   return [ptr, cv.matFromArray(from_shape[0], from_shape[1], cv.CV_8UC3, bytes)];
}

function free_ptr(ptr){
   pyodide._module._PyMem_Free(ptr);
}

function resize_img (img_bytes /* auto converted to Uint8ClampedArray already */, from_shape, to_shape) {
   const img=cv.matFromArray(from_shape[0], from_shape[1], cv.CV_8UC3, img_bytes);
   // Somehow you need to remember to free_ptr later, otherwise you'll leak this memory
   const [ptr, scaledImg] = alloc_matrix_3d(to_shape[1], to_shape[0], cv.CV_8UC3);
   cv.resize(img, scaledImg, new cv.Size(to_shape[1], to_shape[0]), 0, 0, cv.INTER_NEAREST);
   return new Uint8ClampedArray(scaledImg.data);
}
```
The performance issue happens in the following code starting on [jsproxy.c line 173](https://github.com/iodide-project/pyodide/blob/master/src/core/jsproxy.c#L713):

```C
static int
JsBuffer_GetBuffer(PyObject* obj, Py_buffer* view, int flags)
{
  bool success = false;
  JsBuffer* self = (JsBuffer*)obj;
  view->obj = NULL;

  void* ptr;
  if (hiwire_is_on_wasm_heap(JsProxy_REF(self))) {
    // This is fast (no copy)
    ptr = (void*)hiwire_get_byteOffset(JsProxy_REF(self));
  } else {
    // Copy the data onto wasm_heap =(
    ptr = PyBytes_AsString(self->bytes);
    FAIL_IF_NULL(ptr);
    hiwire_copy_to_ptr(JsProxy_REF(self), ptr);
  }
```
So the trick is to pass an object that `is_on_wasm_heap` already.
> Thanks! This is a very interesting example.
> 
> In your particular use case there is a way to fix the performance. I think roughly the following should work:
> 
> ```js
> function alloc_matrix_3d(shape0, shape1, shape2){
>    let num_bytes = shape0*shape1*shape2;
>    let ptr = pyodide._module._PyMem_Malloc(num_bytes);
>    let bytes = pyodie._module.HEAP8.subarray(ptr, ptr + num_bytes);
>    return [ptr, cv.matFromArray(from_shape[0], from_shape[1], cv.CV_8UC3, bytes)];
> }
> 
> function free_ptr(ptr){
>    pyodide._module._PyMem_Free(ptr);
> }
> 
> function resize_img (img_bytes /* auto converted to Uint8ClampedArray already */, from_shape, to_shape) {
>    const img=cv.matFromArray(from_shape[0], from_shape[1], cv.CV_8UC3, img_bytes);
>    // Somehow you need to remember to free_ptr later, otherwise you'll leak this memory
>    const [ptr, scaledImg] = alloc_matrix_3d(to_shape[1], to_shape[0], cv.CV_8UC3);
>    cv.resize(img, scaledImg, new cv.Size(to_shape[1], to_shape[0]), 0, 0, cv.INTER_NEAREST);
>    return new Uint8ClampedArray(scaledImg.data);
> }
> ```

It looks so amazing!  I'll check it out, thanks!
I think we should make a public API to allocate / free on the pyodide heap. This is not the only performance issue in buffer conversions, but at least in this case where you are already explicitly allocating the memory it's really unfortunate to allocate it in the wrong place and then perform an extra copy because there's no API to allocate in the right place.

Thanks a lot for your reports, these conversations are very helpful. Is your code publicly available somewhere? I'd be interested to look at it.
> Thanks a lot for your reports, these conversations are very helpful. Is your code publicly available somewhere? I'd be interested to look at it.

The original python codes are taken from here:  
[https://github.com/STVIR/pysot/blob/2f98bfb6f7b56e4a4263c7024a3a3a7143d630ac/pysot/tracker/base_tracker.py#L88](https://github.com/STVIR/pysot/blob/2f98bfb6f7b56e4a4263c7024a3a3a7143d630ac/pysot/tracker/base_tracker.py#L88)

Example:
```python
pyodide.runPython("x = b'123'");
a = pyodide.pyimport("x"); // a ==> Uint8ClampedArray(3) [49, 50, 51]
pyodide.runPython("del x");
a // ==> Uint8ClampedArray(3) [64, 0, 0]
```
Roger that and thanks a lot.
Some extra comments: In most cases this API (pyodide.getRawBuffer) works well.  When an numpy ndarray is NOT contiguous after some complicated ops, the raw buffer may possibly become  unrecognizable by `cv.matFromArray()`, which will cause a "array is too long" exception.  The resolution is to make the numpy ndarray contingious again by `numpy.ascontingiousarray()`  before passing it as a argument to that opencv method.  And another not-so-efficient way is to convert it into bytes before passing into JS.  However, the issue has nothing to do with the  API.
For instance if you say: `myslice = myarray[::2]`, does `cv.matFromArray` have a problem with it already?
> For instance if you say: `myslice = myarray[::2]`, does `cv.matFromArray` have a problem with it already?


My case is:
```
import numpy as np
a=np.ones(6220800, dtype=np.uint8).reshape(1080,1920,3)
b=a[100:300, 500:800, :]
print(b.flags, b.shape, b.strides)
c=np.ascontiguousarray(b)
print(c.flags, c.shape, c.strides)
```
b is not contiguous and b.strides[0] is not the expected `1 * 3 * 300`, but a wierd 5760.
In JS, if we pass b.getRawBuffer().buffer to cv.matFromArray, an exception will be thrown.
After ascontiguousarray,  c is now a normal ndarray.

> For instance if you say: `myslice = myarray[::2]`, does `cv.matFromArray` have a problem with it already?

after your above op, the flags.c_contiguous is still True though the strides changed.  In this case, cv.matFromArray will be successful if we don't care about the underlying data.
@hoodmane  Is  the getRawBuffer() API in Chrome browser any different from that in Firefox? I found a performance lag in Chrome.   In Firefox it takes 1ms to run the API, but 10~50 ms in Chome. 

python the caller:  
```    
  im_patch = np.ascontiguousarray(im_patch)  
  scaled_img_raw = js.g_tracker.resize_img(im_patch, (model_sz, model_sz))
```

JS the callee:
```
  resize_img (img_pyproxy, to_shape) {
    console.log('img_pyproxy=', img_pyproxy, to_shape);
    const {buffer, shape, strides, release} = img_pyproxy.getRawBuffer('u8');
    console.log('get raw buffer ok');
    const img = cv.matFromArray(shape[0], shape[1], cv.CV_8UC3, buffer);   
    const scaledImg = new cv.Mat();
    cv.resize(img, scaledImg, new cv.Size(to_shape[1], to_shape[0]), 0, 0, cv.INTER_NEAREST);
    img_pyproxy.destroy();
    release();
    return new Uint8ClampedArray(scaledImg.data);
  }
```

Sample output for Firefox:
```

19:19:51.232 img_pyproxy= Proxy { <target>: target(), <handler>: {…} } Array [ 287, 287 ]
19:19:51.233 get raw buffer ok
```

Sample output for Chrome:
```
19:25:01.462 img_pyproxy= Proxy {$$: {…}, toString: ƒ, destroy: ƒ, apply: ƒ, shallowCopyToJavascript: ƒ, …} (2) [287, 287]
19:25:01.488 get raw buffer ok
```


> after your above op, the flags.c_contiguous is still True though the strides changed. 

No it's not:

```python
>>> a = np.arange(24).reshape([8,3])
>>> b =a[::2]
>>> av = memoryview(a)
>>> bv = memoryview(b)
>>> av.c_contiguous 
True
>>> bv.c_contiguous 
False
```

The definition of `_IsCContiguous` in `cpython/Objects/abstract.c` is:

```C
    sd = view->itemsize;
    for (i=view->ndim-1; i>=0; i--) {
        dim = view->shape[i];
        if (dim > 1 && view->strides[i] != sd) {
            return 0;
        }
        sd *= dim;
    }
    return 1;
```

>  Is the getRawBuffer() API in Chrome browser any different from that in Firefox? I found a performance lag in Chrome. In Firefox it takes 1ms to run the API, but 10~50 ms in Chome.

No clue why you're seeing this. Will try to reproduce.
> > after your above op, the flags.c_contiguous is still True though the strides changed.
> 
> No it's not:
> 
> ```python
> >>> a = np.arange(24).reshape([8,3])
> >>> b =a[::2]
> >>> av = memoryview(a)
> >>> bv = memoryview(b)
> >>> av.c_contiguous 
> True
> >>> bv.c_contiguous 
> False
> ```
> 
> The definition of `_IsCContiguous` in `cpython/Objects/abstract.c` is:
> 
> ```c
>     sd = view->itemsize;
>     for (i=view->ndim-1; i>=0; i--) {
>         dim = view->shape[i];
>         if (dim > 1 && view->strides[i] != sd) {
>             return 0;
>         }
>         sd *= dim;
>     }
>     return 1;
> ```

It's not definite, try this:
```
a=np.arange(30, dtype=np.uint8).reshape(2,5,3)
b=a[::2]

```

> > Is the getRawBuffer() API in Chrome browser any different from that in Firefox? I found a performance lag in Chrome. In Firefox it takes 1ms to run the API, but 10~50 ms in Chome.
> 
> No clue why you're seeing this. Will try to reproduce.

Thanks.
Chrome ver: 88.0.4324.150
FireFox ver:  85.0.2 
> > Is the getRawBuffer() API in Chrome browser any different from that in Firefox? I found a performance lag in Chrome. In Firefox it takes 1ms to run the API, but 10~50 ms in Chome.
> 
> No clue why you're seeing this. Will try to reproduce.

Have you reproduced the issue in Chrome?  @hoodmane  
I find that it takes around 5~15ms for JS entering the `getRawBuffer` method. 

>   Module.PyProxyBufferMethods = {
>     getRawBuffer : function(type = "u8"){
>       console.log( "enter function\n");
>       let ArrayType = type_to_array_map.get(type);
>       console.log( "type_to_array_map.get done\n");
>       if(ArrayType === undefined){
>         throw new Error(`Unknown type ${type}`);
>       }
>       console.log( "before _getPtr\n");
>       let this_ptr = _getPtr(this);
>       console.log( "before memview_get_buffer\n");
>       let buffer_struct_ptr = __pyproxy_memoryview_get_buffer(this_ptr);
>       console.log( "after memview_get_buffer\n");
>       if(buffer_struct_ptr === 0){
>         throw new Error("Failed");
>       }
> 

the log is:
![image](https://user-images.githubusercontent.com/34263430/108373578-05bf6b80-723b-11eb-97fe-f59e6152aed3.png)

------PS:------------
It's not always the case, maybe due to some outer factor, so I close it.

@hoodmane I found a bug in your proxy-buffer-procs branch:  
src/core/pyproxy.c, line 667:

>   if(window.BigInt64Array)

will break the web worker's rule (cannot access window in web worker) and throws exception when in web workers. You could fix it before PR merge.
>   if(BigInt64Array)

is just fine to run.
@daoxian Thanks for pointing this out. Could you put comments about improvements to specific PRs on the PR you're talking about?

It should say 
```javascript
if(globalThis.BigInt64Array)
```

Resolving patch:
diff --git a/packages/micropip/micropip/micropip.py b/packages/micropip/micropip/micropip.py
--- a/packages/micropip/micropip/micropip.py
+++ b/packages/micropip/micropip/micropip.py
@@ -33,7 +33,7 @@ async def _get_url(url):
             raise OSError(
                 f"Request for {url} failed with status {resp.status}: {resp.statusText}"
             )
-        return io.BytesIO(await resp.arrayBuffer())
+        return io.BytesIO((await resp.arrayBuffer()).to_py())
 
 
 else:
diff --git a/src/pyodide-py/_pyodide/_core.py b/src/pyodide-py/_pyodide/_core.py
--- a/src/pyodide-py/_pyodide/_core.py
+++ b/src/pyodide-py/_pyodide/_core.py
@@ -82,6 +82,25 @@ def finally_(self, onfinally: Callable) -> "Promise":
             this is needed because ``finally`` is a reserved keyword in Python.
             """
 
+        # There are no types for buffers:
+        # https://github.com/python/typing/issues/593
+        # https://bugs.python.org/issue27501
+        # This is just for docs so lets just make something up?
+
+        def assign(self, rhs: "ReadBuffer"):
+            """Assign from a Python buffer into the Javascript buffer.
+
+            Present only if the wrapped Javascript object is an ArrayBuffer or
+            an ArrayBuffer view.
+            """
+
+        def assign_to(self, to: "ReadWriteBuffer"):
+            """Assign to a Python buffer from the Javascript buffer.
+
+            Present only if the wrapped Javascript object is an ArrayBuffer or
+            an ArrayBuffer view.
+            """
+
     # from pyproxy.c
 
     def create_once_callable(obj: Callable) -> JsProxy:


Test patch:
diff --git a/packages/numpy/test_numpy.py b/packages/numpy/test_numpy.py
--- a/packages/numpy/test_numpy.py
+++ b/packages/numpy/test_numpy.py
@@ -34,7 +34,7 @@ def test_typed_arrays(selenium):
         selenium.run_js(f"window.array = new {jstype}([1, 2, 3, 4]);\n")
         assert selenium.run(
             "from js import array\n"
-            "npyarray = numpy.asarray(array)\n"
+            "npyarray = numpy.asarray(array.to_py())\n"
             f'npyarray.dtype.name == "{npytype}" '
             "and npyarray == [1, 2, 3, 4]"
         )
diff --git a/src/tests/test_jsproxy.py b/src/tests/test_jsproxy.py
--- a/src/tests/test_jsproxy.py
+++ b/src/tests/test_jsproxy.py
@@ -699,6 +699,77 @@ def raises(exc, match=None):
     )
 
 
+def test_buffer(selenium):
+    selenium.run_js(
+        """
+        self.a = new Uint32Array(Array.from({length : 10}, (_,idx) => idx));
+        pyodide.runPython(`
+            from js import a
+            b = a.to_py()
+            b[4] = 7
+            assert b[8] == 8
+            a.assign_to(b)
+            assert b[4] == 4
+            b[4] = 7
+            a.assign(b)
+            assert a[4] == 7
+        `);
+        if(a[4] !== 7){
+            throw Error();
+        }
+        """
+    )
+    selenium.run_js(
+        """
+        self.a = new Uint32Array(Array.from({length : 10}, (_,idx) => idx));
+        pyodide.runPython(`
+            import js
+            from unittest import TestCase
+            raises = TestCase().assertRaisesRegex
+            from array import array
+            from js import a
+            c = array('b', range(30))
+            d = array('b', range(40))
+            with raises(ValueError, "cannot assign to TypedArray"):
+                a.assign(c)
+
+            with raises(ValueError, "cannot assign from TypedArray"):
+                a.assign_to(c)
+
+            with raises(ValueError, "incompatible formats"):
+                a.assign(d)
+
+            with raises(ValueError, "incompatible formats"):
+                a.assign_to(d)
+
+            e = array('I', range(10, 20))
+            a.assign(e)
+        `);
+        for(let [k, v] of a.entries()){
+            if(v !== k + 10){
+                throw new Error([v, k]);
+            }
+        }
+        """
+    )
+
+
+def test_buffer_assign_back(selenium):
+    result = selenium.run_js(
+        """
+        self.jsarray = new Uint8Array([1,2,3, 4, 5, 6]);
+        pyodide.runPython(`
+            from js import jsarray
+            array = jsarray.to_py()
+            array[1::2] = bytes([20, 77, 9])
+            jsarray.assign(array)
+        `);
+        return Array.from(jsarray)
+        """
+    )
+    assert result == [1, 20, 3, 77, 5, 9]
+
+
 def test_memory_leaks(selenium):
     # refcounts are tested automatically in conftest by default
     selenium.run_js(
diff --git a/src/tests/test_typeconversions.py b/src/tests/test_typeconversions.py
--- a/src/tests/test_typeconversions.py
+++ b/src/tests/test_typeconversions.py
@@ -384,15 +384,17 @@ def test_js2python(selenium):
     assert selenium.run("t.jspython is open")
     assert selenium.run(
         """
-        ((t.jsbytes.tolist() == [1, 2, 3])
-         and (t.jsbytes.tobytes() == b"\x01\x02\x03"))
+        jsbytes = t.jsbytes.to_py()
+        ((jsbytes.tolist() == [1, 2, 3])
+         and (jsbytes.tobytes() == b"\x01\x02\x03"))
         """
     )
     assert selenium.run(
         """
+        jsfloats = t.jsfloats.to_py()
         import struct
         expected = struct.pack("fff", 1, 2, 3)
-        (t.jsfloats.tolist() == [1, 2, 3]) and (t.jsfloats.tobytes() == expected)
+        (jsfloats.tolist() == [1, 2, 3]) and (jsfloats.tobytes() == expected)
         """
     )
     assert selenium.run('str(t.jsobject) == "[object XMLHttpRequest]"')
@@ -422,7 +424,6 @@ def test_js2python_bool(selenium):
     )
 
 
-@pytest.mark.parametrize("wasm_heap", (False, True))
 @pytest.mark.parametrize(
     "jstype, pytype",
     (
@@ -437,43 +438,35 @@ def test_js2python_bool(selenium):
         ("Float64Array", "d"),
     ),
 )
-def test_typed_arrays(selenium, wasm_heap, jstype, pytype):
-    if not wasm_heap:
-        selenium.run_js(f"window.array = new {jstype}([1, 2, 3, 4]);\n")
-    else:
-        selenium.run_js(
-            f"""
-             let buffer = pyodide._module._malloc(
-                   4 * {jstype}.BYTES_PER_ELEMENT);
-             window.array = new {jstype}(
-                   pyodide._module.HEAPU8.buffer, buffer, 4);
-             window.array[0] = 1;
-             window.array[1] = 2;
-             window.array[2] = 3;
-             window.array[3] = 4;
-             """
-        )
-    assert selenium.run(
+def test_typed_arrays(selenium, jstype, pytype):
+    assert selenium.run_js(
         f"""
-         from js import array
-         import struct
-         expected = struct.pack("{pytype*4}", 1, 2, 3, 4)
-         print(array.format, array.tolist(), array.tobytes())
-         ((array.format == "{pytype}")
-          and array.tolist() == [1, 2, 3, 4]
-          and array.tobytes() == expected
-          and array.obj._has_bytes() is {not wasm_heap})
-         """
+        window.array = new {jstype}([1, 2, 3, 4]);
+        return pyodide.runPython(`
+            from js import array
+            array = array.to_py()
+            import struct
+            expected = struct.pack("{pytype*4}", 1, 2, 3, 4)
+            print(array.format, array.tolist(), array.tobytes())
+            # Result:
+            ((array.format == "{pytype}")
+            and array.tolist() == [1, 2, 3, 4]
+            and array.tobytes() == expected)
+        `);
+        """
     )
 
 
 def test_array_buffer(selenium):
-    selenium.run_js("window.array = new ArrayBuffer(100);\n")
     assert (
-        selenium.run(
+        selenium.run_js(
             """
-            from js import array
-            len(array.tobytes())
+            window.array = new ArrayBuffer(100);
+            return pyodide.runPython(`
+                from js import array
+                array = array.to_py()
+                len(array.tobytes())
+            `);
             """
         )
         == 100