Repository: pyodide/pyodide

Issue:
Remove or change toJs behavior for buffers?
If we create a PyProxy of a bytes object and then run the `toJs` method on the `PyProxy`, the resulting `TypedArray` allows the user to modify the `bytes` object even though it's read only and might be shared with a bunch of other places. It also leads to potential use after free errors, and can violate expectations because in every other case `toJs` either copies stuff or returns a `PyProxy` (the other biggest problem with `toJs` is that it creates an unpredictable number of `PyProxy` objects that must be located and freed in order to avoid a leak).

Perhaps we should convert `toJs` to exclusively copy the buffer into a Javascript array, so that `toJs` is slow but safe and returns easy to use buffers whereas `getBuffer` is fast and unsafe but only does the one thing so anyone who reads the docs will see the warnings about modifying readonly buffers.

problem with sending bytes to javascript
Is there any restriction on sending bytes from python to javascript?

```javascript
pyodide.runPython('bytes([10])')
> Uint8ClampedArray [10]
pyodide.runPython('bytes([10, 11])')
> Uint8ClampedArray(2) [24, 0]
pyodide.runPython('bytes([10, 11, 12])')
> Uint8ClampedArray(3) [24, 0, 0]
pyodide.runPython('bytes([10, 11, 12, 13])')
> Uint8ClampedArray(4) [10, 11, 12, 13]

pyodide.runPython('b"A"')
> Uint8ClampedArray [65]
pyodide.runPython('b"AB"')
> Uint8ClampedArray(2) [24, 0]
pyodide.runPython('b"ABC"')
> Uint8ClampedArray(3) [24, 0, 0]
pyodide.runPython('b"ABCD"')
> Uint8ClampedArray(4) [65, 66, 67, 68]
```

What's `[24, 0]`?

It's different for different lengths:

```javascript
pyodide.runPython("b'0' * 4")
Uint8ClampedArray(4) [48, 48, 48, 48]
pyodide.runPython("b'0' * 100")
Uint8ClampedArray(100) [48, 48, 48, 48, 48, 48, 48, 48, 48, 48, ...
pyodide.runPython("b'0' * 200")
Uint8ClampedArray(200) [48, 48, 48, 48, 48, 48, 48, 48, 48, 48, ...
pyodide.runPython("b'0' * 300")
Uint8ClampedArray(300) [88, 51, 157, 0, 0, 0, 0, 0, 48, 48, 48, ...
```

It's `[88, 51, 157, 0, 0, 0, 0, 0,` this time.

Seems that there is no problem from javascript to python:

```javascript
f = pyodide.runPython(`
def f(x):
    print([x, x.tobytes()])
    return x
f
`)
> Proxy {$$: {…}, length: 0, name: "target", arguments: null, caller: null, …}
f(new Uint8Array([0]))
> pyodide.asm.js:8 [<memory at 0xa7d308>, b'\x00']
> Uint8Array [0]
f(new Uint8Array([0, 1]))
> pyodide.asm.js:8 [<memory at 0xa7d308>, b'\x00\x01']
> Uint8Array(2) [0, 1]
f(new Uint8Array([0, 1, 2]))
> pyodide.asm.js:8 [<memory at 0xa7d308>, b'\x00\x01\x02']
> Uint8Array(3) [0, 1, 2]
f(new Uint8Array([0, 1, 2, 3]))
> pyodide.asm.js:8 [<memory at 0xa7d308>, b'\x00\x01\x02\x03']
> Uint8Array(4) [0, 1, 2, 3]

```

I tested on this version:

```
pyodide.version(): "0.15.0"
pyodide.runPython('sys.version'): "3.7.4 (default, May 20 2020, 19:25:33) [Clang 6.0.1 ]"
Chrome 83
Firefox 76
```



Discussion:
I think in any case we need a simple way of passing bytes to JS without hacking with the buffer protocol API.

> Perhaps we should convert toJs to exclusively copy the buffer into a Javascript array, so that toJs is slow but safe

Sounds good to me. Generally a single memory copy shouldn't be that bad performance wise. There are plenty copies as it is in Python / scientific python code anyway. 
Okay I'll open a PR for this soon. One question: for outer level arrays we have to use Javascript `Array`, but maybe for the innermost layer we should stick to using a `TypedArray`? 

Currently `python2js_buffer` has three paths: `NOT_SHAREABLE`, `CONTIGUOUS`, and `NOT_CONTIGUOUS`. I think I could just delete the `CONTIGUOUS` and `NOT_CONTIGUOUS` branches and keep the `NOT_SHAREABLE` branch as-is.
Thanks for the report! I can reproduce. For instance `b"AB"` produces indeed,
  - `Uint8ClampedArray [ 24, 0 ]` with v0.15.0 (Python 3.7) 
  - `Uint8ClampedArray [ 0, 8 ]` on master (Python 3.8) 

That doesn't look right. That pattern depending on the input length is also quite strange. There is a test that checks this conversion [here](https://github.com/iodide-project/pyodide/blob/e7dde959b8a6666a257f4d04e0a828c038b7416c/test/test_python.py#L35). It would be good to add another one with your `b"AB"` example.

The conversion should go though [_python2js_bytes](https://github.com/iodide-project/pyodide/blob/e7dde959b8a6666a257f4d04e0a828c038b7416c/src/python2js.c#L153) and then [hiwire_bytes](https://github.com/iodide-project/pyodide/blob/76efcaccf7cb0f478ed1f17fd36cd79f3370980e/src/hiwire.c#L87). I don't see any evident issues that could produce this result there, but it would be certainly worth investigating in more detail. 
Assigning the value to a variable changes this behavior. Not a fix, but may help to find the issue.

```javascript
pyodide.runPython('b"ABC"')
Uint8ClampedArray(3) [24, 0, 0]
pyodide.runPython('x = b"ABC"; x')
Uint8ClampedArray(3) [65, 66, 67]
```
I hit the same bug and tried couple of things.
I assigned the result into Javascript variable and run other Python scripts. The content of the `Uint8ClampedArray` changed. 

I believe it may be related to garbage collector. When it is not assigned to a variable, Python reclaims its memory.   

Resolving patch:
diff --git a/docs/sphinx_pyodide/sphinx_pyodide/jsdoc.py b/docs/sphinx_pyodide/sphinx_pyodide/jsdoc.py
--- a/docs/sphinx_pyodide/sphinx_pyodide/jsdoc.py
+++ b/docs/sphinx_pyodide/sphinx_pyodide/jsdoc.py
@@ -211,7 +211,7 @@ def get_summary_row(self, pkgname, obj):
 
             The output is designed to be input to format_table. The link name
             needs to be set up so that :any:`link_name` makes a link to the
-            actual api docs for this object.
+            actual API docs for this object.
             """
             sig = self.get_sig(obj)
             display_name = obj.name
diff --git a/src/pyodide-py/_pyodide/_core.py b/src/pyodide-py/_pyodide/_core.py
--- a/src/pyodide-py/_pyodide/_core.py
+++ b/src/pyodide-py/_pyodide/_core.py
@@ -48,7 +48,7 @@ def to_py(self) -> Any:
             pass
 
         def then(self, onfulfilled: Callable, onrejected: Callable) -> "Promise":
-            """The ``Promise.then`` api, wrapped to manage the lifetimes of the
+            """The ``Promise.then`` API, wrapped to manage the lifetimes of the
             handlers.
 
             Only available if the wrapped Javascript object has a "then" method.
@@ -57,7 +57,7 @@ def then(self, onfulfilled: Callable, onrejected: Callable) -> "Promise":
             """
 
         def catch(self, onrejected: Callable) -> "Promise":
-            """The ``Promise.catch`` api, wrapped to manage the lifetimes of the
+            """The ``Promise.catch`` API, wrapped to manage the lifetimes of the
             handler.
 
             Only available if the wrapped Javascript object has a "then" method.
@@ -66,7 +66,7 @@ def catch(self, onrejected: Callable) -> "Promise":
             """
 
         def finally_(self, onfinally: Callable) -> "Promise":
-            """The ``Promise.finally`` api, wrapped to manage the lifetimes of
+            """The ``Promise.finally`` API, wrapped to manage the lifetimes of
             the handler.
 
             Only available if the wrapped Javascript object has a "then" method.


Test patch:
diff --git a/packages/numpy/test_numpy.py b/packages/numpy/test_numpy.py
--- a/packages/numpy/test_numpy.py
+++ b/packages/numpy/test_numpy.py
@@ -40,8 +40,23 @@ def test_typed_arrays(selenium):
         )
 
 
-def test_python2js_numpy_dtype(selenium_standalone):
-    selenium = selenium_standalone
+@pytest.mark.parametrize("order", ("C", "F"))
+@pytest.mark.parametrize(
+    "dtype",
+    (
+        "int8",
+        "uint8",
+        "int16",
+        "uint16",
+        "int32",
+        "uint32",
+        "int64",
+        "uint64",
+        "float32",
+        "float64",
+    ),
+)
+def test_python2js_numpy_dtype(selenium, order, dtype):
 
     selenium.load_package("numpy")
     selenium.run("import numpy as np")
@@ -56,59 +71,37 @@ def assert_equal():
                 for k in range(2):
                     assert (
                         selenium.run_js(
-                            f"return pyodide.globals.get('x').toJs()[{i}][{j}][{k}]"
+                            f"return Number(pyodide.globals.get('x').toJs()[{i}][{j}][{k}])"
                         )
                         == expected_result[i][j][k]
                     )
 
-    for order in ("C", "F"):
-        for dtype in (
-            "int8",
-            "uint8",
-            "int16",
-            "uint16",
-            "int32",
-            "uint32",
-            "int64",
-            "uint64",
-            "float32",
-            "float64",
-        ):
-            selenium.run(
-                f"""
-                x = np.arange(8, dtype=np.{dtype})
-                x = x.reshape((2, 2, 2))
-                x = x.copy({order!r})
-                """
-            )
-            assert_equal()
-            classname = selenium.run_js(
-                "return pyodide.globals.get('x').toJs()[0][0].constructor.name"
-            )
-            if order == "C" and dtype not in ("uint64", "int64"):
-                # Here we expect a TypedArray subclass, such as Uint8Array, but
-                # not a plain-old Array
-                assert classname.endswith("Array")
-                assert classname != "Array"
-            else:
-                assert classname == "Array"
-            selenium.run(
-                """
-                x = x.byteswap().newbyteorder()
-                """
-            )
-            assert_equal()
-            classname = selenium.run_js(
-                "return pyodide.globals.get('x').toJs()[0][0].constructor.name"
-            )
-            if order == "C" and dtype in ("int8", "uint8"):
-                # Here we expect a TypedArray subclass, such as Uint8Array, but
-                # not a plain-old Array -- but only for single byte types where
-                # endianness doesn't matter
-                assert classname.endswith("Array")
-                assert classname != "Array"
-            else:
-                assert classname == "Array"
+    selenium.run(
+        f"""
+        x = np.arange(8, dtype=np.{dtype})
+        x = x.reshape((2, 2, 2))
+        x = x.copy({order!r})
+        """
+    )
+    assert_equal()
+    classname = selenium.run_js(
+        "return pyodide.globals.get('x').toJs()[0][0].constructor.name"
+    )
+    # We expect a TypedArray subclass, such as Uint8Array, but not a plain-old
+    # Array
+    assert classname.endswith("Array")
+    assert classname != "Array"
+    selenium.run(
+        """
+        x = x.byteswap().newbyteorder()
+        """
+    )
+    assert_equal()
+    classname = selenium.run_js(
+        "return pyodide.globals.get('x').toJs()[0][0].constructor.name"
+    )
+    assert classname.endswith("Array")
+    assert classname != "Array"
 
     assert selenium.run("np.array([True, False])") == [True, False]
 
@@ -126,13 +119,9 @@ def test_py2js_buffer_clear_error_flag(selenium):
     )
 
 
-def test_python2js_numpy_scalar(selenium_standalone):
-    selenium = selenium_standalone
-
-    selenium.load_package("numpy")
-    selenium.run("import numpy as np")
-
-    for dtype in (
+@pytest.mark.parametrize(
+    "dtype",
+    (
         "int8",
         "uint8",
         "int16",
@@ -143,33 +132,38 @@ def test_python2js_numpy_scalar(selenium_standalone):
         "uint64",
         "float32",
         "float64",
-    ):
-        selenium.run(
-            f"""
-            x = np.{dtype}(1)
-            """
-        )
-        assert (
-            selenium.run_js(
-                """
-            return pyodide.globals.get('x') == 1
-            """
-            )
-            is True
-        )
-        selenium.run(
-            """
-            x = x.byteswap().newbyteorder()
+    ),
+)
+def test_python2js_numpy_scalar(selenium, dtype):
+
+    selenium.load_package("numpy")
+    selenium.run("import numpy as np")
+    selenium.run(
+        f"""
+        x = np.{dtype}(1)
+        """
+    )
+    assert (
+        selenium.run_js(
             """
+        return pyodide.globals.get('x') == 1
+        """
         )
-        assert (
-            selenium.run_js(
-                """
-            return pyodide.globals.get('x') == 1
+        is True
+    )
+    selenium.run(
+        """
+        x = x.byteswap().newbyteorder()
+        """
+    )
+    assert (
+        selenium.run_js(
             """
-            )
-            is True
+        return pyodide.globals.get('x') == 1
+        """
         )
+        is True
+    )
 
 
 def test_runpythonasync_numpy(selenium_standalone):