Repository: pyodide/pyodide

Issue:
Configurable "import js"
Add a `pyodide.mountPackage("name", object)` api that mounts `object` at `name`, so that `import "name"` imports `object`, and `from name import field` imports `object.field` as `field`.

For example, the current `js` import would then be installed by saying `pyodide.mountPackage("js", globalThis)` though we would keep the current behavior as the default. We could get rid of `import js` by saying `pyodide.mountPackage("js", undefined)`.


Discussion:
sounds great.

BTW, have any way to bannd other import ? example about the `micropip` ?

Example, a bool config to enable or disable the `micropip` can or cannot be use ?
You can do that yourself by [using Emscripten FS](https://emscripten.org/docs/api_reference/Filesystem-API.html?highlight=fs#FS.unlink) to delete the `micropip` package from the dummy file system. The filesystem is found either under `pyodide.FS` at start of load or under `pyodide._module.FS` once load is completed.

If you want, you can open another issue requesting that feature.
@hoodmane Oh, thank you~ 

is the `pyodide` package can be delete same as that ?
`micropip` is a pure python package, so there isn't much point in "banning" it; users can simply re-implement it. Further, micropip is only useful for installing pure python packages, and again banning those won't do you much good either. Just by having python around you essentially have full access to the virtual filesystem to insert whatever you want, including compiled C code, by importing `os`.

Thus, I believe any sort of sandboxing along these lines ought to come from restricting along the JS-wasm boundary, and nothing within python itself.
`micropip is a pure python package, so there isn't much point in "banning" it;`

so, this means that i need bannd the HTTP api, not the package ?

so, where is the network API ?
> Thus, I believe any sort of sandboxing along these lines ought to come from restricting along the JS-wasm boundary, and nothing within python itself.

yes , you are right !!!


>so, this means that i need bannd the HTTP api, not the package ?
>
>so, where is the network API ?
 
I believe we use XMLHTTPRequest imported from JavaScript

> I believe we use XMLHTTPRequest imported from JavaScript

so, how to disable it ? example, make a config to disable the import ?
At the moment, we use XMLHTTPRequest via the js module, which currently allows arbitrary js execution. I think a reasonable approach would be to make our network requests use the emscripten fetch api, so that it can be used without the js module, then make the js module a configurable compile time option.

if change to `use the emscripten fetch api` , please let it become disable feature .
@dalcde I don't think it's necessary even to have the js module be a compile time option. Inside `pyodide.js` we can do `mountPackage("js", globalThis)` and then if you don't want that then you can edit `pyodide.js` to remove that. So you wouldn't have to build a separate wasm binary.
I don't think this would work from a security perspective. The functions needed to enable to JS module are already imported into wasm. We need to disable it during compile time so that emscripten does not do the import. Again, users can re-insert the files into the emscripten virtual fs.

@dalcde I thought the point was that the wasm can call any javascript function it has a reference to, but if you don't ever give it a reference then it can't call it?
That is true, but the "reference" is the literal string name of the imported function. When instantiating a wasm module, you supply an object consisting of imports the wasm module is allowed to access, and the wasm module uses the imports by referring to its name (see https://developer.mozilla.org/en-US/docs/WebAssembly/Understanding_the_text_format#Importing_functions_from_JavaScript )

Thus, any import we pass on to a wasm module will be usable, and the list of imports is determined by emscripten at compile time.
Have you checked [RistrictedPython](https://restrictedpython.readthedocs.io/en/latest/)? Its a pure python package and allows applying custom restrictions to the import function.
@oeway 

thanks for your notice ~

---

but you can read in there https://restrictedpython.readthedocs.io/en/latest/idea.html

> There should be additional preventive measures taken to ensure integrity of the application and the server itself, according to information security best practice and unrelated to RestrictedPython.

so, make a sandboxied runtime / limit avaiable import package are still nesssary. 

---

but the `RestrictedPython` seems like still be a good tools to help make a safe OJ runtime.
> but you can read in there
> https://restrictedpython.readthedocs.io/en/latest/idea.html
>
> There should be additional preventive measures taken to ensure integrity
> of the application and the server itself, according to information security
> best practice and unrelated to RestrictedPython.
>
> so, make a sandboxied runtime / limit avaiable import package are still
> nesssary.


Well, ensuring the integrity typically means you need to make
sure the python runtime , module or scripts are downloaded from
reliable source, with hash value verified etc., I don’t think they mean you
need additional sandbox mechanism.

However, they also mentioned:
> RestrictedPython is not a sandbox system or a secured environment, but it helps to define a trusted environment and execute untrusted code inside of it.

Since we have the browser itself as a sandbox system, maybe having this additional restriction layer will be enough for many such applications?

If you read here:
https://restrictedpython.readthedocs.io/en/latest/usage/policy.html#implementing-a-policy
they let you implement your own import policy to limit packages.
If I understand correctly, the goal of this issue is to provide some restrictions on accessing js, if we lift the restriction already in pyodide (either in the JS-wasm boundary or through `mountPackage("js", globalThis)`), many packages patched via js won't work (e.g. a web loop for asyncio, or a http client module that calls js.fetch). 

In my opinion, using `RestrictedPython` to create a restricted environment inside Python with a set of custom policy is much more flexible.
> Well, ensuring the integrity typically means you need to make sure the python runtime , module or scripts are downloaded from reliable source, with hash value verified .

but the code maybe come from a untrusted user.

please see [this ](https://github.com/iodide-project/pyodide/issues/959#issuecomment-751421344) , and [this](https://github.com/iodide-project/pyodide/issues/959#issuecomment-751424230) and [this](https://github.com/iodide-project/pyodide/issues/959#issuecomment-751429757) .

---

Python is a dynamic language, and it can be patch everywhere, no one understand where will be the next sandbox escape vulnerability .

so i think the configurable `import` or a configurable runtime is need to build a great sandbox , make something configurable in the JS-wasm boundary is a best choose . 

and i think that , if we have `mountPackage("js", globalThis)` and `unmountPackage("js")` , between two `runPython` call, the `many packages patched via js won't work (e.g. a web loop for asyncio, or a http client module that calls js.fetch).` maybe not be a issue. also, the `RestrictedPython` still a necessary tools to make the seconed guard  wall .
> but the code maybe come from a untrusted user.

Yes, but you can run untrusted code within RestrictedPython, as it advertised **it helps to define a trusted environment and execute untrusted code inside of it**. In other words, you don't need to do anything within the python runtime, and unstrusted code can run directly within RestrictedPython. It is designed to run untrusted code on the server-side (e.g. with zope which is a web app server) in a native Python environment (possibly without additional protection).

> 
> please see [this ](https://github.com/iodide-project/pyodide/issues/959#issuecomment-751421344) , and [this](https://github.com/iodide-project/pyodide/issues/959#issuecomment-751424230) and [this](https://github.com/iodide-project/pyodide/issues/959#issuecomment-751429757) .

I tried to read the thread, but I got confused as well. @dalcde thinks this might be useful when Pyodide is run in a server which totally make sense to me and I think we can then use RestrictedPython. My feeling is that RestrictedPython is a more thorough solution.

However, it seems you are trying to run Pyodide on the client-side which I don't think any of these will work-- as @hoodmane pointed out, an evil user can always open a browser console and run js code to remove any restrictions you applied. 

If you assume the user cannot access the browser console, then you can simply assume all the code can be executed by RestrictedPython where you can configure any import restrictions, including blacklist the js module and potentially much more, no?

>  i think that , if we have mountPackage("js", globalThis) and unmountPackage("js") , between two runPython call, the `many packages patched via js won't work (e.g. a web loop for asyncio, or a http client module that calls js.fetch). ` maybe not be a issue. also, the RestrictedPython still a necessary tools to make the seconed guard wall .

Unfortunately, many js patches will be applied during runtime, not all of them import everything when the package is loaded.
@oeway If you want to argue that it's impossible to do what @Lyoko-Jeremie is doing, can you please do it on the original thread #959?
(What @Lyoko-Jeremie is doing is code obfuscation, which is impossible in theory but possible to do well enough for many practical purposes. If people care to break it they will eventually do so though.)
@oeway The reason I suggested this feature is because I think it is easy to implement (looking at jsimport.c it's a very small change) and it adds useful flexibility. In particular, I'm less interested in unmounting `js` as @Lyoko-Jeremie wants to do, and more interested in being able to set up several different javascript implemented "packages".
> Thus, any import we pass on to a wasm module will be usable, and the list of imports is determined by emscripten at compile time.

This is impossible. It is possible to pass a wasm function a js callback defined at runtime. The reference to the js callback is some sort of pointer. The compiler can't know at compile time what arguments it will be called with at runtime.
> @oeway If you want to argue that it's impossible to do what @Lyoko-Jeremie is doing, can you please do it on the original thread #959?

OK, It's indeed related to that, but I meant for proposing using RestrictedPython instead of switching to the fetch api of emscripten and apply the restrictions to the js-wasm interface. 

> The reason I suggested this feature is because I think it is easy to implement (looking at jsimport.c it's a very small change) and it adds useful flexibility. In particular, I'm less interested in unmounting js as @Lyoko-Jeremie wants to do, and more interested in being able to set up several different javascript implemented "packages".

I see, if that's easy enough, totally make sense to me. Bing able to mountPackage js packages are certainly interesting to me, I am actually thinking along that, for example importing tensorflow.js. For that I will need to also patch the python to js and js to python data type conversion though.
  

> proposing using RestrictedPython

Still seems like it's more on topic in the original thread.

> I will need to also patch the python to js and js to python data type conversion though.

This sounds interesting. I opened another issue #964 to discuss this.
>This is impossible. It is possible to pass a wasm function a js callback defined at runtime. The reference to the js callback is some sort of pointer. The compiler can't know at compile time what arguments it will be called with at runtime.

Fair point. You can add functions to the table at runtime. However, the wasm imports used by the js module lets you access javascript globals, hence window.eval. Thus, we already have full access to the javascript runtime before we add any new functions at runtime.

> for example importing tensorflow.js. For that I will need to also patch the python to js and js to python data type conversion though.

about this, i have a similar idea in my brain , the official Python version OpenCV seems like dependence the OpenCV binary runtime, try to port it into pyodide maybe wast many proof. but we have a [OpenCV.js](https://docs.opencv.org/3.4/d5/d10/tutorial_js_root.html) , and now it now on dev, you can setup it [follow this](https://docs.opencv.org/3.4/d0/d84/tutorial_js_usage.html) , or [download opencv.js from there](https://docs.opencv.org/3.4.0/opencv.js) .

so , we can simple make the Python version OpenCV running in Brower use OpenCV.js and a simple Python Wrapper . 
> However, the wasm imports used by the js module lets you access javascript globals, hence window.eval. Thus, we already have full access to the javascript runtime before we add any new functions at runtime.

so we need a way to limit the `import js` or disable it like the topic title, right ?
I'm working on a PR for this.
I think as a first step we could make disabling js disable network requests too. We can later re-enable it by hand, using the fetch API or otherwise

> make disabling js disable network requests too

Is disabling network requests a compile time thing? What I'm doing is entirely at runtime.
network requests (specifically micropip) are done via the js module, so if the js module is gone that would break too

@oeway 

> Have you checked `RistrictedPython` ? Its a pure python package and allows applying custom restrictions to the import function.

now i try it, it always throw `__import__ not found` like follow, i guess that the pyodide have a custom __import__  impl that different from normal CPython , so the RistrictedPython patch not work as expected.

```
Traceback (most recent call last):
  File "<exec>", line 77, in importRestrictedPython
  File "code.RPY", line 42, in <module>
ImportError: __import__ not found
```




---

Edit:

i fixed it now, it work well !!! thank all the people who was help me.

@hoodmane 

> You can do that yourself by using Emscripten FS to delete the micropip package from the dummy file system. The filesystem is found either under pyodide.FS at start of load or under pyodide._module.FS once load is completed.


Hi, i find it in the path `pyodide._module.FS.root.contents.lib.contents['python3.8'].contents['site-packages'].contents` , can i direct delete it use `FS.rmdir('/lib/python3.8/site-packages/xxxxx')` ?

---

Edit:

seems like i can use `pyodide._module.FS.destroyNode(pyodide._module.FS.root.contents.lib.contents['python3.8'].contents['site-packages'].contents.numpy)` to delete a file dir
As a concrete example of my objection, I've added a JS2 package in my branch here: https://github.com/dalcde/pyodide/tree/js2 . It's source code is basically copied from `jsimport.c`, and it acts exactly the same as the existing `js`. If you build the branch, you can import `js2` and get the same functionality.

Now a python package is just a bunch of files in the virtual filesystem. Even if you hide the `js` package, a malicious user can still write the same bytes into the virtual fs using `open/write`, and now you have full javascript access. This will work as long as the relevant functions in hiwire and js2python exist and is compiled.
That said, I'm still a fan of #972

Resolving patch:
diff --git a/src/pyodide-py/pyodide/__init__.py b/src/pyodide-py/pyodide/__init__.py
--- a/src/pyodide-py/pyodide/__init__.py
+++ b/src/pyodide-py/pyodide/__init__.py
@@ -1,5 +1,12 @@
 from ._base import open_url, eval_code, find_imports, as_nested_list
 from ._core import JsException  # type: ignore
+from ._importhooks import JsFinder
+import sys
+
+jsfinder = JsFinder()
+register_js_module = jsfinder.register_js_module
+unregister_js_module = jsfinder.unregister_js_module
+sys.meta_path.append(jsfinder)  # type: ignore
 
 
 __version__ = "0.16.1"
@@ -10,4 +17,6 @@
     "find_imports",
     "as_nested_list",
     "JsException",
+    "register_js_module",
+    "unregister_js_module",
 ]
diff --git a/src/pyodide-py/pyodide/_importhooks.py b/src/pyodide-py/pyodide/_importhooks.py
new file mode 100644
--- /dev/null
+++ b/src/pyodide-py/pyodide/_importhooks.py
@@ -0,0 +1,96 @@
+from ._core import JsProxy
+from importlib.abc import MetaPathFinder, Loader
+from importlib.util import spec_from_loader
+import sys
+
+
+class JsFinder(MetaPathFinder):
+    def __init__(self):
+        self.jsproxies = {}
+
+    def find_spec(self, fullname, path, target=None):
+        [parent, _, child] = fullname.rpartition(".")
+        if parent:
+            parent_module = sys.modules[parent]
+            if not isinstance(parent_module, JsProxy):
+                # Not one of us.
+                return None
+            try:
+                jsproxy = getattr(parent_module, child)
+            except AttributeError:
+                raise ModuleNotFoundError(
+                    f"No module named {fullname!r}", name=fullname
+                ) from None
+            if not isinstance(jsproxy, JsProxy):
+                raise ModuleNotFoundError(
+                    f"No module named {fullname!r}", name=fullname
+                )
+        else:
+            try:
+                jsproxy = self.jsproxies[fullname]
+            except KeyError:
+                return None
+        loader = JsLoader(jsproxy)
+        return spec_from_loader(fullname, loader, origin="javascript")
+
+    def register_js_module(self, name, jsproxy):
+        """
+        Registers the Js object ``module`` as a Js module with ``name``. The module
+        can then be imported from Python using the standard Python import system.
+        If another module by the same name has already been imported, this won't
+        have much effect unless you also delete the imported module from
+        ``sys.modules``. This is called by the javascript API
+        ``pyodide.registerJsModule``.
+
+        Parameters
+        ----------
+        name : str
+            Name of js module
+        jsproxy : JsProxy
+            Javascript object backing the module
+        """
+        if not isinstance(name, str):
+            raise TypeError(
+                f"Argument 'name' must be a str, not {type(name).__name__!r}"
+            )
+        if not isinstance(jsproxy, JsProxy):
+            raise TypeError(
+                f"Argument 'jsproxy' must be a JsProxy, not {type(jsproxy).__name__!r}"
+            )
+        self.jsproxies[name] = jsproxy
+
+    def unregister_js_module(self, name):
+        """
+        Unregisters a Js module with given name that has been previously registered
+        with `js_api_pyodide_registerJsModule` or ``pyodide.register_js_module``. If
+        a Js module with that name does not already exist, will raise an error. Note
+        that if the module has already been imported, this won't have much effect
+        unless you also delete the imported module from ``sys.modules``. This is
+        called by the javascript API ``pyodide.unregisterJsModule``.
+
+        Parameters
+        ----------
+        name : str
+            Name of js module
+        """
+        try:
+            del self.jsproxies[name]
+        except KeyError:
+            raise ValueError(
+                f"Cannot unregister {name!r}: no javascript module with that name is registered"
+            ) from None
+
+
+class JsLoader(Loader):
+    def __init__(self, jsproxy):
+        self.jsproxy = jsproxy
+
+    def create_module(self, spec):
+        return self.jsproxy
+
+    def exec_module(self, module):
+        pass
+
+    # used by importlib.util.spec_from_loader
+    def is_package(self, fullname):
+        return True


Test patch:
diff --git a/src/tests/test_jsproxy.py b/src/tests/test_jsproxy.py
--- a/src/tests/test_jsproxy.py
+++ b/src/tests/test_jsproxy.py
@@ -471,3 +471,158 @@ def test_window_isnt_super_weird_anymore():
     assert js.window.Array == Array
     assert js.window.window.window.window == window
     assert window.window.window.window.Array == Array
+
+
+def test_mount_object(selenium):
+    result = selenium.run_js(
+        """
+        function x1(){
+            return "x1";
+        }
+        function x2(){
+            return "x2";
+        }
+        function y(){
+            return "y";
+        }
+        let a = { x : x1, y, s : 3, t : 7};
+        let b = { x : x2, y, u : 3, t : 7};
+        pyodide.registerJsModule("a", a);
+        pyodide.registerJsModule("b", b);
+        return pyodide.runPython(`
+            from a import x
+            from b import x as x2
+            result = [x(), x2()]
+            import a
+            import b
+            result += [a.s, dir(a), dir(b)]
+            result
+        `)
+        """
+    )
+    assert result[:3] == ["x1", "x2", 3]
+    assert set([x for x in result[3] if len(x) == 1]) == set(["x", "y", "s", "t"])
+    assert set([x for x in result[4] if len(x) == 1]) == set(["x", "y", "u", "t"])
+
+
+@pytest.mark.xfail
+def test_mount_map(selenium):
+    result = selenium.run_js(
+        """
+        function x1(){
+            return "x1";
+        }
+        function x2(){
+            return "x2";
+        }
+        function y(){
+            return "y";
+        }
+        let a = new Map(Object.entries({ x : x1, y, s : 3, t : 7}));
+        let b = new Map(Object.entries({ x : x2, y, u : 3, t : 7}));
+        pyodide.registerJsModule("a", a);
+        pyodide.registerJsModule("b", b);
+        return pyodide.runPython(`
+            from a import x
+            from b import x as x2
+            result = [x(), x2()]
+            import a
+            import b
+            result += [a.s, dir(a), dir(b)]
+            import sys
+            del sys.modules["a"]
+            del sys.modules["b"]
+            result
+        `)
+        """
+    )
+    assert result[:3] == ["x1", "x2", 3]
+    # fmt: off
+    assert set(result[3]).issuperset(
+        [
+            "x", "y", "s", "t",
+            "__dir__", "__doc__", "__getattr__", "__loader__",
+            "__name__", "__package__", "__spec__",
+            "jsproxy",
+        ]
+    )
+    # fmt: on
+    assert set(result[4]).issuperset(["x", "y", "u", "t", "jsproxy"])
+
+
+def test_unregister_jsmodule(selenium):
+    selenium.run_js(
+        """
+        let a = new Map(Object.entries({ s : 7 }));
+        let b = new Map(Object.entries({ t : 3 }));
+        pyodide.registerJsModule("a", a);
+        pyodide.registerJsModule("a", b);
+        pyodide.unregisterJsModule("a")
+        pyodide.runPython(`
+            try:
+                import a
+                assert False
+            except ImportError:
+                pass
+        `)
+        """
+    )
+
+
+def test_unregister_jsmodule_error(selenium):
+    selenium.run_js(
+        """
+        try {
+            pyodide.unregisterJsModule("doesnotexist");
+            throw new Error("unregisterJsModule should have thrown an error.");
+        } catch(e){
+            if(!e.message.includes("Cannot unregister 'doesnotexist': no javascript module with that name is registered")){
+                throw e;
+            }
+        }
+        """
+    )
+
+
+def test_nested_import(selenium):
+    assert (
+        selenium.run_js(
+            """
+            window.a = { b : { c : { d : 2 } } };
+            return pyodide.runPython("from js.a.b import c; c.d");
+            """
+        )
+        == 2
+    )
+
+
+def test_register_jsmodule_docs_example(selenium):
+    selenium.run_js(
+        """
+        let my_module = {
+        f : function(x){
+            return x*x + 1;
+        },
+        g : function(x){
+            console.log(`Calling g on argument ${x}`);
+            return x;
+        },
+        submodule : {
+            h : function(x) {
+            return x*x - 1;
+            },
+            c  : 2,
+        },  
+        };
+        pyodide.registerJsModule("my_js_module", my_module);
+        """
+    )
+    selenium.run(
+        """
+        import my_js_module
+        from my_js_module.submodule import h, c
+        assert my_js_module.f(7) == 50
+        assert h(9) == 80
+        assert c == 2
+        """
+    )