Repository: pyodide/pyodide
Pull request: ENH Add micropip support for loading wheels from emfs
Resolves #2731.

### Checklists

- [x] Add a [CHANGELOG](https://github.com/pyodide/pyodide/blob/main/docs/project/changelog.md) entry
- [x] Add / update tests
- [x] Add new / update outdated documentation


Problem statement:
Micropip should be able to install wheels from the file system
## 🚀 Feature

Micropip should be able to install a wheel which is in the emscripten file system from a path.

### Motivation

In node (and some day maybe in chrome), we can mount the local file system into the Emscripten file system. It would be useful to install these, for instance when testing whether a wheel was built correctly.

Hints from issue discussion:
Posting here for visibility.

The latest JupyterLite release adds support for accessing files from the Python (Pyodide-based) kernel: 

- https://github.com/jupyterlite/jupyterlite/releases/tag/v0.1.0b9
- https://github.com/jupyterlite/jupyterlite/pull/655

While the mounting logic is a bit specific to JupyterLite / JupyterLab and enabled via a `ServiceWorker`, making `micropip` able to install packages from a local path would still be useful.

For example one could upload a wheel (or ship it as part of the default content) and install it with:

```py
import micropip
await micropip.install('./snowballstemmer-2.2.0-py2.py3-none-any.whl')
```

![image](https://user-images.githubusercontent.com/591645/174308580-85a6303a-ad42-46da-84cc-725ddeaebe94.png)
Maybe it can be `micropip.install_file`? `install_from_filesystem`?
Probably from a user point of view keeping the same `micropip.install()` would be more convenient? (like regular `pip`)
We currently interpret `micropip.install("a/b/c.whl")` as a relative URL, so there is no syntax space for that. Unless you want to do `micropip.install(from_path="a/b/c.whl")`?
What about intercepting `file://` before fetch as meaning _file in the WASM machine_? I'd think pyodide itself could not be usefully loaded when "hosted" from `file://` because `.json`, much less `.wasm`, would not be served properly. Further no _other_ protocol would be able to resolve it.
What about relative paths though?
Well, URIs don't _necessarily_ have to start with `<protocol>:/`: there's certainly precedent for non-slashed paths, e.g. python's [sqlite3]() module, which inherit's [sqlite's implementation](https://www.sqlite.org/uri.html):

```py
# Open a database in read-only mode.
con = sqlite3.connect("file:template.db?mode=ro", uri=True)
```

So, with the default settings, `file:some-package.whl` could be interpreted the same as `file:/home/pyodide/some-package.whl`. 

Indeed, allowing `<protocol>` to be extended is a _whacking_ good idea: `ipfs://` springs to mind as an _almost perfect_ tool for peer-to-peer package distribution.
> What about intercepting file:// before fetch as meaning file in the WASM machine?

The problem is that when run in Node.js, `file://`  [means host file system](https://nodejs.org/api/fs.html#file-url-paths). Though for Python `file://` would probably still mean the whatever FS Python is installed in. We would want to keep a consistent behavior in the browser and Node. Maybe `memfs://` at least as far as micropip is concerned?
hey guys! awesome to see this being on the table so recently
got here while searching for a way to use my own packages in pyscript
for example i've got a package called xo-gd, which i want to use, but i cant seem to import it

seems like this is a requested feature not yet available seemlessly,
does anyone know of a way for me to do this? do i need to build pyodide myself with this package? or should i just copy all the python files from my package locally ? (it's already all on pip, so i thought it should be straight-forward)

any info would be appreciated! thx & have a good one! 
@fire17 Would you mind opening a separate issue about it? It's a bit orthogonal to the current discussion. The general documentation for adding a package can be found [here](https://pyodide.org/en/stable/development/new-packages.html). Looking at the xo-gd description, it looks like a lof of the functionality it exposes will not be supported in the browser (let's discuss the details in a separate issue).

Commit message: Update docs

Patch to review:
diff --git a/docs/project/changelog.md b/docs/project/changelog.md
index 55bd718803b..87c60e6748b 100644
--- a/docs/project/changelog.md
+++ b/docs/project/changelog.md
@@ -16,6 +16,11 @@ substitutions:
 
 - {{ Fix }} `micropip` supports extra markers in packages correctly now.
   {pr}`2584`
+
+- {{ Enhancement }} `micropip` supports loading wheels from the Emscripten file
+  system using the `emfs:` protocol now.
+  {pr}`2767`
+
 - {{ Enhancement }} Integrity of Pyodide packages are now verified before
   loading them. This is for now only limited to browser environments.
   {pr}`2513`
diff --git a/packages/micropip/src/micropip/_micropip.py b/packages/micropip/src/micropip/_micropip.py
index 170c452fc60..3d731041326 100644
--- a/packages/micropip/src/micropip/_micropip.py
+++ b/packages/micropip/src/micropip/_micropip.py
@@ -8,9 +8,10 @@
 from importlib.metadata import PackageNotFoundError
 from importlib.metadata import distributions as importlib_distributions
 from importlib.metadata import version as importlib_version
-from io import BytesIO
+from io import BufferedReader, BytesIO
 from pathlib import Path
 from typing import Any
+from urllib.parse import ParseResult, urlparse
 from zipfile import ZipFile
 
 from packaging.markers import default_environment
@@ -54,9 +55,10 @@ class WheelInfo:
     build: tuple[int, str] | tuple[()]
     tags: frozenset[Tag]
     url: str
+    parsed_url: ParseResult
     project_name: str | None = None
     digests: dict[str, str] | None = None
-    data: BytesIO | None = None
+    data: BytesIO | BufferedReader | None = None
     _dist: Any = None
     dist_info: Path | None = None
     _requires: list[Requirement] | None = None
@@ -67,7 +69,8 @@ def from_url(url: str) -> "WheelInfo":
 
         See https://www.python.org/dev/peps/pep-0427/#file-name-convention
         """
-        file_name = Path(url).name
+        parsed_url = urlparse(url)
+        file_name = Path(parsed_url.path).name
         name, version, build, tags = parse_wheel_filename(file_name)
         return WheelInfo(
             name=name,
@@ -76,6 +79,7 @@ def from_url(url: str) -> "WheelInfo":
             build=build,
             tags=tags,
             url=url,
+            parsed_url=parsed_url,
         )
 
     def is_compatible(self):
@@ -86,23 +90,28 @@ def is_compatible(self):
                 return True
         return False
 
-    async def download(self, fetch_kwargs):
+    async def _fetch_bytes(self, fetch_kwargs):
         try:
-            wheel_bytes = await fetch_bytes(self.url, fetch_kwargs)
+            return await fetch_bytes(self.url, fetch_kwargs)
         except OSError as e:
-            if self.url.startswith("https://files.pythonhosted.org/"):
+            if self.parsed_url.hostname == "files.pythonhosted.org":
                 raise e
             else:
                 raise ValueError(
-                    f"Can't fetch wheel from '{self.url}'."
+                    f"Can't fetch wheel from '{self.url}'. "
                     "One common reason for this is when the server blocks "
-                    "Cross-Origin Resource Sharing (CORS)."
+                    "Cross-Origin Resource Sharing (CORS). "
                     "Check if the server is sending the correct 'Access-Control-Allow-Origin' header."
                 ) from e
 
-        self.data = BytesIO(wheel_bytes)
-
-        with ZipFile(self.data) as zip_file:
+    async def download(self, fetch_kwargs):
+        data: BytesIO | BufferedReader
+        if self.parsed_url.scheme == "emfs":
+            data = open(self.parsed_url.path, "rb")
+        else:
+            data = BytesIO(await self._fetch_bytes(fetch_kwargs))
+        self.data = data
+        with ZipFile(data) as zip_file:
             self._dist = pkg_resources_distribution_for_wheel(
                 zip_file, self.name, "???"
             )
@@ -116,9 +125,10 @@ def validate(self):
             # No checksums available, e.g. because installing
             # from a different location than PyPI.
             return
+        if not isinstance(self.data, BytesIO):
+            return
         sha256 = self.digests["sha256"]
         m = hashlib.sha256()
-        assert self.data
         m.update(self.data.getvalue())
         if m.hexdigest() != sha256:
             raise ValueError("Contents don't match hash")
@@ -412,13 +422,23 @@ async def install(
         A requirement or list of requirements to install. Each requirement is a
         string, which should be either a package name or URL to a wheel:
 
-        - If the requirement ends in ``.whl`` it will be interpreted as a URL.
-          The file must be a wheel named in compliance with the
-          `PEP 427 naming convention <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
+        - If the requirement ends in ``.whl``, the part of the requirement after
+          the last ``/``  must be a valid wheel name in compliance with the `PEP
+          427 naming convention.
+
+        - If the requirement ends in ``.whl`` and starts with ``emfs:``, it will
+          be interpreted as a path in the Emscripten file system (Pyodide's file
+          system). E.g., `emfs:../relative/path/wheel.whl` or
+          `emfs:/absolute/path/wheel.whl`.
+
+        - If the requirement ends in ``.whl`` and does not start with ``emfs:``
+          it will be interpreted as a URL. The file must be a wheel named in
+          compliance with the `PEP 427 naming convention
+          <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
 
-        - If the requirement does not end in ``.whl``, it will interpreted as the
-          name of a package. A package by this name must either be present in the
-          Pyodide repository at :any:`indexURL <globalThis.loadPyodide>` or on PyPI
+        - If the requirement does not end in ``.whl``, it will interpreted as
+          the name of a package. A package with this name must either be present
+          in the Pyodide lock file or on PyPI.
 
     keep_going : ``bool``, default: False
 
@@ -426,28 +446,29 @@ async def install(
         Python package without a pure Python wheel while doing dependency
         resolution:
 
-        - If ``False``, an error will be raised on first package with a missing wheel.
+        - If ``False``, an error will be raised on first package with a missing
+          wheel.
 
-        - If ``True``, the micropip will keep going after the first error, and report a list
-          of errors at the end.
+        - If ``True``, the micropip will keep going after the first error, and
+          report a list of errors at the end.
 
     deps : ``bool``, default: True
 
-        If ``True``, install dependencies specified in METADATA file for
-        each package. Otherwise do not install dependencies.
+        If ``True``, install dependencies specified in METADATA file for each
+        package. Otherwise do not install dependencies.
 
     credentials : ``Optional[str]``
 
         This parameter specifies the value of ``credentials`` when calling the
-        `fetch() <https://developer.mozilla.org/en-US/docs/Web/API/fetch>`__ function
-        which is used to download the package.
+        `fetch() <https://developer.mozilla.org/en-US/docs/Web/API/fetch>`__
+        function which is used to download the package.
 
         When not specified, ``fetch()`` is called without ``credentials``.
 
     pre : ``bool``, default: False
 
-        If ``True``, include pre-release and development versions.
-        By default, micropip only finds stable versions.
+        If ``True``, include pre-release and development versions. By default,
+        micropip only finds stable versions.
 
     Returns
     -------
@@ -511,7 +532,7 @@ async def install(
     await gather(*wheel_promises)
 
 
-def _generate_package_hash(data: BytesIO) -> str:
+def _generate_package_hash(data: BytesIO | BufferedReader) -> str:
     sha256_hash = hashlib.sha256()
     data.seek(0)
     while chunk := data.read(4096):
diff --git a/packages/micropip/test_micropip.py b/packages/micropip/test_micropip.py
index ee7aa02098e..e79bf661df3 100644
--- a/packages/micropip/test_micropip.py
+++ b/packages/micropip/test_micropip.py
@@ -18,6 +18,7 @@
 except ImportError:
     pass
 
+from conftest import DIST_PATH
 from pyodide_build import common
 
 
@@ -53,6 +54,11 @@ def wheel_base(monkeypatch):
             WHEEL_BASE = None
 
 
+@pytest.fixture
+def regex_wheel_name():
+    return list(DIST_PATH.glob("regex*.whl"))[0].name
+
+
 def _mock_importlib_distributions():
     return (Distribution.at(p) for p in WHEEL_BASE.glob("*.dist-info"))  # type: ignore[union-attr]
 
@@ -215,27 +221,48 @@ def test_install_simple(selenium_standalone_micropip):
     )
 
 
-def test_parse_wheel_url():
+SNOWBALL_WHEEL = "snowballstemmer-2.0.0-py2.py3-none-any.whl"
+
+
+@pytest.mark.parametrize(
+    "path",
+    [
+        SNOWBALL_WHEEL,
+        f"/{SNOWBALL_WHEEL}" f"a/{SNOWBALL_WHEEL}",
+        f"/a/{SNOWBALL_WHEEL}",
+        f"//a/{SNOWBALL_WHEEL}",
+    ],
+)
+@pytest.mark.parametrize("protocol", ["https:", "file:", "emfs:", ""])
+def test_parse_wheel_url1(protocol, path):
     pytest.importorskip("packaging")
     from micropip._micropip import WheelInfo
 
-    url = "https://a/snowballstemmer-2.0.0-py2.py3-none-any.whl"
+    url = protocol + path
     wheel = WheelInfo.from_url(url)
     assert wheel.name == "snowballstemmer"
     assert str(wheel.version) == "2.0.0"
     assert wheel.digests is None
-    assert wheel.filename == "snowballstemmer-2.0.0-py2.py3-none-any.whl"
+    assert wheel.filename == SNOWBALL_WHEEL
     assert wheel.url == url
     assert wheel.tags == frozenset(
         {Tag("py2", "none", "any"), Tag("py3", "none", "any")}
     )
 
+
+def test_parse_wheel_url2():
+    from micropip._micropip import WheelInfo
+
     msg = r"Invalid wheel filename \(wrong number of parts\)"
     with pytest.raises(ValueError, match=msg):
         url = "https://a/snowballstemmer-2.0.0-py2.whl"
-        wheel = WheelInfo.from_url(url)
+        WheelInfo.from_url(url)
 
-    url = "http://scikit_learn-0.22.2.post1-cp35-cp35m-macosx_10_9_intel.whl"
+
+def test_parse_wheel_url3():
+    from micropip._micropip import WheelInfo
+
+    url = "http://a/scikit_learn-0.22.2.post1-cp35-cp35m-macosx_10_9_intel.whl"
     wheel = WheelInfo.from_url(url)
     assert wheel.name == "scikit-learn"
     assert wheel.tags == frozenset({Tag("cp35", "cp35m", "macosx_10_9_intel")})
@@ -416,7 +443,7 @@ async def test_install_non_pure_python_wheel():
 
     msg = "not a pure Python 3 wheel"
     with pytest.raises(ValueError, match=msg):
-        url = "http://scikit_learn-0.22.2.post1-cp35-cp35m-macosx_10_9_intel.whl"
+        url = "http://a/scikit_learn-0.22.2.post1-cp35-cp35m-macosx_10_9_intel.whl"
         transaction = create_transaction(Transaction)
         await transaction.add_requirement(url)
 
@@ -748,3 +775,13 @@ async def test_freeze(mock_fetch: mock_fetch_cls) -> None:
     assert pkg_metadata["imports"] == toplevel[0]
     assert dep1_metadata["imports"] == toplevel[1]
     assert dep2_metadata["imports"] == toplevel[2]
+
+
+@run_in_pyodide(packages=["micropip"])
+async def test_emfs_micropip(selenium_standalone_micropip, regex_wheel_name):
+    import micropip
+    from pyodide.http import pyfetch
+
+    resp = await pyfetch(regex_wheel_name)
+    await resp._into_file(open(regex_wheel_name, "wb"))
+    await micropip.install("emfs:" + regex_wheel_name)


Code review comments:
diff_hunk: @@ -412,42 +422,53 @@ async def install(
         A requirement or list of requirements to install. Each requirement is a
         string, which should be either a package name or URL to a wheel:
 
-        - If the requirement ends in ``.whl`` it will be interpreted as a URL.
-          The file must be a wheel named in compliance with the
-          `PEP 427 naming convention <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
+        - If the requirement ends in ``.whl``, the part of the requirement after
+          the last ``/``  must be a valid wheel name in compliance with the `PEP
+          427 naming convention.
+
+        - If the requirement ends in ``.whl`` and starts with ``emfs:``, it will
+          be interpreted as a path in the Emscripten file system (Pyodide's file
+          system). E.g., `emfs:../relative/path/wheel.whl` or
+          `emfs:/absolute/path/wheel.whl`.
+
+        - If the requirement ends in ``.whl`` and does not start with ``emfs:``
+          it will be interpreted as a URL. The file must be a wheel named in
+          compliance with the `PEP 427 naming convention
+          <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
 
-        - If the requirement does not end in ``.whl``, it will interpreted as the
-          name of a package. A package by this name must either be present in the
-          Pyodide repository at :any:`indexURL <globalThis.loadPyodide>` or on PyPI
+        - If the requirement does not end in ``.whl``, it will interpreted as
+          the name of a package. A package with this name must either be present
+          in the Pyodide lock file or on PyPI.
path: packages/micropip/src/micropip/_micropip.py
text: @user1:
```suggestion
        - Otherwise, it will be interpreted as
          the name of a package. A package with this name must either be present
          in the Pyodide lock file or on PyPI.
```

diff_hunk: @@ -412,42 +422,53 @@ async def install(
         A requirement or list of requirements to install. Each requirement is a
         string, which should be either a package name or URL to a wheel:
 
-        - If the requirement ends in ``.whl`` it will be interpreted as a URL.
-          The file must be a wheel named in compliance with the
-          `PEP 427 naming convention <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
+        - If the requirement ends in ``.whl``, the part of the requirement after
+          the last ``/``  must be a valid wheel name in compliance with the `PEP
+          427 naming convention.
+
+        - If the requirement ends in ``.whl`` and starts with ``emfs:``, it will
+          be interpreted as a path in the Emscripten file system (Pyodide's file
+          system). E.g., `emfs:../relative/path/wheel.whl` or
+          `emfs:/absolute/path/wheel.whl`.
path: packages/micropip/src/micropip/_micropip.py
text: @user1:
Trying to simplify the logic a bit.

```suggestion
        - If the requirement starts with ``emfs:``, it will
          be interpreted as a path in the Emscripten file system (Pyodide's file
          system). E.g., `emfs:../relative/path/wheel.whl` or
          `emfs:/absolute/path/wheel.whl`. In this case, only .whl files are supported.
```

diff_hunk: @@ -748,3 +775,13 @@ async def test_freeze(mock_fetch: mock_fetch_cls) -> None:
     assert pkg_metadata["imports"] == toplevel[0]
     assert dep1_metadata["imports"] == toplevel[1]
     assert dep2_metadata["imports"] == toplevel[2]
+
+
+@run_in_pyodide(packages=["micropip"])
+async def test_emfs_micropip(selenium_standalone_micropip, regex_wheel_name):
path: packages/micropip/test_micropip.py
text: @user1:
Maybe use [snowballstemmer wheel](https://github.com/pyodide/pyodide/tree/main/packages/micropip/test) instead, to make the test little more self-contained?

Some other tests use it such as:

https://github.com/pyodide/pyodide/blob/8c8109f1dfa9f3716f6391351152bcef23ffc006/packages/micropip/test_micropip.py#L248-L251

@author:
I'm not sure I understand. Isn't the test already self contained?

@author:
Oh I guess I see what you are after.

diff_hunk: @@ -86,23 +90,28 @@ def is_compatible(self):
                 return True
         return False
 
-    async def download(self, fetch_kwargs):
+    async def _fetch_bytes(self, fetch_kwargs):
         try:
-            wheel_bytes = await fetch_bytes(self.url, fetch_kwargs)
+            return await fetch_bytes(self.url, fetch_kwargs)
         except OSError as e:
-            if self.url.startswith("https://files.pythonhosted.org/"):
+            if self.parsed_url.hostname == "files.pythonhosted.org":
path: packages/micropip/src/micropip/_micropip.py
text: @user1:
Maybe also,

```suggestion
            if self.parsed_url.hostname in ["files.pythonhosted.org", "cdn.jsdelivr.net"]:
```
as we have wheels there.

@author:
Not strictly related to this PR, but sure it seems like an improvement. Maybe store the known-CORS domains into a global variable. Relatedly, we should ask pypi to add CORs to their 404 page...

diff_hunk: @@ -86,23 +90,28 @@ def is_compatible(self):
                 return True
         return False
 
-    async def download(self, fetch_kwargs):
+    async def _fetch_bytes(self, fetch_kwargs):
         try:
-            wheel_bytes = await fetch_bytes(self.url, fetch_kwargs)
+            return await fetch_bytes(self.url, fetch_kwargs)
         except OSError as e:
-            if self.url.startswith("https://files.pythonhosted.org/"):
+            if self.parsed_url.hostname == "files.pythonhosted.org":
                 raise e
             else:
                 raise ValueError(
-                    f"Can't fetch wheel from '{self.url}'."
+                    f"Can't fetch wheel from '{self.url}'. "
                     "One common reason for this is when the server blocks "
-                    "Cross-Origin Resource Sharing (CORS)."
+                    "Cross-Origin Resource Sharing (CORS). "
                     "Check if the server is sending the correct 'Access-Control-Allow-Origin' header."
                 ) from e
 
-        self.data = BytesIO(wheel_bytes)
-
-        with ZipFile(self.data) as zip_file:
+    async def download(self, fetch_kwargs):
+        data: BytesIO | BufferedReader
+        if self.parsed_url.scheme == "emfs":
+            data = open(self.parsed_url.path, "rb")
path: packages/micropip/src/micropip/_micropip.py
text: @user1:
I wonder if it wouldn't be better to directly add this to fetch_bytes (used in `_fetch_bytes`). Yes, it's potentially one extra copy but that function already handles `file://` protocol, so it would make sense to have a function that is able to open URLs with different protocols rather than have this logic through the code.

No strong opinion about this though.

@author:
This is even possible without an extra copy if we make a few improvements to the code.

diff_hunk: @@ -116,9 +125,10 @@ def validate(self):
             # No checksums available, e.g. because installing
             # from a different location than PyPI.
             return
+        if not isinstance(self.data, BytesIO):
path: packages/micropip/src/micropip/_micropip.py
text: @user1:
Similarly, probably better to be explicit that we don't want to check the checksum if `self.parsed_url.scheme == "emfs":` rather than if it's not BytesIO which is an implementation detail.

Actually, is this necessary? It shouldn't have a digest with `emfs:`  URL and should return already above, no?

@author:
It's needed by mypy.

diff_hunk: @@ -412,42 +422,53 @@ async def install(
         A requirement or list of requirements to install. Each requirement is a
         string, which should be either a package name or URL to a wheel:
 
-        - If the requirement ends in ``.whl`` it will be interpreted as a URL.
-          The file must be a wheel named in compliance with the
-          `PEP 427 naming convention <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
+        - If the requirement ends in ``.whl``, the part of the requirement after
+          the last ``/``  must be a valid wheel name in compliance with the `PEP
+          427 naming convention.
+
+        - If the requirement ends in ``.whl`` and starts with ``emfs:``, it will
+          be interpreted as a path in the Emscripten file system (Pyodide's file
+          system). E.g., `emfs:../relative/path/wheel.whl` or
+          `emfs:/absolute/path/wheel.whl`.
+
+        - If the requirement ends in ``.whl`` and does not start with ``emfs:``
+          it will be interpreted as a URL. The file must be a wheel named in
+          compliance with the `PEP 427 naming convention
+          <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
path: packages/micropip/src/micropip/_micropip.py
text: @user1:
```suggestion
        - If the requirement starts with ``http:``, ``http:`` or ends with `.whl`
          it will be interpreted as a URL. 
```
We already say what are requirements for wheel URL above.

diff_hunk: @@ -412,42 +422,53 @@ async def install(
         A requirement or list of requirements to install. Each requirement is a
         string, which should be either a package name or URL to a wheel:
 
-        - If the requirement ends in ``.whl`` it will be interpreted as a URL.
-          The file must be a wheel named in compliance with the
-          `PEP 427 naming convention <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
+        - If the requirement ends in ``.whl``, the part of the requirement after
+          the last ``/``  must be a valid wheel name in compliance with the `PEP
+          427 naming convention.
path: packages/micropip/src/micropip/_micropip.py
text: @user1:
```suggestion
        - If the requirement is a URL it must end in ``.whl`` and the part of the requirement after
          the last ``/``  must be a valid wheel name in compliance with the 
          `PEP 427 naming convention
          <https://www.python.org/dev/peps/pep-0427/#file-format>`_.
```