Repository: pyodide/pyodide

Issue:
Relative URLs in pyodide.loadPackage
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
The documentation states that [pyodide.loadPackage](https://pyodide.org/en/stable/usage/api/js-api.html#pyodide.loadPackage) supports relative URLs. I'm trying to load an out-of-tree wheel from my local webserver, but this doesn't seem to work out well.

### To Reproduce

<!-- Minimal code example to reproduce the bug. -->
```js
await pyodide.loadPackage("dist/igraph-0.9.11-cp310-cp310-emscripten_3_1_14_wasm32.whl");
```
or
```js
await pyodide.loadPackage("./dist/igraph-0.9.11-cp310-cp310-emscripten_3_1_14_wasm32.whl");
```
Pyodide tries to load the wheel from `https://cdn.jsdelivr.net/pyodide/v0.21.1/full/dist/igraph-0.9.11-cp310-cp310-emscripten_3_1_14_wasm32.whl`.

### Expected behavior

<!-- FILL IN -->
Load the wheel from the relative URL.

### Environment

- Pyodide Version<!-- (e.g. 1.8.1) -->: 0.21.1
- Browser version<!-- (e.g. Chrome 95.0.4638.54) -->: Firefox ESR 91.12.0, Chromium 104.0.5112.101
- Any other relevant information:

<!-- If you are building Pyodide by yourself, please also include these information: -->

<!--
- Commit hash of Pyodide git repository:
- Build environment<!--(e.g. Ubuntu 18.04, pyodide/pyodide-env:19 docker)- ->:
-->

### Additional context

<!-- Add any other context about the problem here. -->

Pyodide crashes when loaded in a Jest Unittest.
## 🐛 Bug

I tried to write unittests for some code utilizing pyodide and vue. I did run into a whole set of errors until I failed to resolve `TextEncoder is not defined` while trying to load pyodide.

Summary:
* pyodide with `node` works like a charm.
* jest only starts with node option `--experimental-vm-modules`
* with jest `indexURL` has to be defined, otherwise the result is a `Error: Cannot find module '.../pyodide-test/node_modules/src/js/pyodide.asm.js' from 'node_modules/pyodide/pyodide.js'`
* `pyodide.asm.js` crashes with `Error: thrown: bad input to processPackageDataError:` unless modified (see 'To Reproduce')
* (currently unresolved) pyodide crashes with `ReferenceError: TextEncoder is not defined`

If anyone could hint me how to resolve the current showstopper I am ready to look into any followup issue (even if I am not sure if I could resolve it).

### To Reproduce

Enable vue testing with jest via
```
module.exports = {
  preset: '@vue/cli-plugin-unit-jest'
}
```

Modify `pyodide.asm.js` by disabling 
`assert(arrayBuffer instanceof ArrayBuffer, "bad input to processPackageData");` in `function processPackageData(arrayBuffer)`, because the `arrayBuffer instanceof ArrayBuffer` is `false` even if `arrayBuffer.constructor.name == "ArrayBuffer"` is `true` (or one could change the assertion to work in this case, but the file is auto-generated anyway...).

Insert you local path in the code below as `indexURL`

```
describe('pyodide', () => {
    let pyodide = null;
    beforeAll(async () => {
        let pyodide_pkg = await require("pyodide/pyodide.js");
        pyodide = await pyodide_pkg.loadPyodide({
            indexURL: "/this/is/your/path/to/the/local/project/node_modules/pyodide/",
        })
    })
    it('can load', () => {
        expect(pyodide).not.toBe(null)
    });
})
```

run `node --experimental-vm-modules node_modules/jest/bin/jest.js`

### Expected behavior

Test successful.

### Observed behavior

Test fails with error message 
```
TextEncoder is not defined
ReferenceError: TextEncoder is not defined
    at createStdinWrapper (/home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/src/js/module.ts:63:19)
    at /home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/src/js/module.ts:54:22
    at callRuntimeCallbacks (/home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/pyodide/pyodide.asm.js:9796:25)
    at preRun (/home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/pyodide/pyodide.asm.js:6586:17)
    at run (/home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/pyodide/pyodide.asm.js:56432:17)
    at runCaller (/home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/pyodide/pyodide.asm.js:56385:33)
    at removeRunDependency (/home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/pyodide/pyodide.asm.js:6664:25)
    at receiveInstance (/home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/pyodide/pyodide.asm.js:6762:21)
    at receiveInstantiationResult (/home/sebastian/workspace/pyodide_bug/pyodide-test/node_modules/pyodide/pyodide.asm.js:6768:21)
```

### Environment

node: v16.13.1
pyodide: 0.20.1-alpha.2

`package.json`:
```
{
  "name": "pyodide-test",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "serve": "vue-cli-service serve",
    "build": "vue-cli-service build",
    "lint": "vue-cli-service lint",
    "test": "jest",
    "offline": "node src/offline.js"
  },
  "dependencies": {
    "@vue/cli-plugin-unit-jest": "^5.0.4",
    "@vue/test-utils": "^2.0.0",
    "core-js": "^3.6.5",
    "pyodide": "^0.20.1-alpha.2",
    "vue": "^3.0.0"
  },
  "devDependencies": {
    "@vue/cli-plugin-babel": "~4.5.13",
    "@vue/cli-plugin-eslint": "~4.5.13",
    "@vue/cli-service": "~4.5.13",
    "@vue/compiler-sfc": "^3.0.0",
    "@vue/vue3-jest": "^27.0.0",
    "babel-eslint": "^10.1.0",
    "eslint": "^6.7.2",
    "eslint-plugin-vue": "^7.0.0"
  },
  "eslintConfig": {
    "root": true,
    "env": {
      "node": true
    },
    "extends": [
      "plugin:vue/vue3-essential",
      "eslint:recommended"
    ],
    "parserOptions": {
      "parser": "babel-eslint"
    },
    "rules": {}
  },
  "browserslist": [
    "> 1%",
    "last 2 versions",
    "not dead"
  ]
}
```

### Additional context

Nice piece of software.

I made a test repository in https://github.com/msebas/pyodide-test to simplify reproducing the problem above.



Discussion:

Looks like there is a solution here:
https://stackoverflow.com/questions/68468203/why-am-i-getting-textencoder-is-not-defined-in-jest
Hello and thanks. 

Seems like this does resolve the problem. Now the test finishes without issues. Sadly I do not know how to modify the build configuration to automate this process, so I close this issue. 

~~On top it seems like when now using a new package install out of the bat the only problem remaining is defining IndexURL. It seems like a dependent package was causing the `processPackageDataError` and this seems to be updated now. As a result I could not reproduce the error in any new installation.~~

Kind regards
I made a mistake. The error does NOT occur with the given setup. I edited the setup and made a test repository in https://github.com/msebas/pyodide-test to simplify reproduction. 

The problem does only occur if jest is configured to use 
```
module.exports = {
  preset: '@vue/cli-plugin-unit-jest'
}
```

I do not know if this is something that should work, but from my point of view this reopens the problem itself.

Resolving patch:
diff --git a/packages/micropip/src/micropip/_compat_in_pyodide.py b/packages/micropip/src/micropip/_compat_in_pyodide.py
--- a/packages/micropip/src/micropip/_compat_in_pyodide.py
+++ b/packages/micropip/src/micropip/_compat_in_pyodide.py
@@ -23,7 +23,7 @@ async def fetch_bytes(url: str, kwargs: dict[str, str]) -> IO[bytes]:
     if parsed_url.scheme == "emfs":
         return open(parsed_url.path, "rb")
     if parsed_url.scheme == "file":
-        result_bytes = (await loadBinaryFile("", parsed_url.path)).to_bytes()
+        result_bytes = (await loadBinaryFile(parsed_url.path)).to_bytes()
     else:
         result_bytes = await (await pyfetch(url, **kwargs)).bytes()
     return BytesIO(result_bytes)
diff --git a/pyodide-build/pyodide_build/common.py b/pyodide-build/pyodide_build/common.py
--- a/pyodide-build/pyodide_build/common.py
+++ b/pyodide-build/pyodide_build/common.py
@@ -109,8 +109,8 @@ def find_matching_wheels(wheel_paths: Iterable[Path]) -> Iterator[Path]:
     "fpcast-test",
     "sharedlib-test-py",
     "cpp-exceptions-test",
-    "_ssl",
-    "_lzma",
+    "ssl",
+    "lzma",
     "pytest",
     "tblib",
 }
diff --git a/src/py/_pyodide/_importhook.py b/src/py/_pyodide/_importhook.py
--- a/src/py/_pyodide/_importhook.py
+++ b/src/py/_pyodide/_importhook.py
@@ -152,7 +152,12 @@ def __init__(self) -> None:
         self.stdlibs = sys.stdlib_module_names | {"test"}
 
         # TODO: put list of unvendored stdlibs to somewhere else?
-        self.unvendored_stdlibs = {"distutils", "test", "_ssl", "_lzma"} & self.stdlibs
+        self.unvendored_stdlibs = {
+            "distutils",
+            "test",
+            "ssl",
+            "lzma",
+        } & self.stdlibs
 
     def find_spec(
         self,


Test patch:
diff --git a/packages/_lzma/test_lzma.py b/packages/lzma/test_lzma.py
similarity index 85%
rename from packages/_lzma/test_lzma.py
rename to packages/lzma/test_lzma.py
--- a/packages/_lzma/test_lzma.py
+++ b/packages/lzma/test_lzma.py
@@ -1,7 +1,7 @@
 from pytest_pyodide import run_in_pyodide
 
 
-@run_in_pyodide(packages=["test", "_lzma"], pytest_assert_rewrites=False)
+@run_in_pyodide(packages=["test", "lzma"], pytest_assert_rewrites=False)
 def test_lzma(selenium):
     # TODO: libregrtest.main(["test_lzma"]) doesn't collect any tests for some unknown reason.
 
diff --git a/packages/_ssl/test_ssl.py b/packages/ssl/test_ssl.py
similarity index 89%
rename from packages/_ssl/test_ssl.py
rename to packages/ssl/test_ssl.py
--- a/packages/_ssl/test_ssl.py
+++ b/packages/ssl/test_ssl.py
@@ -1,7 +1,7 @@
 from pytest_pyodide import run_in_pyodide
 
 
-@run_in_pyodide(packages=["test", "_ssl"], pytest_assert_rewrites=False)
+@run_in_pyodide(packages=["test", "ssl"], pytest_assert_rewrites=False)
 def test_ssl(selenium):
     import platform
     import unittest
@@ -17,6 +17,7 @@ def test_ssl(selenium):
         "test_verify_flags",
         "test_subclass",
         "test_lib_reason",
+        "test_unwrap",
     ]
 
     try:
diff --git a/src/tests/test_package_loading.py b/src/tests/test_package_loading.py
--- a/src/tests/test_package_loading.py
+++ b/src/tests/test_package_loading.py
@@ -2,15 +2,18 @@
 from pathlib import Path
 
 import pytest
+from pytest_pyodide.fixture import selenium_common
+from pytest_pyodide.server import spawn_web_server
+from pytest_pyodide.utils import parse_driver_timeout, set_webdriver_script_timeout
 
-from conftest import DIST_PATH
+from conftest import DIST_PATH, ROOT_PATH
 
 
-def get_pyparsing_wheel_name():
+def get_pyparsing_wheel_name() -> str:
     return list(DIST_PATH.glob("pyparsing*.whl"))[0].name
 
 
-def get_pytz_wheel_name():
+def get_pytz_wheel_name() -> str:
     return list(DIST_PATH.glob("pytz*.whl"))[0].name
 
 
@@ -58,10 +61,33 @@ def test_load_from_url(selenium_standalone, web_server_secondary, active_server)
     selenium.run("import pytz")
 
 
-def test_load_relative_url(selenium_standalone):
-    print(get_pytz_wheel_name())
-    selenium_standalone.load_package(f"./{get_pytz_wheel_name()}")
-    selenium_standalone.run("import pytz")
+def test_load_relative_url(
+    request, runtime, web_server_main, playwright_browsers, tmp_path
+):
+    url, port, _ = web_server_main
+    test_html = (ROOT_PATH / "src/templates/test.html").read_text()
+    test_html = test_html.replace("./pyodide.js", f"http://{url}:{port}/pyodide.js")
+    (tmp_path / "test.html").write_text(test_html)
+    pytz_wheel = get_pytz_wheel_name()
+    pytz1_wheel = pytz_wheel.replace("pytz", "pytz1")
+    shutil.copy(DIST_PATH / pytz_wheel, tmp_path / pytz1_wheel)
+
+    with spawn_web_server(tmp_path) as web_server, selenium_common(
+        request,
+        runtime,
+        web_server,
+        load_pyodide=True,
+        browsers=playwright_browsers,
+        script_type="classic",
+    ) as selenium, set_webdriver_script_timeout(
+        selenium, script_timeout=parse_driver_timeout(request.node)
+    ):
+        if selenium.browser == "node":
+            selenium.run_js(f"process.chdir('{tmp_path.resolve()}')")
+        selenium.load_package(pytz1_wheel)
+        selenium.run(
+            "import pytz; from pyodide_js import loadedPackages; print(loadedPackages.pytz1)"
+        )
 
 
 def test_list_loaded_urls(selenium_standalone):
diff --git a/src/tests/test_pyodide.py b/src/tests/test_pyodide.py
--- a/src/tests/test_pyodide.py
+++ b/src/tests/test_pyodide.py
@@ -1,11 +1,13 @@
 import re
 from collections.abc import Sequence
+from pathlib import Path
 from textwrap import dedent
 from typing import Any
 
 import pytest
 from pytest_pyodide import run_in_pyodide
 
+from conftest import ROOT_PATH
 from pyodide.code import CodeRunner, eval_code, find_imports, should_quiet  # noqa: E402
 
 
@@ -1197,7 +1199,7 @@ def test_unvendored_stdlib(selenium_standalone):
 
     import pytest
 
-    unvendored_stdlibs = ["test", "_ssl", "_lzma"]
+    unvendored_stdlibs = ["test", "ssl", "lzma"]
     removed_stdlibs = ["pwd", "turtle", "tkinter"]
 
     for lib in unvendored_stdlibs:
@@ -1234,3 +1236,59 @@ def test_unvendored_stdlib(selenium_standalone):
             ModuleNotFoundError, match="removed from the Python standard library"
         ):
             finder.find_spec(lib, None)
+
+
+@pytest.mark.xfail_browsers(chrome="Node only", firefox="Node only")
+def test_relative_index_url(selenium, tmp_path):
+    tmp_dir = Path(tmp_path)
+    import subprocess
+
+    version_result = subprocess.run(
+        ["node", "-v"], capture_output=True, encoding="utf8"
+    )
+    extra_node_args = []
+    if version_result.stdout.startswith("v14"):
+        extra_node_args.append("--experimental-wasm-bigint")
+
+    import shutil
+
+    shutil.copy(ROOT_PATH / "dist/pyodide.js", tmp_dir / "pyodide.js")
+    shutil.copytree(ROOT_PATH / "dist/node_modules", tmp_dir / "node_modules")
+
+    result = subprocess.run(
+        [
+            "node",
+            *extra_node_args,
+            "-e",
+            rf"""
+            const loadPyodide = require("{tmp_dir / "pyodide.js"}").loadPyodide;
+            async function main(){{
+                py = await loadPyodide({{indexURL: "./dist"}});
+                console.log("\n");
+                console.log(py._module.API.config.indexURL);
+            }}
+            main();
+            """,
+        ],
+        cwd=ROOT_PATH,
+        capture_output=True,
+        encoding="utf8",
+    )
+    import textwrap
+
+    def print_result(result):
+        if result.stdout:
+            print("  stdout:")
+            print(textwrap.indent(result.stdout, "    "))
+        if result.stderr:
+            print("  stderr:")
+            print(textwrap.indent(result.stderr, "    "))
+
+    if result.returncode:
+        print_result(result)
+        result.check_returncode()
+
+    try:
+        assert result.stdout.strip().split("\n")[-1] == str(ROOT_PATH / "dist") + "/"
+    finally:
+        print_result(result)