Repository: pyodide/pyodide

Issue:
micropip.install not allowing for other wheel URLs other than PyPI
## 🐛 Bug

Hi. `pyodide` is great, so thanks to the dev team for this amazing work. I'm marking this as a "bug" as the docs seem to indicate a different behavior then I'm seeing, but apologies if you think this should be a GitHub Discussion instead (and please feel free to move it).

In the `pyodide` [`v0.19.0` docs it mentions for `micropip.install`](https://pyodide.org/en/stable/usage/api/micropip-api.html#module-micropip) that the `requirements` arg can be

> A requirement or list of requirements to install. Each requirement is a string, which should be either a package name or URL to a wheel:
>   * If the requirement ends in .whl it will be interpreted as a URL. The file must be a wheel named in compliance with the [PEP 427 naming convention](https://www.python.org/dev/peps/pep-0427/#file-format).
>   * If the requirement does not end in .whl, it will interpreted as the name of a package. A package by this name must either be present in the Pyodide repository at indexURL <globalThis.loadPyodide> or on PyPI

The docs also mention "other URLS" here

https://github.com/pyodide/pyodide/blob/950c1dcbd271f90faea0d205a7c95fcd40710a69/docs/usage/loading-packages.md?plain=1#L9-L10

This doesn't seem to be the case for at least TestPyPI. I have wheels for my team's [pure-Python project](https://pypi.org/project/pyhf) there that are built with `build` in the same workflow that we use to upload to PyPI. The installation [from PyPI](https://pypi.org/project/pyhf/) and [from PyPI with a wheel URL](https://files.pythonhosted.org/packages/a4/49/23b30e056d807d8ff0c5de9c419c6becd560be1a676a2349f34727fe9aca/pyhf-0.6.3-py3-none-any.whl) works fine, but the installation [from a wheel URL from TestPyPI](https://test-files.pythonhosted.org/packages/4d/54/a749a9ed7840800dabf2f5eff340dcbd32524a21b673f35f8a7582658c31/pyhf-0.6.4.dev77-py3-none-any.whl) fails with

```
JsException: TypeError: Failed to fetch
```

### To Reproduce

<!-- Minimal code example to reproduce the bug. -->

Running on the [Pyodide terminal emulator](https://pyodide.org/en/stable/console.html) you have made nicely available:

```python
>>> import micropip
>>> await micropip.install("https://test-files.pythonhosted.org/packages/4d/54/a749a9ed7840800dabf2f5eff340dcbd32524a21b673f3
5f8a7582658c31/pyhf-0.6.4.dev77-py3-none-any.whl")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.9/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
    future.result()
  File "/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.9/asyncio/tasks.py", line 258, in __step
    result = coro.throw(exc)
  File "/lib/python3.9/site-packages/micropip/_micropip.py", line 185, in install
    transaction = await self.gather_requirements(requirements, ctx, keep_going)
  File "/lib/python3.9/site-packages/micropip/_micropip.py", line 175, in gather_requirements
    await gather(*requirement_promises)
  File "/lib/python3.9/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
    future.result()
  File "/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/lib/python3.9/asyncio/tasks.py", line 258, in __step
    result = coro.throw(exc)
  File "/lib/python3.9/site-packages/micropip/_micropip.py", line 249, in add_requirement
    await self.add_wheel(name, wheel, version, (), ctx, transaction)
  File "/lib/python3.9/site-packages/micropip/_micropip.py", line 299, in add_wheel
    wheel_bytes = await fetch_bytes(wheel["url"])
  File "/lib/python3.9/site-packages/micropip/_micropip.py", line 57, in fetch_bytes
    return await (await pyfetch(url, **kwargs)).bytes()
  File "/lib/python3.9/site-packages/pyodide/http.py", line 232, in pyfetch
    url, await _jsfetch(url, to_js(kwargs, dict_converter=Object.fromEntries))
  File "/lib/python3.9/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
    future.result()
  File "/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
JsException: TypeError: Failed to fetch
```

Here's a replite example as well:
* Failure to install `pyhf-0.6.4.dev77-py3-none-any.whl` from TestPyPI from wheel URL [![lite-badge]][test_pypi_fail]

[lite-badge]: https://jupyterlite.rtfd.io/en/latest/_static/badge.svg
[test_pypi_fail]: https://replite.vercel.app/retro/consoles/?toolbar=1&kernel=python&code=import%20micropip%0Aawait%20micropip.install%28%22https%3A//test-files.pythonhosted.org/packages/4d/54/a749a9ed7840800dabf2f5eff340dcbd32524a21b673f35f8a7582658c31/pyhf-0.6.4.dev77-py3-none-any.whl%22%29

### Expected behavior

<!-- FILL IN -->

`pyodide` should install the wheel from TestPyPI as the name is valid (built by `build`) and is a valid URL that ends in `.whl`.

Examples of working behavior:
* Install `pyhf` `v0.6.3` successfully from PyPI [![lite-badge]][pypi_name]
* Install `pyhf` `v0.6.3` successfully from PyPI from wheel URL [![lite-badge]][pypi_wheel_url]

[pypi_name]: https://replite.vercel.app/retro/consoles/?toolbar=1&kernel=python&code=import%20micropip%0Aawait%20micropip.install%28%22pyhf%22%29%0Aimport%20pyhf
[pypi_wheel_url]: https://replite.vercel.app/retro/consoles/?toolbar=1&kernel=python&code=import%20micropip%0Aawait%20micropip.install%28%22https%3A//files.pythonhosted.org/packages/a4/49/23b30e056d807d8ff0c5de9c419c6becd560be1a676a2349f34727fe9aca/pyhf-0.6.3-py3-none-any.whl%22%29%0Aimport%20pyhf

### Environment

- Pyodide Version: 1.9.0
- Browser version: Google Chrome 98.0.4758.80
- Any other relevant information:

<!-- If you are building Pyodide by yourself, please also include these information: -->

<!--
- Commit hash of Pyodide git repository:
- Build environment<!--(e.g. Ubuntu 18.04, pyodide/pyodide-env:19 docker)- ->:
-->

### Additional context

None


Discussion:
This is a problem with headers. The server must set the header `access-control-allow-origin: *` on the response. `files.python-hosted.org` sets this header but `test-files.python-hosted.org` does not.


Probably we should update the docs to indicate this. If you are serving the wheel from the same origin as the rest of the website, you don't need this header.
> This is a problem with headers. The server must set the header `access-control-allow-origin: *` on the response. `files.python-hosted.org` sets this header but `test-files.python-hosted.org` does not.

> Probably we should update the docs to indicate this

Thanks for the fast response and explanation! I agree that if things could be updated with a small note on this in the docs that would be helpful.

Resolving patch:
diff --git a/packages/micropip/src/micropip/_micropip.py b/packages/micropip/src/micropip/_micropip.py
--- a/packages/micropip/src/micropip/_micropip.py
+++ b/packages/micropip/src/micropip/_micropip.py
@@ -295,7 +295,20 @@ async def add_requirement(
 
     async def add_wheel(self, name, wheel, version, extras, ctx, transaction):
         transaction["locked"][name] = PackageMetadata(name=name, version=version)
-        wheel_bytes = await fetch_bytes(wheel["url"])
+
+        try:
+            wheel_bytes = await fetch_bytes(wheel["url"])
+        except Exception as e:
+            if wheel["url"].startswith("https://files.pythonhosted.org/"):
+                raise e
+            else:
+                raise ValueError(
+                    f"Couldn't fetch wheel from '{wheel['url']}'."
+                    "One common reason for this is when the server blocks "
+                    "Cross-Origin Resource Sharing (CORS)."
+                    "Check if the server is sending the correct 'Access-Control-Allow-Origin' header."
+                ) from e
+
         wheel["wheel_bytes"] = wheel_bytes
 
         with ZipFile(io.BytesIO(wheel_bytes)) as zip_file:  # type: ignore


Test patch:
diff --git a/packages/micropip/test_micropip.py b/packages/micropip/test_micropip.py
--- a/packages/micropip/test_micropip.py
+++ b/packages/micropip/test_micropip.py
@@ -345,6 +345,22 @@ def test_install_keep_going(monkeypatch):
         )
 
 
+def test_fetch_wheel_fail(monkeypatch):
+    pytest.importorskip("packaging")
+    from micropip import _micropip
+
+    def _mock_fetch_bytes(*args, **kwargs):
+        raise Exception("Failed to fetch")
+
+    monkeypatch.setattr(_micropip, "fetch_bytes", _mock_fetch_bytes)
+
+    msg = "Access-Control-Allow-Origin"
+    with pytest.raises(ValueError, match=msg):
+        asyncio.get_event_loop().run_until_complete(
+            _micropip.install("htps://x.com/xxx-1.0.0-py3-none-any.whl")
+        )
+
+
 def test_list_pypi_package(monkeypatch):
     pytest.importorskip("packaging")
     from micropip import _micropip