Repository: pyodide/pyodide

Issue:
Embed external shared libraries into the wheel
## Problem statement

Normally, native linux Python wheels include its dependent external shared libraries. It is done by [`auditwheel`](https://github.com/pypa/auditwheel) which copies shared libraries into the wheel and repairs the RPATH of elf binaries.

On the other hand, we are serving external shared libraries separately from the Python wheel. This is a bit annoying since it makes wheels not portable themselves. One other partially related problem is that if the wheel contains shared libraries inside its wheel, it is not working well because we have to load external shared libraries globally (unless we manage to make #3054 work) but the libraries inside the wheel are not loaded globally.

## Proposal

I think it would be nice if we can embed shared libraries into the wheels so wheels can be distributed and be used without downloading additional shared libraries. 

For this, I have been recently working on a tool called [`auditwheel-emscripten`](https://github.com/ryanking13/auditwheel-emscripten) which does jobs similar to auditwheel but for Emscripten / Pyodide wheels: It checks the external libraries that Emscripten WASM module depends and can copy them into the wheel file.

## Implementation issues

One issue is that there is no such thing as RPATH in Emscripten WASM modules. [They only contain the name of the shared library](https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md). So we will need to manually handle how the WASM module should find its dependent libraries. But [since we are doing some heuristic path resolution already](https://github.com/pyodide/pyodide/blob/6214bafe5dacae7190781bb6a0091de867aa35db/src/js/load-package.ts#L374-L397), I believe it is not a big deal.

One other issue is how we should distinguish Python c-extension modules and shared libraries. We need to load shared libraries globally (for now), but we want to load Python extension modules locally. So we need some ways to distinguish them.
A naive idea that I am thinking of is using the prefix and suffix of the filename: Checking if the filename startswith `lib` or endswith an extension suffix (`cpython-310-wasm32-emscripten`). But it would be nice if there is a more proper way.
Note that loading a python c-extension module globally will not cause an error (unless there is a symbol collision), but loading a shared library locally will raise an error.



BLD Add option to embed shared libraries into the wheel
### Description

This is an implementation of #3167. It works, but it is still work-in-progress as this change adds load time overhead of checking if a shared library needs to be loaded locally or globally. I am looking for a better and faster way.

- Adds a new key in meta.yaml: `vendor-sharedlib`.
  - If it is set to true, `auditwheel-emscripten` will find shared libraries used by the wheel and copy them into the wheel.
  - In this PR, I only applied this key to `h5py` and `shapely`. Other packages are not affected.


### Checklists

- [ ] Add a [CHANGELOG](https://github.com/pyodide/pyodide/blob/main/docs/project/changelog.md) entry
- [ ] Add / update tests
- [ ] Add new / update outdated documentation



Discussion:
Thanks for starting this discussion! This is indeed the Python way and probably something we really have to do for packages built out of tree. So having auditwheel-emscripten is indeed very valuable.

For packages we built in tree, I'm less sure, as this would lead to potentially redundant downloads (e.g. BLAS downloaded multiple times for numpy and scipy once we switch to a better BLAS). Though it would be worth analysing how many dependent packages each shared library has in the current dependency graph, and whether indeed vendoring it inside the wheel would be that significant. 

If we only have wheels and no standalone shared  dependencies, this would also make hosting easier #3049 as it means we can more easily reuse standard Python tooling.

> A naive idea that I am thinking of is using the prefix and suffix of the filename: Checking if the filename startswith lib or endswith an extension suffix (cpython-310-wasm32-emscripten). But it would be nice if there is a more proper way.

Maybe listing symbols and checking if there is anything matching `PyInit_*`?

 


> One other issue is how we should distinguish Python c-extension modules and shared libraries. We need to load shared libraries globally (for now), but we want to load Python extension modules locally. So we need some ways to distinguish them. A naive idea that I am thinking of is using the prefix and suffix of the filename: Checking if the filename startswith `lib` or endswith an extension suffix (`cpython-310-wasm32-emscripten`).

Isn't an issue with depending on the name that sometimes packages will depend on the extension modules of other packages, e.g., how `reboundx` depends on `rebound`'s (#2909)?


> For packages we built in tree, I'm less sure, as this would lead to potentially redundant downloads (e.g. BLAS downloaded multiple times for numpy and scipy once we switch to a better BLAS). Though it would be worth analysing how many dependent packages each shared library has in the current dependency graph, and whether indeed vendoring it inside the wheel would be that significant.

Yes, that could be an issue for some large shared libs like BLAS. In that case probably we need an option in `meta.yaml` to choose whether to vendor shared libs into the wheel or not.

> Maybe listing symbols and checking if there is anything matching `PyInit_*`?

Thanks for the suggestion! That sound like a nice option. But then we should check how much it increases the load time because we have to look at the content of every shared library.
> Isn't an issue with depending on the name that sometimes packages will depend on the extension modules of other packages, e.g., how `reboundx` depends on `rebound`'s (#2909)?

I think it won't be a problem in rebound case. Currently, we load `librebound.so` locally first, but its symbols become globally visible when reboundx calls dlopen (called via `ctypes.cdll.LoadLibrary`) to load `libreboundx.so` and `librebound.so`.
I think I managed to make this (almost) work. I'll split this up into multiple PRs to make review easier.

Resolving patch:
diff --git a/pyodide-build/pyodide_build/buildpkg.py b/pyodide-build/pyodide_build/buildpkg.py
--- a/pyodide-build/pyodide_build/buildpkg.py
+++ b/pyodide-build/pyodide_build/buildpkg.py
@@ -566,11 +566,6 @@ def package_wheel(
     # to maximize sanity.
     replace_so_abi_tags(wheel_dir)
 
-    vendor_sharedlib = build_metadata.vendor_sharedlib
-    if vendor_sharedlib:
-        lib_dir = Path(common.get_make_flag("WASM_LIBRARY_DIR"))
-        copy_sharedlibs(wheel, wheel_dir, lib_dir)
-
     post = build_metadata.post
     if post:
         print("Running post script in ", str(Path.cwd().absolute()))
@@ -580,6 +575,11 @@ def package_wheel(
             print("ERROR: post failed")
             exit_with_stdio(result)
 
+    vendor_sharedlib = build_metadata.vendor_sharedlib
+    if vendor_sharedlib:
+        lib_dir = Path(common.get_make_flag("WASM_LIBRARY_DIR"))
+        copy_sharedlibs(wheel, wheel_dir, lib_dir)
+
     python_dir = f"python{sys.version_info.major}.{sys.version_info.minor}"
     host_site_packages = Path(host_install_dir) / f"lib/{python_dir}/site-packages"
     if build_metadata.cross_build_env:


Test patch: