Repository: pyodide/pyodide

Issue:
Importing reboundx fails occasionally
## 🐛 Bug

Importing reboundx fails occasionally in the stable/latest branch.

The reason for the failure is that we download and install packages simultaneously.
(To be more precise, we install shared libraries first then install normal wheel packages.)

https://github.com/pyodide/pyodide/blob/b497ce26ed94774a8d695a3e61016a7a26e45e3b/src/js/load-package.ts#L436-L448

The problem is that `reboundx` is dependent on another non-shared library: `rebound`. When loading `reboundx`, it loads `libreboundx.so` which load `librebound.so` which is in `rebound` package. So if `reboundx` is downloaded and installed "before" `rebound`, it fails to load because there is no `librebound.so` in the virtual file system.

I think the `reboundx` needs to bundle `librebound.so` inside its wheel. However, we can try to fix this issue on Pyodide side.

The possible solution can be:

1. When `pyodide.loadPackage()` is called, load dependencies first, then load the requested package.
2. Separate out downloading and dynlib loading process, so downloading all packages happens first, then loading dynlibs happens next.

Personally, I think option 2 is better since if we use option 1, it might be tough to handle cases like `pyodide.loadPyodide(["reboundx", "rebound"])`. The downside of option 2 is that there is a potential slowdown in loading packages.

### To Reproduce

```
await loadPackage("reboundx")
```

![image](https://user-images.githubusercontent.com/24893111/184254798-379a2dce-1c3b-4068-98b4-3c984ee66794.png)


### Environment

- Pyodide Version: 0.21.0

cc: @hannorein 


Discussion:
Maybe a mixed method would work: try to load straight away, but if a dependency is missing throw an error and try again after all packages are downloaded?
Is this just an error message in the console or does the package not work? Is it possible the package itself works fine despite the error message?

Here's a short test to see if it works:

```
import rebound, reboundx
sim = rebound.Simulation()
sim.add(m=1.)
sim.add(m=3e-3,a=4)
sim.add(m=5e-3,a=10)
rebx = reboundx.Extras(sim)
mof = rebx.load_force(“modify_orbits_forces")
rebx.add_force(mof)
sim.particles[2].params["tau_a"] = -200000
sim.particles[2].params["tau_e"] = -400000
sim.integrate(1e5)
```
> Maybe a mixed method would work: try to load straight away, but if a dependency is missing throw an error and try again after all packages are downloaded?

Sound good. Open a PR with retrying approach #2963.

> Is this just an error message in the console or does the package not work? Is it possible the package itself works fine despite the error message?

![image](https://user-images.githubusercontent.com/24893111/184456753-f20015a7-86c3-4c45-a4fe-eb0fa9f4d70b.png)

This is what I get in the console.
(I still cannot reproduce the issue.)

In reboundx's `__init__.py` there is an `import rebound` statement, but after the line that throws the above error. Would it make sense to just move the import statement further up? Then rebound would have to be installed by the time the library is used.
> In reboundx's `__init__.py` there is an `import rebound` statement, but after the line that throws the above error. Would it make sense to just move the import statement further up? Then rebound would have to be installed by the time the library is used.

Well, it is not related to the import order. The problem happens because `libreboundx.so` (inside reboundx) depends on `librebound.so` (inside rebound) but we download them simultaneously in a random order so if `libreboundx.so` is downloaded and loaded before `librebound.so`, it will fail.

Hopefully, #2963 would fix it.

But if reboundx imports rebound (which loads `librebound.so`) before it loads `libreboundx.so` shouldn't this solve the problem?
Nope. The problem occurs in the loading system before any python code is executed. `a.so` contains metadata saying it depends on `b.so` so when `a.so` is loaded it needs to see the symbols from `b.so`. if `b.so` isn't around yet, an error is raised due to missing symbols.

In particular, the issue is that wasm cannot be loaded synchronously, so everything needs to be loaded ahead of time.
Oh. Ok. Thanks. That was the part I was missing..

Resolving patch:
diff --git a/pyodide-build/pyodide_build/buildall.py b/pyodide-build/pyodide_build/buildall.py
--- a/pyodide-build/pyodide_build/buildall.py
+++ b/pyodide-build/pyodide_build/buildall.py
@@ -522,11 +522,6 @@ def generate_packagedata(
             }
             packages[name.lower() + "-tests"] = pkg_entry
 
-    # Workaround for circular dependency between soupsieve and beautifulsoup4
-    # TODO: FIXME!!
-    if "soupsieve" in packages:
-        packages["soupsieve"]["depends"].append("beautifulsoup4")
-
     # sort packages by name
     packages = dict(sorted(packages.items()))
     return packages
diff --git a/pyodide-build/pyodide_build/common.py b/pyodide-build/pyodide_build/common.py
--- a/pyodide-build/pyodide_build/common.py
+++ b/pyodide-build/pyodide_build/common.py
@@ -163,10 +163,6 @@ def _parse_package_subset(query: str | None) -> set[str]:
         packages |= CORE_PACKAGES | CORE_SCIPY_PACKAGES
         packages.discard("min-scipy-stack")
 
-    # Hack to deal with the circular dependence between soupsieve and
-    # beautifulsoup4
-    if "beautifulsoup4" in packages:
-        packages.add("soupsieve")
     packages.discard("")
     return packages
 


Test patch: