Repository: pyodide/pyodide

Issue:
Issue in building rust-based packages in parallel
We are having some issue when building multiple rust-based packages in parallel.

For example, see: https://app.circleci.com/pipelines/github/pyodide/pyodide/5328/workflows/a0ced892-ac1a-4519-984c-d98a84fabe29/jobs/66900

It is expected behavior since `rustup` was not designed for concurrent use (https://github.com/rust-lang/rustup/issues/988).

Some possible workarounds that I can think of are:

1. Add some logic to not build rust-baed packages in parallel
2. Create a fresh local rust build environment everytime when building rust-based package. I found a Python package called [rustenv](https://github.com/chriskuehl/rustenv), which is designed for that purpose. It is a ~150LoC script, which basically installs rust every time from `https://sh.rustup.rs` into a local directory.

I slightly prefer option 1, as we don't have to be responsible for installing build requirements. But I have no strong opinion.




Discussion:
+1 for option 1 as well.

Resolving patch:
diff --git a/pyodide-build/pyodide_build/buildall.py b/pyodide-build/pyodide_build/buildall.py
--- a/pyodide-build/pyodide_build/buildall.py
+++ b/pyodide-build/pyodide_build/buildall.py
@@ -427,6 +427,13 @@ def job_priority(pkg: BasePackage) -> int:
         return 1
 
 
+def is_rust_package(pkg: BasePackage) -> bool:
+    """
+    Check if a package requires rust toolchain to build.
+    """
+    return any([q in pkg.executables_required for q in ("rustc", "cargo", "rustup")])
+
+
 def format_name_list(l: list[str]) -> str:
     """
     >>> format_name_list(["regex"])
@@ -538,14 +545,30 @@ def build_from_graph(
     built_queue: Queue[BasePackage | Exception] = Queue()
     thread_lock = Lock()
     queue_idx = 1
+    building_rust_pkg = False
     progress_formatter = ReplProgressFormatter(len(needs_build))
 
     def builder(n: int) -> None:
-        nonlocal queue_idx
+        nonlocal queue_idx, building_rust_pkg
         while True:
-            pkg = build_queue.get()[1]
+            _, pkg = build_queue.get()
 
             with thread_lock:
+                if is_rust_package(pkg):
+                    # Don't build multiple rust packages at the same time.
+                    # See: https://github.com/pyodide/pyodide/issues/3565
+                    # Note that if there are only rust packages left in the queue,
+                    # this will keep pushing and popping packages until the current rust package
+                    # is built. This is not ideal but presumably the overhead is negligible.
+                    if building_rust_pkg:
+                        build_queue.put((job_priority(pkg), pkg))
+
+                        # Release the GIL so new packages get queued
+                        sleep(0.1)
+                        continue
+
+                    building_rust_pkg = True
+
                 pkg._queue_idx = queue_idx
                 queue_idx += 1
 
@@ -569,6 +592,11 @@ def builder(n: int) -> None:
                 progress_formatter.remove_package(pkg_status)
 
             built_queue.put(pkg)
+
+            with thread_lock:
+                if is_rust_package(pkg):
+                    building_rust_pkg = False
+
             # Release the GIL so new packages get queued
             sleep(0.01)
 


Test patch:
diff --git a/pyodide-build/pyodide_build/tests/test_buildall.py b/pyodide-build/pyodide_build/tests/test_buildall.py
--- a/pyodide-build/pyodide_build/tests/test_buildall.py
+++ b/pyodide-build/pyodide_build/tests/test_buildall.py
@@ -153,3 +153,28 @@ def test_requirements_executable(monkeypatch):
         m.setattr(shutil, "which", lambda exe: "/bin")
 
         buildall.generate_dependency_graph(RECIPE_DIR, {"pkg_test_executable"})
+
+
+@pytest.mark.parametrize("exe", ["rustc", "cargo", "rustup"])
+def test_is_rust_package(exe):
+
+    pkg = buildall.BasePackage(
+        pkgdir=Path(""),
+        name="test",
+        version="1.0.0",
+        disabled=False,
+        meta=None,  # type: ignore[arg-type]
+        package_type="package",
+        run_dependencies=[],
+        host_dependencies=[],
+        host_dependents=set(),
+        dependencies=set(),
+        unbuilt_host_dependencies=set(),
+        executables_required=[exe],
+    )
+
+    assert buildall.is_rust_package(pkg)
+
+    pkg.executables_required = []
+
+    assert not buildall.is_rust_package(pkg)