Repository: pyodide/pyodide-build
Pull request #152: Add support for building without isolation
Author: login: agriyakhetarpal
Created: 2025-03-29T01:12:38Z | Merged: 2025-04-05T22:29:12Z

## Description

This PR is the first in a series of PRs aiming to unblock #109 and also towards strides in other areas; such as being able to build a package from one directory twice, see https://github.com/pyodide/pyodide/pull/5031. In particular, this adds two new flags: `--no-isolation`/`-n` and `--skip-dependency-check`/`-x`, to the `pyodide build` CLI. These will allow us not to use an isolated virtual environment when building a wheel for a package if required.

Closes #136

### See also

- https://docs.scipy.org/doc/scipy/building/redistributable_binaries.html
- xref #138

### Some TODO items

- [x] Get "unisolated" builds working
- [x] Get tests passing
- [x] Investigate and fix the issue where the compiler wrappers are ending up in temporary directories
- [x] Gotta add tests!!!
	- [x] Basic tests that see if the isolation is working
	- [x] Integration test with NumPy/SciPy/etc.
- [x] Add a CHANGELOG entry

Linked issue #136: Support `--no-isolation` flag from pypa/build for "unisolated" package builds
This feature request is based on a requirement that came up in https://github.com/pyodide/pyodide-build/pull/109 – building a package without isolation would mean that:
- users install PEP 508 build dependencies independently from a requirements file, an environment file, etc. prior to building the package, so that those dependencies are available
    - this is not easy to handle with `pyodide-build` and Pyodide's recipe format upstream, so we should probably not do this and keep installing the dependencies beforehand.
- we don't create a PEP 517 isolated virtual environment for building if `pyodide build --no-isolation`
    - should we also have a boolean `isolated:` field for the recipes that want to opt in to this?

So, supporting this feature for us would be limited to point 2, similar to `-n` option, but we would deviate from `pypa/build`'s behaviour and would actually be `-nx` (where `-x` stands for skipping build dependency checks – see https://build.pypa.io/en/stable/#python--m-build-options).

A package to be built without isolation would, therefore, go through this process:
- we still install their build-time dependencies into the environment
- at the time of building the package, we pass the equivalent args for the `-nx` flags to `DefaultIsolatedEnv`, or we monkeypatch [`IsolatedEnv`](https://build.pypa.io/en/stable/api.html#build.env.IsolatedEnv) to create a persistent venv for installation that we can reuse.





Diff:
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 5b183d6a..91b0c14f 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -105,8 +105,11 @@ jobs:
           ]
         os: [ubuntu-latest, macos-latest]
         pyodide-version: [stable]
-        # Run Pyodide minimum version testing only for the pip installer and on Linux
         include:
+        # Run no-isolation tests and Pyodide minimum version testing only
+        # for the pip installer and on Linux
+          - task: {name: test-src-no-isolation, installer: pip} # installer doesn't matter
+            os: ubuntu-latest
           - task: { name: test-recipe, installer: pip }
             os: ubuntu-latest
             pyodide-version: minimum
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4d587b3c..b13c4d92 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added basic support for uv. `uv tool install pyodide-cli --with pyodide-build`, or `uvx --from pyodide-cli --with pyodide-build pyodide --help`, or using `pyodide-build` in `uv`-managed virtual environments will now work.
   [#132](https://github.com/pyodide/pyodide-build/pull/132)
 
+- Added support for building without a wheel without build isolation: `pyodide build` no accepts
+  the `--no-isolation`/`-n` and/or `--skip-dependency-check`/`-x` flags to customise the wheel
+  building behaviour, similar to `pypa/build`.
+
 ### Changed
 
 - The Rust toolchain version has been updated to `nightly-2025-01-18`.
diff --git a/integration_tests/Makefile b/integration_tests/Makefile
index d50dc9be..05dd79b4 100644
--- a/integration_tests/Makefile
+++ b/integration_tests/Makefile
@@ -23,6 +23,25 @@ test-src: check
 
 	@echo "... Passed"
 
+.PHONY: test-src-no-isolation
+test-src-no-isolation: check
+	@echo "... Running integration tests for building src with --no-isolation --skip-dependency-check"
+
+	# Some virtualenv workarounds from https://stackoverflow.com/a/24736236
+	# to make sure that we are using the right environment
+	@( \
+        set -e; \
+		python -m venv ./test_venv; \
+		. ./test_venv/bin/activate; \
+		pip install meson-python meson cython; \
+		pip install -e ../; \
+		./src/numpy_no_isolation.sh; \
+		deactivate; \
+	)
+
+	@rm -rf ./test_venv
+
+	@echo "... Passed"
 .PHONY: check
 check:
 	@echo "... Checking dependencies"
@@ -36,5 +55,9 @@ check:
 clean:
 	rm -rf .pyodide-xbuildenv*
 	rm -rf recipes/*/build
+	rm -rf test_venv
 	rm -rf src/numpy-*
+	rm -rf src/numpy-*.tar.gz
+	rm -rf numpy-*
+	rm -rf numpy-*.tar.gz
 	rm -rf dist
diff --git a/integration_tests/src/numpy_no_isolation.sh b/integration_tests/src/numpy_no_isolation.sh
new file mode 100755
index 00000000..82379d8c
--- /dev/null
+++ b/integration_tests/src/numpy_no_isolation.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# The same as "numpy.sh", but without the isolation, and
+# builds both NumPy and numpy-tests from a persistent
+# build directory.
+
+set -e
+
+VERSION="2.0.2"
+URL="https://files.pythonhosted.org/packages/source/n/numpy/numpy-${VERSION}.tar.gz"
+
+wget $URL
+tar -xf numpy-${VERSION}.tar.gz
+cd numpy-${VERSION}
+
+MESON_CROSS_FILE=$(pyodide config get meson_cross_file)
+
+# Build in a persistent build directory
+${UV_RUN_PREFIX} pyodide build \
+    -Csetup-args=-Dallow-noblas=true \
+    -Csetup-args=--cross-file="${MESON_CROSS_FILE}" \
+    -Cinstall-args=--tags=runtime,python-runtime,devel \
+    -Cbuild-dir="build" \
+    --no-isolation --skip-dependency-check
+
+sed -i 's/numpy/numpy-tests/g' pyproject.toml
+
+${UV_RUN_PREFIX} pyodide build \
+    -Csetup-args=-Dallow-noblas=true \
+    -Csetup-args=--cross-file="${MESON_CROSS_FILE}" \
+    -Cinstall-args=--tags=tests \
+    -Cbuild-dir="build" \
+    --no-isolation --skip-dependency-check
+
+echo "Successfully built wheels for numpy-${VERSION} and numpy-tests-${VERSION}."
diff --git a/pyodide_build/cli/build.py b/pyodide_build/cli/build.py
index dd1496d8..a018a7a7 100644
--- a/pyodide_build/cli/build.py
+++ b/pyodide_build/cli/build.py
@@ -46,6 +46,8 @@ def pypi(
     output_directory: Path,
     exports: str,
     config_settings: ConfigSettingsType,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
 ) -> Path:
     """Fetch a wheel from pypi, or build from source if none available."""
     with tempfile.TemporaryDirectory() as tmpdir:
@@ -65,6 +67,8 @@ def pypi(
             output_directory,
             convert_exports(exports),
             config_settings,
+            isolation=isolation,
+            skip_dependency_check=skip_dependency_check,
         )
         return built_wheel
 
@@ -86,6 +90,8 @@ def url(
     output_directory: Path,
     exports: str,
     config_settings: ConfigSettingsType,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
 ) -> Path:
     """Fetch a wheel or build sdist from url."""
     with tempfile.TemporaryDirectory() as tmpdir:
@@ -102,7 +108,12 @@ def url(
             # unzipped into subfolder
             builddir = files[0]
         wheel_path = build.run(
-            builddir, output_directory, convert_exports(exports), config_settings
+            builddir,
+            output_directory,
+            convert_exports(exports),
+            config_settings,
+            isolation=isolation,
+            skip_dependency_check=skip_dependency_check,
         )
         return wheel_path
 
@@ -112,10 +123,17 @@ def source(
     output_directory: Path,
     exports: str,
     config_settings: ConfigSettingsType,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
 ) -> Path:
     """Use pypa/build to build a Python package from source"""
     built_wheel = build.run(
-        source_location, output_directory, convert_exports(exports), config_settings
+        source_location,
+        output_directory,
+        convert_exports(exports),
+        config_settings,
+        isolation=isolation,
+        skip_dependency_check=skip_dependency_check,
     )
     return built_wheel
 
@@ -164,6 +182,20 @@ def main(
     compression_level: int = typer.Option(
         6, help="Compression level to use for the created zip file"
     ),
+    no_isolation: bool = typer.Option(
+        False,
+        "--no-isolation",
+        "-n",
+        help="Disable building the project in an isolated virtual environment. "
+        "Build dependencies must be installed separately when this option is used",
+    ),
+    skip_dependency_check: bool = typer.Option(
+        False,
+        "--skip-dependency-check",
+        "-x",
+        help="Do not check that the build dependencies are installed. This option "
+        "is only useful when used with --no-isolation.",
+    ),
     config_setting: list[str] | None = typer.Option(
         None,
         "--config-setting",
@@ -232,6 +264,8 @@ def main(
                 # dependencies? Not sure this makes sense...
                 convert_exports(exports),
                 config_settings,
+                isolation=not no_isolation,
+                skip_dependency_check=skip_dependency_check,
                 output_lockfile=output_lockfile,
             )
         except BaseException as e:
@@ -247,17 +281,43 @@ def main(
             source_location = source_location[0 : source_location.find("[")]
     if not source_location:
         # build the current folder
-        wheel = source(Path.cwd(), outpath, exports, config_settings)
+        wheel = source(
+            Path.cwd(),
+            outpath,
+            exports,
+            config_settings,
+            isolation=not no_isolation,
+            skip_dependency_check=skip_dependency_check,
+        )
     elif source_location.find("://") != -1:
-        wheel = url(source_location, outpath, exports, config_settings)
+        wheel = url(
+            source_location,
+            outpath,
+            exports,
+            config_settings,
+            isolation=not no_isolation,
+            skip_dependency_check=skip_dependency_check,
+        )
     elif Path(source_location).is_dir():
         # a folder, build it
         wheel = source(
-            Path(source_location).resolve(), outpath, exports, config_settings
+            Path(source_location).resolve(),
+            outpath,
+            exports,
+            config_settings,
+            isolation=not no_isolation,
+            skip_dependency_check=skip_dependency_check,
         )
     elif source_location.find("/") == -1:
         # try fetch or build from pypi
-        wheel = pypi(source_location, outpath, exports, config_settings)
+        wheel = pypi(
+            source_location,
+            outpath,
+            exports,
+            config_settings,
+            isolation=not no_isolation,
+            skip_dependency_check=skip_dependency_check,
+        )
     else:
         raise RuntimeError(f"Couldn't determine source type for {source_location}")
     # now build deps for wheel
@@ -271,6 +331,8 @@ def main(
                 # dependencies? Not sure this makes sense...
                 convert_exports(exports),
                 config_settings,
+                isolation=not no_isolation,
+                skip_dependency_check=skip_dependency_check,
                 output_lockfile=output_lockfile,
                 compression_level=compression_level,
             )
diff --git a/pyodide_build/common.py b/pyodide_build/common.py
index a8f5d0e2..f75d7b73 100644
--- a/pyodide_build/common.py
+++ b/pyodide_build/common.py
@@ -383,6 +383,19 @@ def _get_sha256_checksum(archive: Path) -> str:
     return h.hexdigest()
 
 
+def _format_dep_chain(dep_chain: Sequence[str]) -> str:
+    return " -> ".join(dep.partition(";")[0].strip() for dep in dep_chain)
+
+
+def _format_missing_dependencies(missing: set[tuple[str, ...]]) -> str:
+    return "".join(
+        "\n\t" + dep
+        for deps in missing
+        for dep in (deps[0], _format_dep_chain(deps[1:]))
+        if dep
+    )
+
+
 def unpack_wheel(
     wheel_path: Path, target_dir: Path | None = None, verbose=True
 ) -> None:
diff --git a/pyodide_build/out_of_tree/build.py b/pyodide_build/out_of_tree/build.py
index 6d450849..8b1935e4 100644
--- a/pyodide_build/out_of_tree/build.py
+++ b/pyodide_build/out_of_tree/build.py
@@ -1,5 +1,6 @@
 import os
 from pathlib import Path
+from textwrap import dedent
 
 from build import ConfigSettingsType
 
@@ -8,11 +9,40 @@
 from pyodide_build.spec import _BuildSpecExports
 
 
+def _create_ignore_files(directory: Path) -> None:
+    directory.joinpath(".gitignore").write_text(
+        dedent("""\
+        # Created by pyodide-build
+        *
+    """),
+        encoding="utf-8",
+    )
+
+    directory.joinpath(".hgignore").write_text(
+        dedent("""\
+        # Created by pyodide-build
+        syntax: glob
+        **/*
+    """),
+        encoding="utf-8",
+    )
+
+
+def _prepare_build_dir(build_dir: Path) -> None:
+    # create a persistent build dir in the source dir
+    build_dir.mkdir(exist_ok=True)
+    # don't track the build dir in version control,
+    # helps if building in a git/mercurial repo
+    _create_ignore_files(build_dir)
+
+
 def run(
     srcdir: Path,
     outdir: Path,
     exports: _BuildSpecExports,
     config_settings: ConfigSettingsType,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
 ) -> Path:
     outdir = outdir.resolve()
     cflags = build_env.get_build_flag("SIDE_MODULE_CFLAGS")
@@ -28,6 +58,9 @@ def run(
     env = os.environ.copy()
     env.update(build_env.get_build_environment_vars(get_pyodide_root()))
 
+    build_dir = srcdir / ".pyodide_build"
+    _prepare_build_dir(build_dir)
+
     build_env_ctx = pypabuild.get_build_env(
         env=env,
         pkgname="",
@@ -36,10 +69,18 @@ def run(
         ldflags=ldflags,
         target_install_dir=target_install_dir,
         exports=exports,
+        build_dir=build_dir,
     )
 
     with build_env_ctx as env:
-        built_wheel = pypabuild.build(srcdir, outdir, env, config_settings)
+        built_wheel = pypabuild.build(
+            srcdir,
+            outdir,
+            env,
+            config_settings,
+            isolation=isolation,
+            skip_dependency_check=skip_dependency_check,
+        )
 
     wheel_path = Path(built_wheel)
     if "emscripten" in wheel_path.name:
diff --git a/pyodide_build/out_of_tree/pypi.py b/pyodide_build/out_of_tree/pypi.py
index a5750b1d..a4c58153 100644
--- a/pyodide_build/out_of_tree/pypi.py
+++ b/pyodide_build/out_of_tree/pypi.py
@@ -70,12 +70,12 @@ def stream_redirected(to=os.devnull, stream=None):
             to = None
 
 
-def get_built_wheel(url):
-    return _get_built_wheel_internal(url)["path"]
+def get_built_wheel(url, isolation=True, skip_dependency_check=False):
+    return _get_built_wheel_internal(url, isolation, skip_dependency_check)["path"]
 
 
 @cache
-def _get_built_wheel_internal(url):
+def _get_built_wheel_internal(url, isolation=True, skip_dependency_check=False):
     parsed_url = urlparse(url)
     gz_name = Path(parsed_url.path).name
 
@@ -107,6 +107,8 @@ def _get_built_wheel_internal(url):
                 build_path / "dist",
                 PyPIProvider.BUILD_EXPORTS,
                 PyPIProvider.BUILD_FLAGS,
+                isolation=isolation,
+                skip_dependency_check=skip_dependency_check,
             )
         except BaseException as e:
             logger.error(" Failed\n Error is:")
@@ -199,11 +201,15 @@ def get_project_from_pypi(package_name, extras):
 
 
 def download_or_build_wheel(
-    url: str, target_directory: Path, compression_level: int = 6
+    url: str,
+    target_directory: Path,
+    compression_level: int = 6,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
 ) -> None:
     parsed_url = urlparse(url)
     if parsed_url.path.endswith("gz"):
-        wheel_file = get_built_wheel(url)
+        wheel_file = get_built_wheel(url, isolation, skip_dependency_check)
         shutil.copy(wheel_file, target_directory)
         wheel_path = target_directory / wheel_file.name
     elif parsed_url.path.endswith(".whl"):
@@ -333,6 +339,8 @@ def _resolve_and_build(
     build_dependencies: bool,
     extras: list[str],
     output_lockfile: str | None,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
     compression_level: int = 6,
 ) -> None:
     requirements = []
@@ -362,7 +370,7 @@ def _resolve_and_build(
     if output_lockfile is not None and len(output_lockfile) > 0:
         version_file = open(output_lockfile, "w")
     for x in result.mapping.values():
-        download_or_build_wheel(x.url, target_folder)
+        download_or_build_wheel(x.url, target_folder, compression_level)
         if len(x.extras) > 0:
             extratxt = "[" + ",".join(x.extras) + "]"
         else:
@@ -380,7 +388,10 @@ def build_wheels_from_pypi_requirements(
     skip_dependency: list[str],
     exports: _BuildSpecExports,
     config_settings: ConfigSettingsType,
-    output_lockfile: str | None,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
+    output_lockfile: str | None = None,
+    compression_level: int = 6,
 ) -> None:
     """
     Given a list of package requirements, build or fetch them. If build_dependencies is true, then
@@ -395,6 +406,9 @@ def build_wheels_from_pypi_requirements(
         build_dependencies,
         extras=[],
         output_lockfile=output_lockfile,
+        isolation=isolation,
+        skip_dependency_check=skip_dependency_check,
+        compression_level=compression_level,
     )
 
 
@@ -404,7 +418,9 @@ def build_dependencies_for_wheel(
     skip_dependency: list[str],
     exports: _BuildSpecExports,
     config_settings: ConfigSettingsType,
-    output_lockfile: str | None,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
+    output_lockfile: str | None = None,
     compression_level: int = 6,
 ) -> None:
     """Extract dependencies from this wheel and build pypi dependencies
@@ -435,6 +451,8 @@ def build_dependencies_for_wheel(
         build_dependencies=True,
         extras=extras,
         output_lockfile=output_lockfile,
+        isolation=isolation,
+        skip_dependency_check=skip_dependency_check,
         compression_level=compression_level,
     )
     # add the current wheel to the package-versions.txt
diff --git a/pyodide_build/pypabuild.py b/pyodide_build/pypabuild.py
index 392b0c09..3a639722 100644
--- a/pyodide_build/pypabuild.py
+++ b/pyodide_build/pypabuild.py
@@ -8,7 +8,6 @@
 from contextlib import contextmanager
 from itertools import chain
 from pathlib import Path
-from tempfile import TemporaryDirectory
 from typing import Literal, cast
 
 from build import BuildBackendException, ConfigSettingsType
@@ -54,7 +53,7 @@
 
 def _gen_runner(
     cross_build_env: Mapping[str, str],
-    isolated_build_env: _DefaultIsolatedEnv,
+    isolated_build_env: _DefaultIsolatedEnv = None,
 ) -> Callable[[Sequence[str], str | None, Mapping[str, str] | None], None]:
     """
     This returns a slightly modified version of default subprocess runner that pypa/build uses.
@@ -79,7 +78,15 @@ def _runner(cmd, cwd=None, extra_environ=None):
 
         # Some build dependencies like cmake, meson installs binaries to this directory
         # and we should add it to the PATH so that they can be found.
-        env["BUILD_ENV_SCRIPTS_DIR"] = isolated_build_env.scripts_dir
+        if isolated_build_env:
+            env["BUILD_ENV_SCRIPTS_DIR"] = isolated_build_env.scripts_dir
+        else:
+            # For non-isolated builds, set a fallback path or use the current Python path
+            import sysconfig
+
+            scripts_dir = sysconfig.get_path("scripts")
+            env["BUILD_ENV_SCRIPTS_DIR"] = scripts_dir
+
         env["PATH"] = f"{cross_build_env['COMPILER_WRAPPER_DIR']}:{env['PATH']}"
         # For debugging: Uncomment the following line to print the build command
         # print("Build backend call:", " ".join(str(x) for x in cmd), file=sys.stderr)
@@ -191,6 +198,30 @@ def _build_in_isolated_env(
             )
 
 
+def _build_in_current_env(
+    build_env: Mapping[str, str],
+    srcdir: Path,
+    outdir: str,
+    distribution: Literal["sdist", "wheel"],
+    config_settings: ConfigSettingsType,
+    skip_dependency_check: bool = False,
+) -> str:
+    with common.replace_env(build_env):
+        builder = _ProjectBuilder(srcdir, runner=_gen_runner(build_env))
+
+        if not skip_dependency_check:
+            missing = builder.check_dependencies(distribution, config_settings or {})
+            if missing:
+                dependencies = common._format_missing_dependencies(missing)
+                _error(f"Missing dependencies: {dependencies}")
+
+        return builder.build(
+            distribution,
+            outdir,
+            config_settings,
+        )
+
+
 def parse_backend_flags(backend_flags: str | list[str]) -> ConfigSettingsType:
     config_settings: dict[str, str | list[str]] = {}
 
@@ -253,21 +284,17 @@ def make_command_wrapper_symlinks(symlink_dir: Path) -> dict[str, str]:
     return env
 
 
+# TODO: a context manager is no longer needed here
 @contextmanager
-def _create_symlink_dir(env: dict[str, str], build_dir: Path | None):
-    if build_dir:
-        # If we're running under build-recipes, leave the symlinks in
-        # the build directory. This helps with reproducing.
-        symlink_dir = build_dir / "pywasmcross_symlinks"
-        shutil.rmtree(symlink_dir, ignore_errors=True)
-        symlink_dir.mkdir()
-        yield symlink_dir
-        return
-
-    # Running from "pyodide build". Put symlinks in a temporary directory.
-    # TODO: Add a debug option to save the symlinks.
-    with TemporaryDirectory() as symlink_dir_str:
-        yield Path(symlink_dir_str)
+def _create_symlink_dir(
+    env: dict[str, str],
+    build_dir: Path,
+):
+    # Leave the symlinks in the build directory. This helps with reproducing.
+    symlink_dir = build_dir / "pywasmcross_symlinks"
+    shutil.rmtree(symlink_dir, ignore_errors=True)
+    symlink_dir.mkdir()
+    yield symlink_dir
 
 
 @contextmanager
@@ -281,6 +308,7 @@ def get_build_env(
     target_install_dir: str,
     exports: _BuildSpecExports,
     build_dir: Path | None = None,
+    no_isolation: bool = False,
 ) -> Iterator[dict[str, str]]:
     """
     Returns a dict of environment variables that should be used when building
@@ -328,12 +356,24 @@ def build(
     outdir: Path,
     build_env: Mapping[str, str],
     config_settings: ConfigSettingsType,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
 ) -> str:
     try:
         with _handle_build_error():
-            built = _build_in_isolated_env(
-                build_env, srcdir, str(outdir), "wheel", config_settings
-            )
+            if isolation:
+                built = _build_in_isolated_env(
+                    build_env, srcdir, str(outdir), "wheel", config_settings
+                )
+            else:
+                built = _build_in_current_env(
+                    build_env,
+                    srcdir,
+                    str(outdir),
+                    "wheel",
+                    config_settings,
+                    skip_dependency_check,
+                )
             print("{bold}{green}Successfully built {}{reset}".format(built, **_STYLES))
             return built
     except Exception as e:  # pragma: no cover
diff --git a/pyodide_build/pywasmcross.py b/pyodide_build/pywasmcross.py
index a766b15d..13b2ee64 100755
--- a/pyodide_build/pywasmcross.py
+++ b/pyodide_build/pywasmcross.py
@@ -521,10 +521,6 @@ def handle_command_generate_args(  # noqa: C901
             "emcmake",
             "cmake",
             *flags,
-            # Since we create a temporary directory and install compiler symlinks every time,
-            # CMakeCache.txt will contain invalid paths to the compiler when re-running,
-            # so we need to tell CMake to ignore the existing cache and build from scratch.
-            "--fresh",
         ]
         return line
     elif cmd == "meson":
diff --git a/pyodide_build/tests/test_cli.py b/pyodide_build/tests/test_cli.py
index 09e534a1..b3a86873 100644
--- a/pyodide_build/tests/test_cli.py
+++ b/pyodide_build/tests/test_cli.py
@@ -355,7 +355,14 @@ def test_py_compile(tmp_path, target, compression_level):
 def test_build1(tmp_path, monkeypatch, dummy_xbuildenv, mock_emscripten):
     from pyodide_build import pypabuild
 
-    def mocked_build(srcdir: Path, outdir: Path, env: Any, backend_flags: Any) -> str:
+    def mocked_build(
+        srcdir: Path,
+        outdir: Path,
+        env: Any,
+        backend_flags: Any,
+        isolation=True,
+        skip_dependency_check=False,
+    ) -> str:
         results["srcdir"] = srcdir
         results["outdir"] = outdir
         results["backend_flags"] = backend_flags
@@ -431,7 +438,14 @@ def unpack_archive_shim(*args):
 
     exports_ = None
 
-    def run_shim(builddir, output_directory, exports, backend_flags):
+    def run_shim(
+        builddir,
+        output_directory,
+        exports,
+        backend_flags,
+        isolation=True,
+        skip_dependency_check=False,
+    ):
         nonlocal exports_
         exports_ = exports
 
@@ -491,7 +505,14 @@ def test_build_config_settings(monkeypatch, dummy_xbuildenv):
 
     config_settings_passed = None
 
-    def run(srcdir, outdir, exports, config_settings):
+    def run(
+        srcdir,
+        outdir,
+        exports,
+        config_settings,
+        isolation=True,
+        skip_dependency_check=False,
+    ):
         nonlocal config_settings_passed
         config_settings_passed = config_settings
 
@@ -657,3 +678,201 @@ def test_build_constraint(tmp_path, dummy_xbuildenv, mock_emscripten, capsys):
     build_dir = RECIPE_DIR / pkg / "build"
     assert (build_dir / "setuptools.version").read_text() == "74.1.3"
     assert (build_dir / "pytest.version").read_text() == "7.0.0"
+
+
+@pytest.mark.parametrize(
+    "isolation_flag",
+    [
+        None,
+        "--no-isolation",
+    ],
+)
+def test_build_isolation_flags(
+    tmp_path, monkeypatch, dummy_xbuildenv, mock_emscripten, isolation_flag
+):
+    """Test that build works with different isolation flags."""
+    from pyodide_build import pypabuild
+
+    build_calls = []
+
+    def mocked_build(
+        srcdir,
+        outdir,
+        env,
+        config_settings,
+        isolation=True,
+        skip_dependency_check=False,
+    ):
+        build_calls.append(
+            {
+                "srcdir": srcdir,
+                "isolation": isolation,
+                "skip_dependency_check": skip_dependency_check,
+            }
+        )
+        dummy_wheel = outdir / "package-1.0.0-py3-none-any.whl"
+        return str(dummy_wheel)
+
+    monkeypatch.setattr(pypabuild, "build", mocked_build)
+    monkeypatch.setattr(build_env, "check_emscripten_version", lambda: None)
+    monkeypatch.setattr(build_env, "replace_so_abi_tags", lambda whl: None)
+    monkeypatch.setattr(
+        common, "retag_wheel", lambda wheel_path, platform: Path(wheel_path)
+    )
+
+    from contextlib import nullcontext
+
+    monkeypatch.setattr(common, "modify_wheel", lambda whl: nullcontext())
+
+    app = typer.Typer()
+    app.command(**build.main.typer_kwargs)(build.main)  # type:ignore[attr-defined]
+
+    srcdir = tmp_path / "in"
+    outdir = tmp_path / "out"
+    srcdir.mkdir()
+
+    args = [str(srcdir), "--outdir", str(outdir)]
+    if isolation_flag:
+        args.append(isolation_flag)
+
+    result = runner.invoke(app, args)
+
+    assert result.exit_code == 0, result.stdout
+    assert len(build_calls) == 1
+
+    # Check that isolation was properly passed
+    expected_isolation = isolation_flag is None
+    assert build_calls[0]["isolation"] == expected_isolation
+
+
+@pytest.mark.parametrize(
+    "skip_check_flag",
+    [
+        None,  # Default: with dependency check
+        "--skip-dependency-check",
+        "-x",
+    ],
+)
+def test_build_skip_dependency_check(
+    tmp_path, monkeypatch, dummy_xbuildenv, mock_emscripten, skip_check_flag
+):
+    """Test that build works with different skip dependency check flags."""
+    from pyodide_build import pypabuild
+
+    build_calls = []
+
+    def mocked_build(
+        srcdir,
+        outdir,
+        env,
+        config_settings,
+        isolation=True,
+        skip_dependency_check=False,
+    ):
+        build_calls.append(
+            {
+                "srcdir": srcdir,
+                "isolation": isolation,
+                "skip_dependency_check": skip_dependency_check,
+            }
+        )
+        dummy_wheel = outdir / "package-1.0.0-py3-none-any.whl"
+        return str(dummy_wheel)
+
+    monkeypatch.setattr(pypabuild, "build", mocked_build)
+    monkeypatch.setattr(build_env, "check_emscripten_version", lambda: None)
+    monkeypatch.setattr(build_env, "replace_so_abi_tags", lambda whl: None)
+    monkeypatch.setattr(
+        common, "retag_wheel", lambda wheel_path, platform: Path(wheel_path)
+    )
+
+    from contextlib import nullcontext
+
+    monkeypatch.setattr(common, "modify_wheel", lambda whl: nullcontext())
+
+    app = typer.Typer()
+    app.command(**build.main.typer_kwargs)(build.main)  # type:ignore[attr-defined]
+
+    srcdir = tmp_path / "in"
+    outdir = tmp_path / "out"
+    srcdir.mkdir()
+
+    args = [str(srcdir), "--outdir", str(outdir)]
+    if skip_check_flag:
+        args.append(skip_check_flag)
+
+    result = runner.invoke(app, args)
+
+    assert result.exit_code == 0, result.stdout
+    assert len(build_calls) == 1
+
+    # Check that skip_dependency_check was properly passed
+    expected_skip = skip_check_flag is not None
+    assert build_calls[0]["skip_dependency_check"] == expected_skip
+
+
+@pytest.mark.parametrize(
+    "isolation,skip_check",
+    [
+        (True, False),  # default: with isolation, without skipping dependency checking
+        (False, False),
+        (True, True),
+        (False, True),
+    ],
+)
+def test_build_combined_flags(
+    tmp_path, monkeypatch, dummy_xbuildenv, mock_emscripten, isolation, skip_check
+):
+    """Test combinations of isolation and skip dependency check flags."""
+    from pyodide_build import pypabuild
+
+    build_calls = []
+
+    def mocked_build(
+        srcdir,
+        outdir,
+        env,
+        config_settings,
+        isolation=True,
+        skip_dependency_check=False,
+    ):
+        build_calls.append(
+            {
+                "srcdir": srcdir,
+                "isolation": isolation,
+                "skip_dependency_check": skip_dependency_check,
+            }
+        )
+        dummy_wheel = outdir / "package-1.0.0-py3-none-any.whl"
+        return str(dummy_wheel)
+
+    monkeypatch.setattr(pypabuild, "build", mocked_build)
+    monkeypatch.setattr(build_env, "check_emscripten_version", lambda: None)
+    monkeypatch.setattr(build_env, "replace_so_abi_tags", lambda whl: None)
+    monkeypatch.setattr(
+        common, "retag_wheel", lambda wheel_path, platform: Path(wheel_path)
+    )
+
+    from contextlib import nullcontext
+
+    monkeypatch.setattr(common, "modify_wheel", lambda whl: nullcontext())
+
+    app = typer.Typer()
+    app.command(**build.main.typer_kwargs)(build.main)  # type:ignore[attr-defined]
+
+    srcdir = tmp_path / "in"
+    outdir = tmp_path / "out"
+    srcdir.mkdir()
+
+    args = [str(srcdir), "--outdir", str(outdir)]
+    if not isolation:
+        args.append("--no-isolation")
+    if skip_check:
+        args.append("--skip-dependency-check")
+
+    result = runner.invoke(app, args)
+
+    assert result.exit_code == 0, result.stdout
+    assert len(build_calls) == 1
+    assert build_calls[0]["isolation"] == isolation
+    assert build_calls[0]["skip_dependency_check"] == skip_check
diff --git a/pyodide_build/tests/test_pypabuild.py b/pyodide_build/tests/test_pypabuild.py
index 02bab233..5c6af469 100644
--- a/pyodide_build/tests/test_pypabuild.py
+++ b/pyodide_build/tests/test_pypabuild.py
@@ -77,6 +77,7 @@ def test_get_build_env(tmp_path, dummy_xbuildenv):
         ldflags="",
         target_install_dir=str(tmp_path),
         exports="pyinit",
+        build_dir=tmp_path,
     )
 
     with build_env_ctx as env:
diff --git a/pyodide_build/tests/test_pywasmcross.py b/pyodide_build/tests/test_pywasmcross.py
index 6b7381c2..f6bf9a90 100644
--- a/pyodide_build/tests/test_pywasmcross.py
+++ b/pyodide_build/tests/test_pywasmcross.py
@@ -257,7 +257,6 @@ def test_get_cmake_compiler_flags():
 
 def test_handle_command_cmake(build_args):
     args = build_args
-    assert "--fresh" in handle_command_generate_args(["cmake", "./"], args)
 
     build_cmd = ["cmake", "--build", "." "--target", "target"]
     assert handle_command_generate_args(build_cmd, args) == build_cmd


Reviews:
user: login: ryanking13
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-30T06:22:38Z
user: login: ryanking13
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-30T06:41:12Z
user: login: hoodmane
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-30T09:09:16Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-30T23:00:00Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-30T23:13:07Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-30T23:22:53Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T00:32:19Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T00:32:58Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T00:33:25Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T00:33:41Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T00:34:02Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T00:34:17Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T00:35:16Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T02:34:15Z
user: login: ryanking13
body: I left some minor comments, but it looks good to me overall.

> I was able to compile NumPy in 77 seconds on my machine, and taking advantage of the build caching through the persistence of the build dir, numpy-tests compiles in 8 seconds.

Nice!

BTW, some of the Google's Bazel-based packages were relying on no-isolated builds, such as JAX. There were a lot of other build issues as well, but I think would make one step closer to supporting those packages. Thank you!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-05T05:21:49Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-05T11:36:20Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-05T11:48:20Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-05T11:49:18Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-05T11:50:44Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-05T12:44:38Z
user: login: agriyakhetarpal
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-05T12:46:48Z

Review comments:
diff_hunk: @@ -23,6 +23,19 @@ test-src: check
 
 	@echo "... Passed"
 
+.PHONY: test-src-no-isolation
+test-src-no-isolation: check
+	@echo "... Running integration tests for building src with --no-isolation --skip-dependency-check"
+
+	# Ensure the necessary build dependencies are installed in the
+	# host environment as we need to do so ourselves.
+
+	pip install meson-python meson cython
path: integration_tests/Makefile
user: login: ryanking13
body: How about installing these dependencies in a virtualenv to prevent polluting the host environment?
created_at: 2025-03-30T06:02:35Z
author_association: MEMBER
diff_hunk: @@ -2,6 +2,17 @@
 
 set -e
 
+NO_ISOLATION=""
+SKIP_DEPS=""
+
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --no-isolation) NO_ISOLATION="--no-isolation"; shift ;;
+        --skip-dependency-check) SKIP_DEPS="--skip-dependency-check"; shift ;;
+        *) echo "Unknown parameter: $1"; exit 1 ;;
+    esac
+done
path: integration_tests/src/numpy.sh
user: login: ryanking13
body: Maybe we can just forward `$@` to `pyodide build`, instead of handling these parameters separately?
created_at: 2025-03-30T06:03:51Z
author_association: MEMBER
diff_hunk: @@ -5,6 +5,16 @@
 import sys
 import traceback
 from collections.abc import Callable, Iterator, Mapping, Sequence
+
+
+# A helper function from pypa/build/__main__.py since
+# it's not in the vendorized code we have
+# TODO: we should move this to a new file. it's out of place
+# between the other imports ;-)
+def _format_dep_chain(dep_chain: Sequence[str]) -> str:
+    return " -> ".join(dep.partition(";")[0].strip() for dep in dep_chain)
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: Maybe you can put it in `common.py`.

We need to reorganize common.py too, as it contains too much code anyway.
created_at: 2025-03-30T06:16:47Z
author_association: MEMBER
diff_hunk: @@ -164,6 +182,19 @@ def main(
     compression_level: int = typer.Option(
         6, help="Compression level to use for the created zip file"
     ),
+    no_isolation: bool = typer.Option(
+        False,
+        "--no-isolation",
+        "-n",
+        help="Disable building the project in an isolated virtual environment. "
+        "Build dependencies must be installed separately when this option is used",
+    ),
+    skip_dependency_check: bool = typer.Option(
+        False,
+        "--skip-dependency-check",
+        "-x",
+        help="Do not check that build dependencies are installed",
path: pyodide_build/cli/build.py
user: login: ryanking13
body: Could you please mention that it is no-op when it is not used with `--no-isolation` IIUC?
created_at: 2025-03-30T06:18:36Z
author_association: MEMBER
diff_hunk: @@ -79,7 +89,15 @@ def _runner(cmd, cwd=None, extra_environ=None):
 
         # Some build dependencies like cmake, meson installs binaries to this directory
         # and we should add it to the PATH so that they can be found.
-        env["BUILD_ENV_SCRIPTS_DIR"] = isolated_build_env.scripts_dir
+        if isolated_build_env:
+            env["BUILD_ENV_SCRIPTS_DIR"] = isolated_build_env.scripts_dir
+        else:
+            # For non-isolated builds, set a fallback path or use the current Python path
+            import sysconfig
+
+            scripts_dir = sysconfig.get_path("scripts")
+            env["BUILD_ENV_SCRIPTS_DIR"] = scripts_dir
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: I am curious if this is adapted from pypa/build.

created_at: 2025-03-30T06:20:30Z
author_association: MEMBER
diff_hunk: @@ -264,10 +323,22 @@ def _create_symlink_dir(env: dict[str, str], build_dir: Path | None):
         yield symlink_dir
         return
 
-    # Running from "pyodide build". Put symlinks in a temporary directory.
-    # TODO: Add a debug option to save the symlinks.
-    with TemporaryDirectory() as symlink_dir_str:
-        yield Path(symlink_dir_str)
+    # TODO: FIXME: compiler wrappers are still ending up in a temporary
+    # directory, which breaks persistent builds. This is not ideal, but
+    # it is better than nothing so this is non-blocking for now. It has
+    # to be investigated further.
+
+    if no_isolation:
+        # For non-isolated builds, create a persistent directory in the current working directory
+        # or in a well-known location like ~/.pyodide/compiler_wrappers
+        symlink_dir = Path.cwd() / ".pyodide_compiler_wrappers"
+        symlink_dir.mkdir(exist_ok=True)
+        yield symlink_dir
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: I don't think we need to branch out this part. I gues we can pass the persistent directory for both no-isolated and isolated build (as I commented in https://github.com/pyodide/pyodide-build/pull/152#issuecomment-2764406264)
created_at: 2025-03-30T06:22:23Z
author_association: MEMBER
diff_hunk: @@ -404,7 +418,9 @@ def build_dependencies_for_wheel(
     skip_dependency: list[str],
     exports: _BuildSpecExports,
     config_settings: ConfigSettingsType,
-    output_lockfile: str | None,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
path: pyodide_build/out_of_tree/pypi.py
user: login: ryanking13
body: (No action is required for this PR)

It may be weird to build all the dependencies in a no-isolation mode. But let's go as-is for now. I personally want to deprecate the feature to build the dependencies if package maintainers in Python ecosystem starts to build Pyodide wheel themselves.
created_at: 2025-03-30T06:25:16Z
author_association: MEMBER
diff_hunk: @@ -191,6 +209,45 @@ def _build_in_isolated_env(
             )
 
 
+def _build_in_current_env(
+    build_env: Mapping[str, str],
+    srcdir: Path,
+    outdir: str,
+    distribution: Literal["sdist", "wheel"],
+    config_settings: ConfigSettingsType,
+    skip_dependency_check: bool = False,
+) -> str:
+    with common.replace_env(build_env):
+        # Setup sysconfigdata path in environment
+        sysconfigdata_name = get_build_flag("SYSCONFIG_NAME")
+        sysconfig_dir = Path(get_build_flag("TARGETINSTALLDIR")) / "sysconfigdata"
+        env = os.environ.copy()
+        env["_PYTHON_SYSCONFIGDATA_NAME"] = sysconfigdata_name
+        env["PYTHONPATH"] = f"{str(sysconfig_dir)}:{env.get('PYTHONPATH', '')}"
+
+        with common.replace_env(env):
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: I guess `build_env` created from `get_build_env` would already have these (`_PYTHON_SYSCONFIGDATA_NAME`, `PYTHONPATH`) values in the map. Why does this need to re-calculated?
created_at: 2025-03-30T06:30:24Z
author_association: MEMBER
diff_hunk: @@ -191,6 +209,45 @@ def _build_in_isolated_env(
             )
 
 
+def _build_in_current_env(
+    build_env: Mapping[str, str],
+    srcdir: Path,
+    outdir: str,
+    distribution: Literal["sdist", "wheel"],
+    config_settings: ConfigSettingsType,
+    skip_dependency_check: bool = False,
+) -> str:
+    with common.replace_env(build_env):
+        # Setup sysconfigdata path in environment
+        sysconfigdata_name = get_build_flag("SYSCONFIG_NAME")
+        sysconfig_dir = Path(get_build_flag("TARGETINSTALLDIR")) / "sysconfigdata"
+        env = os.environ.copy()
+        env["_PYTHON_SYSCONFIGDATA_NAME"] = sysconfigdata_name
+        env["PYTHONPATH"] = f"{str(sysconfig_dir)}:{env.get('PYTHONPATH', '')}"
+
+        with common.replace_env(env):
+            builder = _ProjectBuilder(srcdir, runner=_gen_runner(build_env))
+
+            if not skip_dependency_check:
+                missing = builder.check_dependencies(
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: 👍 

[`check_dependencies`](https://github.com/pypa/build/blob/811c9db0dcac14b7b32f08ee57d09af3666979d4/src/build/_builder.py#L242) looks like a union of

1. get_requires_for_build (== dependencies for the build backend such as setuptools, meson)
2. build_system_requires (== dependencies in the pyproject.toml)

In the isolated build, we've been [dealing with them separately](https://github.com/pyodide/pyodide-build/blob/71aa0b5ae55b196c4b3444434c67a9541dd382c9/pyodide_build/pypabuild.py#L160-L167). I guess we can use `check_dependencies` there too (no action required for this PR).
created_at: 2025-03-30T06:37:16Z
author_association: MEMBER
diff_hunk: @@ -191,6 +209,45 @@ def _build_in_isolated_env(
             )
 
 
+def _build_in_current_env(
+    build_env: Mapping[str, str],
+    srcdir: Path,
+    outdir: str,
+    distribution: Literal["sdist", "wheel"],
+    config_settings: ConfigSettingsType,
+    skip_dependency_check: bool = False,
+) -> str:
+    with common.replace_env(build_env):
+        # Setup sysconfigdata path in environment
+        sysconfigdata_name = get_build_flag("SYSCONFIG_NAME")
+        sysconfig_dir = Path(get_build_flag("TARGETINSTALLDIR")) / "sysconfigdata"
+        env = os.environ.copy()
+        env["_PYTHON_SYSCONFIGDATA_NAME"] = sysconfigdata_name
+        env["PYTHONPATH"] = f"{str(sysconfig_dir)}:{env.get('PYTHONPATH', '')}"
+
+        with common.replace_env(env):
+            builder = _ProjectBuilder(srcdir, runner=_gen_runner(build_env))
+
+            if not skip_dependency_check:
+                missing = builder.check_dependencies(
+                    distribution, config_settings or {}
+                )
+                if missing:
+                    dependencies = "".join(
+                        "\n\t" + dep
+                        for deps in missing
+                        for dep in (deps[0], _format_dep_chain(deps[1:]))
+                        if dep
+                    )
+                    _error(f"Missing dependencies:{dependencies}")
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: How about putting all these in a single function?
created_at: 2025-03-30T06:40:28Z
author_association: MEMBER
diff_hunk: @@ -404,7 +418,9 @@ def build_dependencies_for_wheel(
     skip_dependency: list[str],
     exports: _BuildSpecExports,
     config_settings: ConfigSettingsType,
-    output_lockfile: str | None,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
path: pyodide_build/out_of_tree/pypi.py
user: login: hoodmane
body: Joe Marshall added this and we were always a bit unsure if it made sense as a feature.
created_at: 2025-03-30T09:09:15Z
author_association: MEMBER
diff_hunk: @@ -404,7 +418,9 @@ def build_dependencies_for_wheel(
     skip_dependency: list[str],
     exports: _BuildSpecExports,
     config_settings: ConfigSettingsType,
-    output_lockfile: str | None,
+    isolation: bool = True,
+    skip_dependency_check: bool = False,
path: pyodide_build/out_of_tree/pypi.py
user: login: agriyakhetarpal
body: Yes, I added these extra args to normalise what passes through these functions, even if unused. I agree that it makes sense to drop it entirely. I don't think I've used it personally, and I I haven't seen it being used in the main Pyodide repo.
created_at: 2025-03-30T23:00:00Z
author_association: MEMBER
diff_hunk: @@ -191,6 +209,45 @@ def _build_in_isolated_env(
             )
 
 
+def _build_in_current_env(
+    build_env: Mapping[str, str],
+    srcdir: Path,
+    outdir: str,
+    distribution: Literal["sdist", "wheel"],
+    config_settings: ConfigSettingsType,
+    skip_dependency_check: bool = False,
+) -> str:
+    with common.replace_env(build_env):
+        # Setup sysconfigdata path in environment
+        sysconfigdata_name = get_build_flag("SYSCONFIG_NAME")
+        sysconfig_dir = Path(get_build_flag("TARGETINSTALLDIR")) / "sysconfigdata"
+        env = os.environ.copy()
+        env["_PYTHON_SYSCONFIGDATA_NAME"] = sysconfigdata_name
+        env["PYTHONPATH"] = f"{str(sysconfig_dir)}:{env.get('PYTHONPATH', '')}"
+
+        with common.replace_env(env):
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: > Why does this need to re-calculated?

It doesn't; this was an oversight. Thanks for the catch!
created_at: 2025-03-30T23:13:07Z
author_association: MEMBER
diff_hunk: @@ -79,7 +89,15 @@ def _runner(cmd, cwd=None, extra_environ=None):
 
         # Some build dependencies like cmake, meson installs binaries to this directory
         # and we should add it to the PATH so that they can be found.
-        env["BUILD_ENV_SCRIPTS_DIR"] = isolated_build_env.scripts_dir
+        if isolated_build_env:
+            env["BUILD_ENV_SCRIPTS_DIR"] = isolated_build_env.scripts_dir
+        else:
+            # For non-isolated builds, set a fallback path or use the current Python path
+            import sysconfig
+
+            scripts_dir = sysconfig.get_path("scripts")
+            env["BUILD_ENV_SCRIPTS_DIR"] = scripts_dir
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: Yes, sort of. `pypa/build` has a more robust check for where the `bin/` dir is usually placed. It's here: https://github.com/pypa/build/blob/811c9db0dcac14b7b32f08ee57d09af3666979d4/src/build/env.py#L336-L376. I used a subset of it through `sysconfig.get_path("scripts")`, as we don't need to support all sorts of Python distributions that might be packaged differently.
created_at: 2025-03-30T23:22:53Z
author_association: MEMBER
diff_hunk: @@ -191,6 +209,45 @@ def _build_in_isolated_env(
             )
 
 
+def _build_in_current_env(
+    build_env: Mapping[str, str],
+    srcdir: Path,
+    outdir: str,
+    distribution: Literal["sdist", "wheel"],
+    config_settings: ConfigSettingsType,
+    skip_dependency_check: bool = False,
+) -> str:
+    with common.replace_env(build_env):
+        # Setup sysconfigdata path in environment
+        sysconfigdata_name = get_build_flag("SYSCONFIG_NAME")
+        sysconfig_dir = Path(get_build_flag("TARGETINSTALLDIR")) / "sysconfigdata"
+        env = os.environ.copy()
+        env["_PYTHON_SYSCONFIGDATA_NAME"] = sysconfigdata_name
+        env["PYTHONPATH"] = f"{str(sysconfig_dir)}:{env.get('PYTHONPATH', '')}"
+
+        with common.replace_env(env):
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: Addressed in 8530171ac2f30bbb18099089382cffb89ca7c51d.
created_at: 2025-03-31T00:32:19Z
author_association: MEMBER
diff_hunk: @@ -191,6 +209,45 @@ def _build_in_isolated_env(
             )
 
 
+def _build_in_current_env(
+    build_env: Mapping[str, str],
+    srcdir: Path,
+    outdir: str,
+    distribution: Literal["sdist", "wheel"],
+    config_settings: ConfigSettingsType,
+    skip_dependency_check: bool = False,
+) -> str:
+    with common.replace_env(build_env):
+        # Setup sysconfigdata path in environment
+        sysconfigdata_name = get_build_flag("SYSCONFIG_NAME")
+        sysconfig_dir = Path(get_build_flag("TARGETINSTALLDIR")) / "sysconfigdata"
+        env = os.environ.copy()
+        env["_PYTHON_SYSCONFIGDATA_NAME"] = sysconfigdata_name
+        env["PYTHONPATH"] = f"{str(sysconfig_dir)}:{env.get('PYTHONPATH', '')}"
+
+        with common.replace_env(env):
+            builder = _ProjectBuilder(srcdir, runner=_gen_runner(build_env))
+
+            if not skip_dependency_check:
+                missing = builder.check_dependencies(
+                    distribution, config_settings or {}
+                )
+                if missing:
+                    dependencies = "".join(
+                        "\n\t" + dep
+                        for deps in missing
+                        for dep in (deps[0], _format_dep_chain(deps[1:]))
+                        if dep
+                    )
+                    _error(f"Missing dependencies:{dependencies}")
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: Yes, makes sense. Addressed with 0aed06b6b02ba02dd08d9686a7c00d4e54437129 as I moved them to `pyodide_build.common`.
created_at: 2025-03-31T00:32:58Z
author_association: MEMBER
diff_hunk: @@ -264,10 +323,22 @@ def _create_symlink_dir(env: dict[str, str], build_dir: Path | None):
         yield symlink_dir
         return
 
-    # Running from "pyodide build". Put symlinks in a temporary directory.
-    # TODO: Add a debug option to save the symlinks.
-    with TemporaryDirectory() as symlink_dir_str:
-        yield Path(symlink_dir_str)
+    # TODO: FIXME: compiler wrappers are still ending up in a temporary
+    # directory, which breaks persistent builds. This is not ideal, but
+    # it is better than nothing so this is non-blocking for now. It has
+    # to be investigated further.
+
+    if no_isolation:
+        # For non-isolated builds, create a persistent directory in the current working directory
+        # or in a well-known location like ~/.pyodide/compiler_wrappers
+        symlink_dir = Path.cwd() / ".pyodide_compiler_wrappers"
+        symlink_dir.mkdir(exist_ok=True)
+        yield symlink_dir
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: Thanks for the pointer. I addressed this in 8325e803d78cc3e3833edceeb652aee2e7974891.
created_at: 2025-03-31T00:33:24Z
author_association: MEMBER
diff_hunk: @@ -164,6 +182,19 @@ def main(
     compression_level: int = typer.Option(
         6, help="Compression level to use for the created zip file"
     ),
+    no_isolation: bool = typer.Option(
+        False,
+        "--no-isolation",
+        "-n",
+        help="Disable building the project in an isolated virtual environment. "
+        "Build dependencies must be installed separately when this option is used",
+    ),
+    skip_dependency_check: bool = typer.Option(
+        False,
+        "--skip-dependency-check",
+        "-x",
+        help="Do not check that build dependencies are installed",
path: pyodide_build/cli/build.py
user: login: agriyakhetarpal
body: Addressed in 6c5da50434a00862e3e7d732504d62780fc6c3f8.
created_at: 2025-03-31T00:33:41Z
author_association: MEMBER
diff_hunk: @@ -5,6 +5,16 @@
 import sys
 import traceback
 from collections.abc import Callable, Iterator, Mapping, Sequence
+
+
+# A helper function from pypa/build/__main__.py since
+# it's not in the vendorized code we have
+# TODO: we should move this to a new file. it's out of place
+# between the other imports ;-)
+def _format_dep_chain(dep_chain: Sequence[str]) -> str:
+    return " -> ".join(dep.partition(";")[0].strip() for dep in dep_chain)
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: I agree. For the time being, I addressed this in 0aed06b6b02ba02dd08d9686a7c00d4e54437129.
created_at: 2025-03-31T00:34:02Z
author_association: MEMBER
diff_hunk: @@ -2,6 +2,17 @@
 
 set -e
 
+NO_ISOLATION=""
+SKIP_DEPS=""
+
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --no-isolation) NO_ISOLATION="--no-isolation"; shift ;;
+        --skip-dependency-check) SKIP_DEPS="--skip-dependency-check"; shift ;;
+        *) echo "Unknown parameter: $1"; exit 1 ;;
+    esac
+done
path: integration_tests/src/numpy.sh
user: login: agriyakhetarpal
body: Addressed in e38a704953522fcdc7366f612d52b4c4e77e798b.
created_at: 2025-03-31T00:34:17Z
author_association: MEMBER
diff_hunk: @@ -23,6 +23,19 @@ test-src: check
 
 	@echo "... Passed"
 
+.PHONY: test-src-no-isolation
+test-src-no-isolation: check
+	@echo "... Running integration tests for building src with --no-isolation --skip-dependency-check"
+
+	# Ensure the necessary build dependencies are installed in the
+	# host environment as we need to do so ourselves.
+
+	pip install meson-python meson cython
path: integration_tests/Makefile
user: login: agriyakhetarpal
body: 🤔 I think I can't activate a virtual environment inside a Makefile for some reason. Maybe that's not possible. I'll look for a workaround.

Edit: found one in 4f793fdb6bac38cbf6613fbe4708d0b3e6d42b25!
created_at: 2025-03-31T00:35:16Z
author_association: MEMBER
diff_hunk: @@ -10,4 +10,4 @@ tar -xf numpy-${VERSION}.tar.gz
 cd numpy-${VERSION}
 
 MESON_CROSS_FILE=$(pyodide config get meson_cross_file)
-${UV_RUN_PREFIX} pyodide build -Csetup-args=-Dallow-noblas=true -Csetup-args=--cross-file="${MESON_CROSS_FILE}"
+${UV_RUN_PREFIX} pyodide build -Csetup-args=-Dallow-noblas=true -Csetup-args=--cross-file="${MESON_CROSS_FILE}" "$@"
path: integration_tests/src/numpy.sh
user: login: agriyakhetarpal
body: ```suggestion
${UV_RUN_PREFIX} pyodide build -Csetup-args=-Dallow-noblas=true -Csetup-args=--cross-file="${MESON_CROSS_FILE}"
```
created_at: 2025-03-31T02:23:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,33 @@
+#!/bin/bash
+
+# The same as "numpy.sh", but without the isolation.
path: integration_tests/src/numpy_no_isolation.sh
user: login: agriyakhetarpal
body: ```suggestion
# The same as "numpy.sh", but without the isolation, and
# builds both NumPy and numpy-tests from a persistent
# build directory.
```
created_at: 2025-03-31T02:24:04Z
author_association: MEMBER
diff_hunk: @@ -383,6 +383,19 @@ def _get_sha256_checksum(archive: Path) -> str:
     return h.hexdigest()
 
 
+def _format_dep_chain(dep_chain: Sequence[str]) -> str:
+    return " -> ".join(dep.partition(";")[0].strip() for dep in dep_chain)
+
+
+def _format_missing_dependencies(missing) -> str:
path: pyodide_build/common.py
user: login: ryanking13
body: Could you please add a type for the parameter?
created_at: 2025-04-05T05:06:23Z
author_association: MEMBER
diff_hunk: @@ -521,10 +521,6 @@ def handle_command_generate_args(  # noqa: C901
             "emcmake",
             "cmake",
             *flags,
-            # Since we create a temporary directory and install compiler symlinks every time,
path: pyodide_build/pywasmcross.py
user: login: ryanking13
body: Oh, yes, we can remove this now. 👍 
created_at: 2025-04-05T05:07:03Z
author_association: MEMBER
diff_hunk: @@ -28,6 +31,19 @@ def run(
     env = os.environ.copy()
     env.update(build_env.get_build_environment_vars(get_pyodide_root()))
 
+    # create a persistent build dir in the source dir
+    build_dir = srcdir / ".pyodide_build"
+    build_dir.mkdir(exist_ok=True)
+    # don't track the build dir (helps if building in a git repo)
+    gitignore_path = build_dir / ".gitignore"
+    if not gitignore_path.exists():
+        with gitignore_path.open("w") as f:
+            f.write(
+                dedent("""\
+                    # Created by pyodide-build
+                    *""")
+            )
path: pyodide_build/out_of_tree/build.py
user: login: ryanking13
body: Could you extract this out to a function, such as `prepare_build_dir(build_dir)`?
created_at: 2025-04-05T05:12:17Z
author_association: MEMBER
diff_hunk: @@ -79,7 +89,15 @@ def _runner(cmd, cwd=None, extra_environ=None):
 
         # Some build dependencies like cmake, meson installs binaries to this directory
         # and we should add it to the PATH so that they can be found.
-        env["BUILD_ENV_SCRIPTS_DIR"] = isolated_build_env.scripts_dir
+        if isolated_build_env:
+            env["BUILD_ENV_SCRIPTS_DIR"] = isolated_build_env.scripts_dir
+        else:
+            # For non-isolated builds, set a fallback path or use the current Python path
+            import sysconfig
+
+            scripts_dir = sysconfig.get_path("scripts")
+            env["BUILD_ENV_SCRIPTS_DIR"] = scripts_dir
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: Makes sense. Thanks for the explanation!
created_at: 2025-04-05T05:14:22Z
author_association: MEMBER
diff_hunk: @@ -254,20 +285,14 @@ def make_command_wrapper_symlinks(symlink_dir: Path) -> dict[str, str]:
 
 
 @contextmanager
-def _create_symlink_dir(env: dict[str, str], build_dir: Path | None):
-    if build_dir:
-        # If we're running under build-recipes, leave the symlinks in
-        # the build directory. This helps with reproducing.
-        symlink_dir = build_dir / "pywasmcross_symlinks"
-        shutil.rmtree(symlink_dir, ignore_errors=True)
-        symlink_dir.mkdir()
-        yield symlink_dir
-        return
-
-    # Running from "pyodide build". Put symlinks in a temporary directory.
-    # TODO: Add a debug option to save the symlinks.
-    with TemporaryDirectory() as symlink_dir_str:
-        yield Path(symlink_dir_str)
+def _create_symlink_dir(
+    env: dict[str, str], build_dir: Path, no_isolation: bool = False
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: no_isolation parameter is not used anymore
created_at: 2025-04-05T05:15:51Z
author_association: MEMBER
diff_hunk: @@ -254,20 +285,14 @@ def make_command_wrapper_symlinks(symlink_dir: Path) -> dict[str, str]:
 
 
 @contextmanager
-def _create_symlink_dir(env: dict[str, str], build_dir: Path | None):
-    if build_dir:
-        # If we're running under build-recipes, leave the symlinks in
-        # the build directory. This helps with reproducing.
-        symlink_dir = build_dir / "pywasmcross_symlinks"
-        shutil.rmtree(symlink_dir, ignore_errors=True)
-        symlink_dir.mkdir()
-        yield symlink_dir
-        return
-
-    # Running from "pyodide build". Put symlinks in a temporary directory.
-    # TODO: Add a debug option to save the symlinks.
-    with TemporaryDirectory() as symlink_dir_str:
-        yield Path(symlink_dir_str)
+def _create_symlink_dir(
path: pyodide_build/pypabuild.py
user: login: ryanking13
body: We are not using a temporary directory anymore, so I don't think this needs to be a context manager anymore. No cleanup is done at the function end.
created_at: 2025-04-05T05:18:18Z
author_association: MEMBER
diff_hunk: @@ -254,20 +285,14 @@ def make_command_wrapper_symlinks(symlink_dir: Path) -> dict[str, str]:
 
 
 @contextmanager
-def _create_symlink_dir(env: dict[str, str], build_dir: Path | None):
-    if build_dir:
-        # If we're running under build-recipes, leave the symlinks in
-        # the build directory. This helps with reproducing.
-        symlink_dir = build_dir / "pywasmcross_symlinks"
-        shutil.rmtree(symlink_dir, ignore_errors=True)
-        symlink_dir.mkdir()
-        yield symlink_dir
-        return
-
-    # Running from "pyodide build". Put symlinks in a temporary directory.
-    # TODO: Add a debug option to save the symlinks.
-    with TemporaryDirectory() as symlink_dir_str:
-        yield Path(symlink_dir_str)
+def _create_symlink_dir(
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: I tried this, and it still requires a temporary directory to be created in one more case, as `get_build_env` also allows `build_dir` to be `None`. It looks like it's a little more complex than expected, so I hope it's okay to leave this and try it out in a follow-up PR?
created_at: 2025-04-05T11:36:20Z
author_association: MEMBER
diff_hunk: @@ -383,6 +383,19 @@ def _get_sha256_checksum(archive: Path) -> str:
     return h.hexdigest()
 
 
+def _format_dep_chain(dep_chain: Sequence[str]) -> str:
+    return " -> ".join(dep.partition(";")[0].strip() for dep in dep_chain)
+
+
+def _format_missing_dependencies(missing) -> str:
path: pyodide_build/common.py
user: login: agriyakhetarpal
body: Done, thanks! Addressed in 206a28e68c8b731827e7876048beb90f16179607.
created_at: 2025-04-05T11:48:20Z
author_association: MEMBER
diff_hunk: @@ -28,6 +31,19 @@ def run(
     env = os.environ.copy()
     env.update(build_env.get_build_environment_vars(get_pyodide_root()))
 
+    # create a persistent build dir in the source dir
+    build_dir = srcdir / ".pyodide_build"
+    build_dir.mkdir(exist_ok=True)
+    # don't track the build dir (helps if building in a git repo)
+    gitignore_path = build_dir / ".gitignore"
+    if not gitignore_path.exists():
+        with gitignore_path.open("w") as f:
+            f.write(
+                dedent("""\
+                    # Created by pyodide-build
+                    *""")
+            )
path: pyodide_build/out_of_tree/build.py
user: login: agriyakhetarpal
body: Yes, this makes sense – thanks. I extracted it out to `_prepare_build_dir` and also added a VCS-ignore file for Mercurial in de93d2b62c6119ea245dbeeb674d0b0301f20bca and 6ef79be4e11702e9139af8001b989859548ea2d8 respectively, as I noticed meson-python sets that as well.
created_at: 2025-04-05T11:49:18Z
author_association: MEMBER
diff_hunk: @@ -254,20 +285,14 @@ def make_command_wrapper_symlinks(symlink_dir: Path) -> dict[str, str]:
 
 
 @contextmanager
-def _create_symlink_dir(env: dict[str, str], build_dir: Path | None):
-    if build_dir:
-        # If we're running under build-recipes, leave the symlinks in
-        # the build directory. This helps with reproducing.
-        symlink_dir = build_dir / "pywasmcross_symlinks"
-        shutil.rmtree(symlink_dir, ignore_errors=True)
-        symlink_dir.mkdir()
-        yield symlink_dir
-        return
-
-    # Running from "pyodide build". Put symlinks in a temporary directory.
-    # TODO: Add a debug option to save the symlinks.
-    with TemporaryDirectory() as symlink_dir_str:
-        yield Path(symlink_dir_str)
+def _create_symlink_dir(
+    env: dict[str, str], build_dir: Path, no_isolation: bool = False
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: Good catch – dropped in 920aec7a56a778df4bf25961ebff9056f1655803.
created_at: 2025-04-05T11:50:44Z
author_association: MEMBER
diff_hunk: @@ -191,6 +209,45 @@ def _build_in_isolated_env(
             )
 
 
+def _build_in_current_env(
+    build_env: Mapping[str, str],
+    srcdir: Path,
+    outdir: str,
+    distribution: Literal["sdist", "wheel"],
+    config_settings: ConfigSettingsType,
+    skip_dependency_check: bool = False,
+) -> str:
+    with common.replace_env(build_env):
+        # Setup sysconfigdata path in environment
+        sysconfigdata_name = get_build_flag("SYSCONFIG_NAME")
+        sysconfig_dir = Path(get_build_flag("TARGETINSTALLDIR")) / "sysconfigdata"
+        env = os.environ.copy()
+        env["_PYTHON_SYSCONFIGDATA_NAME"] = sysconfigdata_name
+        env["PYTHONPATH"] = f"{str(sysconfig_dir)}:{env.get('PYTHONPATH', '')}"
+
+        with common.replace_env(env):
+            builder = _ProjectBuilder(srcdir, runner=_gen_runner(build_env))
+
+            if not skip_dependency_check:
+                missing = builder.check_dependencies(
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: Moved this to a new issue: #175.
created_at: 2025-04-05T12:44:38Z
author_association: MEMBER
diff_hunk: @@ -254,20 +285,14 @@ def make_command_wrapper_symlinks(symlink_dir: Path) -> dict[str, str]:
 
 
 @contextmanager
-def _create_symlink_dir(env: dict[str, str], build_dir: Path | None):
-    if build_dir:
-        # If we're running under build-recipes, leave the symlinks in
-        # the build directory. This helps with reproducing.
-        symlink_dir = build_dir / "pywasmcross_symlinks"
-        shutil.rmtree(symlink_dir, ignore_errors=True)
-        symlink_dir.mkdir()
-        yield symlink_dir
-        return
-
-    # Running from "pyodide build". Put symlinks in a temporary directory.
-    # TODO: Add a debug option to save the symlinks.
-    with TemporaryDirectory() as symlink_dir_str:
-        yield Path(symlink_dir_str)
+def _create_symlink_dir(
path: pyodide_build/pypabuild.py
user: login: agriyakhetarpal
body: Moved it to a new issue: #176
created_at: 2025-04-05T12:46:48Z
author_association: MEMBER

PR comments:
user: login: agriyakhetarpal
created_at: 2025-03-30T01:45:08Z
body: Hi @ryanking13, I've been able to build and test out the feature(s) I've added in this PR, but one problem still exists: the compiler wrappers for Emscripten are still being placed inside temporary directories as a part of the build process for some reason: https://github.com/pyodide/pyodide-build/actions/runs/14151180877/job/39644562112?pr=152#step:11:449

What this means in action is that if I were to build NumPy without isolation via the below command invocation in a persistent build directory, say, in `build/` inside the NumPy repo root:

```bash
pyodide build . \
-Csetup-args="--cross-file=$(pyodide config get meson_cross_file)" \
-Csetup-args="-Dblas=none" -Csetup-args="-Dlapack=none" \
-nx \ # i.e., --no-isolation and --skip-dependency-check args, which this PR adds
-Cbuild-dir=build`
```

I am able to build a NumPy wheel. However, if I were to rerun the same command, hoping to make use of the caching provided by the persistent build directory and achieve a speedup for the subsequent run(s) , it would fail because Meson stores its build metadata inside the build directory, which also contains information about the compiler, etc. Meson then fails to find the compiler, as the temporary directory used the previous wheel build where the compiler wrappers were stored no longer exists.

When I build NumPy with a persistent build dir with native compilation on my macOS machine, the compilers are picked up correctly from conda-forge each time – this makes me think that this is a `pyodide-build` specific issue. Maybe it has something to do with `pywasmcross`?

I think you would be the best person to know why this is the case and on how we can address it, so I thought I should ask you about it! :)
author_association: MEMBER
user: login: ryanking13
created_at: 2025-03-30T06:14:53Z
body: Hi @agriyakhetarpal, thanks for working on this.

> the compiler wrappers for Emscripten are still being placed inside temporary directories as a part of the build process for some reason:

Yep, we are using temporary directory for now, but I think you can change it. We've already changed it in recipe build to store the symlinks in the build directory (https://github.com/pyodide/pyodide-build/pull/96), so I think you can do the similar thing in the `pyodide build`.

The directory where the symlinks are stored is decided [here](https://github.com/pyodide/pyodide-build/blob/71aa0b5ae55b196c4b3444434c67a9541dd382c9/pyodide_build/pypabuild.py#L302-L303) and [here](https://github.com/pyodide/pyodide-build/blob/71aa0b5ae55b196c4b3444434c67a9541dd382c9/pyodide_build/pypabuild.py#L257-L265). I think you can pass the `build_dir` parameter [here](https://github.com/pyodide/pyodide-build/blob/71aa0b5ae55b196c4b3444434c67a9541dd382c9/pyodide_build/out_of_tree/build.py#L31-L39) to store the symlink.
author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-03-31T00:44:52Z
body: > Yep, we are using temporary directory for now, but I think you can change it. We've already changed it in recipe build to store the symlinks in the build directory (#96), so I think you can do the similar thing in the `pyodide build`.
> 
> The directory where the symlinks are stored is decided [here](https://github.com/pyodide/pyodide-build/blob/71aa0b5ae55b196c4b3444434c67a9541dd382c9/pyodide_build/pypabuild.py#L302-L303) and [here](https://github.com/pyodide/pyodide-build/blob/71aa0b5ae55b196c4b3444434c67a9541dd382c9/pyodide_build/pypabuild.py#L257-L265). I think you can pass the `build_dir` parameter [here](https://github.com/pyodide/pyodide-build/blob/71aa0b5ae55b196c4b3444434c67a9541dd382c9/pyodide_build/out_of_tree/build.py#L31-L39) to store the symlink.

Thanks a lot for the pointer! I'm able to get subsequent builds for NumPy working. :D

I was able to compile NumPy in 77 seconds on my machine, and taking advantage of the build caching through the persistence of the build dir, `numpy-tests` compiles in 8 seconds.
author_association: MEMBER
user: login: ryanking13
created_at: 2025-04-02T14:16:31Z
body: Thanks again Agriya! I didn't forget this PR, but I've been a little busy this week and haven't been keeping up with all the PRs. I would like to review this in detail, so please wait for me until the weekend! 🙏 
author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-02T19:50:10Z
body: No rush, thank you! :)
author_association: MEMBER
user: login: agriyakhetarpal
created_at: 2025-04-05T22:26:53Z
body: Many thanks for the detailed review, @ryanking13! I'll merge this now, and I've opened issues for adding follow-ups.
author_association: MEMBER