Repository: canonical/charmcraft

Issue:
Building multiple charms simultaneously can cause caching errors
### Bug Description

Using `charmcraft` from `latest/edge` in GH runners results in istio charms failing to be built. More specifically, this results in the [following zipfile bug](https://github.com/canonical/bundle-kubeflow/issues/1005). This used to happen intermittently but with `latest/edge` it happens 90% of the time ([example](https://github.com/canonical/istio-operators/actions/runs/10246771376/job/28639271698?pr=498) where the CI has been re-run 7 times).

Note that, as mentioned in this https://github.com/canonical/bundle-kubeflow/issues/1005#issuecomment-2289148457:
>Building the charm in a fresh LXC container using charmcraft both from `latest/edge` and `latest/candidate` is successful.

A pointer by @lengau was that it could be an issue with the GH runner space, but it was attempted to free up some space before running the tests https://github.com/canonical/istio-operators/pull/532 (44G up from 21G), but the issue persisted.

Apart from logs pasted below, here's from another run for [reference](https://pastebin.canonical.com/p/jbtCsfRrBF/).

### To Reproduce

Rerun the CI in this draft PR https://github.com/canonical/istio-operators/pull/506 (where charmcraft form `latest/edge` is being used) compared to when [using latest/candidate](https://github.com/canonical/istio-operators/pull/510).

### Environment

Microk8s 1.25,1.26
Juju 3.4.5
Some charmcraft versions that have been failing:
* charmcraft (edge) 3.1.2.post11+g0d3b2713 from Canonical** installed
* charmcraft (edge) 3.1.2.post84+g3f0dbc6c from Canonical** installed

But the charms are built with 3.1.2.

### charmcraft.yaml

```shell
type: charm
bases:
  - build-on:
    - name: "ubuntu"
      channel: "20.04"
    run-on:
    - name: "ubuntu"
      channel: "20.04"
parts:
  charm:
    charm-python-packages: [setuptools, pip]
    build-packages: [git, rustc, cargo, libffi-dev, libssl-dev, pkg-config]
  istioctl:
    plugin: dump
    source: https://github.com/istio/istio/releases/download/1.17.3/istioctl-1.17.3-linux-amd64.tar.gz
    source-type: tar

---

type: charm
bases:
  - build-on:
      - name: 'ubuntu'
        channel: '20.04'
    run-on:
      - name: 'ubuntu'
        channel: '20.04'
parts:
  charm:
    charm-python-packages: [setuptools, pip]
```


### Relevant log output

```shell
::    ::   Successfully built setuptools wheel
::    ::   Installing collected packages: setuptools, wheel
::    ::   ERROR: Exception:
::    ::   Traceback (most recent call last):
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
::    ::       status = self.run(options, args)
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 421, in run
::    ::       installed = install_given_reqs(
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 67, in install_given_reqs
::    ::       requirement.install(
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 809, in install
::    ::       install_wheel(
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 606, in install_wheel
::    ::       unpack_file(wheel_path, unpacked_dir.path)
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 249, in unpack_file
::    ::       unzip_file(
::    ::     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 139, in unzip_file
::    ::       shutil.copyfileobj(fp, destfp)
::    ::     File "/usr/lib/python3.8/shutil.py", line 205, in copyfileobj
::    ::       buf = fsrc_read(length)
::    ::     File "/usr/lib/python3.8/zipfile.py", line 940, in read
::    ::       data = self._read1(n)
::    ::     File "/usr/lib/python3.8/zipfile.py", line 1008, in _read1
::    ::       data += self._read2(n - len(data))
::    ::     File "/usr/lib/python3.8/zipfile.py", line 1043, in _read2
::    ::       raise EOFError
::    ::   EOFError
::    ::   ----------------------------------------
::    :: ERROR: Command errored out with exit status 2: /root/parts/charm/build/staging-venv/bin/python3 /root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_x_j7juu/overlay --no-warn-script-location --no-binary anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jsonschema,lightkube,lightkube-models,markupsafe,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
:: WARNING: Initial package installation failed. Falling back to older method, which may leave your charm in an un-runnable state.
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', 'jinja2']
::    :: Collecting jinja2
::    ::   Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
::    :: Collecting MarkupSafe>=2.0
::    ::   Downloading MarkupSafe-2.1.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26 kB)
::    :: Installing collected packages: MarkupSafe, jinja2
::    :: Successfully installed MarkupSafe-2.1.5 jinja2-3.1.4
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', 'wheel', 'setuptools', 'pip']
::    :: Collecting wheel
::    ::   Using cached wheel-0.43.0.tar.gz (99 kB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting setuptools
::    ::   Using cached setuptools-69.5.1.tar.gz (2.3 MB)
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Installing backend dependencies: started
::    ::   Installing backend dependencies: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Collecting pip
::    ::   Downloading pip-24.0.tar.gz (2.1 MB)
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::     Preparing wheel metadata: started
::    ::     Preparing wheel metadata: finished with status 'done'
::    :: Building wheels for collected packages: wheel, setuptools, pip
::    ::   Building wheel for wheel (PEP 517): started
::    ::   Building wheel for wheel (PEP 517): finished with status 'done'
::    ::   Created wheel for wheel: filename=wheel-0.43.0-py3-none-any.whl size=65775 sha256=55c570405f142630c6b9f72fe09d9b67cf1477fcf543ae5b8dcb1f5b7377da81
::    ::   Stored in directory: /root/.cache/pip/wheels/d4/a3/b2/6faade3bb04cd720bb432a6c21ec7baa2542801334a4d47a66
::    ::   Building wheel for setuptools (PEP 517): started
::    ::   Building wheel for setuptools (PEP 517): finished with status 'done'
::    ::   Created wheel for setuptools: filename=setuptools-69.5.1-py3-none-any.whl size=894566 sha256=075e940cfa3659352ca81f87fabf8b4b75ccdbb5f99f2a71a4f77df1b6707fbc
::    ::   Stored in directory: /root/.cache/pip/wheels/57/b6/41/311c955c1ca3eed447666adf5ca9680ddcdd11dc785a746bf6
::    ::   Building wheel for pip (PEP 517): started
::    ::   Building wheel for pip (PEP 517): finished with status 'done'
::    ::   Created wheel for pip: filename=pip-24.0-py3-none-any.whl size=2110223 sha256=7176c5cddbc465dc78dddac6545a4635901a7c9a08a959f4f3bbb3ca0bee944a
::    ::   Stored in directory: /root/.cache/pip/wheels/c1/d6/d4/2f7765d82b17d8c25425875b2dce5af9f22746668c3add50f6
::    :: Successfully built wheel setuptools pip
::    :: Installing collected packages: wheel, setuptools, pip
::    ::   Attempting uninstall: setuptools
::    ::     Found existing installation: setuptools 44.0.0
::    ::     Uninstalling setuptools-44.0.0:
::    ::       Successfully uninstalled setuptools-44.0.0
::    ::   Attempting uninstall: pip
::    ::     Found existing installation: pip 20.0.2
::    ::     Uninstalling pip-20.0.2:
::    ::       Successfully uninstalled pip-20.0.2
::    :: Successfully installed pip-24.0 setuptools-69.5.1 wheel-0.43.0
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', '--requirement=requirements.txt']
::    :: Collecting anyio==3.7.1 (from -r requirements.txt (line 7))
::    ::   Downloading anyio-3.7.1.tar.gz (142 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.9/142.9 kB 3.9 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'done'
::    ::   Getting requirements to build wheel: started
::    ::   Getting requirements to build wheel: finished with status 'done'
::    ::   Installing backend dependencies: started
::    ::   Installing backend dependencies: finished with status 'done'
::    ::   Preparing metadata (pyproject.toml): started
::    ::   Preparing metadata (pyproject.toml): finished with status 'done'
::    :: Collecting attrs==23.1.0 (from -r requirements.txt (line 9))
::    ::   Downloading attrs-23.1.0.tar.gz (212 kB)
::    ::      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.9/212.9 kB 29.5 MB/s eta 0:00:00
::    ::   Installing build dependencies: started
::    ::   Installing build dependencies: finished with status 'error'
::    ::   error: subprocess-exited-with-error
::    ::
::    ::   × pip subprocess to install build dependencies did not run successfully.
::    ::   │ exit code: 1
::    ::   ╰─> [103 lines of output]
::    ::       Collecting hatchling
::    ::         Downloading hatchling-1.24.2.tar.gz (63 kB)
::    ::            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.0/64.0 kB 2.0 MB/s eta 0:00:00
::    ::         Getting requirements to build wheel: started
::    ::         Getting requirements to build wheel: finished with status 'done'
::    ::         Installing backend dependencies: started
::    ::         Installing backend dependencies: finished with status 'error'
::    ::         error: subprocess-exited-with-error
::    ::
::    ::         × pip subprocess to install backend dependencies did not run successfully.
::    ::         │ exit code: 1
::    ::         ╰─> [81 lines of output]
::    ::             Collecting packaging>=23.2
::    ::               Using cached packaging-24.0-py3-none-any.whl
::    ::             Collecting pathspec>=0.10.1
::    ::               Downloading pathspec-0.12.1.tar.gz (51 kB)
::    ::                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.0/51.0 kB 895.3 kB/s eta 0:00:00
::    ::               Installing build dependencies: started
::    ::               Installing build dependencies: finished with status 'done'
::    ::               Getting requirements to build wheel: started
::    ::               Getting requirements to build wheel: finished with status 'done'
::    ::               Preparing metadata (pyproject.toml): started
::    ::               Preparing metadata (pyproject.toml): finished with status 'done'
::    ::             Collecting trove-classifiers
::    ::               Downloading trove-classifiers-2024.4.10.tar.gz (16 kB)
::    ::               Installing build dependencies: started
::    ::               Installing build dependencies: finished with status 'done'
::    ::               Getting requirements to build wheel: started
::    ::               Getting requirements to build wheel: finished with status 'done'
::    ::               Installing backend dependencies: started
::    ::               Installing backend dependencies: finished with status 'done'
::    ::               Preparing metadata (pyproject.toml): started
::    ::               Preparing metadata (pyproject.toml): finished with status 'done'
::    ::             Collecting tomli>=1.2.2
::    ::               Using cached tomli-2.0.1-py3-none-any.whl
::    ::             Collecting pluggy>=1.0.0
::    ::               Downloading pluggy-1.5.0.tar.gz (67 kB)
::    ::                  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.0/68.0 kB 3.5 MB/s eta 0:00:00
::    ::               Installing build dependencies: started
::    ::               Installing build dependencies: finished with status 'error'
::    ::               error: subprocess-exited-with-error
::    ::
::    ::               × pip subprocess to install build dependencies did not run successfully.
::    ::               │ exit code: 2
::    ::               ╰─> [38 lines of output]
::    ::                   Collecting setuptools>=45.0
::    ::                     Using cached setuptools-69.5.1-py3-none-any.whl
::    ::                   Collecting setuptools-scm>=6.2.3 (from setuptools-scm[toml]>=6.2.3)
::    ::                     Using cached setuptools_scm-8.1.0-py3-none-any.whl
::    ::                   Collecting packaging>=20 (from setuptools-scm>=6.2.3->setuptools-scm[toml]>=6.2.3)
::    ::                     Using cached packaging-24.0-py3-none-any.whl
::    ::                   Collecting typing-extensions (from setuptools-scm>=6.2.3->setuptools-scm[toml]>=6.2.3)
::    ::                     Using cached typing_extensions-4.11.0-py3-none-any.whl
::    ::                   Collecting tomli>=1 (from setuptools-scm>=6.2.3->setuptools-scm[toml]>=6.2.3)
::    ::                     Using cached tomli-2.0.1-py3-none-any.whl
::    ::                   Installing collected packages: typing-extensions, tomli, setuptools, packaging, setuptools-scm
::    ::                   ERROR: Exception:
::    ::                   Traceback (most recent call last):
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
::    ::                       status = run_func(*args)
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
::    ::                       return func(self, options, args)
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 452, in run
::    ::                       installed = install_given_reqs(
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 72, in install_given_reqs
::    ::                       requirement.install(
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 856, in install
::    ::                       install_wheel(
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 725, in install_wheel
::    ::                       _install_wheel(
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 585, in _install_wheel
::    ::                       file.save()
::    ::                     File "/root/parts/charm/build/staging-venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 384, in save
::    ::                       shutil.copyfileobj(f, dest)
::    ::                     File "/usr/lib/python3.8/shutil.py", line 205, in copyfileobj
::    ::                       buf = fsrc_read(length)
::    ::                     File "/usr/lib/python3.8/zipfile.py", line 940, in read
::    ::                       data = self._read1(n)
::    ::                     File "/usr/lib/python3.8/zipfile.py", line 1008, in _read1
::    ::                       data += self._read2(n - len(data))
::    ::                     File "/usr/lib/python3.8/zipfile.py", line 1043, in _read2
::    ::                       raise EOFError
::    ::                   EOFError
::    ::                   [end of output]
::    ::
::    ::               note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::             error: subprocess-exited-with-error
::    ::
::    ::             × pip subprocess to install build dependencies did not run successfully.
::    ::             │ exit code: 2
::    ::             ╰─> See above for output.
::    ::
::    ::             note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::             [end of output]
::    ::
::    ::         note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::       error: subprocess-exited-with-error
::    ::
::    ::       × pip subprocess to install backend dependencies did not run successfully.
::    ::       │ exit code: 1
::    ::       ╰─> See above for output.
::    ::
::    ::       note: This error originates from a subprocess, and is likely not a problem with pip.
::    ::       [end of output]
::    ::
::    ::   note: This error originates from a subprocess, and is likely not a problem with pip.
::    :: error: subprocess-exited-with-error
::    ::
::    :: × pip subprocess to install build dependencies did not run successfully.
::    :: │ exit code: 1
::    :: ╰─> See above for output.
::    ::
::    :: note: This error originates from a subprocess, and is likely not a problem with pip.
:: Traceback (most recent call last):
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 260, in _install_dependencies
::     _process_run(
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 408, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jsonschema,lightkube,lightkube-models,markupsafe,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] execution failed with retcode 1
::
:: During handling of the above exception, another exception occurred:
::
:: Traceback (most recent call last):
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 477, in <module>
::     main()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 472, in main
::     builder.build_charm()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 101, in build_charm
::     self.handle_dependencies()
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 354, in handle_dependencies
::     self._install_dependencies(staging_venv_dir)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/instrum.py", line 152, in _f
::     return func(*args, **kwargs)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 289, in _install_dependencies
::     _process_run(cmd)
::   File "/snap/charmcraft/3454/lib/python3.10/site-packages/charmcraft/charm_builder.py", line 408, in _process_run
::     raise RuntimeError(f"Subprocess command {cmd} execution failed with retcode {retcode}")
:: RuntimeError: Subprocess command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--upgrade', '--no-binary', ':all:', '--requirement=requirements.txt'] execution failed with retcode 1
Parts processing error: Failed to run the build script for part 'charm'.
Failed to build charm for bases index '0'.
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20240509-152917.976217.log'
```


Discussion:

Patch:
diff --git a/charmcraft/services/provider.py b/charmcraft/services/provider.py
index cefcb9ad..443e1fb8 100644
--- a/charmcraft/services/provider.py
+++ b/charmcraft/services/provider.py
@@ -17,19 +17,55 @@
 """Service class for creating providers."""
 from __future__ import annotations
 
+import contextlib
+import io
+from collections.abc import Generator
+
+from craft_application.models import BuildInfo
+
+try:
+    import fcntl
+except ModuleNotFoundError:  # Not available on Windows.
+    fcntl = None  # type: ignore[assignment]
 import os
 import pathlib
+from typing import cast
 
+import craft_application
 import craft_providers
 from craft_application import services
+from craft_cli import emit
 from craft_providers import bases
 
-from charmcraft import env
+from charmcraft import env, models
 
 
 class ProviderService(services.ProviderService):
     """Business logic for getting providers."""
 
+    def __init__(
+        self,
+        app: craft_application.AppMetadata,
+        services: craft_application.ServiceFactory,
+        *,
+        project: models.CharmcraftProject,
+        work_dir: pathlib.Path,
+        build_plan: list[BuildInfo],
+        provider_name: str | None = None,
+        install_snap: bool = True,
+    ) -> None:
+        super().__init__(
+            app,
+            services,
+            project=project,
+            work_dir=work_dir,
+            build_plan=build_plan,
+            provider_name=provider_name,
+            install_snap=install_snap,
+        )
+        self._cache_path: pathlib.Path | None = None
+        self._lock: io.TextIOBase | None = None
+
     def setup(self) -> None:
         """Set up the provider service for Charmcraft."""
         super().setup()
@@ -56,12 +92,61 @@ class ProviderService(services.ProviderService):
 
         If no cache_path is included, adds one.
         """
+        self._cache_path = cast(
+            pathlib.Path, kwargs.get("cache_path", env.get_host_shared_cache_path())
+        )
+        self._lock = _maybe_lock_cache(self._cache_path)
+
         # Forward the shared cache path.
-        if "cache_path" not in kwargs:
-            kwargs["cache_path"] = env.get_host_shared_cache_path()
+        kwargs["cache_path"] = self._cache_path if self._lock else None
         return super().get_base(
             base_name,
             instance_name=instance_name,
             # craft-application annotation is incorrect
             **kwargs,  # type: ignore[arg-type]
         )
+
+    @contextlib.contextmanager
+    def instance(
+        self,
+        build_info: BuildInfo,
+        *,
+        work_dir: pathlib.Path,
+        allow_unstable: bool = True,
+        **kwargs: bool | str | None,
+    ) -> Generator[craft_providers.Executor, None, None]:
+        """Instance override for Charmcraft."""
+        with super().instance(
+            build_info, work_dir=work_dir, allow_unstable=allow_unstable, **kwargs
+        ) as instance:
+            try:
+                yield instance
+            finally:
+                if fcntl is not None and self._lock:
+                    fcntl.flock(self._lock, fcntl.LOCK_UN)
+                    self._lock.close()
+
+
+def _maybe_lock_cache(path: pathlib.Path) -> io.TextIOBase | None:
+    """Lock the cache so we only have one copy of Charmcraft using it at a time."""
+    if fcntl is None:  # Don't lock on Windows - just don't cache.
+        return None
+    cache_lock_path = path / "charmcraft.lock"
+
+    emit.trace("Attempting to lock the cache path")
+    lock_file = cache_lock_path.open("w+")
+    try:
+        # Exclusive lock, but non-blocking.
+        fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except OSError:
+        emit.progress(
+            "Shared cache locked by another process; running without cache.", permanent=True
+        )
+        return None
+    else:
+        pid = str(os.getpid())
+        lock_file.write(pid)
+        lock_file.flush()
+        os.fsync(lock_file.fileno())
+        emit.trace(f"Cache path locked by this process ({pid})")
+        return lock_file
diff --git a/pyproject.toml b/pyproject.toml
index 128bfd86..f967a85a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -330,6 +330,9 @@ lint.ignore = [
 # Allow Pydantic's `@validator` decorator to trigger class method treatment.
 classmethod-decorators = ["pydantic.validator"]
 
+[tool.ruff.lint.pydocstyle]
+ignore-decorators = ["overrides.overrides", "overrides.override", "typing.overload", "typing.override"]
+
 [tool.ruff.lint.per-file-ignores]
 "tests/**.py" = [  # Some things we want for the moin project are unnecessary in tests.
     "D",  # Ignore docstring rules in tests


Test patch:
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
index 33f2ae18..d8252bcf 100644
--- a/.github/workflows/tests.yaml
+++ b/.github/workflows/tests.yaml
@@ -80,6 +80,9 @@ jobs:
         run: |
           sudo apt update
           sudo apt install -y python3-pip python3-setuptools python3-wheel python3-venv libapt-pkg-dev
+      - name: Setup LXD
+        uses: canonical/setup-lxd@v0.1.1
+        if: ${{ runner.os == 'Linux' }}
       - name: Install skopeo (mac)
         # This is only necessary for Linux until skopeo >= 1.11 is in repos.
         # Once we're running on Noble, we can get skopeo from apt.
diff --git a/tests/conftest.py b/tests/conftest.py
index 1eeecf33..8b66dd28 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -109,16 +109,19 @@ def service_factory(
 
 
 @pytest.fixture
-def default_build_plan():
+def default_build_info() -> models.BuildInfo:
     arch = util.get_host_architecture()
-    return [
-        models.BuildInfo(
-            base=bases.BaseName("ubuntu", "22.04"),
-            build_on=arch,
-            build_for="arm64",
-            platform="distro-1-test64",
-        )
-    ]
+    return models.BuildInfo(
+        base=bases.BaseName("ubuntu", "22.04"),
+        build_on=arch,
+        build_for="arm64",
+        platform="distro-1-test64",
+    )
+
+
+@pytest.fixture
+def default_build_plan(default_build_info: models.BuildInfo):
+    return [default_build_info]
 
 
 @pytest.fixture
diff --git a/tests/integration/services/conftest.py b/tests/integration/services/conftest.py
index 04704da9..96f232f0 100644
--- a/tests/integration/services/conftest.py
+++ b/tests/integration/services/conftest.py
@@ -14,10 +14,7 @@
 #
 # For further info, check https://github.com/canonical/charmcraft
 """Configuration for services integration tests."""
-import contextlib
-import sys
 
-import pyfakefs.fake_filesystem
 import pytest
 
 from charmcraft import services
@@ -25,20 +22,10 @@ from charmcraft.application.main import APP_METADATA, Charmcraft
 
 
 @pytest.fixture
-def service_factory(
-    fs: pyfakefs.fake_filesystem.FakeFilesystem, fake_path, simple_charm
-) -> services.CharmcraftServiceFactory:
-    fake_project_dir = fake_path / "project"
+def service_factory(simple_charm, new_path) -> services.CharmcraftServiceFactory:
+    fake_project_dir = new_path / "project"
     fake_project_dir.mkdir()
 
-    # Allow access to the real venv library path.
-    # This is necessary because certifi lazy-loads the certificate file.
-    for python_path in sys.path:
-        if not python_path:
-            continue
-        with contextlib.suppress(OSError):
-            fs.add_real_directory(python_path)
-
     factory = services.CharmcraftServiceFactory(app=APP_METADATA)
 
     app = Charmcraft(app=APP_METADATA, services=factory)
diff --git a/tests/integration/services/test_package.py b/tests/integration/services/test_package.py
index cfe9fc0b..7f21f98b 100644
--- a/tests/integration/services/test_package.py
+++ b/tests/integration/services/test_package.py
@@ -14,7 +14,6 @@
 #
 # For further info, check https://github.com/canonical/charmcraft
 """Tests for package service."""
-import contextlib
 import datetime
 import pathlib
 
@@ -28,8 +27,8 @@ from charmcraft.application.main import APP_METADATA
 
 
 @pytest.fixture
-def package_service(fake_path, service_factory, default_build_plan):
-    fake_project_dir = fake_path
+def package_service(new_path: pathlib.Path, service_factory, default_build_plan):
+    fake_project_dir = new_path
     svc = services.PackageService(
         app=APP_METADATA,
         project=service_factory.project,
@@ -49,12 +48,10 @@ def package_service(fake_path, service_factory, default_build_plan):
     ],
 )
 @freezegun.freeze_time(datetime.datetime(2020, 3, 14, 0, 0, 0, tzinfo=datetime.timezone.utc))
-def test_write_metadata(monkeypatch, fs, package_service, project_path):
+def test_write_metadata(monkeypatch, new_path, package_service, project_path):
     monkeypatch.setattr(charmcraft, "__version__", "3.0-test-version")
-    with contextlib.suppress(FileExistsError):
-        fs.add_real_directory(project_path)
-    test_prime_dir = pathlib.Path("/prime")
-    fs.create_dir(test_prime_dir)
+    test_prime_dir = new_path / "prime"
+    test_prime_dir.mkdir()
     expected_prime_dir = project_path / "prime"
 
     project = models.CharmcraftProject.from_yaml_file(project_path / "project" / "charmcraft.yaml")
@@ -75,23 +72,21 @@ def test_write_metadata(monkeypatch, fs, package_service, project_path):
     ],
 )
 @freezegun.freeze_time(datetime.datetime(2020, 3, 14, 0, 0, 0, tzinfo=datetime.timezone.utc))
-def test_overwrite_metadata(monkeypatch, fs, package_service, project_path):
+def test_overwrite_metadata(monkeypatch, new_path, package_service, project_path):
     """Test that the metadata file gets rewritten for a charm.
 
     Regression test for https://github.com/canonical/charmcraft/issues/1654
     """
     monkeypatch.setattr(charmcraft, "__version__", "3.0-test-version")
-    with contextlib.suppress(FileExistsError):
-        fs.add_real_directory(project_path)
-    test_prime_dir = pathlib.Path("/prime")
-    fs.create_dir(test_prime_dir)
+    test_prime_dir = new_path / "prime"
+    test_prime_dir.mkdir()
     expected_prime_dir = project_path / "prime"
 
     project = models.CharmcraftProject.from_yaml_file(project_path / "project" / "charmcraft.yaml")
     project._started_at = datetime.datetime.now(tz=datetime.timezone.utc)
     package_service._project = project
 
-    fs.create_file(test_prime_dir / const.METADATA_FILENAME, contents="INVALID!!")
+    (test_prime_dir / const.METADATA_FILENAME).write_text("INVALID!!")
 
     package_service.write_metadata(test_prime_dir)
 
@@ -100,20 +95,18 @@ def test_overwrite_metadata(monkeypatch, fs, package_service, project_path):
 
 
 @freezegun.freeze_time(datetime.datetime(2020, 3, 14, 0, 0, 0, tzinfo=datetime.timezone.utc))
-def test_no_overwrite_reactive_metadata(monkeypatch, fs, package_service):
+def test_no_overwrite_reactive_metadata(monkeypatch, new_path, package_service):
     """Test that the metadata file doesn't get overwritten for a reactive charm..
 
     Regression test for https://github.com/canonical/charmcraft/issues/1654
     """
     monkeypatch.setattr(charmcraft, "__version__", "3.0-test-version")
     project_path = pathlib.Path(__file__).parent / "sample_projects" / "basic-reactive"
-    with contextlib.suppress(FileExistsError):
-        fs.add_real_directory(project_path)
-    test_prime_dir = pathlib.Path("/prime")
-    fs.create_dir(test_prime_dir)
-    test_stage_dir = pathlib.Path("/stage")
-    fs.create_dir(test_stage_dir)
-    fs.create_file(test_stage_dir / const.METADATA_FILENAME, contents="INVALID!!")
+    test_prime_dir = new_path / "prime"
+    test_prime_dir.mkdir()
+    test_stage_dir = new_path / "stage"
+    test_stage_dir.mkdir()
+    (test_stage_dir / const.METADATA_FILENAME).write_text("INVALID!!")
 
     project = models.CharmcraftProject.from_yaml_file(project_path / "project" / "charmcraft.yaml")
     project._started_at = datetime.datetime.now(tz=datetime.timezone.utc)
diff --git a/tests/integration/services/test_provider.py b/tests/integration/services/test_provider.py
new file mode 100644
index 00000000..a6b149f9
--- /dev/null
+++ b/tests/integration/services/test_provider.py
@@ -0,0 +1,103 @@
+# Copyright 2024 Canonical Ltd.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# For further info, check https://github.com/canonical/charmcraft
+"""Integration tests for the provider service."""
+
+import pathlib
+import shutil
+import subprocess
+import sys
+
+import pytest
+from craft_application.models import BuildInfo
+from craft_cli.pytest_plugin import RecordingEmitter
+
+from charmcraft import services
+from charmcraft.services.provider import _maybe_lock_cache
+
+
+@pytest.mark.skipif(sys.platform == "win32", reason="no cache on windows")
+def test_lock_cache(
+    service_factory: services.CharmcraftServiceFactory,
+    tmp_path: pathlib.Path,
+    default_build_info: BuildInfo,
+    emitter: RecordingEmitter,
+):
+    cache_path = tmp_path / "cache"
+    cache_path.mkdir()
+    lock_file = cache_path / "charmcraft.lock"
+    bash_lock_cmd = ["bash", "-c", f"flock -n {lock_file} true"] if shutil.which("flock") else None
+    provider = service_factory.provider
+    provider_kwargs = {
+        "build_info": default_build_info,
+        "work_dir": pathlib.Path(__file__).parent,
+        "cache_path": cache_path,
+    }
+    assert not lock_file.exists()
+
+    with provider.instance(**provider_kwargs):
+        # Test that the cache lock gets created
+        assert lock_file.is_file()
+        if bash_lock_cmd:
+            with pytest.raises(subprocess.CalledProcessError):
+                # Another process should not be able to lock the file.
+                subprocess.run(bash_lock_cmd, check=True)
+
+    # After exiting we should be able to lock the file.
+    if bash_lock_cmd:
+        subprocess.run(bash_lock_cmd, check=True)
+
+
+@pytest.mark.skipif(sys.platform == "win32", reason="no cache on windows")
+def test_locked_cache_no_cache(
+    service_factory: services.CharmcraftServiceFactory,
+    tmp_path: pathlib.Path,
+    default_build_info: BuildInfo,
+    emitter: RecordingEmitter,
+):
+    cache_path = tmp_path / "cache"
+    cache_path.mkdir()
+    lock_file = cache_path / "charmcraft.lock"
+
+    bash_lock_cmd = ["bash", "-c", f"flock -n {lock_file} true"] if shutil.which("flock") else None
+    # Check that we can lock the file from another process.
+    if bash_lock_cmd:
+        subprocess.run(bash_lock_cmd, check=True)
+    _ = _maybe_lock_cache(cache_path)
+    # And now we can't.
+    if bash_lock_cmd:
+        with pytest.raises(subprocess.CalledProcessError):
+            subprocess.run(bash_lock_cmd, check=True)
+
+    provider = service_factory.provider
+    provider_kwargs = {
+        "build_info": default_build_info,
+        "work_dir": pathlib.Path(__file__).parent,
+        "cache_path": cache_path,
+    }
+
+    with provider.instance(**provider_kwargs) as instance:
+        # Create a file in the cache and ensure it's not visible in the outer fs
+        instance.execute_run(["touch", "/root/.cache/cache_cached"])
+
+        # Because we've already locked the cache, we don't get a subdirectory in
+        # the cache, and thus the touch command inside there only affected the
+        # instance cache and not the shared cache.
+        assert list(cache_path.iterdir()) == [cache_path / "charmcraft.lock"]
+        emitter.assert_progress(
+            "Shared cache locked by another process; running without cache.", permanent=True
+        )
+
+        assert not (tmp_path / "cache_cached").exists()
diff --git a/tests/unit/services/test_provider.py b/tests/unit/services/test_provider.py
index b55f6c48..be6bc72d 100644
--- a/tests/unit/services/test_provider.py
+++ b/tests/unit/services/test_provider.py
@@ -15,12 +15,23 @@
 # For further info, check https://github.com/canonical/charmcraft
 """Unit tests for the provider service."""
 
+try:
+    import fcntl
+except ModuleNotFoundError:  # Windows
+    fcntl = None
+import functools
 import pathlib
+import sys
+from collections.abc import Iterator
+from unittest import mock
 
 import pytest
+from craft_cli.pytest_plugin import RecordingEmitter
 from craft_providers import bases
 
 from charmcraft import models, services
+from charmcraft.application.main import APP_METADATA
+from charmcraft.services.provider import _maybe_lock_cache
 
 
 @pytest.fixture
@@ -42,6 +53,17 @@ def provider_service(
     return service_factory.provider
 
 
+@pytest.fixture
+def mock_register(monkeypatch) -> Iterator[mock.Mock]:
+    register = mock.Mock()
+    monkeypatch.setattr("atexit.register", register)
+    yield register
+
+    # Call the exit hooks as if exiting the application.
+    for hook in register.mock_calls:
+        functools.partial(*hook.args)()
+
+
 @pytest.mark.parametrize(
     "base_name",
     [
@@ -62,6 +84,7 @@ def provider_service(
         bases.BaseName("almalinux", "9"),
     ],
 )
+@pytest.mark.skipif(sys.platform == "win32", reason="no cache on windows")
 def test_get_base_forwards_cache(
     monkeypatch,
     provider_service: services.ProviderService,
@@ -76,3 +99,65 @@ def test_get_base_forwards_cache(
     )
 
     assert base._cache_path == fake_path / "cache"
+
+
+@pytest.mark.parametrize(
+    "base_name",
+    [
+        bases.BaseName("ubuntu", "20.04"),
+        bases.BaseName("ubuntu", "22.04"),
+        bases.BaseName("ubuntu", "24.04"),
+        bases.BaseName("ubuntu", "devel"),
+        bases.BaseName("almalinux", "9"),
+    ],
+)
+@pytest.mark.skipif(sys.platform == "win32", reason="no cache on windows")
+def test_get_base_no_cache_if_locked(
+    monkeypatch,
+    mock_register,
+    tmp_path: pathlib.Path,
+    base_name: bases.BaseName,
+    emitter: RecordingEmitter,
+):
+    cache_path = tmp_path / "cache"
+    cache_path.mkdir(exist_ok=True, parents=True)
+
+    # Make a new path object to work around caching the paths and thus getting the
+    # same file descriptor.
+    locked = _maybe_lock_cache(cache_path)
+    assert locked
+    new_cache_path = pathlib.Path(str(cache_path))
+    monkeypatch.setattr("charmcraft.env.get_host_shared_cache_path", lambda: new_cache_path)
+
+    # Can't use the fixture as pyfakefs doesn't handle locks.
+    provider_service = services.ProviderService(
+        app=APP_METADATA,
+        services=None,  # pyright: ignore[reportArgumentType]
+        project=None,  # pyright: ignore[reportArgumentType]
+        work_dir=tmp_path,
+        build_plan=[],
+    )
+
+    base = provider_service.get_base(
+        base_name=base_name,
+        instance_name="charmcraft-test-instance",
+    )
+
+    assert base._cache_path is None
+    emitter.assert_progress(
+        "Shared cache locked by another process; running without cache.",
+        permanent=True,
+    )
+
+
+@pytest.mark.skipif(sys.platform == "win32", reason="no cache on windows")
+def test_maybe_lock_cache_locks_single_lock(tmp_path: pathlib.Path) -> None:
+    assert _maybe_lock_cache(tmp_path)
+
+
+@pytest.mark.skipif(sys.platform == "win32", reason="no cache on windows")
+def test_maybe_lock_cache_with_another_lock(tmp_path: pathlib.Path) -> None:
+    # Need to save the open file so it's not closed when we try a second time.
+    first_file_descriptor = _maybe_lock_cache(tmp_path)
+    assert first_file_descriptor
+    assert _maybe_lock_cache(tmp_path) is None