Repository: pantsbuild/pants

Issue:
`DOCKER_CONFIG` not respected within `pants test`
**Describe the bug**
In GitHub Actions, I can successfully log in to Azure Container Registry, then do docker pull. But if `pants test` tries to do the same, it results in:

```
Exception: Failed to pull image `***registry.azurecr.io/runner/ubuntu20_***`: Failed to pull Docker image `***registry.azurecr.io/runner/ubuntu20_***`: DockerResponseServerError { status_code: 500, message: "Get \"[https://***registry.azurecr.io/v2/runner/ubuntu20_***/tags/list\](https://***registry.azurecr.io/v2/runner/ubuntu20_***/tags/list/)": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information." }
```

I am logging in to ACR using this step:

```
- name: Login to ACR
  uses: azure/docker-login@v1
  with:
    login-server: *****registry.azurecr.io
    username: ${{ secrets.SP_ACR_PULL_ID }}
    password: ${{ secrets.SP_ACR_PULL_PASSWORD }}
```

I can see that the ACR login steps yields this:
DOCKER_CONFIG=/home/runner/work/_temp/docker_login_1675410905572

This ensures that if I run the following step, that is successful:

```
- name: Test pulling from ACR
  run: docker pull *****registry.azurecr.io/runner/ubuntu20_*****
```

However, `pants test` is not as successful.

The Docker image is configured like this:

```
docker_environment(
    name="docker-env-ubuntu20-*****",
    platform="linux_x86_64",
    image="*****registry.azurecr.io/runner/ubuntu20_*****",
    python_bootstrap_search_path=["<PATH>"],
    docker_env_vars=["DOCKER_CONFIG"],
)
```

I have also tried without `docker_env_vars=["DOCKER_CONFIG"]`, and it doesn't seem to make a difference.

**Pants version**
2.15.0rc2

**OS**
Ubuntu on GitHub Actions

**Additional info**
https://pantsbuild.slack.com/archives/C046T6T9U/p1675414472629759
`DOCKER_CONFIG` not respected within `pants test`
**Describe the bug**
In GitHub Actions, I can successfully log in to Azure Container Registry, then do docker pull. But if `pants test` tries to do the same, it results in:

```
Exception: Failed to pull image `***registry.azurecr.io/runner/ubuntu20_***`: Failed to pull Docker image `***registry.azurecr.io/runner/ubuntu20_***`: DockerResponseServerError { status_code: 500, message: "Get \"[https://***registry.azurecr.io/v2/runner/ubuntu20_***/tags/list\](https://***registry.azurecr.io/v2/runner/ubuntu20_***/tags/list/)": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information." }
```

I am logging in to ACR using this step:

```
- name: Login to ACR
  uses: azure/docker-login@v1
  with:
    login-server: *****registry.azurecr.io
    username: ${{ secrets.SP_ACR_PULL_ID }}
    password: ${{ secrets.SP_ACR_PULL_PASSWORD }}
```

I can see that the ACR login steps yields this:
DOCKER_CONFIG=/home/runner/work/_temp/docker_login_1675410905572

This ensures that if I run the following step, that is successful:

```
- name: Test pulling from ACR
  run: docker pull *****registry.azurecr.io/runner/ubuntu20_*****
```

However, `pants test` is not as successful.

The Docker image is configured like this:

```
docker_environment(
    name="docker-env-ubuntu20-*****",
    platform="linux_x86_64",
    image="*****registry.azurecr.io/runner/ubuntu20_*****",
    python_bootstrap_search_path=["<PATH>"],
    docker_env_vars=["DOCKER_CONFIG"],
)
```

I have also tried without `docker_env_vars=["DOCKER_CONFIG"]`, and it doesn't seem to make a difference.

**Pants version**
2.15.0rc2

**OS**
Ubuntu on GitHub Actions

**Additional info**
https://pantsbuild.slack.com/archives/C046T6T9U/p1675414472629759


Discussion:
@torhovland : It looks like this is because `pantsd` purges all environment variables, and so our Docker library can't read `DOCKER_HOST` or `DOCKER_CONFIG`.

Can you try with `--no-pantsd`, and see if you get further? It's possible that we will also need to enable SSL support to work with your registry, but if that's the case you should get a different error.
@torhovland : It looks like this is because `pantsd` purges all environment variables, and so our Docker library can't read `DOCKER_HOST` or `DOCKER_CONFIG`.

Can you try with `--no-pantsd`, and see if you get further? It's possible that we will also need to enable SSL support to work with your registry, but if that's the case you should get a different error.

Resolving patch:
diff --git a/src/python/pants/pantsd/pants_daemon.py b/src/python/pants/pantsd/pants_daemon.py
--- a/src/python/pants/pantsd/pants_daemon.py
+++ b/src/python/pants/pantsd/pants_daemon.py
@@ -36,6 +36,16 @@
 
 _SHUTDOWN_TIMEOUT_SECS = 3
 
+_PRESERVED_ENV_VARS = [
+    # Controls backtrace behavior for rust code.
+    "RUST_BACKTRACE",
+    # The environment variables consumed by the `bollard` crate as of
+    # https://github.com/fussybeaver/bollard/commit/a12c6b21b737e5ea9e6efe5f0128d02dc594f9aa
+    "DOCKER_HOST",
+    "DOCKER_CONFIG",
+    "DOCKER_CERT_PATH",
+]
+
 
 class PantsDaemon(PantsDaemonProcessManager):
     """A daemon that manages PantsService instances."""
@@ -180,9 +190,9 @@ def run_sync(self):
         # Switch log output to the daemon's log stream, and empty `env` and `argv` to encourage all
         # further usage of those variables to happen via engine APIs and options.
         self._close_stdio(pants_log_path(PurePath(global_bootstrap_options.pants_workdir)))
-        with initialize_stdio(global_bootstrap_options), argv_as(
-            tuple()
-        ), hermetic_environment_as():
+        with initialize_stdio(global_bootstrap_options), argv_as(tuple()), hermetic_environment_as(
+            *_PRESERVED_ENV_VARS
+        ):
             # Install signal and panic handling.
             ExceptionSink.install(
                 log_location=init_workdir(global_bootstrap_options), pantsd_instance=True
diff --git a/src/python/pants/util/contextutil.py b/src/python/pants/util/contextutil.py
--- a/src/python/pants/util/contextutil.py
+++ b/src/python/pants/util/contextutil.py
@@ -68,12 +68,22 @@ def _restore_env(env: Mapping[str, str]) -> None:
 
 
 @contextmanager
-def hermetic_environment_as(**kwargs: str | None) -> Iterator[None]:
-    """Set the environment to the supplied values from an empty state."""
+def hermetic_environment_as(*preserve: str, **override: str | None) -> Iterator[None]:
+    """Mutate the environment of this process, restoring it on exit.
+
+    The given `preserve` environment variable names will have their current values preserved, while
+    the given `override` environment variables will override any values which are already set.
+    """
     old_environment = os.environ.copy()
+    preserve_set = set(preserve)
+    new_environment: dict[str, str | None] = {
+        k: v for k, v in old_environment.items() if k in preserve_set
+    }
+    new_environment.update(override)
+
     _purge_env()
     try:
-        with environment_as(**kwargs):
+        with environment_as(**new_environment):
             yield
     finally:
         _purge_env()


Test patch:
diff --git a/src/python/pants/util/contextutil_test.py b/src/python/pants/util/contextutil_test.py
--- a/src/python/pants/util/contextutil_test.py
+++ b/src/python/pants/util/contextutil_test.py
@@ -74,6 +74,9 @@ def test_hermetic_environment(self) -> None:
         with self.ensure_user_defined_in_environment():
             with hermetic_environment_as():
                 assert "USER" not in os.environ
+        with self.ensure_user_defined_in_environment():
+            with hermetic_environment_as("USER"):
+                assert "USER" in os.environ
 
     def test_hermetic_environment_subprocesses(self) -> None:
         with self.ensure_user_defined_in_environment():