Repository: bazelbuild/bazel
Issue: `PATH` not propagated to many darwin actions
Labels: type: bug
P3
team-Rules-CPP

This is a wider reaching version of https://github.com/bazelbuild/bazel/issues/8425

In the linked issue I did some investigation around default PATH variables for actions that at the time I thought only affected `py_*` targets but it turns out that many other actions on darwin do not inherit any `PATH` and therefore fallback to macOS's default shell env of `/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.` which can allow leakage of `/usr/local/*` binaries (which is specifically an issue because of homebrew).

While the default PATH may be ok for some use cases, if you actually try to restrict it using `--action_env=PATH=something` or by executing bazel with `env -i PATH=something bazel ...`, neither of these are applied to these actions.

For example any action that is executed in the context of Xcode is executed with these environment variables:

```
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.15 \
    XCODE_VERSION_OVERRIDE=11.5.0.11E608c \
  external/local_config_cc/wrapped_clang ...
```

Some actions that don't require Xcode are executed with no environment variables:

```
  exec env - \
  bazel-out/host/bin/external/build_bazel_rules_apple/tools/bundletool/bundletool ...
```

This is not the behavior of some other rules like `genrule` where if you build this target with `--action_env=/usr/bin:/bin` it hits the "Valid path":

```bzl
genrule(
    name = "foo",
    outs = ["foo.txt"],
    cmd = """
if [[ "$$PATH" != "/usr/bin:/bin" ]]; then
  echo "error: path not propagated: $$PATH"
  exit 1
else
  echo "Valid path! $$PATH"
  exit 1
fi
""",
)
```

I think that either all actions should get a very conservative default PATH like `/usr/bin:/bin`, or the flags like `--action_env` should be able to add env vars to these actions (I believe the latter is likely the more flexible path, but I'm not sure about the implementation implications).

For native actions it seems that this can be resolved by adding:

```java
.setInheritedEnvironment(ImmutableSet.of("PATH"));
```

to this logic https://github.com/bazelbuild/bazel/blob/8f678923ebea788fc1e03ba8c6359b51edafe0df/src/main/java/com/google/devtools/build/lib/rules/objc/ObjcRuleClasses.java#L140-L144

But for starlark actions they inherit this environment:

https://github.com/bazelbuild/bazel/blob/8f678923ebea788fc1e03ba8c6359b51edafe0df/tools/osx/crosstool/cc_toolchain_config.bzl#L4546-L4557

When accessing the environment variables for an action with the [`cc_common` api](https://docs.bazel.build/versions/master/skylark/lib/cc_common.html#get_environment_variables). There doesn't seem to be a way to craft an `env_entry` that doesn't impose a value. So I could add `/usr/bin:/bin` there, but it wouldn't allow overrides via `--action_env=PATH=foo`

I'm happy to make a change here but I'm hoping for some guidance on the right path!

### What operating system are you running Bazel on?

macOS

### Bazel version?

8f678923ebea788fc1e03ba8c6359b51edafe0df

Issue comments:
@gregestren is this something you'd be able to advise on?
Actually, this may be an Apple Rules issue, not an --action_env issue. @googlewalt , is this in your wheelhouse?
I think there are two issues:
1. AFAICT, based on code inspection, action_env does not work on bazel.  @keith do you know of instances where it works?
2. bazel does not set a default PATH for actions.

In both cases, the issue seems to be that `BazelRulesModule` does not have some setup code we have internally, that does both (1) and (2).
Sorry.  I found the setup code in `BazelRulesModule`.
Have you tried `--incompatible_strict_action_env`?
I have tried that flag, AFAICT that flag is only read here:

https://github.com/bazelbuild/bazel/blob/84834db3a0d5ac2b1c71c5706f259e7dff4a4f0e/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java#L187-L201

and IIRC this codepath isn't even hit, otherwise action_env would work as well:

https://github.com/bazelbuild/bazel/blob/84834db3a0d5ac2b1c71c5706f259e7dff4a4f0e/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java#L202-L207
[edit: this is wrong, see below]

I verified that this behavior is not specific to apple rules, and it has been this way since at least bazel 1.0.  I would have expected the SHELL_ACTION_ENV to be applied to all the rules that run a shell, but I'm not familiar with exactly how that logic works.  The most recent non-trivial refactoring of this code was done in commit 8626623106fc0f9f83781990f7c235074926d196.  Perhaps someone more familiar with this code can comment?

cc @lberki (author of that commit)
I'm certainly not that "someone familiar with this code" :( The commit https://github.com/bazelbuild/bazel/commit/8626623106fc0f9f83781990f7c235074926d196 was meant to be a refactoring without user-visible effects; do you have a reason to believe that this is what caused the behavior change?

Looking at https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java#L205 , it looks like the variables set in `--action_env` should be applied no matter what, which contrary to the statement in the description of this bug. Are you... sure? Or maybe this is something that's specific to Apple rules (I'm even less familiar with those than I am with the computing of the action environment...)

Apologies for the red herring.  My statements from previous post was wrong (user error).

`--action_env` and `--incompatible_strict_action_env` seem to generally work for bazel.  I tested it for `cc_library`, and it also works for me for Apple wrapped clang invocation for Objc compile and linking:

```
bazel build --action_env PATH=/my_special_path:$PATH -s examples/macos/HelloWorld:HelloWorld
...
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.15 \
    PATH=/my_special_path//usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/Library/Apple/usr/bin \
    XCODE_VERSION_OVERRIDE=11.6.0.11E708 \
  external/local_config_cc/wrapped_clang ...
```

But some other Apple actions don't appear to be have default PATHs, or be hooked up to those flags: StoreboardCompile, AssetCatalogCompile, CompileRootInfoPlist, Processing and signing.
No worries! Unfortunately, I won't be able to help further then :(
No actions in rules_apple seems to employ `use_default_shell_env`, which means `--action_env` will be ignored for them.
Are you suggesting all actions should set that? Is that the right switch for this? It surprises me a bit that that would be the way to make this work. Because that means that all non-native rules would need to set that to `True` (and when would you want to set that to `False` given this issue?)

Somewhat related I guess:

https://github.com/bazelbuild/rules_go/blob/b4c392fcfc0a44f2cc5ee3dbb884121dd2ab6957/proto/compiler.bzl#L126-L130
> Are you suggesting all actions should set that? Is that the right switch for this?

The [documentation recommends](https://cs.opensource.google/bazel/bazel/+/4c0febc02fab5e9fda3f40532a62854fb1c45631:src/main/java/com/google/devtools/build/lib/analysis/actions/SpawnAction.java;l=946) all actions set it.

> It surprises me a bit that that would be the way to make this work. Because that means that all non-native rules would need to set that to `True`

Native rules must also opt in, but some effort was historically put into fixing them (#3320).

> (and when would you want to set that to `False` given this issue?)

The `env` parameter to `actions.run(_shell)` is ignored under `use_default_shell_env=True`.
It looks like this is mutually exclusive with setting `env` on these, which doesn't work for Apple actions since they use env to propagate Xcode version info. I guess we might not be able to have it both ways?
Example of the failures caused by this in the Apple toolchain https://github.com/bazelbuild/rules_swift/pull/497

```

(18:30:51) ERROR: /Users/buildkite/builds/bk-imacpro-16/bazel/rules-swift-swift/examples/xplatform/c_from_swift/BUILD:25:14: Compiling Swift module Counter failed: Worker process did not return a WorkResponse:
--
  |  
  | ---8<---8<--- Start of log, file at /private/var/tmp/_bazel_buildkite/022c118cf0acc0d6f57f13796cd5f4e4/bazel-workers/worker-3-SwiftCompile.log ---8<---8<---
  | Error: DEVELOPER_DIR not set.

```
Yeah, it sounds like it might make sense to go back to your first suggestion, to set a conservative default PATH for Apple-related actions. We do need the environment that's set for Apple rules, but it's ok to add to it.
Using `aquery` (assuming that's a fair way to determine this) here's a quick list of mnemonics in our iOS build that don't have a PATH set even when I pass `--action_env=PATH=something`

Action
AssetCatalogCompile
BundleApp
BundleResources
CcStrip
CompileRootInfoPlist
CppCompile
CppLink
CppModuleMap
ExecutableSymlink
ExtractFromProvisioningProfile
FileWrite
GenProtoDescriptorSet
Genrule
ImportedDynamicFrameworkProcessor
Middleman
ObjcCompile
ObjcLink
ProcessAndSign
ProcessEntitlementsFiles
PythonZipper
SolibSymlink
SourceSymlinkManifest
StoryboardCompile
StoryboardLink
SwiftArchive
SwiftStdlibCopy
Symlink
SymlinkTree
TemplateExpand
XibCompile

I'm surprised by some here looking briefly at the code:

https://github.com/bazelbuild/bazel/blob/ecab72e74fa5e922dc18a5b0d059141d81bacdf8/src/main/java/com/google/devtools/build/lib/rules/cpp/CppHelper.java#L726-L740
Thinking about this more I kinda thing we should make action_env apply to all actions instead. I think that makes sense from a user perspective and it would solve more issues than just the PATH one. Maybe we would also provide a default PATH though. Thoughts?
So there was some confusion with the list above, the problem is that some actions don't print their env with aquery. I'll try to submit a follow up for some of those. This fixes some of my use case https://github.com/bazelbuild/rules_swift/pull/502

but it's not perfect. Firstly piping `ctx.configuration.default_shell_env` through everywhere can be cumbersome. But more importantly that doesn't include everything we actually want. This only includes the "fixed" environment variables, like those passed with `--action_env=FOO=BAR`, not the "variable" environment variables which are attempted to be passed through with `--action_env=FOO`

https://github.com/bazelbuild/bazel/blob/080547f7af998dd45c14f05d426e7c8f60a68a2f/src/main/java/com/google/devtools/build/lib/analysis/config/BuildConfiguration.java#L486-L500

There doesn't seem to be a way to get any of the other variables from starlark, but I might be missing something?
The CppLinkAction is one of the ones where its environment doesn't show up in aquery because it doesn't implement SpawnAction
Ran into a variant of this, switching over to an Apple Silicon machine.

It's worth noting that on Apple Silicon, Homebrew no longer installs into the default PATH but rather to /opt/homebrew/bin.

That said, if we do get into this and change defaults, I think it might well make sense to include Homebrew installs in the strict_action_env default. I get the hermeticity concerns, but it's also probably a good idea to default to the tools the user is more likely to expect, rather than, e.g., use the 2007 era bash, old python, and other os-bundled binaries that I think most users replace with those installed from brew. Always possible to make it more restrictive with --action_env, ofc.
@Keith, what are your thoughts these days on the desirability of having homebrew tools on the action path?
(Sounded negative above, but also now needed to conveniently get e.g. python3 and a version of bash that's newer than the system's ~2007 vintage.) 
At least with python3 macOS has been keeping relatively up to date so I'm happy to be using that version since it's the same for everyone (assuming you only maintain 1 Xcode version). For the bash thing we just deal with the old one. I think the downside we saw before we started excluding the brew path was just people's brew states could be all over the place depending on when things were last updated, so it added a lot of variance.
Got it. Thanks, @Keith.
(We have a brew-update thing+use a bunch of add'l tools in scripts that are more easily installed that way. I think that flips the personal calculus but good to know & I therefore won't try to upstream or anything.)
I also believe that since brew has thousands of formulae in their core repo alone, and most developers are probably using at least _some_ binaries provided by brew (but can be widely-varying and selected according to personal habits), that including it by default opens the door to a ton of subtle variances. I think a lot of developers who are just "using" Bazel as their build system because it's what their projects use, can then easily run into issues that may be hard to debug.

The GNU bash on macOS is old, yes, but since macOS has also shipped with an immutable zsh since 2015 there's now also opportunity for Darwin-specific wrapper scripts to use /bin/zsh if there's a desire for more modern shell scripting language features.
So what's the conclusion here?

Making copious PRs to rule sets to set `use_default_shell_env = True`? Any reason that isn't a good idea?
https://github.com/bazelbuild/bazel/pull/18235 would be the best of both worlds I think. In my case right now we can't get it to work 100% because values like `--action_env=FOO` (where the value should be passed through) isn't something we can fetch in starlark.
If possible definitely set `use_default_shell_env = True` and move on though
A fix for this issue has been included in [Bazel 6.4.0 RC1](https://releases.bazel.build/6.4.0/rc1/index.html). Please test out the release candidate and report any issues as soon as possible. Thanks!
for others note that the change is only if you enable `--incompatible_merge_fixed_and_default_shell_env` and make some patches to rules_apple / rules_swift / apple_support, with bazel 7.x+ this will be the default

Pull request review comments:

Patch that resolved the issue:
diff --git a/src/main/java/com/google/devtools/build/lib/analysis/actions/SpawnAction.java b/src/main/java/com/google/devtools/build/lib/analysis/actions/SpawnAction.java
index 43d34f565f99ff..c2c1cfe9b46736 100644
--- a/src/main/java/com/google/devtools/build/lib/analysis/actions/SpawnAction.java
+++ b/src/main/java/com/google/devtools/build/lib/analysis/actions/SpawnAction.java
@@ -28,6 +28,7 @@
 import com.google.common.collect.Interner;
 import com.google.common.collect.Iterables;
 import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.collect.Sets;
 import com.google.devtools.build.lib.actions.AbstractAction;
 import com.google.devtools.build.lib.actions.ActionContinuationOrResult;
 import com.google.devtools.build.lib.actions.ActionEnvironment;
@@ -683,7 +684,6 @@ public static class Builder {
     private final List<Artifact> outputs = new ArrayList<>();
     private final List<RunfilesSupplier> inputRunfilesSuppliers = new ArrayList<>();
     private ResourceSetOrBuilder resourceSetOrBuilder = AbstractAction.DEFAULT_RESOURCE_SET;
-    private ActionEnvironment actionEnvironment = null;
     private ImmutableMap<String, String> environment = ImmutableMap.of();
     private ImmutableSet<String> inheritedEnvironment = ImmutableSet.of();
     private ImmutableMap<String, String> executionInfo = ImmutableMap.of();
@@ -717,7 +717,6 @@ public Builder(Builder other) {
       this.outputs.addAll(other.outputs);
       this.inputRunfilesSuppliers.addAll(other.inputRunfilesSuppliers);
       this.resourceSetOrBuilder = other.resourceSetOrBuilder;
-      this.actionEnvironment = other.actionEnvironment;
       this.environment = other.environment;
       this.executionInfo = other.executionInfo;
       this.isShellCommand = other.isShellCommand;
@@ -762,12 +761,31 @@ public SpawnAction build(ActionOwner owner, BuildConfigurationValue configuratio
         result.addCommandLine(pair);
       }
       CommandLines commandLines = result.build();
-      ActionEnvironment env =
-          actionEnvironment != null
-              ? actionEnvironment
-              : useDefaultShellEnvironment
-                  ? configuration.getActionEnvironment()
-                  : ActionEnvironment.create(environment, inheritedEnvironment);
+      ActionEnvironment env;
+      if (useDefaultShellEnvironment && environment != null) {
+        // Inherited variables override fixed variables in ActionEnvironment. Since we want the
+        // fixed part of the action-provided environment to override the inherited part of the
+        // user-provided environment, we have to explicitly filter the inherited part.
+        var userFilteredInheritedEnv =
+            ImmutableSet.copyOf(
+                Sets.difference(
+                    configuration.getActionEnvironment().getInheritedEnv(), environment.keySet()));
+        // Do not create a new ActionEnvironment in the common case where no vars have been filtered
+        // out.
+        if (userFilteredInheritedEnv.equals(
+            configuration.getActionEnvironment().getInheritedEnv())) {
+          env = configuration.getActionEnvironment();
+        } else {
+          env =
+              ActionEnvironment.create(
+                  configuration.getActionEnvironment().getFixedEnv(), userFilteredInheritedEnv);
+        }
+        env = env.withAdditionalFixedVariables(environment);
+      } else if (useDefaultShellEnvironment) {
+        env = configuration.getActionEnvironment();
+      } else {
+        env = ActionEnvironment.create(environment, inheritedEnvironment);
+      }
       return buildSpawnAction(
           owner, commandLines, configuration.getCommandLineLimits(), configuration, env);
     }
@@ -984,13 +1002,6 @@ public Builder setResources(ResourceSetOrBuilder resourceSetOrBuilder) {
       return this;
     }
 
-    /** Sets the action environment. */
-    @CanIgnoreReturnValue
-    public Builder setEnvironment(ActionEnvironment actionEnvironment) {
-      this.actionEnvironment = actionEnvironment;
-      return this;
-    }
-
     /**
      * Sets the map of environment variables. Do not use! This makes the builder ignore the 'default
      * shell environment', which is computed from the --action_env command line option.
@@ -1075,6 +1086,18 @@ public Builder executeUnconditionally() {
       return this;
     }
 
+    /**
+     * Same as {@link #useDefaultShellEnvironment()}, but additionally sets the provided fixed
+     * environment variables, which take precedence over the variables contained in the default
+     * shell environment.
+     */
+    @CanIgnoreReturnValue
+    public Builder useDefaultShellEnvironmentWithOverrides(Map<String, String> environment) {
+      this.environment = ImmutableMap.copyOf(environment);
+      this.useDefaultShellEnvironment = true;
+      return this;
+    }
+
     /**
      * Sets the executable path; the path is interpreted relative to the execution root, unless it's
      * a bare file name.
diff --git a/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkActionFactory.java b/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkActionFactory.java
index ca8d46a9ce1671..b6058da30c971b 100644
--- a/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkActionFactory.java
+++ b/src/main/java/com/google/devtools/build/lib/analysis/starlark/StarlarkActionFactory.java
@@ -671,15 +671,24 @@ private void registerStarlarkAction(
     } catch (IllegalArgumentException e) {
       throw Starlark.errorf("%s", e.getMessage());
     }
-    if (envUnchecked != Starlark.NONE) {
-      builder.setEnvironment(
-          ImmutableMap.copyOf(Dict.cast(envUnchecked, String.class, String.class, "env")));
-    }
     if (progressMessage != Starlark.NONE) {
       builder.setProgressMessageFromStarlark((String) progressMessage);
     }
+
+    ImmutableMap<String, String> env = null;
+    if (envUnchecked != Starlark.NONE) {
+      env = ImmutableMap.copyOf(Dict.cast(envUnchecked, String.class, String.class, "env"));
+    }
     if (Starlark.truth(useDefaultShellEnv)) {
-      builder.useDefaultShellEnvironment();
+      if (env != null
+          && getSemantics()
+              .getBool(BuildLanguageOptions.INCOMPATIBLE_MERGE_FIXED_AND_DEFAULT_SHELL_ENV)) {
+        builder.useDefaultShellEnvironmentWithOverrides(env);
+      } else {
+        builder.useDefaultShellEnvironment();
+      }
+    } else if (env != null) {
+      builder.setEnvironment(env);
     }
 
     RuleContext ruleContext = getRuleContext();
diff --git a/src/main/java/com/google/devtools/build/lib/packages/semantics/BuildLanguageOptions.java b/src/main/java/com/google/devtools/build/lib/packages/semantics/BuildLanguageOptions.java
index 2259ab5a68e7d5..469427b6f32856 100644
--- a/src/main/java/com/google/devtools/build/lib/packages/semantics/BuildLanguageOptions.java
+++ b/src/main/java/com/google/devtools/build/lib/packages/semantics/BuildLanguageOptions.java
@@ -662,6 +662,19 @@ public final class BuildLanguageOptions extends OptionsBase {
       help = "If enabled, targets that have unknown attributes set to None fail.")
   public boolean incompatibleFailOnUnknownAttributes;
 
+  @Option(
+      name = "incompatible_merge_fixed_and_default_shell_env",
+      defaultValue = "false",
+      documentationCategory = OptionDocumentationCategory.STARLARK_SEMANTICS,
+      effectTags = {OptionEffectTag.LOADING_AND_ANALYSIS},
+      metadataTags = {OptionMetadataTag.INCOMPATIBLE_CHANGE},
+      help =
+          "If enabled, actions registered with ctx.actions.run and ctx.actions.run_shell with both"
+              + " 'env' and 'use_default_shell_env = True' specified will use an environment"
+              + " obtained from the default shell environment by overriding with the values passed"
+              + " in to 'env'. If disabled, the value of 'env' is completely ignored in this case.")
+  public boolean incompatibleMergeFixedAndDefaultShellEnv;
+  
   @Option(
       name = "incompatible_disable_objc_library_transition",
       defaultValue = "false",
@@ -765,6 +778,9 @@ public StarlarkSemantics toStarlarkSemantics() {
                 EXPERIMENTAL_GET_FIXED_CONFIGURED_ACTION_ENV,
                 experimentalGetFixedConfiguredEnvironment)
             .setBool(INCOMPATIBLE_FAIL_ON_UNKNOWN_ATTRIBUTES, incompatibleFailOnUnknownAttributes)
+            .setBool(
+                INCOMPATIBLE_MERGE_FIXED_AND_DEFAULT_SHELL_ENV,
+                incompatibleMergeFixedAndDefaultShellEnv)
             .setBool(
                 INCOMPATIBLE_DISABLE_OBJC_LIBRARY_TRANSITION,
                 incompatibleDisableObjcLibraryTransition)
@@ -858,6 +874,8 @@ public StarlarkSemantics toStarlarkSemantics() {
       "-experimental_get_fixed_configured_action_env";
   public static final String INCOMPATIBLE_FAIL_ON_UNKNOWN_ATTRIBUTES =
       "-incompatible_fail_on_unknown_attributes";
+  public static final String INCOMPATIBLE_MERGE_FIXED_AND_DEFAULT_SHELL_ENV =
+      "-experimental_merge_fixed_and_default_shell_env";
   public static final String INCOMPATIBLE_DISABLE_OBJC_LIBRARY_TRANSITION =
       "-incompatible_disable_objc_library_transition";
 


Test patch:
diff --git a/src/test/shell/integration/action_env_test.sh b/src/test/shell/integration/action_env_test.sh
index 08af2abd059630..6feaf075740bb0 100755
--- a/src/test/shell/integration/action_env_test.sh
+++ b/src/test/shell/integration/action_env_test.sh
@@ -39,6 +39,15 @@ load("//pkg:build.bzl", "environ")
 
 environ(name = "no_default_env", env = 0)
 environ(name = "with_default_env", env = 1)
+environ(
+    name = "with_default_and_fixed_env",
+    env = 1,
+    fixed_env = {
+        "ACTION_FIXED": "action",
+        "ACTION_AND_CLIENT_FIXED": "action",
+        "ACTION_AND_CLIENT_INHERITED": "action",
+    },
+)
 
 sh_test(
     name = "test_env_foo",
@@ -72,12 +81,16 @@ def _impl(ctx):
   ctx.actions.run_shell(
       inputs=[],
       outputs=[output],
+      env = ctx.attr.fixed_env,
       use_default_shell_env = ctx.attr.env,
       command="env > %s" % output.path)
 
 environ = rule(
     implementation=_impl,
-    attrs={"env": attr.bool(default=True)},
+    attrs={
+        "env": attr.bool(default=True),
+        "fixed_env": attr.string_dict(),
+    },
     outputs={"out": "%{name}.env"},
 )
 EOF
@@ -222,6 +235,52 @@ function test_use_default_shell_env {
          && fail "dynamic action_env used, even though requested not to") || true
 }
 
+function test_use_default_shell_env_and_fixed_env {
+    ACTION_AND_CLIENT_INHERITED=client CLIENT_INHERITED=client \
+        bazel build \
+        --noincompatible_merge_fixed_and_default_shell_env \
+        --action_env=ACTION_AND_CLIENT_FIXED=client \
+        --action_env=ACTION_AND_CLIENT_INHERITED \
+        --action_env=CLIENT_FIXED=client \
+        --action_env=CLIENT_INHERITED \
+        //pkg:with_default_and_fixed_env
+    echo
+    cat bazel-bin/pkg/with_default_and_fixed_env.env
+    echo
+    grep -q ACTION_AND_CLIENT_FIXED=client bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "static action environment not honored"
+    grep -q ACTION_AND_CLIENT_INHERITED=client bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "dynamic action environment not honored"
+    grep -q ACTION_FIXED bazel-bin/pkg/with_default_and_fixed_env.env \
+        && fail "fixed env provided by action should have been ignored"
+    grep -q CLIENT_FIXED=client bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "static action environment not honored"
+    grep -q CLIENT_INHERITED=client bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "dynamic action environment not honored"
+
+    ACTION_AND_CLIENT_INHERITED=client CLIENT_INHERITED=client \
+        bazel build \
+        --incompatible_merge_fixed_and_default_shell_env \
+        --action_env=ACTION_AND_CLIENT_FIXED=client \
+        --action_env=ACTION_AND_CLIENT_INHERITED \
+        --action_env=CLIENT_FIXED=client \
+        --action_env=CLIENT_INHERITED \
+        //pkg:with_default_and_fixed_env
+    echo
+    cat bazel-bin/pkg/with_default_and_fixed_env.env
+    echo
+    grep -q ACTION_AND_CLIENT_FIXED=action bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "action-provided env should have overridden static --action_env"
+    grep -q ACTION_AND_CLIENT_INHERITED=action bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "action-provided env should have overridden dynamic --action_env"
+    grep -q ACTION_FIXED=action bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "action-provided env should have been honored"
+    grep -q CLIENT_FIXED=client bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "static action environment not honored"
+    grep -q CLIENT_INHERITED=client bazel-bin/pkg/with_default_and_fixed_env.env \
+        || fail "dynamic action environment not honored"
+}
+
 function test_action_env_changes_honored {
     # Verify that changes to the explicitly specified action_env in honored in
     # tests. Regression test for #3265.