Repository: pantsbuild/pants

Issue:
Platform/Environment-specific subsystems
## Problem
When running a subprocess in V2 a la `ExecuteProcessRequest`, the default is to use a completely hermetic environment. However, certain processes need environment variables to work properly, such as:

* `LC_ALL / LANG` for Python processes to know to use `UTF-8`
*  `CPPFLAGS / LDFLAGS` when on macOS so that Pex can resolve cryptography

While a naive solution would simply grab the local values, as done in https://github.com/pantsbuild/pants/pull/7721, this presents an issue with remoting that the remote environment should be able to have its values defined. See https://github.com/pantsbuild/pants/pull/7721#discussion_r283988645.

We need some way to
1) Configure what build environment variables Pants should use for subprocesses.
    1. Likely will want to default to the local system's values. 
2) Connect that Subsystem to the specific `ExecuteProcessRequest`.
    1. Each `ExecuteProcessRequest` will not need the entire set of environment variables in the Subsystem(s). For example, PEX requirements processes only need `CPPFLAGS` and `LDFLAGS`, and only if they are on macOS. We want to give the minimum amount of settings as possible to keep the environment more hermetic and to increase cache hits.

## Designs

### Platform subsystems
We discussed creating a scoped instance of the subsystem for both Linux and macOS. We know that certain platforms come with certain expectations, like that macOS will need `CPPFLAGS` and `LDFLAGS` to get cryptography properly resolved but that Linux does not need this.

Because they would be scoped instances, the majority of their values would be populated from the parent subsystem and only their relevant parts would be changed.

Beyond solving this particular goal, it sounds like the `Platform` abstraction would be a useful one for V2 in general.


Discussion:
I believe a lot of this exists verbatim in the native backend? Is there an ideal format in which I can draw out how this could be hooked up? All of it is available immediately in v2 rules, and performs exactly this kind of platform-specific conditional logic -- thinking about the remoting use case is exactly why I designed it that way. Extending it to invoke other processes from other language backends which use native code and adding flags like `CPPFLAGS` to invoke processes hermetically is exactly what's done in #6893, for example.
@Eric-Arellano: The description looks great, thanks!

@cosmicexplorer: If the existing rules might be helpful now, would you mind commenting on #7721 with a minimal set of things that @Eric-Arellano can do to get unblocked?

There is another important aspect to this ticket, which is how the "Platform-specific-Subsystem" instances would be constructed. We had discussed having `@rules` to construct those instances by consuming a `Platform` Param (that would be injected at the root for the relevant platform), but we didn't quite nail down how-or-whether an `ExecuteProcessRequest` would directly consume that param to constrain its execution target platform.
> There is another important aspect to this ticket, which is how the "Platform-specific-Subsystem" instances would be constructed. We had discussed having `@rule`s to construct those instances by consuming a Platform Param (that would be injected at the root for the relevant platform), but we didn't quite nail down how-or-whether an ExecuteProcessRequest would directly consume that param to constrain its execution target platform.

Sure -- but that's what `native_toolchain.py` is for. Does something like: https://github.com/pantsbuild/pants/blob/46c22fa9917dda2b0cf2c1d5880b8223454ae11d/src/python/pants/backend/native/subsystems/native_toolchain.py#L101-L108 seem to do something like what we're talking about? If we're looking for some kind of `Platform` key into the remote execution action (to ensure it runs e.g. in a Linux environment), that would mostly seem to be concern for the remote execution backend (and hopefully not applicable yet). The contents of #6893 seem to pretty precisely mirror the goals of the current PR description, and that code is ~ready for primetime.

> @cosmicexplorer: If the existing rules might be helpful now, would you mind commenting on #7721 with a minimal set of things that @Eric-Arellano can do to get unblocked?

I *thought* I did that -- if there are things that remain unclear, perhaps either of you could please highlight them for me?
Also note that there is specific precedent for setting env vars specifically for subprocesses via subsystem/v2 ruleset in the native backend, starting hereish: https://github.com/pantsbuild/pants/blob/46c22fa9917dda2b0cf2c1d5880b8223454ae11d/src/python/pants/backend/native/config/environment.py#L173-L176
I wrote a very long comment, and then decided to turn it into a doc instead: https://docs.google.com/document/d/1AWIvRRGzmE3q5IXiRCqvLPCo0BkqX_9y4F9B0KV_9h8/edit?usp=sharing (comment access via a `pants-devel@` account).
I've opened #8090 to represent an "MVP" for platform-awareness, where we at least don't accidentally remote something that we cannot.
A common thread between #9760, #10352, the [Multi-Platform Speculation](https://docs.google.com/document/d/1hS0n6vp-hBxy4Xo-q9QA_elofzQtOubjfN4nSeIgp90/edit#) doc, and recent issues with discovering a bootstrap python to use: we need a native facility for per-host/platform/environment configuration.

The description of this ticket remains a seemingly viable solution, so I'm going to look into implementing an MVP of this to add native support for `Subsystems` that are marked "per-environment", and use one such subsystem to set 1) the path PATH for Python executions in #9760, 2) a bootstrap python to use for pex.

EDIT: After looking at this a little bit more deeply, I think that it should be forwards compatible to begin adding these kinds of options on Subsystems, and then scope them later on after actually implementing the `environment`/`platform`-specificity aspect. Will punt for now and pick up #9760 to that end.
related to the example given in description: https://github.com/pantsbuild/pants/pull/10762 
This is related to cross-platform speculation (https://docs.google.com/document/d/1hS0n6vp-hBxy4Xo-q9QA_elofzQtOubjfN4nSeIgp90/edit#), but also to adding native support for executing processes inside of docker containers (which is a form of cross-building).

EDIT: Now covered by #11148.
https://github.com/pantsbuild/pants/issues/13451#issuecomment-956863674 makes the good point that the `interpreter_search_path` is also potentially platform specific.
Following up re https://github.com/pantsbuild/pants/issues/13451#issuecomment-956863674, if we currently have a repo with linux and macos developers how should we handle `interpreter_search_paths`? Make everyone use pyenv?
> Following up re [#13451 (comment)](https://github.com/pantsbuild/pants/issues/13451#issuecomment-956863674), if we currently have a repo with linux and macos developers how should we handle `interpreter_search_paths`? Make everyone use pyenv?

That is one good approach, yea. Another would be to have standardized installs on both macOS and Linux, and then including only those locations in the search path.
Another use case for this functionality I do not believe is addressed yet is when using remote execution with kubernetes pants should respect the remote machines environment variables. 

For example in my case I have configured BuildBarn to run on kubernetes, and installed docker on the runner nodes. This allows pants to build the docker image remotely using REAPI. 

For this implementation we chose the more secure route of creating a side-car container running docker-dind and use the environment variable DOCKER_HOST to point docker to the dind endpoint in the side-car container.

To get this to function currently I have to specify DOCKER_HOST on the local machine to that of what the value is on the remote machine. Ideally pants would respect the remote machines environment variable so a user does not have to change local environment variables to change the remote configuration.

> To get this to function currently I have to specify DOCKER_HOST on the local machine to that of what the value is on the remote machine. Ideally pants would respect the remote machines environment variable so a user does not have to change local environment variables to change the remote configuration.

@boarder7395 Pants can't do that. Remote execution is via the standard specified here: https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto

In particular, for command environment variables, right here: https://github.com/bazelbuild/remote-apis/blob/04784f4a830cc0df1f419a492cde9fc323f728db/build/bazel/remote/execution/v2/remote_execution.proto#L523-L554

There is no way currently in the reapi for a client (Pants) to specify a list of environment variable names to let leak through from the remote environment into the remote command.
#11148 and #13682 cover cross-building using remote execution, and docker (respectively).
This is very close to being done. All that's left is documenting the new feature.

Make public & add help:

https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/python/pants/core/util_rules/environments.py#L133-L142

Finish help:

https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/python/pants/core/util_rules/environments.py#L52-L67

Rename subsystem:

https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/python/pants/core/util_rules/environments.py#L44

The above makes it possible to do things like define different env-specific config for macOS vs Linux, and have Pants automatically use the right config.

--

Environments are also complete for remote execution and docker environments (https://github.com/pantsbuild/pants/issues/13682), which solves an original motivating problem of how it is nearly impossible to have config that works both for remote execution and the localhost.

What remains:

Add help for `environment` field and mark public:

https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/python/pants/core/util_rules/environments.py#L79-L87

Finish help:

https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/python/pants/core/util_rules/environments.py#L145-L156

Finish help and make public:

https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/python/pants/core/util_rules/environments.py#L222-L229

https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/python/pants/core/util_rules/environments.py#L273-L294




Resolving patch:
diff --git a/src/python/pants/core/util_rules/environments.py b/src/python/pants/core/util_rules/environments.py
--- a/src/python/pants/core/util_rules/environments.py
+++ b/src/python/pants/core/util_rules/environments.py
@@ -61,7 +61,9 @@ class EnvironmentsSubsystem(Subsystem):
                 linux_ci = "build-support:linux_ci_env"
                 macos_ci = "build-support:macos_ci_env"
 
-            TODO(#7735): explain how names are used once they are consumed.
+            To use an environment for a given target, specify the name in the `_environment` field
+            on that target. Pants will consume the environment target at the address mapped from
+            that name.
 
             Pants will ignore any environment targets that are not given a name via this option.
             """
@@ -81,8 +83,16 @@ class EnvironmentField(StringField):
     default = LOCAL_ENVIRONMENT_MATCHER
     value: str
     help = softwrap(
-        """
-        TODO(#7735): fill this in.
+        f"""
+        Specify which environment target to consume environment-sensitive options from.
+
+        Once environments are defined in `[environments-preview].names`, you can specify the environment
+        for this target by its name. Any fields that are defined in that environment will override
+        the values from options set by `pants.toml`, command line values, or environment variables.
+
+        You can specify multiple valid environments by using `parametrize`. If
+        `{LOCAL_ENVIRONMENT_MATCHER}` is specified, Pants will fall back to the `local_environment`
+        defined for the current platform, or no environment if no such environment exists.
         """
     )
 
@@ -131,13 +141,20 @@ class LocalFallbackEnvironmentField(FallbackEnvironmentField):
 
 
 class LocalEnvironmentTarget(Target):
-    alias = "_local_environment"
+    alias = "local_environment"
     core_fields = (*COMMON_TARGET_FIELDS, CompatiblePlatformsField, LocalFallbackEnvironmentField)
     help = softwrap(
-        """
-        Configuration of environment variables and search paths for running Pants locally.
+        f"""
+        Configuration of environment-sensitive options, such as environment variables and search
+        paths when Pants runs subprocesses locally.
 
-        TODO(#7735): Explain how this gets used once we allow targets to set environment.
+        To use this environment, map this target's address with a memorable name in
+        `[environments-preview].names`. You can then consume this environment by specifying the name in
+        the `_environment` field defined on other targets.
+
+        Only one `local_environment` may be defined in `[environments-preview].names` per platform, and
+        when `{LOCAL_ENVIRONMENT_MATCHER}` is specified as the environment, the
+        `local_environment` that matches the current platform (if defined) will be selected.
         """
     )
 
@@ -148,10 +165,11 @@ class DockerImageField(StringField):
     value: str
     help = softwrap(
         """
-        The docker image ID to use when this environment is loaded, e.g. `centos6:latest`.
+        The docker image ID to use when this environment is loaded.
 
-        TODO: expectations about what are valid IDs, e.g. if they must come from DockerHub vs other
-        registries.
+        This value may be any image identifier that the local Docker installation can accept.
+        This includes image names with or without tags (e.g. `centos6` or `centos6:latest`), or
+        image names with an immutable digest (e.g. `centos@sha256:<some_sha256_value>`).
         """
     )
 
@@ -213,7 +231,7 @@ class DockerFallbackEnvironmentField(FallbackEnvironmentField):
 
 
 class DockerEnvironmentTarget(Target):
-    alias = "_docker_environment"
+    alias = "docker_environment"
     core_fields = (
         *COMMON_TARGET_FIELDS,
         DockerImageField,
@@ -222,10 +240,13 @@ class DockerEnvironmentTarget(Target):
     )
     help = softwrap(
         """
-        Configuration of a Docker image used for building your code, including the environment
-        variables and search paths used by Pants.
+        Configuration of a Docker environment used for building your code, including the Docker
+        image, along with environment-sensitive options, which include environment variables and
+        search paths, used by Pants to execute processes in this Docker environment.
 
-        TODO(#7735): Explain how this gets used once we allow targets to set environment.
+        To use this environment, map this target's address with a memorable name in
+        `[environments-preview].names`. You can then consume this environment by specifying the name in
+        the `_environment` field defined on other targets.
         """
     )
 
@@ -272,7 +293,7 @@ class RemoteFallbackEnvironmentField(FallbackEnvironmentField):
 
 
 class RemoteEnvironmentTarget(Target):
-    alias = "_remote_environment"
+    alias = "remote_environment"
     core_fields = (
         *COMMON_TARGET_FIELDS,
         RemotePlatformField,
@@ -282,12 +303,17 @@ class RemoteEnvironmentTarget(Target):
     help = softwrap(
         """
         Configuration of a remote execution environment used for building your code, including the
-        environment variables and search paths used by Pants.
+        environment-sensitive options, which include environment variables and
+        search paths, used by Pants to execute processes in this remote environment.
 
         Note that you must also configure remote execution with the global options like
         `remote_execution` and `remote_execution_address`.
 
-        Often, it is only necessary to have a single `_remote_environment` target for your
+        To use this environment, map this target's address with a memorable name in
+        `[environments-preview].names`. You can then consume this environment by specifying the name in
+        the `_environment` field defined on other targets.
+
+        Often, it is only necessary to have a single `remote_environment` target for your
         repository, but it can be useful to have >1 so that you can set different
         `extra_platform_properties`. For example, with some servers, you could use this to
         configure a different Docker image per environment.
@@ -419,7 +445,7 @@ async def determine_local_environment(
     raise AmbiguousEnvironmentError(
         softwrap(
             f"""
-            Multiple `_local_environment` targets from `[environments-preview].names`
+            Multiple `local_environment` targets from `[environments-preview].names`
             are compatible with the current platform `{platform.value}`, so it is ambiguous
             which to use:
             {sorted(tgt.address.spec for _name, tgt in compatible_name_and_targets)}
@@ -615,8 +641,8 @@ async def get_target_for_environment_name(
         raise ValueError(
             softwrap(
                 f"""
-                Expected to use the address to a `_local_environment`, `_docker_environment`, or
-                `_remote_environment` target in the option `[environments-preview].names`, but the name
+                Expected to use the address to a `local_environment`, `docker_environment`, or
+                `remote_environment` target in the option `[environments-preview].names`, but the name
                 `{env_name.val}` was set to the target {address.spec} with the target type
                 `{tgt.alias}`.
                 """


Test patch:
diff --git a/src/python/pants/core/util_rules/environments_integration_test.py b/src/python/pants/core/util_rules/environments_integration_test.py
--- a/src/python/pants/core/util_rules/environments_integration_test.py
+++ b/src/python/pants/core/util_rules/environments_integration_test.py
@@ -19,7 +19,7 @@ def test_unrecognized_build_file_symbols_during_bootstrap() -> None:
 
         # TODO(#7735): Once we migrate the Shell backend to use environments, add one of its
         #  plugin fields here
-        _local_environment(name='env')
+        local_environment(name='env')
         """
     )
     with setup_tmpdir({"BUILD": build_file}) as tmpdir:
@@ -32,7 +32,7 @@ def test_unrecognized_build_file_symbols_during_bootstrap() -> None:
 
 
 def test_environment_sensitive_option_fields_exist() -> None:
-    pants = run_pants(["help", "_local_environment"])
+    pants = run_pants(["help", "local_environment"])
     pants.assert_success()
     assert "python_bootstrap_names" in pants.stdout
     assert "python_bootstrap_search_path" in pants.stdout
diff --git a/src/python/pants/core/util_rules/environments_test.py b/src/python/pants/core/util_rules/environments_test.py
--- a/src/python/pants/core/util_rules/environments_test.py
+++ b/src/python/pants/core/util_rules/environments_test.py
@@ -145,10 +145,10 @@ def test_all_environments(rule_runner: RuleRunner) -> None:
         {
             "BUILD": dedent(
                 """\
-                _local_environment(name='e1')
-                _local_environment(name='e2')
-                _local_environment(name='no-name')
-                _docker_environment(name='docker', image="centos6:latest")
+                local_environment(name='e1')
+                local_environment(name='e2')
+                local_environment(name='no-name')
+                docker_environment(name='docker', image="centos6:latest")
                 """
             )
         }
@@ -169,10 +169,10 @@ def test_choose_local_environment(rule_runner: RuleRunner) -> None:
         {
             "BUILD": dedent(
                 """\
-                _local_environment(name='e1')
-                _local_environment(name='e2')
-                _local_environment(name='not-compatible', compatible_platforms=[])
-                _docker_environment(name='docker', docker_image="centos6:latest")
+                local_environment(name='e1')
+                local_environment(name='e2')
+                local_environment(name='not-compatible', compatible_platforms=[])
+                docker_environment(name='docker', docker_image="centos6:latest")
                 """
             )
         }
@@ -203,14 +203,14 @@ def test_resolve_environment_name(rule_runner: RuleRunner) -> None:
         {
             "BUILD": dedent(
                 """\
-                _local_environment(name='local')
-                _local_environment(
+                local_environment(name='local')
+                local_environment(
                     name='local-fallback', compatible_platforms=[], fallback_environment='local'
                 )
-                _docker_environment(name='docker', image="centos6:latest")
-                _remote_environment(name='remote-no-fallback')
-                _remote_environment(name='remote-fallback', fallback_environment="docker")
-                _remote_environment(name='remote-bad-fallback', fallback_environment="fake")
+                docker_environment(name='docker', image="centos6:latest")
+                remote_environment(name='remote-no-fallback')
+                remote_environment(name='remote-fallback', fallback_environment="docker")
+                remote_environment(name='remote-bad-fallback', fallback_environment="fake")
                 """
             )
         }
@@ -365,7 +365,7 @@ def create_docker_tgt(
 
 
 def test_resolve_environment_tgt(rule_runner: RuleRunner) -> None:
-    rule_runner.write_files({"BUILD": "_local_environment(name='env')"})
+    rule_runner.write_files({"BUILD": "local_environment(name='env')"})
     rule_runner.set_options(
         ["--environments-preview-names={'env': '//:env', 'bad-address': '//:fake'}"]
     )
diff --git a/src/python/pants/engine/internals/platform_rules_test.py b/src/python/pants/engine/internals/platform_rules_test.py
--- a/src/python/pants/engine/internals/platform_rules_test.py
+++ b/src/python/pants/engine/internals/platform_rules_test.py
@@ -141,7 +141,7 @@ def test_docker_complete_env_vars() -> None:
         {
             "BUILD": dedent(
                 """\
-                _docker_environment(
+                docker_environment(
                     name='docker',
                     image='centos@sha256:a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc',
                     platform='linux_x86_64',