Repository: pantsbuild/pants

Issue:
Support for Environment variables in BUILD files.
**Is your feature request related to a problem? Please describe.**
In two cases I've written a plugin that needs to access environment variables, and will need to write a third. Two of which, are documented cases for writing plugins.

- Plugin to take the `$VERSION` variable and insert it into `setup_py()`
- Username/Password for a PyPi repository
- (to come) take `$VERSION` and use it as the `docker_image(version=` .

It would be much easier if by default these fields could take environment variables by default.

**Describe the solution you'd like**

An `Env` 'function' in `BUILD` files to denote the value of that field is to be taken from an environment variable.

```
python_distribution(
  ...
  setup_py(
    version=Env('VERSION'),
  )
)

docker_image(
  ...
  version=Env('VERSION'),
)

pypi_repository(
  password=Env('PYPI_PASSWORD')
)
```
***Other Considerations***

- If the one of the suggested uses for this is secrets, care must be taken to ensure any caching does not inadvertently store the values.
- What should happen when the variable is unset or 'falsly' in the environment?

**Describe alternatives you've considered**

Alternatively, these fields could be made to support environment variable interpolation by default. e.g. `docker_image(version='${VERSION}')`.
While this would also work, it would require Pants developers to second-guess/dictate where environment variables might be used and increase the complexity. 


Discussion:
Thanks for the suggestion! I like this - we want to reduce the need for writing custom plugins when possible. This also semi relates to https://github.com/pantsbuild/pants/issues/10399.

FYI the reason we ban imports of `os.environ` in BUILD files is it would break caching/memoization - your `Env` suggestion meanwhile would allow us to get that caching right.

--

Another consideration is how to handle data types. Would we expect the user to do this?

```python
python_tests(timeout=int(Env("MY_TIMEOUT"))
```

That's legal because BUILD files are Python and `int()` is builtin.

An alternative is for the Target API to start accepting strings for non-strong fields and trying to convert. We only need to update the core definitions of `IntField`, `FloatField`, etc for all consumers to do the right thing. This means this snippet would work, where `Env` is giving back a `str`:

```python
python_tests(timeout=Env("MY_TIMEOUT"))
```
Likely also related to #7022.
> Another consideration is how to handle data types. Would we expect the user to do this?
> 
> ```python
> python_tests(timeout=int(Env("MY_TIMEOUT"))
> ```
> 
> That's legal because BUILD files are Python and `int()` is builtin.
> 
> An alternative is for the Target API to start accepting strings for non-strong fields and trying to convert. We only need to update the core definitions of `IntField`, `FloatField`, etc for all consumers to do the right thing. This means this snippet would work, where `Env` is giving back a `str`:
> 
> ```python
> python_tests(timeout=Env("MY_TIMEOUT"))
> ```

From a users perspective, I would think having the field responsible for string-parsing would be the better option. Why should I, as a user, be responsible for making sure my fields are cast to the right type? 

From a developers perspective, while there are *currently* Int's and Floats as core fields,  what about Lists-of-ints, and maps-of-floats and other more complex data structures?
> Likely also related to #7022.

Sort of. If "target generation" lands, that will mean that you can write much more powerful macros that have access to the Rules API. Rather than using our custom plugin hooks for things like `setup_py()`, you can write a more generic macro to generate a `python_distribution` target. 

But, that still would mean writing a plugin. Iiuc this issue is about allowing you to access env vars _without_ needing to write a plugin. Much lower barrier to entry and maintenance burden for users.

--

> From a users perspective, I would think having the field responsible for string-parsing would be the better option.

Cool, agreed!

> While there are currently Int's and Floats as core fields, what about Lists-of-ints, and maps-of-floats and other more complex data structures?

That should be doable :) For example, with a list of ints, it's probably too hard to implement parsing of `"[1, 2, 3]"` - but we could support `[1, ENV("FOO"), 3]` fairly easily.
> > Likely also related to #7022.
> 
> Sort of. If "target generation" lands, that will mean that you can write much more powerful macros that have access to the Rules API. Rather than using our custom plugin hooks for things like `setup_py()`, you can write a more generic macro to generate a `python_distribution` target.

I meant more that until #7022, there isn't a way to create either a macro or object that safely consumes the Env in a way that will be properly invalidated (without hacking deeply in BUILD parsing to special case this).
With #7022 , how to write a macro to set by the env variable?
> With https://github.com/pantsbuild/pants/issues/7022 , how to write a macro to set by the env variable?

Hey @da-tubi, the result of #7022 was the new Target Generator API added in Pants 2.8, the section "File-level metadata with overrides" of https://blog.pantsbuild.org/introducing-pants-2-8/.

The only way currently consume environment variables in a BUILD files is by writing plugins using the Rules API, like a target generator, or the `setup_py()` kwargs plugin hook.

Resolving patch:
diff --git a/src/python/pants/engine/internals/build_files.py b/src/python/pants/engine/internals/build_files.py
--- a/src/python/pants/engine/internals/build_files.py
+++ b/src/python/pants/engine/internals/build_files.py
@@ -3,12 +3,15 @@
 
 from __future__ import annotations
 
+import ast
 import builtins
 import itertools
+import logging
 import os.path
+import sys
 from dataclasses import dataclass
 from pathlib import PurePath
-from typing import Any, cast
+from typing import Any, Sequence, cast
 
 from pants.build_graph.address import (
     Address,
@@ -19,7 +22,8 @@
     ResolveError,
 )
 from pants.engine.engine_aware import EngineAwareParameter
-from pants.engine.fs import DigestContents, GlobMatchErrorBehavior, PathGlobs, Paths
+from pants.engine.env_vars import CompleteEnvironmentVars, EnvironmentVars, EnvironmentVarsRequest
+from pants.engine.fs import DigestContents, FileContent, GlobMatchErrorBehavior, PathGlobs, Paths
 from pants.engine.internals.defaults import BuildFileDefaults, BuildFileDefaultsParserState
 from pants.engine.internals.dep_rules import (
     BuildFileDependencyRules,
@@ -28,12 +32,13 @@
 )
 from pants.engine.internals.mapper import AddressFamily, AddressMap
 from pants.engine.internals.parser import BuildFilePreludeSymbols, Parser, error_on_imports
+from pants.engine.internals.session import SessionValues
 from pants.engine.internals.synthetic_targets import (
     SyntheticAddressMaps,
     SyntheticAddressMapsRequest,
 )
 from pants.engine.internals.target_adaptor import TargetAdaptor, TargetAdaptorRequest
-from pants.engine.rules import Get, MultiGet, collect_rules, rule
+from pants.engine.rules import Get, MultiGet, collect_rules, rule, rule_helper
 from pants.engine.target import (
     DependenciesRuleApplication,
     DependenciesRuleApplicationRequest,
@@ -45,6 +50,8 @@
 from pants.util.frozendict import FrozenDict
 from pants.util.strutil import softwrap
 
+logger = logging.getLogger(__name__)
+
 
 @dataclass(frozen=True)
 class BuildFileOptions:
@@ -169,6 +176,62 @@ async def ensure_address_family(request: OptionalAddressFamily) -> AddressFamily
     return request.ensure()
 
 
+class BUILDFileEnvVarExtractor(ast.NodeVisitor):
+    def __init__(self, filename: str):
+        super().__init__()
+        self.env_vars: set[str] = set()
+        self.filename = filename
+
+    @classmethod
+    def get_env_vars(cls, file_content: FileContent) -> Sequence[str]:
+        obj = cls(file_content.path)
+        obj.visit(ast.parse(file_content.content, file_content.path))
+        return tuple(obj.env_vars)
+
+    def visit_Call(self, node: ast.Call):
+        is_env = isinstance(node.func, ast.Name) and node.func.id == "env"
+        for arg in node.args:
+            if not is_env:
+                self.visit(arg)
+                continue
+
+            # Only first arg may be checked as env name
+            is_env = False
+
+            if sys.version_info[0:2] < (3, 8):
+                value = arg.s if isinstance(arg, ast.Str) else None
+            else:
+                value = arg.value if isinstance(arg, ast.Constant) else None
+            if value:
+                # Found env name in this call, we're done here.
+                self.env_vars.add(value)
+                return
+            else:
+                logging.warning(
+                    f"{self.filename}:{arg.lineno}: Only constant string values as variable name to "
+                    f"`env()` is currently supported. This `env()` call will always result in "
+                    "the default value only."
+                )
+
+        for kwarg in node.keywords:
+            self.visit(kwarg)
+
+
+@rule_helper
+async def _extract_env_vars(
+    file_content: FileContent, env: CompleteEnvironmentVars
+) -> EnvironmentVars:
+    """For BUILD file env vars, we only ever consult the local systems env."""
+    env_vars = BUILDFileEnvVarExtractor.get_env_vars(file_content)
+    return await Get(
+        EnvironmentVars,
+        {
+            EnvironmentVarsRequest(env_vars): EnvironmentVarsRequest,
+            env: CompleteEnvironmentVars,
+        },
+    )
+
+
 @rule(desc="Search for addresses in BUILD files")
 async def parse_address_family(
     parser: Parser,
@@ -178,6 +241,7 @@ async def parse_address_family(
     registered_target_types: RegisteredTargetTypes,
     union_membership: UnionMembership,
     maybe_build_file_dependency_rules_implementation: MaybeBuildFileDependencyRulesImplementation,
+    session_values: SessionValues,
 ) -> OptionalAddressFamily:
     """Given an AddressMapper and a directory, return an AddressFamily.
 
@@ -235,17 +299,23 @@ async def parse_address_family(
         dependents_rules_parser_state = None
         dependencies_rules_parser_state = None
 
+    all_env_vars = [
+        await _extract_env_vars(fc, session_values[CompleteEnvironmentVars])
+        for fc in digest_contents
+    ]
+
     address_maps = [
         AddressMap.parse(
             fc.path,
             fc.content.decode(),
             parser,
             prelude_symbols,
+            env_vars,
             defaults_parser_state,
             dependents_rules_parser_state,
             dependencies_rules_parser_state,
         )
-        for fc in digest_contents
+        for fc, env_vars in zip(digest_contents, all_env_vars)
     ]
 
     # Freeze defaults and dependency rules
diff --git a/src/python/pants/engine/internals/mapper.py b/src/python/pants/engine/internals/mapper.py
--- a/src/python/pants/engine/internals/mapper.py
+++ b/src/python/pants/engine/internals/mapper.py
@@ -10,6 +10,7 @@
 from pants.backend.project_info.filter_targets import FilterSubsystem
 from pants.base.exceptions import MappingError
 from pants.build_graph.address import Address, BuildFileAddress
+from pants.engine.env_vars import EnvironmentVars
 from pants.engine.internals.defaults import BuildFileDefaults, BuildFileDefaultsParserState
 from pants.engine.internals.dep_rules import (
     BuildFileDependencyRules,
@@ -43,6 +44,7 @@ def parse(
         build_file_content: str,
         parser: Parser,
         extra_symbols: BuildFilePreludeSymbols,
+        env_vars: EnvironmentVars,
         defaults: BuildFileDefaultsParserState,
         dependents_rules: BuildFileDependencyRulesParserState | None,
         dependencies_rules: BuildFileDependencyRulesParserState | None,
@@ -57,6 +59,7 @@ def parse(
                 filepath,
                 build_file_content,
                 extra_symbols,
+                env_vars,
                 defaults,
                 dependents_rules,
                 dependencies_rules,
diff --git a/src/python/pants/engine/internals/parser.py b/src/python/pants/engine/internals/parser.py
--- a/src/python/pants/engine/internals/parser.py
+++ b/src/python/pants/engine/internals/parser.py
@@ -16,6 +16,7 @@
 from pants.base.exceptions import MappingError
 from pants.base.parse_context import ParseContext
 from pants.build_graph.build_file_aliases import BuildFileAliases
+from pants.engine.env_vars import EnvironmentVars
 from pants.engine.internals.defaults import BuildFileDefaultsParserState, SetDefaultsT
 from pants.engine.internals.dep_rules import BuildFileDependencyRulesParserState
 from pants.engine.internals.target_adaptor import TargetAdaptor
@@ -213,6 +214,7 @@ def parse(
         filepath: str,
         build_file_content: str,
         extra_symbols: BuildFilePreludeSymbols,
+        env_vars: EnvironmentVars,
         defaults: BuildFileDefaultsParserState,
         dependents_rules: BuildFileDependencyRulesParserState | None,
         dependencies_rules: BuildFileDependencyRulesParserState | None,
@@ -224,7 +226,11 @@ def parse(
             dependencies_rules=dependencies_rules,
         )
 
-        global_symbols = {**self._symbols, **extra_symbols.symbols}
+        global_symbols: dict[str, Any] = {
+            "env": env_vars.get,
+            **self._symbols,
+            **extra_symbols.symbols,
+        }
 
         if self.ignore_unrecognized_symbols:
             defined_symbols = set()


Test patch:
diff --git a/src/python/pants/engine/internals/build_files_test.py b/src/python/pants/engine/internals/build_files_test.py
--- a/src/python/pants/engine/internals/build_files_test.py
+++ b/src/python/pants/engine/internals/build_files_test.py
@@ -3,6 +3,7 @@
 
 from __future__ import annotations
 
+import logging
 import re
 from textwrap import dedent
 from typing import cast
@@ -13,6 +14,7 @@
 from pants.build_graph.build_file_aliases import BuildFileAliases
 from pants.core.target_types import GenericTarget
 from pants.engine.addresses import Address, AddressInput, BuildFileAddress
+from pants.engine.env_vars import CompleteEnvironmentVars, EnvironmentVars, EnvironmentVarsRequest
 from pants.engine.fs import DigestContents, FileContent, PathGlobs
 from pants.engine.internals.build_files import (
     AddressFamilyDir,
@@ -27,6 +29,7 @@
 from pants.engine.internals.parametrize import Parametrize
 from pants.engine.internals.parser import BuildFilePreludeSymbols, Parser
 from pants.engine.internals.scheduler import ExecutionError
+from pants.engine.internals.session import SessionValues
 from pants.engine.internals.synthetic_targets import (
     SyntheticAddressMaps,
     SyntheticAddressMapsRequest,
@@ -41,6 +44,7 @@
     Target,
 )
 from pants.engine.unions import UnionMembership
+from pants.testutil.pytest_util import assert_logged
 from pants.testutil.rule_runner import (
     MockGet,
     QueryRule,
@@ -49,6 +53,7 @@
     run_rule_with_mocks,
 )
 from pants.util.frozendict import FrozenDict
+from pants.util.strutil import softwrap
 
 
 def test_parse_address_family_empty() -> None:
@@ -69,6 +74,7 @@ def test_parse_address_family_empty() -> None:
             RegisteredTargetTypes({}),
             UnionMembership({}),
             MaybeBuildFileDependencyRulesImplementation(None),
+            SessionValues({CompleteEnvironmentVars: CompleteEnvironmentVars({})}),
         ],
         mock_gets=[
             MockGet(
@@ -86,6 +92,11 @@ def test_parse_address_family_empty() -> None:
                 input_types=(SyntheticAddressMapsRequest,),
                 mock=lambda _: SyntheticAddressMaps(),
             ),
+            MockGet(
+                output_type=EnvironmentVars,
+                input_types=(EnvironmentVarsRequest, CompleteEnvironmentVars),
+                mock=lambda _1, _2: EnvironmentVars({}),
+            ),
         ],
     )
     assert optional_af.path == "/dev/null"
@@ -521,3 +532,67 @@ def uses_undefined():
     # Parse the root BUILD file.
     address_family = rule_runner.request(AddressFamily, [AddressFamilyDir("")])
     assert not address_family.name_to_target_adaptors
+
+
+def test_build_file_env_vars(target_adaptor_rule_runner: RuleRunner) -> None:
+    target_adaptor_rule_runner.write_files(
+        {
+            "BUILD": dedent(
+                """
+                mock_tgt(
+                  description=env("MOCK_DESC"),
+                  tags=[
+                    env("DEF", "default"),
+                    env("TAG", "default"),
+                  ]
+                )
+                """
+            ),
+        },
+    )
+    target_adaptor_rule_runner.set_options([], env={"MOCK_DESC": "from env", "TAG": "tag"})
+    target_adaptor = target_adaptor_rule_runner.request(
+        TargetAdaptor,
+        [TargetAdaptorRequest(Address(""), description_of_origin="tests")],
+    )
+    assert target_adaptor.kwargs["description"] == "from env"
+    assert target_adaptor.kwargs["tags"] == ["default", "tag"]
+
+
+def test_invalid_build_file_env_vars(caplog, target_adaptor_rule_runner: RuleRunner) -> None:
+    target_adaptor_rule_runner.write_files(
+        {
+            "src/bad/BUILD": dedent(
+                """
+                DOES_NOT_WORK = "var_name1"
+                DO_THIS_INSTEAD = env("var_name2")
+
+                mock_tgt(description=env(DOES_NOT_WORK), tags=[DO_THIS_INSTEAD])
+                """
+            ),
+        },
+    )
+    target_adaptor_rule_runner.set_options(
+        [], env={"var_name1": "desc from env", "var_name2": "tag-from-env"}
+    )
+    target_adaptor = target_adaptor_rule_runner.request(
+        TargetAdaptor,
+        [TargetAdaptorRequest(Address("src/bad"), description_of_origin="tests")],
+    )
+    assert target_adaptor.kwargs["description"] is None
+    assert target_adaptor.kwargs["tags"] == ["tag-from-env"]
+    assert_logged(
+        caplog,
+        [
+            (
+                logging.WARNING,
+                softwrap(
+                    """
+                    src/bad/BUILD:5: Only constant string values as variable name to `env()` is
+                    currently supported. This `env()` call will always result in the default value
+                    only.
+                    """
+                ),
+            ),
+        ],
+    )
diff --git a/src/python/pants/engine/internals/mapper_test.py b/src/python/pants/engine/internals/mapper_test.py
--- a/src/python/pants/engine/internals/mapper_test.py
+++ b/src/python/pants/engine/internals/mapper_test.py
@@ -12,6 +12,7 @@
 from pants.build_graph.address import Address
 from pants.build_graph.build_file_aliases import BuildFileAliases
 from pants.core.target_types import GenericTarget
+from pants.engine.env_vars import EnvironmentVars
 from pants.engine.internals.defaults import BuildFileDefaults, BuildFileDefaultsParserState
 from pants.engine.internals.mapper import (
     AddressFamily,
@@ -42,6 +43,7 @@ def parse_address_map(build_file: str, *, ignore_unrecognized_symbols: bool = Fa
         build_file,
         parser,
         BuildFilePreludeSymbols(FrozenDict()),
+        EnvironmentVars({}),
         BuildFileDefaultsParserState.create(
             "", BuildFileDefaults({}), RegisteredTargetTypes({}), UnionMembership({})
         ),
diff --git a/src/python/pants/engine/internals/parser_test.py b/src/python/pants/engine/internals/parser_test.py
--- a/src/python/pants/engine/internals/parser_test.py
+++ b/src/python/pants/engine/internals/parser_test.py
@@ -7,6 +7,7 @@
 
 from pants.build_graph.build_file_aliases import BuildFileAliases
 from pants.core.target_types import GenericTarget
+from pants.engine.env_vars import EnvironmentVars
 from pants.engine.internals.defaults import BuildFileDefaults, BuildFileDefaultsParserState
 from pants.engine.internals.parser import (
     BuildFilePreludeSymbols,
@@ -41,6 +42,7 @@ def test_imports_banned(defaults_parser_state: BuildFileDefaultsParserState) ->
             "dir/BUILD",
             "\nx = 'hello'\n\nimport os\n",
             BuildFilePreludeSymbols(FrozenDict()),
+            EnvironmentVars({}),
             defaults_parser_state,
             dependents_rules=None,
             dependencies_rules=None,
@@ -71,6 +73,7 @@ def perform_test(extra_targets: list[str], dym: str) -> None:
                 "dir/BUILD",
                 "fake",
                 prelude_symbols,
+                EnvironmentVars({}),
                 defaults_parser_state,
                 dependents_rules=None,
                 dependencies_rules=None,
@@ -81,7 +84,7 @@ def perform_test(extra_targets: list[str], dym: str) -> None:
             f" refer to {doc_url('enabling-backends')} for all available"
             " backends to activate.\n\n"
             "All registered symbols: ['__defaults__', '__dependencies_rules__', "
-            f"'__dependents_rules__', 'build_file_dir', 'caof', {fmt_extra_sym}"
+            f"'__dependents_rules__', 'build_file_dir', 'caof', 'env', {fmt_extra_sym}"
             "'obj', 'prelude', 'tgt']"
         )
 
@@ -99,6 +102,7 @@ def perform_test(extra_targets: list[str], dym: str) -> None:
                 "dir/BUILD",
                 "fake",
                 prelude_symbols,
+                EnvironmentVars({}),
                 defaults_parser_state,
                 dependents_rules=None,
                 dependencies_rules=None,