Repository: pantsbuild/pants

Issue:
Consider allowing reading from environment variables in `pants.toml`
We currently allow you to read from environment variables when using command line flags and env variables thanks to the affordances of the shell. But, we don't let you read from them in config files.

It wouldn't be very difficult to support reading environment variables. We could reuse the interpolation syntax used for default values, e.g. `%(foo)s`. Possibly, we could distinguish it a little via something like `$()s`. Now that we only use TOML, we have lots of flexibility to implement interpolation how we want.


Discussion:
I would like something like this to control pylint concurrency because:
- pants does not run each linter itself in parallel (linters run in parallel, but only one process for each)
- pylint can do parallel with `--jobs` commandline arg, but it can't do it directly via env var
- the Makefile + CI I'm working on replacing uses an env var to make this configurable
- but I can't use that env var from within pants.toml

Something like this:
```
[pylint]
args = [
  "--jobs=${PYLINT_CONCURRENCY:-1}",
]
[subprocess-environment]
env_vars.add = [
  "PYLINT_CONCURRENCY",
]
```

Allowing env var interpolation solves part of the problem, however I would still need a way to set a default somehow.

Perhaps the interpolation needs something like an f-string where `env` is the only var available in the context...
```
[pylint]
args = [
  f"--jobs={env.get('PYLINT_CONCURRENCY', 1)}",
]
```
This feature aside, @cognifloyd if you intend the default to be 1 but CI to use something different, then you could just have ci export PANTS_CONFIG_FILES=pants.ci.toml to add pants.ci.toml to pants.toml for CI runs. In the pants.ci.toml you could:
```toml
[pylint]
args.add = ["--jobs=8"]
```
Since the last --jobs wins for pylint, this all works out.

If you _also_ want the CI value to be pluggable without having to dynamically generate a pants.ci.toml or else litter the repo with many of them, you can always just use Pants support for environment variables directly, ie: `PANTS_PYLINT_ARGS="+['--jobs=${PYLINT_CONCURRENCY}']"`. See: https://www.pantsbuild.org/docs/options#list-values

The toml env var interpolation is really just sugar over the already supported env var option syntax thats useful to cut down on line noise for the more complex list and dict cases afaict.
Yeah. Here's what I decided to do: I discovered that using `--jobs=0` with pylint makes it detect the number of processors to use, so that's what I'm doing. If I need to override it, I can set `PANTS_PYLINT_ARGS="+['--jobs=4']"` in the CI workflow. So, I have a working solution. For pylint, the last `--jobs` command line arg seems to win, so that works out just fine in this case.

That said, env var interpolation in config files is a common feature, and I think it would be beneficial for pants as well.
This is needed to support https://github.com/pantsbuild/pex/pull/1716 in any reasonable scenario.
It looks like #7312 will need to be reversed in order to implement anything like this.

Resolving patch:
diff --git a/src/python/pants/option/config.py b/src/python/pants/option/config.py
--- a/src/python/pants/option/config.py
+++ b/src/python/pants/option/config.py
@@ -9,6 +9,7 @@
 import re
 from dataclasses import dataclass
 from functools import partial
+from types import SimpleNamespace
 from typing import Any, Dict, Iterable, List, Mapping, Union, cast
 
 import toml
@@ -61,16 +62,20 @@ def load(
         file_contents: Iterable[ConfigSource],
         *,
         seed_values: SeedValues | None = None,
+        env: Mapping[str, str] | None = None,
     ) -> Config:
         """Loads config from the given string payloads, with later payloads overriding earlier ones.
 
         A handful of seed values will be set to act as if specified in the loaded config file's
         DEFAULT section, and be available for use in substitutions.  The caller may override some of
         these seed values.
+
+        If an `env` is supplied, it is exposed as `env` object available for interpolation via dot
+        access of the environment variable names (e.g.: `env.HOME`).
         """
         config_values = []
         for file_content in file_contents:
-            normalized_seed_values = cls._determine_seed_values(seed_values=seed_values)
+            normalized_seed_values = cls._determine_seed_values(seed_values=seed_values, env=env)
             try:
                 config_values.append(cls._parse_toml(file_content, normalized_seed_values))
             except Exception as e:
@@ -81,7 +86,7 @@ def load(
 
     @classmethod
     def _parse_toml(
-        cls, config_source: ConfigSource, normalized_seed_values: dict[str, str]
+        cls, config_source: ConfigSource, normalized_seed_values: dict[str, Any]
     ) -> _ConfigValues:
         """Attempt to parse as TOML, raising an exception on failure."""
         toml_values = cast(Dict[str, Any], toml.loads(config_source.content.decode()))
@@ -104,19 +109,26 @@ def verify(self, section_to_valid_options: dict[str, set[str]]):
             )
 
     @staticmethod
-    def _determine_seed_values(*, seed_values: SeedValues | None = None) -> dict[str, str]:
+    def _determine_seed_values(
+        *, seed_values: SeedValues | None = None, env: Mapping[str, str] | None = None
+    ) -> dict[str, Any]:
         """We pre-populate several default values to allow %([key-name])s interpolation.
 
         This sets up those defaults and checks if the user overrode any of the values.
+
+        In addition, we pre-populate any supplied env entries to allow %(env.[env-var-name])s
+        interpolation.
         """
         safe_seed_values = seed_values or {}
         buildroot = cast(str, safe_seed_values.get("buildroot", get_buildroot()))
 
-        all_seed_values: dict[str, str] = {
+        all_seed_values: dict[str, Any] = {
             "buildroot": buildroot,
             "homedir": os.path.expanduser("~"),
             "user": getpass.getuser(),
         }
+        if env:
+            all_seed_values["env"] = SimpleNamespace(**env)
 
         def update_seed_values(key: str, *, default_dir: str) -> None:
             all_seed_values[key] = cast(
@@ -187,7 +199,7 @@ def format_str(value: str) -> str:
             # that .format() does not try to improperly interpolate.
             escaped_str = value.replace("{", "{{").replace("}", "}}")
             new_style_format_str = re.sub(
-                pattern=r"%\((?P<interpolated>[a-zA-Z_0-9]*)\)s",
+                pattern=r"%\((?P<interpolated>[a-zA-Z_0-9.]+)\)s",
                 repl=r"{\g<interpolated>}",
                 string=escaped_str,
             )
@@ -205,7 +217,7 @@ def format_str(value: str) -> str:
 
         def recursively_format_str(value: str) -> str:
             # It's possible to interpolate with a value that itself has an interpolation.
-            match = re.search(r"%\(([a-zA-Z_0-9]*)\)s", value)
+            match = re.search(r"%\(([a-zA-Z_0-9.]+)\)s", value)
             if not match:
                 return value
             return recursively_format_str(value=format_str(value))
@@ -290,11 +302,8 @@ def get_value(self, section: str, option: str) -> str | None:
         return remove_val
 
     @property
-    def defaults(self) -> dict[str, str]:
-        return {
-            option: self._stringify_val_without_interpolation(option_val)
-            for option, option_val in self.section_to_values[DEFAULT_SECTION].items()
-        }
+    def defaults(self) -> dict[str, Any]:
+        return self.section_to_values[DEFAULT_SECTION].copy()
 
     def get_verification_errors(self, section_to_valid_options: dict[str, set[str]]) -> list[str]:
         error_log = []
diff --git a/src/python/pants/option/options_bootstrapper.py b/src/python/pants/option/options_bootstrapper.py
--- a/src/python/pants/option/options_bootstrapper.py
+++ b/src/python/pants/option/options_bootstrapper.py
@@ -133,12 +133,11 @@ def filecontent_for(path: str) -> FileContent:
                     read_file(path, binary_mode=True),
                 )
 
-            env = {k: v for k, v in env.items() if k.startswith("PANTS_")}
             bargs = cls._get_bootstrap_args(args)
 
             config_file_paths = cls.get_config_file_paths(env=env, args=args)
             config_files_products = [filecontent_for(p) for p in config_file_paths]
-            pre_bootstrap_config = Config.load(config_files_products)
+            pre_bootstrap_config = Config.load(config_files_products, env=env)
 
             initial_bootstrap_options = cls.parse_bootstrap_options(
                 env, bargs, pre_bootstrap_config
@@ -160,10 +159,9 @@ def filecontent_for(path: str) -> FileContent:
             post_bootstrap_config = Config.load(
                 full_config_files_products,
                 seed_values=bootstrap_option_values.as_dict(),
+                env=env,
             )
 
-            env_tuples = tuple(sorted(env.items(), key=lambda x: x[0]))
-
             # Finally, we expand any aliases and re-populate the bootstrap args, in case there
             # were any from aliases.
             # stuhood: This could potentially break the rust client when aliases are used:
@@ -186,6 +184,11 @@ def filecontent_for(path: str) -> FileContent:
             # remote pants runners.
             os.environ["PANTS_BIN_NAME"] = bootstrap_option_values.pants_bin_name
 
+            env_tuples = tuple(
+                sorted(
+                    (item for item in env.items() if item[0].startswith("PANTS_")),
+                )
+            )
             return cls(
                 env_tuples=env_tuples,
                 bootstrap_args=bargs,


Test patch:
diff --git a/src/python/pants/option/config_test.py b/src/python/pants/option/config_test.py
--- a/src/python/pants/option/config_test.py
+++ b/src/python/pants/option/config_test.py
@@ -21,11 +21,11 @@ class ConfigFile:
     content=dedent(
         """
         [DEFAULT]
-        name = "foo"
+        name = "%(env.NAME)s"
         answer = 42
         scale = 1.2
         path = "/a/b/%(answer)s"
-        embed = "%(path)s::foo"
+        embed = "%(path)s::%(name)s"
         disclaimer = '''
         Let it be known
         that.'''
@@ -62,8 +62,8 @@ class ConfigFile:
     ),
     default_values={
         "name": "foo",
-        "answer": "42",
-        "scale": "1.2",
+        "answer": 42,
+        "scale": 1.2,
         "path": "/a/b/42",
         "embed": "/a/b/42::foo",
         "disclaimer": "Let it be known\nthat.",
@@ -136,6 +136,7 @@ def _setup_config() -> Config:
             FileContent("file2.toml", FILE_2.content.encode()),
         ],
         seed_values={"buildroot": "fake_buildroot"},
+        env={"NAME": "foo"},
     )
     assert ["file1.toml", "file2.toml"] == parsed_config.sources()
     return parsed_config
@@ -146,6 +147,7 @@ def setUp(self) -> None:
         self.config = _setup_config()
         self.default_seed_values = Config._determine_seed_values(
             seed_values={"buildroot": "fake_buildroot"},
+            env={"NAME": "foo"},
         )
         self.expected_combined_values: dict[str, dict[str, list[str]]] = {
             "a": {
@@ -185,8 +187,9 @@ def test_default_values(self) -> None:
         # values for _ConfigValues.defaults are not yet interpolated.
         default_file1_values_unexpanded = {
             **FILE_1.default_values,
+            "name": "%(env.NAME)s",
             "path": "/a/b/%(answer)s",
-            "embed": "%(path)s::foo",
+            "embed": "%(path)s::%(name)s",
         }
         assert file1_values.defaults == {
             **self.default_seed_values,
@@ -196,12 +199,15 @@ def test_default_values(self) -> None:
 
     def test_get(self) -> None:
         # Check the DEFAULT section
+        # N.B.: All values read from config files are read as str and only later converted by the
+        # options parser to the expected destination type; so we ensure we're comparing strings
+        # here.
         for option, value in self.default_seed_values.items():
             # Both config files have the seed values.
-            assert self.config.get(section="DEFAULT", option=option) == [value, value]
+            assert self.config.get(section="DEFAULT", option=option) == [str(value), str(value)]
         for option, value in FILE_1.default_values.items():
             # Only FILE_1 has explicit DEFAULT values.
-            assert self.config.get(section="DEFAULT", option=option) == [value]
+            assert self.config.get(section="DEFAULT", option=option) == [str(value)]
 
         # Check the combined values.
         for section, section_values in self.expected_combined_values.items():