Repository: pantsbuild/pants

Issue:
List typed config values can not be merged between multiple config files (PANTS_CONFIG_FILES)
**Describe the bug**
Originally asked in https://pantsbuild.slack.com/archives/C046T6T9U/p1646199988870439

No matter the syntax I am unable to get docker build args list merging to work.

**Pants version**
2.9.0

**OS**
Linux

**Additional info**

`pants.toml`:
```
...
[docker]
build_args.add = [
    "BASE_PYTHON_IMAGE=foo",
]
```
`pants.ci.toml`:
```
...
[docker]
# Expose aws auth to docker.
build_args.add = [
    "AWS_DEFAULT_REGION",
    "AWS_ACCESS_KEY_ID",
    "AWS_SECRET_ACCESS_KEY",
    "AWS_SESSION_TOKEN",
]
```
`Dockerfile`:
```
ARG BASE_PYTHON_IMAGE
FROM ${BASE_PYTHON_IMAGE}
...
````
Running in CI:
```
$ export PANTS_CONFIG_FILES=pants.ci.toml
$ ./pants package docker/image:target
...
  failed to solve with frontend dockerfile.v0: failed to create LLB definition: base name (${BASE_PYTHON_IMAGE}) should not be blank
...
```



Discussion:
Some more examples:
```
$ PANTS_CONFIG_FILES=pants.ci.toml ./pants help docker | grep -A 12 docker-build-args
  --docker-build-args="[<shell_str>, <shell_str>, ...]"
  PANTS_DOCKER_BUILD_ARGS
  build_args
      default: []
      current value: [
          "AWS_DEFAULT_REGION",
          "AWS_ACCESS_KEY_ID",
          "AWS_ROLE_ARN",
          "AWS_SECRET_ACCESS_KEY",
          "AWS_SESSION_TOKEN",
          "AWS_WEB_IDENTITY_TOKEN_FILE"
      ] (from pants.ci.toml)
      Global build arguments (for Docker `--build-arg` options) to use for all `docker build`

$ PANTS_CONFIG_FILES="+['pants.ci.toml']" ./pants help docker | grep -A 12 docker-build-args
  --docker-build-args="[<shell_str>, <shell_str>, ...]"
  PANTS_DOCKER_BUILD_ARGS
  build_args
      default: []
      current value: [
          "AWS_DEFAULT_REGION",
          "AWS_ACCESS_KEY_ID",
          "AWS_ROLE_ARN",
          "AWS_SECRET_ACCESS_KEY",
          "AWS_SESSION_TOKEN",
          "AWS_WEB_IDENTITY_TOKEN_FILE"
      ] (from pants.ci.toml)
      Global build arguments (for Docker `--build-arg` options) to use for all `docker build`

$ ./pants help docker | grep -A 7 docker-build-args
  --docker-build-args="[<shell_str>, <shell_str>, ...]"
  PANTS_DOCKER_BUILD_ARGS
  build_args
      default: []
      current value: [
          "COMMIT_SHA",
          "BASE_PYTHON_IMAGE=foo"
      ] (from pants.toml)

$ PANTS_CONFIG_FILES="['pants.toml', 'pants.ci.toml']" ./pants help docker | grep -A 12 docker-build-args
  --docker-build-args="[<shell_str>, <shell_str>, ...]"
  PANTS_DOCKER_BUILD_ARGS
  build_args
      default: []
      current value: [
          "AWS_DEFAULT_REGION",
          "AWS_ACCESS_KEY_ID",
          "AWS_ROLE_ARN",
          "AWS_SECRET_ACCESS_KEY",
          "AWS_SESSION_TOKEN",
          "AWS_WEB_IDENTITY_TOKEN_FILE"
      ] (from pants.ci.toml)
      Global build arguments (for Docker `--build-arg` options) to use for all `docker build`

$ PANTS_CONFIG_FILES="['pants.ci.toml', 'pants.toml']" ./pants help docker | grep -A 7 docker-build-args
  --docker-build-args="[<shell_str>, <shell_str>, ...]"
  PANTS_DOCKER_BUILD_ARGS
  build_args
      default: []
      current value: [
          "COMMIT_SHA",
          "BASE_PYTHON_IMAGE=foo"
      ] (from pants.toml)
      Global build arguments (for Docker `--build-arg` options) to use for all `docker build`
```
Merging from env seems to work:
```
PANTS_DOCKER_BUILD_ARGS="+['aaa=bbb']" ./pants help docker  | grep -A 7 docker-build-args
  --docker-build-args="[<shell_str>, <shell_str>, ...]"
  PANTS_DOCKER_BUILD_ARGS
  build_args
      default: []
      current value: [
          "COMMIT_SHA",
          "BASE_PYTHON_IMAGE=foo",
          "aaa=bbb"

```
FWIW this also works:

```
pants.toml
-----------
[docker]
build_args.add = ["foo=bar"]


$ ./pants help docker --build-args="baz=qux"
<snip>
--docker-build-args="[<shell_str>, <shell_str>, ...]"
  PANTS_DOCKER_BUILD_ARGS
  build_args
      default: []
      current value: [
          "foo=bar",
          "baz=qux"
      ] (from pants.toml, from command-line flag)
</snip>
```
OK, I can reproduce this at HEAD, and it's definitely a bug.
It appears to be a bug in the options system (!) not specifically in the docker subsystem.
Yes, I experience the same thing with `[GLOBAL] plugins`. Glad it's not just me :-).
This appears to be working as intended in some sense, but only because we didn't think of this case, apparently.
As intended? I thought the point of `foo.add = [...]` was to support this? It seems like that's being ignored currently.
That works across defaults/env vars/config/flags, but the current semantics of multiple config files is that later files override the values in earlier files, so that only one value per set of config files is presented to the options system. 

But the fact that you expected it to work differently, and I did too (and I wrote the options system back in the day) tells me that the existing logic is not good.
The complication is that we parse config files before we know the type of the option, going to have to think about what to do here.
Thanks for the explanation. Makes sense.

Another use case/motivation:
Enabling certain plugins just for your local setup.

`pants.toml`:
```
[GLOBAL]
backend_packages.add = [
    "pants.backend.python",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.isort",
    "pants.backend.python.lint.pylint",
    "pants.backend.codegen.protobuf.python",
    "pants.backend.python.mixed_interpreter_constraints",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.shell",
    "pants.backend.shell.lint.shfmt",
    "pants.backend.shell.lint.shellcheck",
    "pants.backend.docker",
    "pants.backend.docker.lint.hadolint",
]
```
`~/.config/pants.toml` (enabled via `PANTS_CONFIG_FILES` in `.bashrc`):
```
backend_packages.add = [
    "pants.backend.experimental.python.lint.autoflake",
    "pants.backend.experimental.python.lint.pyupgrade",
]
```
I guess one potential short term workaround is to move away from `PANTS_CONFIG_FILES` pointing to a secondary toml in these situations, and instead just set the overrides using all env vars. e,g,  `PANTS_BACKEND_PACKAGES="+['pants.backend.experimental.python.lint.autoflake', 'pants.backend.experimental.python.lint.pyupgrade']"`.
@benjyw possibly a duplicate of https://github.com/pantsbuild/pants/issues/14679 btw, re: what to close once this is fixed.
Unassigning myself, due to

> It appears to be a bug in the options system (!) not specifically in the docker subsystem.


Resolving patch:
diff --git a/src/python/pants/option/config.py b/src/python/pants/option/config.py
--- a/src/python/pants/option/config.py
+++ b/src/python/pants/option/config.py
@@ -128,17 +128,14 @@ def update_seed_values(key: str, *, default_dir: str) -> None:
 
         return all_seed_values
 
-    def get(self, section, option) -> str | None:
-        """Retrieves option from the specified section (or 'DEFAULT').
-
-        If the specified section does not exist or is missing a definition for the option, the value
-        is looked up in the DEFAULT section.  If there is still no definition found, returns None.
-        """
-        for vals in reversed(self.values):
+    def get(self, section, option) -> list[str]:
+        """Retrieves an option value from each config file in which it appears."""
+        available_vals = []
+        for vals in self.values:
             val = vals.get_value(section, option)
             if val is not None:
-                return val
-        return None
+                available_vals.append(val)
+        return available_vals
 
     def sources(self) -> list[str]:
         """Returns the sources of this config as a list of filenames."""
diff --git a/src/python/pants/option/options_bootstrapper.py b/src/python/pants/option/options_bootstrapper.py
--- a/src/python/pants/option/options_bootstrapper.py
+++ b/src/python/pants/option/options_bootstrapper.py
@@ -168,9 +168,10 @@ def filecontent_for(path: str) -> FileContent:
             # were any from aliases.
             # stuhood: This could potentially break the rust client when aliases are used:
             # https://github.com/pantsbuild/pants/pull/13228#discussion_r728223889
+            alias_vals = post_bootstrap_config.get("cli", "alias")
             alias_dict = parse_expression(
                 name="cli.alias",
-                val=post_bootstrap_config.get("cli", "alias") or "{}",
+                val=alias_vals[-1] if alias_vals else "{}",
                 acceptable_types=dict,
             )
             alias = CliAlias.from_dict(alias_dict)
diff --git a/src/python/pants/option/parser.py b/src/python/pants/option/parser.py
--- a/src/python/pants/option/parser.py
+++ b/src/python/pants/option/parser.py
@@ -581,28 +581,40 @@ def expand(val_or_str):
             else:
                 return val_or_str
 
+        # Helper function to merge multiple values from a single rank (e.g., multiple flags,
+        # or multiple config files).
+        def merge_in_rank(vals):
+            if not vals:
+                return None
+            expanded_vals = [to_value_type(expand(x)) for x in vals]
+            if is_list_option(kwargs):
+                return ListValueComponent.merge(expanded_vals)
+            if is_dict_option(kwargs):
+                return DictValueComponent.merge(expanded_vals)
+            return expanded_vals[-1]  # Last value wins.
+
         # Get value from config files, and capture details about its derivation.
         config_details = None
         config_section = GLOBAL_SCOPE_CONFIG_SECTION if self._scope == GLOBAL_SCOPE else self._scope
-        config_default_val_or_str = expand(self._config.get(DEFAULT_SECTION, dest))
-        config_val_or_str = expand(self._config.get(config_section, dest))
+        config_default_val = merge_in_rank(self._config.get(DEFAULT_SECTION, dest))
+        config_val = merge_in_rank(self._config.get(config_section, dest))
         config_source_files = self._config.get_sources_for_option(config_section, dest)
         if config_source_files:
             config_details = f"from {', '.join(config_source_files)}"
 
         # Get value from environment, and capture details about its derivation.
         env_vars = self.get_env_var_names(self._scope, dest)
-        env_val_or_str = None
+        env_val = None
         env_details = None
         if self._env:
             for env_var in env_vars:
                 if env_var in self._env:
-                    env_val_or_str = expand(self._env.get(env_var))
+                    env_val = merge_in_rank([self._env.get(env_var)])
                     env_details = f"from env var {env_var}"
                     break
 
         # Get value from cmd-line flags.
-        flag_vals = [to_value_type(expand(x)) for x in flag_val_strs]
+        flag_vals = list(flag_val_strs)
         if kwargs.get("passthrough"):
             # NB: Passthrough arguments are either of type `str` or `shell_str`
             # (see self._validate): the former never need interpretation, and the latter do not
@@ -611,39 +623,21 @@ def expand(val_or_str):
             flag_vals.append(
                 ListValueComponent(ListValueComponent.MODIFY, [*passthru_arg_strs], [])
             )
-
-        if is_list_option(kwargs):
-            # Note: It's important to set flag_val to None if no flags were specified, so we can
-            # distinguish between no flags set vs. explicit setting of the value to [].
-            flag_val = ListValueComponent.merge(flag_vals) if flag_vals else None
-        elif is_dict_option(kwargs):
-            # Note: It's important to set flag_val to None if no flags were specified, so we can
-            # distinguish between no flags set vs. explicit setting of the value to {}.
-            flag_val = DictValueComponent.merge(flag_vals) if flag_vals else None
-        elif len(flag_vals) > 1:
+        if len(flag_vals) > 1 and not (is_list_option(kwargs) or is_dict_option(kwargs)):
             raise ParseError(
                 f"Multiple cmd line flags specified for option {dest} in {self._scope_str()}"
             )
-        elif len(flag_vals) == 1:
-            flag_val = flag_vals[0]
-        else:
-            flag_val = None
+        flag_val = merge_in_rank(flag_vals)
         flag_details = None if flag_val is None else "from command-line flag"
 
         # Rank all available values.
-        # Note that some of these values may already be of the value type, but type conversion
-        # is idempotent, so this is OK.
-
         values_to_rank = [
-            (to_value_type(x), detail)
-            for (x, detail) in [
-                (flag_val, flag_details),
-                (env_val_or_str, env_details),
-                (config_val_or_str, config_details),
-                (config_default_val_or_str, config_details),
-                (kwargs.get("default"), None),
-                (None, None),
-            ]
+            (flag_val, flag_details),
+            (env_val, env_details),
+            (config_val, config_details),
+            (config_default_val, config_details),
+            (to_value_type(kwargs.get("default")), None),
+            (None, None),
         ]
         # Note that ranked_vals will always have at least one element, and all elements will be
         # instances of RankedValue (so none will be None, although they may wrap a None value).


Test patch:
diff --git a/src/python/pants/option/config_test.py b/src/python/pants/option/config_test.py
--- a/src/python/pants/option/config_test.py
+++ b/src/python/pants/option/config_test.py
@@ -51,6 +51,13 @@ class ConfigFile:
         add = 0
         remove = 0
         nested = { nested_key = 'foo' }
+
+        [list_merging]
+        list1 = []
+        list2 = [1, 2]
+        list3.add = [3, 4]
+        list4.remove = [5]
+        list5 = [6, 7]
         """
     ),
     default_values={
@@ -70,6 +77,13 @@ class ConfigFile:
             "recursively_interpolated_from_section": "overridden_from_default is interpolated (again)",
         },
         "d": {"dict_val": "{'add': 0, 'remove': 0, 'nested': {'nested_key': 'foo'}"},
+        "list_merging": {
+            "list1": "[]",
+            "list2": "[1, 2]",
+            "list3": "+[3, 4]",
+            "list4": "-[5]",
+            "list5": "[6, 7]",
+        },
     },
 )
 
@@ -88,6 +102,14 @@ class ConfigFile:
         list.remove = [8, 9]
 
         [empty_section]
+
+        [list_merging]
+        list1 = [11, 22]
+        list2.add = [33]
+        list3.add = [8, 9]
+        list3.remove = [4, 55]
+        list4 = [66]
+        list6.add = [77, 88]
         """
     ),
     default_values={},
@@ -96,6 +118,13 @@ class ConfigFile:
         "b": {"preempt": "False"},
         "d": {"list": "+[0, 1],-[8, 9]"},
         "empty_section": {},
+        "list_merging": {
+            "list1": "[11, 22]",
+            "list2": "+[33]",
+            "list3": "+[8, 9],-[4, 55]",
+            "list4": "[66]",
+            "list6": "+[77, 88]",
+        },
     },
 )
 
@@ -118,16 +147,40 @@ def setUp(self) -> None:
         self.default_seed_values = Config._determine_seed_values(
             seed_values={"buildroot": "fake_buildroot"},
         )
-        self.expected_combined_values = {
-            **FILE_1.expected_options,
-            **FILE_2.expected_options,
-            "a": {**FILE_2.expected_options["a"], **FILE_1.expected_options["a"]},
+        self.expected_combined_values: dict[str, dict[str, list[str]]] = {
+            "a": {
+                "list": ['["1", "2", "3", "42"]'],
+                "list2": ["+[7, 8, 9]"],
+                "list3": ['-["x", "y", "z"]'],
+                "fast": ["True"],
+            },
+            "b": {"preempt": ["True", "False"]},
+            "c": {
+                "name": ["overridden_from_default"],
+                "interpolated_from_section": ["overridden_from_default is interpolated"],
+                "recursively_interpolated_from_section": [
+                    "overridden_from_default is interpolated (again)"
+                ],
+            },
+            "d": {
+                "dict_val": ["{'add': 0, 'remove': 0, 'nested': {'nested_key': 'foo'}}"],
+                "list": ["+[0, 1],-[8, 9]"],
+            },
+            "empty_section": {},
+            "list_merging": {
+                "list1": ["[]", "[11, 22]"],
+                "list2": ["[1, 2]", "+[33]"],
+                "list3": ["+[3, 4]", "+[8, 9],-[4, 55]"],
+                "list4": ["-[5]", "[66]"],
+                "list5": ["[6, 7]"],
+                "list6": ["+[77, 88]"],
+            },
         }
 
     def test_default_values(self) -> None:
         # This is used in `options_bootstrapper.py` to ignore default values when validating options.
-        file1_config_values = self.config.values[0]
-        file2_config_values = self.config.values[1]
+        file1_values = self.config.values[0]
+        file2_values = self.config.values[1]
         # NB: string interpolation should only happen when calling _ConfigValues.get_value(). The
         # values for _ConfigValues.defaults are not yet interpolated.
         default_file1_values_unexpanded = {
@@ -135,20 +188,25 @@ def test_default_values(self) -> None:
             "path": "/a/b/%(answer)s",
             "embed": "%(path)s::foo",
         }
-        assert file1_config_values.defaults == {
+        assert file1_values.defaults == {
             **self.default_seed_values,
             **default_file1_values_unexpanded,
         }
-        assert file2_config_values.defaults == self.default_seed_values
+        assert file2_values.defaults == self.default_seed_values
 
     def test_get(self) -> None:
         # Check the DEFAULT section
-        for option, value in {**self.default_seed_values, **FILE_1.default_values}.items():
-            assert self.config.get(section="DEFAULT", option=option) == value
+        for option, value in self.default_seed_values.items():
+            # Both config files have the seed values.
+            assert self.config.get(section="DEFAULT", option=option) == [value, value]
+        for option, value in FILE_1.default_values.items():
+            # Only FILE_1 has explicit DEFAULT values.
+            assert self.config.get(section="DEFAULT", option=option) == [value]
+
         # Check the combined values.
         for section, section_values in self.expected_combined_values.items():
-            for option, value in section_values.items():
-                assert self.config.get(section=section, option=option) == value
+            for option, value_list in section_values.items():
+                assert self.config.get(section=section, option=option) == value_list
 
     def test_empty(self) -> None:
         config = Config.load([])
diff --git a/src/python/pants/option/custom_types_test.py b/src/python/pants/option/custom_types_test.py
--- a/src/python/pants/option/custom_types_test.py
+++ b/src/python/pants/option/custom_types_test.py
@@ -65,10 +65,12 @@ def test_flatten_shlexed_list() -> None:
 
 
 class TestCustomTypes:
-    def assert_list_parsed(self, s: str, *, expected: ParsedList) -> None:
+    @staticmethod
+    def assert_list_parsed(s: str, *, expected: ParsedList) -> None:
         assert expected == ListValueComponent.create(s).val
 
-    def assert_split_list(self, s: str, *, expected: List[str]) -> None:
+    @staticmethod
+    def assert_split_list(s: str, *, expected: List[str]) -> None:
         assert expected == ListValueComponent._split_modifier_expr(s)
 
     def test_unset_bool(self):
@@ -149,7 +151,6 @@ def test_split_list_modifier_expressions(self) -> None:
         self.assert_split_list("+1,2],-[3,4", expected=["+1,2]", "-[3,4"])
         self.assert_split_list("+(1,2],-[3,4)", expected=["+(1,2]", "-[3,4)"])
 
-    #
     @pytest.mark.xfail(
         reason="The heuristic list modifier expression splitter cannot handle certain very unlikely cases."
     )
diff --git a/src/python/pants/option/options_test.py b/src/python/pants/option/options_test.py
--- a/src/python/pants/option/options_test.py
+++ b/src/python/pants/option/options_test.py
@@ -509,8 +509,16 @@ def register(opts: Options) -> None:
 
 class OptionsTest(unittest.TestCase):
     @staticmethod
-    def _create_config(config: dict[str, dict[str, str]] | None = None) -> Config:
-        return Config.load([FileContent("test_config.toml", toml.dumps(config or {}).encode())])
+    def _create_config(
+        config: dict[str, dict[str, str]] | None = None,
+        config2: dict[str, dict[str, str]] | None = None,
+    ) -> Config:
+        return Config.load(
+            [
+                FileContent("test_config.toml", toml.dumps(config or {}).encode()),
+                FileContent("test_config2.toml", toml.dumps(config2 or {}).encode()),
+            ]
+        )
 
     def _parse(
         self,
@@ -518,12 +526,13 @@ def _parse(
         flags: str = "",
         env: dict[str, str] | None = None,
         config: dict[str, dict[str, Any]] | None = None,
+        config2: dict[str, dict[str, Any]] | None = None,
         bootstrap_option_values=None,
     ) -> Options:
         args = ["./pants", *shlex.split(flags)]
         options = Options.create(
             env=env or {},
-            config=self._create_config(config),
+            config=self._create_config(config, config2),
             known_scope_infos=OptionsTest._known_scope_infos,
             args=args,
             bootstrap_option_values=bootstrap_option_values,
@@ -771,10 +780,14 @@ def check(
             flags: str = "",
             env_val: str | None = None,
             config_val: str | None = None,
+            config2_val: str | None = None,
         ) -> None:
             env = {"PANTS_GLOBAL_LISTY": env_val} if env_val else None
             config = {"GLOBAL": {"listy": config_val}} if config_val else None
-            global_options = self._parse(flags=flags, env=env, config=config).for_global_scope()
+            config2 = {"GLOBAL": {"listy": config2_val}} if config2_val else None
+            global_options = self._parse(
+                flags=flags, env=env, config=config, config2=config2
+            ).for_global_scope()
             assert global_options.listy == expected
 
         default = [1, 2, 3]
@@ -795,8 +808,12 @@ def check(
         check(
             flags="--listy=+[8,9]",
             env_val="+[6,7]",
-            config_val="+[4,5]",
-            expected=[*default, 4, 5, 6, 7, 8, 9],
+            config_val="+[4,5],+[45]",
+            expected=[*default, 4, 5, 45, 6, 7, 8, 9],
+        )
+        check(
+            config_val="+[4,5],-[4]",
+            expected=[*default, 5],
         )
 
         # Appending and filtering across env, config and flags (in the right order).
@@ -804,7 +821,8 @@ def check(
             flags="--listy=-[1,5,6]",
             env_val="+[6,7]",
             config_val="+[4,5]",
-            expected=[2, 3, 4, 7],
+            config2_val="+[99]",
+            expected=[2, 3, 4, 99, 7],
         )
         check(
             flags="--listy=+[8,9]",
@@ -812,6 +830,17 @@ def check(
             config_val="+[4,5],-[3]",
             expected=[1, 2, 8, 9],
         )
+        # Appending a value from a fromfile.
+        with temporary_file(binary_mode=False) as fp:
+            fp.write("-[3]")
+            fp.close()
+            check(
+                flags="--listy=+[8,9]",
+                env_val="-[4,5]",
+                config_val="+[4,5]",
+                config2_val=f"@{fp.name}",
+                expected=[1, 2, 8, 9],
+            )
 
         # Overwriting from env, then appending and filtering.
         check(
@@ -826,7 +855,8 @@ def check(
             flags="--listy=+[8,9]",
             env_val="+[6,7]",
             config_val="[4,5]",
-            expected=[4, 5, 6, 7, 8, 9],
+            config2_val="-[4]",
+            expected=[5, 6, 7, 8, 9],
         )
 
         # Overwriting from flags.
@@ -971,9 +1001,13 @@ def check(
             expected: dict[str, str],
             flags: str = "",
             config_val: str | None = None,
+            config2_val: str | None = None,
         ) -> None:
             config = {"GLOBAL": {"dicty": config_val}} if config_val else None
-            global_options = self._parse(flags=flags, config=config).for_global_scope()
+            config2 = {"GLOBAL": {"dicty": config2_val}} if config2_val else None
+            global_options = self._parse(
+                flags=flags, config=config, config2=config2
+            ).for_global_scope()
             assert global_options.dicty == expected
 
         default = {"a": "b"}
@@ -987,6 +1021,12 @@ def check(
 
         check(config_val='{"c": "d"}', expected=specified_args)
         check(config_val='+{"c": "d"}', expected=all_args)
+        check(
+            config_val='+{"c": "d"}',
+            config2_val='+{"e": "f"}',
+            flags='--dicty=\'+{"g": "h"}\'',
+            expected={**all_args, "e": "f", "g": "h"},
+        )
         check(
             config_val='+{"c": "d"}',
             flags='--dicty=\'+{"e": "f"}\'',