Repository: pantsbuild/pants

Issue:
<unrecognized symbol> Error when using marco inside BUILD file
**Describe the bug**
I tried upgrading pants to 2.18.0. I have a macro file `build_file_prelude_globs = ["pants_plugins/macros.py"]` which worked in in pants 2.16.0 and 2.17.0. When running `pants` with version `2.18.0` I get:
```
Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 112, in main
    PantsLoader.main()
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 108, in main
    cls.run_default_entrypoint()
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 91, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 142, in run
    runner = LocalPantsRunner.create(
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 93, in create
    build_config = options_initializer.build_config(options_bootstrapper, env)
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/init/options_initializer.py", line 153, in build_config
    return _initialize_build_configuration(self._plugin_resolver, options_bootstrapper, env)
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/init/options_initializer.py", line 52, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper, env, backends_requirements)
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/init/plugin_resolver.py", line 141, in resolve
    for resolved_plugin_location in self._resolve_plugins(options_bootstrapper, env, request):
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/init/plugin_resolver.py", line 163, in _resolve_plugins
    params = Params(request, determine_bootstrap_environment(session))
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/core/util_rules/environments.py", line 448, in determine_bootstrap_environment
    session.product_request(ChosenLocalEnvironmentName, [Params()])[0],
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 581, in product_request
    return self.execute(request)
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 522, in execute
    self._raise_on_error([t for _, t in throws])
  File "/home/vscode/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 506, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

InvalidTargetException: BUILD:25: The 'subprocess_environment_env_vars' field in target //:local-environment must be an iterable of strings (e.g. a list of strings), but was `env_vars_devcontainer()` with type `<unrecognized symbol>`.

NoneType: None
```
It's a really simple macro which just returns something like `["env1=val1", "...."]`
**Pants version**
2.18.0

**OS**
WSL devcontainer.

**Additional info**
Reminds me of https://github.com/pantsbuild/pants/issues/19156


Discussion:
The issue is with using macros for environment targets. That is currently not supported.
It may have been silently ignored before, where as now you get a proper error message.

Did `pants peek //:local-environment` provide the expected value for `subprocess_environment_env_vars` on 2.17.0?

Edit: yes, it did.
OK, to clarify. Macros are not loaded during the bootstrap phase when environment targets are loaded. However, it seems that in certain cases, they are loaded again later with macros present and those that use environment targets in that setting, it may work using macros.

I'd suggest not relying on this behaviour as it is rather nuanced and may change unexpectedly between pants releases. I have submitted a fix however to re-open the previous loop-hole.

Resolving patch:
diff --git a/src/python/pants/engine/internals/parser.py b/src/python/pants/engine/internals/parser.py
--- a/src/python/pants/engine/internals/parser.py
+++ b/src/python/pants/engine/internals/parser.py
@@ -270,6 +270,12 @@ def __str__(self) -> str:
         return self._type_alias
 
     def __call__(self, **kwargs: Any) -> TargetAdaptor:
+        if self._parse_state.is_bootstrap and any(
+            isinstance(v, _UnrecognizedSymbol) for v in kwargs.values()
+        ):
+            # Remove any field values that are not recognized during the bootstrap phase.
+            kwargs = {k: v for k, v in kwargs.items() if not isinstance(v, _UnrecognizedSymbol)}
+
         # Target names default to the name of the directory their BUILD file is in
         # (as long as it's not the root directory).
         if "name" not in kwargs:


Test patch:
diff --git a/src/python/pants/engine/internals/build_files_test.py b/src/python/pants/engine/internals/build_files_test.py
--- a/src/python/pants/engine/internals/build_files_test.py
+++ b/src/python/pants/engine/internals/build_files_test.py
@@ -711,6 +711,42 @@ def test_default_plugin_field_bootstrap() -> None:
     assert dict(tags=("ok",)) == dict(address_family.defaults["mock_tgt"])
 
 
+def test_environment_target_macro_field_value() -> None:
+    rule_runner = RuleRunner(
+        rules=[QueryRule(AddressFamily, [AddressFamilyDir])],
+        target_types=[MockTgt],
+        is_bootstrap=True,
+    )
+    rule_runner.set_options(
+        args=("--build-file-prelude-globs=prelude.py",),
+    )
+    rule_runner.write_files(
+        {
+            "prelude.py": dedent(
+                """
+                def tags():
+                    return ["foo", "bar"]
+                """
+            ),
+            "BUILD": dedent(
+                """
+                mock_tgt(name="tgt", tags=tags())
+                """
+            ),
+        }
+    )
+
+    # Parse the root BUILD file.
+    address_family = rule_runner.request(AddressFamily, [AddressFamilyDir("")])
+    tgt = address_family.name_to_target_adaptors["tgt"][1]
+    # We're pretending that field values returned from a called macro function doesn't exist during
+    # bootstrap. This is to allow the semi-dubios use of macro calls for environment target field
+    # values that are not required, and depending on how they are used, it may work to only have
+    # those field values set during normal lookup.
+    assert not tgt.kwargs
+    assert tgt == TargetAdaptor("mock_tgt", "tgt", "BUILD:2")
+
+
 def test_build_file_env_vars(target_adaptor_rule_runner: RuleRunner) -> None:
     target_adaptor_rule_runner.write_files(
         {