Repository: pantsbuild/pants

Issue:
Parametrization does not work with address globs `::` and `AllTargets`
We are currently applying parametrization from https://github.com/pantsbuild/pants/issues/13882 lazily, at the same time as generating targets, i.e. going from `UnexpandedTargets -> Targets`.

This means that the `UnexpandedTargets` will try to store the "template" target as an actual `Address`/`Target`, even though it does not exist, by definition. This happens here:

https://github.com/pantsbuild/pants/blob/05ec375c8bfcaa0396c673847bb139326883cc08/src/python/pants/engine/internals/build_files.py#L214-L219

`AddressFamily` parses BUILD files and creates all the `TargetAdaptor` instances belonging to it by calling `AddressMap.parse()`:

https://github.com/pantsbuild/pants/blob/05ec375c8bfcaa0396c673847bb139326883cc08/src/python/pants/engine/internals/mapper.py#L44-L47

https://github.com/pantsbuild/pants/blob/05ec375c8bfcaa0396c673847bb139326883cc08/src/python/pants/engine/internals/mapper.py#L117-L122

We should never have an address for the "template" target, as it fundamentally does not exist. It cannot be a member of `UnexpandedTargets`, and I think ideally wouldn't even have a `TargetAdaptor` for the template target, only for each parametrized target.

To fix this, I think we probably want to start applying parametrization inside `AddressMap.parse()`! That would avoid the rest of our code having to learn about parametrization.

> To fix this, I think we probably want to start applying parametrization inside `AddressMap.parse()`! 

Naively, I was thinking a downside of this would mean things like `./pants peek path/to:template` won't work. But that cannot/should not ever work. Again, it's not a valid target, it's not addressable.

This speaks to how we need to really focus in our docs on explaining that `parametrize` is a special mechanism that is a convenince to create multiple target definitions for you.


Discussion:
> To fix this, I think we probably want to start applying parametrization inside `AddressMap.parse()`! That would avoid the rest of our code having to learn about parametrization.
> 
> > To fix this, I think we probably want to start applying parametrization inside `AddressMap.parse()`!
> 
> Naively, I was thinking a downside of this would mean things like `./pants peek path/to:template` won't work. But that cannot/should not ever work. Again, it's not a valid target, it's not addressable.

This was how I implemented things initially, but I wasn't sure how it was going to work with `overrides`. Thanks for diving in on #14496.
@Eric-Arellano : How does one reproduce the breakage that you're referring to here? AFAICT, https://github.com/pantsbuild/example-jvm/pull/10/commits/fbd18ccdf977e5a0252af0d0237a151a369ae346 passes that repo's CI and generally works as expected, as long as the `jdk` is a `moved_field` (see #14509).
Ok: I was very easily able to reproduce with https://github.com/pantsbuild/example-jvm/pull/12. I haven't yet determined the difference between https://github.com/pantsbuild/example-jvm/pull/10 and that one in terms of this causing an error though. Will investigate further today.

Resolving patch:
diff --git a/src/python/pants/engine/internals/build_files.py b/src/python/pants/engine/internals/build_files.py
--- a/src/python/pants/engine/internals/build_files.py
+++ b/src/python/pants/engine/internals/build_files.py
@@ -15,10 +15,11 @@
 from pants.engine.engine_aware import EngineAwareParameter
 from pants.engine.fs import DigestContents, GlobMatchErrorBehavior, PathGlobs, Paths
 from pants.engine.internals.mapper import AddressFamily, AddressMap, AddressSpecsFilter
+from pants.engine.internals.parametrize import _TargetParametrizations
 from pants.engine.internals.parser import BuildFilePreludeSymbols, Parser, error_on_imports
 from pants.engine.internals.target_adaptor import TargetAdaptor
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
-from pants.engine.target import Targets, UnexpandedTargets, WrappedTarget
+from pants.engine.target import WrappedTarget
 from pants.option.global_options import GlobalOptions
 from pants.util.docutil import bin_name, doc_url
 from pants.util.frozendict import FrozenDict
@@ -212,19 +213,19 @@ async def addresses_from_address_specs(
     )
     dirnames = {os.path.dirname(f) for f in build_file_paths.files}
     address_families = await MultiGet(Get(AddressFamily, AddressFamilyDir(d)) for d in dirnames)
-    candidate_addresses = Addresses(
+    base_addresses = Addresses(
         itertools.chain.from_iterable(
             address_family.addresses_to_target_adaptors for address_family in address_families
         )
     )
 
-    tgts_generators_kept, tgts_generators_replaced = await MultiGet(
-        Get(UnexpandedTargets, Addresses, candidate_addresses),
-        Get(Targets, Addresses, candidate_addresses),
+    target_parametrizations_list = await MultiGet(
+        Get(_TargetParametrizations, Address, base_address) for base_address in base_addresses
     )
     residence_dir_to_targets = defaultdict(list)
-    for tgt in (*tgts_generators_kept, *tgts_generators_replaced):
-        residence_dir_to_targets[tgt.residence_dir].append(tgt)
+    for target_parametrizations in target_parametrizations_list:
+        for tgt in target_parametrizations.all():
+            residence_dir_to_targets[tgt.residence_dir].append(tgt)
 
     matched_globs = set()
     for glob_spec in address_specs.globs:
diff --git a/src/python/pants/engine/internals/graph.py b/src/python/pants/engine/internals/graph.py
--- a/src/python/pants/engine/internals/graph.py
+++ b/src/python/pants/engine/internals/graph.py
@@ -9,7 +9,7 @@
 import os.path
 from dataclasses import dataclass
 from pathlib import PurePath
-from typing import Iterable, Iterator, NamedTuple, Sequence, cast
+from typing import Iterable, NamedTuple, Sequence, cast
 
 from pants.base.deprecated import warn_or_error
 from pants.base.exceptions import ResolveError
@@ -40,6 +40,9 @@
 )
 from pants.engine.internals import native_engine
 from pants.engine.internals.parametrize import Parametrize
+from pants.engine.internals.parametrize import (  # noqa: F401
+    _TargetParametrizations as _TargetParametrizations,
+)
 from pants.engine.internals.target_adaptor import TargetAdaptor
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
 from pants.engine.target import (
@@ -176,37 +179,6 @@ def warn_deprecated_field_type(request: _WarnDeprecatedFieldRequest) -> _WarnDep
     return _WarnDeprecatedField()
 
 
-@dataclass(frozen=True)
-class _TargetParametrizations:
-    """All parametrizations and generated targets for a single input Address.
-
-    If a Target has been parametrized, it might _not_ be present in this output, due to it not being
-    addressable using its un-parameterized Address.
-    """
-
-    original_target: Target | None
-    parametrizations: FrozenDict[Address, Target]
-
-    def get(self, address: Address) -> Target | None:
-        if self.original_target and self.original_target.address == address:
-            return self.original_target
-        return self.parametrizations.get(address)
-
-    def generated_or_generator(self, maybe_generator: Address) -> Iterator[Target]:
-        if not self.original_target:
-            raise ValueError(
-                "A `parametrized` target cannot be consumed without its parameters specified.\n"
-                f"Target `{maybe_generator}` can be addressed as:\n"
-                f"{bullet_list(addr.spec for addr in self.parametrizations)}"
-            )
-        if self.parametrizations:
-            # Generated Targets.
-            yield from self.parametrizations.values()
-        else:
-            # Did not generate targets.
-            yield self.original_target
-
-
 @rule
 async def resolve_target_parametrizations(
     address: Address,
diff --git a/src/python/pants/engine/internals/parametrize.py b/src/python/pants/engine/internals/parametrize.py
--- a/src/python/pants/engine/internals/parametrize.py
+++ b/src/python/pants/engine/internals/parametrize.py
@@ -9,7 +9,10 @@
 
 from pants.build_graph.address import BANNED_CHARS_IN_PARAMETERS
 from pants.engine.addresses import Address
+from pants.engine.target import Target
+from pants.util.frozendict import FrozenDict
 from pants.util.meta import frozen_after_init
+from pants.util.strutil import bullet_list
 
 
 @frozen_after_init
@@ -104,3 +107,43 @@ def __repr__(self) -> str:
         strs = [str(s) for s in self.args]
         strs.extend(f"{alias}={value}" for alias, value in self.kwargs.items())
         return f"parametrize({', '.join(strs)}"
+
+
+# TODO: This is not the right name for this class, nor the best place for it to live. But it is
+# consumed by both `pants.engine.internals.graph` and `pants.engine.internals.build_files`, and
+# shouldn't live in `pants.engine.target` (yet? needs more stabilization).
+@dataclass(frozen=True)
+class _TargetParametrizations:
+    """All parametrizations and generated targets for a single input Address.
+
+    If a Target has been parametrized, it might _not_ be present in this output, due to it not being
+    addressable using its un-parameterized Address.
+    """
+
+    original_target: Target | None
+    parametrizations: FrozenDict[Address, Target]
+
+    def all(self) -> Iterator[Target]:
+        """Iterates over all Target instances which are valid after parametrization."""
+        if self.original_target:
+            yield self.original_target
+        yield from self.parametrizations.values()
+
+    def get(self, address: Address) -> Target | None:
+        if self.original_target and self.original_target.address == address:
+            return self.original_target
+        return self.parametrizations.get(address)
+
+    def generated_or_generator(self, maybe_generator: Address) -> Iterator[Target]:
+        if not self.original_target:
+            raise ValueError(
+                "A `parametrized` target cannot be consumed without its parameters specified.\n"
+                f"Target `{maybe_generator}` can be addressed as:\n"
+                f"{bullet_list(addr.spec for addr in self.parametrizations)}"
+            )
+        if self.parametrizations:
+            # Generated Targets.
+            yield from self.parametrizations.values()
+        else:
+            # Did not generate targets.
+            yield self.original_target


Test patch:
diff --git a/src/python/pants/backend/scala/dependency_inference/rules_test.py b/src/python/pants/backend/scala/dependency_inference/rules_test.py
--- a/src/python/pants/backend/scala/dependency_inference/rules_test.py
+++ b/src/python/pants/backend/scala/dependency_inference/rules_test.py
@@ -13,6 +13,7 @@
 from pants.backend.scala.target_types import rules as scala_target_rules
 from pants.core.util_rules import config_files, source_files
 from pants.engine.addresses import Address, Addresses, UnparsedAddressInputs
+from pants.engine.internals.parametrize import Parametrize
 from pants.engine.target import (
     DependenciesRequest,
     ExplicitlyProvidedDependencies,
@@ -45,6 +46,7 @@ def rule_runner() -> RuleRunner:
             QueryRule(Targets, [UnparsedAddressInputs]),
         ],
         target_types=[ScalaSourcesGeneratorTarget],
+        objects={"parametrize": Parametrize},
     )
     rule_runner.set_options(args=[], env_inherit=PYTHON_BOOTSTRAP_ENV)
     return rule_runner
@@ -241,8 +243,7 @@ def test_multi_resolve_dependency_inference(rule_runner: RuleRunner) -> None:
         {
             "lib/BUILD": dedent(
                 """\
-            scala_sources(name="lib_2_13", resolve="scala-2.13")
-            scala_sources(name="lib_2_12", resolve="scala-2.12")
+            scala_sources(resolve=parametrize("scala-2.13", "scala-2.12"))
             """
             ),
             "lib/Library.scala": dedent(
@@ -258,8 +259,7 @@ def grok(): Unit = {
             ),
             "user/BUILD": dedent(
                 """\
-            scala_sources(name="user_2_13", resolve="scala-2.13")
-            scala_sources(name="user_2_12", resolve="scala-2.12")
+            scala_sources(resolve=parametrize("scala-2.13", "scala-2.12"))
             """
             ),
             "user/Main.scala": dedent(
@@ -285,21 +285,21 @@ def main(args: Array[String]): Unit = {
     )
 
     tgt = rule_runner.get_target(
-        Address("user", relative_file_path="Main.scala", target_name="user_2_13")
+        Address("user", relative_file_path="Main.scala", parameters={"resolve": "scala-2.13"})
     )
     deps = rule_runner.request(
         InferredDependencies, [InferScalaSourceDependencies(tgt[ScalaSourceField])]
     )
     assert deps == InferredDependencies(
-        [Address("lib", relative_file_path="Library.scala", target_name="lib_2_13")]
+        [Address("lib", relative_file_path="Library.scala", parameters={"resolve": "scala-2.13"})]
     )
 
     tgt = rule_runner.get_target(
-        Address("user", relative_file_path="Main.scala", target_name="user_2_12")
+        Address("user", relative_file_path="Main.scala", parameters={"resolve": "scala-2.12"})
     )
     deps = rule_runner.request(
         InferredDependencies, [InferScalaSourceDependencies(tgt[ScalaSourceField])]
     )
     assert deps == InferredDependencies(
-        [Address("lib", relative_file_path="Library.scala", target_name="lib_2_12")]
+        [Address("lib", relative_file_path="Library.scala", parameters={"resolve": "scala-2.12"})]
     )
diff --git a/src/python/pants/engine/internals/build_files_test.py b/src/python/pants/engine/internals/build_files_test.py
--- a/src/python/pants/engine/internals/build_files_test.py
+++ b/src/python/pants/engine/internals/build_files_test.py
@@ -28,6 +28,7 @@
     evaluate_preludes,
     parse_address_family,
 )
+from pants.engine.internals.parametrize import Parametrize
 from pants.engine.internals.parser import BuildFilePreludeSymbols, Parser
 from pants.engine.internals.scheduler import ExecutionError
 from pants.engine.internals.target_adaptor import TargetAdaptor
@@ -40,6 +41,7 @@
     SingleSourceField,
     SourcesPaths,
     SourcesPathsRequest,
+    StringField,
     Tags,
     Target,
     TargetGenerator,
@@ -121,6 +123,10 @@ def make_target():
         run_prelude_parsing_rule(prelude_content)
 
 
+class ResolveField(StringField):
+    alias = "resolve"
+
+
 class MockTgt(Target):
     alias = "mock_tgt"
     core_fields = (Dependencies, MultipleSourcesField, Tags)
@@ -128,14 +134,19 @@ class MockTgt(Target):
 
 class MockGeneratedTarget(Target):
     alias = "generated"
-    core_fields = (Dependencies, SingleSourceField, Tags)
+    core_fields = (
+        Dependencies,
+        ResolveField,
+        SingleSourceField,
+        Tags,
+    )
 
 
 class MockTargetGenerator(TargetGenerator):
     alias = "generator"
     core_fields = (Dependencies, MultipleSourcesField, Tags)
     copied_fields = (Tags,)
-    moved_fields = ()
+    moved_fields = (ResolveField,)
 
 
 class MockGenerateTargetsRequest(GenerateTargetsRequest):
@@ -292,6 +303,7 @@ def address_specs_rule_runner() -> RuleRunner:
             UnionRule(GenerateTargetsRequest, MockGenerateTargetsRequest),
             QueryRule(Addresses, [AddressSpecs]),
         ],
+        objects={"parametrize": Parametrize},
         target_types=[MockTgt, MockGeneratedTarget, MockTargetGenerator],
     )
 
@@ -566,3 +578,28 @@ def test_address_specs_generated_target_does_not_belong_to_generator(
     assert (
         "The address `demo/f.txt:not_owner` was not generated by the target `demo:not_owner`"
     ) in str(exc.value)
+
+
+def test_address_specs_parametrize(
+    address_specs_rule_runner: RuleRunner,
+) -> None:
+    address_specs_rule_runner.write_files(
+        {
+            "demo/f.txt": "",
+            "demo/BUILD": dedent(
+                """\
+                generator(sources=['f.txt'], resolve=parametrize("a", "b"))
+                """
+            ),
+        }
+    )
+
+    assert resolve_address_specs(address_specs_rule_runner, [DescendantAddresses(""),]) == {
+        Address("demo", generated_name="f.txt", parameters={"resolve": "a"}),
+        Address("demo", generated_name="f.txt", parameters={"resolve": "b"}),
+        Address("demo", relative_file_path="f.txt", parameters={"resolve": "a"}),
+        Address("demo", relative_file_path="f.txt", parameters={"resolve": "b"}),
+        # TODO: This needs further work: in order for aliases to be useful, they will need to
+        # include some parameters. But exactly what to do about overrides is an open question.
+        Address("demo"),
+    }
diff --git a/src/python/pants/jvm/dependency_inference/artifact_mapper_test.py b/src/python/pants/jvm/dependency_inference/artifact_mapper_test.py
--- a/src/python/pants/jvm/dependency_inference/artifact_mapper_test.py
+++ b/src/python/pants/jvm/dependency_inference/artifact_mapper_test.py
@@ -17,6 +17,7 @@
 from pants.backend.java.target_types import rules as java_target_rules
 from pants.core.util_rules import config_files, source_files
 from pants.engine.addresses import Address, Addresses
+from pants.engine.internals.parametrize import Parametrize
 from pants.engine.target import Dependencies, DependenciesRequest
 from pants.jvm.dependency_inference.artifact_mapper import (
     FrozenTrieNode,
@@ -52,6 +53,7 @@ def rule_runner() -> RuleRunner:
             QueryRule(Addresses, [DependenciesRequest]),
             QueryRule(ThirdPartyPackageToArtifactMapping, []),
         ],
+        objects={"parametrize": Parametrize},
         target_types=[
             JavaSourceTarget,
             JavaSourcesGeneratorTarget,
@@ -158,15 +160,18 @@ def test_third_party_dep_inference_resolve(rule_runner: RuleRunner) -> None:
         {
             "BUILD": dedent(
                 """\
-                artifact_args = {"group": "joda-time", "artifact": "joda-time", "version": "2.10.10"}
-
-                jvm_artifact(name="artifact_a", **artifact_args, resolve="a")
-                jvm_artifact(name="artifact_b", **artifact_args, resolve="b")
-                jvm_artifact(name="artifact_c", **artifact_args, resolve="c")
-
-                java_source(name='lib_a', source='PrintDate.java', resolve='a')
-                java_source(name='lib_b', source='PrintDate.java', resolve='b')
-                java_source(name='lib_c', source='PrintDate.java', resolve='c')
+                jvm_artifact(
+                    name="artifact",
+                    group="joda-time",
+                    artifact="joda-time",
+                    version="2.10.10",
+                    resolve=parametrize("a", "b", "c"),
+                )
+                java_source(
+                    name="lib",
+                    source="PrintDate.java",
+                    resolve=parametrize("a", "b", "c"),
+                )
                 """
             ),
             "PrintDate.java": dedent(
@@ -179,15 +184,16 @@ def test_third_party_dep_inference_resolve(rule_runner: RuleRunner) -> None:
         }
     )
 
-    def assert_inferred(lib_name: str, expected: list[str]) -> None:
-        lib = rule_runner.get_target(Address("", target_name=lib_name))
+    def assert_inferred(resolve: str) -> None:
+        lib = rule_runner.get_target(
+            Address("", target_name="lib", parameters={"resolve": resolve})
+        )
         assert rule_runner.request(
             Addresses, [DependenciesRequest(lib[Dependencies])]
-        ) == Addresses(Address("", target_name=name) for name in expected)
+        ) == Addresses([Address("", target_name="artifact", parameters={"resolve": resolve})])
 
-    assert_inferred("lib_a", ["artifact_a"])
-    assert_inferred("lib_b", ["artifact_b"])
-    assert_inferred("lib_c", ["artifact_c"])
+    for r in ("a", "b", "c"):
+        assert_inferred(r)
 
 
 @maybe_skip_jdk_test