Repository: pantsbuild/pants

Issue:
Target parametrization should default the values of parameters during address expansion
As alluded to in the [Resolution section of the design doc](https://docs.google.com/document/d/1mzZWnXiE6OkgMH_Dm7WeA3ck9veusQmr_c6GWPb_tsQ/edit#heading=h.b9cow77oy8vk), un-parametrized target addresses for "static" parametrization should be expanded to the relevant parametrization when used in the `dependencies` field of targets.

This is important because it allows a dependee target which is itself parametrized to use an un-parametrized address to refer to an explicitly declared dependency. Otherwise, the dependencies field _itself_ would need to be parametrized in order to be valid. For example:
```python
scala_sources(resolve=parametrize("a", "b"))

deploy_jar(
    ...,
    resolve="a",
    dependencies=[
        ":app@resolve=a",
    ],
)

deploy_jar(
    ...,
    resolve="b",
    dependencies=[
        ":app@resolve=b",
    ],
)
```
... could become:
```python
scala_sources(resolve=parametrize("a", "b"))

deploy_jar(
    ...,
    resolve=parametrize("a", "b"),
    dependencies=[
        # Newly legal, because we defaulted parametrization of addresses.
        ":app",
    ],
)
```


Discussion:
How does the parametrization of addresses know what to include in the address for the dependency?
Example:
```python
scala_sources(resolve=parametrize("a", "b"))

deploy_jar(
    ...,
    unrelated=parametrize("c", "d"),
    resolve=parametrize("a", "b"),
    dependencies=[
        # Newly legal, because we defaulted parametrization of addresses. Does it also include `@unrelated` or not for `:app`.. and why so?
        ":app",
    ],
)
```

> How does the parametrization of addresses know what to include in the address for the dependency?

Good question! I think that resolving explicitly specified dependencies might actually involve needing to compare field values (iff both the consuming target and the consumed target have been parametrized). But we'll see. Starting this one now.
Reopening since this needs some tests (discussed on #14788).
The tests won't need to be cherry-picked, so am removing from `2.11.x`.
Two important issues have been reported for this feature:
1. an explicit dep without a resolve specified on a `parametrize`d generator from a `parametrize`d dependee (`x/y/z.py@resolve=one` -> `a/b`) is not filled in.
2. an explicit dep without a resolve specified on a `parametrize`d generated target (a thirdparty requirement, in particular) from a `parametrize`d dependee (`x/y/z.py@resolve=one` -> `a/b#c`) is not filled in.

Both are bad, but the latter case is worse, because there seemingly is an exact match for the edge listed [in the error message](https://github.com/pantsbuild/pants/blob/59f3e2a927049a345ef4d2427f33cddd34d3f1ae/src/python/pants/engine/internals/parametrize.py#L266-L272).

Resolving patch:
diff --git a/src/python/pants/engine/internals/parametrize.py b/src/python/pants/engine/internals/parametrize.py
--- a/src/python/pants/engine/internals/parametrize.py
+++ b/src/python/pants/engine/internals/parametrize.py
@@ -208,14 +208,19 @@ def get_subset(self, address: Address, consumer: Target) -> Target:
             return instance
 
         def remaining_fields_match(candidate: Target) -> bool:
-            """Returns true if all Fields absent from the candidate's Address match the consumer."""
+            """Returns true if all Fields absent from the candidate's Address match the consumer.
+
+            TODO: This does not account for the computed default values of Fields:
+              see https://github.com/pantsbuild/pants/issues/16175
+            """
+
             unspecified_param_field_names = {
                 key for key in candidate.address.parameters.keys() if key not in address.parameters
             }
 
             return all(
-                consumer.has_field(field_type) and consumer[field_type] == field_value
-                for field_type, field_value in candidate.field_values.items()
+                consumer.has_field(field_type) and consumer[field_type].value == field.value
+                for field_type, field in candidate.field_values.items()
                 if field_type.alias in unspecified_param_field_names
             )
 


Test patch:
diff --git a/src/python/pants/engine/internals/graph_test.py b/src/python/pants/engine/internals/graph_test.py
--- a/src/python/pants/engine/internals/graph_test.py
+++ b/src/python/pants/engine/internals/graph_test.py
@@ -118,7 +118,7 @@ class MockTarget(Target):
     deprecated_alias_removal_version = "9.9.9.dev0"
 
 
-class ResolveField(StringField):
+class ResolveField(StringField, AsyncFieldMixin):
     alias = "resolve"
 
 
@@ -1009,9 +1009,9 @@ def assert_generated(
     address: Address,
     build_content: str,
     files: list[str],
-    expected: set[Target],
+    expected_targets: set[Target] | None = None,
     *,
-    dependencies: dict[str, set[str]] | None = None,
+    expected_dependencies: dict[str, set[str]] | None = None,
 ) -> None:
     rule_runner.write_files(
         {
@@ -1020,11 +1020,14 @@ def assert_generated(
         }
     )
     parametrizations = rule_runner.request(_TargetParametrizations, [address])
-    assert expected == {
-        t for parametrization in parametrizations for t in parametrization.parametrization.values()
-    }
+    if expected_targets is not None:
+        assert expected_targets == {
+            t
+            for parametrization in parametrizations
+            for t in parametrization.parametrization.values()
+        }
 
-    if dependencies is not None:
+    if expected_dependencies is not None:
         # TODO: Adjust the `TransitiveTargets` API to expose the complete mapping.
         #   see https://github.com/pantsbuild/pants/issues/11270
         specs = SpecsParser(rule_runner.build_root).parse_specs(["::"])
@@ -1040,7 +1043,7 @@ def assert_generated(
         )
         assert {
             k.spec: {a.spec for a in v} for k, v in dependency_mapping.mapping.items()
-        } == dependencies
+        } == expected_dependencies
 
 
 def test_generate_multiple(generated_targets_rule_runner: RuleRunner) -> None:
@@ -1129,7 +1132,7 @@ def test_parametrize(generated_targets_rule_runner: RuleRunner) -> None:
                 residence_dir="demo",
             ),
         },
-        dependencies={
+        expected_dependencies={
             "demo@tags=t1": {"demo/f1.ext@tags=t1"},
             "demo@tags=t2": {"demo/f1.ext@tags=t2"},
             "demo/f1.ext@tags=t1": set(),
@@ -1200,7 +1203,7 @@ def test_parametrize_overrides(generated_targets_rule_runner: RuleRunner) -> Non
                 residence_dir="demo",
             ),
         },
-        dependencies={
+        expected_dependencies={
             "demo:demo": {
                 "demo/f1.ext@resolve=a",
                 "demo/f1.ext@resolve=b",
@@ -1231,14 +1234,14 @@ def test_parametrize_atom(generated_targets_rule_runner: RuleRunner) -> None:
                 residence_dir="demo",
             ),
         },
-        dependencies={
+        expected_dependencies={
             "demo@resolve=a": set(),
             "demo@resolve=b": set(),
         },
     )
 
 
-def test_parametrize_partial_atom(generated_targets_rule_runner: RuleRunner) -> None:
+def test_parametrize_partial_atom_to_atom(generated_targets_rule_runner: RuleRunner) -> None:
     assert_generated(
         generated_targets_rule_runner,
         Address("demo", target_name="t2"),
@@ -1258,27 +1261,7 @@ def test_parametrize_partial_atom(generated_targets_rule_runner: RuleRunner) ->
             """
         ),
         ["f1.ext", "f2.ext"],
-        {
-            MockGeneratedTarget(
-                {
-                    SingleSourceField.alias: "f2.ext",
-                    ResolveField.alias: "a",
-                    Dependencies.alias: [":t1"],
-                },
-                Address("demo", target_name="t2", parameters={"resolve": "a"}),
-                residence_dir="demo",
-            ),
-            MockGeneratedTarget(
-                {
-                    SingleSourceField.alias: "f2.ext",
-                    ResolveField.alias: "b",
-                    Dependencies.alias: [":t1"],
-                },
-                Address("demo", target_name="t2", parameters={"resolve": "b"}),
-                residence_dir="demo",
-            ),
-        },
-        dependencies={
+        expected_dependencies={
             "demo:t1@resolve=a": set(),
             "demo:t1@resolve=b": set(),
             "demo:t2@resolve=a": {"demo:t1@resolve=a"},
@@ -1287,7 +1270,9 @@ def test_parametrize_partial_atom(generated_targets_rule_runner: RuleRunner) ->
     )
 
 
-def test_parametrize_partial_generator(generated_targets_rule_runner: RuleRunner) -> None:
+def test_parametrize_partial_generator_to_generator(
+    generated_targets_rule_runner: RuleRunner,
+) -> None:
     assert_generated(
         generated_targets_rule_runner,
         Address("demo", target_name="t2"),
@@ -1307,37 +1292,7 @@ def test_parametrize_partial_generator(generated_targets_rule_runner: RuleRunner
             """
         ),
         ["f1.ext", "f2.ext"],
-        {
-            MockGeneratedTarget(
-                {
-                    SingleSourceField.alias: "f2.ext",
-                    ResolveField.alias: "a",
-                    Dependencies.alias: [":t1"],
-                },
-                Address(
-                    "demo",
-                    relative_file_path="f2.ext",
-                    target_name="t2",
-                    parameters={"resolve": "a"},
-                ),
-                residence_dir="demo",
-            ),
-            MockGeneratedTarget(
-                {
-                    SingleSourceField.alias: "f2.ext",
-                    ResolveField.alias: "b",
-                    Dependencies.alias: [":t1"],
-                },
-                Address(
-                    "demo",
-                    relative_file_path="f2.ext",
-                    target_name="t2",
-                    parameters={"resolve": "b"},
-                ),
-                residence_dir="demo",
-            ),
-        },
-        dependencies={
+        expected_dependencies={
             "demo/f1.ext:t1@resolve=a": set(),
             "demo/f1.ext:t1@resolve=b": set(),
             "demo/f2.ext:t2@resolve=a": {"demo:t1@resolve=a"},
@@ -1358,6 +1313,49 @@ def test_parametrize_partial_generator(generated_targets_rule_runner: RuleRunner
     )
 
 
+def test_parametrize_partial_generator_to_generated(
+    generated_targets_rule_runner: RuleRunner,
+) -> None:
+    assert_generated(
+        generated_targets_rule_runner,
+        Address("demo", target_name="t2"),
+        dedent(
+            """\
+            generator(
+              name='t1',
+              resolve=parametrize('a', 'b'),
+              sources=['f1.ext'],
+            )
+            generator(
+              name='t2',
+              resolve=parametrize('a', 'b'),
+              sources=['f2.ext'],
+              dependencies=['./f1.ext:t1'],
+            )
+            """
+        ),
+        ["f1.ext", "f2.ext"],
+        expected_dependencies={
+            "demo/f1.ext:t1@resolve=a": set(),
+            "demo/f1.ext:t1@resolve=b": set(),
+            "demo/f2.ext:t2@resolve=a": {"demo/f1.ext:t1@resolve=a"},
+            "demo/f2.ext:t2@resolve=b": {"demo/f1.ext:t1@resolve=b"},
+            "demo:t1@resolve=a": {
+                "demo/f1.ext:t1@resolve=a",
+            },
+            "demo:t1@resolve=b": {
+                "demo/f1.ext:t1@resolve=b",
+            },
+            "demo:t2@resolve=a": {
+                "demo/f2.ext:t2@resolve=a",
+            },
+            "demo:t2@resolve=b": {
+                "demo/f2.ext:t2@resolve=b",
+            },
+        },
+    )
+
+
 # -----------------------------------------------------------------------------------------------
 # Test FieldSets. Also see `engine/target_test.py`.
 # -----------------------------------------------------------------------------------------------