Repository: pantsbuild/pants

Issue:
Apply parameter expansion for dependency ignores
We only apply for includes:

https://github.com/pantsbuild/pants/blob/ae9e2c866cf0b41c2de1730046afa3e0021daf1a/src/python/pants/engine/internals/graph.py#L1110-L1132

But maybe we should also apply for ignores:

https://github.com/pantsbuild/pants/blob/ae9e2c866cf0b41c2de1730046afa3e0021daf1a/src/python/pants/engine/internals/graph.py#L1165-L1167

A user is getting issues excluding the target generator `dir:foo` because it parametrized to `dir:foo@resolve=a` and `dir:foo@resolve=b`, along with all of its generated targets like `dir:foo#gen@resolve=a`. 
Pants crashes with error related to parameterization on latest 2.13.x
**Describe the bug**

I upgraded to 757a2664cb82fac6693f13609ce14911d8fd88e3 to pull in the `pex` upgrade. Most CI runs on the PR failed with:
```
20:03:57.76 [INFO] Starting: Resolving plugins: toolchain.pants.plugin==0.20.0
20:04:11.50 [INFO] Completed: Resolving plugins: toolchain.pants.plugin==0.20.0
20:04:12.12 [INFO] [auth-plugin] Successfully loaded Toolchain token from env var 'TOOLCHAIN_AUTH_TOKEN', expiration: 2022-12-20T20:52:09+00:00.
Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 115, in main
    PantsLoader.main()
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 111, in main
    cls.run_default_entrypoint()
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/bin/pants_runner.py", line 99, in run
    runner = LocalPantsRunner.create(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 159, in create
    specs = calculate_specs(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/init/specs_calculator.py", line 94, in calculate_specs
    (changed_addresses,) = session.product_request(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 577, in product_request
    return self.execute(request)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 498, in _raise_on_error
    raise ExecutionError(
Exception message: 1 Exception encountered:
Engine traceback:
  in select
  in pants.vcs.changed.find_changed_owners
  in pants.backend.project_info.dependees.find_dependees
  in pants.backend.project_info.dependees.map_addresses_to_dependees
  in pants.engine.internals.graph.resolve_dependencies (src/projects/rhp/backend:celery)
Traceback (most recent call last):
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/engine/internals/selectors.py", line 653, in native_engine_generator_send
    res = func.send(arg)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/engine/internals/graph.py", line 1146, in resolve_dependencies
    explicitly_provided_includes = [
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/engine/internals/graph.py", line 1147, in <listcomp>
    parametrizations.get_subset(address, tgt, field_defaults).address
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+git757a2664_py38/lib/python3.8/site-packages/pants/engine/internals/parametrize.py", line 271, in get_subset
    raise ValueError(
ValueError: The explicit dependency `src/projects/rhp/backend/celery.py` of the target at `src/projects/rhp/backend:celery` does not provide enough address parameters to identify which parametrization of the dependency target should be used.
Target `src/projects/rhp/backend:backend` can be addressed as:
  * src/projects/rhp/backend:backend
NoneType: None
FATAL: exception not rethrown
FATAL: exception not rethrown
```

The code in question is laid out like:
```
src/projects/rhp/backend/
  celery.py
  other.py
  BUILD.pants
```
With targets:
```
python_sources()

pex_binary(
    name="celery",
    entry_point="celery.bin.celery",
    dependencies=[
        "./celery.py",
    ],
    restartable=True,
)
```

The crash happened when running the `test`, `check`, and `lint --skip-formatters` but not when running `tailor`, `update-build-files --check`, or `lint --only=black --only=isort`. I haven't been able to reproduce it locally.

Note that we don't `parameterize` any of our targets so I'm not sure why Pants is trying to figure out which parameterization to use.

**Pants version**

757a2664cb82fac6693f13609ce14911d8fd88e3

**OS**

Linux



Discussion:

Hm, I don't reproduce this in simple cases, such as:
```diff
diff --git a/testprojects/src/python/print_env/BUILD b/testprojects/src/python/print_env/BUILD
index f6018298ff..35d53f8e41 100644
--- a/testprojects/src/python/print_env/BUILD
+++ b/testprojects/src/python/print_env/BUILD
@@ -1,6 +1,6 @@
 # Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
-pex_binary(name="binary", entry_point="print_env.main", dependencies=[":lib"])
+pex_binary(name="binary", entry_point="print_env.main", dependencies=["./main.py"])
 
-python_sources(name="lib")
+python_sources()
```

...which successfully runs `./pants dependencies testprojects/src/python/print_env:binary` and `--changed-dependees=transitive`.
I believe that we have enough info in [the Slack thread](https://pantsbuild.slack.com/archives/C0D7TNJHL/p1658271672639599?thread_ts=1658263758.236709&cid=C0D7TNJHL) to reproduce this one: I'll take a stab tomorrow.

Thanks again!

Resolving patch:
diff --git a/src/python/pants/engine/internals/graph.py b/src/python/pants/engine/internals/graph.py
--- a/src/python/pants/engine/internals/graph.py
+++ b/src/python/pants/engine/internals/graph.py
@@ -9,7 +9,7 @@
 import os.path
 from dataclasses import dataclass
 from pathlib import PurePath
-from typing import Iterable, NamedTuple, Sequence, cast
+from typing import Iterable, Iterator, NamedTuple, Sequence, cast
 
 from pants.base.deprecated import resolve_conflicting_options, warn_or_error
 from pants.base.specs import AncestorGlobSpec, RawSpecsWithoutFileOwners, RecursiveGlobSpec
@@ -33,7 +33,7 @@
     _TargetParametrizationsRequest as _TargetParametrizationsRequest,
 )
 from pants.engine.internals.target_adaptor import TargetAdaptor, TargetAdaptorRequest
-from pants.engine.rules import Get, MultiGet, collect_rules, rule
+from pants.engine.rules import Get, MultiGet, collect_rules, rule, rule_helper
 from pants.engine.target import (
     AllTargets,
     AllTargetsRequest,
@@ -340,13 +340,7 @@ async def resolve_target(
             base_address, description_of_origin=request.description_of_origin
         ),
     )
-    if address.is_generated_target:
-        # TODO: This is an accommodation to allow using file/generator Addresses for
-        # non-generator atom targets. See https://github.com/pantsbuild/pants/issues/14419.
-        original_target = parametrizations.get(base_address)
-        if original_target and not target_types_to_generate_requests.is_generator(original_target):
-            return WrappedTarget(original_target)
-    target = parametrizations.get(address)
+    target = parametrizations.get(address, target_types_to_generate_requests)
     if target is None:
         raise ResolveError(
             softwrap(
@@ -1067,6 +1061,35 @@ async def determine_explicitly_provided_dependencies(
     )
 
 
+@rule_helper
+async def _fill_parameters(
+    field_alias: str,
+    consumer_tgt: Target,
+    addresses: Iterable[Address],
+    target_types_to_generate_requests: TargetTypesToGenerateTargetsRequests,
+    field_defaults: FieldDefaults,
+) -> tuple[Address, ...]:
+    assert not isinstance(addresses, Iterator)
+
+    parametrizations = await MultiGet(
+        Get(
+            _TargetParametrizations,
+            _TargetParametrizationsRequest(
+                address.maybe_convert_to_target_generator(),
+                description_of_origin=f"the `{field_alias}` field of the target {consumer_tgt.address}",
+            ),
+        )
+        for address in addresses
+    )
+
+    return tuple(
+        parametrizations.get_subset(
+            address, consumer_tgt, field_defaults, target_types_to_generate_requests
+        ).address
+        for address, parametrizations in zip(addresses, parametrizations)
+    )
+
+
 @rule(desc="Resolve direct dependencies")
 async def resolve_dependencies(
     request: DependenciesRequest,
@@ -1130,25 +1153,24 @@ async def resolve_dependencies(
     # parameters. If so, fill them in.
     explicitly_provided_includes: Iterable[Address] = explicitly_provided.includes
     if explicitly_provided_includes:
-        explicit_dependency_parametrizations = await MultiGet(
-            Get(
-                _TargetParametrizations,
-                _TargetParametrizationsRequest(
-                    address.maybe_convert_to_target_generator(),
-                    description_of_origin=(
-                        f"the `{request.field.alias}` field of the target {tgt.address}"
-                    ),
-                ),
-            )
-            for address in explicitly_provided_includes
+        explicitly_provided_includes = await _fill_parameters(
+            request.field.alias,
+            tgt,
+            explicitly_provided_includes,
+            target_types_to_generate_requests,
+            field_defaults,
         )
-
-        explicitly_provided_includes = [
-            parametrizations.get_subset(address, tgt, field_defaults).address
-            for address, parametrizations in zip(
-                explicitly_provided_includes, explicit_dependency_parametrizations
+    explicitly_provided_ignores: FrozenOrderedSet[Address] = explicitly_provided.ignores
+    if explicitly_provided_ignores:
+        explicitly_provided_ignores = FrozenOrderedSet(
+            await _fill_parameters(
+                request.field.alias,
+                tgt,
+                tuple(explicitly_provided_ignores),
+                target_types_to_generate_requests,
+                field_defaults,
             )
-        ]
+        )
 
     # If the target has `SpecialCasedDependencies`, such as the `archive` target having
     # `files` and `packages` fields, then we possibly include those too. We don't want to always
@@ -1192,7 +1214,7 @@ async def resolve_dependencies(
                     *itertools.chain.from_iterable(inferred),
                     *special_cased,
                 )
-                if addr not in explicitly_provided.ignores
+                if addr not in explicitly_provided_ignores
             }
         )
     )
diff --git a/src/python/pants/engine/internals/parametrize.py b/src/python/pants/engine/internals/parametrize.py
--- a/src/python/pants/engine/internals/parametrize.py
+++ b/src/python/pants/engine/internals/parametrize.py
@@ -12,7 +12,7 @@
 from pants.engine.addresses import Address
 from pants.engine.collection import Collection
 from pants.engine.engine_aware import EngineAwareParameter
-from pants.engine.target import Field, FieldDefaults, Target
+from pants.engine.target import Field, FieldDefaults, Target, TargetTypesToGenerateTargetsRequests
 from pants.util.frozendict import FrozenDict
 from pants.util.meta import frozen_after_init
 from pants.util.strutil import bullet_list, softwrap
@@ -193,12 +193,27 @@ def generated_for(self, address: Address) -> FrozenDict[Address, Target]:
 
         raise self._bare_address_error(address)
 
-    def get(self, address: Address) -> Target | None:
+    def get(
+        self,
+        address: Address,
+        target_types_to_generate_requests: TargetTypesToGenerateTargetsRequests | None = None,
+    ) -> Target | None:
         """Find the Target with an exact Address match, if any."""
         for parametrization in self:
             instance = parametrization.get(address)
             if instance is not None:
                 return instance
+
+        # TODO: This is an accommodation to allow using file/generator Addresses for
+        # non-generator atom targets. See https://github.com/pantsbuild/pants/issues/14419.
+        if target_types_to_generate_requests and address.is_generated_target:
+            base_address = address.maybe_convert_to_target_generator()
+            original_target = self.get(base_address, target_types_to_generate_requests)
+            if original_target and not target_types_to_generate_requests.is_generator(
+                original_target
+            ):
+                return original_target
+
         return None
 
     def get_all_superset_targets(self, address: Address) -> Iterator[Address]:
@@ -225,11 +240,15 @@ def get_all_superset_targets(self, address: Address) -> Iterator[Address]:
                     yield parametrized_tgt.address
 
     def get_subset(
-        self, address: Address, consumer: Target, field_defaults: FieldDefaults
+        self,
+        address: Address,
+        consumer: Target,
+        field_defaults: FieldDefaults,
+        target_types_to_generate_requests: TargetTypesToGenerateTargetsRequests,
     ) -> Target:
         """Find the Target with the given Address, or with fields matching the given consumer."""
         # Check for exact matches.
-        instance = self.get(address)
+        instance = self.get(address, target_types_to_generate_requests)
         if instance is not None:
             return instance
 


Test patch:
diff --git a/src/python/pants/engine/internals/graph_test.py b/src/python/pants/engine/internals/graph_test.py
--- a/src/python/pants/engine/internals/graph_test.py
+++ b/src/python/pants/engine/internals/graph_test.py
@@ -1227,6 +1227,55 @@ def test_parametrize_partial_generator_to_generated(
     )
 
 
+def test_parametrize_partial_exclude(generated_targets_rule_runner: RuleRunner) -> None:
+    assert_generated(
+        generated_targets_rule_runner,
+        Address("demo", target_name="t2"),
+        dedent(
+            """\
+            generator(
+              name='t1',
+              resolve=parametrize('a', 'b'),
+              sources=['f1.ext', 'f2.ext'],
+            )
+            generator(
+              name='t2',
+              resolve=parametrize('a', 'b'),
+              sources=['f3.ext'],
+              dependencies=[
+                './f1.ext:t1',
+                './f2.ext:t1',
+                '!./f2.ext:t1',
+              ],
+            )
+            """
+        ),
+        ["f1.ext", "f2.ext", "f3.ext"],
+        expected_dependencies={
+            "demo/f1.ext:t1@resolve=a": set(),
+            "demo/f2.ext:t1@resolve=a": set(),
+            "demo/f1.ext:t1@resolve=b": set(),
+            "demo/f2.ext:t1@resolve=b": set(),
+            "demo/f3.ext:t2@resolve=a": {"demo/f1.ext:t1@resolve=a"},
+            "demo/f3.ext:t2@resolve=b": {"demo/f1.ext:t1@resolve=b"},
+            "demo:t1@resolve=a": {
+                "demo/f1.ext:t1@resolve=a",
+                "demo/f2.ext:t1@resolve=a",
+            },
+            "demo:t1@resolve=b": {
+                "demo/f1.ext:t1@resolve=b",
+                "demo/f2.ext:t1@resolve=b",
+            },
+            "demo:t2@resolve=a": {
+                "demo/f3.ext:t2@resolve=a",
+            },
+            "demo:t2@resolve=b": {
+                "demo/f3.ext:t2@resolve=b",
+            },
+        },
+    )
+
+
 def test_parametrize_16190(generated_targets_rule_runner: RuleRunner) -> None:
     class ParentField(Field):
         alias = "parent"
@@ -1749,7 +1798,7 @@ def dependencies_rule_runner() -> RuleRunner:
             UnionRule(InjectDependenciesRequest, InjectCustomSmalltalkDependencies),
             UnionRule(InferDependenciesRequest, InferSmalltalkDependencies),
         ],
-        target_types=[SmalltalkLibraryGenerator, MockTarget],
+        target_types=[SmalltalkLibrary, SmalltalkLibraryGenerator, MockTarget],
     )
 
 
@@ -1880,6 +1929,7 @@ def test_dependency_injection(dependencies_rule_runner: RuleRunner) -> None:
                 """\
                 smalltalk_libraries(name='target', sources=['*.st'])
                 target(name='provided')
+                target(name='injected2')
                 """
             ),
         }
@@ -2014,6 +2064,35 @@ def test_depends_on_generated_targets(dependencies_rule_runner: RuleRunner) -> N
     )
 
 
+def test_depends_on_atom_via_14419(dependencies_rule_runner: RuleRunner) -> None:
+    """See #14419."""
+    dependencies_rule_runner.write_files(
+        {
+            "src/smalltalk/f1.st": "",
+            "src/smalltalk/f2.st": "",
+            "src/smalltalk/BUILD": dedent(
+                """\
+                smalltalk_library(source='f1.st')
+                smalltalk_library(
+                    name='t2',
+                    source='f2.st',
+                    dependencies=['./f1.st'],
+                )
+                """
+            ),
+        }
+    )
+    # Due to the accommodation for #14419, the file address `./f1.st` resolves to the atom target
+    # with the default name.
+    assert_dependencies_resolved(
+        dependencies_rule_runner,
+        Address("src/smalltalk", target_name="t2"),
+        expected=[
+            Address("src/smalltalk"),
+        ],
+    )
+
+
 def test_resolve_unparsed_address_inputs() -> None:
     rule_runner = RuleRunner(
         rules=[QueryRule(Addresses, [UnparsedAddressInputs])], target_types=[MockTarget]