Repository: pantsbuild/pants

Issue:
Fix transitive target excludes with `python_distribution`
## Context

### setup.py and python_distribution

See https://www.pantsbuild.org/docs/python-distributions. Try it out, such as running `./pants package src/python/pants:pants-packaged`. This will create a `.whl` file, but it will be more useful when iterating to remove the line `setup_py_commands=` in `src/python/pants/BUILD` so that Pants simply generates a loose chroot that you can inspect (https://www.pantsbuild.org/docs/python-distributions#generating-a-setuppy-file).

The `install_requires` part of the generated `setup.py` file is particularly interesting for this case.

### Transitive excludes

See the tooltip at https://www.pantsbuild.org/docs/targets#dependencies-and-dependency-inference for what these are why they're used. 

When building a distribution, it can helpful to ignore certain problematic dependencies. For example, Toolchain's Pants plugin technically depends on `pantsbuild.pants`, but this is causing issues and we want to be able to ignore that dependency.

A kind of confusing point with transitive excludes: they will only ignore the thing explicitly excluded and the dependencies of the excluded thing will still show up. For example, `b` depends on `a`. `c` depends on `b`, so `c` transitively depends both on `a` and `b`. Now, if we tell Pants to transitively exclude `!!b`, `a` will still show up - we only end up ignoring `b` and we still bring in `b`'s own dependencies. This makes the code you're working with simpler, but it's a little confusing.

FYI this is the code for transitive targets:

https://github.com/pantsbuild/pants/blob/2f298b2471633d31e3c396eec274b7e97e474673/src/python/pants/engine/internals/graph.py#L241-L301

## The bug

Right now, transitive excludes are not working when generating `setup.py`. See https://github.com/pantsbuild/pants/pull/11470 for a test @asherf wrote to demonstrate this - you will want to use this test in your PR and make sure that test becomes green to prove this is fixed.

You can also reproduce this manually. Add `"!!3rdparty/python:pex",` to the `dependencies` for `:pants-packaged`, then run `./pants package` and inspect the generated `setup.py` file. You will see that `pex` still shows up in `pex_binary`.

However, the exclude _is_ working when you run `./pants dependencies --transitive src/python/pants:pants-packaged`. So we can infer this is an issue with the code in https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/python/goals/setup_py.py, rather than the transitive targets rule in `graph.py`.

## Investigating what's going on

It's not clear why this is happening. Here is the code where we set `install_requires`:

https://github.com/pantsbuild/pants/blob/2f298b2471633d31e3c396eec274b7e97e474673/src/python/pants/backend/python/goals/setup_py.py#L503-L514

You'll want to work backwards to see why the `requirements` variable is still including the transitive exclude. It will be helpful to add logging statements along the way while running `./pants package`: https://www.pantsbuild.org/docs/rules-api-logging#adding-logging

Once you find the place where it's happening, we can figure out the cleanest way to get transitive excludes to work here.


Discussion:
@Eric-Arellano added a PR with a test that AFAICT repos the issue .
https://github.com/pantsbuild/pants/pull/11470

<img width="748" alt="Screen Shot 2021-01-15 at 1 31 26 PM" src="https://user-images.githubusercontent.com/1268088/104780704-06457d80-5736-11eb-9ea9-c95202c73de6.png">

https://travis-ci.com/github/pantsbuild/pants/jobs/472259333#L707

Resolving patch:
diff --git a/src/python/pants/backend/python/goals/setup_py.py b/src/python/pants/backend/python/goals/setup_py.py
--- a/src/python/pants/backend/python/goals/setup_py.py
+++ b/src/python/pants/backend/python/goals/setup_py.py
@@ -70,6 +70,7 @@
 from pants.util.logging import LogLevel
 from pants.util.memo import memoized_property
 from pants.util.meta import frozen_after_init
+from pants.util.ordered_set import FrozenOrderedSet
 from pants.util.strutil import ensure_text
 
 logger = logging.getLogger(__name__)
@@ -477,9 +478,11 @@ async def generate_chroot(request: SetupPyChrootRequest) -> SetupPyChroot:
 
     owned_deps, transitive_targets = await MultiGet(
         Get(OwnedDependencies, DependencyOwner(exported_target)),
-        Get(TransitiveTargets, TransitiveTargetsRequest([exported_target.target.address])),
+        Get(
+            TransitiveTargets,
+            TransitiveTargetsRequest([exported_target.target.address]),
+        ),
     )
-
     # files() targets aren't owned by a single exported target - they aren't code, so
     # we allow them to be in multiple dists. This is helpful for, e.g., embedding
     # a standard license file in a dist.
@@ -490,7 +493,6 @@ async def generate_chroot(request: SetupPyChrootRequest) -> SetupPyChroot:
         Get(SetupPySources, SetupPySourcesRequest(targets, py2=request.py2)),
         Get(ExportedTargetRequirements, DependencyOwner(exported_target)),
     )
-
     # Generate the kwargs for the setup() call. In addition to using the kwargs that are either
     # explicitly provided or generated via a user's plugin, we add additional kwargs based on the
     # resolved requirements and sources.
@@ -617,9 +619,9 @@ async def get_requirements(
     setup_py_generation: SetupPyGeneration,
 ) -> ExportedTargetRequirements:
     transitive_targets = await Get(
-        TransitiveTargets, TransitiveTargetsRequest([dep_owner.exported_target.target.address])
+        TransitiveTargets,
+        TransitiveTargetsRequest([dep_owner.exported_target.target.address]),
     )
-
     ownable_tgts = [
         tgt for tgt in transitive_targets.closure if is_ownable_target(tgt, union_membership)
     ]
@@ -639,9 +641,25 @@ async def get_requirements(
     direct_deps_tgts = await MultiGet(
         Get(Targets, DependenciesRequest(tgt.get(Dependencies))) for tgt in owned_by_us
     )
+
+    transitive_excludes: FrozenOrderedSet[Target] = FrozenOrderedSet()
+    uneval_trans_excl = [
+        tgt.get(Dependencies).unevaluated_transitive_excludes for tgt in transitive_targets.closure
+    ]
+    if uneval_trans_excl:
+        nested_trans_excl = await MultiGet(
+            Get(Targets, UnparsedAddressInputs, unparsed) for unparsed in uneval_trans_excl
+        )
+        transitive_excludes = FrozenOrderedSet(
+            itertools.chain.from_iterable(excludes for excludes in nested_trans_excl)
+        )
+
+    direct_deps_chained = FrozenOrderedSet(itertools.chain.from_iterable(direct_deps_tgts))
+    direct_deps_with_excl = direct_deps_chained.difference(transitive_excludes)
+
     reqs = PexRequirements.create_from_requirement_fields(
         tgt[PythonRequirementsField]
-        for tgt in itertools.chain.from_iterable(direct_deps_tgts)
+        for tgt in direct_deps_with_excl
         if tgt.has_field(PythonRequirementsField)
     )
     req_strs = list(reqs)
@@ -654,7 +672,6 @@ async def get_requirements(
         f"{kwargs.name}{setup_py_generation.first_party_dependency_version(kwargs.version)}"
         for kwargs in set(kwargs_for_exported_targets_we_depend_on)
     )
-
     return ExportedTargetRequirements(req_strs)
 
 


Test patch:
diff --git a/src/python/pants/backend/python/goals/setup_py_test.py b/src/python/pants/backend/python/goals/setup_py_test.py
--- a/src/python/pants/backend/python/goals/setup_py_test.py
+++ b/src/python/pants/backend/python/goals/setup_py_test.py
@@ -496,42 +496,111 @@ def test_get_requirements() -> None:
         ),
     )
 
-    def assert_requirements(
-        expected_req_strs,
-        addr: Address,
-        *,
-        version_scheme: FirstPartyDependencyVersionScheme = FirstPartyDependencyVersionScheme.EXACT,
-    ):
-        rule_runner.set_options(
-            [f"--setup-py-generation-first-party-dependency-version-scheme={version_scheme.value}"],
-            env_inherit={"PATH", "PYENV_ROOT", "HOME"},
-        )
-        tgt = rule_runner.get_target(addr)
-        reqs = rule_runner.request(
-            ExportedTargetRequirements,
-            [DependencyOwner(ExportedTarget(tgt))],
-        )
-        assert sorted(expected_req_strs) == list(reqs)
-
     assert_requirements(
-        ["ext1==1.22.333", "ext2==4.5.6"], Address("src/python/foo/bar", target_name="bar-dist")
+        rule_runner,
+        ["ext1==1.22.333", "ext2==4.5.6"],
+        Address("src/python/foo/bar", target_name="bar-dist"),
     )
     assert_requirements(
-        ["ext3==0.0.1", "bar==9.8.7"], Address("src/python/foo/corge", target_name="corge-dist")
+        rule_runner,
+        ["ext3==0.0.1", "bar==9.8.7"],
+        Address("src/python/foo/corge", target_name="corge-dist"),
     )
 
     assert_requirements(
+        rule_runner,
         ["ext3==0.0.1", "bar~=9.8.7"],
         Address("src/python/foo/corge", target_name="corge-dist"),
         version_scheme=FirstPartyDependencyVersionScheme.COMPATIBLE,
     )
     assert_requirements(
+        rule_runner,
         ["ext3==0.0.1", "bar"],
         Address("src/python/foo/corge", target_name="corge-dist"),
         version_scheme=FirstPartyDependencyVersionScheme.ANY,
     )
 
 
+def test_get_requirements_with_exclude() -> None:
+    rule_runner = create_setup_py_rule_runner(
+        rules=[
+            determine_setup_kwargs,
+            get_requirements,
+            get_owned_dependencies,
+            get_exporting_owner,
+            SubsystemRule(SetupPyGeneration),
+            QueryRule(ExportedTargetRequirements, (DependencyOwner,)),
+        ]
+    )
+    rule_runner.add_to_build_file(
+        "3rdparty",
+        textwrap.dedent(
+            """
+            python_requirement_library(
+                name='ext1',
+                requirements=['ext1==1.22.333'],
+            )
+            python_requirement_library(
+                name='ext2',
+                requirements=['ext2==4.5.6'],
+            )
+            python_requirement_library(
+                name='ext3',
+                requirements=['ext3==0.0.1'],
+            )
+            """
+        ),
+    )
+    rule_runner.add_to_build_file(
+        "src/python/foo/bar/baz",
+        "python_library(dependencies=['3rdparty:ext1'], sources=[])",
+    )
+    rule_runner.add_to_build_file(
+        "src/python/foo/bar/qux",
+        "python_library(dependencies=['3rdparty:ext2', 'src/python/foo/bar/baz'], sources=[])",
+    )
+    rule_runner.add_to_build_file(
+        "src/python/foo/bar",
+        textwrap.dedent(
+            """
+            python_distribution(
+                name='bar-dist',
+                dependencies=['!!3rdparty:ext2',':bar'],
+                provides=setup_py(name='bar', version='9.8.7'),
+            )
+
+            python_library(
+                sources=[],
+                dependencies=['src/python/foo/bar/baz', 'src/python/foo/bar/qux'],
+            )
+          """
+        ),
+    )
+
+    assert_requirements(
+        rule_runner, ["ext1==1.22.333"], Address("src/python/foo/bar", target_name="bar-dist")
+    )
+
+
+def assert_requirements(
+    rule_runner,
+    expected_req_strs,
+    addr: Address,
+    *,
+    version_scheme: FirstPartyDependencyVersionScheme = FirstPartyDependencyVersionScheme.EXACT,
+):
+    rule_runner.set_options(
+        [f"--setup-py-generation-first-party-dependency-version-scheme={version_scheme.value}"],
+        env_inherit={"PATH", "PYENV_ROOT", "HOME"},
+    )
+    tgt = rule_runner.get_target(addr)
+    reqs = rule_runner.request(
+        ExportedTargetRequirements,
+        [DependencyOwner(ExportedTarget(tgt))],
+    )
+    assert sorted(expected_req_strs) == list(reqs)
+
+
 def test_owned_dependencies() -> None:
     rule_runner = create_setup_py_rule_runner(
         rules=[