Repository: pantsbuild/pants

Issue:
`pex_binary` depending on `python_distribution` is consuming transitive deps
**Describe the bug**
The following repo can be used to reproduce the issue: https://github.com/dimitar-petrov/pants_merge_debug
Included is a docker-compose.yml file to be able to set up the environment independently of OS used.

```
./pants package py/apps/pyapp:venv
14:27:39.91 [ERROR] 1 Exception encountered:

  Exception: Can only merge Directories with no duplicates, but found 2 duplicate entries in :

`MANIFEST.in`: 1.) file digest=64958bd91c39a06f78542a71383436bbf0c798138e4fe46780c8d9c08aa91278 size=179:

include README.org LICENSE VERSION 
include pybind/cnufft/cnufft.c
include pybind/cnufft/cnufft.h
include pybind/cnufft/libcnufft.so
include pybind/cnufft/libfnufft.so
prune *.pyc

`MANIFEST.in`: 2.) file digest=892591bf1e4c7e1337cfd470096a467805fa2cb6f46b792fe6e3a6a8b0a72e0a size=366:

include README.rst
include setup.py
include nufft/__init__.py
include nufft/nufft.py
include nufft/nufft.pyf
include src/nufft/dfftpack.f
include src/nufft/dirft1d.f
include src/nufft/dirft2d.f
include src/nufft/dirft3d.f
include src/nufft/next235.f
include src/nufft/nufft1df90.f
include src/nufft/nufft2df90.f
include src/nufft/nufft3df90.f
include pyproject.toml
```

**Pants version**
pants_version = "2.12.0rc1"

**OS**
Linux

Issue opened after discussion on slack with: @benjyw, @kaos, @thejcannon 
`pex_binary` depending on `python_distribution` is consuming transitive deps
**Describe the bug**
The following repo can be used to reproduce the issue: https://github.com/dimitar-petrov/pants_merge_debug
Included is a docker-compose.yml file to be able to set up the environment independently of OS used.

```
./pants package py/apps/pyapp:venv
14:27:39.91 [ERROR] 1 Exception encountered:

  Exception: Can only merge Directories with no duplicates, but found 2 duplicate entries in :

`MANIFEST.in`: 1.) file digest=64958bd91c39a06f78542a71383436bbf0c798138e4fe46780c8d9c08aa91278 size=179:

include README.org LICENSE VERSION 
include pybind/cnufft/cnufft.c
include pybind/cnufft/cnufft.h
include pybind/cnufft/libcnufft.so
include pybind/cnufft/libfnufft.so
prune *.pyc

`MANIFEST.in`: 2.) file digest=892591bf1e4c7e1337cfd470096a467805fa2cb6f46b792fe6e3a6a8b0a72e0a size=366:

include README.rst
include setup.py
include nufft/__init__.py
include nufft/nufft.py
include nufft/nufft.pyf
include src/nufft/dfftpack.f
include src/nufft/dirft1d.f
include src/nufft/dirft2d.f
include src/nufft/dirft3d.f
include src/nufft/next235.f
include src/nufft/nufft1df90.f
include src/nufft/nufft2df90.f
include src/nufft/nufft3df90.f
include pyproject.toml
```

**Pants version**
pants_version = "2.12.0rc1"

**OS**
Linux

Issue opened after discussion on slack with: @benjyw, @kaos, @thejcannon 


Discussion:
FYI changed the title to reflect the bug
The workaround is to use transitive dep excludes, e.g., 

```
pex_binary(
    name="venv",
    dependencies=[
        "py/libs/nufft:nufft_dist",
        "py/libs/pybind:pybind_dist",
        "!!py/libs/nufft/MANIFEST.in:nufft_files",
        "!!py/libs/nufft/README.rst:nufft_files",
        "!!py/libs/nufft/pyproject.toml:pyproject",
        "!!py/libs/nufft/setup.py:nufft0",
        "!!py/libs/pybind/MANIFEST.in:pybind_files",
        "!!py/libs/pybind/README.rst:pybind_files",
        "!!py/libs/pybind/pyproject.toml:pyproject",
        "!!py/libs/pybind/setup.py:pybind0",
    ]
)
```
(technically you only need to exclude n-1 for an n-way collision, but might as well be symmetric)

Obviously this is very unsatisying, but will work until we can fix more effectively.
FYI changed the title to reflect the bug
The workaround is to use transitive dep excludes, e.g., 

```
pex_binary(
    name="venv",
    dependencies=[
        "py/libs/nufft:nufft_dist",
        "py/libs/pybind:pybind_dist",
        "!!py/libs/nufft/MANIFEST.in:nufft_files",
        "!!py/libs/nufft/README.rst:nufft_files",
        "!!py/libs/nufft/pyproject.toml:pyproject",
        "!!py/libs/nufft/setup.py:nufft0",
        "!!py/libs/pybind/MANIFEST.in:pybind_files",
        "!!py/libs/pybind/README.rst:pybind_files",
        "!!py/libs/pybind/pyproject.toml:pyproject",
        "!!py/libs/pybind/setup.py:pybind0",
    ]
)
```
(technically you only need to exclude n-1 for an n-way collision, but might as well be symmetric)

Obviously this is very unsatisying, but will work until we can fix more effectively.

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_from_targets.py b/src/python/pants/backend/python/util_rules/pex_from_targets.py
--- a/src/python/pants/backend/python/util_rules/pex_from_targets.py
+++ b/src/python/pants/backend/python/util_rules/pex_from_targets.py
@@ -46,6 +46,7 @@
 )
 from pants.backend.python.util_rules.python_sources import rules as python_sources_rules
 from pants.core.goals.generate_lockfiles import NoCompatibleResolveException
+from pants.core.goals.package import TraverseIfNotPackageTarget
 from pants.core.target_types import FileSourceField
 from pants.engine.addresses import Address, Addresses
 from pants.engine.collection import DeduplicatedCollection
@@ -538,7 +539,14 @@ async def create_pex_from_targets(
         sources_digests.append(request.additional_sources)
     if request.include_source_files:
         transitive_targets = await Get(
-            TransitiveTargets, TransitiveTargetsRequest(request.addresses)
+            TransitiveTargets,
+            TransitiveTargetsRequest(
+                request.addresses,
+                should_traverse_deps_predicate=TraverseIfNotPackageTarget(
+                    roots=request.addresses,
+                    union_membership=union_membership,
+                ),
+            ),
         )
         sources = await Get(PythonSourceFiles, PythonSourceFilesRequest(transitive_targets.closure))
 


Test patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_from_targets_test.py b/src/python/pants/backend/python/util_rules/pex_from_targets_test.py
--- a/src/python/pants/backend/python/util_rules/pex_from_targets_test.py
+++ b/src/python/pants/backend/python/util_rules/pex_from_targets_test.py
@@ -16,10 +16,12 @@
 import pytest
 
 from pants.backend.python import target_types_rules
+from pants.backend.python.goals import package_pex_binary
 from pants.backend.python.subsystems import setuptools
 from pants.backend.python.subsystems.setup import PythonSetup
 from pants.backend.python.target_types import (
     EntryPoint,
+    PexBinary,
     PexLayout,
     PythonRequirementTarget,
     PythonSourcesGeneratorTarget,
@@ -70,6 +72,7 @@
 def rule_runner() -> PythonRuleRunner:
     return PythonRuleRunner(
         rules=[
+            *package_pex_binary.rules(),
             *pex_test_utils.rules(),
             *pex_from_targets.rules(),
             *target_types_rules.rules(),
@@ -80,6 +83,7 @@ def rule_runner() -> PythonRuleRunner:
             *setuptools.rules(),
         ],
         target_types=[
+            PexBinary,
             PythonSourcesGeneratorTarget,
             PythonRequirementTarget,
             PythonSourceTarget,
@@ -759,6 +763,52 @@ def test_exclude_sources(include_sources: bool, rule_runner: PythonRuleRunner) -
     assert len(snapshot.files) == (1 if include_sources else 0)
 
 
+def test_include_sources_without_transitive_package_sources(rule_runner: PythonRuleRunner) -> None:
+    rule_runner.write_files(
+        {
+            "src/app/BUILD": dedent(
+                """
+                python_sources(
+                    name="app",
+                    sources=["app.py"],
+                    dependencies=["//src/dep:pkg"],
+                )
+                """
+            ),
+            "src/app/app.py": "",
+            "src/dep/BUILD": dedent(
+                # This test requires a package that has a standard dependencies field.
+                # 'pex_binary' has a dependencies field; 'archive' does not.
+                """
+                pex_binary(name="pkg", dependencies=[":dep"])
+                python_sources(name="dep", sources=["dep.py"])
+                """
+            ),
+            "src/dep/dep.py": "",
+        }
+    )
+
+    rule_runner.set_options(
+        [
+            "--backend-packages=pants.backend.python",
+            "--python-repos-indexes=[]",
+        ],
+        env_inherit={"PATH"},
+    )
+
+    request = PexFromTargetsRequest(
+        [Address("src/app", target_name="app")],
+        output_filename="demo.pex",
+        internal_only=True,
+        include_source_files=True,
+    )
+    pex_request = rule_runner.request(PexRequest, [request])
+    snapshot = rule_runner.request(Snapshot, [pex_request.sources])
+
+    # the packaged transitive dep is excluded
+    assert snapshot.files == ("app/app.py",)
+
+
 @pytest.mark.parametrize("enable_resolves", [False, True])
 def test_cross_platform_pex_disables_subsetting(
     rule_runner: PythonRuleRunner, enable_resolves: bool