Repository: pantsbuild/pants

Issue:
Issue with package local dist into pex
**Describe the bug**
When I attempt to build a pex that depends on a local dist, it fails when PEX is creating a symlink (see logs below).
( Using this feature: #13306 )

Verified with a simpler case in the example repo that what I try to do works.

**Pants version**
2.8.0rc5 (also observed on latest `main`)

**OS**
MacOS

**Additional info**

This is the process and resulting exception from the debug logs:


```
11:52:03.82 [DEBUG] Running Extracting 12 requirements to build demo.pex from repository.pex: PyYAML==5.3.1, backoff==1.10.0, blinker==1.4, click==7.1.2, codetiming==1.2.0, enum34==1.1.10, gevent==20.9.0, marshmallow==3.9.0, packaging==20.9, pluggy==0.13.1, prometheus-client==0.9.0, transitions==0.8.4 under semaphore with concurrency id: 12
11:52:03.85 [DEBUG] spawned local process as Some(82860) for Process { argv: ["/Users/x/.pyenv/versions/3.7.12/bin/python", "./pex", "--cert", ".cacert.pem", "--python-path", "/Users/x/.pyenv/versions/3.7.12/bin:/Users/x/.pyenv/versions/3.8.11/bin:/Users/x/.pyenv/versions/3.9.7/bin:/usr/local/bin:/root/pyenv/bin", "--tmpdir", ".tmp", "--output-file", "demo.pex", "--requirements-pex", "local_dists.pex", "--pex-repository", "repository.pex", "--interpreter-constraint", "CPython<4,>=3.8", "--no-emit-warnings", "--jobs", "6", "--manylinux", "manylinux2014", "--entry-point", "bazooka.cli:main", "--sources-directory=source_files", "--constraints", "constraints.txt", "PyYAML==5.3.1", "backoff==1.10.0", "blinker==1.4", "click==7.1.2", "codetiming==1.2.0", "enum34==1.1.10", "gevent==20.9.0", "marshmallow==3.9.0", "packaging==20.9", "pluggy==0.13.1", "prometheus-client==0.9.0", "transitions==0.8.4"], env: {"CPPFLAGS": "", "LANG": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/Users/x/.cache/pants/pants_dev_deps/Darwin.x86_64.i386.py38.venv/bin:/Users/x/src/svs/python/bazooka/.dev-env/bin:/Users/x/.cargo/bin:/Users/x/.nvm/versions/node/v14.15.1/bin:/Users/x/.pyenv/shims:/Users/x/.pyenv/bin:/Users/x/Library/Python/3.7/bin:/usr/local/opt/ruby/bin:/usr/local/lib/ruby/gems/2.7.0/bin:/usr/local/opt/gettext/bin:/usr/local/opt/curl/bin:/Users/x/bin:/Users/x/src/svs/go/bin/:/Users/x/.jenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/puppetlabs/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON_PATH": "/Users/x/.pyenv/versions/3.7.12/bin:/Users/x/.pyenv/versions/3.8.11/bin:/Users/x/.pyenv/versions/3.9.7/bin:/usr/local/bin:/root/pyenv/bin", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_files: Digest { hash: Fingerprint<5bf6d09f5c6be72dc85edcf960884054b444c8d2cf8f8cab320260704a915adf>, size_bytes: 604 }, output_files: {RelativePath("demo.pex")}, output_directories: {}, timeout: None, execution_slot_variable: None, description: "Extracting 12 requirements to build demo.pex from repository.pex: PyYAML==5.3.1, backoff==1.10.0, blinker==1.4, click==7.1.2, codetiming==1.2.0, enum34==1.1.10, gevent==20.9.0, marshmallow==3.9.0, packaging==20.9, pluggy==0.13.1, prometheus-client==0.9.0, transitions==0.8.4", level: Info, append_only_caches: {CacheName("pex_root"): CacheDest(".cache/pex_root")}, jdk_home: None, platform_constraint: Some(Macos_x86_64), use_nailgun: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, cache_scope: Successful }

[...]

11:52:05.97 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Extracting 12 requirements to build demo.pex from repository.pex: PyYAML==5.3.1, backoff==1.10.0, blinker==1.4, click==7.1.2, codetiming==1.2.0, enum34==1.1.10, gevent==20.9.0, marshmallow==3.9.0, packaging==20.9, pluggy==0.13.1, prometheus-client==0.9.0, transitions==0.8.4' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/Users/x/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/Users/x/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/Users/x/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/Users/x/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 532, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/Users/x/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 668, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/Users/x/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 700, in execute_pkg_resources
    return runner()
  File "/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 672, in main
    env=env,
  File "/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 692, in do_main
    cache=ENV.PEX_ROOT,
  File "/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 600, in build_pex
    pex_builder.add_distribution(installed_dist.distribution)
  File "/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 458, in add_distribution
    dist_hash = self._add_dist_dir(dist.location, dist_name)
  File "/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 421, in _add_dist_dir
    self._copy_or_link(path, target_dir, label=dist_name)
  File "/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/pex_builder.py", line 541, in _copy_or_link
    self._chroot.symlink(src, dst, label)
  File "/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/common.py", line 686, in symlink
    os.symlink(abs_src, abs_dst)
FileExistsError: [Errno 17] File exists: '/Users/x/.cache/pants/named_caches/pex_root/installed_wheels/12138e788771deb6b7f0447e95d3ff330c080b4d/PyYAML-5.3.1-cp38-cp38-macosx_11_0_x86_64.whl' -> '/private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-execution0ijeeU/.tmp/tmp1qqqi41k/.deps/PyYAML-5.3.1-cp38-cp38-macosx_11_0_x86_64.whl'
```

Any pointers as to what to look for etc appreciated. Will keep digging see what I can come up with..



Discussion:
Hmm... I think I see where it goes wrong. The `local_dists.pex` does not only have my local dist, but also all the required dependencies in there... So when the PEX builder assembles the environment, it adds all dists from `local_dists.pex` and then all required dists from `repository.pex`, but those has already been added from the local_dists, and BOOM.

Not sure what the correct solution is here, but it feels like the PEX builder could play nice and realize if adding the same dist again, just skip and move on to the next one.

The correct fix might be that pants should only include local dists and no 3rdparty ones in `local_dists.pex`, as the name suggests.
Could it be this "harmless" feat I've run into?
https://github.com/pantsbuild/pants/blob/41ac4a2c79a19543f80be56f21198b588541ed18/src/python/pants/backend/python/util_rules/pex_from_targets.py#L231-L237

EDIT: Nope.
OK, I now know how to reproduce the issue, and it presents itself as soon as the local dist has _any_ dependencies at all.

So, I don't think it is viable that the `local_dist.pex` is built with only local dists without any 3rdparty deps, as that would result in a "broken" pex. So I'm inclined to think the best way forward is to make PEX aware that requirements may also be satisfied from the `--requirements-pex`.
Confirming that the fix suggested in pantsbuild/pex#1517 solves my use case.
@kaos does this solve your use case? It passes Pants tests, but before investing time in beefing those up to emulate your use case I was hoping you could give this edit a burn.
```diff
$ git diff
diff --git a/src/python/pants/backend/python/util_rules/local_dists.py b/src/python/pants/backend/python/util_rules/local_dists.py
index fc9458445..60033dcbd 100644
--- a/src/python/pants/backend/python/util_rules/local_dists.py
+++ b/src/python/pants/backend/python/util_rules/local_dists.py
@@ -132,6 +132,7 @@ async def build_local_dists(
             interpreter_constraints=request.interpreter_constraints,
             additional_inputs=wheels_digest,
             internal_only=request.internal_only,
+            additional_args=["--intransitive"],
         ),
     )
 
```
> So, I don't think it is viable that the local_dist.pex is built with only local dists without any 3rdparty deps, as that would result in a "broken" pex. So I'm inclined to think the best way forward is to make PEX aware that requirements may also be satisfied from the --requirements-pex.

I think that was actually the right tactic. The local PEX is a hack and can be "broken" since its never exposed to or used by end users. It's just there to emulate building a wheel for each local dist and then adding that wheel to another PEX.
Great. I thought it would require changes to pex itself how PEXes are built to get that behavior where not all dependencies are included. Hence my unviable conclusion.. 

Will try this out, thanks!
Profit!

> The correct fix might be that pants should only include local dists and no 3rdparty ones in local_dists.pex, as the name suggests.

I was onto it before overthinking it in the wrong direction :P 
Should've spotted there was a pex flag for it already ;) 

Thank you @jsirois for digging it out for me.
@benjyw do you see any issues with building the local_dists.pex `--intransitive`? For ~setup.py generated by Pants, the wheel transitive deps will jive with the Pants maintained deps. For hand-crafted setup.py, they might not, but - IIUC, the intent is that the hand-crafted setup.py is for those who know what they're doing / for building native extensions.
Hmmm, on balance I see no problem with using `--intransitive`. The failure mode is - handwritten setup.py with requirements that aren't expressed as BUILD file deps, and there be dragons in other ways too, so I'm fine breaking that case.
Ok. The other thing I was looking at was eliminating the --requirements-pex / PEX_PATH hack in place to add local dists to the resolves in various goals. That would make this all more robust and do the right thing with extra requirements. Did you look into that and reject it? There is some muscle that needs to be applied to get this to work with lockfiles, but that's doable; so I was guessing you just took an expedient approach to unblock real users that did not exercise any of these corners.
IIRC I used PEX_PATH because local dists are first-party code, and users might be iterating on them, so invoking a resolve on every change is a big and ~unexpected performance hit in that iteration. Imagine tweaking your code and having to wait ten minutes to re-resolve a bunch of unrelated third party things.
OK, thanks for that background @benjyw. I'm hesitating on the 1-liner above to fix @kaos's issue though since the current fast solution has caveats / accepted bugs / corners and the 1 line fix will just change these caveats / accepted bugs / corners to new ones. It probably makes sense to fix the whac-a-mole sooner rather than later.

 It seems to me we could have fast and correct with `--intransitive` PEXes that we gathered all top-level requirements from and mixed into resolves. So if the local dists PEX had just 1 local dist, we'd look at that dist and gather its `Requires-Dist` metadata and for each of those:
1. Add to the resolve if the required project is not already in the resolve.
2. Adjust the requirements in the current resolve does have the project already (AND the specifiers).

If only local dist code changes, but not requirements, then resolves will be unperturbed, vice versa, and both combined as well. Also, with 2 you might get an unsatisfiable constraint on a given project and the resolve would blow up fast - which is correct and what we want.
So I'm not sure why just building the local dists pex `--intransitive` isn't enough? Any of its requirements would be brought in via dep inference or explicit dep anyway, no? Since they'd be in the transitive dep closure of whatever root you're building.  
@benjyw that works if and only if the user-provided `setup.py` / `pyproject.toml` does not declare dependencies or if it does and the user makes sure to repeat these in the `python_distribution.dependencies`. @kaos IIUC you must have a `setup.py` or `pyproject.toml` that declares its own dependencies directly. Is that right? Do you also re-declare these as `python_distribution.dependencies`?

The advantage of actually inspecting the local dist wheels after they are built is the `Requires-Dist` metadata cuts through this. You're sure the Pants resolve process is getting all the requirements it needs to succeed or fail.
@jsirois Yes, I have a `pyproject.toml` with a `[tool.poetry.dependencies]` and a `poetry_requirements` target in the BUILD file. Also using a `constraints.txt` file generated by a `poetry export`. 
No custom setup.py, relying on getting that generated.
Also, I don't have any 3rdparty deps listed on the `python_distribution`s, rely on inference for that.
The pex I'm building have deps to all the local dists I want to include.

Some excerpts, in case it helps show my setup more clearly..

```python
pex_binary(
    name="bazooka-cdi",
    description="Bazooka as a CDI server module, with the mongo backend.",
    entry_point="cdi.cli:main",
    restartable=True,
    dependencies=[
        "bazooka:dist",
        "bazooka_storage_mongo:dist",
        ":bazooka-plugins",
    ],
)

target(
    name="bazooka-plugins",
    dependencies=[
        "bazooka_access_control:dist",
        "bazooka_admin:dist",
        "bazooka_cdi:dist",
        "bazooka_cron:dist",
        "bazooka_flask:dist",
        "bazooka_flask_graph:dist",
        "bazooka_flask_rest:dist",
        "bazooka_legacy_api:dist",
        "bazooka_metrics:dist",
        "bazooka_profile:dist",
        "bazooka_puppet:dist",
        "bazooka_puppet_svs:dist",
        "bazooka_hooks:dist",
        # Mutually exclusive plugins, include in final target as required.
        # "bazooka_storage_mongo:dist",
        # "bazooka_storage_pony:dist",
    ],
)
```

Each of those dists are uniform, using this macro (disclaimer: unfinished code):
```python
def bazooka_dist(name, summary, console_scripts=None, plugins=None, entry_points=None, dist=None, provides=None):
    python_sources()
    python_distribution(
        name=f"dist",
        sdist=False,
        wheel=True,
        provides=python_artifact(
            name=name,
            version="0.0.1",
            # version="$VERSION_PYTHON",
            summary=summary,
            classifiers=[
                "Development Status :: 4 - Beta",
                "Environment :: Console",
                "Intended Audience :: Developers",
                #    License :: OSI Approved :: ...
                "Operating System :: Linux",
                "Programming Language :: Python",
            ],
            **(provides or {}),
        ),
        entry_points={
            "console_scripts": console_scripts or {},
            "bazooka": plugins or {},
            **(entry_points or {}),
        },
        **(dist or {}),
    )

def bazooka_plugin(name, summary, entry_point=None, **kwargs):
    bazooka_dist(
        name,
        summary,
        plugins={name.split("-", 1)[-1]: entry_point or name.replace("-", "_")},
        **kwargs
    )
```

And typically used like:
```python
bazooka_dist(
    "bazooka",
    "Deployment Orchestrator for Application Releases",
    console_scripts={"bazooka": "bazooka.cli:main"},
    dist={"pypi_repositories":["@svs"]},
)

# And for the plugins..
bazooka_plugin("bazooka-admin", "Admin plugin for Bazooka")
```

@kaos so you're using dependencies on local dists not to get your own CPython extension (C) code compiled but just to get the wheel metadata (`<name>-<ver>.dist-info/entry_points.txt`) needed for a pure Python plugin mechanism?.
@jsirois Yes, precisely. 
Aha - OK. This is not the use case I've been worrying about. In your case, the 1 line fix will work just fine. Its the C extension code case with custom `setup.py` / `pyproject.toml` that can fail / have install requirements we don't account for properly.
Alright, enough kvetching. I'd rather we did the right thing so this was all bulletproof, but that will take more work than I can invest at the present moment. I'll proceed with the 1-liner.

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/local_dists.py b/src/python/pants/backend/python/util_rules/local_dists.py
--- a/src/python/pants/backend/python/util_rules/local_dists.py
+++ b/src/python/pants/backend/python/util_rules/local_dists.py
@@ -60,6 +60,13 @@ class LocalDistsPex:
 
     Can be consumed from another PEX, e.g., by adding to PEX_PATH.
 
+    The PEX will only contain locally built dists and not their dependencies. For Pants generated
+    `setup.py` / `pyproject.toml`, the dependencies will be included in the standard resolve process
+    that the locally-built dists PEX is adjoined to via PEX_PATH. For hand-made `setup.py` /
+    `pyproject.toml` with 3rdparty dependencies not hand-mirrored into BUILD file dependencies, this
+    will lead to issues. See https://github.com/pantsbuild/pants/issues/13587#issuecomment-974863636
+    for one way to fix this corner which is intentionally punted on for now.
+
     Lists the files provided by the dists on sys.path, so they can be subtracted from
     sources digests, to prevent the same file ending up on sys.path twice.
     """
@@ -132,6 +139,7 @@ async def build_local_dists(
             interpreter_constraints=request.interpreter_constraints,
             additional_inputs=wheels_digest,
             internal_only=request.internal_only,
+            additional_args=["--intransitive"],
         ),
     )
 


Test patch: