Repository: pypa/pipenv
Pull request: Patch for setup.py egg_info issue
It was discovered that requirementslib uses the sys.executable python rather than the pipenv environment python.  Which as @oz123 reminded me should be the same, but due to some weird code paths, it wasn't always the same in my testing when the resolver got invoked as subprocess.

### The issue

Fixes #5753 

### The fix

I don't love the use of the environment variable, but I figured out this was the original intention of how the pipenv <--> requirements lib interface and at some point it got hard-coded to sys.executable in the resolver code which was getting invoked with a different python version at points.

This fixes the problem by ensuring that the correct python of the pipenv environment is being used.

### The checklist

* [X] Associated issue
* [ ] A news fragment in the `news/` directory to describe this fix with the extension `.bugfix.rst`, `.feature.rst`, `.behavior.rst`, `.doc.rst`. `.vendor.rst`. or `.trivial.rst` (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.



Problem statement:
Release v2023.6.26: error: invalid command 'egg_info'

Upgrading to `pipenv` v2023.6.26 generates an `error: invalid command 'egg_info'` in the CircleCI builds of our Django project.

`pipenv` works as expected on local machines but fails on CircleCI until we `pip install --upgrade "pipenv<2023.6.26`

### Expected result

pipenv builds without errors.

### Actual result
```
Successfully installed pipenv-2023.6.26 setuptools-68.0.0
pipenv, version 2023.6.26
Creating a virtualenv for this project...
Pipfile: /home/circleci/circleci-workdir/Pipfile
Using /home/circleci/.pyenv/versions/3.9.17/bin/python3.9 (3.9.17) to create virtualenv...
created virtual environment CPython3.9.17.final.0-64 in 162ms
  creator CPython3Posix(dest=/home/circleci/circleci-workdir/.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/circleci/.local/share/virtualenv)
    added seed packages: pip==23.1.2, setuptools==67.8.0, wheel==0.40.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!

Virtualenv location: /home/circleci/circleci-workdir/.venv
Installing dependencies from Pipfile.lock (ef2d1f)...
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'egg_info'

Exited with code exit status 1
```

When possible, provide the verbose output (`--verbose`), especially for locking and dependencies resolving issues.

### Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

-------------------------------------------------------------------------------

Please run `$ pipenv --support`, and paste the results here. Don't put backticks (`` ` ``) around it! The output already contains Markdown formatting.

If you're on macOS, run the following:

    $ pipenv --support | pbcopy

If you're on Windows, run the following:

    > pipenv --support | clip

If you're on Linux, run the following:

    $ pipenv --support | xclip

Hints from issue discussion:
@cclauss can you please show you dependencies ?

Do you have somewhere in the tree `distribute`.  The seems to be related to one of the packages.
(I guess earlier versions of pipenv were more relaxed about the underlying error).
This was certainly caused by (unintentionally) the large refactor to requirementslib to migrate away from attrs to pydantic.   It seems that you have some requirement that is going down the path of getting the metadata by calling `python setup.py egg_info` -- but it seems something either with this package or otherwise doesn't allow for this command.  It would be helpful to figure out what path for getting the metadata this package should take, and more information about how the package setup is configured that its failing on.
I have a similar issue when locking [this Pipfile](https://github.com/juanitorduz/website_projects/blob/master/Pipfile). Any tips on how to help you debug the error? 

@juanitorduz that Pipfile will be gold for trying to figure it out -- know that this is a top priority bug for pipenv, but it will require some tinkering with requirementslib to get it nailed down.   I'll try to find some time to triage further later this week, but if someone else wants to dig deeper, I think the strategy is:
1.) Narrow down which package type is causing this issue, is the vcs isntall?  
2.) Then I would try comparing in an editable pipenv install from main the two release points to determine how the metadata is determined in the prior version, because clearly in the new version its trying to build the egg_info, we may need a different code path to take precedence in requirementslib for this case.

The easiest way to test a potential fix would be to modify requirementslib within pipenv and open a PR -- if all the tests in pipenv pass, there is a high probability the tests in requirementslib will pass also when the change is ported over the the main library there.
Pipfile:
`{git = "https://github.com/shtalinberg/django-el-pagination.git"}`
@juanitorduz Try commenting out your line 90 and running again.
> sts-jax = {git = "https://github.com/probml/sts-jax.git", editable = true, ref = "main"}
I was able to reproduce it briefly with an existing old virtualenv and just the Pipfile:

```
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"
sts-jax = {git = "https://github.com/probml/sts-jax.git", editable = true, ref = "main"}

[dev-packages]

[requires]
python_version = "3.9"
```

For some reason when I had pipenv create a new virtualenv, it did not get the egg_info issue.   I then wasn't able to lock for a different reason which is equally odd, despite there being a regular release 2.12, I got:

> CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement tensorflow>=2.11 (from versions: 2.13.0rc0, 2.13.0rc1, 2.13.0rc2)

So I noted those that it did find are pre releases, and I ran again:  `pipenv lock -v --pre` and it locked successfully and installed successfully. 

Noting also that in the issue report from @cclauss that it appears to be creating a fresh virtualenv but is getting the no egg_info command -- from what I can tell, python setup.py egg_info should be a valid command for the dependencies that are getting the error, they have a setup.py and use setuptools.


I think I just figured out what is happening, and its not the simplest fix but let me explain.  requirementslib basically is using the system python and not the virtualenv python -- I have a fix, but the problem is, requirementslib starts using pipenv mechanics to figure out the correct python to use.  Your system python has a really old setuptools is my guess which doesn't have the `setup_egg` command.  Please have a try of this branch to see what I mean, for me it fixes the issue I was able to reproduce in my Ubuntu VM:  https://github.com/pypa/pipenv/pull/5759

The problem is, it opens the can of worms of what the heck is requirementslib, and why doesn't it just become part of pipenv.    Or maybe it pushes us to have a better interface to pass in the right python to use within requirementslib, but yeah -- this is a great discovery of an issue and kind of a bummer. 
Hi @matteius 

>  Your system python has a really old setuptools is my guess which doesn't have the setup_egg command. 

Just tested 2023.6.26 with `pipenv lock` command againt different [Python Offical docker images](https://hub.docker.com/_/python).
Looks like the issue happens on python:3.9 or lower. 3.10 and 3.11 has no issue installing the package.

Hope this helps!
@shinkawk I am guessing if you do a regular `pip install setuptools --upgrade` ahead of the test that python:3.9 and lower would actually succeed, but it highlights an implementation flaw in pipenv we want to address.

Commit message: Propose patch to use the correct python in the requirementslib resolve phase.

Patch to review:
diff --git a/pipenv/environment.py b/pipenv/environment.py
index 32eca94ce1..83e41285c4 100644
--- a/pipenv/environment.py
+++ b/pipenv/environment.py
@@ -81,6 +81,7 @@ def __init__(
         if self.is_venv:
             self._base_paths = self.get_paths()
         self.sys_paths = get_paths()
+        os.environ["PIP_PYTHON_PATH"] = self.python
 
     def safe_import(self, name: str) -> ModuleType:
         """Helper utility for reimporting previously imported modules while inside the env"""
diff --git a/pipenv/resolver.py b/pipenv/resolver.py
index 87e76564bc..eb24669ad4 100644
--- a/pipenv/resolver.py
+++ b/pipenv/resolver.py
@@ -795,10 +795,6 @@ def _main(
     parse_only=False,
     category=None,
 ):
-    os.environ["PIPENV_REQUESTED_PYTHON_VERSION"] = ".".join(
-        [str(s) for s in sys.version_info[:3]]
-    )
-    os.environ["PIP_PYTHON_PATH"] = str(sys.executable)
     if parse_only:
         parse_packages(
             packages,
diff --git a/pipenv/utils/dependencies.py b/pipenv/utils/dependencies.py
index 4295c11a2e..0248271c10 100644
--- a/pipenv/utils/dependencies.py
+++ b/pipenv/utils/dependencies.py
@@ -53,25 +53,17 @@ def get_pipfile_category_using_lockfile_section(category):
 
 
 class HackedPythonVersion:
-    """A Beautiful hack, which allows us to tell pip which version of Python we're using."""
+    """A hack, which allows us to tell resolver which version of Python we're using."""
 
-    def __init__(self, python_version, python_path):
-        self.python_version = python_version
+    def __init__(self, python_path):
         self.python_path = python_path
 
     def __enter__(self):
-        # Only inject when the value is valid
-        if self.python_version:
-            os.environ["PIPENV_REQUESTED_PYTHON_VERSION"] = str(self.python_version)
         if self.python_path:
             os.environ["PIP_PYTHON_PATH"] = str(self.python_path)
 
     def __exit__(self, *args):
-        # Restore original Python version information.
-        try:
-            del os.environ["PIPENV_REQUESTED_PYTHON_VERSION"]
-        except KeyError:
-            pass
+        pass
 
 
 def get_canonical_names(packages):
diff --git a/pipenv/utils/resolver.py b/pipenv/utils/resolver.py
index 8fdbd91fbb..bc94cfdf60 100644
--- a/pipenv/utils/resolver.py
+++ b/pipenv/utils/resolver.py
@@ -1155,10 +1155,8 @@ def resolve_deps(
     """
     index_lookup = {}
     markers_lookup = {}
-    python_path = which("python", allow_global=allow_global)
     if not os.environ.get("PIP_SRC"):
         os.environ["PIP_SRC"] = project.virtualenv_src_location
-    backup_python_path = sys.executable
     results = []
     resolver = None
     if not deps:
@@ -1167,7 +1165,7 @@ def resolve_deps(
     req_dir = req_dir if req_dir else os.environ.get("req_dir", None)
     if not req_dir:
         req_dir = create_tracked_tempdir(prefix="pipenv-", suffix="-requirements")
-    with HackedPythonVersion(python_version=python, python_path=python_path):
+    with HackedPythonVersion(python_path=project.environment.python):
         try:
             results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
                 deps,
@@ -1186,8 +1184,7 @@ def resolve_deps(
     # Second (last-resort) attempt:
     if results is None:
         with HackedPythonVersion(
-            python_version=".".join([str(s) for s in sys.version_info[:3]]),
-            python_path=backup_python_path,
+            python_path=project.environment.python,
         ):
             try:
                 # Attempt to resolve again, with different Python version information,
diff --git a/pipenv/vendor/requirementslib/models/setup_info.py b/pipenv/vendor/requirementslib/models/setup_info.py
index 69306d3735..f32a7f9b1b 100644
--- a/pipenv/vendor/requirementslib/models/setup_info.py
+++ b/pipenv/vendor/requirementslib/models/setup_info.py
@@ -58,11 +58,6 @@
 
 CACHE_DIR = os.environ.get("PIPENV_CACHE_DIR", user_cache_dir("pipenv"))
 
-# The following are necessary for people who like to use "if __name__" conditionals
-# in their setup.py scripts
-_setup_stop_after = None
-_setup_distribution = None
-
 
 def pep517_subprocess_runner(cmd, cwd=None, extra_environ=None) -> None:
     """The default method of calling the wrapper subprocess."""
@@ -75,8 +70,9 @@ def pep517_subprocess_runner(cmd, cwd=None, extra_environ=None) -> None:
 
 class BuildEnv(envbuild.BuildEnvironment):
     def pip_install(self, reqs):
+        python = os.environ.get("PIP_PYTHON_PATH", sys.executable)
         cmd = [
-            sys.executable,
+            python,
             "-m",
             "pip",
             "install",
@@ -1141,29 +1137,15 @@ def run_setup(script_path, egg_base=None):
         sys.path.insert(0, target_cwd)
 
         save_argv = sys.argv.copy()
-        try:
-            global _setup_distribution, _setup_stop_after
-            _setup_stop_after = "run"
-            sys.argv[0] = script_name
-            sys.argv[1:] = args
-            with open(script_name, "rb") as f:
-                contents = f.read().replace(rb"\r\n", rb"\n")
-                exec(contents, g)
-        # We couldn't import everything needed to run setup
-        except Exception:
-            python = os.environ.get("PIP_PYTHON_PATH", sys.executable)
+        python = os.environ.get("PIP_PYTHON_PATH", sys.executable)
 
-            sp.run(
-                [python, "setup.py"] + args,
-                cwd=target_cwd,
-                stdout=sp.PIPE,
-                stderr=sp.PIPE,
-            )
-        finally:
-            _setup_stop_after = None
-            sys.argv = save_argv
-            _setup_distribution = get_metadata(egg_base, metadata_type="egg")
-        dist = _setup_distribution
+        sp.run(
+            [python, "setup.py"] + args,
+            cwd=target_cwd,
+            stdout=sp.PIPE,
+            stderr=sp.PIPE,
+        )
+        dist = get_metadata(egg_base, metadata_type="egg")
     return dist
 
 


Code review comments:
diff_hunk: @@ -1141,29 +1137,15 @@ def run_setup(script_path, egg_base=None):
         sys.path.insert(0, target_cwd)
 
         save_argv = sys.argv.copy()
-        try:
-            global _setup_distribution, _setup_stop_after
-            _setup_stop_after = "run"
-            sys.argv[0] = script_name
-            sys.argv[1:] = args
-            with open(script_name, "rb") as f:
-                contents = f.read().replace(rb"\r\n", rb"\n")
-                exec(contents, g)
-        # We couldn't import everything needed to run setup
-        except Exception:
-            python = os.environ.get("PIP_PYTHON_PATH", sys.executable)
+        python = os.environ.get("PIP_PYTHON_PATH", sys.executable)
 
-            sp.run(
-                [python, "setup.py"] + args,
-                cwd=target_cwd,
-                stdout=sp.PIPE,
-                stderr=sp.PIPE,
-            )
-        finally:
-            _setup_stop_after = None
-            sys.argv = save_argv
-            _setup_distribution = get_metadata(egg_base, metadata_type="egg")
-        dist = _setup_distribution
+        sp.run(
+            [python, "setup.py"] + args,
+            cwd=target_cwd,
+            stdout=sp.PIPE,
+            stderr=sp.PIPE,
path: pipenv/vendor/requirementslib/models/setup_info.py
text: @user1:
```suggestion
            capture_output=True
```
---
% `MY_CODE="import subprocess as sp ; sp.run([python, "setup.py"] + args, cwd=target_cwd, stdout=sp.PIPE, stderr=sp.PIPE)"`

% `echo $MY_CODE | ruff --select=UP022 --fix -`
```
import subprocess as sp ; sp.run([python, setup.py] + args, cwd=target_cwd, capture_output=True)
```
% `ruff rule UP022 ` # https://beta.ruff.rs/docs/rules/#pyupgrade-up
# replace-stdout-stderr (UP022)

Derived from the **pyupgrade** linter.

Autofix is always available.

## What it does
Checks for uses of `subprocess.run` that send `stdout` and `stderr` to a
pipe.

## Why is this bad?
As of Python 3.7, `subprocess.run` has a `capture_output` keyword argument
that can be set to `True` to capture `stdout` and `stderr` outputs. This is
equivalent to setting `stdout` and `stderr` to `subprocess.PIPE`, but is
more explicit and readable.

## Example
```python
import subprocess

subprocess.run(["foo"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
```

Use instead:
```python
import subprocess

subprocess.run(["foo"], capture_output=True)
```

## References
- [Python 3.7 release notes](https://docs.python.org/3/whatsnew/3.7.html#subprocess)
- [Python documentation: `subprocess.run`](https://docs.python.org/3/library/subprocess.html#subprocess.run)


@user1:
Was the removal of `cwd=target_cwd` intentional?

@author:
@user1 intentional, as it seemed unnecessary with the `cd(target_cwd)` context manager -- should already be in that directory.

@author:
Well its finally having a good run -- some just turned green.