Repository: pypa/pipenv

Issue:
Breaking changes from 2023.7.23 to 2023.8.20 when using pyproject.toml
### Issue description

We noticed CI jobs began to fail over the weekend when upstream `pipenv` was picked up to perform `pipenv install --dev --deploy` on a repository with no changes other than an upgrade from `pipenv - 2023.7.23` to `pipenv - 2023.8.20`

After some debugging, we found it appears to be due to a breaking change in how dependencies are resolved in `pyproject.toml` between these versions of `pipenv`. Based on the diff (https://github.com/pypa/pipenv/compare/v2023.7.23...v2023.8.20#diff-d5fa37485876aa13851c69343a10f66e9c5b8820257447a0a0b1645776d09172R42), there were a fair amount of modifications made in how requirements are parsed, including deprecation of `requirementslib`. 

### Expected result

`pipenv install` should continue to work for dependencies whose versions are specified in `pyproject.toml` rather than just `Pipfile`

### Actual result

redacted logs from **successful** step using `pipenv version 2023.7.23`

```
pipenv clean
pipenv install --deploy --dev

Loading .env environment variables...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Resolving dependencies...
Success!

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
Resolving dependencies...
Success!

Updated Pipfile.lock (xxx)!
Loading .env environment variables...
Installing dependencies from Pipfile.lock (xx)...
Installing dependencies from Pipfile.lock (xx)...
```

redacted logs from **failing** step using `pipenv version 2023.8.20` 

> only including relevant sections specific to the private dependency where failures occur (stubbed as `github.com/org/repo.git` `v1.0.0`):

```
pipenv clean
pipenv install --deploy --dev

Loading .env environment variables...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
Warning: INFO:pipenv.patched.pip._internal.operations.prepare:Obtaining file:///home/circleci/project (from -r /tmp/pipenv-wrujgjp8-requirements/pipenv-kyt__35u-constraints.txt (line 3))
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Checking if build backend supports build_editable: started
INFO:pipenv.patched.pip._internal.cli.spinners:Checking if build backend supports build_editable: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Getting requirements to build wheel: started
INFO:pipenv.patched.pip._internal.cli.spinners:Getting requirements to build wheel: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Installing backend dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing backend dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (pyproject.toml): started
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (pyproject.toml): finished with status 'done'
...
...
INFO:pipenv.patched.pip._internal.operations.prepare:Collecting common@ git+ssh://git@github.com/org/repo.git@v1.0.0 (from application==0.0.0->-r /tmp/pipenv-wrujgjp8-requirements/pipenv-kyt__35u-constraints.txt (line 3))
INFO:pipenv.patched.pip._internal.vcs.git:Cloning ssh://****@github.com/org/repo.git (to revision v1.0.0) to /tmp/pip-temp-oesk1ice/common_50cbdabbce754467a4c199345000bc82
INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet 'ssh://****@github.com/org/repo.git' /tmp/pip-temp-oesk1ice/common_50cbdabbce754467a4c199345000bc82
INFO:pip.subprocessor:Running command git checkout -q f832c163be4915ae9ebef4590f7d2141d5cd65c1
INFO:pipenv.patched.pip._internal.vcs.git:Resolved ssh://****@github.com/org/repo.git to commit f832c163be4915ae9ebef4590f7d2141d5cd65c1
...
...
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Getting requirements to build wheel: started
INFO:pipenv.patched.pip._internal.cli.spinners:Getting requirements to build wheel: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Installing backend dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing backend dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (pyproject.toml): started
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (pyproject.toml): finished with status 'done'
...
...
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (setup.py): started
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (setup.py): finished with status 'done'
...
...
INFO:pipenv.patched.pip._internal.vcs.git:Cloning ssh://****@github.com/org/repo.git (to revision v1.0.0) to /tmp/tmpkdh9kolc
INFO:pip.subprocessor:Running command git clone --filter=blob:none 'ssh://****@github.com/org/repo.git' /tmp/tmpkdh9kolc
INFO:pip.subprocessor:Cloning into '/tmp/tmpkdh9kolc'...
INFO:pip.subprocessor:Running command git checkout -q f832c163be4915ae9ebef4590f7d2141d5cd65c1
INFO:pipenv.patched.pip._internal.vcs.git:Resolved ssh://****@github.com/org/repo.git to commit f832c163be4915ae9ebef4590f7d2141d5cd65c1
..
..
Updated Pipfile.lock (7e50a95cbc0abb105303e9a25682548342d710c984e31e9566e6aabcae59590d)!
Loading .env environment variables...
Installing dependencies from Pipfile.lock (59590d)...
Installing dependencies from Pipfile.lock (59590d)...
[pipenv.exceptions.InstallError]: Collecting common@ git+ssh://git@github.com/org/repo.git@v1.0.0@f832c163be4915ae9ebef4590f7d2141d5cd65c1 (from -r /tmp/pipenv-24ej7m9_-requirements/pipenv-i2yt9ua0-reqs.txt (line 1))
[pipenv.exceptions.InstallError]:   Cloning ssh://****@github.com/org/platform-python-common.git@v1.0.0 (to revision f832c163be4915ae9ebef4590f7d2141d5cd65c1) to /tmp/pip-install-br8go210/common_f90e1bae5bc44e1e91525d67ec63685e
[pipenv.exceptions.InstallError]: Running command git clone --filter=blob:none --quiet 'ssh://****@github.com/org/repo.git@v1.0.0' /tmp/pip-install-br8go210/common_f90e1bae5bc44e1e91525d67ec63685e
[pipenv.exceptions.InstallError]:   fatal: remote error:
[pipenv.exceptions.InstallError]:    org/repo.git@v1.0.0 is not a valid repository name
[pipenv.exceptions.InstallError]:   Visit https://support.github.com/ for help
[pipenv.exceptions.InstallError]:   error: subprocess-exited-with-error
[pipenv.exceptions.InstallError]:   
[pipenv.exceptions.InstallError]:   × git clone --filter=blob:none --quiet 'ssh://****@github.com/org/repo.git@v1.0.0' /tmp/pip-install-br8go210/common_f90e1bae5bc44e1e91525d67ec63685e did not run successfully.
[pipenv.exceptions.InstallError]:   │ exit code: 128
[pipenv.exceptions.InstallError]:   ╰─> See above for output.
[pipenv.exceptions.InstallError]:   
[pipenv.exceptions.InstallError]:   note: This error originates from a subprocess, and is likely not a problem with pip.
[pipenv.exceptions.InstallError]: error: subprocess-exited-with-error
[pipenv.exceptions.InstallError]: 
[pipenv.exceptions.InstallError]: × git clone --filter=blob:none --quiet 'ssh://****@github.com/org/repo.git@v1.0.0' /tmp/pip-install-br8go210/common_f90e1bae5bc44e1e91525d67ec63685e did not run successfully.
[pipenv.exceptions.InstallError]: │ exit code: 128
[pipenv.exceptions.InstallError]: ╰─> See above for output.
[pipenv.exceptions.InstallError]: 
[pipenv.exceptions.InstallError]: note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Couldn't install package: {}
```


the generated `Pipfile.lock` for `common` when running `2023.8.20` looks correct

```
        "common": {
            "git": "git+ssh://git@github.com/org/repo.git@v1.0.0",
            "ref": "f832c163be4915ae9ebef4590f7d2141d5cd65c1"
        },
```

we tested the latest (`8.20`) `pipenv install` on a _different_ repository where `pyproject.toml` does **not** include dependency versions and they are _only_ contained in `Pipfile`. in this scenario, `pipenv install` succeeds.

however, in the newly failing job where `pyproject.toml` **does include** versions, `pipenv install --deploy` fails.

**example of pyproject.toml**

the same `pyproject.toml` succeeds in `2023.7.23` and fails in `2023.8.20`

```
[project]
name = "application"
version = "0.0.0"
dependencies = [
  "common @ git+ssh://git@github.com/org/repo.git@v1.0.0",
]
```

### Steps to replicate

for a repository with a `pyproject.toml` file that has a `git+ssh` dependency, run `pipenv install --deploy`. 




-------------------------------------------------------------------------------

can't include `pipenv --support` -- relevant redacted outputs included above 



Discussion:

Resolving patch:
diff --git a/pipenv/utils/dependencies.py b/pipenv/utils/dependencies.py
--- a/pipenv/utils/dependencies.py
+++ b/pipenv/utils/dependencies.py
@@ -959,18 +959,29 @@ def expansive_install_req_from_line(
 
 def install_req_from_pipfile(name, pipfile):
     _pipfile = {}
+    vcs = None
     if hasattr(pipfile, "keys"):
         _pipfile = dict(pipfile).copy()
+    else:
+        vcs = next(iter([vcs for vcs in VCS_LIST if pipfile.startswith(f"{vcs}+")]), None)
+        if vcs is not None:
+            _pipfile[vcs] = pipfile
 
     extras = _pipfile.get("extras", [])
     extras_str = ""
     if extras:
         extras_str = f"[{','.join(extras)}]"
-    vcs = next(iter([vcs for vcs in VCS_LIST if vcs in _pipfile]), None)
+    if not vcs:
+        vcs = next(iter([vcs for vcs in VCS_LIST if vcs in _pipfile]), None)
 
     if vcs:
-        _pipfile["vcs"] = vcs
-        req_str = f"{_pipfile[vcs]}{_pipfile.get('ref', '')}{extras_str}"
+        vcs_url = _pipfile[vcs]
+        fallback_ref = ""
+        if "@" in vcs_url:
+            vcs_url_parts = vcs_url.rsplit("@", 1)
+            vcs_url = vcs_url_parts[0]
+            fallback_ref = vcs_url_parts[1]
+        req_str = f"{vcs_url}{_pipfile.get('ref', fallback_ref)}{extras_str}"
         if not req_str.startswith(f"{vcs}+"):
             req_str = f"{vcs}+{req_str}"
         if f"{vcs}+file://" in req_str:
diff --git a/pipenv/utils/requirements.py b/pipenv/utils/requirements.py
--- a/pipenv/utils/requirements.py
+++ b/pipenv/utils/requirements.py
@@ -156,6 +156,12 @@ def requirement_from_lockfile(
         if vcs in package_info:
             url = package_info[vcs]
             ref = package_info.get("ref", "")
+            if "@" in url:
+                url_parts = url.rsplit("@", 1)
+                url = url_parts[0]
+                if not ref:
+                    ref = url_parts[1]
+
             extras = (
                 "[{}]".format(",".join(package_info.get("extras", [])))
                 if "extras" in package_info
@@ -163,7 +169,7 @@ def requirement_from_lockfile(
             )
             include_vcs = "" if f"{vcs}+" in url else f"{vcs}+"
             egg_fragment = "" if "#egg=" in url else f"#egg={package_name}"
-            ref_str = "" if f"@{ref}" in url else f"@{ref}"
+            ref_str = "" if not ref or f"@{ref}" in url else f"@{ref}"
             if is_editable_path(url) or "file://" in url:
                 pip_line = f"-e {include_vcs}{url}{ref_str}{egg_fragment}{extras}"
             else:


Test patch: