Repository: pypa/pipenv

Issue:
pipenv 2022.8.5.13 introduced an issue with platformdirs
### Issue description

We have a job in our CI pipeline to uses `pipenv` to set up an environment with `black` and then run said tool. Our CI job for this in Gitlab-CI is extremely simple:

```yaml
black_linting_check:
  stage: lint
  image: python:3.8
  before_script:
    - python -V
    - pip install pipenv
    - pipenv install --dev
  script:
    - pipenv run black derms derms_core mocks --check
```

Unfortunately it seems like the release of pipenv 2022.8.5.13 introduced some issue with the `platformdirs` package.

The following is a log of a CI successful run of our job from August 12th 2022:
```
Running with gitlab-runner 14.7.0 (98daeee0)
  on ip-17[2](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L2)-[3](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L3)1-5-168 qLr1uLbu
Resolving secrets
00:00
Preparing the "docker" executor
00:09
Using Docker executor with image python:3.8 ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image python:3.8 ...
Using docker image sha2[5](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L5)[6](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L6):be03ed85[7](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L7)d3127[8](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L8)de24758c22[9](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L9)9c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
Preparing environment
01:12
Running on runner-qlr1ulbu-project-22339857-concurrent-1 via ip-172-31-5-168...
Getting source from Git repository
00:38
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/opusonesolutions/gridos/derms/v3/derms-core/.git/
Created fresh repository.
Checking out 197c530c as DR-2998-deadband-settings-api...
Skipping Git submodules setup
Executing "step_script" stage of the job script
04:06
Using docker image sha256:be03ed857d31278de24758c2299c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
$ python -V
Python 3.8.13
$ pip install pipenv
Looking in indexes: https://pypi.org/simple, https://[MASKED]:****@opusonesolutions.jfrog.io/opusonesolutions/api/pypi/pypi-local/simple
Collecting pipenv
  Downloading pipenv-2022.8.5-py2.py3-none-any.whl (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 64.1 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.8/site-packages (from pipenv) (57.5.0)
Collecting certifi
  Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.2/160.2 KB 27.6 MB/s eta 0:00:00
Collecting virtualenv
  Downloading virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 63.1 MB/s eta 0:00:00
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Collecting distlib<1,>=0.3.5
  Downloading distlib-0.3.5-py2.py3-none-any.whl (466 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.0/467.0 KB 59.8 MB/s eta 0:00:00
Collecting platformdirs<3,>=2.4
  Downloading platformdirs-2.5.2-py3-none-any.whl ([14](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L14) kB)
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Installing collected packages: distlib, virtualenv-clone, platformdirs, filelock, certifi, virtualenv, pipenv
Successfully installed certifi-2022.6.[15](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L15) distlib-0.3.5 filelock-3.8.0 pipenv-2022.8.5 platformdirs-2.5.2 virtualenv-20.[16](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L16).3 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
$ pipenv install --dev
Creating a virtualenv for this project...
Pipfile: /builds/opusonesolutions/gridos/derms/v3/derms-core/Pipfile
Using /usr/local/bin/python3.8 (3.8.13) to create virtualenv...
created virtual environment CPython3.8.13.final.0-64 in 6[19](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L19)ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/derms-core-Sx_JEpdN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==[22](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L22).2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/derms-core-Sx_JEpdN
Installing dependencies from Pipfile.lock ([29](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2849857867#L29)83d4)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
$ pipenv run black derms derms_api mocks --check
Loading .env environment variables...
All done! ✨ 🍰 ✨
290 files would be left unchanged.
Cleaning up project directory and file based variables
00:01
Job succeeded
```

The following is a log of a CI failed run of our job from August 13th 2022:
```
Running with gitlab-runner 14.7.0 (98daeee0)
  on ip-17[2](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L2)-[3](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L3)1-5-63 oyxCD8yZ
Resolving secrets
00:00
Preparing the "docker" executor
01:2[5](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L5)
Using Docker executor with image python:3.8 ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image python:3.8 ...
Using docker image sha25[6](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L6):be03ed85[7](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L7)d3127[8](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L8)de24758c22[9](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L9)9c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
Preparing environment
00:03
Running on runner-oyxcd8yz-project-22339857-concurrent-1 via ip-172-31-5-63...
Getting source from Git repository
00:11
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/opusonesolutions/gridos/derms/v3/derms-core/.git/
Checking out ca798f52 as DR-2998-deadband-settings-api...
Skipping Git submodules setup
Executing "step_script" stage of the job script
02:49
Using docker image sha256:be03ed857d31278de24758c2299c563a358de45fd08c92f7d10b7cfbb84fbac0 for python:3.8 with digest python@sha256:3583a2f489ef5350b27ec1f29f494dfd47bf690ebbdf6e74d2ead3b3b14fbe54 ...
$ python -V
Python 3.8.13
$ pip install pipenv
Looking in indexes: https://pypi.org/simple, https://[MASKED]:****@opusonesolutions.jfrog.io/opusonesolutions/api/pypi/pypi-local/simple
Collecting pipenv
  Downloading pipenv-2022.8.13-py2.py3-none-any.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 16.2 MB/s eta 0:00:00
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Collecting certifi
  Downloading certifi-2022.6.15-py3-none-any.whl (160 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.2/160.2 KB 2.4 MB/s eta 0:00:00
Collecting virtualenv
  Downloading virtualenv-20.16.3-py2.py3-none-any.whl (8.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB [12](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L12).9 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=36.2.1 in /usr/local/lib/python3.8/site-packages (from pipenv) (57.5.0)
Collecting platformdirs<3,>=2.4
  Downloading platformdirs-2.5.2-py3-none-any.whl ([14](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L14) kB)
Collecting distlib<1,>=0.3.5
  Downloading distlib-0.3.5-py2.py3-none-any.whl (466 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 467.0/467.0 KB 3.7 MB/s eta 0:00:00
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Installing collected packages: distlib, virtualenv-clone, platformdirs, filelock, certifi, virtualenv, pipenv
Successfully installed certifi-2022.6.[15](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L15) distlib-0.3.5 filelock-3.8.0 pipenv-2022.8.13 platformdirs-2.5.2 virtualenv-20.[16](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L16).3 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version [22](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L22).0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
$ pipenv install --dev
Creating a virtualenv for this project...
Pipfile: /builds/opusonesolutions/gridos/derms/v3/derms-core/Pipfile
Using /usr/local/bin/python3.8 (3.8.13) to create virtualenv...
created virtual environment CPython3.8.13.final.0-64 in 1679ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/derms-core-Sx_JEpdN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/derms-core-Sx_JEpdN
Installing dependencies from Pipfile.lock ([29](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L29)83d4)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
$ pipenv run black derms derms_api mocks --check
Loading .env environment variables...
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/derms-core-Sx_JEpdN/bin/black", line 5, in <module>
    from black import patched_main
  File "src/black/__init__.py", line 42, in <module>
  File "src/black/nodes.py", line [33](https://gitlab.com/opusonesolutions/gridos/derms/v3/derms-core/-/jobs/2877524960#L33), in <module>
  File "src/black/cache.py", line 9, in <module>
ModuleNotFoundError: No module named 'platformdirs'
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
```

I am pretty sure that this issue is caused by some change between `pipenv` 2022.8.5 and 2022.8.5.13 for the following reasons:

1. I noticed this issue initially on a different MR in our Gitlab and so went back and forcibly re-ran the job show above on a different MR that had not changed between the two job runs show above. The result was this previously green job failing with no change other than the version of `pipenv` (Which, you will note from the CI job details was not version locked)
2. Modifying our CI job such that the version of `pipenv` installed was locked to 2022.8.5 corrected the issue


### Expected result

`pipenv` should install `platformdirs` correctly into the environment

### Actual result

`pipenv` seems to fail to install `platformdirs` into the enviroment

### Steps to replicate

This should be replicable using the steps in the CI job listed above along with any old random codebase



Discussion:
This is an interesting one @OOPMan -- I was able to reproduce it, but when I passed the verbose argument, I saw in the output that `platformdirs` was already installed to my site-packages, and so it was ignoring installing it.
```
$ pipenv --no-site-packages install platformdirs -v
Installing platformdirs...
Installing package: platformdirs
Writing supplied requirement line to temporary file: 'platformdirs'
Installing 'platformdirs'
$ '\c\Users\matte\.virtualenvs\pipenv-5254-UiSfqJAX\Scripts\python.exe' 'C:\Users\matte\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\__pip-runner__.py' install --verbose --upgrade --exists-
action=i -r 'c:\users\matte\appdata\local\temp\pipenv-ofc42xsn-requirements\pipenv-_t1s_993-requirement.txt' -i https://pypi.org/simple
Using source directory: '\\c\\Users\\matte\\.virtualenvs\\pipenv-5254-UiSfqJAX\\src'
Adding platformdirs to Pipfile's [packages]...
Installation Succeeded
Installing dependencies from Pipfile.lock (d573fd)...
Writing supplied requirement line to temporary file: 'platformdirs==2.5.2 --hash=sha256:027d8e83a2d7de06bbac4e5ef7e023c02b863d7ea5d079477e722bb41ab25788 --hash=sha256:58c8abb07dcb441e6ee4b11d8df0ac856038f944ab9
8b7be6b27b2a3c7feef19'
Installing 'platformdirs'
$ '\c\Users\matte\.virtualenvs\pipenv-5254-UiSfqJAX\Scripts\python.exe' 'C:\Users\matte\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\__pip-runner__.py' install --verbose --upgrade --require
-hashes --no-deps --exists-action=i -r 'c:\users\matte\appdata\local\temp\pipenv-ofc42xsn-requirements\pipenv-y0ny1g83-requirement.txt' -i https://pypi.org/simple
Using source directory: '\\c\\Users\\matte\\.virtualenvs\\pipenv-5254-UiSfqJAX\\src'
Using pip 22.2.2 from C:\Users\matte\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip (python 3.10)
Requirement already satisfied: platformdirs==2.5.2 in c:\users\matte\appdata\roaming\python\python310\site-packages (from -r c:\users\matte\appdata\local\temp\pipenv-ofc42xsn-requirements\pipenv-y0ny1g83-requir
ement.txt (line 1)) (2.5.2)
```

The only way I was able to get it to work was by first uninstalling `platformdirs` using `pip`:

```
matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pip uninstall platformdirs --yes
Found existing installation: platformdirs 2.5.2
Uninstalling platformdirs-2.5.2:
  Successfully uninstalled platformdirs-2.5.2

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pip uninstall platformdirs --yes
WARNING: Skipping platformdirs as it is not installed.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv run python -c "import platformdirs"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'platformdirs'

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv --no-site-packages install platformdirs -v
Installing platformdirs...
Installing package: platformdirs
Writing supplied requirement line to temporary file: 'platformdirs'
Installing 'platformdirs'
$ '\c\Users\matte\.virtualenvs\pipenv-5254-UiSfqJAX\Scripts\python.exe' 'C:\Users\matte\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\__pip-runner__.py' install --verbose --upgrade --exists-
action=i -r 'c:\users\matte\appdata\local\temp\pipenv-745_7y99-requirements\pipenv-kzsguwdl-requirement.txt' -i https://pypi.org/simple
Using source directory: '\\c\\Users\\matte\\.virtualenvs\\pipenv-5254-UiSfqJAX\\src'
Adding platformdirs to Pipfile's [packages]...
Installation Succeeded
Installing dependencies from Pipfile.lock (d573fd)...
  ================================ 0/0 - 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5254
$ pipenv run python -c "import platformdirs"
```

I am not sure why it would be preferring what is in site-packages and not adding it the virtualenv.   I looked over the changes from the two versions and it doesn't stand out to me yet what would be causing it.

Resolving patch:
diff --git a/pipenv/core.py b/pipenv/core.py
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -1,5 +1,3 @@
-from __future__ import annotations
-
 import json as simplejson
 import logging
 import os
@@ -10,6 +8,7 @@
 import warnings
 from pathlib import Path
 from posixpath import expandvars
+from typing import Dict, List, Optional, Union
 
 import dotenv
 import pipfile
@@ -24,6 +23,7 @@
     install_req_from_parsed_requirement,
 )
 from pipenv.patched.pip._internal.req.req_file import parse_requirements
+from pipenv.project import Project
 from pipenv.utils.constants import MYPY_RUNNING
 from pipenv.utils.dependencies import (
     convert_deps_to_pip,
@@ -50,12 +50,9 @@
 )
 from pipenv.utils.spinner import create_spinner
 from pipenv.vendor import click
+from pipenv.vendor.requirementslib.models.requirements import Requirement
 
 if MYPY_RUNNING:
-    from typing import Dict, List, Optional, Union
-
-    from pipenv.project import Project
-    from pipenv.vendor.requirementslib.models.requirements import Requirement
 
     TSourceDict = Dict[str, Union[str, bool]]
 
@@ -1211,7 +1208,7 @@ def do_purge(project, bare=False, downloads=False, allow_global=False):
         click.echo(fix_utf8(f"Found {len(to_remove)} installed package(s), purging..."))
 
     command = [
-        project_python(project),
+        project_python(project, system=allow_global),
         _get_runnable_pip(),
         "uninstall",
         "-y",
@@ -1529,9 +1526,10 @@ def pip_install(
             )
 
     pip_command = [
-        project._which("python", allow_global=allow_global),
+        project_python(project, system=allow_global),
         _get_runnable_pip(),
         "install",
+        "--ignore-installed",
     ]
     pip_args = get_pip_args(
         project,
@@ -2363,7 +2361,7 @@ def do_uninstall(
         if package_name in packages_to_remove:
             with project.environment.activated():
                 cmd = [
-                    project_python(project),
+                    project_python(project, system=system),
                     _get_runnable_pip(),
                     "uninstall",
                     package_name,
@@ -2669,7 +2667,7 @@ def do_check(
     safety_path = os.path.join(
         os.path.dirname(os.path.abspath(__file__)), "patched", "safety"
     )
-    _cmd = [project_python(project)]
+    _cmd = [project_python(project, system=system)]
     # Run the PEP 508 checker in the virtualenv.
     cmd = _cmd + [Path(pep508checker_path).as_posix()]
     c = run_command(cmd, is_verbose=project.s.is_verbose())
@@ -3017,7 +3015,7 @@ def do_clean(
                 )
             # Uninstall the package.
             cmd = [
-                project_python(project),
+                project_python(project, system=system),
                 _get_runnable_pip(),
                 "uninstall",
                 apparent_bad_package,


Test patch: