Repository: pypa/pipenv

Issue:
pipenv clean fails if PyGObject is installed inside the venv
### Issue description

pipenv clean fails (throws exception) when executed with PyGObject installed inside the virtual environment.

<del>It is possible that there are other packages on PyPI that can trigger this behavior, but so far I haven't found any other. I don't know what's different about the PyGObject package that causes this, either.</del>

Update: it looks like any package that installs data files into the _include_ subdirectory of the virtual environment could trigger the problem.

### Expected result

If the Pipfile doesn't contain PyGObject, or any other package that depends on it, I expect pipenv clean to uninstall it.

### Actual result

```
$ pipenv clean -v
Traceback (most recent call last):
  File "/home/dsimon/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/cli/command.py", line 602, in clean
    do_clean(ctx=ctx, three=state.three, python=state.python, dry_run=dry_run)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/core.py", line 2526, in do_clean
    canonicalize_name(pkg.project_name) for pkg in project.get_installed_packages()
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 320, in get_installed_packages
    packages = [pkg for pkg in workingset if self.dist_is_in_project(pkg)]
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 320, in <listcomp>
    packages = [pkg for pkg in workingset if self.dist_is_in_project(pkg)]
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 311, in dist_is_in_project
    prefix = _normalized(self.env_paths["prefix"])
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 951, in env_paths
    for k, v in sysconfig._INSTALL_SCHEMES[scheme].items()
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 951, in <dictcomp>
    for k, v in sysconfig._INSTALL_SCHEMES[scheme].items()
KeyError: 'py_version_short'
```

### Steps to replicate

Edit: PyGObject's dependencies must be installed (e.g. with the distro's package manager) before attempting to install it with pip. On Ubuntu, these are: `libcairo-dev libdbus-1-dev libgirepository1.0-dev`.


```bash
# 1. create a new environment with no packages
mkdir temp && cd temp
pipenv install
# 2. install PyGObject inside the virtual environment
pipenv run pip install pygobject
# 3. try to run clean
pipenv clean
```

-------------------------------------------------------------------------------

<details><summary>$ pipenv --support</summary>

Pipenv version: `'2018.10.13'`

Pipenv location: `'/home/dsimon/.local/lib/python3.6/site-packages/pipenv'`

Python location: `'/usr/bin/python3'`

Python installations found:

  - `3.7.0`: `/usr/bin/python3.7m`
  - `3.7.0`: `/usr/bin/python3.7`
  - `3.6.6`: `/usr/bin/python3.6m`
  - `3.6.6`: `/usr/bin/python3.6`
  - `3.5.6`: `/usr/bin/python3.5`
  - `3.5.6`: `/usr/bin/python3.5m`
  - `3.4.9`: `/usr/bin/python3.4m`
  - `3.4.9`: `/usr/bin/python3.4`
  - `3.3.7`: `/usr/bin/python3.3`
  - `3.3.7`: `/usr/bin/python3.3m`
  - `2.7.15`: `/usr/bin/python2.7`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.18.13-200.fc28.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Oct 10 17:29:59 UTC 2018',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'linux'}
```

System environment variables:

  - `QTINC`
  - `LS_COLORS`
  - `MC_SKIN`
  - `XDG_MENU_PREFIX`
  - `MODULES_RUN_QUARANTINE`
  - `LANG`
  - `GDM_LANG`
  - `HISTCONTROL`
  - `DISPLAY`
  - `GUESTFISH_RESTORE`
  - `HOSTNAME`
  - `QTDIR`
  - `EDITOR`
  - `COLORTERM`
  - `GUESTFISH_INIT`
  - `USERNAME`
  - `XDG_VTNR`
  - `SSH_AUTH_SOCK`
  - `XDG_SESSION_ID`
  - `MODULES_CMD`
  - `USER`
  - `ENV`
  - `GUESTFISH_PS1`
  - `DESKTOP_SESSION`
  - `WAYLAND_DISPLAY`
  - `GNOME_TERMINAL_SCREEN`
  - `PWD`
  - `HOME`
  - `XDG_SESSION_TYPE`
  - `BASH_ENV`
  - `XDG_DATA_DIRS`
  - `XDG_SESSION_DESKTOP`
  - `GJS_DEBUG_OUTPUT`
  - `LOADEDMODULES`
  - `MAIL`
  - `VISUAL`
  - `QTLIB`
  - `TERM`
  - `VTE_VERSION`
  - `SHELL`
  - `QT_IM_MODULE`
  - `XMODIFIERS`
  - `XDG_CURRENT_DESKTOP`
  - `GNOME_TERMINAL_SERVICE`
  - `SHLVL`
  - `XDG_SEAT`
  - `MODULEPATH`
  - `GDMSESSION`
  - `GNOME_DESKTOP_SESSION_ID`
  - `LOGNAME`
  - `DBUS_SESSION_BUS_ADDRESS`
  - `XDG_RUNTIME_DIR`
  - `MODULEPATH_modshare`
  - `PATH`
  - `GUESTFISH_OUTPUT`
  - `MODULESHOME`
  - `HISTSIZE`
  - `GJS_DEBUG_TOPICS`
  - `SESSION_MANAGER`
  - `LESSOPEN`
  - `BASH_FUNC_module%%`
  - `BASH_FUNC_switchml%%`
  - `BASH_FUNC_scl%%`
  - `BASH_FUNC__moduleraw%%`
  - `_`
  - `OLDPWD`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_SHIMS_BASE_MODULE`
  - `PIP_PYTHON_PATH`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/opt/totalphase/datacenter/bin:/home/dsimon/bin:/home/dsimon/.local/bin:/opt/totalphase/datacenter/bin:/home/dsimon/bin:/home/dsimon/.local/bin:/usr/lib64/qt-3.3/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin`
  - `SHELL`: `/bin/bash`
  - `EDITOR`: `vim`
  - `LANG`: `en_US.UTF-8`
  - `PWD`: `/home/dsimon/temp`


---------------------------

Contents of `Pipfile` ('/home/dsimon/temp/Pipfile'):

```toml
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.6"

```


Contents of `Pipfile.lock` ('/home/dsimon/temp/Pipfile.lock'):

```json
{
    "_meta": {
        "hash": {
            "sha256": "415dfdcb118dd9bdfef17671cb7dcd78dbd69b6ae7d4f39e8b44e71d60ca72e7"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

```
</details>


pipenv clean fails if PyGObject is installed inside the venv
### Issue description

pipenv clean fails (throws exception) when executed with PyGObject installed inside the virtual environment.

<del>It is possible that there are other packages on PyPI that can trigger this behavior, but so far I haven't found any other. I don't know what's different about the PyGObject package that causes this, either.</del>

Update: it looks like any package that installs data files into the _include_ subdirectory of the virtual environment could trigger the problem.

### Expected result

If the Pipfile doesn't contain PyGObject, or any other package that depends on it, I expect pipenv clean to uninstall it.

### Actual result

```
$ pipenv clean -v
Traceback (most recent call last):
  File "/home/dsimon/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/cli/command.py", line 602, in clean
    do_clean(ctx=ctx, three=state.three, python=state.python, dry_run=dry_run)
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/core.py", line 2526, in do_clean
    canonicalize_name(pkg.project_name) for pkg in project.get_installed_packages()
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 320, in get_installed_packages
    packages = [pkg for pkg in workingset if self.dist_is_in_project(pkg)]
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 320, in <listcomp>
    packages = [pkg for pkg in workingset if self.dist_is_in_project(pkg)]
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 311, in dist_is_in_project
    prefix = _normalized(self.env_paths["prefix"])
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 951, in env_paths
    for k, v in sysconfig._INSTALL_SCHEMES[scheme].items()
  File "/home/dsimon/.local/lib/python3.6/site-packages/pipenv/project.py", line 951, in <dictcomp>
    for k, v in sysconfig._INSTALL_SCHEMES[scheme].items()
KeyError: 'py_version_short'
```

### Steps to replicate

Edit: PyGObject's dependencies must be installed (e.g. with the distro's package manager) before attempting to install it with pip. On Ubuntu, these are: `libcairo-dev libdbus-1-dev libgirepository1.0-dev`.


```bash
# 1. create a new environment with no packages
mkdir temp && cd temp
pipenv install
# 2. install PyGObject inside the virtual environment
pipenv run pip install pygobject
# 3. try to run clean
pipenv clean
```

-------------------------------------------------------------------------------

<details><summary>$ pipenv --support</summary>

Pipenv version: `'2018.10.13'`

Pipenv location: `'/home/dsimon/.local/lib/python3.6/site-packages/pipenv'`

Python location: `'/usr/bin/python3'`

Python installations found:

  - `3.7.0`: `/usr/bin/python3.7m`
  - `3.7.0`: `/usr/bin/python3.7`
  - `3.6.6`: `/usr/bin/python3.6m`
  - `3.6.6`: `/usr/bin/python3.6`
  - `3.5.6`: `/usr/bin/python3.5`
  - `3.5.6`: `/usr/bin/python3.5m`
  - `3.4.9`: `/usr/bin/python3.4m`
  - `3.4.9`: `/usr/bin/python3.4`
  - `3.3.7`: `/usr/bin/python3.3`
  - `3.3.7`: `/usr/bin/python3.3m`
  - `2.7.15`: `/usr/bin/python2.7`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.18.13-200.fc28.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Oct 10 17:29:59 UTC 2018',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'linux'}
```

System environment variables:

  - `QTINC`
  - `LS_COLORS`
  - `MC_SKIN`
  - `XDG_MENU_PREFIX`
  - `MODULES_RUN_QUARANTINE`
  - `LANG`
  - `GDM_LANG`
  - `HISTCONTROL`
  - `DISPLAY`
  - `GUESTFISH_RESTORE`
  - `HOSTNAME`
  - `QTDIR`
  - `EDITOR`
  - `COLORTERM`
  - `GUESTFISH_INIT`
  - `USERNAME`
  - `XDG_VTNR`
  - `SSH_AUTH_SOCK`
  - `XDG_SESSION_ID`
  - `MODULES_CMD`
  - `USER`
  - `ENV`
  - `GUESTFISH_PS1`
  - `DESKTOP_SESSION`
  - `WAYLAND_DISPLAY`
  - `GNOME_TERMINAL_SCREEN`
  - `PWD`
  - `HOME`
  - `XDG_SESSION_TYPE`
  - `BASH_ENV`
  - `XDG_DATA_DIRS`
  - `XDG_SESSION_DESKTOP`
  - `GJS_DEBUG_OUTPUT`
  - `LOADEDMODULES`
  - `MAIL`
  - `VISUAL`
  - `QTLIB`
  - `TERM`
  - `VTE_VERSION`
  - `SHELL`
  - `QT_IM_MODULE`
  - `XMODIFIERS`
  - `XDG_CURRENT_DESKTOP`
  - `GNOME_TERMINAL_SERVICE`
  - `SHLVL`
  - `XDG_SEAT`
  - `MODULEPATH`
  - `GDMSESSION`
  - `GNOME_DESKTOP_SESSION_ID`
  - `LOGNAME`
  - `DBUS_SESSION_BUS_ADDRESS`
  - `XDG_RUNTIME_DIR`
  - `MODULEPATH_modshare`
  - `PATH`
  - `GUESTFISH_OUTPUT`
  - `MODULESHOME`
  - `HISTSIZE`
  - `GJS_DEBUG_TOPICS`
  - `SESSION_MANAGER`
  - `LESSOPEN`
  - `BASH_FUNC_module%%`
  - `BASH_FUNC_switchml%%`
  - `BASH_FUNC_scl%%`
  - `BASH_FUNC__moduleraw%%`
  - `_`
  - `OLDPWD`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_SHIMS_BASE_MODULE`
  - `PIP_PYTHON_PATH`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/opt/totalphase/datacenter/bin:/home/dsimon/bin:/home/dsimon/.local/bin:/opt/totalphase/datacenter/bin:/home/dsimon/bin:/home/dsimon/.local/bin:/usr/lib64/qt-3.3/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin`
  - `SHELL`: `/bin/bash`
  - `EDITOR`: `vim`
  - `LANG`: `en_US.UTF-8`
  - `PWD`: `/home/dsimon/temp`


---------------------------

Contents of `Pipfile` ('/home/dsimon/temp/Pipfile'):

```toml
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.6"

```


Contents of `Pipfile.lock` ('/home/dsimon/temp/Pipfile.lock'):

```json
{
    "_meta": {
        "hash": {
            "sha256": "415dfdcb118dd9bdfef17671cb7dcd78dbd69b6ae7d4f39e8b44e71d60ca72e7"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

```
</details>




Discussion:
Hi, see [my comment](https://github.com/pypa/pipenv/issues/3055#issuecomment-430989785) in #3055, it doesn't seem to be related to any specific dependency because I faced this issue for a newly created virtualenv (created with just a `pipenv run pip install pip==18.0`)
Hi @whren,
very interesting! I'm still trying to narrow down what's the actual root of the problem, but I already found that I can reproduce the same exception in `pipenv clean` by just manually creating a new file or directory inside the include directory of the venv created by pipenv. I'll report back if I find out anything more.
Looks like what's triggering the issue is that PyGObject installs _pygobject.h_ as a data file [into the include directory](https://github.com/GNOME/pygobject/blob/ad96fdc7ba2c500b158ba510c00e9054732cab1b/setup.py#L1199). If I create a mock package that does nothing but install something as a data file into _include_ and install that into pipenv's virtual environment, I can already reproduce the exception in pipenv clean. E.g. setup.py:
```python
from setuptools import setup

setup(
    name='repro',
    version='0.0.1',
    description='Reproducer for exception in pipenv clean',
    data_files=[('include', ['repro.h'])]
)
```
(You need to create repro.h in the same dir; its contents don't matter).
I totally agree, the header files (and maybe other platform-dependent files like .so or .dll) seem to cause the problems. `py_version_short` is the one used to get the correct directory for this (e.g. `3.6` in `lib/python3.6/site-packages`), and we’re not populating it correctly in certain code paths.
(I label this with `posix` since Windows don’t have these parameters, so it’s one thing to keep in mind when someone try to fix this.)
I think I've managed to track it down to [this function](https://github.com/pypa/pipenv/blob/87551b9d45e6f208fabb36ecdd81ce6d49ca98cf/pipenv/project.py#L927):
```python
    @property
    def _pyversion(self):
        include_dir = vistir.compat.Path(self.virtualenv_location) / "include"
        python_path = next(iter(list(include_dir.iterdir())), None)
        if python_path and python_path.name.startswith("python"):
            python_version = python_path.name.replace("python", "")
            py_version_short, abiflags = python_version[:3], python_version[3:]
            return {"py_version_short": py_version_short, "abiflags": abiflags}
        return {}
```
The code seems to assume that the name of the first file in the include directory starts with _python_ and ends with the version number and ABI flags. However, this is not always the case if some package puts additional files in there.
Hi, see [my comment](https://github.com/pypa/pipenv/issues/3055#issuecomment-430989785) in #3055, it doesn't seem to be related to any specific dependency because I faced this issue for a newly created virtualenv (created with just a `pipenv run pip install pip==18.0`)
Hi @whren,
very interesting! I'm still trying to narrow down what's the actual root of the problem, but I already found that I can reproduce the same exception in `pipenv clean` by just manually creating a new file or directory inside the include directory of the venv created by pipenv. I'll report back if I find out anything more.
Looks like what's triggering the issue is that PyGObject installs _pygobject.h_ as a data file [into the include directory](https://github.com/GNOME/pygobject/blob/ad96fdc7ba2c500b158ba510c00e9054732cab1b/setup.py#L1199). If I create a mock package that does nothing but install something as a data file into _include_ and install that into pipenv's virtual environment, I can already reproduce the exception in pipenv clean. E.g. setup.py:
```python
from setuptools import setup

setup(
    name='repro',
    version='0.0.1',
    description='Reproducer for exception in pipenv clean',
    data_files=[('include', ['repro.h'])]
)
```
(You need to create repro.h in the same dir; its contents don't matter).
I totally agree, the header files (and maybe other platform-dependent files like .so or .dll) seem to cause the problems. `py_version_short` is the one used to get the correct directory for this (e.g. `3.6` in `lib/python3.6/site-packages`), and we’re not populating it correctly in certain code paths.
(I label this with `posix` since Windows don’t have these parameters, so it’s one thing to keep in mind when someone try to fix this.)
I think I've managed to track it down to [this function](https://github.com/pypa/pipenv/blob/87551b9d45e6f208fabb36ecdd81ce6d49ca98cf/pipenv/project.py#L927):
```python
    @property
    def _pyversion(self):
        include_dir = vistir.compat.Path(self.virtualenv_location) / "include"
        python_path = next(iter(list(include_dir.iterdir())), None)
        if python_path and python_path.name.startswith("python"):
            python_version = python_path.name.replace("python", "")
            py_version_short, abiflags = python_version[:3], python_version[3:]
            return {"py_version_short": py_version_short, "abiflags": abiflags}
        return {}
```
The code seems to assume that the name of the first file in the include directory starts with _python_ and ends with the version number and ABI flags. However, this is not always the case if some package puts additional files in there.

Resolving patch:
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -926,8 +926,8 @@ def proper_case_section(self, section):
     @property
     def _pyversion(self):
         include_dir = vistir.compat.Path(self.virtualenv_location) / "include"
-        python_path = next(iter(list(include_dir.iterdir())), None)
-        if python_path and python_path.name.startswith("python"):
+        python_path = next((x for x in include_dir.iterdir() if x.name.startswith("python")), None)
+        if python_path:
             python_version = python_path.name.replace("python", "")
             py_version_short, abiflags = python_version[:3], python_version[3:]
             return {"py_version_short": py_version_short, "abiflags": abiflags}


Test patch: