Repository: pypa/pipenv

Issue:
pipenv clean does not clean packages from the virtual env, but some packages from the global env
##### Issue description

I am trying to run `pipenv clean` after removing a package from Pipfile and running `pipfile lock`

##### Expected result

That the removed packaged is uninstalled by pipenv clean.

##### Actual result

Pipenv actually tries to remove 4 packages from the global env: pipenv, virtualenv, virtualenv-clone and certifi. It doesn't actually remove them. Maybe it's actually removing them from another virtualenv ?

##### Steps to replicate

Create a project, add a package with pipenv install.

Remove it by editing the Pipfile. 

Run pipfile lock to update Pipfile.lock

Run pipfile clean to remove the package you edited out of Pipfile.

Pipenv clean --dry-run just lists the names of the 4 packages it tries to remove: pipenv, virtualenv, virtualenv-clone, certifi

I then run pip list and the package I removed from Pipfile is still there in the venv. The 4 packages it tried to remove are not even installed in the venv.

I have tested this on both the latest pip release and pipenv master, on Windows. I've tested on both  cmd.exe and git bash. Yeah I know bash is not officially supported, that's why I reproed on cmd.exe

This happens in the venv shell or not, and by running 'pipenv run pipenv clean' too.

Update:
Amusingly, if I do install one of the four packages in the venv, it does uninstall it, which points to the code that gets that list as being faulty. The code that tries to uninstall the packages from that list is at fault for failing silently, though.

I can try to repro on Mac OS later tonight

-------------------------------------------------------------------------------


<details><summary>$ pipenv --support</summary>

Pipenv version: `'2018.7.1.dev0'`

Pipenv location: `'C:\\Users\\safco106950\\AppData\\Roaming\\Python\\Python37\\site-packages\\pipenv'`

Python location: `'c:\\users\\safco106950\\appdata\\local\\programs\\python\\python37\\python.exe'`

Python installations found:

  - `3.7.0`: `C:\Users\safco106950\AppData\Local\Programs\Python\Python37\python.exe`
  - `3.6.5`: `C:\Users\safco106950\AppData\Local\Programs\Python\Python36-32\python.exe`
  - `3.5`: `C:\Users\safco106950\AppData\Local\Programs\Python\Python35\python.exe`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '7',
 'platform_system': 'Windows',
 'platform_version': '6.1.7601',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'win32'}
```

System environment variables:

  - `ALLUSERSPROFILE`
  - `APPDATA`
  - `CLASSPATH`
  - `COMMONPROGRAMFILES`
  - `COMMONPROGRAMFILES(X86)`
  - `COMMONPROGRAMW6432`
  - `COMPUTERNAME`
  - `COMSPEC`
  - `DADIR`
  - `DEFLOGDIR`
  - `FP_NO_HOST_CHECK`
  - `HOMEDRIVE`
  - `HOMEPATH`
  - `LOCALAPPDATA`
  - `LOGONSERVER`
  - `NUMBER_OF_PROCESSORS`
  - `OS`
  - `PATH`
  - `PATHEXT`
  - `PIPENV_ACTIVE`
  - `PIP_PYTHON_PATH`
  - `PROCESSOR_ARCHITECTURE`
  - `PROCESSOR_IDENTIFIER`
  - `PROCESSOR_LEVEL`
  - `PROCESSOR_REVISION`
  - `PROGRAMDATA`
  - `PROGRAMFILES`
  - `PROGRAMFILES(X86)`
  - `PROGRAMW6432`
  - `PROMPT`
  - `PSMODULEPATH`
  - `PUBLIC`
  - `PYTHONDONTWRITEBYTECODE`
  - `QTJAVA`
  - `SAPLOGON_INI_FILE`
  - `SESSIONNAME`
  - `SYSTEMDRIVE`
  - `SYSTEMROOT`
  - `TEMP`
  - `TMP`
  - `UATDATA`
  - `USERDNSDOMAIN`
  - `USERDOMAIN`
  - `USERDOMAIN_ROAMINGPROFILE`
  - `USERNAME`
  - `USERPROFILE`
  - `VIRTUAL_ENV`
  - `VSEDEFLOGDIR`
  - `WINDIR`
  - `WINDOWS_TRACING_FLAGS`
  - `WINDOWS_TRACING_LOGFILE`

Pipenvûspecific environment variables:

 - `PIPENV_ACTIVE`: `1`

Debugûspecific environment variables:

  - `PATH`: `C:\Users\safco106950\.virtualenvs\jupyter_test-UM4uSYuv\Scripts;C:\Users\safco106950\AppData\Roaming\Python\Python37\Scripts;C:\Users\safco106950\AppData\Local\Programs\Python\Python37\Scripts;C:\users-apps\Oracle\ora1123\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Common Files\Hitachi ID;C:\Program Files (x86)\Microsoft Application Virtualization Client;C:\Program Files (x86)\QuickTime\QTSystem\;C:\Program Files (x86)\Enterprise Vault\EVClient\;C:\Program Files (x86)\CheckPoint\Endpoint Security\Endpoint Common\bin;C:\Users\safco106950\AppData\Local\Programs\Python\Launcher\;C:\Users\safco106950\AppData\Local\Programs\Python\Python36-32\Scripts\;C:\Users\safco106950\AppData\Local\Programs\Python\Python36-32\;C:\Users\safco106950\AppData\Local\atom\bin;C:\Users\safco106950\AppData\Local\Pandoc\`
  - `VIRTUAL_ENV`: `C:\Users\safco106950\.virtualenvs\jupyter_test-UM4uSYuv`


---------------------------

Contents of `Pipfile` ('C:\\Users\\safco106950\\work\\jupyter_test\\Pipfile'):

```toml
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.6"

```


Contents of `Pipfile.lock` ('C:\\Users\\safco106950\\work\\jupyter_test\\Pipfile.lock'):

```json
{
    "_meta": {
        "hash": {
            "sha256": "415dfdcb118dd9bdfef17671cb7dcd78dbd69b6ae7d4f39e8b44e71d60ca72e7"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

```
</details>



Discussion:
Just thought of adding verbose to clean:

```
λ pipenv clean -vvv               
Ignoring 'pip'.                   
Ignoring 'setuptools'.            
Uninstalling 'pipenv'…            
Uninstalling 'virtualenv'…        
Uninstalling 'virtualenv-clone'…  
Uninstalling 'certifi'…           
```

Doesn't help all that much.
I assume you have global packages enabled in the virtualenv. Pipenv uses pip for removal, and pip just ignores non-local packages when you try to uninstall them, so my guess is nothing is not actually removed, those are just no-op.

Where are those packages actually installed at? Try printing their `__file__` attribute in Python.
Those packages are not in the local venv:

```
(jupyter_test-UM4uSYuv) C:\Users\safco106950\work\jupyter_test>python -c "import
 certifi"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'certifi'
```

After "exit", going back to global python:

```
C:\Users\safco106950\work\jupyter_test>python -c "import certifi; print(certifi.
__file__)"
C:\Users\safco106950\AppData\Local\Programs\Python\Python36-32\lib\site-packages
\certifi\__init__.py
```
As expected

The fact that they're not in the local venv is further shown by the experiment described above, where I install virtualenv in the local venv using pip, then run pipenv clean. This now succeeds in uninstalling virtualenv (and only virtual env, even if it prints the same 4 as always) from the local venv.
The output of "pipenv run pip list" doesn't show any of those 4. There are lots of packages in the venv, whereas my Pipfile is empty. I've tried with a non-empty pipfile, to check it's not caused by that.

Resolving patch:
diff --git a/pipenv/core.py b/pipenv/core.py
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -2327,8 +2327,6 @@ def do_check(
 
 def do_graph(bare=False, json=False, json_tree=False, reverse=False):
     import pipdeptree
-
-
     try:
         python_path = which("python")
     except AttributeError:
@@ -2390,7 +2388,7 @@ def do_graph(bare=False, json=False, json_tree=False, reverse=False):
             err=True,
         )
         sys.exit(1)
-    cmd = '"{0}" {1} {2}'.format(
+    cmd = '"{0}" {1} {2} -l'.format(
         python_path, escape_grouped_arguments(pipdeptree.__file__.rstrip("cdo")), flag
     )
     # Run dep-tree.
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -9,6 +9,7 @@
 import hashlib
 import contoml
 from first import first
+from cached_property import cached_property
 import pipfile
 import pipfile.api
 import six
@@ -279,26 +280,47 @@ def get_location_for_virtualenv(self):
                     return vistir.compat.Path(name).absolute().as_posix()
         return str(get_workon_home().joinpath(name))
 
-    def get_installed_packages(self):
-        from . import PIPENV_ROOT, PIPENV_VENDOR, PIPENV_PATCHED
-        from .utils import temp_path, load_path, temp_environ
+    @property
+    def working_set(self):
+        from .utils import load_path
+        sys_path = load_path(self.which("python"))
+        import pkg_resources
+        return pkg_resources.WorkingSet(sys_path)
+
+    def find_egg(self, egg_dist):
+        import site
+        from distutils import sysconfig as distutils_sysconfig
+        site_packages = distutils_sysconfig.get_python_lib()
+        search_filename = "{0}.egg-link".format(egg_dist.project_name)
+        try:
+            user_site = site.getusersitepackages()
+        except AttributeError:
+            user_site = site.USER_SITE
+        search_locations = [site_packages, user_site]
+        for site_directory in search_locations:
+                egg = os.path.join(site_directory, search_filename)
+                if os.path.isfile(egg):
+                    return egg
+
+    def locate_dist(self, dist):
+        location = self.find_egg(dist)
+        if not location:
+            return dist.location
+
+    def dist_is_in_project(self, dist):
+        prefix = _normalized(self.env_paths["prefix"])
+        location = self.locate_dist(dist)
+        if not location:
+            return False
+        return _normalized(location).startswith(prefix)
 
+    def get_installed_packages(self):
+        workingset = self.working_set
         if self.virtualenv_exists:
-            with temp_path(), temp_environ():
-                new_path = load_path(self.which("python"))
-                new_path = [
-                    new_path[0],
-                    PIPENV_ROOT,
-                    PIPENV_PATCHED,
-                    PIPENV_VENDOR,
-                ] + new_path[1:]
-                sys.path = new_path
-                os.environ["VIRTUAL_ENV"] = self.virtualenv_location
-                from .vendor.pip_shims.shims import get_installed_distributions
-
-                return get_installed_distributions(local_only=True)
+            packages = [pkg for pkg in workingset if self.dist_is_in_project(pkg)]
         else:
-            return []
+            packages = [pkg for pkg in packages]
+        return packages
 
     @classmethod
     def _sanitize(cls, name):
@@ -903,3 +925,34 @@ def proper_case_section(self, section):
                 del section[dep]
         # Return whether or not values have been changed.
         return changed_values
+
+    @property
+    def _pyversion(self):
+        include_dir = vistir.compat.Path(self.virtualenv_location) / "include"
+        python_path = next(iter(list(include_dir.iterdir())), None)
+        if python_path and python_path.name.startswith("python"):
+            python_version = python_path.name.replace("python", "")
+            py_version_short, abiflags = python_version[:3], python_version[3:]
+            return {"py_version_short": py_version_short, "abiflags": abiflags}
+        return {}
+
+    @property
+    def env_paths(self):
+        import sysconfig
+        location = self.virtualenv_location if self.virtualenv_location else sys.prefix
+        prefix = vistir.compat.Path(location).as_posix()
+        scheme = sysconfig._get_default_scheme()
+        config = {
+            "base": prefix,
+            "installed_base": prefix,
+            "platbase": prefix,
+            "installed_platbase": prefix
+        }
+        config.update(self._pyversion)
+        paths = {
+            k: v.format(**config)
+            for k, v in sysconfig._INSTALL_SCHEMES[scheme].items()
+        }
+        if "prefix" not in paths:
+            paths["prefix"] = prefix
+        return paths


Test patch:
diff --git a/tests/integration/test_cli.py b/tests/integration/test_cli.py
--- a/tests/integration/test_cli.py
+++ b/tests/integration/test_cli.py
@@ -103,8 +103,13 @@ def test_pipenv_clean_pip_no_warnings(PipenvInstance):
     with PipenvInstance(chdir=True) as p:
         with open('setup.py', 'w') as f:
             f.write('from setuptools import setup; setup(name="empty")')
-        p.pipenv('run pip install -e .')
-        assert p.pipenv('clean').out
+        c = p.pipenv('install -e .')
+        assert c.return_code == 0
+        c = p.pipenv('run pip install pytz')
+        assert c.return_code == 0
+        c = p.pipenv('clean')
+        assert c.return_code == 0
+        assert c.out, "{0} -- STDERR: {1}".format(c.out, c.err)
 
 
 @pytest.mark.cli
@@ -114,8 +119,11 @@ def test_pipenv_clean_pip_warnings(PipenvInstance):
             f.write('from setuptools import setup; setup(name="empty")')
         # create a fake git repo to trigger a pip freeze warning
         os.mkdir('.git')
-        p.pipenv('run pip install -e .')
-        assert p.pipenv('clean').out
+        c = p.pipenv("run pip install -e .")
+        assert c.return_code == 0
+        c = p.pipenv('clean')
+        assert c.return_code == 0
+        assert c.err
 
 
 @pytest.mark.cli