Repository: jazzband/pip-tools

Issue:
pip-sync on fresh conda environment with Python 2.7.15 tries to remove Python package
When running pip-sync in a conda environment with python 2.7.15 installed package, it always tries to uninstall non-pip packages like `python`, failing to execute.
It runs ok on conda environments with python 3.x.

##### Environment Versions

1. OS Type: `MacOSX 10.13.6`
1. Python version: `Python 2.7.15 :: Anaconda, Inc.`
1. pip version:  `pip 18.1 from /anaconda2/envs/my-env/lib/python2.7/site-packages/pip (python 2.7)`
1. pip-tools version: `pip-compile, version 3.1.0`

##### Steps to replicate

1. conda create -n my-env python=2.7
2. source activate my-env
3. pip install -U pip
4. pip install pip-tools
5. echo "django" > requirements.in
6. pip-compile requirements.in
7. pip-sync requirements.txt

##### Expected result

pip-sync execution successful. A clean environment with just Django installed to the latest possible version.

##### Actual result

pip-sync error:

````
Cannot uninstall 'Python'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Traceback (most recent call last):
  File "/anaconda2/envs/my-env/bin/pip-sync", line 11, in <module>
    sys.exit(cli())
  File "/anaconda2/envs/my-env/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/anaconda2/envs/my-env/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/anaconda2/envs/my-env/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/anaconda2/envs/my-env/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/anaconda2/envs/my-env/lib/python2.7/site-packages/piptools/scripts/sync.py", line 74, in cli
    install_flags=install_flags))
  File "/anaconda2/envs/my-env/lib/python2.7/site-packages/piptools/sync.py", line 149, in sync
    check_call([pip, 'uninstall', '-y'] + pip_flags + sorted(to_uninstall))
  File "/anaconda2/envs/my-env/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/anaconda2/envs/my-env/bin/pip', 'uninstall', '-y', u'certifi', u'python', u'wsgiref']' returned non-zero exit status 1
````


Discussion:
Sorry, it was me the one from the previous post. Didn't notice I was logged in as one of our bot accounts 😝 
I am having the same issue. 

I have tried this with fresh conda environments, old ones, simple requirements.txt or complicated ones. All of them fail. 

1. OS Type: MacOSX 10.14
2. Python Version: Python 2.7.14 |Anaconda, Inc.
3. pip version: pip 10.0.1 from /Users/username_here/miniconda3/envs/api/lib/python2.7/site-packages/pip (python 2.7)
4. pip-tools version: pip-tools==3.1.0

**Steps to Replicate:**

1. Make a fresh conda environment with `conda create --name example python=2.7.14 pip`
2. `pip install pip-tools`
3. navigate to my directory with a `requirements.txt` in it
4. pip-sync

**Expected result:**
Install all of the python packages in requirements.txt

**Actual results:**

```
$ pip-sync
Cannot uninstall 'Python'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/Users/peterhoburg/miniconda3/envs/api/bin/pip-sync", line 11, in <module>
    sys.exit(cli())
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/piptools/scripts/sync.py", line 74, in cli
    install_flags=install_flags))
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/site-packages/piptools/sync.py", line 149, in sync
    check_call([pip, 'uninstall', '-y'] + pip_flags + sorted(to_uninstall))
  File "/Users/peterhoburg/miniconda3/envs/api/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Users/peterhoburg/miniconda3/envs/api/bin/pip', 'uninstall', '-y', u'python', u'wsgiref']' returned non-zero exit status 1
```
This is reproducible in a generic Python 2.7 Docker container as well. Rolling back to pip 9.x  is a work-around:

1.  Start a Python 2.7 Docker container

    ```sh
    docker run --rm -it python:2.7 /bin/bash
    ```

1.  Try to install requirements using stock pip

    ```sh
    $ pip --version
    pip 18.1 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)

    $ cd /tmp

    $ echo "requests" > requirements.in

    $ pip install pip-tools
    Collecting pip-tools
      Downloading https://files.pythonhosted.org/packages/4a/16/7d7e908a657b8b41ce665e38caaa1a4415707bdb7b06524deec9802f0b1c/pip_tools-3.2.0-py2.py3-none-any.whl (43kB)
        100% |████████████████████████████████| 51kB 3.4MB/s
    Collecting click>=6 (from pip-tools)
      Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
        100% |████████████████████████████████| 81kB 6.9MB/s
    Collecting six (from pip-tools)
      Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
    Installing collected packages: click, six, pip-tools
    Successfully installed click-7.0 pip-tools-3.2.0 six-1.12.0

    $ pip-compile
    #
    # This file is autogenerated by pip-compile
    # To update, run:
    #
    #    pip-compile --output-file requirements.txt requirements.in
    #
    certifi==2018.11.29       # via requests
    chardet==3.0.4            # via requests
    idna==2.8                 # via requests
    requests==2.21.0
    urllib3==1.24.1           # via requests

    $ pip-sync
    Cannot uninstall 'Python'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
    Traceback (most recent call last):
      File "/usr/local/bin/pip-sync", line 11, in <module>
        sys.exit(cli())
      File "/usr/local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
        return self.main(*args, **kwargs)
      File "/usr/local/lib/python2.7/site-packages/click/core.py", line 717, in main
        rv = self.invoke(ctx)
      File "/usr/local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/usr/local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
        return callback(*args, **kwargs)
      File "/usr/local/lib/python2.7/site-packages/piptools/scripts/sync.py", line 74, in cli
        install_flags=install_flags))
      File "/usr/local/lib/python2.7/site-packages/piptools/sync.py", line 149, in sync
        check_call([pip, 'uninstall', '-y'] + pip_flags + sorted(to_uninstall))
      File "/usr/local/lib/python2.7/subprocess.py", line 190, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/usr/local/bin/pip', 'uninstall', '-y', u'python', u'virtualenv', u'wsgiref']' returned non-zero exit status 1
    ```

1.  Roll back to pip 9.x

    ```sh
    $ pip install "pip<10"
    Collecting pip<10
      Downloading https://files.pythonhosted.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl (1.4MB)
        100% |████████████████████████████████| 1.4MB 8.5MB/s
    Installing collected packages: pip
      Found existing installation: pip 18.1
        Uninstalling pip-18.1:
          Successfully uninstalled pip-18.1
    Successfully installed pip-9.0.3

    $ pip-sync
    DEPRECATION: Uninstalling a distutils installed project (python) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling Python-2.7.15:
      Successfully uninstalled Python-2.7.15
    Uninstalling virtualenv-16.1.0:
      Successfully uninstalled virtualenv-16.1.0
    Not uninstalling wsgiref at /usr/local/lib/python2.7, as it is in the standard library.
    You are using pip version 9.0.3, however version 18.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    Collecting certifi==2018.11.29
      Cache entry deserialization failed, entry ignored
      Cache entry deserialization failed, entry ignored
      Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
        100% |████████████████████████████████| 163kB 4.6MB/s
    Collecting chardet==3.0.4
      Cache entry deserialization failed, entry ignored
      Cache entry deserialization failed, entry ignored
      Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
        100% |████████████████████████████████| 143kB 5.3MB/s
    Collecting idna==2.8
      Cache entry deserialization failed, entry ignored
      Cache entry deserialization failed, entry ignored
      Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
        100% |████████████████████████████████| 61kB 5.0MB/s
    Collecting requests==2.21.0
      Cache entry deserialization failed, entry ignored
      Cache entry deserialization failed, entry ignored
      Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
        100% |████████████████████████████████| 61kB 10.0MB/s
    Collecting urllib3==1.24.1
      Cache entry deserialization failed, entry ignored
      Cache entry deserialization failed, entry ignored
      Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
        100% |████████████████████████████████| 122kB 6.4MB/s
    Installing collected packages: certifi, chardet, idna, urllib3, requests
    Successfully installed certifi-2018.11.29 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.1
    You are using pip version 9.0.3, however version 18.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    ```

Resolving patch:
diff --git a/piptools/_compat/__init__.py b/piptools/_compat/__init__.py
--- a/piptools/_compat/__init__.py
+++ b/piptools/_compat/__init__.py
@@ -29,4 +29,6 @@
     PyPI,
     install_req_from_line,
     install_req_from_editable,
+    stdlib_pkgs,
+    DEV_PKGS,
 )
diff --git a/piptools/_compat/pip_compat.py b/piptools/_compat/pip_compat.py
--- a/piptools/_compat/pip_compat.py
+++ b/piptools/_compat/pip_compat.py
@@ -36,6 +36,8 @@ def do_import(module_path, subimport=None, old_path=None):
 cmdoptions = do_import('cli.cmdoptions', old_path='cmdoptions')
 get_installed_distributions = do_import('utils.misc', 'get_installed_distributions', old_path='utils')
 PyPI = do_import('models.index', 'PyPI')
+stdlib_pkgs = do_import('utils.compat', 'stdlib_pkgs', old_path='compat')
+DEV_PKGS = do_import('commands.freeze', 'DEV_PKGS')
 
 # pip 18.1 has refactored InstallRequirement constructors use by pip-tools.
 if pkg_resources.parse_version(pip.__version__) < pkg_resources.parse_version('18.1'):
diff --git a/piptools/sync.py b/piptools/sync.py
--- a/piptools/sync.py
+++ b/piptools/sync.py
@@ -4,6 +4,7 @@
 import tempfile
 from subprocess import check_call
 
+from piptools._compat import stdlib_pkgs, DEV_PKGS
 from . import click
 from .exceptions import IncompatibleRequirements, UnsupportedConstraint
 from .utils import flat_map, format_requirement, key_from_ireq, key_from_req, get_hashes_from_ireq
@@ -14,9 +15,7 @@
     'pip-tools',
     'pip-review',
     'pkg-resources',
-    'setuptools',
-    'wheel',
-]
+] + list(stdlib_pkgs) + list(DEV_PKGS)
 
 
 def dependency_tree(installed_keys, root_key):


Test patch:
diff --git a/tests/test_sync.py b/tests/test_sync.py
--- a/tests/test_sync.py
+++ b/tests/test_sync.py
@@ -111,7 +111,8 @@ def test_diff_should_uninstall(fake_dist):
 
 def test_diff_should_not_uninstall(fake_dist):
     ignored = ('pip==7.1.0', 'pip-tools==1.1.1', 'pip-review==1.1.1',
-               'pkg-resources==0.0.0', 'setuptools==34.0.0', 'wheel==0.29.0')
+               'pkg-resources==0.0.0', 'setuptools==34.0.0', 'wheel==0.29.0',
+               'python==3.0', 'distribute==0.1', 'wsgiref==0.1', 'argparse==0.1')
     installed = [fake_dist(pkg) for pkg in ignored]
     reqs = []