Repository: jazzband/pip-tools

Issue:
pip-sync uninstalls pkg-resources, breaks psycopg2
`pip-sync` uninstalls `pkg-resources`, which in turn breaks installation of many other packages. `pkg-resources` is a new "system" package that was recently extracted from `setuptools` (since version 31, I believe). I think it must be handled similarly to `setuptools`.

##### Steps to replicate

On a fully updated Ubuntu 16.04 LTS:

```console
semenov@dev2:~/tmp$ rm -rf ~/.cache/pip
semenov@dev2:~/tmp$ virtualenv --python=$(which python3) test
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/semenov/tmp/test/bin/python3
Also creating executable in /home/semenov/tmp/test/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
semenov@dev2:~/tmp$ cd test
semenov@dev2:~/tmp/test$ . bin/activate
(test) semenov@dev2:~/tmp/test$ pip install pip-tools
Collecting pip-tools
  Downloading pip_tools-1.8.0-py2.py3-none-any.whl
Collecting six (from pip-tools)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting first (from pip-tools)
  Downloading first-2.0.1-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools)
  Downloading click-6.6-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 559kB/s
Installing collected packages: six, first, click, pip-tools
Successfully installed click-6.6 first-2.0.1 pip-tools-1.8.0 six-1.10.0
(test) semenov@dev2:~/tmp/test$ echo psycopg2 > requirements.in
(test) semenov@dev2:~/tmp/test$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
psycopg2==2.6.2
(test) semenov@dev2:~/tmp/test$ pip-sync
Uninstalling pkg-resources-0.0.0:
  Successfully uninstalled pkg-resources-0.0.0
Collecting psycopg2==2.6.2
  Downloading psycopg2-2.6.2.tar.gz (376kB)
    100% |████████████████████████████████| 378kB 2.4MB/s
Could not import setuptools which is required to install from a source distribution.
Traceback (most recent call last):
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/pip/req/req_install.py", line 387, in setup_py
    import setuptools  # noqa
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/setuptools/__init__.py", line 10, in <module>
    from setuptools.extern.six.moves import filter, filterfalse, map
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in <module>
    from pkg_resources.extern import VendorImporter
ImportError: No module named 'pkg_resources.extern'

Traceback (most recent call last):
  File "/home/semenov/tmp/test/bin/pip-sync", line 11, in <module>
    sys.exit(cli())
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/piptools/scripts/sync.py", line 72, in cli
    install_flags=install_flags))
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/piptools/sync.py", line 157, in sync
    check_call([pip, 'install'] + pip_flags + install_flags + sorted(to_install))
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'install', 'psycopg2==2.6.2']' returned non-zero exit status 1
```

##### Expected result

`pip-sync` keeps `pkg-resources` in place, `psycopg2` installs normally.

##### Actual result

`pip-sync` uninstalls `pkg-resources`, then `psycopg2` installation fails with: `ImportError: No module named 'pkg_resources.extern'`


Discussion:
Debian and derivatives have split pkg-resources from setuptools for a long time.  I ran into this issue in my first ten minutes of trying out pip-tools.
I have a similar issue:

`rnwolf  ~  workspace  cpptest4  pip-sync                      
Uninstalling pkg-resources-0.0.0:
  Successfully uninstalled pkg-resources-0.0.0

 rnwolf  ~  workspace  cpptest4  pip-compile > requirements.txt
Traceback (most recent call last):
  File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/bin/pip-compile", line 7, in <module>
    from piptools.scripts.compile import cli
  File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/lib/python3.5/site-packages/piptools/scripts/compile.py", line 16, in <module>
    from ..repositories import LocalRequirementsRepository, PyPIRepository
  File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/lib/python3.5/site-packages/piptools/repositories/__init__.py", line 3, in <module>
    from .pypi import PyPIRepository
  File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 14, in <module>
    from ..cache import CACHE_DIR
  File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/lib/python3.5/site-packages/piptools/cache.py", line 9, in <module>
    from pkg_resources import Requirement
ImportError: No module named 'pkg_resources'
`
Work-around: pip uninstall setuptools && pip install -U setuptools
A work-around that worked for me: add a line `pkg-resources==0.0.0` to `requirements.in`.

@merwok 's workaround didn't work for me.
> A work-around that worked for me: add a line pkg-resources==0.0.0 to requirements.in.

doesn't work for me.
`pip-sync` output:
```
Collecting pkg-resources==0.0.0
  Could not find a version that satisfies the requirement pkg-resources==0.0.0 (from versions: )
No matching distribution found for pkg-resources==0.0.0
```

@merwok 's workaround is ugly, but worked here.
Adding pkg-resources==0.0.0 to requirements.in worked for me. Tested in following way on debian stretch:
`mkvirtualenv tmp`
`mkdir tmp && cd tmp`
`echo pkg-resources==0.0.0 > requirements.in`
`pip install pip-tools`
`echo flask >> requirements.in`
`pip-compile`
`pip-sync`

After that flask has been installed and pkg-resources is still there. I tested it also by adding bpython to requirements.in (and pip-compile && pip-sync) and removing it followed by pip-compile and pip-sync. It worked without any problems, i.e. pkg-resources was preserved.
It not only breaks psycopg2, but also pip-tools itself!

Resolving patch:
diff --git a/piptools/sync.py b/piptools/sync.py
--- a/piptools/sync.py
+++ b/piptools/sync.py
@@ -11,6 +11,7 @@
     'pip',
     'pip-tools',
     'pip-review',
+    'pkg-resources',
     'setuptools',
     'wheel',
 ]


Test patch:
diff --git a/tests/test_sync.py b/tests/test_sync.py
--- a/tests/test_sync.py
+++ b/tests/test_sync.py
@@ -100,6 +100,16 @@ def test_diff_should_uninstall(fake_dist):
     assert to_uninstall == {'django'}  # no version spec when uninstalling
 
 
+def test_diff_should_not_uninstall(fake_dist):
+    ignored = ('pip==7.1.0', 'pip-tools==1.1.1', 'pip-review==1.1.1',
+               'pkg-resources==0.0.0', 'setuptools==34.0.0', 'wheel==0.29.0')
+    installed = [fake_dist(pkg) for pkg in ignored]
+    reqs = []
+
+    to_uninstall = diff(reqs, installed)[1]
+    assert to_uninstall == set()
+
+
 def test_diff_should_update(fake_dist, from_line):
     installed = [fake_dist('django==1.7')]
     reqs = [from_line('django==1.8')]