Repository: pypa/pip

Issue:
pip fail after upgrade to 1.5.5
```
$ pip install --upgrade pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-1.5.5-py2.py3-none-any.whl#md5=03a932d6f82a3887d8de1cdb837c87ed
  Using download cache from /Users/robinho/Library/Caches/pip-downloads/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fpy2.py3%2Fp%2Fpip%2Fpip-1.5.5-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...
$ pip
Traceback (most recent call last):
  File "/usr/local/bin/pip", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 2749, in <module>
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 446, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 459, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 628, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pip==1.5.4
```

Using a python installed by homebrew.

```
$ python --version
Python 2.7.6
$ brew --version
0.9.5
$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.9.2
BuildVersion:   13C1021
```



Discussion:
Hmm, for some reason your `pip` command wasn't updated. I wasn't able to reproduce with a quick attempt here locally. You could try do `python -m pip --upgrade --force-reinstall pip` to see if that fixes it.

@dstufft don't work.

```
$ python -m pip install --upgrade --force-reinstall pip
Downloading/unpacking pip
  Using download cache from /Users/robinho/Library/Caches/pip-downloads/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fpy2.py3%2Fp%2Fpip%2Fpip-1.5.5-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 1.5.5
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...
$ pip
Traceback (most recent call last):
  File "/usr/local/bin/pip", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 2749, in <module>
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 446, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 459, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 628, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pip==1.5.4
```

Maybe it is a bug on homebrew install =/. What's your opinion about it?

```
$ mkvirtualenv test
New python executable in test/bin/python2.7
Also creating executable in test/bin/python
Installing setuptools, pip...done.

$ pip list --outdated
pip (Current: 1.5.4 Latest: 1.5.5)
setuptools (Current: 2.2 Latest: 3.5.1)

$ pip install --upgrade pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-1.5.5-py2.py3-none-any.whl#md5=03a932d6f82a3887d8de1cdb837c87ed
  Using download cache from /Users/robinho/Library/Caches/pip-downloads/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fpy2.py3%2Fp%2Fpip%2Fpip-1.5.5-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...

$ pip --version
pip 1.5.5 from /Users/robinho/.virtualenvs/test/lib/python2.7/site-packages (python 2.7)
```

What happens if you do `which pip`?

Before upgrade:

```
$ which pip
/usr/local/bin/pip

$ pip --version
pip 1.5.4 from /usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg (python 2.7)
```

After upgrade:

```
$ pip install --upgrade pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-1.5.5-py2.py3-none-any.whl#md5=03a932d6f82a3887d8de1cdb837c87ed
  Using download cache from /Users/robinho/Library/Caches/pip-downloads/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fpy2.py3%2Fp%2Fpip%2Fpip-1.5.5-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...

$ which pip
/usr/local/bin/pip

$ pip --version
Traceback (most recent call last):
  File "/usr/local/bin/pip", line 5, in <module>
    from pkg_resources import load_entry_point
  File "build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py", line 2749, in <module>
    working_set = WorkingSet._build_master()
  File "build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py", line 446, in _build_master
    return cls._build_from_requirements(__requires__)
  File "build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py", line 459, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py", line 628, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pip==1.5.4
```

Either your pythons and paths are horribly messed up, or this might just be an old distribute/setuptools blowup.

Try these:

```
# Get rid of them
$ python -m pip uninstall -y setuptools
$ python -m pip uninstall -y distribute
# the next command should FAIL now because the above should be uninstalled
$ python -c "import pkg_resources"
# reinstall stuff now
$ python -m pip install -U --force-reinstall setuptools
$ python -m pip install -U --force-reinstall pip
# now check things work and are the same
$ python -m pip --version
$ pip --version
$ pip list
```

Does pip try to check/upgrade the setuptools version to avoid these situations?

Not work. I don't known why, but the `uninstall` keep the `pkg_resources`.

```
$ python -m pip uninstall -y setuptools
Uninstalling setuptools:
  Successfully uninstalled setuptools

$ python -m pip uninstall -y distribute
Cannot uninstall requirement distribute, not installed
Storing debug log for failure in /Users/robinho/.pip/pip.log

$ python -c "import pkg_resources"

$ python -m pip install -U --force-reinstall setuptools
Downloading/unpacking setuptools
  Using download cache from /Users/robinho/Library/Caches/pip-downloads/https%3A%2F%2Fpypi.python.org%2Fpackages%2F3.4%2Fs%2Fsetuptools%2Fsetuptools-3.6-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools
Cleaning up...
$ python -m pip install -U --force-reinstall pip
Downloading/unpacking pip
  Using download cache from /Users/robinho/Library/Caches/pip-downloads/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fpy2.py3%2Fp%2Fpip%2Fpip-1.5.5-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...

$ python -m pip --version
pip 1.5.5 from /usr/local/lib/python2.7/site-packages (python 2.7)

$ pip --version
Traceback (most recent call last):
  File "/usr/local/bin/pip", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 2829, in <module>
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 451, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 464, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/site-packages/pkg_resources.py", line 639, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pip==1.5.4

$ ls -l /usr/local/bin/pip
lrwxr-xr-x  1 robinho  admin  32 May  7 14:22 /usr/local/bin/pip -> ../Cellar/python/2.7.6_1/bin/pip

$ cat /usr/local/bin/pip
#!/usr/local/opt/python/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'pip==1.5.4','console_scripts','pip'
__requires__ = 'pip==1.5.4'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
    )
```

I think **Upgrade setuptools and pip** Pull Request #29039 probably solves this issue.  

A similar pip 1.5.4 to 1.5.5 problem is at the root of [/homebrew/python issue](https://github.com/Homebrew/homebrew/issues/26900) #26900

@wem3 The Pull Request https://github.com/Homebrew/homebrew/pull/29039 was merged.
Now the pip 1.5.5 are working.

Thanks for the attention, guys!

I see `pkg_resources.DistributionNotFound:` error popping out at SO from time to time. My [diagnosis](http://stackoverflow.com/a/21314526/95735) at one point in the past was

> The root of the problem are often outdated scripts in the bin (Linux) or Scripts (Windows) subdirectory.

However I'm not sure what causes these scripts to not be updated when upgrading given distribution/package.

To provide a bit more information, Homebrew's version of Python currently bundles setuptools 3.6; before yesterday it provided setuptools 3.4.1.

I can confirm that setuptools 3.6 does not fix this problem, and it does occur even if no other setuptools/distutils installations are present.

I've noticed this only happens when pip tries to upgrade itself by downloading a wheel (which is the default); `pip install --upgrade --no-use-wheel pip` does produce a working pip.

I can provide more information if needed.

I can also see two separate `site-packages` in the logs shown on the issue, which suggests to me that some migration hasn't worked properly (some scripts might have been left behind?). You only want one `site-packages` for one python..

@mistydemeo pip provides a `RECORD` file in the `egg-info` / `dist-info` folder of a package when its installed, in the python's `site-packages` folder. e.g

```
/usr/lib/python2.7/site-packages/setuptools-3.5.1-py2.7.egg-info/RECORD
```

It lists the files installed by pip for a package. If you compare those between pip install between `--no-use-wheel` and not, we could find some of the problem.

The contents are identical between the working and non-working (non-wheel and wheel) installs; even the checksums of the files are identical.

Well, damn. @mistydemeo thanks for trying that! fwiw the reason to manually look up the `RECORD`s is that listing files installed with `pip show --list <pkg>` doesn't work for wheels in 1.5.x, it's fixed in 1.6.

I'd still consider a possibility of @robsonpeixoto is that his installation seems to have gathered two `site-packages` (two python install locations? :S),

```
/usr/local/lib/python2.7/site-packages/
```

and

```
/usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/
```

I'm not sure if this is caused from homebrew workings or what. Leftover files in there might also be a problem. Having multiple `setuptools` (and `pkg_resources.py`) in these and 'wrong one' / old one being found in different situations has happened to people before.

It seems weird to me that even though `python -m pip` came from the correct `site-packages`, it installed (or not?) a script that is symlinked to an old one :S

One of the two site-packages directories is actually just a symlink: they're the same directory. Homebrew's python is only configured to use one `site-packages`.

> It seems weird to me that even though python -m pip came from the correct site-packages, it installed (or not?) a script that is symlinked to an old one :S

To explain the symlink structure:

Homebrew installs software to private versioned directories within `/usr/local/Cellar`, then symlinked to the standard locations. In the case of Python 2.7.6 (revision 1), it is installed to `/usr/local/Cellar/python/2.7.6_1`, then symlinked into `/usr/local`. `pip` is not actually installed as a symlink; rather `/usr/local/bin/pip` just happens to be the public symlink to the real script in `/usr/local/Cellar/python/2.7.6_1/bin/pip`. When the working non-wheel pip upgrade occurs, it's updating the script in the longer private path, while the non-working upgrade leaves that old script in place. Neither upgrade touches the `/usr/local/bin/pip` symlink.

Homebrew download pip tarball and execute setup.py.
`pip install --upgrade pip` installs pip from wheel.

Installing from setup.py makes versioned script.
I think upgrading pip from wheel doesn't overwrite versioned script.
This is why DistributionNotFound happens.

For more detail:

```
$ rm -rf /usr/local/lib/python2.7/
$ brew uninstall python
Uninstalling /usr/local/Cellar/python/2.7.6_1...

$ brew install python
==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/python-2.7.6_1.mavericks.bottle.tar.gz
Already downloaded: /Library/Caches/Homebrew/python-2.7.6_1.mavericks.bottle.tar.gz
==> Pouring python-2.7.6_1.mavericks.bottle.tar.gz
==> Caveats
Setuptools and Pip have been installed. To update them
  pip install --upgrade setuptools
  pip install --upgrade pip

You can install Python packages with
  pip install <package>

They will install into the site-package directory
  /usr/local/lib/python2.7/site-packages

See: https://github.com/Homebrew/homebrew/wiki/Homebrew-and-Python

.app bundles were installed.
Run `brew linkapps` to symlink these to /Applications.
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
You can try again using:
  brew link python

Possible conflicting files are:
/usr/local/bin/pip
==> /usr/local/Cellar/python/2.7.6_1/bin/python -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/2.7.6_1/bin --install-lib=/usr/local/lib/python2.7/
==> /usr/local/Cellar/python/2.7.6_1/bin/python -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python/2.7.6_1/bin --install-lib=/usr/local/lib/python2.7/
==> Summary
🍺  /usr/local/Cellar/python/2.7.6_1: 4907 files, 80M

$ ll /usr/local/bin/pip
lrwxr-xr-x  1 inada-n  admin  32  5 29 09:18 /usr/local/bin/pip@ -> ../Cellar/python/2.7.6_1/bin/pip

$ ll /usr/local/bin/pip2
lrwxr-xr-x  1 inada-n  admin  33  5 29 09:18 /usr/local/bin/pip2@ -> ../Cellar/python/2.7.6_1/bin/pip2

$ rm /usr/local/bin/pip2

$ pip install --upgrade pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-1.5.6-py2.py3-none-any.whl#md5=4d4fb4b69df6731c7aeaadd6300bc1f2
  Using download cache from /Users/inada-n/.pip/cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fpy2.py3%2Fp%2Fpip%2Fpip-1.5.6-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...

$ ll /usr/local/bin/pip
lrwxr-xr-x  1 inada-n  admin  32  5 29 09:18 /usr/local/bin/pip@ -> ../Cellar/python/2.7.6_1/bin/pip

$ ll /usr/local/bin/pip2
-rwxr-xr-x  1 inada-n  admin  305  5 29 09:19 /usr/local/bin/pip2*

$ pip
Traceback (most recent call last):
  File "/usr/local/bin/pip", line 5, in <module>
    from pkg_resources import load_entry_point
  File "build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py", line 2749, in <module>
  File "build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py", line 446, in _build_master
  File "build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py", line 459, in _build_from_requirements
  File "build/bdist.macosx-10.9-x86_64/egg/pkg_resources.py", line 628, in resolve
pkg_resources.DistributionNotFound: pip==1.5.4

$ pip2

Usage:
  pip <command> [options]
...
```

When using `--no-use-wheel`, pip overwrites to symlinked script.

```
$ pip install -U --no-use-wheel pip --log ~/pip.log
Downloading/unpacking pip from https://pypi.python.org/packages/source/p/pip/pip-1.5.6.tar.gz#md5=01026f87978932060cc86c1dc527903e
  Downloading pip-1.5.6.tar.gz (938kB): 938kB downloaded
  Storing download in cache at ./.pip/cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpip%2Fpip-1.5.6.tar.gz
  Running setup.py (path:/private/var/folders/11/3pzjxxcd7q53dbfsm95ms3d00000gn/T/pip_build_inada-n/pip/setup.py) egg_info for package pip

    warning: no files found matching 'pip/cacert.pem'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
Installing collected packages: pip
  Found existing installation: pip 1.5.4
    Uninstalling pip:
      Successfully uninstalled pip
  Running setup.py install for pip

    warning: no files found matching 'pip/cacert.pem'
    warning: no files found matching '*.html' under directory 'docs'
    warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
    no previously-included directories found matching 'docs/_build/_sources'
    Installing pip script to /usr/local/bin
    Installing pip2.7 script to /usr/local/bin
    Installing pip2 script to /usr/local/bin
Successfully installed pip
Cleaning up...

$ ll /usr/local/bin/pip
lrwxr-xr-x  1 inada-n  admin  32  5 29 09:22 /usr/local/bin/pip@ -> ../Cellar/python/2.7.6_1/bin/pip
```

What is the correct behavior when file exists while installing script?
1. Overwrite (follow symlink, current hehavior of installing from source)
2. Overwrite (replace symlink)
3. Do nothing (Current behavior of installing from wheel)
4. Report error

FYI, `pip --exists-action=b install -U pip` doesn't fix this problem.


Resolving patch:
diff --git a/pip/wheel.py b/pip/wheel.py
--- a/pip/wheel.py
+++ b/pip/wheel.py
@@ -242,6 +242,10 @@ def is_entrypoint_wrapper(name):
 
     maker = ScriptMaker(None, scheme['scripts'])
 
+    # Ensure old scripts are overwritten.
+    # See https://github.com/pypa/pip/issues/1800
+    maker.clobber = True
+
     # Ensure we don't generate any variants for scripts because this is almost
     # never what somebody wants.
     # See https://bitbucket.org/pypa/distlib/issue/35/


Test patch: