Repository: pypa/pip

Issue:
pip install pgraph - AssertionError: Multiple .dist-info directories
- Pip version: 8.1.0
- Python version: 2.7.11+
- Operating System: Linux lab 4.3.0-kali1-686-pae
### Description:

Please try to install **pgraph** with command: `sudo pip install pgraph --upgrade`, 
### What I've run:

```
root@lab:~/# uname -a 
Linux lab 4.3.0-kali1-686-pae #1 SMP Debian 4.3.5-1kali1 (2016-02-11) i686 GNU/Linux

root@lab:~/# pip install pgraph --upgrade
Collecting pgraph
Requirement already up-to-date: pyramid-chameleon in /usr/local/lib/python2.7/dist-packages (from pgraph)
Requirement already up-to-date: pyramid in /usr/local/lib/python2.7/dist-packages (from pgraph)
Requirement already up-to-date: webtest in /usr/local/lib/python2.7/dist-packages (from pgraph)
Requirement already up-to-date: waitress in /usr/local/lib/python2.7/dist-packages (from pgraph)
Requirement already up-to-date: mock in /usr/local/lib/python2.7/dist-packages (from pgraph)
Collecting py-deps[memcache]>=0.5.5 (from pgraph)
Requirement already up-to-date: celery in /usr/local/lib/python2.7/dist-packages (from pgraph)
Requirement already up-to-date: pyramid-debugtoolbar in /usr/local/lib/python2.7/dist-packages (from pgraph)
Requirement already up-to-date: funcsigs in /usr/local/lib/python2.7/dist-packages (from pgraph)
Requirement already up-to-date: Chameleon in /usr/local/lib/python2.7/dist-packages (from pyramid-chameleon->pgraph)
Requirement already up-to-date: setuptools in /usr/local/lib/python2.7/dist-packages (from pyramid->pgraph)
Requirement already up-to-date: repoze.lru>=0.4 in /usr/local/lib/python2.7/dist-packages (from pyramid->pgraph)
Requirement already up-to-date: zope.interface>=3.8.0 in /usr/lib/python2.7/dist-packages (from pyramid->pgraph)
Requirement already up-to-date: PasteDeploy>=1.5.0 in /usr/local/lib/python2.7/dist-packages (from pyramid->pgraph)
Requirement already up-to-date: WebOb>=1.3.1 in /usr/local/lib/python2.7/dist-packages (from pyramid->pgraph)
Requirement already up-to-date: translationstring>=0.4 in /usr/local/lib/python2.7/dist-packages (from pyramid->pgraph)
Requirement already up-to-date: zope.deprecation>=3.5.0 in /usr/local/lib/python2.7/dist-packages (from pyramid->pgraph)
Requirement already up-to-date: venusian>=1.0a3 in /usr/local/lib/python2.7/dist-packages (from pyramid->pgraph)
Requirement already up-to-date: six in /usr/lib/python2.7/dist-packages (from webtest->pgraph)
Requirement already up-to-date: beautifulsoup4 in /usr/lib/python2.7/dist-packages (from webtest->pgraph)
Requirement already up-to-date: pbr>=0.11 in /usr/local/lib/python2.7/dist-packages (from mock->pgraph)
Requirement already up-to-date: networkx in /usr/local/lib/python2.7/dist-packages (from py-deps[memcache]>=0.5.5->pgraph)
Requirement already up-to-date: wheel in /usr/local/lib/python2.7/dist-packages (from py-deps[memcache]>=0.5.5->pgraph)
Requirement already up-to-date: pip>=6.0.1 in /usr/local/lib/python2.7/dist-packages (from py-deps[memcache]>=0.5.5->pgraph)
Requirement already up-to-date: pylibmc in /usr/local/lib/python2.7/dist-packages (from py-deps[memcache]>=0.5.5->pgraph)
Requirement already up-to-date: kombu<3.1,>=3.0.34 in /usr/local/lib/python2.7/dist-packages (from celery->pgraph)
Requirement already up-to-date: pytz>dev in /usr/lib/python2.7/dist-packages (from celery->pgraph)
Requirement already up-to-date: billiard<3.4,>=3.3.0.23 in /usr/local/lib/python2.7/dist-packages (from celery->pgraph)
Requirement already up-to-date: pyramid-mako>=0.3.1 in /usr/local/lib/python2.7/dist-packages (from pyramid-debugtoolbar->pgraph)
Collecting Pygments (from pyramid-debugtoolbar->pgraph)
  Using cached Pygments-2.1.3-py2.py3-none-any.whl
Collecting decorator>=3.4.0 (from networkx->py-deps[memcache]>=0.5.5->pgraph)
  Using cached decorator-4.0.9-py2.py3-none-any.whl
Requirement already up-to-date: anyjson>=0.3.3 in /usr/local/lib/python2.7/dist-packages (from kombu<3.1,>=3.0.34->celery->pgraph)
Requirement already up-to-date: amqp<2.0,>=1.4.9 in /usr/local/lib/python2.7/dist-packages (from kombu<3.1,>=3.0.34->celery->pgraph)
Collecting Mako>=0.8 (from pyramid-mako>=0.3.1->pyramid-debugtoolbar->pgraph)
Requirement already up-to-date: MarkupSafe>=0.9.2 in /usr/lib/python2.7/dist-packages (from Mako>=0.8->pyramid-mako>=0.3.1->pyramid-debugtoolbar->pgraph)
Installing collected packages: py-deps, pgraph, Pygments, decorator, Mako
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 732, in install
    **kwargs
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 835, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 1030, in move_wheel_files
    isolated=self.isolated,
  File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 344, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 304, in clobber
    assert not info_dir, 'Multiple .dist-info directories'
AssertionError: Multiple .dist-info directories
```



Discussion:
As described in issue #1964, the error message needs to be corrected:

> The error is very confusing and doesn't include any hint on which is the real problem, which are the "multiple" ones, these are not listed even in the pip.log

Please correct the error message, to give the necessary information for the user to know what to do to address the problem.

Closing this, I believe this is an issue with the specific package file on PyPI and is not a pip issue.
This continues to happen for me, on different code bases.

An example from today:

```
$ python3 -m pip install .
Processing /home/bignose/Projects/maildrake/trunk
Collecting docutils (from maildrake===NEXT)
  Using cached docutils-0.13.1-py3-none-any.whl
Collecting fasteners (from maildrake===NEXT)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting setuptools (from maildrake===NEXT)
  Using cached setuptools-34.3.2-py2.py3-none-any.whl
Collecting sqlparse (from maildrake===NEXT)
  Using cached sqlparse-0.2.3-py2.py3-none-any.whl
Collecting monotonic>=0.1 (from fasteners->maildrake===NEXT)
  Using cached monotonic-1.3-py2.py3-none-any.whl
Collecting six (from fasteners->maildrake===NEXT)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting packaging>=16.8 (from setuptools->maildrake===NEXT)
  Using cached packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools->maildrake===NEXT)
  Using cached appdirs-1.4.3-py2.py3-none-any.whl
Collecting pyparsing (from packaging>=16.8->setuptools->maildrake===NEXT)
  Using cached pyparsing-2.2.0-py2.py3-none-any.whl
Installing collected packages: docutils, monotonic, six, fasteners, pyparsing, packaging, appdirs, setuptools, sqlparse, maildrake
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 360, in run
    prefix=options.prefix_path,
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 305, in clobber
    ', '.join(info_dir))
AssertionError: Multiple .dist-info directories: /home/bignose/.local/lib/python3.5/site-packages/monotonic-1.2.dist-info, /home/bignose/.local/lib/python3.5/site-packages/monotonic-1.3.dist-info

$ ls /home/bignose/.local/lib/python3.5/site-packages/monotonic-*
ls: cannot access '/home/bignose/.local/lib/python3.5/site-packages/monotonic-*': No such file or directory
```

So, for some reason the “Multiple .dist-info directories” message reports two files that *both don't exist* after the command exits. This makes it difficult to know what even went wrong.

This continues to happen even in a clean virtualenv, and when I explicitly clear the Pip cache:

```
(temp_venv) $ rm -r ~/.cache/pip/*

(temp_venv) $ python3 -m pip install ../trunk/dist/maildrake-0.1.2-py2.py3-none-any.whl
Processing /home/bignose/Projects/maildrake/trunk/dist/maildrake-0.1.2-py2.py3-none-any.whl
Requirement already satisfied: semver in ./lib/python3.5/site-packages (from maildrake==0.1.2)
Requirement already satisfied: docutils in ./lib/python3.5/site-packages (from maildrake==0.1.2)
Requirement already satisfied: setuptools in ./lib/python3.5/site-packages (from maildrake==0.1.2)
Collecting fasteners (from maildrake==0.1.2)
  Downloading fasteners-0.14.1-py2.py3-none-any.whl
Requirement already satisfied: sqlparse in ./lib/python3.5/site-packages (from maildrake==0.1.2)
Collecting monotonic>=0.1 (from fasteners->maildrake==0.1.2)
  Downloading monotonic-1.3-py2.py3-none-any.whl
Collecting six (from fasteners->maildrake==0.1.2)
  Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: monotonic, six, fasteners, maildrake
Exception:
Traceback (most recent call last):
  File "/home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/pip/commands/install.py", line 360, in run
    prefix=options.prefix_path,
  File "/home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/pip/wheel.py", line 305, in clobber
    ', '.join(info_dir))
AssertionError: Multiple .dist-info directories: /home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/monotonic-1.2.dist-info, /home/bignose/Projects/maildrake/temp_venv/lib/python3.5/site-packages/monotonic-1.3.dist-info
```

There is no information given to help trace where these “multiple .dist-info directories” are coming from, what's making them appear, and what can be done to stop them from appearing.

How can this be diagnosed further?

@bignose-debian Can you publish ``maildrake-0.1.2-py2.py3-none-any.whl`` somewhere?
ping, we would need the aforementioned wheel.
I suspect two `.dist-info` in the wheel, just like in #1964
I managed to resolve this, temporarily. I deleted files from inside the cache for Pip.

`$ rm /var/local/cache/pip/*`

The cache is in a directory that is completely different from any of the filenames in the error message, so it's not at all clear whether or why this would change the behaviour.

My Pip configuration is:

```
$ cat ~/.pip/pip.conf
# $HOME/.pip/pip.conf
# User-specific configuration for PIP installer for Python packages.

[global]
download-cache = /var/local/cache/pip

[install]
build-directory = /var/local/devel/bignose/virtualenvs/build
source-directory = /var/local/devel/bignose/virtualenvs/src
```

Ah, that means a broken wheel got cached at one point and then kept getting reused.
> […] a broken wheel got cached at one point and then kept getting reused.

So, from my position the bug here is that Pip is giving a message that is totally unhelpful for diagnosing that problem.

What should it say, instead, that would allow the reader to immediately know that (and not something else) is the problem?

> […] a broken wheel got cached at one point and then kept getting reused.

So, how can I prevent this from happening?

I am in the position of needing to frequently delete all files under the configured ‘build-directory’ in order to avoid the unhelpful error message here.

While #1964 was closed and we now have a better error, I think that this is still a bug.

I have double checked and the wheels have a single .dist-info folder.
I also got the same issue for tar.gz files

This is triggered only when I am using a custom 'build' folder.

The build folder does contain multiple .dist-info, one for the existing version, and one for the version which is going to be installed.

For my project I am pinning everything, so when switching between branches it is common for the CI to reinstall older versions.

An easy way to reproduce it 

```
$ virtualenv pip-build-3564
$ cd pip-build-3564/
$ . bin/activate
$ mkdir build
$ pip install -b build bunch==1.0.0
# All good
$ pip install -b build bunch==1.0.1
AssertionError: Multiple .dist-info directories: /home/adi/tmp/pip-build-3564/lib/python2.7/site-packages/bunch-1.0.0.dist-info, /home/adi/tmp/pip-build-3564/lib/python2.7/site-packages/bunch-1.0.1.dist-info
```

I have standard Ubuntu 16.04-x64
```
$ python --version
Python 2.7.12
$ pip --version
pip 9.0.1
```

If I remove the build folder, all is fine.

Thanks!

Hmm, that sounds like a simple case of you not cleaning up the build directory before doing a rebuild (I guess the point here is to allow incremental builds?). I don't really have much experience with the ```-b``` option and the reasons why people might need it, so I can't say if that's something you'd expect people to do. @pypa/pip-committers any thoughts?
I am using a custom build folder so that the CI runs in as much isolation as possible.
I am not using Travis, but buildbot, so the build system is not erased between builds.... and the same build system is running multiple tests in parallel, (for multiple projects), each in its won virtualenv 

---------

It might be the case that the build folder needs a manual build.

I noted that there is the `--no-clean` argument. I am not using it, so I was assuming that pip will automatically clean.

--------

Note that for the above example, those are the actual steps that I am running...but a simple upgrading of the package
> Hmm, that sounds like a simple case of you not cleaning up the build directory before doing a rebuild

The ‘pip’ configuration setting, ‘build-directory’, specifies the build directory for Pip. Are you saying that Pip will crash if that directory is not *manually* cleaned, between ‘pip install’ invocations?

If so, it seems you're describing what needs to be corrected in Pip's behaviour to resolve this bug.

See https://github.com/pypa/pip/issues/3564#issuecomment-297950139

It contains a set of commands which executed will trigger the failure. You can try to run them on your system.

But yes. Pip fails if the directory is not manually cleaned.... but only when using a "custom" build folder with `pip install -b some/custom/path/which/needs/manual/clean SOME-PACKAGE`
@bignose-debian I'm more suggesting that reusing pip's build directory is a bit more complex than supplying an empty directory to ```--build-directory```. Sure, pip could clean the build directory before use. But that would prevent any sort of incremental build optimisations - which is what some people use ```--build-directory``` for, I believe. But *not* cleaning out the build directory means that pip needs to be prepared for any sort of unexpected content in there (which as this issue shows, isn't always easy).

I don't think there's a simple solution. At the moment, the rules are (roughly):

1. Without ```--build-directory```, pip will use a newly created empty directory for the build.
2. If the user specifies an empty directory in ```--build-directory```, pip will produce the same results as in (1), but will leave the output of the build process in the specified directory.
3. If the ```--build-directory``` is not empty, it's possible for pip to fail because there's unexpected content in there. It's the user's responsibility to resolve this issue.

The problem here is (3). There may be room for some heuristics to improve the situation - for example, pip will delete any subdirectories named ```*.dist-info``` from the build directory before starting. But those heuristics may not always be right (an unlikely case, but consider an incremental build system where it's the building of the dist-info file that's complex). And there will always be the possibility of error (imagine if the build directory contained a write-protected file called ```setup.py```). And documenting precisely what pip will do to the supplied build directory (in a way that people will notice *before* pip corrupts something they want to keep!) will be important.

I'm sure that PRs which improve things in case (3) will be of interest. I expect they'll cause a certain amount of debate, though, it's not obvious how (or even if) we should change things.
Thanks @pfmoore  for the explanation.

But if pip install is not expected to clean the build directory, then why do we have this option:

```
  --no-clean                  Don't clean up build directories.
```

I have never used incremental builds, but my expectation is that for incremental build you will use `--build-directory` together with `--no-clean`

When the `--no-clean` flag is omitted, based on the current documentation I am expecting that the build directory to be cleaned. 

I am triggering this error with a happy path. The initial pip command is successfully executed and I am not passing the `--no-clean` flag... but the next call will fail

```
$ pip install -b build bunch==1.0.0
# All good
$ pip install -b build bunch==1.0.1
AssertionError: Multiple .dist-info directories: /home/adi/tmp/pip-build-3564/lib/python2.7/site-packages/bunch-1.0.0.dist-info, /home/adi/tmp/pip-build-3564/lib/python2.7/site-packages/bunch-1.0.1.dist-info
```
@adiroiban Honestly, I don't know. There's a lot of historical stuff that's around this area that I wasn't involved in. My explanation may not be completely accurate - it's possible I've misunderstood stuff.

Maybe there is a bug in that ```--build-directory``` should result in the build dir being emptied if ```--no-clean``` isn't specified. I don't know if the docs say anything specific here?
@pfmoore don't worry. Your feedback is much appreciated.

So for now, I don't see any difference between using install with or without `--no-clean`

I have never used that incremental build thing... so I have no idea what is that :)

So, in order to get a PR rolling it would help to get some feedback regarding the behavior for the `--no-clean` flag

1.pip install, custom build directory, without `--no-clean`
-----------------------------------------------------------------------

```
$ ls ./build-here
ls: cannot access './build-here': No such file or directory

$ pip install -b ./build-here  bunch==1.0.0

$ ls ./build-here
ls: cannot access './build-here': No such file or directory
```

This is what I am expecting to get... but for now I get the same results as for test case 2.
That is, the build directory is not cleaned.


2.pip install, custom build directory, with `--no-clean`
---------------------------------------------------------------------

```
$ ls ./build-here
ls: cannot access './build-here': No such file or directory

$ pip install -b ./build-here --no-clean bunch==1.0.0

$ ls ./build-here/
bunch  pip-delete-this-directory.txt
```

this is what I get now ... and based on the current documentation is what I am expecting

The current expectation is that if you're specifying ``--build-directory`` then you're managing the lifecycle of that directory, not pip.
@dstufft then the current implemenation is valid... maybe only not well documented.... and a bit awkward in relational tot the `--no-clean` flag

Please note that I am talking about the `pip install`  `  -b, --build <dir> ` option... as from

```
Install Options:

  -b, --build <dir>           Directory to unpack packages into and build in.

  --no-clean                  Don't clean up build directories.

```

Not sure what `--build-directory` is.

So, maybe the fix for this would be to extend the documentation to

```
  -b, --build <dir>           Directory to unpack packages into and build in.
                              When in used, pip will not do any cleaning, even when
                              --no-clean is NOT passed. 
                              You will have to do your own cleaning of this folder.

  --no-clean                  Don't clean up build directories.
                              Build directory is never cleaned when a custom build directory is used.
```

@dstufft What do you say, does my comment make sense? Thanks!
Documenting the current behavior better is always a +1 in my book. To be clear, I'm not dead set on maintaining that property, that is just the current expectation, a compelling argument *could* see it changed. One note though is that pip uses randomized build directories by default, so you shouldn't need to use ``--build`` at all to get isolated installs. There's also https://github.com/pypa/pip/issues/4371.
> The current expectation is that if you're specifying --build-directory then you're managing the lifecycle of that directory, not pip.
> […] that is just the current expectation, a compelling argument could see it changed.

A build can sometimes crash; then the build directory is left in place, and nothing cleans it up.

I set the ‘build-directory’ option in Pip's configuration file, so that when a build crashes the cruft isn't littering my virtualenv directory.

So I do expect that Pip will at least not complain when the build directory is not empty; that it will gracefully handle the presence of previous build artefacts in that directory.

> I set the ‘build-directory’ option in Pip's configuration file, so that when a build crashes the cruft isn't littering my virtualenv directory.

pip uses a build directory located in ``TMPDIR`` by default, if the build crashes and leaves something behind presumably at some point the system will reclaim space by deleting temporary files.

Resolving patch:
diff --git a/src/pip/_internal/cmdoptions.py b/src/pip/_internal/cmdoptions.py
--- a/src/pip/_internal/cmdoptions.py
+++ b/src/pip/_internal/cmdoptions.py
@@ -440,7 +440,8 @@ def only_binary():
     help='Directory to unpack packages into and build in. Note that '
          'an initial build still takes place in a temporary directory. '
          'The location of temporary directories can be controlled by setting '
-         'the TMPDIR environment variable (TEMP on Windows) appropriately.'
+         'the TMPDIR environment variable (TEMP on Windows) appropriately. '
+         'When passed, build directories are not cleaned in case of failures.'
 )  # type: Any
 
 ignore_requires_python = partial(


Test patch: