Repository: pypa/pip

Issue:
pip doesn't clean up after itself in /tmp/
After a few months of deployments, our servers /tmp folder is filling up with lingering pip builds. (see below).

pip should clean up after itself, maybe after a successful build it should remove all old temp folders, otherwise it can easily fill up the hard drive, making servers that don't reboot often very unhappy.

```
[root@nasa-mainframe tmp]# ll | grep pip
drwx------ 12 apache apache     4096 Sep 19 18:17 pip-02SaCT-build
drwx------ 11 apache apache     4096 Sep 18 22:46 pip-02ZJbU-build
drwx------ 11 apache apache     4096 Nov  3 13:56 pip-0GWM9j-build
drwx------ 11 apache apache     4096 Sep 10 18:13 pip-0nhDES-build
drwx------  8 apache apache     4096 Dec 15 12:03 pip-0TuvMR-build
drwx------ 11 apache apache     4096 Sep  7 22:10 pip-11FKKs-build
drwx------ 11 apache apache     4096 Dec  9 11:13 pip-39gSFC-build
drwx------ 11 apache apache     4096 Nov  3 10:56 pip-3EK81q-build
drwx------ 11 apache apache     4096 Sep  7 22:14 pip-3L6bvP-build
drwx------ 11 apache apache     4096 Oct 27 15:34 pip-4d0jFD-build
drwx------ 11 apache apache     4096 Nov 15 15:57 pip-6AS5_a-build
drwx------ 11 apache apache     4096 Dec  9 11:02 pip-6jWQP4-build
drwx------ 11 apache apache     4096 Oct  5 17:58 pip-6LmEvH-build
drwx------ 11 apache apache     4096 Sep 14 14:28 pip-7cBgOj-build
drwx------  2 apache apache     4096 Dec 12 17:44 pip-7jDhp0-record
drwx------ 12 apache apache     4096 Sep 19 18:15 pip-7jO4xC-build
drwx------ 11 apache apache     4096 Oct  7 15:36 pip-7mNzFF-build
drwx------ 11 apache apache     4096 Nov  3 17:54 pip-7x1sBs-build
drwx------ 11 apache apache     4096 Nov  1 12:49 pip-8FpTMv-build
drwx------ 11 apache apache     4096 Dec  9 10:48 pip-8nuHew-build
drwx------ 11 apache apache     4096 Oct 26 11:14 pip-9PzLSo-build
drwx------ 11 apache apache     4096 Oct 26 16:25 pip-9Wyekd-build
drwx------ 11 apache apache     4096 Dec  9 11:26 pip-aFFRCH-build
drwx------ 11 apache apache     4096 Dec 12 18:00 pip-BDtvpn-build
drwx------ 11 apache apache     4096 Oct 27 15:37 pip-bqfuzI-build
drwx------ 11 apache apache     4096 Oct 11 13:46 pip-CG6iL9-build
drwx------ 11 apache apache     4096 Nov  2 13:25 pip-cQsiFq-build
drwx------ 11 apache apache     4096 Sep 13 18:15 pip-cuQCwH-build
drwx------ 11 apache apache     4096 Sep  9 17:02 pip-cwTDmU-build
drwx------ 11 apache apache     4096 Dec  9 11:10 pip-D4lgz2-build
drwx------ 11 apache apache     4096 Sep 13 18:21 pip-DAZ6pr-build
```



Discussion:
Thanks for the report. In general pip does clean up after itself; I use pip frequently and my tmp-dir has no such lingering build dirs in it. There may be specific cases or flags that cause it to fail to clean up; you'll need to provide a more specific reproduction case.

Same with our installation. Running under jenkins 
tmp filled with :
/tmp/pip-PSIqAe-build

Versions:
pip 1.0.1 , python 2.7)

Our build steps include:
virtualenv --no-site-packages $PYENV_HOME --distribute
. $PYENV_HOME/bin/activate
pip install --quiet --use-mirrors nosexcover
pip install --quiet --use-mirrors pylint
pip install --quiet --use-mirrors mercurial
pip bundle dist/project-$JOB_REVISION.$BUILD_NUMBER.pybundle -r requirements.txt ./
pip install dist/project-$JOB_REVISION.$BUILD_NUMBER.pybundle
nosetests --with-xcoverage --with-xunit --cover-package=project --cover-erase --cover-html
pylint -f parseable project/ | tee pylint.out

It appears that installation from a bundle fails to clean up a temporary directory. Pull request welcome; in the meantime, the bundle doesn't appear to serve any actual purpose in the listed build steps; may as well just "pip install -r requirements.txt" and skip the bundle, and the problem will go away.

Actually in our case there's no bundles involved.

Our pip step simply does: `pip install -E ve -r requirements.txt`

cat requirements.txt:

```
./ext/src/django-1.3.1.tar.gz
./ext/src/gevent-1.0a2.tar.gz

cssmin==0.1.4
jinja2==2.6
lxml==2.3.2
markupsafe==0.15
pygeoip==0.2.2
python-amazon-product-api==0.2.5
python-dateutil==1.5
python-memcached==1.48
nose==1.1.2
coverage==3.5.1
django-nose==0.1.3
pycrypto==2.4.1
iptools==0.4.0
webassets==0.6
coffin==0.3.6
simplejson==2.2.1
six==1.0.0
gunicorn==0.13.1
greenlet==0.3.2
```

Looks like it's only the direct path-referenced tar.gz packages that don't get cleaned up.

Using pip 1.0.2 I'm unable to reproduce that result using a requirements.txt with relative-path-referenced tarballs; there are no pip build directories left in /tmp after the install.

BTW you probably want to stop using -E; it's inherently broken with --no-site-packages virtualenvs (and probably in some other cases as well), and will be gone in a future release of pip. Just use the copy of pip that's automatically installed in every virtualenv's bin dir instead.

Sweet, using the virtualenv's pip directly without -E seems to resolve the issue. 
Thanks!

Hmm, I can't reproduce it with -E either, so I don't think -E was actually (directly) the culprit in this case. Is it possible your global install of pip was an older version of pip that didn't clean up correctly, and the inside-the-virtualenv pip you are using now is the most recent pip (1.0.2)?

That makes sense, the server had pip 0.7.x. Thanks for your help

carljm: "bundle doesn't appear to serve any actual purpose in the listed build steps"

We have purpose here -  bundle is used as main build artifact to distribute application together with dependencies around server park and eliminate pip download dependencies on each server we provide deployment (as it does with usual "pip install -r requirements.txt").

As I mentioned this steps part of jenkins job:
1) We create bundle/ build artifact:
pip bundle dist/project-$JOB_REVISION.$BUILD_NUMBER.pybundle -r requirements.txt ./
2) As part of build verification install content from bundle:
pip install dist/project-$JOB_REVISION.$BUILD_NUMBER.pybundle 
3) Run all tests on application installed from a bundle.

I also have this issue on a Ubuntu server.

pip 1.0/python2.7 with virtualenv

Since version 1.0.5 of uwsgi isn't available on pypi, we install it from a .tar.gz file like so:
`$ pip-2.7 install 'http://projects.unbit.it/downloads/uwsgi-1.0.5.tar.gz'`

That always leaves behind a build dir in /tmp 
(pip 1.0.2)

Yes, this seem to be happening again for us too.
All "tar.gz" packages are leaving temp folders behind, even when using the virtualenv's pip instance.

I'm still not able to reproduce; I just did both `pip install http://projects.unbit.it/downloads/uwsgi-1.0.5.tar.gz` and `pip install http://pypi.python.org/packages/source/f/factory_boy/factory_boy-1.1.3.tar.gz` and there is no pip-related directory leftover in `/tmp/` afterwards . Leaving open, since clearly people are experiencing this, but there's apparently some factor that has yet to be identified here.

After messing around for a while trying to repro, I found the trigger: this only happens if the package is already installed.
I can repro the issue like this:

```
$ wget http://pypi.python.org/packages/source/s/six/six-1.1.0.tar.gz
$ pip install ./six-1.1.0.tar.gz
$ pip install ./six-1.1.0.tar.gz
# Now there will be a /tmp/pip-xxx
```

Thanks @adamJLev!

Is anyone working on a fix, or is it fair game for me to tackle? :)

Fair game AFAIK.


Resolving patch:
diff --git a/pip/req.py b/pip/req.py
--- a/pip/req.py
+++ b/pip/req.py
@@ -895,7 +895,7 @@ def uninstall(self, auto_confirm=False):
             req.commit_uninstall()
 
     def locate_files(self):
-        ## FIXME: duplicates code from install_files; relevant code should
+        ## FIXME: duplicates code from prepare_files; relevant code should
         ##        probably be factored out into a separate method
         unnamed = list(self.unnamed_requirements)
         reqs = list(self.requirements.values())
@@ -1003,7 +1003,9 @@ def prepare_files(self, finder, force_root_egg_info=False, bundle=False):
                     ##occurs when the script attempts to unpack the
                     ##build directory
 
+                    # NB: This call can result in the creation of a temporary build directory
                     location = req_to_install.build_location(self.build_dir, not self.is_download)
+
                     ## FIXME: is the existance of the checkout good enough to use it?  I don't think so.
                     unpack = True
                     url = None
@@ -1084,7 +1086,7 @@ def prepare_files(self, finder, force_root_egg_info=False, bundle=False):
                             self.add_requirement(subreq)
                     if req_to_install.name not in self.requirements:
                         self.requirements[req_to_install.name] = req_to_install
-                    if self.is_download:
+                    if self.is_download or req_to_install._temp_build_dir is not None:
                         self.reqs_to_cleanup.append(req_to_install)
                 else:
                     self.reqs_to_cleanup.append(req_to_install)


Test patch:
diff --git a/tests/test_cleanup.py b/tests/test_cleanup.py
--- a/tests/test_cleanup.py
+++ b/tests/test_cleanup.py
@@ -96,6 +96,26 @@ def test_no_install_and_download_should_not_leave_build_dir():
     assert not os.path.exists(env.venv_path/'/build'), "build/ dir should be deleted"
 
 
+def test_temp_build_dir_removed():
+    """
+    It should remove the temp build dir created by pip.
+
+    TestFileEnvironment provides a mechanism to ensure that
+    no temporary files are left after a test script executes,
+    so we can take advantage of that here by simply running
+    `run_pip` twice (which uses TestFileEnvironment).
+    """
+    env = reset_env()
+
+    # Install a requirement into a fresh environment. The temporary build
+    # directory should be removed.
+    result = run_pip('install', 'https://bitbucket.org/ianb/initools/get/tip.zip')
+
+    # Now install the requirement again, exercising a different code path,
+    # and ensure that the temporary build directory is still removed.
+    result = run_pip('install', 'https://bitbucket.org/ianb/initools/get/tip.zip')
+
+
 def test_download_should_not_delete_existing_build_dir():
     """
     It should not delete build/ if existing before run the command
diff --git a/tests/test_pip.py b/tests/test_pip.py
--- a/tests/test_pip.py
+++ b/tests/test_pip.py
@@ -362,6 +362,10 @@ def __init__(self, environ=None, use_distribute=None, sitecustomize=None):
         if sitecustomize:
             self._add_to_sitecustomize(sitecustomize)
 
+        # Ensure that $TMPDIR exists
+        if self.temp_path and not os.path.exists(self.temp_path):
+            os.makedirs(self.temp_path)
+
     def _ignore_file(self, fn):
         if fn.endswith('__pycache__') or fn.endswith(".pyc"):
             result = True
@@ -510,6 +514,10 @@ def __init__(self, environ=None, sitecustomize=None):
 
         assert self.root_path.exists
 
+        # Ensure that $TMPDIR exists
+        if self.temp_path and not os.path.exists(self.temp_path):
+            os.makedirs(self.temp_path)
+
     def __del__(self):
         pass # shutil.rmtree(str(self.root_path), ignore_errors=True)
 
@@ -663,5 +671,5 @@ def main():
 
 
 if __name__ == '__main__':
-    sys.stderr.write("Run pip's tests using nosetests. Requires virtualenv, ScriptTest, and nose.\n")
+    sys.stderr.write("Run pip's tests using nosetests. Requires virtualenv, ScriptTest, mock, and nose.\n")
     sys.exit(1)
diff --git a/tests/test_unicode.py b/tests/test_unicode.py
--- a/tests/test_unicode.py
+++ b/tests/test_unicode.py
@@ -20,6 +20,6 @@ def test_install_package_that_emits_unicode():
 
     env = reset_env()
     to_install = os.path.abspath(os.path.join(here, 'packages', 'BrokenEmitsUTF8'))
-    result = run_pip('install', to_install, expect_error=True)
+    result = run_pip('install', to_install, expect_error=True, expect_temp=True, quiet=True)
     assert '__main__.FakeError: this package designed to fail on install' in result.stdout
     assert 'UnicodeDecodeError' not in result.stdout