Repository: pypa/pip

Issue:
--target option causes issues when upgrading or re-installing
If you install using the "--target" option:

```
pip install --target lib requests
```

And then repeat the command a second time, it doesn't say "requirement satisfied" it installs it again as a subdirectory of the existing installation (lib/requests/requests and lib/requests-2.2.0.dist-info/requests-2.2.0.dist-info

Repeat the command a third time and it appears the nesting stops but it chokes and still doesn't report "requirement satisfied":

```
Downloading/unpacking requests
  Downloading requests-2.2.0-py2.py3-none-any.whl (623kB): 623kB downloaded
Installing collected packages: requests
Successfully installed requests
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 307, in run
    os.path.join(options.target_dir, item)
  File "/usr/lib/python2.7/shutil.py", line 291, in move
    raise Error, "Destination path '%s' already exists" % real_dst
Error: Destination path '/tmp/piptest/lib/requests/requests' already exists

Storing debug log for failure in /tmp/tmpcBDoql
```



Discussion:
Has anyone had a chance to look at this since January? I just ran across this issue with the newest pip release.

@dstufft, there seems to be a cluster of issues relating to this one. Has anyone looked at it?

Hey guys, this issue has been open for some time. Is there anything I can do to help this along?

@brosner, @dholth, @dstufft, @jezdez, @pfmoore, @pnasrat, @r1chardj0n3s

This is a longstanding issue that is _really_ annoying.

@jonparrott Apart from the obvious "patches welcome", not much...

The nesting behaviour is really odd. On the other hand, the fact that pip doesn't recognise that requests is installed is understandable (after all, it _isn't_ installed in the sense that `import requests` will fail) - I can see that the behaviour is not what you'd want, but it's at least understandable (unlike the nesting!)

Personally, I only ever use `--target` to dump a copy of a distribution somewhere (for vendoring, for example) and I never want to "manage" it - I just blow away the directory and re-download if needed. Even the `dist-info` directories are unwanted clutter for that usage.

To be honest, I could argue that `--target` is just broken, and should be removed. Maybe we should replace it with a command, "pip unpack" that just unpacked the distribution into the target and made no pretense that it was "installed"? That might better match the reality.

@pfmoore thanks for the response!

`--target` is currently rather important for one environment in particular, Google App Engine (eg. https://github.com/GoogleCloudPlatform/appengine-python-flask-skeleton and https://github.com/GoogleCloudPlatform/appengine-python-bottle-skeleton

An 'unpack' command, that still retrieved dependencies, updated in place, etc. could be a workable replacement, but you might want to talk to the folks in that community for feedback.

The 'blow away and re-download' approach is what @jonparrott's tool [Gaelic](https://bitbucket.org/jonparrott/gaelic) does as a workaround, BTW.

@pfmoore The vendoring use case is exactly what the app engine community is trying to do, as mentioned by @webmaven. The problem stems that even though --target works great for the initial vendoring, running it again just breaks everything. I've even written a [tool](https://bitbucket.org/jonparrott/gaelic) to do exactly what you described- nuke the target directory and re-"install" everything.

Personally I think a command like "unpack" would perfectly suit the app engine community. I'm unfamiliar with pip internals and the particulars behind package installation, but I'm willing to dive in and take a first stab at implementing it. What are your thoughts, @webmaven?

Of course, if Google would add proper virtualenv support to app engine we'd be fine where we are. :(

@webmaven as you say, it's probably the AppEngine community that need to discuss alternative approaches. If we can fix upgrades with `--target` we will, but I'm not promising anything (it feels like a hard problem).

OK, here is the bare minimum of functionality that I think a notional `pip unpack` would need. For all operations, unpack needs to get dependencies by default in the exact same way install does:

`pip unpack <packagename> --target lib/`

`pip unpack -r requirements.txt --target lib/`

A current pain point not yet addressed by pip: deriving a list of installed packages and their versions from a target folder. Because it isn't installing anything, you need an alternate method of creating expanded requirements files that include the unpacked dependencies. Either of the following would work: `pip unpack -r requirements.txt --freeze > requirements.txt` or `pip freeze --target lib/ > requirements.txt`  Frankly, in the absence of an unpack command, the latter spelling would still be a good feature for pip, for the same reasons.

@jonparrott, can you think of anything else that is required to replace `pip install --target` for GAE's use case?

All you need to scan a directory for what's installed is:

``` python
import pkg_resources
env = pkg_resources.Environment([dirname])
for project in env:
    for dist in env[project]:
        print("{}=={}".format(dist.project_name, dist.version))
```

That's effectively what you were wanting from `pip freeze --target dirname`. It doesn't deal quite right with editable installations, but I suspect you'd have problems installing those with `--target` anyway.

Oh, and the `pip unpack` invocations you show work now, using `pip install` rather than `pip unpack`. Modulo the bug originally reported for this issue, of course ;-)

So it seems to me that you can probably use the "blow away the directory" approach along with the script to generate a requirements file as a workaround.

> It doesn't deal quite right with editable installations, but I suspect you'd have problems installing those with --target anyway.

@jonparrott, you have more experience than I in this area, but it doesn't seem to me, provided that the local install is in a virtualenv, that the `--editable` flag would be particularly problematic, unless you forgot to change it back before you deployed to GAE.

> Oh, and the pip unpack invocations you show work now, using pip install rather than pip unpack. Modulo the bug originally reported for this issue, of course ;-)

Of course.

But then, you were asking what it would take to replace `--target dirname` on the install command with something like `pip unpack`, well, the answer is that it needs to support those relevant parts of the install command (such as `--target`) and actually work for reinstalls (ie. 'modulo the bug'). It isn't very surprising that the resulting commands look so similar.

> So it seems to me that you can probably use the "blow away the directory" approach along with the script to generate a requirements file as a workaround.

Blowing away the contents of the directory only works for `pip install -r requirements.txt --target lib/` and not for `pip install <packagename> --target lib/`.

OK. I just looked at the code. If `--target` is set, all pip does is:
1. Set `ignore-installed` to true. (So that stuff already in `site-packages` doesn't block the install).
2. Create a temporary directory.
3. Do the install with `--home=the_temp_dir`
4. Move everything from the `lib_dir` of the temp home into the target dir.
5. Delete the temp dir.

So the nesting comes from step (4) as if the destination of `shutil.move` is a directory, the source is moved inside that directory. That's a clear bug, and fixable (although fixing is a bit fiddly). I may have a go at it, although I'm not promising when, as it's not a key issue for me (fixing the code isn't the hard bit, we'd want a test adding to confirm that the fix works). PRs are obviously welcome.

The "install into a temp dir" aspect explains most of the other inconsistencies, too. Multiple installs don't say "this is already there" because it isn't. Upgrade won't work because it's not there to upgrade. Editable would probably be a mess, I have no idea how it would work. "Fragile" is the most polite term I can think of for the behaviour...

Thanks for looking into this Paul and finding the source of one of the problems.

I think for App Engine's purposes, fixing upgrade isn't really necessary - if there's a way for us to overwrite anything in the destination directory instead of failing that would be sufficient.

I can try to see if I can write a test/PR this weekend for the fix if you don't beat me to it.

@pfmoore thanks for investigating! Like @jonparrott says, upgrading is desired but not required (desired because it makes using app engine that much more like the rest of the Python ecosystem). If we just get to idempotency for repeated installs, I think we'll both be quite pleased.

What happens if you skip steps 2, 4, and 5, and do the install with `--home=target_dir`?


Resolving patch:
diff --git a/pip/commands/install.py b/pip/commands/install.py
--- a/pip/commands/install.py
+++ b/pip/commands/install.py
@@ -51,7 +51,11 @@ def __init__(self, *args, **kw):
             dest='target_dir',
             metavar='dir',
             default=None,
-            help='Install packages into <dir>.')
+            help='Install packages into <dir>. '
+                 'By default this will not replace existing files/folders in '
+                 '<dir>. Use --upgrade to replace existing packages in <dir> '
+                 'with new versions.'
+        )
 
         cmd_opts.add_option(
             '-d', '--download', '--download-dir', '--download-directory',
@@ -344,9 +348,32 @@ def run(self, options, args):
                     os.makedirs(options.target_dir)
                 lib_dir = distutils_scheme('', home=temp_target_dir)['purelib']
                 for item in os.listdir(lib_dir):
+                    target_item_dir = os.path.join(options.target_dir, item)
+                    if os.path.exists(target_item_dir):
+                        if not options.upgrade:
+                            logger.warn(
+                                'Target directory %s already exists. Specify '
+                                '--upgrade to force replacement.'
+                                % target_item_dir
+                            )
+                            continue
+                        if os.path.islink(target_item_dir):
+                            logger.warn(
+                                'Target directory %s already exists and is '
+                                'a link. Pip will not automatically replace '
+                                'links, please remove if replacement is '
+                                'desired.'
+                                % target_item_dir
+                            )
+                            continue
+                        if os.path.isdir(target_item_dir):
+                            shutil.rmtree(target_item_dir)
+                        else:
+                            os.remove(target_item_dir)
+
                     shutil.move(
                         os.path.join(lib_dir, item),
-                        os.path.join(options.target_dir, item),
+                        target_item_dir
                     )
                 shutil.rmtree(temp_target_dir)
             return requirement_set


Test patch:
diff --git a/tests/functional/test_install.py b/tests/functional/test_install.py
--- a/tests/functional/test_install.py
+++ b/tests/functional/test_install.py
@@ -527,6 +527,33 @@ def test_install_package_with_target(script):
         str(result)
     )
 
+    # Test repeated call without --upgrade, no files should have changed
+    result = script.pip('install', '-t', target_dir, "initools==0.1")
+    assert not Path('scratch') / 'target' / 'initools' in result.files_updated
+
+    # Test upgrade call, check that new version is installed
+    result = script.pip('install', '--upgrade', '-t',
+                        target_dir, "initools==0.2")
+    assert Path('scratch') / 'target' / 'initools' in result.files_updated, (
+        str(result)
+    )
+    egg_folder = (
+        Path('scratch') / 'target' / 'INITools-0.2-py%s.egg-info' % pyversion)
+    assert egg_folder in result.files_created, (
+        str(result)
+    )
+
+    # Test install and upgrade of single-module package
+    result = script.pip('install', '-t', target_dir, 'six')
+    assert Path('scratch') / 'target' / 'six.py' in result.files_created, (
+        str(result)
+    )
+
+    result = script.pip('install', '-t', target_dir, '--upgrade', 'six')
+    assert Path('scratch') / 'target' / 'six.py' in result.files_updated, (
+        str(result)
+    )
+
 
 def test_install_package_with_root(script, data):
     """