Repository: pypa/pip

Issue:
pip install --user should check that the user's $PATH is correct
On Unix, `pip install --user ...` drops scripts into `~/.local/bin`. Historically, this directory has _not_ been on the default Debian/Ubuntu `$PATH`.

This is hopefully on its way to being fixed ([bash 4.3-15](http://metadata.ftp-master.debian.org/changelogs//main/b/bash/bash_4.3-15_changelog) is on its way into Debian unstable now, and should hopefully land in Debian testing in a few weeks, [bug report here](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820856); the Ubuntu bug that will hopefully be used to justify a backport of this fix into Xenial is [here](https://bugs.launchpad.net/ubuntu/+source/bash/+bug/1588562)). But, even once the fix lands, all it will do is change the default for _newly created_ accounts -- so for a long time, there are going to be people who try running `pip install --user` and end up with it dropping executables into a directory that's not on `$PATH`.

When `pip install --user` installs a script into `~/.local/bin` (or the equivalent on other OSes, why not), then it should check the current `os.environ["PATH"]`, and if this directory is not on the `PATH`, then it should print an explanatory message that warns the user and gives them some information on how to fix the problem.

(Or heck, it could even suggest they run `pip fix-user-path` to have pip automatically add tihs to their `~/.profile` -- it's a little wonky but not too hard to make reasonably robust, and if it's a manual action that tells the user what it's doing as it runs then it should be reasonable helpful.)

(This would be a reasonable first fix for someone new to pip -- does pip's issue tracker have a tag for those?)



Discussion:
I think this is a good idea, although my one worry is that some RC files exclude directories that don't already exist so that's a bit of a weird case where it would be in their `$PATH` if they opened a new shell but it's not now.

I'm sort of hovering between -0 and -1 on something like `pip fix-user-path` just because I find those automatic profile things to be error prone.

I guess I'm currently +0 on `pip fix-user-path` on the following rationale:
- For users who understand how `~/.profile` and friends work, then it's a small convenience that they'll use or ignore as they prefer, and if it breaks things then they'll be able to fix it.
- For users who don't understand how `~/.profile` and friends work, then the best they can hope to do is to blindly follow some complicated instructions, and if those instructions have an error then they're doomed regardless. So we might as well implement those instructions in the form of code, to at least eliminate human error and human pain.

For concreteness, I'm imagining a user interaction that would look something like

```
$ pip install --user virtualenv
Collecting virtualenv
  [...]
Successfully installed virtualenv-15.0.2

WARNING: this package provides a script called "virtualenv", and I put this
into your personal script directory, /home/njs/.local/bin/
BUT, your command line is not set up to look for scripts in this directory!
You won't be able to run the "virtualenv" script until you fix this.
For more information, see

   https://...

or, if you want me to try fixing your configuration for you automatically, run:

   pip fix-user-path

(You only have to do this once.)
$ pip fix-user-path
It looks like you're using the "bash" shell! Great, I know how to handle that.
I've added the following lines to the file /home/njs/.profile:

    # -- start lines automatically added by pip fix-user-path --
    export PATH=$HOME/.local/bin:$PATH
    # -- end lines automatically added by pip fix-user-path --

If this looks wrong, you can edit that file directly in your favorite text editor.

PLEASE NOTE: You probably need to log out and log back in before the
new configuration will take effect!
$ pip fix-user-path
It looks like you're using the "bash" shell! Great, I know how to handle that.
...I just checked the file /home/njs/.profile, and it looks like I've already fixed
your configuration for you, but it hasn't taken effect yet.
Most likely, this means you need to log out and log back in again.
If that doesn't help, [...]
$ logout
<start new shell>
$ pip fix-user-path
It looks like /home/njs/.local/bin is already on your $PATH, so there's
nothing for me to do -- you're all set!
```

> some RC files exclude directories that don't already exist

Oh, and on this point:

Fedora/RH unconditionally add `~/.local/bin` to `$PATH`: 

Debian/Ubuntu have historically done the annoying thing you mention with `~/bin`, _but_ that's irrelevant to us; the new patch that's working its way through the system to add `~/.local/bin` to the default `$PATH` configuration adds it unconditionally (and actually makes `~/bin` unconditional as well): https://sources.debian.net/src/bash/4.3-15/debian/skel.profile/

Is there anyone else you're worried about in particular?

I've seen plenty of other 3rd party python distributions maul people's `profile` scripts adding what they saw fit to paths which then made them a nightmare when later "uninstalled" or someone used something else. Definitely -1 on messing with that, especially given pip has X many linux distros to all work on as well as macOS.

@Ivoz: the difference is that (a) this path is always supposed to be there, there's no sense in which a user might "uninstall" their `~/.local` directory, and all we'd be doing is fixing existing user accounts to match how they were supposed to be set up all along, (b) pip by definition knows what the standard user script directory is on all linux distros and macOS, so there's no problem there, and (c) this is something that users would explicitly opt into.

"Don't mess with user profiles" isn't an option -- unless we want to disable `--user` entirely. The options are "force users to mess with their profiles by manually cutting-and-pasting things" or "provide a tool that does the cutting-and-pasting for them".

So here's a question, it seems to me that we can get most of the way there by just having a suggested snippet in the error message without any sort of fix-me-up command. Yes it'll require them to copy/paste but I think it's a fairly trivial operation for them to do that, and we avoid any sort of "pip broke my profile" issues.

Er, my question being, why not just do that?

That would also be a lot better than what we have now :-).

(Do keep in mind though that a large chunk of pip's userbase does not know what a "shell" is, or what a "dot-file" is, or even what a "text editor" is ("you mean like Google Docs?")... trivial is a relative term :-))

A cross-platform bash `source`-able `fix-me-up` would be useful for CI scripts
e.g.

``` bash
$ source <(pip fix-me-up-source)
```

e.g. https://github.com/travis-ci/travis-ci/issues/6247 could use something like that, so they dont need to hard-code the path names.

And it could be the basis for users automating the `.profile` fix themselves.

``` bash
$ pip fix-me-up-source >> ~/.profile
```

Alternatively, is there some way to to ask pip to emit the relevant directories it will use, in a machine  readable format, sort of like what `pkg-config` does for C libraries.

For CI scripts you probably just want to use virtualenvs or something instead of `--user`? Though yeah, it sucks if they just randomly allow `--user` but it's broken.

> Alternatively, is there some way to to ask pip to emit the relevant directories it will use, in a machine readable format, sort of like what pkg-config does for C libraries.

Maybe that should be a different bug, since this one is aimed at trying to fix a papercut that bites new users? (While you're at it, a similar thing I've been annoyed by is that there's no way to ask pip to say what wheels it thinks can be installed in the current environment.)

for info, I think there is a similar problem on windows?  I just tried on win 10 VM, and pip install --user put my script in `~/AppData/Roaming/Python/Scripts/`, which is not on the path (either in dos or bash)
Cross linking to #3813 because it has a dependency on this.
@pradyunsg 
Something must have gone wrong as you linked this issue to itself above.
Well, yes. I posted this on the wrong one. I'm sleepy.
Cross linking to #1668 because it has a dependency on this.
> (This would be a reasonable first fix for someone new to pip -- does pip's issue tracker have a tag for those?)

It does. I'll tag it. :)

Resolving patch:
diff --git a/src/pip/_internal/commands/install.py b/src/pip/_internal/commands/install.py
--- a/src/pip/_internal/commands/install.py
+++ b/src/pip/_internal/commands/install.py
@@ -158,6 +158,14 @@ def __init__(self, *args, **kw):
             help="Do not compile Python source files to bytecode",
         )
 
+        cmd_opts.add_option(
+            "--no-warn-script-location",
+            action="store_false",
+            dest="warn_script_location",
+            default=True,
+            help="Do not warn when installing scripts outside PATH",
+        )
+
         cmd_opts.add_option(cmdoptions.no_binary())
         cmd_opts.add_option(cmdoptions.only_binary())
         cmd_opts.add_option(cmdoptions.no_clean())
@@ -290,6 +298,7 @@ def run(self, options, args):
                         global_options,
                         root=options.root_path,
                         prefix=options.prefix_path,
+                        warn_script_location=options.warn_script_location,
                     )
 
                     possible_lib_locations = get_lib_location_guesses(
diff --git a/src/pip/_internal/req/req_install.py b/src/pip/_internal/req/req_install.py
--- a/src/pip/_internal/req/req_install.py
+++ b/src/pip/_internal/req/req_install.py
@@ -725,7 +725,7 @@ def match_markers(self, extras_requested=None):
             return True
 
     def install(self, install_options, global_options=None, root=None,
-                prefix=None):
+                prefix=None, warn_script_location=True):
         global_options = global_options if global_options is not None else []
         if self.editable:
             self.install_editable(
@@ -735,7 +735,10 @@ def install(self, install_options, global_options=None, root=None,
             version = wheel.wheel_version(self.source_dir)
             wheel.check_compatibility(version, self.name)
 
-            self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
+            self.move_wheel_files(
+                self.source_dir, root=root, prefix=prefix,
+                warn_script_location=warn_script_location,
+            )
             self.install_succeeded = True
             return
 
@@ -927,7 +930,8 @@ def check_if_exists(self):
     def is_wheel(self):
         return self.link and self.link.is_wheel
 
-    def move_wheel_files(self, wheeldir, root=None, prefix=None):
+    def move_wheel_files(self, wheeldir, root=None, prefix=None,
+                         warn_script_location=True):
         move_wheel_files(
             self.name, self.req, wheeldir,
             user=self.use_user_site,
@@ -936,6 +940,7 @@ def move_wheel_files(self, wheeldir, root=None, prefix=None):
             prefix=prefix,
             pycompile=self.pycompile,
             isolated=self.isolated,
+            warn_script_location=warn_script_location,
         )
 
     def get_dist(self):
diff --git a/src/pip/_internal/wheel.py b/src/pip/_internal/wheel.py
--- a/src/pip/_internal/wheel.py
+++ b/src/pip/_internal/wheel.py
@@ -3,6 +3,7 @@
 """
 from __future__ import absolute_import
 
+import collections
 import compileall
 import copy
 import csv
@@ -37,8 +38,12 @@
 )
 from pip._internal.utils.setuptools_build import SETUPTOOLS_SHIM
 from pip._internal.utils.temp_dir import TempDirectory
+from pip._internal.utils.typing import MYPY_CHECK_RUNNING
 from pip._internal.utils.ui import open_spinner
 
+if MYPY_CHECK_RUNNING:
+    from typing import Dict, List, Optional
+
 wheel_ext = '.whl'
 
 VERSION_COMPATIBLE = (1, 0)
@@ -140,8 +145,64 @@ def _split_ep(s):
     return console, gui
 
 
+def message_about_scripts_not_on_PATH(scripts):
+    # type: (List[str]) -> Optional[str]
+    """Determine if any scripts are not on PATH and format a warning.
+
+    Returns a warning message if one or more scripts are not on PATH,
+    otherwise None.
+    """
+    if not scripts:
+        return None
+
+    # Group scripts by the path they were installed in
+    grouped_by_dir = collections.defaultdict(set)  # type: Dict[str, set]
+    for destfile in scripts:
+        parent_dir = os.path.dirname(destfile)
+        script_name = os.path.basename(destfile)
+        grouped_by_dir[parent_dir].add(script_name)
+
+    path_env_var_parts = os.environ["PATH"].split(os.pathsep)
+    # Warn only for directories that are not on PATH
+    warn_for = {
+        parent_dir: scripts for parent_dir, scripts in grouped_by_dir.items()
+        if parent_dir not in path_env_var_parts
+    }
+    if not warn_for:
+        return None
+
+    # Format a message
+    msg_lines = []
+    for parent_dir, scripts in warn_for.items():
+        scripts = sorted(scripts)
+        if len(scripts) == 1:
+            start_text = "script {} is".format(scripts[0])
+        else:
+            start_text = "scripts {} are".format(
+                ", ".join(scripts[:-1]) + " and " + scripts[-1]
+            )
+
+        msg_lines.append(
+            "The {} installed in '{}' which is not on PATH."
+            .format(start_text, parent_dir)
+        )
+
+    last_line_fmt = (
+        "Consider adding {} to PATH or, if you prefer "
+        "to suppress this warning, use --no-warn-script-location."
+    )
+    if len(msg_lines) == 1:
+        msg_lines.append(last_line_fmt.format("this directory"))
+    else:
+        msg_lines.append(last_line_fmt.format("these directories"))
+
+    # Returns the formatted multiline message
+    return "\n".join(msg_lines)
+
+
 def move_wheel_files(name, req, wheeldir, user=False, home=None, root=None,
-                     pycompile=True, scheme=None, isolated=False, prefix=None):
+                     pycompile=True, scheme=None, isolated=False, prefix=None,
+                     warn_script_location=True):
     """Install a wheel"""
 
     if not scheme:
@@ -391,9 +452,16 @@ def _get_script_text(entry):
 
     # Generate the console and GUI entry points specified in the wheel
     if len(console) > 0:
-        generated.extend(
-            maker.make_multiple(['%s = %s' % kv for kv in console.items()])
+        generated_console_scripts = maker.make_multiple(
+            ['%s = %s' % kv for kv in console.items()]
         )
+        generated.extend(generated_console_scripts)
+
+        if warn_script_location:
+            msg = message_about_scripts_not_on_PATH(generated_console_scripts)
+            if msg is not None:
+                logger.warn(msg)
+
     if len(gui) > 0:
         generated.extend(
             maker.make_multiple(


Test patch:
diff --git a/tests/functional/test_install.py b/tests/functional/test_install.py
--- a/tests/functional/test_install.py
+++ b/tests/functional/test_install.py
@@ -1224,3 +1224,26 @@ def test_install_pep508_with_url_in_install_requires(script):
     res = script.pip('install', pkga_path, expect_error=True)
     assert "Direct url requirement " in res.stderr, str(res)
     assert "are not allowed for dependencies" in res.stderr, str(res)
+
+
+def test_installing_scripts_outside_path_prints_warning(script):
+    result = script.pip_install_local(
+        "--prefix", script.scratch_path, "script_wheel1", expect_error=True
+    )
+    assert "Successfully installed script-wheel1" in result.stdout, str(result)
+    assert "--no-warn-script-location" in result.stderr
+
+
+def test_installing_scripts_outside_path_can_suppress_warning(script):
+    result = script.pip_install_local(
+        "--prefix", script.scratch_path, "--no-warn-script-location",
+        "script_wheel1"
+    )
+    assert "Successfully installed script-wheel1" in result.stdout, str(result)
+    assert "--no-warn-script-location" not in result.stderr
+
+
+def test_installing_scripts_on_path_does_not_print_warning(script):
+    result = script.pip_install_local("script_wheel1")
+    assert "Successfully installed script-wheel1" in result.stdout, str(result)
+    assert "--no-warn-script-location" not in result.stderr
diff --git a/tests/unit/test_wheel.py b/tests/unit/test_wheel.py
--- a/tests/unit/test_wheel.py
+++ b/tests/unit/test_wheel.py
@@ -9,7 +9,6 @@
 from pip._internal.compat import WINDOWS
 from pip._internal.exceptions import InvalidWheelFilename, UnsupportedWheel
 from pip._internal.utils.misc import unpack_file
-
 from tests.lib import DATA_DIR
 
 
@@ -371,3 +370,86 @@ def test_skip_building_wheels(self, caplog):
             wb.build(Mock())
             assert "due to already being wheel" in caplog.text
             assert mock_build_one.mock_calls == []
+
+
+class TestMessageAboutScriptsNotOnPATH(object):
+
+    def _template(self, paths, scripts):
+        with patch.dict('os.environ', {'PATH': os.pathsep.join(paths)}):
+            return wheel.message_about_scripts_not_on_PATH(scripts)
+
+    def test_no_script(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=[]
+        )
+        assert retval is None
+
+    def test_single_script__single_dir_not_on_PATH(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=['/c/d/foo']
+        )
+        assert retval is not None
+        assert "--no-warn-script-location" in retval
+        assert "foo is installed in '/c/d'" in retval
+
+    def test_two_script__single_dir_not_on_PATH(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=['/c/d/foo', '/c/d/baz']
+        )
+        assert retval is not None
+        assert "--no-warn-script-location" in retval
+        assert "baz and foo are installed in '/c/d'" in retval
+
+    def test_multi_script__multi_dir_not_on_PATH(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=['/c/d/foo', '/c/d/bar', '/c/d/baz', '/a/b/c/spam']
+        )
+        assert retval is not None
+        assert "--no-warn-script-location" in retval
+        assert "bar, baz and foo are installed in '/c/d'" in retval
+        assert "spam is installed in '/a/b/c'" in retval
+
+    def test_multi_script_all__multi_dir_not_on_PATH(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=[
+                '/c/d/foo', '/c/d/bar', '/c/d/baz',
+                '/a/b/c/spam', '/a/b/c/eggs'
+            ]
+        )
+        assert retval is not None
+        assert "--no-warn-script-location" in retval
+        assert "bar, baz and foo are installed in '/c/d'" in retval
+        assert "eggs and spam are installed in '/a/b/c'" in retval
+
+    def test_two_script__single_dir_on_PATH(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=['/a/b/foo', '/a/b/baz']
+        )
+        assert retval is None
+
+    def test_multi_script__multi_dir_on_PATH(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=['/a/b/foo', '/a/b/bar', '/a/b/baz', '/c/d/bin/spam']
+        )
+        assert retval is None
+
+    def test_multi_script__single_dir_on_PATH(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=['/a/b/foo', '/a/b/bar', '/a/b/baz']
+        )
+        assert retval is None
+
+    def test_single_script__single_dir_on_PATH(self):
+        retval = self._template(
+            paths=['/a/b', '/c/d/bin'],
+            scripts=['/a/b/foo']
+        )
+        assert retval is None