Repository: xonsh/xonsh

Issue:
xpip doesn't detect/support "pip install --user" installs of xonsh
## xonfig

<details>

```
+------------------+----------------------+
| xonsh            | 0.9.27               |
| Git SHA          | 71fe9014             |
| Commit Date      | Jan 29 08:58:58 2021 |
| Python           | 3.9.5                |
| PLY              | 3.11                 |
| have readline    | True                 |
| prompt toolkit   | 3.0.19               |
| shell type       | prompt_toolkit       |
| pygments         | 2.9.0                |
| on posix         | True                 |
| on linux         | True                 |
| distro           | ubuntu               |
| on darwin        | False                |
| on windows       | False                |
| on cygwin        | False                |
| on msys2         | False                |
| is superuser     | False                |
| default encoding | utf-8                |
| xonsh encoding   | utf-8                |
| encoding errors  | surrogateescape      |
| on jupyter       | False                |
| jupyter kernel   | None                 |
| xontrib 1        | apt_tabcomplete      |
| xontrib 2        | direnv               |
| xontrib 3        | kitty                |
| xontrib 4        | linuxbrew            |
+------------------+----------------------+
```

</details>

## Expected Behavior
After installing xonsh via `pip3 install --user xonsh` (and ensuring that `~/.local/bin` is on `$PATH`, etc), xonsh works and runs just fine. Since `xpip` is supposed to manage the Python environment where xonsh itself is defined, I would expect it to wrap a non-root `pip`, ideally invoked in a way to install in the user's dir.

## Current Behavior
```
$ which xpip
sudo /usr/bin/python3 -m pip
```
Instead, `xpip` wraps a `sudo` invocation that will install things globally systemwide, which is not at all how xonsh itself was installed. And, if the user tries to do something "smart" like `xpip install --user xontrib-whatever`, I'm not sure quite what it will do but surely nothing good.

## Steps to Reproduce
1. Install xonsh via `pip3 install --user xonsh`
2. Run `xpip` to install something like a xonfig
3. Sadness and an unexpected `sudo` that might do undesired things to your system

## For community
⬇️  **Please click the 👍 reaction instead of leaving a `+1` or 👍  comment**


Discussion:
Hi! I agree this is pretty annoying and I can't think of any justification
for it.
PR welcome!

On Sat, Jul 17, 2021, 06:23 Daniel Egnor ***@***.***> wrote:

> xonfig
>
> +------------------+----------------------+
>
> | xonsh            | 0.9.27               |
>
> | Git SHA          | 71fe9014             |
>
> | Commit Date      | Jan 29 08:58:58 2021 |
>
> | Python           | 3.9.5                |
>
> | PLY              | 3.11                 |
>
> | have readline    | True                 |
>
> | prompt toolkit   | 3.0.19               |
>
> | shell type       | prompt_toolkit       |
>
> | pygments         | 2.9.0                |
>
> | on posix         | True                 |
>
> | on linux         | True                 |
>
> | distro           | ubuntu               |
>
> | on darwin        | False                |
>
> | on windows       | False                |
>
> | on cygwin        | False                |
>
> | on msys2         | False                |
>
> | is superuser     | False                |
>
> | default encoding | utf-8                |
>
> | xonsh encoding   | utf-8                |
>
> | encoding errors  | surrogateescape      |
>
> | on jupyter       | False                |
>
> | jupyter kernel   | None                 |
>
> | xontrib 1        | apt_tabcomplete      |
>
> | xontrib 2        | direnv               |
>
> | xontrib 3        | kitty                |
>
> | xontrib 4        | linuxbrew            |
>
> +------------------+----------------------+
>
>
> Expected Behavior
>
> After installing xonsh via pip3 install --user xonsh (and ensuring that
> ~/.local/bin is on $PATH, etc), xonsh works and runs just fine. Since xpip
> is supposed to manage the Python environment where xonsh itself is defined,
> I would expect it to wrap a non-root pip, ideally invoked in a way to
> install in the user's dir.
> Current Behavior
>
> $ which xpip
>
> sudo /usr/bin/python3 -m pip
>
>
> Instead, xpip wraps a sudo invocation that will install things globally
> systemwide, which is not at all how xonsh itself was installed. And, if the
> user tries to do something "smart" like xpip install --user
> xontrib-whatever, I'm not sure quite what it will do but surely nothing
> good.
> Steps to Reproduce
>
>    1. Install xonsh via pip3 install --user xonsh
>    2. Run xpip to install something like a xonfig
>    3. Sadness and an unexpected sudo that might do undesired things to
>    your system
>
> For community
>
> ⬇️ *Please click the 👍 reaction instead of leaving a +1 or 👍 comment*
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/xonsh/xonsh/issues/4369>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AELF3BMRFZSUVQ3OYRYF7X3TYDZT7ANCNFSM5AQT4QKQ>
> .
>

I completely agree. For me this is a dealbreaker that makes me refuse to use ```xpip```. I think many install issues are related to this, and the logical consequence that people look for other methods than ```xpip``` (see for example #4365  ).
@daniel-shimon, can you provide starting pointers / suggestions for how one might address this? (How is `xpip` set up currently, any gotchas one might look out for?)
I'm not familiar with it too much, the tricky part will probably be to decide whether to use `--user` or not
Notes to myself and/or anyone who works on this: Currently, xpip is defined in [`detect_xpip_alias()` in `aliases.py`](https://github.com/xonsh/xonsh/blob/main/xonsh/aliases.py#L762). Right now it's quite simple:

```
def detect_xpip_alias():
    """
    Determines the correct invocation to get xonsh's pip
    """
    if not getattr(sys, "executable", None):
        return lambda args, stdin=None: (
            "",
            "Sorry, unable to run pip on your system (missing sys.executable)",
            1,
        )

    basecmd = [sys.executable, "-m", "pip"]
    try:
        if ON_WINDOWS or IN_APPIMAGE:
            # XXX: Does windows have an installation mode that requires UAC?
            return basecmd
        elif not os.access(os.path.dirname(sys.executable), os.W_OK):
            return ["sudo"] + basecmd
        else:
            return basecmd
    except Exception:
        # Something freaky happened, return something that'll probably work
        return basecmd
```

(The silent `except Exception` seems unfortunate to me...) The current heuristic is: If the directory containing the "python" binary is writable, xpip==pip, otherwise xpip==sudo pip (both cases using the python binary that's running xonsh). This is clearly incomplete (understatement).

Important use cases:
- xonsh installed by `pip install` into a systemwide location that requires `sudo`
- xonsh installed by `pip install` into a systemwide location that does not require `sudo` (typical on Mac)
- xonsh installed by `pip install --user` into a per-user location (shouldn't require `sudo`)
- xonsh installed with one of the above and also `-e` (dev install), source dir referenced by `.pth` file
- xonsh installed with an "egg" referenced by `.pth` file
- xonsh installed by package manager into dist-packages (as "egg" or flat source)

Proposed stab at a better heuristic:
- If `__file__` is a descendant of any `sys.path` entry at or after (in path order) anything from `sys.getsitepackages()`, then apparently the running xonsh was installed systemwide. Check that `sys.path` entry's permissions for whether `sudo` is required. Use `[interpreter] -m pip install` (with no `--user`).
- Else, if `__file__` is a descendant of any `sys.path` entry at or after (in path order) the `site.getusersitepackages()` return, AND that directory is writable by the current user, then the running xonsh was installed locally. Use `[interpreter] -m pip install --user` (with no `sudo`).

The reason for the "at or after in path order" check is to handle the case where a `.pth` file is redirecting somewhere else, specifically this is common for installs with `-e`. This depends on the fact that the `site` module adds systemwide site packages and their `.pth` redirects _after_ user site packages and their redirects.

Regardless, when using `sudo`, I would suggest printing out a message and using `sudo -k` to always prompt for a password, rather than potentially surprising the user by taking actions as root; err on the side of safety when it comes to root. Also, we should stop swallowing Exception.

If this all sounds sane, I can certainly draft a PR, but I'd need some help testing on various configurations.
That sounds very interesting.

I was wondering about the following. Do you think that it may be possible, in the meantime, to implement some sort of ```--user-force``` flag that forces the use of the ```pip install --user``` command? That may allow advanced users to bypass these problems until fully solved with a better solution maybe? :)
Advanced users can just run `pip install --user` directly. I don't think anything ever forces you to use `xpip`, it's just a wrapper intended for convenience/robustness.
@egnor -- your proposal sounds reasonable to me.  A PR improving the heuristics is most welcome!

Resolving patch:
diff --git a/xonsh/aliases.py b/xonsh/aliases.py
--- a/xonsh/aliases.py
+++ b/xonsh/aliases.py
@@ -815,7 +815,7 @@ def detect_xpip_alias():
             # XXX: Does windows have an installation mode that requires UAC?
             return basecmd
         elif not os.access(os.path.dirname(sys.executable), os.W_OK):
-            return ["sudo"] + basecmd
+            return basecmd.extend(["--user"])
         else:
             return basecmd
     except Exception:


Test patch: