Repository: pypa/pip

Issue:
When --python option is placed after the subcommand name in pip install, it gets ignored
### Description

Apologies if this is a duplicate issue, but I cannot for the life of me figure out how to use either GitHub or Google's search to search for the string "--python", since they either ignore the `--` or treat it as a negative search, and the string "python" is obviously not specific enough to identify just the CLI option.

From reading the [documentation on the `--python` option](https://pip.pypa.io/en/stable/topics/python-option/), it sounds like it exactly matches my use case of wanting to use pip to install a package into a different virtual environment. However, the commands in the example code block on that page don't work for me when I try to use a pip that was itself installed into a different virtual environment, since instead of installing to the virtual environment pointed to by the `--python` option, it installs to the virtual environment that pip itself is installed to.

I can't use the pip that is bundled with my python installation, since that version of pip is too old (22.2.1) to have the `--python` option. In my environment, we are running RHEL 8.7, but we are using a version of Python 3.10.6 that was compiled from source, and we cannot upgrade the version of pip that we installed at the time for reproducibility reasons. For all intents and purposes, you can treat this Python and pip as part of the "system", even if they were not packaged by Red Hat.

My use case is that I would like the ability to create minimal virtual environments that don't have pip installed and still be able to tell a separate installation of pip to install packages into that virtual environment. Since my "system" pip is too old to have the `--python` option, I would like to use a pip that was installed in one virtual environment to install packages into a different virtual environment.

### Expected behavior

I expected that the packages would be installed to the virtual environment pointed to by the `--python` option, not the virtual environment that `pip` itself is installed to. I would also at least expect a warning message if it is just going to ignore the `--python` CLI option.

### pip version

23.1.2

### Python version

3.10.6

### OS

RHEL 8.7

### How to Reproduce

```sh
# First create a virtual environment to just hold the specific version of pip
$ python3 -m venv venv-pip
$ ./venv-pip/bin/pip install pip==23.1.2
# Confirm that the virtual environment pip was correctly updated
$ ./venv-pip/bin/pip --version
# Then create a new virtual environment without pip
$ python3 -m venv venv-app --without-pip
$ ./venv-pip/bin/pip install --python ./venv-app/bin/python3 black
# venv-app doesn't have black installed
$ ls venv-app/bin/
# But venv-pip _does_ have black installed
$ ls venv-pip/bin/
```

### Output

```sh
# First create a virtual environment to just hold the specific version of pip
$ python3 -m venv venv-pip
$ ./venv-pip/bin/pip install pip==23.1.2
Collecting pip==23.1.2
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2.1
    Uninstalling pip-22.2.1:
      Successfully uninstalled pip-22.2.1
Successfully installed pip-23.1.2
# Confirm that the virtual environment pip was correctly updated
$ ./venv-pip/bin/pip --version
pip 23.1.2 from /home/rxia/tmp/venv-pip/lib/python3.10/site-packages/pip (python 3.10)

# Then create a new virtual environment without pip
$ python3 -m venv venv-app --without-pip
$ ./venv-pip/bin/pip install --python ./venv-app/bin/python3 black
Collecting black
  Using cached black-23.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)
Collecting click>=8.0.0 (from black)
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting mypy-extensions>=0.4.3 (from black)
  Using cached mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Collecting packaging>=22.0 (from black)
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting pathspec>=0.9.0 (from black)
  Using cached pathspec-0.11.1-py3-none-any.whl (29 kB)
Collecting platformdirs>=2 (from black)
  Using cached platformdirs-3.5.1-py3-none-any.whl (15 kB)
Collecting tomli>=1.1.0 (from black)
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
Installing collected packages: tomli, platformdirs, pathspec, packaging, mypy-extensions, click, black
Successfully installed black-23.3.0 click-8.1.3 mypy-extensions-1.0.0 packaging-23.1 pathspec-0.11.1 platformdirs-3.5.1 tomli-2.0.1

# venv-app doesn't have black installed
$ ls venv-app/bin/
activate  activate.csh  activate.fish  Activate.ps1  python  python3  python3.10

# But venv-pip _does_ have black installed
$ ls venv-pip/bin/
activate  activate.csh  activate.fish  Activate.ps1  black  blackd  python  python3  python3.10
```

### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
Sorry, user error. I realized that the `--python` option has to come _before_ the word `install`.

But I think it's still misleading for pip to silently ignore the `--python` option when it appears after the word `install` but not before. Should I leave this issue open to address that issue, or should I close this and file a new issue that more correctly describes the bug?
This sounds like a bug. I would expect the global `--python` option to be picked up anywhere on the command line (that's how other global options like `--verbose` work). But pip's option parsing is complex, and it looks like the `--python` option isn't getting recognised in that position. The problem is that we have to process `--python` before we identify the subcommand, but at that point we don't parse any arguments after the subcommand name.

I don't know how best to fix this. One option is to simply document the behaviour (and if you have any suggestions on how we could do that in a way that would have helped you understand what was going on here, that would be great). Or we could raise an error if `--python` is found in the subcommand options[^1]. But I'm not sure we could recognise the `--python` argument in the subcommand options without a significant restructuring of our parsing code.

I've retitled this issue to reflect the actual problem here.

[^1]: A brief outline of how we might do this, based on my initial investigations. I'm adding this here mainly so I don't forget it - I don't have time to make a PR now, but I don't want to lose the information. The `--python` option is recognised by subcommand parsers because it's in the `general_options` listing, so we need to remove it from there. Then, in order to have it recognised by the main parser, it needs to be added in as a special case in `pip._internal.cli.main_parser.create_main_parser`, just after the code marked "add the general options". I think that should do it, we'd need tests and probably still some documentation of this special case. I'm also not clear how this would get displayed in the help - that's something that would need to be checked.

Resolving patch:
diff --git a/src/pip/_internal/cli/base_command.py b/src/pip/_internal/cli/base_command.py
--- a/src/pip/_internal/cli/base_command.py
+++ b/src/pip/_internal/cli/base_command.py
@@ -131,6 +131,17 @@ def _main(self, args: List[str]) -> int:
                 ", ".join(sorted(always_enabled_features)),
             )
 
+        # Make sure that the --python argument isn't specified after the
+        # subcommand. We can tell, because if --python was specified,
+        # we should only reach this point if we're running in the created
+        # subprocess, which has the _PIP_RUNNING_IN_SUBPROCESS environment
+        # variable set.
+        if options.python and "_PIP_RUNNING_IN_SUBPROCESS" not in os.environ:
+            logger.critical(
+                "The --python option must be placed before the pip subcommand name"
+            )
+            sys.exit(ERROR)
+
         # TODO: Try to get these passing down from the command?
         #       without resorting to os.environ to hold these.
         #       This also affects isolated builds and it should.


Test patch:
diff --git a/tests/functional/test_install.py b/tests/functional/test_install.py
--- a/tests/functional/test_install.py
+++ b/tests/functional/test_install.py
@@ -1157,7 +1157,7 @@ def test_install_nonlocal_compatible_wheel(
         "--find-links",
         data.find_links,
         "--only-binary=:all:",
-        "--python",
+        "--python-version",
         "3",
         "--platform",
         "fakeplat",
@@ -1177,7 +1177,7 @@ def test_install_nonlocal_compatible_wheel(
         "--find-links",
         data.find_links,
         "--only-binary=:all:",
-        "--python",
+        "--python-version",
         "3",
         "--platform",
         "fakeplat",
diff --git a/tests/functional/test_python_option.py b/tests/functional/test_python_option.py
--- a/tests/functional/test_python_option.py
+++ b/tests/functional/test_python_option.py
@@ -39,3 +39,15 @@ def test_python_interpreter(
     script.pip("--python", env_path, "uninstall", "simplewheel", "--yes")
     result = script.pip("--python", env_path, "list", "--format=json")
     assert json.loads(result.stdout) == before
+
+
+def test_error_python_option_wrong_location(
+    script: PipTestEnvironment,
+    tmpdir: Path,
+    shared_data: TestData,
+) -> None:
+    env_path = os.fspath(tmpdir / "venv")
+    env = EnvBuilder(with_pip=False)
+    env.create(env_path)
+
+    script.pip("list", "--python", env_path, "--format=json", expect_error=True)