Repository: pypa/pip

Issue:
Disallow explicitly passing install-location-related arguments in --install-options
**What's the problem this feature will solve?**

Currently the `--install-option` parameter allows passing arbitrary arguments to `setup.py` for packages using the legacy-installation path. If any options/flags that control install location are passed, such as:

* `--exec-prefix`
* `--home`
* `--install-base`
* `--install-data`
* `--install-headers`
* `--install-lib`
* `--install-platlib`
* `--install-purelib`
* `--install-scripts`
* `--prefix`
* `--root`
* `--user`

then it causes issues. Not every package uses `setup.py`, so in the general case this causes legacy packages to be installed with the the scheme provided in `--install-option` and PEP 517/wheel packages to be installed with the scheme pip is aware of. This has already hit two users (#7253, #7240).

Pip knows how to use scheme information properly for PEP 517, wheel, and legacy installs. Enforcing that pip is the only one passing this kind of information will make the user experience more consistent.

**Describe the solution you'd like**

Explicitly fail `pip install` if `--install-option` includes any of the above arguments, whether provided on the command-line or in a requirements file.

In addition to the consistent user experience, this would also allow pip to pass the more explicit

* `--install-{purelib,platlib,headers,scripts,data}` (for `install`)
* `--install-dir`, `--script-dir` (for `install -e`)

to `setup.py` instead of the [current](https://github.com/pypa/pip/blob/53770a7184ddeb26d133ba77ddbb6a0c35af1ed5/src/pip/_internal/utils/setuptools_build.py#L165-L172) `--prefix`, `--home`, `--root`, `--user`, and `--install-headers`.

This may also be required to implement [PEP 582](https://www.python.org/dev/peps/pep-0582/) (`__pypackages__`) since doing legacy installs to a plain `lib` directory isn't possible with the current arguments being passed to `setup.py`, and passing an explicit `--install-lib` would break any usages of the other arguments in `--install-option`.

IMO this could be implemented without deprecation because the current behavior demonstrated in #7253 and #7240 causes failures in non-obvious ways, and the remediation is simple in most cases - just pass the parameters to pip instead.

**Alternative Solutions**

* do nothing - as we accumulate more features that conflict with these arguments being passed to `--install-option` we will get more support tickets. These will taper off eventually as people stop using these flags in order to install non-legacy packages. We eventually forget that the user can pass these options, and implement new features without worrying about it.
* automatically determine a scheme based on `--install-option` - this defeats the purpose of having pip-level arguments
* only fail the install if the scheme in `--install-option` conflicts with the one specified at the pip level - this is more friendly, but I feel that it would also be more confusing and add to the implementation burden in pip (reconciling `--install-option` across different packages in a requirements file)

**Additional context**

* #7253 - user already having a related issue
* #7240 - user already having a related issue
* #6052 - without executing the current proposal the refactoring mentioned here gets much more difficult, since we must continue to pass `--home`, `--prefix`, `--user`, and `--root` to setup.py


Discussion:
Sounds like a good idea to me.

We'd have to go through a regular deprecation cycle here but I don't see any reason to not do this.
I would argue that this is an extension of #6606/fix for #7240, and thus doesn't require a deprecation cycle. As mentioned above, the current behavior when using these flags is broken unless all builds are going through the legacy code path.
*nods*

I don't think this is an extension of #6606 -- that PR changed when pip used PEP 517, which is something we're still working on. This, however, affects everyone on the legacy code path. We'd be removing functionality that user might depend on directly, so we should go through the deprecation cycle for it.

FWIW, I now know that basically any change in pip's build logic will affect someone (i.e. break their deployment / CI pipeline) and they're usually not happy about it. With the power of retrospect, I feel we should've gone for at least short deprecation cycle (1 release) for #6606 since that was a (minor) change in end-user-affecting behavior as well.

OK. After thinking on it more, we can communicate the workaround in the deprecation notice. Hopefully that means currently impacted users will not have to search the issue tracker when some packages are missing, so I don't mind that approach as much as I did initially. It might even be worth including in a fix release for 19.3? Then we could also close #7240.
Given that we've already closed that issue (which is a form of communication on our release processes), I'm inclined to say no -- but it's really an RM's decision so... pinging @xavfernandez! :P

FWIW, I'm in no hurry to close #7240 but yes, even the deprecation messaging is enough to justify closing it.

I'm on board with forbidding conflicting options in `--install-options` and also in favor of a deprecation period.
Concerning a 19.3.2 release, it seems strange to make a patch release just to include a warning... but if everyone agrees (and if the patch is clean), why not :)
Note that #7002 (unreleased) will also break some usages of these options in `--install-option`, since we only take into account the `--user`, `--prefix`, `--target`, and `--root` options explicitly passed to pip when determining whether to do a user install.
After talking this over with @pradyunsg, our conclusions are:

1. we should maintain the 6 month deprecation period for this (so if we make a patch release or get this in 20.0, we remove in 20.2)
1. #6606 and #7002 should not be reverted, and we can treat the deprecation notice as a warning that things may not work as expected (with wheels and automatically-determined user-mode installation)
1. in the interim, we may be able to distill the options passed to `--install-option` into input for our scheme (potentially blocking if there is some conflict between `--install-option` and arguments passed directly to pip), which would let us press forward with other refactoring that requires pip to have more control over the setuptools arguments
For the more fine-grained option handling, I'm thinking the following:

## install options mapped to pip options

if provided in any `--install-option`, these options would be compared against all other requirements' `--install-option`s and the options provided to pip itself. If the pip equivalents are unset then they would be set to the provided options and the options removed from `--install-option`. If the pip equivalents are set to a different value, then we raise an error. If other requirements have a different value or do not have the same value in their `--install-option`s then we raise an error.

1. `--home` maps to `--target`
1. `--prefix` maps to `--prefix`
1. `--root` maps to `--root`
1. `--user` maps to `--user`

## install options that override scheme values directly

If provided in any `--install-option`, these options would be compared against all other requirements' `--install-option`s. If the other requirements have a different value or do not have the same value in their `--install-option` then we raise an error. If there are any pip-level location options then we raise an error. Otherwise we explicitly set the corresponding field on our install scheme after we generate the default one and remove the argument from `--install-option`.

1. `--install-data`
1. `--install-headers`
1. `--install-lib`
1. `--install-platlib`
1. `--install-purelib`
1. `--install-scripts`

## unsupported install options

These options, which do not have an equivalent in the scheme that we install packages to, are unsupported and if they are provided in any `--install-option`s then we raise an error

1. `--exec-prefix`
1. `--install-base`

---

I think this is the minimum (in terms of user-facing impact) we can do while still being able to move forward on creating an explicit scheme for install.

Just to reiterate why this is important: once the above is in place, we would be able to do the following:

1. reconcile input arguments
1. map input arguments to a [`Scheme`](https://github.com/pypa/pip/blob/59289f58f82a60389d3d4d8852e2337460c75125/src/pip/_internal/models/scheme.py#L9) object in `InstallCommand.run`
1. use that `Scheme` to instantiate a working set
1. use that working set for finding installed distributions during dependency resolution (cleaning up a lot of `Resolver`/`RequirementPreparer`/`InstallRequirement` in the process)
1. use that scheme as the basis for installation (and conflicting requirement uninstallation), which should fix several bugs (#1056, #4296, #5595)
Added to milestone 20.2.
Was a deprecation warning added in 19.3/20.2? Or is this an optimistic "we'll add the warning in 20.2"? 
@pradyunsg There is a [deprecation warning](https://github.com/pypa/pip/blob/7d875faff908e2131108e1bcc71475ecc031098e/src/pip/_internal/commands/install.py#L635) pointing to this issue, with gone_in="20.2". No feedback was reported here so I assume it's now appropriate to convert the warning into an error.
Awesome!
The relevant bit of code: https://github.com/pypa/pip/blob/4645ecbdb096373d6711406527b3b1fdc4fb785c/src/pip/_internal/commands/install.py#L635-L649

Resolving patch:
diff --git a/src/pip/_internal/commands/install.py b/src/pip/_internal/commands/install.py
--- a/src/pip/_internal/commands/install.py
+++ b/src/pip/_internal/commands/install.py
@@ -21,7 +21,6 @@
 from pip._internal.operations.check import check_install_conflicts
 from pip._internal.req import install_given_reqs
 from pip._internal.req.req_tracker import get_requirement_tracker
-from pip._internal.utils.deprecation import deprecated
 from pip._internal.utils.distutils_args import parse_distutils_args
 from pip._internal.utils.filesystem import test_writable_dir
 from pip._internal.utils.misc import (
@@ -290,7 +289,7 @@ def run(self, options, args):
         try:
             reqs = self.get_requirements(args, options, finder, session)
 
-            warn_deprecated_install_options(
+            reject_location_related_install_options(
                 reqs, options.install_options
             )
 
@@ -606,7 +605,7 @@ def decide_user_install(
     return True
 
 
-def warn_deprecated_install_options(requirements, options):
+def reject_location_related_install_options(requirements, options):
     # type: (List[InstallRequirement], Optional[List[str]]) -> None
     """If any location-changing --install-option arguments were passed for
     requirements or on the command-line, then show a deprecation warning.
@@ -639,20 +638,12 @@ def format_options(option_names):
     if not offenders:
         return
 
-    deprecated(
-        reason=(
-            "Location-changing options found in --install-option: {}. "
-            "This configuration may cause unexpected behavior and is "
-            "unsupported.".format(
-                "; ".join(offenders)
-            )
-        ),
-        replacement=(
-            "using pip-level options like --user, --prefix, --root, and "
-            "--target"
-        ),
-        gone_in="20.2",
-        issue=7309,
+    raise CommandError(
+        "Location-changing options found in --install-option: {}."
+        " This is unsupported, use pip-level options like --user,"
+        " --prefix, --root, and --target instead.".format(
+            "; ".join(offenders)
+        )
     )
 
 


Test patch:
diff --git a/tests/functional/test_install_reqs.py b/tests/functional/test_install_reqs.py
--- a/tests/functional/test_install_reqs.py
+++ b/tests/functional/test_install_reqs.py
@@ -1,3 +1,4 @@
+import json
 import os
 import textwrap
 
@@ -5,6 +6,7 @@
 
 from tests.lib import (
     _create_test_package_with_subdirectory,
+    create_basic_sdist_for_package,
     create_basic_wheel_for_package,
     need_svn,
     path_to_url,
@@ -15,6 +17,51 @@
 from tests.lib.path import Path
 
 
+class ArgRecordingSdist(object):
+    def __init__(self, sdist_path, args_path):
+        self.sdist_path = sdist_path
+        self._args_path = args_path
+
+    def args(self):
+        return json.loads(self._args_path.read_text())
+
+
+@pytest.fixture()
+def arg_recording_sdist_maker(script):
+    arg_writing_setup_py = textwrap.dedent(
+        """
+        import io
+        import json
+        import os
+        import sys
+
+        from setuptools import setup
+
+        args_path = os.path.join(os.environ["OUTPUT_DIR"], "{name}.json")
+        with open(args_path, 'w') as f:
+            json.dump(sys.argv, f)
+
+        setup(name={name!r}, version="0.1.0")
+        """
+    )
+    output_dir = script.scratch_path.joinpath(
+        "args_recording_sdist_maker_output"
+    )
+    output_dir.mkdir(parents=True)
+    script.environ["OUTPUT_DIR"] = str(output_dir)
+
+    def _arg_recording_sdist_maker(name):
+        # type: (str) -> ArgRecordingSdist
+        extra_files = {"setup.py": arg_writing_setup_py.format(name=name)}
+        sdist_path = create_basic_sdist_for_package(
+            script, name, "0.1.0", extra_files
+        )
+        args_path = output_dir / "{}.json".format(name)
+        return ArgRecordingSdist(sdist_path, args_path)
+
+    return _arg_recording_sdist_maker
+
+
 @pytest.mark.network
 def test_requirements_file(script):
     """
@@ -259,28 +306,21 @@ def test_wheel_user_with_prefix_in_pydistutils_cfg(
     assert 'installed requiresupper' in result.stdout
 
 
-def test_install_option_in_requirements_file(script, data, virtualenv):
-    """
-    Test --install-option in requirements file overrides same option in cli
-    """
-
-    script.scratch_path.joinpath("home1").mkdir()
-    script.scratch_path.joinpath("home2").mkdir()
-
-    script.scratch_path.joinpath("reqs.txt").write_text(
-        textwrap.dedent(
-            """simple --install-option='--home={home}'""".format(
-                home=script.scratch_path.joinpath("home1"))))
+def test_install_option_in_requirements_file_overrides_cli(
+    script, arg_recording_sdist_maker
+):
+    simple_sdist = arg_recording_sdist_maker("simple")
 
-    result = script.pip(
-        'install', '--no-index', '-f', data.find_links, '-r',
-        script.scratch_path / 'reqs.txt',
-        '--install-option=--home={home}'.format(
-            home=script.scratch_path.joinpath("home2")),
-        expect_stderr=True)
+    reqs_file = script.scratch_path.joinpath("reqs.txt")
+    reqs_file.write_text("simple --install-option='-O0'")
 
-    package_dir = script.scratch / 'home1' / 'lib' / 'python' / 'simple'
-    result.did_create(package_dir)
+    script.pip(
+        'install', '--no-index', '-f', str(simple_sdist.sdist_path.parent),
+        '-r', str(reqs_file), '--install-option=-O1',
+    )
+    simple_args = simple_sdist.args()
+    assert 'install' in simple_args
+    assert simple_args.index('-O1') < simple_args.index('-O0')
 
 
 def test_constraints_not_installed_by_default(script, data):
@@ -605,7 +645,7 @@ def test_install_unsupported_wheel_file(script, data):
     assert len(result.files_created) == 0
 
 
-def test_install_options_local_to_package(script, data):
+def test_install_options_local_to_package(script, arg_recording_sdist_maker):
     """Make sure --install-options does not leak across packages.
 
     A requirements.txt file can have per-package --install-options; these
@@ -613,27 +653,41 @@ def test_install_options_local_to_package(script, data):
     packages.  This needs to be a functional test because the bug was around
     cross-contamination at install time.
     """
-    home_simple = script.scratch_path.joinpath("for-simple")
-    test_simple = script.scratch.joinpath("for-simple")
-    home_simple.mkdir()
+
+    simple1_sdist = arg_recording_sdist_maker("simple1")
+    simple2_sdist = arg_recording_sdist_maker("simple2")
+
     reqs_file = script.scratch_path.joinpath("reqs.txt")
     reqs_file.write_text(
-        textwrap.dedent("""
-            simple --install-option='--home={home_simple}'
-            INITools
-            """.format(**locals())))
-    result = script.pip(
+        textwrap.dedent(
+            """
+            simple1 --install-option='-O0'
+            simple2
+            """
+        )
+    )
+    script.pip(
         'install',
-        '--no-index', '-f', data.find_links,
+        '--no-index', '-f', str(simple1_sdist.sdist_path.parent),
         '-r', reqs_file,
-        expect_stderr=True,
     )
 
-    simple = test_simple / 'lib' / 'python' / 'simple'
-    bad = test_simple / 'lib' / 'python' / 'initools'
-    good = script.site_packages / 'initools'
-    result.did_create(simple)
-    assert result.files_created[simple].dir
-    result.did_not_create(bad)
-    result.did_create(good)
-    assert result.files_created[good].dir
+    simple1_args = simple1_sdist.args()
+    assert 'install' in simple1_args
+    assert '-O0' in simple1_args
+    simple2_args = simple2_sdist.args()
+    assert 'install' in simple2_args
+    assert '-O0' not in simple2_args
+
+
+def test_location_related_install_option_fails(script):
+    simple_sdist = create_basic_sdist_for_package(script, "simple", "0.1.0")
+    reqs_file = script.scratch_path.joinpath("reqs.txt")
+    reqs_file.write_text("simple --install-option='--home=/tmp'")
+    result = script.pip(
+        'install',
+        '--no-index', '-f', str(simple_sdist.parent),
+        '-r', reqs_file,
+        expect_error=True
+    )
+    assert "['--home'] from simple" in result.stderr
diff --git a/tests/unit/test_command_install.py b/tests/unit/test_command_install.py
--- a/tests/unit/test_command_install.py
+++ b/tests/unit/test_command_install.py
@@ -7,8 +7,9 @@
 from pip._internal.commands.install import (
     create_env_error_message,
     decide_user_install,
-    warn_deprecated_install_options,
+    reject_location_related_install_options,
 )
+from pip._internal.exceptions import CommandError
 from pip._internal.req.req_install import InstallRequirement
 
 
@@ -44,16 +45,15 @@ def test_most_cases(
         assert decide_user_install(use_user_site=None) is result
 
 
-def test_deprecation_notice_for_pip_install_options(recwarn):
+def test_rejection_for_pip_install_options():
     install_options = ["--prefix=/hello"]
-    warn_deprecated_install_options([], install_options)
+    with pytest.raises(CommandError) as e:
+        reject_location_related_install_options([], install_options)
 
-    assert len(recwarn) == 1
-    message = recwarn[0].message.args[0]
-    assert "['--prefix'] from command line" in message
+    assert "['--prefix'] from command line" in str(e.value)
 
 
-def test_deprecation_notice_for_requirement_options(recwarn):
+def test_rejection_for_location_requirement_options():
     install_options = []
 
     bad_named_req_options = ["--home=/wow"]
@@ -67,18 +67,16 @@ def test_deprecation_notice_for_requirement_options(recwarn):
         None, "requirements2.txt", install_options=bad_unnamed_req_options
     )
 
-    warn_deprecated_install_options(
-        [bad_named_req, bad_unnamed_req], install_options
-    )
-
-    assert len(recwarn) == 1
-    message = recwarn[0].message.args[0]
+    with pytest.raises(CommandError) as e:
+        reject_location_related_install_options(
+            [bad_named_req, bad_unnamed_req], install_options
+        )
 
     assert (
         "['--install-lib'] from <InstallRequirement> (from requirements2.txt)"
-        in message
+        in str(e.value)
     )
-    assert "['--home'] from hello (from requirements.txt)" in message
+    assert "['--home'] from hello (from requirements.txt)" in str(e.value)
 
 
 @pytest.mark.parametrize('error, show_traceback, using_user_site, expected', [