Repository: pypa/pip (Python)

Issue:
Pip accepts requirements with extras that don't conform to PEP 508
**Description**
See #8285. The form `setuptools_scm>=3.5[toml]` does not conform to [PEP 508](https://www.python.org/dev/peps/pep-0508/#grammar). Pip accepts it, presumably for historical reasons.

We should deprecate and ultimately reject this form.

**Expected behavior**
Pip should only accept standard (PEP 508) formats for requirements.

**How to Reproduce**
See the linked issue.


Discussion:
I'm trying to get familiar with pip sources and can take a look at this.

I'll probably need a bit of advice along the way though: I've looked at this enough to see that the requirements/specifier handling is a bit redundant: it gets parsed several times with slightly different results and it's not 100% clear which of these parsing passes should check for this. More importantly it looks like the non-deprecated form `base[extra]>=1.0` gets mis-parsed in two places...

I'll make a potential fix for just this issue and then comment on the other weirdness once I understand it better.

Actually commenting now on the strange behaviour I'm seeing so I remember later:
* `parse_req_from_line()` returns no extras for input like "base[extra]>=1.0" in the returned RequirementParts
```
>>> req_parts = parse_req_from_line("base[extra]>=1.0", None)
>>> print (req_parts.requirement.extras, req_parts.extras)
{'extra'} set()
>>> req_parts = parse_req_from_line("base>=1.0[extra]", None)
>>> print (req_parts.requirement.extras, req_parts.extras)
set() {'extra'}
```
* that RequirementParts is used to build an InstallRequirement: that will however end up having extras (even if it was explicitly given an empty extras set as input)
* `parse_req_from_line()` uses `_strip_extras()` that is clearly buggy in context of "base[extra]>=1.0" input -- maybe this shouldn't be executed for plain requirement specifiers?

I think the major confusion with the methods in question is that there seems to be undocumented redundancy:
* what does it mean when a RequirementParts or InstallRequirement has both "extras" and "requirement.extras"
* what does it mean when a RequirementParts or InstallRequirement has both "markers" and "requirement.marker"

So InstallRequirements does this:

        if extras:
            self.extras = extras
        elif req:
            self.extras = {
                pkg_resources.safe_extra(extra) for extra in req.extras
            }
        else:
            self.extras = set()
I still can't find an explanation -- why are there two ways to define extras?

EDIT: after lots of git blame I think this is the origin of multiple ways to give extras to InstallRequirement: https://github.com/pypa/pip/issues/1236. Still don't understand why.
Man, I think we should just rewrite `RequirementParts` and get rid of `parts.extras` and `parts.marker` entirely. It is only used by `parse_req_from_line` and `parse_req_from_editable`, both are well-scoped and can be replaced safely.

I am not entirely sure, but assume most of the `extras` and `marker` cruft here are due to history baggage. `InstallRequirement` and its various constructors predate PEP 508 and the `packaging` library. pip had its own parsing logic at the time, and was refactored to use `packaging` afterwards, but a lot of the logic lingers to avoid breaking stuff (and because nobody really have much time to do the cleanups).
I'm on board. We will have to go through a deprecation cycle though, since this is changing user-facing behavior.
Right, that's the feel I got from the code as well -- although for a newcomer parse_req_from_line() is **not** well specified: refactoring anything seems scary because the input can be so many different things...
Trust me, I feel as uneasy as you are to the prospect. I feel less afriad of breaking things with experience, but still break as many things as when I started.
+1 from me too. And @jku don't worry, those nervous feelings are the sign of someone who knows what they are doing! The people I'd be worried about are people who look at this code and think it all seems reasonably straightforward 🙂 

On a more serious note, please feel free to ask if you're unsure. And trust your instincts - they've been good so far.
For others information: I have some ideas and might try the refactoring at some point but for now I'm not working on it -- feel free to have a go. 

I'll have another go with the PR #8424 now though: getting this specific misparsing marked as deprecated will be useful in any case.



Patch:
diff --git a/news/8288.removal b/news/8288.removal
new file mode 100644
index 00000000000..830d91aab95
--- /dev/null
+++ b/news/8288.removal
@@ -0,0 +1,1 @@
+Add deprecation warning for invalid requirements format "base>=1.0[extra]"
diff --git a/src/pip/_internal/req/constructors.py b/src/pip/_internal/req/constructors.py
index 46b1daa902e..857f7fff5aa 100644
--- a/src/pip/_internal/req/constructors.py
+++ b/src/pip/_internal/req/constructors.py
@@ -23,6 +23,7 @@
 from pip._internal.models.wheel import Wheel
 from pip._internal.pyproject import make_pyproject_path
 from pip._internal.req.req_install import InstallRequirement
+from pip._internal.utils.deprecation import deprecated
 from pip._internal.utils.filetypes import ARCHIVE_EXTENSIONS
 from pip._internal.utils.misc import is_installable_dir, splitext
 from pip._internal.utils.typing import MYPY_CHECK_RUNNING
@@ -370,6 +371,17 @@ def with_source(text):
             if add_msg:
                 msg += '\nHint: {}'.format(add_msg)
             raise InstallationError(msg)
+        else:
+            # Deprecate extras after specifiers: "name>=1.0[extras]"
+            # This currently works by accident because _strip_extras() parses
+            # any extras in the end of the string and those are saved in
+            # RequirementParts
+            for spec in req.specifier:
+                spec_str = str(spec)
+                if spec_str.endswith(']'):
+                    msg = "Extras after version '{}'.".format(spec_str)
+                    replace = "moving the extras before version specifiers"
+                    deprecated(msg, replacement=replace, gone_in="21.0")
     else:
         req = None
 


Test patch:
diff --git a/tests/functional/test_install_extras.py b/tests/functional/test_install_extras.py
index ba03f1e4aab..d70067b6bca 100644
--- a/tests/functional/test_install_extras.py
+++ b/tests/functional/test_install_extras.py
@@ -104,6 +104,27 @@ def test_nonexistent_options_listed_in_order(script, data):
     assert matches == ['nonexistent', 'nope']
 
 
+def test_install_deprecated_extra(script, data):
+    """
+    Warn about deprecated order of specifiers and extras.
+
+    Test uses a requirements file to avoid a testing issue where
+    the specifier gets interpreted as shell redirect.
+    """
+    script.scratch_path.joinpath("requirements.txt").write_text(
+        "requires_simple_extra>=0.1[extra]"
+    )
+    simple = script.site_packages / 'simple'
+
+    result = script.pip(
+        'install', '--no-index', '--find-links=' + data.find_links,
+        '-r', script.scratch_path / 'requirements.txt', expect_stderr=True,
+    )
+
+    result.did_create(simple)
+    assert ("DEPRECATION: Extras after version" in result.stderr)
+
+
 def test_install_special_extra(script):
     # Check that uppercase letters and '-' are dealt with
     # make a dummy project
diff --git a/tests/functional/test_new_resolver.py b/tests/functional/test_new_resolver.py
index 202a4b2b404..9329180334c 100644
--- a/tests/functional/test_new_resolver.py
+++ b/tests/functional/test_new_resolver.py
@@ -200,8 +200,6 @@ def test_new_resolver_ignore_dependencies(script):
     [
         "base[add]",
         "base[add] >= 0.1.0",
-        # Non-standard syntax. To deprecate, see pypa/pip#8288.
-        "base >= 0.1.0[add]",
     ],
 )
 def test_new_resolver_installs_extras(tmpdir, script, root_dep):
@@ -228,6 +226,32 @@ def test_new_resolver_installs_extras(tmpdir, script, root_dep):
     assert_installed(script, base="0.1.0", dep="0.1.0")
 
 
+def test_new_resolver_installs_extras_deprecated(tmpdir, script):
+    req_file = tmpdir.joinpath("requirements.txt")
+    req_file.write_text("base >= 0.1.0[add]")
+
+    create_basic_wheel_for_package(
+        script,
+        "base",
+        "0.1.0",
+        extras={"add": ["dep"]},
+    )
+    create_basic_wheel_for_package(
+        script,
+        "dep",
+        "0.1.0",
+    )
+    result = script.pip(
+        "install", "--use-feature=2020-resolver",
+        "--no-cache-dir", "--no-index",
+        "--find-links", script.scratch_path,
+        "-r", req_file,
+        expect_stderr=True
+    )
+    assert "DEPRECATION: Extras after version" in result.stderr
+    assert_installed(script, base="0.1.0", dep="0.1.0")
+
+
 def test_new_resolver_installs_extras_warn_missing(script):
     create_basic_wheel_for_package(
         script,