Repository: jazzband/pip-tools

Issue:
--upgrade-packages fails to constrain subdeps, if either absent from a preexisting output file, or if --upgrade is also passed
The documentation states that the `--upgrade` and `--upgrade-packages` [can be used together](https://github.com/jazzband/pip-tools/blob/6.4.0/README.rst#updating-requirements), however the `--upgrade-packages` flag is ignored when `--upgrade` is supplied.

#### Environment Versions

1. OS Type: macOS and Ubuntu 18.04
2. Python version: `python 3.9.0`
3. pip version: `pip 21.3.1`
4. pip-tools version: `pip-compile 6.4.0`

#### Steps to replicate

_Using different args to the documentation, as `requests` is not in the `django` requirements, and `requests<=3.0.0` does not affect the "latest" release._

```bash
# Create requirements.in file
echo "django" > requirements.in
# Run pip-compile
pip-compile --upgrade --upgrade-package 'sqlparse<=0.4.0'
```

#### Expected result

```
asgiref==3.4.1
    # via django
django==4.0.1
    # via -r requirements.in
sqlparse==0.4.0
    # via django
```

#### Actual result

```
asgiref==3.4.1
    # via django
django==4.0.1
    # via -r requirements.in
sqlparse==0.4.2 
    # via django
```

`sqlparse` is not restricted to `<=0.4.0`. If I omit the `--upgrade` flag, only supplying `--upgrade-packages 'sqlparse<=0.4.0'`, I get the expected result (but my other packages are not upgraded to later versions). This also doesn't work if the argument is `sqlparse==0.4.0`.

My current workaround is just to delete the requirements file(s) before running `pip-compile`, and omitting the `--upgrade` flag so the existing files aren't used as a cache. I suspect either the `--upgrade-packages` constraint logic is inside the `if not upgrade...` [conditional](https://github.com/jazzband/pip-tools/blob/6.4.0/piptools/scripts/compile.py#L354), or that `--upgrade-packages` only [extend](https://github.com/jazzband/pip-tools/blob/6.4.0/piptools/scripts/compile.py#L437), not constrain the complete set of packages.

This feature should be permitted, or the documentation changed to avoid stating that this is possible


Discussion:
After some more investigation, this issue occurs without the `--upgrade` flag if there is no existing output file. It only "works" if `--upgrade` is not given and the output file is present (therefore the pip-compile [conditional branch](https://github.com/jazzband/pip-tools/blob/6.4.0/piptools/scripts/compile.py#L354) occurs. 

This actually affects my workaround above, as I'd have to run `pip-compile` twice after deleting the previous requirements files; first to use the latest versions, second to recognise the `--upgrade-packages` versions.
Looks like this behavior was changed in https://github.com/jazzband/pip-tools/pull/1031. There's relevant [discussion](https://github.com/jazzband/pip-tools/pull/1031#pullrequestreview-344925098) in that PR, but I can't tell whether this exact case was intentionally changed. @AndydeCleyre might be able to share more.
I did not realize they were intended to potentially be used together, as I didn't see sense in the sentiment "keep/install an old version of package x AND upgrade all packages to their latest" -- I figured the upgrade-all bit would trump the other bit. I was wrong.

I'll try to review this whole thing.
Until then, does it help to put your restricted versions into the input file?
That makes sense. The `requirements.in` file approach would probably work, yeah. This night just be a documentation issue then, if it's not the intended behaviour, because the docs have an example showing the use together.

I can see why the sentiment of upgrading all and pinning a specific package would seem strange. In my specific case, I'm actually trying to force the requirements file to contain a specific, CPU-only flavour of PyTorch with `torch==1.10.0+cpu`, but upgrade all the other "standard" packages. At the moment, I end up with `torch==1.10.0` in my `requirements.txt`. So it's a bit of a non-standard use-case
OK, reviewing the discussion at #759, I think a key note is right at the start. @atugushev opened with:

> Pip-compile should raise an error while trying to upgrade a package which doesn't exist in a requirements.in.

Through the discussion, the "penalty" was softened and we did not end up raising any errors in this case.

But the expectation there illuminated is that packages passed to `--upgrade-package` are either present in the input file, or considered irrelevant. In other words, any package for which you want to control the version should be in the input file. That's my understanding, anyway.

With that in mind:

`requirements.in`:
```ini
django
sqlparse  # for django
```

```console
$ pip-compile --upgrade-package 'sqlparse<=0.4.0'
asgiref==3.5.0
    # via django
django==4.0.2
    # via -r requirements.in
sqlparse==0.4.0
    # via
    #   -r requirements.in
    #   django
$ pip-compile --upgrade --upgrade-package 'sqlparse<=0.4.0'
asgiref==3.5.0
    # via django
django==4.0.2
    # via -r requirements.in
sqlparse==0.4.0
    # via
    #   -r requirements.in
    #   django
$ rm requirements.txt
$ pip-compile --upgrade --upgrade-package 'sqlparse<=0.4.0'
asgiref==3.5.0
    # via django
django==4.0.2
    # via -r requirements.in
sqlparse==0.4.0
    # via
    #   -r requirements.in
    #   django
```

So

- is that a reasonable and good expectation?
- if so
  - does it address the issues here?
  - we probably should make that clearer in the docs
> Pip-compile should raise an error while trying to upgrade a package which doesn't exist in a requirements.in.

> is that a reasonable and good expectation?

For what it's worth, I relied on the behavior requested here (upgrading specific packages not declared in reqs.in) for a while and didn't realize it was no longer available! Certainly a question is whether the complexity is worthwhile, but I found it really valuable to be able to easily upgrade a specific sub-dependency that had recently had a security fix release. `pip-compile` ensured I still had a set of compatible packages without forcing me to upgrade everything. 

Of course, that use case can be worked around by temporarily adding the sub-dependency to reqs.in and then removing it, but it would be nice to have that automated. I didn't want the particular package in reqs.in permanently - if the relevant direct dependency no longer needed it, I would want it to disappear from the output.
I'm not saying things should be one way or another here, but want to mention a relevant alternative:

`security.txt`:
```ini
sqlparse<=0.4.0  # See https://...
```

`requirements.in`:
```ini
django
-c security.txt
```
Yep, definitely an option as well!
So is there a good reason not to treat `--upgrade-package` items identically to a `-c constraints.txt` file containing those items? 
Oh right yes: because of the simple case of upgrading to the max. Never mind.

Resolving patch:
diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -425,10 +425,7 @@ def cli(
         key_from_ireq(install_req): install_req for install_req in upgrade_reqs_gen
     }
 
-    existing_pins_to_upgrade = set()
-
-    # Exclude packages from --upgrade-package/-P from the existing
-    # constraints, and separately gather pins to be upgraded
+    # Exclude packages from --upgrade-package/-P from the existing constraints
     existing_pins = {}
 
     # Proxy with a LocalRequirementsRepository if --upgrade is not specified
@@ -446,9 +443,7 @@ def cli(
 
         for ireq in filter(is_pinned_requirement, ireqs):
             key = key_from_ireq(ireq)
-            if key in upgrade_install_reqs:
-                existing_pins_to_upgrade.add(key)
-            else:
+            if key not in upgrade_install_reqs:
                 existing_pins[key] = ireq
         repository = LocalRequirementsRepository(
             existing_pins, repository, reuse_hashes=reuse_hashes
@@ -515,6 +510,27 @@ def cli(
                 )
             )
 
+    if upgrade_packages:
+        constraints_file = tempfile.NamedTemporaryFile(mode="wt", delete=False)
+        constraints_file.write("\n".join(upgrade_packages))
+        constraints_file.flush()
+        try:
+            reqs = list(
+                parse_requirements(
+                    constraints_file.name,
+                    finder=repository.finder,
+                    session=repository.session,
+                    options=repository.options,
+                    constraint=True,
+                )
+            )
+        finally:
+            constraints_file.close()
+            os.unlink(constraints_file.name)
+        for req in reqs:
+            req.comes_from = None
+        constraints.extend(reqs)
+
     extras = tuple(itertools.chain.from_iterable(ex.split(",") for ex in extras))
 
     if extras and not setup_file_found:
@@ -525,9 +541,8 @@ def cli(
         key_from_ireq(ireq) for ireq in constraints if not ireq.constraint
     }
 
-    allowed_upgrades = primary_packages | existing_pins_to_upgrade
     constraints.extend(
-        ireq for key, ireq in upgrade_install_reqs.items() if key in allowed_upgrades
+        ireq for key, ireq in upgrade_install_reqs.items() if key in primary_packages
     )
 
     constraints = [req for req in constraints if req.match_markers(extras)]


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -894,13 +894,20 @@ def test_upgrade_packages_version_option_no_existing_file(pip_conf, runner):
     assert "small-fake-b==0.2" in out.stderr
 
 
-def test_upgrade_packages_version_option_and_upgrade(pip_conf, runner):
+@pytest.mark.parametrize(
+    "reqs_in",
+    (
+        pytest.param("small-fake-a\nsmall-fake-b", id="direct reqs"),
+        pytest.param("small-fake-with-unpinned-deps", id="parent req"),
+    ),
+)
+def test_upgrade_packages_version_option_and_upgrade(pip_conf, runner, reqs_in):
     """
     piptools respects --upgrade-package/-P inline list with specified versions
     whilst also doing --upgrade.
     """
     with open("requirements.in", "w") as req_in:
-        req_in.write("small-fake-a\nsmall-fake-b")
+        req_in.write(reqs_in)
     with open("requirements.txt", "w") as req_in:
         req_in.write("small-fake-a==0.1\nsmall-fake-b==0.1")
 
@@ -1895,6 +1902,7 @@ def test_unreachable_index_urls(runner, cli_options, expected_message):
     assert expected_message in stderr_lines
 
 
+@pytest.mark.parametrize("subdep_already_pinned", (True, False))
 @pytest.mark.parametrize(
     ("current_package", "upgraded_package"),
     (
@@ -1903,7 +1911,7 @@ def test_unreachable_index_urls(runner, cli_options, expected_message):
     ),
 )
 def test_upgrade_packages_option_subdependency(
-    pip_conf, runner, current_package, upgraded_package
+    pip_conf, runner, current_package, upgraded_package, subdep_already_pinned
 ):
     """
     Test that pip-compile --upgrade-package/-P upgrades/downgrades subdependencies.
@@ -1914,7 +1922,8 @@ def test_upgrade_packages_option_subdependency(
 
     with open("requirements.txt", "w") as reqs:
         reqs.write("small-fake-a==0.1\n")
-        reqs.write(current_package + "\n")
+        if subdep_already_pinned:
+            reqs.write(current_package + "\n")
         reqs.write("small-fake-with-unpinned-deps==0.1\n")
 
     out = runner.invoke(