Repository: jazzband/pip-tools

Issue:
Pip-compile upgrades a package even it doesn't exist in a requrements.*
Pip-compile should raise an error while trying to upgrade a package which doesn't exist in a requirements.in.

##### Environment Versions

1. OS Type: any
1. Python version: any
1. pip version:  any
1. pip-tools version: any

##### Steps to replicate

1. echo "click" > requirements.in
2. pip-compile
3. pip-compile -P django

##### Expected result

Raise a error/warning or skip non-existing package.

##### Actual result

```
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile -P pytz
#
click==7.0
django==1.11.20
pytz==2018.9
```

Pip-compile upgrades a package even it doesn't exist in a requrements.*
Pip-compile should raise an error while trying to upgrade a package which doesn't exist in a requirements.in.

##### Environment Versions

1. OS Type: any
1. Python version: any
1. pip version:  any
1. pip-tools version: any

##### Steps to replicate

1. echo "click" > requirements.in
2. pip-compile
3. pip-compile -P django

##### Expected result

Raise a error/warning or skip non-existing package.

##### Actual result

```
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile -P pytz
#
click==7.0
django==1.11.20
pytz==2018.9
```



Discussion:
It's not clear if `-P` should or shouldn't work with SRC *in and/or DST *.txt requirements files. Would it make sense if `pip-compile -P django requirements.in` raise error but not `pip-compile -P django` ?

Hello @pawciobiel,

> if `pip-compile -P django requirements.in` raise error but not `pip-compile -P django`

What's the difference between those commands? Why they should have different behavior?



IMO it would be best for this to skip non-existing packages with a warning, rather than an error. That way I could safely loop through different `.in` files, upgrading a specified package if and only if it exists, without needing to check if a failure is a "real" problem or just an inappropriate upgrade attempt. 

FWIW, I'm using these Zsh functions to wrap `pip-compile -P` and `pip-compile -U`:

```bash
# recompile *requirements.txt with upgraded versions of all or specified packages (upgrade)
pipu () {  # [req...]
    local reqs=($@)
    for reqsin in *requirements.in(N); do
        print -rP "%F{cyan}> upgrading ${reqsin:r}.txt <- $reqsin . . .%f"
        if [[ $# -gt 0 ]]; then
            pip-compile --no-header ${${@/*/-P}:^reqs} $reqsin 2>&1 | hpype
            pipc $reqsin  # can remove if https://github.com/jazzband/pip-tools/issues/759 gets fixed
        else
            pip-compile --no-header -U $reqsin 2>&1 | hpype
        fi
    done
}
# upgrade with hashes
pipuh () {  # [req...]
    local reqs=($@)
    for reqsin in *requirements.in(N); do
        print -rP "%F{cyan}> upgrading ${reqsin:r}.txt <- $reqsin . . .%f"
        if [[ $# -gt 0 ]]; then
            pip-compile --no-header --generate-hashes ${${@/*/-P}:^reqs} $reqsin 2>&1 | hpype
            pipch $reqsin  # can remove if https://github.com/jazzband/pip-tools/issues/759 gets fixed
        else
            pip-compile --no-header -U --generate-hashes $reqsin 2>&1 | hpype
        fi
    done
}
```
@AndydeCleyre, probably, warning+skip is better than an error indeed. Thanks for the suggestion!
I second the need for _not_ raising error. And in my case (#950), I wouldn't want even warning.

Maybe `--upgrade-package` and `--upgrade-only-package` or something?
It's not clear if `-P` should or shouldn't work with SRC *in and/or DST *.txt requirements files. Would it make sense if `pip-compile -P django requirements.in` raise error but not `pip-compile -P django` ?

Hello @pawciobiel,

> if `pip-compile -P django requirements.in` raise error but not `pip-compile -P django`

What's the difference between those commands? Why they should have different behavior?



IMO it would be best for this to skip non-existing packages with a warning, rather than an error. That way I could safely loop through different `.in` files, upgrading a specified package if and only if it exists, without needing to check if a failure is a "real" problem or just an inappropriate upgrade attempt. 

FWIW, I'm using these Zsh functions to wrap `pip-compile -P` and `pip-compile -U`:

```bash
# recompile *requirements.txt with upgraded versions of all or specified packages (upgrade)
pipu () {  # [req...]
    local reqs=($@)
    for reqsin in *requirements.in(N); do
        print -rP "%F{cyan}> upgrading ${reqsin:r}.txt <- $reqsin . . .%f"
        if [[ $# -gt 0 ]]; then
            pip-compile --no-header ${${@/*/-P}:^reqs} $reqsin 2>&1 | hpype
            pipc $reqsin  # can remove if https://github.com/jazzband/pip-tools/issues/759 gets fixed
        else
            pip-compile --no-header -U $reqsin 2>&1 | hpype
        fi
    done
}
# upgrade with hashes
pipuh () {  # [req...]
    local reqs=($@)
    for reqsin in *requirements.in(N); do
        print -rP "%F{cyan}> upgrading ${reqsin:r}.txt <- $reqsin . . .%f"
        if [[ $# -gt 0 ]]; then
            pip-compile --no-header --generate-hashes ${${@/*/-P}:^reqs} $reqsin 2>&1 | hpype
            pipch $reqsin  # can remove if https://github.com/jazzband/pip-tools/issues/759 gets fixed
        else
            pip-compile --no-header -U --generate-hashes $reqsin 2>&1 | hpype
        fi
    done
}
```
@AndydeCleyre, probably, warning+skip is better than an error indeed. Thanks for the suggestion!
I second the need for _not_ raising error. And in my case (#950), I wouldn't want even warning.

Maybe `--upgrade-package` and `--upgrade-only-package` or something?

Resolving patch:
diff --git a/piptools/scripts/compile.py b/piptools/scripts/compile.py
--- a/piptools/scripts/compile.py
+++ b/piptools/scripts/compile.py
@@ -282,6 +282,8 @@ def cli(
         key_from_req(install_req.req): install_req for install_req in upgrade_reqs_gen
     }
 
+    existing_pins_to_upgrade = set()
+
     # Proxy with a LocalRequirementsRepository if --upgrade is not specified
     # (= default invocation)
     if not upgrade and os.path.exists(output_file.name):
@@ -293,13 +295,14 @@ def cli(
         )
 
         # Exclude packages from --upgrade-package/-P from the existing
-        # constraints
-        existing_pins = {
-            key_from_req(ireq.req): ireq
-            for ireq in ireqs
-            if is_pinned_requirement(ireq)
-            and key_from_req(ireq.req) not in upgrade_install_reqs
-        }
+        # constraints, and separately gather pins to be upgraded
+        existing_pins = {}
+        for ireq in filter(is_pinned_requirement, ireqs):
+            key = key_from_req(ireq.req)
+            if key in upgrade_install_reqs:
+                existing_pins_to_upgrade.add(key)
+            else:
+                existing_pins[key] = ireq
         repository = LocalRequirementsRepository(existing_pins, repository)
 
     ###
@@ -345,7 +348,10 @@ def cli(
         key_from_ireq(ireq) for ireq in constraints if not ireq.constraint
     }
 
-    constraints.extend(upgrade_install_reqs.values())
+    allowed_upgrades = primary_packages | existing_pins_to_upgrade
+    constraints.extend(
+        ireq for key, ireq in upgrade_install_reqs.items() if key in allowed_upgrades
+    )
 
     # Filter out pip environment markers which do not match (PEP496)
     constraints = [


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -370,6 +370,22 @@ def test_upgrade_packages_option(pip_conf, runner):
     assert "small-fake-b==0.3" in out.stderr
 
 
+def test_upgrade_packages_option_irrelevant(pip_conf, runner):
+    """
+    piptools ignores --upgrade-package/-P items not already constrained.
+    """
+    with open("requirements.in", "w") as req_in:
+        req_in.write("small-fake-a")
+    with open("requirements.txt", "w") as req_in:
+        req_in.write("small-fake-a==0.1")
+
+    out = runner.invoke(cli, ["--upgrade-package", "small-fake-b"])
+
+    assert out.exit_code == 0
+    assert "small-fake-a==0.1" in out.stderr.splitlines()
+    assert "small-fake-b==0.3" not in out.stderr
+
+
 def test_upgrade_packages_option_no_existing_file(pip_conf, runner):
     """
     piptools respects --upgrade-package/-P inline list when the output file
@@ -385,20 +401,30 @@ def test_upgrade_packages_option_no_existing_file(pip_conf, runner):
     assert "small-fake-b==0.3" in out.stderr
 
 
-def test_upgrade_packages_version_option(pip_conf, runner):
+@pytest.mark.parametrize(
+    "current_package, upgraded_package",
+    (
+        pytest.param("small-fake-b==0.1", "small-fake-b==0.3", id="upgrade"),
+        pytest.param("small-fake-b==0.3", "small-fake-b==0.1", id="downgrade"),
+    ),
+)
+def test_upgrade_packages_version_option(
+    pip_conf, runner, current_package, upgraded_package
+):
     """
     piptools respects --upgrade-package/-P inline list with specified versions.
     """
     with open("requirements.in", "w") as req_in:
         req_in.write("small-fake-a\nsmall-fake-b")
     with open("requirements.txt", "w") as req_in:
-        req_in.write("small-fake-a==0.1\nsmall-fake-b==0.1")
+        req_in.write("small-fake-a==0.1\n" + current_package)
 
-    out = runner.invoke(cli, ["-P", "small-fake-b==0.2"])
+    out = runner.invoke(cli, ["--no-annotate", "--upgrade-package", upgraded_package])
 
     assert out.exit_code == 0
-    assert "small-fake-a==0.1" in out.stderr
-    assert "small-fake-b==0.2" in out.stderr
+    stderr_lines = out.stderr.splitlines()
+    assert "small-fake-a==0.1" in stderr_lines
+    assert upgraded_package in stderr_lines
 
 
 def test_upgrade_packages_version_option_no_existing_file(pip_conf, runner):
@@ -870,3 +896,36 @@ def test_options_in_requirements_file(runner, options):
 
     with open("requirements.txt") as reqs_txt:
         assert options in reqs_txt.read().splitlines()
+
+
+@pytest.mark.parametrize(
+    "current_package, upgraded_package",
+    (
+        pytest.param("small-fake-b==0.1", "small-fake-b==0.2", id="upgrade"),
+        pytest.param("small-fake-b==0.2", "small-fake-b==0.1", id="downgrade"),
+    ),
+)
+def test_upgrade_packages_option_subdependency(
+    pip_conf, runner, current_package, upgraded_package
+):
+    """
+    Test that pip-compile --upgrade-package/-P upgrades/dpwngrades subdependencies.
+    """
+
+    with open("requirements.in", "w") as reqs:
+        reqs.write("small-fake-with-unpinned-deps\n")
+
+    with open("requirements.txt", "w") as reqs:
+        reqs.write("small-fake-a==0.1\n")
+        reqs.write(current_package + "\n")
+        reqs.write("small-fake-with-unpinned-deps==0.1\n")
+
+    out = runner.invoke(
+        cli, ["--no-annotate", "--dry-run", "--upgrade-package", upgraded_package]
+    )
+
+    stderr_lines = out.stderr.splitlines()
+    assert "small-fake-a==0.1" in stderr_lines, "small-fake-a must keep its version"
+    assert (
+        upgraded_package in stderr_lines
+    ), "{} must be upgraded/downgraded to {}".format(current_package, upgraded_package)
diff --git a/tests/test_data/minimal_wheels/small_fake_with_unpinned_deps-0.1-py2.py3-none-any.whl b/tests/test_data/minimal_wheels/small_fake_with_unpinned_deps-0.1-py2.py3-none-any.whl
new file mode 100644
Binary files /dev/null and b/tests/test_data/minimal_wheels/small_fake_with_unpinned_deps-0.1-py2.py3-none-any.whl differ
diff --git a/tests/test_data/packages/small_fake_with_unpinned_deps/setup.py b/tests/test_data/packages/small_fake_with_unpinned_deps/setup.py
new file mode 100644
--- /dev/null
+++ b/tests/test_data/packages/small_fake_with_unpinned_deps/setup.py
@@ -0,0 +1,7 @@
+from setuptools import setup
+
+setup(
+    name="small_fake_with_unpinned_deps",
+    version=0.1,
+    install_requires=["small-fake-a", "small-fake-b"],
+)