Repository: jazzband/pip-tools (Python)

Issue:
pip-compile bumps setuptools within commented-out spec
If you have a `requirements.in` which contains a package which depends on `setuptools`, for example:
```
$ cat requirements.in 
protobuf  # or google-api-core, ipython, pandas-gbq, pydata-google-auth
```
Then after running `pip-compile`, you get:
```
$ cat requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
protobuf==3.10.0
six==1.12.0               # via protobuf

# The following packages are considered to be unsafe in a requirements file:
# setuptools==41.4.0        # via protobuf
```

As you can see, `pip-compile` refuses to include `setuptools` directly in a generated `requirements.txt`, instead adding a section with the comment "The following packages are considered to be unsafe in a requirements file:".
This behaviour is fine in the general (and I don't disagree with the comment or behaviour).

However because the commented out `setuptools` line includes a version number, when `setuptools` releases an update then the `requirements.txt` file ends up looking out of date.

Say you had committed a `requirements.txt` which looked like:
```
# requirements...

# The following packages are considered to be unsafe in a requirements file:
# setuptools==41.4.0        # via protobuf
```

Then `setuptools` releases `41.5.0` and now when you run `pip-compile` you get:
```
# requirements...

# The following packages are considered to be unsafe in a requirements file:
# setuptools==41.5.0        # via protobuf
```

This is inconvenient if you happen to regenerate your `requirements.txt` at some point after an upstream releases, however if you're also using `pip-compile` to check that your `requirements.txt` is in step with your `requirements.in` (see #882) in CI then it means that your build now fails for what is essentially a non-change.

I'm not sure what the right solution is here, though perhaps if there was a way for pip-compile to understand its own commented out dependency that might help?


Discussion:
Hello @PeterJCLaw,

Sorry, I didn't quite understand what you are trying to achieve and how you use pip-compile in that way. Could you add more details, attach the `requirements.txt`, provide commands you run?
@atugushev sorry, yeah, on reflection I didn't include enough info. I've updated the description -- does that help?
I see. The option `--allow-unsafe` might help. Have you tried it?
Aha, thanks. I wasn't aware of that option -- I'll check that out. I think it would still be useful if there was a solution here which didn't involve doing something considered unsafe.
> Aha, thanks. I wasn't aware of that option -- I'll check that out. 

This gave me an idea to add a mention about `--allow-unsafe` in the comment line, like we have when you try to compile with `--generate-hashes` unsafe packages, for example:

```bash
$ echo -e "six\nsetuptools" | pip-compile - -qo- --generate-hashes
The generated requirements file may be rejected by pip install. See # WARNING lines for details.
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes --output-file=- -
#
six==1.12.0 \
    --hash=sha256:3350809f0555b11f552448330d0b52d5f24c91a322ea4a15ef22629740f3761c \
    --hash=sha256:d16a0141ec1a18405cd4ce8b4613101da75da0e9a7aec5bdd4fa804d0e0eba73

# WARNING: The following packages were not pinned, but pip requires them to be
# pinned when the requirements file includes hashes. Consider using the --allow-unsafe flag.
# setuptools==41.5.0
```
> I think it would still be useful if there was a solution here which didn't involve doing something considered unsafe.

The solution might be to unpin unsafe packages without `--allow-unsafe`, for example:

```bash
$ echo setuptools | pip-compile - -qo-
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file=- -
#

# The following packages are considered to be unsafe in a requirements file:
# setuptools
```

And pin unsafe packages with `--allow-unsafe`:

```bash
echo setuptools | pip-compile - -qo- --allow-unsafe
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --allow-unsafe --output-file=- -
#

# The following packages are considered to be unsafe in a requirements file:
setuptools==41.5.0
```
@PeterJCLaw 

I've submitted #975 which implements https://github.com/jazzband/pip-tools/issues/973#issuecomment-547010219. Please, review/test it if you have time.
> The solution might be to unpin unsafe packages without `--allow-unsafe`

FTR, it was already discussed before, see https://github.com/jazzband/pip-tools/pull/276#issuecomment-171567920, and fixed in #294, but looks like it has been broken later unintentionally.


@atugushev thanks for fixing this, do you know when this will be available in a release? (`setuptools` has been bumped recently, so would be great if this could be out soon)
@PeterJCLaw here is the tracking issue #1005. We'll release it very soon.
`pip-tools v4.3.0` is released.
Awesome, thanks!



Patch:
diff --git a/piptools/writer.py b/piptools/writer.py
index f54b58fa5..47cfbbc4a 100644
--- a/piptools/writer.py
+++ b/piptools/writer.py
@@ -189,17 +189,18 @@ def _iter_lines(
                 yield MESSAGE_UNSAFE_PACKAGES
 
             for ireq in unsafe_requirements:
-                req = self._format_requirement(
-                    ireq,
-                    reverse_dependencies,
-                    primary_packages,
-                    marker=markers.get(key_from_ireq(ireq)),
-                    hashes=hashes,
-                )
+                ireq_key = key_from_ireq(ireq)
                 if not self.allow_unsafe:
-                    yield comment("# {}".format(req))
+                    yield comment("# {}".format(ireq_key))
                 else:
-                    yield req
+                    line = self._format_requirement(
+                        ireq,
+                        reverse_dependencies,
+                        primary_packages,
+                        marker=markers.get(ireq_key),
+                        hashes=hashes,
+                    )
+                    yield line
 
         # Yield even when there's no real content, so that blank files are written
         if not yielded:


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
index 60624b922..e45456578 100644
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -691,8 +691,7 @@ def test_annotate_option(pip_conf, runner, option, expected):
 
 
 @pytest.mark.parametrize(
-    "option, expected",
-    [("--allow-unsafe", "\nsetuptools=="), (None, "\n# setuptools==")],
+    "option, expected", [("--allow-unsafe", "\nsetuptools=="), (None, "\n# setuptools")]
 )
 @pytest.mark.network
 def test_allow_unsafe_option(runner, option, expected):
diff --git a/tests/test_writer.py b/tests/test_writer.py
index 167226a69..f64e6380f 100644
--- a/tests/test_writer.py
+++ b/tests/test_writer.py
@@ -129,7 +129,7 @@ def test_iter_lines__unsafe_dependencies(writer, from_line, allow_unsafe):
         "test==1.2",
         "",
         MESSAGE_UNSAFE_PACKAGES,
-        "setuptools==1.10.0" if allow_unsafe else comment("# setuptools==1.10.0"),
+        "setuptools==1.10.0" if allow_unsafe else comment("# setuptools"),
     )
     assert tuple(lines) == expected_lines
 
@@ -147,7 +147,7 @@ def test_iter_lines__unsafe_with_hashes(writer, from_line):
         "test==1.2 \\\n    --hash=FAKEHASH",
         "",
         MESSAGE_UNSAFE_PACKAGES_UNPINNED,
-        comment("# setuptools==1.10.0"),
+        comment("# setuptools"),
     )
     assert tuple(lines) == expected_lines