Repository: jazzband/pip-tools

Issue:
Dependencies are droped when using a combination of -r & -c in layered requirements
First of all, thanks a lot for developing and maintaining the package! It is a great tool!

<!-- Describe the issue briefly here. -->
#### Background

Regarding the issue, when using layered requirements (eg `test.txt` being a superset of `main.txt`), two referencing modes are available `-r` that defines requirements, and `-c` that defines constraints.

Including both `-r main.in` and `-c main.txt` in `test.in` should ensure that `test.txt` is a superset of `main.txt`. It should produce the same results as including just `-r main.txt` except by the fact that it keeps the comments indicating where indirect dependencies came from.


#### Issue description

However, **using  both `-r main.in` and `-c main.txt` at the same time drops some dependencies from `main.txt` that are not propagated to `test.txt`**.

When installing `pip install -r test.txt`, the missing packages are installed anyway (they were dependencies, indeed). And in fact, the `--verbose` flag  of pip-tools shows that the missing packages are taken into account until the very end of the compilation process but are not included in the output.

On the other hand, using only `-r main.in` does not miss any package (but does not ensure that the versions are the same).

#### Environment Versions

1. OS version: docker image python:3.7-slim, but also in other environments.
1. Python version: Python 3.7.6
1. pip version:  19.3.1
1. pip-tools version: pip-compile, version 4.4.0

#### Steps to replicate
To ensure reproducibility I ran this in a docker, but I was able to reproduce it elsewhere:
```
docker run -it --rm python:3.7-slim bash
```
1. Install pip-tools
```
pip install pip-tools
```
2. Create two simple layered requirements with
```
echo pandas > main.in
echo -e "-r main.in\n-c main.txt" > test.in
```
2. Compile `main.in` requirements with:  ->  [[captured output]](https://github.com/jazzband/pip-tools/files/4104410/output0.txt)
```
python -m piptools compile --verbose --allow-unsafe --build-isolation --no-header --no-index main.in --output-file main.txt
```

3. Compile `test.in` requirements with:  ->  [[captured output]](https://github.com/jazzband/pip-tools/files/4104397/output1.txt)
```
python -m piptools compile --verbose --allow-unsafe --build-isolation --no-header --no-index test.in --output-file test.txt
```

#### Expected result
Every package listed in `main.txt` and its version should appear also in `test.txt`. `test.txt` could include **additional** packages if `test.in` specified additional dependencies.

This is the content of `main.txt` and the **expected**  content of `test.txt`:
```
numpy==1.18.1             # via pandas
pandas==0.25.3
python-dateutil==2.8.1    # via pandas
pytz==2019.3              # via pandas
six==1.14.0               # via python-dateutil
```
#### Actual result
Instead, `test.txt` has **less packages** than `main.txt`. The actual content of `test.txt` is:
```
numpy==1.18.1             # via pandas
pandas==0.25.3
python-dateutil==2.8.1    # via pandas
pytz==2019.3              # via pandas
```
This happened with other library combinations, have not figured out the pattern yet.

As mentioned, `pip install -r test.txt` will also install `six`, and `--verbose` shows that pip-tools is considering `six` until the very end of the compilation process.


Discussion:
Hello @mgab, 

Thanks for the perfectly described issue! It's candy to my eyes ☺️ Regarding the issue, it's not a bug and this is how pip-compile currently works with constraints files. Please see this [section](https://github.com/jazzband/pip-tools#workflow-for-layered-requirements) and related comment https://github.com/jazzband/pip-tools/pull/1037#issuecomment-578470265. 
@atugushev: In this [section](https://github.com/jazzband/pip-tools#workflow-for-layered-requirements), I think it would be very reasonable for users to expect `pytz==2019.2` to be added to `dev-requirements.txt`, via this lineage: `django-debug-toolbar==2.0 -> django==2.1.12 -> pytz==2019.2`. 

This example fails if used with `--generate-hashes`:
```
# requirements.in
django<2.2
```
```
# requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes requirements.in
#
django==2.1.15 \
    --hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
    --hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c
pytz==2019.3 \
    --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
    --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
    # via django
```
```
# dev-requirements.in
-c requirements.txt
django-debug-toolbar
```
```
# dev-requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes dev-requirements.in
#
django-debug-toolbar==2.1 \
    --hash=sha256:24c157bc6c0e1648e0a6587511ecb1b007a00a354ce716950bff2de12693e7a8 \
    --hash=sha256:77cfba1d6e91b9bc3d36dc7dc74a9bb80be351948db5f880f2562a0cbf20b6c5
django==2.1.15 \
    --hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
    --hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c \
    # via django-debug-toolbar
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
    # via django-debug-toolbar
```

Failure when installing `dev-requirements.txt`:
```
$ pip install -r dev-requirements.txt
Collecting django-debug-toolbar==2.1
  Using cached django_debug_toolbar-2.1-py3-none-any.whl (198 kB)
Collecting django==2.1.15
  Using cached Django-2.1.15-py3-none-any.whl (7.3 MB)
Collecting sqlparse==0.3.0
  Using cached sqlparse-0.3.0-py2.py3-none-any.whl (39 kB)
Collecting pytz
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    pytz from https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl#sha256=1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d (from django==2.1.15->-r dev-requirements.txt (line 10))
```
@jeevb 

In this case ` pip install -r requirements.txt -r dev-requirements.txt` must be used. It would be nice to have it mentioned in README too.
@atugushev 

If a constraints file is NOT specified in `dev-requirements.in`, the resulting `dev-requirements.txt` works on its own:

```
# dev-requirements.in
django-debug-toolbar
```

Notice how the below file is different from the one generated with `-c requirements.txt`:
```
# dev-requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes dev-requirements.in
#
django-debug-toolbar==2.1 \
    --hash=sha256:24c157bc6c0e1648e0a6587511ecb1b007a00a354ce716950bff2de12693e7a8 \
    --hash=sha256:77cfba1d6e91b9bc3d36dc7dc74a9bb80be351948db5f880f2562a0cbf20b6c5
django==2.1.15 \
    --hash=sha256:48522428f4a285cf265af969f4744c5ebb027c7f41958ba48b639ace2068ffe7 \
    --hash=sha256:a794f7a2f4b7c928eecfbc4ebad03712ff27fb545abe269bf01aa8500781eb1c \
    # via django-debug-toolbar
pytz==2019.3 \
    --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
    --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
    # via django
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \
    # via django-debug-toolbar
```

This works:
```
$ pip install -r dev-requirements.txt
Collecting django-debug-toolbar==2.1
  Using cached django_debug_toolbar-2.1-py3-none-any.whl (198 kB)
Collecting django==2.1.15
  Using cached Django-2.1.15-py3-none-any.whl (7.3 MB)
Collecting pytz==2019.3
  Using cached pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting sqlparse==0.3.0
  Using cached sqlparse-0.3.0-py2.py3-none-any.whl (39 kB)
Installing collected packages: sqlparse, pytz, django, django-debug-toolbar
Successfully installed django-2.1.15 django-debug-toolbar-2.1 pytz-2019.3 sqlparse-0.3.0
```

Is it expected that adding `-c requirements.txt` to `dev-requirements.in` breaks the install of the `dev-requirements.txt` file on its own?
@jeevb 

> Is it expected that adding `-c requirements.txt` to `dev-requirements.in` breaks the install of the individual `dev-requirements.txt` file?

In this workflow, yes it is. If you use `-r requirements.txt`, then you'd be able to install `dev-requirements.txt` file on its own.
> In this case ` pip install -r requirements.txt -r dev-requirements.txt` must be used. It would be nice to have it mentioned in README too.

FTR, addressed in a tracking issue #1043.
@atugushev Thanks for the clarification!

Resolving patch:
diff --git a/piptools/resolver.py b/piptools/resolver.py
--- a/piptools/resolver.py
+++ b/piptools/resolver.py
@@ -332,6 +332,17 @@ def _iter_dependencies(self, ireq):
         Editable requirements will never be looked up, as they may have
         changed at any time.
         """
+        # Pip does not resolve dependencies of constraints. We skip handling
+        # constraints here as well to prevent the cache from being polluted.
+        # Constraints that are later determined to be dependencies will be
+        # marked as non-constraints in later rounds by
+        # `combine_install_requirements`, and will be properly resolved.
+        # See https://github.com/pypa/pip/
+        # blob/6896dfcd831330c13e076a74624d95fa55ff53f4/src/pip/_internal/
+        # legacy_resolve.py#L325
+        if ireq.constraint:
+            return
+
         if ireq.editable or is_url_requirement(ireq):
             for dependency in self.repository.get_dependencies(ireq):
                 yield dependency


Test patch:
diff --git a/tests/test_cli_compile.py b/tests/test_cli_compile.py
--- a/tests/test_cli_compile.py
+++ b/tests/test_cli_compile.py
@@ -1016,3 +1016,27 @@ def test_remove_outdated_options(runner, input_opts, output_opts):
 
     assert out.exit_code == 0, out
     assert out.stderr.strip() == input_opts
+
+
+def test_sub_dependencies_with_constraints(pip_conf, runner):
+    # Write constraints file
+    with open("constraints.txt", "w") as constraints_in:
+        constraints_in.write("small-fake-a==0.1\n")
+        constraints_in.write("small-fake-b==0.2\n")
+        constraints_in.write("small-fake-with-unpinned-deps==0.1")
+
+    with open("requirements.in", "w") as req_in:
+        req_in.write("-c constraints.txt\n")
+        req_in.write("small_fake_with_deps_and_sub_deps")  # require fake package
+
+    out = runner.invoke(cli)
+
+    assert out.exit_code == 0
+
+    req_out_lines = set(out.stderr.splitlines())
+    assert {
+        "small-fake-a==0.1         # via small-fake-with-unpinned-deps",
+        "small-fake-b==0.2         # via small-fake-with-unpinned-deps",
+        "small-fake-with-deps-and-sub-deps==0.1",
+        "small-fake-with-unpinned-deps==0.1  # via small-fake-with-deps-and-sub-deps",
+    }.issubset(req_out_lines)
diff --git a/tests/test_data/fake-index.json b/tests/test_data/fake-index.json
--- a/tests/test_data/fake-index.json
+++ b/tests/test_data/fake-index.json
@@ -1,4 +1,7 @@
 {
+  "aiohttp": {
+    "3.6.2": {"": ["yarl"]}
+  },
   "anyjson": {
     "0.3.3": {"": []}
   },
@@ -79,6 +82,9 @@
       "setuptools>=18.5"
     ]}
   },
+  "idna": {
+    "2.8": {"": []}
+  },
   "ipython": {
     "2.1.0": {
       "": ["gnureadline"],
@@ -163,5 +169,8 @@
     "0.6": {"": []},
     "0.10": {"": []},
     "0.10.4": {"": []}
+  },
+  "yarl": {
+    "1.4.2": {"": ["idna"]}
   }
 }
diff --git a/tests/test_data/minimal_wheels/small_fake_with_deps_and_sub_deps-0.1-py2.py3-none-any.whl b/tests/test_data/minimal_wheels/small_fake_with_deps_and_sub_deps-0.1-py2.py3-none-any.whl
new file mode 100644
Binary files /dev/null and b/tests/test_data/minimal_wheels/small_fake_with_deps_and_sub_deps-0.1-py2.py3-none-any.whl differ
diff --git a/tests/test_data/packages/small_fake_with_deps_and_sub_deps/setup.py b/tests/test_data/packages/small_fake_with_deps_and_sub_deps/setup.py
new file mode 100644
--- /dev/null
+++ b/tests/test_data/packages/small_fake_with_deps_and_sub_deps/setup.py
@@ -0,0 +1,7 @@
+from setuptools import setup
+
+setup(
+    name="small_fake_with_deps_and_sub_deps",
+    version=0.1,
+    install_requires=["small-fake-with-unpinned-deps"],
+)
diff --git a/tests/test_resolver.py b/tests/test_resolver.py
--- a/tests/test_resolver.py
+++ b/tests/test_resolver.py
@@ -172,6 +172,11 @@
                     "pytz==2016.4 (from celery==4.0.2)",
                 ],
             ),
+            # Check that dependencies of relevant constraints are resolved
+            (
+                ["aiohttp", ("yarl==1.4.2", True)],
+                ["aiohttp==3.6.2", "idna==2.8 (from yarl==1.4.2)", "yarl==1.4.2"],
+            ),
         ]
     ),
 )
@@ -253,6 +258,19 @@ def test_iter_dependencies(resolver, from_line):
         next(res._iter_dependencies(ireq))
 
 
+def test_iter_dependencies_results(resolver, from_line):
+    res = resolver([])
+    ireq = from_line("aiohttp==3.6.2")
+    assert next(res._iter_dependencies(ireq)).comes_from == ireq
+
+
+def test_iter_dependencies_ignores_constraints(resolver, from_line):
+    res = resolver([])
+    ireq = from_line("aiohttp==3.6.2", constraint=True)
+    with pytest.raises(StopIteration):
+        next(res._iter_dependencies(ireq))
+
+
 def test_combine_install_requirements(from_line):
     celery30 = from_line("celery>3.0", comes_from="-r requirements.in")
     celery31 = from_line("celery==3.1.1", comes_from=from_line("fake-package"))