Repository: jazzband/pip-tools

Issue:
pip-compile may add unwanted requirements from future versions of package dependencies
pip-compile may add unwanted requirements from future versions of package dependencies.
E.g. when collecting third-level dependences (`celery`->`amqp`->`vine`), when second-level
dependency changes (`ampq` 2.0.2 -> 1.4.9), 3rd-level requirement (`vine`) _might_ remain in resulting output. In fact second compilation gives expected results (don't know why).
##### Steps to replicate

Tried on fresh virtulaenv, OS X with pip==8.1.1 and pip-tools==1.6.5

```
$ echo "celery\nlibrabbitmq" > requirements.in
$ pip-compile -r -v
```
##### Expected result

No vine package in requirements.txt

```
amqp==1.4.9               # via kombu, librabbitmq
anyjson==0.3.3            # via kombu
billiard==3.3.0.23        # via celery
celery==3.1.23
kombu==3.0.35             # via celery
librabbitmq==1.6.1
pytz==2016.4              # via celery
```
##### Actual result (unwanted vine requirement is present)

```
amqp==1.4.9               # via kombu, librabbitmq
anyjson==0.3.3            # via kombu
billiard==3.3.0.23        # via celery
celery==3.1.23
kombu==3.0.35             # via celery
librabbitmq==1.6.1
pytz==2016.4              # via celery
vine==1.1.0
```

Full log (first run, actual result): https://gist.github.com/coagulant/db486bd554bd34d2074d9e2df8e25105
Full log (second run, expected result): https://gist.github.com/coagulant/e66e9ad23e2572757ca710cd0865b0d0



Discussion:
Great catch, and great report, too! This behaviour can definitely be explained by the current implementation. After a few rounds of dependency resolution, some package version (in this case `ampq` 2.x) may indeed include some extra packages (in this case `vine`). Then after some more rounds, it may turn out `ampq` 2.x cannot satisfy the dependencies, but 1.x can, so it "drops down". This action should then also remove `vine` from the dependencies again of course, but this is not checked in the current implementation.

The proper way to implement this is probably to trace all dependencies back to all the instances that referred to it. So that, as soon as we remove `ampq==2.0.2` from the graph, the `vine` package becomes unreachable and gets dropped, too.

The explanation for why it works the second time is that it's not run with the `--upgrade` option, which would re-hit PyPI to get the latest versions. If you omit an explicit `--upgrade`, the normal behaviour is to only add/remove lines but not touch versions (unless we have to). The way it achieves this is by pretending that the existing `requirements.txt` is the "first PyPI" server it checks. By that time, it concludes that there's only `ampq==1.4.9`, and as such the `vine` dependency is unused and thus removed. Two ways to replicate the initial behaviour would be to either: (1) edit the `requirements.txt` manually to point `ampq` to 2.0.2 and then re-run `pip-compile`. This will bump the version down again, but you'll note the `vine` dep is not removed; or (2) call `pip-compile --upgrade`.

So yes, this is a bug, and I welcome a PR for it for anyone that wants to give it a try.


Resolving patch:
diff --git a/piptools/resolver.py b/piptools/resolver.py
--- a/piptools/resolver.py
+++ b/piptools/resolver.py
@@ -181,22 +181,29 @@ def _resolve_one_round(self):
         # Find the new set of secondary dependencies
         log.debug('')
         log.debug('Finding secondary dependencies:')
-        theirs = set(dep
+        # Grouping constraints to make clean diff between rounds
+        theirs = set(self._group_constraints(dep
                      for best_match in best_matches
-                     for dep in self._iter_dependencies(best_match))
+                     for dep in self._iter_dependencies(best_match)))
 
         # NOTE: We need to compare RequirementSummary objects, since
         # InstallRequirement does not define equality
         diff = {RequirementSummary(t.req) for t in theirs} - {RequirementSummary(t.req) for t in self.their_constraints}
-        has_changed = len(diff) > 0
+        removed = ({RequirementSummary(t.req) for t in self.their_constraints} -
+                   {RequirementSummary(t.req) for t in theirs})
+
+        has_changed = len(diff) > 0 or len(removed) > 0
         if has_changed:
             log.debug('')
             log.debug('New dependencies found in this round:')
             for new_dependency in sorted(diff, key=lambda req: key_from_req(req.req)):
                 log.debug('  adding {}'.format(new_dependency))
+            log.debug('Removed dependencies in this round:')
+            for removed_dependency in sorted(removed, key=lambda req: key_from_req(req.req)):
+                log.debug('  removing {}'.format(removed_dependency))
 
         # Store the last round's results in the their_constraints
-        self.their_constraints |= theirs
+        self.their_constraints = theirs
         return has_changed, best_matches
 
     def get_best_match(self, ireq):


Test patch:
diff --git a/tests/fixtures/fake-index.json b/tests/fixtures/fake-index.json
--- a/tests/fixtures/fake-index.json
+++ b/tests/fixtures/fake-index.json
@@ -1,8 +1,25 @@
 {
+  "anyjson": {
+    "0.3.3": {"": []}
+  },
+  "amqp": {
+    "1.4.9": {"": []},
+    "2.0.2": {"": ["vine>=1.1.1"]}
+  },
   "arrow": {
     "0.5.0": {"": ["python-dateutil"]},
     "0.5.4": {"": ["python-dateutil"]}
   },
+  "billiard": {
+    "3.3.0.23": {"": []}
+  },
+  "celery": {
+    "3.1.23": {"": [
+      "kombu>=3.0.34",
+      "pytz>dev",
+      "billiard>=3.3.0.23"
+    ]}
+  },
   "click": {
     "3.3": {"": []},
     "4.0": {"": []}
@@ -55,6 +72,15 @@
       "markupsafe"
     ]}
   },
+  "kombu": {
+    "3.0.35": {"": [
+      "anyjson>=0.3.3",
+      "amqp>=1.4.9,<2.0"
+    ]}
+  },
+  "librabbitmq": {
+    "1.6.1": {"": ["amqp>=1.4.6"]}
+  },
   "markupsafe": {
     "0.23": {"": []}
   },
@@ -68,6 +94,9 @@
   "pyzmq": {
     "2.1.12": {"": []}
   },
+  "pytz": {
+    "2016.4": {"": []}
+  },
   "six": {
     "1.6.1": {"": []},
     "1.9.0": {"": []}
@@ -83,6 +112,9 @@
   "tornado": {
     "3.2.2": {"": []}
   },
+  "vine": {
+    "1.1.1": {"": []}
+  },
   "werkzeug": {
     "0.6": {"": []},
     "0.10": {"": []},
diff --git a/tests/test_resolver.py b/tests/test_resolver.py
--- a/tests/test_resolver.py
+++ b/tests/test_resolver.py
@@ -66,6 +66,19 @@
              'pygments==1.5',
              'sphinx==0.3']
          ),
+
+        # We must remove child dependencies from result if parent is removed (e.g. vine from amqp>=2.0)
+        # See: GH-370
+        (['celery', 'librabbitmq'],
+         [
+             'amqp==1.4.9',
+             'anyjson==0.3.3',
+             'billiard==3.3.0.23',
+             'celery==3.1.23',
+             'kombu==3.0.35',
+             'librabbitmq==1.6.1',
+             'pytz==2016.4']
+         ),
     ])
 )
 def test_resolver(resolver, from_line, input, expected, prereleases):