Repository: pypa/pipenv

Issue:
pipenv requirement resolution failure for the ipython package
I think that you are cutting off the resolution algorithm in `pipenv/patched/piptools/resolver.py` too soon.  The three observations that lead me to this belief are:

(1)  In a clean empty python27 pipenv, typing `pipenv install ipython` succeeds in installing ipython but fails to create `Pipfile.lock` with the following error message:
```
RuntimeError: No stable configuration of concrete packages could be found for the given constraints after 16 rounds of resolving.
This is likely a bug.
```

(2) In a new empty python27 pipenv, typing `pipenv install appnope backports.shutil-get-terminal-size decorator enum34 ipython-genutils pathlib2 pexpect pickleshare prompt-toolkit ptyprocess pygments scandir simplegeneric six traitlets wcwidth ipython` succeeds, in both the installation and in the creation of `Pipfile.lock`.  The magic list of packages I used here was just the packages that step (1) told me were necessary to install for `ipython`.  This suggests that the problem is with the resolution method, not the interdependencies of this particular set of packages.

(3) Indeed, rerunning experiment (1), but after manually changing `pipenv/patched/piptools/resolver.py` so as to allow it an extra 100 rounds of resolution, succeeds in both the installation of `ipython` and in the creation of `Pipfile.lock`.


Discussion:
I'm unable to reproduce this. I have no trouble resolving `ipython` dependencies within 10 rounds (it takes 4).
Could you give a `--verbose` output to see what's going on during the resolving?
I've seen some issue with the stabilisation of the resolver, so let's see this.
Unfortunately the `--verbose` flag does not seem to create any more input (on stdout or stderr) in the part where `Pipfile.lock` is being created.  Nonetheless, here is the output for your perusal: [pipenv.verbose.stdout.txt](https://github.com/kennethreitz/pipenv/files/1360814/pipenv.verbose.stdout.txt)

In case this is useful in debugging it, here is the `Pipfile.lock` that was produced in step (2), as that should some information about what weird kind of graph the code is looking at (as well as some info about my computer's setup): [Pipfile.lock.txt](https://github.com/kennethreitz/pipenv/files/1360819/Pipfile.lock.txt)

Incidentally, when I increased the resolver round limit I noticed that the resolver code has a fair number of logging statements in it.  However, I was not able to figure out how to run the code so that those logging statements are active.  If you can tell me that, I'd be happy to generate those logs and append them here.

Thank you!
My bad, I'd actually need you to run `pipenv lock --verbose` (where you have a `Pipfile` specyfing only ipython of course).
The "verbosity" is no applied on the `lock` part when running `install`, that was my mistake.
Thanks for the tip!  Indeed this seems much more useful: [pipenv.lock.verbose.stdout.txt](https://github.com/kennethreitz/pipenv/files/1360889/pipenv.lock.verbose.stdout.txt)

This is pretty similar to #792. 
There's definitely a bug here:
(I sorted the "Finding secondary dependencies" lines for comparision)

Round 15:
```
Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was ==0.1.0)
  found candidate backports.shutil-get-terminal-size==1.0.0 (constraint was <any>)
  found candidate decorator==4.1.2 (constraint was <any>)
  found candidate enum34==1.1.6 (constraint was <any>)
  found candidate ipython==5.5.0 (constraint was <any>)
  found candidate ipython-genutils==0.2.0 (constraint was <any>)
  found candidate pathlib2==2.3.0 (constraint was ==2.3.0)
  found candidate pexpect==4.2.1 (constraint was <any>)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.4,<2.0.0)
  found candidate ptyprocess==0.5.2 (constraint was >=0.5)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate scandir==1.6 (constraint was ==1.6)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate six==1.11.0 (constraint was >=1.9.0)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate wcwidth==0.1.7 (constraint was <any>)

Finding secondary dependencies:
  appnope==0.1.0            requires -
  backports.shutil-get-terminal-size==1.0.0 requires -
  decorator==4.1.2          requires -
  enum34==1.1.6             requires -
  ipython-genutils==0.2.0   requires -
  ipython==5.5.0            requires appnope; sys_platform == "darwin", backports.shutil-get-terminal-size; python_version == "2.7", decorator, pathlib2; python_version == "2.7" or python_version == "3.3", pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.4, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2
  pathlib2==2.3.0           requires scandir; python_version < "3.5", six
  pexpect==4.2.1            requires ptyprocess>=0.5
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
  prompt-toolkit==1.0.15    requires six>=1.9.0, wcwidth
  ptyprocess==0.5.2         requires -
  pygments==2.2.0           requires -
  scandir==1.6              requires -
  simplegeneric==0.8.1      requires -
  six==1.11.0               requires -
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six
  wcwidth==0.1.7            requires -
```

Round 16:
```
Finding the best candidates:
  found candidate appnope==0.1.0 (constraint was ==0.1.0)
  found candidate backports.shutil-get-terminal-size==1.0.0 (constraint was ==1.0.0)
  found candidate decorator==4.1.2 (constraint was <any>)
  found candidate enum34==1.1.6 (constraint was ==1.1.6)
  found candidate ipython==5.5.0 (constraint was <any>)
  found candidate ipython-genutils==0.2.0 (constraint was <any>)
  found candidate pathlib2==2.3.0 (constraint was ==2.3.0)
  found candidate pexpect==4.2.1 (constraint was ==4.2.1)
  found candidate pickleshare==0.7.4 (constraint was <any>)
  found candidate prompt-toolkit==1.0.15 (constraint was >=1.0.4,<2.0.0)
  found candidate ptyprocess==0.5.2 (constraint was >=0.5)
  found candidate pygments==2.2.0 (constraint was <any>)
  found candidate scandir==1.6 (constraint was <any>)
  found candidate simplegeneric==0.8.1 (constraint was >0.8)
  found candidate six==1.11.0 (constraint was >=1.9.0)
  found candidate traitlets==4.3.2 (constraint was >=4.2)
  found candidate wcwidth==0.1.7 (constraint was <any>)

Finding secondary dependencies:
  appnope==0.1.0            requires -
  backports.shutil-get-terminal-size==1.0.0 requires -
  decorator==4.1.2          requires -
  enum34==1.1.6             requires -
  ipython-genutils==0.2.0   requires -
  ipython==5.5.0            requires appnope; sys_platform == "darwin", backports.shutil-get-terminal-size; python_version == "2.7", decorator, pathlib2; python_version == "2.7" or python_version == "3.3", pexpect; sys_platform != "win32", pickleshare, prompt-toolkit<2.0.0,>=1.0.4, pygments, setuptools>=18.5, simplegeneric>0.8, traitlets>=4.2
  pathlib2==2.3.0           requires scandir; python_version < "3.5", six
  pexpect==4.2.1            requires ptyprocess>=0.5
  pickleshare==0.7.4        requires pathlib2; python_version in "2.6 2.7 3.2 3.3"
  prompt-toolkit==1.0.15    requires six>=1.9.0, wcwidth
  ptyprocess==0.5.2         requires -
  pygments==2.2.0           requires -
  scandir==1.6              requires -
  simplegeneric==0.8.1      requires -
  six==1.11.0               requires -
  traitlets==4.3.2          requires decorator, enum34; python_version == "2.7", ipython-genutils, six
  wcwidth==0.1.7            requires -
```

No candidate change, no dependencies change, yet the constraints are detected as changing. There's a bug in detecting the constraint changes.

I'm now able to reproduce for some reason. I'll try to spot where the diff is failing while it reproduces.
I found something in the patched version of `pip-tools` here.
In `Resolver. _iter_dependencies`:
```
        elif ireq.markers:
            for dependency in self.repository.get_dependencies(ireq):
                yield dependency
        elif ireq.extras:
            for dependency in self.repository.get_dependencies(ireq):
                yield dependency
            return
```
these call to `self.repository.get_dependencies(ireq)` will **sometime** return a dependency object pinned (like `enum34==1.1.6`), sometime a similar object including markers, and sometime nothing (and empty set) for the "same `ireq`" between rounds.

That will cause some "fake variation", potentially never stabilising, like above.

As I say "same `ireq`", it's not quite the same, as sometime the underlying req has a marker, and sometime not.

Commenting out this patch fixes the issue on my side.

@kennethreitz Question for you: what was the goal of this patch exactly?
The same for me. It actually bothers me quite hard!

My `Pipfile`: 

```
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[requires]
python_version = '3.6'

[dev-packages]

"django-debug-toolbar" = "*"
"pep8-naming" = "*"
"flake8-builtins" = "==0.2"
"flake8-commas" = "==0.4.3"
"flake8-quotes" = "==0.9.0"
"flake8-blind-except" = "*"
"flake8-bugbear" = "*"
"flake8-comprehensions" = "*"
"flake8-pep3101" = "*"
pytest = "==3.0.7"
"pytest-flake8" = "==0.8.1"
"pytest-isort" = "==0.1.0"
"pytest-django" = "*"
"pytest-cov" = "*"
gitlint = "==0.8.2"
"pre-commit" = "*"
xenon = "*"
radon = "*"

[packages]

django = "<1.12"
"django-split-settings" = "*"
"django-axes" = "*"
"psycopg2" = "*"
gunicorn = "*"
unipath = "*"
"python-decouple" = "*"
```

A workaround:
1. `pipenv install -d`
2. `pip install ipython`
@vphilippon making pip-tools resolve extras makers
@jc159 find out exactly how many rounds you need. we may need to up our default. 
this patch allows pip-tools to have "508 specifiers" in it's final results, which are incredibly useful for us, and our users. It's essential, in fact. It does require more rounds, however, which is why we upped the default. 

It sounds like, in this case, we may need to increase it further. 
@kennethreitz Upping the default number of round will not fix the real issue here: There are "random" changes in the constraints while they should be stable. The number of round required will depend on your "luck" here. It will at best reduce the risk of never stabilising.
Without changing a thing, clearing my cache every time, I've managed to resolve the `ipython` dependencies in 3, 4, 7 rounds, and sometime failing in over 16 rounds. That's is not ok.

I'm 100% convinced, after debugging and going line-by-line, that there's a bug here. The number of round required should be stable, not changing between attempts, which it does.

Also, even if the extras and markers are resolved here, half of them can be dropped in the `_group_constraint` section. There's no markers combination done, which means the only markers that will get through will be the one on the first ireq for the combination.
I just stumbled across this too. Python 3.6.2 / pipenv 8.2.6
@kennethreitz The number of rounds needed to resolve `ipython` is not deterministic.  When I ran it a half-dozen times, the lowest I saw was 13, the highest 145.  I do not think that upping the default number of rounds is a good fix to this problem.  The worst example I know of is the `jupyter` package, for which the resolver falls into a practically infinite loop.
I've found the source of the variation:
```
elif ireq.markers:
    for dependency in self.repository.get_dependencies(ireq):
        yield dependency
```
the yielded `dependency` is the same object instance as the `ireq` here. this means:
- `dependency.prepared == True`
- `dependency` will be the `ireq` object in the next iteration.
- On next iteration, `ireq.prepared == True`
- Calling `self.repository.get_dependencies` with a prepared `ireq` returns an empty set of dependencies, hence the change in constraints (more specifically, `reqset._prepare_file(self.finder, ireq)` returns and empty set).
- This did not occur before this patch because `yield InstallRequirement.from_line(dependency_string, constraint=ireq.constraint)` always yielded a new instance of `ireq`, unprepared.

Quickfix: Setting `dependency.prepared = False` before the yield stabilised the resolve.

Maybe there's something better to do here though, but at least I think I got what's happening figured out.
Can you prepare a PR and I'll play with it to see if it still meets our requirements?
@kennethreitz Yes, I'll get on it.
While exploring a bit, I noticed that even if I remove the patch, the markers are still there, sometime simply being different (pointing back to my comment about needing to combine markers in `_group_constraints`)

It looks like the patch in `repositories/pypi.py` and `utils.py` is sufficient to keep the markers, if the combining is done. I'll try to tackle this, that might be desirable in `pip-tools` too.

Resolving patch:
diff --git a/pipenv/patched/piptools/resolver.py b/pipenv/patched/piptools/resolver.py
--- a/pipenv/patched/piptools/resolver.py
+++ b/pipenv/patched/piptools/resolver.py
@@ -276,9 +276,12 @@ def _iter_dependencies(self, ireq):
             return
         elif ireq.markers:
             for dependency in self.repository.get_dependencies(ireq):
+                dependency.prepared = False
                 yield dependency
+            return
         elif ireq.extras:
             for dependency in self.repository.get_dependencies(ireq):
+                dependency.prepared = False
                 yield dependency
             return
         elif not is_pinned_requirement(ireq):


Test patch: