Repository: pypa/pipenv

Issue:
Implicit post release requirements are stripped which causes errors if there is no corresponding non-post-release pypi package
When I install a package with the intent to upgrade in the future (>=), if the package has a dash in its version, Pipenv strips the dash when generating the lockfile which results in failed installation.

```
~/git $ mkdir tmp
~/git $ cd tmp

 ~/g/tmp $ pipenv install --python=python3.6 "ibm-db-sa-py3>=0.3.1-1"
Creating a Pipfile for this project...
Creating a virtualenv for this project...
Using python3.6 to create virtualenv...
⠋Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/hawk/.local/share/virtualenvs/tmp-f_tdiakk/bin/python3.6
Also creating executable in /home/hawk/.local/share/virtualenvs/tmp-f_tdiakk/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/hawk/.local/share/virtualenvs/tmp-f_tdiakk
Installing ibm-db-sa-py3>=0.3.1-1...
Collecting ibm-db-sa-py3>=0.3.1-1
Collecting sqlalchemy>=0.7.3 (from ibm-db-sa-py3>=0.3.1-1)
Installing collected packages: sqlalchemy, ibm-db-sa-py3
Successfully installed ibm-db-sa-py3-0.3.1.post1 sqlalchemy-1.1.13

Adding ibm-db-sa-py3>=0.3.1-1 to Pipfile's [packages]...
```

```
 ~/g/tmp $ pipenv lock
Locking [dev-packages] dependencies...
⠹Locking [packages] dependencies...
⠦Updated Pipfile.lock!
```

```
 ~/g/tmp $ grep ibm Pipfile
ibm-db-sa-py3 = ">=0.3.1-1"
```

```
 ~/g/tmp $ grep -a1 ibm Pipfile.lock
        "ibm-db-sa-py3": {
            "version": "==0.3.1"
```

```
 ~/g/tmp $ pipenv uninstall --all
Un-installing all packages from virtualenv...
Found 2 installed package(s), purging...
Uninstalling ibm-db-sa-py3-0.3.1.post1:
  Successfully uninstalled ibm-db-sa-py3-0.3.1.post1
Uninstalling SQLAlchemy-1.1.13:
  Successfully uninstalled SQLAlchemy-1.1.13

Environment now purged and fresh!
```
```
 ~/g/tmp $ pipenv install
No package provided, installing all dependencies.
Pipfile found at /home/hawk/git/tmp/Pipfile. Considering this to be the project home.
Installing dependencies from Pipfile.lock...
An error occured while installing! 0/2 - 00:00:00
Collecting ibm-db-sa-py3==0.3.1

  Could not find a version that satisfies the requirement ibm-db-sa-py3==0.3.1 (from versions: 0.3.0, 0.3.0.post1, 0.3.1.post1)
No matching distribution found for ibm-db-sa-py3==0.3.1

[================================] 0/2 - 00:00:00
```

Lockfile does not get the correct version for functools32
pipenv fails trims the `-2` off the end of the version for functools32 resulting in the following error:

```
An error occured while installing!
  Could not find a version that satisfies the requirement functools32==3.2.3 (from -r /var/folders/hh/1yl543hs1cvdqtt2kl5z6m280000gn/T/tmpXl2UKq-requirements.txt (line 34)) (from versions: 3.2.3.post1, 3.2.3.post2)
No matching distribution found for functools32==3.2.3 (from -r /var/folders/hh/1yl543hs1cvdqtt2kl5z6m280000gn/T/tmpXl2UKq-requirements.txt (line 34))
```

when searching for the package with pip
```
$ pip search functools32
functools32 (3.2.3-2)  - Backport of the functools module from Python 3.2.3 for use on 2.7 and PyPy.
```

In the lock file
```
        "functools32": {
            "version": "==3.2.3"
        },
```


To reproduce the issue:
```
$ echo "functools32" > requirements.txt ;
$ pipenv install
```

This works fine:
```
$ echo "functools32==3.2.3-2" > requirements.txt ;
$ pipenv install
```


Discussion:
Let's take TDD here: write a failing test in a branch, then fix it. Can you do that @techalchemy?
Before we go any farther here, there is a lengthy issue about this but it seems to have been closed. This is an issue with the pypi endpoint. Some packages do post-releases and use the proper notation with a dash, others do a post release but don't work with the dash notation. We removed the dash notation because it seems the latter was more common but that's clearly not a workable solution.

We'll need some form of a retry solution for this case I think. I'll add the appropriate issue number when I've got a moment to find it.
To be clear — this IBM's fault for using such a dumb versioning scheme. 
@kennethreitz, this is actually a valid versioning scheme and there's even a pep explicitly defining its use. The problem is the implementation isn't consistent.
Grrrrr
Semver or Calver all the things
if we have good examples of each case it might not be that difficult, but might be ugly
Ok, #270 was the original issue and #314 was the patch that is causing us to hit this undesired behaviour. I paged Donald on why this was happening but never got a response. We need a way to try the full version and then strip the post release version and try again.
@nateprewitt I believe this is an issue in this specific case because there is no non-post-release version of the package 'ibm-db-sa-py3' -- note there is only 0.3.1.post1, 0.3.0.post1, and 0.3.0, but not 0.3.1

It's possible that pypi would correctly resolve 0.3.1 to 0.3.1.post1 if there were a release package available for 0.3.1.  As you mentioned this could be as simple as a fallback check for the post-release itself
Closing, as this seems to be a pypi issue. I'd ask in `#pypa` on irc. 
I think it is and it isnt, but I think we can test-resolve the package with pip-tools -- it has functionality to determine the correct version to 'pin' given a requirement.  So to solve the issue I originally posted I think it's as simple as asking pypi for the correct pin or at least checking whether a version can be resolved before pinning it in the lockfile

As a sidenote, I wonder if its really correct to drop explicit post release references in requirements files or pipfiles when they are not pinned... in my case above, we are explicitly looking for the post release or newer.
I think there's more we can do here. I'm thinking #314 was a mistake or at the very least it's incomplete. I think we can do better than we currently are so I'd like to keep this open while we look at alternatives.
Hmm, I actually can't reproduce this. What version of pipenv are you currently using @NorthIsUp? This was an issue early on in 3.0 but I was under the impression we'd since solved it.

Neither of these are causing issues for me on 5.0.0:
```
$ echo "functools32==3.2.3-2" > requirements.txt ;
$ pipenv install
```

or

```
$ pipenv install functools32==3.2.3-2
$ pipenv lock
```
specifying the version works fine, it is when there is no version specified in the requirements.txt or in the Pipfile..

```bash
$ echo "functools32" > requirements.txt ;
$ pipenv install
Requirements file found, instead of Pipfile! Converting...
Creating a virtualenv for this project...
⠋New python executable in /Users/adam/.virtualenvs/toxicity-learn-gHgF3mqz/bin/python2.7
Also creating executable in /Users/adam/.virtualenvs/toxicity-learn-gHgF3mqz/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/adam/.virtualenvs/toxicity-learn-gHgF3mqz
No package provided, installing all dependencies.
Pipfile found at /Users/adam/src/toxicity-learn/Pipfile. Considering this to be the project home.
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
⠸Locking [packages] dependencies...
⠴Updated Pipfile.lock!
Installing dependencies from Pipfile.lock...
An error occured while installing!
  Could not find a version that satisfies the requirement functools32==3.2.3 (from -r /var/folders/hh/1yl543hs1cvdqtt2kl5z6m280000gn/T/tmp1nSR_i-requirements.txt (line 1)) (from versions: 3.2.3.post1, 3.2.3.post2)
No matching distribution found for functools32==3.2.3 (from -r /var/folders/hh/1yl543hs1cvdqtt2kl5z6m280000gn/T/tmp1nSR_i-requirements.txt (line 1))

$ cat Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
functools32 = "*"
```
confirmed that it is the same issue on v5

```bash
$ pipenv --version
pipenv, version 5.0.0
$ echo "functools32" > requirements.txt ;
$ pipenv install
Requirements file found, instead of Pipfile! Converting...
No package provided, installing all dependencies.
Pipfile found at /Users/adam/src/toxicity-learn/Pipfile. Considering this to be the project home.
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
⠹Locking [packages] dependencies...
⠴Updated Pipfile.lock!
Installing dependencies from Pipfile.lock...
An error occured while installing! 0/1 - 00:00:00
Collecting functools32==3.2.3

  Could not find a version that satisfies the requirement functools32==3.2.3 (from versions: 3.2.3.post1, 3.2.3.post2)
No matching distribution found for functools32==3.2.3

[================================] 0/1 - 00:00:00
$ cat Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
functools32 = "*"
```
Sorry, my reading comprehension is apparently not up to snuff this evening. I can indeed reproduce this and this is oddly enough an actual test in our test suite to verify this *is* happening.

I was pretty on the fence about merging #314 and here it is to bite us. So it appears pip barfs on some packages when an explicit post release is provided, while others need it. I'm going to have to think about a better option for this, but it seems pretty crazy `pip` doesn't have a way to enforce something like this. @kennethreitz do you have any background here?
Try again with latest. 

Resolving patch:
diff --git a/pipenv/utils.py b/pipenv/utils.py
--- a/pipenv/utils.py
+++ b/pipenv/utils.py
@@ -26,6 +26,10 @@ def shellquote(s):
     return "'" + s.replace("'", "'\\''") + "'"
 
 
+def clean_pkg_version(version):
+    return six.u(pep440_version(str(version))).replace('==', '')
+
+
 def resolve_deps(deps, sources=None, verbose=False, hashes=False):
 
     constraints = []
@@ -53,10 +57,20 @@ def resolve_deps(deps, sources=None, verbose=False, hashes=False):
 
     r = Resolver(constraints=constraints, repository=pypi)
     results = []
-
-    for result in r.resolve():
+    # pre-resolve instead of iterating to avoid asking pypi for hashes of editable packages
+    resolved_tree = r.resolve()
+    hashes = r.resolve_hashes((req for req in resolved_tree if not req.editable))
+    # convert to a dictionary indexed by package names instead of install req objects
+    resolved_hashes = {}
+    for req, pypi_hash in hashes.items():
+        resolved_hashes[pep423_name(req.name)] = {
+            'version': clean_pkg_version(req.specifier),
+            'hashes': pypi_hash
+        }
+
+    for result in resolved_tree:
         name = pep423_name(result.name)
-        version = six.u(pep440_version(str(result.specifier))).replace('==', '')
+        version = clean_pkg_version(result.specifier)
 
         if hashes:
             try:
@@ -66,6 +80,12 @@ def resolve_deps(deps, sources=None, verbose=False, hashes=False):
                     collected_hashes.append(release['digests']['sha256'])
 
                 collected_hashes = ['sha256:' + s for s in collected_hashes]
+                # Add pypi resolved hashes
+                if name in resolved_hashes and resolved_hashes[name]['version'] == version:
+                    # Eliminate potential duplicate hashes
+                    resolved = resolved_hashes[name]['hashes']
+                    resolved |= set(collected_hashes)
+                    collected_hashes = list(resolved)
 
                 results.append({'name': name, 'version': version, 'hashes': collected_hashes})
             except ValueError:


Test patch: