Repository: pypa/pipenv

Issue:
Lock file differs depending on Python version of Pipenv installation
Pipenv seems to create a different lock file depending on whether Pipenv was installed with `pip2` or `pip3`.

Example with pipenv installed using `pip2 install --user pipenv`:
```
$ which pipenv
/Users/martin/Library/Python/2.7/bin/pipenv

$ pipenv install apscheduler
Creating a virtualenv for this project…
⠋Using base prefix '/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/martin/.local/share/virtualenvs/foo-mzLZXOua/bin/python3.6
Also creating executable in /Users/martin/.local/share/virtualenvs/foo-mzLZXOua/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/martin/.local/share/virtualenvs/foo-mzLZXOua
Creating a Pipfile for this project…
Installing apscheduler…
Collecting apscheduler
  Using cached APScheduler-3.3.1-py2.py3-none-any.whl
Collecting tzlocal>=1.2 (from apscheduler)
Collecting pytz (from apscheduler)
  Using cached pytz-2017.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=0.7 in /Users/martin/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.6/site-packages (from apscheduler)
Collecting six>=1.4.0 (from apscheduler)
  Using cached six-1.11.0-py2.py3-none-any.whl
Installing collected packages: pytz, tzlocal, six, apscheduler
Successfully installed apscheduler-3.3.1 pytz-2017.2 six-1.11.0 tzlocal-1.4

Adding apscheduler to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (019ead)!

$ cat Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
apscheduler = "*"

$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "10b2336ecdb1ebf47f350b4b23ada696f0662969a53d4c798a51bf073c019ead"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.3",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "17.0.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64",
            "python_full_version": "3.6.3",
            "python_version": "3.6",
            "sys_platform": "darwin"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "apscheduler": {
            "hashes": [
                "sha256:bc9f96e498adb362beb5f1d3715a2570d100183add4ace5227c1a7d5dbaac900",
                "sha256:f68874dff1bdffcc6ce3adb7840c1e4d162c609a3e3f831351df30b75732767b"
            ],
            "version": "==3.3.1"
        },
        "funcsigs": {
            "hashes": [
                "sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca",
                "sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50"
            ],
            "markers": "python_version == '2.7'",
            "version": "==1.0.2"
        },
        "futures": {
            "hashes": [
                "sha256:c4884a65654a7c45435063e14ae85280eb1f111d94e542396717ba9828c4337f",
                "sha256:51ecb45f0add83c806c68e4b06106f90db260585b25ef2abfcda0bd95c0132fd"
            ],
            "markers": "python_version == '2.7'",
            "version": "==3.1.1"
        },
        "pytz": {
            "hashes": [
                "sha256:c883c2d6670042c7bc1688645cac73dd2b03193d1f7a6847b6154e96890be06d",
                "sha256:03c9962afe00e503e2d96abab4e8998a0f84d4230fa57afe1e0528473698cdd9",
                "sha256:487e7d50710661116325747a9cd1744d3323f8e49748e287bc9e659060ec6bf9",
                "sha256:43f52d4c6a0be301d53ebd867de05e2926c35728b3260157d274635a0a947f1c",
                "sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67",
                "sha256:54a935085f7bf101f86b2aff75bd9672b435f51c3339db2ff616e66845f2b8f9",
                "sha256:39504670abb5dae77f56f8eb63823937ce727d7cdd0088e6909e6dcac0f89043",
                "sha256:ddc93b6d41cfb81266a27d23a79e13805d4a5521032b512643af8729041a81b4",
                "sha256:f5c056e8f62d45ba8215e5cb8f50dfccb198b4b9fbea8500674f3443e4689589"
            ],
            "version": "==2017.2"
        },
        "six": {
            "hashes": [
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb",
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9"
            ],
            "version": "==1.11.0"
        },
        "tzlocal": {
            "hashes": [
                "sha256:05a2908f7fb1ba8843f03b2360d6ad314dbf2bce4644feb702ccd38527e13059"
            ],
            "version": "==1.4"
        }
    },
    "develop": {}
}

```

And pipenv installed with `pip3 install --user pipenv`:
```
$ which pipenv
/Users/martin/Library/Python/3.6/bin/pipenv

$ pipenv update
Updating all dependencies from Pipfile…
Found 3 installed package(s), purging…
Uninstalling APScheduler-3.3.1:
  Successfully uninstalled APScheduler-3.3.1
Uninstalling pytz-2017.2:
  Successfully uninstalled pytz-2017.2
Uninstalling tzlocal-1.4:
  Successfully uninstalled tzlocal-1.4

Environment now purged and fresh!
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (019ead)!
Installing dependencies from Pipfile.lock (019ead)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 4/4 — 00:00:01
To activate this project's virtualenv, run the following:
 $ pipenv shell
All dependencies are now up-to-date!

$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "10b2336ecdb1ebf47f350b4b23ada696f0662969a53d4c798a51bf073c019ead"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "0",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "17.0.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64",
            "python_full_version": "2.7.14",
            "python_version": "2.7",
            "sys_platform": "darwin"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "apscheduler": {
            "hashes": [
                "sha256:bc9f96e498adb362beb5f1d3715a2570d100183add4ace5227c1a7d5dbaac900",
                "sha256:f68874dff1bdffcc6ce3adb7840c1e4d162c609a3e3f831351df30b75732767b"
            ],
            "version": "==3.3.1"
        },
        "pytz": {
            "hashes": [
                "sha256:c883c2d6670042c7bc1688645cac73dd2b03193d1f7a6847b6154e96890be06d",
                "sha256:03c9962afe00e503e2d96abab4e8998a0f84d4230fa57afe1e0528473698cdd9",
                "sha256:487e7d50710661116325747a9cd1744d3323f8e49748e287bc9e659060ec6bf9",
                "sha256:43f52d4c6a0be301d53ebd867de05e2926c35728b3260157d274635a0a947f1c",
                "sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67",
                "sha256:54a935085f7bf101f86b2aff75bd9672b435f51c3339db2ff616e66845f2b8f9",
                "sha256:39504670abb5dae77f56f8eb63823937ce727d7cdd0088e6909e6dcac0f89043",
                "sha256:ddc93b6d41cfb81266a27d23a79e13805d4a5521032b512643af8729041a81b4",
                "sha256:f5c056e8f62d45ba8215e5cb8f50dfccb198b4b9fbea8500674f3443e4689589"
            ],
            "version": "==2017.2"
        },
        "six": {
            "hashes": [
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb",
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9"
            ],
            "version": "==1.11.0"
        },
        "tzlocal": {
            "hashes": [
                "sha256:05a2908f7fb1ba8843f03b2360d6ad314dbf2bce4644feb702ccd38527e13059"
            ],
            "version": "==1.4"
        }
    },
    "develop": {}
}
```

Using `pipenv --two update` in the last example gives the same result.

This effectively means that it's not possible to use a Python 3 Pipenv to generate a `Pipfile.lock` or  `requirements.txt` that is usable in a Python 2 environment.


Discussion:
Are you using the latest version?
Yes:

```
$ pipenv --version
pipenv, version 8.2.6
```
This may be related to #848. The Python 3 installation of pipenv seems to ignore the environment markers in the `Pipfile` created with the Python 2 installation.
Even if `pipenv` was to respect environment markers, the `Pipfile.lock` would not be guaranteed to be unchanged between Python 2 and Python 3 (nor across different OS).

To list and resolve the dependencies, we fetch the wheels of the packages. These wheels can be different, *and provide different dependencies*, depending on whether we are on Python 2 or Python 3 (ex: `foo-1.0.0-py2-none-any.whl` vs `foo-1.0.0-py3-none-any.whl`). This check and selection is done by `pip`.

To make/compile a "universal" `Pipfile.lock` (or `requirements.txt`), we'd need to respect environment markers, and patch `pip` to iterate through every wheel/sdist available for a given package version and compute the full set of dependencies for each (as each might yield different dependencies), tagging each of those with the appropriate environment markers. This is not a one-line-fix 😄.

We've hit the same issue with `pip-tools`, which `pipenv` uses for dependency resolving (with a few patches).
Thanks, I understand the perfect solution might be complicated.

What surprised me was that `pipenv lock --two` does not do what I expect either (create a Python 2-compatible lock file).

Maybe `lock` could use the Python 2 interpreter to generate the lock file when `--two` is given? Or do you think that is just as complicated to solve?
From the Readme:
`--three / --two  Use Python 3/2 when creating virtualenv.`
I would assume these options only apply when first creating the virtualenv, which would likely only be done once, but that goes beyond my knowledge of `pipenv` and what's the intended use for these options. You'll need the input of someone else here.
Closing issue!  I think the docs make it clear what `--two` and `--three` do.  
The issue is not resolved though. The documentation should at the very least state that to create Python 2 compatible `Pipfile.lock` or `requirements.txt` requires a Python 2 installation of Pipenv.
@mpolden I think that's pretty self explanatory.  I'm open to feedback from some other maintainers though!  @vphilippon @nateprewitt @techalchemy 
Saw this the other day but wasn’t sure what to say... how could something you installed using python 3 and therefore runs in python 3 then run itself in the python 2 interpreter? I mean, there are conceivably ways but it’s a horrible practice and it doesn’t work very well... why would you want this when you can do a python2 install of pipenv?
> why would you want this when you can do a python2 install of pipenv?

In my case I think didn't think much about it. I use a Python 3 environment in my dev environment, but deploy to a Python 2 environment.

There's also the options `--two` and `--three` that states:
> Use Python 3/2 when creating virtualenv

which I assumed would provide the needed compatibility.

Additionally, the docs seems to nudge the user towards Python 3. From https://docs.pipenv.org/install.html#make-sure-you-ve-got-python-pip:

> This guide is written for Python 3, however, these instructions should work fine on Python 2.7—if you are still using it, for some reason.

And finally, deterministic builds seems to be a key selling point in the features list. From https://docs.pipenv.org/#pipenv-features

> Enables truly deterministic builds, while easily specifying only what you want.
> Generates and checks file hashes for locked dependencies.

It was not obvious to me that the results of dependency resolution would depend on the Python version used by pipenv itself (and not the virtualenv) , and I couldn't find any mention of this behavior in the documentation.
> It was not obvious to me that the results of dependency resolution would depend on the Python version used by pipenv itself (and not the virtualenv) , and I couldn't find any mention of this behavior in the documentation.

Well, my thought on this is that if `pipenv` is going to offer to create/manage virtualenvs that can be either Python 2 or Python 3, then the dependency resolution and locking for that environment should respect the Python version of that environment, of course.
If the dependency resolution is based on the root version of python used to install `pipenv` itself, I'd call it broken, and users will get into trouble for sure.
"You can create Python 2 or Python 3 venv, but the locking will be valid if your venv Python version matches the root python version used to install `pipenv` to begin with." Oh hell no 😨 .

I actually thought this was dealt with somehow with a `pip-tools` and `pip` patch done by Kenneth. I haven't looked deeply into that though yet. If it's effectively not dealt with, this really needs to be addressed (and it sounds like I'll have to find some free time to actually go and look into that 😄 ).
Yeah, there's something weird going on here. Which version of Python you install pipenv on should not be affecting the lock file. A venv established with --two vs --three will have differing results and the os can affect the final output, but the system Python version is supposed to be irrelevant.
So if I'm understanding correctly:  If Python3 is the system python and you create a virtualenv using python2 from a pipenv that's installed with py3, the lock file will be different than if pipenv was installed with a system python of 2.7?  If so what are the changes between the lock file?
@erinxocon You're correct. The difference would be that the lockfile would contain Python 3 dependencies instead of Python 2 dependencies.

In short: the dependencies would be computed based on the python environment that `pipenv` is in *itself* (likely, the root env outside of any venv), instead of the environment for *which* its supposed to comput the dependencies.

But, it might be a bit more complex than that. Python version specific dependencies can be determined from various means. It can be with environment markers, or by the py2 and py3 wheels listing different dependencies, or some specific code in the setup.py when dealing with sdists. At this point, I cannot tell if one or all of the above are impacted by the "root python version".

The whole "get stuff for the right environnement" is dealt by `pip`. It all boils down to what `pip` thinks the environment is for each of those variant.
@vphilippon I know he was working on this around the time of this commit: 7d2927f3 -- might be worth looking around that time, i can dig into this in a little bit
@vphilippon @erinxocon here is the list of commits I can find with regard to locking / patching pip / python version hacks / dependency resolution: 

- 9c12ba52
- 234931bf
- 70103836
- e4cba868
- ae4591b2
- 4c000e42
- d69d116c

Not sure what to make of all of this but the answer to this issue lies somewhere in all of these commits, I would guess.
Similar discussion in https://github.com/jazzband/pip-tools/issues/563 (especially [this](https://github.com/jazzband/pip-tools/issues/563#issuecomment-330927748), [this](https://github.com/jazzband/pip-tools/issues/563#issuecomment-331533568) and [this](https://github.com/jazzband/pip-tools/issues/563#issuecomment-333887440) comment).

If it's not possible, or feasible, to support "universal" lock files (which is just ok), it should be at least clearly documented, so user doesn't get false expectations. You simply get the lock file for the python version / OS platform you compiled on (until future improvements may or may be introduced)?

(But I assume this is offtopic, as the original issue was about using virtualenv python environment and not pipenv installation environment to calculate the lock file.)
So I've been thinking about this @vphilippon @techalchemy and @nateprewitt.  Is this related to #1014  with the wrong python version being used to create a lock file?
@erinxocon yes, for sure.  It's the same issue.  I know @vphilippon is looking at this some when he can, I've spent a few spare minutes on this when I have them but it really requires a deep dive and a lot of focused effort to figure out where the last bit of trickery needs to happen.
Small update: This is in my mind (days and night 😄 ), and I hope to be able to dive into that, but this won't be a mere 10 minutes task. I have to find the time to dive into this, so sorry for the delay.
@vphilippon Good to hear you're working on it. Thanks!
Just leaving out here that this affects me as well, but the other way around. Pipenv was installed with Python 2, and I'm developing an application in Python 3. The lock file sets a dependency of Python 2 (`python-openid`) in lieu of the dependency of Python 3 (`python3-openid`).

I made my way here from there: python-social-auth/social-core#142

Edit: My workaround was to re-install pipenv with `pip3 install pipenv --user`
I believe this should now be fixed in master!  Can someone test and let me know?  Will be releasing this fix in the next version.

Resolving patch:
diff --git a/pipenv/utils.py b/pipenv/utils.py
--- a/pipenv/utils.py
+++ b/pipenv/utils.py
@@ -1,4 +1,5 @@
 # -*- coding: utf-8 -*-
+from collections import namedtuple
 import os
 import hashlib
 import tempfile
@@ -393,6 +394,9 @@ def clean_pkg_version(version):
 
 class HackedPythonVersion(object):
     """A Beautiful hack, which allows us to tell pip which version of Python we're using."""
+
+    PatchedSysVersion = namedtuple('PatchedSysVersion', ['major', 'minor', 'micro'])
+
     def __init__(self, python_version, python_path):
         self.python_version = python_version
         self.python_path = python_path
@@ -400,10 +404,13 @@ def __init__(self, python_version, python_path):
     def __enter__(self):
         os.environ['PIP_PYTHON_VERSION'] = str(self.python_version)
         os.environ['PIP_PYTHON_PATH'] = str(self.python_path)
+        self.backup_version_info = sys.version_info
+        sys.version_info = self.PatchedSysVersion(*map(int, self.python_version.split('.')))
 
     def __exit__(self, *args):
         # Restore original Python version information.
         del os.environ['PIP_PYTHON_VERSION']
+        sys.version_info = self.backup_version_info
 
 
 def prepare_pip_source_args(sources, pip_args=None):
@@ -522,17 +529,18 @@ def resolve_deps(deps, which, which_pip, project, sources=None, verbose=False, p
             resolved_tree = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
         except RuntimeError:
             # Don't exit here, like usual.
-            pass
+            resolved_tree = None
 
     # Second (last-resort) attempt:
-    with HackedPythonVersion(python_version=''.join([str(s) for s in sys.version_info[:3]]), python_path=backup_python_path):
+    if resolved_tree is None:
+        with HackedPythonVersion(python_version='.'.join([str(s) for s in sys.version_info[:3]]), python_path=backup_python_path):
 
-        try:
-            # Attempt to resolve again, with different Python version information,
-            # particularly for particularly particular packages.
-            resolved_tree = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
-        except RuntimeError:
-            sys.exit(1)
+            try:
+                # Attempt to resolve again, with different Python version information,
+                # particularly for particularly particular packages.
+                resolved_tree = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
+            except RuntimeError:
+                sys.exit(1)
 
 
 


Test patch: