Repository: pypa/pipenv

Issue:
pipenv install --deploy does not work when using a private pypi server
`pipenv install --deploy` refuses to run despite Pipenv.lock was created a second ago. It claims the Pipenv.lock is outdated. It may be related to the interpolation of environment variables.

<details><summary>$ python -m pipenv.help output</summary>

Pipenv version: `'11.9.0'`

Pipenv location: `'/usr/local/lib/python3.6/site-packages/pipenv'`

Python location: `'/usr/local/bin/python'`

Other Python installations in `PATH`:

  - `2.7`: `/usr/bin/python2.7`
  - `2.7`: `/usr/bin/python2.7`
  - `3.6`: `/usr/local/bin/python3.6m`
  - `3.6`: `/usr/local/bin/python3.6`
  - `3.6`: `/usr/local/bin/python3.6`

  - `3.6.4`: `/usr/local/bin/python`
  - `3.6.4`: `/usr/local/bin/python`
  - `2.7.9`: `/usr/bin/python`
  - `2.7.9`: `/usr/bin/python2`
  - `3.6.4`: `/usr/local/bin/python3`
  - `3.6.4`: `/usr/local/bin/python3`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.6.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.9.60-linuxkit-aufs',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Mon Nov 6 16:00:12 UTC 2017',
 'python_full_version': '3.6.4',
 'python_version': '3.6',
 'sys_platform': 'linux'}
```

System environment variables:

  - `PYPI_USERNAME`
  - `HOSTNAME`
  - `GPG_KEY`
  - `TERM`
  - `PYPI_PASSWORD`
  - `OLDPWD`
  - `PYTHON_VERSION`
  - `PATH`
  - `PWD`
  - `LANG`
  - `PYTHON_PIP_VERSION`
  - `SHLVL`
  - `HOME`
  - `_`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_PYTHON_PATH`
  - `PYTHONUNBUFFERED`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`
  - `LANG`: `C.UTF-8`
  - `PWD`: `/build`


---------------------------

Contents of `Pipfile` ('/build/Pipfile'):

```toml
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${PYPI_USERNAME}:${PYPI_PASSWORD}@pypi.python.org/simple"
verify_ssl = true
name = "example"

[packages]
requests = "*"

```


Contents of `Pipfile.lock` ('/build/Pipfile.lock'):

```json
{
    "_meta": {
        "hash": {
            "sha256": "bfcfe32eca28d264087028c2a4defa699fce33c27421ffa8ca18c371876ebe9f"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            },
            {
                "name": "example",
                "url": "https://${PYPI_USERNAME}:${PYPI_PASSWORD}@pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
            ],
            "version": "==2018.1.18"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "idna": {
            "hashes": [
                "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f",
                "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4"
            ],
            "version": "==2.6"
        },
        "requests": {
            "hashes": [
                "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
            ],
            "index": "pypi",
            "version": "==2.18.4"
        },
        "urllib3": {
            "hashes": [
                "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
            ],
            "version": "==1.22"
        }
    },
    "develop": {}
}

```
</details>

------------

##### Expected result

`pipenv install --deploy` should install the packages from Pipfile.lock.

##### Actual result

`pipenv install --deploy` claims that Pipfile.lock is outdated, despite it was generated one second ago.

```
# pipenv install --deploy
Your Pipfile.lock (edd7a5) is out of date. Expected: (6a2984).
Aborting deploy.
# pipenv lock
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (6ebe9f)!
# pipenv install --deploy
Your Pipfile.lock (6ebe9f) is out of date. Expected: (492c98).
Aborting deploy.
```

##### Steps to replicate

- have PYPI_USERNAME and PYPI_PASSWORD defined
- use my Pipfile
- run `pipenv lock`
- run `pipenv install --deploy`

I noticed that:

- the expected hash changes if you change the value of PYPI_USERNAME and PYPI_PASSWORD
- if you unset PYPI_USERNAME and PYPI_PASSWORD, the expected hash matches the one in Pipfile.lock, but of course you miss the credentials to install from private pypi server.



Discussion:

Resolving patch:
diff --git a/pipenv/core.py b/pipenv/core.py
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 import contextlib
-import codecs
 import logging
 import os
 import sys
@@ -1312,20 +1311,14 @@ def do_init(
         )
     # Write out the lockfile if it doesn't exist, but not if the Pipfile is being ignored
     if (project.lockfile_exists and not ignore_pipfile) and not skip_lock:
-        # Open the lockfile.
-        with codecs.open(project.lockfile_location, 'r') as f:
-            lockfile = simplejson.load(f)
-        # Update the lockfile if it is out-of-date.
-        p = pipfile.load(project.pipfile_location)
-        # Check that the hash of the Lockfile matches the lockfile's hash.
-        if not lockfile['_meta'].get('hash', {}).get('sha256') == p.hash:
-            old_hash = lockfile['_meta'].get('hash', {}).get('sha256')[-6:]
-            new_hash = p.hash[-6:]
+        old_hash = project.get_lockfile_hash()
+        new_hash = project.calculate_pipfile_hash()
+        if new_hash != old_hash:
             if deploy:
                 click.echo(
                     crayons.red(
                         'Your Pipfile.lock ({0}) is out of date. Expected: ({1}).'.format(
-                            old_hash, new_hash
+                            old_hash[-6:], new_hash[-6:]
                         )
                     )
                 )
@@ -1338,7 +1331,7 @@ def do_init(
                 click.echo(
                     crayons.red(
                         u'Pipfile.lock ({0}) out of date, updating to ({1})…'.format(
-                            old_hash, new_hash
+                            old_hash[-6:], new_hash[-6:]
                         ),
                         bold=True,
                     ),
@@ -1655,19 +1648,13 @@ def ensure_lockfile(keep_outdated=False):
         keep_outdated = project.settings.get('keep_outdated')
     # Write out the lockfile if it doesn't exist, but not if the Pipfile is being ignored
     if project.lockfile_exists:
-        # Open the lockfile.
-        with codecs.open(project.lockfile_location, 'r') as f:
-            lockfile = simplejson.load(f)
-        # Update the lockfile if it is out-of-date.
-        p = pipfile.load(project.pipfile_location)
-        # Check that the hash of the Lockfile matches the lockfile's hash.
-        if not lockfile['_meta'].get('hash', {}).get('sha256') == p.hash:
-            old_hash = lockfile['_meta'].get('hash', {}).get('sha256')[-6:]
-            new_hash = p.hash[-6:]
+        old_hash = project.get_lockfile_hash()
+        new_hash = project.calculate_pipfile_hash()
+        if new_hash != old_hash:
             click.echo(
                 crayons.red(
                     u'Pipfile.lock ({0}) out of date, updating to ({1})…'.format(
-                        old_hash, new_hash
+                        old_hash[-6:], new_hash[-6:]
                     ),
                     bold=True,
                 ),
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -1,4 +1,5 @@
 # -*- coding: utf-8 -*-
+import codecs
 import json
 import os
 import re
@@ -640,3 +641,16 @@ def add_index_to_pipfile(self, index):
 
     def recase_pipfile(self):
         self.write_toml(recase_file(self._pipfile))
+
+    def get_lockfile_hash(self):
+        if not os.path.exists(self.lockfile_location):
+            return
+        # Open the lockfile.
+        with codecs.open(self.lockfile_location, 'r') as f:
+            lockfile = json.load(f)
+        return lockfile['_meta'].get('hash', {}).get('sha256')
+
+    def calculate_pipfile_hash(self):
+        # Update the lockfile if it is out-of-date.
+        p = pipfile.load(self.pipfile_location, inject_env=False)
+        return p.hash


Test patch:
diff --git a/tests/test_pipenv.py b/tests/test_pipenv.py
--- a/tests/test_pipenv.py
+++ b/tests/test_pipenv.py
@@ -12,6 +12,7 @@
 )
 from pipenv.vendor import toml
 from pipenv.vendor import delegator
+from pipenv.patched import pipfile
 from pipenv.project import Project
 from pipenv.vendor.six import PY2
 if PY2:
@@ -1118,3 +1119,37 @@ def test_clean_on_empty_venv(self, pypi):
         with PipenvInstance(pypi=pypi) as p:
             c = p.pipenv('clean')
             assert c.return_code == 0
+
+
+    @pytest.mark.install
+    def test_environment_variable_value_does_not_change_hash(self, pypi, monkeypatch):
+        with PipenvInstance(chdir=True, pypi=pypi) as p:
+            with open(p.pipfile_path, 'w') as f:
+                f.write("""
+[[source]]
+url = 'https://${PYPI_USERNAME}:${PYPI_PASSWORD}@pypi.python.org/simple'
+verify_ssl = true
+name = 'pypi'
+
+[requires]
+python_version = '2.7'
+
+[packages]
+flask = "==0.12.2"
+""")
+            monkeypatch.setitem(os.environ, 'PYPI_USERNAME', 'whatever')
+            monkeypatch.setitem(os.environ, 'PYPI_PASSWORD', 'pass')
+            assert Project().get_lockfile_hash() is None
+            c = p.pipenv('install')
+            lock_hash = Project().get_lockfile_hash()
+            assert lock_hash is not None
+            assert lock_hash == Project().calculate_pipfile_hash()
+            # sanity check on pytest
+            assert 'PYPI_USERNAME' not in str(pipfile.load(p.pipfile_path))
+            assert c.return_code == 0
+            assert Project().get_lockfile_hash() == Project.calculate_pipfile_hash()
+            monkeypatch.setitem(os.environ, 'PYPI_PASSWORD', 'pass2')
+            assert Project().get_lockfile_hash() == Project.calculate_pipfile_hash()
+            with open(p.pipfile_path, 'a') as f:
+                f.write('requests = "==2.14.0"\n')
+            assert Project().get_lockfile_hash() != Project.calculate_pipfile_hash()