Repository: pypa/pipenv

Issue:
Environment variables are stored in plain text when installing packages from a private repository
[Injecting credentials into Pipfiles via environment variables](https://docs.pipenv.org/advanced/#injecting-credentials-into-pipfiles-via-environment-variables) works as advertised when doing `pipenv install`. However, it expands and stores the environment variables in both the `Pipfile` and the `Pipfile.lock` files when installing a package via `pipenv install package-name`.

<details><summary>$ python -m pipenv.help output (partial)</summary>

Pipenv version: `'11.10.1'`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.5.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST '
                     '2018; root:xnu-4570.51.1~1/RELEASE_X86_64',
 'python_full_version': '3.5.4',
 'python_version': '3.5',
 'sys_platform': 'darwin'}
```

Contents of `Pipfile` ('/Users/ixai/tmp/pipenv-credentials/Pipfile'):

```toml
[[source]]
name = "pripypi"
verify_ssl = true
url = "https://${USERNAME}:${PASSWORD}@private.host.tld/simple"

[packages]

[dev-packages]

[requires]
python_version = "3.5"
```

</details>

##### Expected result

The environment variables shouldn't be stored in neither the `Pipfile` nor the `Pipfile.lock` files.

##### Actual result

The environment variables are stored in both the `Pipfile` and the `Pipfile.lock` files.

##### Steps to replicate

* Add a new private package through `pipenv install private-package`

<details><summary>$ pipenv install private-package -v output (redacted) </summary>

```
Installing private-package…
⠋Installing 'private-package'
⠙$ "/Users/ixai/.local/share/virtualenvs/pipenv-credentials-Xvn7BU9A/bin/pip" install   --verbose    "private-package" -i https://USERNAME:PASSWORD@private.host.tld/simple --exists-action w
Created temporary directory: /private/var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pip-ephem-wheel-cache-thklj_04
Created temporary directory: /private/var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pip-install-uesxs639
Looking in indexes: https://USERNAME:PASSWORD@private.host.tld/simple
Collecting private-package
  1 location(s) to search for versions of private-package:
  * https://USERNAME:PASSWORD@private.host.tld/simple/private-package/
  Getting page https://USERNAME:PASSWORD@private.host.tld/simple/private-package/
  Looking up "https://private.host.tld/simple/private-package/" in the cache
  Current age based on date: 429608
  Ignoring unknown cache-control directive:
  Freshness lifetime from request max-age: 600
  Starting new HTTPS connection (1): private.host.tld
  https://private.host.tld:443 "GET /simple/private-package/ HTTP/1.1" 200 317
  Ignoring unknown cache-control directive:
  Updating cache with response from "https://private.host.tld/simple/private-package/"
  Analyzing links from page https://private.host.tld/simple/private-package/
    Found link https://private.host.tld/packages/private-package-1.0.2.tar.gz#md5=261952776ead30da5b3420004f490d05 (from https://private.host.tld/simple/private-package/), version: 1.0.2
  Using version 1.0.2 (newest of versions: 1.0.2)
  Using cached wheel link: file:///Users/ixai/Library/Caches/pip/wheels/f8/13/ba/3dc0cb6fd49185d9bd60bfc67993fb38d78c04b797290f34b5/custom_types-1.0.2-cp35-none-any.whl
Requirement already satisfied: SQLAlchemy>=1.0 in /Users/ixai/.local/share/virtualenvs/pipenv-credentials-Xvn7BU9A/lib/python3.5/site-packages (from private-package) (1.2.7)
Installing collected packages: private-package

Successfully installed private-package-1.0.2
Cleaning up...

Adding private-package to Pipfile's [packages]…
Pipfile.lock (aba785) out of date, updating to (4bc74b)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (4bc74b)!
Installing dependencies from Pipfile.lock (4bc74b)…
Installing 'private-package==1.0.2 --hash=sha256:13371aeaa49534191b86bd3e98641edf3090f942faf80547949ecdd384aa9e48  '
$ "/Users/ixai/.local/share/virtualenvs/pipenv-credentials-Xvn7BU9A/bin/pip" install   --verbose  --no-deps  -r "/var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pipenv-lzb62ayi-requirements/pipenv-k1btpmay-requirement.txt" --require-hashes -i https://USERNAME:PASSWORD@private.host.tld/simple --exists-action w
Installing 'sqlalchemy==1.2.7 --hash=sha256:d6cda03b0187d6ed796ff70e87c9a7dce2c2c9650a7bc3c022cd331416853c31  '
$ "/Users/ixai/.local/share/virtualenvs/pipenv-credentials-Xvn7BU9A/bin/pip" install   --verbose  --no-deps  -r "/var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pipenv-lzb62ayi-requirements/pipenv-ldq_4mj9-requirement.txt" --require-hashes -i https://USERNAME:PASSWORD@private.host.tld/simple --exists-action w
Created temporary directory: /private/var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pip-ephem-wheel-cache-m6n5ks7b
Created temporary directory: /private/var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pip-install-wl64aesl
Looking in indexes: https://USERNAME:PASSWORD@private.host.tld/simple
Requirement already satisfied: private-package==1.0.2 in /Users/ixai/.local/share/virtualenvs/pipenv-credentials-Xvn7BU9A/lib/python3.5/site-packages (from -r /var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pipenv-lzb62ayi-requirements/pipenv-k1btpmay-requirement.txt (line 1)) (1.0.2)
  Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatable environment, install into an empty virtualenv.
Cleaning up...

Created temporary directory: /private/var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pip-ephem-wheel-cache-u8m9v8gr
Created temporary directory: /private/var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pip-install-5sg0ey7z
Looking in indexes: https://USERNAME:PASSWORD@private.host.tld/simple
Requirement already satisfied: sqlalchemy==1.2.7 in /Users/ixai/.local/share/virtualenvs/pipenv-credentials-Xvn7BU9A/lib/python3.5/site-packages (from -r /var/folders/ns/fvdlrncs7v946ch8ktv_3wyc0000gn/T/pipenv-lzb62ayi-requirements/pipenv-ldq_4mj9-requirement.txt (line 1)) (1.2.7)
  Since it is already installed, we are trusting this package without checking its hash. To ensure a completely repeatable environment, install into an empty virtualenv.
Cleaning up...

  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
```

</details>
.

* See your credentials in both the `Pipfile` and the `Pipfile.lock` files.


Discussion:
Wasn’t this already fixed in the dev release?
So to clarify— your pipfile has the normal un-interpolated variables, but when you install something that uses the index it expands them and rewrites? If so this seems pretty bad
I've replicated this - running `pipenv install package-name` replaces the environment variables in the Pipfile and Pipfile.lock with actual values.
@techalchemy yes, that's correct

Resolving patch:
diff --git a/pipenv/core.py b/pipenv/core.py
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -1452,7 +1452,7 @@ def pip_install(
         sources = [{'url': index}]
         if extra_indexes:
             if isinstance(extra_indexes, six.string_types):
-                extra_indexes = [extra_indexes,]
+                extra_indexes = [extra_indexes]
             for idx in extra_indexes:
                 try:
                     extra_src = project.find_source(idx).get('url')
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -34,6 +34,7 @@
     is_valid_url,
     normalize_drive,
     python_version,
+    safe_expandvars,
 )
 from .environments import (
     PIPENV_MAX_DEPTH,
@@ -647,17 +648,15 @@ def write_lockfile(self, content):
 
     @property
     def pipfile_sources(self):
-        if 'source' in self.parsed_pipfile:
-            sources = []
-            for i, s in enumerate(self.parsed_pipfile['source']):
-                for k in s.keys():
-                    if k == 'verify_ssl':
-                        continue
-                    val = os.path.expandvars(self.parsed_pipfile['source'][i][k])
-                    s[k] = val
-                sources.append(s)
-            return sources
-        return [DEFAULT_SOURCE]
+        if 'source' not in self.parsed_pipfile:
+            return [DEFAULT_SOURCE]
+        # We need to make copies of the source info so we don't
+        # accidentally modify the cache. See #2100 where values are
+        # written after the os.path.expandvars() call.
+        return [
+            {k: safe_expandvars(v) for k, v in source.items()}
+            for source in self.parsed_pipfile['source']
+        ]
 
     @property
     def sources(self):
diff --git a/pipenv/utils.py b/pipenv/utils.py
--- a/pipenv/utils.py
+++ b/pipenv/utils.py
@@ -1340,3 +1340,11 @@ def atomic_open_for_write(target, binary=False, newline=None, encoding=None):
         except OSError:
             pass
         os.rename(f.name, target)  # No os.replace() on Python 2.
+
+
+def safe_expandvars(value):
+    """Call os.path.expandvars if value is a string, otherwise do nothing.
+    """
+    if isinstance(value, six.string_types):
+        return os.path.expandvars(value)
+    return value


Test patch:
diff --git a/tests/integration/test_install_basic.py b/tests/integration/test_install_basic.py
--- a/tests/integration/test_install_basic.py
+++ b/tests/integration/test_install_basic.py
@@ -1,3 +1,7 @@
+import contextlib
+import os
+
+from pipenv.utils import temp_environ
 from pipenv.vendor import delegator
 
 import pytest
@@ -234,3 +238,29 @@ def test_clean_on_empty_venv(PipenvInstance, pypi):
     with PipenvInstance(pypi=pypi) as p:
         c = p.pipenv('clean')
         assert c.return_code == 0
+
+
+@pytest.mark.install
+def test_install_does_not_extrapolate_environ(PipenvInstance, pypi):
+    with temp_environ(), PipenvInstance(pypi=pypi, chdir=True) as p:
+        os.environ['PYPI_URL'] = pypi.url
+
+        with open(p.pipfile_path, 'w') as f:
+            f.write("""
+[[source]]
+url = '${PYPI_URL}/simple'
+verify_ssl = true
+name = 'mockpi'
+            """)
+
+        # Ensure simple install does not extrapolate.
+        c = p.pipenv('install')
+        assert c.return_code == 0
+        assert p.pipfile['source'][0]['url'] == '${PYPI_URL}/simple'
+        assert p.lockfile['_meta']['sources'][0]['url'] == '${PYPI_URL}/simple'
+
+        # Ensure package install does not extrapolate.
+        c = p.pipenv('install six')
+        assert c.return_code == 0
+        assert p.pipfile['source'][0]['url'] == '${PYPI_URL}/simple'
+        assert p.lockfile['_meta']['sources'][0]['url'] == '${PYPI_URL}/simple'