Repository: pypa/pipenv

Issue:
pipenv install with private source doesn't use provided environment variables
##### Expected result

Have configured a private source which require username and password to authenticate.  (It's a privately hosted https://jfrog.com/artifactory/ pypi repo.)
```
[[source]]
url = "http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@my-private-repo"
verify_ssl = false
name = "artifactory"
```
When running `ARTIFACTORY_USERNAME=myuser ARTIFACTORY_PASSWORD=mypass pipenv install` from the same directory as the pipfile on a project, I would expect the env-variables in the private source to use the values I provided.

I've also tried to provide env-variables in other way:
- Setting env variable inline on the command (as shown in the example command above)
- Exporting env variables
- Putting env variables into a '.env' file that look like this:
```
ARTIFACTORY_USERNAME=myuser
ARTIFACTORY_PASSWORD=mypass
```
All of the above give me the same result error result (below).

 When I provide the username and password directly inline in the pipfile it works fine, but when I use environment variables instead it fails with the following error:

```
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@my-private-repo --trusted-host my-private-repo

                          ROUND 1                           
Current constraints:
  aiopubsub==1.0.0
  asynctest
  pytest==3.3.2

Finding the best candidates:
  found candidate aiopubsub==1.0.0 (constraint was ==1.0.0)
User for artifactory.firmglobal.com: 
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): artifactory.firmglobal.com
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 82, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 71, in main
    clear=do_clear,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 63, in resolve
    verbose=verbose,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 425, in resolve_deps
    pre,
  File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 336, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 116, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/piptools/repositories/pypi.py", line 101, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 722, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/patched/notpip/index.py", line 831, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

/usr/local/lib/python2.7/dist-packages/pipenv/utils.py:1230: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/pipenv-ip4PTc-requirements'>
  warnings.warn(warn_message, ResourceWarning)
```
I assume it's just a confusing error message to tell me that the authentication fails, since I get the exact same error message if I put in an invalid username or password directly in the pipfile private source description.

##### Steps to replicate

pipfile which use source that require authentication. Use env variables for credentials:
```
[[source]]
url = "http://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_PASSWORD}@private-repo"
verify_ssl = false
name = "my repo"
```
Run pipenv install where you provide credentials:
`ARTIFACTORY_USERNAME=user ARTIFACTORY_PASSWORD=pass pipenv install`

What am I doing wrong?

<details><summary>$ python -m pipenv.help output</summary>

Pipenv version: `'11.9.0'`

Pipenv location: `'/usr/local/lib/python2.7/dist-packages/pipenv'`

Python location: `'/usr/bin/python'`

Other Python installations in `PATH`:

  - `2.7`: `/usr/bin/python2.7`
  - `2.7`: `/usr/bin/python2.7`
  - `3.5`: `/usr/bin/python3.5m`
  - `3.5`: `/usr/bin/python3.5`
  - `3.6`: `/usr/bin/python3.6m`
  - `3.6`: `/usr/bin/python3.6`

  - `2.7.12`: `/usr/bin/python`
  - `2.7.12`: `/usr/bin/python2`
  - `3.5.2`: `/usr/bin/python3`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.4.0-116-generic',
 'platform_system': 'Linux',
 'platform_version': '#140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018',
 'python_full_version': '2.7.12',
 'python_version': '2.7',
 'sys_platform': 'linux2'}
```

System environment variables:

  - `LC_NUMERIC`
  - `MANDATORY_PATH`
  - `CONSCRIPT_OPTS`
  - `_LXSESSION_PID`
  - `XDG_GREETER_DATA_DIR`
  - `SPARK_HOME`
  - `PROJECT_HOME`
  - `LC_CTYPE`
  - `PYTHONDONTWRITEBYTECODE`
  - `XDG_CURRENT_DESKTOP`
  - `LC_PAPER`
  - `LOGNAME`
  - `XDG_SEAT`
  - `PATH`
  - `XDG_VTNR`
  - `QT_PLATFORM_PLUGIN`
  - `PYTHONUNBUFFERED`
  - `VIRTUALENVWRAPPER_SCRIPT`
  - `ZSH`
  - `DISPLAY`
  - `XDG_SESSION_DESKTOP`
  - `LANG`
  - `CONFIRMIT_CFG_DIR`
  - `TERM`
  - `SHELL`
  - `XDG_SESSION_PATH`
  - `XAUTHORITY`
  - `LANGUAGE`
  - `SHLVL`
  - `QT_QPA_PLATFORMTHEME`
  - `TERMINATOR_UUID`
  - `PIP_REQUIRE_VIRTUALENV`
  - `WINDOWID`
  - `_`
  - `SSH_AGENT_PID`
  - `HOME`
  - `LC_MONETARY`
  - `XDG_CONFIG_HOME`
  - `SAL_USE_VCLPLUGIN`
  - `CONSCRIPT_HOME`
  - `XDG_RUNTIME_DIR`
  - `WORKON_HOME`
  - `VIRTUALENVWRAPPER_PROJECT_FILENAME`
  - `LC_ADDRESS`
  - `SSH_AUTH_SOCK`
  - `VIRTUALENV_PYTHON`
  - `GDMSESSION`
  - `VIRTUALENVWRAPPER_WORKON_CD`
  - `CONFIRMIT_CFG_HOME`
  - `IBUS_DISABLE_SNOOPER`
  - `XDG_SEAT_PATH`
  - `PIP_PYTHON_PATH`
  - `XDG_SESSION_ID`
  - `DBUS_SESSION_BUS_ADDRESS`
  - `ORBIT_SOCKETDIR`
  - `VIRTUALENVWRAPPER_HOOK_DIR`
  - `LC_IDENTIFICATION`
  - `DESKTOP_SESSION`
  - `LSCOLORS`
  - `XDG_CONFIG_DIRS`
  - `DEFAULTS_PATH`
  - `XDG_SESSION_TYPE`
  - `OLDPWD`
  - `LS_COLORS`
  - `GDM_LANG`
  - `LC_TELEPHONE`
  - `LC_MEASUREMENT`
  - `PWD`
  - `COLORTERM`
  - `LC_NAME`
  - `XDG_MENU_PREFIX`
  - `LC_TIME`
  - `LESS`
  - `PAGER`
  - `XDG_DATA_DIRS`
  - `USER`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/home/andreas/.conscript/bin:/opt/spark-2.2.1/bin:/opt/flink/bin:/home/andreas/activator-dist-1.3.12/bin:/home/andreas/bin:/home/andreas/bin:/home/andreas/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games`
  - `SHELL`: `/usr/bin/zsh`
  - `LANG`: `en_US.UTF-8`
  - `PWD`: `/path-to-project`


---------------------------

Contents of `Pipfile` ('/path-to-project/Pipfile'):

```toml
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "http://$USERNAME:$PASSWORD@my-private-repo"
verify_ssl = false
name = "artifactory"

[dev-packages]
pytest = "==3.3.2"
asynctest = "*"
aiopubsub = "==1.0.0"

[packages]
aio-pika = "==1.8.1"
deepdiff = "==3.3.0"
pika = "*"
async-timeout = "*"
async-generator = "*"
"e1839a8" = {path = ".", editable = true}

[requires]
python_version = "3.6"

```

</details>




Discussion:
Dotenv are only loaded with `run` and `shell`, not `install` (and other commands). Yes, I know it’s weird, but that’s a design decision I don’t have the authority change ¯\\\_(ツ)\_/¯ You’ll need to load your environs in other forms.
It's fine that the .env approach doesn't work, but shouldn't the other ones work?
I am experiencing the same issue, it's downloading packages but failing to update/create Pipfile.lock
Yes if these are set outside of .env this should work just fine.

Tested this against current master and it is definitely working, so if it isn't for you try master

Hi. It's not working with latest code, at least in my setup.

Here is my Pipfile
```
[[source]]
url = "https://${PYPI_USERNAME}:${PYPI_PASSWORD}@pypi.xpertsea.com/simple/"
verify_ssl = true
name = "pypi"

[packages]
dpath = "*"
scipy = "*"

[dev-packages]
mypy = "*"
pylint = "*"

[requires]
python_version = "3.6"
```
Here is what I get using `pipenv lock` :
```
Locking [dev-packages] dependencies…
candidates = self.finder.find_all_candidates(req_name)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/home/jbg/venvtest/lib/python3.5/site-packages/pipenv-11.9.1-py3.5.egg/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line
```

Drilling down the code it appears that at line 1061 in core.py when `venv_resolve_deps` is called, it'll eventually use the Project object that has been explicitly told to not expand venvs when parsed, so the call to PyPI fails with invalid credentials and produce this weird error. It actually happens down the line in utils.py, around line 345 where the actual PyPI call is.

I'm just not sure where the best place is to actually fix this issue without causing another bug later down the row..
gotcha, so the issue occurs during locking because of `project.py`:

```python
    @property
    def _lockfile(self):
        """Pipfile.lock divided by PyPI and external dependencies."""
        pfile = pipfile.load(self.pipfile_location, inject_env=False)
        lockfile = json.loads(pfile.lock())
        for section in ('default', 'develop'):
            lock_section = lockfile.get(section, {})
            for key in list(lock_section.keys()):
                norm_key = pep423_name(key)
                lockfile[section][norm_key] = lock_section.pop(key)
        return lockfile
```
I don't know if it's psosible to address this without also patching the piptools resolver debug logging functionality to prevent it from logging the interpolated index urls to the console...
I am also affected by this problem. Are there any news?
@eolo999 not yet unfortunately. We will update here when we get this sorted

Resolving patch:
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -615,8 +615,12 @@ def write_toml(self, data, path=None):
     def pipfile_sources(self):
         if 'source' in self.parsed_pipfile:
             sources = []
-            for s in self.parsed_pipfile['source']:
-                s['url'] = os.path.expandvars(s['url'])
+            for i, s in enumerate(self.parsed_pipfile['source']):
+                for k in s.keys():
+                    if k == 'verify_ssl':
+                        continue
+                    val = os.path.expandvars(self.parsed_pipfile['source'][i][k])
+                    s[k] = val
                 sources.append(s)
             return sources
         return [DEFAULT_SOURCE]
@@ -629,10 +633,8 @@ def sources(self):
             if sources_:
                 return sources_
 
-        if 'source' in self.parsed_pipfile:
-            return self.parsed_pipfile['source']
         else:
-            return [DEFAULT_SOURCE]
+            return self.pipfile_sources
 
     def find_source(self, source):
         """given a source, find it.


Test patch:
diff --git a/tests/integration/test_project.py b/tests/integration/test_project.py
--- a/tests/integration/test_project.py
+++ b/tests/integration/test_project.py
@@ -2,6 +2,30 @@
 import pytest
 import os
 from pipenv.project import Project
+from pipenv.utils import temp_environ
+from pipenv.patched import pipfile
+
+
+@pytest.mark.project
+@pytest.mark.sources
+@pytest.mark.environ
+def test_pipfile_envvar_expansion(PipenvInstance):
+    with PipenvInstance(chdir=True) as p:
+        with temp_environ():
+            with open(p.pipfile_path, 'w') as f:
+                f.write("""
+[[source]]
+url = 'https://${TEST_HOST}/simple'
+verify_ssl = false
+name = "pypi"
+
+[packages]
+pytz = "*"
+                """.strip())
+            os.environ['TEST_HOST'] = 'localhost:5000'
+            project = Project()
+            assert project.sources[0]['url'] == 'https://localhost:5000/simple'
+            assert 'localhost:5000' not in str(pipfile.load(p.pipfile_path))
 
 
 @pytest.mark.project