Repository: pypa/pipenv

Issue:
Pipenv doesn't resolve --system python correctly.
##### Describe you environment

1. OS Type `Docker Alpine Python` -> https://github.com/docker-library/python/blob/1d59eb2dd813c64891bf554a8ea01754aba25816/3.6/alpine3.6/Dockerfile
1. Python version: `Python 3.6.3`
1. Pipenv version: `pipenv, version 8.3.1`

##### Expected result

If I set PIP_PYTHON_PATH it's ignored and overrided by pipenv (my question is why, what's reasoning behind that?) because https://github.com/kennethreitz/pipenv/blob/master/pipenv/utils.py#L367 It's overriding this ENV variable when I use flag `--system`. I've catched that issue because when building my docker image it was failing on `pipenv install --system`

Why I know this? Because there is other function that works incorrectly.
```(Pdb) python_path
'/bin/python'
(Pdb) which('python')
'/bin/python'
(Pdb) print(sys.executable)
'/usr/local/bin/python'

/ # /bin/python
sh: /bin/python: not found
```

I think issue is somewhere here: https://github.com/kennethreitz/pipenv/blob/1bf6f5101a3bf1dca78be814e742411cc5bdb2dd/pipenv/cli.py#L1440
```
            p = os.sep.join([location] + ['bin/{0}'.format(command)])
```
this should return `/usr/local/bin/python` but for some strange reason it's returning `/bin/python`

##### Actual result

`pipenv install --system` should install not failed that /bin/python is not found.

##### Steps to replicate

`pipenv install --system` on that docker image will fail


Discussion:
@galuszkak this is interesting. So `sys.executable` is set according to pdb, but we aren’t passing the fact that you used `—system` when we actually call `which()`, so it’s trying to use the project virtualenv location which presumably doesn’t exist. This is the relevant invocation which should be easy enough to fix when I get into the office: https://github.com/kennethreitz/pipenv/blob/master/pipenv/utils.py#L406

As for why we are doing all of this, it’s mainly so that we can trick the pip-fools package resolver to resolve python packages for versions of python that are different from the active interpreter 
Using this Dockerifle, which is based on the one commited to this repo I cannot reproduce this issue, not saying it doesn't exist, looks like something is up, buy this dockerfile seems to work:

```Dockerfile
FROM python:alpine3.6

# -- Install Pipenv:
RUN set -ex && pip install pipenv --upgrade

# -- Install Application into container:
RUN set -ex && mkdir /app

WORKDIR /app

# -- Adding Pipfiles
ONBUILD COPY Pipfile Pipfile
ONBUILD COPY Pipfile.lock Pipfile.lock

# -- Install dependencies:
ONBUILD RUN set -ex && pipenv install --deploy --system
```
@erinxocon I didn't use --deploy flag + You aren't using $PIP_PYTHON_PATH which is original issue here which is overriden by pipenv.
@galuszkak Apologies; I understand that, I guess I am unsure why this needs to be set in this case.  We should definitely address this bug and fix it.  I'm just wondering if there is more going on with docker alpine which required the PIP_PYTHON_PATH being set.  
@erinxocon nothing to apologise :), thanks for looking to my issue 👍 . The fix that I supply in #1005 is working for me right now.

But I tried your image and it's working perfectly fine with same set of things. Trying to naildown what is causing my problem in first place, because sam Pipfile and Pipfile.lock and same .env file sounds work on Your Dockerfile, and I don't see a difference because I source same image...
Got it @erinxocon ! Here is how can You reproduce that:
```
FROM python:3.6-alpine3.6

RUN set -ex && pip install pipenv --upgrade
RUN set -ex && mkdir /app
ENV PIP_PYTHON_PATH=/usr/local/bin/python
COPY Pipfile /app/Pipfile

WORKDIR /app
```
Pipfile:
```
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
flask = "*"
"flask-restful" = "*"

[dev-packages]
pytest = "*"
"pytest-flask" = "*"

[requires]
python_version = "3.6"
```

```
$ docker build . -t 'pipenv_issue:latest' --no-cache
$ docker run -it pipenv_issue:latest sh
/app # pipenv install --system
CRITICAL:pip.utils:Error [Errno 2] No such file or directory: '/bin/python': '/bin/python' while executing command python setup.py egg_info
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1855, in install
    do_init(dev=dev, allow_global=system, ignore_pipfile=ignore_pipfile, system=system, skip_lock=skip_lock, verbose=verbose, concurrent=concurrent, deploy=deploy, pre=pre)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1320, in do_init
    do_lock(system=system, pre=pre)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1050, in do_lock
    pre=pre
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 459, in resolve_deps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 200, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 296, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 153, in get_dependencies
    result = reqset._prepare_file(self.finder, ireq)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/req/req_set.py", line 639, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/req/req_set.py", line 134, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/req/req_install.py", line 438, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/utils/__init__.py", line 667, in call_subprocess
    cwd=cwd, env=env)
  File "/usr/local/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/bin/python': '/bin/python'
```

Seems to be a problem with using ONBUILD RUN vs just RUN.  Not being a docker expert I'll have to figure out what the difference is between them.
@erinxocon I don't think that's the problem. In example above I show You problem inside running container not on any step building image.
@erinxocon the problem is with setting the variable in the first place (which doesn't actually need to be set but that's beside the point) -- by setting `PIP_PYTHON_PATH` it breaks resolver functionality when you need to pass `--system`:

```python
def which(command, location=None, allow_global=False):
    if location is None:
        location = project.virtualenv_location

    if not allow_global:
        if os.name == 'nt':
            p = find_windows_executable(os.path.join(location, 'Scripts'), command)
        else:
            p = os.sep.join([location] + ['bin/{0}'.format(command)])
    else:
        if command == 'python':
            p = sys.executable

return p
```

Since @galuszkak is using `--system` but `allow_global` doesn't get passed through by the copy of `which` that is passed to the resolver, but there is also no `Project.virtualenv_location` due to the `--system` flag, we wind up having simultaneously:

1. `location` is `None`
2. `Project.virtualenv_location` is `None` (`location` is still `None`)
3. _not_ `allow_global`
4. `p = os.sep.join([None] + ['bin/{0}'.format(python)])`

That's how we wind up with `/bin/python` as a command attempt.
@galuszkak Apologies I didn't see the -it sh flag passed in your example.  

@techalchemy makes sense. Do you have a patch in the works, or should I take that on?
@erinxocon @galuszkak already wrote a patch (#1005) which @nateprewitt had expressed some concerns about but I'm not sure where that leaves it
@erinxocon again, nothing to apologies and really thanks for looking into my issue.

Can we agree/disagree on #1005 as a fix? I will try move discussion further on that PR.
This issue hit me again today, but on OS X in totally different conditions. Stack trace below.
```
➜  saudi git:(dev) ✗ pipenv install -r requirements.txt
Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
Creating a virtualenv for this project…
⠋Traceback (most recent call last):
  File "/usr/local/bin/pew", line 7, in <module>
    from pew.pew import pew
  File "/usr/local/lib/python3.6/site-packages/pew/__init__.py", line 11, in <module>
    from . import pew
  File "/usr/local/lib/python3.6/site-packages/pew/pew.py", line 42, in <module>
    from pew._utils import (check_call, invoke, expandpath, own, env_bin_dir,
  File "/usr/local/lib/python3.6/site-packages/pew/_utils.py", line 22, in <module>
    encoding = locale.getlocale()[1] or 'ascii'
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/locale.py", line 581, in getlocale
    return _parse_localename(localename)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/locale.py", line 490, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: UTF-8

Virtualenv location:
Requirements.txt found, instead of Pipfile! Converting…
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did.
We recommend updating your Pipfile to specify the "*" version, instead.
Requirements file provided! Importing into Pipfile…
Creating a virtualenv for this project…
⠋Traceback (most recent call last):
  File "/usr/local/bin/pew", line 7, in <module>
    from pew.pew import pew
  File "/usr/local/lib/python3.6/site-packages/pew/__init__.py", line 11, in <module>
    from . import pew
  File "/usr/local/lib/python3.6/site-packages/pew/pew.py", line 42, in <module>
    from pew._utils import (check_call, invoke, expandpath, own, env_bin_dir,
  File "/usr/local/lib/python3.6/site-packages/pew/_utils.py", line 22, in <module>
    encoding = locale.getlocale()[1] or 'ascii'
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/locale.py", line 581, in getlocale
    return _parse_localename(localename)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/locale.py", line 490, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: UTF-8

Virtualenv location:
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
CRITICAL:pip.utils:Error [Errno 2] No such file or directory: '/bin/python': '/bin/python' while executing command python setup.py egg_info
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1866, in install
    do_init(dev=dev, allow_global=system, ignore_pipfile=ignore_pipfile, system=system, skip_lock=skip_lock, verbose=verbose, concurrent=concurrent, deploy=deploy, pre=pre)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1330, in do_init
    do_lock(system=system, pre=pre)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 1121, in do_lock
    pre=pre
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 461, in resolve_deps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 200, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 274, in _iter_dependencies
    for dependency in self.repository.get_dependencies(ireq):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 153, in get_dependencies
    result = reqset._prepare_file(self.finder, ireq)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/req/req_set.py", line 523, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/req/req_set.py", line 134, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/req/req_install.py", line 438, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/pip/utils/__init__.py", line 667, in call_subprocess
    cwd=cwd, env=env)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/bin/python': '/bin/python'
```
My shell was ZSH. After setting LC_ALL and LANG in .zshrc everything works fine. But I'm not sure why LANG/LC_ALL should have any impact on choosing python interpreter.
@galuszkak it is specifically an issue on Mac because of locale issues, I'll try to remember tomorrow to track down an early issue in this project where Kenneth discussed the problem with the default shell config but basically if your locale isn't set right python dies horribly

As for the PR I think we needed nate to sign off, let me ping him over in the PR
Pass `system` to the function, where `which` is being passed, and pass it to the passed `which`. 

Resolving patch:
diff --git a/pipenv/cli.py b/pipenv/cli.py
--- a/pipenv/cli.py
+++ b/pipenv/cli.py
@@ -1038,7 +1038,8 @@ def do_lock(verbose=False, system=False, clear=False, pre=False):
         which=which,
         which_pip=which_pip,
         project=project,
-        pre=pre
+        pre=pre,
+        allow_global=system
     )
 
     # Add develop dependencies to lockfile.
@@ -1099,7 +1100,8 @@ def do_lock(verbose=False, system=False, clear=False, pre=False):
         which=which,
         which_pip=which_pip,
         project=project,
-        pre=pre
+        pre=pre,
+        allow_global=system
     )
 
     # Add default dependencies to lockfile.
@@ -2046,7 +2048,6 @@ def uninstall(
 @click.option('--clear', is_flag=True, default=False, help="Clear the dependency cache.")
 @click.option('--pre', is_flag=True, default=False, help=u"Allow pre–releases.")
 def lock(three=None, python=False, verbose=False, requirements=False, dev=False, clear=False, pre=False):
-
     # Ensure that virtualenv is available.
     ensure_project(three=three, python=python)
 
diff --git a/pipenv/utils.py b/pipenv/utils.py
--- a/pipenv/utils.py
+++ b/pipenv/utils.py
@@ -526,7 +526,7 @@ class PipCommand(pip.basecommand.Command):
     return resolved_tree, resolver
 
 
-def resolve_deps(deps, which, which_pip, project, sources=None, verbose=False, python=False, clear=False, pre=False):
+def resolve_deps(deps, which, which_pip, project, sources=None, verbose=False, python=False, clear=False, pre=False, allow_global=False):
     """Given a list of dependencies, return a resolved list of dependencies,
     using pip-tools -- and their hashes, using the warehouse API / pip.
     """
@@ -534,7 +534,7 @@ def resolve_deps(deps, which, which_pip, project, sources=None, verbose=False, p
     index_lookup = {}
     markers_lookup = {}
 
-    python_path = which('python')
+    python_path = which('python', allow_global=allow_global)
     backup_python_path = shellquote(sys.executable)
 
     results = []


Test patch:
diff --git a/tests/test_pipenv.py b/tests/test_pipenv.py
--- a/tests/test_pipenv.py
+++ b/tests/test_pipenv.py
@@ -894,6 +894,34 @@ def test_complex_deps_lock_and_install_properly(self):
             assert c.return_code == 0
 
     @pytest.mark.lock
+    @pytest.mark.install
+    @pytest.mark.system
+    @pytest.mark.skipif(os.name != 'posix', reason="Windows doesn't have a root")
+    def test_resolve_system_python_no_virtualenv(self):
+        """Ensure we don't error out when we are in a folder off of / and doing an install using --system,
+        which used to cause the resolver and PIP_PYTHON_PATH to point at /bin/python
+        
+        Sample dockerfile:
+        FROM python:3.6-alpine3.6
+
+        RUN set -ex && pip install pipenv --upgrade
+        RUN set -ex && mkdir /app
+        COPY Pipfile /app/Pipfile
+
+        WORKDIR /app
+        """
+        with temp_environ():
+            os.environ['PIPENV_IGNORE_VIRTUALENVS'] = '1'
+            os.environ['PIPENV_USE_SYSTEM'] = '1'
+            with PipenvInstance(chdir=True) as p:
+                os.chdir('/tmp')
+                c = p.pipenv('install --system xlrd')
+                assert c.return_code == 0
+                for fn in ['Pipfile', 'Pipfile.lock']:
+                    path = os.path.join('/tmp', fn)
+                    if os.path.exists(path):
+                        os.unlink(path)
+
     @pytest.mark.requirements
     @pytest.mark.complex
     def test_complex_lock_changing_candidate(self):