Repository: linkedin/shiv

Issue:
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both
With shiv 0.0.14 from PyPI:

```console
$ shiv aws -c aws -p $(which python3.6) -o blergh
 shiv! 🔪
Collecting aws
Collecting fabric>=1.6 (from aws)
Collecting boto (from aws)
  Using cached https://files.pythonhosted.org/packages/bd/b7/a88a67002b1185ed9a8e8a6ef15266728c2361fcb4f1d02ea331e4c7741d/boto-2.48.0-py2.py3-none-any.whl
Collecting prettytable>=0.7 (from aws)
Collecting paramiko<3.0,>=1.10 (from fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/3e/db/cb7b6656e0e7387637ce850689084dc0b94b44df31cc52e5fc5c2c4fd2c1/paramiko-2.4.1-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl
Collecting pynacl>=1.0.1 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/74/8e/a6c0d340972d9e2f1a405aaa3f2460950b4c0337f92db0291a4355974529/PyNaCl-1.2.1-cp36-cp36m-macosx_10_6_intel.whl
Collecting bcrypt>=3.1.3 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/7e/59/d48fd712941da1a5d6490964a37bb3de2e526965b6766273f6a7049ee590/bcrypt-3.1.4-cp36-cp36m-macosx_10_6_intel.whl
Collecting cryptography>=1.5 (from paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/40/87/acdcf84ce6d25a7db1c113f4b9b614fd8d707b7ab56fbf17cf18cd26a627/cryptography-2.2.2-cp34-abi3-macosx_10_6_intel.whl
Collecting cffi>=1.4.1 (from pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/8e/be/40b1bc2c3221acdefeb9dab6773d43cda7543ed0d8c8df8768f05af2d01e/cffi-1.11.5-cp36-cp36m-macosx_10_6_intel.whl
Collecting six (from pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting idna>=2.1 (from cryptography>=1.5->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=1.5->paramiko<3.0,>=1.10->fabric>=1.6->aws)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.4.1->pynacl>=1.0.1->paramiko<3.0,>=1.10->fabric>=1.6->aws)
Installing collected packages: pyasn1, pycparser, cffi, six, pynacl, bcrypt, idna, asn1crypto, cryptography, paramiko, fabric, boto, prettytable, aws
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 335, in run
    use_user_site=options.use_user_site,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 49, in install_given_reqs
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 748, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 961, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/wheel.py", line 216, in move_wheel_files
    prefix=prefix,
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/locations.py", line 165, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

Pip install failed!
``` 

Here's the packages installed systemwide alongside shiv:

```
click==6.7
importlib-resources==0.5
pip==10.0.1
setuptools==39.1.0
shiv==0.0.14
wheel==0.31.0
```

OS X, Python 3.6.5 from Homebrew.

Discussion:
coveralls: ## Pull Request Test Coverage Report for [Build 56](https://coveralls.io/builds/16907439)

* **0** of **0**  **(NaN%)** changed or added relevant lines in **0** files are covered.
* **2** unchanged lines in **1** file lost coverage.
* Overall coverage increased (+**0.9%**) to **70.652%**

---


|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [.tox/py36/lib/python3.6/site-packages/shiv/pip.py](https://coveralls.io/builds/16907439/source?filename=.tox%2Fpy36%2Flib%2Fpython3.6%2Fsite-packages%2Fshiv%2Fpip.py#L36) | 2 | 92.86% |
<!-- | **Total:** | **2** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/16907439/badge)](https://coveralls.io/builds/16907439) |
| :-- | --: |
| Change from base [Build 55](https://coveralls.io/builds/16906049): |  0.9% |
| Covered Lines: | 195 |
| Relevant Lines: | 276 |

---
##### 💛  - [Coveralls](https://coveralls.io)

sixninetynine: Thanks @rouge8 !

Couple things:

According to [distutils](https://github.com/python/cpython/blob/master/Lib/distutils/dist.py#L339-L343) there's _three_ places this config file can butt it's head in and ruin stuff:

        There are three possible config files: distutils.cfg in the
        Distutils installation directory (ie. where the top-level
        Distutils __inst__.py file lives), a file in the user's home
        directory named .pydistutils.cfg on Unix and pydistutils.cfg
        on Windows/Mac; and setup.cfg in the current directory.

a modified `distutils.cfg` is what homebrew ships
so I think if you update this PR to check for _either_ `pydistutils.cfg` or `.pydistutils.cfg` we'll be in good shape.

Maybe we could kill two birds with one stone (existence check and getting a Path object):

```
def pydistutils_cfg() -> Path:
    try:
        return next(Path.home().glob('*pydistutils.cfg'))
    except StopIteration:
        ...

pydistutils = pydistutils_cfg()

if pydistutils is not None and pydistutils.exists():
    # stuff
```

rouge8: yikes. updated ✅ 

Patch:
diff --git a/src/shiv/constants.py b/src/shiv/constants.py
index 1b82477..33d742b 100644
--- a/src/shiv/constants.py
+++ b/src/shiv/constants.py
@@ -18,3 +18,4 @@ BLACKLISTED_ARGS: Dict[Tuple[str, ...], str] = {
     ("-d", "--download"): "Shiv needs to actually perform an install, not merely a download.",
     ("--user", "--root", "--prefix"): "Which conflicts with Shiv's internal use of '--target'.",
 }
+DISTUTILS_CFG_NO_PREFIX = "[install]\nprefix="
diff --git a/src/shiv/pip.py b/src/shiv/pip.py
index 3be1339..9deab31 100644
--- a/src/shiv/pip.py
+++ b/src/shiv/pip.py
@@ -3,9 +3,10 @@ import os
 import subprocess
 import sys
 
+from pathlib import Path
 from typing import Generator, List
 
-from .constants import PIP_REQUIRE_VIRTUALENV, PIP_INSTALL_ERROR
+from .constants import PIP_REQUIRE_VIRTUALENV, PIP_INSTALL_ERROR, DISTUTILS_CFG_NO_PREFIX
 
 
 @contextlib.contextmanager
@@ -17,12 +18,28 @@ def clean_pip_env() -> Generator[None, None, None]:
     """
     require_venv = os.environ.pop(PIP_REQUIRE_VIRTUALENV, None)
 
+    # based on
+    # https://github.com/python/cpython/blob/8cf4b34b3665b8bb39ea7111e6b5c3410899d3e4/Lib/distutils/dist.py#L333-L363
+    pydistutils = Path.home() / (".pydistutils.cfg" if os.name == "posix" else "pydistutils.cfg")
+    pydistutils_already_existed = pydistutils.exists()
+
+    if not pydistutils_already_existed:
+        # distutils doesn't support using --target if there's a config file
+        # specifying --prefix. Homebrew's Pythons include a distutils.cfg that
+        # breaks `pip install --target` with any non-wheel packages. We can
+        # work around that by creating a temporary ~/.pydistutils.cfg
+        # specifying an empty prefix.
+        pydistutils.write_text(DISTUTILS_CFG_NO_PREFIX)
+
     try:
         yield
 
     finally:
         if require_venv is not None:
             os.environ[PIP_REQUIRE_VIRTUALENV] = require_venv
+        if not pydistutils_already_existed:
+            # remove the temporary ~/.pydistutils.cfg
+            pydistutils.unlink()
 
 
 def install(interpreter_path: str, args: List[str]) -> None:


Test patch:
diff --git a/test/conftest.py b/test/conftest.py
index 9229abb..3454d5f 100644
--- a/test/conftest.py
+++ b/test/conftest.py
@@ -1,11 +1,23 @@
+import os
+
 from pathlib import Path
 
 import pytest
 
 
-@pytest.fixture
-def package_location():
-    return Path(__file__).absolute().parent / 'package'
+@pytest.fixture(params=[True, False], ids=['.', 'absolute-path'])
+def package_location(request):
+    package_location = Path(__file__).absolute().parent / 'package'
+
+    if request.param is True:
+        # test building from the current directory
+        cwd = os.getcwd()
+        os.chdir(package_location)
+        yield Path('.')
+        os.chdir(cwd)
+    else:
+        # test building an absolute path
+        yield package_location
 
 
 @pytest.fixture
diff --git a/test/test_cli.py b/test/test_cli.py
index ad2f6bb..a38017c 100644
--- a/test/test_cli.py
+++ b/test/test_cli.py
@@ -45,11 +45,20 @@ class TestCLI:
         # assert we got the correct reason
         assert strip_header(result.output) == DISALLOWED_PIP_ARGS.format(arg=arg, reason=reason)
 
-    def test_hello_world(self, tmpdir, runner, package_location):
+    # /usr/local/bin/python3.6 is a test for https://github.com/linkedin/shiv/issues/16
+    @pytest.mark.parametrize('interpreter', [None, Path('/usr/local/bin/python3.6')])
+    def test_hello_world(self, tmpdir, runner, package_location, interpreter):
+        if interpreter is not None and not interpreter.exists():
+            pytest.skip(f'Interpreter "{interpreter}" does not exist')
+
         with tempfile.TemporaryDirectory(dir=tmpdir) as tmpdir:
             output_file = Path(tmpdir, 'test.pyz')
 
-            result = runner(['-e', 'hello:main', '-o', output_file.as_posix(), package_location.as_posix()])
+            args = ['-e', 'hello:main', '-o', output_file.as_posix(), package_location.as_posix()]
+            if interpreter is not None:
+                args = ['-p', interpreter.as_posix()] + args
+
+            result = runner(args)
 
             # check that the command successfully completed
             assert result.exit_code == 0
diff --git a/test/test_pip.py b/test/test_pip.py
new file mode 100644
index 0000000..aba1721
--- /dev/null
+++ b/test/test_pip.py
@@ -0,0 +1,48 @@
+import os
+
+from pathlib import Path
+
+import pytest
+
+from shiv.constants import PIP_REQUIRE_VIRTUALENV, DISTUTILS_CFG_NO_PREFIX
+from shiv.pip import clean_pip_env
+
+
+@pytest.mark.parametrize("pydistutils_path, os_name", [
+    ("pydistutils.cfg", "nt"),
+    (".pydistutils.cfg", "posix"),
+    (None, os.name),
+])
+def test_clean_pip_env(monkeypatch, tmpdir, pydistutils_path, os_name):
+    home = tmpdir.join("home").ensure(dir=True)
+    monkeypatch.setenv("HOME", home)
+
+    # patch os.name so distutils will use `pydistutils_path` for its config
+    monkeypatch.setattr(os, 'name', os.name)
+
+    if pydistutils_path:
+        pydistutils = Path.home() / pydistutils_path
+        pydistutils_contents = "foobar"
+        pydistutils.write_text(pydistutils_contents)
+    else:
+        pydistutils = Path.home() / ".pydistutils.cfg"
+        pydistutils_contents = None
+
+    before_env_var = "foo"
+    monkeypatch.setenv(PIP_REQUIRE_VIRTUALENV, before_env_var)
+
+    with clean_pip_env():
+        assert PIP_REQUIRE_VIRTUALENV not in os.environ
+
+        if not pydistutils_path:
+            # ~/.pydistutils.cfg was created
+            assert pydistutils.read_text() == DISTUTILS_CFG_NO_PREFIX
+        else:
+            # ~/.pydistutils.cfg was not modified
+            assert pydistutils.read_text() == pydistutils_contents
+
+    assert os.environ.get(PIP_REQUIRE_VIRTUALENV) == before_env_var
+
+    # If a temporary ~/.pydistutils.cfg was created, it was deleted. If
+    # ~/.pydistutils.cfg already existed, it still exists.
+    assert pydistutils.exists() == bool(pydistutils_path)