Repository: pypa/pip

Issue:
pip 19.0.2: IndexError when building wheel on osx
**Environment**

* pip version: 19.0.2
* Python version: 3.6
* OS: macos 10.13
* Platform: Travis-ci

The error was observed on travis-ci build's using Miniconda with an environment that contains both conda and pip packages.

**Description**
It seems release 19.0.2 breaks something related to wheel building under certain circumstances.
The problem started happening yesterday and it popped up exactly after 19.0.2 was released. The build was succeeding on 19.0.1, and as soon as 19.0.2 was available, the build started failing. See [successful build #10](https://travis-ci.com/gramaziokohler/robotic_assembly_workshop/builds/100329545) and [failed build #11](https://travis-ci.com/gramaziokohler/robotic_assembly_workshop/builds/100330616). After pinning pip back to 19.0.1, the build started working again.

I don't have a clear idea of what exactly fails, but it seems to be related to `cvxpy` and its dependency [`scs`](https://github.com/cvxgrp/cvxpy/blob/master/setup.py#L70), but it is not clear to me if this is in any way connected to the root cause, or is only the visible side-effect.

**How to Reproduce**

1. Build [using this travis file](https://github.com/gramaziokohler/robotic_assembly_workshop/blob/master/.travis.yml)
2. If pip version is not pinned, and it uses 19.0.2, an error occurs (see below for tracebacks and full output).

**Output**
The full build output is available directly on travis: https://travis-ci.com/gramaziokohler/robotic_assembly_workshop/builds/100338039

```
  Stored in directory: /Users/travis/Library/Caches/pip/wheels/1d/2f/99/1594715f229f93f235d8b370a3b6f16e2b00fbac1087a99902
  Building wheel for scs (setup.py) ... done
Exception:
Traceback (most recent call last):
  File "/Users/travis/miniconda3/envs/workshop/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/Users/travis/miniconda3/envs/workshop/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 355, in run
    session=session, autobuilding=True
  File "/Users/travis/miniconda3/envs/workshop/lib/python3.6/site-packages/pip/_internal/wheel.py", line 980, in build
    python_tag=python_tag,
  File "/Users/travis/miniconda3/envs/workshop/lib/python3.6/site-packages/pip/_internal/wheel.py", line 813, in _build_one
    python_tag=python_tag)
  File "/Users/travis/miniconda3/envs/workshop/lib/python3.6/site-packages/pip/_internal/wheel.py", line 821, in _build_one_inside_env
    wheel_path = builder(req, temp_dir.path, python_tag=python_tag)
  File "/Users/travis/miniconda3/envs/workshop/lib/python3.6/site-packages/pip/_internal/wheel.py", line 898, in _build_one_legacy
    return os.path.join(tempd, sorted(os.listdir(tempd))[0])
IndexError: list index out of range
```


Discussion:
The line causing this exception was added in PR #6236.
It looks like before, the build in question (of scs) was hitting this line in `_build_one_inside_env()`:
https://github.com/pypa/pip/pull/6236/files#diff-79304b3f74b6bf37043da4d96ccd3755L831
and then causing `_build_one_inside_env()` to return `None`. The line in question looked like this, so you can see the exception was being swallowed--
```python
except Exception:
    pass
```
Now, however, the exception is bubbling up without being handled because the code causing the `IndexError` was refactored out of that try-except block.

If you look at the Travis log for the successful case (the older one), you can see here: https://travis-ci.com/gramaziokohler/robotic_assembly_workshop/builds/100329545#L284
that it says scs indeed failed to build:
```
Building wheel for scs (setup.py) ... done
Running setup.py clean for scs
Building wheel for future (setup.py) ... done
Stored in directory: /Users/travis/Library/Caches/pip/wheels/0c/61/d2/d6b7317325828fbb39ee6ad559dbe4664d0896da4721bf379e
Successfully built compas-assembly compas-rbe cvxpy future
Failed to build scs
```
But this wasn't causing pip to error out before because of the exception swallowing I mentioned above.




Resolving patch:
diff --git a/src/pip/_internal/wheel.py b/src/pip/_internal/wheel.py
--- a/src/pip/_internal/wheel.py
+++ b/src/pip/_internal/wheel.py
@@ -779,6 +779,42 @@ def should_use_ephemeral_cache(
     return True
 
 
+def get_legacy_build_wheel_path(
+    names,  # type: List[str]
+    temp_dir,  # type: str
+    req,  # type: InstallRequirement
+    command_args,  # type: List[str]
+    command_output,  # type: str
+):
+    # type: (...) -> Optional[str]
+    """
+    Return the path to the wheel in the temporary build directory.
+    """
+    # Sort for determinism.
+    names = sorted(names)
+    if not names:
+        # call_subprocess() ensures that the command output ends in a newline.
+        msg = (
+            'Failed building wheel for {} with args: {}\n'
+            'Command output:\n{}'
+            '-----------------------------------------'
+        ).format(req.name, command_args, command_output)
+        logger.error(msg)
+        return None
+    if len(names) > 1:
+        # call_subprocess() ensures that the command output ends in a newline.
+        msg = (
+            'Found more than one file after building wheel for {} '
+            'with args: {}\n'
+            'Names: {}\n'
+            'Command output:\n{}'
+            '-----------------------------------------'
+        ).format(req.name, command_args, names, command_output)
+        logger.warning(msg)
+
+    return os.path.join(temp_dir, names[0])
+
+
 class WheelBuilder(object):
     """Build wheels from a RequirementSet."""
 
@@ -888,14 +924,21 @@ def _build_one_legacy(self, req, tempd, python_tag=None):
                 wheel_args += ["--python-tag", python_tag]
 
             try:
-                call_subprocess(wheel_args, cwd=req.setup_py_dir,
-                                show_stdout=False, spinner=spinner)
+                output = call_subprocess(wheel_args, cwd=req.setup_py_dir,
+                                         show_stdout=False, spinner=spinner)
             except Exception:
                 spinner.finish("error")
                 logger.error('Failed building wheel for %s', req.name)
                 return None
-            # listdir's return value is sorted to be deterministic
-            return os.path.join(tempd, sorted(os.listdir(tempd))[0])
+            names = os.listdir(tempd)
+            wheel_path = get_legacy_build_wheel_path(
+                names=names,
+                temp_dir=tempd,
+                req=req,
+                command_args=wheel_args,
+                command_output=output,
+            )
+            return wheel_path
 
     def _clean_one(self, req):
         base_args = self._base_setup_args(req)


Test patch:
diff --git a/tests/lib/__init__.py b/tests/lib/__init__.py
--- a/tests/lib/__init__.py
+++ b/tests/lib/__init__.py
@@ -21,6 +21,10 @@
 pyversion_tuple = sys.version_info
 
 
+def assert_paths_equal(actual, expected):
+    os.path.normpath(actual) == os.path.normpath(expected)
+
+
 def path_to_url(path):
     """
     Convert a path to URI. The path will be made absolute and
diff --git a/tests/unit/test_wheel.py b/tests/unit/test_wheel.py
--- a/tests/unit/test_wheel.py
+++ b/tests/unit/test_wheel.py
@@ -15,7 +15,7 @@
 from pip._internal.req.req_install import InstallRequirement
 from pip._internal.utils.compat import WINDOWS
 from pip._internal.utils.misc import unpack_file
-from tests.lib import DATA_DIR
+from tests.lib import DATA_DIR, assert_paths_equal
 
 
 @pytest.mark.parametrize(
@@ -38,25 +38,13 @@ def test_contains_egg_info(s, expected):
     assert result == expected
 
 
-@pytest.mark.parametrize(
-    "base_name, autobuilding, cache_available, expected",
-    [
-        ('pendulum-2.0.4', False, False, False),
-        # The following cases test autobuilding=True.
-        # Test _contains_egg_info() returning True.
-        ('pendulum-2.0.4', True, True, False),
-        ('pendulum-2.0.4', True, False, True),
-        # Test _contains_egg_info() returning False.
-        ('pendulum', True, True, True),
-        ('pendulum', True, False, True),
-    ],
-)
-def test_should_use_ephemeral_cache__issue_6197(
-    base_name, autobuilding, cache_available, expected,
-):
+def make_test_install_req(base_name=None):
     """
-    Regression test for: https://github.com/pypa/pip/issues/6197
+    Return an InstallRequirement object for testing purposes.
     """
+    if base_name is None:
+        base_name = 'pendulum-2.0.4'
+
     req = Requirement('pendulum')
     link_url = (
         'https://files.pythonhosted.org/packages/aa/{base_name}.tar.gz'
@@ -76,6 +64,30 @@ def test_should_use_ephemeral_cache__issue_6197(
         link=link,
         source_dir='/tmp/pip-install-9py5m2z1/pendulum',
     )
+
+    return req
+
+
+@pytest.mark.parametrize(
+    "base_name, autobuilding, cache_available, expected",
+    [
+        ('pendulum-2.0.4', False, False, False),
+        # The following cases test autobuilding=True.
+        # Test _contains_egg_info() returning True.
+        ('pendulum-2.0.4', True, True, False),
+        ('pendulum-2.0.4', True, False, True),
+        # Test _contains_egg_info() returning False.
+        ('pendulum', True, True, True),
+        ('pendulum', True, False, True),
+    ],
+)
+def test_should_use_ephemeral_cache__issue_6197(
+    base_name, autobuilding, cache_available, expected,
+):
+    """
+    Regression test for: https://github.com/pypa/pip/issues/6197
+    """
+    req = make_test_install_req(base_name=base_name)
     assert not req.is_wheel
     assert req.link.is_artifact
 
@@ -87,6 +99,59 @@ def test_should_use_ephemeral_cache__issue_6197(
     assert ephem_cache is expected
 
 
+def call_get_legacy_build_wheel_path(caplog, names):
+    req = make_test_install_req()
+    wheel_path = wheel.get_legacy_build_wheel_path(
+        names=names,
+        temp_dir='/tmp/abcd',
+        req=req,
+        command_args=['arg1', 'arg2'],
+        command_output='output line 1\noutput line 2\n',
+    )
+    return wheel_path
+
+
+def test_get_legacy_build_wheel_path(caplog):
+    actual = call_get_legacy_build_wheel_path(caplog, names=['name'])
+    assert_paths_equal(actual, '/tmp/abcd/name')
+    assert not caplog.records
+
+
+def test_get_legacy_build_wheel_path__no_names(caplog):
+    actual = call_get_legacy_build_wheel_path(caplog, names=[])
+    assert actual is None
+    assert len(caplog.records) == 1
+    record = caplog.records[0]
+    assert record.levelname == 'ERROR'
+    assert record.message.splitlines() == [
+        "Failed building wheel for pendulum with args: ['arg1', 'arg2']",
+        "Command output:",
+        "output line 1",
+        "output line 2",
+        "-----------------------------------------",
+    ]
+
+
+def test_get_legacy_build_wheel_path__multiple_names(caplog):
+    # Deliberately pass the names in non-sorted order.
+    actual = call_get_legacy_build_wheel_path(
+        caplog, names=['name2', 'name1'],
+    )
+    assert_paths_equal(actual, '/tmp/abcd/name1')
+    assert len(caplog.records) == 1
+    record = caplog.records[0]
+    assert record.levelname == 'WARNING'
+    assert record.message.splitlines() == [
+        ("Found more than one file after building wheel for pendulum "
+         "with args: ['arg1', 'arg2']"),
+        "Names: ['name1', 'name2']",
+        "Command output:",
+        "output line 1",
+        "output line 2",
+        "-----------------------------------------",
+    ]
+
+
 @pytest.mark.parametrize("console_scripts",
                          ["pip = pip._internal.main:pip",
                           "pip:pip = pip._internal.main:pip"])