Repository: pypa/pipenv

Issue:
Pipenv sync in sequential mode ignores pip install results
### Issue description

`pipenv sync --sequential` may silently fail to install a package.
`pipenv sync --sequential -v` doesn't emit verbose logs of any `pip install` invocations but the first.

Fixed in #3537.

### Expected result

An error:
```
[user@dev repro]$ ../.venv/bin/pipenv sync --sequential
Creating a virtualenv for this project…
Pipfile: /home/user/pipenv/repro/Pipfile
Using /home/user/pipenv/.venv/bin/python3 (3.7.2) to create virtualenv…
⠇ Creating virtual environment...Already using interpreter /home/user/pipenv/.venv/bin/python3
Using base prefix '/usr'
New python executable in /home/user/pipenv/repro/.venv/bin/python3
Also creating executable in /home/user/pipenv/repro/.venv/bin/python
Installing setuptools, pip, wheel...
done.

✔ Successfully created virtual environment! 
Virtualenv location: /home/user/pipenv/repro/.venv
Installing dependencies from Pipfile.lock (63bec0)…
An error occurred while installing requests==2.21.0 --hash=sha256:0000000000000000000000000000000000000000000000000000000000000000! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:03
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/home/user/pipenv/pipenv/core.py", line 2599, in do_sync
[pipenv.exceptions.InstallError]:       system=system,
[pipenv.exceptions.InstallError]:   File "/home/user/pipenv/pipenv/core.py", line 1228, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/home/user/pipenv/pipenv/core.py", line 834, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "/home/user/pipenv/pipenv/core.py", line 740, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "/home/user/pipenv/pipenv/core.py", line 661, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting requests==2.21.0 (from -r /tmp/pipenv-4aqevpfx-requirements/pipenv-zu00lfak-requirement.txt (line 1))', '  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl']
[pipenv.exceptions.InstallError]: ['THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.', '    requests==2.21.0 from https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl#sha256=7bf2a778576d825600030a110f3c0e3e8edc51dfaafe1c146e39a2027784957b (from -r /tmp/pipenv-4aqevpfx-requirements/pipenv-zu00lfak-requirement.txt (line 1)):', '        Expected sha256 0000000000000000000000000000000000000000000000000000000000000000', '             Got        7bf2a778576d825600030a110f3c0e3e8edc51dfaafe1c146e39a2027784957b']
ERROR: ERROR: Package installation failed...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:00
```

### Actual result

Silent failure to install `requests`:
```
[user@dev repro]$ ../.venv/bin/pipenv sync --sequential
Creating a virtualenv for this project…
Pipfile: /home/user/pipenv/repro/Pipfile
Using /home/user/pipenv/.venv/bin/python3 (3.7.2) to create virtualenv…
⠧ Creating virtual environment...Already using interpreter /home/user/pipenv/.venv/bin/python3
Using base prefix '/usr'
New python executable in /home/user/pipenv/repro/.venv/bin/python3
Also creating executable in /home/user/pipenv/repro/.venv/bin/python
Installing setuptools, pip, wheel...
done.

✔ Successfully created virtual environment! 
Virtualenv location: /home/user/pipenv/repro/.venv
Installing dependencies from Pipfile.lock (63bec0)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:04
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!
[user@dev repro]$ ../.venv/bin/pipenv run pip list
Package    Version   
---------- ----------
certifi    2018.11.29
chardet    3.0.4     
idna       2.8       
pip        19.0.3    
setuptools 40.8.0    
urllib3    1.24.1    
wheel      0.33.1    
```

### Steps to replicate

Cause a hash mismatch one of the packages.
```
        "requests": {
            "hashes": [
                "sha256:0000000000000000000000000000000000000000000000000000000000000000"
            ],
```
Run `pipenv sync --sequential`.

<details><summary>$ pipenv --support</summary>

Pipenv version: `'2018.11.27.dev0'`

Pipenv location: `'/home/user/pipenv/pipenv'`

Python location: `'/home/user/pipenv/.venv/bin/python3'`

Python installations found:

  - `3.7.2`: `/usr/bin/python3.7m`
  - `3.7.2`: `/usr/bin/python3.7`
  - `3.5.2`: `/home/user/.pyenv/versions/3.5.2/bin/python`
  - `3.5.2`: `/home/user/.pyenv/versions/3.5.2/bin/python3.5m`
  - `2.7.15`: `/usr/bin/python`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.7.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.14.74-1.pvops.qubes.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Mon Oct 8 17:14:24 UTC 2018',
 'python_full_version': '3.7.2',
 'python_version': '3.7',
 'sys_platform': 'linux'}
```

System environment variables:

  - `QUBES_KEYMAP`
  - `LS_COLORS`
  - `QREXEC_SERVICE_ARGUMENT`
  - `LANG`
  - `HISTCONTROL`
  - `DISPLAY`
  - `HOSTNAME`
  - `PYENV_ROOT`
  - `OLDPWD`
  - `COLORTERM`
  - `XDG_VTNR`
  - `SSH_AUTH_SOCK`
  - `QT_X11_NO_MITSHM`
  - `XDG_SESSION_ID`
  - `QUBES_ENV_SOURCED`
  - `USER`
  - `VMTYPE`
  - `GNOME_TERMINAL_SCREEN`
  - `UPDTYPE`
  - `PWD`
  - `QREXEC_AGENT_PID`
  - `HOME`
  - `SSH_AGENT_PID`
  - `QUBES_USER_KEYMAP`
  - `XDG_DATA_DIRS`
  - `MAIL`
  - `WINDOWPATH`
  - `SHELL`
  - `TERM`
  - `VTE_VERSION`
  - `QREXEC_SERVICE_FULL_NAME`
  - `QREXEC_REMOTE_DOMAIN`
  - `XDG_SESSION_CLASS`
  - `GNOME_TERMINAL_SERVICE`
  - `XDG_SEAT`
  - `SHLVL`
  - `PYENV_SHELL`
  - `GNOME_DESKTOP_SESSION_ID`
  - `LOGNAME`
  - `DBUS_SESSION_BUS_ADDRESS`
  - `XDG_RUNTIME_DIR`
  - `PATH`
  - `PS1`
  - `HISTSIZE`
  - `LESSOPEN`
  - `_`
  - `PIP_DISABLE_PIP_VERSION_CHECK`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_SHIMS_BASE_MODULE`
  - `PIP_PYTHON_PATH`
  - `PYTHONFINDER_IGNORE_UNSUPPORTED`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/home/user/.local/bin:/home/user/bin:/home/user/.cargo/bin:/home/user/.cargo/bin:/home/user/.pyenv/shims:/home/user/.pyenv/bin:/home/user/.cargo/bin:/home/user/.local/bin:/home/user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin`
  - `SHELL`: `/bin/bash`
  - `LANG`: `C.UTF-8`
  - `PWD`: `/home/user/pipenv/repro`


---------------------------

Contents of `Pipfile` ('/home/user/pipenv/repro/Pipfile'):

```toml
[packages]
requests = "*"

```


Contents of `Pipfile.lock` ('/home/user/pipenv/repro/Pipfile.lock'):

```json
{
    "_meta": {
        "hash": {
            "sha256": "70e8bf6bc774f5ca177467cab4e67d4264d0536857993326abc13ff43063bec0"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:47f9c83ef4c0c621eaef743f133f09fa8a74a9b75f037e8624f83bd1b6626cb7",
                "sha256:993f830721089fef441cdfeb4b2c8c9df86f0c63239f06bd025a76a7daddb033"
            ],
            "version": "==2018.11.29"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "idna": {
            "hashes": [
                "sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407",
                "sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c"
            ],
            "version": "==2.8"
        },
        "requests": {
            "hashes": [
                "sha256:0000000000000000000000000000000000000000000000000000000000000000"
            ],
            "index": "pypi",
            "version": "==2.21.0"
        },
        "urllib3": {
            "hashes": [
                "sha256:61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39",
                "sha256:de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22"
            ],
            "version": "==1.24.1"
        }
    },
    "develop": {}
}

```
</details>



Discussion:

Resolving patch:
diff --git a/pipenv/core.py b/pipenv/core.py
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -645,10 +645,10 @@ def do_where(virtualenv=False, bare=True):
             click.echo(location)
 
 
-def _cleanup_procs(procs, concurrent, failed_deps_queue, retry=True):
+def _cleanup_procs(procs, failed_deps_queue, retry=True):
     while not procs.empty():
         c = procs.get()
-        if concurrent:
+        if not c.blocking:
             c.block()
         failed = False
         if c.return_code != 0:
@@ -682,7 +682,7 @@ def _cleanup_procs(procs, concurrent, failed_deps_queue, retry=True):
 def batch_install(deps_list, procs, failed_deps_queue,
                   requirements_dir, no_deps=False, ignore_hashes=False,
                   allow_global=False, blocking=False, pypi_mirror=None,
-                  nprocs=PIPENV_MAX_SUBPROCESS, retry=True):
+                  retry=True):
     from .vendor.requirementslib.models.utils import strip_extras_markers_from_requirement
     failed = (not retry)
     if not failed:
@@ -753,12 +753,13 @@ def batch_install(deps_list, procs, failed_deps_queue,
                 extra_indexes=extra_indexes,
                 use_pep517=not failed,
             )
-            if procs.qsize() < nprocs:
-                c.dep = dep
-                procs.put(c)
+            c.dep = dep
+            if dep.is_vcs or dep.editable:
+                c.block()
 
+            procs.put(c)
             if procs.full() or procs.qsize() == len(deps_list):
-                _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
+                _cleanup_procs(procs, failed_deps_queue, retry=retry)
 
 
 def do_install_dependencies(
@@ -782,7 +783,6 @@ def do_install_dependencies(
     from six.moves import queue
     if requirements:
         bare = True
-    blocking = not concurrent
     # Load the lockfile if it exists, or if only is being used (e.g. lock is being used).
     if skip_lock or only or not project.lockfile_exists:
         if not bare:
@@ -818,19 +818,19 @@ def do_install_dependencies(
         )
         sys.exit(0)
 
-    procs = queue.Queue(maxsize=PIPENV_MAX_SUBPROCESS)
+    if concurrent:
+        nprocs = PIPENV_MAX_SUBPROCESS
+    else:
+        nprocs = 1
+    procs = queue.Queue(maxsize=nprocs)
     failed_deps_queue = queue.Queue()
     if skip_lock:
         ignore_hashes = True
 
     install_kwargs = {
         "no_deps": no_deps, "ignore_hashes": ignore_hashes, "allow_global": allow_global,
-        "blocking": blocking, "pypi_mirror": pypi_mirror
+        "blocking": not concurrent, "pypi_mirror": pypi_mirror
     }
-    if concurrent:
-        install_kwargs["nprocs"] = PIPENV_MAX_SUBPROCESS
-    else:
-        install_kwargs["nprocs"] = 1
 
     # with project.environment.activated():
     batch_install(
@@ -838,7 +838,7 @@ def do_install_dependencies(
     )
 
     if not procs.empty():
-        _cleanup_procs(procs, concurrent, failed_deps_queue)
+        _cleanup_procs(procs, failed_deps_queue)
 
     # Iterate over the hopefully-poorly-packaged dependencies…
     if not failed_deps_queue.empty():
@@ -850,7 +850,6 @@ def do_install_dependencies(
             failed_dep = failed_deps_queue.get()
             retry_list.append(failed_dep)
         install_kwargs.update({
-            "nprocs": 1,
             "retry": False,
             "blocking": True,
         })
@@ -858,7 +857,7 @@ def do_install_dependencies(
             retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
         )
     if not procs.empty():
-        _cleanup_procs(procs, False, failed_deps_queue, retry=False)
+        _cleanup_procs(procs, failed_deps_queue, retry=False)
 
 
 def convert_three_to_python(three, python):
@@ -2507,6 +2506,8 @@ def do_run(command, args, three=None, python=False, pypi_mirror=None):
     previous_pipenv_active_value = os.environ.get("PIPENV_ACTIVE")
     os.environ["PIPENV_ACTIVE"] = vistir.misc.fs_str("1")
 
+    os.environ.pop("PIP_SHIMS_BASE_MODULE", None)
+
     try:
         script = project.build_script(command, args)
         cmd_string = ' '.join([script.command] + script.args)


Test patch:
diff --git a/tests/integration/test_sync.py b/tests/integration/test_sync.py
--- a/tests/integration/test_sync.py
+++ b/tests/integration/test_sync.py
@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import, print_function
+import json
 import os
 
 import pytest
@@ -68,3 +69,46 @@ def test_sync_should_not_lock(PipenvInstance, pypi):
         c = p.pipenv('sync')
         assert c.return_code == 0
         assert lockfile_content == p.lockfile
+
+
+@pytest.mark.sync
+@pytest.mark.lock
+def test_sync_sequential_detect_errors(PipenvInstance, pypi):
+    with PipenvInstance(pypi=pypi) as p:
+        with open(p.pipfile_path, 'w') as f:
+            contents = """
+[packages]
+requests = "*"
+        """.strip()
+            f.write(contents)
+
+        c = p.pipenv('lock')
+        assert c.return_code == 0
+
+        # Force hash mismatch when installing `requests`
+        lock = p.lockfile
+        lock['default']['requests']['hashes'] = ['sha256:' + '0' * 64]
+        with open(p.lockfile_path, 'w') as f:
+            json.dump(lock, f)
+
+        c = p.pipenv('sync --sequential')
+        assert c.return_code != 0
+
+
+@pytest.mark.sync
+@pytest.mark.lock
+def test_sync_sequential_verbose(PipenvInstance, pypi):
+    with PipenvInstance(pypi=pypi) as p:
+        with open(p.pipfile_path, 'w') as f:
+            contents = """
+[packages]
+requests = "*"
+        """.strip()
+            f.write(contents)
+
+        c = p.pipenv('lock')
+        assert c.return_code == 0
+
+        c = p.pipenv('sync --sequential --verbose')
+        for package in p.lockfile['default']:
+            assert 'Successfully installed {}'.format(package) in c.out