Repository: pypa/pipenv

Issue:
Pipenv lock still takes way too long to complete
`pipenv lock` is still taking up to 30 minutes to complete for my medium-sized project.

While #4403 was closed as fixed and indeed no full tree copies are made any more, `pipenv lock` is still unworkably slow. What used to take 50 seconds under `2018.11.26`, now takes 30 minutes to figure out that nothing changed:

```
$ export PIPENV_INSTALL_TIMEOUT=10000
$ cp Pipfile.lock Pipfile.lock.before
$ time pipenv lock
Locking [dev-packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (0e1198)!

real	29m19.701s
user	28m39.300s
sys	4m8.222s
$ diff -su Pipfile.lock.old Pipfile.lock
Files Pipfile.lock.old and Pipfile.lock are identical
```

This is making it untenable to keep using Pipenv.

Full [`pipenv lock -v` output in a gist](https://gist.github.com/mjpieters/715b825a5fded073adc2945d802f5177).

Pipfile:

```ini
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
flake8 = "*"
flake8-bugbear = "*"
black = "==19.10b0"
pre-commit = "*"
pytest = "*"
pytest-cov = "*"

[packages]
project-name = {editable = true,path = "."}
"zc.buildout" = "*"

[requires]
python_version = "3.6"
```

where `project-name` (redacted name) lists the following `install-requires` lines:

```ini
install_requires =
    apache-airflow[aws,celery,postgres,redis] >= 1.10.11
    airflow_multi_dagrun
    airflow-prometheus-exporter
```

-----------

<details><summary>$ pipenv --support</summary>

Pipenv version: `'2020.8.13'`

Pipenv location: `'/usr/local/Cellar/pipenv/2020.8.13/libexec/lib/python3.8/site-packages/pipenv'`

Python location: `'/usr/local/Cellar/pipenv/2020.8.13/libexec/bin/python3.8'`

Python installations found:

  - `3.9.0`: `/Users/mj/bin/python3.9`
  - `3.8.5`: `/usr/local/bin/python3`
  - `3.8.5`: `/usr/local/bin/python3.8`
  - `3.8.3`: `/Users/mj/bin/python3.8`
  - `3.7.7`: `/Users/mj/bin/python3.7`
  - `3.7.6`: `/opt/miniconda3/bin/python3`
  - `3.7.6`: `/opt/miniconda3/bin/python3.7m`
  - `3.7.6`: `/opt/miniconda3/bin/python3.7`
  - `3.7.3`: `/Users/mj/.pyenv/versions/3.7.3/bin/python3`
  - `3.6.10`: `/Users/mj/bin/python3.6`
  - `3.6.1`: `/Users/mj/bin/pypy3`
  - `3.5.9`: `/Users/mj/bin/python3.5`
  - `3.4.10`: `/Users/mj/bin/python3.4`
  - `3.3.7`: `/Users/mj/bin/python3.3`
  - `3.2.6`: `/Users/mj/bin/python3.2`
  - `2.7.18`: `/Users/mj/bin/python2.7`
  - `2.7.16`: `/usr/bin/python2.7`
  - `2.6.9`: `/Users/mj/bin/python2.6`
  - `2.5.6`: `/Users/mj/bin/python2.5`
  - `2.4.6`: `/Users/mj/bin/python2.4`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.8.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Wed May 27 17:00:02 PDT '
                     '2020; root:xnu-4570.71.80.1~1/RELEASE_X86_64',
 'python_full_version': '3.8.5',
 'python_version': '3.8',
 'sys_platform': 'darwin'}
```

System environment variables:

  - `PATH`
  - `AUTOJUMP_ERROR_PATH`
  - `NVM_INC`
  - `TERM_PROGRAM`
  - `GIT_PROMPT_END`
  - `rvm_bin_path`
  - `AUTOJUMP_SOURCED`
  - `GEM_HOME`
  - `NVM_CD_FLAGS`
  - `SHELL`
  - `TERM`
  - `HISTSIZE`
  - `TMPDIR`
  - `COPYFILE_DISABLE`
  - `IRBRC`
  - `Apple_PubSub_Socket_Render`
  - `CONDA_SHLVL`
  - `PERL5LIB`
  - `TERM_PROGRAM_VERSION`
  - `CONDA_PROMPT_MODIFIER`
  - `TERM_SESSION_ID`
  - `MY_RUBY_HOME`
  - `PERL_MB_OPT`
  - `LC_ALL`
  - `USER`
  - `HISTFILESIZE`
  - `NVM_DIR`
  - `COMMAND_MODE`
  - `CONDA_EXE`
  - `rvm_path`
  - `SSH_AUTH_SOCK`
  - `__CF_USER_TEXT_ENCODING`
  - `PTPIMG_API_KEY`
  - `_CE_CONDA`
  - `rvm_prefix`
  - `_`
  - `CONDA_PREFIX`
  - `PWD`
  - `HOMEBREW_GITHUB_API_TOKEN`
  - `EDITOR`
  - `LANG`
  - `ITERM_PROFILE`
  - `PYTHONSTARTUP`
  - `XPC_FLAGS`
  - `ITERM_ORIG_PS1`
  - `XPC_SERVICE_NAME`
  - `HISTCONTROL`
  - `_CE_M`
  - `FIGNORE`
  - `rvm_version`
  - `COLORFGBG`
  - `HOME`
  - `SHLVL`
  - `LC_TERMINAL_VERSION`
  - `ITERM_PREV_PS1`
  - `ITERM_SESSION_ID`
  - `PERL_LOCAL_LIB_ROOT`
  - `LOGNAME`
  - `CONDA_PYTHON_EXE`
  - `LC_CTYPE`
  - `GEM_PATH`
  - `GIT_PROMPT_ONLY_IN_REPO`
  - `CONDA_DEFAULT_ENV`
  - `NVM_BIN`
  - `DISPLAY`
  - `LC_TERMINAL`
  - `HGSRC`
  - `SECURITYSESSIONID`
  - `RUBY_VERSION`
  - `PERL_MM_OPT`
  - `COLORTERM`
  - `PIP_DISABLE_PIP_VERSION_CHECK`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_SHIMS_BASE_MODULE`
  - `PIP_PYTHON_PATH`
  - `PYTHONFINDER_IGNORE_UNSUPPORTED`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/usr/local/Cellar/pipenv/2020.8.13/libexec/tools:/usr/local/google-sdks/google-cloud-sdk/bin:/Users/mj/perl5/bin:/Users/mj/.nvm/versions/node/v12.14.1/bin:/Users/mj/.rvm/gems/ruby-2.3.0/bin:/Users/mj/.rvm/gems/ruby-2.3.0@global/bin:/Users/mj/.rvm/rubies/ruby-2.3.0/bin:/opt/miniconda3/bin:/opt/miniconda3/condabin:/Users/mj/bin:/Users/mj/Development/Library/zopatista_setup/bin:/Users/mj/Development/Library/zopatista_setup/audio:/usr/local/Cellar/ruby/2.7.1_2/bin:/Users/mj/.cargo/bin:/Users/mj/.poetry/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/usr/local/sbin:/opt/X11/bin:/Applications/Postgres.app/Contents/Versions/latest/bin:/Users/mj/.local/bin:/Users/mj/.rvm/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin`
  - `SHELL`: `/bin/bash`
  - `EDITOR`: `vim`
  - `LANG`: `en_GB.UTF-8`
  - `PWD`: `/Users/mj`


---------------------------

</details>



Discussion:
Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to `2018.11.26` speeds things up massively for me too
> Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to `2018.11.26` speeds things up massively for me too

Yes, +1! Thanks for looking into this, Martijn!  We had exactly the same issue (lots of big tmp directories), but sorta gave up for the time being… I couldn't really argue much when `pipenv lock` took 10+ minutes for simply adding another package.

(and hi, btw 👋)
A "workaround" that may work (and what we have used from time to time) is to remove the lock file and rerun the lock.  This may also hint at the underlying problem, which may have something to do with how pipenv is using the existing lock file during the lock process.
Deleting the `Pipfile.lock` speeds things up for sure. I just install all the packages from scratch and run `pipenv lock` without changing anything and it gets stuck eventhough there is no change at all
Can you have a try on the master branch? I recently disabled the copying of the local source tree.
> Can you have a try on the master branch? I recently disabled the copying of the local source tree.

Do you mean #4421? That's been [part of the 2020.8.13 release](https://github.com/pypa/pipenv/commit/9299ae1f7353bdd523a1829f3c7cad0ee67c2e3b) that I tested with for this issue. I did note that the tree copying no longer takes place.

I also don't see any changes on master since 2020.8.13 was released that look relevant. Did I miss something?
If I remember correctly, 2018.11.26 didn't resolve transient dependencies of an editable package. You can compare the lock files generated by both versions.
> If I remember correctly, 2018.11.26 didn't resolve transient dependencies of an editable package. You can compare the lock files generated by both versions.

I see no differences that look problematic. With `2020.8.13.dev0` (master branch) I see `markers` entries added. The dependencies I see added are ones that were due to other dependencies having been upgraded (I deleted the lock file to see if there was much speed difference).

With the lock file deleted and using `2020.8.13.dev0`, a `pipenv lock` took 10 minutes.
I removed all other dependencies except the editable one. It turns out no significant difference is seen in the time costs of 2018.11.26 and 2020.8.13. It will be great if you can narrow down which package brings the performance degrading.

I created a demo repo to showcase the issue, which you can clone and adjust to reproduce the performance issue:
https://github.com/frostming/pipenv-lock-performance

```bash
$ docker build . --build-arg PIPENV_VER=2020.8.13 -t pipenv-bug
$ docker build . --build-arg PIPENV_VER=2018.11.26 -t pipenv-bug
```

Resolving patch:
diff --git a/pipenv/patched/piptools/repositories/pypi.py b/pipenv/patched/piptools/repositories/pypi.py
--- a/pipenv/patched/piptools/repositories/pypi.py
+++ b/pipenv/patched/piptools/repositories/pypi.py
@@ -81,6 +81,9 @@ def get_hash(self, location):
         if can_hash:
             # hash url WITH fragment
             hash_value = self.get(new_location.url)
+            if not hash_value:
+                hash_value = "{}:{}".format(new_location.hash_name, new_location.hash)
+                hash_value = hash_value.encode('utf8')
         if not hash_value:
             hash_value = self._get_file_hash(new_location) if not new_location.url.startswith("ssh") else None
             hash_value = hash_value.encode('utf8') if hash_value else None
diff --git a/tasks/vendoring/__init__.py b/tasks/vendoring/__init__.py
--- a/tasks/vendoring/__init__.py
+++ b/tasks/vendoring/__init__.py
@@ -50,12 +50,12 @@
 HARDCODED_LICENSE_URLS = {
     "pytoml": "https://github.com/avakar/pytoml/raw/master/LICENSE",
     "cursor": "https://raw.githubusercontent.com/GijsTimmers/cursor/master/LICENSE",
-    "delegator.py": "https://raw.githubusercontent.com/kennethreitz/delegator.py/master/LICENSE",
+    "delegator.py": "https://raw.githubusercontent.com/amitt001/delegator.py/master/LICENSE",
     "click-didyoumean": "https://raw.githubusercontent.com/click-contrib/click-didyoumean/master/LICENSE",
     "click-completion": "https://raw.githubusercontent.com/click-contrib/click-completion/master/LICENSE",
     "parse": "https://raw.githubusercontent.com/techalchemy/parse/master/LICENSE",
     "semver": "https://raw.githubusercontent.com/k-bx/python-semver/master/LICENSE.txt",
-    "crayons": "https://raw.githubusercontent.com/kennethreitz/crayons/master/LICENSE",
+    "crayons": "https://raw.githubusercontent.com/MasterOdin/crayons/master/LICENSE",
     "pip-tools": "https://raw.githubusercontent.com/jazzband/pip-tools/master/LICENSE",
     "pytoml": "https://github.com/avakar/pytoml/raw/master/LICENSE",
     "webencodings": "https://github.com/SimonSapin/python-webencodings/raw/"


Test patch:
diff --git a/tests/pypi b/tests/pypi
--- a/tests/pypi
+++ b/tests/pypi
@@ -1 +1 @@
-Subproject commit 1881ecb45431952d2e18e2be3416a8835e53778a
+Subproject commit 77faecd45e85b14448667bc16af32ddfe6f8d42d
diff --git a/tests/pytest-pypi/pytest_pypi/app.py b/tests/pytest-pypi/pytest_pypi/app.py
--- a/tests/pytest-pypi/pytest_pypi/app.py
+++ b/tests/pytest-pypi/pytest_pypi/app.py
@@ -16,7 +16,7 @@
 from flask import Flask, redirect, abort, render_template, send_file, jsonify
 
 
-ReleaseTuple = collections.namedtuple("ReleaseTuple", ["path", "requires_python"])
+ReleaseTuple = collections.namedtuple("ReleaseTuple", ["path", "requires_python", "hash"])
 
 app = Flask(__name__)
 session = requests.Session()
@@ -86,14 +86,18 @@ def add_release(self, path_to_binary):
         path_to_binary = os.path.abspath(path_to_binary)
         path, release = os.path.split(path_to_binary)
         requires_python = ""
+        hash_value = ""
         if path_to_binary.endswith(".whl"):
             pkg = distlib.wheel.Wheel(path_to_binary)
             md_dict = pkg.metadata.todict()
             requires_python = md_dict.get("requires_python", "")
             if requires_python.count(".") > 1:
                 requires_python, _, _ = requires_python.rpartition(".")
-        self.releases[release] = ReleaseTuple(path_to_binary, requires_python)
-        self._package_dirs.add(ReleaseTuple(path, requires_python))
+        if os.path.isfile(path_to_binary + ".sha256"):
+            with open(path_to_binary + ".sha256") as f:
+                hash_value = f.read().strip()
+        self.releases[release] = ReleaseTuple(path_to_binary, requires_python, hash_value)
+        self._package_dirs.add(ReleaseTuple(path, requires_python, hash_value))
 
 
 class Artifact(object):
@@ -155,9 +159,9 @@ def prepare_packages(path):
                     packages[package_name] = Package(package_name)
 
                 packages[package_name].add_release(os.path.join(root, file))
-    remaining = get_pypi_package_names() - set(list(packages.keys()))
-    for pypi_pkg in remaining:
-        packages[pypi_pkg] = Package(pypi_pkg)
+    # remaining = get_pypi_package_names() - set(list(packages.keys()))
+    # for pypi_pkg in remaining:
+    #     packages[pypi_pkg] = Package(pypi_pkg)
 
 
 @app.route('/')
diff --git a/tests/pytest-pypi/pytest_pypi/templates/package.html b/tests/pytest-pypi/pytest_pypi/templates/package.html
--- a/tests/pytest-pypi/pytest_pypi/templates/package.html
+++ b/tests/pytest-pypi/pytest_pypi/templates/package.html
@@ -7,7 +7,7 @@
 <body>
 	<h1>Links for {{ package.name }}</h1>
 	{% for release, value in package.releases.items() %}
-		<a href="/{{ package.name }}/{{ release }}"{%- if value.requires_python %} data-requires-python="{{ value.requires_python }}"{% endif %}>{{ release }}</a>
+		<a href="/{{ package.name }}/{{ release }}{%- if value.hash %}#sha256={{ value.hash }}{% endif %}"{%- if value.requires_python %} data-requires-python="{{ value.requires_python }}"{% endif %}>{{ release }}</a>
 		<br>
 	{% endfor %}
 </body>