Repository: pypa/pipenv

Issue:
pipenv install --deploy now fails after having worked for months
### Issue

pipenv install --deploy now fails in my docker build step due to out of date lockfile. 

However whenever I run pipenv lock, it fails to update the hash and instead spits out the original hash that it already was.

### Expected

I expect the dependencies to install, as they have been for almost 6 months with no issue.

### Actual result
```console
✔ Successfully created virtual environment! 
Virtualenv location: /root/.local/share/virtualenvs/-x-v5uFv0
Your Pipfile.lock (ab547e) is out of date. Expected: (9d352e).
Usage: pipenv install [OPTIONS] [PACKAGES]...

ERROR:: Aborting deploy
```
### Steps to replicate

- RUN pipenv install --deploy (in a Docker build step, having copied in the Pipfile and Pipfile.lock)
-------------------------------------------------------------------------------
[output.txt](https://github.com/pypa/pipenv/files/9742751/output.txt)



Discussion:
@Nick-Yazdani You are failing on the hash check -- when is the last time you relocked?  It worked when when I re-locked your Pipfile in my testing.
@matteius I relocked several times but it just kept reverting back to the previous hash that was out of date. I think perhaps there is an issue with my system configurations rather than pipenv. (Perhaps my use of pyenv etc.). Feel free to close the issue.
@Nick-Yazdani Would love to see us figure out the root of the problem -- we can leave it open for now.   Do you have a prior version of pipenv that this works with your existing setup on?   Are you on latest ubuntu?
@matteius  It was working with whatever release the docker build step was pulling prior to today, sorry I just pull the latest in my build step so I'm not exactly sure of the pipenv version. Everything works fine on my local its just the docker build part that is causing issues. I'm on Ubuntu 20.04. 

As a sanity check I will run the build on our GCP pipeline and see if it fails the same way, that way we can narrow it down to my OS?
It failed in the GCP pipeline, and that build had previously succeeded (yesterday). I ran it on the last successful commit, the image it uses is python:3.9-bullseye. Perhaps this only happens when building in a Docker environment?
@Nick-Yazdani The biggest changes between the `2022.9.4` and  `2022.9.10` were in the named package categories PR.   That might explain a possible new hash generation.  When I diff the lock file I generated and the one in your output file, I see the only change is with the markers, which was a few releases ago fixed to always generate the same markers.   Perhaps the markers affected the hash generation.  If it has been several months since locking there are a few reasons that come to mind a new hash maybe generated.

Could you describe more about:
>  I relocked several times but it just kept reverting back to the previous hash that was out of date.

How does it revert back to the prior hash, in what step?

@Nick-Yazdani For debugging purposes, would you be able to change the Pipeline to `pipenv sync` instead of `pipenv install --deploy` which verifies the hash.  I would like to narrow down if its failing on simply the hash validation, or anything additional in the install phase.
I downgraded to the previous release on my local system and now it works. I will however as a courtesy reinstall the newest release and do what you have asked so as to be of help :)
@Nick-Yazdani Can you share part or all of the Dockerfile that builds this?
@matteius I can't get it to break now even on the latest (it installs perfectly and so does sync option), I think the issue was that for some reason when I ran pipenv lock with the newest release it wasn't actually updating the hash as evidenced by the output being the same out of date hash that it began with.

When I downgraded to the previous release and ran pipenv lock, it successfully updated the hash and thus now has resolved the problem.
```console
FROM base AS python-deps

RUN python -m pip install --user pipx && python -m pipx ensurepath

SHELL ["bash", "-lc"] 

RUN pipx install pipenv

COPY Pipfile .
COPY Pipfile.lock .
RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy
```
@Nick-Yazdani Ah wow, thank you for your report -- somehow in your example lock file, re-locking it doesn't update the hash.  This is definitely going to be related to the named categories work, there were a number of places I had to refactor how the lock file was accessed and updated, so I am sure that this is it.  Thanks for your help in pin-pointing the issue.
```
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5392$ pipenv install --deploy
Your Pipfile.lock (ab547e) is out of date. Expected: (9d352e).
Usage: pipenv install [OPTIONS] [PACKAGES]...

ERROR:: Aborting deploy
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5392$ pipenv lock
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (2f38753c2c827a9636d10619569c9aa25970d6fcade1f04d3758edebd0ab547e)!
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5392$ pipenv install --deploy
Your Pipfile.lock (ab547e) is out of date. Expected: (9d352e).
Usage: pipenv install [OPTIONS] [PACKAGES]...

ERROR:: Aborting deploy
```

Resolving patch:
diff --git a/pipenv/core.py b/pipenv/core.py
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -1138,6 +1138,7 @@ def do_lock(
                 overwrite_with_default(lockfile.get("default", {}), lockfile[category])
             )
     if write:
+        lockfile.update({"_meta": project.get_lockfile_meta()})
         project.write_lockfile(lockfile)
         click.echo(
             "{}".format(
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -10,8 +10,11 @@
 import re
 import sys
 import urllib.parse
+from json.decoder import JSONDecodeError
 from pathlib import Path
 
+import click
+
 from pipenv.cmdparse import Script
 from pipenv.environment import Environment
 from pipenv.environments import Setting, is_in_virtualenv, normalize_pipfile_path
@@ -582,8 +585,6 @@ def _lockfile(self, categories=None):
                 )
                 lockfile = lockfile._data
 
-        # with open(self.pipfile_location) as pf:
-        #     lockfile = plette.Lockfile.with_meta_from(plette.Pipfile.load(pf))
         if categories is None:
             categories = self.get_package_categories(for_lockfile=True)
         for category in categories:
@@ -760,7 +761,11 @@ def get_lockfile_meta(self):
         from .vendor.plette.lockfiles import PIPFILE_SPEC_CURRENT
 
         if self.lockfile_exists:
-            sources = self.lockfile_content.get("_meta", {}).get("sources", [])
+            sources = (
+                self.load_lockfile(expand_env_vars=False)
+                .get("_meta", {})
+                .get("sources", [])
+            )
         elif "source" in self.parsed_pipfile:
             sources = [dict(source) for source in self.parsed_pipfile["source"]]
         else:
@@ -994,8 +999,17 @@ def recase_pipfile(self):
     def load_lockfile(self, expand_env_vars=True):
         lockfile_modified = False
         with io.open(self.lockfile_location, encoding="utf-8") as lock:
-            j = json.load(lock)
-            self._lockfile_newlines = preferred_newlines(lock)
+            try:
+                j = json.load(lock)
+                self._lockfile_newlines = preferred_newlines(lock)
+            except JSONDecodeError:
+                click.secho(
+                    "Pipfile.lock is corrupted; ignoring contents.",
+                    fg="yellow",
+                    bold=True,
+                    err=True,
+                )
+                j = {}
         if not j.get("_meta"):
             with open(self.pipfile_location) as pf:
                 default_lockfile = plette.Lockfile.with_meta_from(
diff --git a/pipenv/vendor/plette/pipfiles.py b/pipenv/vendor/plette/pipfiles.py
--- a/pipenv/vendor/plette/pipfiles.py
+++ b/pipenv/vendor/plette/pipfiles.py
@@ -81,7 +81,7 @@ def get_hash(self):
             "develop": self._data.get("dev-packages", {}),
         }
         for category, values in self._data.items():
-            if category in PIPFILE_SECTIONS or category in ("default", "develop"):
+            if category in PIPFILE_SECTIONS or category in ("default", "develop", "pipenv"):
                 continue
             data[category] = values
         content = json.dumps(data, sort_keys=True, separators=(",", ":"))


Test patch: