Repository: pypa/pipenv

Issue:
Pipenv 2022.10.25 does not appear to correctly prune obsolete dependencies
### Issue description

When updating packages in a Pipfile to versions that vendor away certain dependencies, these dependencies are not pruned from the Pipfile.lock

### Expected result

The Pipfile.lock doesn't contain dependencies that are not listed in the Pipfile and not depended on by any (sub-)dependencies of listed packages, either. Pipenv version 2022.6.7 does this correctly.

### Actual result

Obsoleted package dependencies remain in the Pipfile.lock

### Steps to replicate

We start with a Pipfile containing

```toml
[packages]
pytest = "==7.1.2"
pytest-xdist = "==2.5.0"
```

The lock file generated with those dependencies contains, among others, `py` and `pytest-forked`. After updating the Pipfile's contents to

```toml
[packages]
pytest = "==7.2.0"
pytest-xdist = "==3.0.2"
```

and running `pipenv lock`, the packages mentioned above are still listed in the lock file, despite not being depended on any more by either package (as evidenced by `pipenv graph`).

-------------------------------------------------------------------------------

<details><summary>$ pipenv --support</summary>

Pipenv version: `'2022.10.25'`

Pipenv location: `'/usr/local/lib/python3.10/site-packages/pipenv'`

Python location: `'/usr/local/bin/python'`

OS Name: `'posix'`

User pip version: `'22.3'`

user Python installations found:

  - `3.10.5`: `/usr/local/bin/python`
  - `3.10.5`: `/usr/local/bin/python3`
  - `3.9.2`: `/usr/bin/python3.9`
  - `3.9.2`: `/usr/bin/python3`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.10.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.15.0-10052-tuxedo',
 'platform_system': 'Linux',
 'platform_version': '#58tux1 SMP Thu Oct 20 16:29:39 UTC 2022',
 'python_full_version': '3.10.5',
 'python_version': '3.10',
 'sys_platform': 'linux'}
```

System environment variables:

  - `SHELL`
  - `COLORTERM`
  - `TERM_PROGRAM_VERSION`
  - `HOSTNAME`
  - `PYTHON_VERSION`
  - `SSH_AUTH_SOCK`
  - `REMOTE_CONTAINERS_IPC`
  - `PWD`
  - `PYTHON_SETUPTOOLS_VERSION`
  - `HOME`
  - `LANG`
  - `REMOTE_CONTAINERS`
  - `GPG_KEY`
  - `PYTHONPATH`
  - `TERM`
  - `REMOTE_CONTAINERS_SOCKETS`
  - `SHLVL`
  - `PYTHON_PIP_VERSION`
  - `PYTHON_GET_PIP_SHA256`
  - `PYTHON_GET_PIP_URL`
  - `BROWSER`
  - `PATH`
  - `PIP_NO_CACHE_DIR`
  - `TERM_PROGRAM`
  - `VSCODE_IPC_HOOK_CLI`
  - `_`
  - `PIP_DISABLE_PIP_VERSION_CHECK`
  - `PIP_PYTHON_PATH`
  - `PYTHONDONTWRITEBYTECODE`
  - `PYTHONFINDER_IGNORE_UNSUPPORTED`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/vscode/vscode-server/bin/linux-x64/8fa188b2b301d36553cbc9ce1b0a146ccb93351f/bin/remote-cli:/root/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`
  - `SHELL`: `/bin/bash`
  - `LANG`: `C.UTF-8`
  - `PWD`: `/workspaces/pipenv-break`


---------------------------

Contents of `Pipfile` ('/workspaces/pipenv-break/Pipfile'):

```toml
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pytest = "==7.2.0"
pytest-xdist = "==3.0.2"

[requires]
python_version = "3.10"

[pipenv]
allow_prereleases = true

```


Contents of `Pipfile.lock` ('/workspaces/pipenv-break/Pipfile.lock'):

```json
{
    "_meta": {
        "hash": {
            "sha256": "513621fbfbc31236089e939f4c8c8079530cf7a18514d593e9cfece60d5c947f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "attrs": {
            "hashes": [
                "sha256:29adc2665447e5191d0e7c568fde78b21f9672d344281d0c6e1ab085429b22b6",
                "sha256:86efa402f67bf2df34f51a335487cf46b1ec130d02b8d39fd248abfd30da551c"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==22.1.0"
        },
        "exceptiongroup": {
            "hashes": [
                "sha256:4d6c0aa6dd825810941c792f53d7b8d71da26f5e5f84f20f9508e8f2d33b140a",
                "sha256:73866f7f842ede6cb1daa42c4af078e2035e5f7607f0e2c762cc51bb31bbe7b2"
            ],
            "markers": "python_version < '3.11'",
            "version": "==1.0.1"
        },
        "execnet": {
            "hashes": [
                "sha256:8f694f3ba9cc92cab508b152dcfe322153975c29bda272e2fd7f3f00f36e47c5",
                "sha256:a295f7cc774947aac58dde7fdc85f4aa00c42adf5d8f5468fc630c1acf30a142"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
            "version": "==1.9.0"
        },
        "iniconfig": {
            "hashes": [
                "sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3",
                "sha256:bc3af051d7d14b2ee5ef9969666def0cd1a000e121eaea580d4a313df4b37f32"
            ],
            "version": "==1.1.1"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==21.3"
        },
        "pluggy": {
            "hashes": [
                "sha256:4224373bacce55f955a878bf9cfa763c1e360858e330072059e10bad68531159",
                "sha256:74134bbf457f031a36d68416e1509f34bd5ccc019f0bcc952c7b909d06b37bd3"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==1.0.0"
        },
        "py": {
            "hashes": [
                "sha256:51c75c4126074b472f746a24399ad32f6053d1b34b68d2fa41e558e6f4a98719",
                "sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
            "version": "==1.11.0"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        },
        "pytest": {
            "hashes": [
                "sha256:892f933d339f068883b6fd5a459f03d85bfcb355e4981e146d2c7616c21fef71",
                "sha256:c4014eb40e10f11f355ad4e3c2fb2c6c6d1919c73f3b5a433de4708202cade59"
            ],
            "index": "pypi",
            "version": "==7.2.0"
        },
        "pytest-forked": {
            "hashes": [
                "sha256:8b67587c8f98cbbadfdd804539ed5455b6ed03802203485dd2f53c1422d7440e",
                "sha256:bbbb6717efc886b9d64537b41fb1497cfaf3c9601276be8da2cccfea5a3c8ad8"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==1.4.0"
        },
        "pytest-xdist": {
            "hashes": [
                "sha256:688da9b814370e891ba5de650c9327d1a9d861721a524eb917e620eec3e90291",
                "sha256:9feb9a18e1790696ea23e1434fa73b325ed4998b0e9fcb221f16fd1945e6df1b"
            ],
            "index": "pypi",
            "version": "==3.0.2"
        },
        "tomli": {
            "hashes": [
                "sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc",
                "sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f"
            ],
            "markers": "python_version < '3.11'",
            "version": "==2.0.1"
        }
    },
    "develop": {}
}

```
</details>



Discussion:
Thanks for the report @bootrecords , I was able to reproduce it.

Resolving patch:
diff --git a/pipenv/core.py b/pipenv/core.py
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -1157,6 +1157,13 @@ def do_lock(
                 err=True,
             )
 
+        # Prune old lockfile category as new one will be created.
+        if not keep_outdated:
+            try:
+                del lockfile[category]
+            except KeyError:
+                pass
+
         from pipenv.utils.resolver import venv_resolve_deps
 
         # Mutates the lockfile
@@ -2026,7 +2033,7 @@ def do_outdated(project, pypi_mirror=None, pre=False, clear=False):
         project, clear=clear, pre=pre, write=False, pypi_mirror=pypi_mirror
     )
     for category in project.get_package_categories(for_lockfile=True):
-        for package in lockfile[category]:
+        for package in lockfile.get(category, []):
             try:
                 updated_packages[package] = lockfile[category][package]["version"]
             except KeyError:


Test patch:
diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py
--- a/tests/integration/conftest.py
+++ b/tests/integration/conftest.py
@@ -219,11 +219,11 @@ def __init__(self, path):
         self.document["source"] = self.document.get("source", tomlkit.aot())
         self.document["requires"] = self.document.get("requires", tomlkit.table())
         self.document["packages"] = self.document.get("packages", tomlkit.table())
-        self.document["dev_packages"] = self.document.get("dev_packages", tomlkit.table())
+        self.document["dev-packages"] = self.document.get("dev-packages", tomlkit.table())
         super().__init__()
 
     def install(self, package, value, dev=False):
-        section = "packages" if not dev else "dev_packages"
+        section = "packages" if not dev else "dev-packages"
         if isinstance(value, dict):
             table = tomlkit.inline_table()
             table.update(value)
@@ -233,10 +233,10 @@ def install(self, package, value, dev=False):
         self.write()
 
     def remove(self, package, dev=False):
-        section = "packages" if not dev else "dev_packages"
+        section = "packages" if not dev else "dev-packages"
         if not dev and package not in self.document[section]:
-            if package in self.document["dev_packages"]:
-                section = "dev_packages"
+            if package in self.document["dev-packages"]:
+                section = "dev-packages"
         del self.document[section][package]
         self.write()
 
diff --git a/tests/integration/test_uninstall.py b/tests/integration/test_uninstall.py
--- a/tests/integration/test_uninstall.py
+++ b/tests/integration/test_uninstall.py
@@ -252,5 +252,5 @@ def test_uninstall_multiple_categories(pipenv_instance_private_pypi):
         c = p.pipenv('uninstall six --categories="prereq after"')
         assert c.returncode == 0
 
-        assert "six" not in p.lockfile["prereq"]
+        assert "six" not in p.lockfile.get("prereq", {})
         assert "six" not in p.lockfile["default"]