Repository: pypi/warehouse

Issue:
Wrong key name used for PEP 658 metadata files in the JSON index
**Describe the bug**

[PEP 691](https://peps.python.org/pep-0691/#project-detail) states that the key name for metadata files in the JSON index should be `dist-info-metadata`:

> `dist-info-metadata`: An optional key that indicates that metadata for this file is available, via the same location as specified in [PEP 658](https://peps.python.org/pep-0658) (`{file_url}.metadata`).

However, warehouse is providing it under the `data-dist-info-metadata` key instead:

```
$ curl -H 'Accept: application/vnd.pypi.simple.v1+json' https://pypi.org/simple/fluffy-server/ | jq .files
[...]
  {
    "data-dist-info-metadata": {
      "sha256": "4db99543165cbdeef42ccb6257545911ccd7865d65e304e3e056f383a25f309c"
    },
    "filename": "fluffy_server-1.39.2-py3-none-any.whl",
    [...]
```

This is causing pip to not use the metadata files as it is looking for the `dist-info-metadata` key only:
https://github.com/pypa/pip/blob/f25f8fffbbd16fdb13a4f8977946afe9a3248453/src/pip/_internal/models/link.py#L265


**Additional context**

There are two bugs discovered recently in pip which may make this tricky to fix:

* https://github.com/pypa/pip/issues/12042
* https://github.com/pypa/pip/issues/12038

I believe if we simply fix the key name in pypi.org, it will break existing pip versions as it will cause users to encounter these bugs. It may be necessary to coordinate this fix with fixes to the above bugs in pip to avoid disruption?


Discussion:
Are there versions of pip currently using the incorrect name? This was just shipped last week: https://github.com/pypi/warehouse/pull/13649
I don't think any versions of pip are using the incorrect name. https://github.com/pypa/pip/commit/bad03ef931d9b3ff4f9e75f35f9c41f45839e2a1#diff-3919ca53335487395177edeffff8b60bd360f1ad18c95593287f6814b9ecefb8R249 is the commit which first introduced the support in pip, and it is using `dist-info-metadata` already. (It does reference `data-dist-info-metadata` but that is for the PEP 658 HTML-based index and not the JSON index.)

In my testing I couldn't get pip to use the metadata files at all with pypi.org when using the JSON index. My guess is that the current key is not being widely used at the moment since if it was, other people would probably have encountered the pip bugs I referenced above before I did.
Ah, I understand.

For what its worth, pip is not the only consumer of PEP 658 data via the JSON api.

I suppose we _will_ need to coordinate with the pip team though. @pfmoore do you have thoughts on how you'd anticipate such coordination happening?
See the discussion in https://github.com/pypa/pip/issues/12042

Specifically, as `dist-info-metadata` is optional, I think it should be omitted unless there's an actual file (rather than including it with a value `false`). And the backfill to existing wheels shouldn't be started until we have a fixed pip released, and people have a chance to upgrade.

Resolving patch:
diff --git a/warehouse/packaging/utils.py b/warehouse/packaging/utils.py
--- a/warehouse/packaging/utils.py
+++ b/warehouse/packaging/utils.py
@@ -73,6 +73,9 @@ def _simple_detail(project, request):
                 "data-dist-info-metadata": {"sha256": file.metadata_file_sha256_digest}
                 if file.metadata_file_sha256_digest
                 else False,
+                "core-metadata": {"sha256": file.metadata_file_sha256_digest}
+                if file.metadata_file_sha256_digest
+                else False,
             }
             for file in files
         ],


Test patch:
diff --git a/tests/unit/api/test_simple.py b/tests/unit/api/test_simple.py
--- a/tests/unit/api/test_simple.py
+++ b/tests/unit/api/test_simple.py
@@ -272,6 +272,7 @@ def test_with_files_no_serial(self, db_request, content_type, renderer_override)
                     "size": f.size,
                     "upload-time": f.upload_time.isoformat() + "Z",
                     "data-dist-info-metadata": False,
+                    "core-metadata": False,
                 }
                 for f in files
             ],
@@ -319,6 +320,7 @@ def test_with_files_with_serial(self, db_request, content_type, renderer_overrid
                     "size": f.size,
                     "upload-time": f.upload_time.isoformat() + "Z",
                     "data-dist-info-metadata": False,
+                    "core-metadata": False,
                 }
                 for f in files
             ],
@@ -407,6 +409,9 @@ def test_with_files_with_version_multi_digit(
                     }
                     if f.metadata_file_sha256_digest is not None
                     else False,
+                    "core-metadata": {"sha256": "deadbeefdeadbeefdeadbeefdeadbeef"}
+                    if f.metadata_file_sha256_digest is not None
+                    else False,
                 }
                 for f in files
             ],