Repository: pypi/warehouse

Issue:
Support Metadata Version 2.2
**What's the problem this feature will solve?**
Since PEP 643 is accepted, PyPI should start accepting Metadata 2.2 distributions. See  PyO3/maturin#564.

**Describe the solution you'd like**
Allow `Metadata-Version: 2.2` to pass the legacy upload endpoint’s form validation, and validate the corresponding `Dynamic` field.

The latter part could be a bit tricky since `Dynamic` is not allowed for wheels. Not sure how it could be best handled (I don’t have much experience with WTForm).

**Additional context**
https://www.python.org/dev/peps/pep-0643/

Add support for PEP 685 (Metadata Version 2.3)
**What's the problem this feature will solve?**
[PEP 685](https://peps.python.org/pep-0685/) specifies rules for normalisation of extra names in core metadata, and in turn bumps the metadata version to 2.3. Currently it is not possible to upload wheels with metadata 2.3 because warehouse rejects them as having an unsupported metadata version.

**Describe the solution you'd like**
The PEP suggests tools consuming metadata (like warehouse) emit warnings if invalid extra names are found for *older* metadata versions, although the tool may raise an error. I assume there's no mechanism in the upload API to return a warning, only an error? I don't think an error is appropriate here as it will likely break uploads of packages with previously valid metadata.

Tools should also warn (but may error) for invalid extras with metadata version 2.3. 

Therefore I propose leaving the validation unchanged for older versions, but returning an error if the metadata version is 2.3 (or later) and an extra name is invalid.

**Additional context**
<!-- Add any other context, links, etc. about the feature here. -->

I have an initial implementation here: https://github.com/pypa/warehouse/pull/11380/commits/48b5063a65a382d3bba622351858aea9c058f6c0

Add support for metadata 2.2 and 2.3
Fixes #9660 and #11526

I couldn't find any existing tests for the `Metadata-Version` validation so I haven't added any yet.
There are tests for the new validation functions for `Dynamic` and `Provides-Extra` though.


Discussion:
There's a couple levels of 'validation' that could happen for the `Dynamic` field here:

1. Validate that `Name` and `Version` are not marked as `Dynamic` (feasible)
2. Validate that wheels do not set `Dynamic` (probably feasible)
3. Validation across multiple artifacts (probably infeasible):
    - If a field is not marked as `Dynamic`, then the value of the field in any wheel built from the sdist MUST match the value in the sdist. If the field is not in the sdist, and not marked as `Dynamic`, then it MUST NOT be present in the wheel.
    - If a field is marked as `Dynamic`, it may contain any valid value in a wheel built from the sdist (including not being present at all).
    - Backends MUST NOT mark a field as `Dynamic` if they can determine that it was generated from data that will not change at build time.

Are we OK with only doing 1 & 2? (cc @pfmoore)
From my perspective just allowing 2.2 when no `Dynamic` is used would already be great and solve https://github.com/PyO3/maturin/pull/564
I think (1) is reasonable. The PEP allows wheels to set `Dynamic`, so I think (2) is wrong, and should be omitted. I'm happy that Warehouse shouldn't try to do (3).
> The PEP allows wheels to set Dynamic, so I think (2) is wrong, and should be omitted.

Got it. I was taking this from the original issue and thought I was missing something in the PEP. I'll update the OP accordingly.

If all we need to do is 1) then this is straightforward.
What's the next step for this? Is it just waiting on a PR?
Yes.

Any update on this?
@domdfcoding gentle ping, are you still interested in working on this?
I'm trying to revive this in #13606

Resolving patch:
diff --git a/warehouse/forklift/legacy.py b/warehouse/forklift/legacy.py
--- a/warehouse/forklift/legacy.py
+++ b/warehouse/forklift/legacy.py
@@ -59,6 +59,7 @@
     Dependency,
     DependencyKind,
     Description,
+    DynamicFieldsEnum,
     File,
     Filename,
     JournalEntry,
@@ -395,6 +396,37 @@ def _validate_classifiers(form, field):
             )
 
 
+def _validate_dynamic(_form, field):
+    declared_dynamic_fields = {str.title(k) for k in field.data or []}
+    disallowed_dynamic_fields = {"Name", "Version", "Metadata-Version"}
+    if invalid := (declared_dynamic_fields & disallowed_dynamic_fields):
+        raise wtforms.validators.ValidationError(
+            f"The following metadata field(s) are valid, "
+            f"but cannot be marked as dynamic: {invalid!r}",
+        )
+    allowed_dynamic_fields = set(DynamicFieldsEnum.enums)
+    if invalid := (declared_dynamic_fields - allowed_dynamic_fields):
+        raise wtforms.validators.ValidationError(
+            f"The following metadata field(s) are not valid "
+            f"and cannot be marked as dynamic: {invalid!r}"
+        )
+
+
+_extra_name_re = re.compile("^([a-z0-9]|[a-z0-9]([a-z0-9-](?!--))*[a-z0-9])$")
+
+
+def _validate_provides_extras(form, field):
+    metadata_version = packaging.version.Version(form.metadata_version.data)
+
+    if metadata_version >= packaging.version.Version("2.3"):
+        if invalid := [
+            name for name in field.data or [] if not _extra_name_re.match(name)
+        ]:
+            raise wtforms.validators.ValidationError(
+                f"The following Provides-Extra value(s) are invalid: {invalid!r}"
+            )
+
+
 def _construct_dependencies(form, types):
     for name, kind in types.items():
         for item in getattr(form, name).data:
@@ -419,7 +451,7 @@ class MetadataForm(forms.Form):
                 # Note: This isn't really Metadata 2.0, however bdist_wheel
                 #       claims it is producing a Metadata 2.0 metadata when in
                 #       reality it's more like 1.2 with some extensions.
-                ["1.0", "1.1", "1.2", "2.0", "2.1"],
+                ["1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3"],
                 message="Use a known metadata version.",
             ),
         ],
@@ -470,6 +502,9 @@ class MetadataForm(forms.Form):
     author = wtforms.StringField(
         description="Author", validators=[wtforms.validators.Optional()]
     )
+    supported_platform = wtforms.StringField(
+        description="Supported-Platform", validators=[wtforms.validators.Optional()]
+    )
     description_content_type = wtforms.StringField(
         description="Description-Content-Type",
         validators=[wtforms.validators.Optional(), _validate_description_content_type],
@@ -495,6 +530,10 @@ class MetadataForm(forms.Form):
         description="Classifier",
         validators=[_validate_no_deprecated_classifiers, _validate_classifiers],
     )
+    dynamic = ListField(
+        description="Dynamic",
+        validators=[_validate_dynamic],
+    )
     platform = wtforms.StringField(
         description="Platform", validators=[wtforms.validators.Optional()]
     )
@@ -564,6 +603,10 @@ class MetadataForm(forms.Form):
         description="Requires-Dist",
         validators=[wtforms.validators.Optional(), _validate_legacy_dist_req_list],
     )
+    provides_extra = ListField(
+        description="Provides-Extra",
+        validators=[wtforms.validators.Optional(), _validate_provides_extras],
+    )
     provides_dist = ListField(
         description="Provides-Dist",
         validators=[wtforms.validators.Optional(), _validate_legacy_dist_req_list],
@@ -613,6 +656,15 @@ def full_validate(self):
                 "Include at least one message digest."
             )
 
+        # Dynamic is only allowed with metadata version 2.2+
+        if self.dynamic.data:
+            metadata_version = packaging.version.Version(self.metadata_version.data)
+            if metadata_version and metadata_version < packaging.version.Version("2.2"):
+                raise wtforms.validators.ValidationError(
+                    "'Dynamic' is only allowed in metadata version 2.2 and higher, "
+                    f"but you declared {self.metadata_version.data}"
+                )
+
 
 def _validate_filename(filename, filetype):
     # Our object storage does not tolerate some specific characters
@@ -1107,6 +1159,8 @@ def file_upload(request):
                     "home_page",
                     "download_url",
                     "requires_python",
+                    "dynamic",
+                    "provides_extra",
                 }
             },
             uploader=request.user if request.user else None,
diff --git a/warehouse/legacy/api/json.py b/warehouse/legacy/api/json.py
--- a/warehouse/legacy/api/json.py
+++ b/warehouse/legacy/api/json.py
@@ -155,12 +155,16 @@ def _json_data(request, project, release, *, all_releases):
             "requires_dist": (
                 list(release.requires_dist) if release.requires_dist else None
             ),
+            "provides_extra": (
+                list(release.provides_extra) if release.provides_extra else None
+            ),
             "docs_url": project.documentation_url,
             "bugtrack_url": None,
             "home_page": release.home_page,
             "download_url": release.download_url,
             "yanked": release.yanked,
             "yanked_reason": release.yanked_reason or None,
+            "dynamic": list(release.dynamic) if release.dynamic else None,
         },
         "urls": releases[release.version],
         "vulnerabilities": vulnerabilities,
diff --git a/warehouse/packaging/models.py b/warehouse/packaging/models.py
--- a/warehouse/packaging/models.py
+++ b/warehouse/packaging/models.py
@@ -25,6 +25,7 @@
 from sqlalchemy import (
     BigInteger,
     CheckConstraint,
+    Column,
     FetchedValue,
     ForeignKey,
     Index,
@@ -36,7 +37,7 @@
     orm,
     sql,
 )
-from sqlalchemy.dialects.postgresql import CITEXT, UUID as PG_UUID
+from sqlalchemy.dialects.postgresql import ARRAY, CITEXT, ENUM, UUID as PG_UUID
 from sqlalchemy.exc import MultipleResultsFound, NoResultFound
 from sqlalchemy.ext.associationproxy import association_proxy
 from sqlalchemy.ext.hybrid import hybrid_property
@@ -465,6 +466,32 @@ class ReleaseURL(db.Model):
     url: Mapped[str]
 
 
+DynamicFieldsEnum = ENUM(
+    "Platform",
+    "Supported-Platform",
+    "Summary",
+    "Description",
+    "Description-Content-Type",
+    "Keywords",
+    "Home-page",
+    "Download-URL",
+    "Author",
+    "Author-email",
+    "Maintainer",
+    "Maintainer-email",
+    "License",
+    "Classifier",
+    "Requires-Dist",
+    "Requires-Python",
+    "Requires-External",
+    "Project-URL",
+    "Provides-Extra",
+    "Provides-Dist",
+    "Obsoletes-Dist",
+    name="release_dynamic_fields",
+)
+
+
 class Release(HasObservations, db.Model):
     __tablename__ = "releases"
 
@@ -517,6 +544,12 @@ def __table_args__(cls):  # noqa
 
     yanked_reason: Mapped[str] = mapped_column(server_default="")
 
+    dynamic = Column(  # type: ignore[var-annotated]
+        ARRAY(DynamicFieldsEnum),
+        nullable=True,
+        comment="Array of metadata fields marked as Dynamic (PEP 643/Metadata 2.2)",
+    )
+
     _classifiers: Mapped[list[Classifier]] = orm.relationship(
         secondary="release_classifiers",
         order_by=Classifier.ordering,
@@ -569,6 +602,12 @@ def __table_args__(cls):  # noqa
     _provides_dist = _dependency_relation(DependencyKind.provides_dist)
     provides_dist = association_proxy("_provides_dist", "specifier")
 
+    provides_extra = Column(  # type: ignore[var-annotated]
+        ARRAY(Text),
+        nullable=True,
+        comment="Array of extra names (PEP 566/685|Metadata 2.1/2.3)",
+    )
+
     _obsoletes_dist = _dependency_relation(DependencyKind.obsoletes_dist)
     obsoletes_dist = association_proxy("_obsoletes_dist", "specifier")
 


Test patch:
diff --git a/tests/unit/forklift/test_legacy.py b/tests/unit/forklift/test_legacy.py
--- a/tests/unit/forklift/test_legacy.py
+++ b/tests/unit/forklift/test_legacy.py
@@ -388,6 +388,72 @@ def test_validate_classifiers_invalid(self, db_request, data):
         with pytest.raises(ValidationError):
             legacy._validate_classifiers(form, field)
 
+    @pytest.mark.parametrize(
+        "data", [["Requires-Dist"], ["Requires-Dist", "Requires-Python"]]
+    )
+    def test_validate_dynamic_valid(self, db_request, data):
+        form = pretend.stub()
+        field = pretend.stub(data=data)
+
+        legacy._validate_dynamic(form, field)
+
+    @pytest.mark.parametrize(
+        "data",
+        [
+            ["Version"],
+            ["Name"],
+            ["Version", "Name"],
+            ["Provides-Extra", "I-Am-Not-Metadata"],
+        ],
+    )
+    def test_validate_dynamic_invalid(self, db_request, data):
+        form = pretend.stub()
+        field = pretend.stub(data=data)
+
+        with pytest.raises(ValidationError):
+            legacy._validate_dynamic(form, field)
+
+    @pytest.mark.parametrize("data", [["dev"], ["dev-test"]])
+    def test_validate_provides_extras_valid(self, db_request, data):
+        form = pretend.stub(
+            provides_extra=pretend.stub(data=data),
+            metadata_version=pretend.stub(data="2.3"),
+        )
+        field = pretend.stub(data=data)
+
+        legacy._validate_provides_extras(form, field)
+
+    @pytest.mark.parametrize("data", [["dev_test"], ["dev.lint", "dev--test"]])
+    def test_validate_provides_extras_invalid(self, db_request, data):
+        form = pretend.stub(
+            provides_extra=pretend.stub(data=data),
+            metadata_version=pretend.stub(data="2.3"),
+        )
+        field = pretend.stub(data=data)
+
+        with pytest.raises(ValidationError):
+            legacy._validate_provides_extras(form, field)
+
+    @pytest.mark.parametrize("data", [["dev"], ["dev-test"]])
+    def test_validate_provides_extras_valid_2_2(self, db_request, data):
+        form = pretend.stub(
+            provides_extra=pretend.stub(data=data),
+            metadata_version=pretend.stub(data="2.2"),
+        )
+        field = pretend.stub(data=data)
+
+        legacy._validate_provides_extras(form, field)
+
+    @pytest.mark.parametrize("data", [["dev_test"], ["dev.lint", "dev--test"]])
+    def test_validate_provides_extras_invalid_2_2(self, db_request, data):
+        form = pretend.stub(
+            provides_extra=pretend.stub(data=data),
+            metadata_version=pretend.stub(data="2.2"),
+        )
+        field = pretend.stub(data=data)
+
+        legacy._validate_provides_extras(form, field)
+
 
 def test_construct_dependencies():
     types = {"requires": DependencyKind.requires, "provides": DependencyKind.provides}
@@ -478,6 +544,26 @@ def test_requires_python(self):
         form = legacy.MetadataForm(MultiDict({"requires_python": ">= 3.5"}))
         form.requires_python.validate(form)
 
+    @pytest.mark.parametrize(
+        "data",
+        [
+            {
+                "filetype": "bdist_wheel",
+                "metadata_version": "2.1",
+                "dynamic": "requires",
+            },
+            {
+                "metadata_version": "1.2",
+                "sha256_digest": "dummy",
+                "dynamic": "requires",
+            },
+        ],
+    )
+    def test_dynamic_wrong_metadata_version(self, data):
+        form = legacy.MetadataForm(MultiDict(data))
+        with pytest.raises(ValidationError):
+            form.full_validate()
+
 
 class TestFileValidation:
     def test_defaults_to_true(self):
@@ -3417,6 +3503,121 @@ def test_upload_succeeds_creates_release(
             ),
         ]
 
+    @pytest.mark.parametrize(
+        "version, expected_version",
+        [
+            ("1.0", "1.0"),
+            ("v1.0", "1.0"),
+        ],
+    )
+    def test_upload_succeeds_creates_release_metadata_2_3(
+        self, pyramid_config, db_request, metrics, version, expected_version
+    ):
+        user = UserFactory.create()
+        EmailFactory.create(user=user)
+        project = ProjectFactory.create()
+        RoleFactory.create(user=user, project=project)
+
+        db_request.db.add(Classifier(classifier="Environment :: Other Environment"))
+        db_request.db.add(Classifier(classifier="Programming Language :: Python"))
+
+        filename = "{}-{}.tar.gz".format(project.name, "1.0")
+
+        pyramid_config.testing_securitypolicy(identity=user)
+        db_request.user = user
+        db_request.user_agent = "warehouse-tests/6.6.6"
+        db_request.POST = MultiDict(
+            {
+                "metadata_version": "2.3",
+                "name": project.name,
+                "version": version,
+                "summary": "This is my summary!",
+                "filetype": "sdist",
+                "md5_digest": _TAR_GZ_PKG_MD5,
+                "content": pretend.stub(
+                    filename=filename,
+                    file=io.BytesIO(_TAR_GZ_PKG_TESTDATA),
+                    type="application/tar",
+                ),
+                "supported_platform": "i386-win32-2791",
+            }
+        )
+        db_request.POST.extend(
+            [
+                ("classifiers", "Environment :: Other Environment"),
+                ("classifiers", "Programming Language :: Python"),
+                ("requires_dist", "foo"),
+                ("requires_dist", "bar (>1.0)"),
+                ("project_urls", "Test, https://example.com/"),
+                ("requires_external", "Cheese (>1.0)"),
+                ("provides_extra", "testing"),
+                ("provides_extra", "plugin"),
+                ("dynamic", "Supported-Platform"),
+            ]
+        )
+
+        storage_service = pretend.stub(store=lambda path, filepath, meta: None)
+        db_request.find_service = lambda svc, name=None, context=None: {
+            IFileStorage: storage_service,
+            IMetricsService: metrics,
+        }.get(svc)
+
+        resp = legacy.file_upload(db_request)
+
+        assert resp.status_code == 200
+
+        # Ensure that a Release object has been created.
+        release = (
+            db_request.db.query(Release)
+            .filter(
+                (Release.project == project) & (Release.version == expected_version)
+            )
+            .one()
+        )
+        assert release.summary == "This is my summary!"
+        assert release.classifiers == [
+            "Environment :: Other Environment",
+            "Programming Language :: Python",
+        ]
+        assert set(release.requires_dist) == {"foo", "bar (>1.0)"}
+        assert release.project_urls == {"Test": "https://example.com/"}
+        assert set(release.requires_external) == {"Cheese (>1.0)"}
+        assert release.version == expected_version
+        assert release.canonical_version == "1"
+        assert release.uploaded_via == "warehouse-tests/6.6.6"
+        assert set(release.provides_extra) == {"testing", "plugin"}
+        assert set(release.dynamic) == {"Supported-Platform"}
+
+        # Ensure that a File object has been created.
+        db_request.db.query(File).filter(
+            (File.release == release) & (File.filename == filename)
+        ).one()
+
+        # Ensure that a Filename object has been created.
+        db_request.db.query(Filename).filter(Filename.filename == filename).one()
+
+        # Ensure that all of our journal entries have been created
+        journals = (
+            db_request.db.query(JournalEntry)
+            .options(joinedload(JournalEntry.submitted_by))
+            .order_by("submitted_date", "id")
+            .all()
+        )
+        assert [(j.name, j.version, j.action, j.submitted_by) for j in journals] == [
+            (
+                release.project.name,
+                release.version,
+                "new release",
+                user,
+            ),
+            (
+                release.project.name,
+                release.version,
+                f"add source file {filename}",
+                user,
+            ),
+        ]
+
     def test_all_valid_classifiers_can_be_created(self, db_request):
         for classifier in classifiers:
             db_request.db.add(Classifier(classifier=classifier))
diff --git a/tests/unit/legacy/api/test_json.py b/tests/unit/legacy/api/test_json.py
--- a/tests/unit/legacy/api/test_json.py
+++ b/tests/unit/legacy/api/test_json.py
@@ -232,6 +232,7 @@ def test_renders(self, pyramid_config, db_request, db_session):
                 "docs_url": "/the/fake/url/",
                 "download_url": None,
                 "downloads": {"last_day": -1, "last_week": -1, "last_month": -1},
+                "dynamic": None,
                 "home_page": None,
                 "keywords": None,
                 "license": None,
@@ -242,6 +243,7 @@ def test_renders(self, pyramid_config, db_request, db_session):
                 "platform": None,
                 "project_url": "/the/fake/url/",
                 "project_urls": expected_urls,
+                "provides_extra": None,
                 "release_url": "/the/fake/url/",
                 "requires_dist": None,
                 "requires_python": None,
@@ -483,6 +485,8 @@ def test_detail_renders(self, pyramid_config, db_request, db_session):
                 description=DescriptionFactory.create(
                     content_type=description_content_type
                 ),
+                dynamic=["Platform", "Supported-Platform"],
+                provides_extra=["testing", "plugin"],
             )
         ]
 
@@ -540,6 +544,7 @@ def test_detail_renders(self, pyramid_config, db_request, db_session):
                 "docs_url": "/the/fake/url/",
                 "download_url": None,
                 "downloads": {"last_day": -1, "last_week": -1, "last_month": -1},
+                "dynamic": ["Platform", "Supported-Platform"],
                 "home_page": None,
                 "keywords": None,
                 "license": None,
@@ -550,6 +555,7 @@ def test_detail_renders(self, pyramid_config, db_request, db_session):
                 "platform": None,
                 "project_url": "/the/fake/url/",
                 "project_urls": expected_urls,
+                "provides_extra": ["testing", "plugin"],
                 "release_url": "/the/fake/url/",
                 "requires_dist": None,
                 "requires_python": None,
@@ -630,6 +636,7 @@ def test_minimal_renders(self, pyramid_config, db_request):
                 "docs_url": None,
                 "download_url": None,
                 "downloads": {"last_day": -1, "last_week": -1, "last_month": -1},
+                "dynamic": None,
                 "home_page": None,
                 "keywords": None,
                 "license": None,
@@ -640,6 +647,7 @@ def test_minimal_renders(self, pyramid_config, db_request):
                 "platform": None,
                 "project_url": "/the/fake/url/",
                 "project_urls": None,
+                "provides_extra": None,
                 "release_url": "/the/fake/url/",
                 "requires_dist": None,
                 "requires_python": None,