Repository: pypa/pip
Pull request #11095 (2022-05-07 10:11:05)

Add json_metadata property to BaseDistribution
As suggested by @pfmoore in https://github.com/pypa/pip/pull/10771#issuecomment-1113971667, this copies the `msg_to_json` function from his `pkg_metadata` project.

This function will be used in `pip install --report` (and possibly `pip list --format=json`).

Towards #53 and [#10771](https://github.com/pypa/pip/pull/10771#issuecomment-1114585840).


pfmoore: How important is it that `msg_to_json` takes a `Message` argument? I'm considering changing it in `pkg_metadata` to take a `bytes` object (and hence hiding the dependency on `email.message`) as that makes the normal usage (read a metadata file) marginally simpler. But if conversion from `Message` format is useful, I'll leave that available as well.
sbidoul: @pfmoore In this case, it was convenient to have it since `BaseDistribution` exposes metadata as an `email.message.Message`. If `pkg_metada` would not have had that capability I'd have needed to refactor more.

So from this unique sample, I say this is useful. In other places where I've had to handle package metadata, email.message.Message is also involved, so it is certainly a convenient intermediate representation.

That said, exposing functions that handle metadata file encodings properly in `pkg_metadata` would certainly be useful too as I remember it is a topic that is not obvious in the PEP texts.

pfmoore: Sounds good. I can expose `msg_to_json` and a thin wrapper for `bytes_to_json` easily enough.
sbidoul: Hi @uranusjr, thanks for the review!

Actually this json conversion function is copied verbatim from @pfmoore 's [pkg_metadata](https://github.com/pfmoore/pkg_metadata), as the goal is to vendor it when Paul says it is good enough for that. That is also the reason why there are no tests for it here. See also [comment](https://github.com/pypa/pip/pull/10771#issuecomment-1113971667)

The `BaseDistribution.json_metadata` property is the way I chose to expose it to pip internals and the rest is an implementation detail that will change as soon as we vendor `pkg_metadata`. What I could do is rename `json.py` to `_json.py` to make it more explicit that it is "private".

I felt a property was appropriate since there was already a `metadata` property returning a `Message`. 

I agree `json_metadata` is slightly misleading as the value is a dict and not a json string or bytes. But TBH I have no better idea that would also express the fact that this dictionary is json-serializable according to the transformation specified in PEP 566. Perhaps `json_metadata_dict` would be better ?

sbidoul: @uranusjr @pfmoore how do you think we should move forward with this one ?
pfmoore: I’ll probably change the name to `msg_to_dict` in `pkg_metadata`. I don’t really have a strong opinion on the name to use here. I’m fine with keeping `json_metadata`, but I’d also be ok with something like `metadata_dict`. Whatever we choose, we can easily change it later, after all… 🤷 

Patch:
diff --git a/src/pip/_internal/metadata/_json.py b/src/pip/_internal/metadata/_json.py
new file mode 100644
index 000000000..336b52f1e
--- /dev/null
+++ b/src/pip/_internal/metadata/_json.py
@@ -0,0 +1,84 @@
+# Extracted from https://github.com/pfmoore/pkg_metadata
+
+from email.header import Header, decode_header, make_header
+from email.message import Message
+from typing import Any, Dict, List, Union
+
+METADATA_FIELDS = [
+    # Name, Multiple-Use
+    ("Metadata-Version", False),
+    ("Name", False),
+    ("Version", False),
+    ("Dynamic", True),
+    ("Platform", True),
+    ("Supported-Platform", True),
+    ("Summary", False),
+    ("Description", False),
+    ("Description-Content-Type", False),
+    ("Keywords", False),
+    ("Home-page", False),
+    ("Download-URL", False),
+    ("Author", False),
+    ("Author-email", False),
+    ("Maintainer", False),
+    ("Maintainer-email", False),
+    ("License", False),
+    ("Classifier", True),
+    ("Requires-Dist", True),
+    ("Requires-Python", False),
+    ("Requires-External", True),
+    ("Project-URL", True),
+    ("Provides-Extra", True),
+    ("Provides-Dist", True),
+    ("Obsoletes-Dist", True),
+]
+
+
+def json_name(field: str) -> str:
+    return field.lower().replace("-", "_")
+
+
+def msg_to_json(msg: Message) -> Dict[str, Any]:
+    """Convert a Message object into a JSON-compatible dictionary."""
+
+    def sanitise_header(h: Union[Header, str]) -> str:
+        if isinstance(h, Header):
+            chunks = []
+            for bytes, encoding in decode_header(h):
+                if encoding == "unknown-8bit":
+                    try:
+                        # See if UTF-8 works
+                        bytes.decode("utf-8")
+                        encoding = "utf-8"
+                    except UnicodeDecodeError:
+                        # If not, latin1 at least won't fail
+                        encoding = "latin1"
+                chunks.append((bytes, encoding))
+            return str(make_header(chunks))
+        return str(h)
+
+    result = {}
+    for field, multi in METADATA_FIELDS:
+        if field not in msg:
+            continue
+        key = json_name(field)
+        if multi:
+            value: Union[str, List[str]] = [
+                sanitise_header(v) for v in msg.get_all(field)
+            ]
+        else:
+            value = sanitise_header(msg.get(field))
+            if key == "keywords":
+                # Accept both comma-separated and space-separated
+                # forms, for better compatibility with old data.
+                if "," in value:
+                    value = [v.strip() for v in value.split(",")]
+                else:
+                    value = value.split()
+        result[key] = value
+
+    payload = msg.get_payload()
+    if payload:
+        result["description"] = payload
+
+    return result
diff --git a/src/pip/_internal/metadata/base.py b/src/pip/_internal/metadata/base.py
index d226dec8b..2823b6754 100644
--- a/src/pip/_internal/metadata/base.py
+++ b/src/pip/_internal/metadata/base.py
@@ -9,8 +9,10 @@ import zipfile
 from typing import (
     IO,
     TYPE_CHECKING,
+    Any,
     Collection,
     Container,
+    Dict,
     Iterable,
     Iterator,
     List,
@@ -38,6 +40,8 @@ from pip._internal.utils.misc import is_local, normalize_path
 from pip._internal.utils.packaging import safe_extra
 from pip._internal.utils.urls import url_to_path
 
+from ._json import msg_to_json
+
 if TYPE_CHECKING:
     from typing import Protocol
 else:
@@ -379,6 +383,17 @@ class BaseDistribution(Protocol):
         """
         return self._metadata_cached()
 
+    @property
+    def metadata_dict(self) -> Dict[str, Any]:
+        """PEP 566 compliant JSON-serializable representation of METADATA or PKG-INFO.
+
+        This should return an empty dict if the metadata file is unavailable.
+
+        :raises NoneMetadataError: If the metadata file is available, but does
+            not contain valid metadata.
+        """
+        return msg_to_json(self.metadata)
+
     @property
     def metadata_version(self) -> Optional[str]:
         """Value of "Metadata-Version:" in distribution metadata, if available."""


Test patch:
diff --git a/tests/unit/metadata/test_metadata.py b/tests/unit/metadata/test_metadata.py
index 57e09acf7..0e8f3ffc1 100644
--- a/tests/unit/metadata/test_metadata.py
+++ b/tests/unit/metadata/test_metadata.py
@@ -6,8 +6,14 @@ from unittest import mock
 import pytest
 from pip._vendor.packaging.utils import NormalizedName
 
-from pip._internal.metadata import BaseDistribution, get_directory_distribution
+from pip._internal.metadata import (
+    BaseDistribution,
+    get_directory_distribution,
+    get_wheel_distribution,
+)
+from pip._internal.metadata.base import FilesystemWheel
 from pip._internal.models.direct_url import DIRECT_URL_METADATA_NAME, ArchiveInfo
+from tests.lib.wheel import make_wheel
 
 
 @mock.patch.object(BaseDistribution, "read_text", side_effect=FileNotFoundError)
@@ -82,3 +88,17 @@ def test_dist_get_direct_url_valid_metadata(mock_read_text: mock.Mock) -> None:
     mock_read_text.assert_called_once_with(DIRECT_URL_METADATA_NAME)
     assert direct_url.url == "https://e.c/p.tgz"
     assert isinstance(direct_url.info, ArchiveInfo)
+
+
+def test_metadata_dict(tmp_path: Path) -> None:
+    """Basic test of BaseDistribution metadata_dict.
+
+    More tests are available in the original pkg_metadata project where this
+    function comes from, and which we may vendor in the future.
+    """
+    wheel_path = make_wheel(name="pkga", version="1.0.1").save_to_dir(tmp_path)
+    wheel = FilesystemWheel(wheel_path)
+    dist = get_wheel_distribution(wheel, "pkga")
+    metadata_dict = dist.metadata_dict
+    assert metadata_dict["name"] == "pkga"
+    assert metadata_dict["version"] == "1.0.1"