Repository: pyodide/pyodide
Pull request: ENH Add micropip.list()
### Summary

Closes: #1967

Adds a new API `micropip.list()` which returns a list of installed packages.

### Detail

- `PackageMetadata`: a dataclass object that contains (name, version, source) of a package
- `PackageDict`: a dict-like object that maps package name to PackageMetadata
  - has custom `__repr__()` to print list of packages in tabularized format
- `micropip.list()` returns `PackageList` instance which contains list of installed packages
- open to better names and better container design 😀

```
>>> import micropip
>>> await micropip.install("black")
>>> await micropip.install("https://files.pythonhosted.org/packages/89/06/2c2d3034b4d6bf22f2a4ae546d16925898658a33b4400cfb7e2c1e2871a3/pytz-2020.5-py2.py3-none-any.whl")
>>> pkgs = micropip.list()
>>> print(pkgs)
| Name              | Version  | Source                                                                                                                                      |
| ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| regex             | 2021.7.6 | pyodide                                                                                                                                     |
| click             | 8.0.3    | pypi                                                                                                                                        |
| typing-extensions | 4.0.1    | pypi                                                                                                                                        |
| platformdirs      | 2.4.0    | pypi                                                                                                                                        |
| pathspec          | 0.9.0    | pypi                                                                                                                                        |
| tomli             | 1.2.2    | pypi                                                                                                                                        |
| mypy-extensions   | 0.4.3    | pypi                                                                                                                                        |
| black             | 21.11b1  | pypi                                                                                                                                        |
| pytz              | 2020.5   | https://files.pythonhosted.org/packages/89/06/2c2d3034b4d6bf22f2a4ae546d16925898658a33b4400cfb7e2c1e2871a3/pytz-2020.5-py2.py3-none-any.whl |
>>> "regex" in pkgs
True
>>> "pytz" in pkgs
True
>>> "numpy" in pkgs
False
```

### Checklists

- [x] Add a [CHANGELOG](https://github.com/pyodide/pyodide/blob/main/docs/project/changelog.md) entry
- [x] Add / update tests
- [ ] `WHEEL_BASE` path
- [ ] Update `pyodide.loadedPackages docs`
<!-- - [ ] Add new / update outdated documentation -->


Problem statement:
Add micropip.list
It would be nice to add `micropip.list` function to list all installed packages and their versions, by analogy with the `pip list` command.

The output could be a list like object of dictionaries (or data class objects) with the following fields,
```py
    name: str
    version : str
    path : Path
```
and a `__repr__`  method to display this nicely (except for maybe the path). Actually, tt could be either,
 1. a custom list like object with a `__repr__` containing standard dict objects
 2. a standard list with dataclasses that each has a custom `__repr__` method

or some mixture of the two. Option 2 would likely be easiest to use/implement IMO.

Hints from issue discussion:
Should the output of this API include js packages like numpy? I think so if we are going to merge `micropip` and `pyodide.loadPackage`
Yes, I was thinking we could support both, and indicate the source in a separate column.
```
# Name        Version                   Source
package_1     0.2.2                     pypi
package_2     1.3.1                     pyodide
package_3     0.1.4                     <some-custom-url.whl>
```
This is what `conda list` is doing for instance by listing both pip and conda install packages.

Not 100% sure yet what would be the best output type to both have this repr and a usable container object with the results though.
> I think so if we are going to merge micropip and pyodide.loadPackage

Answered in https://github.com/pyodide/pyodide/issues/1470#issuecomment-970079464

Commit message: Implement micropip.list()

Patch to review:
diff --git a/packages/micropip/src/micropip/__init__.py b/packages/micropip/src/micropip/__init__.py
index 8833180c86d..524499beee2 100644
--- a/packages/micropip/src/micropip/__init__.py
+++ b/packages/micropip/src/micropip/__init__.py
@@ -1,3 +1,3 @@
-from .micropip import install
+from .micropip import install, list
 
-__all__ = ["install"]
+__all__ = ["install", "list"]
diff --git a/packages/micropip/src/micropip/micropip.py b/packages/micropip/src/micropip/micropip.py
index be1a26dded5..ddd44e33a4b 100644
--- a/packages/micropip/src/micropip/micropip.py
+++ b/packages/micropip/src/micropip/micropip.py
@@ -1,4 +1,5 @@
 import asyncio
+import copy
 import hashlib
 import importlib
 import io
@@ -13,6 +14,7 @@
 from zipfile import ZipFile
 
 from .externals.pip._internal.utils.wheel import pkg_resources_distribution_for_wheel
+from .package import PackageList, PackageMetadata
 
 from pyodide import IN_BROWSER, to_js
 
@@ -140,10 +142,13 @@ async def _install_wheel(name, fileinfo):
 
 class _PackageManager:
     def __init__(self):
-        self.installed_packages = {}
+        self.installed_packages = PackageList()
 
     async def gather_requirements(
-        self, requirements: Union[str, List[str]], ctx=None, keep_going: bool = False
+        self,
+        requirements: Union[str, List[str]],
+        ctx=None,
+        keep_going: bool = False,
     ):
         ctx = ctx or default_environment()
         ctx.setdefault("extra", None)
@@ -153,7 +158,7 @@ async def gather_requirements(
         transaction: Dict[str, Any] = {
             "wheels": [],
             "pyodide_packages": [],
-            "locked": dict(self.installed_packages),
+            "locked": copy.deepcopy(self.installed_packages),
             "failed": [],
             "keep_going": keep_going,
         }
@@ -185,11 +190,11 @@ async def install(
         if len(pyodide_packages):
             # Note: branch never happens in out-of-browser testing because in
             # that case BUILTIN_PACKAGES is empty.
-            self.installed_packages.update(pyodide_packages)
+            self.installed_packages.update({pkg.name: pkg for pkg in pyodide_packages})
             wheel_promises.append(
                 asyncio.ensure_future(
                     pyodide_js.loadPackage(
-                        to_js([name for [name, _] in pyodide_packages])
+                        to_js([name for [name, _, _] in pyodide_packages])
                     )
                 )
             )
@@ -197,7 +202,13 @@ async def install(
         # Now install PyPI packages
         for name, wheel, ver in transaction["wheels"]:
             wheel_promises.append(_install_wheel(name, wheel))
-            self.installed_packages[name] = ver
+
+            # detect whether the wheel metadata is from PyPI or from custom location
+            # wheel metadata from PyPI has SHA256 checksum digest.
+            wheel_source = "pypi" if wheel["digests"] is not None else wheel["url"]
+            self.installed_packages[name] = PackageMetadata(
+                name=name, version=str(ver), source=wheel_source
+            )
         await gather(*wheel_promises)
 
     async def add_requirement(
@@ -230,7 +241,9 @@ async def add_requirement(
             and BUILTIN_PACKAGES[req.name]["version"] in req.specifier
         ):
             version = BUILTIN_PACKAGES[req.name]["version"]
-            transaction["pyodide_packages"].append((req.name, version))
+            transaction["pyodide_packages"].append(
+                PackageMetadata(name=req.name, version=version, source="pyodide")
+            )
             return
 
         if req.marker:
@@ -241,7 +254,7 @@ async def add_requirement(
 
         # Is some version of this package is already installed?
         if req.name in transaction["locked"]:
-            ver = transaction["locked"][req.name]
+            ver = transaction["locked"][req.name].version
             if ver in req.specifier:
                 # installed version matches, nothing to do
                 return
@@ -264,7 +277,7 @@ async def add_requirement(
             await self.add_wheel(req.name, wheel, ver, req.extras, ctx, transaction)
 
     async def add_wheel(self, name, wheel, version, extras, ctx, transaction):
-        transaction["locked"][name] = version
+        transaction["locked"][name] = PackageMetadata(name=name, version=version)
         wheel_bytes = await fetch_bytes(wheel["url"])
         wheel["wheel_bytes"] = wheel_bytes
 
@@ -307,6 +320,9 @@ def find_wheel(
 
         return None, None
 
+    def list(self) -> PackageList:
+        return copy.deepcopy(self.installed_packages)
+
 
 # Make PACKAGE_MANAGER singleton
 PACKAGE_MANAGER = _PackageManager()
@@ -365,7 +381,30 @@ def install(requirements: Union[str, List[str]], keep_going: bool = False):
     )
 
 
-__all__ = ["install"]
+def list():
+    """Get the list of installed packages through micropip.
+
+    Returns
+    -------
+    ``PackageMetadataList``
+        A list of installed packages.
+
+        >>> package_list = micropip.list()
+        >>> print(package_list)
+        | Name              | Version  | Source  |
+        | ----------------- | -------- | ------- |
+        | regex             | 2021.7.6 | pyodide |
+        | tomli             | 1.2.2    | pypi    |
+        | mypy-extensions   | 0.4.3    | pypi    |
+        ...
+        >>> "regex" in package_list
+        True
+    """
+    return PACKAGE_MANAGER.list()
+
+
+# prevent sphinx autodoc from generating duplicated docs
+__all__: List[str] = []
 
 
 if __name__ == "__main__":
diff --git a/packages/micropip/src/micropip/package.py b/packages/micropip/src/micropip/package.py
new file mode 100644
index 00000000000..695dfa56b94
--- /dev/null
+++ b/packages/micropip/src/micropip/package.py
@@ -0,0 +1,71 @@
+from collections.abc import Collection
+from dataclasses import dataclass, field, astuple
+from pathlib import Path
+from typing import List, Dict, Iterable
+
+__all__: List[str] = []
+
+
+def _format_table(headers: List[str], table: List[Iterable]) -> str:
+    def format_row(values, widths, filler=""):
+        columns = " | ".join(f"{x:{filler}<{w}}" for x, w in zip(values, widths))
+        return f"| {columns} |"
+
+    col_width = [max(len(x) for x in col) for col in zip(headers, *table)]
+    rows = []
+
+    rows.append(format_row(headers, col_width))
+    rows.append(format_row([""] * len(col_width), col_width, filler="-"))
+    for line in table:
+        rows.append(format_row(line, col_width))
+
+    return "\n".join(rows)
+
+
+@dataclass
+class PackageMetadata:
+    name: str
+    version: str = ""
+    source: str = ""
+
+    def __iter__(self):
+        return iter(astuple(self))
+
+    @staticmethod
+    def keys():
+        return [key.capitalize() for key in PackageMetadata.__dataclass_fields__.keys()]
+
+
+class PackageList(Collection):
+    def __init__(self):
+        self.packages: Dict[str, PackageMetadata] = {}
+
+    def __repr__(self):
+        return self.tabularize()
+
+    def __len__(self):
+        return len(self.packages)
+
+    def __iter__(self):
+        return iter(self.packages.values())
+
+    def __contains__(self, pkg_name: str):  # type: ignore
+        return pkg_name in self.package_names
+
+    def __setitem__(self, key: str, item: PackageMetadata):
+        self.packages[key] = item
+
+    def __getitem__(self, key: str):
+        return self.packages[key]
+
+    @property
+    def package_names(self):
+        return [pkg.name for pkg in self]
+
+    def update(self, pkg: Dict[str, PackageMetadata]):
+        self.packages.update(pkg)
+
+    def tabularize(self):
+        headers = PackageMetadata.keys()
+        table = list(self.packages.values())
+        return _format_table(headers, table)


Code review comments:
diff_hunk: @@ -365,7 +381,30 @@ def install(requirements: Union[str, List[str]], keep_going: bool = False):
     )
 
 
-__all__ = ["install"]
+def list():
path: packages/micropip/src/micropip/micropip.py
text: @user1:
Maybe let's call it `_list` and rename to `list` when importing in `__init__` ? As renaming the builtin `list` can probably have bad side effects..

Actually, I think we could also move the whole file to `_micropip.py` as everything here is private, and public objects are re-exported to the top-level namespace.

@author:
Yeah, I was also thinking that using the name `list` could be problematic.

diff_hunk: @@ -185,19 +190,25 @@ async def install(
         if len(pyodide_packages):
             # Note: branch never happens in out-of-browser testing because in
             # that case BUILTIN_PACKAGES is empty.
-            self.installed_packages.update(pyodide_packages)
+            self.installed_packages.update({pkg.name: pkg for pkg in pyodide_packages})
             wheel_promises.append(
                 asyncio.ensure_future(
                     pyodide_js.loadPackage(
-                        to_js([name for [name, _] in pyodide_packages])
+                        to_js([name for [name, _, _] in pyodide_packages])
                     )
                 )
             )
 
         # Now install PyPI packages
         for name, wheel, ver in transaction["wheels"]:
             wheel_promises.append(_install_wheel(name, wheel))
-            self.installed_packages[name] = ver
+
+            # detect whether the wheel metadata is from PyPI or from custom location
+            # wheel metadata from PyPI has SHA256 checksum digest.
+            wheel_source = "pypi" if wheel["digests"] is not None else wheel["url"]
+            self.installed_packages[name] = PackageMetadata(
path: packages/micropip/src/micropip/micropip.py
text: @user1:
Not introduced in this PR; but it could be possible that the above _install_wheel would fail, and yet we would still add this to the list of installed packages? Maybe better to do it via some callback to `_install_wheel ` after the installation succeeded there ?

diff_hunk: @@ -185,19 +190,25 @@ async def install(
         if len(pyodide_packages):
             # Note: branch never happens in out-of-browser testing because in
             # that case BUILTIN_PACKAGES is empty.
-            self.installed_packages.update(pyodide_packages)
+            self.installed_packages.update({pkg.name: pkg for pkg in pyodide_packages})
             wheel_promises.append(
                 asyncio.ensure_future(
                     pyodide_js.loadPackage(
-                        to_js([name for [name, _] in pyodide_packages])
+                        to_js([name for [name, _, _] in pyodide_packages])
                     )
                 )
             )
 
         # Now install PyPI packages
         for name, wheel, ver in transaction["wheels"]:
             wheel_promises.append(_install_wheel(name, wheel))
-            self.installed_packages[name] = ver
+
+            # detect whether the wheel metadata is from PyPI or from custom location
+            # wheel metadata from PyPI has SHA256 checksum digest.
+            wheel_source = "pypi" if wheel["digests"] is not None else wheel["url"]
path: packages/micropip/src/micropip/micropip.py
text: @author:
This (checking digest) is a hacky way to detect whether the wheel is from PyPI or from a custom URL.
Another way can be adding a key that explicitly indicates the wheel is from a custom URL.

diff_hunk: @@ -0,0 +1,71 @@
+from collections.abc import Collection
+from dataclasses import dataclass, field, astuple
+from pathlib import Path
+from typing import List, Dict, Iterable
+
+__all__: List[str] = []
+
+
+def _format_table(headers: List[str], table: List[Iterable]) -> str:
+    def format_row(values, widths, filler=""):
+        columns = " | ".join(f"{x:{filler}<{w}}" for x, w in zip(values, widths))
+        return f"| {columns} |"
+
+    col_width = [max(len(x) for x in col) for col in zip(headers, *table)]
+    rows = []
+
+    rows.append(format_row(headers, col_width))
+    rows.append(format_row([""] * len(col_width), col_width, filler="-"))
+    for line in table:
+        rows.append(format_row(line, col_width))
+
+    return "\n".join(rows)
+
+
+@dataclass
+class PackageMetadata:
+    name: str
+    version: str = ""
+    source: str = ""
+
+    def __iter__(self):
+        return iter(astuple(self))
+
+    @staticmethod
+    def keys():
+        return [key.capitalize() for key in PackageMetadata.__dataclass_fields__.keys()]
+
+
+class PackageList(Collection):
path: packages/micropip/src/micropip/package.py
text: @user1:
Since the expected bahavior is that of a custom dict, I wonder if it wouldn't be simpler to inherit from 
`collections.UserDict` which will make it unnecessary to implement most of the following `__` methods 

Also because it's returned by the list API, we might need to add it to sphinx documentation. Which means it would need a short docstring.

diff_hunk: @@ -0,0 +1,71 @@
+from collections.abc import Collection
+from dataclasses import dataclass, field, astuple
+from pathlib import Path
+from typing import List, Dict, Iterable
+
+__all__: List[str] = []
+
+
+def _format_table(headers: List[str], table: List[Iterable]) -> str:
+    def format_row(values, widths, filler=""):
+        columns = " | ".join(f"{x:{filler}<{w}}" for x, w in zip(values, widths))
+        return f"| {columns} |"
+
+    col_width = [max(len(x) for x in col) for col in zip(headers, *table)]
+    rows = []
+
+    rows.append(format_row(headers, col_width))
+    rows.append(format_row([""] * len(col_width), col_width, filler="-"))
+    for line in table:
+        rows.append(format_row(line, col_width))
+
+    return "\n".join(rows)
+
+
+@dataclass
+class PackageMetadata:
+    name: str
+    version: str = ""
+    source: str = ""
+
+    def __iter__(self):
+        return iter(astuple(self))
+
+    @staticmethod
+    def keys():
+        return [key.capitalize() for key in PackageMetadata.__dataclass_fields__.keys()]
+
+
+class PackageList(Collection):
+    def __init__(self):
+        self.packages: Dict[str, PackageMetadata] = {}
+
+    def __repr__(self):
+        return self.tabularize()
+
+    def __len__(self):
+        return len(self.packages)
+
+    def __iter__(self):
+        return iter(self.packages.values())
+
+    def __contains__(self, pkg_name: str):  # type: ignore
+        return pkg_name in self.package_names
+
+    def __setitem__(self, key: str, item: PackageMetadata):
+        self.packages[key] = item
+
+    def __getitem__(self, key: str):
+        return self.packages[key]
+
+    @property
+    def package_names(self):
+        return [pkg.name for pkg in self]
+
+    def update(self, pkg: Dict[str, PackageMetadata]):
+        self.packages.update(pkg)
+
+    def tabularize(self):
path: packages/micropip/src/micropip/package.py
text: @user1:
This can probably be the `__repr__` method directly or a `_tabularize` method to keep it private.

diff_hunk: @@ -365,7 +381,30 @@ def install(requirements: Union[str, List[str]], keep_going: bool = False):
     )
 
 
-__all__ = ["install"]
+def list():
+    """Get the list of installed packages through micropip.
+
+    Returns
+    -------
+    ``PackageMetadataList``
+        A list of installed packages.
+
+        >>> package_list = micropip.list()
+        >>> print(package_list)
+        | Name              | Version  | Source  |
+        | ----------------- | -------- | ------- |
+        | regex             | 2021.7.6 | pyodide |
+        | tomli             | 1.2.2    | pypi    |
+        | mypy-extensions   | 0.4.3    | pypi    |
path: packages/micropip/src/micropip/micropip.py
text: @user1:
Looks nice. How about,
```suggestion
        Name              | Version  | Source
        ----------------- | -------- | -------
        regex             | 2021.7.6 | pyodide
        tomli             | 1.2.2    | pypi   
        mypy-extensions   | 0.4.3    | pypi 
```
as something a bit lighter? Unless the goal was to make markdown compatible tables?