Repository: pyodide/pyodide

Issue:
[Discussion] Changing the default behavior of loading unvendored stdlib
## Proposed refactoring or deprecation

- Related: #1542

We unvendors some stdlib packages in order to reduce the size of the main module.

- distutils
- tests
- ssl
- lzma (#2939)
- (maybe more in the future, e.g. sqlite3, unicodedata, ..., #1826)

The `pyodide.loadPyodide` API accepts `fullstdlib` parameter, which is currently true in default. What `fullstdlib` currently does is [load distutils package if true](https://github.com/pyodide/pyodide/blob/14708295cd1d2a38235e2dd3727ed7760e29eaa0/src/js/pyodide.ts#L319-L321). It does not affect loading other unvendored stdlibs like ssl, which I think is a bit confusing for users.

IMO not loading ssl and some other unvendored stdlibs are reasonable since the sizes of those packages are quite big.

```
960K    distutils.tar
840K    ssl-1.0.0.zip
```

So I think it would be great to
1. Change the default value of fullstdlib to “False”.
2. If fullstdlib is True, load all unvendored packages including distutils, ssl, lzma (except for `test` which is too big (~22MB) and mostly not used by users).
3. Add unvendored stdlibs to “always packages” when building.

WDYT?



Discussion:
We should discuss this at https://discuss.python.org/c/webassembly/28. I wonder how scalable our current approach is and whether it wouldn't break people's assumptions about what the stdlib should contain (particularly now that Wasm is an officially supported CPython target).
> We should discuss this at https://discuss.python.org/c/webassembly/28. I wonder how scalable our current approach is and whether it wouldn't break people's assumptions about what the stdlib should contain (particularly now that Wasm is an officially supported CPython target).

Yes, I think you are right. Actually, I found that you have already opened a discussion about this last year: https://discuss.python.org/t/minifying-the-stdlib-in-pyodide/8414.
Opened a new discussion here: https://discuss.python.org/t/unvendoring-some-of-stdlib-modules/18076
Back in this topic, my opinion is that we can start with not loading distutils by default (i.e. setting default value of fullstdlib to false), while unvendoring other packages can be discussed case by case later.
Yes, agreed. It was good to get feedback from CPython devs. It sounded like there is no strong opposition to this idea but I agree that we should be restrained about the number of stdlib modules we unvendor otherwise user experience is going to be worse when installing dependencies.

Resolving patch:
diff --git a/pyodide-build/pyodide_build/common.py b/pyodide-build/pyodide_build/common.py
--- a/pyodide-build/pyodide_build/common.py
+++ b/pyodide-build/pyodide_build/common.py
@@ -97,6 +97,9 @@ def find_matching_wheels(wheel_paths: Iterable[Path]) -> Iterator[Path]:
     "micropip",
     "distutils",
     "test",
+    "ssl",
+    "lzma",
+    "sqlite3",
 }
 
 CORE_PACKAGES = {
@@ -109,11 +112,8 @@ def find_matching_wheels(wheel_paths: Iterable[Path]) -> Iterator[Path]:
     "fpcast-test",
     "sharedlib-test-py",
     "cpp-exceptions-test",
-    "ssl",
-    "lzma",
     "pytest",
     "tblib",
-    "sqlite3",
 }
 
 CORE_SCIPY_PACKAGES = {
diff --git a/src/py/_pyodide/_importhook.py b/src/py/_pyodide/_importhook.py
--- a/src/py/_pyodide/_importhook.py
+++ b/src/py/_pyodide/_importhook.py
@@ -138,6 +138,10 @@ def register_js_finder() -> None:
     sys.meta_path.append(jsfinder)
 
 
+UNVENDORED_STDLIBS = ["distutils", "ssl", "lzma", "sqlite3"]
+UNVENDORED_STDLIBS_AND_TEST = UNVENDORED_STDLIBS + ["test"]
+
+
 class UnvendoredStdlibFinder(MetaPathFinder):
     """
     A MetaPathFinder that handles unvendored and removed stdlib modules.
@@ -150,15 +154,10 @@ class UnvendoredStdlibFinder(MetaPathFinder):
     def __init__(self) -> None:
         # `test`` is not a stdlib module, but we unvendors in anyway.
         self.stdlibs = sys.stdlib_module_names | {"test"}
-
-        # TODO: put list of unvendored stdlibs to somewhere else?
-        self.unvendored_stdlibs = {
-            "distutils",
-            "test",
-            "ssl",
-            "lzma",
-            "sqlite3",
-        } & self.stdlibs
+        self.unvendored_stdlibs = set(UNVENDORED_STDLIBS_AND_TEST)
+        assert not (
+            self.unvendored_stdlibs - self.stdlibs
+        ), "unvendored stdlibs not in stdlibs"
 
     def find_spec(
         self,


Test patch:
diff --git a/src/tests/test_pyodide.py b/src/tests/test_pyodide.py
--- a/src/tests/test_pyodide.py
+++ b/src/tests/test_pyodide.py
@@ -1130,6 +1130,27 @@ def test_sys_path0(selenium):
     )
 
 
+def test_fullstdlib(selenium_standalone_noload):
+    selenium = selenium_standalone_noload
+    selenium.run_js(
+        """
+        let pyodide = await loadPyodide({
+            fullStdLib: true,
+        });
+
+        await pyodide.loadPackage("micropip");
+
+        pyodide.runPython(`
+            import _pyodide
+            import micropip
+            loaded_packages = micropip.list()
+            print(loaded_packages)
+            assert all((lib in micropip.list()) for lib in _pyodide._importhook.UNVENDORED_STDLIBS)
+        `);
+        """
+    )
+
+
 @run_in_pyodide
 def test_run_js(selenium):
     from unittest import TestCase