Repository: pyodide/pyodide
Pull request: Change the name of repodata.json to pyodide-lock.json
Resolves #3788

- [x] Add a [CHANGELOG](https://github.com/pyodide/pyodide/blob/main/docs/project/changelog.md) entry


Problem statement:
Change `repodata.json` name to something else?
I am told by conda people that repodata.json is supposed to be the input to a solver. Our repodata.json is a lock file which is the output of a solver. This is a misleading name clash. We will need to spend a lot of time explaining to conda people that this file isn't what they expect to be. Can we rename it?

Hints from issue discussion:
@katietz
In the default mode, I would say it's still the input to a solver, it's just that given that Pyodide is a distribution ( with a single version for each package) dependency resolution is trivial. In terms of contents, it's  taking `meta.yaml` and copying them to this file, so same as conda as far as I understand?

Though yes, if one starts to generate a `repodata.json` for a specific application, then it really starts to look like a lock file.  Is there anything like a repodata.json but used for lock files in the conda ecosystem?
But 1. we guarantee existence and uniqueness of "solutions" and 2. we guarantee that the solution is found in linear time in the solution size. The solver is in fact trivial. What we actually have is more like a virtual environment that we lazily load packages from as they are needed. We will only load exactly one version of each package. Kai was suggesting adding multiple versions and I said I valued a lot the speed, the existence and the uniqueness which are all properties we lose. But this is proof that we are not similar to conda repodata.json.

Real solvers handle multiple versions of each package and version constraints on dependencies. There may be any number of solutions, 0, one, or many. The time to find the solution is potentially exponential in the solution size. 

As a useless theoretical aside, I think solving is NP hard and probably also co NP hard -- the decision problem is "does there exist a subset of packages and versions such that for every package in the set and for every dependency of that package, the constraint is satisfied". This sounds to me like Sigma Pi in the polynomial time hierarchy, so that it is conjecturally strictly harder than NP complete (it's possible of course that P = NP or that P != NP but NP = CoNP, and in either case it wouldn't be any harder). I'm curious whether it could be Sigma Pi complete. Have to ask my complexity theory friends.

Commit message: Change the name of repodata.json to pyodide-lock.json

Patch to review:
diff --git a/Makefile b/Makefile
index a16097803c6..d4c4229f6c2 100644
--- a/Makefile
+++ b/Makefile
@@ -15,7 +15,7 @@ all: check \
 	dist/package.json \
 	dist/python \
 	dist/console.html \
-	dist/repodata.json \
+	dist/pyodide-lock.json \
 	dist/python_stdlib.zip \
 	dist/test.html \
 	dist/module_test.html \
@@ -246,7 +246,7 @@ $(CPYTHONLIB): emsdk/emsdk/.complete
 	date +"[%F %T] done building cpython..."
 
 
-dist/repodata.json: FORCE pyodide_build
+dist/pyodide-lock.json: FORCE pyodide_build
 	date +"[%F %T] Building packages..."
 	make -C packages
 	date +"[%F %T] done building packages..."
diff --git a/docs/development/new-packages.md b/docs/development/new-packages.md
index f77f96d3be4..a5a19f20018 100644
--- a/docs/development/new-packages.md
+++ b/docs/development/new-packages.md
@@ -293,7 +293,7 @@ build. We automate the following steps:
   `<package name>-tests.zip`
 - Repack the wheel with `python -m wheel pack`
 
-Lastly, a `repodata.json` file is created containing the dependency tree of all
+Lastly, a `pyodide-lock.json` file is created containing the dependency tree of all
 packages, so {js:func}`pyodide.loadPackage` can load a package's dependencies
 automatically.
 
diff --git a/pyodide-build/pyodide_build/_py_compile.py b/pyodide-build/pyodide_build/_py_compile.py
index 8feaa3faf99..ccedc223439 100644
--- a/pyodide-build/pyodide_build/_py_compile.py
+++ b/pyodide-build/pyodide_build/_py_compile.py
@@ -261,7 +261,7 @@ def _get_py_compiled_archive_name(path: Path) -> str | None:
 def _update_repodata(
     input_dir: Path, repodata: dict[str, Any], name_mapping: dict[str, str]
 ) -> dict[str, Any]:
-    """Update repodata.json with the new names of the py-compiled wheels.
+    """Update pyodide-lock.json with the new names of the py-compiled wheels.
 
     Also update the checksums of the updated wheels
     """
@@ -321,7 +321,7 @@ def _py_compile_archive_dir(
             )
             name_mapping[file_path.name] = output_name
 
-    repodata_path = input_dir / "repodata.json"
+    repodata_path = input_dir / "pyodide-lock.json"
     if name_mapping and repodata_path.exists():
         if verbose:
             print(f"Updating {repodata_path.name}")
diff --git a/pyodide-build/pyodide_build/buildall.py b/pyodide-build/pyodide_build/buildall.py
index 0659de53a5c..1034e8fe838 100755
--- a/pyodide-build/pyodide_build/buildall.py
+++ b/pyodide-build/pyodide_build/buildall.py
@@ -783,7 +783,7 @@ def install_packages(
     """
     Install packages into the output directory.
     - copies build artifacts (wheel, zip, ...) to the output directory
-    - create repodata.json
+    - create pyodide_lock.json
 
 
     pkg_map
@@ -800,8 +800,8 @@ def install_packages(
         pkg_map.values(), output_dir, compression_level=compression_level
     )
 
-    repodata_path = output_dir / "repodata.json"
-    logger.info(f"Writing repodata.json to {repodata_path}")
+    repodata_path = output_dir / "pyodide-lock.json"
+    logger.info(f"Writing pyodide-lock.json to {repodata_path}")
 
     package_data = generate_repodata(output_dir, pkg_map)
     with repodata_path.open("w") as fd:
diff --git a/pyodide-build/pyodide_build/cli/build_recipes.py b/pyodide-build/pyodide_build/cli/build_recipes.py
index 875c7d04ae7..7c9d5e27c69 100644
--- a/pyodide-build/pyodide_build/cli/build_recipes.py
+++ b/pyodide-build/pyodide_build/cli/build_recipes.py
@@ -27,7 +27,7 @@ def recipe(
     ),
     install_dir: str = typer.Option(
         None,
-        help="Path to install built packages and repodata.json. "
+        help="Path to install built packages and pyodide-lock.json. "
         "If not specified, the default is `./dist`.",
     ),
     cflags: str = typer.Option(
@@ -66,7 +66,7 @@ def recipe(
         help="Level of zip compression to apply when installing. 0 means no compression.",
     ),
 ) -> None:
-    """Build packages using yaml recipes and create repodata.json"""
+    """Build packages using yaml recipes and create pyodide-lock.json"""
     initialize_pyodide_root()
 
     if common.in_xbuildenv():
diff --git a/pyodide-build/pyodide_build/cli/py_compile.py b/pyodide-build/pyodide_build/cli/py_compile.py
index 0768e6f3247..e4b9a713327 100644
--- a/pyodide-build/pyodide_build/cli/py_compile.py
+++ b/pyodide-build/pyodide_build/cli/py_compile.py
@@ -17,7 +17,7 @@ def main(
 ) -> None:
     """Compile .py files to .pyc in a wheel, a zip file, or a folder with wheels or zip files.
 
-    If the provided folder contains the `repodata.json` file, it will be
+    If the provided folder contains the `pyodide-lock.json` file, it will be
     rewritten with the updated wheel / zip file paths and sha256 checksums.
     """
     if not path.exists():
diff --git a/pyodide-build/pyodide_build/create_pypa_index.py b/pyodide-build/pyodide_build/create_pypa_index.py
index 74b22bf24c4..b8c1ebef64a 100644
--- a/pyodide-build/pyodide_build/create_pypa_index.py
+++ b/pyodide-build/pyodide_build/create_pypa_index.py
@@ -24,7 +24,7 @@ def create_pypa_index(
     ----------
     packages:
         A dictionary of packages that we want to index. This should be the
-        "packages" field from repodata.json.
+        "packages" field from pyodide-lock.json.
 
     target_dir:
         Where to put the  index. It will be placed in a subfolder of
diff --git a/pyodide-build/pyodide_build/create_xbuildenv.py b/pyodide-build/pyodide_build/create_xbuildenv.py
index ad54db7cc4a..19e683f46b3 100644
--- a/pyodide-build/pyodide_build/create_xbuildenv.py
+++ b/pyodide-build/pyodide_build/create_xbuildenv.py
@@ -54,7 +54,7 @@ def get_relative_path(pyodide_root: Path, flag: str) -> Path:
         sysconfig_dir,
         Path("Makefile.envs"),
         wasm_lib_dir / "cmake",
-        Path("dist/repodata.json"),
+        Path("dist/pyodide-lock.json"),
         Path("dist/python"),
         Path("dist/python_stdlib.zip"),
     ]
diff --git a/pyodide-build/pyodide_build/install_xbuildenv.py b/pyodide-build/pyodide_build/install_xbuildenv.py
index 59459c0ced9..f1bd96b02ea 100644
--- a/pyodide-build/pyodide_build/install_xbuildenv.py
+++ b/pyodide-build/pyodide_build/install_xbuildenv.py
@@ -62,15 +62,15 @@ def install_xbuildenv(version: str, xbuildenv_path: Path) -> None:
         xbuildenv_path / "site-packages-extras", host_site_packages, dirs_exist_ok=True
     )
     cdn_base = f"https://cdn.jsdelivr.net/pyodide/v{version}/full/"
-    if (repodata_json := xbuildenv_root / "dist" / "repodata.json").exists():
-        repodata_bytes = repodata_json.read_bytes()
+    if (lockfile := xbuildenv_root / "dist" / "pyodide-lock.json").exists():
+        lockfile_bytes = lockfile.read_bytes()
     else:
-        repodata_url = cdn_base + "repodata.json"
-        with urlopen(repodata_url) as response:
-            repodata_bytes = response.read()
-    repodata = json.loads(repodata_bytes)
-    version = repodata["info"]["version"]
-    create_pypa_index(repodata["packages"], xbuildenv_root, cdn_base)
+        lockfile_url = cdn_base + "pyodide-lock.json"
+        with urlopen(lockfile_url) as response:
+            lockfile_bytes = response.read()
+    lockfile = json.loads(lockfile_bytes)
+    version = lockfile["info"]["version"]
+    create_pypa_index(lockfile["packages"], xbuildenv_root, cdn_base)
 
 
 def install(path: Path, *, download: bool = False, url: str | None = None) -> None:
diff --git a/pyodide-build/pyodide_build/out_of_tree/pypi.py b/pyodide-build/pyodide_build/out_of_tree/pypi.py
index 79ae971f52e..82cfc682ae3 100644
--- a/pyodide-build/pyodide_build/out_of_tree/pypi.py
+++ b/pyodide-build/pyodide_build/out_of_tree/pypi.py
@@ -302,7 +302,7 @@ def get_dependencies(self, candidate):
 def _get_json_package_list(fname: Path) -> Generator[str, None, None]:
     json_data = json.load(fname.open())
     if "packages" in json_data:
-        # pyodide repodata.json format
+        # pyodide-lock.json format
         yield from json_data["packages"].keys()
     else:
         # jupyterlite all.json format
diff --git a/pyodide-build/pyodide_build/tests/fixture.py b/pyodide-build/pyodide_build/tests/fixture.py
index 6d66e495f26..351b9840abd 100644
--- a/pyodide-build/pyodide_build/tests/fixture.py
+++ b/pyodide-build/pyodide_build/tests/fixture.py
@@ -81,7 +81,7 @@ def temp_xbuildenv(tmp_path_factory):
 """  # noqa: W191
     )
     (pyodide_root / "dist").mkdir()
-    (pyodide_root / "dist" / "repodata.json").write_text(
+    (pyodide_root / "dist" / "pyodide-lock.json").write_text(
         json.dumps(mock_repodata_json())
     )
 
diff --git a/pyodide-build/pyodide_build/tests/test_py_compile.py b/pyodide-build/pyodide_build/tests/test_py_compile.py
index 0e162131c0e..6333ef4ed6b 100644
--- a/pyodide-build/pyodide_build/tests/test_py_compile.py
+++ b/pyodide-build/pyodide_build/tests/test_py_compile.py
@@ -191,7 +191,7 @@ def test_py_compile_archive_dir(tmp_path, with_repodata):
         "packageB", base_dir=tmp_path, data=wheel_data, tag="py3-none-any"
     )
 
-    repodata_path = tmp_path / "repodata.json"
+    repodata_path = tmp_path / "pyodide-lock.json"
     repodata = {
         "info": {"arch": "wasm32"},
         "packages": {
@@ -210,8 +210,8 @@ def test_py_compile_archive_dir(tmp_path, with_repodata):
     if with_repodata:
         with open(repodata_path, "w") as fh:
             json.dump(repodata, fh)
-        expected_in.add("repodata.json")
-        expected_out.add("repodata.json")
+        expected_in.add("pyodide-lock.json")
+        expected_out.add("pyodide-lock.json")
 
     assert set(el.name for el in tmp_path.glob("*")) == expected_in
 
diff --git a/src/js/api.ts b/src/js/api.ts
index fc4b10b640c..c5245e5d55f 100644
--- a/src/js/api.ts
+++ b/src/js/api.ts
@@ -248,7 +248,7 @@ export class PyodideAPI {
    *
    *    let result = await pyodide.runPythonAsync(`
    *        from js import fetch
-   *        response = await fetch("./repodata.json")
+   *        response = await fetch("./pyodide-lock.json")
    *        packages = await response.json()
    *        # If final statement is an expression, its value is returned to JavaScript
    *        len(packages.packages.object_keys())
diff --git a/src/js/load-package.ts b/src/js/load-package.ts
index e324bbdcd7e..bcb209176d9 100644
--- a/src/js/load-package.ts
+++ b/src/js/load-package.ts
@@ -17,28 +17,28 @@ import { makeWarnOnce } from "./util";
 
 /**
  * Initialize the packages index. This is called as early as possible in
- * loadPyodide so that fetching repodata.json can occur in parallel with other
+ * loadPyodide so that fetching pyodide-lock.json can occur in parallel with other
  * operations.
  * @param lockFileURL
  * @private
  */
 async function initializePackageIndex(lockFileURL: string) {
-  let repodata;
+  let lockfile;
   if (IN_NODE) {
     await initNodeModules();
     const package_string = await nodeFsPromisesMod.readFile(lockFileURL);
-    repodata = JSON.parse(package_string);
+    lockfile = JSON.parse(package_string);
   } else {
     let response = await fetch(lockFileURL);
-    repodata = await response.json();
+    lockfile = await response.json();
   }
-  if (!repodata.packages) {
+  if (!lockfile.packages) {
     throw new Error(
-      "Loaded repodata.json does not contain the expected key 'packages'.",
+      "Loaded pyodide lock file does not contain the expected key 'packages'.",
     );
   }
-  API.repodata_info = repodata.info;
-  API.repodata_packages = repodata.packages;
+  API.repodata_info = lockfile.info;
+  API.repodata_packages = lockfile.packages;
   API.repodata_unvendored_stdlibs_and_test = [];
 
   // compute the inverted index for imports to package names
@@ -97,7 +97,7 @@ type PackageLoadMetadata = {
   installPromise?: Promise<void>;
 };
 
-// Package data inside repodata.json
+// Package data inside pyodide-lock.json
 export type PackageData = {
   file_name: string;
   shared_library: boolean;
@@ -210,7 +210,7 @@ function recursiveDependencies(
 
 /**
  * Download a package. If `channel` is `DEFAULT_CHANNEL`, look up the wheel URL
- * relative to indexURL from `repodata.json`, otherwise use the URL specified by
+ * relative to indexURL from `pyodide-lock.json`, otherwise use the URL specified by
  * `channel`.
  * @param name The name of the package
  * @param channel Either `DEFAULT_CHANNEL` or the absolute URL to the
diff --git a/src/js/package.json b/src/js/package.json
index 2e05f3e7963..b1b08858e0b 100644
--- a/src/js/package.json
+++ b/src/js/package.json
@@ -51,7 +51,7 @@
     "./pyodide.mjs": "./pyodide.mjs",
     "./pyodide.js": "./pyodide.js",
     "./package.json": "./package.json",
-    "./repodata.json": "./repodata.json"
+    "./pyodide-lock.json": "./pyodide-lock.json"
   },
   "files": [
     "pyodide.asm.js",
@@ -62,7 +62,7 @@
     "pyodide.mjs.map",
     "pyodide.d.ts",
     "ffi.d.ts",
-    "repodata.json",
+    "pyodide-lock.json",
     "console.html"
   ],
   "browser": {
diff --git a/src/js/pyodide.ts b/src/js/pyodide.ts
index 9a779beb0de..8407536f3e2 100644
--- a/src/js/pyodide.ts
+++ b/src/js/pyodide.ts
@@ -196,9 +196,9 @@ export async function loadPyodide(
     indexURL?: string;
 
     /**
-     * The URL from which Pyodide will load the Pyodide ``repodata.json`` lock
+     * The URL from which Pyodide will load the Pyodide ``pyodide-lock.json`` lock
      * file. You can produce custom lock files with :py:func:`micropip.freeze`.
-     * Default: ```${indexURL}/repodata.json```
+     * Default: ```${indexURL}/pyodide-lock.json```
      */
     lockFileURL?: string;
 
@@ -265,7 +265,7 @@ export async function loadPyodide(
     jsglobals: globalThis,
     stdin: globalThis.prompt ? globalThis.prompt : undefined,
     homedir: "/home/pyodide",
-    lockFileURL: indexURL! + "repodata.json",
+    lockFileURL: indexURL! + "pyodide-lock.json",
     args: [],
     _node_mounts: [],
   };
diff --git a/src/tests/test_asyncio.py b/src/tests/test_asyncio.py
index e12ecf9b9d4..b19ce9f40db 100644
--- a/src/tests/test_asyncio.py
+++ b/src/tests/test_asyncio.py
@@ -406,7 +406,7 @@ def test_await_pyproxy_eval_async(selenium):
         """
         let c = pyodide._api.pyodide_code.eval_code_async(`
             from js import fetch
-            await (await fetch('repodata.json')).json()
+            await (await fetch('pyodide-lock.json')).json()
         `);
         let result = await c;
         c.destroy();
@@ -421,7 +421,7 @@ def test_await_pyproxy_async_def(selenium):
         let packages = await pyodide.runPythonAsync(`
             from js import fetch
             async def temp():
-                return await (await fetch('repodata.json')).json()
+                return await (await fetch('pyodide-lock.json')).json()
             await temp()
         `);
         return (!!packages.packages) && (!!packages.info);
diff --git a/src/tests/test_jsproxy.py b/src/tests/test_jsproxy.py
index 22b08aa42a8..b7372eaf956 100644
--- a/src/tests/test_jsproxy.py
+++ b/src/tests/test_jsproxy.py
@@ -509,14 +509,14 @@ def temp():
     from pyodide.ffi import create_once_callable
 
     js.setTimeout(create_once_callable(temp), 100)
-    js.fetch("repodata.json")
+    js.fetch("pyodide-lock.json")
 
 
 @run_in_pyodide
 def test_import_bind(selenium):
     from js import fetch
 
-    fetch("repodata.json")
+    fetch("pyodide-lock.json")
 
 
 @run_in_pyodide
diff --git a/src/tests/test_pyodide.py b/src/tests/test_pyodide.py
index 1eb06db5864..bacad8849ad 100644
--- a/src/tests/test_pyodide.py
+++ b/src/tests/test_pyodide.py
@@ -401,7 +401,7 @@ def test_run_python_async_toplevel_await(selenium):
         """
         await pyodide.runPythonAsync(`
             from js import fetch
-            resp = await fetch("repodata.json")
+            resp = await fetch("pyodide-lock.json")
             json = (await resp.json()).to_py()["packages"]
             assert "micropip" in json
         `);


Code review comments:
diff_hunk: @@ -17,28 +17,28 @@ import { makeWarnOnce } from "./util";
 
 /**
  * Initialize the packages index. This is called as early as possible in
- * loadPyodide so that fetching repodata.json can occur in parallel with other
+ * loadPyodide so that fetching pyodide-lock.json can occur in parallel with other
  * operations.
  * @param lockFileURL
  * @private
  */
 async function initializePackageIndex(lockFileURL: string) {
-  let repodata;
+  let lockfile;
   if (IN_NODE) {
     await initNodeModules();
     const package_string = await nodeFsPromisesMod.readFile(lockFileURL);
-    repodata = JSON.parse(package_string);
+    lockfile = JSON.parse(package_string);
   } else {
     let response = await fetch(lockFileURL);
-    repodata = await response.json();
+    lockfile = await response.json();
   }
-  if (!repodata.packages) {
+  if (!lockfile.packages) {
     throw new Error(
-      "Loaded repodata.json does not contain the expected key 'packages'.",
+      "Loaded pyodide lock file does not contain the expected key 'packages'.",
     );
   }
-  API.repodata_info = repodata.info;
-  API.repodata_packages = repodata.packages;
+  API.repodata_info = lockfile.info;
+  API.repodata_packages = lockfile.packages;
path: src/js/load-package.ts
text: @user1:
Probably we should update the variable name in `micropip` and here together.

@author:
For compatibility reasons we could store it into both `repodata_xxx` and `lockfile_xxx` for a while.