Repository: pyodide/pyodide

Issue:
Improve `pywasmcross` to handle response files
@ryanking13 mentioned [here](https://github.com/pyodide/pyodide/issues/3427#issuecomment-1374418990) that:

> pywasmcross calculates what symbols need to be exported by looking at objects files (.a, .o). But it doesn't handle response files (.rsp). So we need to add a logic to parse response files there.

I actually didn't even know what a response file was, so I just asked ChatGPT to do it for me, and here's what it says:

---

A response file is a file that contains a list of arguments that can be passed to a command line tool. For example, suppose we have a command line tool called "tool" that accepts a list of input files and processes them. Instead of typing out the long list of input files every time we want to run the tool, we can put the list of input files in a response file and pass the response file to the tool using the "@" symbol. For example:
```
tool @response_file.rsp
```
The contents of response_file.rsp might look something like this:
```
input_file1.txt
input_file2.txt
input_file3.txt
```
To handle response files in the `calculate_exports` function, we can modify the function to parse the response file and extract the list of arguments contained within it. Here is an example of how this could be done:

```py
def calculate_exports(line: list[str], export_all: bool) -> Iterable[str]:
    """
    Collect up all the object files and archive files being linked and list out
    symbols in them that are marked as public. If ``export_all`` is ``True``,
    then return all public symbols. If not, return only the public symbols that
    begin with `PyInit`.
    """
    # Parse response files and extract list of arguments
    objects = []
    for arg in line:
        if arg.endswith(".rsp"):
            with open(arg, "r") as f:
                objects.extend(f.read().splitlines())
        else:
            objects.append(arg)
    
    # Filter objects to only include .a and .o files
    objects = [obj for obj in objects if obj.endswith((".a", ".o"))]
    
    exports = None
    # Using emnm is simpler but it cannot handle bitcode. If we're only
    # exporting the PyInit symbols, save effort by using nm.
    if export_all:
        exports = calculate_object_exports_readobj(objects)
    if exports is None:
        # Either export_all is false or we are linking at least one bitcode
        # object. Fall back to a more conservative estimate of the symbols
        # exported. This can export things with `__visibility__("hidden")`
        exports = calculate_object_exports_nm(objects)
    if export_all:
        return exports
    return (x for x in exports if x.startswith("PyInit"))
```

---

Seems simple enough? It just replaced [this line](https://github.com/pyodide/pyodide/blob/aeb066bbcef310a419bf058347bbc4d23ca52ec1/pyodide-build/pyodide_build/pywasmcross.py#L501):

```py
   objects = [arg for arg in line if arg.endswith((".a", ".o"))]
```

with this:

```py
    # Parse response files and extract list of arguments
    objects = []
    for arg in line:
        if arg.endswith(".rsp"):
            with open(arg, "r") as f:
                objects.extend(f.read().splitlines())
        else:
            objects.append(arg)
    
    # Filter objects to only include .a and .o files
    objects = [obj for obj in objects if obj.endswith((".a", ".o"))]
```


Discussion:

Resolving patch:
diff --git a/pyodide-build/pyodide_build/pywasmcross.py b/pyodide-build/pyodide_build/pywasmcross.py
--- a/pyodide-build/pyodide_build/pywasmcross.py
+++ b/pyodide-build/pyodide_build/pywasmcross.py
@@ -415,14 +415,27 @@ def calculate_object_exports_nm(objects: list[str]) -> list[str]:
     return result.stdout.splitlines()
 
 
+def filter_objects(line: list[str]) -> list[str]:
+    """
+    Collect up all the object files and archive files being linked.
+    """
+    return [
+        arg
+        for arg in line
+        if arg.endswith((".a", ".o"))
+        or arg.startswith(
+            "@"
+        )  # response file (https://gcc.gnu.org/wiki/Response_Files)
+    ]
+
+
 def calculate_exports(line: list[str], export_all: bool) -> Iterable[str]:
     """
-    Collect up all the object files and archive files being linked and list out
-    symbols in them that are marked as public. If ``export_all`` is ``True``,
-    then return all public symbols. If not, return only the public symbols that
-    begin with `PyInit`.
+    List out symbols from object files and archive files that are marked as public.
+    If ``export_all`` is ``True``, then return all public symbols.
+    If not, return only the public symbols that begin with `PyInit`.
     """
-    objects = [arg for arg in line if arg.endswith((".a", ".o"))]
+    objects = filter_objects(line)
     exports = None
     # Using emnm is simpler but it cannot handle bitcode. If we're only
     # exporting the PyInit symbols, save effort by using nm.


Test patch:
diff --git a/pyodide-build/pyodide_build/tests/test_pywasmcross.py b/pyodide-build/pyodide_build/tests/test_pywasmcross.py
--- a/pyodide-build/pyodide_build/tests/test_pywasmcross.py
+++ b/pyodide-build/pyodide_build/tests/test_pywasmcross.py
@@ -5,6 +5,7 @@
 from pyodide_build.pywasmcross import (
     BuildArgs,
     calculate_exports,
+    filter_objects,
     get_cmake_compiler_flags,
     get_library_output,
     handle_command_generate_args,
@@ -186,6 +187,33 @@ def test_conda_unsupported_args(build_args):
     )
 
 
+@pytest.mark.parametrize(
+    "line, expected",
+    [
+        ([], []),
+        (
+            [
+                "obj1.o",
+                "obj2.o",
+                "slib1.so",
+                "slib2.so",
+                "lib1.a",
+                "lib2.a",
+                "-o",
+                "test.so",
+            ],
+            ["obj1.o", "obj2.o", "lib1.a", "lib2.a"],
+        ),
+        (
+            ["@dir/link.txt", "obj1.o", "obj2.o", "test.so"],
+            ["@dir/link.txt", "obj1.o", "obj2.o"],
+        ),
+    ],
+)
+def test_filter_objects(line, expected):
+    assert filter_objects(line) == expected
+
+
 @pytest.mark.xfail(reason="FIXME: emcc is not available during test")
 def test_exports_node(tmp_path):
     template = """