Repository: pyodide/pyodide

Issue:
Building a Python package that uses Rust
## 🚀 Feature
Tooling and/or docs to support building Python packages that contain Rust code.

### Motivation
I was in the process of (attempting to) create a JS/wasm build of [Hugging Face Tokenizers](https://github.com/huggingface/tokenizers/) when I realised that it might just be easier/better for my use case to load it with Pyodide (especially since I might want to use Python packages that depend upon Tokenizers. The [docs show](https://github.com/pyodide/pyodide/blob/main/docs/development/new-packages.md) how to build a Python package that uses C (with emscripten), but not Rust.

Rust has great, simple wasm build support via [`wasm-pack`](https://github.com/rustwasm/wasm-pack). After annotating the Rust functions/modules that you want to export (and installing wasm pack with `cargo install wasm-pack`) it's just a single `wasm-pack build --target=web --out-dir=wasm` to get the wasm and glue files. [Here's a hello world tutorial](https://developer.mozilla.org/en-US/docs/WebAssembly/Rust_to_wasm).

So I'm wondering: Is this feasible now with the right configuration, or is there some extra work on the Pyodide dev tooling to get this working? If it's not possible with the current tooling, what might be the steps required to make it possible?

Thanks!


Discussion:
Thanks for proposing it! There was an earlier analysis of feasibility in https://github.com/pyodide/pyodide/issues/1282#issuecomment-785459008 and it doesn't look very promising. 

Maybe a start could be trying to build a very simple PyO3 Python library. Related https://github.com/PyO3/pyo3/issues/1522

Even if the end we manage to compile that library say for RustPython using the wasm-pack, you end up with a Python extension built for a different Python interpreter and the compatibility story there is not very good. 
I wonder if it could be possible to convince Rust to emit llvm ir and then use emcc to do the backend compilation and linkage. Rust also has a wasm32-emscripten target but it's apparently pretty broken.
@hoodmane @rth Thanks for your tips! Having looked into this, it seems to be a bit over my head at my current level of knowledge. That said, it would be great if I could make a dent in at least part of the problem.

My understanding is that if we get a simple Python+Rust library compiled with PyO3 (with the Rust converted to wasm), then we'll be able to swap out Python with Pyodide? But from reading some of those Github issues sounds like PyO3 may create different interfaces compared to what Pyodide expects? Would someone be able to list the rough high-level problems that would likely need to be solved here?

Question: Since it should be pretty easy to compile the Rust code to wasm (+ js glue) with wasm pack (at least, going by my past experiences with wasm pack), I was wondering whether it would be possible to then import that JS module into Python/Pyodide in such a way that Pyodide essentially sees that JS module as it would the PyO3 Python module? Or would there be "interface" issues due to diferences in the way wasm pack and PyO3 generate glue code?

> I wonder if it could be possible to convince Rust to emit llvm ir

[The answer here](https://stackoverflow.com/questions/39004513/how-to-emit-llvm-ir-from-cargo) says you can emit llvm-ir with `cargo rustc -- --emit=llvm-ir`


>  it seems to be a bit over my head at my current level of knowledge

One of the fun parts of Pyodide is that a lot of it is over our heads, at least until we figure it out =)

> Maybe a start could be trying to build a very simple PyO3 Python library.

I think perhaps even better might be if we could write a trivial rust extension without using PyO3. I just tried to do this for a while, but I keep getting errors at import time.
Here's my partial work on this:
https://github.com/hoodmane/minimal-rust-pyton-ext/blob/master/src/lib.rs
You can build with `cargo build` and then use
`cp target/debug/libpython_min_test.so mymod.cpython-39-x86_64-linux-gnu.so` 
then can try to `import mymod` but there are errors.

Anyways, if we can get a module like this to work, then we could try to compile it to work with Pyodide.

Resolving patch:
diff --git a/cpython/adjust_sysconfig.py b/cpython/adjust_sysconfig.py
--- a/cpython/adjust_sysconfig.py
+++ b/cpython/adjust_sysconfig.py
@@ -23,7 +23,7 @@ def adjust_sysconfig(config_vars: dict[str, str]):
         MAINCC="cc",
         LDSHARED="cc",
         LINKCC="cc",
-        BLDSHARED="cc",
+        BLDSHARED="emcc -sSIDE_MODULE=1",  # setuptools-rust looks at this
         CXX="c++",
         LDCXXSHARED="c++",
     )
diff --git a/pyodide-build/pyodide_build/buildpkg.py b/pyodide-build/pyodide_build/buildpkg.py
--- a/pyodide-build/pyodide_build/buildpkg.py
+++ b/pyodide-build/pyodide_build/buildpkg.py
@@ -135,6 +135,7 @@ def get_bash_runner():
             "PYTHONINCLUDE",
             "NUMPY_LIB",
             "PYODIDE_PACKAGE_ABI",
+            "HOME",
             "HOSTINSTALLDIR",
             "TARGETINSTALLDIR",
             "SYSCONFIG_NAME",
@@ -149,6 +150,11 @@ def get_bash_runner():
             "UNISOLATED_PACKAGES",
             "WASM_LIBRARY_DIR",
             "WASM_PKG_CONFIG_PATH",
+            "CARGO_BUILD_TARGET",
+            "CARGO_HOME",
+            "CARGO_TARGET_WASM32_UNKNOWN_EMSCRIPTEN_LINKER",
+            "RUSTFLAGS",
+            "PYO3_CONFIG_FILE",
         ]
     } | {"PYODIDE": "1"}
     if "PYODIDE_JOBS" in os.environ:
diff --git a/pyodide-build/pyodide_build/pywasmcross.py b/pyodide-build/pyodide_build/pywasmcross.py
--- a/pyodide-build/pyodide_build/pywasmcross.py
+++ b/pyodide-build/pyodide_build/pywasmcross.py
@@ -10,6 +10,7 @@
 """
 import json
 import os
+import shutil
 import sys
 
 IS_MAIN = __name__ == "__main__"
@@ -30,7 +31,7 @@
 from pyodide_build import common
 from pyodide_build._f2c_fixes import fix_f2c_input, fix_f2c_output, scipy_fixes
 
-symlinks = {"cc", "c++", "ld", "ar", "gcc", "gfortran"}
+symlinks = {"cc", "c++", "ld", "ar", "gcc", "gfortran", "cargo"}
 
 
 def symlink_dir():
@@ -537,6 +538,13 @@ def handle_command(
     if returncode != 0:
         sys.exit(returncode)
 
+    # Rust gives output files a `.wasm` suffix, but we need them to have a `.so`
+    # suffix.
+    if line[0:2] == ["cargo", "rustc"]:
+        p = Path(args.builddir)
+        for x in p.glob("**/*.wasm"):
+            shutil.move(x, x.with_suffix(".so"))
+
     sys.exit(returncode)
 
 


Test patch:
diff --git a/packages/cryptography/test_cryptography.py b/packages/cryptography/test_cryptography.py
--- a/packages/cryptography/test_cryptography.py
+++ b/packages/cryptography/test_cryptography.py
@@ -17,149 +17,6 @@ def test_cryptography(selenium):
     assert f1.decrypt(f.encrypt(b"abc")) == b"abc"
 
 
-@run_in_pyodide(packages=["cryptography", "pytest"])
-def test_der_reader_basic(selenium):
-    import pytest
-    from cryptography.hazmat._der import DERReader
-
-    reader = DERReader(b"123456789")
-    assert reader.read_byte() == ord(b"1")
-    assert reader.read_bytes(1).tobytes() == b"2"
-    assert reader.read_bytes(4).tobytes() == b"3456"
-
-    with pytest.raises(ValueError):
-        reader.read_bytes(4)
-
-    assert reader.read_bytes(3).tobytes() == b"789"
-
-    # The input is now empty.
-    with pytest.raises(ValueError):
-        reader.read_bytes(1)
-    with pytest.raises(ValueError):
-        reader.read_byte()
-
-
-@run_in_pyodide(packages=["cryptography", "pytest"])
-def test_der(selenium):
-    import pytest
-    from cryptography.hazmat._der import (
-        INTEGER,
-        NULL,
-        OCTET_STRING,
-        SEQUENCE,
-        DERReader,
-        encode_der,
-        encode_der_integer,
-    )
-
-    # This input is the following structure, using
-    # https://github.com/google/der-ascii
-    #
-    # SEQUENCE {
-    #   SEQUENCE {
-    #     NULL {}
-    #     INTEGER { 42 }
-    #     OCTET_STRING { "hello" }
-    #   }
-    # }
-    der = b"\x30\x0e\x30\x0c\x05\x00\x02\x01\x2a\x04\x05\x68\x65\x6c\x6c\x6f"
-    reader = DERReader(der)
-    with pytest.raises(ValueError):
-        reader.check_empty()
-
-    with pytest.raises(ValueError):
-        with reader:
-            pass
-
-    with pytest.raises(ZeroDivisionError):
-        with DERReader(der):
-            raise ZeroDivisionError
-
-    # Parse the outer element.
-    outer = reader.read_element(SEQUENCE)
-    reader.check_empty()
-    assert outer.data.tobytes() == der[2:]
-
-    # Parse the outer element with read_any_element.
-    reader = DERReader(der)
-    tag, outer2 = reader.read_any_element()
-    reader.check_empty()
-    assert tag == SEQUENCE
-    assert outer2.data.tobytes() == der[2:]
-
-    # Parse the outer element with read_single_element.
-    outer3 = DERReader(der).read_single_element(SEQUENCE)
-    assert outer3.data.tobytes() == der[2:]
-
-    # read_single_element rejects trailing data.
-    with pytest.raises(ValueError):
-        DERReader(der + der).read_single_element(SEQUENCE)
-
-    # Continue parsing the structure.
-    inner = outer.read_element(SEQUENCE)
-    outer.check_empty()
-
-    # Parsing a missing optional element should work.
-    assert inner.read_optional_element(INTEGER) is None
-
-    null = inner.read_element(NULL)
-    null.check_empty()
-
-    # Parsing a present optional element should work.
-    integer = inner.read_optional_element(INTEGER)
-    assert integer.as_integer() == 42
-
-    octet_string = inner.read_element(OCTET_STRING)
-    assert octet_string.data.tobytes() == b"hello"
-
-    # Parsing a missing optional element should work when the input is empty.
-    inner.check_empty()
-    assert inner.read_optional_element(INTEGER) is None
-
-    # Re-encode the same structure.
-    der2 = encode_der(
-        SEQUENCE,
-        encode_der(
-            SEQUENCE,
-            encode_der(NULL),
-            encode_der(INTEGER, encode_der_integer(42)),
-            encode_der(OCTET_STRING, b"hello"),
-        ),
-    )
-    assert der2 == der
-
-
-@run_in_pyodide(packages=["cryptography"])
-def test_der_lengths(selenium):
-
-    from cryptography.hazmat._der import OCTET_STRING, DERReader, encode_der
-
-    for [length, header] in [
-        # Single-byte lengths.
-        (0, b"\x04\x00"),
-        (1, b"\x04\x01"),
-        (2, b"\x04\x02"),
-        (127, b"\x04\x7f"),
-        # Long-form lengths.
-        (128, b"\x04\x81\x80"),
-        (129, b"\x04\x81\x81"),
-        (255, b"\x04\x81\xff"),
-        (0x100, b"\x04\x82\x01\x00"),
-        (0x101, b"\x04\x82\x01\x01"),
-        (0xFFFF, b"\x04\x82\xff\xff"),
-        (0x10000, b"\x04\x83\x01\x00\x00"),
-    ]:
-        body = length * b"a"
-        der = header + body
-
-        reader = DERReader(der)
-        element = reader.read_element(OCTET_STRING)
-        reader.check_empty()
-        assert element.data.tobytes() == body
-
-        assert encode_der(OCTET_STRING, body) == der
-
-
 @settings(suppress_health_check=[HealthCheck.too_slow], deadline=None)
 @given(data=binary())
 def test_fernet(selenium_module_scope, data):