Repository: pyodide/pyodide

Issue:
Cryptography Support
Pyodide is genius. It will certainly influence and broaden the world impact of all these data science tools and Python in so many new contexts. 

Any plan on adding 'cryptography' to the list of supported packages?

It seems many cool modules depend on it for fundamental encryption and security. All applications that would be awesome in the browser with Pyodide. 

e.g.
```console
>>> micropip.install('ccxt')
Couldn't find a pure Python 3 wheel for 'cryptography'
>>> micropip.install('alias')
Couldn't find a pure Python 3 wheel for 'cryptography'
>>> micropip.install('keycache')
Couldn't find a pure Python 3 wheel for 'cryptography'
```


Discussion:
The issue with adding the cryptography package is that it uses CFFI for C-extensions and we currently don't support CFFI (https://github.com/iodide-project/pyodide/issues/681#issuecomment-639413545). 
Is this a limitation of pyodide or would we be able to add this in with a PR?
Same applies for [`pynacl`](https://github.com/pyca/pynacl/) package. It took some time to figure out that it uses CFFI (for libsodium), and pyodide is not able to load dynamic libraries for that reason. It would be amazing to be able to use such packages.
Same for google/tink btw:
```
ValueError: Couldn't find a pure Python 3 wheel for 'tink'
```

Note that we do support cffi since #1656 was merged.
I meant the https://pypi.org/project/cffi package not the C FFI in general. I don't know what would be the practical difference in making it work as compared to ctypes, but we should try adding it as a first step then.
I see. Well libffi support was a blocker. Looking at the cffi github there's a lot of stuff that I don't understand at all. It might well be quite a complex task still.
Okay well cffi only asks for the libffi library, so there is a chance it might just work if we try to build it.
I am trying to build it and so far I am running into the problem that it is trying to link with `-lffi` but `libffi` can only be linked into the main module. Somehow need to modify config to filter out that flag or convince it not to issue it. I guess for now I can just modify pywasmcross. With that stripped out it at least builds correctly. It imports successfully too.
We're presumably going to start having trouble when cffi tries to compile things, since we don't have a C compiler.
Okay well even with the cffi blocker cleared, there is a bigger problem which is that `cryptography` has a rust/pyo3 module. Building rust modules seems to be a pretty daunting task.
It's by no means a long term solution but we could try building cryptography 3.3.2 (from Feb 7, 2021) which was before they added the rust/pyo3  dependency.
That is probably a good idea. Another concern about rust is that it generates very large build artifacts: a Rust Python module that defines a single API that returns a constant compiles to a 3.4Mb .so, though obviously the size is something we can concern ourselves with we figure out how to actually build stuff...
Perhaps I'll open a draft PR with some of my partial progress toward building rust modules. I might be getting close, a lot of problems come up though.
Next issue with cryptography is lots of `six`.
> some of my partial progress toward building rust modules.

Even if it works, and personally I like playing with Rust, I would be a bit hesitant to add it to the default build. People have already enough trouble building Pyodide as is, without adding the Rust dependency. Or at least we would need to revisit the idea that all included packages are built by default.

> a Rust Python module that defines a single API that returns a constant compiles to a 3.4Mb .so

Hmm, interesting. I was under mistaken impression that Rust would make smaller wasm binaries given all the work being done in that area. But even https://rustpython.github.io/demo/ is still 10MB compressed, which is larger than us.
>  at least we would need to revisit the idea that all included packages are built by default.

Yeah, might be helpful to divide them up by capabilities required.

> I was under mistaken impression that Rust would make smaller wasm binaries

I think they compress it very well, it just comes with TONS of extra stuff in it by default. For instance, all of the fancy panic messages. Their `print!` macro generates a whole pile of support code. Generics rapidly increase code size. In C there just aren't any constructs that generate enormous amounts of code behind the scenes. I think if you strip out all of this stuff, it might make a smaller binary but it is not clear how to do it. It also might not be fair because I made no effort at all to shrink this. 

In any case, Rust seems much harder to port than C.
Okay so next issue: we probably need to port openSSL for this.
There should be a "pure" python implementation. Did you look for such alternative?
Why? It's even possible to implement PGP in Javascript.
See   https://openpgpjs.org/
How about this route?
https://github.com/ricmoo/pyaes
@planctron Yeah that would be a reasonable approach. One would have to implement [custom backend interfaces](https://cryptography.io/en/latest/hazmat/backends/interfaces/) with pgpjs. I most likely won't do this myself but if someone is interested in contributing that would be welcome.
> @planctron Yeah that would be a reasonable approach. ..

My idea was to use **another library**, which is pure python, like _pyaes_.
Can you not use _**pyaes**_ to encrypt  instead of _crypography_?

@planctron In the original post, the rational given for wanting the cryptography package is that a large number of other packages are dependent on it (ccxt, alias, and keycache are given as examples).
Would it be possible to polyfill this module using Subtle.Crypto ?  ( https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto ) 
Perhaps this has already been considered.
> Would it be possible to polyfill this module using Subtle.Crypto ?

Thanks for the proposal! For one thing, all those methods are async which currently would be difficult to call synchronously from Python. Also doing experimental things could work for some less critical packages but doing that for cryptography is probably not a great idea. That being said people can indeed use SubtleCrypto from Python (or make a package on top of it), as long as it's not presented as an identical package to the cryptography library IMO.
Ok, good points. Brain storming; Would it be possible to wrap those async methods in promises and then use 'await' inside the wrapper methods?  ( To make them sync.)  Or compile the original cryptography library within the webassembly dependency but map the missing strong entropy to use subtle.crypto instead of whatever is causing the build to not work in webassembly?
I think the best way forward is still to build the original cryptography library version 3.4.8 (2021-08-24) before rust was added as a build requirement. To do that one would need to build openssl as a dynamic library https://github.com/pyodide/pyodide/issues/2124. As far as I can tell that's the only remaining blocker.
I think I can get cryptography to work. They actually introduced the rust dependency in v3.4.0 so we will need to use v3.3.2 until we can resolve https://github.com/rust-lang/rust/issues/93015 or hack our way around it.
Well I successfully built and imported cryptography v3.3.2, but there are some dynamic linking errors that need to be debugged.

Resolving patch:
diff --git a/pyodide-build/pyodide_build/buildpkg.py b/pyodide-build/pyodide_build/buildpkg.py
--- a/pyodide-build/pyodide_build/buildpkg.py
+++ b/pyodide-build/pyodide_build/buildpkg.py
@@ -136,6 +136,9 @@ def get_bash_runner():
             "HOSTINSTALLDIR",
             "PYMAJOR",
             "PYMINOR",
+            "CPYTHONBUILD",
+            "STDLIB_MODULE_CFLAGS",
+            "SIDE_MODULE_LDFLAGS",
         ]
     } | {"PYODIDE": "1"}
     if "PYODIDE_JOBS" in os.environ:
diff --git a/pyodide-build/pyodide_build/common.py b/pyodide-build/pyodide_build/common.py
--- a/pyodide-build/pyodide_build/common.py
+++ b/pyodide-build/pyodide_build/common.py
@@ -3,7 +3,36 @@
 from pathlib import Path
 from typing import Optional
 
-UNVENDORED_STDLIB_MODULES = ["test", "distutils"]
+UNVENDORED_STDLIB_MODULES = {"test", "distutils"}
+
+ALWAYS_PACKAGES = {
+    "pyparsing",
+    "packaging",
+    "micropip",
+}
+
+CORE_PACKAGES = {
+    "micropip",
+    "pyparsing",
+    "pytz",
+    "packaging",
+    "Jinja2",
+    "regex",
+    "fpcast-test",
+    "sharedlib-test-py",
+    "cpp-exceptions-test",
+    "ssl",
+}
+
+CORE_SCIPY_PACKAGES = {
+    "numpy",
+    "scipy",
+    "pandas",
+    "matplotlib",
+    "scikit-learn",
+    "joblib",
+    "pytest",
+}
 
 
 def _parse_package_subset(query: Optional[str]) -> set[str]:
@@ -32,35 +61,15 @@ def _parse_package_subset(query: Optional[str]) -> set[str]:
     if query is None:
         query = "core"
 
-    core_packages = {
-        "micropip",
-        "pyparsing",
-        "pytz",
-        "packaging",
-        "Jinja2",
-        "regex",
-        "fpcast-test",
-        "sharedlib-test-py",
-        "cpp-exceptions-test",
-    }
-    core_scipy_packages = {
-        "numpy",
-        "scipy",
-        "pandas",
-        "matplotlib",
-        "scikit-learn",
-        "joblib",
-        "pytest",
-    }
     packages = {el.strip() for el in query.split(",")}
-    packages.update(["pyparsing", "packaging", "micropip"])
+    packages.update(ALWAYS_PACKAGES)
     packages.update(UNVENDORED_STDLIB_MODULES)
     # handle meta-packages
     if "core" in packages:
-        packages |= core_packages
+        packages |= CORE_PACKAGES
         packages.discard("core")
     if "min-scipy-stack" in packages:
-        packages |= core_packages | core_scipy_packages
+        packages |= CORE_PACKAGES | CORE_SCIPY_PACKAGES
         packages.discard("min-scipy-stack")
 
     # Hack to deal with the circular dependence between soupsieve and


Test patch:
diff --git a/packages/cryptography/test_cryptography.py b/packages/cryptography/test_cryptography.py
new file mode 100644
--- /dev/null
+++ b/packages/cryptography/test_cryptography.py
@@ -0,0 +1,224 @@
+from hypothesis import HealthCheck, given, settings
+from hypothesis.strategies import binary, integers
+
+from conftest import selenium_context_manager
+from pyodide_build.testing import run_in_pyodide
+
+
+@run_in_pyodide(packages=["cryptography"])
+def test_cryptography():
+    import base64
+
+    from cryptography.fernet import Fernet, MultiFernet
+
+    f1 = Fernet(base64.urlsafe_b64encode(b"\x00" * 32))
+    f2 = Fernet(base64.urlsafe_b64encode(b"\x01" * 32))
+    f = MultiFernet([f1, f2])
+
+    assert f1.decrypt(f.encrypt(b"abc")) == b"abc"
+
+
+@run_in_pyodide(packages=["cryptography", "pytest"])
+def test_der_reader_basic():
+    import pytest
+    from cryptography.hazmat._der import DERReader
+
+    reader = DERReader(b"123456789")
+    assert reader.read_byte() == ord(b"1")
+    assert reader.read_bytes(1).tobytes() == b"2"
+    assert reader.read_bytes(4).tobytes() == b"3456"
+
+    with pytest.raises(ValueError):
+        reader.read_bytes(4)
+
+    assert reader.read_bytes(3).tobytes() == b"789"
+
+    # The input is now empty.
+    with pytest.raises(ValueError):
+        reader.read_bytes(1)
+    with pytest.raises(ValueError):
+        reader.read_byte()
+
+
+@run_in_pyodide(packages=["cryptography", "pytest"])
+def test_der():
+    import pytest
+    from cryptography.hazmat._der import (
+        INTEGER,
+        NULL,
+        OCTET_STRING,
+        SEQUENCE,
+        DERReader,
+        encode_der,
+        encode_der_integer,
+    )
+
+    # This input is the following structure, using
+    # https://github.com/google/der-ascii
+    #
+    # SEQUENCE {
+    #   SEQUENCE {
+    #     NULL {}
+    #     INTEGER { 42 }
+    #     OCTET_STRING { "hello" }
+    #   }
+    # }
+    der = b"\x30\x0e\x30\x0c\x05\x00\x02\x01\x2a\x04\x05\x68\x65\x6c\x6c\x6f"
+    reader = DERReader(der)
+    with pytest.raises(ValueError):
+        reader.check_empty()
+
+    with pytest.raises(ValueError):
+        with reader:
+            pass
+
+    with pytest.raises(ZeroDivisionError):
+        with DERReader(der):
+            raise ZeroDivisionError
+
+    # Parse the outer element.
+    outer = reader.read_element(SEQUENCE)
+    reader.check_empty()
+    assert outer.data.tobytes() == der[2:]
+
+    # Parse the outer element with read_any_element.
+    reader = DERReader(der)
+    tag, outer2 = reader.read_any_element()
+    reader.check_empty()
+    assert tag == SEQUENCE
+    assert outer2.data.tobytes() == der[2:]
+
+    # Parse the outer element with read_single_element.
+    outer3 = DERReader(der).read_single_element(SEQUENCE)
+    assert outer3.data.tobytes() == der[2:]
+
+    # read_single_element rejects trailing data.
+    with pytest.raises(ValueError):
+        DERReader(der + der).read_single_element(SEQUENCE)
+
+    # Continue parsing the structure.
+    inner = outer.read_element(SEQUENCE)
+    outer.check_empty()
+
+    # Parsing a missing optional element should work.
+    assert inner.read_optional_element(INTEGER) is None
+
+    null = inner.read_element(NULL)
+    null.check_empty()
+
+    # Parsing a present optional element should work.
+    integer = inner.read_optional_element(INTEGER)
+    assert integer.as_integer() == 42
+
+    octet_string = inner.read_element(OCTET_STRING)
+    assert octet_string.data.tobytes() == b"hello"
+
+    # Parsing a missing optional element should work when the input is empty.
+    inner.check_empty()
+    assert inner.read_optional_element(INTEGER) is None
+
+    # Re-encode the same structure.
+    der2 = encode_der(
+        SEQUENCE,
+        encode_der(
+            SEQUENCE,
+            encode_der(NULL),
+            encode_der(INTEGER, encode_der_integer(42)),
+            encode_der(OCTET_STRING, b"hello"),
+        ),
+    )
+    assert der2 == der
+
+
+@run_in_pyodide(packages=["cryptography"])
+def test_der_lengths():
+
+    from cryptography.hazmat._der import OCTET_STRING, DERReader, encode_der
+
+    for [length, header] in [
+        # Single-byte lengths.
+        (0, b"\x04\x00"),
+        (1, b"\x04\x01"),
+        (2, b"\x04\x02"),
+        (127, b"\x04\x7f"),
+        # Long-form lengths.
+        (128, b"\x04\x81\x80"),
+        (129, b"\x04\x81\x81"),
+        (255, b"\x04\x81\xff"),
+        (0x100, b"\x04\x82\x01\x00"),
+        (0x101, b"\x04\x82\x01\x01"),
+        (0xFFFF, b"\x04\x82\xff\xff"),
+        (0x10000, b"\x04\x83\x01\x00\x00"),
+    ]:
+        body = length * b"a"
+        der = header + body
+
+        reader = DERReader(der)
+        element = reader.read_element(OCTET_STRING)
+        reader.check_empty()
+        assert element.data.tobytes() == body
+
+        assert encode_der(OCTET_STRING, body) == der
+
+
+@settings(suppress_health_check=[HealthCheck.too_slow], deadline=None)
+@given(data=binary())
+def test_fernet(selenium_module_scope, data):
+    sbytes = list(data)
+    with selenium_context_manager(selenium_module_scope) as selenium:
+        selenium.load_package("cryptography")
+        selenium.run(
+            f"""
+            from cryptography.fernet import Fernet
+            data = bytes({sbytes})
+            f = Fernet(Fernet.generate_key())
+            ct = f.encrypt(data)
+            assert f.decrypt(ct) == data
+            """
+        )
+
+
+@settings(suppress_health_check=[HealthCheck.too_slow], deadline=None)
+@given(block_size=integers(min_value=1, max_value=255), data=binary())
+def test_pkcs7(selenium_module_scope, block_size, data):
+    sbytes = list(data)
+    with selenium_context_manager(selenium_module_scope) as selenium:
+        selenium.load_package("cryptography")
+        selenium.run(
+            f"""
+            from cryptography.hazmat.primitives.padding import ANSIX923, PKCS7
+            block_size = {block_size}
+            data = bytes({sbytes})
+            # Generate in [1, 31] so we can easily get block_size in bits by
+            # multiplying by 8.
+            p = PKCS7(block_size=block_size * 8)
+            padder = p.padder()
+            unpadder = p.unpadder()
+
+            padded = padder.update(data) + padder.finalize()
+
+            assert unpadder.update(padded) + unpadder.finalize() == data
+            """
+        )
+
+
+@settings(suppress_health_check=[HealthCheck.too_slow], deadline=None)
+@given(block_size=integers(min_value=1, max_value=255), data=binary())
+def test_ansix923(selenium_module_scope, block_size, data):
+    sbytes = list(data)
+    with selenium_context_manager(selenium_module_scope) as selenium:
+        selenium.load_package("cryptography")
+        selenium.run(
+            f"""
+            from cryptography.hazmat.primitives.padding import ANSIX923, PKCS7
+            block_size = {block_size}
+            data = bytes({sbytes})
+            a = ANSIX923(block_size=block_size * 8)
+            padder = a.padder()
+            unpadder = a.unpadder()
+
+            padded = padder.update(data) + padder.finalize()
+
+            assert unpadder.update(padded) + unpadder.finalize() == data
+            """
+        )
diff --git a/packages/ssl/test_ssl.py b/packages/ssl/test_ssl.py
new file mode 100644
--- /dev/null
+++ b/packages/ssl/test_ssl.py
@@ -0,0 +1,32 @@
+from pyodide_build.testing import run_in_pyodide
+
+
+@run_in_pyodide(packages=["test", "ssl"])
+def test_ssl():
+    import platform
+    import unittest
+    import unittest.mock
+    from test import libregrtest
+
+    platform.platform(aliased=True)
+    name = "test_ssl"
+    ignore_tests = [
+        "*test_context_custom_class*",
+        "*ThreadedTests*",
+        "*ocket*",
+        "test_verify_flags",
+        "test_subclass",
+        "test_lib_reason",
+    ]
+
+    try:
+        with unittest.mock.patch(
+            "test.support.socket_helper.bind_port",
+            side_effect=unittest.SkipTest("nope!"),
+        ):
+            libregrtest.main(
+                [name], ignore_tests=ignore_tests, verbose=True, verbose3=True
+            )
+    except SystemExit as e:
+        if e.code != 0:
+            raise RuntimeError(f"Failed with code: {e.code}")
diff --git a/pyodide-build/pyodide_build/tests/test_common.py b/pyodide-build/pyodide_build/tests/test_common.py
--- a/pyodide-build/pyodide_build/tests/test_common.py
+++ b/pyodide-build/pyodide_build/tests/test_common.py
@@ -1,4 +1,8 @@
 from pyodide_build.common import (
+    ALWAYS_PACKAGES,
+    CORE_PACKAGES,
+    CORE_SCIPY_PACKAGES,
+    UNVENDORED_STDLIB_MODULES,
     _parse_package_subset,
     get_make_environment_vars,
     get_make_flag,
@@ -6,77 +10,54 @@
 
 
 def test_parse_package_subset():
-    # micropip is always included
-    assert _parse_package_subset("numpy,pandas") == {
-        "pyparsing",
-        "packaging",
-        "micropip",
-        "numpy",
-        "pandas",
-        "test",
-        "distutils",
-    }
+    assert (
+        _parse_package_subset("numpy,pandas")
+        == {
+            "numpy",
+            "pandas",
+        }
+        | UNVENDORED_STDLIB_MODULES
+        | ALWAYS_PACKAGES
+    )
 
     # duplicates are removed
-    assert _parse_package_subset("numpy,numpy") == {
-        "pyparsing",
-        "packaging",
-        "micropip",
-        "numpy",
-        "test",
-        "distutils",
-    }
+    assert (
+        _parse_package_subset("numpy,numpy")
+        == {
+            "numpy",
+        }
+        | UNVENDORED_STDLIB_MODULES
+        | ALWAYS_PACKAGES
+    )
 
     # no empty package name included, spaces are handled
-    assert _parse_package_subset("x,  a, b, c   ,,, d,,") == {
-        "pyparsing",
-        "packaging",
-        "micropip",
-        "test",
-        "distutils",
-        "x",
-        "a",
-        "b",
-        "c",
-        "d",
-    }
+    assert (
+        _parse_package_subset("x,  a, b, c   ,,, d,,")
+        == {
+            "x",
+            "a",
+            "b",
+            "c",
+            "d",
+        }
+        | UNVENDORED_STDLIB_MODULES
+        | ALWAYS_PACKAGES
+    )
 
-    assert _parse_package_subset("core") == {
-        "pyparsing",
-        "packaging",
-        "pytz",
-        "Jinja2",
-        "micropip",
-        "regex",
-        "fpcast-test",
-        "test",
-        "distutils",
-        "sharedlib-test-py",
-        "cpp-exceptions-test",
-    }
+    assert (
+        _parse_package_subset("core")
+        == CORE_PACKAGES | UNVENDORED_STDLIB_MODULES | ALWAYS_PACKAGES
+    )
     # by default core packages are built
     assert _parse_package_subset(None) == _parse_package_subset("core")
 
-    assert _parse_package_subset("min-scipy-stack") == {
-        "pyparsing",
-        "packaging",
-        "pytz",
-        "Jinja2",
-        "micropip",
-        "regex",
-        "fpcast-test",
-        "numpy",
-        "scipy",
-        "pandas",
-        "matplotlib",
-        "scikit-learn",
-        "joblib",
-        "pytest",
-        "test",
-        "distutils",
-        "sharedlib-test-py",
-        "cpp-exceptions-test",
-    }
+    assert (
+        _parse_package_subset("min-scipy-stack")
+        == CORE_SCIPY_PACKAGES
+        | CORE_PACKAGES
+        | UNVENDORED_STDLIB_MODULES
+        | ALWAYS_PACKAGES
+    )
     # reserved key words can be combined with other packages
     assert _parse_package_subset("core, unknown") == _parse_package_subset("core") | {
         "unknown"
diff --git a/src/tests/test_core_python.py b/src/tests/test_core_python.py
--- a/src/tests/test_core_python.py
+++ b/src/tests/test_core_python.py
@@ -30,7 +30,7 @@ def test_cpython_core(python_test, selenium, request):
         else:
             pytest.xfail('known failure with code "{}"'.format(",".join(error_flags)))
 
-    selenium.load_package(UNVENDORED_STDLIB_MODULES)
+    selenium.load_package(list(UNVENDORED_STDLIB_MODULES))
     try:
         selenium.run(
             """