Repository: pyodide/pyodide
Pull request: Add cryptography, openssl, and _ssl packages
Resolves #761.

ssl passes 46/168 tests. 108 fail due to inability to start a socket and another 14 failing with attribute errors due to missing library functions.

### Checklists

- [x] Add a [CHANGELOG](https://github.com/pyodide/pyodide/blob/main/docs/project/changelog.md) entry
- [x] Add / update tests


Problem statement:
Cryptography Support
Pyodide is genius. It will certainly influence and broaden the world impact of all these data science tools and Python in so many new contexts. 

Any plan on adding 'cryptography' to the list of supported packages?

It seems many cool modules depend on it for fundamental encryption and security. All applications that would be awesome in the browser with Pyodide. 

e.g.
```console
>>> micropip.install('ccxt')
Couldn't find a pure Python 3 wheel for 'cryptography'
>>> micropip.install('alias')
Couldn't find a pure Python 3 wheel for 'cryptography'
>>> micropip.install('keycache')
Couldn't find a pure Python 3 wheel for 'cryptography'
```

Hints from issue discussion:
The issue with adding the cryptography package is that it uses CFFI for C-extensions and we currently don't support CFFI (https://github.com/iodide-project/pyodide/issues/681#issuecomment-639413545). 
Is this a limitation of pyodide or would we be able to add this in with a PR?
Same applies for [`pynacl`](https://github.com/pyca/pynacl/) package. It took some time to figure out that it uses CFFI (for libsodium), and pyodide is not able to load dynamic libraries for that reason. It would be amazing to be able to use such packages.
Same for google/tink btw:
```
ValueError: Couldn't find a pure Python 3 wheel for 'tink'
```

Note that we do support cffi since #1656 was merged.
I meant the https://pypi.org/project/cffi package not the C FFI in general. I don't know what would be the practical difference in making it work as compared to ctypes, but we should try adding it as a first step then.
I see. Well libffi support was a blocker. Looking at the cffi github there's a lot of stuff that I don't understand at all. It might well be quite a complex task still.
Okay well cffi only asks for the libffi library, so there is a chance it might just work if we try to build it.
I am trying to build it and so far I am running into the problem that it is trying to link with `-lffi` but `libffi` can only be linked into the main module. Somehow need to modify config to filter out that flag or convince it not to issue it. I guess for now I can just modify pywasmcross. With that stripped out it at least builds correctly. It imports successfully too.
We're presumably going to start having trouble when cffi tries to compile things, since we don't have a C compiler.

Commit message: Make ssl as part of core build

Patch to review:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index aeeb8db2577..2e5af905828 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -74,7 +74,7 @@ jobs:
             source pyodide_env.sh
 
             ccache -z
-            PYODIDE_PACKAGES="core" make
+            PYODIDE_PACKAGES="core, ssl" make
             ccache -s
 
       - run:
diff --git a/Makefile.envs b/Makefile.envs
index dec3170dff2..ccf75bf5be2 100644
--- a/Makefile.envs
+++ b/Makefile.envs
@@ -19,6 +19,7 @@ export HOSTPYTHON=$(HOSTPYTHONROOT)/bin/python$(PYMAJOR).$(PYMINOR)
 
 export CPYTHONROOT=$(PYODIDE_ROOT)/cpython
 export CPYTHONLIB=$(CPYTHONROOT)/installs/python-$(PYVERSION)/lib/python$(PYMAJOR).$(PYMINOR)
+export CPYTHONBUILD=$(PYODIDE_ROOT)/cpython/build/Python-$(PYVERSION)
 
 export TARGETINSTALLDIR=$(PYODIDE_ROOT)/cpython/installs/python-$(PYVERSION)
 export HOSTINSTALLDIR=$(PYODIDE_ROOT)/packages/.artifacts
@@ -109,6 +110,7 @@ export MAIN_MODULE_CFLAGS= $(CFLAGS_BASE) \
 	-I$(PYTHONINCLUDE) \
 	-s EXCEPTION_CATCHING_ALLOWED=['we only want to allow exception handling in side modules']
 
+export STDLIB_MODULE_CFLAGS= $(SIDE_MODULE_CFLAGS) -I Include/ -I .
 
 .output_vars:
 	set
diff --git a/docs/project/changelog.md b/docs/project/changelog.md
index 74c4bb4c9ec..1d72d51eb23 100644
--- a/docs/project/changelog.md
+++ b/docs/project/changelog.md
@@ -17,7 +17,7 @@ substitutions:
 ### Packages
 
 - New packages: sqlalchemy {pr}`2112`, pydantic {pr}`2117`, wrapt {pr}`2165`,
-  boost-histogram {pr}`2174`
+  boost-histogram {pr}`2174`, cryptography v3.3.2 {pr}`2263`, the standard library ssl module {pr}`2263`
 
 - Upgraded packages: distlib (0.3.4), lxml (4.8.0) {pr}`2239`,
 
diff --git a/packages/cryptography/meta.yaml b/packages/cryptography/meta.yaml
new file mode 100644
index 00000000000..349a0fe0e17
--- /dev/null
+++ b/packages/cryptography/meta.yaml
@@ -0,0 +1,25 @@
+package:
+  name: cryptography
+  version: 3.3.2
+
+source:
+  url: https://files.pythonhosted.org/packages/d4/85/38715448253404186029c575d559879912eb8a1c5d16ad9f25d35f7c4f4c/cryptography-3.3.2.tar.gz
+  sha256: 5a60d3780149e13b7a6ff7ad6526b38846354d11a15e21068e57073e29e19bed
+build:
+  cflags: |
+    -I$(PYODIDE_ROOT)/packages/openssl/build/openssl-1.1.1m/include/
+    -Wno-implicit-function-declaration
+  ldflags: |
+    -L$(PYODIDE_ROOT)/packages/openssl/build/openssl-1.1.1m/dist/
+requirements:
+  run:
+    - openssl
+    - six
+    - cffi
+test:
+  imports:
+    - cryptography
+    - cryptography.fernet
+    - cryptography.hazmat
+    - cryptography.utils
+    - cryptography.x509
diff --git a/packages/cryptography/test_cryptography.py b/packages/cryptography/test_cryptography.py
new file mode 100644
index 00000000000..59fec71e7c8
--- /dev/null
+++ b/packages/cryptography/test_cryptography.py
@@ -0,0 +1,224 @@
+from hypothesis import HealthCheck, given, settings
+from hypothesis.strategies import binary, integers
+
+from conftest import selenium_context_manager
+from pyodide_build.testing import run_in_pyodide
+
+
+@run_in_pyodide(packages=["cryptography"])
+def test_cryptography():
+    import base64
+
+    from cryptography.fernet import Fernet, MultiFernet
+
+    f1 = Fernet(base64.urlsafe_b64encode(b"\x00" * 32))
+    f2 = Fernet(base64.urlsafe_b64encode(b"\x01" * 32))
+    f = MultiFernet([f1, f2])
+
+    assert f1.decrypt(f.encrypt(b"abc")) == b"abc"
+
+
+@run_in_pyodide(packages=["cryptography", "pytest"])
+def test_der_reader_basic():
+    import pytest
+    from cryptography.hazmat._der import DERReader
+
+    reader = DERReader(b"123456789")
+    assert reader.read_byte() == ord(b"1")
+    assert reader.read_bytes(1).tobytes() == b"2"
+    assert reader.read_bytes(4).tobytes() == b"3456"
+
+    with pytest.raises(ValueError):
+        reader.read_bytes(4)
+
+    assert reader.read_bytes(3).tobytes() == b"789"
+
+    # The input is now empty.
+    with pytest.raises(ValueError):
+        reader.read_bytes(1)
+    with pytest.raises(ValueError):
+        reader.read_byte()
+
+
+@run_in_pyodide(packages=["cryptography", "pytest"])
+def test_der():
+    import pytest
+    from cryptography.hazmat._der import (
+        INTEGER,
+        NULL,
+        OCTET_STRING,
+        SEQUENCE,
+        DERReader,
+        encode_der,
+        encode_der_integer,
+    )
+
+    # This input is the following structure, using
+    # https://github.com/google/der-ascii
+    #
+    # SEQUENCE {
+    #   SEQUENCE {
+    #     NULL {}
+    #     INTEGER { 42 }
+    #     OCTET_STRING { "hello" }
+    #   }
+    # }
+    der = b"\x30\x0e\x30\x0c\x05\x00\x02\x01\x2a\x04\x05\x68\x65\x6c\x6c\x6f"
+    reader = DERReader(der)
+    with pytest.raises(ValueError):
+        reader.check_empty()
+
+    with pytest.raises(ValueError):
+        with reader:
+            pass
+
+    with pytest.raises(ZeroDivisionError):
+        with DERReader(der):
+            raise ZeroDivisionError
+
+    # Parse the outer element.
+    outer = reader.read_element(SEQUENCE)
+    reader.check_empty()
+    assert outer.data.tobytes() == der[2:]
+
+    # Parse the outer element with read_any_element.
+    reader = DERReader(der)
+    tag, outer2 = reader.read_any_element()
+    reader.check_empty()
+    assert tag == SEQUENCE
+    assert outer2.data.tobytes() == der[2:]
+
+    # Parse the outer element with read_single_element.
+    outer3 = DERReader(der).read_single_element(SEQUENCE)
+    assert outer3.data.tobytes() == der[2:]
+
+    # read_single_element rejects trailing data.
+    with pytest.raises(ValueError):
+        DERReader(der + der).read_single_element(SEQUENCE)
+
+    # Continue parsing the structure.
+    inner = outer.read_element(SEQUENCE)
+    outer.check_empty()
+
+    # Parsing a missing optional element should work.
+    assert inner.read_optional_element(INTEGER) is None
+
+    null = inner.read_element(NULL)
+    null.check_empty()
+
+    # Parsing a present optional element should work.
+    integer = inner.read_optional_element(INTEGER)
+    assert integer.as_integer() == 42
+
+    octet_string = inner.read_element(OCTET_STRING)
+    assert octet_string.data.tobytes() == b"hello"
+
+    # Parsing a missing optional element should work when the input is empty.
+    inner.check_empty()
+    assert inner.read_optional_element(INTEGER) is None
+
+    # Re-encode the same structure.
+    der2 = encode_der(
+        SEQUENCE,
+        encode_der(
+            SEQUENCE,
+            encode_der(NULL),
+            encode_der(INTEGER, encode_der_integer(42)),
+            encode_der(OCTET_STRING, b"hello"),
+        ),
+    )
+    assert der2 == der
+
+
+@run_in_pyodide(packages=["cryptography"])
+def test_der_lengths():
+
+    from cryptography.hazmat._der import OCTET_STRING, DERReader, encode_der
+
+    for [length, header] in [
+        # Single-byte lengths.
+        (0, b"\x04\x00"),
+        (1, b"\x04\x01"),
+        (2, b"\x04\x02"),
+        (127, b"\x04\x7f"),
+        # Long-form lengths.
+        (128, b"\x04\x81\x80"),
+        (129, b"\x04\x81\x81"),
+        (255, b"\x04\x81\xff"),
+        (0x100, b"\x04\x82\x01\x00"),
+        (0x101, b"\x04\x82\x01\x01"),
+        (0xFFFF, b"\x04\x82\xff\xff"),
+        (0x10000, b"\x04\x83\x01\x00\x00"),
+    ]:
+        body = length * b"a"
+        der = header + body
+
+        reader = DERReader(der)
+        element = reader.read_element(OCTET_STRING)
+        reader.check_empty()
+        assert element.data.tobytes() == body
+
+        assert encode_der(OCTET_STRING, body) == der
+
+
+@settings(suppress_health_check=[HealthCheck.too_slow], deadline=None)
+@given(data=binary())
+def test_fernet(selenium_module_scope, data):
+    sbytes = list(data)
+    with selenium_context_manager(selenium_module_scope) as selenium:
+        selenium.load_package("cryptography")
+        selenium.run(
+            f"""
+            from cryptography.fernet import Fernet
+            data = bytes({sbytes})
+            f = Fernet(Fernet.generate_key())
+            ct = f.encrypt(data)
+            assert f.decrypt(ct) == data
+            """
+        )
+
+
+@settings(suppress_health_check=[HealthCheck.too_slow], deadline=None)
+@given(block_size=integers(min_value=1, max_value=255), data=binary())
+def test_pkcs7(selenium_module_scope, block_size, data):
+    sbytes = list(data)
+    with selenium_context_manager(selenium_module_scope) as selenium:
+        selenium.load_package("cryptography")
+        selenium.run(
+            f"""
+            from cryptography.hazmat.primitives.padding import ANSIX923, PKCS7
+            block_size = {block_size}
+            data = bytes({sbytes})
+            # Generate in [1, 31] so we can easily get block_size in bits by
+            # multiplying by 8.
+            p = PKCS7(block_size=block_size * 8)
+            padder = p.padder()
+            unpadder = p.unpadder()
+
+            padded = padder.update(data) + padder.finalize()
+
+            assert unpadder.update(padded) + unpadder.finalize() == data
+            """
+        )
+
+
+@settings(suppress_health_check=[HealthCheck.too_slow], deadline=None)
+@given(block_size=integers(min_value=1, max_value=255), data=binary())
+def test_ansix923(selenium_module_scope, block_size, data):
+    sbytes = list(data)
+    with selenium_context_manager(selenium_module_scope) as selenium:
+        selenium.load_package("cryptography")
+        selenium.run(
+            f"""
+            from cryptography.hazmat.primitives.padding import ANSIX923, PKCS7
+            block_size = {block_size}
+            data = bytes({sbytes})
+            a = ANSIX923(block_size=block_size * 8)
+            padder = a.padder()
+            unpadder = a.unpadder()
+
+            padded = padder.update(data) + padder.finalize()
+
+            assert unpadder.update(padded) + unpadder.finalize() == data
+            """
+        )
diff --git a/packages/openssl/meta.yaml b/packages/openssl/meta.yaml
new file mode 100644
index 00000000000..acba459dcbf
--- /dev/null
+++ b/packages/openssl/meta.yaml
@@ -0,0 +1,19 @@
+package:
+  name: openssl
+  version: 1.1.1m
+
+source:
+  url: https://www.openssl.org/source/openssl-1.1.1m.tar.gz
+  sha256: f89199be8b23ca45fc7cb9f1d8d3ee67312318286ad030f5316aca6462db6c96
+build:
+  sharedlibrary: true
+  script: |
+    emconfigure ./Configure gcc -no-ui-console -DHAVE_FORK=0 -DOPENSSL_NO_SECURE_MEMORY -DNO_SYSLOG -fPIC
+    sed -i 's!^CROSS_COMPILE=.*!!g' Makefile
+    make build_generated
+    make -j ${PYODIDE_JOBS:-3} libcrypto.a
+    make -j ${PYODIDE_JOBS:-3} libssl.a
+    emar -d libcrypto.a liblegacy-lib-bn_asm.o liblegacy-lib-des_enc.o liblegacy-lib-fcrypt_b.o
+    mkdir dist
+    emcc -sSIDE_MODULE=1 libcrypto.a -o dist/libcrypto.so
+    emcc -sSIDE_MODULE=1 libssl.a -o dist/libssl.so
diff --git a/packages/ssl/empty/.keep b/packages/ssl/empty/.keep
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/packages/ssl/meta.yaml b/packages/ssl/meta.yaml
new file mode 100644
index 00000000000..c8e9da7d0cd
--- /dev/null
+++ b/packages/ssl/meta.yaml
@@ -0,0 +1,21 @@
+package:
+  name: ssl
+  version: 1.0.0 # Nonesense
+source:
+  path: empty
+build:
+  sharedlibrary: true
+  script: |
+    mkdir dist
+    export DISTDIR=$(pwd)/dist
+    cd $CPYTHONBUILD
+    emcc $STDLIB_MODULE_CFLAGS -c Modules/socketmodule.c -o Modules/socketmodule.o
+    emcc $STDLIB_MODULE_CFLAGS -c Modules/_ssl.c -o Modules/_ssl.o -I$PYODIDE_ROOT/packages/openssl/build/openssl-1.1.1m/include/ -DOPENSSL_THREADS
+    emcc Modules/_ssl.o -o $DISTDIR/_ssl.so $SIDE_MODULE_LDFLAGS
+    emcc Modules/socketmodule.o -o $DISTDIR/socketmodule.so $SIDE_MODULE_LDFLAGS
+requirements:
+  run:
+    - openssl
+test:
+  imports:
+    - ssl
diff --git a/packages/ssl/test_ssl.py b/packages/ssl/test_ssl.py
new file mode 100644
index 00000000000..e05ebc9cd39
--- /dev/null
+++ b/packages/ssl/test_ssl.py
@@ -0,0 +1,32 @@
+from pyodide_build.testing import run_in_pyodide
+
+
+@run_in_pyodide(packages=["test", "ssl"])
+def test_ssl():
+    import platform
+    import unittest
+    import unittest.mock
+    from test import libregrtest
+
+    platform.platform(aliased=True)
+    name = "test_ssl"
+    ignore_tests = [
+        "*test_context_custom_class*",
+        "*ThreadedTests*",
+        "*ocket*",
+        "test_verify_flags",
+        "test_subclass",
+        "test_lib_reason",
+    ]
+
+    try:
+        with unittest.mock.patch(
+            "test.support.socket_helper.bind_port",
+            side_effect=unittest.SkipTest("nope!"),
+        ):
+            libregrtest.main(
+                [name], ignore_tests=ignore_tests, verbose=True, verbose3=True
+            )
+    except SystemExit as e:
+        if e.code != 0:
+            raise RuntimeError(f"Failed with code: {e.code}")
diff --git a/pyodide-build/pyodide_build/buildpkg.py b/pyodide-build/pyodide_build/buildpkg.py
index 3014763d23f..a1af88f45b3 100755
--- a/pyodide-build/pyodide_build/buildpkg.py
+++ b/pyodide-build/pyodide_build/buildpkg.py
@@ -136,6 +136,9 @@ def get_bash_runner():
             "HOSTINSTALLDIR",
             "PYMAJOR",
             "PYMINOR",
+            "CPYTHONBUILD",
+            "STDLIB_MODULE_CFLAGS",
+            "SIDE_MODULE_LDFLAGS",
         ]
     } | {"PYODIDE": "1"}
     if "PYODIDE_JOBS" in os.environ:
diff --git a/src/js/load-package.ts b/src/js/load-package.ts
index b80711189dd..a25dcf6aace 100644
--- a/src/js/load-package.ts
+++ b/src/js/load-package.ts
@@ -237,6 +237,7 @@ async function loadDynlib(lib: string, shared: boolean) {
     const module = await Module.loadWebAssemblyModule(byteArray, {
       loadAsync: true,
       nodelete: true,
+      allowUndefined: true,
     });
     Module.preloadedWasm[lib] = module;
     if (shared) {


Code review comments:
diff_hunk: @@ -74,7 +74,7 @@ jobs:
             source pyodide_env.sh
 
             ccache -z
-            PYODIDE_PACKAGES="core" make
+            PYODIDE_PACKAGES="core, ssl" make
path: .circleci/config.yml
text: @user1:
Maybe let's add ssl to core packages?