Repository: freedomofpress/securedrop

Issue:
Packaging process builds Rust code
Our `make build-debs` packaging process needs to build a Rust wheel and install it into the virtualenv. As part of this we need to make sure that the built wheel is fully reproducible so we aren't regressing on reproducible builds.


Discussion:
Just noting that in my dev Qubes VM with 6 CPU cores assigned, it takes a little over 4 minutes to build redwood in release mode.
So here's what maturin is doing under the hood, as discovered by `RUST_LOG=debug maturin build -m redwood/Cargo.toml -v -r`

```
2023-06-28T20:05:49.562717Z DEBUG maturin::project_layout: Using cargo manifest path from command line argument: "/home/user/github/freedomofpress/securedrop/redwood/Cargo.toml"
2023-06-28T20:05:49.562741Z DEBUG maturin::project_layout: Resolving cargo metadata from "/home/user/github/freedomofpress/securedrop/redwood/Cargo.toml"
2023-06-28T20:05:49.680016Z DEBUG maturin::project_layout: Found pyproject.toml at "/home/user/github/freedomofpress/securedrop/redwood/pyproject.toml"
2023-06-28T20:05:49.680486Z DEBUG maturin::project_layout: Resolving cargo metadata from "/home/user/github/freedomofpress/securedrop/redwood/Cargo.toml"
2023-06-28T20:05:49.798743Z DEBUG maturin::project_layout: Project layout resolved project_root=/home/user/github/freedomofpress/securedrop/redwood python_dir=/home/user/github/freedomofpress/securedrop/redwood rust_module=/home/user/github/freedomofpress/securedrop/redwood/redwood python_module=/home/user/github/freedomofpress/securedrop/redwood/redwood extension_name=redwood module_name=redwood
🔗 Found pyo3 bindings
2023-06-28T20:05:49.892508Z DEBUG maturin::python_interpreter: Found CPython interpreter at /home/user/github/freedomofpress/securedrop/.venv/bin/python3
🐍 Found CPython 3.8 at /home/user/github/freedomofpress/securedrop/.venv/bin/python3
📡 Using build options compatibility from pyproject.toml
2023-06-28T20:05:49.892782Z DEBUG maturin::compile: Setting PYO3_PYTHON to /home/user/github/freedomofpress/securedrop/.venv/bin/python3
2023-06-28T20:05:49.892800Z DEBUG maturin::compile: Running CARGO_ENCODED_RUSTFLAGS="-C\u{1f}link-arg=-fuse-ld=/usr/bin/mold" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.8-64bit" PYO3_PYTHON="/home/user/github/freedomofpress/securedrop/.venv/bin/python3" PYTHON_SYS_EXECUTABLE="/home/user/github/freedomofpress/securedrop/.venv/bin/python3" "cargo" "rustc" "--message-format" "json-render-diagnostics" "-v" "--manifest-path" "/home/user/github/freedomofpress/securedrop/redwood/Cargo.toml" "--release" "--lib"
...
2023-06-28T20:08:25.756120Z DEBUG maturin::module_writer: Adding redwood-0.1.0.dist-info/METADATA
2023-06-28T20:08:25.756369Z DEBUG maturin::module_writer: Adding redwood-0.1.0.dist-info/WHEEL
2023-06-28T20:08:25.756691Z DEBUG maturin::module_writer: Adding redwood/__init__.py
📖 Found type stub file at redwood.pyi
2023-06-28T20:08:25.756900Z DEBUG maturin::module_writer: Adding redwood/__init__.pyi from /home/user/github/freedomofpress/securedrop/redwood/redwood.pyi
2023-06-28T20:08:25.757153Z DEBUG maturin::module_writer: Adding redwood/py.typed
2023-06-28T20:08:25.757345Z DEBUG maturin::module_writer: Adding redwood/redwood.cpython-38-x86_64-linux-gnu.so from /home/user/.cargo/target/release/maturin/libredwood.so
2023-06-28T20:08:26.382540Z DEBUG maturin::module_writer: Adding redwood-0.1.0.dist-info/RECORD
📦 Built wheel for CPython 3.8 to /home/user/.cargo/target/wheels/redwood-0.1.0-cp38-cp38-linux_x86_64.whl
```

Here's what the `__init__.py` file contains:
```
from .redwood import *

__doc__ = redwood.__doc__
if hasattr(redwood, "__all__"):
    __all__ = redwood.__all__
```

for reference:
```python3
>>> redwood.__all__
['generate_source_key_pair', 'encrypt_message', 'encrypt_file', 'decrypt', 'RedwoodError']
>>> redwood.__doc__
'A Python module implemented in Rust.'
```

I thiiiiink we could just do this ourselves?? Will keep poking.
It ended up being like 40 lines of Python, which is pretty good I think!

maturin does have some nice safety features, like it checks that the `PyInit_redwood` symbol is present (https://github.com/PyO3/maturin/blob/3ea1d0f165351d44f4c02afed98a226334401558/src/compile.rs#L539), so I think we should eventually switch back to it, but I think we can just wait until it lands in Debian - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999850

And since it's statically compiled, hopefully we'll just be able to use it directly out of unstable/testing (might run into glibc version issues?).

Resolving patch:
diff --git a/redwood/build-wheel.py b/redwood/build-wheel.py
new file mode 100644
--- /dev/null
+++ b/redwood/build-wheel.py
@@ -0,0 +1,60 @@
+#!/usr/bin/python3
+"""
+Build a Python wheel containing the Rust redwood code
+
+Ideally we'd use maturin for this, but it has a big dependency tree
+that's not really worth reviewing, so this is mostly a stop-gap until
+we can use a Debian-packaged version. This is only intended to run in
+the SecureDrop dev and packaging environments.
+
+At a high level, this script:
+* Compiles the Rust code into a shared library (libredwood.so)
+* Copy and rename that into the Python package structure
+* Build a Python wheel using setuptools
+"""
+import argparse
+import os
+import shutil
+import subprocess
+import sys
+import sysconfig
+from pathlib import Path
+
+parser = argparse.ArgumentParser(description="Build a wheel")
+parser.add_argument("--release", action="store_true", help="Build in release mode")
+parser.add_argument("--redwood", type=Path, help="Path to redwood folder")
+parser.add_argument("--target", type=Path, help="Path to target folder")
+args = parser.parse_args()
+
+os.chdir(args.redwood)
+
+if args.release:
+    flavor = "release"
+    cargo_flags = ["--release"]
+else:
+    flavor = "debug"
+    cargo_flags = []
+
+env = {"CARGO_TARGET_DIR": str(args.target), **os.environ}
+
+print("Starting to build Rust code")
+subprocess.check_call(["cargo", "build", "--lib", "--locked"] + cargo_flags, env=env)
+print("Copying libredwood.so into package")
+libredwood = args.target / flavor / "libredwood.so"
+if not libredwood.exists():
+    print(f"Error: Can't find libredwood.so (looked for {libredwood})")
+    sys.exit(1)
+
+so_destination = args.redwood / "redwood" / ("redwood" + sysconfig.get_config_var("EXT_SUFFIX"))
+shutil.copy2(libredwood, so_destination)
+
+print("Building wheel")
+subprocess.check_call([sys.executable, "-m", "pip", "wheel", "."])
+# Yes the wheel name is wrong (this is not a pure-Python wheel),
+# but it doesn't matter, this is only an intermediate step as it will
+# be immediately unpacked and installed into the virtualenv.
+whl = Path("redwood-0.1.0-py3-none-any.whl")
+if not whl.exists():
+    print("Error: Can't find the wheel")
+    sys.exit(1)
+print(f"Built at: {whl.absolute()}")
diff --git a/redwood/redwood/__init__.py b/redwood/redwood/__init__.py
new file mode 100644
--- /dev/null
+++ b/redwood/redwood/__init__.py
@@ -0,0 +1,6 @@
+# flake8: noqa
+# Wrapper to Rust .so
+from .redwood import *
+
+__doc__ = redwood.__doc__
+__all__ = redwood.__all__


Test patch:
diff --git a/builder/tests/test_securedrop_deb_package.py b/builder/tests/test_securedrop_deb_package.py
--- a/builder/tests/test_securedrop_deb_package.py
+++ b/builder/tests/test_securedrop_deb_package.py
@@ -9,6 +9,7 @@
     subprocess.check_output(["git", "rev-parse", "--show-toplevel"]).decode().strip()
 )
 DEB_PATHS = list((SECUREDROP_ROOT / "build/focal").glob("*.deb"))
+SITE_PACKAGES = "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages"
 
 
 @pytest.fixture(scope="module")
@@ -69,6 +70,7 @@ def test_deb_package_contains_expected_conffiles(deb: Path):
         "/var/www/securedrop/.well-known/pki-validation/",
         "/var/www/securedrop/translations/messages.pot",
         "/var/www/securedrop/translations/de_DE/LC_MESSAGES/messages.mo",
+        f"{SITE_PACKAGES}/redwood/redwood.cpython-38-x86_64-linux-gnu.so",
     ],
 )
 def test_app_code_paths(securedrop_app_code_contents: str, path: str):
@@ -89,7 +91,7 @@ def test_app_code_paths(securedrop_app_code_contents: str, path: str):
         "/var/www/securedrop/static/.webassets-cache/",
         "/var/www/securedrop/static/gen/",
         "/var/www/securedrop/config.py",
-        "/var/www/static/i/custom_logo.png",
+        "/var/www/securedrop/static/i/custom_logo.png",
         ".j2",
     ],
 )
diff --git a/molecule/testinfra/app/test_smoke.py b/molecule/testinfra/app/test_smoke.py
--- a/molecule/testinfra/app/test_smoke.py
+++ b/molecule/testinfra/app/test_smoke.py
@@ -1,6 +1,8 @@
 """
 Basic smoke tests that verify the apps are functioning as expected
 """
+import json
+
 import pytest
 import testutils
 
@@ -31,3 +33,19 @@ def test_interface_up(host, name, url, curl_flags):
                 assert "nopenopenope" in f.content_string
     assert "200 OK" in response
     assert "Powered by" in response
+
+
+def test_redwood(host):
+    """
+    Verify the redwood wheel was built and installed properly and basic
+    functionality works
+    """
+    response = host.run(
+        "/opt/venvs/securedrop-app-code/bin/python3 -c "
+        "'import redwood; import json; print("
+        'json.dumps(redwood.generate_source_key_pair("abcde", "test@invalid")))\''
+    )
+    parsed = json.loads(response.stdout)
+    assert "-----BEGIN PGP PUBLIC KEY BLOCK-----" in parsed[0]
+    assert "-----BEGIN PGP PRIVATE KEY BLOCK-----" in parsed[1]
+    assert len(parsed[2]) == 40