Repository: woodruffw/pyrage (language: python)

Issue:
Streaming decryption
Any plans to support streaming decryption from a file-like object? We want to stream large encrypted files from an sftp server and decrypt them on the fly as we write them somewhere else.

It looks like `decrypt_file` is basically doing that, we just need an API that takes the actual file-like fileobj.
https://github.com/woodruffw/pyrage/blob/aab0f86bca80eb263b8d7b5fa4fdb06dfb52f095/src/lib.rs#L224

Maybe using something like this https://github.com/omerbenamram/pyo3-file


Discussion / hints:
Thanks for opening an issue!

Could you say a bit more about your requirements here? We added the `decrypt_file` API with https://github.com/woodruffw/pyrage/pull/32; that originally included support for file-like objects, but I held off on it due to a lack of clarity on end-user needs. Is the problem here that your SFTP streaming operation produces a Python fileobj, and not something with a file descriptor or path that could otherwise be streamed from?
Correct, we have no fd or path. In this case it's actually a custom object that implements file-like methods (it inherits [`io.BufferedIOBase`](https://docs.python.org/3.10/library/io.html#io.BufferedIOBase))

But if you are doing buffering yourself then I would expect probably any python object that just implements `read(size=-1)` to work.
Got it, thanks for explaining!

I'm open to supporting this, but I need to think a bit more about how I want to do it: I'd *generally* prefer to keep this package's dependencies to a bare minimum, and I want to make sure the downsides of using a file-like object API are communicated well to users (specifically, there's much more "critical" time spent in the CPython interpreter, so fileobj I/O is a lot slower than acquiring a file on just the Rust side.)


I gave this some more thought: 

1. If at all possible with your use case, I'd _prefer_ not to have a dedicated file-like API here (for the reasons mentioned above). In particular: since you're streaming from a server, have you looked into using `memfd` or a FIFO for this? If your use case enables system-level file descriptors as an abstraction, then I'd be open to supporting an API that takes raw file descriptors and does I/O over them. This is arguably less Pythonic, but IMO it avoids a lot of the abstraction leakage and interpreter overhead problems that I've seen with fileobjs in the past.
2. If the above is absolutely not an option for you, then I'd consider adding fileobj support through `pyo3-file`. *Ideally* this would be in the form of an extra (i.e. `pip install pyrage[fileobj]`), but I'm not sure if pyO3 or Maturin supports that. 

Let me know if this sounds reasonable to you! TL;DR: I'd be willing to accept a PR for fileobj support, but only if your other options here are exhausted :slightly_smiling_face: 
We would prefer not to route it thru the filesystem if possible.

Another approach could be chunked decryption, where we would get bytes however we want in python and repeatedly pass an incremental `byte` array into some decryption API that would return the decrypted bytes.

It looks like the rust age library has an API for this https://github.com/str4d/rage/blob/bccc4bc9497e0a4542219787ec8183414d1bd9a5/age/src/primitives/stream.rs#L197

I guess you would need to expose `age::Decryptor` via some python wrapper since it will need to maintain state across chunks.
> We would prefer not to route it thru the filesystem if possible.

That's understandable; does `memfd` not suffice for your use case? When used correctly a memfd descriptor is similar to a mmap descriptor, and should never touch the filesystem. 

Re: chunked decryption: this would also be reasonable! 
Hello, I'm looking for this too. We have a decryption service that reads file from remote storage and writes the decrypted bytes to the caller. This service has no disk available to store the encrypted or decrypted state (plus we pass that stream through other filters like decompression and checksumming).

We have an API that basically looks like this on our end and all decryptors implement it:

```py
def filter(input: io.BufferedReader, output: io.BufferedWriter):
    ...
```

The simplest implementation of this is what we have for zstd decompression (also rust bindings):

```py
def zstd_decompress_filter(input: io.IOBase, output: io.IOBase) -> str:
    dctx = zstandard.ZstdDecompressor()
    dctx.copy_stream(input, output)
```

https://github.com/indygreg/python-zstandard/blob/0063333790a853360c816101511635865405834f/rust-ext/src/compressor.rs#L219
> This service has no disk available to store the encrypted or decrypted state (plus we pass that stream through other filters like decompression and checksumming).

Could you check and see if `memfd` would work for your use case, as mentioned in https://github.com/woodruffw/pyrage/issues/36#issuecomment-1661261179? That would be my preferred approach to implementing this, and wouldn't require disk access or usage.
I don't know memfd, never used it. Unless I'm missing something, I will create an input and output anonymous files and I will have to write the whole file into that anonymous file, basically loading it fully in memory. That's not streaming.

Here is what I came up with, take it as pseudocode.

```py
def age_decrypt_filter(path: str, request_output: io.BytesIO) -> str:
    s3_input = s3.get_stream(path)
    inputfd = os.memfd_create("s3-input")
    outputfd = os.memfd_create("age-output")

    recipient = pyrage.x25519.Recipient.from_str("age1z...")

    with open(inputfd, "rb") as age_input:
        shutil.copyfileobj(s3_input, age_input)

        with open(outputfd, "wb") as age_output:
            age_input.seek(0, os.SEEK_SET)
            pyrage.decrypt_file(age_input, age_output, [recipient])

            age_output.seek(0, os.SEEK_SET)
            zstd_decompress_filter(age_output, request_output)

def zstd_decompress_filter(input: io.IOBase, output: io.IOBase) -> str:
    dctx = zstandard.ZstdDecompressor()
    dctx.copy_stream(input, output)
```


> I don't know memfd, never used it. Unless I'm missing something, I will create an input and output anonymous files and I will have to write the whole file into that anonymous file, basically loading it fully in memory. That's not streaming.

The trick here would be to use demand-paging with `mmap` + `memfd`, which would be true streaming. This is traditionally not possible with network FDs because they aren't arbitrarily seekable, but Linux actually *does* support it with TCP [as of a few years back](https://lwn.net/Articles/752188/). The constraints there (such as having to configure the MTU on your interface, and needing to guarantee that inbound data is page-sized and page-aligned) may not be worth it for your case, though.

TL;DR this is do-able, but probably not practically. Sorry for taking you down a rabbit hole 🙂 

Looks like a fileobj-based interface is probably for the best, after all. Would you be interested in submitting a patch for that, potentially based on the work in #32?

Fix patch:
diff --git a/Cargo.lock b/Cargo.lock
index 0167c63..9ce2c31 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -940,6 +940,15 @@ dependencies = [
  "pyo3-build-config",
 ]
 
+[[package]]
+name = "pyo3-file"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "488563e2317157edd6e12c3ef23e10363bd079bf8630e3de719e368b4eb02a21"
+dependencies = [
+ "pyo3",
+]
+
 [[package]]
 name = "pyo3-macros"
 version = "0.22.3"
@@ -972,6 +981,7 @@ dependencies = [
  "age",
  "age-core",
  "pyo3",
+ "pyo3-file",
 ]
 
 [[package]]
diff --git a/Cargo.toml b/Cargo.toml
index 15a8106..09296d5 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -25,3 +25,4 @@ pyo3 = { version = "0.22.3", features = [
     "abi3-py38",
     "py-clone",
 ] }
+pyo3-file = "0.9.0"
diff --git a/pyrage-stubs/pyrage-stubs/__init__.pyi b/pyrage-stubs/pyrage-stubs/__init__.pyi
index 8119866..1282a58 100644
--- a/pyrage-stubs/pyrage-stubs/__init__.pyi
+++ b/pyrage-stubs/pyrage-stubs/__init__.pyi
@@ -1,3 +1,4 @@
+from io import BufferedIOBase
 from typing import Sequence, Union
 
 from pyrage import ssh, x25519, passphrase, plugin
@@ -15,5 +16,7 @@ class IdentityError(Exception):
 
 def encrypt(plaintext: bytes, recipients: Sequence[Recipient]) -> bytes: ...
 def encrypt_file(infile: str, outfile: str, recipients: Sequence[Recipient]) -> None: ...
+def encrypt_io(in_io: BufferedIOBase, recipients: Sequence[Recipient]) -> bytes: ...
 def decrypt(ciphertext: bytes, identities: Sequence[Identity]) -> bytes: ...
 def decrypt_file(infile: str, outfile: str, identities: Sequence[Identity]) -> None: ...
+def decrypt_io(in_io: BufferedIOBase, out_io: BufferedIOBase, recipient: Sequence[Recipient]) -> None: ...
diff --git a/src/lib.rs b/src/lib.rs
index 179316b..7c1198f 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -17,6 +17,7 @@ use pyo3::{
     py_run,
     types::PyBytes,
 };
+use pyo3_file::PyFileLikeObject;
 
 mod passphrase;
 mod plugin;
@@ -259,6 +260,64 @@ fn decrypt_file(
     Ok(())
 }
 
+fn from_pyobject(file: PyObject, read_only: bool) -> PyResult<PyFileLikeObject> {
+    // is a file-like
+    PyFileLikeObject::with_requirements(file, read_only, !read_only, false, false)
+}
+
+#[pyfunction]
+fn encrypt_io(
+    reader: PyObject,
+    writer: PyObject,
+    recipients: Vec<Box<dyn PyrageRecipient>>,
+) -> PyResult<()> {
+    // This turns each `dyn PyrageRecipient` into a `dyn Recipient`, which
+    // is what the underlying `age` API expects.
+    let recipients = recipients.into_iter().map(|pr| pr.as_recipient()).collect();
+    let reader = from_pyobject(reader, true)?;
+    let writer = from_pyobject(writer, false)?;
+    let mut reader = std::io::BufReader::new(reader);
+    let mut writer = std::io::BufWriter::new(writer);
+    let encryptor = Encryptor::with_recipients(recipients)
+        .ok_or_else(|| EncryptError::new_err("expected at least one recipient"))?;
+    let mut writer = encryptor
+        .wrap_output(&mut writer)
+        .map_err(|e| EncryptError::new_err(e.to_string()))?;
+    std::io::copy(&mut reader, &mut writer).map_err(|e| EncryptError::new_err(e.to_string()))?;
+    writer
+        .finish()
+        .map_err(|e| EncryptError::new_err(e.to_string()))?;
+    Ok(())
+}
+
+#[pyfunction]
+fn decrypt_io(
+    reader: PyObject,
+    writer: PyObject,
+    identities: Vec<Box<dyn PyrageIdentity>>,
+) -> PyResult<()> {
+    let identities = identities.iter().map(|pi| pi.as_ref().as_identity());
+    let reader = from_pyobject(reader, true)?;
+    let writer = from_pyobject(writer, false)?;
+    let reader = std::io::BufReader::new(reader);
+    let mut writer = std::io::BufWriter::new(writer);
+    let decryptor = match age::Decryptor::new_buffered(reader)
+        .map_err(|e| DecryptError::new_err(e.to_string()))?
+    {
+        age::Decryptor::Recipients(d) => d,
+        age::Decryptor::Passphrase(_) => {
+            return Err(DecryptError::new_err(
+                "invalid ciphertext (encrypted with passphrase, not identities)",
+            ))
+        }
+    };
+    let mut reader = decryptor
+        .decrypt(identities)
+        .map_err(|e| DecryptError::new_err(e.to_string()))?;
+    std::io::copy(&mut reader, &mut writer)?;
+    Ok(())
+}
+
 #[pymodule]
 fn pyrage(py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
     // HACK(ww): pyO3 modules are not packages, so we need this nasty
@@ -298,9 +357,11 @@ fn pyrage(py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
     m.add("EncryptError", py.get_type_bound::<EncryptError>())?;
     m.add_wrapped(wrap_pyfunction!(encrypt))?;
     m.add_wrapped(wrap_pyfunction!(encrypt_file))?;
+    m.add_wrapped(wrap_pyfunction!(encrypt_io))?;
     m.add("DecryptError", py.get_type_bound::<DecryptError>())?;
     m.add_wrapped(wrap_pyfunction!(decrypt))?;
     m.add_wrapped(wrap_pyfunction!(decrypt_file))?;
+    m.add_wrapped(wrap_pyfunction!(decrypt_io))?;
 
     Ok(())
 }


Test patch:
diff --git a/test/test_pyrage.py b/test/test_pyrage.py
index df4ddcf..4d1211e 100644
--- a/test/test_pyrage.py
+++ b/test/test_pyrage.py
@@ -1,6 +1,7 @@
 import os
 import tempfile
 import unittest
+from io import BytesIO
 
 import pyrage
 
@@ -23,6 +24,59 @@ def test_roundtrip(self):
 
         self.assertEqual(b"test", decrypted)
 
+    def test_roundtrip_io_fh(self):
+        identity = pyrage.x25519.Identity.generate()
+        recipient = identity.to_public()
+        with tempfile.TemporaryFile() as unencrypted:
+            unencrypted.write(b"test")
+            unencrypted.seek(0)
+            with tempfile.TemporaryFile() as encrypted:
+                pyrage.encrypt_io(unencrypted, encrypted, [recipient])
+                encrypted.seek(0)
+                with tempfile.TemporaryFile() as decrypted:
+                    pyrage.decrypt_io(encrypted, decrypted, [identity])
+                    decrypted.seek(0)
+                    unencrypted.seek(0)
+                    self.assertEqual(unencrypted.read(), decrypted.read())
+
+    def test_roundtrip_io_bytesio(self):
+        identity = pyrage.x25519.Identity.generate()
+        recipient = identity.to_public()
+        unencrypted = BytesIO(b'test')
+        encrypted = BytesIO()
+        decrypted = BytesIO()
+        pyrage.encrypt_io(unencrypted, encrypted, [recipient])
+        encrypted.seek(0)
+        pyrage.decrypt_io(encrypted, decrypted, [identity])
+        decrypted.seek(0)
+        unencrypted.seek(0)
+        self.assertEqual(unencrypted.read(), decrypted.read())
+
+    def test_roundtrip_io_fail(self):
+        identity = pyrage.x25519.Identity.generate()
+        recipient = identity.to_public()
+
+        with self.assertRaises(TypeError):
+            input = 'test'
+            output = BytesIO()
+            pyrage.encrypt_io(input, output, [recipient])
+
+        with self.assertRaises(TypeError):
+            input = BytesIO()
+            output = 'test'
+            pyrage.encrypt_io(input, output, [recipient])
+
+        with self.assertRaises(TypeError):
+            input = 'test'
+            output = BytesIO()
+            pyrage.decrypt_io(input, output, [recipient])
+
+        with self.assertRaises(TypeError):
+            input = BytesIO()
+            output = 'test'
+            pyrage.decrypt_io(input, output, [recipient])
+
+
     def test_roundtrip_file(self):
         identity = pyrage.x25519.Identity.generate()
         recipient = identity.to_public()