Sequoia: A New OpenPGP Implementation inRust
An Experience Report

Neal H. Walfield and Justus WinterRust Fest, Nov. 24, 2018https://sequoia-pgp.org/talks/2018-11-rust-fest

SequoiaÉ

A new OpenPGP implementation in Rust
É

É

First commit: October 16, 2017

Motivation
ÉGnuPG is hard to modify
Code and API grew organically over 21 years
Lack of unit tests
É Tight component coupling
É
É

Many developers unsatisfied with GnuPG’s API
Rust is memory safe
É GnuPG can’t be used on iOS due to GPLÉÉWho We Are

ÉNeal, Justus, KaiÉÉÉFormer GnuPG developers (2–2.5 years at g10code)At p≡p since Fall 2017Fundingp≡p (primary)Wau Holland Stiftung (secondary)É (We’re actively looking to diversify funding base!)
ÉÉ

OpenPGP

É

Encryption and Data Authentication & Integrity Standard
É

É

RFC 4880

Not just for email. . .
Package Signing
Commit Signing
É Document Signing (integrated in LibreOffice)É Backups, Archives
É Encrypted Storage in the Cloud
É Encrypted Sneaker Net
É Password Manager
É Remote Authentication (e.g., ssh agent)
ÉÉ

Packet-Based Format

ÉAn OpenPGP message is composed of packets:
Literal Data Packet
Signature + One Pass Signature Packet
É Compression Container
É Symmetrically Encrypted Data Packet (SEIP)
É Public-Key Encrypted Session Key Packet (PKESK)
É ...
ÉÉ

An OpenPGP Message

Hello!

É

Some Data

É

Encapsulate in an OpenPGP packet

É

Sign itÉ

Encrypt it

É

Looks like a pipeAn OpenPGP Message

Hello!
Literal Data

É

Some Data

É

Encapsulate in an OpenPGP packet

É

Sign itÉ

Encrypt it

É

Looks like a pipeAn OpenPGP Message

OPSHello!
Literal Data

É

Some DataÉEncapsulate in an OpenPGP packet

É

Sign itÉ

Encrypt it

É

Looks like a pipe

SIG

An OpenPGP Message

Encryption (SEIP)
PKESKOPSHello!
Literal Data

ÉSome Data

É

Encapsulate in an OpenPGP packet

É

Sign itÉ

Encrypt it

É

Looks like a pipe

SIG

An OpenPGP Message

Encryption (SEIP)
PKESK

OPSHello!
Literal Data

É

Some Data

É

Encapsulate in an OpenPGP packet

É

Sign itÉ

Encrypt it

É

Looks like a pipe

SIG

Aside: OpenPGP Messages Also Used for Key Exchange

É

Public-Key Packet

É

Public-Subkey Packet

É

User ID PacketÉ...

Introduction
Sequoia
OpenPGP

Challenges
Processing a Pipeline
A Generic Reader Stack
A Better Parser Interface
Callbacks that Save State
Smuggling Failures in io::Errors

Challenge: Processing a Pipeline
Decryptor
read()

Hashed Reader
read()

Literalread()

Application

ÉÉ

Create a stack of readers!
Readers can also be used to deal with framing
ÉÉ

Enforce packet boundaries
Chunked encodingDepth-First Traversal of an OpenPGP Message

ÉVisitor Pattern

É

Uses the stackIdea
impl<R: Reader> CompressedData<R> {fn parse(mut reader: R) {if let Some(algo) = reader.read_u8() {let reader = match algo {1 => CompressedData { inner: reader },...
};
parse(reader);}}}fn parse<R: Reader>(mut reader: R) {if let Some(tag) = reader.read_u8() {match tag {8 => CompressedData::parse(reader),...
}}}

error: reached the recursion limit while instantiating,→`parse::<CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<CompressedData< c,→CompressedData<CompressedData<GenericReader< c,→std::fs::File> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Lessons

Generics can result in a lot of invisible types
É There is no way to articulate a base case to the compiler
ÉÉÉÉ

Can limit recursion at run-time
No way to express this to the compiler=⇒ need dynamic dispatch

Challenge: A Generic Reader Stack

É

Need an into_inner() for trait objects

ÉBut trait objects are unsized!

ÉCan use a special version of self:
pub trait BufferedReader {fn into_inner<'a>(self: Box<Self>)-> Option<Box<BufferedReader + 'a> >where Self: 'a;}É

Need an Option to handle the base case

into_inner and a Transparent Forwarder

É

Working with boxed objects is uglyÉ

Often requires unnecessary boxing and unboxingÉ

Can use a transparent forwarder:
impl<'a> BufferedReader for Box<BufferedReader + 'a>{fn buffer(&self) -> &[u8] {self.as_ref().buffer()
}
}

É...

Can now pass a Box<BufferedReader> wherever a
BufferedReader is neededBut, this creates a linked-list with into_inner()
impl<R: BufferedReader> BufferedReaderLimitor<R> {pub fn new(reader: R) -> Self {...
}}
impl<R: BufferedReader> BufferedReader
for BufferedReaderLimitor<R>{
fn into_inner<'a>(self: Box<Self>)-> Option<Box<BufferedReader + 'a>>where Self: 'a {Some(Box::new(self.reader))}}into_inner boxes the inner reader
The inner reader is of type R
É If R is a Box<BufferedReader>, we now have two boxes
É Each new / into_inner adds another box!
É The constructor needs to take a Box<BufferedReader> toavoid thisÉÉChallenge: A Better Parser Interface

É

Using the visitor pattern requires callbacks

É

Would prefer an iterator-like API

É

OpenPGP messages can be huge
ÉRequires streaming operations

An Iterator Interface
let pp = PacketParser::from_reader(r).unwrap();
for packet in pp.iter() {eprintln!("{:?}", packet);
if let Packet::LiteralData(l) = packet {// This doesn't work!
io::copy(&mut l, &mut io::stdout()).expect("Decryption failed");}}
É

Doesn’t allow streaming
ÉÉ

É

The returned item can’t reference the original object
But the reader has to stay embedded in thePacketParser to get the next packet!

Flattens tree structureAn Iterator-like Interface
let mut ppr = PacketParser::from_reader(r).unwrap();
while let PacketParserResult::Some(mut pp) = ppr {// Streaming operations...
match pp.packet {Packet::Literal(_) =>io::copy(&mut pp, output)?,...
}let (packet, ppr_) = pp.recurse()?;ppr = ppr_;// We own the packet & can save it without copying.
match packet {...
}}

Three phases
Similar enough to Rust’s Iterator API to be familiar
É If we don’t want to recurse into a container, can use nextÉÉ

Challenge: Callbacks that Save State
É

Often want to collect some state, but how to propagate it?

ÉIn C:struct callback_state {...
}
void callback(void *cookie) {struct callback_state *state = cookie;...
}void g() {struct callback_state state;function(&state, callback);}

ÉIn Rust, don’t use a cookie, use a trait!

Example
trait CallbackHelper {fn callback(&mut self);}
fn function<CB: CallbackHelper>(cb: &mut CB) {cb.callback()}
struct Callback { }impl CallbackHelper for Callback {fn callback(&mut self) {println!("Hello, world!");}}fn main() {function(&mut Callback { });}Challenge: Smuggling Failures in io::Errors

ÉWe use failures

ÉWe also implement general purpose traits (e.g., io::Read)

ÉFailures can be returned via an io::Error using
failure::compat!

Converting a Failure to an io::Error

match result {Ok(r) => Ok(r),Err(e) => match e.downcast::<io::Error>() {// An io::Error. Pass as-is.
Ok(e) => Err(e),// A failure. Create a compat object & wrap it.
Err(e) =>Err(io::Error::new(io::ErrorKind::Other,e.compat())),},}

Recovering the Failure

let result = io::copy(&mut verifier, output).map_err(|e| if e.get_ref().is_some() {// Wrapped failure::Error. Recover it.
failure::Error::from_boxed_compat(e.into_inner().unwrap())} else {// Plain io::Error.
e.into()})?;É

verifier is a custom reader

É

Using this pattern, it is still able to use rich errors!

Interested in Sequoia?

É

Sequoia’s low-level API is 98% feature complete
ÉÉ. . . including a C FFIPort to p≡p Engine nearly complete
ÉAbout 60% as much code as version using GPGgpgv replacement
É New keyserver implementation
É Experiments porting other software
ÉFirst release was. . .
É Join us!
ÉÉ(We’re looking to hire people to help
with Android and iOS integration.)

Interested in Sequoia?

É

Sequoia’s low-level API is 98% feature completeÉÉ. . . including a C FFIPort to p≡p Engine nearly complete
É

About 60% as much code as version using GPG

gpgv replacement
É New keyserver implementation
É Experiments porting other software
ÉFirst release was. . . right now!
É Join us!
ÉÉ(We’re looking to hire people to help
with Android and iOS integration.)

Andy Wright, CC BY 2.0

Summaryhttps://sequoia-pgp.org

É

Sequoia is a new OpenPGP
implementation

É

User-centric development

É

Strong focus on security

É

Portable & highly integratedÉ

Low-level API is already usable

É

Join us on. . .
irc: #sequoia on Freenodemailing list: devel@sequoia-pgp.org
É gitlab:
gitlab.com/sequoia-pgp/
ÉÉ

Sequoia by steve lyon, CC BY-SA 2.0