MOMMIE Knows Best: Systematic Optimizations for Verifiable Distributed Algorithms

Complex distributed algorithms become running systems through an integration with optimizations that target the system’s deployment environment. Although expedient, this approach has disadvantages. First, this often makes implementing the algorithm difficult, since its logic must be composed with the optimizations. Second, proving the guarantees of the implementation is tedious, because the proofs must be derived for the composed algorithm, which may not be directly mappable to the original, unoptimized algorithm. Finally, retargeting the implementation to a different deployment—requiring a different set of optimizations—can be wasteful, since a new composed algorithm must be derived to include this different set of optimizations, including their correctness proofs. We fault the tussle between abstraction and performance as the fundamental cause for this problem. On one hand, algorithm designers need abstraction to simplify the job of identifying and proving invariants or liveness properties about their algorithms. On the other hand, algorithm implementers have faced an age-old conviction (grounded in at least some truth) that abstraction hurts performance, which for the complexity and latencies involved in distributed algorithms often leads to unusability. This trend is evident in the level of detail present in the pseudo-code descriptions of many replicated systems. For example, the PBFT system [4], a replicated state machine based on Byzantine consensus, gives an I/O Automaton specification—arguably a clean formalism intended for proving properties—that nevertheless contains explicit details about cryptographic tools (digital signatures vs. MAC authenticators), content dissemination (push vs. pull, direct broadcast vs. one-hop flood), deduplication (send whole object vs. object digest), transport mechanisms (retransmissions, negative acknowledgments) and group communication (manual collection of identical messages from many sources to form quorums). Beyond specification, these implementation details feature prominently in the correctness proofs prepared by the authors; in fact, the authors had to prepare distinct proofs for different versions of their algorithm that differed in the choice of implementation details (crypto for message authentication) but ostensibly not in the fundamental “business logic” of the algorithm. This was not an isolated example: in the decade since that work, dozens of other proposals have been presented that push into the business logic further optimizations for distinct execution environments and workloads (e.g., mostly non-faulty or trusted nodes, or even mostly read-only or contention-free write requests). Although such overspecialization is successful at improving performance, at an algorithmic level it increases complexity and hurts dependability: implementations appear with missing features, and correctness proofs end up lacking rigor. In this paper we challenge the validity of the tussle at the heart of this trend. We argue for a different way to build replicated systems that are both rigorous in specification and guarantees, but also optimized to perform well in their particular execution environment and workload. We make the case for separating implementation details from the algorithms they optimize—arguably as old an idea as computer science. In particular, we advocate a replicated algorithm abstraction model that admits provably safe, plug-and-play, fine-grained optimizations that can be chosen and applied automatically, maintaining any safety and liveness guarantees proven for the optimization-free algorithm specification.