1 Verifying Concurrent Search Structure Templates

spec. Set ADT Templates Give-up Lock coupling Link Implementations Hash table (give-up) B+ tree Lock coupling list B-link tree Hash table (link) Fig. 1. The structure of our proofs. synchronization and memory representation for such data structures. The second ingredient is the recently proposed flow framework [Krishna et al. 2018], an SL-based abstraction mechanism that allows one to reason about global inductive invariants of general graphs in a local manner. The flow framework enables us to formalize the correctness argument of Shasha and Goodman [1988] within separation logic. Krishna et al. [2018] use the flow framework to verify a template algorithm based on the so-called give-up technique. However, the proof is only on paper and it depends on a bespoke program logic. This logic requires new reasoning primitives that are not supported by the logics implemented in existing SL-based tools. In addition, the underlying semantic model of the logic makes it hard to use equational reasoning, which is a critical prerequisite for enabling proof automation. To enable tool support for our proposed methodology, we therefore adapt and simplify the flow framework so that it fits within the confines of existing SL-based verification tools and engineer solutions for specific challenges that arise in the formalization of the complex invariants that the template algorithms depend on. Specifically, our new flow framework presented in §3 simplifies the meta theory of the original flow framework and can now be embedded directly into existing separation logic tools like GRASShopper [Piskac et al. 2013, 2014] and the Iris higher-order separation logic framework [Jung et al. 2016, 2017, 2015; Krebbers et al. 2017]. The former enables proof automation via SMT solvers and the latter can be used to mechanically check proofs using Coq [Coq Development Team 2017]. This showcases the flexibility of the new framework, as it allows the verifier to choose either highly-trusted but labor-intensive tools or tools that provide more automation but have a larger trusted code base, depending on the needs at hand. We demonstrate our methodology by formalizing three template algorithms for concurrent search structures based on the link, the give-up, and the lock-coupling technique of synchronization (Fig. 1). For these, we derive concrete implementations based on B-trees, hash tables, and sorted linked lists, resulting in five different data structure implementations. §4 discusses the proof of the link template in detail. A summary of our verification effort is provided in §5. We perform the template proofs in Iris/Coq and verify the implementations in GRASShopper, in order to bring the proofs of highly complicated implementations such as B-link trees within reach. We note that there is no formal connection between the proofs done in Coq and GRASShopper. If one desires end-to-end certified proofs, one can parametrize Iris by the programming language used in GRASShopper and use GRASShopper as an oracle for implementation proofs, or even perform both template and implementation proofs in Iris/Coq (albeit with subtantial manual effort). The only other trusted component is the meta theory of our flow framework. These proofs are simple facts about graphs and graph properties that need only be proven once, apply to every data structure proof in our evaluation and beyond, and are proved in detail in §A. By contrast, all the proofs pertaining to our individual case studies are performed and verified by either Coq or GRASShopper. All the algorithms we have considered use fine-grained node-level locking. This is representative of real-world applications, many of which prefer lock-based over lock-free algorithms as the latter Proc. ACM Program. Lang. , Vol. 1, No. CONF, Article 1. Publication date: January 2018. 1:4 Siddharth Krishna, Dennis Shasha, and Thomas Wies tend to copy data more1. On the other hand, our methodology does not require locking, and can be extended to prove lock-free algorithms such as the Bw-tree [Levandoski and Sengupta 2013]. The proofs we obtain are both more modular and simpler than existing proofs of such concurrent data structures. In fact, we are the first to obtain a mechanically verified proof of concurrent B-link trees. Unlike the proof of da Rocha Pinto et al. [2011], which is not mechanized, our proof does not assume node-level operations to be given as primitives. In particular, we also verify the challenging split operation. The only other comparable proof is that of a B+ tree in [Malecha et al. 2010]. However, this work only considers a sequential implementation of B-trees and the proof is considerably more complex than ours (encompassing more than 5000 lines of proof for roughly 500 lines of code). The contributions of this paper can be summarized as follows: • We propose a methodology for verifying concurrent search structure templates that enables proofs to be compositional in terms of program structure and state, and exploit both thread and algorithmic modularity. • We present an improved and simplified flow framework for reasoning compositionally about the complex invariants needed for the template and implementation proofs. The framework is designed so that it can be used with off-the-shelf separation logic verification tools. • Wemechanically prove several complex real-world data structures such as the B-link tree that are beyond the capability of existing techniques for mechanized or automated formal proofs. The obtained proofs are much simpler and more reusable than prior (pencil-and-paper) proofs of comparable structures.