A Log-Structured Store for Streaming Data

Event stream processing systems have focused on in-memory databases to achieve high performance, but there are several classes of application that require the persistence of some subset of the events processed and/or results produced at very high data rates. This paper describes a facility for streaming data to disk using a versioned AVL-tree index layered over an append-only log-structured store that operates at between 45-60% the throughput of an in-memory red-black tree or hash table. 1. HISTORY AND MOTIVATION Event pattern matching and situation detection systems like RAPIDE [5,6] have long been able to make use of persistent storage, but most high-performance event stream processing (ESP) systems, which were developed in the context of telecommunications infrastructure to handle the event rates generated by call data records [1,2] and network packet traffic [3], were unable to tolerate the latency and loss of throughput involved with storing data to disk. Consequently, ESP systems have used chronicles, estimation, and load-shedding to meet performance requirements, storing short-lived data using inmemory databases or custom indices. The needs of many ESP applications can be met with purely inmemory systems, but there are classes of application that require persistence of some subset of their data — often at uncomfortably high through-puts that range into hundreds of thousands of events per second — for correctness, and many situations in which application restart time would be unacceptably long if recalculation via an historical replay were required to re-establish current state. Our streaming data system was developed to address a broad range of applications, including some that require high-throughput persistence and others that require retrospective queries over retained historical data while new data continues to stream into the system. 2. THE LOG STRUCTURED APPROACH Storage systems are typically organized into a read-optimized repository for long-lived data with a write-optimized journal as a front-end to absorb and then batch updates to the read-optimized repository. In contrast, a log-structured store (LSS) is constructed entirely from a write-optimized journal. The structure of an LSS is conceptually identical to a ring buffer where writes progress forward until reaching the end of the buffer then wrap around and continue from the beginning. The primary advantage of this design is that the disk head is always in the right place to perform the next write, thus eliminating expensive disk seeks. A secondary advantage is that recovery of a log can be performed using a very simple “roll forward” algorithm that replays or rejects any uncommitted writes following the last checkpoint of the log. Ousterhout and Rosenblum [8,9] began work on LSS filesystems after observing that most filesystems spend a great deal of time updating on-disk index structures for data that will only be overwritten or deleted before it leaves the buffer cache. Around the same time Stonebraker exploited the database recovery advantages of an append-only LSS for the Postgres Storage System [14]. LSSs were initially thought to be a general solution to improve the performance of write-intensive applications, but performance was ultimately hindered by the need for a background garbage collector, known as the “cleaner,” to reclaim free space from the log. The Berkeley LFS team documented the difficulties presented by the cleaner: performance degrades as the ratio of live data to free space increases because more and more time is spent waiting for the cleaner to re-locate live data in order to make room for new writes [10,11]. Fortunately, ESP workloads typically feature enormous numbers of writes that only need be retained for short periods (usually defined by a window over some ordering property of the incoming data), which allows us to avoid excessive cleaner overhead while enjoying the throughput advantages of an LSS. 3. IMPLEMENTATION 3.1 System Resources We use 64-bit addressing over memory mapped log files to store our data, making use of the operating system’s virtual memory subsystem to handle caching. This approach provides us with a directly addressable buffer pool with very little bookkeeping overhead, no double buffering and no calls to malloc(). This Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the VLDB copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Very Large Database Endowment. To copy otherwise, or to republish, to post on servers or to redistribute to lists, requires a fee and/or special permissions from the publisher, ACM. VLDB ’07, September 23-28, 2007, Vienna, Austria. Copyright 2007 VLDB Endowment, ACM 978-1-59593-649-3/07/09. method also provides us with file-local object identifiers (OID) for free in the form of simple 64-bit file offsets. Our approach to system resource utilization is in many ways an update of the one used by the Dali storage manager [4], however updates to Dali’s store were done in-place rather via an appendonly log, which limited its persistence performance. 3.2 Versioned Trees In order to avoid read/write lock contention in the storage system, we have implemented a versioned tree data structure that allows writes to continue while reads (including ad-hoc queries against internal table state) are performed against against short-lived snapshots. It was easy to implement this efficiently by exploiting the non-destructive write property of the LSS. A great deal of prior work has been done on multitemporal [12] and versioned tree data-structures [15] for in-place storage systems, and on techniques for efficiently updating metadata in a versioned log [13], but because our needs are focused on asneeded snapshots rather than historical range queries we are able to avoid most of the difficulties that that work sought to overcome and thus use very simple data structures. numbers of writes that only need be retained for short periods (usually defined by a window over some ordering property of the incoming data), which allows us to avoid excessive cleaner overhead while enjoying the throughput advantages of an LSS.