Reducing the Footprint of Main Memory HTAP Systems: Removing, Compressing, Tiering, and Ignoring Data

Gracefully reducing the main memory footprint (e.g., via compression and data tiering) is an unsolved challenge for HTAP database systems, where most traditional reduction methods are no longer applicable. Since advantages of a reduced footprint are manyfold, the issue is of high importance for main memory-resident databases. In this paper, we present our work on Hyrise and discuss how we break down this challenge into three aspects in order to reduce the main memory consumption without losing the performance advantage of in-memory databases. First, we reduce existing allocations by efficiently selecting indices and workload-driven compression configurations for table data. Second, we use hybrid table layouts to tier data with minimal impacts on the runtime performance. Third, we employ data structures that efficiently eliminate unnecessary accesses to secondary storage. As an outlook, we depict our vision to eventually unite these aspects in a holistic footprint optimization. 1. MEMORY FOOTPRINT REDUCTION Database systems that store their entire working set in DRAM have a clear performance advantage over their diskbased counterparts. But DRAM can easily become the main cost driver for large server systems. Its capacity still remains limited, especially when compared to modern storage arrays with petabytes of storage. Moreover, DRAM is responsible for up to 40% of a server’s energy consumption [15]. Besides cost considerations, growing data volumes limit the applicability of main memory-resident database systems. Basically, all in-memory database vendors have added capabilities to handle data volumes larger than DRAM capacities, which underlines the need of efficient means to lower the memory footprint (i.e., main memory consumption). A lower footprint provides obvious benefits such as decreased hardware costs but also enables faster recovery times. On top of that, gained free space can be used to execute faster algorithms with larger memory requirements or for adding auxiliary data structures such as secondary indexes. One self-imposed condition that makes our project particularly interesting is the focus on hybrid transactional and analytical workloads (so-called HTAP, OLxP, or mixed workloads). Most existing approaches, e.g., pageor tuple-based eviction, incur disastrous performance penalties for workloads that are mostly dominated by sequential processing. Proceedings of the VLDB 2018 PhD Workshop, August 27, 2018. Rio de Janeiro, Brazil. Copyright (C) 2018 for this paper by its authors. Copying permitted for private and academic purposes. We implement our research in the open source database Hyrise. Hyrise is a columnar in-memory database optimized for HTAP workloads. The database is ACID-compliant and uses MVCC concurrency control. Data in Hyrise is horizontally partitioned into so-called chunks with a predefined size (comparable to [14]). Data is inserted in the most recent mutable chunk until this chunk reaches its limit. At this point, a new empty chunk is created and the recently filled chunk becomes immutable (using append-only for updates and MVCC invalidations for deletions). We approach tiering in Hyrise by separating the problem into three aspects: 1. Before any data is evicted (to presumably significantly slower storage tiers), existing data structures are compressed or even removed (cf. Section 3). 2. We use hybrid table layouts to vertically partition and evict data to secondary storage (cf. Section 4). 3. We employ auxiliary data structures to ignore irrelevant data (i.e., data skipping or pruning, cf. Section 5). In this paper, we will discuss these three aspects which are already published or currently work in progress. The final objective of the pursued thesis is to create a holistic view onto these three aspects and properly balance them.