Superseding traditional indexes by orchestrating learning and geometry

The explosion of big data and the new generation of applications and computing paradigms, possibly related to mobile and IoT scenarios, raise new challenges, such as strict latency, energy and storage constraints, which usually vary among devices, users and time because of intrinsic and extrinsic conditions not predictable in advance. The algorithmic community addressed these challenges by focusing, among the others, on memory hierarchy utilisation [15, 38], query processing on streams [11], space efficiency [30, 35], parallel and distributed processing [19], and software auto-tuning [29], just to mention a few. In this paper, we contribute to these studies in a twofold manner. We design the first learned index (à la [27]) that solves the dictionary problem with provably better asymptotic time and space complexity than classic indexing data structures for hierarchical memories, such as B-trees and Cache-Sensitive Search trees [34], and modern learned indexes [16, 27]. We call our solution the Piecewise Geometric Model index (shortly, PGM-index) because it turns the indexing of a sequence of keys into the coverage of a sequence of 2D-points via linear models (i.e. segments) suitably learned to trade, in a principled way, query time vs space efficiency. This idea comes from the heuristic results of [16, 27] which we strengthen by showing that the minimal number of such segments can be computed via known and optimal streaming algorithms [32, 42]. The PGM-index is then obtained by recursively applying this geometric idea and by guaranteeing a smoothed adaptation to the “geometric complexity” of the input data. Finally, we propose a variant of the PGM-index that adapts itself not only to the distribution of the dictionary keys but also to their access frequencies, thus obtaining the first distribution-aware learned index. The second main contribution of this paper is the proposal and study of the concept of Multicriteria Data Structure, namely one that adds to the classic requirements of being space and time efficient, the novel feature of being flexible enough to dynamically adapt itself to the constraints imposed by the application of use. We show that the PGM-index is a multicriteria data structure because its significant flexibility in storage and query time can be exploited by a properly designed optimisation algorithm that efficiently finds its best design setting in order to match the input constraints (either in time or in space). Such theoretical contributions are corroborated by a thorough experimental analysis over three known and large datasets, showing that the PGM-index and its multicriteria variant improve uniformly, over both time and space, classic and learned indexes up to several orders of magnitude. This makes our geometrically-learned approach potentially useful in modern DBscenarios [26] and paves the way to novel investigations in the classic realm of data structure design for other problems, some of which are stated in the concluding section of this paper. :2 Superseding traditional indexes by orchestrating learning and geometry 2012 ACM Subject Classification Information systems → Data structures, Theory of computation → Data structures and algorithms for data management, Theory of computation → Data compression