Robust Data Transformations

Background. Massively parallel data processing systems are ubiquitous in today’s big data era. Examples include Hadoop, Spark, Stratosphere, and a number of tools developed on top of them. Users of these systems upload their datasets to a distributed le system and run their analysis in a distributed fashion. However, several analyses require a variety of data preparation steps in order to perform the actual analysis eciently. Examples include creating a variety of data layouts, indexes, or partitioning for analytical queries; collecting randomor stratied sampling for approximate queries; compressing, erasure coding, or exibly replicating data for tuning the storage space; and cleaning data, i.e. violation detection and repair, to guarantee consistencywith respect to a given set of business rules. Traditionally, such data transformations start with the assumption of a query workload, i.e. a representative set of queries provided upfront. Alternative approaches collect the query workload over time and apply transformations based on the observed query workload. Problem. Unfortunately, the assumption of having a workload is not true in many of the modern data analyses which are ad-hoc and exploratory in nature. For example, an analyst looking for anomalies in a web server log might compute a variety of aggregates in an ad-hoc manner. Such analysis has no predened query workload. And collecting the workload is tedious since the analyst would typically want to start using the data as soon as it is available, perform his analysis, and move on to the next dataset, possibly before the workload gets collected. e other problem with workload-based data transformation is that they benet only a small subset of the possible queries. In case the workload changes, the workload-based approaches need to re-transform the data. For example, the data needs to be partitioned all over again when the group-by queries change or cleaned all over again when the business rules change. is is highly undesirable for running ad-hoc data analyses. Our Idea. We ask ourselves two critical questions: (i) can we transform a dataset right at the beginning and without having a query workload, and (ii) can the transformations be robust in that they benet ad-hoc queries in the future. To address these questions, we propose to shi from workload-driven data transformations to data-driven data transformations. is means that the data transformations depend only on the data itself. One way to do this is to exhaustively transform the data for all combinations of transformations. Consider data partitioning as a concrete example. To create robust partitioning, imagine partitioning a dataset on all attributes, i.e. ad-hoc group-by or join queries will nownot need to re-partition this dataset. Alternatively, we can pick uniformly distributed points in the search space. For example, one could imagine creating all posThis article is published under a Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0/), which permits distribution and reproduction in any medium as well allowing derivative works, provided that you attribute the original work to the author(s) and CIDR