Compression Methods for Structured Floating-Point Data and their Application in Climate Research

The use of new technologies, such as GPU boosters, have led to a dramaticincrease in the computing power of High-Performance Computing (HPC)centres. This development, coupled with new climate models that can betterutilise this computing power thanks to software development and internaldesign, led to the bottleneck moving from solving the differential equationsdescribing Earth’s atmospheric interactions to actually storing the variables.The current approach to solving the storage problem is inadequate: eitherthe number of variables to be stored is limited or the temporal resolutionof the output is reduced. If it is subsequently determined that another vari-able is required which has not been saved, the simulation must run again.This thesis deals with the development of novel compression algorithmsfor structured floating-point data such as climate data so that they can bestored in full resolution.Compression is performed by decorrelation and subsequent coding ofthe data. The decorrelation step eliminates redundant information in thedata. During coding, the actual compression takes place and the data iswritten to disk. A lossy compression algorithm additionally has an approx-imation step to unify the data for better coding. The approximation stepreduces the complexity of the data for the subsequent coding, e.g. by usingquantification. This work makes a new scientific contribution to each of thethree steps described above.This thesis presents a novel lossy compression method for time-seriesdata using an Auto Regressive Integrated Moving Average (ARIMA) modelto decorrelate the data. In addition, the concept of information spaces andcontexts is presented to use information across dimensions for decorrela-tion. Furthermore, a new coding scheme is described which reduces theweaknesses of the eXclusive-OR (XOR) difference calculation and achievesa better compression factor than current lossless compression methods forfloating-point numbers. Finally, a modular framework is introduced thatallows the creation of user-defined compression algorithms.The experiments presented in this thesis show that it is possible to in-crease the information content of lossily compressed time-series data byapplying an adaptive compression technique which preserves selected datawith higher precision. An analysis for lossless compression of these time-series has shown no success. However, the lossy ARIMA compression modelproposed here is able to capture all relevant information. The reconstructeddata can reproduce the time-series to such an extent that statistically rele-vant information for the description of climate dynamics is preserved.Experiments indicate that there is a significant dependence of the com-pression factor on the selected traversal sequence and the underlying datamodel. The influence of these structural dependencies on prediction-basedcompression methods is investigated in this thesis. For this purpose, theconcept of Information Spaces (IS) is introduced. IS contributes to improv-ing the predictions of the individual predictors by nearly 10% on average.Perhaps more importantly, the standard deviation of compression results ison average 20% lower. Using IS provides better predictions and consistentcompression results.Furthermore, it is shown that shifting the prediction and true value leadsto a better compression factor with minimal additional computational costs.This allows the use of more resource-efficient prediction algorithms toachieve the same or better compression factor or higher throughput duringcompression or decompression. The coding scheme proposed here achievesa better compression factor than current state-of-the-art methods.Finally, this paper presents a modular framework for the developmentof compression algorithms. The framework supports the creation of user-defined predictors and offers functionalities such as the execution of bench-marks, the random subdivision of n-dimensional data, the quality evalua-tion of predictors, the creation of ensemble predictors and the execution ofvalidity tests for sequential and parallel compression algorithms.This research was initiated because of the needs of climate science, butthe application of its contributions is not limited to it. The results of this the-sis are of major benefit to develop and improve any compression algorithmfor structured floating-point data.