Scheduling Aggregation Operations Chunk management Chunks for each cube are arranged on disk in a given order of dimensions. In this work we explore both alternatives. To support large data sets with a large number of dimensions we store data in chunks. BESS is used to store sparse data.
Chunks can be dense if the density is above a certain threshold.
When more memory is required, minichunks are written to disk and their memory is utilized. Depending on the density of the dataset considered and the data distribution, an appropriate minichunk size is chosen. Too large a minichunk size will accommodate sparser and fewer minichunks in memory. A file is kept to store the minichunks of the chunks in each cube. For larger sizes either support for 64 bit long pointers is required or a parallel file system can be used.