Patent ID: 8626725

Claim:
A method for processing data, comprising: receiving raw data organized according to rows or records having fields corresponding to columns; reorganizing the raw data organized as rows or records according to columns; encoding data values in each column according to dictionary and/or value encoding as a first data reduction layer; performing a hybrid run length encoding (RLE) and bit packing (BP) compression iteratively over all columns as a second data reduction layer; in response to a query, receiving a subset of the data as integer encoded and compressed sequences of values corresponding to different columns of the data; defining processing buckets that span over the subset of the data received as integer encoded and compressed sequences of values based on changes of compression type occurring in any of the integer encoded and compressed sequences of values of the subset of data, wherein the compression type is evaluated for each column in a compression analysis and, based on the compression analysis, is selected for each column according to which compression type results in the highest rate of compression for that column; and performing operations defined by the query on a bucket-by-bucket basis, according to the current bucket type when processing the integer encoded and compressed sequences of values to return query results.