Patent ID: 8027983

Claim:
A method, comprising: receiving, by a data processing apparatus, a data record from a collection of data records, the data record having multiple dimensions, each dimension corresponding to a respective field in the data record and specifying a value; receiving, by a data processing apparatus, a set of hashing options, the set of hashing options specifying hash lengths for hashes of the multiple dimensions of the data record, a hash length being specified for each dimension based on a number of different possible values of data in the dimension; and creating, by a data processing apparatus, a composite fixed length data hash of the data record, the composite fixed length data hash being a concatenation of respective hashes of values specified by respective dimensions of the data record, each of the respective hashes being in a respective section of the composite fixed length data hash, each respective section having a section length that corresponds to the hash length of a particular dimension to which the respective section corresponds, the composite fixed length data hash having a fixed length at least equal to a sum of the section lengths of respective sections in the composite fixed length data hash, wherein one section of the composite fixed length data hash includes multiple segments that store respective hash values corresponding to multiple values in a field of the data record, each of the segments having a segment length that is shorter than the section length of the section in which the segments are included.