Patent ID: 8095548

Claim:
A method of managing a large data set, the method comprising the steps of: determining a maximum number of file descriptors that can be open simultaneously by an operating system operating on a selected computer; opening, by a computer, a finite number of containers, the finite number of containers being no more than, but approximately equal to, the determined maximum number of file descriptors, and being substantially less than the number of the plurality of records; sorting, by the computer, a dataset of a plurality of records by unique key field value for each of the plurality of records to the finite number of data containers, at least an initial pass of the sorting comprising an incomplete top-down radix sort to thereby substantially reduce computer resources required to perform a record search, the step of sorting comprising: determining the key field value of the key field of the respective record, generating a digest of the key field value, the digest having a same fixed size for each of the plurality of records, the step of generating the digest comprising the step of applying a hash function to the determined key field value, modifying the digest to determine a container address for the respective record, the step of modifying the digest comprising the step of applying a bit-level hash mask to the digest to determine the container address for the respective record, the bit-level hash mask having a size equal to an integer value of the size of the digest minus an integer value related to log 2 (finite number of containers), and associating the respective record with the determined container address; and when it is desired to sort the records into an increased number of containers, performing the steps of: opening a finite number of next-pass data containers for each of the finite number of data containers associated with the sorted plurality of records defining prior-pass data containers, decreasing the size of the hash mask to define a next-pass hash mask, applying the next-pass hash mask to the digest of the key field value for each subset of the plurality of sorted records associated with a separate one of the prior-pass data containers to determine a next-pass container address for each respective record, and associating each respective record with the respective determined next-pass container address.