Patent ID: 7370068

Claim:
A method of processing a file in a system that has a size larger than a size of a buffer in volatile storage in the system, the method comprising: retrieving segments of the file from persistent storage into the buffer in the volatile storage; sorting each segment in the buffer; removing duplicate records while sorting each segment in the buffer; after the sorting and duplicate records removing, determining whether the buffer contains space to retrieve at least a further segment from the file in the persistent storage; in response to determining that the buffer contains the space, retrieving the at least further segment into the buffer to perform further sorting and duplicate records removing to compact the segments in the buffer; writing the sorted and compacted segments to the persistent storage of the system, wherein writing the sorted and compacted segments to the persistent storage comprises writing just full pages of the sorted and compacted segments from the buffer to the persistent storage; leaving a partial page of the sorted and compacted segments in the buffer; and retrieving at least another segment of the file from the persistent storage into the buffer to sort and compact, by performing removal of duplicate records, with the partial page in the buffer.