Patent ID: 8655833

Claim:
A system for continually updating a large and substantially continually changing dataset that is accessed by an application thread, comprising: a first server configured to selectively control a substantially continually changing first dataset, the first server including a checkpointing thread, the first server further including a checkpointing cache and a data-index table; and a second server configured to selectively send new data from a second data set to the first server for inclusion into the first dataset, the new data comprised of discrete, ordered blocks, the second server being remote from the first server, wherein the data-index table further contains the ordering information of data blocks held within the second data set, and the first server is configured to update the data-index table as new data blocks are received at the first server from the second server, and wherein the first server is further configured such that: as new data blocks are received at the first server from the second server, at least the ordering information of the data blocks is placed into the checkpointing cache, and then for each block of new data to be included in the first data set, the checkpointing thread compares the ordering information of the data block in the data-index table to the ordering information for the data block in the checkpointing cache, upon the comparison indicating that the ordering information for the order of the data block to be included in the first data set is correct, the checkpointing thread including that data block into the first data set, and upon the comparison indicating that the ordering information for the data block to be included in the first data set is incorrect, the checkpointing thread discarding that data block and awaiting the data block of the correct order from the checkpointing cache.