Patent ID: 9558199
Date: 2017-01-31
CPC Classifications: G06F

Claim:
1. A computer-implemented method for deduplicating a dataset, the method comprising: receiving a number indicating a count of unique data items in an expected dataset to be received per time unit, prior to analyzing any data items in the expected data set, wherein the unique data items are unique within the expected dataset; in response to receiving the number, creating a plurality of deduplication bit array partitions of a size corresponding to the number of unique data items in the expected dataset to be received per time unit, the respective deduplication bit array partitions tracking whether data items expected to be received during one time unit have been received, wherein a size of the respective deduplication bit array partitions corresponds to the received number indicating the count of unique data items in an expected dataset to be received per time unit, wherein the plurality of deduplication bit array partitions are stored in memory; receiving a data item in the expected dataset; identifying a specific deduplication bit array partition, of the plurality of deduplication bit array partitions, that tracks whether data items, including the received data item, expected to be received during a specific time unit have already been received; accessing the specific deduplication bit array partition to determine if the received data item has already been received; upon determining that the received data item has not already been received, changing a value of a bit in the specific deduplication bit array partition corresponding to the received data item; creating a new deduplication bit array partition; writing an oldest of the plurality of deduplication bit array partitions to permanent storage; and deleting the oldest of the plurality of deduplication bit array partitions from memory.