Patent ID: 8180739

Claim:
A data processing method comprising: collecting a stream of data records from one or more devices connected in a computing network, wherein a first data record in the stream is uniquely identifiable based on an associated identifier having a first part and a second part, wherein the first part corresponds to a source of data generation that is common to the first data record and also to at least a second data record generated from the same source that generated the first data record, and the second part corresponds to an attribute of the first data record that is unique to the first data record; loading one or more persistent indexes associated with the stream of data records into memory, wherein an index associated with the first data record is configured such that the first part of the identifier associated with the first data record is stored in a first data structure as a first entry, and the second part of the identifier associated with the first data record is stored in a second data structure in association with a data file comprising content of the first data record, wherein the first entry in the first data structure points to the data file comprising the content of the first data record; identifying a duplicate data record in the stream of data records using the in-memory indexes, in response to determining that a first part and a second part of a second data record match the first part and the second part of the first data record, respectively; and updating a repository such that the duplicate data record is not stored in the repository.