Patent ID: 7836054

Claim:
A system for processing a message store for near duplicate messages, comprising: a deduper module configured to identify near duplicate messages in a plurality of messages in a message store, comprising: a message digester module configured to generate a message sequence taken from metadata for, and of content contained in, each of the messages and to generate an attachment sequence for at least part of at least one attachment associated with one or more of the messages; a concatenator module configured to concatenate the message sequence and the attachment sequence into a compound digest for each message; a comparer module configured to compare the message sequences and the compound digests of the messages in the message store; and a message sequence marker module configured to mark each such message having a message sequence not matching the message sequence of any other such message as unique; and mark each such message having a message sequence matching the message sequence of at least one other such message as an exact duplicate; and a classifier module configured to group those messages remaining unmarked and having similar content into sets that each comprise one or more near duplicate messages, wherein the marker is further configured to designate a first of the near duplicate messages in each of the sets as unique and each remaining near duplicate message in the set as a near duplicate; a compound sequence maker module to mark each exact duplicate message and each near duplicate message having a compound digest not matching any other compound digest as a unique message; and a processor to execute each of the modules, which are stored on a computer-readable storage medium.