Patent ID: 8914331
Filing Date: 2014-12-16
Classification: G06F,G06Q,H04L,Y10S

Abstract:
1. A computer-implemented system for identifying duplicate and near duplicate messages, comprising: a set of messages; an extraction module to extract metadata from each of the messages; a compiler to compile the messages into a master array based on the extracted metadata; a topic module to determine topics of the messages in the master array; a topic sorting module to sort the messages in the master array based on the topics; a message selection module to select one of the messages and to compare the selected message with a next message; a conversation module to determine that the selected and next messages are from a common conversation thread when the topics match; a comparison module to compare a body of one such message with the body of each other message; a duplicate identification module to identify those messages having matching bodies as exact duplicates and to remove the exact duplicate messages from the set; a sort module to sort the remaining messages in order of message length; a recursive determination module to compare a shorter message with a longer message and to determine that the body of the shorter message is included in the body of the longer message; a near-duplicate module to mark the shorter message as a near-duplicate of the longer message; and a processor to execute the modules.