Patent ID: 8046372
Filing Date: 2011-10-25
Classification: G06F

Abstract:
1. A computer-implemented method for determining a set of duplicate documents describing subject matter similar to subject matter described by a first document, the method comprising, under control of instructions executed by one or more computing devices: generating a set of tokens for the first document, each token in the set of tokens representing a series of characters found in the first document; selecting one or more tokens of the set of tokens having an occurrence frequency in the first document lower than a frequency threshold; executing a non-fielded relevance search on a token index using the selected one or more tokens, wherein the token index comprises an index of tokens corresponding to strings in documents of a document corpus, and wherein the non-fielded relevance search returns candidate documents from the document corpus with scores for each candidate document; for each candidate document with a score above a scoring threshold, performing filtering on the candidate document and disqualifying the candidate document if the candidate document is a mutually exclusive variation of the first document, wherein the mutually exclusive variation indicates that, while otherwise similar to the first document, the candidate document is not a duplicate of the first document; and providing the set of duplicate documents, wherein the set of duplicate documents includes candidate documents that have scores above the scoring threshold and that have not been disqualified.