Patent ID: 8250079

Claim:
A method for identifying near and exact-duplicate documents in a document collection, the method comprising: for each document in the collection performing by a computer the steps of: reading textual content from the document; filtering the textual content based on user settings; determining N most frequent words from the filtered textual content of the document to generate a first most frequent, word- 1 , to an Nth most frequent word, word-N, sorted from highest to lowest frequency; performing a quorum search using the N most frequent words from the filtered textual content in the document with a threshold M, wherein the threshold M is used to retrieve documents from the document collection having a number M of the N most frequent words; sorting results from the quorum search based on relevancy, whereby based on the values of N and M near and exact-duplicate documents are identified in the document collection; and determining the relevancy by taking a number of hits for the quorum search in the document and dividing the number of hits by a size of the document in kilobytes of text in the document or a size in kilobytes for the entire document.