document: Improved robustness of signature-based near-replica detection via lexicon randomization Detection of near duplicate documents is an important problem in many data mining and information filtering applications . When faced with massive quantities of data , traditional duplicate detection techniques relying on direct inter-document similarity computation ( e.g. , using the cosine measure ) are often not feasible given the time and memory performance constraints . On the other hand , fingerprint-based methods , such as I-Match , are very attractive computationally but may be brittle with respect to small changes to document content . We focus on approaches to near-replica detection that are based upon large-collection statistics and present a general technique of increasing their robustness via multiple lexicon randomization . In experiments with large web-page and spam-email datasets the proposed method is shown to consistently outperform traditional I-Match , with the relative improvement in duplicate-document recall reaching as high as 40-60 % . The large gains in detection accuracy are offset by only small increases in computational requirements .
doc_bio_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O