We consider how to efficiently compute the overlap between all pairs of web documents. This information can be used to improve web crawlers, web archivers and in the presentation of search results, among others. Introduction Many documents are being replicated across the world wide web. For instance, there are several copies of JAVA FAQs and Linux manuals on the net. The fraction of the total WWW collection consisting of duplicates and near duplicates has been estimated at 30 to 45 .
Duplication has both positive and negative aspects.
Finding near-replicas of documents on the web.