Patent ID: 6547829
Filing Date: 2003-04-15
Classification: G06F,Y10S

Abstract:
A computer-based method for use in crawling a computer-readable document store, and particularly for detecting duplicate documents during a crawl so as to avoid unnecessarily retrieving and processing such duplicates, comprising the following acts:(a) obtaining from the document store a content identifier (CID) corresponding to a particular document, wherein the CID is characterized in that: (1) the CID can be fetched independently of the document itself, (2) the CID uniquely identifies the physical document in that no two different documents would have equal CIDs, and (3) the same document accessible through different URLs would have the same CID; (b) determining whether the value of the CID is the same as the value of a previously obtained CID corresponding to another document; and (c) if the value of the CID is not the same as the value of a previously obtained CID, fetching the particular document from the document store.