Patent ID: 7305610
Filing Date: 2007-12-04
Classification: G06F

Abstract:
1. A computer implemented method of crawling hyperlinked documents, comprising: sending a request for additional links to hyperlinked documents to a link manager; receiving a plurality of links to hyperlinked documents to be crawled, the plurality of links being selected by the link manager based on priority; grouping the plurality of links to hyperlinked documents by host; grouping hosts into buckets according to a number of hyperlinked documents to be crawled at each host; sorting the hosts in each bucket based on a stall time of each host; selecting a host from one of the buckets to crawl next according to the stall time of the host; crawling a hyperlinked document from the selected host; determining a retrieval time for crawling the hyperlinked document from the selected host; and adjusting a subsequent stall time for the selected host according to the retrieval time.