Patent ID: 8812478
Filing Date: 2014-08-19
Classification: G06F

Abstract:
1. A method comprising: receiving, at one or more processors, a plurality of links to hyperlinked documents to be crawled; grouping, by the one or more processors, the plurality of links to the hyperlinked documents based on hosts associated with the plurality of links to the hyperlinked documents; grouping, by the one or more processors, the hosts into buckets according to quantities of links to hyperlinked documents to be crawled at the hosts, selecting, by the one or more processors, a particular host, from a first bucket of the buckets, to crawl next by going through one or more of the buckets until the first bucket is found to include the particular host that satisfies a particular condition, crawling, by the one or more processors, a link to a hyperlinked document, of the plurality of links to the hyperlinked documents, from the particular host; and moving, by the one or more processors and based on crawling the link, the particular host from the first bucket to a second bucket, of the buckets, that includes second hosts with fewer links to hyperlinked documents to be crawled than the first hosts in the first bucket.