Patent ID: 8812478

Claim:
A method comprising: receiving, at one or more processors, a plurality of links to hyperlinked documents to be crawled; grouping, by the one or more processors, the plurality of links to the hyperlinked documents based on hosts associated with the plurality of links to the hyperlinked documents; grouping, by the one or more processors, the hosts into buckets according to quantities of links to hyperlinked documents to be crawled at the hosts, each of the buckets being identified based on a respective number of links to hyperlinked documents to be crawled per host; selecting, by the one or more processors, a particular host, from a first bucket of the buckets, to crawl next by going through one or more of the buckets until the first bucket is found to include the particular host that satisfies a particular condition, the first bucket including first hosts of the hosts, and the first hosts including the particular host; crawling, by the one or more processors, a link to a hyperlinked document, of the plurality of links to the hyperlinked documents, from the particular host; and moving, by the one or more processors and based on crawling the link, the particular host from the first bucket to a second bucket, of the buckets, that includes second hosts with fewer links to hyperlinked documents to be crawled than the first hosts in the first bucket.