Patent ID: 8266134

Claim:
A computer-implemented method comprising: receiving, by one or more processors, a plurality of links to hyperlinked documents to be crawled; grouping, by one or more processors, the plurality of links to hyperlinked documents by host, each of the plurality of links being associated with one of a plurality of hosts; grouping, by one or more processors, hosts into buckets, by storing host identifiers, associated with the plurality of hosts, in the buckets, according to a number of links to hyperlinked documents to be crawled at each host; sorting, by one or more processors, the hosts in each bucket based on a stall time of each host, the stall time of each host indicating an earliest time that each host should be crawled; crawling, by one or more processors, a link to a first hyperlinked document from a host in a bucket based on a stall time associated with the host; determining, by one or more processors, a time to obtain the first hyperlinked document from the host; adjusting, by one or more processors, the stall time based on the determined time to obtain the first hyperlinked document from the host, a load factor associated with the host, and a current time; moving, by one or more processors, the host from the bucket to another bucket that includes hosts with fewer links to hyperlinked documents to be crawled; and crawling, by one or more processors, a link to a second hyperlinked document based on the adjusted stall time.