Patent ID: 7886042

Claim:
A method comprising the steps of: while performing a web crawl, detecting a newly discovered hyperlink to a first searchable item on a particular host, in response to detecting said hyperlink, scheduling said first searchable item for downloading by placing said hyperlink to said first searchable item in a download queue based at least in part on (a) a number of other searchable items on said particular host that are scheduled for downloading and (b) a freshness target for said first searchable item; wherein, after said hyperlink to said first searchable item is placed in said download queue, the download queue includes a plurality of hyperlinks to a plurality of searchable items that includes said first searchable item; after placing said hyperlink to said first searchable item in said download queue, detecting a newly discovered hyperlink to a second searchable item on said particular host, wherein the second searchable item is different than the first searchable item; in response to detecting said hyperlink to said second searchable item, scheduling said second searchable item for downloading by placing said hyperlink to said second searchable item in said download queue based at least in part on (a) a second number of searchable items on said particular host that are scheduled for downloading and (b) a freshness target for said second searchable item; wherein said hyperlink to said second searchable item is at a position in said download queue that is before said hyperlink to said first searchable item in said download queue; wherein, based on the positions of said hyperlinks to said first and second searchable items in said download queue, said second searchable item will be downloaded before said first searchable item; prior to downloading said first and second searchable items, determining, based on the positions of said hyperlinks to said first and second searchable items in said download queue, a time at which said first searchable item is scheduled to be downloaded and a time at which said second searchable item is scheduled to be downloaded; wherein the steps are performed by one or more computing devices.