Patent ID: 8868541

Claim:
A method implemented by data processing apparatus, the method comprising: crawling a plurality of resources multiple times; determining a respective prominent section for each resource based on, at least, visual dimensions of a rendering of the section; determining a respective change frequency for each resource, wherein the change frequency is based on, at least, detected changes to the respective prominent section of the resource; determining a crawl interval for each of the resources based on, at least, the respective change frequency of the resource, the crawl interval for each resource indicating a time between crawls of the resource; and for a first resource of the plurality of resources: determining that the crawl interval has elapsed, determining an error count of unsuccessful crawl attempts since a last successful crawl attempt does not exceed a maximum number of allowed failures, and in response to determining the error count does not exceed a maximum number of allowed failures, increasing the crawl interval of the first resource as a modified crawl interval based on an amount of time that has elapsed since a last successful crawl of the first resource.