Patent ID: 7599920

Claim:
A computer-implemented method of indexing documents in websites, the method comprising: on a server system having one or more processors and memory storing programs to be executed by the one or more processors: for each website of a multiplicity of websites, each website having a corresponding current crawl rate limit: crawling the respective website, in accordance with the current crawl rate limit corresponding to the respective website, to download documents from the respective website for inclusion in a database; storing crawl data associated with the crawling of the respective website; providing, for display, a crawl rate control mechanism to a respective owner of the respective website, including providing for display to the respective owner at least a portion of the crawl data, and enabling selection of a new crawl rate limit corresponding to the respective website by the respective owner; comparing a maximum crawl rate for the respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; and in response to a request to increase a current crawl rate for crawling the respective website, increasing the current crawl rate limit only when the current crawl rate limit is a limiting factor in crawling the respective website.