Patent Document ID: 8458163
Application ID: 13252149

Base Claim:
1. A method of indexing documents in websites, the method comprising: on a server system having one or more processors and memory storing programs to be executed by the one or more processors: storing, for each website of a multiplicity of websites, a corresponding current crawl rate limit; comparing a maximum crawl rate for a respective website over a defined period of time with the current crawl rate limit for crawling the respective website to determine if the current crawl rate limit is a limiting factor in crawling the respective website; performing a website crawling management function in accordance with the determination of whether the current crawl rate limit is the limiting factor in crawling the respective website; and providing a crawl rate control mechanism to a respective owner of the respective website, wherein the crawl rate control mechanism enables selection of a new crawl rate limit corresponding to the respective website by the respective owner.

---

Claim 2:
2. The method of claim 1 , further including crawling the respective website at a rate no greater than the current crawl rate limit for crawling the respective website.