Patent Document ID: 8037054
Application ID: 12823358

Base Claim:
1. A method of scheduling documents for crawling, performed on a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising: receiving from a website a notification that includes a sitemap URL corresponding to a sitemap for the website; in response to the notification: accessing the sitemap at the sitemap URL; and retrieving from the sitemap document location information and metadata for a plurality of documents associated with the website; scheduling for downloading documents, from among the plurality of documents, based at least in part on the metadata retrieved from the sitemap; and downloading at least a subset of the documents scheduled for downloading.

---

Claim 4:
4. The method of claim 1 , the metadata providing information related to at least one of prioritizing documents for crawling by a web crawler and selecting documents for inclusion in a crawl; the scheduling including at least one of prioritizing documents for crawling and selecting documents for crawling in accordance with the metadata.