Patent ID: 7536389

Claim:
An automated machine-implemented method for crawling dynamic content, the method comprising: based on first data identifying a plurality of pages to crawl, a processor determining that information about a first page at a first location should be indexed; wherein said first location is identified in said first data by a first link; in response to determining that information about the first page should be indexed, prior to indexing said information, determining that the first link is associated with a first prerequisite uniform resource locator (URL) for the first page; wherein the first prerequisite URL identifies the location of a first prerequisite page that must be visited prior to visiting the first page in order to retrieve dynamic content that the first page will contain; wherein determining the first prerequisite URL for the first page is based on second data, said second data identifying, for each particular page of one or more pages, a particular prerequisite URL, the particular prerequisite URL identifying a location of a particular prerequisite page that must be visited prior to visiting the particular page; in response to determining that the first link is associated with a first prerequisite URL for the first page, visiting the first prerequisite page at the location identified by the first prerequisite URL; and after visiting the first prerequisite page, indexing said information about the first page; wherein indexing said information about the first page comprises visiting the first page and indexing information based upon content received while visiting the first page.