Patent ID: 7610267
Filing Date: 2009-10-27
Classification: G06F,Y10S

Abstract:
1. A computer-implemented method comprising: retrieving, from a site host, pages associated with the site, wherein the pages contain content; determining how dynamic the content of the site is, based on the degree to which the content of the retrieved pages changed since a previous crawl of the site; if the content of the site is determined dynamic, in relation to a corresponding threshold, then continuing retrieving, from the site host, pages associated with the site; and if the content of the site is determined not dynamic, in relation to the corresponding threshold, then not retrieving, from the site host, a subset of pages associated with the domain; wherein determining how dynamic the content of the site is comprises maintaining a count of the number of continuous pages for which a same prerequisite page must be retrieved to retrieve the continuous pages; wherein retrieving a first page of the continuous pages without first retrieving the same prerequisite page causes a link to the first page to appear as a dead link but retrieving the first page after retrieving the same prerequisite page causes the link to the first page to appear as an alive link; wherein retrieving the same prerequisite page causes information required for retrieving the first page to be retrieved from the site host; wherein the method is performed by one or more computing devices comprising one or more processors.