Patent ID: 8886621

Claim:
A computer implemented method for adaptive feedback ensuring timeliness of a collection of web pages retrieved from servers in a computer network, the method comprising: in a computer system having access to the servers in the computer network, extracting one or more universal resource locators (URLs) from a result of searching for web pages that are served by servers in the computer network; and for each URL extracted, determining whether or not a web page corresponding to the URL is present in whole or in part in the collection which is a cache of URLs and corresponding web pages accessed by a crawler, wherein, when the web page is determined to be present in the collection, refreshing by the crawler the web page in the collection by requesting a current copy of the web page from a corresponding one of the servers in the computer network in accordance with a first probability, such that due to the first probability a frequency of refreshing the web page over a period of time by the crawler is a function of a frequency with which the URL that is extracted appears in a plurality of the results of searching over the period of time, and when the web page is determined not to be present in the collection, downloading by the crawler the web page from a corresponding one of the servers in the computer network and including the web page in the collection.