Patent ID: 7454410

Claim:
A computer-implemented method for Web crawler data collection to assist a search engine in deciding on the optimal crawling frequencies and the optimal times to crawl each Web page, comprising the steps of: collecting information associated with a plurality of queries, wherein the information relates to results of the queries or responses to the queries or both results of and responses to the queries; using the collected information to determine a return probability b i,j,k that the search engine will return a Web page i in a position j of query result page k, wherein the return probability b i,j,k is determined by estimating a temporary return probability Tb i,j,k and then updating an existing return probability b i,j,k using a smoothing process; and using the collected information to determine an incorrect response probability d i relating to the probability that a query to a stale version of a particular Web page i yields an incorrect or vacuous response, wherein determining an incorrect response probability d i comprises: determining that a Previously Presented Web page i is in a sample queue that has not been processed; reading an index value i of the Web page i and determining if the Web page i exists; when the Web page exists, determining if the Web page is current; when the Web page i is not current, reading a client query from the sample queue and determining if the Web page i contains a correct response to the client query; and when the Web page i does not contain a correct response to the client query, updating d i =D·d i +(1−D), where D is a smoothing parameter.