Patent ID: 8255385

Claim:
A system for adaptively deploying a Web-crawler to a Web source at a crawl rate based on historical publication data for the Web source, the system comprising: a computing device associated with a search engine having one or more processors and one or more computer-readable storage media; and a data store coupled with the search engine, wherein the search engine: determines an update frequency estimation for a particular Web page, the update frequency estimation being determined according to: F i =Σ k=1 i w k F i-k A) wherein F i is the update frequency estimation for the particular Web page for a time period, i, B) wherein w k is a weight factor given to a particular time segment, k, included in the time period, and C) wherein F i-k is a publication rate for a given time segment included in the current time period; and calculates the adaptive crawl rate by multiplying the update frequency estimation for the particular Web page by a constant.