Patent ID: 7310632

Claim:
A computer implemented system that facilitates web-crawling, comprising at least a processor, one or more memories with following components stored thereon: a managing component that performs a predictive analysis to predict when a web page will change, and determines when, and how to perform web-crawling; a server computer component that implements a web-crawling component that crawls subsets of web pages as a function of the predictive analysis, discovers and updates the pages in a catalogue of possible search results; and a decision-theoretic component that determines an appropriate time to crawl the at least one web page and makes predictions regarding changes in at least one web page based at least in part on: a probability that a particular outcome will occur, Pr; and a utility factor associated with each outcome, Utility(O); an action, a, selected from a set of possible actions, A, to be performed on the at least one web page, which maximizes the value of: ∑ o ∈ O ⁢ Pr ⁡ ( o ❘ a ) × Utility ⁡ ( o ) where o is an outcome selected from a set of possible outcomes, O, wherein the outcome o maximizes the efficiency of the web-crawling component in discovering and updating changed web pages.