Patent ID: 8745183

Claim:
A computer system for adaptively refreshing a web page, comprising: a web crawler refreshing the web page; a processor device operating as a page refresh policy manager implementing a page refresh policy to determine a refresh period for the web page using a divergence measurement between a base version and one or more subsequent versions of the web page, by performing: receiving a base version of the web page to be indexed; partitioning the base version of the web page into a first collection of fragments; receiving a subsequent version of the web page; and partitioning the subsequent version of the web page into a second collection of fragments; a processor device operating as a fragment comparator performing: comparing the fragments from the first collection to corresponding fragments from the second collection to determine the divergence measurement representing a numeric degree of difference between the fragments from the first and second collections; recording the divergence measurement in a change profile for the web page; using the divergence measurement between the fragments as the web page divergence between web page versions; updating a refresh period for scheduling refreshing the web page using the determined divergence measurement; and scheduling a time for refreshing the web page using the refresh period; wherein updating the refresh period of the web page using the web page divergence for scheduling refreshing of the web page comprises applying an adaptive refresh policy to adjust the refresh period by determining a lower bound on utility and an upper bound on utility of refreshing the web page at expiration of the refresh period by examining the change history of the web page and increasing the refresh period if the upper bound of utility is less than the utility threshold or decreasing the refresh period if the lower bound of utility is greater than or equal to the utility threshold; wherein the utility threshold is less than a risk control parameter.