Patent ID: 7941421
Filing Date: 2011-05-10
Classification: G06F

Abstract:
1. A computer implemented method of operating a search system, the method comprising: executing a crawl of a plurality of web pages; analyzing results of the crawl and detecting web pages of the plurality belonging to at least one similarity class; determining clusters of the plurality of web pages based on characteristics of the content of the web pages; determining, for each of the determined clusters, at least one metric indicative of similarity among resource locators associated with the web pages of that cluster, wherein the metric indicative of similarity among resource locators is an entropy metric; determining the web pages belonging to the at least one similarity class based on the determined clusters and the determined similarity metrics, including determining a cluster having web pages belonging to the at least one similarity class based on local entropy of selected keys of the resource locators for the web pages of that cluster and a statistical measure of the similarity metrics; and making a notation, based at least in part upon the similarity class, for a web page and corresponding resource locator.