Patent ID: 8099408

Claim:
A method in a computing device for identifying informative links of a web site for use in crawling the web site, the method comprising: collecting sample web pages of the web site; generating a site map of the web site based on the sample web pages, the site map representing clusters of similar web pages and indications of inter-cluster and intra-cluster links between web pages; analyzing the generated site map to identify skeletal links of the web site, the skeletal links being links that support the structure of the web site; and after analyzing the generated site map to identify skeletal links of the web site, crawling the web site by following links that are similar to the identified skeletal links and not following other links wherein the identifying of skeletal links of the web site is based on assessing the coverage and informativeness of the web pages of a cluster, the coverage of the web pages of a cluster is based on the number of unique web pages in a cluster and the number of unique web pages in the web site, the informativeness of web pages of a cluster is based on the number of web pages in the cluster that are duplicates of other web pages in the cluster and the number of web pages in the cluster.