Patent ID: 7827166
Filing Date: 2010-11-02
Classification: G06F

Abstract:
1. A method comprising the steps of: identifying a particular set of one or more specified parameters that each do not determine a majority of the content of a particular webpage of a particular website; wherein identifying said particular set of one or more parameters includes: during a web-crawling procedure, accessing a first webpage that belongs to said particular website, wherein said first webpage corresponds to a first uniform resource locator (URL) that comprises a first plurality of parameters; determining whether each non-specified parameter and corresponding value, of a second URL that comprises a second plurality of parameters, matches a non-specified parameter and corresponding value of said first URL; if each non-specified parameter and corresponding value of the second URL-matches a non-specified parameter and corresponding value of said first URL, then excluding, from a set of webpages that are to be accessed during said web-crawling procedure, a second webpage to which said second URL refers; if each non-specified parameter and corresponding value of the second URL does not match a non-specified parameter and corresponding value of said first URL, then including said second webpage in said set of webpages; and accessing said second webpage during said web-crawling procedure only if said set of webpages includes said second webpage; wherein the steps are performed by one or more computing devices.