Patent ID: 8712992
Filing Date: 2014-04-29
Classification: G06F

Abstract:
1. A method, comprising: receiving a set of webpage identifiers, a first webpage identifier of the set identifying a first webpage, a second webpage identifier of the set identifying a second webpage, and a third webpage identifier of the set identifying a third webpage; performing an ordering of webpage identifiers of the set to derive a first order for crawling the first webpage, the second webpage, and the third webpage, the first order providing for crawling the first webpage before crawling the second webpage and providing for crawling the second webpage before crawling the third webpage, the ordering based at least in part upon: determining, while crawling the first webpage, that the first webpage and the third webpage are associated with a common host and that the second webpage is not associated with the common host, the determining comprising searching the set, while crawling the first webpage, for one or more webpage identifiers that are associated with the common host, the common host based upon an internet protocol address; and performing, while crawling the first webpage, a first reordering of the webpage identifiers of the set based at least in part upon the determining to derive a second order, the second order providing for crawling the first webpage and the third webpage at least one of concurrently or contiguously and before crawling the second webpage, at least some of the method implemented at least in part via a processing unit.