Patent ID: 8930946
Filing Date: 2015-01-06
Classification: G06F

Abstract:
1. A computer-implemented method comprising: defining tasks to be performed by one or more worker processes of a plurality of parallel worker processes and for each of a plurality of uniform resource locators to be crawled; distributing the tasks into a plurality of groups, each group being based on a certain type of task being performed, wherein each group is associated with a respective group identifier; prioritizing the tasks within each group; receiving a request from a worker process of the plurality of parallel worker processes for at least one task; in response to the request, leasing a number of highest-priority tasks within a first group to the worker process for a first time period such that the highest priority tasks are unavailable to other worker processes during the first time period; receiving from the worker process a designation of one or more new tasks, each new task defining a new uniform resource locator to be crawled, the new uniform resource locator being identified by the worker process during execution of one of the highest priority tasks in the first group; and distributing the one or more new tasks to the plurality of groups.