Patent ID: 8484286

Claim:
A method for collecting data from sites having addresses on a network comprising the steps of: (a) providing a plurality of data fetchers, each of said data fetchers residing on a different volunteered private personal computer that is not dedicated to fetching; (b) providing a central address dispatcher that maintains a list of site addresses; (c) providing at least one deployed address dispatcher that maintains a list of site addresses; (d) said each of said data fetchers sending a request for a site address to one of said address dispatchers, receiving said site address, retrieving data from said site at said site address, and sending said data to a data store; (e) said central address dispatcher obtaining new site addresses for said central address dispatcher list extracted from said data sent to said data store; (f) said at least one deployed address dispatcher obtaining new site addresses for said at least one deployed address dispatcher list from a network hosting service provider; and (g) synchronizing said central address dispatcher list and said at least one deployed address dispatcher list to avoid duplicate site addresses.