Patent ID: 8560521

Claim:
A method, comprising: identifying a first file including a prefix tree, wherein the prefix tree includes an ordered hierarchical data structure which stores arrays of data with a common prefix, wherein the prefix tree stores at least a portion of uniform resource locators, each of the uniform resource locators having a common prefix; selecting an agent to process the first file; and processing the first file utilizing the agent, wherein the agent is selected based on a type of the prefix tree included in the first file, and wherein the type of the prefix tree is indicated by a file extension of the first file in which the prefix tree is included; wherein selecting the agent includes identifying that the prefix tree is a crawl trie based upon the file extension and selecting an agent configured to perform a crawling function based upon identifying that the prefix tree is a crawl trie; and wherein processing the first file utilizing the agent includes downloading a second file identified using a uniform resource locator stored in the prefix tree and analyzing the second file to identify unwanted content in the second file.