## 5 Analysis for Crawl Guidance and Enrichment
### Crawler Guidance  
Consider first the simplified view of the architecture of a traditional Web crawler depicted in Figure 9 (top). In a traditional Web crawler, such as Heritrix [43], the archiving task is described using a crawl definition configuration file that specifies the list of seed URLs to start the crawl from, patterns that specify which URLs to include or exclude from the crawl, _etc_. At runtime, URLs are managed in a priority queue, which ensures optimal allocation of the bandwidth available to the crawler given the URLs of the frontier (URLs discovered, but not yet crawled) and politeness constraints. Web pages are then fetched one after the other; links are extracted from these Web pages, and the crawl definition is used to determine whether the URLs found on a Web page should be added to the frontier.  
We expand on this architecture in the ARCOMEM project, adding new functionalities to the crawling system, as shown in the bottom part of Figure 9. As already noted, the traditional crawl definition file is replaced by an intelligent crawl definition, which allows the specification of relevance scores and the referencing of the particular kinds of Web applications and ETOEs that define the scope of the archiving task. Queue management functions similarly as in a traditional architecture, but the page fetching module is replaced by some more elaborate resource fetching component that is able to retrieve resources that are not simply accessible by a simple HTTP GET request, but only by other means: a succession of such requests, a POST request, the use of an API (which is a very common situation on Social Web networks) or the selection of individual Web objects inside a Web page (e.g., blog posts, individual comments, _etc._). Each content item obtained by the crawler is stored in the ARCOMEM object store for use by analysis modules and archivist tools.  
After a resource (for instance, a Web page) is fetched, an application-aware helper module is used in place of the usual link extraction function, in order to identify the Web application that is currently being crawled, decide on and categorize crawling actions (e.g., URL fetching, using an API) that can be performed on this particular Web application and the kind of Web objects that can be extracted. This is a critical phase for using clues from the Social Web to crawl content, because, depending on the kind of Web application that is being crawled (traditional static Web site, Web forum managed by some content management system, wiki, social networking sites, such as Twitter or Facebook), the kind of relevant crawling actions and Web objects to be extracted vary dramatically.  
The crawling actions thus obtained are, depending on their nature (embeds or other Web resources), either directly forwarded to the selection component or sent for further analysis and ranking to the online analysis modules. Since crawling actions are more complex than in a traditional crawler, and since we want to prioritize the crawl in an intelligent manner, the URL selection module is replaced by a resource selection and prioritization module that makes use of the intelligent crawling definition and the feedback from the online analysis modules to prioritize the crawl. This is the step where semantic analysis can make an impact on the guidance of the crawl. For example, if a topic relevant to the intelligent crawl specification is found in the anchor text of a link pointing to an external Web site, this link may be prioritized over other links on the page. More details about the crawling strategy can be found in [44].