## 4 Module Description
### Adaptive Heritrix  
In addition to the large-scale crawler developed by IMF, we have also investigated to what extent Heritrix, a widely-used open source crawler [14], can be adapted to ARCOMEM's crawling architecture. Heritrix implements a typical centralized crawling process, where a URL is prioritized only when it is added to the frontier. In order to adapt Heritrix to the needs of ARCOMEM, we have implementeda frontier that supports updating the priorities of already scheduled URLs and receiving scored URLs from external processes, possibly running on different servers. As a result, Heritrix can be used as a fetching service for selective Web harvesting. Overall, we have extended Heritrix with a range of functionalities, regarding the storing of crawled content, the extraction of anchor text with links, _etc_. All the modifications are available as open source software in the releases of the ARCOMEM project. In this article we focus on the two main features mentioned above. The default frontier of Heritrix employs a Berkeley DB backed hash table for storing URLs, typically grouped according to the domain or host they belong to. The key of a URL's record in the frontier is computed based on its domain, a flag indicating whether the URL should be crawled immediately, its priority (or precedence in Heritrix terminology), and a counter, which increases for every URL that is inserted in the frontier (Figure 5). The frontier implementation provides a next method for obtaining the next URL to crawl from a given domain or host, but there is no method to update the priority of an already scheduled URL. To overcome this limitation, we have implemented a frontier that extends the default frontier of Heritrix, adding a hash table that maps a URL already scheduled to the key with which it was scheduled. When we need to update the priority of a URL already scheduled, we use this hash table to locate the corresponding record from the frontier, update its priority, recalculate a key and insert it in the frontier data structure at a new position. The fact that Heritrix employs an increasing counter to calculate the key for each URL ensures that there are no collisions. Figure 6 shows as an example the state of the frontier during a crawl for a domain example.com, where there are three URLs queued for crawling. The first one is flagged for immediate downloading and has a priority of 64, while the other two URLs have equal priority of 48. Upon an update of the priority of the URL [[http://example.com/copyright.html](http://example.com/copyright.html)]([http://example.com/copyright.html](http://example.com/copyright.html)) from 48 to 32, the order of the two URLs is reversed, as shown in Figure 7.  
Figure 5: The structure of the key corresponding to an entry of a URL scheduled for crawling in the default frontier of Heritrix. Figure 6: Example of Adaptive Heritrix frontier data structures and URL index. In the frontier we have developed, when a URL \(u\) is scheduled for crawling, first we have to check whether the hash table mapping URLs to entries in the frontier contains an entry for \(u\). If \(u\) is found, then we update the priority, otherwise we need to check whether \(u\) has already been crawled. The second feature we discuss enables Heritrix to receive prioritized URLs from other processes. Heritrix provides an action directory, where processes having access to the same filesystem can write files with seeds or URLs to be crawled. In order to fit Heritrix into the ARCOMEM crawling architecture and receive URLs with priority scores from the online analysis phase, we have implemented a Web service which receives scored URLs in an ARCOMEM-specific JSON format. In the simplest case, the required information is an identifier for the crawl, the URL, a score in the range \([0,1]\), and optionally a flag indicating whether this URL should be blacklisted, _i.e._, not crawled at all.