## 5 Evaluation
### Comparison of ARCOMEM Versus Standard Crawl  
For evaluating the quality of the crawls in terms of how focused the collected documents are, we conducted a series of experiments. The main task of these experiments is to give an overview of the number of the crawled documents match the keywords given by the ICD, and how similar the textual content of the crawled documents is compared to the seed documents. Since there is no ground truth for the actual relevance of a document we have investigated some alternatives on how to measure the relevance of a document with respect to the given crawl definition. The dataset used for the experiment consists of two different crawls in the financial domain. One crawl was performed using the described ARCOMEM architecture while the other crawl is a standard Heritrix crawl. The results we are presenting in this section are based on the document score generated by the Solr scoring module [52]. This score is computed as follows. For each crawl, we create an inverted index of the crawled documents. Next, we form a query with the keywords provided by the ICD, or the most representative keywords from the textual content of the seed documents. The idea of these settings is to find documents with similar textual content to the ICD keywords or to the content of the seed documents. We assume that the documents matching the query are relevant and that their relevance is represented by their similarity to the query, or in other words, their retrieval status value (RSV). Since the standard Solr scoring is based on TF/IDF, the whole document collection is taken into account for normalization. One drawback of this method may arise if all analyzed documents contain the relevant keywords. The IDF score which is used for normalization will be relatively low if many documents of the collection contain these words, and due to this, relevant documents may get a lower similarity score. In order to deal with this factor we checked how many of the documents contained our keywords. The results are shown in Table 3. The ARCOMEM crawl consist of 234,749 documents and the Heritrix Crawl contained 366,806 documents. In order to make both crawls comparable we analyzed the percentage of documents containing keywords from the ICD. The results show that most of the keywords do not appear inside most documents which allows us to use a TF/IDF based scoring approach. It becomes also visible that the percentage of occurrences of the keywords from the ICD in the crawled documents is in nearly all cases higher when crawling with the ARCOMEM framework than when crawling with Heritrix. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Batch** & \(bl\) & **Update Function** & **Harvest Ratio** & **Average Similarity** & **DMOZ Topics** \\ \hline No & 1 & FIRST & 0.3317 & 0.2945 & 0.4979 \\ No & 5 & FIRST & 0.2948 & 0.2819 & 0.4677 \\ No & 1 & AVG & 0.3609 & 0.3024 & 0.5779 \\ No & 5 & AVG & 0.3200 & 0.2897 & 0.5420 \\ Yes & 1 & AVG & 0.3556 & 0.3013 & 0.5260 \\ Yes & 5 & AVG & 0.3347 & 0.2952 & 0.5176 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Results from simulated crawls with and without batch priority updating. The results of the experiments to assess the quality of the ARCOMEM and Heritrix crawls are shown in Tables 4 and 5, respectively. We compared four different settings for describing what makes a document relevant. These settings differ in how the terms are selected and the number of terms that are used for calculating the similarity score of crawled documents to the topic of the crawl. For the experiment in the first row (ICD), we use the terms from the ICD. For the experiments in the last three rows, we choose the terms based on their TF/IDF value for the seed documents; maxQt = 10 indicates that the 10 most representative keywords are taken into account, which gives a very narrow definition of relevance. Additionally we used 50 and 100 words giving us a broader definition of relevance, since more words are considered to be important.