## 5 Evaluation
### Comparison of ARCOMEM Versus Standard Crawl
When comparing the results of the two crawls we see some obvious differences. The maximum similarity score per document was always the highest inside the ARCOMEM crawl, no matter which \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Ground Truth** & **Relevant Documents (\%)** & **Average Similarity** & **Maximum Similarity** & **Standard Deviation** \\ \hline ICD & 14.29 & 0.0178 & 0.8440 & 0.0426 \\ Seeds maxQt = 10 & 10.81 & 0.0179 & 0.4830 & 0.0263 \\ Seeds maxQt = 50 & 44.06 & 0.0049 & 0.4299 & 0.0149 \\ Seeds maxQt = 100 & 48.63 & 0.0060 & 0.5279 & 0.0172 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Statistics on ARCOMEM Crawl. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Keyword** & **ARCOMEM** & **Heritrix** \\ \hline Antonis Samaras & 0.0175 & 0.0052 \\ European Union & 0.6011 & 0.4346 \\ EU & 1.7870 & 1.8986 \\ financial crisis & 0.8443 & 0.3645 \\ Eurozone & 0.3157 & 0.1238 \\ bailouts & 0.0826 & 0.0393 \\ austerity & 0.3838 & 0.1638 \\ measures & 1.4880 & 0.7993 \\ strategy & 1.9123 & 0.9362 \\ growth & 3.9319 & 1.9752 \\ public & 6.7357 & 3.5990 \\ investments & 1.3406 & 0.5545 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Percentage of crawled documents containing keywords from ICD. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Ground Truth** & **Relevant Documents (\%)** & **Average Similarity** & **Maximum Similarity** & **Standard Deviation** \\ \hline ICD & 7.61 & 0.0196 & 0.7580 & 0.0440 \\ Seeds maxQt = 10 & 6.14 & 0.0175 & 0.2957 & 0.0264 \\ Seeds maxQt = 50 & 56.26 & 0.0026 & 0.3401 & 0.0114 \\ Seeds maxQt = 100 & 59.13 & 0.0021 & 0.3122 & 0.0099 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Statistics on Heritrix Crawl. ground truth was chosen. For the first two setups, with a limited number of relevant keywords, the percentage of relevant documents was much higher inside the ARCOMEM crawl; when many different keywords are taken into account this changes. In contrast to that, the average similarity of the documents was higher for the Heritrix crawl when only few keywords are considered. This can be explained by looking at the absolute number of relevant documents for the different setups. The focused crawler found many more documents related to the small set of keywords. As a result, the average relevance has dropped, since this is calculated based only on the relevant documents. Beside the results describing the relevance of all the crawled documents, we also analyzed how the relevance evolves over time. Figure 8 shows how the relevance of the crawled documents evolves over time. The relative relevance is calculated using the average similarity over 1000 crawled documents and dividing this value by the maximum of the averages. We see that the relevance of the crawled documents exhibits large fluctuations during the crawl. While the Heritrix crawl does not show a certain tendency over the crawl, we see that the relative relevance of the documents of the ARCOMEM crawl increases up to the maximum after around 34,000 documents and then drops. Overall the relevance of the ARCOMEM based crawl was higher for most of the time. Since one of the ideas of the ARCOMEM architecture is to use social networks to get additional content for the crawl we also used the described methods for analyzing the quality of links posted within Twitter. For this experiment we collected a total of 14,703 tweets related to our topic, out of which 7677 contained at least one URL. The content of these URLs was crawled and indexed in the same way as we did it with the standard Heritrix and ARCOMEM crawl. Overall we performed this experiment on a set of 2.2 million crawled documents. The overall number of documents was much larger, but for this experiment we only took documents with textual content into account. Table 6 shows some basic statistics on the crawl. Figure 8: Focused (ARCOMEM) and unfocused (Heritrix) crawling over time.