# A Knowledge-Driven Multimedia Retrieval System Based on Semantics and Deep Features
## 4 Experimental Results  
In this section, we present the used document collections, the testing strategy and the obtained results. We used three datasets--_20 Newsgroups_[57], _PASCAL VOC2012_[58] and _DMOZ_[59]. The specific characteristic in terms of document features, contents and size allow us to evaluate in deep all the component of our framework. In particular, we use the first dataset to evaluate and select retrieval metrics based on semantic similarity, the second one to evaluate and select the deep features in _CBIR_ context and the last one to perform the final tests in which we combine the different strategies. _20 Newsgroups_ is one of the most used document collections in literature. It is available in _scikit-learn_ and managed by Python language. This version allows us to get the dataset by removing headers, footers and quotes. The 20 newsgroups collection contains twenty topics, as shown in Table 1.  
The document collection has been pre-processed by the normalization module. The query set used for the evaluation of semantic similarity metrics is composed of ten queries, for each one has been recognized the set of relevant categories of 20 Newsgroups. For example for the query \"mars\" as planet the corresponding category is \"sci.space\". The _PASCAL VOC2017_ is an image database composes by 20 objects as shown in Table 2.  
The query set used for the evaluation of the visual descriptors is composed of thirteen classes each consisting of five images. The classes have a direct correspondence with the classes pre-assigned in the dataset. _DMOZ_ has been one of the most popular and rich multilingual open-source web directories. The project initially called ODP--Open Directory Project was born in 1998. The purpose of DMOZ was to collect and index URLs to create a directory of hierarchically organized web contents. From the DMOZ dump both image and text has been extracted as described in Section 3. In Tables 3 and 4 are reported the statistics downstream of this process. We use as evaluation metrics Precision-Recall curve and Mean Average Precision (MAP) [1]. \begin{table}
\begin{tabular}{c c} \hline \hline
**Topic** & **Documents** \\ \hline alt.atheism & 799 \\ comp.graphics & 973 \\ comp.os.ms-windows.misc & 985 \\ comp.sys.ibm.pc.hardware & 982 \\ comp.sys.mac.hardware & 961 \\ comp.windows.x & 980 \\ misc.forsale & 972 \\ rec.autos & 990 \\ rec.motorcycles & 994 \\ rec.sport.baseball & 994 \\ rec.sport.hockey & 999 \\ sci.crypt & 991 \\ sci.electronics & 981 \\ sci.med & 990 \\ sci.space & 987 \\ soc.religion.christian & 997 \\ talk.politics.guns & 910 \\ talk.politics.mideast & 940 \\ talk.politics.misc & 775 \\ talk.religion.misc & 628 \\ \hline Total & 18,828 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of documents for each category in 20 Newsgroups dataset. It is the percentage of retrieved relevant documents compared with all retrieved documents. Recall is expressed as:  
\[Recall=\frac{|\{relevant\ documents\}\cap\{retrieved\ documents\}|}{|\{relevant\ documents\}|}. \tag{9}\]  
It is the percentage of relevant retrieved documents compared with all relevant documents in the document collection. The Precision-Recall curve is obtained as an interpolation of the Precision values for 11 standard Recall values ranging from 0 to 1 with step 0.1. Interpolation is estimated with the following criteria:  
\[P_{interp}(r)=\max_{r_{i}\geq r}p(r_{i}). \tag{10}\]  
We use The Precision-Recall curve to compare different retrieval algorithms considering the whole set of retrieved documents. Moreover, one of the most used measures in literature to measure the web information retrieval performances is the Mean Average Precision on the first top k results. We choice MAP, because in a web search engine the user is generally interested in the first k results.