# A Knowledge-Driven Multimedia Retrieval System Based on Semantics and Deep Features
## 3 The Proposed System
\[sim_{LC}(c_{1},c_{2})=-\log\frac{len(c_{1},c_{2})}{2\cdot deep_{max}}.\] (3)  
Semantic measures based on information content (IC) assume that each concept includes much information in a knowledge based (i.e., WordNet [46]). Similarity measures are based on the information content of each concept. The measures are:  
* Resnik's Measure: it assumes that for two given concepts, similarity is depended on the information content that subsumes them in the taxonomy \[sim_{resnik}(c_{1},c_{2})=-\log p(Iso(c_{1},c_{2}))=IC(Iso(c_{1},c_{2})).\] (4)
* Jiang's Measure: Jiang's Measure uses both the amount of information needed to state the shared information between the two concepts and the information needed to fully describe these terms. The value is a semantic distance between two concepts. Semantic similarity is the opposite of the semantic distance. \[sim_{jiang}(c_{1},c_{2})=\frac{2\cdot IC(Iso(c_{1},c_{2}))}{IC(c_{1})+IC(c_{2} )}.\] (5)  
Image Extractor module fetches the images in each HTML document. The retrieved images are analyzed and filtered by size and format; in our case, we consider only JPEG images but we can easily consider other format. The same module extracts visual features from these images. In our framework, we consider three types of features--local, global and deep. We use ORB as local feature  
Figure 3: Text Normalization. due to its performance compared with other similar descriptors [26]. The same consideration is made regrading the global descriptor PHOG [47]. We consider four deep descriptors derived from CNNs--VGG16, ResNet50, InceptionV3, MobileNetV2 due to the novelty in the use of this kind of features. For each CNNs we have implemented a global average pooling and a global max pooling. In this work the CNNs are pre-trained on ImageNet. Multimedia Web Document Composer module builds the documents to store in our NoSQL database (i.e., MongoDB). In particular, we use a json document consists of the following fields:  
* Name: web site name;
* Url: web site url;
* Images: array like structure, each element is a json like structure (see Figure 4);
* Image name;
* Image path: image storage path;
* PHOG: PHOG feature;
* ORB: 2-D array of ORB key point;
* Deep Descriptor: array of deep descriptors; _id:ObjectId(\"Sda2e85a9a96f1812 dbb1847\") name: \"113600\" url: \"[[http://www.cottagecraft.co.uk/](http://www.cottagecraft.co.uk/)]([http://www.cottagecraft.co.uk/](http://www.cottagecraft.co.uk/))\" topic: \"Top/Sports/Equestrian/Equipment_and_Apparel\" cleaned_text: \"javascript seems disabled browser for best experience site sure turn j...\" title: \"Cottage Craft\" summary: \"Manufactures of girths, headcollars, numnahs, and pads. Site offers p...\" images: Array <0: Object  path: \"/113600_1.jpg\" size: 617880 > orb:Array > deep_desc:Array >1:Object  path: \"/113600_0.jpg\" size: 617880 > orb:Array > deep_desc:Array  
The Retrieval subsystem has been configured in three different search cases. The first case (_Case A_) is the text only query. In Figure 5 the activity diagram is shown. The system normalizes the text of the query posed by the user and considers the domain to extract the concept from the knowledge Base. It computes the semantic similarity between the concept and each term of the documents. In a previous step, we apply a word sense disambiguation algorithm, and for the assignment of the score to the single documents the Equation (6) is used. \[sim_{doc}(tq,d)=\frac{\sum_{i=1}^{N}sim(td,d_{i})}{N}, \tag{6}\]  
where \(tq\) is a concept, \(d\) is a document, \(N\) is the number of tokens in the document and \(d_{i}\) is a document term. Figure 4: Document Example. The second case (_Case B_) is the visual query. In Figure 6 the activity diagram of this step is shown. The system extracts the feature from the image query and computes its similarity with regards to the document collection images.