# A Knowledge-Driven Multimedia Retrieval System Based on Semantics and Deep Features
## 3 The Proposed System  
In this section, we present the architecture of the implemented system and describe in detail the modules. The Figure 1 shows the system at a glance. The system consists of two main subsystems. The first one has in charge the creation and population of the database from a document collection; moreover, it extracts images and their descriptors and normalizing the text. The second one implements the retrieval process, using semantic measure and image retrieval techniques. _Multimedia Web Documents Repository Processor_ creates the document collection used in the retrieval tasks. The Figure 2 shows an activity diagram of this subsystem. Web documents are processed to obtain a structured object, then is stored in a NoSQL document-based database. In particular, the subsystem extracts image and text from the document normalizing the text, while the images are processed to extract features. The result of the this tasks is then merged into a single structured object if the document has at least one image and the text is composed of at least 25 terms. We remove documents without images because we want evaluate a combination of textual and visual descriptors. The _Text normalization_ is a fundamental step in an information retrieval systems because it allows to obtain documents with only clean text without stopwords and other kind of \"noise\" as for example misspelling terms [1]. Moreover, we use stemming and lemmatization tasks to have each word in its basic form. As shown in the Figure 3 the _Tags Removal_ module removes the HTML code from the text. The second module transforms the text into lowercase text and removes special chars (i.e., all chars except number, letter, _ and _ ). _Stopwords Removal_ processes the normalizedtext removing words without any particular meaning (e.g., articles, adverbs, conjunctions,...). _Stemming-Lemmatization_ module analyzes the output of the previous blocks and performs text transformations. Stemming transforms words into their canonical form to achieve more effective matching. The problem in using stemming is that words with different meanings can be associated with the same root. To solve this issue, we use a more sophisticated algorithm which uses lemmas instead of the root. The use of lemmas is more efficient because it represents the canonical form of the word (e.g., for verbs it uses infinitive form).The last step before document storing is performed by the _Out of vocabulary words removal_ module. It removes all words that do not have sense in English language (e.g., misspelling words). The semantic similarity module is in charge of compute the similarity on the textual document information. In our work we use different kinds of similarity metrics as described in the previous section. Figure 1: Architecture: Proposed Framework. Figure 2: Activity diagram: Multimedia web Documents Repository Processor. The semantic measures based on path are:  
* Shortest Path based Measure: this measure only takes into account \(len(c_{1},c_{2})\). It assumes that the \(sim(c1,c2)\) depend on closeness of two concepts are in the taxonomy, and that a similarity between two terms is proportional to the number of edges between them. \[sim_{path}(c_{1},c_{2})=2\cdot deep_{max}-len(c_{1},c_{2}).\] (1)
* Wu & Palmer's Measure: it introduced a scaled measure. This similarity measure considers the position of concepts \(c1\) and \(c2\) in the taxonomy relatively to the position of the most specific common concept \(Iso(c_{1},c_{2})\). It assumes that the similarity between two concepts is the function of path length and depth in path-based measures. \[sim_{WP}(c_{1},c_{2})=\frac{2\cdot depth(Iso(c_{1},c_{2}))}{len(c_{1},c_{2})+2 \cdot depth(Iso(c_{1},c_{2}))}.\] (2)
* Leakcock & Chodorow's Measure: it uses the maximum depth of taxonomy from the considered terms.