# Amharic _Adhoc_ Information Retrieval System Based on Morphological Features
## 4 Amharic IR System Design and Text Representation
### Matching and Ranking  
In the proposed system, the extraction of index and query terms from documents and user queries uses the same workflow with text preprocessing, morphological analysis and stopword removal. Searching for relevant documents is based on matching query terms (representing information need of users) with index terms (representing documents). We used exact vocabulary term matching which searches documents that contain the query terms without analyzing the semantics of words and without considering the semantic connections between them. The retrieval probability of a relevant document for a given query is different in case of stem-based and root-based retrievals. In general, the retrieval probability of a relevant document for a given query in case of root-based matching is higher than stem-based matching. For example, consider the sample document (Appendix A) and the following four query terms derived from \(\hbar\)-\(\Omega\)C /\(s\)-\(b\)-\(r\)/:  
1. _Mk-/ sibirati_ 'the state of something which is broken'/
2. _mnc / mosabori_ 'to break'/
3. _Mk-/ sabara_'something which is broken'/
4. _mnc /?osabari_ 'cause to break'/.  
The document (Appendix A) is relevant to the four queries. The retrieval probability of the document in the case of stem-based matching is 0.032 for term query (1), 0.017 for (2), 0.004 for (3) and 0 for (4). However, in case of root-based matching, the retrieval probability of the document (at Appendix A) to the four queries is similar (0.057), which is better than stem-based matching. Furthermore, the stem-based text representation retrieves non-relevant documents since it may conflate semantically unrelated words.  
We use Lemur toolkit for ranking. For a given query \(Q\) and a collection of retrieved documents \(D\), the Lemur toolkit ranks retrieval results based on their possible relevance. It implements both BM25 and language modeling, where the document length is considered. BM25 ranks documents based on the following equation:  
\[\textit{Score}\ (D,Q)=\sum_{i=1}^{n}IDF(qi).\frac{f(qi,D).(k1+1)}{f(qi,D)+k1.(1-b+b. \ \frac{|D|}{\alpha qyll})} \tag{10}\]  
where \(f\ (qi\),\(D)\) is \(q_{i}\)'s term frequency in the document \(D\), \(\mid D\mid\) is the length of the document \(D\) in words, and \(\textit{avgdl}\) is the average document length in the text collection from which documents are drawn whereas \(k_{1}\) and \(b\) are free parameters. \(IDF\ (q_{i})\) is the inverse document frequency weight of the query term \(q_{i}\). From Equation (10), _Score_ (\(D\),\(Q\)) would be higher when query terms in \(Q\) have higher frequencies in document \(D\). For each \(qi\) in \(Q\), the following inequality holds true for Amharic documents.  
\[f_{r}(qi\),\(D)>=f_{s}(qi\),\(D)>=f_{w}(qi\),\(D) \tag{11}\]  
where \(f_{r}\), \(f_{s}\), and \(f_{w}\) denote term frequency in root-based, stem-based and word-based representations, respectively. Thus, it can be inferred that the root-based representation of queries and documents provide better information for ranking as \(f_{r}(qi\),\(D)\) provides the highest possible score.  
For language modeling, the similarity between a document \(D\) and a query \(Q\) is measured by the Kullback-Leibler (KL) divergence between the document model \(D\theta\) and the query model \(Q\theta\). The Kullback-Leibler (KL) divergence ranking function captures the term occurrence distributions and computed as:  
\[\text{KL}\ (Q\theta,D\theta)=\sum_{w\in V}p(w|Q\theta)\log\frac{p(w|Q\theta)}{p(w| D\theta)} \tag{12}\]  
where \(w\) is word, \(v\) is word vector, \(p(w\mid Q\theta)\) is estimated query term, \(p\ (w\mid D\theta)\) is the smoothed probability of a term seen in the document.