# A Personalized Ontology Recommendation System to Effectively Support Ontology Development by Reuse
## 6 Evaluation
### Evaluation Metrics  
We used the test users' responses and various evaluation metrics to evaluate the effectiveness of our recommendation system by directly evaluating the rankings' quality. We considered the following evaluation metrics, which are discussed in [34; 35]:  
(1) Mean Absolute Error (_MAE_): used to measure the effectiveness of a recommendation algorithm in terms of accuracy. It computes the average of the absolute value differences between the rank predicted by the recommender and the actual rank given by the user over all of the recommendation items. The _MAE_ is calculated using Equation (10), the lower the _MAE_ the better the accuracy:  
\[MAE=\frac{1}{n}\sum_{i=1}^{n}\left|d_{i}-d^{\prime}{}_{i}\right| \tag{10}\]  
where \(d_{i}\) is the actual rank, \(d^{\prime}{}_{i}\) is the predicted rank, and \(n\) is the number of items in the recommendation list. (2) Mean Reciprocal Rank (_MRR_): an information retrieval measure that evaluates the recommendation list based on the position of the first relevant item. It measures if the recommender system places the user's most relevant item at the top of the list. The larger the _MRR_ value, the better the performance of the recommender system. The _MRR_ is defined using Equation (11):  
\[MRR=\frac{1}{q}\sum_{i=1}^{q}\frac{1}{rank_{i}} \tag{11}\]  
where \(q\) is the number of recommendation lists (i.e., iterations), and \(rank_{i}\) is the rank of the user's most relevant item (ranked 1) in the list of recommendations generated by the system. (3) Mean Average Precision (_MAP_): in the context of recommender systems, precision \(P\) measures the fraction of the recommended items which are relevant. It is defined as:  
\[P=\frac{number\ of\ relevant\ items}{total\ number\ of\ items\ in\ recommendation\ list} \tag{12}\]  
while _P@k_ is the fraction of relevant items in the top-k-recommended results, defined as:  
\[P@k=\frac{number\ of\ relevant\ items\ in\ top\ k\ results}{k} \tag{13}\]  
The average _P@k_ (_AP@k_) calculates the precision for only the relevant items that are recommended. It is defined as:  
\[AP@k=\sum_{k=1}^{k}P@k*rel(k) \tag{14}\]  
where _rel(k)_ is an indicator function that equals 1 if the \(k\)th item is relevant, and 0 otherwise. An item is relevant if the user is satisfied with its ranking in the list (i.e., does not change its position), and is considered to be non-relevant otherwise. \[AP@k=\sum_{k=1}^{k}P@k*rel(k) \tag{15}\]  
The Mean Average Precision (_MAP_) measures the _AP@k_ averaged over the recommendation lists of all users. It is defined as:  
\[MAP=\frac{1}{n}{\sum_{i=1}^{n}AP_{i}} \tag{16}\]  
where \(n\) is the number of recommendation lists generated by the test users. (4) Normalized Discounted Cumulative Gain (_NDCG_): a metric to measure the rankings' quality, where the gain \(G_{i}\) for an item \(i\) is the same as the ranking relevance score, and it was used to measure the effectiveness of a system's ranking of some items compared with the user's ranking for the same items. The Cumulative Gain _CG(k)_ is defined as the sum of gains up to a position \(k\) in the recommendation list. \[CG(k)=\sum_{i=1}^{k}G_{i} \tag{17}\]  
The drawback of _CG_ is that it neglects the importance of the ordering of items (i.e., by swapping the relative order of any two items, the _CG_ would be unaffected). To overcome the drawback of _CG_, Discounted Cumulative Gain (_DCG_) has been introduced to penalize highly relevant items being placed at the bottom. \[DCG(k)=\sum_{i}^{k}\frac{G_{i}}{log_{2}(i+1)} \tag{18}\]  
The _DCG_ score still has a drawback, since it adds up with the length of the recommendation list. Therefore, if we compare the _DCG_ score for a system recommending the top 5 and the top 10 items, the latter is always the higher score not because of its recommendation quality but because of its length.