# A Personalized Ontology Recommendation System to Effectively Support Ontology Development by Reuse
## 6 Evaluation
### Results and Discussion  
In this subsection, we report the results of applying the evaluation metrics to measure the performance of our ontology recommender system. As mentioned earlier, 16 evaluators tried our system and made 46 iterations in total. In each iteration, the evaluator received a recommendation list of ontologies and their concepts that best match his query and preferences. The user could check the information about each ontology (e.g., name and description) and its matched class (e.g., label, synonyms, and sub-classes) to decide if the system ranks the recommendations properly or if they needed to be re-ranked. We collected the test users' responses in log files to record their choices in each iteration. The differences in the rankings between the recommended list and the user's ranked list (i.e., the absolute error) for each iteration done by each user, are shown in Table 2. An absolute error value equal to 0 means that the test user is satisfied with the algorithm's ranking, and he did not perform any re-rank. If the user made a minor change in the rankings by moving one or two ontologies a step downward or upward (usually in the first positions), we would obtain a reasonable absolute error value (e.g., 2 or 4). High absolute error values for some test users (e.g., 14, 16, and 20) come when they move an ontology from a bottom position to a top one, where this causes a shift for all the in-between ontologies and thus the absolute error value becomes high. We also notice that the evaluators conducted most of the re-ranking of the recommendation list at the first iterations, while conducting a minor re-ranking in later iterations. This is an indicator that our recommender investigates user preferences in the first iterations and then learns from the user's selections, whereas, in later iterations, the recommendations gradually become much more appropriate to user preferences. We measured the accuracy of our recommendations by applying the _MAE_ metric, and we obtained a score of 3.3. We expect a lower _MAE_ score if the evaluators try the system for other iterations. In statistics, the reciprocal rank of a query response is the multiplicative \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**Absolute Error of Every Iteration**} & \multicolumn{1}{c}{**Total**} \\  & \multicolumn{1}{c}{
\begin{tabular}{c} **No. of** \\ **Iterations** \\ \end{tabular} } & **Iteration 1** & **Iteration 2** & **Iteration 3** & **Iteration 4** & **Iteration 5** & **Absolute Error** \\ \hline Evaluator 1 & 2 & 0 & 0 & - & - & - & 0 \\ Evaluator 2 & 2 & 0 & 0 & - & - & - & 0 \\ Evaluator 3 & 3 & 6 & 2 & 0 & - & - & 8 \\ Evaluator 4 & 2 & 0 & 0 & - & - & - & 0 \\ Evaluator 5 & 3 & 0 & 0 & 0 & - & - & 0 \\ Evaluator 6 & 4 & 0 & 14 & 0 & 4 & - & 18 \\ Evaluator 7 & 4 & 12 & 0 & 14 & 0 & - & 26 \\ Evaluator 8 & 2 & 4 & 0 & - & - & - & 4 \\ Evaluator 9 & 3 & 20 & 10 & 0 & - & - & 30 \\ Evaluator 10 & 2 & 0 & 0 & - & - & - & 0 \\ Evaluator 11 & 5 & 6 & 10 & 0 & 0 & 0 & 16 \\ Evaluator 12 & 2 & 0 & 0 & - & - & - & 0 \\ Evaluator 13 & 2 & 8 & 0 & - & - & - & 8 \\ Evaluator 14 & 3 & 12 & 4 & 2 & - & - & 18 \\ Evaluator 15 & 4 & 0 & 0 & 2 & 0 & - & 2 \\ Evaluator 16 & 3 & 16 & 6 & 0 & - & - & 22 \\ \hline \multicolumn{6}{c}{**Total = 46**} & \multicolumn{1}{c}{**Total = 152**} \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluators’ responses and calculating Mean Absolute Error (_MAE_). inverse of the rank of the first correct answer. For example, 1 for first place, 1/2 for second place, 1/3 for third place, and so on. Table 3 shows the reciprocal rank for each iteration, where the mean reciprocal rank (\(MRR\)) is the average of the reciprocal ranks across all 46 iterations. We obtained an \(MRR\) score value equal to 0.88, which means, in about 88% of the users' iterations, the system successfully managed to place the user's most relevant item at the top of the list.