# Towards a Semantics-Based Recommendation System for Cultural Heritage Collections
## 3 Developing a Recommendation System for Cultural Heritage Collections
### Evaluation  
Evaluation plays a crucial role in the development of recommendation systems. It usually focuses on the accuracy of the recommendations, using metrics such as precision  
Figure 8: Part of the example result of step 3.3. Screenshot from Neo4j. and recall, which measure the quality and quantity of recommendations, respectively [19]. Depending on the purpose and context of the recommendations, evaluation goals can also include coverage, trust, novelty, serendipity, diversity, etc. [16]. Due to time restrictions and insufficient resources, we were not able to conduct a user-based evaluation of the proposed recommendation algorithms. In this section, we describe how the proposed algorithms can be evaluated in the future. For the Search Module, the evaluation concentrates on assessing the extent to which the search results match the user's interests (precision) and the variety of topics they are related to (diversity). For the Recommendation Module, in addition to precision, efficiency should also be considered, as the system is designed to capture the users' additional topics of interest, which are reflected by their viewing behaviours, and locate target objects in more than one round of interaction. The assessment of recall is also important for both modules, although it is difficult to achieve with the example dataset because the available semantic metadata does not describe all the visual features of some objects. The following sections introduce the evaluation process, including the aspects of evaluation for each module and the related measures. Most procedures concern analysis of results produced using the proposed system; a few procedures require additional actions implemented with Cypher statements, which are presented in the Appendix A. 1. Evaluation of the Search Module  
The evaluation of the Search Module focuses on the precision, recall and diversity of the results. The top 30 results (results on the first page, as set in step 2.3) are suggested to be assessed for each aspect, as users are most likely to view an object on the first page of the search results. Precision is the fraction of relevant instances among the retrieved instances [42]. The precision (\(R(pre)\)) of the Search Module is the number of objects in the list of search results that match the user's interests (\(O(n)\)) out of all the objects the module returns (\(O(m)\)). \[R(pre)=\frac{O(n)}{O(m)} \tag{3}\]  
The precision rate can be misleading since it can be affected by the number of items that meet the user's interests in the dataset. For example, if there is only one object in the dataset that meets the user's interests, the precision rate would be low inevitably; in such cases, precision would not be an appropriate indicator for the performance of the system. Another metric that should be considered in those cases is the recall rate. Recall is the fraction of relevant instances that were retrieved [42]. For the Search Module, the recall rate (\(R(rec)\)) would be the number of objects returned by the Search Module that match the user's interests (\(O(n)\)) out of all the objects that the user would find interesting in the database (\(O(s)\)), as calculated using the formula below. Due to the incomplete semantic annotations in the AS-NTUE-School-Art-Textbooks dataset, it would be difficult to calculate \(O(s)\), but this metric would be applicable to other datasets with complete semantic descriptions. \[R(rec)=\frac{O(n)}{O(s)} \tag{4}\]  
The diversity of the results is assessed based on the similarity among the objects in the list of search results. The Node Similarity algorithm (unweighted) is used to compare any two objects among the first 30 objects returned by the Search Module based on the subjects and keywords they are associated with. The diversity rate (\(D\)) is then calculated using the formula shown below, where \(S(avg)\) is the average similarity score, and the similarity scores vary from 0 to 1. The assumption here is that the higher the average similarity score is, the less diverse the results will be.