# Re-Enrichment Learning: Metadata Saliency for the Evolutive Personalization of a Recommender System
## 4 Experiment and Discussion
### Experimental Setup
#### 4.1.2 Evaluation  
Learning from implicit user feedback, e.g., click and dwell time, has been an important factor in improving recommender systems [18]. The common implicit retention metric [19], the click-through rate (CTR) [46; 47; 48; 49; 6], was adopted to intuitively and directly evaluate the effectiveness of the methods for user satisfaction or retention on digital platforms. Specifically, the metric was used to measure the ratio of clicks to recommendations, as in Equation (12).  
Figure 4: Examples of movies from different genres (MovieLens dataset).  
Figure 5: Examples of goods from different categories (Amazon dataset).  
\[CTR=\frac{\text{Number of click-throughs}}{\text{Number of recommendations}}\times 100\;(\%) \tag{12}\]  
Furthermore, the metric of mean average precision [50], \(MAP@K\), supports the consistency of an experiment. This metric indicates how many items a user engages with among those recommended to them, as CTR does, but also uses the sequence of feedback (see Equation (13), where \(S\) indicates the number of samples). To begin with, average precision, \(AP@K\), is defined as the summation of the engaged precision values divided by the number of engagements, \(m\), where \(P(i)\) is the precision at \(i\) and \(\delta(i)\) is a bivariate function for engagement (as in Equations (14)-(16)).  
\[MAP@K=\frac{1}{S}\sum_{n=1}^{S}(AP@K)_{n} \tag{13}\]  
\[AP@K=\frac{1}{m}\sum_{i=1}^{K}P(i)\cdot\delta(i) \tag{14}\]  
\[P(i)=\frac{\text{Number of relevant recommendations}\big{|}_{i}}{\text{Number of recommendations}\big{|}_{i}} \tag{15}\]  
\[\delta(i)=\begin{cases}1&\text{if $i^{th}$ recommendation is engaged,}\\ 0&\text{otherwise.}\end{cases} \tag{16}\]  
A/B experiments [5; 19; 51] were conducted to verify the difference between applying and not applying re-enrichment learning to baseline methods. Five baseline methods are used for comparison: SVD [26], BPR [27], L-GCN [28], L-FM [35], and W&D [36]. The baseline methods are either collaborative or hybrid approaches because our proposed method treats the category preferences of users, e.g., movie genre and product category, rather than the internal content of an item, e.g., the words in a news article. We collected data from 32 participants, who clicked on all the items that seemed to be interesting. For fairness, the participants were informed of which method was used in each experiment. The time required to finish one sequence of collecting implicit feedback ranged from 10 to 30 min depending on the participant. The sequence of collecting feedback was repeated four times for every participant. A total of 4532 feedbacks were collected for the MovieLens dataset and 4830 feedbacks were collected for Amazon dataset were collected.  
Lastly, a non-parametric statistical hypothesis test, Wilcoxon test [52], was performed to check the significance of improvements. We adopted a significance level \(\alpha\) of 0.05, which indicates a 5% risk of concluding that a difference exists when there is no actual difference. The interpretation of a result in the Wilcoxon test is that the null hypothesis, \"The population median (\(\eta\)) equals the hypothesized median (\(\eta_{0}\))\", is wrong when the \(p\)-value is smaller than the significance level (i.e., the difference is significant).