False-positive metrics for recommender systems offline evaluation
Creators: name: Mena Maldonado, Elisa
Publisher: name: RMIT University
Year: 2024
Subjects: classificationCode: 460510
subject: Recommender systems

description: Recommender Systems (RS) support users’ decision-making by suggesting items they may find useful. As the primary goal of a recommender is to suggest content users like, evaluation methods have focused on measuring liked recommendations. However, disliked recommendations have adverse effects on user experience. Psychological studies have described a negativity bias in human perception, whereby wrong impressions may sometimes outweigh good ones in our overall assessment of an experience. Hence, we could also avoid recommending what users dislike. Research in recommender systems is mostly focused on evaluation with respect to the number of liked recommendations while disliked recommendations are often used in practice for specific business domains. For instance, the skip rate in music recommendation measures the tracks a user has skipped from a playlist.

The recommendation tasks, settings and datasets for RS offline evaluation substantially differ from the ones used in offline evaluation. Online evaluation and user studies are the most reliable methods for commercial settings. Still, offline evaluation represents an affordable previous step to filter out algorithms that will be used later with other evaluation methodologies. Offline evaluation involves gathering user interactions -for some time- from a deployed system and using them to assess system performance. Statistical biases arise during data collection as users tend to give a small rating proportion to the many items a recommender manages, resulting in a data sparsity problem. Additionally, a small set of popular items get most of these ratings, so the data collected is missing-not-at-random (MNAR).

Thus, we find motivation for employing evaluation measures that assess poor and good suggestions. After decades of development, evaluating recommender systems remains challenging due to the statistical biases involved. In a list of recommended items, three variables are considered, true-positives, false-positives and the recommended items that were not assessed. The presence of missing judgements causes a loss of complementarity between true and false-positive metrics. Hence, it is essential to understand how this loss affects the system rankings given by these metrics and to what extent these metrics are useful to evaluate a particular recommendation task.

In this dissertation, we investigate systematic disagreements between false and true-positive metrics, identify elements to discern which metrics are more reliable for offline evaluation. We do this by developing a formal analysis of the the expected biased and unbiased metrics, the optimal rankings for precision and anti-precision, and the definition of agreement and the steepness of the popularity and relevance distributions. We apply literature methods for debiasing true-positive metrics on false-positive metrics and find out they are not suitable to debias them. Thus, we redefine current true and false-positive metrics to consider missing judgements properly and avoid any discrepancies due to (biased) MNAR test data. Then, we study the relation of true, false-positives and unjudged items involved in system ranking. We analyse this situation by characterising the variation of the three variables involved in the system ranking. We disentangle item popularity and relevance to quantify their participation in system ranking construction.

We find that the recommendation of popular items is rewarded by true-positive metrics while penalised by false-positives. The relationship between true and false-positive metrics strongly correlates with the absence of relevance data. Our redefined metrics —that consider missing judgements in their calculation— are useful to correct system rankings discrepancies. And that the influence of relevance in system ranking decreases as the relation of unjudged items between systems increases in standard recommendation problems.