Personalized Beyond-accuracy Calibration in Recommendation

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-19 13:04:13

Abstract: Recommender systems usually aim to optimize accuracy in a supervised setting. Due to various data biases, they often fail to enhance
other critical qualities that go beyond accuracy, such as diversity, novelty, and serendipity. Prior studies focus on addressing the bias in beyond-accuracy metrics from the provider’s perspective, such as increasing the overall diversity of recommendations to com-
bat popularity bias. In this work, we take a user-centric approach to this problem and demonstrate that users have distinct preferences for beyond-accuracy metrics. We hypothesize that users have an implicit behavioral model that goes beyond optimizing their
choices only for accuracy. For instance, we assume that a user’s purchase behavior is a mix of items that are more familiar to the
user (optimizing for accuracy), and new items that are aimed for exploration (optimizing for novelty). We argue that a recommender
system should reflect users’ interest in such beyond-accuracy metrics. This perspective allows for a more holistic understanding of users’ behavior and preferences leading to more fine-grained personalized recommendations. To this end, we propose a post-ranking greedy optimization algorithm that ensures recommendations are not only accurate but also meet users’ beyond-accuracy preferences. Through extensive experiments, we demonstrate our proposed method’s ability to balance the trade-off between ranking accuracy and user-centric beyond-accuracy preferences.

Main paper: 
 introduction: 
 Many systems deploy recommender systems (RSs) to assist users in finding the target relevant content among the vast array of options available. One of the most widely used classes of recommender algorithms is collaborative filtering. These algorithms often exploit the user's prior interactions to enhance the effectiveness of Figure 1 : Mismatch between user's performance expectations based on their prior interactions and the actual performance of the recommendation algorithm that may not be personalized to meet those expectations.the recommendations, measured by typical ranking-based metrics (e.g., nDCG, Recall). Recent literature [8, 11] suggests that such a methodology fails to account for crucial aspects of recommendation, including propagation of natural biases in data (e.g., popular items) that may significantly impact the overall quality of recommendations "beyond accuracy" (e.g., novelty, diversity) [6, 37] . For instance, such algorithms might end up with high accuracy metrics by recommending a small set of popular items while carrying the risk of gradually narrowing down the users' areas of interest, creating an effect similar to echo chambers [9] .The propagation of data biases in beyond-accuracy measurements is extensively studied from the perspective of item providers [39, 40, 43] . A recurring theme in the debiasing literature often focuses on a particular criterion to group items, framing the problem as a building algorithm that fulfills a recommendation objective within these groups. Grouping criteria are often either based on a demographic attribute (e.g., provider continent [15] or gender [5] ) or an interaction-based attribute (e.g., popularity [3] ). The objective could be goal oriented [27] or based on (weighted) parity among groups [25] .In this work, we focus on goal-oriented beyond-accuracy debiasing from the users' perspective; that is, users should receive recommendations tailored to their preferences and/or behavior regarding beyond-accuracy measurements. For instance, we see in Fig. 1 that a user who has interacted with a sufficiently large set of items in their profile (so excluding a cold-start user). Computing beyond-accuracy measures on the set of items in the user profile shows that the user has a preference for interacting with a wide range of categories multiple times (having high redundancy and catalog coverage [37] ) rather than novel and surprising items. Accordingly, it is intuitive to expect that the beyond-accuracy measures proportionality remain the same in a personalized recommendation list [38] . Therefore, recommender systems should calibrate their recommendations to account for it. We refer to this task as personalized beyond-accuracy calibration (PBAC). A simplified use-case example of PBAC can be found in the next-basket recommendation literature [4, 24] where each user's recommended basket is divided into a repeat and exploration part. The division is inspired by user grocery shopping behavior that consists of regular weekly items (repeat), as well as new items that have not appeared in the user's basket before (explore). We see this as a specific case of PBAC, already demonstrating the importance and significance of it in the next-basket recommendation domain.In particular, we aim to answer the following research questions:RQ1 What is the extent of variations in user preferences for beyondaccuracy metrics and is it possible to effectively group users based on these preferences? RQ2 How well do widely used recommendation algorithms address beyond-accuracy variations in user preferences? Are there any user groups that receive favorable recommendations regarding beyond-accuracy calibration? RQ3 Can our proposed PBAC algorithm be used to determine the optimal balance between accuracy and beyond-accuracy calibration in a recommender system for different user groups? RQ4 To what extent does the underlying recommendation algorithm impact the effectiveness of our proposed PBAC reranking model?In order to answer RQ1 and RQ2, we utilize 𝐾-means clustering as a criterion to group similar users according to their propensity toward beyond-accuracy metrics. Our experiments on two real-world datasets with diverse domains showcase that users significantly differ in their preferences regarding beyond-accuracy metrics. However, state-of-the-art collaborative filtering techniques such as VAECF [26] and WMF [16] fail to account for this. We argue that incorporating beyond-accuracy calibration into the recommendation process is essential to ensure that the recommendations meet the diverse needs and expectations of all users. Recommender systems are often trained in a pairwise or pointwise manner, making it challenging to include calibration objectives in the training phase as it is a list-wise property [35, 42] . As a result, in response to RQ3, we propose a re-ranking mixed-integer optimization framework aiming at simultaneously optimizing for accuracy and multi-objective PBAC metrics to achieve the best trade-off between these objectives. We further propose a greedy algorithm for solving this problem in polynomial time and evaluate the performance of our algorithm for all clusters of users on Movielens1M and Yelp datasets (Experimental setup is outlined in Sec. 5). Results demonstrate our proposed method's effectiveness in satisfying PBAC property for all clusters of users, while maintaining the accuracy of recommendation. Moreover, to address RQ4, we compare the performance of several state-of-the-art recommendation algorithms in satisfying users' preferences in terms of beyond-accuracy metrics after applying the proposed model. Our key contributions, in summary, are as follows:• We highlight the importance of beyond-accuracy calibration for users by analyzing their preferences and the performance of state-of-the-art collaborative filtering algorithms in representing these preferences. • We propose a re-ranking optimization algorithm capable of simultaneously optimizing accuracy and beyond-accuracy calibration property in polynomial time. • Through extensive experiments, we evaluate our algorithm on two real-world datasets and compare it against five baseline algorithms. 
related work and concepts: 
 User-oriented algorithmic bias in recommendation. Algorithmic bias is the bias induced or propagated by algorithms' mechanisms [8] . Algorithmic bias is often studied either from the user or item perspectives. User-oriented algorithmic bias is particularly concerning as it can reinforce stereotypes and discrimination based on various factors such as race, gender, and age [1, 12] . From the users' perspective, Ekstrand et al. [12] explore the disparities of accuracies within groups of users with different demographic characteristics and then propose a re-sampling method to address the issue. Wang and Chen [38] demonstrate a disparity in the distribution of beyond-accuracy metrics with respect to different user characteristics and highlight the need for algorithms capable of addressing it. Moreover, Li et al. [25] propose an optimization method to create parity between active and inactive user groups. Rahmani et al. [30] perform extensive experiments on the generalizability of the user-oriented algorithmic fairness method proposed by Li et al. [25] under different grouping assumptions and diverse domains.Their results indicate that grouping users based on a single attribute or a cutoff (e.g., top 5% of most active users) is domain-dependent with poor generalizability, requiring more nuanced and multidisciplinary approaches to algorithmic bias. Similarly, this paper examines the biases induced by algorithms from the user's perspective. Furthermore, we utilize K-means clustering to structurally divide similar users in their tendency regarding beyond-accuracy metrics into clusters by minimizing within-cluster distance. In contrast to the previous works on user-oriented algorithmic bias, this work aims to present each user with recommendations that match their expectations.Beyond-accuracy metrics in recommendation. Researchers propose several beyond-accuracy metrics to evaluate the quality of a recommendation algorithm. Smyth and McClave [34] propose the average pairwise distance between items in a recommendation list to measure recommender system diversity. A similar metric is proposed by Ziegler et al. [44] to measure "intra-list similarity" scores in which high values denote low diversity. Another line of research in beyond-accuracy metrics focuses on improving such qualities in recommender systems. For instance, Kamishima et al. [20] and Abdollahpouri et al. [2] propose methods to enhance the long-tail coverage of the recommendation algorithm. Pardos and Jiang [29] investigate the filter bubble problem in a university course recommendation system and propose a method for bursting the bubble by recommending a more diverse set of items. This study utilizes the metrics defined by prior works to extract users' propensity towards these qualities. However, we optimize recommendation algorithms to meet users' expectations by calibrating the recommendation list accordingly.Calibration fairness. There has recently been a renewed focus on the calibration concept in the context of fairness in machine learning and classification tasks [7] . Calibrated algorithms are those in which the predicted proportions of the various classes match the actual proportions of data points contained in the training data [13] . This ensures that the algorithm is not biased towards any one class, and that the predictions it makes are accurate and reflective of the actual data. In recommendation tasks, Steck [36] proposes an iterative greedy algorithm for ensuring the distribution of recommended movie genres is consistent with the users' historical data. Rojas et al. [32] examine how the method proposed by Steck [35] performs when dealing with bias in several collaborative recommendation algorithms and the effect of genre calibration on accuracy. Klimashevskaia et al. [22] study the impact of three different re-ranking-based (post-processing) calibration approaches, analyzing how they can amplify or mitigate popularity bias in the movie recommendation domain. Ariannezhad et al. [4] apply the concept of calibration in the next-basket recommendation scenario, where the division of recommended baskets into repeat and exploration sections [24] is based proportionally on the historical interactions of the users in grocery shopping domain. Contrary to the works surveyed, this paper focuses on calibrating beyondaccuracy measurements in the recommendation list with the same proportion as users' profiles in training data. 
beyond-accuracy calibration: 
 In this section, we motivate the personalized beyond-accuracy calibration concepts by undertaking both data and algorithmic analyses. Firstly, we discuss the background and beyond-accuracy metrics, and then we analyze users' historical interactions to derive their tendency towards beyond-accuracy qualities. In addition, we group users according to their similar tendencies. Finally, we assess how well state-of-the-art recommendation algorithms can match the beyond-accuracy offered in their recommended list to the historical preferences of the users. This assessment allows us to determine the degree to which the algorithms are able to personalize recommendations to individual user preferences.Prior studies [14, 19] define several metrics for evaluating the quality of recommender systems that go beyond accuracy. These metrics are selected based on the application and objective of the recommender system. Following the most widely adopted beyondaccuracy metrics and preliminary correlation analysis, we select the metrics with the least correlation with one another to capture extractable characteristics. Hence, we choose four beyond-accuracy metrics along with profile length as grouping criteria. This is because user profile length plays a crucial role in determining the confidence of our estimates of tendencies toward beyond-accuracy metrics. Although our selection is not comprehensive, we believe these four metrics are representative of the full range of extractable characteristics and are suitable for use in our experiments. The idea and analysis we present are trivially applicable to any arbitrary set of beyond-accuracy metrics.(1) Global long-tail novelty (Nov.) measures the user's willingness to interact with unpopular items. The value ranges from 0 to infinity. Higher values indicate a higher tendency for the user to interact with unpopular items. Following [6] , we define novelty as:EQUATIONwhere P indicates the set of items in the user history and 𝑝 (𝑖) is the popularity score of item 𝑖.(2) Co-occurrence-based surprise (Sur.) measures the user's interest in interacting with unexpected items. Formally defined as:EQUATIONwhere 𝑃𝑀𝐼 (𝑖, 𝑗) denotes the pairwise mutual information score between items 𝑖 and 𝑗, i.e., the extent to which observed cooccurrences differ from what is expected [18] . Given a pair of items 𝑖 and 𝑗, 𝑃𝑀𝐼 (𝑖, 𝑗) is calculated as follows:EQUATIONwhere 𝑝 (𝑖) and 𝑝 ( 𝑗) indicate the probabilities that a user interacts with items 𝑖 and 𝑗, respectively, and 𝑝 (𝑖, 𝑗) shows the probability that a user interacts with both items PMI values range from -1 to 1, with -1 indicating no user interacts with the two items together (negative co-occurrence) and 1 signifying complete co-occurrence of the two items. Therefore, 𝑆 𝑐𝑜 -𝑜𝑐𝑐 also takes values from -1 to 1. The greater value of 𝑆 𝑐𝑜 -𝑜𝑐𝑐 is, the more independent items a user has in their profile leading to high surprise score. In this work, 𝑆 𝑐𝑜 -𝑜𝑐𝑐 of a user profile is calculated by taking the mean value of the co-occurrence scores based on all items rated before a specific item in the user profile. (3) Coverage (Cov.) measures the user's interest in the diversity of item categories. Following [37] , we define coverage as the percentage of interacted categories in user profile:EQUATIONwhere 𝐶𝑎𝑡 (𝑖) and | 𝐶𝑎𝑡 | indicate set of categories covered by item 𝑖 and the total number of categories, respectively. Coverage ranges within (0, 1], with 0 indicating coverage of no categories, and 1 of all categories. (4) Redundancy (Red.) measures how many redundant categories the users have interacted with. Redundancy complements coverage, as coverage alone does not indicate the user's willingness to accept diverse recommendations [37] . Formally:EQUATIONranging within [0, 1), where 0 indicates that no redundant item categories are in the user profile. The greater the value, the more redundant item categories are in the user profile.After defining the beyond-accuracy metrics of our interest, here we leverage the historical interactions and calculate beyond-accuracy metrics on the user profile based on the previously rated items.Our goal is to examine how much existing datasets support our hypotheses and how much users' historical interactions differ in terms of beyond-accuracy metrics.To this aim, for each user, we calculate the beyond-accuracy metrics (i.e., features used for user clustering) mentioned in Sec. 3.1. Fig. 2 shows the min-max-normalized distribution of each value for all users in the MovieLens1M and Yelp datasets. The Yelp dataset shows more variation in user propensities. Among the analyzed metrics, co-occurrence-based surprise shows the highest variation in Yelp due to the high sparsity of the interaction matrix. In the MovieLens1M dataset, we see a higher variation among coverage and redundancy features indicating the role these metrics can play in characterizing users' expectations. The considerable variation of extracted features from user preferences highlights the potential benefit of clustering users. By grouping users with similar preferences, the underlying behavior and common expectations of these groups can be revealed. This increased understanding of user behavior can inform personalized recommendations that better meet their individual needs and preferences.Therefore, we cluster users based on the extracted features with the 𝐾-means algorithm. To find the optimal number of clusters, we perform a preliminary experiment using the Elbow method alongside the Silhouettes score for different values of 𝐾. Our analysis of both datasets reveals the presence of four unique user groups with distinct behavior patterns. One cluster, for example, encompasses users who are highly active and have a preference for novel items, as observed in cluster 3 in the Yelp dataset. Another cluster consists of less active users who show interest in a wide range of categories, as seen in cluster 4 in the Yelp dataset (see Fig. 3 ). A greater number of clusters would result in clusters with smaller user sizes and little added value in minimizing inter-cluster variations. Table 1 shows the statistics of clustering results. We depict the center of each cluster in Fig. 3 . There is a considerable distance between the cluster centroids. A closer look at the centroids in the MovieLens1M reveals some interesting results. As an example, users in cluster 4 are highly active users (with 479.57 average interactions) and are particularly interested in specific items with low coverage and redundancy and the highest novelty score. In contrast, clusters 2 and 3 contain inactive users (24.33 and 47.95 average interactions, respectively) with the difference that users in cluster 3 are interested in a narrower range of categories. The Yelp dataset seems to exhibit a similar pattern, suggesting that the 𝐾-means algorithm can cluster users to capture their behavioral differences, as intended.Next, we intend to analyze how different recommendation algorithms, ranging from traditional to deep recommendation models, serve different user clusters in calibrating their expectations in the recommended lists. Particularly, we include three traditional methods (MF [23] , BPR [31] , and WMF [16] ) and a deep recommendation model (VAECF [26] ). In addition, we include a non-personalized baseline method, MostPop (ranking items based on their global popularity), for further investigation and comparison. To this end, we extract the beyond-accuracy features from the recommendation lists generated by each recommendation algorithm. Based on a feature vector for each user in the train data (expected output) and the recommendation list (algorithm's output), we can calculate the distance between these two vectors as a measure of algorithm miscalibration. Miscalibration can be formally defined as the deviation of the algorithm's recommended items from the user's historical preferences in terms of beyond-accuracy metrics. Fig. 4 illustrates the miscalibration of algorithms regarding beyond-accuracy metrics using Euclidean distance 1 averaged over all users. The values of this figure illustrate the incapability of current recommendation algorithms in satisfying users' beyond-accuracy expectations. Generally, all models have higher miscalibration on the Movie-Lens1M dataset than Yelp. Among models, VAECF seems to be the best-performing in beyond-accuracy calibration, followed by MF and WMF, although this capability is domain-dependent. Moreover, as expected, the MostPop model provides users with the highest level of miscalibration by recommending the same popular items to all clusters. Hence, it is imperative to devise techniques to increase the system's engagement rate by providing personalized recommendations tailored to each user's taste. We summarize our findings as follows:• Answer to RQ1: We calculate the beyond-accuracy metrics for each user. The distribution of these metrics shows significant variations among users in both MovieLens1M and Yelp, with Yelp showing more variations. We use the 𝐾-means algorithm to cluster users based on these metrics and find that there are four distinct groups of users. The clustering results reveal that users in different clusters have distinct preferences for beyond-accuracy metrics, and the 𝐾-means algorithm successfully separates them into clusters with meaningful differences. Such variations highlight the need for personalizing beyond-accuracy qualities accordingly. • Answer to RQ2: Based on the analysis of the miscalibration of recommendation algorithms in terms of beyond-accuracy metrics, it is clear that current models have a limited ability to satisfy individual variations in user preferences. This is evident in the high miscalibration values observed for all models, particularly on MovieLens1M. Our analysis shows that there is no user group that receives notably favorable recommendations in terms of beyond-accuracy calibration. Even among the most active users represented by clusters 1 and 4, the recommendations are highly miscalibrated. This highlights the importance of devising techniques that can effectively address individual variations in user preferences for beyond-accuracy metrics. 
personalized beyond-accuracy calibration algorithm 4.1 problem formulation: 
 Let U and I be the set of users and items, respectively. We assume user 𝑢 ∈ U is provided with a list of top-𝑆 items, 𝐿 𝑆 𝑢 , given a predicted relevance score matrix, 𝑅 | U | ×𝑆 . As analyzed in the previous section, state-of-the-art recommender systems fail to generate beyond-accuracy-calibrated recommendations as they are accuracy-optimized. We provide a re-ranking framework to calibrate the baseline recommendation list, such that the new list aligns better with users' preferences considering both PBAC and accuracy objectives.Let M = {M 1 , M 2 , ...M H } be a set of metrics that measures the quality of recommendation beyond accuracy. Let P 𝑢 be the set of items user 𝑢 has interacted with in the training data. To measure the miscalibration in a list over metric M k ∈ M, we calculate the distance between the user's expected outcome extracted from the user profile, 𝑐 𝑘 𝑢 = M k (P 𝑢 ), and the recommended list, 𝑜 𝑘 𝑢 = M k (𝐿 𝑆 𝑢 ), 𝐺 𝑘 (𝐿 𝑆 , U ) = 1 | U | ∑︁ 𝑢 ∈U |d(𝑐 𝑘 𝑢 , 𝑜 𝑘 𝑢 ) | ≤ 𝜖 ∀𝑢 ∈ Uwhere 𝐿 𝑆 = {𝐿 𝑆 𝑢 } 𝑢 ∈U . Also, it satisfies 𝜖-PBAC requirement when 𝐺 𝑘 (𝐿 𝑆 , U ) ≤ 𝜖 ∀M k ∈ MIn other words, a recommender system satisfies the 𝜖-PBAC requirement if 𝐺 𝑘 is less than or equal 𝜖 for all beyond-accuracy metrics, where 𝐺 𝑘 is the average distance (over users) between the beyond-accuracy metrics of the recommended list and the user's profile. According to this definition, 𝜖 represents the degree of strictness imposed by PBAC; when epsilon is zero the optimal calibration is achieved. Note, nonetheless, excessive adherence to PBAC may result in a loss of accuracy. Therefore, we optimize for both PBAC and accuracy metrics.This part outlines an algorithm that generates a calibrated list by properly selecting 𝐾 items for each user 𝑢 among its top-𝑆 items, 𝐿 𝑆 𝑢 , where 𝐾 ≤ 𝑆 ≤ |I|. We define a binary decision matrix 𝑋 = [𝑋 𝑢𝑖 ] | U | ×𝑆 to denote whether the item 𝑖 is recommended to user 𝑢 in the calibrated list. For each user 𝑢, we represent the beyond-accuracy calibrated top-𝐾 list as a binary vector 𝑋 𝑢 = [𝑋 𝑢1 , 𝑋 𝑢2 , ..., 𝑋 𝑢𝑆 ] when 𝑆 𝑖=1 𝑋 𝑢𝑖 = 𝐾. We formulate this calibration as an optimization problem to maximize the total sum of relevance scores constrained by 𝜖-calibration requirement as:EQUATIONwhere 𝐿 𝑆 (𝑋 ) is the top-𝑆 lists, 𝐿 𝑆 , induced from 𝑋 . Any feasible solution of problem 6 recommends exactly 𝐾 items to each user and is guaranteed to satisfy 𝜖-PBAC requirement. The optimal solution has the highest sum of relevance to users possible under the constraints. It can be challenging to select the appropriate 𝜖 value since a small value leads to an empty solution space, while large values for 𝜖 have little impact on PBAC. Thus, we append the PBAC constraint to the optimization objective as a regularized penalty term with importance weight (hyper-parameter) 𝜆 𝑘 for each M k . We also apply a weighting parameter 𝛼 𝑢 for each user 𝑢.We assume all the users are of the same importance and apply same 𝛼 𝑢 to maximize total relevance score 2 . In summary, we replace the objective function in probelm 6 with:EQUATIONWe note that the proposed optimization problem is a mixed-integerlinear programming (MILP) problem. Although MILP is an NP-hard problem, we can use a wide variety of effective heuristic solvers that provide a satisfactory and feasible solution in practice. 3 We also provide a greedy algorithm to make it suitable for the largescale recommendation settings by reducing the MILP to a Linear Programming (LP) problem. This can be achieved by replacing the binary constraint 𝑋 𝑢𝑖 ∈ {0, 1} by 0 ≤ 𝑋 𝑢𝑖 ≤ 1 (Lines 1 and 2 in Algorithm 1) and rounding the optimal 𝑋 𝑢𝑖 values to nearest integer (Line 3). For each user 𝑢, while the total number of recommended items is not equal to 𝐾 (i.e., 𝑆 𝑖=1 𝑋 𝑢𝑖 ≠ 𝐾), we iteratively add (remove) items with the highest (lowest) relevance scores (Lines 8 and 11). This iterative adding and removing naturally results in the desired 𝐾 total number of items recommended to each user, while only selecting the most relevant items. While our algorithm does not provide the global optimum, our experiments indicate that it can provide a near-optimal solution relatively quickly. The pseudo-code is presented in Algorithm 1. 
experimental setup: 
 In this section, we describe our experiments. To ensure reproducibility, we used the open-source recommendation toolkit Cornac [33] , and make our code open source upon acceptance.Datasets. We use two public datasets, MovieLens1M and Yelp. We apply 20-core pre-processing on the datasets to make sure each user/item has sufficient feedback; i.e., each training user/item has at least 20 ratings. Table 2 shows the specs of the datasets, where |C𝑎𝑡 | is the number of categories in the dataset. 2 We leave the impact of varying weights for different user groups to future work 3 For instance, the Gurobi solver (https://www.gurobi.com). Evaluation. We perform a 80/20 temporal split on each dataset for train and test data, respectively. For clustering, we extract the users' beyond-accuracy using the metrics introduced in Sec. 3 and perform a 𝐾-means clustering. We use 𝐾 = 4 based on the elbow method and silhouette score. To evaluate the models' top-10 recommendation list, we employ well-known accuracy-based metrics such as nDCG, Precision, and Recall, as well as the beyond-accuracy metrics of Sec. 3.1. For the optimization experiment, we use the same regularization weight for all beyond-accuracy metrics, i.e., 𝜆 𝑗 = 𝜆 for all 𝑗, and re-ranking is performed on baseline top-50 (𝑆 = 50) recommendation list 4 . Since accuracy and beyond-accuracy metrics usually oppose each other [17] , we define weighted average recommendation performance (ARP) metric to evaluate the overall performance of the models in terms of both types of metrics:EQUATIONΔ(%𝐷𝑖𝑠𝑡 .) is the percentage of improvement in miscalibration degree (see Definition 4.1) and Δ(%𝑛𝐷𝐶𝐺) is the percentage of drop in nDCG metric after optimization. The weighted ARP represents the overall improvement of the recommendation system after optimization. We considers three scenarios for evaluation: higher priority for beyond-accuracy calibration (w=0.3), equal priority for both (w=0.5), and higher priority for accuracy (w=0.7). We tune 𝜆 as a hyperparameter to maximizes 𝐴𝑅𝑃 (0.5) on train data. The results of an ablation study on various 𝜆 is in Sec. 6.Baselines. We compare our approach to various traditional and modern deep learning models, as suggested by Dacrema et al. [10] including:• MostPop: The most popular items are recommended to users using a simple and non-personalized method. The measure of popularity is the number of interactions. • MF [23] : This is an ordinary matrix factorization which decomposes the user-item interaction matrix into lower dimension rectangular matrices. • WMF [16] : This matrix factorization method assumes latent features are independent for a pair of items and assigns smaller weights to negative samples. • BPR [31] : BPR instantiates a zero mean Gaussian prior based on the latent factors of users and items interaction matrix and adds it to the supervised loss as an 𝐿 2 regularization term. • VAECF [26] : This method uses variational autoencoders with multinomial Bayesian inference to estimate the parameters of a generative model. We implement our algorithm using these baselines to show how it achieves the desirable performance on calibration metrics in overall recommendation performance. Evaluation settings. We adopt the baseline algorithms with the default parameter settings of their papers with embedding size of 50 for all baseline algorithms. We set the learning rate to 0.001. Early stopping is applied and the best models are selected based on cross-validation. The model parameters are updated using Adam [21] as the optimization algorithm. 
results and discussion: 
 In this section, notation N is used to represent the original baseline top-10 recommendation while PBAC refers to our optimized reranking method.Improving beyond-accuracy. Fig. 5 presents the average miscalibration across all beyond-accuracy metrics for the users in various clusters, both before and after optimization (see Definition 4.1). The results of the baseline demonstrate that conventional collaborative filtering algorithms are not effective for beyond-accuracy calibration (as mentioned in Sec. 3). Our proposed algorithm improves (reduces) the miscalibration of recommendations for all users in the Yelp dataset, as in Fig. 5b . The degree of improvement varies across models and clusters, with the highest improvement observed in the VAECF method and Cluster 4. The Dist. column in Table 3 shows average improvement on miscalibration for Yelp across all baselines. A lower distance indicates less miscalibration. The reduction of miscalibration degree among clusters (𝐶 1 , 𝐶 datasets, it can be observed that the Yelp dataset exhibits less variation among clusters for reducing miscalibration, with the PBAC algorithm performing relatively consistent across all clusters.The analysis of PBAC algorithm on different clusters in Movie-Lens1M reveals a higher degree of variation, with some clusters experiencing only slight improvement, while others show substantial improvement. Notably, Fig. 5a , highlights that cluster 𝐶 4 has a significant reduction in miscalibration compared to other clusters. This variation in improvement is attributed to the preferences of users in that cluster. It can be seen in Fig. 3a that the center of beyond-accuracy metrics of cluster 𝐶 4 has a higher distance to other clusters, indicating a greater degree of variance in user tastes. This is due to underlying data characteristics, as Yelp has higher sparsity but more balanced interactions (see [41] ). In contrast, the MovieLens1M dataset is heavily influenced by popularity, with certain items receiving the majority of ratings. This popularity bias, as described in [3] , limits the ability of baseline algorithms to recommend novel and unexpected items to satisfy the diverse tastes of users. As a result, the effectiveness of the PBAC algorithm in reducing miscalibration may be limited compared to its performance on the Yelp dataset.Next, we investigate whether our proposed PBAC algorithm improves the overall performance of recommendation with respect to accuracy and beyond-accuracy calibration simultaneously, i.e., whether our algorithm is capable of balancing multiple objectives optimally. This can be addressed by observing the patterns in 𝐴𝑅𝑃 (𝑤) columns in Table 3 , which are the weighted average between the percentage of improvement in miscalibration and the changes in nDCG values after optimization (formally defined at Eq. ( 8)); hence positive values indicate an overall improvement. When assigning higher importance to beyond-accuracy calibration in comparison to accuracy (ARP(0.3) column), all PBAC models improve the overall recommendation performance. In the case of equality of importance between the two objectives (ARP(0.5) column), the PBAC model outperforms the baselines in clusters 𝐶 1 , 𝐶 2 , and 𝐶 3 . As baseline models are already optimized for accuracy, one may expect PBAC models to perform worse than baselines when a higher weight is assigned to the accuracy objective (ARP(0.7) column). Surprisingly, in many cases, PBAC models outperform the accuracy-optimized baseline models as can be seen in the 𝐴𝑅𝑃 (0.7) column in Table 3 , e.g., the MF model in cluster 𝐶 1 with the value of 86.72. Further, the average 𝐴𝑅𝑃 (𝑤) values for weights (0.3, 0.5, 0.7) is (6.8886, 5.0767, 3.2648) indicating an average improvement over baselines in all three scenarios. Hence, our proposed PBAC algorithm effectively balances the trade-off between accuracy and beyond-accuracy in recommender systems for various user clusters. This addresses RQ3 and demonstrates the robustness and versatility of the PBAC model in optimizing multiple objectives in a real-world setting.Baseline recommender model dependency. The underlying recommendation algorithm has a significant impact on the effectiveness of the proposed PBAC re-ranking model. Because baseline recommendation algorithms are inputs to PBAC re-ranking models, so their performance is highly correlated with their effectiveness. Investigating the reduction in miscalibration degree, Fig. 5b illustrate that in Yelp PBAC performs considerably better on the VAECF baseline, followed by MF and BPR, with an average reduction of (25.59%, 12.51%, and 5.38%) among the four clusters. Likewise, the VAECF model exhibits a more substantial average miscalibration reduction than other methods on the MovieLens1M dataset, as shown in Fig. 5a . Essentially this shows that VAECF is learning the underlying user-item representation more accurately and in the most unbiased way. This finding confirms the results reported in [28] on the predictive ability of the VAECF model for beyond-accuracy metrics. On the other hand, among all clusters and datasets, the MostPop algorithm shows the lowest reduction in miscalibration with an average improvement of 1.12%. This was expected as this algorithm is non-personalized, and the items recommended by the baseline recommendation list are similar for all users. Thus, calibrating over parameters such as novelty and surprise for users with different tastes may be less effective since there is not enough variety in recommended items. Comparing the average recommendation performance (ARP), as can be seen in Table 3 , the VAECF model improves beyond accuracy calibration significantly; however, it sacrifices accuracy in the process. This results in negative ARP when higher importance is assigned to accuracy objective, i.e., ARP(0.7). Considering equal importance at ARP(0.5), we see positive values for all models except VAECF in cluster 𝐶 1 , 𝐶 2 , and 𝐶 3 . In addition, when beyond-accuracy calibration is prioritized, ARP(0.3), we can see positive average ARP values among all clusters and baseline algorithms as expected from the PBAC model performance (see Table Table 3 ). This result addresses RQ4 and shows that collaborative filtering algorithms may have difficulty learning user preferences and assigning high relevance scores to items that appear to be relevant to users from both accuracy and beyond-accuracy perspectives simultaneously. Consequently, the effectiveness of the PBAC re-ranking model is affected by the recommendation algorithm and is often associated with trade-offs between accuracy and beyond-accuracy calibration Ablation study. We analyze the impact of the optimization regularization parameter 𝜆 from Eq. (7) . The larger values of 𝜆 are expected to produce recommendations calibrated to a user's taste in beyond-accuracy metrics. However, the excessive pursuit of calibration is unnecessary and could have an adverse impact on the overall recommendation performance. Therefore, we are interested in studying how different values of 𝜆 in Eq. ( 7) can influence the overall performance of the recommender system, i.e., total accuracy (nDCG, Recall), and beyond-accuracy metrics defined in Sec. 3.As indicated in Fig. 6 , 𝜆 exhibits a more "accuracy-centric" behavior, indicating that this parameter affects overall system accuracy more significantly than beyond-accuracy metrics, especially with respect to coverage and redundancy. The reason why an excessively high value of 𝜆 does not benefit us in the overall performance (the combination of beyond-accuracy calibration and accuracy objective), can be explained by the fact that many long-tail and surprising items lack sufficient preference scores (interactions), making it unclear whether including these items in recommendation lists will actually result in user appreciation. Thus, choosing the appropriate model parameters is essential to increasing the level of personalization and overall utility of the marketplace. 
conclusion and future work: 
 This paper addresses the beyond-accuracy metrics calibration from the users' perspective. We demonstrated that users have eclectic preferences when it comes to beyond accuracy qualities such as diversity and novelty. We further showed that recommender systems designed to maximize accuracy fail to calibrate the generated recommendations with respect to beyond-accuracy metrics. We address this issue by utilizing user profile interactions and an efficient re-ranking algorithm; the calibrated recommendations should match user expectations for all clusters of users within a range of tolerance. Extensive experiments confirm that our method can achieve a desirable point in the trade-off between ranking accuracy and beyond-accuracy calibration.For future work, we plan to extend our analysis to multi stakeholder settings, including producers or side-stakeholders, as an additional objective in the optimization framework. We can also analyze personalized beyond-accuracy calibrations in online settings as users' tastes change dynamically over time. 
 
 Appendix: The results obtained with other distance metrics (e.g., L1 norm and cosine distance) are similar.Our preliminary experiments show an increase in 𝑆 enhances the model's performance, however, this comes with a much higher computational cost. 𝑆= 50 appears to be at the balance point of this trade-off. 


Decision: The paper presents a re-ranking approach for recommender systems that emphasize beyond-accuracy metrics, such as diversity, novelty, and serendipity, calibrated to user-specific preferences. The authors propose a post-ranking greedy optimization algorithm to balance accuracy with user preferences. The study is based on the hypothesis that users have distinct, implicit behavioral models that value different metrics in recommendation systems. Experiments are conducted on two real-world datasets.

The major concerns on this paper is its clarity and completeness. Both reviewer rNwv and Yhrb find multiple places where the authors need to clarify more in order for readers to understand the paper clearly, including several problems on tables, figures, and definitions (see the detailed reviews). Yet, while the reviewers have different opinions on the reliability and significance of this paper’s contributions, they agree that the overall merits is above the threshold, so we recommend an accept at the end. If this paper is accepted, we would strongly suggest the authors to revise the paper and address the reviewers’ questions and concerns before submitting the final version.

Reviews: review: main_review: main_review: Figure 2 is incomplete

"Our goal is to examine how much existing datasets support our hypotheses" - no explicit hypotheses is given and second-guessing authors is difficult
 
" we cluster users based on the extracted features with the 𝐾-means algorithm.  " - no description of features given and if you meant the parameters in section 3.1, then say so.
 
Table 1 - data in this table is given in an ad-hoc fashion. what does this mean to say 1 cluster? is this the entire collection?- Please explain.
 
"Figure 4: Distance between user expectations and the provided recommendations." - Please explian this figure properly.
 
 state of the art recommenders  ignores personalised systems - for example, https://arxiv.org/pdf/1706.04148

PBAC is a post-optimization method whereas current personalisation systems are end-to-end
main_review: main_review: The research topic is interesting and relevant to the scientific community. It would be appropriate to extend the experiments conducted and provide a formal and practical pipeline to follow to adapt one's recommendation algorithm to the methodology proposed here.
main_review: main_review: The paper is rather complex, and I got lost in a few places.  I *think* I have understood it correct (viz my summary above), but an initial schematic that introduced that flow would have been nice.  So I encourage you to think carefully about the story-line, and how you are presenting it.

Why is Definition 4.1 in a smaller font than the rest of the text?
And in deed, why is there a single thing given this formal treatment?

In Table 3, is there a "bottom line" summary possible wrt to net effect for the overall set of users, weighting the four clusters by their size to get a "population calculation" outcome?

Why didn't you suppress all the zeros in every second position in the last col of Table 3?

Are the discovered clusters in the two parts of Figure 3 reflecting similar properties?  Looks like not?  Can you give a verbal summary of what type of user is reflected in each of the eight clusters that are depicted, and are there then any similarities between any elements of the two sets of discovered user clusters?  (And, could Figure 3 possibly be made any smaller??)

"Note, nonetheless" in Section 4 is at least one word too long.

I didn't understand what I was to infer from Figure 5, and it needs more explanation/interpretation to take the reader through it.

Check Eqn 5, it didn't feel right to be summing up the sizes of the clusters that the items are members of in the denominator.

Why is there a midline in the top part of Table 1?

Why are the values in table 3 expressed to different numbers of digits?  I understand why the columns would be different, but would normally expect that within a column all would have the same level of accuracy.  (Indeed, when I see different numbers of decimal places in data tables, I start fearing cut-paste errors of various sorts, which is definitely a worry in a research paper.)
main_review: main_review: The paper is easy to read. The idea is (to my knowledge) novel and interesting. The clustering analysis is interesting and provides a novel perspective on a widely studied dataset. There is potential in developing the work further, in particular in relation to temporal drift in accuracy and beyond-accuracy metrics. There is also a sufficient connection with theory - the paper is good fit for this conference.