ENSUR: Equitable and Statistically Unbiased Recommendation
Authors: Nitin Bisht
Xiuwen Gong
Guandong Xu
Venue: ICML 2025 poster (2025)
Keywords: Dynamic Prediction Sets
Primary area: general_machine_learning->supervised_learning

Abstract: Although Recommender Systems (RS) have been well-developed for various fields of applications, they often suffer from a crisis of platform credibility with respect to RS confidence and fairness, which may drive users away, threatening the platform's long-term success. In recent years, some works have tried to solve these issues; however, they lack strong statistical guarantees. Therefore, there is an urgent need to solve both issues with a unifying framework with robust statistical guarantees. In this paper, we propose a novel and reliable framework called Equitable and Statistically Unbiased Recommendation (ENSUR)) to dynamically generate prediction sets for users across various groups, which are guaranteed 1) to include ground-truth items with user-predefined high confidence/probability (e.g., 90\%); 2) to ensure user fairness across different groups; 3) to have minimum efficient average prediction set sizes.
We further design an efficient algorithm named Guaranteed User Fairness Algorithm (GUFA) to optimize the proposed method and derive upper bounds of risk and fairness metrics to speed up the optimization process.
Moreover, we provide rigorous theoretical analysis concerning risk and fairness control and minimum set size. Extensive experiments validate the effectiveness of the proposed framework, which aligns with our theoretical analysis.

ENSUR: Equitable and Statistically Unbiased Recommendation
Nitin Bisht Xiuwen Gong'
Guandong Xu 2
Abstract
the platform's credibility. In recent years, few recommen- dation approaches (Naghiaci et al., 2022; KWEON et al., 2024) are developed for model confidence. However, these methods are heuristic modeling without statistical guaran- tee. Meanwhile, fairness is another critical issue that may harm user experience and undermine platform reliability, Some fairness- based recommendation models have been developed in recent years (Li et al., 2023; Han et al., 2024). While these papers alleviate fairness issues in recommenda- tion systems, they are typically empirically validated with- out statistical guarantees for both performance and faimess.
Although Recommender Systems (RS) have been well-developed for various fields of applications, they often suffer from a crisis of platform credi- bility with respect to RS confidence and faimess,, which may drive users away, threatening the plat- form's long-term success. In recent years, some works have tried to solve these issues; however, they lack strong statistical guarantees. Therefore, there is an urgent need to solve both issues with a unifying framework with robust statistical guar- antees. In this paper, we propose a novel and reli- able framework called Equitable and Statistically Unbiased Recommendation (ENSUR)) to dynam- ically generate prediction sets for users across var- ious groups, which are guaranteed 1) to include ground-truth -tttn items with user-predefined high con- fidencceprobabilityy (e.g., 90%); 2) to ensure user fairness across different groups; 3) to have mini- mum efficient average prediction set sizes. We fur- ther design an efficient algorithm named Guaran- teed User Fairness Algorithm (GUFA) to optimize the proposed method and derive upper bounds of risk and fairness metrics to speed up optimization process. Moreover, we provide rigorous theoreti- cal analysis concerning risk and faimess control and minimum set size. Extensive experiments val- idate the effectiveness of the proposed framework, which aligns with our theoretical analysis.
As a result, we are motivated to develop a complete and statistically guaranteed recommendation framework that considers both model confidence and fairness issues as a whole in this paper. Our overall goal is to construct set pre- dictors that can generate minimum prediction set for each user while guaranteeing model confidence and ensuring user fairness among different groups Thus, objectives of our framework are threefold: (1) to construct prediction sets that cover true item with high user pre-defined probability, say 90% (i.e., confidence level); (2) to guarantee user fair- ness across different groups; and (3) to guarantee minimum average set size while ensuring (1) and (2).
Inspired by Risk-Controlling Prediction Sets (RCPS) (Bates et al., 2021b) a powerful statistical tool, we propose a reliable and fair framework called Equitable and Statisti- cally Unbiased Recommendation (ENSUR) to achieve the above-mentioned objectives. However, RCPS in its natural form, is designed only ensure coverage guarantees. As result, it does not address our key objectives, specifically: 1) how to guarantee faimess among different user groups defi- nitions in a statistical way? 2) how to improve the efficiency of constructing prediction sets when the search range is so large? 3) how to produce recommendation sets with mini- mum size? 4) how to theoretically guarantee the constructed prediction sets meet the risk control and fairness definition as well as minimum set size? To address these gaps, we first define an estimator called faimess metric, which is required to meet the Chimess-Conntollie fined in a similar way as that of risk control. We then build our objective function by minimizing average prediction set while making it meet both RCPS and FCPS constraints for all users across different groups. Subsequently, we derive upper bounds for both the risk and fairness to accelerate
1. Introduction
Recommender Systems (RS) (Aggarwal, 2016; Fan et al., 2022: Sharma et al., 2024) are a type of information filtering system designed to provide suggestions to users based on their preferences. While much effort goes into improving accuracy of these recommendation models, less attention has been paid to model confidence, affecting users' trust in
Prediction Sets (FCPS) de-
Universitt of Technology, Sydney 'The Education Univer- sity of Hong Kong. Correspondence to: Xiuwen Gong <Xi- uwen. Oven.Gong wet@@uusssuuuut.. edu.au>> Guandong Xu <gdxu@ eduhk.hk>.
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ENSUR: Equitable and Statistically Unbiased Recommendation
14
Ug
GUFAAAgonthhm
Ug
lgg
Ug
No
GT
A aFRANNUFF)) sq
M4
14
Ceppl
1g
M4
14
Gr U4
M5
kpuc Dotnents
Grouped Datzerts
Orguus
Prodetion Sets
Figure 1. The proposed ENSUR Framework. Red check marks indicate the true relevant items.
optimization process for prediction set construction. Lastly, we provide theoretical analysis to prove effectiveness set predictors with respect to RCPS and FCPS, and minimum set size. The proposed framework is depicted in Figure 1.
et al., 1999), media streaming (Chang et al., 2017), social networks (He et al., 2024) etc. Credibility and fairness are two crucial factors ensuring the satisfaction of customers and the long-term success of these systems. Traditional rec- ommendation models primarily focused accuracy (Ado- mavicius & Tuzhilin, 2005; Ricci et al., 2010), however, aligning with broader trends in machine learning (Huang et al., 2021; Liu et al., 2019; Zou & Liu, 2023), there is a growing appreciation that model confidence, the reliability of a recommendation, is equally important. However, most of these methods are heuristic modeling without statisti- cal guarantees (Naghiaei et al., 2022). Meanwhile, some fairness-based recommendation models have been devel- oped in recent years, which usually focus on a particular fairness issue in specific fields of application. Fairness in RS can be viewed from diverse perspectives (Li et al., 2023). One such perspective is Individual fairness and Group fair- ness. Individual fairness requires that similar individuals receive comparable treatment. However, defining this simi- larity is challenging due disagreements over task-specific similarity metrics (Dwork et al., 201 1). Group faireess, on the other hand, ensures that protected groups receive treat- ment comparable to that of advantaged groups or the general population (Pedreschi et al., 2009), thus ensuring equitable treatment across predefined groups. It can be further clas- sified from the user side item/platform side. Focusing User-Side group Fairness, it can be defined based on sensitive features like age, gender, race, etc. (Yao & Huang, 2017) utilized gender to distinguish between advantaged and disadvantaged user groups and measured prediction dis- crepancies. Another approach utilizes differentiating groups based on user interactions as defined by (Li et al., 2021a) and (Abdollahpouri al., 2019). To ensure fairness, exist- ing works apply several techniques such as regularization and constrained optimization (Li et al., 2021a; Islam et al., 2021). Some other approaches use Reinforcement Learn- ing by formulating the problem as a Constrained Markov Decision Process (Ge et al., 2021; 2022). To evaluate the fairness, (Yao & Huang, 2017) introduced four group met- ries evaluate collaborative filtering recommender models. (Fu et al., 2020) employed the Group Recommendation Un- fairness (GRU) metric to assess disparities across these user groups. Rahmani et al. (2022) depicted this approach bal- ances faimess with utility under certain conditions. Unlike
Our contributions are summarized as follows:
Firstly, we formulate the recommendation problem from statistically guaranteed perspectives in terms of risk control guarantee and fairness control guarantee, and propose a reliable and fair recommendation frame- work, i.e., Equitable and Statistically Unbiased Rec- ommendation (ENSUR)), which is able to construct minimum prediction set while ensuring the risk control and fairness guarantee for all users in different groups. Secondly, we design an efficient optimization algo- rithm, i.c. Greedy User Fairness Algorithm (GUFA) to optimize the objective function of ENSUR. To ac- celerate the optimization process, we derive the upper bounds for both the defined expected risk and fairness metric via concentration inequalities in Theorem 4.1 and Theorem 4.2 and then make them approach their respective thresholds in a greedy way.
Next, we establish rigorous theoretical guarantees for the proposed framework ENSUR. We prove that the constructed prediction set can achieve risk control and faimess guarantees in Theorem 5.1 while achieving minimal set sizes in Theorem 5.2, which theoretically verifies the effectiveness of ENSUR.
Finally, we conduct comprehensive experiments on top of five commonly used recommendation models and various datasets across multiple domains and fair- ness definitions, demonstrating the empirical efficiency and effectiveness of the proposed ENSUR, which also aligns with our theoretical analysis.
2. Related Works
2.1. Recommendation
Recommender systems (RS) (Ko et al., 2022; Lu et al., 2015) help users make decisions via personalized content in dif- ferent fields of application, such as e-commerce (Schafer
The code and implementation details are available at http:///gtthh.com//aallleeetuuun
2

---

ENSUR: Equitable and Statistically Unbiased Recommendation
the more robust statistical frameworks utilized in general machine learning works (Gong et al., 2023b;a; 2021), these fairness methods do not have notion of statistical guaran- tees. Addressing that gap is focus of our work.
closed set A C IR such that d(.) is nested i.e.
A < A2 = (u) c Φx (u).
(1)
Considering the recommendation setting with implicit feed- back (Hu et al., 2008; Zhu et al., 2024), we define the loss function between the relevant item iime of user u and the prediction set ox to be 0-1 loss as follows:
2.2. Misk-Controlling Prediction Sets
We develop uncertainty quantification for the model con- fidence and faimess based on Risk-Controlln Prediction Sets (RCPS) (Bates et al., 2021b). RCPS is general frame- work, not a specific algorithm, for producing predictive sets that satisty the risk control in Definition 1. Different con- texts require different designs of risk or other estimators to achieve best performance. For example, in the context of medical diagnosis, if set S()) represents plausible diag- noses based on patient features X and R(S) is expected risk of loss from missing true diagnoses, then RCPS ensures this risk to remain below a with confidence 1 - 5. This enables doctors to automatically screen for many diseases (e.g., via a blood sample) and refer the patient to relevant specialists. We will apply framework of RCPS to the designed risk and faimess in the context of recommendation.
{
if iuue & Φx
L(tuue xx(2))
{0
(2)
if imm E Φx.
Using (Bates et al., 2021b), the loss function L L(muse @x(u)) is assumed to also satisfy the following property:
ΦA (w) C (u) = L(buue Φx, (u)) 2 L (fuve a (u))
Based the above loss function, we define the expected risk of not including ground-truth item in the prediction set for all users as follows:
R(AG) = E(L Oxa (u)))-
(4)
Defimition (Risk-controllingg prediction sets (RCPS) (Bates et al., 2021b)). Let S be a random function taking values in space of functions X + y" (e.g., a functional estimator trained on data). We say that is a (a, 6) RCPS if, with probability at least 1 5, we have R(S) S.
Subsequently, require defined risk to meet risk- controlling prediction sets (RCPS), which ensures the proba- bility of risk lower than user-specified threshold a is no less than user-defined confidence level 1 8, namely, reliability of recommendation. This can be formulated as follows:
Pr Pr(R(Aa)= S a)) 2 1 5.
3. The Proposed Framework
Meanwhile, faimess among users in the advantaged groups and the disadvantaged groups is another challenge that needs to be tackled. Notably, in recommendation settings, "ad- vantaged" or "disadvantaged" can stem from various fac- tors tors-such as demographics, engagement patterns, or other Komain-specitte attributes. Thus, we define a fairness metric AF(-) via the difference between the normally used recom- mendation metric (such as hit rate (HR) and DCG) of the advantaged group G1 and the disad group G2, to evaluate user fairness as follows:
In this section, we formulate the objective functions that our framework, i.c., Equitable and Statistically Unbiased Recommendation (ENSUR)), aims to achieve. Firstly, we introduce the notations used in the paper. Consider ni items, denoted as i = [ where each item iy is an element of the item space I. Similarly, we have m users, represented by = [u]m_ where each user uy belongs to the user space U. For brevity, we use u and i for user and item, respectively. The group information G of each user u is known, and following (Li et al., 2021b), we partition users into two groups. G1 and G2. such that G1 n nG2- U to ensure exclusivity. Here, G1 and G2 represent the advantaged and disadvantaged groups, respectively.
and G2 UG2 =
A A = |G1
Σ uEG,
Miosa, (w))
The recommendation is conducted via relevance model m u x I + [o. 1]. which maps a user u and an item i to an estimate score m(u, i), and items with the highest scores are usually the most relevant recommendations. However, there no theoretical guarantee to ensure the confidence of the model's 's output, and so the reliability of the recommended items remains uncertain. In the following, we will follow the framework of cast-oonroolling Prediction Sets (RCPS) Bates et al. (2021a) to solve this gap. We define our set predictor to be o : u + Y, where i' C I is a set-valued output guided by parameter X. This lambda takes values in
M M(xxag (w))
(6)
mEGa
Here, (.) denotes generalized function representing rec- ommendation metric (such as HR DCG) that measures performance of recommendation set dx. (u) for any user u.
For example, when we use hit rate (HR) or DCG as the recommendation metric, we can express them as:
HR(G.) = G
Σ REG.
[(relevant item in dxa, (u)).
3

---

ENSUR: Equitable and Statistically Unbiased Recommendation
upper bounds (AG 8) and AF+ (AG, AG2 5) respec- tively, then it will become more efficient approach the Following
DCG = G MEG,
(OAG, (u));
threshold a and rj by adjusting the the upper bound strategy to accelerate the optimization pro- cedures in (Bates et al., 2021b), we have the optimized risk constraint as follows:
Thus, the fairness metrics can be expressed as:
AHR H HR(G2)), ADEG = [D) DCG(G2)).
Pr(R(Ac) S (RG, 2 2
This design makes our proposed framework more flexible by accommodating different types of RS metrics and diverse user-group definitions.
and () aa for all G e (G1, G2).
Similarly, the optimized fairness metric constraint can be reformulated follows:
Similarly, we require the defined fairness metric to meet the fairness-controlling prediction sets (FCPS), that is, the probability of the fairness metric lower than a user-specified threshold yy no less than user-pre-define confidence level 1- 8, namely, the reliability of fairness. The detailed for- mulation can be expressed as follows:
Pr(AF(Aa: >G,) S AFF (AG AG2. 0)) 2 8 and AFt()G,, Ac, 8) n.
(10)
Consequently, we can choose Ā as the largest value of À such that the entire confidence region the left of À falls below the target risk level a and 1), and the set size will achieve the minimum value. The optimized objective function can be formulated as follows:
(AG, s s) 2 8-
Moreover, we hope constructed prediction sets to be as small as possible while they meet the risk-controlling guarantee as well as the fairness-controlling guarantee. This is because a smaller but more relevant set not only reduces uncertainty in recommendations (Coserato & Bridge, 2023) but also enhances user satisfaction and eases cognitive load (Chen et al., 2022), ultimately improving the usability and effec- tiveness of the RS. Therefore, our goal is to find the optimal (Aa,, AG.) that minimizes the average size of the recom- mendation sets, satisfying the risk (coverage) and fairness guarantees for all users in groups G1 and Gg. The objective function can be formulated follows:
(AG, = (
Ao AG, E [0,
R ()G,8)) Sa, AAG, )c.. s mnj
(11)
To optimize the above objective function and output the optimal solution for (ÅG, (G,) that dominate the validity of set predictor, we design a novel greedy seeed--straegy-baae -based algorithm called Greedy User Fairness Algorithm (GUFA). The complete procedures of the optimization algorithm are summarized in Algorithm 1.
arg min Σ
1
Σ
1 (u)
However, it still remains unknown that what the upper bounds of risk and faimess metric look like. In the fol- lowing part, we will derive the upper bounds in Theorem 4.1 and Theorem 4.2 respectively.
|G|
(Do doa)) ) G2)
NEG
s.t. Pr(R()a) S a)) 2 1 for all G E (Gi, G2),
Pr(AF(AG, Ac.) 5 m) 2 8.
Theorem 4.1 ( (Upper Bound for Risk). Assume loss func- tion Liiuee Oxa (u)) follows a Bernoulli distribution, then upper bound for the risk R(XG) can be found as follows:
(8)
Here, a and 17 are the user pre-specified risk and fairesss thresholds, say 10%; 1 1 8 and - 5 are the user pre-defined confidence level for the risk and fairness, say 90%.
R (Aa, = suu (((AG) (ChomCDF(nBRMaa) n, a) $ {6}
(12)
4. The Optimization Algorithm
where is the number of samples; G E [G1, G2}; R()g)
To optimize the objective function in Equation (8), we need to ensure the risk and fairness metric in the constraints are below decision-maker's pre-defined value a and 17 respec- tively, and finally obtain the optimal prediction set with minimum size. It is not efficient to directly apply the greedy algorithm as the range of risk and fairness values that ap- proach the threshold a and 17 by adjusting the (Aa, AGx) is very large. If we can derive the upper bounds of both risk and fairness metric and take values at their corresponding
denotes the empirical risk of R(AG), which can be calcu- lated as follows:
R((A))
Σ
Mimoes ((imees4rch (r)
(13)
G
WEG
Here, |G] denotes the number of users in group G.
Proof. Proof can be found in Appendix B.I.

---

ENSUR: Equitable and Statistically Unbiased Recommendation
Algorithm 1 Guaranteed User Faimess Algorithm (GUFA)
5. Theoretical Analysis
1: Initialization:
In this section, we provide theoretical analysis on the risk and fairness control guarantee in Theorem 5.1, as well as the minimum set size guarantee in Theorem 5.2.
2: Initialize control parameters for two groups ÀGI AG2
3: Initialize user pre-specified parameters a, 7) 6,
8, A2
Theorem 5.1 (Risk and Fairness Control Guarantee). For all group G E {G1, and E (0,1), with probability of at least 1 8 for risk threshold a, and with probability of at least 1 8 for fairness threshold r), have:
4: Define Loss as in Equation (2)
5: Define Fairness metric as in Equation (6)
6: Adjustment Loop:
7: for users in each group G E {G1, G2) do
8: Calculate (Ag, 6) such that Pr(R(AG) 2
Pr(B(Ac) sa) 2 1
(AG. 5)) 2 1 -
9:
Compute AF(AG)) AG2) and calculate AFt (AG AG2, 5) such that Pr(AF(Aa,, AG²) S
A APr(AF(GG dox) Sn) 2 8.
(15)
AFT (Aai Aa2, 6)) 2 1 1-8
10:
if RG(AG,0) 8) > a OR AF 0) > 1
Proof. Proof can be found in Appendix B.3.
then
Update sG, t AG. AI, t AG2- -2
Remark. In Theorem 5.1, we prove that the optimal s, )G, obtained from Algorithm 1 are indeed able to control the expected risk to below the decision makers' defined values of a with confidence 1 1 6, and control the faimess metric AF to below the decision makers' defined values of with confidence 1 ò. This theoretically validate the recommendation reliability and faimesss of the proposed ENSUR framework.
12: end if
13: end for
14: λG
XGI AG2 Get the optimal Aaa Ag2
15: Construct Prediction Sets:
16: for each user u in group G do
17: 23o0 (u) + {il m(u,i) i) 2 Aa}
end for
Output: the optimal solution ÅG₁ Åo, and prediction sets Sxo, (u) and &xan (u) for all users in different groups.
Theorem 5.2 (Minimum Set Size Guarantee). Let (6xe)' @Ncz) ) be any set predictor and let (Øxa,' Phaz) be the optimal predictor obtained from Algorithm 1 such that R(Xg) S R(Aa) and AF Nox)) S AF(AG, )
Then for each G E (GG, G22,, we have:
Theorem 4.2 (Upper Bound for Fairness Metric). The upper bound for fairness metric AF (AG,, AG,) can be derived by applying Bernstein inequality (Maurer & Pontil, 2009) as follows:
11l0so (w)l s E [1xx
(16)
where 16x u in group G.
(u) denotes the predicted set size for any user
AFt AFt( Acc,,),,,8) 8)
20 log
(2)
+
2/2 log
(
(14)
Proof. Proof can be found in Appendix B.4.
n1 + 12
Remark. In Theorem 5.2, we prove that set predictor learned by our algorithm output the minimal prediction set size for any user u in group G, which theoretically vali- date the effectiveness of the proposed ENSUR framework.
where ny and n2 denote the number of samples for group Gi and G2; of denotes the variance associated with the fairness metric Amr or ApcG- The detailed formulation of the variance can be referred to in Appendix A.
To sum up, set predictors constructed by Algorithm 1 can modify any black-box recommendation models to output prediction sets for new customers that are strictly guaranteed to satisfy the risk control as defined in Equation (5) and the faimess control defined in Equation (7) while ensuring the minimum prediction sets in Equation (8).
Proof. Proof can be found in Appendix B.2.
Recommendation After obtaining optimal (Aa Aga) from Algorithm 1, we can recommend new items for users. For example, when user u comes, we first decide on group G that they belong to and then utilize corresponding Xg to calculate their prediction set via step 17.
6. Experiments
In this section, we conduct experiments to validate the effec- tiveness of the proposed framework (ENSUR). We design
5

---

ENSUR: Equitable and Statistically Unbiased Recommendation
experiments 1) validate whether the framework can pro- vide desired coverage guarantee in terms of risk, better performance in terms of average set size, and improved fairness in terms of Hit Rate Difference (Hit Rate Diff.) and DCG Difference (DCG Diff.) across various datasets with sensitive attributes; 2) analyze how the parameters ( i.e a, 6, 7 and 8) influence the performance; 3) analyze the time-efficiency of ENSUR compared to other fairness baselines.
Table I. Performances and fairness comparisons with base mod- els and fairness baselines on AmazonOffice Dataset grouped by the Interactions terms of risk, average set size, and Hit Rate Diff/DCG Diff, respectively. Bold indicates best result, underline indicates the second best and i marks threshold exceeded cases.
Meehod
Averase Size
DCG
bill
DasPEM
0,418
0.155
0.17
10277
34
10725
0.248
DeepEM ENSUR
0.190
0.900
0,401
0.881
0J303
01389
0.788
GMP
0,121
0.3281
10361
30
0.69
0.214
GMF ENSUR
0.197
0.80
0,428
0.16
00363
01.3
6.1. Datasets and Base Models
LigHTENN
00223
0,477
01126
0.218
33
We conduct experiments on four datasets with specific sen- sitive user attributes: (1) AmazonOffice dataset (eCom- merce) (MeAuley et al., 2015) grouped by item interac- tions; (2) Last.fm dataset (music streaming) (Cantador et al.,
LibiGCN ENSUR
0067
0
03913
0,474
0.276
0.087
0.198
MLP
6.409
0.219
0.19
0.81
26
0.61
0,210
MIP + ENSUR
0.197
0.307
0.013
0.14
0.184
0816
0.257
NauMF
0.155
01945
0,414
0.235
0.185
2011) grouped by region (developed and other countries;
00379
24
D621
0,339
NOMME ENSUR
0/406
0.012
0.143
(3) MovieLens dataset (movie ratings) (Harper& & Konstan, 2015) grouped by gender; and (4) Book-Crossing dataset (book ratings) (Ziegler et al., 2005) grouped by age. We implement the proposed framework on five base recommen- dation models: DeepFM (Guo al., 2017), GMF (Koren et al., 2009), MLP (Zhang et al., 2019), NeuMF (He et al., 2017), and LightGCN (He et al., 2020). Additionally, we compare our framework ENSUR with four fairness base- lines: 1) NFCF (Islam et al., 2021) 2) MFCF (Islam et al., 2021) 3) GMF-UFR (Li et al., 2021a) 4) NCF-UFR (Li et al., 2021a). The implementation details and the details of all the datasets, base models, and faimess baselines can be found in Appendices anddI D.
Other Hosellaes
NKC
0.1W
28
00800
0.991
0.334
0.361
BAW
6.357
MFCF
BLFS
90
0022
0,402
01128
0151
NAUME-UFR
26
0330
0.356
0.183
0.309
GME-UER
00:00
30
0.995
0.395
0.13
0.157
0066
0.278
and fairness guarantees across all datasets. The ENSUR-enhanced models always meet risk below defined thresholds. For base models, the minimum risk threshold criteria is frequently not met. For example, in the AmazonOffice Dataset, we notice, as depicted by 1. that risk thresholds are not met for at least one group, i... the disadvantaged group across all the base models. In fairness baselines, we observe criteria are not met for both the groups in most cases across all datasets, which may be because of their emphasis on trading off performance for accuracy.
6.2. Experimental Results
6.2.1. RESULTS W.R.T PERFORMANCE AND FAIRNESS
We compare the performance and fairness of the ENSUR framework with five base recommendation models and four fairness baselines. We set the predefined risk threshold a = 0.20, fairness threshold v = 0.20 via manual valida- tion. The error rates = 0.1 and 8 0.1 are representatively set following (Bates et al., 2021b). The coverage guarantee measured in terms of risk: performance measured using average set size, and fairness is compared using disparity in these metrics between user groups (Difference in Hit Rate and Difference in DCG). The results for the AmazonOffice dataset (grouped by interactions) are provided in Table whereas results for MovieLens dataset (grouped by gender), Last.fM dataset (grouped by region), and Book-Crossing dataset (grouped by age) are provided in Tables 3 to 5 re- spectively in Appendix E.
We also observe that the ENSUR- enhanced models can get the best results in average set size on all the datasets, but the best model varies among different datasets. For example, MLP + ENSUR achieves the best recommendations in terms of average set size on the AmazonOffice dataset. Similar trends are observed for MovieLens, fM and Book-Crossing datasets as depicted in Tables 3 to 5 in Appendix E. All ENSUR censuReen models meet the fairness thresh- old for both the Hit Rate Diff and DCG Diff across all datasets. However, the best-performing models vary by dataset. For example. MLP + ENSUR achieves the best fairness on the AmazonOffiice dataset under the Hit Rate Diff while DeepFM + ENSUR outperforms all the other models in terms of DCG Diff. Tables 3 to 5 in Appendix E depict similar trends for remaining datasets. In addition, the base models do not always achieve the faimess metrics and exceed the fairness threshold marked by f. Meanwhile, the fairness base-
The results, presented in Table lead us to the following key observations:
The ENSUR framework ensures that all base models generate prediction sets that satisfy both risk control
6

---

ENSUR: Equitable and Statistically Unbiased Recommendation
line models do achieve faimess metrics after sacrificing their accuracy, but they are still inferior the ENSUR- enhanced models.
the other datasets, see Figures 12 14 in Appendix E.2.
Finally, we examine trends average prediction set sizes by varying fairness confidence 8 from 0.10 to 0.50 (in in- crements of 0.05) under fixed faimess thresholds (17 = 0. 0515, 0.20, 0.25) measured on Last.fm dataset grouped by region (Figure 5). We notice that as value of 8 increases, for given faimesss threshold, model becomes less conservative, and hence prediction set size decreases. This phenomenon further validates effectiveness of our framework in balanc- ing between producing tight average prediction set size and ensuring fairness. Similarly, results for other datasets can be found in Figures 15 to 17 in Appendix E.2.
Overall, the ENSUR framework effectively ensures both recommendation performance and fairness while guaranteeing risk control, providing valuable insights for real-world applications. We further discuss the generalizability of grouping strategies and practical ap- plicability in Appendix F and Appendix G respectively.
6.2.2. PARAMETER ANALYSIS
We further analyze influence of pre-defined risk-related pa- rameters a and ó and fairness-related parameters and on the prediction sets generated by ENSUR framework.
Overall, this parameter analysis guides real-world applica- tions in balancing performance and fairness with confidence guarantees.
Effect of Risk Control Parameters a and on Prediction Set Sizes We first evaluate the impact of error rate o varying from 0.10 to 0.50 (in increments of 0.05) on average prediction set sizes under fixed risk confidence thresholds 5 = 0.05, 0.10, 0.15 using AmazonOffice dataset, grouped by interactions in Figure 2. It can be easily observed that as a increases, the average set size across all models decreases. The decreasing trend demonstrates the framework's ability to generate valid prediction sets that adapt to the error rate a. Similar trends can be observed remaining datasets, see Figures 6 to 8 in Appendix E.2.
6.2.3, TIME EFFICIENCY COMPARISON
We analyze the computational cost (training time) of the ENSUR framework in comparison with other faimesss base- lines. Specifically, for in-processing fairness baselines such as NFCF and MFCF, we consider the fine-tuning step to calculate the training time. For post-processing fairness baselines such as NeuMF-UFR and GMF-UFR, we take the re-reranking step as the training time. For proposed ENSUR framework, we take the calibration step as the training time. We measure the time of ENSUR, averaged on top of all the base models. Our experiments are conducted via 10-fold cross validation to ensure statistical reliability. The results are presented in Table 2.
We further evaluate effect of varying risk confidence ò from 0.10 to 0.50 (in increments of 0.05) on average prediction set sizes under fixed risk thresholds (a- = 0.15, 0.20, 0.25) using Croks-Cossing dataset, grouped by age (see Figure 3). In general, all the models show a decreasing trend which validates effectiveness of the proposed framework. Interest- ingly, prediction set sizes do not seem to fluctuate much for smaller values of 6, while a decreasing trend occurs with increasing 6. This is because relaxing confidence of risk constraints makes our predictions less conservative, thereby reducing the number of items included in prediction set. Similar phenomenon can be obtained on the other datasets, see Figures 9 to 11 in Appendix E.2.
From the results, we can observe that proposed frame- work ENSUR is significantly more time-efficient than other faimnsss baselines, which indicates scalability of our method. This is because in-processing methods like NFCF and MFCF involves model refitting which substantially in- creases the computational cost. By contrast, ENSUR oper- ates independently of the training phase, eliminating this overhead. Additionally, ENSUR is substantially faster than NeuMF-UFR and GMF-UFR, other post-proeee processsonssing meth- ods, because these models involve solving constrained and complex optimization problem, whereas ENSUR employs a simple yet effective greedy- algorithm.
Effect of Fairness Control Parameters 1) and on Pre- diction Set Sizes: : We analyze how varying 7) measured by the Hit Rate Diff. and DCG Diff. from 00 to 0.50 (in increments of 0.05) on the average prediction set sizes un- der fixed fairness confidence (8 0.15, 0.20, 0.25) affects average prediction set sizes, measured the Movielens dataset grouped by gender (Figure 4). With increasing 17, the prediction set size decreases, validating model's capacity to have smaller prediction sets for less strict condition. The prediction set sizes usually stabilize after an initial decrease as 1) rises, suggesting that the framework's 's fairness sensi- tivity to 17 diminishes beyond a certain point. This offers guidance on selecting appropriate fairness thresholds while maintaining usability. Similar results can be observed
Table 2. Training time (minutes) comparison of our framework ENSUR with four fairness baselines.
Dataset
MFCF
NFCF
GMF-UFR
NouMF-UER
ENSUR(OUrS)
AmazonOttice
45
50
25
22
MovieLens
78
90
49
45
12
Last.fm
50
58
35
30
8
Book-Crossie Crossing
110
65
7

---

ENSUR: Equitable and Statistically Unbiased Recommendation
DoepFM +GLFR
DespFM + +GURR
25
GMF
+GURS
Average Set
Neulif +BUPR
Keulir +GUFR
25
DeeoFM +GUES.
-9
00
20
UgMtSCN
HLP +GUFR
15
Neute +GUPR
01
0.2
0.3
0.4
0.5
0.2
0.2
0.3
0.4
0.5
0.2
0.3
0.4
0.
(a) = 0.05
(b) 8 = 0.10
(c) 8 = 0.15
Figure 2. Analysis of base models after applying the ENSUR framework in terms of average set size with varying a
(0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on AmazonOffiie datasct grouped by Inferactions under different 8.
26
35
34
Average
Deeol'M +GUFS
Deepf
Deeof'M +OUER
22
-CUFI
UgNiSCN
UgC
UgMAGCN
HIP +GUR
MLP +GUFR
+GUFR
Kouse IGUFR
NouMF (GUFA
Nou'F GGUFA
28
0.1
0.2
03
0.4
0.5
0.3
0.2
03
0.4
05
01
0.2
D3
0.4
05
5
(a) a = 0.15
(b) = 0.20
(c) a = 0.25
Figure 3. Analysis of base models after applying the ENSUR framework terms average set size with varying
(0.10, 0.15 15, 0.20, 0.25, 0.30. 0.35, 0.40, 0.45, 0.50} on Book-Crossing dataset grouped by Age under different
50
Deepli GGUER
Avrrigg Set Sam
40
GMF GURR
LightSCh
18
20.0
DeepFM
DeepFN
+GLF
GMF HOUFR
3
GUPR
LightCh IGUFR
LightDON rGUF
12
NeuMt -CUPR
MLP GSUF
Averege
12.5
Neuhr +GUIR
MLP +CUFR
10.0
10
0.1
6.2
0.3
0.4
0.5
0.3
0.2
0.3
0.4
05
01
022
0.3
0.4
03
0
(a) 8 0.15
(b) 8 0.20
(c) 8 0.25
Figure 4. Analysis of base models after applying the ENSUR framework in terms of average set size with varying 1 (0.10, 15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on Movielens dataset grouped by Gender under different 8.
45
DeepFM
48
40
+GUF
Averoge Set 52e
UgreSCN +GUFR
Average Set Size
LightSC +GUF
DeepPM GUFR
MLP +SUFR
MLP +GUFR Noutte +UUP
GMF +GUFR
NouMt +OUPH
35.
Average
30
MLP
Weutt +GURR
25
20
25
11 0.15
(b) 1) 0.20
(c) 1) 0.25
Figure 5. Analysis of base models after applying the ENSUR framework in terms of average set size with varying î (0.10, 051 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on Last.fm dataset grouped by Region under different 7)
8

---

Chen, D., Yan, Q. Chen, C.,, Zheng, Z., Liu, Y., Ma, Z., Yu, C., Xu, J., and Zheng, B. Hierarchically constrained adap- tive ad exposure in feeds. In Proceedings of the 31 ACM International Conference on Information & &Knolldgge Association for ENSUR: Equitable and Statistically Unbiased Recommendation
7. Conclusion
Bates, S., Angelopoulos, A., Lei, L.,, Malik, J., and Jordan, M. I. Distribution-free, risk-controlling prediction sets, 2021b.
This paper investigates two principle issues that affect the credibility of RS with respect to confidence and fairmess. We integrate the two factors into a unified framework called Equitable and Statistically Unbiased Recommendation (EN- SUR)), which dynamically outputs prediction sets that are guaranteed to have the risk and fairness below a threshold with pre-specified high confidence, such as 90%, while re- taining the minimum average size. We conduct theoretical analysis and empirical studies, which are consistent in vali- dating the effectiveness. is noteworthy that the efficiency of optimizing the ENSUR also depends on the tightness of the derived upper bounds for our risk and fairness, thus, we leave the question whether there exists tighter upper bounds for the future work. Moreover, the proposed framework can work on top of any recommendation model by taking them as black-box, which offers robust foundation for advanc- ing fairness and reliability in RS, paving the way for future research and development in this field.
Cantador, I., Brusilovsky, P., and Kuflik, T. Second work- shop on information heterogeneity and fusion in recom- mender systems (hetrec2011), In Proceedings of the Fifth ACM Conference on Recommender Systems, RecSys "11, pp. 387-388. Association for Computing Machinery, 2011.
Chang. S., Zhang, Y., Tang, J., Yin, D., Chang, Y. Hasegawa-Johnson, M. A., and Huang, T. S. Stream- ing recommender systems. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pp. 381-389. International World Wide Web Conferences Steering Committee, 2017.
Acknowledgments
Management, CIKM '22, pp. Computing Machinery, 2022.
This work is partially supported by the Australian Re- search Council (ARC) Under Grants DP220103717 and LE220100078, and the National Natural Science Founda- tion of China under Grants No.62072257.
Coscrato, V. and Bridge, D. Estimating and evaluating the uncertainty of rating predictions and top-n recommen- dations in recommender systems. ACM Trans. Recomm. Syst., April 2023.
Impact Statement
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness, 2011.
Our framework dynamically tailors prediction set sizes in recommender systems, ensuring fairness and performance guarantees while reducing cognitive overload and resource inefficiencies. By addressing disparities in user experiences and promoting equitable recommendations, it advances in- clusivity and transparency in user-centric platforms, with applications across e-commerce, streaming, and beyond.
Fan, W., Liu, X., Jin, W., Zhao, x., Tang, J., and Li, Q Graph trend filtering networks for recommendation. In Proceedings of the 45th International ACM SIGIR Con- ference Research and Development in Information Retrieval, SIGIR '22, pp. 112-121, 2022.
References
Fu, 7., Xian, Y., Gao, R., Zhao, J., Huang, Q., Ge, Y., Xu, S., Geng, S., Shah, C., Zhang, Y., and de Melo, G. Fairess-awaae explainable recommendation over knowl- edge graphs, 2020.
Abdollahpouri, H., Mansoury, M., Burke, R., and Mobasher, B. The unfairness of popularity bias in recommendation, 2019.
Ge, Y., Liu, S., Gao, R., Xian, Y., Li, Y., Zhao, X., Pei, C., Sun, F., Ge, J., Ou, W., and Zhang, Y. Towards long-term fairness in recommendation.
Adomavicius, G. and Tuzhilin, A. Toward the next gener- ation of recommender systems: a survey of the state-of- the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering, 17(6):734-749, 2005.
Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM '21. ACM, March 2021.
Aggarwal, C. C. Recommender Systems: The Textbook. Springer Publishing Company, Incorporated, 1st edition, 2016.
Ge, Y., Zhao, X., Yu, L.,, Paul, S., Hu, D., Hsieh, C.-C., and Zhang, Y. Toward pareto efficient fairness-utility trade- off in recommendation through reinforcement learning. In Proceedings of the Fifteenth ACM International Con- ference on Web Search and Data Mining, WSDM '22. ACM, February 2022.
Bates, S., Angelopoulos, A., Lei, L., Malik, J., and Jordan, M. Distribution-fret risk contooonng prediction sets. J. ACM, 68(6), sep 2021a
9

---

ENSUR: Equitable and Statistically Unbiased Recommendation
Gong, X., Yuan, D., and Bao, W. Understanding partial multi-label learning via mutual information. In Advances in Neural Information Processing Systems, volume 34, 2021.
Huang. X., Du, B., and Liu, W. Multichannel color image denoising via weighted schatten p-norm minimization. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI'20, 2021. ISBN 9780999241166
Gong, X., Yuan, D., and Bao, W. Discriminative metric learning for partial label learning. IEEE Transactions Neural Networks and Learning Systems, 34(8):4428- 4439, 2023a.
Islam, R., Keya, K. N., Zeng, Z., Pan, S., and Foulds, J. Debiasing career recommendations with neural fair col- laborative filtering. In Proceedings of the Web Conference 2021, www '21, pp. 3779-3790, New York, NY, USA, 2021. Association for Computing Machinery.
Gong, X., Yuan, D., Bao, W., and Luo, unifying probabilistic framework for partially labeled data learn- ing. IEEE Transactions Pattern Analysis and Machine Intelligence, (5(7))8036-80444 2023b.
Ko, H., Lee, S., Park, Y., and Choi, A. A survey of recom- mendation systems: recommendation models, techniques, and application fields. Electronics, 11(1):141, 2022.
Guo, H., Tang, R., Ye, Y., Li, Z., and He, X. Deepfm: A factorization-maching based neural network for prediction, 2017.
Koren, Y., Bell, R., and Volinsky, C. Matrix factorization techniques for recommender systems. Computer, 42(8): 30-37, 2009.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2024. URL https: : / www. . gurobi com.
KWEON, W., Kang, S., Jang, S., and Yu, H. Top- personalized-K recommendation. In The Web Conference 2024, 2024.
Han, Z., Chen, C., Zheng. X., Liu, W., Wang, J.. Cheng, W., and Li, Y. In-processing user constrained dominant sets for user-oriented fairness in recommender systems. In Proceedings of the 31 ACM International Conference on Multimedia, MM 23, pp. 6190-6201. ACM, October 2023.
Li, Y., Chen, H., Fu, Z., Ge. Y., and Zhang. Y. User-oriented fairness in recommendation. In Proceedings of the Web Conference 2021, WWW '21. ACM, April 202 la.
Han, Z., Chen, C., Zheng, X., Zhang, L., and Li, Y. Hyper- graph convolutional network for user-oriented fairness in recommender systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, pp- 903-913. Association for Computing Machinery, 2024.
Li, Y., Chen, H., Fu, Z., Ge, Y., and Zhang, Y. User-oriented fairness in recommendation. In Proceedings of the Web Conference 2021, WWW '21, pp. 624-632. Association for Computing Machinery, 2021b.
Li, Y., Chen, H., Xu, S., Ge. Y., Tan, J., Liu, S., and Zhang, Y. Fairness in recommendation: Foundations, methods and applications, 2023.
Harper, F. M. and Konstan, J. A. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 2015.
Liu, W., Shen, X., Du, B., Tsang, I. W., Zhang, W., and Lin, X. Hyperspectral imagery classification via stochastic hhsvms. IEEE Transactions on Image Processing, 28(2): 577-588, 2019. doi: 10.1109/T1120018288666111
Hc, X., Liao, L.. Zhang, H., Nic, L., Hu, X., and Chua, T.-S. Neural collaborative filtering, 2017.
He, X., Deng, K., Wang, X., Li, Y., Zhang, Y., and Wang. M. Lightgen: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd
Lu, J.. Wu, D., Mao, M., Wang, W., and Zhang, G. Rec- ommender system application developments: a survey. Decision support systems, 74:12- 2015.
International ACM SIGIR conference on research and development Information Retrieval, pp. 639-6699966 00986488222 2020.
Ma, H., Xie, R., Meng. I.,, Feng, F,, Du, X., Sun, X., Kang, Z., and Meng. X. Negative sampling in recommendation: A survey and future directions, 2024.
He, X., Liu, Q., and Jung, S. The impact of recommenda- tion system on user satisfaction: A moderated mediation approach. Journal of Theoretical and Applied Electronic Commerce Research, 19:448 466, 02 2024.
Maurer, A. and Pontil, M. Empirical bernstein bounds and sample variance penalization, 2009.
Hu, Y., Koren, Y., and Volinsky, C. Collaborative filtering for implicit feedback datasets. 2008 Eighth IEEE international conference on data mining, pp. 263-272. leee, 2008
McAuley, J., Targett, C., Shi, Q. and van den Hengel, A. Image-based recommendations on styles and substitutes, 2015.
10

---

ENSUR: Equitable and Statistically Unbiased Recommendation
Naghiaei, M., Rahmani, H. A., Aliannejadi, M., and Sonboli, N. Towards confidence-aware calibrated recommendation. Proceedings of the st ACM International Conference on Information & Knowledge Management, 2022.
Appendix
A. Assumptions
Assumption A.I. theorem 4.2, we assume that the groups
Pedreschi, D., Ruggieri, S., and Turini, F. Measuring dis- crimination in socially-sensitig decision records. In Pro- ceedings of the 2009 SIAM International Conference Data Mining, pp. 581-581151 592. SIAM, 2009.
G₁ and G2 are mutually independent and that hit rates or DCG scores independently distributed within each group. Under these assumptions, the variances for the fairness met- rics are calculated as follows:
P1(1 - P1)
P2(1 - P2)
Rahmani, H. A., Naghiaei, M., Dehghan, M., and Alianne- jadi, M. Experiments on generalizability of user-oriented fairness in recommender systems, 2022.
T
112
81
$2
DCG
Ricci, F., Rokach, L.,, and Shapira, Recommender Sys- tems Handbook, volume 1-35, pp. 1-35. Springer-Veriie Verlag Berlin, Heidelberg, 10 2010.
71
122
where Pr and P2 are the observed hit rates, and of and $2 are the sample variances of DCG scores for groups G1 and G2, respectively.
Santos, H. G. and Toffolo, T. Mixed integer linear program- ming with python. Accessed: Apr, 2020.
This assumption ensures that the application of Bernstein's inequality is valid, allowing us derive the Upper Confi- dence Bound (UCB) for faimess metrics as shown in 14.
Schafer, B., Konstan, J., and Riedl, J. Recommender sys- tems in e-commerce. 1st ACM Conference on Electronic Commerce, Denver, Colorado, United States, 10 1999.
Assumption A.2. Throughout the Theorem 5.1, make mild assumption A i.e., the minimum value the parameter À can take, as follows:
Sharma, A., Li, H., Li, X., and Ji Optimizing novelty of top-k recommendations using large language models and reinforcement learning. arXiv preprint arXiv:2406. 14169, 2024.
Pr Pr(Rc(AA") S a) 2 1-8APT(AF min min C1 < 2 1-6
where X is the group-specific minimum value of the pa- rameter for risk control, and mim and A are the minimum values for fairness control across the groups.
Yao, S. and Huang, B. Beyond parity: Faimeess objectives for collaborative filtering, 2017.
This assumption depicts the belief that we can control any user-defined risk and fairness e by taking valid values in a closed set A C R² U {=o0).
Zhang, S., Yao, L.,, Sun, A., and Tay, Y. Deep learning based recommender system: survey and new perspectives. ACM computing surveys (CSUR), 52(1):1-38, 2019.
Zhu, H., Xiong, F., Chen, H., Xiong, X., and Wang, L. Incorporating a triple graph neural network with multi- ple implicit feedback for social recommendation. ACM
B. Proofs
B.1. Proof of Theorem 4.1
Transactions the Web, 2024.
Proof. We focus finding some RG such that out of samples, RG yields atmost k: = nRG successes (where success is defined observing risk) with a significance level of atleast 1 8. The CDF of the binomial distribution is given by:
Ziegler, C.-N., MeNee, S. M., Konstan, J. A., and Lausen, G. Improving recommendation lists through topic di- versification. In Proceedings of the 14th International Conference World Wide Web, WWW '05. Association for Computing Machinery, 2005.
k
Zou, X. and Liu, W. Generalization bounds for adversar- ial contrastive learning, 2023. URL https: /arviv. org/abs /232. 10633.
P(Binom(a,p) <
Σ ==0
(1)n''
12 p²
p)-
Let us assume we know Pt and we seek Rig such that:
P(Binom(n,, (tt) S nlia) 2 d.
Replacing R&G with the user-defined risk value a, the equa- tion becomes:
P(Binom(n, a) S nRa) 2

---

ENSUR: Equitable and Statistically Unbiased Recommendation
or
B.3. Proof of Theorem 5.1
P(Binom(n,.)) a)) S nRe)= 8
Proof. Let Xg be the highest parameter value for each group Ge {G1, G2] such that the expected risk of not including truly relevant items and the fairness metric is less than and yy i...
which can be reformulated as:
BinomCDF(nRRG. n, a) S
To solve for RG, we find the root of this equation which is also the UCB at a: i.e.,
N/= max(Ag E [miin, >max,c]
BinonCDF(nRR,, n, 8=0 0
Ra(Aa) I a AFo(Aa, A6a) S n) (18)
Formally,
Assume for a parameter value AG, we have Ra(Aa) > a or ('G,,, AG2) > a.
R = sup RG BinomCDF(nnR a) {o}
Then by the definition of Xg. we have,
Hence Proved.
Ra S SaA No.) Sn
B.2. Proof of Theorem 4.2
which implies,
Proof. Bernstein's inequality for a sum of independent ran- dom variables X, with mean μe, variance o², and bounded by U states:
Ro(Aa) S < Ra(Aa)V
AF(AG) S < AF(Aa, )oy)
(X, 1)) 21) S 2exp (-op nt
Using Equation (3), we have:
n
ÅY N/2
(17)
where is the number of observations, X, is the i-th random variable, t is the deviation threshold, o² is the variance of X, and is the upper bound on the range of X,.
Since ÅG and X are within the range of real numbers, consider some & > 0 such that
(Na + E) 2 Åo.
Analogously, we consider, with some decision-make con- fidence value 6, that the empirical fairness metric differs from the true fairesss metric by the threshold This can be mathematically represented
Utilizing the definition of X/ and do in Equation 18, we get,
+ S < < Ra VAF+ (A No, + S < AF(?G Xo. + E)
(-
nt²
8 = 2ex
o7 + +11
(19)
which rearranges to:
According to the principles of Upper Confidence Bound (UCB), i.e., eq. 11, the events R {, S or (2 No. +, S 7) can only occur with probabil- ities not exceeding and â respectively. Specifically, the UCB ensures that the probability of observing Ro (Aa) > is bounded by 6, the probability of AF )G,) " is bounded by 8.
n1
log
8
o +
UL'
solving for f gives:
t= 2 log 2 + 2
log
n
Therefore, with complementary probability condition, under Assumption and the defined ranges of å and ô, we can conclude with confidence that:
Assuming U conservatively and n2,, obtain the UCB as:
Pr(Ro(Aa) 5a) 5 ^
A (AG1: 8) 8) AF()G AG2)
(20)
Pr(AF(Aa, So,) 5)) > 1 8.
20 log(? { + log(?)
+
This validates the assertions of Theorem 3, thereby formally proving the theorem.
n1 + n2
Hence Proved.
12

---

ENSUR: Equitable and Statistically Unbiased Recommendation
B.4. Proof of Theorem 5.2
D. Detailed Experimenation Details
R and A May) I AF xo, 'o,), this relationship
Proof. Since Ralds..))
D.1. Datasets and Grouping Methods
In the main paper, we introduced four user grouping strate- gies to evaluate the fairness and performance of our frame- work: (1) grouping based on interaction count with items, (2) grouping based user age, (3) grouping based on user gender, and (4) grouping based geographic categoriza- tion into developed and other countries. These strategies were applied to the AmazonOffice, Book-Crossing, Movie- Lens, and Last.fm datasets, respectively. Below, we provide further details on the grouping methodology:
is expressed through the sum of relevance scores m(u,a) i) over the items in the respective prediction sets for users:
EE m(u,i)
1)
Σ
Σ
m((u
BEG SEGSE (w)
BEG ( (1)
indicating that the accumulated scores of included items in Øxg are greater.
This is equivalent to:
Following Li et al. (2021a), users were initially evenly split into two groups, with 50% assigned to each group. The groups were then dynamically adjusted to ensure that the min- imum interaction count in the advantaged group ex- ceeded the maximum count in the disadvantaged group by at least one.
Grouping by interaction count:
Σ
Σ
m(u,)) 2 Σ
Σ
m(u, i).
UEG ( (u)ios (w)
BEG vEdsa (u)losz( (u)
For some items E @x (u) \ &io (u), m(u.0) 1) Åo and for all items i @2 (u) \ 0ag (w), m(u, 2 Åc based Algorithm 1.
Grouping by age: Users were divided into two age groups: younger users ($ 60 years) and older users (> 60 years).
This condition is satisfied if:
Grouping by gender: Users were grouped into bi- nary categories based on identified gender (male and female).
(xx(")) 2 1Φsa (u)l-
Thus, the expected size of the set using &xa is optimized to be minimal, i.c.,
Geographic categorization: Users were categorized based on their country of origin into developed (e.g.. USA, UK, Europe, Japan etc.) and other countries.
E' I0 (w)) s E [10x((1)]]
(21)
Furthermore, we conducted additional grouping exper- iment on the Last.fm dataset. We extended the interaction count-based grouping to incorporate interactions with popu- lar items, following Abdollahpouri et al. (2019). The results of this experiment are provided in Appendix F.
thereby proving the theorem.
LI
C. Implementation Details
All base recommender models are trained for 20 epochs with a batch size of 256, a learning rate of 0.001, the Adam optimizer, and Binary Cross Entropy Loss (BCELoss). For the NFCF and MFCF models, we modified the original code to generalize grouping logic for diverse criteria (e.g., inter- action count, age, gender, and geography) and adapted the debiasing process to compute bias directions dynamically for various groups. To ensure consistency, we reused the score files generated by our base models for the GMF-UFR and NFUMF-UFR models. In order to enhance the repro- ducibility of the results, we utilized MIP (Santos & Toffolo, 2020), free light-weight Python library for modeling and optimization, instead of Gurobi (Gurobi Optimization, LLC, 2024) optimization solver, commercially licensed soft- ware. For fair and sound comparisons with the base models and fairness baselines, instead of using arbitrary top-k pre- dictions, we utilized the average optimal prediction set size returned by the ENSUR framework on top of the given base recommendation model.
D.2. Sampling and Data Splitting
We followed the following sampling and splitting method:
Negative sampling:
Following Ma et al. (2024), we selected 50 non-interacted items per user through neg- ative sampling for training, validation, and testing. Data splitting: We employed the Leave-One- Out (LOO) strategy (He et al., 2017; Han et al., 2023) to partition the dataset into training, calibration, and test- ing sets. Specifically, for each user, one interaction was isolated for calibration and testing. while the remaining interactions were used for training.
Multiple trials: To account for variability in sampling and splitting, we repeated the experiments over 20 inde- pendent trials. For each trial, random negative samples were drawn for training, validation, and testing. The results were averaged across all the trials.
13

---

ENSUR: Equitable and Statistically Unbiased Recommendation
D.3. Model Configurations and Fairness Baselines
These results reaffirm the main paper's observations regard- ing ENSUR's ability to balance fairness and performance while adapting effectively across diverse datasets.
To evaluate the effectiveness of our framework, we imple- mented it on top of the five base recommender models spec- ified in the main paper. Here, we provide specific architec- tural and training details of the models used:
Table 3. Performances and fairness comparisons with base models and fairness haselines on the MovieLens Dataset grouped by the
Base Recommendation Models
gender in terms of risk, average set size, and Hit Rate DCf Diff, respectively. Bold indicates the best result, underline indi-
DeepFM: Combines 8 latent factors with deep layers of [50, 25, 10] and ReLU activation.
cates the second best and marks threshold exceeded cases.
Methol
Group Rok.
Average Set Size
Hin Rate
DCG
|ii Rate Dur
PCG DuT
GMF: Utilizes an embedding size of 8 for capturing
DapPEM
0.2
0.5
00.53
0017
0
0.187
99325
linear interactions between user and item embeddings.
DespEM- ENSUR
0002
0.18
MLP: Employs layers of [64, 32, 16] with ReLU acti-
0.187
0.113
10.52
0.141
[0378]
vation for modeling non-linear interactions.
0.105
9
0.359
GME ENSUR
0.133
0.843
0.303
00443
0,006
NeuMF: Integrates GMF and MLP with a GMF em- bedding size of 8 and MLP layers of [64, 32, 16], using ReLU activation. LightGCN: Configured with an embedding size of 8 and 3 graph convolution layers.
LishGON
0007
0.443
19
LigHGON
0.125
03375
0.001
0.126
0146
MLP
0131
0.066
0.007
0.135
7
ENSUR
0.014
0.107
01855
0.595
To validate our framework further, we compared it with four fairness baseline approaches. The baselines are based the most commonly adopted methods in fairness literature i.e in- processesin and post processonn methods (Li et al., 2023):
NouMF
0.19
0.199
00002
0.357
0.543
0.022
0.015
NoME ENSUR
0.149
0.05
0
Other Pairmas Bowalines
0.198
[0.339
0.01
0.01
0 1
1x795
0..40
MFCF
[0241]
9
0.000
0,009
10.55
NaMME-UUR
8
10754
10.25
0.044
0.1R
0.35
GME-UER
03151
9
0.765
0.537
0.03
0.022
Fairness Baselines
0118
NFCF and MFCF(n-proceesingg (Islam et al., 2021): The authors utilize a pre-training and fine- tuning approach induce user-sided group fairness. Initially, the user embeddings sensitive interactions, followed by a de-biasing step to mitigate the embedding bias. Finally, the models are fine-tuned on sensitive item recommendations with a fairness penalty to reduce systemic bias in predictions. Neumf-UFR AND GMF-UFR proccessc))))) (Li et al., 2021a): This post-hoe re-ranking approach uti- lizes an integer programming solver to balance fairness and utility disparity between advantaged and disad- vantaged user groups. The method optimizes prefer- ence scores while enforcing a faimess constraint, en- suring that recommendation quality differences (e.g., DCG@ 10, @ 10) between groups remain below a specified threshold.
learned from non-
Table 4. Performances and fairness comparisons with base models and faimess baselines on the Last.fM Dataset grouped by the Re- gion in terms of risk, average set size, and Hit Rate Diff/DCG Diff.
respectively. Bold indicates the best result, underline indicates the second best and marks threshold exceeded cases.
Melhodd
Gruu
DCG
bilf
DapFM
0.171
0.29
0,363
0,799
22
D.221
0,252
DeppM FNSUR
DJIKT
0.35%
0.16
0006
0,32
GMF
0.100
0.011
02933
45
0.197
ENSUR
0.156
0.273
0,019
0.023
BLFF%
11994
0.35
I LIgHIGNN
02271
0.382
0.006
0003
0,360
LightGCN
0.164
0.566
0.392
600
6LW
10806
01300
MLP
0
0.79
6.358
0.019
0.013
32
1976
6.715
MLP + ENSUR
0.607
0.005
619
0.325
0:001
0179
0.335
0.021
E. Additional Experiments
0.59
22
0.903
NaME + ENSUR
0,310
11001
E.1. Remaining Experiments -Continued
Other Bosellaes
NFCF
00344
00752
0.344
0.024
0.04)
0.273
6.705
Tables 3 to 5 extend the analysis provided in the main paper. These tables support the key findings: the ENSUR frame- work consistently achieves both risk control (a = 0.20) and fairness (17 = 0.20) thresholds across all datasets, outper- forming base models and fairness baselines.
MFCF
45
D1789
6.359
0.066
NaMME-IFR
[0215 [
30
07887
0.306
2
00255T |
10742
0.357
GME-UER
45
0.797
0,345
0.007
0.045
19.27%
0.197
14

---

ENSUR: Equitable and Statistically Unbiased Recommendation
set size as å ranges from t. 0.50 (in increments of 0.05), with fixed thresholds of 7) 0.15, 0.20, 0.25. These find- ings are based on AmazonOffice, MovieLens and Last.fm datasets respectively.
Table Performances and fairesss comparisons with hase models and fairness baselines on the Book-Crossing Dataset grouped by the Age in terms of risk, average set size, and Hit Rate Dift/DCG Diff, respectively. Bold indicates the best result, underline indi-
cates the second best and f marks threshold exceeded cases.
These results further validate variations in 1) and ō exhibit consistent patterns, emphasizing our framework's ability to adapt prediction set sizes effectively based on fairness constraints.
Methed
Ridk
Size
Dur DCC
DMIFM
0.3021
0.11
BASE
0.571
00176
DogFM e ENSUR
13813
(0251
0.00
202
0,329
00257
GMF
CIETT
0.813
0.777
0.129
0.11
3
0.604
GMF
GJES
01815
0.706
o05
GEW
112011
00.522
F. Generalizablity of Grouping Methods
LIgHGGN
0.854
0.646
0.189
0.217
0051
0.083
01.1%
LIgHGNN KNSUR
0.019
0005
We validate if ENSUR is adaptable to practitioners' de- mands for customized user groups based on specific biases or fairness concerns relevant to their context. Specifically, we test our framework using different grouping techniques on single dataset i.e. Last.fm by grouping users based on item interactions and grouping by both item interactions and interactions with popular items on the Last.fm dataset. The results could be found in Table 6 and Table 7. The results demonstrate that the ENSUR framework can dynamically generate prediction sets for users grouped by any condition. This is particularly useful in real-world scenarios, where different applications may have different definitions of fair- ness. By allowing any grouping method, the framework can support dynamic fairness criteria that can evolve with changing societal norms or organizational policies, thereby allowing practitioners to define user groups based on the specific biases or fairness concerns relevant to their context.
GLW
0.301
MLP
0.124
0.576
0.225
0.192
0.093
03165
0.684
0.152
MIP ENSUR
0183
01:00
1.119
NeaMP
0.44
0.835
0.253
0.2141
0.00
3M
0.346
NeuMP ENSUR
11613
00357
0.008
0.096
GPTT
0,507
00200
Ofher Falmes Baxdiinss
NFCF
102167
0.744
(0766
008
MFCF
0,202
00382
0.087
0.04
U.ST
01635
NeIMP-LFRR
0.817
0.143
0641
10335T
0,674
0.185
GMF-UE
0,00
0.76
00007
1667
11166
E.2. Parameters Analysis -Continued
Effect of Risk Control Parameters and 5 on Prediction Set Sizes.
Figures 6 to 8 illustrate the trends in the average predic- tion set size as a varies from 0.10 to 0.50 (in increments of 0.05), while keeping the risk confidence thresholds fixed at 5 0.05, 0.10, 0.15, using the Book Brok-Crrsssnn, MovieLens, and fa datasets respectively. Similarly, Figures 9 to 11 present the trends in the average prediction set size as 6 varies from 0.10 to 0.50 (in increments of 0.05), while keep- ing the confidence thresholds fixed at a = 0.15, 0.20, 0.25, using the AmazonOffice, Movielens, and Last,fm datasets respectively.
G. Practical Applicability of the Framework
We now analyze the practical applicability of our framework. In real-world recommendation systems, prediction sets are often fixed to a specific size k and applied uniformly across all users. This fixed size is typically determined heuristi- cally or through trial and error, aiming to maximize the likelihood of including items that users may interact with while prioritizing and ranking items by relevance. However, this heuristic approach has several limitations:
The observed trends in Figures 6 to 8 (variation in a) and Figures 9 to 11 (variation in 5) are consistent with the ob- servations reported in Figure 2 (AmazonOffice dataset) and Figure 3
Fixed-size sets can lead to cognitive overload for users when the size is too large or fail to meet individual user needs when the size is too small.
dataset) in the main paper. These results reinforce the consistency of our framework's 's behav- ior across different datasets and grouping methods.
They do not account for disparities in user engagement or group fairesss, potentially disadvantaging certain user groups.
Effect of Fairness Control Parameters v) and å on Predic- tion Set Sizes Figures 12 to 14 illustrate how the average prediction set size changes 1) varies from 0.10 to 0.50 (in increments of 0.05), while holding the fairesss confidence thresholds fixed at å 0.15. 0.20, 0.25. These results are based on the AmazonOffice dataset, Book-Crossing and Book- Sook-Crossing datasets respectively.
Recommending unnecessary items results in resource inefficiencies for platforms.
Our framework addresses these challenges by dynamically determining the minimum prediction set size for each user, satisfying fairness and performance guarantees with statisti- cal confidence (e.g., 95%). This complements the existing recommender systems can employ an appropriate ag-
In contrast, Figures 15 to 17 display the trends in prediction
15

---

ENSUR: Equitable and Statistically Unbiased Recommendation
DoepFM +GLFR GVF +OUER
Average Set Sine
Average Seh Size
NeuMf -UUP
DeepFM +CUFI
DeesFM +GUF
LightGCN +GUFR
UgMISCN
HLP +GUFR
MLP +GUFR
Neute +GUPR
NeuMF -GUFR
01
0.2
0.3
0.4
0.5
0.3
0.2
0.3
0.4
0.5
0.1
0.2
03
0.4
05
(a) 6 = 0.05
(b) 6 = 0.10
(c) 8 = 0.15
Figure 6. Analysis of base models after applying the ENSUR framework in terms average set size with varying a
(0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on Last.fm dataset grouped by Region under different 8.
40
40
Average Set Sine
Set Size
20
30
20
DeepFM +GUFN
25
CMI +GUPa
DeesFM +GUFR
GMI -ELIA
DoepFM +GUFR
25
GMF +GUFR
LightGCN +GUFR
LigHSEN +GJFR
Ug GGUR
+GUFR
MLP +OUFR
MLP +GUFR
NeuMF -GUFR
15.
15
0.05
(b) 0.10
(c) 6 0.15
Figure 7. Analysis of base models after applying the ENSUR framework in terms of average set size with varying (0.10, 0,15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on Book-Crossing dataset grouped by Age under different d.
DeepFM +GURR
DoepFM +GLFR
Deepf'M
GMF +GUFR
GVF
GMF +GUFR
Average Set Size
20
MLP
00
LightGCN +GUFR
Neuter +GUFR
NeuMf -GUPR
+GUFR
25
35
15
NeuMF +GUER
Average
10
30
10
0.1
0.2
0.3
0.4
0.5
0.3
0.2
0.3
0.4
0.5
01
0.2
03
0.4
05
0
(a) 8 = 0.05
(b) 8 = 0.10
(c) 5 = 0.15
Figure 8. Analysis of base models after applying the ENSUR framework in terms of average set size with varying a (0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on MovieLens dataset grouped by Gender under different 5.
gregation method (for example, mean) to compute global This global k, obtained with the theoretical guarantees, can then be applied to recommend unseen items users, ensur- ing that faimess and performance guarantees hold across the system.
dynamically adjusting & cold-start scenarios ensures con- cise and personalized recommendations, preventing user overwhelm and aligning with platform resource constraints.
Additionally, the calculated k can serve as benchmark to fine-tune recommendation models, enabling iterative im- provements that enhance fairness and accuracy across di- verse user groups. By tailoring prediction set sizes dynami- cally, our framework provides a practical, scalable solution for modern recommendation systems.
For example, in e-commerce platforms such as Amazon, in- stead of heuristically fixing & = 10 for all users, our frame- 7) that balances
work identifies an optimal l (e.g.. fairness and accuracy, reducing unnecessary recommenda- tions and enhancing satisfaction while optimizing plat- form resources. Similarly, in streaming services like Netflix,
16

---

ENSUR: Equitable and Statistically Unbiased Recommendation
DeepFM
DeepFM GGER+
26
37.5
Average Set Stze
Lighttch
LigeGCNN GGUER +UUF
14
MLFR+GUFR
35.0
DevoFM +GUFK
Neuter GUFR
32.5
Neulr -OUFR
GMF rGUFA
UgHGEN +GUFR
30.0
20-
27.5
28
25.0
6
(a) 0.15
(b) 0.20
(c) 0.25
Figure 9. Analysis of base models after applying the ENSUR framework in terms of average set size with varying d (0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on AfazonOffice dataset grouped by Interactions under different a.
45
DeepPM +GUER LightGN
Deepl'M +GUFR.
Deeptit -GUFR- LightCh
Averoge Set 5.28
40
UpMaSCN +SUFR NeuMt -OUPH
Set Size
MLP +GUFR
HLP +GUFR Noufe: GGUP
Noute #OUPR
80
20
20
0.3
0.4
(a) 0.15
(b) 0.20
(c) 0.25
Figure 10. Analysis of base models after applying the ENSUR framework in terms of average set size with varying 5
10.10, 0.15, 0.20, 0.25, (1300, 0.35, 0.40, 0.45, 0.503 Last.fm dataset grouped by Region under different
DeepFM +GUFR
20
GNER
10
Average Set Size
LightGEN
Average Set 5i5e
DeapFN
Neater +GUPA
34
GMF -OURR
GMF GUFR
UigESSN +GUER
LighEOH +GUBR
NLP GUFR
MLP GGUAA
20
NzUMF
NeuNF
0.1
0.2
03
0.4
0.5
0.3
0.2
03
0.4
0.5
0.
0.2
0.3
0.4
05
5
(a) 0.15
(b) a = 0.20
(c) a = 0.25
Figure 11 Analysis of base models after applying the ENSUR framework in terms of average set size with varying 5 (0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on MovieLens datasct grouped by Gender under different
GMF LUUR
Deeplit +GUER
37.5
GMP GUPR
DeepfiM +GURR
76
GMF cOUP
Deeplitt GUIR
LightSc
35.0
UgreSCN +GUFR
UphtSiC GUJFR
32.5
rGLF
30.0
30:
23.8
22.5
01
62
0.1
0.4
0.5
63
0.2
0.3
0.4
0.5
01
0.2
1.3
0.4
03
0)
(a) 0.15
(b) 8 = 0.20
(c) 8 0.25
Figure 12. Analysis of base models after applying the ENSUR framework in terms of average set size with varying
(0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on AfazonOffice dataset grouped by Interactions under different 8.
17

---

ENSUR: Equitable and Statistically Unbiased Recommendation
DeapFM LightGch MLP +GUFR
DeepFM +GUFR
DeepFM +GUR
Nveragg Set Size
Averagee Set Size
LightGhh
42:
+UUF NeuMr -CUFR
HLP-+GUFR.
40
Neuter GUFR
Seh
Neuld +OUFR
18
26
1
(a) 0.15
(h) 0.20
(c) 8 0.25
Figure Analysis of base models after applying the ENSUR framework in terms of average set size with varying ") {0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50 on Book-Crossing datasct grouped by Age under different 6.
45
DeepFM GNE +GUPR
DoepFM GMF HOUPR.
DeapFM GUUR
GMF +GURR
Average Set Size
40
LightGEN HEZIF
82
LightEN 4GUP
Lightoch 4GUFR
MLP GUFR
MLP +GUF
HLP +GURR
35
Neuti +ULRR
Average Set
NeuMr -CUPR
Set
Keuft +GURR
-0
80
30
(a) = 0.15
(b) = 0.20
(c) 8 = 0.25
Figure Analysis of base models after applying ENSUR framework in terms of average set size with varying " {0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on Last.M dataset grouped by Region under different
DeepFM +GUFR Gan -GUR
28
MLP +GUFR
LightCC GUFR
27
DoepFM HGUFR MLP GUFA
26
Deepi Mt +GBER
NeuMF -OUFR
GMF rGUFR UihhSCN +GUFR
27.5
25.0
0.1
0.2
0.3
0.4
0.5
033
0.2
0.
0.2
03
0.4
05
(a) = 0.15
(b) 1 = 0.20
(c) 11 = 0.25
Figure 15. Analysis of base models after applying the ENSUR framework in terms of average size with varying å
10.10, 0515, 0.20, 0.25, 0.30., 0.35, 0.40, 0.45, 0.50} on AmazonOffice dataset grouped by Interactions under different r7.
DeegFM
GMF +GUFR
60
GMF +GUFR
GMI +GURR
Avtrngge Set Sip
Lightoch HGURR
LightSCN EGUFR
LightGCk GUFF
40
MLP +GUFR
MLP +SUFR
MLP +GUFR
Neattr +GLPR
Set
Neutt -CUPH
Neafte +GUPR
00
-0
25
20
(a) 17 0.15
(b) 17 0.20
(c) 1) 0.25
Figure 16. Analysis of base models after applying the ENSUR framework in terms of average set size with varying å 10.10, 0515, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on Book-Cront mook-Crossing datasct grouped by Age under different y7.
18

---

ENSUR: Equitable and Statistically Unbiased Recommendation
10
18
Averoge Set Sibe
awernge Set size
04:
GNE
14
+GUF
LightCh +GURR
UgMEECN +GUFR
LightSC +GUFR
02
MLP +GUER
NLP +GUER
HLP +GURR
NeuttF GLFR
10
NeuMf GUUF
Neute GGLR
0.1
0.2
0.3
05
0.3
0.2
0.3
0.5
0.2
0.3
0.4
0.
8
(a) 0.15
(b) 1) 0.20
(c) 1) = 0.25
Figure Analysis of models after applying the ENSUR framework in terms average set size varying å (0.10, 0515, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50} on MovieLens dataset grouped by Gender under different 1).
19

---

ENSUR: Equitable and Statistically Unbiased Recommendation
Table Performance and fairness comparisons with base models and fairness baselines on the Last.fM Dataset grouped by the Item Interactions terms of risk, average set size, and Hit Rate Diff/DGG Diff, respectively. Bold indicates the best result, un- derline indicates the second best and marks threshold exceeded cases.
Methel
Group
Rik:
Acrrage
Rate BCG
EXIr)
DOG EMIT
Grouped by sumber interattione
DeepPM
10.00
0005
0255
DeepMM- +
0.090
0.183
0.507
0038
0.06
0.243
0.133
0.56
0.45
LABHGGN
0.179
01821
0.547
0.18
0.099
LAMIGEN ENSUR
0.201
0.799
0.402
0.003
0.06
MLP
0.92
0.808
0.44
0.077
0.76
MLP ENSUR
0.51
ILS40
0.448
0.041
0.67
NeoMP
0.51
ILS40
0.58
0.067
0.811
NcoMP ENSUR
0144
ILBS8
0.581
0.007
0.67
Other Faimoss Bowelinns
0053
[0277)
0.383
0.516
[0231
(0707)
0.775
0.
0.900
[02131
0:700
0.515
[0201T]
[
0,772
0.517
Table Performance, and fairness comparisons with models and fairness baselines on the Last.fM Dataset grouped by the Item Interactions Interaction with Popular Items terms of risk, average set size, and Hit Rate DifI/DCG Diff, respectively. Bold indicates the best result, underline indicates the second best and marks threshold exceeded cases,
Melad
Rib.
Rute
DCG
Groeped by number of rotal Intecaations & peputar Items Interactions
DeepFM
0:001
00350
DeepM--
0.162
6.534
0.54
GMI
B1663
8,437
0.60
0.12
0131
0.178
30
0.202
GME EASEIR
0.163
0.663
0.485
0.955
0.387
01126
0.505
MLP
0443
0,522
0.132
01299
31
0.233
MLP ENSUR
0.165
0.
0.143
0.179
0.322
0217
0,703
0.308
0.4
0.442
Other Ralmes Baaeiase
0:195
0.157
10
0.63
0.500
NoMMFAIER
0.111
29
0.99)
0.588
0,777
0.166
01425
GME-TFR
ILT7X
BX22
0.308
20


Reviews:
[{"review_id": "TP7bVSbmBn", "reviewer": "Reviewer_zLvX", "summary": "The paper introduces a comprehensive framework named ENSUR (Ensuring Statistical Fairness and Confidence in Recommendation Systems), which is designed to statistically ensure both fairness and confidence in the outcomes generated by recommendation systems. The authors propose that by utilizing two key components, RCPS (Recommendation Confidence Probability Sets) and FCPS (Fairness Constrained Probability Sets), the framework can generate high-confidence dynamic recommendation sets tailored to individual users. These sets not only meet the stringent requirements of confidence levels but also rigorously satisfy group fairness criteria, ensuring that the recommendations are equitable across different groups.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to the reviewer for their positive and encouraging feedback. We sincerely appreciate their recognition of our framework's rigor, versatility, and substantial empirical validation.  \n\na) Hyperparameter Selection and Tuning:  \nWe acknowledge a detailed explanation presenting hyperparameter selection and tuning will significantly benefit the paper's clarity.  The choice of our hyperparameter parameters (e.g., risk threshold $\\alpha$, fairness threshold $\\eta$, confidence parameters $\\delta$ and $\\hat{\\delta}$) is primarily guided by extensive empirical validation and standard practices from relevant literature such as Bates et al. (2021).  For example, on the AmazonOffice Dataset, the risk threshold ($\\alpha$) is set as 0.2  to control over-coverage while ensuring robustness, and η is chosen as 0.2 by selecting the smallest value satisfying fairness constraints across groups, based on interaction count using defined fairness metrics, verified on validation sets. Similarly, the confidence parameters δ and $\\hat{\\delta}$ were chosen from the set $\\{0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5}\\$ based on the value that consistently achieved the desired confidence level without unnecessarily large recommendation sizes. We followed this same procedure across the other datasets. We will include a detailed discussion of these guidelines in the final version.  \n\nb) We thank the reviewer for their valuable suggestion to improve readability. In the final version, we will include intuitive explanations of the theoretical contributions in Section 5 alongside the formal results. Similar to the existing remarks for Theorems 5.1 and 5.2, we will explain how Theorems 4.1 and 4.2 provide a high-probability surrogate for population-level constraints and how these bounds are later used in Theorems 5.1 and 5.2, which further guide the design of the GUFA optimization procedure. We will illustrate this flow clearly for accessibility. \n\nBased on the valuable feedback, we will (i) add a dedicated paragraph in Section 6 explaining hyperparameter selection across datasets, supported by our sensitivity experiments from Section 6.2.2, and (ii) give intuitive summaries of the main theoretical results for broader understanding. \n\nAgain, we are deeply thankful to the reviewer for the positive outlook of our paper and their insightful suggestions for further improving its accessibility and clarity.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I think there is no problem.", "experimental_designs_or_analyses": "The experimental designs are thorough and sound.", "relation_to_broader_scientific_literature": "The paper extends the prior works in fairness in recommendation systems.", "essential_references_not_discussed": "I think the essential references are discussed in the paper.", "other_strengths_and_weaknesses": "The proposed framework, ENSUR, is designed to be both model-agnostic and dataset-agnostic, meaning it can seamlessly integrate with any base recommender model and adapt to diverse datasets across various domains without requiring specific modifications or constraints. One of the key strengths of ENSUR is its ability to provide rigorous theoretical guarantees for fairness, a critical aspect that has been notably absent in prior works, thereby addressing a significant gap in the field of fair recommendation systems. To validate the effectiveness of the proposed framework, the authors conduct extensive and comprehensive experiments that span multiple datasets and compare against numerous fairness baselines. Additionally, the study includes thorough and complete hyperparameter sensitivity testing, which meticulously examines the impact of different parameter settings on the framework's performance, further solidifying the reliability and generalizability of the results. The experimental outcomes demonstrate that ENSUR achieves significant improvements over existing baselines, not only in terms of recommendation performance but also in computational efficiency, highlighting its practical utility and superiority in real-world applications. Overall, the framework's versatility, theoretical rigor, and empirical validation make it a substantial advancement in the pursuit of fair and efficient recommendation systems.\n\n\nWhile the paper presents a robust and well-structured framework, it could benefit from providing more detailed explanations regarding the selection and tuning of hyperparameters within the main body of the text, particularly addressing how these choices might vary across different dataset settings and domains. Such insights would offer readers a clearer understanding of the practical considerations involved in implementing the framework and how to adapt it to specific use cases. Additionally, while the supplementary material is comprehensive and contains valuable technical details, its density and complexity might make it less accessible to readers who are not deeply familiar with the mathematical foundations of the work. To improve readability and accessibility, the authors could consider summarizing the key proof ideas and theoretical insights in the main paper, using intuitive explanations and high-level overviews to convey the core concepts without overwhelming readers who may not have a strong mathematical background. This approach would make the paper more inclusive and engaging for a broader audience, while still preserving the rigor and depth of the technical content for experts in the field.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tU75gMAaEJ", "reviewer": "Reviewer_xCED", "summary": "This paper proposes a novel and reliable framework called Equitable and Statistically Unbiased Recommendation (ENSUR)) to dynamically generate prediction sets for users across various groups. This paper further designs an efficient algorithm named Guaranteed User Fairness Algorithm (GUFA) to optimize the proposed method and derive upper bounds of risk and fairness metrics to speed up optimization process. Rigorous theoretical analysis and extensive experiments are also provided.\n\n## update after rebuttal\n\nAuthors have addressed my concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their encouraging and supportive feedback. We are glad they found our paper well-written, rigorous, and efficient. Below, we aim to address their insightful queries and suggestions:  \n\na) Technical Innovation:  \nOur technical innovation lies in ENSUR being a unified statistical framework for generating recommendation sets that satisfy both confidence and group fairness constraints. We achieve this by formulating the recommendation problem into the Risk-Controlling Prediction Sets (RCPS) and Fairness-Constrained Prediction Sets (FCPS) paradigm, thereby integrating them into a single calibration process. A key innovation is the development of the GUFA (Guaranteed User Fairness Algorithm), a powerful yet efficient algorithm that enables post hoc calibration without retraining the base recommender model. GUFA leverages theoretically derived upper bounds on risk and fairness violations (presented in Theorems 4.1 and 4.2), using derived Binomial and Bernstein concentration inequalities to make the optimization tractable. These bounds are further used to construct guarantees for risk and fairness control in Theorems 5.1 and 5.2 while ensuring minimal prediction set size. \nWhile prior methods usually rely on heuristic or empirical approaches, our approach thereby results in a model and dataset-agnostic pipeline that is theoretically sound, computationally efficient, and practically deployable, with no need for any architecture-specific modifications or retraining. \n\nb) Selection of Fairness Threshold ($\\eta$) by practitioners:  \nWe thank the reviewer for the insightful question. To effectively choose a fairness threshold (η), the practitioners can: 1) assess historical performance disparity between user groups. 2) conduct preliminary analyses on validation datasets to balance fairness requirements against recommendation performance, and 3) finally align the thresholds with institutional fairness standards. We will add a detailed discussion about the approach to paper revision. For example, in the AmazonOffice Dataset, we set $\\eta = 0.2\\$ by selecting the smallest value satisfying fairness constraints across groups based on interaction count, using our defined fairness metrics verified on validation sets. \n\nc) Real-world Value of Minimizing Prediction Set Size:  \nMinimizing average prediction set size significantly improves recommendation quality as: 1) It reduces cognitive overload on users, thereby reducing the problem of ad blindness 2) On a streaming platform, a minimum yet accurate prediction set could help in skipping prefetching and caching trailers for irrelevant content thereby saving bandwidth and compute. 3) a smaller yet better prediction set size in ads will lead to higher click-through rates (CTR) and better returns.  \n\nWe also thank the reviewer for highlighting the minor issue in Eq. (12) and suggesting improving our framework illustration. We will address them in the final revision.  \n\nOnce again, we thank the reviewer for their thoughtful questions and encouragement, and we look forward to strengthening the final version of the paper based on the valuable feedback.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I checked the proofs of this paper and do not find the errors.", "experimental_designs_or_analyses": "Yes. I check the setting of experiments in Section 6.1.", "relation_to_broader_scientific_literature": "The paper builds on Risk-Controlling Prediction Sets (Bates et al., 2021) and fairness-aware recommendation literature.", "essential_references_not_discussed": "In my view, related works are currently cited/discussed in this paper.", "other_strengths_and_weaknesses": "Strengths:\n\na. This paper is well-written and easy to follow.\n\nb. The proposed method and theory in this paper is solid.\n\nc. Unlike existing frameworks, this paper offers rigorous theoretical guarantees.\n\nd. It is highly efficient compared to current baselines.\n\ne. The framework is lightweight and is model and dataset-agnostic.\n\nf. Comprehensive experiments are conducted across multiple datasets, models, and user groupings. Extensive hyperparameters experiments are done to show how they impact coverage, performance and fairness.\n\nWeakness:\n\na. I appreciate the proposed method and theory, but in my view, the technique in method or proof process of theorems is not surprising.\n\nb. The authors could explain further on how the choice of hyperparameters is made in the main paper.\n\nc. The framework illustration could be elaborated further to make a reader understand the work quickly.", "other_comments_or_suggestions": "Some minor issues: Eq.(12) in line 226.", "questions_for_authors": "a. I need a complete overview of the technical innovations of this paper. In order to prove the theory in the article, what technique is used in this article, what problems does this technique solve, and how innovative is this technique?\n\nb. How should the practitioners determine appropriate fairness thresholds (eta)?\n\nc. How does guaranteeing a minimum average prediction set size improve recommendation quality? Some real-world example will be helpful", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X7gUPVJiQa", "reviewer": "Reviewer_BirQ", "summary": "The paper introduces ENSUR (Equitable and Statistically Unbiased Recommendation), a framework designed to enhance fairness and confidence in recommender systems. The core idea is to generate dynamic prediction sets that (1) ensure a high-confidence inclusion of the true item, (2) guarantee fairness across diverse user groups, and (3) minimize set sizes for efficiency. To achieve this, the authors propose the Guaranteed User Fairness Algorithm (GUFA), which optimizes prediction sets while maintaining statistical risk and fairness bounds. The paper provides a rigorous theoretical foundation, derives upper bounds for risk and fairness control, and supports claims with extensive empirical evaluations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and for recognizing our framework's novelty, theoretical rigor, and practical efficiency. We appreciate their insightful comments and questions and address them below:   \n\na) Adaptation of ENSUR when fairness groups are inferred dynamically:  \nWe thank the reviewer for the important point. While ENSUR assumes predefined user groups, we can naturally extend it to dynamically inferred groups using clustering techniques based on user interaction patterns (e.g., click history, content preferences, time of activity).  We can rerun the ENSUR's statistical calibration step periodically or upon significant distributional changes in user behavior, reflecting these evolving group definitions. Since GUFA performs per-group calibration and is computationally efficient, such updates are feasible at scale. This approach ensures that fairness and confidence guarantees are preserved even as the definitions of user groups evolve over time. In the final version of the paper, we will add a Discussion on  this adaptation strategy.\n\nb) Extension to multi-sided fairness constraints   \nWe are thankful for suggesting this interesting direction. Our framework can be generalized to multi-sided fairness by adding multiple fairness constraints simultaneously— i.e., fairness for users as well as content providers. Each side (user or provider) will have associated fairness thresholds ($\\eta$ values), and the GUFA optimization strategy will jointly ensure fairness guarantees are maintained across all these dimensions while ensuring that the combined objective remains tractable. This extension maintains the modularity of the current approach and is practically beneficial in many multi-stakeholder platforms such as marketplaces.  \n\nc) Limitations when applying ENSUR to highly imbalanced user groups:  \nWe sincerely appreciate this critical observation. In highly imbalanced settings, ENSUR will continue to maintain valid fairness guarantees as the calibration of risk and fairness constraints is done independently for every group. This will ensure statistical robustness. However, in such settings, due to underrepresentation, the minority group may require a more conservative learned parameter $\\lambda$ to ensure statistical guarantees. This may eventually result in larger prediction set sizes for underrepresented groups. We recognize that extreme imbalance is a real-world challenge and will discuss ENSUR behavior under such cases and its implications in the Discussion section of the revised version. \n\nWe are again deeply grateful to the reviewer for their positive acknowledgment of our work and for their valuable questions.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Theorem 4.1 and Theorem 4.2 establish upper bounds for risk and fairness, which are correct.", "experimental_designs_or_analyses": "The experimental evaluation is comprehensive, covering multiple datasets (AmazonOffice, MovieLens, Last.fm, Book-Crossing) and a variety of base models (DeepFM, GMF, MLP, NeuMF, LightGCN).", "relation_to_broader_scientific_literature": "The authors draw on concepts from risk-controlling prediction sets (Bates et al., 2021) and fairness-aware collaborative filtering (e.g., Yao & Huang, 2017; Abdollahpouri et al., 2019).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Novel contribution: The combination of risk control and fairness guarantees in recommendation is innovative. Rigorous theoretical foundation: Well-structured proofs and upper-bound derivations.Strong empirical validation: Comprehensive experiments demonstrate real-world applicability. Computational efficiency: ENSUR is significantly faster than existing fairness-aware baselines.\n\nFairness group assumptions: The approach assumes predefined user groups, which may not always be straightforward in practice.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. How does ENSUR adapt when fairness groups are not predefined but inferred dynamically from user interactions?\n2. Could ENSUR be extended to handle multi-sided fairness constraints (e.g., fairness for both users and content providers)?\n3. Are there any observed limitations when applying ENSUR to highly imbalanced user groups?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XFX8oYVcb2", "reviewer": "Reviewer_at4n", "summary": "This paper introduces ENSUR (Equitable and Statistically Unbiased Recommendation), a novel framework aimed at ensuring confidence and fairness in recommender systems. The authors propose a dynamic method for generating prediction sets that guarantee:\n1.\tA user-predefined confidence level (e.g., 90%) for including the true item,\n2.\tFairness across different user groups,\n3.\tMinimal average prediction set sizes.\nTo achieve these goals, the authors develop the Guaranteed User Fairness Algorithm (GUFA), which optimizes fairness and risk constraints efficiently. They establish theoretical guarantees for fairness control, risk control, and minimal prediction set size. Extensive experiments validate the ENSUR framework across multiple datasets and base models, demonstrating improved fairness and reliability without sacrificing performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their encouraging feedback and for appreciating the relevance, rigor, and practical efficiency of our framework. Below, we address the questions:  \n\na) ENSUR's performance under significant group size imbalance:  \nWhile the empirical results presented in the paper cover moderately imbalanced scenarios, ENSUR remains robust under more severe imbalances. This is primarily due to its design: each user group's risk and fairness thresholds are independently optimized. As a result, ENSUR ensures stable statistical guarantees—even when groups differ significantly in size—by tailoring the learned calibration parameter ($\\lambda$) to each group’s distribution. However, underrepresented groups may require more conservative λ values, leading to slightly larger prediction sets. We will clarify this behavior in the Discussion in the revised version. \n\nb) Extending fairness constraints to multi-group settings:  \nThank you for this insightful question. The optimization in GUFA can effectively handle multiple fairness constraints simultaneously by introducing a unique fairness threshold (η) for each group. Each group's constraint will correspond to its threshold ($\\eta$), and the optimization problem remains structurally the same. For example, if a dataset has three user subgroups (e.g., by age, gender, and region), GUFA jointly calibrates all three using a separate fairness constraint. This makes the extension to multi-group fairness both practical and computationally efficient.  \n\nc) Practical limitations in applying ENSUR to online recommendation scenarios:  \nWe appreciate this critical point. ENSUR, as a statistical calibration framework, remains practically applicable to online recommendation scenarios, but given dynamic changes in user behavior online, ENSUR would require periodic recalibration (e.g., daily, weekly, or monthly, depending on system dynamics and data drift). Given ENSUR's efficiency, such recalibration is feasible in large-scale systems. We will expand this point further in the Discussion section and explore this direction in our future work. \n\nOnce again, we are incredibly grateful to the reviewers for their valuable comments, suggestions, and positive recognition of our work.", "claims_and_evidence": "The paper’s claims are well-supported by rigorous theoretical analysis and empirical validation. The authors provide detailed proofs for fairness and risk control guarantees (Theorem 5.1) and minimal prediction set size (Theorem 5.2). Additionally, they derive upper bounds for risk and fairness metrics to accelerate optimization (Theorem 4.1 and Theorem 4.2). The experimental results across four datasets confirm the framework’s effectiveness, showing improved fairness and risk control compared to baseline methods.", "methods_and_evaluation_criteria": "The proposed methodology is well-founded and aligns with established fairness-aware recommendation frameworks. The authors adapt the Risk-Controlling Prediction Sets (RCPS) approach while incorporating fairness constraints, making their contributions novel and practically relevant. The evaluation criteria—risk control, fairness control, and minimal prediction set size—are well-justified, and the results demonstrate meaningful improvements over baseline models.", "theoretical_claims": "The theoretical claims in this paper are sound and rigorously proved. Theorems 4.1 and 4.2 provide upper bounds for risk and fairness metrics, facilitating efficient optimization. Theorems 5.1 and 5.2 ensure fairness and risk constraints while maintaining minimal prediction set size. The derivations are mathematically solid, and the assumptions are well-motivated and clearly stated.", "experimental_designs_or_analyses": "The experimental setup is comprehensive, covering four diverse datasets (e.g., MovieLens, AmazonOffice) and five base recommendation models. The comparisons with fairness baselines (NFCF, MFCF, GMF-UFR, NeuMF-UFR) are appropriate, and the results consistently support the authors’ claims. The parameter analysis further enhances the credibility of the findings. The efficiency comparison demonstrates ENSUR’s computational advantage over existing fairness methods.", "relation_to_broader_scientific_literature": "This paper extends prior work on fairness in recommendation (e.g., Yao & Huang, 2017; Abdollahpouri et al., 2019) and uncertainty quantification via risk-controlling prediction sets (e.g., Bates et al., 2021). ENSUR builds upon these foundations by integrating fairness and confidence guarantees into a unified optimization framework. The approach is novel in its combination of theoretical guarantees and practical implementation, making it a valuable contribution to fairness-aware recommendation research.", "essential_references_not_discussed": "The paper covers essential prior work, particularly in fairness-aware recommendation and statistical risk control. No significant omissions were identified.", "other_strengths_and_weaknesses": "Strengths:\n\n•\tTheoretical rigor: Provides strong mathematical guarantees for fairness, risk control, and efficiency.\n\n•\tPractical relevance: Demonstrates applicability across multiple datasets and base models.\n\n•\tComputational efficiency: ENSUR outperforms fairness baselines in training time.\n\n•\tClear writing and well-structured methodology.\n\nWeaknesses:\n\n•\tAssumptions: Some fairness constraints may not hold universally across all domains.\n\n•\tEvaluation scope: While diverse datasets are used, real-world deployment studies would further strengthen the impact.", "other_comments_or_suggestions": "1.\tIt would be useful to discuss potential limitations when fairness groups are highly imbalanced.\n\n2.\tConsider elaborating on real-world applicability beyond academic datasets.\n\n3.\tFuture work could explore alternative fairness constraints and their implications.", "questions_for_authors": "1.\tHow does ENSUR perform when fairness groups have significantly different sizes? Does it remain stable under severe imbalance?\n\n2.\tCould the fairness constraints be extended to multi-group settings beyond binary group definitions?\n\n3.\tAre there practical limitations to applying ENSUR in online recommendation scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a new framework ENSUR to statistically ensure both fairness and confidence in recommendation systems.  All the reviewers think that this paper did a good job in deriving the rigorous theoretical guarantees for fairness and agree to accept this paper.
Decision: Accept (poster)