Equal Experience in Recommender Systems
Authors: Jaewoong Cho
Moonseok Choi
Changho Suh
Venue: ICLR 2022 Submitted (2022)
Keywords: Fairness
Recommender systems

Abstract: We explore the fairness issue that arises in recommender systems. Biased data due to inherent stereotypes of particular groups (e.g., male students' average rating on mathematics is often higher than that on humanities, and vice versa for females) may yield a limited scope of suggested items to a certain group of users. Our main contribution lies in the introduction of a novel fairness notion (that we call equal experience), which can serve to regulate such unfairness in the presence of biased data. The notion captures the degree of the equal experience of item recommendations across distinct groups. We propose an optimization framework that incorporates the fairness notion as a regularization term, as well as introduce computationally-efficient algorithms that solve the optimization. Experiments on synthetic and benchmark real datasets demonstrate that the proposed framework can indeed mitigate such unfairness while exhibiting a minor degradation of recommendation accuracy.

Under review as a conference paper at ICLR 2022
EQUAL EXPERIENCE IN RECOMMENDER SYSTEMS
Anonymous authors Paper under double-bling review
ABSTRACT
We explore the fairness issue that arises in recommender systems. Biased data due to inherent stereotypes of particular groups (e.g., male students' average rating on mathematics often higher than that on humanities, and vice versa for females) may yield a limited scope of suggested items to certain group of users. Our main contribution lies in the introduction of a novel fairness notion (that we call equal experience), which can serve regulate such unfairness in the presence of biased data. The notion captures the degree of the equal experience of item recommendations across distinct groups. We propose an optimization framework that incorporates the fairness notion as regularization term, as well as introduce computationaally-effice algorithms that solve the optimization. Experiments on synthetic and benchmark real datasets demonstrate that the proposed frame- work can indeed mitigate such unfairness while exhibiting a minor degradation of recommendation accuracy.
INTRODUCTION
Recommender systems are everywhere, playing crucial role to support decision making and to decide what we experience in daily life. One recent challenge concerning fairness arises when the systems are built upon biased historical data. Biased data due to polarized preferences of particular groups for certain items may often yield limited recommendation service. For instance, if female students exhibit high ratings on literature subjects and less interest in math and science relative to males, the subject recommender system trained based such data may provide a narrow scope of recommended subjects to the female group, thereby yielding unequal experience. This unequal experience across groups may result amplifying the gender gap issue in science, technology, engineering, and mathematics (STEM) fields.
Among various works for fair recommender systems (aao & Huang 2017 Li al. 2021 Kamishima & Akahoo| 2017 Xiao et al., 2017] Beutel et al. 2019; Burke 2017), 7), one recent and most relevant work is Yao & Huang (017). They focus on a scenario which unfairness occurs mainly due to distinct recommendation accuracies across different groups. They propose novel fairness measures that quantify the degree of such unfairness via the difference between recommendation accuracies, and also develop optimization framework that well trades the fairness measures against the average accuracy. However, it comes with a challenge in ensuring fairness w.r.t. the unequal experience. This is because similar accuracy performances between different groups do not guarantee a variety of recommendations to an underrepresented group with historical data bearing low preferences and/or scarce ratings for certain items. For instance, in the subject recommendation, the fairness notion may not serve properly, as long as female students exhibit low ratings (and/or lack of ratings) on math and science subjects due to societal/cultural influences (and/or sampling biases). Furthermore, if the recommended items are selected only according to the overall preference, the biased preference for a specific item group will further increase, and the exposure the unpreferred item group will gradually decrease.
Contribution: In effort to address the challenge, we introduce new fairness notion that we call equal experience. At high level, the notion represents how equally various items are suggested even for an underrepresented group preserving such biased historical data. Inspired by an information- theoretic notion "mutual information" (Cover 1999) and its key property "chain rule", we quantify our notion so as to control the level of independence between preference predictions and items for any group of users. Specifically, the notion encourages prediction Y (c.g., 1 if a user prefers an item; O otherwise) be independent of the following two: (i) user group Zuser (e.g., O for male; and 1 for

---

Under review as a conference paper at ICLR 2022
female); and (ii) item group Zitem (e.g., 0 for mathematies; and for literature). In other words, it promotes Y 1 (Zuser; Zicem): ): which in turns ensures all of the following four types of independence that one can think of: Y 1 Zitem. Y I Zuser. Y I. Zitem Zuser, and Y 1. Zuser Zitem- This inspired by the fact that mutual information being zero is equivalent to the independence between associated random variables, as well as the chain rule:
I(Y; Zuser: Zitem) = I(V; Zitem) + IY; Zuser Zitem)
I(Y; Zuser) + I(Y: Zitem Zuserr
See Section 3.1 for details. The higher independence, the more diverse recommendation services are offered for every group. We also develop an optimization framework that incorporates the quantified notion as a regularization term into conventional optimization for collaborative filtering in recommender systems (e.g., the one based on matrix completion Koren (2008); Koren et al. (2009)) Here one noticeable feature of our framework is that the fairness performances w.r.f. the above four types of independence conditions can be gracefully controlled via single unified regularization term. This is in stark contrast to prior works (Yao & Huang, 2017; Li et al., 2021 Kamishima & Akaho 2017 Mehrotra et al. 2018), , each of which promotes only independence condition two via two separate regularization terms. See below Related works for details. In order to enable an efficient implementation of the fairness constraint, we employ recent methodologies developed in the context of fair classifiers, such as the ones building upon kernel density estimation (Cho et al.) (020a)) mutual information Zhang al., 2018: Kamishima et al. 2012 Cho et al. 2020b). or covariance (Zafar et al. 2017a b). We also conduct extensive experiments both on synthetic and two benchmark real datasets: MovieLens IM (Harper & Konstan 2013) and Last FM 360K (Celma) 2010). As result, we first identify two primary sources of biases that incur unequal experience; population imbalance and observation bias (Yao & Huang/ 20.7). In addition, we demonstrate that our faimess notion can help improve the fairness measure w.r.t. equal experience (to be defined in Section 3.1 see
Definition 2)
while exhibiting a small degradation of recommendation accuracy.
Related works: In addition to Yao & Huang 2017). : numerous fairness notions and algorithms have
been proposed for fair recommender systems Xiao et al. 2017 Beutel et al. [019 Singh & Joachims
2018 Zehlike et al. 22177
et al. |020 Biega et al. 2018]] Li et al. 202 Kamishima
& Akaho 2011) Mehrotra et al. [018)] 8 |Schnabel et al. 2016)_ Xiao et al. (201 7) develop fairness
notions that encourage similar recommendations for users within the same group Beutel et al. (2019)
consider similar metrics as that in Yao & Huang (2117) yet in the context of pairwise recommender
systems wherein pairewise preferences are given as training data. Li et al. (2021) propose a fairness measure that quantifies the irrelevancy of preference predictions to user groups, like demogruphic parity in the fairness literature (Feldman et al. 2015 Zafar et al.) 2017a b). Specifically, they consider the independence condition between prediction Y and user group Zuser: Y Zuser. Actually this was also considered as another fairness measure in Yao & Huang (2017). Similarly, other works with a different direction consider the similar notion concerning the independence w.r.t. item group Zicem
y Zitem (Kamishima& & Akaho 2017 Singh & Joachims 2018 Biega et al. 2018). Mehrotra et al. (2018) incorporate both measures to formulate multi-objective optimization. In Section 2.2 we will elaborate on why the above prior fairness notions cannot fully address the challenge w.r.t. unequal experience.
There has been a proliferation of fairness notions in the context of fair classifiers: (i) group fair-
ness (Feldman et al. [2015) Zafar et al. 2017bb Hardt et al. 2016 Woodworth al. 2017); (ii)
individual fairness (Dwork et al. 2012 Garg et al. 2018); (iii) causality-basee fairness (Kusner et al.) 2017 Nabi & Shpitser| 2018)) Russell et al. 2017 Wu al. 2019; Zhang & Bareinboim [2 2018h a).
Among various prominent group fairness notions, demographic parity and equalized odds give an inspiration to our work in the process of applying the chain rule, reflected in equation [ Concurrently, multitude of fairness algorithms have been developed with the use of covariance (Zafar et al. b). mutual information (Zhang et al. [0018] Kamishima et al., 2012 Cho et al.) 2020b), kernel density estimation (Cho et al., 2020a) or Rényi correlation (Mary et al. 2019) to name few. In this work, we also demonstrate that our proposed framework (to be presented in Section 3) embraces
many of these approaches; See Remark for details.
2

---

Under review as a conference paper at ICLR 2022
2 PROBLEM FORMULATION
As key technique for operating recommender systems, consider collaborative filtering which estimates user ratings on items. We first formulate an optimization problem for collaborative filtering building upon one prominent approach, matrix completion. We then introduce couple of fairness measures proposed by recent prior works ( (Yao & Huang 2017 Li et al. 2021 Kamishima & Akaho 2017), and present an extended optimization framework that incorporates the fairness measures as regularization terms.
2.1 OPTIMIZATION BASED ON MATRIX COMPLETION
As a well-known approach for operating recommender systems, we consider matrix completion (Fazel 2002: Koren et al. 2009 Candès & Recht 2009). Let M e [g= be the ground-truth rating matrix where n and m denote the number of users and items respectively. Each entry, denoted by Méj. can be of any type. It could be binary, five-star rating, any real number. Denote by 12 the set of observed entries of M. For simplicity, assume noiseless observation. Denote by M guxm an estimate of the rating matrix.
Matrix completion can be done via the rank minimization that exploits the low-rank structure of the rating matrix. However, since the problem is NP-hard (Fazel) 2002), we consider relaxation approach that intends to minimize instead the squared error between M and M in the observed entries:
min Σ (My M,,) M
(i.j)ER
There are two well-known approaches for solving the optimization in equation 2 (i) matrix fac- torization (Abadir & Magnus 2005] Koren et al., (200)); and (ii) neural-net-based parameteriza- tion (Salakhutdinov et al. 2007] Sedhain et 2015 He et al., 2017). Matrix factorization assumes certain structure on the rating matrix: M LR where L E Raxt and 1 One natural way to search for optimal L and Rº is to apply gradient descent (Robbins & Monro 1951) w.r.t. all of the Lij's 's and R,j"s is although it does not ensure the convergence of the optimal point due to non-convexity. The second approach is to parameterize M via neural networks such as restricted Boltzmann machine (Salakhutdinov al. 2007) and autoencoder (Sedhain et al.) 2015 Lee et al. 2018). For instance, one may employ an autoencoder-type neural network which outputs a completed matrix M fed by the partially-observe 2013), an observed now vector of M is fed into the autoencoder, while an observed column vector serves an input for item-based autoencoder (Sedhain et al.) 2015). In this work, consider the two approaches in our experiments: matrix factorization with gradient descent; and autoencoder-base parameterization.
version of M. For user-based autoencoder (Sedhain et al.
One common way to promote a fair recommender system is to incorporate a fairesss measure, say Ctair (which we will relate to estimated matrix M), as regularization term into the above base optimization in equation]]
min (1-A) M
Σ (M, - -M,)² + A Liair (o,))e
(3)
where A E [o, 1] denotes a normalized regularization factor that balances prediction accuracy against the fairness constraint. For the fuimesss-regularizatio term Lfair several fairness measures have been introduced.
2.2 FAIRNESS MEASURES IN PRIOR WORKS YAo & HUANG 2017 KAMISHIMA & AKAHO 2017
LI ET AL. 2021
We list three of them, which are mostly relevant to our framework to be presented in Section 3 For illustrative purpose, we will explain them in simple setting where there are two groups of users. say the male group M and the female group F. The first is value unfairness proposed by Yao & Huang (2017). It quantifies the difference between prediction errors across the groups of users over the
3

---

Under review as a conference paper at ICLR 2022
Zitam
0
hem
M(M 7)
Zuerr
Y: a generic random variable w.r.t. Mg
1==18 2r)
1
Zuser
1 1
1
Figure 1: An example in which Y Zitem but Y Zitem Zueer- Here V = 1(8 2 r): 8
generic random variable w.r.t estimated ratings Mij and indicates certain threshold. The (i.j) entry of an estimated preference matrix with 6 users (row) and 6 items (column) indicates MM 2 T).
entire items:
VAL 1 m
1
Σ
(Mo Mby
(Mij Ma) (KJ))
Σ
(4)
MMl
Fr
(4J)ER:EM
prediction errue w.r.t M
prediction error w.r.l.
where Mg and Fi denote the male and female group w.r.t. observed entries, respectively. While the measure promotes fairness w.r.t. prediction accuracy across distinct groups, it may not ensure fairness w.r.t. the diversity of recommended items users. To this clearly, consider extreme scenario in which the ground truth rating is very small Mi5- = for a certain item j" (say science subject) for all i F. In this case, minimizing VAL may encourage Maj- 0 for all i F. This then incurs almost the science subject to the females, thus giving no opportunity to experience
no recommendation the subject. This motivates to propose helps mitigate such unfairness.
fairness measure (to be presented in Section 3.1 that
On the other hand, Kamishima & Akaho (2017) introduce another faimess measure, which bears a similar spirit to demographic parity in the fairness literature (Feldman et al.) 2015 Zafar et al. 2017a b). The measure, named Calders and Verwer's discrimination score (CVS), quantifies the level of irrelevancy between preference predictions and item groups. To describe it in detail, let us introduce some notations. Let Zitem be sensitive attribute w.r.t. item groups, e.g., Zite (literature) and Zitem = (science). Let Y be a generic random variable w.r.t. estimated ratings Miy's 's. To capture the preference prediction, let us consider a simple binary preference setting in which Y = 1 2 r where indicates certain threshold. Specializing the measure into the one like demographic parity, can be quantified as:
CVS = IP = 1 1 = -) P(Ý 1/ = 0)].
(5)
Minimizing the measure encourages the independence between Y and Zitem. thereby promoting the same rating statistics across different groups However, it does not necessarily ensure the same statisties when we focus on certain group of users. It guarantees the independence only in the average sense. To see this clearly, consider a simple scenario in which there are two groups of users, say female and male. Let Zuser be another sensitive attribute w.r.t. user groups, e.g., Zuser (female) and Zuser = (male). Fig. 1 iillustrates a concrete example where is independent of Zitem Notice that the number of I's w.r.t. Zitem 0 (over the entire
groups) is the same that w.r.t. Zitem 1. However, focusing on a certain user group, say Zuser Y is highly correlated with Zitem Observe in the case Zuser 0 that the number of I's is 9 for Zienm = O, while it reads 0 for Zitem
Li et al. (2021) consider similar measure, named user-oriented group fairness (UGF), yet which targets the independence w.r.t. user groups. Similar to CVS, we can define it by replacing Zitem with Zuser in equation
UGF IP(Y 1| =1) -P(( 1 0)].
4

---

Under review as a conference paper at ICLR 2022
However, by symmetry, the high correlation issue discussed via Fig. I still arises.
3 PROPOSED FRAMEWORK
We first propose new fairness measures that can regulate fairness w.r.t. the opportunity to experience inherently preference items, well as address the high correlation issue discussed as above. We then develop an integrated optimization framework that unifies the faimess measures as a single regularization term. Finally we introduce concrete methodologies that can implement the proposed optimization.
3.1 NEW FAIRNESS MEASURES
The common limitation of the prior fairness measures (Yao Huang 2017: Kamishima & Akaho 2017; Li et al. 2021) is that the independence between preference predictions and item groups may not be guaranteed for certain group of users. This motivates us consider the conditional independence as a new fairness notion, formally defined as below.
Definition (Equalized Recommendation) recommender system is said respect "equalized recommendation" " if prediction Y is independent of item's 's sensitive attribute Zitem given user's sensitive attribute Zuser: Y Zitem Zuser.
Inspired by the quantification methods w.r.t equalized odds in the faimess literature (Jiang et al. 2019 Donini et al. 2018) Hardt et al. 2016 Woodworth et al. (017), we quantify the new notion
DER Σ
Σ
PPV-
7user 21) (P 7item 22, Zuser 21)
2mm2
for arbitrary alphabet sizes |uueer| and 1teml Here DER stands for the difference w.r.t. two interested probabilities that arise in equalized recommendation. and this naming is similar to those in prior fairness metrics (Donini et al. 2018 Jiang et al., 2019). ) captures the degree of violating equalized recommendation via the difference between the conditional probability and its marginal given Zuser- Notice that the minimum DER = 0 is achieved under "equalized recommendation" One may consider another measure which takes "max" operation instead of "2" in equation 7 7o different measure based on the ratio of the two associated probabilities. We focus on DER in equation 7 for tractability of associated optimization problem that we will explain in Section 3.2
The constraint of "equalized recommendation" encourages the same prediction statistics of items for every user group. thereby promoting the equal chances of experiencing a variety of items for all individuals. However, the notion comes with a limitation. The limitation comes from the fact that conditional independence does not necessarily imply independence (Cover) 1999):
Y Zitem Zuser +
Y Zitem-
(8)
Actually, the ultimate goal of fair recommender system is to ensure all of the following four types of independence:
Y Zitem:
Ý Zuser Zitem:
T Zuser:
i Zitem Zuser-
One natural question that arises is then: What is proper fairness notion which allows us to respect all of the above four conditions preferably in one shot? In attempt to succinctly represent all of the four conditions, we invoke an information-thhoree notion, mutual information (Cover) 1999). One key property of mutual information, called the chain rule, gives an insight:
I(;; Zusar, Zram) I(Ý; Zitem + I(Y; Zuser Zitem) I(;; Zuser) ) + I(;; Zitem 1user)
(10)
From this, we can readily see that
I IY; Zuser. Zitem) = 0
((;; Zitem) = 0, I(Y; Zuser Zitem) = O. (11)
1(Y; Zuser) O, 1(;; Zitem Zuser) 0.
This is due to the non-nenn negaatiiv property of mutual information. The key observation in equation motivates us propose fairness notion that call equal experience.
I]
5

---

Under review as a conference paper at ICLR 2022
Definition 2 (Equal Experience) A recommender system is said respect "equal experience" if its prediction Y is independent of both Zitem and Zuser: Y
(Zitem: Z
Similar to DER, we also quantify the notion as the difference between the conditional probability and its marginal:
DEE
Σ
Σ
P(Y =) P(Y 1/ 22, Zuser 21)
(12)
E Zuee 22222
for arbitrary alphabet sizes user and neeml We also coin the similar naming: DEE (difference w.r.t. two interested probabilities that arise in equal experience).
3.2 SHRRNESS-REULLRIIE OPTIMIZATION
Taking DEE as the fairness-regularization term Lfair in the focused framework (equation 3 we get:
min (1 A)
Σ
(Mg Mrj)² A DEE
(13)
M
(i.j)en
where À E [0, 1] denotes normalized regularization factor. Here one challenge that arises in equa- tion 13 that expressing DEE terms of an optimization variable M not that straightforward.
To overcome the challenge, we take the kernel density estimation (KDE) technique (Cho et al. 2020a) which enables faithful quantification approach is that the computed measures based on KDE is differentiable w.r.t. model parameters, thus enjoying family of gradient- optimizers (Géron 2019 Kingma Ba, 2014b). Since the problem context where the KDE technique (Cho et al. (2020a) was introduced is different from ours, we describe below details on the technique, tailoring to our framework.
carresssregularizztion terms. One key benefit of the KDE
Implementation of DEE via the KDE technique (Cho et al. 2020a): We first parameterize pre- diction output M via matrix factorization neural network. Let be collection of parameters w.r.t. M. It could be collection of matrix entries of L. and R when employing matrix factorization M = LR. Or it could be a collection of neural network parameters in the latter case.
The key idea of the KDE technique is to approximate the interested probability distributions via kernel density estimator defined below:
Definition (Kernel Density Estimator (KDE) (Davis et al., 2011)) Let (g......,
g(()) be i.i.d.
examples drawn from distribution with an unknown density f. Its KDE is defined as: í( =
I sh Ei=it fk
8 glo)"
where fu is kernel function (e.g., Gaussian kernel function (Davis et al. is a smoothing parameter called bandwidth.
2017)) and
For DEE, the interested probability distributions are IP( = and P( = 1| = 22, Zuser = 21). Let us first consider IP(Y = 1). Remember Y 1( 2 J. so Y should be taken into consideration initially. Using the KDE, can estimate the probability density function of Y. say fg (0):
fx(0) 1
R m
Mij
fk
(14)
nmh
1=
This together with γ = -1{8= 2 t} gives:
PΦ 1) [ (o(
1
1
m
Mi
Fk
nm
where Fo(0) = 0 (t)dt. Since the approach relies upon family of gradient-based optimizers,
the gradients of IP(Ý 1) and PP 1 Zitem 22, Zuser 21) need to be computed explicitly. Using the technique in Cho et al. (2020a) (Proposition therein), we can readily approximate V DEE. See Appendix A.2 for details.

---

Under review as a conference paper at ICLR 2022
Remark 1 (Other choices for measure of "equal experience") Instead of DEE, one resort
to other measures based on prominent tools employed in the fairness literature: covariance (Zafar
al. b), mutual information (Zhang al.) 2018) Kamishima et al. 2012 Cho et al. 2020b),
Wasserstein distance (Jiang el al. (020) and Rényi correlation (Mary el al. 2019). We leave a more
detailed explanation in Appendix A.3
4 EXPERIMENTS
We conduct experiments both synthetic and two benchmark real datasets: MovieLens 1M (Harper & Konstan 2013) and Last FM 360K (Celma 2010). We generate synthetic data so as to pose faimess issues. Algorithms are implemented in PyTorch (Paszke al., 2019), and experiments are performed on a server with Titan RTX GPUs. All the simulation results (to be reported) are the ones averaged over five trials with distinct random seeds. In Appendix AA6 we also present the running times of our algorithm and baselines on the synthetic and real datasets.
4.1 SYNTHETIC DATASET
Here we highlight two types of bias: population imbalance and observation bias (Yao & Huang 2017). For illustrative purpose, let us explain them in simple subjjct-recommenddee
example where there are two user groups (Zuser = 0 for male and Zuser = 1 for female) and two item groups (Zitem for science and Zitem = 1 for literature). Population imbalance refers to the difference in the ground-truth preferences between two user groups, e.g.. for the science subject, male students exhibit higher ratings relative females. Observation bias is the one that occurs due to the stereotype formed by societal and cultural influences. To understand what it means, let us consider a scenario where male student equally likes science and literature subjects. But due to the stereotype that male students prefer science to literature in general, there may be very sparse ratings from male students for literature. The system trained based on such data might incorrectly interpret as if male students dislike literature. Such data is said to have observation bias.
We generate synthetic data that bear the two biases in the context of binary ratings, i.e., Mij E {+1 (like), 11 (dislike)). We divide n users into male and female groups each of 7. say M and F. Items are also divided into two groups of 3. say male-preferred group M and female-preferred group F. To account for population imbalance, we first generate the ground-truth rating matrix M E Rmxm using the following four probabilities: {PMM' MMMM,,PFF.. PPM. }FF} where PMM' indicates the probability that male student likes male nneee-reeerred subject (science). More precisely, for E M and j E M',
{{ +1, w.p. PMM:
Mij -
(15)
-1, w.p. 1 PMM"
Similarly the other probabilities are defined. To ensure the low-rank structure, say rank r, of the rating matrix, we generate _ basis rating vectors for male group as per the above preference probabilities, and similarly another set of E basis rating vectors is generated for female group. Every male student picks one of the ş basis vectors w.r.t. the male group uniformly at random, and similarly for every female student. This then yields rank(M) =
To control observation bias, we introduce another probability set: {qMM., COA. MMF'.FMM (FM. where q.v denotes the probability that male student's 's rating is observed for male-preferred subject. More precisely, for i E M and j E M',
{ Ω 2
(i,5) E
w.p. 9MM'
(16)
w.p. 1 IMM.
Similarly the other probabilities are defined. For simplicity, throughout all the synthetic data sim- ulations, we assume a symmetric setting in which PMM = == po), PMF = (= Pt).
qMM' = (= qo), and qMF" -F q1).
We consider a setting in which (r. n, m) = (20, 600, 400). We leave a more detailed explanation of experiments the synthetic dataset in Appendix [A.4] First, we check whether the bias actually incurs unfair recommendations. For ease of understanding, we consider two scenarios: (i) population imbalance varies without observation bias; and (ii) observation bias varies without population
7

---

Under review as a conference paper at ICLR 2022
Population imbalance
Observation bias
1.40
0.30
1.20
0.20
0.40
0.10
0.20
0.05
0.00
00.4,0.4) 00.4,03) (0.4.02) (0.4,0.1) (Po.Ps)
(30 (1)
(0.2, (0.2, (13))(2 (M.2)(0.2, 0.01)
Figure 2: (Left) DEE function of (po. p1) which controls the degree of population imbalance while fixing go = = 0.2: (Right) DEE as a function of (go: 21) w.r.t. observation bias while
fixing PD p1 0.4. Here po PMM = PFF. = PMF PFM' qo QMM' = QFF and
JMF = IF
Table 1: Prediction error (RMSE) and fairness performances on the synthetic dataset preserving ob- servation bias (yo, /1) = (0.2, 0.01) while exhibiting no population imbalance (po: P1) = (0.4, 0.4). The boldface indicates the best result and the underline denotes the second best.
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0.8889 + 0.0111
0.1201
0.0405
0.4644 + 0.0108
0.0120 + 0.0050
0.0118 + 0.0047
Ours (DEE)
0.9020 0.0081
0.0025
0.0006
0.4609 + 0.0076
0.0006
0.0004
0.0002
0.0001
Ours (DER)
0,8887
0.0042
0.0401
0.0056
0.4540 = 0.0110
0.0201
0.0028
0.0002
0.0002
VAL-based
0.8837
0.0045
0.1099
0.0045
0.0003 6.48c-6
0.0090
0.0004
0.0088
0.0015
UGF-based
0.8961
0.0067
0.0144
0.0144
0.4709 = 0.0182
0.0004
0.0003
0.0217
0.0027
CVS-hased
0.9003 0.0061
0.1390
0.0413
0.4722 + 0.0055
0.0206
0.0022
0.0002 0.0001
imbalance. Fig. 2 (Left) presents the 1st scenario, demonstrating that the fairness performance measured in DEE decreases with an increase in population imbalance, controlled by IPO P1 Fig (Right) considers the 2nd scenario. We see the same trend yet now w.r.t. the variation of observation bias.
Table presents the prediction error (RMSE) and fairness performances on the synthetic dataset
having observation bias (20, 21) (0.2, 0.01) yet without population imbalance (po, p1) = (0.4, 0.4).
We consider four fairness measures: (i) DEE (in equation 12 (ii) DER (in equation 7 (iii) VAL
(in equation 4): (iv) UGF (in equation ; (v) CVS (in equation 5). We also compare our algorithm
with four baselines: (i unfair (no fairness constraint); (ii) VAL-based algorithm (Yao Huang 2017):
(iii) UGF-based algorithm (Li et al. 2021I: (iv) CVS-based algorithm (Kamishima & Akaho 2017). Each baseline, say VAL-based algorithm, achieves the best faimess performance only for VAL, while it does not work well under the other faimess measures. On the other hand, our algorithm offers great fairness performances for all the measures, except for VAL, which our framework does not target.
4.2 REAL DATASETS
We consider two benchmark datasets: MovieLens 1M (Harper Konstan 2015). and Last FM 360K (Celmal 2010). We run experiments employing both matrix factorization (MF) based and autoencoder (AE) based techniques. In Appendix [A.5] we leave a more detailed explanation of experiments on the real datasets and the results of AE-based technique. The results of real data experiments are listed in Tables 2 and As in the synthetic data setting, we can make two relevant metric (DEE) while meeting their own. We also see that our algorithm exhibits significant performances for all the fairness measures except for VAL which does not have close relationship with the equal experience that aim
observations. All baseline algonthmss fail to respect
8

---

Under review as a conference paper at ICLR 2022
Table 2: Prediction error (RMSE) and faireess performances of the matrix factorization based algorithm on MovieLens IM dataser. The boldface indicates the best result and the underline denotes the second best. Our algorithm offers great fairness performances for all the measures, except for VAL, which our framework does not target.
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0.8541 + 0.0033
0.2447
0.0134
0.3227 + 0.0031
0.0058 + 0.0042
0.1291 = 0.0079
Ours (DEE)
0.8641 = 0.0047
0.0014
0.0008
0.2941 + 0.0024
0.0018 + 0.0016
0.0007 = 0.0004
Ours (DER)
0.8526 + 0.0029
0.0114
0.0041
0.3332 + 0.0050
0.0055 + 0.0022
0.0014 = 0.0001
VAL-hased
0.8529 + 0.0011
0.3659
0.0033
0.0942 + 0.0016
0.0261 + 0.0020
0.1388 + 0.0030
UGF-based
0,8550 0.0015
0.2492
0.0100
0.3285 + 0.0051
0.0001 + 0.0001
0.1355 + 0.0038
CV5-based
0.8549 0.0018
0.0721 0.0069
0.3319 0.0046
0.0065 0.0042
0.0002 3.45e-5
Table 3: Prediction error (RMSE) and fairesss performances of the matrix factorization based algorithm on Last FM 360K dataset.
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0.6720 0.0024
0.0840 0.0110
0.2297 0.0020
0.0404 1: 0.0033
0.0204 0.0104
Ours (DEE)
0.6892 0.0040
0.0082 = 0.0023
0.2777
0.0048
0.0161 + 0.0009
0.0040
0.0011
Ours (DER)
0.6830 0.0033
0.0711
0.0140
0.2588
0.0024
0.0356 : 0.0072
0.0047 4 0.0007
VAL-based
0.6802 0.0006
0.1461 0.0216
0.0016 + 2.03e5
0.0234 + 0.0020
0.0324 0.0202
UGF-based
0.6705
0.0030
0.0644 0.0313
0.2366
0.0016
0.0011 4.03e-5
0.3221 0.0157
CVS- -aSeb
0.6758 0.0037
0.0791 = 0.0136
0.2448 0.0034
0.0373 = 0.0063
0.0012 0.0003
5 EXTENSION
In this section, we discuss the extension of our work: (i) introducing a new notion that bears a similar spirit equalized odds, and (ii) applying our notion the fair ranking context.
Our fairness notion equal experience aims at recommending a variety of items for all user groups. Demographic parity is similar our notion in the sense of considering the imrlevancy of predictions to groups. On the other hand, equalized odds is the fairness notion that concerns equal error rates (e.g.. true/negative positive rate) across user groups by employing the ground-truth label Y. Similar to equalized odds, our measure in recommender systems can readily be extended to a setting in which the ground-truth label is available to exploit. The key idea is to promote Y 1 (Zuser, Zitem) See Appendix A.7 for details.
Many end-to-end recommender systems offer recommendation list via two processes: (i) candidate generation, and (ii) ranking. In this work, we focus on the first candidate generation which we built collaborative filtering for. But our proposed notion can also be applicable in generating end ranked list The idea behind the end-ranked list generation is define Y an indicator function which returns 1 when the item of interest belongs to, say top-k item set (0 otherwise). In this case, the same notion Y 1 (Zuser, Zitem) serves proper role. role. We leave detailed implementation of this notion under the fair ranking context for future work.
CONCLUSION
We introduced a novel fairness notion, equal experience, capable of respecting the desired require- ments for fair recommender systems: independence between preference predictions and user groups; conditional independence for a certain user group, and vice versa for item groups. The notion also seamlessly integrates into prior fairnsss algorithms. Extensive experiments revealed the existence of unfairness (or bias) w.r.t. equal experience, and our fair optimization framework successfully mitigates such unfairness with minimal prediction accuracy loss. Our future work of interest is four-folded: (i) merging our notion with unexamined algorithms relying upon Rényi correlation or Wasserstein distance; (ii) constructing robust and fair recommender system in the presence of data poisoning: (iii) developing a blind fair recommender system without sensitive attributes; and (iv) extending our notion to fair ranking.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
The proposed fairness notion and framework (which faithfully respects the notion) will enable recommender system to offer an enough opportunity for users to experience various groups of items against stereotypes. Users are further influenced by the various experiences when making more essential choices such as choosing job. Hence, it can give significant impacts upon users who can expand their options via diverse experiences. On other hand, users provide sensitive information to service provider of fair recommender systems, and there is possibility that the user's sensitive information could be abused. Hence, one future work of great potential might be to develop a framework that ensures fairesss in the blind setting where sensitive information is not available.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of experiments, we provide step-by-step guideline synthetic data construction in Section 4.1 We also provide a thorough description regarding two real datasets, MovieLens 1M (Harper & Konstan 2015) and Last FM 360K (Celmal 2010), along with data preprocessing methods in Section 4.2 All the hyperparameter settings w.r.t. synthetic and real data experiments are listed in Appendix A.4 and A.5 respectively. We present the average running time of all algorithms together with specific computer configuration details in Appendix A.6 Lastly, the codes are available at httttps: /tttt::///ithbbb http:///...coooi////////i.r.ee
REFERENCES
Karim M Abadir and Jan R Magnus. Matrix algebra, volume 1. Cambridge University Press, 2005.
Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Li Wei, Yi Wu, Lukasz Heldt, Zhe Zhao, Lichan Hong, Ed H Chi, et al. Faimess in recommendation ranking through pairwise comparisons. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2212 2220, 2019.
Asia Biega, Krishna P Gummadi, and Gerhard Weikum, Equity of attention: Amortizing individual fairesss in rankings. In The 41st international sigir conference research & development in information retrieval, pp. 405-414, 2018.
Robin Burke. Multisided fairness for recommendation. arXiv preprint arXiv: arXiv: (70.::0077000093, 2017.
Emmanuel J Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foun- dations of Computational mathematics, 9()):177777, 2009.
Oscar Celma. Music Recommendation and Discovery in the Long Tail. Springer, 2010.
Jaewoong Cho, Gyeongjo Hwang, and Changho Suh. A fair classifier using kernel density estimation. Advances in Neural Information Processing Systems, 33:15088-15099 15099, 2020a.
Jaewoong Cho, Gyeongjo Hwang, and Changho Suh. fair classifier using mutual information. In 2020 IEEE International Symposium on Information Theory (ISIT), pp. 2521-2526 IEEE, 2020b.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Richard A Davis, Keh-Shin Lii, and Dimitris N Politis. Remarks some nonparametric estimates of density function. In Selected Works of Murray Rosenblau, pp. 95-100. Springer, 201
Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. In Advances in Neural Information Processing Systems 31 (NeurIPS). 2018.
C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. S. Zemel. Fairness through awareness. Innova- tions in Theoretical Computer Science Conferennce (ITCS), 2012.
Maryam Fazel. Matrix rank minimization with applications. PhD thesis, PhD thesis, Stanford University, 2002.
10

---

Under review as a conference paper at ICLR 2022
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference knowledge discovery and data mining, pp. 259 399-268,10 2015.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed Huai hsin Chi, and Alex Beutel. Counterfactual faimess in text classification through robustness. AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES), 2018
Aurélien Géron. Hands-On Machine Learning with Scikit-Learn & TensorFlow. O'Reilly, 2017.
Aurélien Géron, Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems. O' O'Reilly Media, 2019.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances Neural Information Processing Systems 29 (NeurIPS). 2016.
Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TūS), 2015.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pp. 173-182, 2017.
Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492-518.
Springer, 1992.
Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification. In Proceedings of the Thirty-Fifth Conference Uncertainty in Artificial Intelligence, (UAI), 2019.
Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classificatio.. In Uncertainty in Artificial Intelligence, pp. 882-87227 PMLR, 2020.
Toshihiro Kamishima and Shotaro Akaho. Considerations recommendation independence for find-good-itent task. 2017.
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 35-50. Springer, 2012.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 142.::411..690.. 2014a.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014b.
Ychuda Koren. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 426-434, 2008
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Neural Information Processing Systems 30 (NeurIPS), 2017.
Kiwon Lee, Yong H Lee, and Changho Suh. Alternating autoencoders for matrix completion. In 2018 IEEE Data Science Workshop (DSW), pp. 130-134. IEEE, 2018
Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. User-oriented fairness in recommendation. arXiv preprint arXiv:2104.10677 2021.
Jérémie Mary, Clément Calauzenes, and Noureddine El Karoui. Faimess-aware learning for continu- ous attributes and treatments. In International Conference on Machine Learning, pp. 4382-4391. PMLR, 2019.
11

---

Under review as a conference paper at ICLR 2022
Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. Towards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairnsss & satisfaction in recommendation systems. In Proceedings of the 27th acm international conference on information and knowledge management, pp- 2243-2251, 2018.
Razieh Nabi and Ilya Shpitser. Fair inference on outcomes. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), 2018.
Harikrishna Narasimhan, Andrew Cotter, Maya Gupta, and Serena Wang. Pairwise fairness for ranking and regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5248-5255, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 32 (NeurIPS), 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Chris Russell, Matt J Kusner, Joshua Loftus, and Ricardo Silva. When worlds collide: Integrating different counterfactual assumptions in fairness. In Advances in Neural Information Processing Systems 30 (NeurIPS). 2017.
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted boltzmann machines for collaborative filtering. In Proceedings of the 24th international conference on Machine learning, pp. 791-798, 2007.
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations treatments: Debiasing learning and evaluation. In International Conference on Machine Learning, pp. 1670-1679. PMLR, 2016.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th international conference on World Wide Web, pp. 111-112, 2015.
Ashudeep Singh and Thorsten Joachims. Fairness of exposure in rankings. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 2219-2228, 2018.
Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro. Learning non- discriminatory predictors. In Conference on Learning Theory, pp. 1920-1953, PMLR, 2017.
Yongkai Wu, Lu Zhang, and Xintao Wu. Counterfactual fairness: Unidentification, bound and algorithm. In Proceedings of the Twenty-Eigg Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, 2019.
Lin Xiao. Zhang Min, Zhang Yongfeng, Gu Zhaoquan, Liu Yiqun, and Ma Shaoping. Fairness- aware group recommendation with pareto-efficiency. In Proceedings of the Eleventh ACM Conference on Recommender Systems, pp. 107-115, 2017.
Sirui Yao and Bert Huang. Beyond parity: Fairness objectives for collaborative filtering. arXiv preprint 17X5v:1705.00m 088044 2017.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreat- ment. In Proceedings of the 26th international conference on world wide web, pp. 1171-1180.
2017a.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Faimess constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics, pp. 962-970. PMLR, 2017b.
12

---

Under review as a conference paper at ICLR 2022
Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Megahed, and Ricardo Baeza- Yates. Fa ir: A fair top-k ranking algorithm. In Proceedings of the 2017 ACM on Conference Information and Knowledge Management, pp. 1569-1578, 2017.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335- 340, 2018.
Junzhe Zhang and Elias Bareinboim. Fairness in decision- - the causal explanation formula. In Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence (AAAI), 2018a.
Junzhe Zhang and Elias Bareinboim. Equality of opportunity in classification: causal approach. In Advances in Neural Information Processing Systems 31 (NeurIPS), 201 8b.
A APPENDIX
A.1 OUTLINE
In the appendix, we first provide further explanation for the implementation of KDE technique. We also present other choices for measure of equal experience in further detail. Next, we provide detailed explanation of experimental settings on both synthetic and real datasets: MovieLens IM (Harper & Konstan (015) and Last FM 360K (Celmal 2010). We then present additional experimental results which are not included in the main paper due to space limitation, and as well as provide complexity analysis of our approach. We also provide detailed explanation for the extension of our measure to the faimesss notion: Y 1 (Zueer Zitem) and present experimental results for this extension.
A.2 IMPLEMENTATION OF KDE TECHNIQUE
In order to approximate V DEE, we first calculate the gradient of P( = 1) employing the technique in Cho et al. (0200a):
m
Mij
PY nmhh
fk
V V.
(17)
We can apply the same procedures w.r.t. the second interested probability P(Ý = 1 Zitem
22, Zuser 21). Merging equation and the counterpart w.r.t. the second probability, we can readily
obtain:
DEE RV
Σ Σ
H's (P(2- = 1/ = 22, Zuser = 21) 1P(5 = =1)) 1))
21 Eu 2rte
(18)
(P(
1 22, Zuser 21) -pp = 1))
where Hs(x) denotes the Huber loss 5))
Huber 1992 that takes f when |x| S 6; otherwise 8([x]
A.3 OTHER CHOICES FOR A MEASURE OF "equal experience"
Instead of DEE in equation 12 one can resort other measures based prominent tools employed in the faimess literature: covariance (Zafar et al. 2017a|b), mutual information (Zhang et al. 2018 Kamishima et a 2012, Cho al., 2020b). Wasserstein distance (Jiang et al. (2020) and Rényi corrllation (Mary et al. 2019). For instance, the covariance-basse approach allows to take Ltair as:
Ltair
E
where Z == (Zitem Zuser).
(19)
Here use & instead of V, as Y incurs non-differentiabiiity. hindering implementation. In the case of mutual information, one can take Ltair as:
Ltair I(Y; Zitem, Zuser) > I(Ý; Zitem Zuser)
(20)
13

---

Under review as a conference paper at ICLR 2022
Again, for of implementation, we employ Y. This choice is also relevant because it serves as an upper bound of T(Y; Zitem: Z. Notice that Y is function of V. Reducing T T(Y; Zitem: )
yields the minimization of the interested quantity I(Y; Zitam: Zuser). For implementation of
[(;; Zitem. Zuser), we may employ the variational optimization technique in (Zhang et al. al. 2020b) translate into a function optimization which can also be parameterized. Other choices can also be dealt with properly relying upon the associated techniques in (Jiang et al. Mary al. 2
2018 Cho
2020
A.4 SYNTHETIC DATASET EXPERIMENTS
We consider a setting in which (r, n, m) (20, 600, 400). The synthetic data generated under the setting is randomly split into two subsets: 90% train set and 10% test set. Since Mi E (+1,-1}, we set the threshold = 0, i.e., Y 1( 2 0}. We train a matrix factorization (MF) based recommender system with the same rank that of the dataset, i.c., L E 2000x20 and R E R 2000000 We set hyperparameters (ó, h) = (0.01, 0.01) and À = 0.99 for KDE-based algorithm implementation. We use Adam optimizer for 1. 1,000 iterations using full gradient. The learning rate set to le-3 and (/1, [2) (0.9, 0.999). The additional experimental results in a variety of scenarios are listed from Table 4 to We also visualize how the predicted preference rate of item groups for
every user group Pr(Y
Zuser; Zitem) changes under our framework. See Fig. and
Table 4: Prediction error (RMSE) and fairness performances on the synthetic dataset. The dataset preserves observation bias (qo. Φ1) = (0.2, 0.02) while exhibiting no population imbalance (Po. (P).P) (0.4. 0.4). The boldface indicates the best result and the underline denotes the second best. Each baseline, say VAL-based approach, enjoys the best fairness performance only for the measure focused therein, VAL, while it does not work well under the other fairness measures. On the other hand, our algorithm offers great fairness performances for all the measures, except for VAL, which our framework does not target.
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0,8423 + 0.0086
0.1002 + 0.0264
0.2992 0.0142
0.0160 + 0.0031
0.0066 + 0.0034
Ours (DEE)
0,8611 = 0.0116
0,0022 = 0.0003
0.3273
0.0043
0.0006 = 0.0002
0.0003 = 0.0001
Ours (DER)
0.8605 + 0.0069
0,0414 + 0.0059
0.3130
0.0103
0.0207 = 0.0030
0.0001 = 0.0001
VAL-based
0.8460 + 0.0081
0,0829 + 0.0035
0.0003 + 1.25e-5
0.0138 + 0.0019
0.0066 + -0.0014
UGF-hased
0.8546 + 0.0050
0.1137 + 0.0149
0.3163
0.0036
0.0004 + 0.0001
0.0195 + -0.0031
EVSeha
0.8622 + 0.0072
0.1189 + 0.0222
0.3228
0.0076
0.0220 + 0.0035
0.0001 + 4.76e-5
Table 5: Prediction error (RMSE) and fairness performances on the synthetic dataset. The dataset preserves observation bias (4n, ạ = (0.2, 0.03) while exhibiting no population imbalance (Po- Pi) 0.44..
Mcasure
RMSE
DEE
VAL
UGF
CVS
Unfair
0,8031
0.0127
0.0783 0.0053
0.2248 + 0.0068
0.0161 0.0043
0.0073 0.0040
Ours (DEE)
0.8240 0.0074
0.0045 0.0032
0.2427 = 0.0089
0.0018
0.0018
0.0005
0.0004
Ours (DER)
0.8287 0.0080
0.0507
0.0086
0.2495 + 0.0063
0.0253
0.0043
0.0001
0.0001
VAL-based
0.8155 +: 0.0100
0.0779 = 0.0042
0.0003 1 9.26e-6
0.0087 0.0014
0.0082 0.0013
Uased
0.8150 + 0.0111
0.0910
0.0109
0.2338
1 0.0092
0.0008 1 0.0003
0.0211 1 0.0021
CVS-based
0.8151 0.0115
0.0791 = 0.0110
0.2374
+ 0.0092
0.0234 4 0.0014
0.0002 1 0.0001
14

---

Under review as a conference paper at ICLR 2022
Table 6: Prediction error (RMSE) and fairness performances on the synthetic dataset. The dataset preserves population imbalance (Po. p1) (0.4, 0.1) while exhibiting no observation bias (go, 91) (0.2,0.2) 0.2).
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0.0837 0.0149
0.58591: 0.0011
0.0727
0.0009
0.0193 0.0004
0.0018 0.0005
Ours (DEE)
0.6821 0.0025
0.0123 0.0004
0.1865
0.0096
0.0004 1 0.0002
0.0003
0.0002
Ours (DER)
0.6761
0.0507 0.0052
0.1885
0.0100
0.0254 + 0.0026
0.0004 0.0002
VAL-based
0.3436 0.0110
0,5648 = 0.0022
0.0002 + 9.40c-6
0.0182 = 0.0007
0.0018 0.0006
UGF-based
0,5640 0.2033
0.4495 = 0.1660
0.0935
0.0527
0.0001 3.62e-5
0.0047 0.0024
CVS-bused
0.1277 0.0107
0.5856 0.0015
0.0690
0.0005
0.0188 + 0.0006
0.0002 0.0001
Table Prediction error (RMSE) and fairness performances on the synthetic dataset. The dataset
preserves population imbalance (Po, P1) (0.4, 0.2) while exhibiting no observation bias (go, 91) (0.2,0.2) 0.2).
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0.0600 0.0102
0.3789 + 0.0007
0.0697 0.0004
0.0217 0.0005
0.0007 0.0003
Ours (DEE)
0.6125 + 0.0038
0.0115 + 0.0009
0.0938 1 0.0029
0.0005
0.0001
0.0003 1: 0.0002
Ours (DER)
0.6187 = 0.0067
0.0622 - 0.0015
0.0894 : 0.0016
0.0311
0.0008
0.0006 + 0.0002
VAL-based
0,3451 = 0.0109
1.684400000
0.0002 3.03e-6
0.0207
0.0011
0.0033 + 0.0011
UGF-based
0.4221 0.0130
0.3713 0.0041
0.0592 0.0020
0.0002
2.38e-5
0.0058 0.0011
WSs-based
0.1046 0.0059
0.3790 + 0.0007
0.0672 0.0012
0.0219 0.0004
0.0001 4.43e-5
Table 8: Prediction error (RMSE) and fairness performances on the synthetic dataset. The dataset preserves population imbalance (Po. P1) (0.4, 0.3) while exhibiting no observation bias (go. 91) (0.2,0.2) 0.2).
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0.0582 = 0.0098
0.1952
0.0002
0.0678
0.0006
0.0233 0.0002
0.0022 + 0.0005
Ours (DEE)
0.4525 + 0.0058
0.0049
0.0028
0.0767
0.0023
0.0004
0.0003
0.0002 0.0001
Ours (DER)
0.4946 = 0.0058
0.0609
0.0049
0.0827
0.0021
0.0305
0.0025
0.0003 + 0.0002
VAL-based
0.3460 0.0046
0.1863
0.0018
0.0002
6.32e-6
0.0246 0.0010
0.0030 + 0.0012
UGF-based
0.4031 0.0095
0.1830
0.0040
0.0683
0.0024
3.84e-5
2.62e-5
0.0039 + 0.0006
CVS-based
0.1077 + 0.0178
0.1927 0.0014
0.0660 0.0008
0.0228 0.0011
0.0002 1: 2.30e-5
unfair
0.5
0.5
Pr(Y =1 Zam))
0.4
0.4
0.3
0.3
M
F
M
F
M
F
M
F
0.2
0.2
0.1
0.1
0
M
F
M
F
Figure 3: Predicted preference rate item groups for every user group Pr(Ý = 1 Ziemm on the synthetic dataset. The dataset preserves observation bias (go, q1) (0.2, 0.01) while exhibiting no population imbalance (po.Pi) = (0.4, 0.4).
A.5 REAL DATASETS EXPERIMENTS
MovieLens IM: The associated task is to predict the movie rating on a 5-star scale. This dataset contains 6,040 users, 3,900 movies, and 1, 1,000,209 ratings, i.e., rating matrix is 4.26%
15

---

Under review as a conference paper at ICLR 2022
unfair
ours
0.45
Pref = 1 Ziumm
04
0.35
0.35
0.3
0.3
0.25
02
0.15
0.15
0.1
0.05
M F
M
F
0.05
M
F
M
F
0
M
F
M
F
Figure 4: Predicted preference rate of item groups for every user group Pr(Y -1|Zuser> Zitem) on the synthetic dataset. The dataset preserves population imbalance (Po.P1) = (0.4, 0.1) while exhibiting
no observation bias (go. 91) = (0.2, 0.2).
full We divide user and item groups based on gender and genre, respectively. Action, crime, filme-noir, war are selected as male-preferred genre, whereas children, fantasy, musical, romance are selected female-prefered genre. We can select male-preferred and female- preferred genres in a variety of ways based on ratings and observations. For various scenarios, the experimental results with similar trends are obtained, so we report the results for one representative scenario. If we assume that the real dataset is generated from the same model as the synthetic dataset, we can estimate the following probabilities. We empirically estimate the interested probabilities w.r.t. population imbalance as: PMM = 0.627, PMr 0.517, Pr.M 0.622, and fr = 0.595. Similarly we obtain the estimates for the other probabilities w.r.t. observation bias: GMM 0.053, @MF 0.037, OF 0.037, and @F = 0.046.
Last FM 360K: The associated task to predict whether the user likes the artist or not. This dataset contains 359,347 users, 294,015 artists, and 17,559.530 play counts, i.c., rating matrix 0.02% F The data for play counts is converted to binary rating: +1 if counts > average, otherwise -1. We divide user and item groups based on gender and genre, respectively. Since this dataset only contains gender information, we use Last.fm AP collect the genre of corresponding artist's music; the tag was associated with 5,706 artists. We also randomly select 5000 male and 5000 female users. Among 10 genres, we choose hip-hop and musical for male and female preferred genres, respectively. The final rating matrix of 10,000 users and 5,706 artists is 0.55% full. From the real data, we obtain empirical estimates for the interested probabilities w.r.t. population imbalance: PMM = 0.548, PMr 0.421, P. 0.438 and prr = 0.529. Similarly we obtain the estimates for the other probabilities w.r.t. observation bias: TMM = 0.0054, @MP 0.0011, TF = 0.0036 and @ドF = 0.0038.
We randomly split the real datasets into 90% train set and 10% test set. In case of MovieLens data, the rating is five-star based, so we set the threshold T 3, i.e., Y = 1(8 > 3). On the other hand, for LastEM dataset, we set 7 = 0 as My E {+1,- -1.. We run experiments employing both matrix factorization (MF) based and autoencoder (AE) based techniques. We set the rank as 512 for MF-based algorithm as was found by hyperparameter search. The structure of the employed autoencoder (Sedhain et al. 2015) is follows: (i) encoder has two linear layers: 512 nodes with ReLU actiavation and 512 nodes with dropout layer (rate = 0.7) and ReLU activation; (ii) decoder has one layer with 512 nodes. For MovieLens 1M data (five-star ratings), we apply the clipping to the decoder output to fit into the range. For LastFM 360K data (binary rating: +1 and -1), we apply tanh activation. Hyperparameters for KDs- algorithm are to (6, h) (0.01, 0.01) and A = 0.9. We use Adam optimizer for 1,000 iterations using full gradient, and the learning rate is
hhtp://www.movilees..om http:///ccemmm.nc/ee http://www.lattffft.
16

---

Under review as a conference paper at ICLR 2022
set to 1e-3. Since the main paper contains mostly MF- experiments, here only present the performances of autoencoder based algorithm on both real datasets.
Table 9: Prediction error (RMSE) and fairness performances of the autoencoder based algorithm MovieLens IM dataset. We observe the same performance trends as those in Table
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0.8369 1 0.0012
0.2477 + 0.0175
0.3412 = 0.0031
0.0419 + 0.0042
0.1158
0.0025
Ours (DEE)
0.8437 + 0.0042
0.0120 + 0.0028
0.3338 = 0.0037
0.0039 + 0.0022
0.0042 + 0.0010
Ours (DER)
0.8411 + 0.0027
0.0285 + 0.0084
0.3395 = 0.0048
0.0144 + 0.0046
0.0061
0.0023
VAL-hased
0.8433 + 0.0022
0.2138 + 0.0363
0.2128 + 0.0143
0.0299 + 0.0172
0.0918
0.0070
based
0.8491 + 0.0056
0.1934 + 0.0109
0.3391 = 0.0040
0.0011 + 0.0006
0.0982 0.0055
CVS-based
0.8495 + 0.0050
0.0808 + 0.0225
0.3424
= 0.0069
0.0343 + 0.0105
0.0023 0.0010
Table 10: Prediction error (RMSE) and fairness performances of the autoencoder based algorithm Last FM 360K dataset.
Measure
RMSE
DEE
VAL
UGF
CVS
Unfair
0.6534 0.0024
0.1003 + 0.0172
0.2253 + 0.0031
0.0501 0.0086
0.0006 + 0.0002
Ours (DEE)
0.6649 +: 0.0212
0.0024 + 0.0005
0.2213 - 0.0133
0.0012 1: 0.0008
0.0007 = 0.0006
Ours (DER)
0.6501 + 0.0004
0.0878 = 0.0015
0.2204 + 0.0004
0.0439 0.0008
0.0008 = 0.0001
VAL-based
0.6828 + 0.0202
0.0571 = 0.0054
0.1915 + 0.0038
0.0288 0.0029
0.0063 = 0.0029
UGF-based
0.6861 + 0.0310
0.0485 = 0.0025
0.2098 + 0.0143
0.0001 = 5.00e-5
0.0021 = 0.0016
CVS-hased
0.6685 + 0.0079
0.0960 + 0.0096
0.2421 + 0.0105
0.0480 + 0.0047
0.0002 = 3.24e-5
unfair
ours
Pr(y 1|/ Znwn)
0.6
0.6
0.5
0.5
0.4
0.4
0.3
M
F
M
F
0.3
M F
M
F
0.2
0.2
0.1
0.1
M
F
M
F
Figure 5: Predicted preference rate of item groups for every user group Pr(y Zuser: Zitem) of the matrix factorization based algorithm MovieLens IM dataset.
unfair
ours
0.35
0.35
Pr(Y = 1 Zueer Zaem)
0.3
0.3
0.25
0.25
02
0.2
M
F
M
F
M
F
M
F
0.15
0.15
0.1
0.05
M
F
M
F
Figure 6: Predicted preference rate item groups for every user group Pr(y Zuueer Zitem) of the matrix factorization based algorithm Last FM 360K dataset.
17

---

Under review as a conference paper at ICLR 2022
unfair
ours
0.7
0.7
Pr(Y = 1| Zuen)
0.4
0.4
0.3
M
F
M
JF
0.3
M
F
M
F
M
F
M
F
Figure 7: Predicted preference rate of item groups for every group Pr(Y 7 Zitem) ) of the autoencoder based algorithm on MovieLens 1M dataset.
unfair
ours
0,2
0.35
Pr(Y = 1 Ziemm
0.3
0.15
0.25
0.2
0.15
0.05
M'
F
M
F
0.1
M
F
M
F
0.05
M
F
M
F
Figure 8: Predicted preference rate of item groups for every user group Pr(ý the autoencoder based algorithm on Last FM 360K dataset.
1 Zuser: Zitem) of
A.6 COMPLEXITY ANALYSIS
We do complexity analysis of ours in light of other baselines. For comparison, we consider the running time measured under Pytorch on Xeon Silver 4210R CPU and TITAN RTX GPU. Table IT presents the running times of matrix factorization based algorithms on the synthetic and real datasets. While our approach provides better fairness performance w.r.t. DEE (as in the above tables), it comes at a cost of an increased complexity, around twice relative to the CVS-based algorithm.
Table 11: The running time (in seconds) of our algorithm and baselines on the synthetic and two real datasets: MovieLens IM and LastFM 360K.
Dataset
Synthetic
MovieLens 1M
LastFM 360K
Unfair Ours (DEE)
2.15
6.72
16.27
13.23
86.82
192.02
VAL-based (Yao & Huang 2017
5,83
201.08
477.14
UGF- (Li et al. 2021
7.05
60.14
136.69
CVS-based (Kamishuma & Akaho 2017
7.16
47.89
104.42
A.7 EXTENSION TO THE FAIRNESS NOTION: Y
Zuser: Zitem Y
Similar to DEE in equation
12 quantify the notion via:
Σ E Σ
P(Ý 1|Y = 4) - P(Ý = 1|2 = Zitem 22, Zuser = =1)
(21)
(0.1) 21 2ine 2rm
for arbitrary alphabet sizes |uuser| and Z
18

---

Under review as a conference paper at ICLR 2022
Using the KDE approach, similarly obtain:
P(- = 1/Y 1) {- (o
1
Σ
Ps
T Mij
[Tyl
h
(Kj)Et,
where Fx(0) = l fx(t)dt; Yy := 1{M 2 JJ and by {(i,j) (i,j) Q Yis y). We can then compute the gradien w.r.t. w as:
V. 1
Σ Σ 1T ((.))1,
1
Mij V.. Mi..
(22)
h
We can then enjoy family of gradient-based optimizers (Géron 2017 Kingma & Ba 2014a). We
provide experimental results for the extension on Moviel.nns IM (Harper & Konstan 2015) real dataset. We run experiments employing both matrix factorization (MF) based and autoencoder (AE) based techniques. We demonstrate that the framework based on the extended notion can indeed mitigate such unfairness while exhibiting minor degradation of recommendation accuracy. The results are listed Table [2 and
Table 12: Prediction error (RMSE) and fairness performances of the matrix factorization based algorithm on MovieLens IM dataset. The boldface indicates the best result and the underline denotes the second best. The approach based on the extended fairnsss notion (conditioning on Y), enjoys the best fairness performance for the measure focused therein.
Measure
RMSE
DEE
Conditioning on Y 21
VAL
UGF
CVS
Unfair
0.2447 0.00134
0.3494 0.0077
0.3227
0.1291
Ours (DEE)
0,8641 1: 0.0047
0,0014
= 0.9008
0.3005 = 0.0048
0.2941 = 0.0002
0.0018
: 0.0016
0.0007
D.00044
Conditioning Y
0.8576 1 0.0011
0.1451 1: 0.0079
0,0283 1 0.0057
0.3230 1: 0.0012
0.0052
0.0029
0,0668
0.0051
VAL-based
0.8529 1 0.0011
0.3659 + 0.0033
0.4679 + 0.0098
0,0942 : 0.0016
0.0261
0.0020
0.1388
0.0030
UGF -uF-b
0,8550
1: 0.0015
0.2492
3: 0.0100
0.3657 1: 0.0084
0.3285
4: 0.0051
0.0001
1: 0,0001
0.1355
0.0038
CVS-hased
0.0018
0.0069
0.3260 0.0135
0.3319
0.0046
0.0065
0.0042
0.0002
3.45e-5
Table 13: Prediction error (RMSE) and fairnsss performances of the autooncoder based algorithm on MovieLens 1M dataset.
Measure
RMSE
DEE
Conditioning on
CVS
Unfaiz
0.8369 = 0.0012
0.2477 = 0.0175
0.3557 = 0.0086
0.3412
2: 0.0031
0.0419 1: 0.0042
0.1155 2: 0.0025
Ours
0.8437 1: 0.0042
0.0120
0.0028
0.2260
0.0203
0.3338
0.0037
0.0039 0.0022
0.0042
0.0010
Conditioning on Y
0.8467 + 0.0012
0.1157
0.0203
0.0505 : 0.0264
0.3305
0.0039
0.0131
0.0139
0,0551
2: 0.0046
VAL hased
0.8433 1: 0.0022
0.2138
1: 0,0363
0.3372 : 0.0117
0.2128
0.0143
0.0299
0.0172
0.0918
1: 0.0070
UGF-based
0.3391
0,0011
CVS-hased
0.8495
0.0050
0.0808
0.0225
0.2408 0.0165
0.3424
0.0069
0.0343
0.0105
0.0023
0,0010
19


Reviews:
[{"review_id": "xQ_24pVfBoD", "reviewer": "Reviewer_dddS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think that the paper solves an interesting problem and generalizes previous work. The main concern here seems to be with the weight of the contribution. Specifically, the new notion of fairness is not very complicated and seems to follow from previous work. Further, the optimization algorithms also seems to follow from previous work and are not novel. If there were significant technical issues it does not seem to me that they were well-emphasized in the paper.  \n\nHave the authors thought about including any theoretical guarantees. For example, convergence of the optimization. Or a guarantee that the fairness notion would hold at least approximately. \n\nSome minor representation issues:\n1-Eq(2) on page 3, shouldn't \\hat{M} belong to a constraint set such low rank.\n2-Eq(4) on page 3, I think (M_{ij}-\\hat{M}_{ij}) is missing a square. \n\n\n\n", "summary_of_the_paper": "The paper is concerned with fairness in recommendations. Specifically, there are groups of users and groups of items. Previous work has modelled fairness as the constraint of all user groups having the same accuracy or as the prediction probability being independent of the item group or the user group. In this work, the notion is generalized so that the prediction is independent of both the item and user group. Optimization algorithms are shown to solve this problem along with experimental results.  ", "summary_of_the_review": "While the paper introduces a new meaningful notion of fairness. The notion closely follows the previous work. The optimization methods used also follow the previous work. Overall, this makes the contribution of the paper very incremental. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gHqJvBPby6N", "reviewer": "Reviewer_85io", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The presented study did not analyse the diversity of the recommended items in the experimental results, which the new proposed fairness notion aims to improve. Also, it would further strengthen the study if the authors could analyse who could benefit from the proposed method, e.g., females who enjoyed action or crime movies but did not have much historical data, or others?\n\nAlso, as this study focused on the problem of rating prediction in recommender systems, it would good to also test the effectiveness of the proposed method in the ranking setting.", "summary_of_the_paper": "The presented study argued that a fair recommendation should be independent of both user and item. Therefore, the study introduced a new fairness notion, i.e., equal experience, and further incorporated this fairness notion as a regularisation term in the matrix completion framework to construct a fair recommender system. The proposed method was evaluated with three datasets (one synthetic and two real datasets). ", "summary_of_the_review": "The new fairness notion introduced by the presented study can provide certain new knowledge and insights on how to construct a fair recommender system, and the effectiveness of the proposed recommendation method seemed to be supported by the experimental results. However, the study can be further strengthened if more in-depth analysis on the recommendation results can be provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TECFkWhFGda", "reviewer": "Reviewer_zEp7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The result due to chain rule is pretty interesting and neat in that joint independence of preference prediction user-item groups ensures multiple independence notions together. The authors do a good job of giving examples of the definition in terms of all of the independence notions.\n\nEmpirically, the method proposed in the paper gives good fairness performance on all fairness metrics compared (except one), while the specific papers only specialize in reducing one form of unfairness at a time.\n\nWeaknesses: \nThe normative goal of the fairness notion is not well-motivated in the paper. From what I understand, making the preference prediction independent of both the item group and the user group means that different base rates of preference for item groups among different user groups should not mean different rates of recommendation. The authors give an example of science vs literature preference for male and female users, where according to this measure, science and literature would be recommended at the rate at which they appear in the item set (because of the independence between Y~ and Z_item). In my opinion, this notion needs to be further motivated and included with a proper discussion about the scope of what item and user groups could mean. For example, if there are more literature courses to be recommended than science courses, they might not have the same capacity or preference among the general user group as science courses. While it may make sense to recommend science and literature equally to male and female users, wouldn't recommending science and literature at proportional rates only make sense when they are equally preferred in the general population?\n\nIn the spirit of the above question, I would like the authors to elaborate on their criticism of fairness notions based on the difference in recommendation accuracies (Paragraph 2 of Introduction). However, later in the future work (Sec 5), the authors seem to advocate for Equalized odds fairness notion which is an accuracy-based notion.\n\nThe setup uses Y>threshold as a preference prediction isn't the most used form of recommendation, while something like the relative ranking/ordering of the items is. The paper does not talk about how the defined fairness metrics would apply if predictions were used in terms of ranking the different items for each user/user type.\n\nIn the experiments section, would it also make sense to use an importance weighting approach (e.g. [1]) to tackle the selection bias created by the data generation process? [1] Schnabel, Tobias, et al. \"Recommendations as treatments: Debiasing learning and evaluation.\" ICML, 2016.\n\nMinor:\n- In paragraph 2 of the Introduction, the authors say: “female students exhibit low ratings on math and science subjects due to …. sampling bias”. It is not clear if the term \"ratings\" means the predictions or scores observed in a dataset. However, the other factor of societal/cultural differences mentioned by the authors does explain the observation well. Perhaps, the authors meant the lack of ratings instead of lower ratings.\n- In related work, some literature from debiasing word embeddings could be relevant to include because those works also include an independence-based notion between embeddings of occupations and gender-specific words. In the current work, since matrix completion is used as the collaborative filtering method, it also corresponds to user and item embeddings and the notions of orthogonality and independence might be relevant to mention (or describe).", "summary_of_the_paper": "The paper defines a fairness notion for a recommender system that is based on the independence of preference predictions to the user group and the item groups. The paper defines a mutual information-based mathematical expression to measure unfairness, called equal experience metric, and then optimize it along with a matrix factorization-based collaborative filtering.", "summary_of_the_review": "The paper tackles an important problem of fairness in recommender systems, and it defines a fairness notion that contains both the user groups and items groups. The metric defined is pretty straightforward to understand. The kernel-based probability density estimation to compute the difference between conditional and marginal probabilities is not a very standard method but it seems to work for the purposes of the experiment. Overall, the fairness notions are not very strongly motivated (as highlighted above), and the paper is missing the guidance around what groupings of items and users are meaningful for such a recommendation task.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7J0_qkqx0J-", "reviewer": "Reviewer_TDXy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is well written and easy to follow.\n+ The related works is well described, especially the representative ones. Fair comparisons on these representative baselines are provided in the experiments.\n+ It is interesting to utilize mutual information for fairness problems.\n\nWeaknesses:\n- I suggest to include DER to be a baseline.\n- I am wondering if there are any difficulties when facing more than group of users and items.\n- As shown in the experiments, the proposed method keeps receiving the worst RMSE performance. Also, from table 6, we can observe that when the proposed performs much better than  the baselines for fairness comparisons, it also performs much worse than others for recommendation comparisons. There is tradeoff between fairness and accuracy, which may need a more detailed analysis. ", "summary_of_the_paper": "This paper proposes a new notion “equal experience” to measure the fairness among groups in recommender systems, and provides an method to optimize this new notion based on matrix completion. Experiments demonstrate the effective of the proposed optimization framework.", "summary_of_the_review": "- Since I am not very familiar with recent work on fairness, the proposed notion \"equal experience\" seems novel to me. \n- The authors provide the optimization method for the \"equal experience\"  based on both traditional ML and deep learning techniques.\n- The results prove that the proposed method can receive better \"equal experience\" optimization performance compared to other baselines.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new (un)fairness metric for recommender systems based on mutual information and then develop an algorithm to account for this metric in matrix factorization-based collaborative filtering. The reviewers all agree that the proposed metric and algorithm are sound at a technical level, however, they have concerns regarding the motivation of the introduced metric as well as the experimental evaluation. The rebuttal by the authors did not persuade the reviewers to reconsider their original assessment and they still argued that their concerns remained. In the final recommendation, the simplicity of the metric was not seeing as a weakness of the work.
Decision: Reject