Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper defines a fairness notion for a recommender system that is based on the independence of preference predictions to the user group and the item groups. The paper defines a mutual information-based mathematical expression to measure unfairness, called equal experience metric, and then optimize it along with a matrix factorization-based collaborative filtering.
main_review: Strengths: The result due to chain rule is pretty interesting and neat in that joint independence of preference prediction user-item groups ensures multiple independence notions together. The authors do a good job of giving examples of the definition in terms of all of the independence notions.

Empirically, the method proposed in the paper gives good fairness performance on all fairness metrics compared (except one), while the specific papers only specialize in reducing one form of unfairness at a time.

Weaknesses: 
The normative goal of the fairness notion is not well-motivated in the paper. From what I understand, making the preference prediction independent of both the item group and the user group means that different base rates of preference for item groups among different user groups should not mean different rates of recommendation. The authors give an example of science vs literature preference for male and female users, where according to this measure, science and literature would be recommended at the rate at which they appear in the item set (because of the independence between Y~ and Z_item). In my opinion, this notion needs to be further motivated and included with a proper discussion about the scope of what item and user groups could mean. For example, if there are more literature courses to be recommended than science courses, they might not have the same capacity or preference among the general user group as science courses. While it may make sense to recommend science and literature equally to male and female users, wouldn't recommending science and literature at proportional rates only make sense when they are equally preferred in the general population?

In the spirit of the above question, I would like the authors to elaborate on their criticism of fairness notions based on the difference in recommendation accuracies (Paragraph 2 of Introduction). However, later in the future work (Sec 5), the authors seem to advocate for Equalized odds fairness notion which is an accuracy-based notion.

The setup uses Y>threshold as a preference prediction isn't the most used form of recommendation, while something like the relative ranking/ordering of the items is. The paper does not talk about how the defined fairness metrics would apply if predictions were used in terms of ranking the different items for each user/user type.

In the experiments section, would it also make sense to use an importance weighting approach (e.g. [1]) to tackle the selection bias created by the data generation process? [1] Schnabel, Tobias, et al. "Recommendations as treatments: Debiasing learning and evaluation." ICML, 2016.

Minor:
- In paragraph 2 of the Introduction, the authors say: “female students exhibit low ratings on math and science subjects due to …. sampling bias”. It is not clear if the term "ratings" means the predictions or scores observed in a dataset. However, the other factor of societal/cultural differences mentioned by the authors does explain the observation well. Perhaps, the authors meant the lack of ratings instead of lower ratings.
- In related work, some literature from debiasing word embeddings could be relevant to include because those works also include an independence-based notion between embeddings of occupations and gender-specific words. In the current work, since matrix completion is used as the collaborative filtering method, it also corresponds to user and item embeddings and the notions of orthogonality and independence might be relevant to mention (or describe).
summary_of_the_review: The paper tackles an important problem of fairness in recommender systems, and it defines a fairness notion that contains both the user groups and items groups. The metric defined is pretty straightforward to understand. The kernel-based probability density estimation to compute the difference between conditional and marginal probabilities is not a very standard method but it seems to work for the purposes of the experiment. Overall, the fairness notions are not very strongly motivated (as highlighted above), and the paper is missing the guidance around what groupings of items and users are meaningful for such a recommendation task.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The presented study argued that a fair recommendation should be independent of both user and item. Therefore, the study introduced a new fairness notion, i.e., equal experience, and further incorporated this fairness notion as a regularisation term in the matrix completion framework to construct a fair recommender system. The proposed method was evaluated with three datasets (one synthetic and two real datasets). 
main_review: The presented study did not analyse the diversity of the recommended items in the experimental results, which the new proposed fairness notion aims to improve. Also, it would further strengthen the study if the authors could analyse who could benefit from the proposed method, e.g., females who enjoyed action or crime movies but did not have much historical data, or others?

Also, as this study focused on the problem of rating prediction in recommender systems, it would good to also test the effectiveness of the proposed method in the ranking setting.
summary_of_the_review: The new fairness notion introduced by the presented study can provide certain new knowledge and insights on how to construct a fair recommender system, and the effectiveness of the proposed recommendation method seemed to be supported by the experimental results. However, the study can be further strengthened if more in-depth analysis on the recommendation results can be provided.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is concerned with fairness in recommendations. Specifically, there are groups of users and groups of items. Previous work has modelled fairness as the constraint of all user groups having the same accuracy or as the prediction probability being independent of the item group or the user group. In this work, the notion is generalized so that the prediction is independent of both the item and user group. Optimization algorithms are shown to solve this problem along with experimental results.  
main_review: I think that the paper solves an interesting problem and generalizes previous work. The main concern here seems to be with the weight of the contribution. Specifically, the new notion of fairness is not very complicated and seems to follow from previous work. Further, the optimization algorithms also seems to follow from previous work and are not novel. If there were significant technical issues it does not seem to me that they were well-emphasized in the paper.  

Have the authors thought about including any theoretical guarantees. For example, convergence of the optimization. Or a guarantee that the fairness notion would hold at least approximately. 

Some minor representation issues:
1-Eq(2) on page 3, shouldn't \hat{M} belong to a constraint set such low rank.
2-Eq(4) on page 3, I think (M_{ij}-\hat{M}_{ij}) is missing a square. 




summary_of_the_review: While the paper introduces a new meaningful notion of fairness. The notion closely follows the previous work. The optimization methods used also follow the previous work. Overall, this makes the contribution of the paper very incremental.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new notion “equal experience” to measure the fairness among groups in recommender systems, and provides an method to optimize this new notion based on matrix completion. Experiments demonstrate the effective of the proposed optimization framework.
main_review: Strengths:
+ The paper is well written and easy to follow.
+ The related works is well described, especially the representative ones. Fair comparisons on these representative baselines are provided in the experiments.
+ It is interesting to utilize mutual information for fairness problems.

Weaknesses:
- I suggest to include DER to be a baseline.
- I am wondering if there are any difficulties when facing more than group of users and items.
- As shown in the experiments, the proposed method keeps receiving the worst RMSE performance. Also, from table 6, we can observe that when the proposed performs much better than  the baselines for fairness comparisons, it also performs much worse than others for recommendation comparisons. There is tradeoff between fairness and accuracy, which may need a more detailed analysis. 
summary_of_the_review: - Since I am not very familiar with recent work on fairness, the proposed notion "equal experience" seems novel to me. 
- The authors provide the optimization method for the "equal experience"  based on both traditional ML and deep learning techniques.
- The results prove that the proposed method can receive better "equal experience" optimization performance compared to other baselines.
review_title: null

Meta-review: This paper introduces a new (un)fairness metric for recommender systems based on mutual information and then develop an algorithm to account for this metric in matrix factorization-based collaborative filtering. The reviewers all agree that the proposed metric and algorithm are sound at a technical level, however, they have concerns regarding the motivation of the introduced metric as well as the experimental evaluation. The rebuttal by the authors did not persuade the reviewers to reconsider their original assessment and they still argued that their concerns remained. In the final recommendation, the simplicity of the metric was not seeing as a weakness of the work.
Decision: Reject