Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a 3-part algorithm for an efficient and fair federated recommender system that preserves privacy. They promote privacy and efficiency by randomly sampling item gradients from only a fraction of clients. The authors present a theorem to give bounds on how the sampling rate affects accuracy. They then develop a global fair model by only using sensitive information about a client's group from a fraction of the population. The global model is then used to regularize item representations in local models to maintain fairness without exposing further sensitive information. The authors show this leads to increased model accuracy, reduced bias, and reduced communication on MovieLens datasets.
strength_and_weaknesses: Strengths:
1. Maintaining fairness while protecting privacy is a challenging problem, and this paper shows good progress on a real-world dataset toward achieving this aim.
2. Multi-part algorithm is intuitive and well presented.
3. Proof of affect of sampling rate on federated accuracy fills a gap in existing literature.

Weaknesses:
1. The problem of maintaining fairness and privacy is incompletely solved as their solution requires a fraction (20%) of user's to sacrifice privacy. The remaining 80% of users only enjoy partial fairness as they are only regularized by a fair model without theoretical guarantees. The authors should explain how realistically they would violate 20% of users' privacy. Perhaps this fraction is comparable to the number of users who opt out / do not opt in to privacy permissions, though this group may not be a representative sample of users overall.
2. The proof of the error bound when sampling clients is unsurprising and seems somewhat obvious. This section can perhaps be moved to an appendix. Furthermore, there is an assumption that users are uniformly distributed among clusters, but in general in fairness, groups tend to be of unequal size.
3. Evaluation is performed on two different sized MovieLens datasets, so the paper is closer to validating on just a single problem domain. The authors should consider other recommender system benchmark datasets.
clarity,_quality,_novelty_and_reproducibility: Clarity - Overall the paper is quite clear and easy to follow. Some interpretations were ambiguous to me, such as why their method would lead to more accurate recommendations when additional fairness constraints are imposed.
Quality - Offline experiments are conducted in a competent way. 
Novelty - Techniques are modestly novel. They are not a far departure from existing techniques, but are combined in a novel and coherent way.
Reproducibility - Although the techniques are described clearly and use an open dataset, I do not believe the authors open-sourced their implementation.
summary_of_the_review: I am marginally inclined to accept this paper. The authors address an important issue with an intuitive and well-explained algorithm. They demonstrate their technique on a real-world dataset and show improvements in efficiency, fairness, and accuracy. However, their solution still manages to violate privacy for 20% of the population while providing only incomplete fairness for the remaining users. Also the novlety of their technique is somewhat modest. The authors should use the feedback from reviews to strengthen the final version of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors improve a recent federated recommendation method for rating prediction with explicit feedback, i.e., FedRec [Lin et al., 2021], by reducing the communication cost and improving the fairness.
strength_and_weaknesses: Strength:
1 The authors derive some complexity bound in client sampling for reducing the communication cost.
2 The authors propose a dual-fair vector update technique for improving the fairness.

Weakness:
1 The authors focus on a very specific problem, i.e. rating prediction with explicit feedback, and a specific type of method, i.e., matrix factorization.
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is well presented.

Novelty: the proposed two techniques (client sampling and dual-fair vector update) are novel

Reproducibility: It is not difficult to reproduce the results on the two public datasets, i.e., MovieLens 100K and MovieLens 1M.

summary_of_the_review: The proposed techniques are indeed novel for the specific problem, i.e., federated recommendation with explicit feedback, and are effective as shown by the empirical studies. However, the technical contribution is not very significant since the authors only focus on a very specific problem among various more important recommendation problems, e.g., matrix factorization for recommendation with implicit feedback.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to improve the communication efficiency of the federated recommender system by incorporating random sampling. Specifically, the paper conducts a theoretical analysis to investigate how the reduced number of sampled clients will impact the model’s accuracy. Upon the above framework, the paper further proposes a de-biased recommender system by adding the fairness constraints in the training loss.

strength_and_weaknesses: Strength:

1. The targeting problem is an important branch of federated learning. 
2. The paper is well written.
3. The proposed method is novel.


Weaknesses:
1. The proposed method relies on two assumptions: 1) clustering clients in the recommendation system, and 2) availability of 20% data at server. These two assumptions are too strong that limits the value of the proposed method. 
2. It is unclear how the clients will be clustered. Because the mentioned fairness is group-wise, the clustering result is very important to decide which client will contribute to the corresponding group. 
3. In Algorithm 1, there are two subfunctions, FairMF and ClientFilling,  that need to be cleary described.
4. Some experiment results need to further analyzed. For exampmle, in Figure 3, why all curves are very unstable at the first server rounds, and the communication rounds is too few to demonstrated that the MF is converged. In Table 3, please analyse why RS-FairFRS(100%) performs worse than RS-FairFRS(20%).

clarity,_quality,_novelty_and_reproducibility: The paper is not clearly described its details. Some of key method relies on the other papers. The paper proposes source code for evaluation. 

summary_of_the_review: The paper is a combination of two mechanisms: 1) random sampling for improving efficiency, and 2) regularization term for fairness. The proposed methods relies on two impractical assumptions. Moreover, the proposed method is reuse the existing technology without limited novelty. The experimental analysis needs to be strengthened to better support its claim.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper propose a federated recommender system which guarantees fairness and communication efficiency while trying to minimize the accuracy drop.
strength_and_weaknesses: Strength:
1. The authors derive the first sample complexity for federated recommender, despite under strong assumptions.
2. The authors evaluate the proposed the method on two real-world datasets and the experimental results seem promising.

Weakness:
1. Some of the assumptions in the paper are not validated. For example, in the first paragraph of 4.1, the authors make a pretty strong assumption that the distribution of the gradients are a mixture distribution by several homogeneous distributions. I would like to see this validated on a real-world dataset.
Another assumption worth validating is ": We assume the availability of data of very few (20%) clients Dserver at the server for evaluating the fairness loss." 20% is a non-negligible percentage. Could the author give a practical scenario where such a big portion of users' data are available?
2. Algorithm 1 is too sketchy. It is understandable if the reader fully understand the context but as a pseudocode it is not self-contained.
clarity,_quality,_novelty_and_reproducibility: Clarity: The writing of the draft is ok but can be further improved. For example, the terms "FairMF" and "FO-ClientBatch" suddenly appear as paragraph titles. I would recommend using one to two sentences to outline the dual update process and the terms before those paragraphs.

Quality: Good. I like the motivation of the paper as fairness is an important yet challenging topic in federated systems. The authors propose a solution with some trade-offs and the experimental results show it works.
 
Novelty: Low novelty. The protocol is a combination of known techniques. The proof for the sample complexity is novel but not very challenging given the strong assumptions.

Reproducibility: The code is included in the supplementary material. I did not have the chance to reproduce the results in the paper.
summary_of_the_review: Overall this is a borderline paper. 

The quality of the paper, including the motivation, the design of the protocol, the evaluation, is okay. The writing can be further improved as I pointed out in the weakness and clarity sections.

The novelty is low. Like I mentioned in the novelty section, the proposed protocol is composed of known techniques. The math for deriving sample complexity is ok but not very challenging.

-----------------------------------------------------------------
I would like to thank the authors for their answers to my question. I decide to keep the score after reading the rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies fairness and efficiency. in the context of Federated Recommender Systems (FRS). Past literature on federated supervised learning shows that sampling improves communication efficiency without jeopardizing accuracy. The authors provide sampling complexity guarantees for this process in the context of FRS, and extend their analysis and sampling approach to address demographic bias and incorporate fairness. They show empirically their proposed method reduces communication costs while maintaining fairness and accuracy.

Reviewers found the problem interesting, but contributions to be somewhat too incremental for ICLR. Broadening the setting considered would help both w.r.t. challenge behind contributions but also the paper's scope. Presentation also needs polishing: crucial details and algorithmic implementations were unclear.
justification_for_why_not_higher_score: Please see above.
justification_for_why_not_lower_score: NA
Decision: Reject