Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on Collaborative Metric Learning (CML) based recommendation systems (RS). The authors point out that the current CML-based studies might result in a limited recommendation performance due to the insufficient consideration of the multiple categories of user preferences in practical RS. To alleviate this problem, they propose a novel algorithm named Diversity-Promoting Collaborative Metric Learning (DPCML). The secret of success behind the DPCML is to design multiple representations for each user to focus on different interest groups (i.e., both majority and minority). Going a step further, the authors theoretically prove that the proposed DPCML outperforms the existing CML-based methods and could generalize well to unseen test data. Finally, experiments over a wide range of benchmark datasets demonstrate the efficacy of DPCML.
strengths_and_weaknesses: Strengths:
1. The motivations and contributions of this paper are clear. 
2. The proposed method is novel. Concretely, this work presents a simple but effective method, DPCML, which leverages multiple user representations to capture user interests and adopt a diversity control regularization term to serve their purpose better.
3. Significantly, this paper is technically valuable. The authors further demonstrate the effectiveness of the proposed algorithm from a challenging theoretical perspective. In my opinion, the generalization analysis of CML-based algorithms is not an easy task, which is barely explored and could motivate more effective work in this direction. 
4. The experiment is persuasive.  They conduct comprehensive experiments on four benchmarks and compare DPCML with 14 state-of-the-art competitors. The empirical results show the superiority of DPCML.

Weaknesses:
1. Some parts of the paper should be reorganized to help readers understand this work better. For example, the proof of Corollary 1 should be mentioned in the main paper. Moreover, the empirical verification should be followed with Corollary 1 at the end of that part. 
2. There are some typos in this paper, such as:
a) Row 186: “when the training is completed…” Here “completed” should be “complete”.
b) Row 274: “…the performance of the validation/test set…” Here “of” should be “on”.
c) Row 716 (in Appendix): “NCF” should be NeuMF.

questions: 1. Some parts of the paper should be reorganized to help readers understand this work better.
2. The authors should complement a further explanation of the Joint Accessibility problem to make this concept concise and understandable.
More questions should be found in the above weakness part.

limitations: None.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a technique for item recommendation which augments a standard collaborative metric learning (CML) with multiple user representations. An additional loss term is introduced in order to encourage diversity among representations for the same user. The proposed approach is compared to alternative matrix factorization (MF) and CML based techniques on common benchmark datasets. Additionally, theoretical justification for the proposed approach in the form of a generalization bound is presented.
strengths_and_weaknesses: ### Strengths
Item recommendation is an important problem, and the experiments demonstrate that the proposed approach performs well in the settings considered.

The work here suggests that it may be worth revisiting ideas previously developed in the context of MF based techniques in the CMF framework.

### Weaknesses
The idea of using multiple user representations is not new and goes back to at least [1], which is identical to the approach presented in [2] (only the later is cited in this work). Showing that something that works in the MF setting also works in the CML setting is a limited contribution.

The diversity-promoting regularizer is not sufficiently explored. The paper does not adequately motivate the desire to prevent representations for the same user from being too different, the second term of the unnumbered equation between (7) and (8). I find this counterintuitive and would have appreciated more commentary on this, and further exploration of the regularization strategy in general. Does including this regularizer in an MF setting like that explored in [1, 2] improve those results? Were alternative regularizers considered? How sensitive are the results to $\eta$?

The extent to which the propose system actually promotes diversity (the motivation for the approach) is not explored outside the result in Figure 5. Providing more more evidence that the proposed approach actually results in more diverse recommendations (for example, using well know measure of diversity [3]) would significantly strengthen the paper.

The paper introduces several additional hyperparameters ($C$, $\eta$, $\delta_1$, $\delta_2$), and the results here seem highly dependent on their exact choice (Figures 6, 8, 9). This is a significant drawback and I wonder if similar computational budgets were allocated to tuning baselines?

I found the paper somehow difficult to read and poorly organized. There are numerous grammatical, incomplete sentences, and unreadable figures. A couple I explicitly noted are listed below.

- Line 22: “giving them the relevant recommendations” => “giving them relevant recommendations”
- Line 268-269: The sentence “Moreover, there would induce different performances with different diversity values.” is missing a noun.
- Figure 6 and 7 are impossible to read without zooming to around 400%.
- It would help if the numerical order of Figures 6-9 should match the order they are discussed in the subsequent paragraphs.

### References
1. Jason Weston, Ron J Weiss, and Hector Yee. Nonlinear latent factorization by embedding multiple user interests. In ACM Conference on Recommender systems, 2013.
2. Guo, Wenshuo, et al. The Stereotyping Problem in Collaboratively Filtered Recommender Systems. Equity and Access in Algorithms, Mechanisms, and Optimization. 2021.
3. Borodin, Allan, Hyun Chul Lee, and Yuli Ye. Max-sum diversification, monotone submodular functions and dynamic updates. ACM SIGMOD-SIGACT-SIGAI symposium on Principles of Database Systems. 2012.

### Update Post Rebuttal
In their reply, the authors have provided thorough answers to my questions and further experimental results which address my concerns. I have raised my score accordingly.
questions: - What is the intuition behind wanting to ensure representations for the same user aren't too dissimilar in $\psi_g(u_i)$?
- Were other diversity-promoting regularizers explored?
- Does the diversity-promoting regularizer improve results for MF based systems like those in [1,2]?
- Does the proposed approach actually result in more diverse recommendations?
- Were similar computational resources allocated to tuning baseline hyperparameters?


limitations: The paper does not discuss any limitations of the proposed approach. Providing some discussion of failure modes would strengthen the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work attacks the collaborative metric learning (CML) based recommendation algorithms. The critical point is that the current studies might overlook the minority taste of users in practice due to the limited representation capability, leading to inevitable preference degradation. Based on this recognition, this paper presents a simple and effective approach (i.e., Diversity-Promoting CML, DPCML) to alleviate this preference bias. Then, a decent theoretical guarantee is derived, suggesting the superiority of DPCML against CML in theory. Finally, empirical studies ascertain the effectiveness of the proposed method.
strengths_and_weaknesses: Strength:
1.	Clear motivation and good novelty.  
2.	Sound/important theoretical analysis. The provided theoretical results present novel insights into the strengths of DPCML against the current CML-based recommendation approaches. This is challenging to derive the generalization bound and may arise much more research attention in the community. 
3.	Convincing validations. The authors conduct comprehensive experiments to show the effectiveness of DPCML against the current CML-based algorithms, and the experiment results seem promising.
4.	The presentation of this paper is good and easy to understand.

Weakness:
The writing of this paper is roughly good but could be further improved. For example, there are a few typos and mistakes in grammar:
1.	Row 236 in Page 4, “…show its superiority.”: I think this sentence should be polished.  
2.	Row 495 in Supp. Page 15: “Hard” should be “hard”.
3.	Row 757 in Supp. Page 29: “…training/validation/test” should be “…training/validation/test sets”. 
4.	Row 821 in Supp. Page 31: “Fig.7” should be “Fig.12”.
Last but not least, each theorem and corollary appearing in the main paper should be attached to its corresponding proof link to make it easy for the reader to follow.

The primary concerns are motivation, methodology soundness, and experiment persuasion. I believe this is a qualified paper with good novelty, clear theoretical guarantees, and convincing empirical results. 


questions: The authors should correct my minor concerns listed in the weakness to make their work more complete. 
limitations: There are no potential societal impacts of this paper.
review_title: null

Meta-review: metareview: After the author response, all three reviewers are in favor of accepting the paper. The proposed collaborative metric learning approach was appreciated in terms of novelty by two of the reviewers, whereas one considered the contribution limited. The theoretical analysis was appreciated and the experiment was considered convincing. Overall, this was considered a strong paper.

Reviewers seem to have some remaining concerns: 
- improving clarity be reorganization and some typo etc. fixes 
- some additional explanation, including clarifying the role of the diversity-promoting regularizer and discussion of variant regularizers
- fairness of hyperparameter tuning versus baselines
- additional discussion of results in terms of promoting diversity
- discussion of limitations
In their responses, authors have clarified some of the issues, provided several additional experiment results including results on parameter sensitivity and potential application to the M2F method, and agreed to carry out changes improving clarity. The thorough responses were appreciated. The additional information in the author responses should be taken into account in the final paper version to the extent possible.

confidence: Certain
Decision: Accept