The Minority Matters: A Diversity-Promoting Collaborative Metric Learning Algorithm

Abstract: Collaborative Metric Learning (CML) has recently emerged as a popular method in recommendation systems (RS), closing the gap between metric learning and Collaborative Filtering. Following the convention of RS, existing methods exploit unique user representation in their model design. This paper focuses on a challenging scenario where a user has multiple categories of interests. Under this setting, we argue that the unique user representation might induce preference bias, especially when the item category distribution is imbalanced. To address this issue, we propose a novel method called Diversity-Promoting Collaborative Metric Learning (DPCML), with the hope of considering the commonly ignored minority interest of the user. The key idea behind DPCML is to include a multiple set of representations for each user in the system. Based on this embedding paradigm, user preference toward an item is aggregated from different embeddings by taking the minimum item-user distance among the user embedding set. Furthermore, we observe that the diversity of the embeddings for the same user also plays an essential role in the model. To this end, we propose a diversity control regularization term to accommodate the multi-vector representation strategy better. Theoretically, we show that DPCML could generalize well to unseen test data by tackling the challenge of the annoying operation that comes from the minimum value. Experiments over a range of benchmark datasets speak to the efficacy of DPCML.

Reviews and author responses:

 I would like to thank the authors for their thoughtful reply and willingness to meaningfully engage with my questions and concerns. You have gone above and beyond in your response. I believe the additional results provided above significantly strengthen the empirical experiments and further validate the theoretical results presented in the paper. I will be raising my score accordingly and look forward to reading the final version of the paper.

 We sincerely express our gratitude for your constructive comments and make the following responses to your primary concerns: 

> **(Q1)**: Clarification of contribution/novelty.

**(A1)**: Thank you so much for your nice question! We will make the following clarification of contributions to our work: 
1. Although the concept of recommendation diversity has been studied in the RS community, we argue that there are few attentions to the diverse interests of users in the CML framework. Motivated by this, we start the first exploration and propose a simple but effective method to handle the diverse interests of users. The empirical results support the advantage of the CML paradigm.
2. In addition, we observe that simply leveraging multiple vectors for users is not necessarily effective. Specifically, there is a chance to find trivial solutions where the multiple vectors are almost identical. To avoid these trivial solutions, we propose a diversity-controlling regularization scheme to accommodate users' preferences better.
3. Most importantly, to our knowledge, we are the first to prove the effectiveness of multiple user representations **in theory**. Specifically, we show that DPCML with multiple embeddings could enjoy a smaller generalization error than single-vector CML. Specifically, we have the following results:

> [**Theorem 1**]. [Generalization Upper Bound of DPCML] Let $\mathbb{E}[\hat{\mathcal{L}}\_{\mathcal{D}}(\boldsymbol{g})]$ be the population risk of $\hat{\mathcal{L}}_{\mathcal{D}}(\boldsymbol{g})$. Then, $\forall \boldsymbol{g} \in \mathcal{H}_R$, with high probability, the following inequation holds: 
$$
\begin{aligned}
\left| \hat{\mathcal{L}}\_{\mathcal{D}}(\boldsymbol{g}) - \mathbb{E} [\hat{\mathcal{L}}\_{\mathcal{D}}(\boldsymbol{g})]\right| \le \sqrt{\frac{2d\log \left(3r \tilde{N}\right)}{\tilde{N}}},
\end{aligned}
$$
where we have 

$$
\tilde{N} = \left(4r^2\sqrt{\left(\frac{(4 + \eta)^2}{|\mathcal{U}|} + \frac{2}{|\mathcal{U}|^2} \sum_{u\_i \\ \in \\ \mathcal{U}} \left(\frac{1}{n\_i^+} + \frac{1}{n\_i^-}\right)\right)}\right)^{-2}.
$$
> Intriguingly, we see that our derived bound does not depend on $C$. This is consistent with the over-parameterization phenomenon. On top of Theorem 1, we have the following corollary.

> [**Corollary 1.**] DPCML could enjoy a smaller generalization error bound than CML.

> **(Q2)**: Intuition behind the proposed diversity-promoting regularization term $\psi_{\boldsymbol{g}}(u_i)$.

**(A2)**: Thanks for your interesting question! We are sorry that our unclear expression for this makes you confused. 
As you can see, $\psi_{\boldsymbol{g}}(u_i)$ includes two terms below, i.e., 

$$
\psi\_{\boldsymbol{g}}(u\_i) = \max(0, \delta\_1 - \delta\_{\boldsymbol{g}, u_i}) + \max(0, \delta\_{\boldsymbol{g}, u\_i} - \delta\_2),
$$
where 

$$
\delta\_{\boldsymbol{g}, u\_i} = \frac{1}{2C(C-1)} \sum_{\substack{c_1, c_2 \\ \in \\ C \\ c_1 \neq c_2}} \\|\boldsymbol{g}_{u_i}^{c_1}-\boldsymbol{g}\_{u_i}^{c\_2}\\|^2,
$$
and $C$ is the number of user embeddings.

We argue that one should minimize $\psi_{\boldsymbol{g}}(u_i)$ to get a good performance. Our point here is that extremely large/small values of $\delta_{\boldsymbol{g}, u_i}$ might be harmful for the generalization error. It is easy to see that if it is extremely small, then the embeddings for a given user are very close to each other, increasing the model complexity with few performance gains. This obviously will induce overfitting. On the other hand, a too large diversity might also induce overfitting. It might be a bit confusing at first glance. But, imagine that when some noise observations or extremely rare interests far away from the normal patterns exist in the data, having a large diversity will make it easier to overfit such data. Moreover, it is also a natural assumption that a user's interests should not be too different. Hence, we argue that the diversity measure should remain at a moderate magnitude. We have also validated this assumption in Fig.1 in our main paper. We hope the new explanation will help you understand our idea. Moreover, in reply to your question Q3-Q5, we present a more detailed performance comparison to validate this assumption.

 >**(Q3)**: Were other diversity-promoting regularizers explored?

**(A3)**: Thank you for your nice question! We are sorry that we do not explore other alternative regularizers directly in the original paper. But here, we attempt three variants of our proposed Diversity Control Regularization Scheme (DCRS): 
1.  $\textbf{w/o DCRS}$: This is a variant of our method where no regularization is adopted at all. 
2.   $\textbf{DCRS}-\delta_1$: This is a variant of our method where the punishment on a **large** diversity is **removed**. In other words, we will use the following regularization term:
$$
\psi_{\boldsymbol{g}}^1(u\_i) = \max(0, \delta_1 - \delta_{\boldsymbol{g}, u\_i}).
$$ 
3. $\textbf{DCRS}-\delta_2$: This is a variant of our method where the punishment on a **small** diversity is **removed**. In other words, we will use the following regularization term:
$$
\psi_{\boldsymbol{g}}^2(u\_i) = \max(0, \delta_{\boldsymbol{g}, u\_i} - \delta_2).
$$

Note that w.r.t. cases 2 and 3, $\psi_{\boldsymbol{g}}(u_i)$ in Eq.(7) is replaced with $\psi_{\boldsymbol{g}}^1(u_i)$ and $\psi_{\boldsymbol{g}}^2(u_i)$, respectively. 
The ablation studies are conducted on the Steam-200k dataset to show the effects of these variants. The empirical results are listed below:
> Ablation results of DCRS for DPCML1 are listed as follows:
| **Method**         | **P@3** | **R@3** | **NDCG@3** | **P@5** | **R@5** | **NDCG@5** | **MAP** | **MRR** |
|:------------------:|:-------:|:-------:|:----------:|:-------:|:-------:|:----------:|:-------:|:-------:|
| **w/o DCRS**       | 23.86   | 13.06   | 24.90      | 23.57   | 11.56   | 24.77      | 20.37   | 44.38   |
| **DCRS-$\delta\_1$** | 24.28   | 14.38   | 25.61      | 22.48   | 11.35   | 24.13      | 21.02   | 45.76   |
| **DCRS-$\delta\_2$** | 24.56   | 14.36   | 25.41      | 23.82   | 11.97   | 24.74      | 21.67   | 45.14   |
| **DCRS**           | 25.39   | 14.84   | 26.56      | 23.88   | 12.11   | 25.25      | 22.26   | 46.79   |

> Ablation results of DCRS for DPCML2 are listed as follows:
| **Method**        | **P@3** | **R@3** | **NDCG@3** | **P@5** | **R@5** | **NDCG@5** | **MAP** | **MRR** |
|:-----------------:|:-------:|:-------:|:----------:|:-------:|:-------:|:----------:|:-------:|:-------:|
| **w/o DCRS**       | 27.96   | 15.68   | 29.42      | 27.85   | 13.94   | 29.56      | 22.50   | 50.03   |
| **DCRS-$\delta\_1$** | 28.28   | 16.05   | 29.60      | 27.25   | 13.75   | 29.17      | 22.63   | 50.00   |
| **DCRS-$\delta\_2$** | 29.26   | 16.83   | 30.61      | 28.47   | 14.28   | 30.16      | 23.86   | 51.14   |
| **DCRS**           | 29.88   | 17.13   | 31.22      | 28.70   | 14.51   | 30.56      | 24.10   | 51.95   

From the above results, we can see that: in most cases, only employing one of the two terms of DCRS could still improve the recommendation performance. However, none of them could outperform our proposed method. This strengthens the effectiveness of our proposed regularization scheme.

 > **(Q4)**: Sensitivity analysis of $\eta$.

**(A4)**: Thanks for your helpful question! We investigate the sensitivity of $\eta \in \\{0,1, 3, 5, 10, 20, 30\\}$ for recommendation results on the Steam-200k dataset. The experimental results are listed below for DPCML1 and DPCML2, respectively. We can conclude that a proper $\eta$ (roughly $10$) could significantly improve the performance, suggesting the essential role of the proposed diversity control regularization scheme.

> The Sensitivity results of $\eta$ for DPCML1 are listed as follows:
| **$\eta$** | **P@3** | **R@3** | **NDCG@3** | **P@5** | **R@5** | **NDCG@5** | **MAP** | **MRR** |
|:----------:|:-------:|:-------:|:----------:|:-------:|:-------:|:----------:|:-------:|:-------:|
| **0**       | 23.86   | 13.06   | 24.90      | 23.57   | 11.56   | 24.77      | 20.37   | 44.38   |
| **1**      | 25.04   | 14.65   | 26.01      | 24.60   | 12.55   | 25.81      | 21.65   | 45.55   |
| **3**      | 24.67   | 14.43   | 25.50      | 23.88   | 12.25   | 24.96      | 21.56   | 44.73   |
| **5**      | 25.24   | 14.91   | 26.65      | 23.80   | 12.17   | 25.34      | 22.17   | 47.23   |
| **10**     | 25.39   | 14.84   | 26.56      | 23.88   | 12.11   | 25.25      | 22.26   | 46.79   |
| **20**     | 24.60   | 14.34   | 25.79      | 24.03   | 12.05   | 25.17      | 21.87   | 46.20   |
| **30**     | 25.23   | 14.69   | 26.19      | 24.25   | 12.08   | 25.58      | 21.94   | 46.00   |

> The Sensitivity results of $\eta$ for DPCML2 are listed as follows:
| **$\eta$** | **P@3** | **R@3** | **NDCG@3** | **P@5** | **R@5** | **NDCG@5** | **MAP** | **MRR** |
|:----------:|:-------:|:-------:|:----------:|:-------:|:-------:|:----------:|:-------:|:-------:|
| **0**       | 27.96   | 15.68   | 29.42      | 27.85   | 13.94   | 29.56      | 22.50   | 50.03   |
| **1**      | 28.55   | 16.35   | 29.92      | 27.82   | 13.94   | 29.65      | 22.90   | 50.57   |
| **3**      | 28.68   | 16.32   | 29.96      | 27.71   | 13.90   | 29.59      | 23.13   | 50.19   |
| **5**      | 29.34   | 16.82   | 30.45      | 27.98   | 13.95   | 29.75      | 23.42   | 50.62   |
| **10**     | 29.88   | 17.13   | 31.22      | 28.70   | 14.51   | 30.56      | 24.10   | 51.95   |
| **20**     | 29.81   | 17.12   | 31.08      | 29.11   | 14.65   | 30.77      | 24.35   | 51.90   |
| **30**     | 29.43   | 16.99   | 30.67      | 28.96   | 14.53   | 30.56      | 24.50   | 51.36   |

> **(Q5)**: The effectiveness of regularizer for MF-based systems.

**(A5)**: Thank you for your interesting question! To see this, we attempt to apply the proposed diversity control regularization scheme (DCRS) for M2F [2] method (Literature [3] provided by you will be cited/discussed in our paper in the final version). In addition, we further explore the effectiveness of DCRS for the general framework of joint accessibility (GFJA, Eq.(10) in the main paper). Here we also conduct a grid search to choose the best performance of M2F with DCRS on the Steam-200k and MovieLens-1m datasets, where the parameter space stays the same as DPCML. The experimental results are summarized in the following tables:
|**Steam-200k**   |       |       |        |       |       |        |       |       |
|:--------------:|:-------:|:-------:|:--------:|:-------:|:-------:|:--------:|:-------:|:-------:|
| **Method**       | P@3   | R@3   | NDCG@3 | P@5   | R@5   | NDCG@5 | MAP   | MRR   |
| **M2F**          | 11.33 | 5.69  | 11.95  | 11.44 | 5.73  | 12.98  | 6.43  | 25.05 |
| **M2F+DCRS**     | 10.92 | 5.58  | 11.49  | 10.89 | 5.48  | 12.37  | 6.25  | 24.26 |
| **GFJA**         | 21.53 | 12.60 | 22.52  | 20.37 | 10.16 | 21.49  | 19.32 | 40.69 |
| **GFJA+DCRS**    | 21.63 | 12.40 | 22.72  | 20.38 | 9.98  | 21.74  | 19.53 | 40.92 |
| **MovieLens-1m** |       |       |        |       |       |        |       |       |
| **M2F**          | 8.61  | 1.84  | 9.36   | 7.60  | 2.30  | 8.67   | 2.95  | 20.40 |
| **M2F+DCRS**     | 7.59  | 1.49  | 8.16   | 7.10  | 2.02  | 7.92   | 2.53  | 18.51 |
| **GFJA**         | 15.79 | 3.19  | 16.11  | 16.02 | 4.77  | 16.66  | 11.04 | 32.54 |
| **GFJA+DCRS**    | 16.71 | 3.54  | 16.94  | 17.24 | 5.27  | 17.71  | 11.75 | 33.87 |

From the above results, we can draw the following observations: 1) The proposed DCRS does not work well for MF-based models. A possible reason here is that the metric space of MF-based and CML-based methods are intrinsically different. MF adopts the inner-product space while CML adopts the Euclidean space. In this paper, we merely consider the DCRS for Euclidean space. The corresponding strategy for the inner-product space is left as future work. 2) In most metrics, GFJA+DCRS could outperform GFJA significantly, which supports the advantages of our proposed DCRS. 3) Compared with M2F, the performance gain of GFJA is sharp on both datasets. This suggests the superiority of our proposed method against the current multi-vector-based competitors.

 > **(Q6)** Does the proposed approach result in more diverse recommendations?

**(A6)**: Thanks for your constructive suggestion! Following your valuable comments, to show the improvement of DPCML in promoting diversity, we evaluate the performance of DPCML against CML-based competitors with a new metric called max-sum diversification (MaxDiv) from [4]:

$$
\textit{MaxDiv}@N = \frac{1}{|\mathcal{U}|}\sum\_{u \\ \\in \\ \mathcal{U}} \\ \sum\_{v\_i, v\_j \\ \\in \\ \mathcal{I}^N\_{u},\\ v\_i \neq v\_j} s(v\_i, v\_j),
$$
where $s(v\_i, v\_j)=\\|\boldsymbol{g}\_{v\_i} - \boldsymbol{g}\_{v\_j}\\|^2$ is the square of euclidean distance between item $v\_i$ and $v\_j$, and $\mathcal{I}^N\_{u}$ is the top-$N$ recommendation items for user $u$.

Generally speaking, MaxDiv@$N$ measures the recommendation diversification by considering item-side similarity, where a high value implies that the recommendation results are relatively diverse. Then, we compare DPCML with the following competitors for a fair evaluation: a) UniS. b) HarS. c) DPCML1 without (w/o) DCRS and d) DPCML2 without (w/o) DCRS. The experiments are conducted on the Steam-200k and MovieLens-1m datasets with $N \in \\{3, 5, 10, 20\\}$.
The diversity results are shown as follows:
| **Steam-200k**      |          |          |           |           |
|:-------------------:|:--------:|:--------:|:---------:|:---------:|
| **Method**          | MaxDiv@3 | MaxDiv@5 | MaxDiv@10 | MaxDiv@20 |
| **UniS**            | 1.354    | 4.750    | 23.520    | 117.927   |
| **HarS**            | 1.752    | 6.809    | 40.378    | 236.794   |
| **DPCML1 w/o DCRS** | 1.643    | 5.857    | 30.425    | 155.193   |
| **DPCML1**          | 1.822    | 6.713    | 34.727    | 179.065   |
| **DPCML2 w/o DCRS** | 2.958    | 11.398   | 65.398    | 365.458   |
| **DPCML2**          | 2.977    | 11.472   | 65.952    | 369.876   |
| **MovieLens-1m**    |          |          |           |           |
| **UniS**            | 1.739    | 6.142    | 30.127    | 140.095   |
| **HarS**            | 2.443    | 8.826    | 46.390    | 244.078   |
| **DPCML1 w/o DCRS** | 1.623    | 5.857    | 29.500    | 140.057   |
| **DPCML1**          | 1.744    | 6.195    | 30.755    | 145.615   |
| **DPCML2 w/o DCRS** | 2.827    | 10.423   | 55.612    | 292.089   |
| **DPCML2**          | 3.144    | 11.498   | 60.696    | 313.086   |

We observe that a) for methods within the same negative sampling strategy (i.e., UniS and DPCML1, HarS and DPCML2), our proposed DPCML could achieve relatively higher max-sum values. This suggests the improvement of DPCML in terms of promoting recommendation diversity. b) In most cases (except for DPCML1 w/o DCRS on the MovieLens-1m dataset), DPCML outperforms other competitors even without regularization. c) Most importantly, equipped with the regularization term DCRS, DPCML could achieve better diversification results against w/o DCRS. This once again shows the rationality/importance of DCRS. 

> **(Q7)**: About additional hyperparameters. 

**(A7)**: Thank you very much for your valuable comment! Since DPCML adopts multiple user representations, it will inevitably introduce a few additional parameters. Moreover, we notice that, ever since the paradigm of CML appeared in [1], the subsequent algorithms that improve CML are also included some extra hyperparameters in their work, such as the number of memory blocks (LRML[5], HLR[6]) and regularization coefficients $\lambda_{nbr}$, $\lambda_{dist}$ (TransCF[7]). Besides, introducing hyperparameters does not always work. Only a proper design of the method can lead to such improvements. To check the effectiveness of such hyperparameters, we have also conducted a series of experiments to answer your questions (Q3)-(Q6). Therefore, we believe this may not be a big problem because the number of parameters in DPCML is not particularly large. In addition, according to our experiments, we find that fixing the hyperparameter with $C=5$ and $\eta=10$  has obtained a good performance improvement in most datasets. This also shows that our method is not so sensitive to hyperparameter choices.

 > **(Q8)**: Computational resources of baselines. 

**(A8)**: Thanks for your nice question! Of course, we do. We are sorry for the unclear expressions in Sec.C.4 in the supplementary materials. To ensure a reasonable comparison for all baselines, we also conduct a grid search to determine their best performance. Specifically, in terms of the same parameters in CML-based competitors (such as batch size, learning rate, margin $\lambda$, and dimension $d$), we search them within the same space as DPCML. Then, we follow the parameter space provided in their original papers for the remaining different parameters. In addition, for all MF-based approaches, we also follow the same parameter space in the paper therein. For example, we search the predictive factors in MF within $[8, 16, 32, 64]$, etc. We refer the reviewer to referred studies for more experimental details (All literature has already been listed in Sec.C.3). 

> **(Q9)**: Some minor concerns. 

**(A9)**: Thank you very much for your helpful suggestions, and we are very sorry that some minor issues make some parts of our paper difficult to read. In the final version, we will carefully fix all typos, grammatical errors and improper size of figures, etc. Furthermore, we will invite native speakers to help improve our writing. Finally, we will adjust the order of Figures 6-9, and clear versions of them will be further attached to the supplementary material. 

> **(Q10)**: Limitations of DPCML.

**(A10)**: Thanks for your nice question! We are sorry that we do not discuss any limitations of DPCML. Following the paradigm of CML, a possible limitation of our proposed diversity-promoting algorithm is that it generally applies to implicit feedback but not explicit feedback since CML only cares about the relative preference ranking instead of concrete magnitude. In the future, we will explore how to improve the recommendation diversity based on explicit feedback (such as rating records).

> **Reference**: 
- [1] Collaborative metric learning. In WWW, 2017.

- [2] The Stereotyping Problem in Collaboratively Filtered Recommender Systems. In EAAMO, 2021. 

- [3] Nonlinear latent factorization by embedding multiple user interests. In RecSys, 2013.

- [4] Max-sum diversication, monotone submodular functions and dynamic updates. In SIGMOD/PODS, 2012. 
- [5] Latent relational metric learning via memory-based attention for collaborative ranking. In WWW, 2018.
- [6] Hierarchical latent relation modeling for collaborative metric learning. In RecSys, 2021.
- [7] Collaborative translational metric learning. In ICDM, 2018.



 We sincerely express our gratitude for your acceptance and constructive comments. The responses to your suggestions are listed below:

> **(Q1)** The writing of this paper is roughly good but could be further improved, including a few typos and mistakes in grammar.

**(A1)** Thank you for your nice suggestion! In the final version, we will fix all typos and grammar mistakes. Furthermore, we will invite native speakers to help cross-check the paper to improve our writing further. 

> **(Q2)** Each theorem and corollary appearing in the main paper should be attached to its corresponding proof link to make it easy for the reader to follow.

**(A2)** Thank you for your valuable comment! In the final version, we would carefully check all hyperlinks in the paper, such as the proof link and the figure/table link, to make the paper easy to follow. 

 We sincerely express our gratitude for your acceptance and valuable suggestions for our work. We would make the following responses to your concerns:
> **(Q1)**: Some parts of the paper should be reorganized to help readers understand this work better. 

**(A1)**: Thank you very much for your constructive comment! In the final version, we will reorganize Corollary.1 to make theoretical/empirical results more precise following your comments.

> **(Q2)**: The authors should complement a further explanation of the Joint Accessibility problem to make this concept concise and understandable.

**(A2)**: Thank you for your nice question! We are so sorry for the unclear explanations. The notion of joint accessibility is first proposed by [1]. It measures whether an item candidate with size $K$ could be jointly accessed by a user in a Top-$K$ recommendation. In other words, joint accessibility also somewhat captures a fundamental requirement of content diversity. If there are sufficient preference records of users, they should be able to be recommended any combination of $K$ items they may be interested in. More discussions of related work in this direction could be found in the paper (Sec.A.2 in the supplementary materials).

> **(Q3)**: There are some typos in this paper.

**(A3)** Thank you for your valuable comment! We will fix all typos and polish our writing in the final version. 

> **Reference**:
-  [1] The Stereotyping Problem in Collaboratively Filtered Recommender Systems. In EAAMO, 2021. 

 This paper focuses on Collaborative Metric Learning (CML) based recommendation systems (RS). The authors point out that the current CML-based studies might result in a limited recommendation performance due to the insufficient consideration of the multiple categories of user preferences in practical RS. To alleviate this problem, they propose a novel algorithm named Diversity-Promoting Collaborative Metric Learning (DPCML). The secret of success behind the DPCML is to design multiple representations for each user to focus on different interest groups (i.e., both majority and minority). Going a step further, the authors theoretically prove that the proposed DPCML outperforms the existing CML-based methods and could generalize well to unseen test data. Finally, experiments over a wide range of benchmark datasets demonstrate the efficacy of DPCML. Strengths:
1. The motivations and contributions of this paper are clear. 
2. The proposed method is novel. Concretely, this work presents a simple but effective method, DPCML, which leverages multiple user representations to capture user interests and adopt a diversity control regularization term to serve their purpose better.
3. Significantly, this paper is technically valuable. The authors further demonstrate the effectiveness of the proposed algorithm from a challenging theoretical perspective. In my opinion, the generalization analysis of CML-based algorithms is not an easy task, which is barely explored and could motivate more effective work in this direction. 
4. The experiment is persuasive.  They conduct comprehensive experiments on four benchmarks and compare DPCML with 14 state-of-the-art competitors. The empirical results show the superiority of DPCML.

Weaknesses:
1. Some parts of the paper should be reorganized to help readers understand this work better. For example, the proof of Corollary 1 should be mentioned in the main paper. Moreover, the empirical verification should be followed with Corollary 1 at the end of that part. 
2. There are some typos in this paper, such as:
a) Row 186: “when the training is completed…” Here “completed” should be “complete”.
b) Row 274: “…the performance of the validation/test set…” Here “of” should be “on”.
c) Row 716 (in Appendix): “NCF” should be NeuMF.
 1. Some parts of the paper should be reorganized to help readers understand this work better.
2. The authors should complement a further explanation of the Joint Accessibility problem to make this concept concise and understandable.
More questions should be found in the above weakness part.
 None.

 This paper presents a technique for item recommendation which augments a standard collaborative metric learning (CML) with multiple user representations. An additional loss term is introduced in order to encourage diversity among representations for the same user. The proposed approach is compared to alternative matrix factorization (MF) and CML based techniques on common benchmark datasets. Additionally, theoretical justification for the proposed approach in the form of a generalization bound is presented. ### Strengths
Item recommendation is an important problem, and the experiments demonstrate that the proposed approach performs well in the settings considered.

The work here suggests that it may be worth revisiting ideas previously developed in the context of MF based techniques in the CMF framework.

### Weaknesses
The idea of using multiple user representations is not new and goes back to at least [1], which is identical to the approach presented in [2] (only the later is cited in this work). Showing that something that works in the MF setting also works in the CML setting is a limited contribution.

The diversity-promoting regularizer is not sufficiently explored. The paper does not adequately motivate the desire to prevent representations for the same user from being too different, the second term of the unnumbered equation between (7) and (8). I find this counterintuitive and would have appreciated more commentary on this, and further exploration of the regularization strategy in general. Does including this regularizer in an MF setting like that explored in [1, 2] improve those results? Were alternative regularizers considered? How sensitive are the results to $\eta$?

The extent to which the propose system actually promotes diversity (the motivation for the approach) is not explored outside the result in Figure 5. Providing more more evidence that the proposed approach actually results in more diverse recommendations (for example, using well know measure of diversity [3]) would significantly strengthen the paper.

The paper introduces several additional hyperparameters ($C$, $\eta$, $\delta_1$, $\delta_2$), and the results here seem highly dependent on their exact choice (Figures 6, 8, 9). This is a significant drawback and I wonder if similar computational budgets were allocated to tuning baselines?

I found the paper somehow difficult to read and poorly organized. There are numerous grammatical, incomplete sentences, and unreadable figures. A couple I explicitly noted are listed below.

- Line 22: “giving them the relevant recommendations” => “giving them relevant recommendations”
- Line 268-269: The sentence “Moreover, there would induce different performances with different diversity values.” is missing a noun.
- Figure 6 and 7 are impossible to read without zooming to around 400%.
- It would help if the numerical order of Figures 6-9 should match the order they are discussed in the subsequent paragraphs.

### References
1. Jason Weston, Ron J Weiss, and Hector Yee. Nonlinear latent factorization by embedding multiple user interests. In ACM Conference on Recommender systems, 2013.
2. Guo, Wenshuo, et al. The Stereotyping Problem in Collaboratively Filtered Recommender Systems. Equity and Access in Algorithms, Mechanisms, and Optimization. 2021.
3. Borodin, Allan, Hyun Chul Lee, and Yuli Ye. Max-sum diversification, monotone submodular functions and dynamic updates. ACM SIGMOD-SIGACT-SIGAI symposium on Principles of Database Systems. 2012.

### Update Post Rebuttal
In their reply, the authors have provided thorough answers to my questions and further experimental results which address my concerns. I have raised my score accordingly. - What is the intuition behind wanting to ensure representations for the same user aren't too dissimilar in $\psi_g(u_i)$?
- Were other diversity-promoting regularizers explored?
- Does the diversity-promoting regularizer improve results for MF based systems like those in [1,2]?
- Does the proposed approach actually result in more diverse recommendations?
- Were similar computational resources allocated to tuning baseline hyperparameters?

 The paper does not discuss any limitations of the proposed approach. Providing some discussion of failure modes would strengthen the paper.

 This work attacks the collaborative metric learning (CML) based recommendation algorithms. The critical point is that the current studies might overlook the minority taste of users in practice due to the limited representation capability, leading to inevitable preference degradation. Based on this recognition, this paper presents a simple and effective approach (i.e., Diversity-Promoting CML, DPCML) to alleviate this preference bias. Then, a decent theoretical guarantee is derived, suggesting the superiority of DPCML against CML in theory. Finally, empirical studies ascertain the effectiveness of the proposed method. Strength:
1.	Clear motivation and good novelty.  
2.	Sound/important theoretical analysis. The provided theoretical results present novel insights into the strengths of DPCML against the current CML-based recommendation approaches. This is challenging to derive the generalization bound and may arise much more research attention in the community. 
3.	Convincing validations. The authors conduct comprehensive experiments to show the effectiveness of DPCML against the current CML-based algorithms, and the experiment results seem promising.
4.	The presentation of this paper is good and easy to understand.

Weakness:
The writing of this paper is roughly good but could be further improved. For example, there are a few typos and mistakes in grammar:
1.	Row 236 in Page 4, “…show its superiority.”: I think this sentence should be polished.  
2.	Row 495 in Supp. Page 15: “Hard” should be “hard”.
3.	Row 757 in Supp. Page 29: “…training/validation/test” should be “…training/validation/test sets”. 
4.	Row 821 in Supp. Page 31: “Fig.7” should be “Fig.12”.
Last but not least, each theorem and corollary appearing in the main paper should be attached to its corresponding proof link to make it easy for the reader to follow.

The primary concerns are motivation, methodology soundness, and experiment persuasion. I believe this is a qualified paper with good novelty, clear theoretical guarantees, and convincing empirical results. 

 The authors should correct my minor concerns listed in the weakness to make their work more complete.  There are no potential societal impacts of this paper.

Meta-review: After the author response, all three reviewers are in favor of accepting the paper. The proposed collaborative metric learning approach was appreciated in terms of novelty by two of the reviewers, whereas one considered the contribution limited. The theoretical analysis was appreciated and the experiment was considered convincing. Overall, this was considered a strong paper.

Reviewers seem to have some remaining concerns: 
- improving clarity be reorganization and some typo etc. fixes 
- some additional explanation, including clarifying the role of the diversity-promoting regularizer and discussion of variant regularizers
- fairness of hyperparameter tuning versus baselines
- additional discussion of results in terms of promoting diversity
- discussion of limitations
In their responses, authors have clarified some of the issues, provided several additional experiment results including results on parameter sensitivity and potential application to the M2F method, and agreed to carry out changes improving clarity. The thorough responses were appreciated. The additional information in the author responses should be taken into account in the final paper version to the extent possible.


Decision: Accept