Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a novel formulation of the triplet objective function by explicitly regularizing the distance between the positive and negative samples in a triplet. 
They evaluate their approach on (Fahsion) MNIST and EHR data. 


strength_and_weaknesses: While the approach is interesting, I have major concerns regarding the empirical analysis. 


clarity,_quality,_novelty_and_reproducibility: - How stable is the model? In table 5, it would be more meaningful to also re-train the embeddings. 
- It is unclear whether the very small improvements in a toy-like dataset such as MNIST are meaningful. It would be more insightful to see results on a slightly more complex dataset such as CIFAR-10 or CIFAR-100. 
- How does the proposed approach scale? 
- How does the proposed approach work for EHR data in an unsupervised setting (online generation of triplets via negative sampling)? I don't really see the practical relevance of the supervised setting here and importantly comparisons to unsupervised methods PCA and ICA are not meaningful.  
- PCA and ICA are very poor baselines as dimensionality reduction methods. It would be more meaningful to see results based on eg MDS, a VAE, kernel PCA/GP-LVM, diffusion maps,...
- In Fig 3, how was the UMAP for non-transformed data computed? Based on the PCA representation?
summary_of_the_review: Major flaws in empirical analysis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A variation of a triplet loss for deep metric learning, dubbed NPLB, is proposed. It is inspired by the condition that distance between positive and negative examples should be bigger than distance between anchor and positive. It is illustrated that optimizing such objective leads to more compact clusters. Empirical results on benchmark datasets (MNIST and fashionMNIST) and Biobank dataset show increased prediction accuracy (weighted F1 score), when using embeddings from NPLB.
strength_and_weaknesses: Strengths
Motivation for, and derivation of the new triplet objective is clear.
Empirical results for cluster density and separability, as well as embeddings utility in downstream prediction tasks looks compelling

Weaknesses
At moments it is not easy to follow the flow of the paper. Last paragraph of the Introduction appears to have redundancy in summarizing the main 3 aspects of the contribution: novel variation of triplet objective, embeddings’ utility for classification tasks, and distance based risk indicators. Claim that this approach does not require additional hyperparameters, as it is based on distances, might be misleading. In equation 3, the added “regularization term”, ie the square of distance, could have a hyperparameter multiplicator to tune the impact on overall loss. Just in this case it is set to 1 (similarly as it is set to 0.1 in MDR case).  

Minor suggestions
Equation 1 has an extra right bracket.

clarity,_quality,_novelty_and_reproducibility: The paper seems to contribute incrementally to the spectrum of triplet loss objectives. The empirical evaluation setting could have been explained a bit more clearly. For example, it is not clear if all the test subjects are covered by classes “Normal”, “Low Risk” and “High Risk” in section 5.3? Why is the dataset split based on the gender and presented separately for female (and male in appendix)? Also, why is DeepPatient transformation R^n -> R^n and not R^n -> R^d like others? Given that repository will be shared, the results seem reproducible.
summary_of_the_review: Given that modeling novelty seem to be adding distance based regularization to known objective, and novel application of using the learned metrics as health risk is not quite clear to me, I am leaning towards the rejection, although open to turn to the positive side if my concerns are addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors introduce a simple modification to a metric-learning triplet loss by adding penalty term which encourages the distance between the anchor sample and negative  sample to be similar to the distance between the positive sample and the negative sample. 


strength_and_weaknesses: Strengths
- The authors provide results for interesting applications of the method on healthcare data. 

Weaknesses

- The proposed idea is extremely simple. In essence, the submission uses the fact that a triplet contains two negative pairs: (a,n) and (p,n). In addition to the standard triplet loss, the authors add a loss that encourages the distance between the p and n pair (a negative pair) to be similar to the distance between the a and the n pair (another negative pair). Thus, the paper just adds an extra loss to the typical triplet loss that encourages all negative pairs in the dataset to be roughly equidistant. 

- Given the extreme simplicity of this idea, in my opinion the paper is lacking: (A) a thorough analysis how it compares to other recent triplet/contrastive losses, and in what cases equidistant negative pairs are desirable, (B) a thorough benchmarking comparing to more related work on triplet and contrastive losses*, and (C) an evaluation on a much larger set of common (metric learning) benchmark datasets used in related contrastive/triplet methods, rather than just MNIST and FashionMNIST. 

      *To just name a few methods for which I think comparisons are lacking: Lifted Structured Loss [1], Multi-Class N-pair loss [2], Noise Contrastive Estimation [3], InfoNCE [4], Circle Loss [5]

      [1] Oh Song, H., Xiang, Y., Jegelka, S., & Savarese, S. (2016). Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4004-4012). 
 
      [2] Sohn, K. (2016). Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29. 

      [3] Gutmann, M., & Hyvärinen, A. (2010, March). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics (pp. 297-304). JMLR Workshop and Conference Proceedings. 

      [4] Oord, A. V. D., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

      [5] Sun, Yifan, et al. "Circle loss: A unified perspective of pair similarity optimization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
     


- Finally, the motivation via the definition of uniform in-class embeddings is unclear to me, in part because the notation is not well defined at the moment. I encourage the authors to elaborate more on this point. Notation issue: Rojas-Thomas & Santos (2021)  define the local density (LD) over a point x_i in c_k. The authors in this submission define LD on c_k directly, without specifying if p_j is also in c_k or any data point in the dataset. 
	
	
clarity,_quality,_novelty_and_reproducibility: - The novelty of the method is very limited. Connections to other contrastive losses that make uses of more negative pairs are not explored by the authors. 

- The results on MNIST and FashionMNIST should be easy to reproduce. 
summary_of_the_review: I vote reject as I do not think that this work is ready for publication, given that the comparison to related triplet/contrastive methods on public benchmark datasets is severely limited. Adding more related methods and experiments should not present a significant hurdle. 


--------
## Post Rebuttal Update
I thank the authors for working to improve their submission. I am keeping my score since metric learning work published at top ML conferences should follow a more thorough benchmarking on diverse datasets and against recent competitive metric learning approaches. In addition, in its current form, the submission is still lacking convincing arguments as to why the proposed approach should work better than alternative metric learning approaches that also make use of additional relationships between positive and/or negative pairs. I encourage the authors to take this feedback into account to mature the submission.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission has been thoroughly reviewed by three knowledgeable reviewers. All of them assessed it below the threshold for acceptance. The reviewers brought up several concerns, including incrementality of the contribution to the field, and concerns with the scope and soundness of the empirical evaluation framework. This paper is unfortunately not ready for publication at ICLR at this point. 
justification_for_why_not_higher_score: Paper not ready for publication at ICLR due to limited novelty and too narrow scope of empirical evaluation.
justification_for_why_not_lower_score: n/a
Decision: Reject