Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper proposes a new contrastive learning objective for learning representations called Contrasim. Contrasim aims to learn representations that maximize inner-products between "should-be-similar" inputs and minimize them between "should-be-dissimilar" inputs. Contrasim's claim to novelty is that it adds the additional minimization objective for the "should-be-dissimilar" representation pairs whereas other representation learning objectives like project-weighted CCA (PWCCA) and centered kernel alignment (CKA) do not.

The authors test Contrasim on 3 representation learning tasks:
1) Determining which encodings of sentences from two different languages are translations of each other.
2) Determining which sentence and image encodings are (caption,image) pairs.
3) For neural networks with the same architecture and training data, but different initialization, determine which layers of the neural networks correspond.

Experimental results show that Contrasim dramatically outperforms the chosen baselines: CKA, PWCCA, deep dot-product and deep CKA; and the performance gap increases when using hard negative mining with the FAISS library.

Last, low dimensional projection plots show that Contrasim improves the proximity of should-be-similar inputs relative to the original representations and relative to the aforementioned baselines.
strength_and_weaknesses: 
The strength of this paper is that it emphasizes the importance of negative examples in training similarity functions and shows that they are critical in contrastic learning. I also liked that the authors experimented on embeddings produced by SOTA models like RoBerta and ViT.

However, the weakness of the paper is that using negative pairs is very common in contrastive learning, despite the paper's claiming the opposite. For instance, in SimCLR (which this paper cites), the objective maximizes the dot products of should-be positive pairs and minimizes the dot products of negative pairs (see equation (1) in the SimCLR paper).

Likewise, well known contrastive learning criteria like noise contrastive estimation (NCE), InfoNCE and triplet loss seek to minimize dot products of negative pairs relative to positive pairs.

These contrastive learning objectives need to be compared against because they are all highly similar to the proposed Contrasim.

In particular, equation (3) Contrasim can be seen as maximizing the log-odds probability of positive versus negative pairs if we assume there is a normalizing factor "Z" which has been divided out from both the numerator and denominator:

  eq (3) = \sum_i log [ \sum_p\~i [ exp(ai . ap)/Z ] / [ \sum_n!\~i exp(ai . an)/Z ].

where "\~" denotes the similarity relation and "!\~" denotes the not-similar relation.

SimCLR can be seen as maximizing the very similar odds ratio:

  \sum_i \sum_p\~i log { [ exp(ai . ap)/Z ] / [ \sum_n!\~i exp(ai . an)/Z ] },

so it is not clear if Contrasim presents a notable advance, especially in absence of empirical comparisons and/or theoretical justifications.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to read. And it should be easily reproducible once the authors' code is made public, as they have promised to make it upon publication.

As described above, the lack of novelty is the major shortcoming of this work. And the missing empirical evaluations against highly similar contrastive learning methods critically impact the quality of the work as well.

summary_of_the_review: While Contrasim is a moderately intriguing variant of existing contrastive learning losses, it is too similar to existing methods like SimCLR and NCE and not well enough distinguished from them to warrant acceptance at ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a similarity metric learned from similar and dissimilar examples using a loss inspired by contrastive learning. The metric is tested on three benchmarks (i.e., layer prediction, multilingual language modeling, image captioning), among which are two new benchmarks proposed by the authors on which they test the metric. 

strength_and_weaknesses: Strengths:
- Evaluation of the created metric on different types of data. 

Weaknesses:
- Clarity of the work and its goals. 
- Limited technical contribution.
- Evaluation only on three benchmarks, two of which are created. 
- Comparison with existing similarity metrics is limited.

clarity,_quality,_novelty_and_reproducibility: The clarity and the writing could be improved. Some spelling errors should be corrected. 
The technical novelty is limited as the proposed model makes use of existing methods of contrastive learning. 
Reproducibility is constrained by the use of retrieved examples, but when train data are provided by the authors, this problem is mitigated.
summary_of_the_review: 
When defining a new similarity metric, one should first define its theoretical properties and evaluate these properties empirically with data. These important parts are lacking in the paper. 

The motivation of the work is not clear. Is it the purpose to learn an encoder based on a contrastive loss or a similarity metric that can be used in and transferred to all kinds of applications?

What would be the effect if another similarity metric than cosine similarity would be used to compare representations during training?

The novelty of the work seems limited. Learning representations from positive and negative examples based on a contrastive loss and using these to create a semantic space where similarity is measured has been done for almost a decade. The number of baselines and comparisons with other similarity metrics seem limited. 

The authors rely on the FAISS library for similarity search of train examples. What is the influence of this initial search and the underlying methods that FAISS uses on the quality of the examples used in the training and consequently on the quality of the learned metric?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a parameterized similarity measure that shows significant improvement over existing closed-form similarity measures. 
strength_and_weaknesses: Strength:
  - The paper is well-written and easy to follow. 
  - The method is simple yet effective.
  - The empirical analysis is comprehensive. 
  - The provided benchmark can benefit future research.
  - This paper provides a novel perspective in providing similarity measures compared to the existing closed-form ones.

Weakness:
  - As a learnable similarity measure, due to the randomness of model training, this metric cannot guarantee consistent results, which may limit its use cases and reliability to some extent.
clarity,_quality,_novelty_and_reproducibility: Clarity: Good.

Quality: Good.

Novelty: Good.

Reproducibility: Fully reproducable.
summary_of_the_review: This paper provides a novel similarity measure based on contrastive learning, which greatly outperform existing closed-form measures in different scenarios.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a learned measure of representation similarity, ContraSim, as opposed to analytical ones like centered kernel alignment (CKA) and canonical correlation analysis (CCA). ContraSim is specific to a task (training distribution), and uses a simple MLP to project the representation to a different subspace. Experiments show that contrastive loss is a key component to achieving good results. 

Paper also introduces two additional benchmarks based on multi-lingual similarity and image-caption similarity.  ContraSim performs well on these tasks compared to baselines.
strength_and_weaknesses: Strengths:
- Paper is well-written, easy to follow. 
- Introduction of two benchmarks adds a different perspective to this research area compared to literature
- Proposed method outperforms baselines by a significant margin

Weaknesses:
- The analyses in the paper can dive deeper into the problem space. How much does ContraSim generalize? Can you use the same  ContraSim model across all benchmarks instead of training a separate one for each?
- It would be good to discuss the trade-offs introduced by ContraSim. E.g. we need to access to labeled training data. We need different models for each domain. 
- The paper does not discuss the implications of this research. Why is learning based measure useful? What did we learn that's new? 
clarity,_quality,_novelty_and_reproducibility: The paper is excellent in clarity, and reproducibility. 

The novelty is above par, but a simple extension of existing ideas.
The quality can be improved with deeper analysis, as mentioned in weaknesses.
summary_of_the_review: I think the paper is above the acceptance bar for ICLR. I would like to see some additional analyses and discussion in the final version of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an empirically successful similarity measure called ContraSim. It takes fairly large positive and negative sets as input, trains a separate encoder using contrastive loss, and then evaluate the final similarity as a dot product of these encoded representations. This paper only learns the similarity function without modifying the underlying representations themselves, but instead the parameters are in the trainable encoder. 

There was some confusion among the reviewers that this work is learning representations, which it is not. The authors addressed this adequately and some reviewers agreed. I suspect a source of this confusion is that in eq 3), a_i was used instead of z_i, which makes it seem like the original representation a_i was being learned. The authors were not very clear on the benefits of learning this separate encoder z=e(a), though since e is just a 2 layer NN and it seems easy to imagine some benefits over relearning the whole representation for each layer.  It is also perhaps not surprising that additional negative examples can be helpful for these tasks, but still showing this is a clear contribution.

The paper is clear in that it is a trainable similarity measure, and it seems to do well on the advertised goal by outperforming baselines by a clear margin. 2 reviewers liked the paper and many good comments (confusing points about FAISS and SimCLR, further impact) were raised by the negative reviewers.  The main quantitative results are on artificial layer prediction task or pick-a-neighbor binary contrast task, it is unclear what are the impacts of these clear improvements. The paper can be improved by addressing these confusion points and perhaps expanded to include results on downstream tasks.
justification_for_why_not_higher_score: lack of strong support in the reviews, the limited overall scope of learning a similarity function
Decision: Reject