ContraSim -- A Similarity Measure Based on Contrastive Learning
Authors: Adir Rahamim
Yonatan Belinkov
Venue: Submitted to ICLR 2023 (2023)
Keywords: Interpretability
similarity measure
analysis
language models
multilingual
image captioning

Abstract: Recent work has compared neural network representations via similarity-based analyses, shedding light on how different aspects (architecture, training data, etc.) affect models' internal representations. The quality of a similarity measure is typically evaluated by its success in assigning a high score to representations that are expected to be matched.  However, existing similarity measures  perform mediocrely on standard benchmarks. In this work, we develop a new similarity measure, dubbed ContraSim, based on contrastive learning. In contrast to common closed-form similarity measures, ContraSim learns a parameterized measure by using both similar and dissimilar examples. We perform an extensive experimental evaluation of our method, with both language and vision models, on the standard layer prediction benchmark and two new benchmarks that we develop: the multilingual benchmark and the image--caption benchmark. In all cases, ContraSim achieves much higher accuracy than previous similarity measures, even when presented with challenging examples. 

Under review as a conference paper at ICLR 2023
CONTRASIM A SIMILARITY MEASURE BASED ON CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has compared neural network representations via similarity-based analyses, shedding light on how different aspects (architecture, training data, etc.) affect models' internal representations. The quality of a similarity measure is typically evaluated by its success in assigning high score to representations that are expected to be matched. However, existing similarity measures perform mediocrely on standard benchmarks. In this work, develop new similarity measure, dubbed ContraSim, based on contrastive learning. In contrast to com- mon closed-form similarity measures, ContraSim learns parameterized measure by using both similar and dissimilar examples. We perform an extensive experi- mental evaluation of our method, with both language and vision models, on the standard layer prediction benchmark and two new benchmarks that we develop: the multilingual benchmark and the image-caption benchmark. In cases, Con- traSim achieves much higher accuracy than previous similarity measures, even when presented with challenging examples.
INTRODUCTION
Representation learning is key property in deep neural networks. But how can we assess the simi- larity of representations learned by two models? A recent line of work is concerned with developing similarity measures and using them to analyze the models' internal representations. Similarity- based analyses may shed light on how different datasets, architectures, etc., change the model's 's learned representations. For example, similarity analysis showed that lower layers different models are more similar to each other, while fine-tuning affects mostly the top layers (Wu et al.) 2020).
Various similarity measures have been proposed for comparing representations, among them the most popular ones are based on centered kernel alignment (CKA) (Kornblith et al.] (2099) and canon- ical correlation analysis (CCA) (Hotelling 1936; Morcos et al. 2018). They all share a similar methodology: given pair of feature representations of the same imput, they estimate the similar- ity between them, without considering other examples. However, they all perform mediocrely on standard benchmarks. Motivated by that, we propose new learnable similarity measure.
In this paper, we introduce ContraSim, a new similarity measure, based on contrastive learning (CL) (Chen al. 2020 He et al. 2020). In contrast prior work, which defines closed-form general- purpose similarity measures, ContraSim is a task-specific learnable similarity measure that uses examples high similarity (the positive set) and examples that have low similarity (the negative set), to train an encoder that maps representations to the space where similarity is measured. In the projected space, we maximize the representation similarity with examples from the positive set, and minimize it with examples from the negative set.
We experimentally evaluate ContraSim on one standard similarity metries benchmark and two new benchmarks we introduce in this paper, and demonstrate its superiority compared to common simi- larity measures. First, we use the known layer prediction benchmark (Kornblith et al., 2209)), which assesses whether high similarity is assigned to two els differing only in their weight initialization. Second, in our proposed multilingual benchmark, we assume multilingual model and parallel dataset of translations in two languages. A good simi- larity measure should assign a higher similarity to the in language A and its translation in language B, compared to the similarity of the same sentence in language A and a random sentence in language B. Third, we design the image-caption benchmark, based similar idea. Given image and its text caption, and correspondingly vision model
layers in two mod-
representations of a sentence

---

Under review as a conference paper at ICLR 2023
and language model, good similarity measure should assign a high similarity to representations of the image and its caption, compared to the similarity of the same image and a random caption.
In both of our new benchmarks, investigate more challenging scenario, where instead of choos- ing a random sentence, we retrieve highly similar sentences as confusing examples, using the Face- book AI Similarity Search (FAISS) library (Johnson al., 2019). While other similarity measures are highly affected by this change, our method maintains a high accuracy with a very small degra- dation. We attribute this the highly separable representations that our method learns.
Finally, in all benchmarks, we show that if we change the training procedure of the encoder to only maximize the similarity of similar examples, the projected representations have poor separation, indicating that the CL procedure is a crucial part of the method's 's success.
In summary, this work makes the following contributions:
We introduce a new similarity measure - ContraSim. Inspired by contrastive learning, it uses positive and negative sets to train an encoder that maps representations to the space where similarity is measured.
We propose two new benchmarks for the evaluation of similarity measures: the multilingual benchmark and the image-caption benchmark.
We show that ContraSim outperforms existing similarity measures in all benchmarks, and maintains a high accuracy even when faced with more challenging examples.
RELATED WORK
Comparing different models allows one to analyze how different aspects like network architecture, training set, and model size affect the model's s learned representations. For instance, Kornblith et al. (2019) showed that adding too many layers to convolutional neural network, trained for image classification, hurts its performance. Using CKA, they found that more than half of the network's layers are very similar to the last. They further found that two models trained on different image datasets (CIFAR- 10 and CIFAR- 100. Krizhevsky et al. 2009) learn representations that are similar in the shallow layers. Similar findings were noted for language models by Wu et al. (2020). The latter also evaluated the effect of fine-tuning language models, and found that the top layers most affected by fine-tuning.
Investigating the effect of layer width, Kornblith et al. (2019) and Morcos al. 2018 found that increasing the model's layer width results in more similar representations between models, and that networks are generally more similar to networks with the same layer width than networks with a relatively larger width. RRahu et al.] (2017) provided an interpretation of the learning process by comparing the similarity of representations at some layer during the training process compared to the final representations. They found that networks converge from bottom to top, i.e., layers closer to the input converge their final representation faster than deeper layers. Based on that insight, they proposed frozen training, where they successively freeze lower layers during training, updating only the deeper layers. They found that frozen training leads to classifiers with higher generalization.
Cianfarani et al. (2022) used similarity measures to analyze the effect of adversarial training on deep neural networks trained for image classification. Using CKA, they compared representations of adversarially trained neural networks with representations of regularly trained and discovered that adversarial examples have little effect on early layers. They further found that deeper layers overfit during adversarial training. Moreover, they found high similarity between representations of adversarial images generated with different threat model.
All prior work computes similarity only between examples that are similar, using functional closed- form measures. In contrast, we utilize both positive and negative samples in learnable similarity measure, which allows adaptation to specific tasks.
PROBLEM SETUP
Let X (( 2 denote set of N examples, and A {(a ( the set of repre- sentations generated for the examples in X. A representation is high- -igee-innnsional vector of neuron activations. Representations may be created by the same or different models, by different layers of
2

---

Under review as a conference paper at ICLR 2023
the same model, etc. For instance, 21 and ziz may be the same input, with aj and a² represen- lations of that input in different layers. Alternatively, 41 can be an image and 22 its caption, with aj
(4)
(t)
(4) and ai (8) their representations from vision model and a language model, respectively.
Our goal is to obtain a scalar similarity score, which represents the similarity between the two sets (6) and ranges from 0 (no similarity) to 1 (identical representations). That is, we define X Xpx as a matrix of pl activations of N data points, and X2 E IR xpa as another matrix of P2 activations of data points. We seek a similarity measure, s (X1, X2).
()) and a2
of representations, aj
4 CONTRASIM
In this section introduce ContraSim, similarity index for measuring the similarity of neural network representations. Our method uses a trainable encoder, which first maps representations to a new space and then measures the similarity of the projected representations. Formally, let ey denote an encoder with trainable parameters 0, and assume two representations d₁ and ag In order to obtain similarity score between 0 and 1, we first apply L.2 normalization to the encoder outputs: ž = es(a))/ heel//ee((a))) (a)) || (and similarly for a2). Then their similarity is calculated as:
s(11, x2)
where s is simple closed-form similarity measure for two vectors. Throughout this work use dot product for 8.
For efficiency reasons, calculate the similarity between batches of the normalized encoder rep- resentations, dividing by the batch size n:
n
2/2)
7
Training. None of the current similarity measures uses negative examples estimate the sim- ilarity of a given pair. Using two examples, their output is usually a scalar that represents the similarity between them, without leveraging data from other examples. However, based on knowl- edge from other examples, we can construct better similarity index. In particular, for given example (+( E X with its representation a; we construct a set of positive examples indices, P(i) (PI,...,,Pyl and a set of negative examples indices, N() {ni,..,, .... nt). The choice of these sets is task-specific and allows one to add inductive bias to the training process.
We train the encoder to maximize the similarity of a; with all the positive examples, while the same time making it dis-similar from the negative examples. We leverage ideas from contrastive learning (Chen al. 2020 He et al. 2020), and minimize the following objective:
Eperio exp(a; ap/T)
C
log
(3)
P())
EnEN(i) exp(a; an/r)
NE1
with scalar temperature parameter T > 0. Here ap and a, are representations from the positive and negative groups, respectively.
Our work uses negative examples and a trainable encoder for constructing similarity measure. We evaluate these aspects in the experimental section, and show that using negative examples is an important aspect of our method. We show that the combination of the two leads to similarity measure with superior results over current similarity measures.
SIMILARITY MEASURE EVALUATION
We use three benchmarks to evaluate similarity measures: the known layer prediction benchmark, and two new benchmarks we design: the multilingual benchmark and the magge-caption benchmark. We further propose strengthened version of the last using the FAISS software.
3

---

Under review as a conference paper at ICLR 2023
Network
Layer
Layer 2
Layer
Layer 4
Layer 5
Layer
Network B
Layer
Layer 2
Layer
Layer 4
Layer 5
Layer 6
Figure 1: Layer prediction benchmark. Given two models differing only in weight initialization, A and B, for each layer in the first model, among all layers of the second model, a good similarity measure is expected assign the highest similarity for the architectrrall-ccorsspoon
layer.
5.1 LAYER PREDICTION BENCIMARR
Proposed by Kornblith et al. 2019). basic and intuitive benchmark is to assess the invariance of similarity measure against changes to the random seed. Given two models, with the same architec- ture and training configuration (i.e., same training set, same hyperparameters, etc.), differing only in their weight initializations, for each layer in the first model, among all layers of the second model, one expect that a good similarity measure assigns the highest similarity for the architecturally- corresponding layer. Formally, let f and be two models with & layers, and define as f; and gi as the jth layer of and y models, respectively. After caleulating the similarity of with each layer of (g),..., fi)) the pair with the highest similarity is expected to be (Jis 9i). The benchmark counts the number of layers for which this pair was indeed assigned the highest similarity, and divides by the total number of pairs. An illustration is found in Figure [
The intuition behind this benchmark is that each layer captures different information about the input data. For example Jawahar et al. (2019) showed that different layers of the BERT model (Devlin al., 2018) capture different semantic information. Korbblith al. (2019) showed, using CKA similarity, that indeed there exists a correspondence between layers of models trained with different seeds. Thus, although trained from different seeds, the same layers are expected to capture the same information and therefore have a high similarity.
5.2 MULTILINGUAL BENCHMARK
Multilingual models, such as Mulilingual-BERT (Devlin et al. 2018). learn represent texts in different languages in the same representation space. Interestingly, these models show cross-lingual zero-shot transferability, where a model fine-tuned in one language and evaluated in a different language (Pires et al., 2019). Muller et al.] (2021) analyzed this transferability and found that lower layers of the Multilingual-BEll BERT align the representations between sentences in different languages.
Since multilingual models share similarities between representations of different languages, we ex- pect that a good similarity measure should assign a high similarity to two representations of sen- tence in two different languages. In other words, we expect similarity measures to be invariant to the sentence source language. Consider multilingual model ff and dataset X, where each entry in (f) (i) E X be sentence written it consists of the same sentence in different languages. Let (x1") .22
in two languages- - language and language B. The similarity between f(x and f(x!")) should be higher than the similarity between f(z(") and the representation of a sentence in language B randomly chosen from X. i.c., f(x("),, where (ze)) (("aa))))
(.al))) E X is a randomly chosen example from X. The benchmark calculates the fraction of cases for which the correct translation assigned the highest similarity. An illustration is found in Figure
Additionally, suggest strengthened version of the multilingual benchmark, using FAISS, library for efficient similarity search. Instead of sampling random sentences in language B, we use FAISS to find the pair (2(3) ...5)) E X. where x-2
(1) with the representation f fxx2") that is most similar to f f(x)") ). out of a large set of vectors pre-indexed by FAISS. This leads to a more (7) is expected be higher challenging scenario, the similarity between 2 and FAISS sampled x
(f) * x-2
than the similarity between 2 and randomly chosen x/ increasing the difficulty of the similarity measure correctly identify the pair (2]4) (()) as the highest-similarity pair.
4

---

Under review as a conference paper at ICLR 2023
S
S
S₁
S2
1
21
rz
CV model
Mutilingual
Multliiguua model
Muttlinuual
a pencil drawing of zebra and her baby.
very typical bus
Product and geography aze what nake czear skiming wozk.
Produkt und Geogzaphie das, was czene abschöpfen Arbeit nacht.
The tannis shees have range of prices.
Original caption
Image
Other caption
The multilingual benchmark. of and rf denote the representations of the same sentence in different languages, and Si their similarity. rá represents the random sentence representation, and S2 is the simi- larity between and rf. We expect be higher than S2.
Figure 2:
Figure 3: The image cageion benchmark. rf and r' denote the representations of the cap- tion and the image pair, respectively, and Si is their similarity. r denotes the random caption representation, and S2 the similar- ity between it and r². We expect S to be higher than S2.
5.3 IMAGE-CAPTION BENCHMARK
Let be a dataset of images and their textual descriptions (captions), / be a computer vision model and g a language model. Given pair of image and its caption, (m E X, good similarity ff (mm g(c()). In partic- ular, this similarity should be higher than that of the pair of the same image representation ff (mm()) and some random caption's representation g(c(5)), where e(d) randomly chosen caption from dataset X. The intuition behind this benchmark is that an image and its caption represent the same scene different way. Thus, their representations should have a higher similarity than that of the same image and some random caption. An illustration is found in Figure
measure is expected assign a high similarity their representations -
As in the multilingual benchmark, we also propose a strengthened variant for the Image-caption benchmark using FAISS, Rather than sampling random caption cl5), we use FAISS to find the pair (ml)), cii) X, where cli) + clO), with the representation g(c()) that is most similar to g(c()))
EXPERIMENTS
Baselines and Ablations.
We compare ContraSim with the following standard baselines.
Centered Kernel Alignment (CKA): Proposed by Korbblith al. (2019). CKA computes kernel matrix for each matrix representation input, and defines the scalar similarity index as the two kernel matrices' alignment. We use linear kernel for CKA evaluation, as the original paper reveals similar results for both linear and RBF kernels. CKA is our main point of comparison due to its success in prior work and wide applicability.
PWCCA: Proposed by Morcos et al. 2018), PWCCA an extension of Canonical Cor- relation Analysis (CCA). Given two matrices, CCA finds bases for those matrices, such that after projecting them to those bases the correlation between the projected matrices is maximized. While in CCA the scalar similarity index is computed as the mean correlation coefficient, in PWCCA that mean is weighted by the importance each canonical correlation has the representation
See Appendix A.4 for more details on each method. Appendix A.5 report additional baseline results.
In addition, report the results of new similarity measures, which encoder map representations to the space where similarity is measured. However, in both methods we train ʃe to only maximize the similarity between positive pairs:
Lmaz - -(21,
(4)
PPCCAA requires the number of examples to be larger than the feature vector dimension, which is not possible achieve in all benchmarks. Therefore, we compare with PWCCA in subset of experiments.
5

---

Under review as a conference paper at ICLR 2023
where 21 and 22 are representations whose similarity we wish maximize, where for batch of size n similarity is measured as: - 8 x1). We experiment with two functions for s- dot-product and CKA-au and accordingly name these similarity measures DeepDot and DeepCKA. These methods provide point of comparison where the similarity measure is trained, but without
negative examples, examine whether contrastive learning is crucial our method's ' success.
Encoders details. In all experiments, the encoder fo is two-layer multi-layered perceptron with hidden layer dimensions of 512 and 256, and output dimension of 128. We trained the encoder for 50 epochs for the layer prediction and 30 epochs for the multilingual and We used the Adam optimizer (Kingma & Ba 2014) with a learning rate 0.001 and a batch size of 1024 representations. We used
benchmarks.
0.07 for ContraSim training.
6.1
LAYER PREDICTION BENCHMARK
6.1.1
SETUP
Recall that this benchmark evaluates whether certain layer from one model is deemed most similar to its siaittecccrallycoorrsppneeee layer from another model, where the two models differ only in their weight initialization. We repeat this process for all layers and 5 different model pairs, and report average accuracy. We experiment with both language and vision setups.
Models. For language experiments, we use the recent MultiBERT's (Sellam et al. 2021), a set of 25 BERT models, differing only in their initial random weights. For vision experiments, we pre-train 10 visual transformer (ViT) models (Dosovitskiy et al. 2020) on the ImageNet- 1k dataset (Russakovsky et al. 2015). Then we fine-tune them CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., Further training details are available in Appendix A.3
2009
Datasets. For language experiments, we experiment with word-level contextual representations generated on two English text datasets: the Penn Treebank (Marcus et al. 1993) and Wiki'Text (Merity et al. 2016). For Penn TreeBank we generate 5005 and 10019 representations for the test and training sets, respectively; for WikiText we generate 5024/10023 test/training representations. For vision experiments, we experiment with representations generated on CIFAR-10 and CIFAR- 100. For both generate 5000 and 10000 test and training representations, respectively.
Positive and Negative sets. Given batch of representations of some model at layer j. we define its positive set the representations the same layer j of models that differ from i. The negative set is all representations from layers that differ from j (including from model i).
6.1.2 RESULTS
The results are shown in Table 1 In both language and vision evaluations. CKA achieves better results than PWCCA, consistent with the findings by (Ding et al. 2021). DeepDot and DeepCKA A perform poorly, with much lower results than PWCCA and CKA, revealing that maximizing the similarity is not satisfactory for similarity measure purposes. Our method, ContraSim, achieves excellent results. When trained on one dataset's training set and evaluated on the same dataset's s test set, ContraSim achieves perfect accuracy under this benchmark, with large margin over CKA results. This holds for both language and vision cases.
Even when trained on one dataset and evaluated over another dataset, ContraSim surpasses other similarity measures results, showing the transferability of the learned encoder projection between datasets. This true both when transferring across domains (in text, between news texts from the Penn Treebank and Wikipedia texts), and when transferring across classification tasks (in images, between the 10-label CIFAR- 10 and the 100-label CIFAR-100).
6.2 MULTILINGUAL BENCHMARK
6.2.1 SETUP
This benchmark assesses whether a similarity measure assigns a high similarity to multilingual rep- resentations of the same sentence in different languages. Given batch of (representations of) sen-

---

Under review as a conference paper at ICLR 2023
Table 1: Layer prediction benchmark accuracy results for language and vision cases. For encoder- based methods we report mean and std over 5 random initializations. For ContraSim, we experiment with training with different datasets (rows) and evaluating on same or different datasets (columns).
Language
Vision
Penn TreeBank
WikiText
CIFAR-10
CIFAR-100
PWCCA
38.33
55.00
PWCCA
47.27
45.45
CKA
71.66
76.66
CKA
78.18
74.54
DeepDot
15.55 1.69
14.00
2.26
DeepDot
14.90 + 1.78
14.18
+ 2.67
DeepCKA
16.66
3.16
19.66
1.63
DeepCKA
17.09 + 2.95
13.09
+ 4.20
ContraSim
ContraSim
Penn
100 + 0
85.45 + 1.62
CIFAR-10
100 + 0
90.54 + 2.90
Wiki
94.00 4.66
100 + 0
CIFAR-100
85.81 + 5.68
100 + 0
tences 6(0) in language L; and their translations 800 in language L,. we compute the similarity between ((0) and b(8), and the similarities between 8(0) and 10 randomly chosen batches of repre- sentations in language L,. If ((0) is more similar to (65) than it is to all other batches, then we mark success. (Alternatively, in a more challenging scenario, we use FAISS find for each representation in each layer the 10 most similar representations in that layer.) We repeat this process separately for representations from different layers of a multilingual model, over many sentences and multiple language pairs, and report average accuracy per layer Appendix A.1 gives more details.
Model and Data. We use two multilingual models: multilingual BERT (Devlin et al., 2018 and XLM-R (Conneau et al. 2019). We use the XNLI dataset (Conneau al. 2018). which has natural language inference examples, parallel in multiple languages. Each example in our dataset is sentence taken from either the premise or hypothesis sets. We experiment with 5 typologically- different languages: English, Arabic, Chinese, Russian, and Turkish. We created sentence-level representations, with 5000 test 10000 training representations. As sentence representation, we experiment with [CLS] token representations and with mean pooling of token representations, since Del & Fishell (22T)) noted difference in similarity in these two cases. We report results with [CLS] representations in the main paper and mean pooling results in Appendix(A.) A.] the trends are similar.
(222))
Positive and Negative sets. Given a pair of languages and a batch of representations at some layer, for each representation we define its positive pair the representation of the sentence in the different language, and its negative set as all other representations in the batch.
6.2.2 RESULTS
Results with multilingual BERT representations in Table 2 show the effectiveness of our method. (trends with XLM-R are consistent; Appendix (A.13)). Under random sampling evaluation (left block), ContraSim shows superior results over other similarity measures, although always being evaluated on language pairs it has not seen at training time. Using FAISS-basec based sampler (right block) further extends the gaps. While CKA results dropped by Rs 45%, DeepCKA dropped by 51%, and DeepDot dropped by 40%, ContraSim was much less affected by FAISS sampling (s 17% drop on average and practically no drop in most layers). This demonstrates the high separability between examples of ContraSim, enabling it to distinguish even very similar examples and assign a higher similarity to the correct pair. For all other methods, mid-layers have the highest accuracy, whereas for our method almost all layers are near 100% accuracy, except for the first 3 or 4 layers.
Evaluation results reveals interesting insights that were not found using previous similarity mea- sures. In FAISS results, we see that there much greater difference accuracy between shallow and deep layers in ContraSim compared to previous similarity measures. This means that using previous similarity measures we might infer that there is no difference the ability to detect the
Foor deep similarity measures (DeepCKA, DeepDot, and ContraSim), upon training the encoder on exam- ples from a pair of languages, (be, Ing), r & g, we evaluate it over all other distinct pairs of languages. https:/hugginggfccccoo -awhherrbaae-mmllllnngnn
7

---

Under review as a conference paper at ICLR 2023
Table 2: Multilingual benchmark accuracy results. With random sampling (left block), ContraSim outperforms other similarity measures. Using FAISS (right block) further extends the gaps.
Random
FAISS
Layer
CKA
DeepCKA
DeepDot
ContraSim
CKA
DeepCKA
DeepDot
ContraSim
71.7 5.3
82.0 +
63.3 + 10.4
95.5 5.4
20.1 + 4.0
10.7 2.6
29.9
36.0 10.7
78.7 +
4.14
68.5 9.9
95.0
27.2 5.5
2.9
46,9
=9
33.0 14.8
3.0
8.2
70.4 9.7
41.9 6.7
4.2
51.5 10.3
20.5
92.6
91.5
3.4
99.9
33.4
3.72
72.4 8.6
88.3
83.5
94.7 4.8
99.9
49.3
6.3
4
99.1.
88.6
86.4
92.5
100
51.4
39,9
42.1
99.5.
88.8 3.7
5.0
92.6
100
53.0
41.1
11.7
99.6.
85.2 3.6
91.4 7.0
100
56. 1
8.7
43.8 13.4
99.7.
8..
5.66
89.1 1..5
100
49
8.5
39.2
99.6.
10
3.
80.3 5.9
85.34 10.3
51.5 5.3
42.4 7.8
34.3 12.2
99.5.
4.2
76.6 6.4
79.74 13.9
9
52.4 5.3
43.3 13.3885
31.4 12.8
99.3. 0.5
12
86.4 + 3.4
63.8 7.9
64.3+ 19.7
99.9 +
52.8 +: 4.5
32.3 + 8.7
26.1 + 21.9
98.9.
correct pair across different layers. However, ContraSim shows that the difference in the ability to detect the correct pair dramatically changes from shallow to deep layers.
To further analyze this, we compare the original multilingual representations from the last layer with their projections by ContraSim's s trained encoder. Figure from Appendix A.1.2 shows UMAP (Meinnes et al.] 2018) projections of representations of sentences in English and sentences in Arabic, before and after ContraSim encoding. The ContraSim encoder trained Arabic and English languages. The original representations are organized according to the source language (by shape), whereas ContraSim projects translations of the same sentence close each other (clustered
by color).
6.3 IMAGE-CAPTION BENCIIMARK
6.3.1 SETUP
Given test set X, consisting of pairs of an image representation generated by computer vi- sion model and its caption representation from a language model, we split X to batches of size 64. For each batch, we compute the similarity between the image representations and their cor- responding caption representations. We then sample 10 different caption batches, either randomly or using FAISS (as before), and compute the similarity between the image representation and each aoddo//FAIS--retiivvee caption representation. If the highest similarity is between the image rep- resentation and the original caption representation, we mark a success. For trainable similarity measures, we train with 5 different random seeds and average the results.
Models and Data. We use two vision models for image representations: ViT and ConvNext (Liu et al. 2022); and two language models for text representations: BERT and GPT2 (Radford et al. 2019 We use the Conceptual Captions dataset (Sharma et al. 2018), which has =3.3M pairs of images and English captions. We use 5000 and 10000 pairs as test and training sets, respectively.
Positive and Negative sets. Given batch of image representations with their corresponding cap- tion representations, for each image representation we define as a positive set its corresponding caption representation, and negative set all other representations in the batch.
6.3.2 RESULTS
Figure demonstrates the strength of ContraSim. Under random sampling (green boxes), Deep- CKA achieves comparable results to ContraSim, while DeepDot and CKA achieve lower results. However, using FAISS (red boxes) causes a big decrease DeepCKA accuracy, while ContraSim maintains high accuracy.
Another interesting result is that in 3 of pairs we tested, CKA accuracy using FAISS for sampling was higher than using random sampling. This contradicts the intuition that using similar examples the sampling stage should make it difficult for similarity measures to distinguish between examples. This might indicate that CKA suffers from stability issues.
8

---

Under review as a conference paper at ICLR 2023
ConvNext/GPT2
ConvNext/BERT
100
100
f
80
Accuracy
60
60
40
f
20
random FAISS
20
f
random
FAISS
0
DeepCKA
DeepDot
ContraSim
DeepCKA
DeepDot
ContraSim
VIT/GPT2
VIT/BERT
100
80
Accuracy
60
40
40
20
random
20
f
random
FAISS
FAISS
CKA
DeepCKA
Deep
ContraSim
DeepCKA
DeepDot
ContraSim
Method
Method
Figure 4: Image-caption benchmark results for 4 different model pairs. ContraSim works best, and the only measure robust FAISS sampling.
Looking CKA results we might observe that BERT representations are more similar to computer vision representations rather than GPT2 model representations. That is because using CKA we see that the pairs with BERT model achieve higher accuracy compared to pairs with GPT2 model. Higher accuracy means that it is easier to detect the matching pair, which means that the representa- tions are more similar. However, ContraSim achieves high accuracy both BERT pairs and GPT2 pairs, which means that both models share the same similarity to computer vision models, in contrast to what we may infer from previous similarity measures.
Finally, we report results with the multi-modal CLIP model (Radford et al. 202T) in Appendix 5 Because the model was pre-trained with contrastive learning, simple dot-product similarity works very well, there is need to leam similarity measure in this case.
CONCLUSION
We proposed similarity measure, ContraSim, based ideas from contrastive learning. By defining the positive and negative sets we learn an encoder that maps representation to space where similarity measured. Our method outperformed other similarity measures under the common layer prediction benchmark, and two new benchmarks we proposed: the multilingual benchmark and the image-caption benchmark. It particularly shines in strengthened versions of said benchmarks, where random sampling is replaced with finding the most similar examples using FAISS.
Our new similarity measure benchmarks can facilitate work on a similarity -based based analysis of deep neural networks. The multilingual benchmark is useful for work in multilingual language models, while the image-caption benchmark may help work in multi-mode modiil settings. drawback of our method is that it can only compare representations of the same dimensionality. This can be addressed by using a dimensionality reduction to a shared space. We leave the research of a more sophisticated method for future work. Moreover, compared to existing methods, ContraSim needs access to training set for the encoder training procedure. Compared with closed-fome similarity measures, train time is another trade-off. In addition, since method learns parameterized measure, it may help train models with similarity objectives. We also leave that for future work. Finally, considering ContraSim's superiority in all evaluations, will better fit for interpretability of neural networks.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Our work adds to the body of literature on the interpretability of neural networks and may mitigate their opacity. We do not foresee major risks associated with this work. However, malicious actor could train ContraSim adversarially, assign poor similarity estimates, and lead to false analyses.
REPRODUCIBILITY STATEMENT
All the evaluations conducted throughout this paper are fully reproducible. ContraSim training scheme and architecture were described in Sections and 6 Detailed information regarding each evaluation also provided in the Section 6 with further details in Appendix Code to reproduce all the paper results will be made publicly available upon publication.
REFERENCES
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Christian Cianfarani, Arjun Nitin Bhagoji, Vikash Sehwag, Ben Zhao, and Prateek Mittal. Un- derstanding robust learning through the lens representation similarities. arXiv preprint 0rX1::2206.098688 2022.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02.
2019.
Maksym Del and Mark Fishel. Establishing interlingua in multilingual language models. arXiv preprint arXiv: 109.0/2099 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scabl scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint urXiv:/810.04880 2018.
Frances Ding, Jtan-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity with statistical testing. arXiv preprint 10Xiv:208.00161, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /929, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9729-9738, 2020.
Harold Hotelling. Relations between two sets of variates. Biometrika, 28:321-377, 19366
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL 20/9-57th Annual Meeting of the Association for Computational Linguisties, 2019.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. JEEE Transactions Big Data, 7(3):535-5477 2019.
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Simon Korblith,, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11976-11986, 2022.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguisties, 19(2):313-330, 1993. URL
org/ 193-2004
Leland Mclnnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. First align, then predict: Under- standing the cross-lingual ability of multilingual bert. arXiv preprint vrXiv:2101.11109 2021.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? arXiv preprint 1rXiv: 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 3748-8763. PMLR, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV, 115(3):211 25222, 2015. doi: 10. 1007/1 1263-015-08166
Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Ture, Jacob Eisenstein, Dipanjan Das, et al. The multiberts: Bert repro- ductions for robustness analysis. arXiv preprint arXiv:2106. 16163, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
John WU, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Similar- ity analysis of contextual word representation models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4638-4655, 2020.

---

Under review as a conference paper at ICLR 2023
Table 3: Multilingual benchmark results with mean pooling.
Random
FAISS
Layer
CKA
DeepCKA
DeepDot
ContraSim
CKA
DeepCKA
DeepDot
ContraSim
87.7
86.3 +: 9.7
43.4 17.7
98.7 2.2
67.6 + 14.3
54.1
10.9
41.7 + 19.1
94.2 10.7
89.0
88.7 +
$1.5
20.0
99.5 0.8
68.2 = 13.2
4.
38.9 = 17.2
6.0
20.8
20.8
99.9 0.1
72.2 = 11.5
8.1
44.8 = 16.6
20.5
93.7 3.3
5.0
73.1 1944
+.0
74.3 10.0
8.0
45.7 16.5
99.5
4.01
83.9 15.6
99.9
4.22
8.1
53.2 = 17.5
99.8 1
2,44
91.2 91
100
7.
8.1
60.1 18.2
99.8 1.7
94.8 2.5
90.6
9.2
100
7.99
7.
63.5
99.9
4.3
90.3
76.7 8.1
524
61.0
9.
34
5.0
15.1
100
5.4
55.5
99.9
92.6222
5.
80.7 10.7-1888
100
4.
49.2
9..9
91.1
6.5
72.2
+
70.6
48.8
43.2
99.8
90.8
71.0 7.6
99.9
72.7
4.
42.7
9..4
A APPENDIX
A.1 MULTILINGUAL BENCHMARK
A.1.1 EVALUATION PARAMETERS
We split the test set, X, into equally sized batches of size 8, (b(1), ((2), ((m),, where each batch consists of multilingual BERT representations of the same sentence in 5 different languages: L (L,, Ls}. Given pair of different languages, (Li, L,), j. and a batch of representations, b, we consider the representation of those languages in the batch, (b[i]. 6[]]), and compute the similarity between b[i] and b[j] as Su sbbi1, b[jl). We also compute the similarity between b[i] and 10 randomly chosen batches (or, 10 batches chosen using FAISS) of representations in language Lj as {s; = <bb[i) b, [- If arg max, { = O, we count it as a correct prediction. Each layer's accuracy defined as the number of successful predictions over the number of batches, n. We average results over all possible pairs of different languages.
A.1.2 CONTRASIM PROJECTIONS
Original Representations
ContraSim Representations
Arabic
English
English
Figure 5: Original representations (left) are clustered by the source language (by shape). ContraSim (right) projects representations of the same sentence different languages close by (by color).
A.1.3 FURTHER EVALUATIONS
In addition to using the [CLS] token representation as a sentence representation, we also evaluated the multilingual benchmark using mean pooling sentence representation. We used the same evalu- ation process as described in Section 6.2 The results, summarized in Table 3 are consistent with the results in the main paper (Table 2). Under random sampling, ContraSim outperforms all other similarity measures. Using FAISS causes a big degradation in all other methods' accuracy, while ContraSim maintains high accuracy across all layers.
12

---

Under review as a conference paper at ICLR 2023
In addition, evaluated the multilingual benchmark with another multilingual model- the XLM-R (Coneeau et al. 2019) model. Results, summarized in Table 4 show similar pattern to Tables and with ContraSim achieving the highest accuracies across all layers, in both random sampling and FAISS sampling scenarios.
Table 4: Multilingual benchmark results on XLMMR model.
Random
FAISS
Layer
CKA
DeepCKA
DcepDot
ContraSim
CKA
DeepCKA
DeepDot
ContraSim
89.7 +. 4.6
88.2 + 3.1
45.0 = 22.1
99.89 0.31
60.6 + 11.4
24.6 + 4.7
24.8 124
98.1
90.6
2.0
63.1 23.4
99.99 0.02
57.1 + 11.4
30.7 5.7
31.1 = 13.5
99.5 0.7
3.1
1.7
79.5 18.9
99.99
55.5 11.1
33.8 33.8664
39.3
99.9
2.
1.7
91.7
100
6.88
55.1 1162
99.9
2.1
1.66
94.0
100
66.2
64.2 3..2
99.9
8.
100+0
66.7
7.6
2.
1.
9.5
100
7..7
44.1
68.8
9
9.11
1.7
9..0
100
68.1
43.6
18.
99.9
94.8
1.
99.
100
8.7
8.0
116
99.9
10
93.
14
92.7
100
462
7..
18.6
99
92.0+28
93.3
92.7
10.9
100
35.5
39.1 7.3
57.2 183
9
12
4.7
89.5
81.6 13.8
100
2.8
32.3 7.0
34.7 15.1
9
A.2 IMAGE-CAPTION
In addition to the four model pairs we evaluated Figure 4 we assessed the multi-modal vision and language CLIP model (Radford et al. 2021), which was trained using contrastive learning on pairs of images and their captions. Results in Table [ show interesting findings. Under random sampling, dot product, DeepCKA and ContraSim achieve perfect accuracy. However, using FAISS causes significant degradation DeepCKA accuracy, and only small degradation in dot product and ContraSim results, with equal accuracy for both. We attribute this high accuracy for simple product to the fact that CLIP training was done using contrastive learning, thus observing high separability between examples.
Table 5: Image-caption benchmark accuracy results using CLIP model
Random
FAISS
CKA
93.67
25.31
Dot Product
100
98.73
DeepCKA
100
13.92
DeepDot
29.11
25.31
ContraSim
100
98.73
A.3 VIT TRAINING DETAILS
We used the ViT-base (Dosovitskiy al. 2020 architecture. We pretrained 10 models on the ImgaeNet-1K K dataset (Deng et al. 2009), differing only in their weight initializations by using ran- dom seeds from 9. We used the AdamW optimizer (Kingma Ba 2014) with Ir 0.001, weight decay- 1e 3, batch size 128, and cosine learning scheduler. We trained each model for 150 epochs and used the final checkpoint.
Then, fine-tuned the pretrained models CIFAR-10 and CIFAR-100 datasets (Krizhevsky al. 2009). We used Adam W optimizer with = 2e - 5, weight decay = 0.01, batch size = 10, and linear scheduler. For models fine-tuned CIFAR-10. the average accuracy the CIFAR-10 test set is 96.33%. For models tnnettunddde on CIFAR- 100,,- the average accuracy on the CIFAR-1O1 test set is 78.87%.
13

---

Under review as a conference paper at ICLR 2023
A.4 DETAILS OF PRIOR SIMILARITY MEASURES
Canonical Correlation Analysis (CCA). Given two matrices, CCA finds bases for those matrices, such that after projecting them to those bases the projected matrices' correlation is maximized. For 1 S i S Pi. the canonical correlation coefficient is given by:
Pi max corr( (X w Yuk)
8.I. Vjsi Xw 1 Xw
(5)
Vjsi Yuy Y Yuy.
where corr(X,Y)
- Given the vector of correlation coefficients corrs (p1, .... Pp:),
(X.Y)
(P......pp.))
the final scalar similarity index is computed as the mean correlation coefficient:
P1
ScCa (X, Y) = PCCA
Pi
(6)
P1
previously used in (Raghu et al. 2017 Korbblith et al. 2019).
Weiegten-Weigteeddd CCA (PWCCA). Morcos et al. (2018) proposed a different approach to
transform the vector of correlation coefficients, corrs, into scalar similarity index. Instead of
defining the similarity as the mean correlation coefficient, PWCCA uses weighted mean and the similarity defined as:
Spw
Dg Σ O4
OGPi
Σ 1y))
where 2, is the jth column of X, and h2 Xww is the vector observed upon projecting X the canonical direction. Code available at:
Centered Kernel Alignment (CKA). CKA, Proposed by Kornblith et al. (2019), suggests com- puting kernel matrix for each matrix representation input, and defining the scalar similarity index as the two kernel matrices' alignment. For linear kernel, CKA is defined as:
Y
SCKa
(8)
| X
Code available at: https::/github.co http:/ttthucccoomeoeeeommceeceee ctw.:/hgu..ccooooggg......o research/ research/ :ree/mmatter/rpppeeentaaaooo stt.:ionn_mmiiaaiii..mm
A.5 ADDITIONAL SIMILARITY MEASURES
We evaluated three more similarity measures: singular vector CCA (SVCCA) Raghu et al. (017)). dot product and L2 norm between the difference of the normalized representations (dubbed Norm in the paper). Given two batch representations, X and Y. SVCCA performs CCA on the truncated singular value decomposition (SVD) of X and
For two representations, z and we defined dis-similarity measure as:
DisNorm (z, y) = [</||<>]]] l/IlI
(9)
Since this is dis-similarity measure, we defined the norm similarity measure as:
S = 1 Dis (x,z)
(10)
For batch of representations, define batch similarity as the mean of pairwise norm similarity. In addition, we evaluated ContraSim with a different similarity measure than dot not-product and replaced it with the norm similarity measure.
14

---

Under review as a conference paper at ICLR 2023
Table 6: Layer prediction benchmark with additional similarity measures.
Penn TreeBank
WikiText
SVCCA
46.66
56.66
Dot product
8.33
6.66
Norm
10.00
11.66
ContraSim norm
Penn
100.00 100.00
90.00
Wiki
100.00
Table 7: Multilingual benchmark with additional similarity measures. Left block is with random sampling, and right block is FAISS sampling.
Random
FAISS
Dot product
Dot Product
Layer
Norm
ContraSim
Norm
ContraSim.norm
1
48.93 = 7.07
68.67 10.86
89.39 + 14.46
15,00 + 6.41
20.42 + 9.32
15.76 6.56
2
31.71 3.31
74.19 1 9.73
85.40 1 18.00
20.14 2.75
21.16 + 11.43
18.83 1 11.19
3
49.32 1 6.53
83.92 13.12
85.68 1 18.88
13.00 4.500
29.42 1: 22.99
26.36 1 16.01
4
29.60 + 2.44
99.61 + 0.41
96.81 +
16.20 4.00
57.98 + 15.68
28.79 = 9.80
5
99.75 + 0.29
99.86 + 0.33
99.86 0.23
82.17 7.36
82.39
74.04 7.11
6
99.75 + 0.35
99.84 + 0.27
0.27
0.11
83.38 7.70
18.24=6..15
77.00 + 6.68
7
9
99.85
9.13
89.72 5.57
89.23
78.21 6.77
8
0.11
9.15
9.11
93.49 4.07
89.70
81.97 = 6.63
99.61 + 0.38
9.47
99.91
82.48 9.32
84.85
81.37 6.53
10
96.64 2.46
99.38
9.18
55.02 + 15.42
$1.43
80.59 6.83
11
8.133
98.40
99.
29.62 = 13.82
82.20 = 9.46
80.74 7.08
12
76.86 = 15.23
87.25 = 12.39
12.39
99.73 0.36
25.75 = 26.55
50.11 + 32.55
80.24 6.63
Similar to PWCCA, SVCCA requires that the number of examples is larger than the vector dimen- sion, thus could only evaluate in the layer prediction benchmark. All other similarity measures were evaluated with all evaluations the layer prediction benchmark, the multilingual benchmark and the image-caption benchmark.
Table shows layer prediction benchmar results. We can observe that SVCCA achieves slightly better results than PWCCA, and lower than CKA and ContraSim. Both dot product and norm achieve low accuracies. ContraSim_norm achieves same or better results than ContraSim.
Table 8: Image-caption benchmark results for additional similarity measures, 4 different model pairs.
Vision Model
ViT
ConvNext
Language Model
BERT
GPT2
BERT
GPT2
Dot
6.32
6.32
11.39
8.86
Product
Random
Norm
6.32
7.59
15.19
7.59
ContraSim norm
100
100
100
100
Dot
5.06
2.53
7.59
6.32
Product
FAISS
Norm
5.06
5.06
6.32
10.12
ContraSim
93.67
98.73
81.03
93.67
15

---

Under review as a conference paper at ICLR 2023
Multilingual benchmark results, summarized in better results than CKA, although achieve low results under layer prediction and image-caption benchmarks. This emphasizes the importance of multiple evaluations for similarity measures. Com- pared to ContraSim, both methods achieve lower results. ContraSim norm achieves lower results compared to ContraSim, under both random and FAISS sampling.
that both dot product and norm achieve
Image-caption benchmark results, summarized Table g show that under both random sampling and FAISS sampling dot product and norm achieve low accuracy. Under random sampling, Con- traSim_nore achieves perfect accuracy, while using FAISS sampling shows slight degradation com- pared ContraSim.
16


Reviews:
[{"review_id": "_q2OpLCN7Q", "reviewer": "Reviewer_m4jx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a parameterized similarity measure that shows significant improvement over existing closed-form similarity measures. ", "summary_of_the_review": "This paper provides a novel similarity measure based on contrastive learning, which greatly outperform existing closed-form measures in different scenarios.", "strength_and_weaknesses": "Strength:\n  - The paper is well-written and easy to follow. \n  - The method is simple yet effective.\n  - The empirical analysis is comprehensive. \n  - The provided benchmark can benefit future research.\n  - This paper provides a novel perspective in providing similarity measures compared to the existing closed-form ones.\n\nWeakness:\n  - As a learnable similarity measure, due to the randomness of model training, this metric cannot guarantee consistent results, which may limit its use cases and reliability to some extent.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RbCnbzqNmZ6", "reviewer": "Reviewer_1R2q", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a learned measure of representation similarity, ContraSim, as opposed to analytical ones like centered kernel alignment (CKA) and canonical correlation analysis (CCA). ContraSim is specific to a task (training distribution), and uses a simple MLP to project the representation to a different subspace. Experiments show that contrastive loss is a key component to achieving good results. \n\nPaper also introduces two additional benchmarks based on multi-lingual similarity and image-caption similarity.  ContraSim performs well on these tasks compared to baselines.", "summary_of_the_review": "I think the paper is above the acceptance bar for ICLR. I would like to see some additional analyses and discussion in the final version of the paper.", "strength_and_weaknesses": "Strengths:\n- Paper is well-written, easy to follow. \n- Introduction of two benchmarks adds a different perspective to this research area compared to literature\n- Proposed method outperforms baselines by a significant margin\n\nWeaknesses:\n- The analyses in the paper can dive deeper into the problem space. How much does ContraSim generalize? Can you use the same  ContraSim model across all benchmarks instead of training a separate one for each?\n- It would be good to discuss the trade-offs introduced by ContraSim. E.g. we need to access to labeled training data. We need different models for each domain. \n- The paper does not discuss the implications of this research. Why is learning based measure useful? What did we learn that's new? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "O1gJqY1tfXd", "reviewer": "Reviewer_rucx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a similarity metric learned from similar and dissimilar examples using a loss inspired by contrastive learning. The metric is tested on three benchmarks (i.e., layer prediction, multilingual language modeling, image captioning), among which are two new benchmarks proposed by the authors on which they test the metric. \n", "summary_of_the_review": "\nWhen defining a new similarity metric, one should first define its theoretical properties and evaluate these properties empirically with data. These important parts are lacking in the paper. \n\nThe motivation of the work is not clear. Is it the purpose to learn an encoder based on a contrastive loss or a similarity metric that can be used in and transferred to all kinds of applications?\n\nWhat would be the effect if another similarity metric than cosine similarity would be used to compare representations during training?\n\nThe novelty of the work seems limited. Learning representations from positive and negative examples based on a contrastive loss and using these to create a semantic space where similarity is measured has been done for almost a decade. The number of baselines and comparisons with other similarity metrics seem limited. \n\nThe authors rely on the FAISS library for similarity search of train examples. What is the influence of this initial search and the underlying methods that FAISS uses on the quality of the examples used in the training and consequently on the quality of the learned metric?\n", "strength_and_weaknesses": "Strengths:\n- Evaluation of the created metric on different types of data. \n\nWeaknesses:\n- Clarity of the work and its goals. \n- Limited technical contribution.\n- Evaluation only on three benchmarks, two of which are created. \n- Comparison with existing similarity metrics is limited.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4y7bkV5Ub-", "reviewer": "Reviewer_xBqQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper proposes a new contrastive learning objective for learning representations called Contrasim. Contrasim aims to learn representations that maximize inner-products between \"should-be-similar\" inputs and minimize them between \"should-be-dissimilar\" inputs. Contrasim's claim to novelty is that it adds the additional minimization objective for the \"should-be-dissimilar\" representation pairs whereas other representation learning objectives like project-weighted CCA (PWCCA) and centered kernel alignment (CKA) do not.\n\nThe authors test Contrasim on 3 representation learning tasks:\n1) Determining which encodings of sentences from two different languages are translations of each other.\n2) Determining which sentence and image encodings are (caption,image) pairs.\n3) For neural networks with the same architecture and training data, but different initialization, determine which layers of the neural networks correspond.\n\nExperimental results show that Contrasim dramatically outperforms the chosen baselines: CKA, PWCCA, deep dot-product and deep CKA; and the performance gap increases when using hard negative mining with the FAISS library.\n\nLast, low dimensional projection plots show that Contrasim improves the proximity of should-be-similar inputs relative to the original representations and relative to the aforementioned baselines.", "summary_of_the_review": "While Contrasim is a moderately intriguing variant of existing contrastive learning losses, it is too similar to existing methods like SimCLR and NCE and not well enough distinguished from them to warrant acceptance at ICLR.\n", "strength_and_weaknesses": "\nThe strength of this paper is that it emphasizes the importance of negative examples in training similarity functions and shows that they are critical in contrastic learning. I also liked that the authors experimented on embeddings produced by SOTA models like RoBerta and ViT.\n\nHowever, the weakness of the paper is that using negative pairs is very common in contrastive learning, despite the paper's claiming the opposite. For instance, in SimCLR (which this paper cites), the objective maximizes the dot products of should-be positive pairs and minimizes the dot products of negative pairs (see equation (1) in the SimCLR paper).\n\nLikewise, well known contrastive learning criteria like noise contrastive estimation (NCE), InfoNCE and triplet loss seek to minimize dot products of negative pairs relative to positive pairs.\n\nThese contrastive learning objectives need to be compared against because they are all highly similar to the proposed Contrasim.\n\nIn particular, equation (3) Contrasim can be seen as maximizing the log-odds probability of positive versus negative pairs if we assume there is a normalizing factor \"Z\" which has been divided out from both the numerator and denominator:\n\n  eq (3) = \\sum_i log [ \\sum_p\\~i [ exp(ai . ap)/Z ] / [ \\sum_n!\\~i exp(ai . an)/Z ].\n\nwhere \"\\~\" denotes the similarity relation and \"!\\~\" denotes the not-similar relation.\n\nSimCLR can be seen as maximizing the very similar odds ratio:\n\n  \\sum_i \\sum_p\\~i log { [ exp(ai . ap)/Z ] / [ \\sum_n!\\~i exp(ai . an)/Z ] },\n\nso it is not clear if Contrasim presents a notable advance, especially in absence of empirical comparisons and/or theoretical justifications.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject