Robust Similarity Learning with Difference Alignment Regularization
Authors: Shuo Chen
Gang Niu
Chen Gong
Okan Koc
Jian Yang
Masashi Sugiyama
Venue: ICLR 2024 poster (2024)
Keywords: contrastive learning
metric learning
regularization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A new regularization approach to deal with inconsistent differences in similarity learning

Abstract: Similarity-based representation learning has shown impressive capabilities in both supervised (e.g., metric learning) and unsupervised (e.g., contrastive learning) scenarios. Existing approaches effectively constrained the representation difference (i.e., the disagreement between the embeddings of two instances) to fit the corresponding (pseudo) similarity supervision. However, most of them can hardly restrict the variation of representation difference, sometimes leading to overfitting results where the clusters are disordered by drastically changed differences. In this paper, we thus propose a novel difference alignment regularization (DAR) to encourage all representation differences between inter-class instances to be as close as possible, so that the learning algorithm can produce consistent differences to distinguish data points from each other. To this end, we construct a new cross-total-variation (CTV) norm to measure the divergence among representation differences, and we convert it into an equivalent stochastic form for easy optimization. Then, we integrate the proposed regularizer into the empirical loss for difference-aligned similarity learning (DASL), shrinking the hypothesis space and alleviating overfitting. Theoretically, we prove that our regularizer tightens the error bound of the traditional similarity learning. Experiments on multi-domain data demonstrate the superiority of DASL over existing approaches in both supervised metric learning and unsupervised contrastive learning tasks.

Published as a conference paper at ICLR 2024
ROBUST SIMILARITY LEARNING WITH DIFFERENCE ALIGNMENT REGULARIZATION
Shuo Chen', Gang Niu', Chen Gong", Okan Koc', Jian Yang", Masashi Sugiyama' 1.2
'RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
The University of Tokyo, Tokyo, Japan
3 Nanjing University of Science and Technology, Nanjing, China
ABSTRACT
Similarity- representation learning has shown impressive capabilities in both supervised (e.g., metric learning) and unsupervised (e.g., contrastive learning) scenarios. Existing approaches effectively constrained the representation differ- ence (i.e., the disagreement between the embeddings of two instances) to fit the corresponding (pseudo) similarity supervision. However, most of them can hardly restrict the variation of representation difference. sometimes leading overfitting results where the clusters are disordered by drastically changed differences. In this paper, we thus propose novel difference alignment regularization (DAR) to encourage all representation differences between inter-class instances to be as close as possible, so that the learning algorithm can produce consistent differ- ences to distinguish data points from each other. To this end, we construct a new vros-tooallvarriation (CTV) norm measure the divergence among representa- tion differences, and we convert it into an equivalent stochastic form for easy op- timization. Then, we integrate the proposed regularizer into the empirical loss for difference-aligned similarity learning (DASL), shrinking the hypothesis space and alleviating overfitting Theoretically, we prove that our regularizer tightens the er- ror bound of the traditional similarity learning. Experiments on multi-domain data demonstrate the superiority of DASL over existing approaches in both supervised metric learning and unsupervised contrastive learning tasks.
INTRODUCTION
Recently, representation learning with pairwise similarity has demonstrated great effectiveness many different types of data (Weinberger et al., 2006; Zadeh et al., 2016; Xu et al., 2022; Wu et al., 2022). In this problem setting, we usually consider the pairwise relationship (e.g., similar or dissim- ilar) between instances from the training data, and we aim to learn a generalizable feature represen- tation to predict the pairwise similarity for the test data. This topic of similarity learning has already attracted much attention and has also been applied to many specific tasks such as classification, verification, and clustering (Jing& & Tian, 2020; Kou et al., 2022; Zhong et al., 2020).
In the supervised case, the similarity labels of training data are usually provided by humans. This supervised setting is also referred to as metric learning (Xing et al., 2002). where a distance metric is learned to measure the pairwise similarity between instances in both the training and test phases. Al- though some existing approaches can successfully learn the corresponding feature representation of similarity metric via the nonlinear convolutional neural networks (CNNs) (a.k.a. SiameseNer (Kaya & Bilge, 2019)), the correctly predicted distances will not be penalized any more in their empirical loss (e.g., triplet loss (Hoffer & Ailon, 2015) and ProxyAnchor loss (Kim et al., 2020b)). This means that these inter-class distances/diffeereness will vary in an underconstrained range and thus may lead to the unstable predictions that affect the model generalization on the unseen test data.
In the unsupervised (self-supervised) case, the positive (similar) pairs and negative (dissimilar) pairs typically generated by the pseudo supervision, e.g. the instance discrimination (Dosovitskiy et al., 2014) and data augmentation (Chen et al., 2020) used in contrastive learning to build negative
*Correspondence Shuo Chen (shuo.chen.yer www..chhen.a@riicc.......

---

Published as a conference paper at ICLR 2024
Data Points
Differences
Away Points
Pull In Differences
Class-1
Class-1
Class- 1
Higher-Order Difference
(Supervisto))
(Reqularicatoon) Diffremnce Between Differmoes
Differmoc Retwee Peints
Class-2
Class-3
Class-2
Class-3
Claes-2
Claes-3
(c). Regularization ceterence-alignenn regularization (DAR). Our regularizer minimizes the higher-order difference to find the similar (first-oreer) representation dif- ferences, so that we can use consistent differences for data discrimination.
(8). Before Training
(b) A fter Training
Figure/ 1: Conceptual illustration of our proposed
and positive pairs, respectively. Nevertheless, the above issue of inconsistent distances/differenees in supervised metric learning may also exist in self-super contrastive learning. The abrupt changes in the distances between different clusters make it difficult for the learning algorithm to capture category information effectively, because the clusters with relatively small mutual distances single class. In this case, the discriminability of the pairwise distance
tend to be misconstrued is also affected, and thereby reducing the reliability of the learned representation.
For both supervised and unsupervised scenarios, similarity learning has shown promising results, especially in vision and language tasks. However, most existing approaches usually focus on how constrain the representation difference (i.e., the disagreement between the embeddings of two instances) to be consistent with the corresponding similarity supervision (as shown in Fig. ((a)-bb))- They can hardly ensure the consistency among representation differences (i.e., the difference be- tween differences), and these drastically changed differences may disorder the cluster distributions in the representation space. This will lead to some overfitting results where the model prediction is unstable/umsmoothh w.r.t. the input data, and thus degrading the final classification performance. The popular adversarial training (Kim et al., 2020a; Jiang et al., 2020) and classical regularization tech- niques (e.g., the f2/Ezrrgullarizrr (Ying et al., 2009; Zadeh et al., 2016) and label smoothing (Wu et al., 2022)) are able to mitigate this impact by enriching the critical training data or constraining the hypotheses. However, they cannot solve this issue in essence avoid the inconsistency among representation differences. Recent works restricted the local neighbors to having similar distance distributions (Cho et al., 2010; Tian al., 2019; Wei al., 2021), but they can only consider the single scale of the distance value, and their constraints are limited to the adjacent relationship.
In this paper, follow the general principle of Occam's 's razor (Rasmussen & Ghahramani, 2000) to propose a novel difference aligument regularization (DAR) that explicitly encourages representation differences to as close as possible. Specifically, we first construct a cross-tovs pooss-total-vee notal-variation (CTV) norm equipped with a generic distance function to measure the disagreement between any two rep- resentation differences, and this effectively characterizes the higher-order difference information. Such CTV norm is converted to equivalent stochastic form whose computational complexity determined only by the small batch size. We jointly learn the traditional empirical loss with proposed regularizer form the difference-aligned similarity learning (DASL), that can obtain consistent representation differences to distinguish data points from each other (as shown in Fig. (())). Theoretically, prove that the stability and generalizability of DASL can be successfully improved with our proposed regularizer. Since DAR is quite a general technique and applicable to many empirical losses, implement it in both metric learning and contrastive learning approaches, and the baseline results can be significantly improved by our method (e.g., the 2.1% improvement for the self-supervised learning task on ImageNet-IK3
Our main contributions are as follows. (i) We propose a novel regularization technique to solve the problem of inconsistent differences with complete theoretical guarantees. (ii) We construct strictly defined norm to measure the higher-order difference, which can be easily optimized with the stochastic gradient. (iii) Experiments demonstrate that our method can outperform the state-of-the- art approaches in both supervised and unsupervised tasks.
2 BACKGROUND AND RELATED WORK
As the supervised and unsupervised settings of similarity- based representation learning, the related works of metric learning and contrastive learning reviewed in this section. Throughout this paper,
2

---

Published as a conference paper at ICLR 2024
Inconsistent Distance (unregular) (1 Consistent Distance (regular.
inconsistent Direction (unregular) (d1). Consistent Direction (regular)
-10
Carsl
Camd
Osend
(82). Incomsistent Distance
(b2). Consistent Distance
(62). Inconsistent Direction
(02). Consistent Direction
300
Freguency
16
Frequency
130
Frequency
Prequency
10
100
230
130
60 04 06
03
24
0-
0.8
0.3
06 DB
02
E4
0.
0.8
Deference of Distance
Dfference of Distance
Diffrennce Directon
Differncce of Directin
Figure 2: Toy examples to show the consisten/incooniitent differences (including distance values and distance directions). Each column is data distribution and the corresponding difference distri- bution of distances (including distance values or distance directions).
we write matrices, vectors, and three-order tensors as bold uppercase characters, bold lowercase characters, and bold calligraphic uppercase characters, respectively. ||>]|| and ||æ||| denote the 4 norm and f2-norm of the vector æ, respectively.
Supervised Metric Learning. Here the pairwise similarities of training data annotated by hu- mans, and the central problem how to learn a distance metric or feature representation that faith- fully reflects the pairwise similarity between each pair of instances. Both linear (Weinberger al., 2006; Xu et al., 201 8) and nonlinear approaches (Chen et al., 2019; Liao et al., 2023) learn generic feature representation &0 IT R (where h is the feature dimensionality), and the correspond- p( which measures the similarity between
ing learnable distance is de (x, ②) = l/<p(x) instances æ, E R. The basic learning objective is reduce the distance (ar, @)) and are similar and to enlarge it if æ and o are dissimilar. To alleviate the overfitting issue caused by the strong nonlinearity of yp implemented by deep neural networks, various loss functions (e.g., Augular loss (Wang al., 2017) and multi-tupple/NNpat loss (Sohn, 2016)) and metric forms (e.g.. multi-local metrics (Ye et al., 2016; 2019)) have been proposed to construct the generalized data pair containing three more data points, which successfully improve the diversity of training data. The regular- ization technique is another way to reduce overfitting, and recent works have further introduced adversarial training (Chen et al., 2018: Duan et al., 2018) neighborhood consistency (Cho al., 2010; Tian et al., 2019; Wei et al., 2021) to constrain the hypothesis space of learning algorithms.
Unsupervised Supervis)) Contrastive Learning. This task has similar training phase (i.e., considering the pairwise relationship) with metric learning. Existing noise contrastive esti- mation (NCE) loss based methods usually have two critical components: instance discrimination to generate negative pairs (Wu et al., 2018; Dosovitskiy et al., 2014) and data augmentation to generate positive pairs (Chen et al., 2020; Jiang al., 2020). Contrastive learning reduces the distance dep (x, x x² for positive pair (x. x*) and enlarges the distance (x, Fzy for negative pairs {ax, 2n, }} where n the (negative) batch size. Many works further enriched the con- trast information within the positive pairs based on different perspectives such as adversarial per- turbation (Kim et al., 2020a; Jiang et al., 2020) and wttiiiiiewwwiiiimoeeeeee
learning (Tian et al., 2020b; Sun et al., 2022). Since and Øs, might be semantically similar, yet they undeservedly pushed away from each other, recent works put forward to correct/reduce the false negative pairs by conventional techniques such as uunlaele learning (Chuang et al., 2020) and pseudo- labeling (Zheng et al., 2021). Distance regularization approaches have also been proposed to reduce the inconsistency of distance distributions (Wei al., 2021) constrain the reliable range of distance values (Chen et al., 2021).
METHODOLOGY
In this section, we formulate our proposed regularizer and the corresponding learning objective.
3.1 MOTIVATION
We start with toy examples intuitively understand the importance of the consistent differences. In the first case, we generate data points from three Gaussian distributions, where the parameters of the three distributions are configured with different sets of means and covariances. Here, the
3

---

Published as a conference paper at ICLR 2024
inter-cluster distances more consistent in Fig. 2(b1) than in Fig. 2(a1). In the second case, we further generate data points from different distributions to form clusters in different directions. The direction distributions in Fig. 2(d1) (i.e., the single direction) are more consistent than those in Fig. 1(c For the two cases above, we record the corresponding difference distributions of distance values or distance directions (as shown in Fig. 2(a2)- Meanwhile, sample additional data points from the same distribution to collect the test data.
The Strong Consistency Promotes the Stable Generalizability. In the above Fig. 2, all the dif- ferent settings can successfully distinguish clusters from each other, i.e., the training data is always discriminated well. However, the inconsistent and consistent cases have different generalizability. When the distances/directions are more consistent (i.e., the differences in Fig. 2(b2) and (d2) are close to zero), the clusters become more uniformly distributed, so that the classification reliability on the unseen test data can be improved (see the corresponding test accuracies in Appendix). Our theoretical results also reveal that the consistent distances promote stable generalizability. This is a critical property of a reliable similarity metric, but it is neglected in most existing approaches, propose to explicitly characterize this property with cenerallpurpoos regularizer.
3.2 FORMULATION
As the above issue of inconsistent difference may exist in both supervised metric learning and unsu- pervised contrastive learning tasks, here we first show that their loss functions can be simply unified.
A Unified Form of Supervised and Unsupervised Losses. For the widely used (n+ 1)-tuleet/Npair loss in metric learning and the NCE loss in contrastive learning, based on the given training set _ f Rm i 1, ...... } (m the data dimensionality and N is the sample size), we aim ... to learn a generic feature representation & Rm + Ra with the following empirical loss
d-de //7
Cemp (4p; 1
E. ( log
d xt 1// + Σz= dh (r. (o, 2/7
where 7 0 is temperature parameter, and the anchor instance o is sampled from X. Here x+ is directly obtained from the perturbed x for the unsupervised (contrastive learn- ing) case, or it is chosen randomly from the intra-class set 2*+(x) = {=1yz= = V/ z E 2 [((x]}] for the supervised (metric learning) case. Correspondingly, the mini-batch instances (as, Tb....., Lo.. are directly selected from 2') 2 {{x} for the unsupervised case, or from the inter- class set 2' (x) = {2 Vx * Ye, = E 2'} for the supervised case (y/ is the class label of æ).
Minimizing the above Eq. (1) will constrain the first-order difference V(61 (z. 2)
Fasture Vecten Dallennse
Foeward Plase
(xxx)
Rachward Phase
(Φ)) to be consistent with the corresponding (pseudo) supervision of the two instances and ã because the distance value (æ, T) is ac- tually the f2-norm of the first-order difference (x, 2). Here we want to reduce the incon- sistency among these first-order differences (as shown in Fig. 3), so we need to further consider the higher-order difference
Empirical
Risk
Chesso Clasee
Assett
Encoder
Hligher Order
Differance
Stared Parameters
Afloxt
Encodee
Stared Panomators
Reps
Encodor
Flast-Order
2
(2, (2)
Difference
Differrnce Alignment
Inpot
Output
Regularizatinn
where (xr, 3) and 2z are two given data pairs, and we use a vector-valued even function (()): R + R measure such higher-order difference, namely G(V(2)
Figure 3: The overall framework of our proposed
difference-aligned similarity learning.
(2) & (æ, a. 2, 2)). Then we formulate our regularization term in detail.
Difference Alignment Regularization. For training examples of 2", we have C2 first-order differences. However, as the intra-class differences are already limited small values by the empir- ical loss, we only need to consider the inter-class differences (v(') (z,, 22,,,;)ee 2;)hiissss tu Cor- respondingly, when further consider the higher-order differences, we calculate the following
4

---

Published as a conference paper at ICLR 2024
Algorithm Solving Eq. (8) via SGD,
Input: training set 2 = (e. J step size 7 > 0; regularization parameter À 0; batch size
e N4; randomly initialized ( maximum iteration number T. For from to T':
1). Uniformly pick (n + instances ( 1 from 2
2). Compute the gradients Lemp( (2 Jy=u) and Ratign (4;; ( Jy=o) (namely
e4p: ) and aalignn (4:; (ab; )==0)/ /0kp respectively);
3). Update the learning parameter:
sp (11-1)) m(@eemp (as,, /0=0 Ad (pp: (Eo, );=0)/Oup);
End
Output: the converged (
cumulative sum of all cases:
EisiCiSN ISKCISN, (I.)) /n g ( 2, wky (x,, mi))
(3)
We define the cross-total totl--vaiaaiionn (CTV) norm on a matrix M E R (H E N as
M = En 9(M, - M&),
(4)
where Mj is the j-th column of M, and can rewrite Eq. (3) to a value of CTV norm:
E Eigieisn (ISKEsN
(Tiy Fg Dk mn)
v 1NN-1: (TN-1. 2N) =2 - ..... V (a,, (xr,
(5)
ISKIS
Here the even function G (2) (I,, Ry F2 x3) = C (ats 3i, Bú, xj)) and each element will be twice calculated in Eq. (3), so the corresponding CTV norm has additional coefficient 2. Furthermore, we have the following theorem to ensure that Eq. (4) is a strictly defined norm that satisfies the non-negativity, homogeneity, and triangle property (the proof is given in Appendix).
Theorem 1. The function || llev R + R is a strictly defined norm if and only the measure function ((): Rdth + IR is strictly defined norm.
In practice, we can use the popular fnoormm or l2-norm to implement the function g. However, the calculation of Eq. (5) involves O(N²) vectors which can be computationally costly for large- scale data. Instead we can consider the stochastic form of the regularizer w.r.t. a mini batch (aby .b....,, Lo. }, where is the batch size and 2h, denotes the anchor instance in Eq. (1) for simplicity. Then we define the following difference alignment regularization (DAR)
Raliga (pp: 2¹) = (bb fll (1) (I Lb, Db.),..., (1) (aX,, Ib.r))
(
(6)
and we can easily get that Ralign (4p: 2) = kIll & (x; DigissssN llav where K > 0 is independent of the train data 2 and the feature representation sp implies that minimizing Eq. (5) and Eq. (6) is mathematically equivalent (see Appendix for more details).
Objective & Optimization. We integrate the above regularizer in Eq. (5) into the conventional em- pirical loss in Eq. (I), and obtain the final learning objective of our sifference-aligneer similarity
learning (DASL):
min (F(() = Lemp 2¹) + RRalggn (sp; 2'},
PER
where À > 0 is a trade-off parameter and H. the hypothesis space of the learning parameter. Then are able to optimize our learning objective Eq. (8) in stochastic way (as shown in Fig. 3). We need to specify the stochastic terms of both Cemp(yps 2") and Ratign (4p: 2 for a given mini batch. Here the traditional empirical loss Lemp 2.) already has its stochastic form (emp (æs, }
5

---

Published as a conference paper at ICLR 2024
and our DAR also has its stochastic form Ralign (4;; (wb; 12-0) of which the computational complex- ity only depends on the batch size !. Therefore, each iteration, we can use the stochastic gradient instead of the full gradient for fast computation (see the running time comparison in Appendix).
The detailed iteration steps based on stochastic gradient descent (SGD) (Reddi et al., 2016) are summarized in Algorithm 1, and we have the corresponding theoretical result (in Appendix) to investigate the convergence behavior of iteration points cp"), ,g!!) ...., ....,, obtained by Eq. (7).
4 THEORETICAL ANALYSES
In this section, we theoretically investigate the robustness and generalizability of our method.
Model Robustness. For given data pair (ær, .). our model prediction is the distance value Ap(2;, )). which measures the similarity between x and S. Therefore, we would like to analyze the stability of such distance function learned from DASL in Eq. (8) to investigate the robustness of our method. Intuitively, for two given data pair (ær, 3) and (z, 2), our regularizer Ralign has al- ready reduced the higher-order difference, so the inconsistency between the first-order difference V (az, 3) and (( 2) (as well as the divergence between (a and du 2)) should be bounded. To be more rigorous, we have the following theorem to reveal such an upper bound.
Theorem 2. Suppose that the instances x, ã, z, and & are independently sampled from the same distribution as the training set X. Then, for any feature representation yp learned from the objective F(()), we have that with probability at least 1 5,
(ar ))- d1p(2, ()1 <(())[ @0|2 + || 2112) max 0)1,, 2 (ln2)))2)))
where E(A) L Low 2))))))0; is monotonically decreasing w.r.t the regularization parameter X, and the constant L > 0 is independent of & and X.
From the above Eq. (9), can clearly observe that the distance error bound is first affected by the original distances || - 2 Ø112 and || - -2l|2, and it gradually decreases with the increase of the sample size N. This is consistent with our intuitions and empirical knowledge. Meanwhile, with the proposed regularizer, such a bound can be further reduced, and the increased regularization parameter will shrink the bound and make the prediction more stablc/robust w.r.t. the input data.
Generalization Error. We would like to further prove that our learning algorithm also tightens the generalization error bound (GEB) (Chen et al., 2021) of the conventional similarity learning approach. As we know, GEB usually has convergence rate of O(L/VN) for an empirical risk minimization model. Here we do not investigate the convergence rate as a function of sample size, but we show that tightened GEB benefits from the regularization Raiign to validate the effectiveness of our method. Specifically, for the underlying data distribution G, we denote the expected risk Llp;: 2) =Eqe, t~ [C(pp: (N, and discuss how far it is from the empirical risk [((p).
Theorem 3. For any yp learned from the objective F(yp) and any given constant ô (0, 1), we have that with probubility at least 5,
[((p)) [((p; S ((A))(n))lg(1 + max(d E) 2TVVVnnn///00 (222))2222/22))
(10)
c² where B(A) = (C 2/C)/> monotonically decreasing W.r.l. À and w(n) log
n +
monotonically decreasing w.r.f. n. Here the constant C > 0 is independent of 4p and X.
From the above result in Eq. (10), easy to observe that the error bound dominated by two main aspects. First, the error bound gradually decreases with the increase of the sampling number N as well as the batch size n. This is consistent with the observations in existing works (Saunshi et al., 2019). More importantly, we can find that such an error bound becomes tighter as À increases, and thus the regularization term Ralign can accelerate the empirical risk convergence to the expected risk. Therefore, Theorem 3 shows that our method successfully improves the generalization ability of conventional similarity learning algorithms.
Specifically, Lemp (sp: (11,, 1 = logle dd a0 1/7 /(c -dp(top -d + Eソ -aplos a 1,7173 and Raiign (yp; {as, 10) [ (Ibb )..).... v(61 (arby Fb, ), ..... v (abas Fb.. ]1lav.

---

Published as a conference paper at ICLR 2024
00 Untegularized CL on STL-10
01 Reggulbrizedd CL jour DASLI on STL/10
(c), Unreguiarize ledd CL on CIFAR-10
(1). Reguiarieed CL (our DASL) on CIFAR-16
-8
-00
Figure 4: The t-SNE visualizations of our DASL and the conventional (unregularized) contrastive learning (CL) method on STL-10 and CIFAR-10 10 datasets, where we can clearly observe that our method obtains the better clustering results than the unregularized method.
EXPERIMENTAL RESULTS
In this section, we show experimental results on real-world datasets to validate the effectiveness of DASL in both the supervised metric learning and unsupervised contrastive learning tasks. We first provide ablation studies and visualization results. Then, we compare our method with existing state-of-the-ar methods. Both the training and test processes are implemented on Pytorch (Paszke et al., 2019) with TeslaV100 GPUs, where the regularization parameter À is set to 0.5. The dimen- sionality h and the parameter y in Eq. (1) are set 512 and 0.2, respectively. The hyper-parameters of compared methods are set to the recommended values according to their original papers.
5.1 ABLATION STUDIES & VISUALIZATION RESULTS
In this subsection, we would like to first investigate the usefulness of our proposed regularizer in both supervised and unsupervised scenarios. Here we use the typical fn-norm (considering its smoothness in the optimization) to implement the measure function G() in our regularizer.
Supervised Metric Learning. We employ different backbones (BN-Inception (Ioffe & Szegedy, 2015) for Npair (Sohn, 2016), and Re (He et al., 2016) for Prox- yAnchor (Kim et al., 2020b) and MetricFormer (Yan et al., 2022)) to validate the effectiveness of our approach in various settings. We conduct experiments on CAR (Krause et al., 2013) and CUB (Welinder et al., 2010) datasets and record the test accuracy of compared methods (with 500 epochs, learning rate 10-3, and batch size 512 (Zhou et al., 2021; 2022a)) in Tab. 1. We can observe that our method can work well with all the three baseline methods. Our DASL obtains relatively stable performance in various cases with different embedding sizes. When we increase the regularization parameter À from 0.1 to 0.5, the corresponding recognition accuracy rates are also improved, which clearly validates the significance of our regularization term.
Table I: Classification accuracy rates of baseline methods and our method on CAR and CUB datasets (feature em- bedding sizes are 128 and 512).
METHOD
CAR
CUB
128-dim
512-dim.
128-dim.
512-lim.
RAD1 1 R388
R001
R08
R001
R8s
RMI
R48
Njuin(RN) win Reg. (A=
68.36
86.01
82.37
95.12
58.12
78.72
65.58
90.82
DASSLNNpaiIBNHHAA DASL MAsLLNNpir(BBHH((( 0055)
88.281
80.351
66.33
70.381 90.361
89.120
96.307
62,021
82.301
69.121
92.531
ProuA.1550 wlo Keg. (A DJ
69.24
87.86
97.86
62,12
69.72 92.41
DASI. [PoxA -011
69.24
583661
98.23
80.951
11.131992.771
DASL||PoxAA +[A 0.0)
70.381 91.211 92.31 98.907
63.137
82.171
73,967 94.211
F.R5) wio Rey 0)
74.42 92.53
DASI (MarieFF(A=O..)
73.561
91.121 91.56 97.531
70,861
86.251 74.551 92,977
DASL[MeendP+(A=0..00 0.5)
73.561 92.321 92_127 98.411 72,351 88.281 75.511 93.311
Unsupervised Contrastive Learning. Here we only adopt the ResNet-50 backbone for all com- pared methods. In Fig. we show the classification accuracy rates of all compared methods on CIFAR-IO (Krizhevsky et al., 2009) and STL-10 datasets (Coates et al., 2011), where we can ob- serve that our method consistently improves the corresponding bascline results (SimCLR (Chen et al., 2020), HCL (Robinson et al., 2021), and BYOL (Grill et al., 2020)) in all cases. In order have a more intuitive understanding of the regularizer's effect, we conduct the i-SNE embed- ding (Van der Maaten & Hinton, 2008) to obtain the 2-dimensional data points which help us better understand the usefulness of our introduced new term. As shown in Fig. 4, DASL can successfully obtain more accurate separations than the baseline results.
5.2 EXPERIMENTS ON SUPERVISED METRIC LEARNING
Image Retrieval. Here investigate the capability of DASL with the popular retrieval tasks on CAR (Krause et al., 2013). CUB (Welinder et al., 2010), and SOP (Oh Song et al., 2016). The further compared methods are JDR (Chu et al., 2020), IBC (Seidenschwarz et al., 2021),
7

---

Published as a conference paper at ICLR 2024
(al, STL-10 with 100 epochs
[b). STL-TLL-L with 400 epochs
CIFAR-I0 with 100 epochs
(d), CIFAR-10 with 400 epochs
Accurncy
Accurcy
Accurccy
Accurwcy
30
ICL
RCL
RYOL
Brot
EVO.
2r01
BASLINCLO
DASLIPCLI
DA5UMCLI
30
DASLI810LM
DNSLI810L
385
46c
200 400
600
20C
E00
203
403
Batth
Sire
Batch Size
Batth
Size
Batch
Figure 5: Classification accuracy of all methods on STL-10 and CIFAR-10 datasets, where the (negative) batch size is from 32 to 512.
AVSL (Zhang al., 2022), and ContextSimilarity (Liao et al., 2023). In our experiments, all compared methods are incorporated into the ResNet-50 backbone to train end-to-end net- work as the final distance metries. We refer the combinations of our method with Npair loss and ProxyAnchor loss as DASL-NP and DASL-PA, respectively. The NMI and Recall@ R scores of all compared methods are shown in Tab. 2. From the quantitative results, we clearly observe that MetricFormer, ContextSimilarity, and our methods obtain the higher accuracies than other comapred methods. Compared with the two strong baseline methods, our DASL, can achieve either better or competitive NMI and Recall@R scores on the three datasets.
Table 2: Performance of all compared methods (with ResNet-50 backbone) on CAR-196, CUB-200, and SOP datasets. The best two results are bolded and underlined, respectively.
METHOD
CAR 196
CUB 200
SOP
NMI
R@1
R@4
R@B
NMI
R@1 1
R@4
R@B
NM1
R@0 1
R@P10
R 100
Spain 2016)
50
82.57
94.97
95. 95.92
69.53 53
64.52
85.63
91.15
91.11
76.21
88,43
92.08
Proxy A.kim et al, 20200)
75.72
87.71
95.76
97.86
72.31
69.72
87.01
92.41
91.02
78.39
90.48
96.16
[Du al. 2020)
70.56
84.86
94. 56
97.21
70:32
69.44
87.01
91.33
92.21
79.21
90.53
96.01
al. 2021)
74.82
86.1
96.21
98.21
74.01
70.32
87.61
92.72
92.61
81.42
91 91.32
95.89
AVSI. (7hong al, 2023)
25.86
91.51
97.02
9R
71.21
71.91
88.1
93.21
91.21
79.61
91.40
96.40
MetricF-YYance ol., 2022)
76.23
91.76
96.31
97.21
75.41
74.42
85.75
92.53
92.71
62.23
92.62
96.13
ContextS
76.32
91.80
97.14
98.41
74.01
71.91
88.82
93.42
92.61
62.63
92.56
96.74
DASL-NP (ours)
75.961
86. 86.34t
57.
98.871
73.521
69.63t
19.627
93.611
92.85t
79.21
93.21
97.86*
DASL-PA (ours)
77.321
92.311
97.821
98301
76.501
73.961
90.54
94,21
93.861
83,32
93.86
97.951
Face Recognition. WebFace (Yi et al., 2014) is em- ployed as the training set while the test sets include AgeDB30 (Moschoglou et al., 2017), CFP-FP (Sengupta et al., 2016), and MegaFace (Kemelmacher- Shlizerman et al., 2016). The batch size set to 256 and embedding size to 512 for all methods (with ResNet-50 back- bone). The compared methods are dif- ferent regularized versions of Sphereface (Zhang et al., 2020) and Arcface (Deng et al., 2019). As shown in Tab. 3, the two cosine similarity based approaches can always perform better than the orig- inal softmax. We also observe that our regularizer improves the performance of both Sphereface and Arcface in all cases. For example, on MegaFace with 10th distractors, the accuracies of Sphereface and Areface are boosted by 1.44% and 1.20%, respectively.
Here CASIA-
Table 3: Accuracy rates of all compared methods on AgeDB30, CFPFP, and MegaFace datasets.
METHOD
Fice Vesification
Face Ideutifcctiion (MegnPace)
Age.
Maga-10
Mcgs-10
Mego-10*
Sodumax
91.30
93.19
80:43
87.11
92.83
Sph. Reggt in al. 2017)
93.42
94.30
88. 38.38
92.86
95.93
Sph. -SEchhann et 2000)
93.45
94.39
92.70
95.88
An RegiDnng et al, 2019)
93.93
94.77
90.68
94.34
96.83
Arc ShcC/zanng et al., 2020)
93.82
94.91
90.91
94.56
96.95
DASL -APR
94.211
95.212
89-661
93.96/
96.25"
DASI (Arc +DAR)
95.331
96.157
91.351
95.767
97.28"
5.3 EXPERIMENTS ON UNSUPERVISED CONTRASTIVE LEARNING
Image Classification. We employ ResNet-50 as our backbone and implement our method based on SimCLR (Chen al., 2020) and SWAV (Caron al., 2020), and the corresponding results are denoted as DASL (cluster-free) and DASL (cluster-used) respectively. We train our method on ImageNet- -00 and ImageNet- JK (Russakovsky et al., 2015), and compare it with existing representative approaches including contrastive multiview coding (CMC) (Tian et al., 2020a), hard negative based contrastive learning (HCL) (Robinson et al., 2021), prototypical contrastive learning (PCL. (Li et al., 2021), BYOL (Grill et al., 2020), and meta augmentation (MetAug) (Li et al., 2022). Then we also im- plement our method on the popular ViT-B/16 backbone and compare it with three more methods including DINO (Caron et al., 2021). iBOT (Zhou et al., 2022b), and PQCL (Zhang et al., 2023). We conduct comprehensive evaluations by recording the classification accuracy rates of all methods obtained with the fine-tuning linear softmax (i.e., the Top-1 score and Top-5 score of linear probing) and the K-NN classification (here h = 8). From Tab. 4, we can observe that our method successfully improves the SimCLR and HCI. by least 4% in different cases of batch size the two datasets.
8

---

Published as a conference paper at ICLR 2024
Table 4: Classification accuracy (%) of all methods on ImageNet-100 and ImageNet-IK datasets. The batch sizes are set to 1024 and 512 for ResNet-50 and ViT-B/16 backbones, respectively, Here the best and second- best results are bolded and underlined, respectively.
ImageNet- 100
ImigeNett IK
METTIOD
#Arch.
100 epochs
400 epochs
300 epochs
800 epochs
K-NN
'Top-1
Top-5
A-NN
Top-1
Top-5
k-NN
Top-1
Top-5
A-NN
Top-1
Top-5
RRChenn or 2020)
70.6
75.2
64.2
87.9
69.1
Res.50
BYOL(Gnil 2020)
90.1
66.9
71.2
90.5
67.2
73.2
91.5
CMC et 2020g)
57.7
60.2
79 2
71.6
73.6
92.
63.2
68.2
87.2
67.
71.2
89.9
Res.50
2021)
Sw.AViCaroe 2020)
58.2
61.0
79.4
72.1
75.8
92.9
65.4
73.1
91.2
65.7
75.3
91.5
Res.50
HCL[Rooinsoon 2023)
55.9
60.8
79.3
70.2
74.6
92.3
64.2
71.2
91.2
67.2
11.7
90.7
Res.50
MetAng( es 2023)
59.2
61.1
79.4
69.8
75.6
93.2
65.4
74.2
91.1
67.8
76.0
92.9
Res.50
DASL (cluster-fie free)
60.5
65.2
79.8
73.5
76.6
93.9
68.3
72.7
91.9
68.2
76.7
92.9
Res.50
(cluster- used)
80.1
94.3
89.I
74.8
92.4
69.1
76.6
Res.50
BYOLIGall et 2020)
57.2
62.8
77.9
72.1
76.9
93.8
66.6
71.4
91.2
68.2
74.2
92.8
VIT-B/16
SwAAiCarcn et 2020)
60.1
62.5
80.5
14.1
77.8
94.2
64.7
71.8
911
69.
75.6
91
VIT R/16
DINO et 2021)
61.5
67.5
8188
78.2
79.2
95.5
72.3
76.1
92.4
76.2
78.2
94,2
VIT-8/16
iBOT al., 3022b)
61.5
68.2 2
82-2
77-5
78.5
95.2
71.5
75.0
91.9
75.2
76.0
92.6
VIT-B/16
PQCLiZhang et al, 2023)
62.3
66.7
82.3
78.5
79.5
94.8
70.8
76.5
91.9
78.
76.9
93.0
B/16
DASL (cluster-Iree)
62.5
69.5
SLT
754
80.1
96.1
71.5
77.8
92.7
76.2
79.2
94.5
VIT-B/16
DASL (cluster-used)
82.8
79.3
82.3
96.8
72.9
76.8
933
77.9
79.9
96.3
VIT-8/16
Based on the powerful ViT-B/16 encoder, our method also consistently boosts the baseline meth- ods and outperforms the three state-of-the-an art methods (DINO, iBOT, and PQCL) in most cases.
Sentence Embedding. For the BookCorpus dataset which includes six sub-tasks MR, CR, SUBJ, MPQA, TREC, and MSRP (Kiros et al., 2015), we follow the experimental settings in the baseline method quick-thought (QT) (Lo- geswaran & Lee, 2018) to choose the neighbor- ing sentences as positive pairs. Then, we fur- ther compare our DASL, with consistent con- trast (CO2) (Wei et al., 2021), and uncertainty representativeness mixing (UnReMix) (Tabassum et al., 2022), and the corresponding average classification accuracy rates are shown in Fig. 6. Our method improves the classification ac- curacy of baseline method QT by at least 2% on most classification benchmarks. This clearly demonstrates that the consistent difference is not only useful in the image data but also in the text data, and our method is a good solution to utilize such a property for model training.
DCL
EOO
BASE Juavi
MR
CR
500
MPDA
FEC
NSR
Figure 6: Classification accuracy rates (%) of all compared methods on the BookCorpus dataset including six text classification tasks.
For the STS dataset (Agirre et al., 2016), we fol- low the common practice in SimCSE (Gao et al., 2021) to use the pre-trained checkpoints of BERT (Devlin et al., 2018), and we further com- pare method with information minimiza- tion contrastive learning (InforMin-CLL (Chen et 2022), misCSE (Klein & Nabi, 2022), and smoothed contrastive learning (SCL) (Wu et al., 2022). As we can observe from Tab. 5, DASL obtains considerable improvements on the baseline method SiMCSE. Mcanwhile, our method can outperform the other two representative methods misCSE and InforMin in most cases. Our DASL also achieves the best average score in all compared methods.
Table 5: Classification accuracy rates (%) of all compared methods on the STS dataset including five tasks and the corresponding average scores.
METHOD
STS12
.SS13
STS14
STS15
STS16
Aver.
SinCSR
68.69
82.05
31.15
79.30
76.84
PCL
72.74
83.36
76.05
83.07
79.26
78.90
TaCMin
70.23
83.48
75.51
81.72
79.88
78.16
miCSe
71.71
83.09
75.46
83.13
80.22
78.72
SCL
72.86
84.91
76.79
84.35
81.74
80:13
SACL (ours)
72.70
81.53
78.32
85,55
82,54
80,73
6 CONCLUSION
In this paper, we investigated the issue of inconsistent representation differences in traditional sim- ilarity learning. We proposed a novel DAR regularizer to encourage the learning algorithm seek for the close differences for data discrimination. To this end. we built a new CTV norm to mea- sure the divergence among representation differences, and we showed that it can be easily solved by SGD. Our proposed DAR is quite generic technique and we can easily deploy it in both met- ric learning and contrastive learning tasks with negligible computation overhead. We conducted extensive theoretical analyses that guarantee the effectiveness of our method. Comparison experi- ments on real-world datasets across multiple domains indicated that our learning algorithm acquires more reliable feature representations than state-of-the-art methods. As this paper merely focused on the unsupervised and fully supervised cases, in the future we plan to explore the inconsistent representation differences in semi-supervised and weakly supervised scenarios.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
M.S. was supported by JST CREST Grant Number JPMJCRI8AZ and grant from Apple, Inc. Any views, opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and should not be interpreted as reflecting the views, policies or position, either expressed or implied. of Apple Inc. J.Y. was supported by the National Science Fund of China under Grant No. 62361 166670.
REFERENCES
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. Semeval-2016 task I: Semantic textual similarity, monolingual and cross-lingual evaluation. In International Workshop on Semantic Evaluation, 2016. 5.3
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Advances in neural information processing systems (NeurIPS), pp. 1401-1413, 2020. 5.3, 4,
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision (ICCV), 2021. 5.3,
Shaobin Chen, Jie Zhou, Yuling Sun, and Liang He. An information minimization based contrastive learning model for unsupervised sentence embeddings learning. In International Conference of Computational Linguistics (COLING), 2022. 5.3
Shuo Chen, Chen Gong, Jian Yang, Xiang Li, Yang Wei, and Jun Li. Adversarial metric learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
Shuo Chen, Lei Luo, Jian Yang, Chen Gong, Jun Li, and Heng Huang. Curvilinear distance metric learning. In Advances in neural information processing systems (NeurIPS), 2019.
Shuo Chen, Gang Niu, Chen Gong, Jun Li, Jian Yang, and Masashi Sugiyama. Large-margin trastive learning with distance polarization regularizer. In International Conference on Machine Learning (ICML), pp. 1673-1683, 2021. 2,
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597-1607, 2020. 1, 2, 5. 5.3, 8
Minsu Cho, Jungmin Lee, and Kyoung Mu Lee, Reweighted random walks for graph matching. In European Conference on Computer Vision (ECCV), 2010. 1.2
Xu Chu, Yang Lin, Yasha Wang. Xiting Wang, Hailong Yu, Xin Gao, and Qi Tong. Distance metric learning with joint representation diversification. In International Conference Machine Learning (ICML), pp. 1962-1973, 2020. 5.2, 5.2
Ching Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020. 2
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature lcarning. In International conference on artificial intelligence and statistics (AISTATS), pp. 215-223, 2011. 5.1
Corina Cortes and Vladimir Vapnik. Support-vectur networks. machine learning. Machine Learn- ing, 1995. 6
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3, 5.2
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805, 2018. 5.3
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimina- tive unsupervised feature learning with convolutional neural networks. Advances neural information processing systems (NeurIPS), volume 27, pp. 766-774, 2014. 1,2
Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou. Deep adversarial metric learn- ing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.0882 2021. 5.3
Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 33:2 1271 21284, 2020. 5.1, 5.3,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016. 5.1
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Similarity-Based Pattern Recognition: Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, October 12- 14, 2015. Proceedings Springer. 2015.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learnin (ICML), pp- 448-456, 2015. 5.1
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 2020. 1,2
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Mahmut Kaya and Hasan Şakir Bilge. Deep metric learning: A survey. Symmetry, 11(9):1066, 2019. 1
Ira Saaellmaahhr-Shizeemaaa Steven M Seitz, Daniel Miller, and Evan Brossard. The megaface benchmark: million faces for recognition scale. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4873-4882, 2016. 5.2
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning.
Advances in Neural Information Processing Systems (NeurIPS), 33, 2020a. 1,
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3238- 3247, 2020b. 1, 5.2,
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip thought vectors. Advances in neural information processing systems (NeurIPS), 18:3294-3306 2015. 5.3
Tassilo Klein and Moin Nabi. miese: Mutual information contrastive learning for low-shot sentence embeddings. arXiv preprint / 1.09928, 2022. 5.3
Xuan Kou, Chenghao Xu, Xu Yang, and Cheng Deng. Attention-guided contrastive hashing for tailed image retrieval. In International joint conference on artificial intelligence (ICCAI), 2022. 1
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ine-grained categorization. In 3dRR, 2013. 5.2
11

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 5.1
Jiangmeng Li, Wenwen Qiang. Changwen Zheng, Bing Su, and Hui Xiong. Metaug: Contrastive learning via meta feature augmentation. In International Conference Machine Learning (ICML), 2022. 5.3, 4
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representa- tions (ICLR), 2021. 5.3, 4
Christopher Liao, Theodoros Tsiligkaridis, and Brian Kulis. Supervised metric learning to rank for retrieval via contextual similarity optimization. In International Conference on Machine Learnin (ICML), 2023. 5.2,
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In International Conference on Learning Representations (ICLR), 2018. 5.3
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, the--wild age database. In CVPR workshops, 2017. 5.2
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4004-4012, 2016. 5.2
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems (NeurIPS), 32, 2019.
Carl Rasmussen and Zoubin Ghahramani. Occam's 's razor. Advances neural information process- ing systems (NeurIPS), 13, 2000. 1
Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In International Conference on Machine Learnin (ICML), pp. 314-323, 2016. 3.2
Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations (ICLR), 2021.
5.1, 5.3,
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision. 115(3):211 22522,221122,, 2015. 5.3
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Con- ference on Machine Learning (ICML), pp. 5628 5628-5637, 2019. 4
Jenny Denise Seidenschwarz, Ismail Elezi, and Laura Leal-Taixe. Learning intra-batch connections for deep metric learning. In International Conference on Machine Learnin (ICML), 2021. 5.2, 5.2
Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter confer- ence on applications of computer vision (WACV), 2016. 5.2
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems (NeurIPS), 19:1877-1865. 2016. 2, 5.1
12

---

Published as a conference paper at ICLR 2024
Fan- Sun, Jordan Hoffmann Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference Learning Representations (ICLR), 2020. 6
Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video- language pre-training with multimodal temporal contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 2
Afrina Tabassum, Muntasir Wahed, Hoda Eldardiry, and Ismini Lourentzou. Hard negative sampling strategies for contrastive representation learning. arXiv preprint arXiv:2206.01 197, 2022. 5.3
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European
Conference on Computer Vision (ECCV), pp. 1-18, 2020a. 5.3, 4
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola, What makes for good views for contrastive learning. Advances in Neural Information Processing Sys- tems (NeurIPS), 33, 2020b. 2
Yurun Tian, Xin Yu, Bin Fan, Fuchao Wu, Huub Heijnen, and Vassileios Balntas. Sosnet: Second order similarity regularization for local descriptor learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 1,2
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 5.1
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. In IEEE International Conference Computer Vision (ICCV), 2017. 2
Chen Wei, Huiyu Wang, Wei Shen, and Alan Yuille. Co2:
Consistent contrast for unsupervised visual representation learning. In International Conference on Learning Representations (ICLR), 2021. 1, 2, 5.3
Kilian Q Weinberger, John Blitzer, and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. In Advances in neural information processing systems (NeurIPS), pp. 1473-1480, 2006. 1,2
Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010. 5.1, 5.2
Xing Wu, Chaochen Gao, Yipeng Su, Jizhong Han, Zhongyuan Wang, and Songlin Hu. Smoothed contrastive learning for unsupervised sentence embedding. In Proceedings of the 29th Interna- tional Conference on Computational Linguistics (ACL), 2022. 1, 5.3
Zhirong WU, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. IEEE Conference on Computer Vision and Puttern Recog- nition (CVPR), pp. 3733-3742 2018. 2
Eric P Xing, Andrew Y Ng, Michael I Jordan, and Stuart Russell. Distance metric learning with application to clustering with side-ing information. In Advances in neural information processing
systems (NeurIPS), volume 15, pp. 12, 2002. 1
Jie Xu, Lei Luo, Cheng Deng, and Heng Huang. Bilevel distance metric learning for robust image recognition. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 2
Xinyi Xu, Cheng Deng, Yaochen Xie, and Shuiwang Ji. Group contrastive self-supervised learning on graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 1, 6
Jiexi Yan, Erkun Yang, Cheng Deng, and Heng Huang. Metricformer: A unified perspective of correlation exploring in similarity learning. Advances in neural information processing systems (NeurIPS), 2022. 5.1, 5.2
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (KDD), 2015. 6
13

---

Published as a conference paper at ICLR 2024
Han-Jia Ye, De-Chuan Zhan, Xue-Min Si, Yuan Jiang, and Zui-Hua Zhou. What makes objects similar: A unified multi-metric learning approach. Advances in neural information processing systems (NeurIPS), 29, 2016. 2
Han-Jia Ye, De-Chuan Zhan, Nan Li, and Yuan Jiang. Learning multiple local metrics: Global
consideration helps. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019. 2
Dong Yi, Zhen Lei, Shengeai Liao, and Stan Li. Learning face representation from scratch. arXiv preprint Arxivv1411. 1.993, 2014. 5.2
Yiming Ying, Kaizhu Huang, and Colin Campbell. Sparse metric learning via smooth optimization.
Advances in neural information processing systems (NeurIPS), 2009.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Advances in neural information processing systems (NeurIPS), 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning auto- mated. In International Conference on Machine Learning (ICML), 2021. 6
Pourya Zadeh, Reshad Hosseini, and Suvrit Sra. Geometric mean metric learning. In International Conference on Machine Learnin (ICML), 2016.
Borui Zhang, Wenzhao Zheng, Jie Zhou, and Jiwen Lu. Attributable visual similarity learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 5.2, 5.2
Dingyi Zhang, Yingming Li, and Zhongfei Zhang. Deep metric learning with spherical embedding. Advances in neural information processing systems (NeurIPS), 2020. 5.2
Shaofeng Zhang, Qiang Zhou, Zhibin Wang, Fan Wang, and Junchi Yan. Patch-level contrastive learning via positional query for visual pre-training. In International Conference on Machine Learmin (ICML), 2023. 5.3, 4
Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 2
Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. Deep robust clustering by contrastive learning. arXiv preprint 2030000008.03000 2020.
Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 4401-4410, 2021. 5.1
Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class- inassmnncemmetal learning. In IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 9046 90569056, 2022a. 5.1
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In International Conference Learning Repre- sentations (ICLR), 2022b. 5.3,
14

---

Published as a conference paper at ICLR 2024
APPENDIX
This part is the appendix of our manuscript. It includes the additional experiments and the mathe- matical proofs of theorems.
A ADDITIONAL EXPERIMENTS
A.I NUMERICAL. RESULTS OF FIG.
In Fig. 2, the 1000 points of each cluster (as the training data) are generated from the Gaussian distributions, and the corresponding distribution parameters are listed in Tab. 6. Meanwhile, we sample additional data points from the same distribution to collect the test data (200 points for each cluster).
We conduct k-NN both training and test data record the corresponding classification accuracy rates. The results are listed in Tab. 7, where we can clearly find that all cases have achieved the satisfied training accuracy. However, the cases of consistent differences can obtain better test results than the cases of inconsistent differences.
Table The distribution parameters (the mean vector and covariance matrix) of clusters in Fig. 2.
(al), inconsistent
(bl), consistent
(el), incensistent
(d1), consistent
0.76
Clusterl
N
0.75
Cluster2
M
N
J
JV
4
10
10
0.76
Cluster3
35
N
20
20
2
20
0.76
0.75
Table 7: Training and test accuracy rates of all cases (%).
(a1), Training/Tes
(bl), Training/Tes Test
(cl). Training/Test
(dl), Training/Test
Clusterl
99.1 99.1/75.3
19..1/85.6
98.5/85.1
98.5/85.1
Cluster2
99.2/ 76.3
08.5
99.1 /85.2
98.5/86.4
Cluster3
99.8/89.3
98.8/95.3
98.1/74.1
98.1/85.2
Average
98.8/89.2
98.6/81.5
98.4/85.6
A.2 EXPERIMENTS ON GRAPH EMBEDDING
We test our method on challenging graph data to further investigate the generalizability of our method. Here two types datasets considered, those the biochemical molecules and the so- cial networks, including DD, PTC, IMDB-B, IMDB-M, RDT-B, PROTEINS, NCI1, and MUTAG (Ya- nardag & Vishwanathan, 2015).
Here we use the representative method InfoGraph (Sun et al., 2020) as the baseline, and we follow the common practice for the downstream graph-level classification task on datasets. Note here we fine-tune support vecior machine (SVM) (Cortes & Vapnik, 1995) on the learned feature represen- tation to evaluate the final classification performance by using the 10-fold cross- vols-validation.. We split the dataset into the training, test, and validation sets at the proportion of 8/1/1 and report the mean classification accuracy with standard deviation after runs followed by a linear SVM classifier. The SVM is trained using tross-validation by the best validation performance. Our compared methods include HCL, GraphCL (You et al., 2020), JOAO (You al., 2021), and GroupCL (Xu et al., 2022).
training folds of data and the model for testing is selected
From the results shown in Fig. 7, we observe that our DASL consistently improves the baseline method InfoGraph in all eight cases. Meanwhile, compared with the other graph contrastive learning
15

---

Published as a conference paper at ICLR 2024
(a). Average Accuracy DD
(b), Average Accuracy PTC
0.65
0.85
Accuracy
0.8
Accuracy
0.6
0.75
0.55
0.7
InfoGraph HCL GraphCL J OAO GouuCC DASL
Info HCL GraphCLJ OAO @rropC DASL
(d). Average Accuracy on IMDB-M
(c). Average Accuracy on IMDB-B
0.54
0.8
0.52
Accuracy
Accuracy
0.5
0.75
0.48
0.7
+
0.44
InfoGraph HCL GraphCL OAO GroupCL DASL
InfoGraph HCL GraphCLJ OAO GroupCL DASL
(f). Average Accuracy PROTEINS
(e). Average Accuracy on RDT-B
0.8
0.95
T
Accuracy
0.9
Accuracy
0.76
0.85
0.72
0.8
0.7
InfoGraph HCL GraphCL OAO GroupCL DASL
InfoGraph HCL GraphCLJ OAO GroupCL DASL
(g). Average Accuracy on NCI1
(h). Average Accuracy MUTAG
0.84
0.95
0.82
Accuracy
0.8
Accuracy
0.9
0.78
0.76
0.74
0.72
InfoGraph HCL GraphCL JOAO GroupCL DASL
InfoGraph HCL GraphCLJ DAO GroupCL DASL
Figure 7: Accuracy rates of compared methods on traditional unsupervised graph embedding tasks including eight popular datasets.
approaches, method does well GraphCL, JOAO, and GroupCL. Furthermore, for most of the eight datasets, our method can outperform the three methods with a higher accuracy mean and lower accuracy variance.
A.3 VISUALIZATION OF THE TRAINING CURVES
In our experiments, we further record the values of empirical losses and regularization terms of the traditional methods (including the Npair loss with BN and the PrchyAnchor loss with ResNet-50) and our approach in each epoch.
We can find from Fig. that the loss functions traditional approaches can always decrease well and converge to stable values, but the inconsistencies among representation differences (i.e., the CTV norm values) are still large. After introducing the DAR term in our approach, the empirical loss can still converge to stable value. and more importantly the corresponding CTV norm is effectively reduced and controlled, so that the final classification performance is successfully improved (as shown in Tab. l).
16

---

Published as a conference paper at ICLR 2024
(a). Training loss of Npair loss
(b). Training loss of DASL
Loss value
Loss value
C CTV-norm
CTV-nom
loss valuee
2
loss value
1.5
1
0.5
100
200
400
300
iteration epoch
iteration epoch
(c). Training loss of ProxyA nchor loss
(d). Training loss our DASL
2.5
Loss value
Loss value
2.5
CTV-nom
CTV-norm
loss value
loss value
2
1.5
1.5
1-
0.5
0.5
100
200
300
400
100
200
300
400
iteration epoch
iteration epoch
Figure& 8: Training curves of the baseline methods and our method on the CAR dataset.
Table 8: Training time of the baseline methods and our proposed method (100 epochs, in hours).
CIFAR-10
ImageNet-
METHOD
512
1024
512
1024
SimCLR (Chen et al., 2020)
2.3
1.3
10.9
5.5
SWAV (Caron al., 2020)
2.6
1.7
11.5
5,8
DASL SimCLRR+DAR)
2.5
1.6
11.3
5.9
DASL (DAR)+DAR)
2.6
1.9
11.8
5.9
A.4 RUNNING TIME COMPARISON
In our regularization term, we adopted the f2-norm to implement the measure function G(-) of CTV norm. We would like to investigate if the efficiency of the learning algorithm will be affected by the calculation of CTV norm. Therefore, here we further provide experiments to record the training time of our method as well as the corresponding baseline method. Specifically, we use two NVIDIA TeslaV 0100 GPUs to train our method based on SimCLR and SwAV with 100 epochs, respectively. For each case, we set the batch size to 512 and 1024.
In Tab. 8, we can find that the proposed regularizer only brings in very little additional time con- sumption. This because the gradient calculation of CTV norm independent to the size training data, so the training time is still acceptable in practice use.
A.5 PARAMETRIC SENSITIVITY
Here we investigate the parametric sensitivity of the regularization parameter À in our method. Specifically, we change À in 50. and we record the classification accuracy of our method on STL-10 and CIFAR- 10 datasets (batch size=256/5 12/0224, epochs=100). Tab. 9 shows that the accuracy variation of our method is smaller than 2%. These results clearly demonstrate that the regularization parameters À is very stable within a given range. implies that the parametr of our method can be easily tuned in practice use.
17

---

Published as a conference paper at ICLR 2024
Table 9: Parametric sensitivity of X on the STL-10 and CIFAR datasets (%). Here changed
within [0.01,5] 5].
dataset (batchsize)
0.01
0.1
0.5
1.5
5
STL-10 (256)
75.8
76.5
77.2
76.9
76.2
STL-10 (512)
77.6
78.3
79.5
79.5
78.2
STL-10 (1024)
79.9
80.5
81.3
80.9
80.5
CIFAR-10 10 (256)
87.5
87.5
88.3
88.1
87.5
CIFAR-10 (512)
90.7
91.2
91.0
90.8
CIFAR-I0 10 (1024)
90.2
91.6
92.5
92.6
91.5
A.6 CONVERGENCE ANALYSIS
We have the following theoretical result to investigate the convergence behavior of iteration points (0)) (2) (T( obtained by Eq. (7).
Theorem 4. We assume that function F((p) has 8-bounded gradient (IVVF(())lz < 6) and let n --) ()(52)) and then for iteration points in Algorithm 1, have minosisT ([I 112) /VSAF/FFTTT where AF - and S is a Lips-
chitz constant such that ( S 4'112-
Notice that in the above theorem, the variables S, AF, and ő are all independent of T. The gradients of the iteration points of our final learning objective F(sp) will gradually decrease It means that the iteration points (p(1,, (2(), ......... ......!!)) will converge a stationary point of the learning objective F with a convergence rate O(1/VT), where T is the number of iterations.
B PROOFS
B. THE EQUIVALENCE BETWEEN EQ (5) AND EQ. (6)
According to the definition of CVT norm, we have that
Ralign( 2¹)
Egbs
(2by Tb.). ((1) (16,
Wby), ( (xb 1b..) |1mb
Σ
N c(++))-gg ( (o xz))
(20 (x)
(o
CN
Σ
c clx+1)-4g (22) (x; t. xx 211)
Cm-1
Igi<jsN. 1ScesN..skdsMM))HGbbb..
CV
c(++1)-4
Σ
v (2)) (I; (Ig tx
C++l
Vil)
(21, 22),
(m.
" Isicis )
(x;,
Cn+1
(11)
where the constant c(x+1) (n+1)-4
/C only depends on the batch size and sample size.
B.2 DISCUSSION ON THE TRIVIAL SOLUTION RALION (4p; 2) =
Here we would like to discuss our learning algorithm will obtain the trivial solution Ralign (4p; 2¹) O. We prove that there exists Xn such that AA (0, Ao). Ratign 2.) 0.
18

---

Published as a conference paper at ICLR 2024
where 6 E arg min Cemp (4p; 2)) aliga (4;: 2). To be specific, we suppose that
s E arg min Cemp (9;; 2²),
(12)
and by using the definition of empirical loss, we have that
( 2 > 0.
(13)
We let A(A) Cemp(12 (4;; 2) + Ralign (s;; 22), and we have that A(A) is always continuous w.r.. X, and thus have
lim A(A) = A(0) = Cemp(4pp 2))
(14)
and
lim Ratign Ralign('33 2 Ratign (48 2) Q. A-0
(15)
By the property of positive limitation, we have that there exists Ag > 0 such that AA (0, An)
Q
3Q 2
Ratiiga(P3 2) -
(16)
where Å is sufficiently small number satisfying RRalign 2) Q S 3. It clearly reveals that the trivial solution will not be incurred by our learning algorithm with the controllable regularization parameter X.
B.3 PROOF FOR THEOREM
Proof. For a matrix M E R the non-negativity of CTV norm can be directly obtained from the non-negative element (MM M&). For the homogeneity, we have that V/u E IR.
M 1ettv
Σ ISICKSH 9(uM; wMM)
Σ ISICHHH Hun(M Md))
Σ EiSRAKH M -Ma)
= [lats
(17)
where the third equation is based on the homogeneity of the measure function G(-). Then, for the triangle property, we have that VP, gaxff
+ Qliav
Σ nsjekgh 9((P, Q,) (P, + Qc))
Σ LisjegHH C((P; P.) + (Q, Qx))
Σ -IIjekKn g(P, Pk) + +QQ - Q+)
Σ 1SjCKHH C(P, - Pk) + Σ i5jckLL g(Q, - Qk)
Pllay + 1aav
(18)
which completes the proof.
B.4 PROOF FOR THEOREM 2
Lemma 1. For independent random variables tı .2,...,,t, tn E T and a given function w T - R,
if T 1,2, n), the function satisfies
(w(ti,..., t.......... tn) w(t.,... ((t,..........,t)) S Pi
(19)
then for any given > 0. it holds that P Pf[w(t,..... -
# s
2e 2e-2p² 2/² E2
19

---

Published as a conference paper at ICLR 2024
Proof. Firstly, denote that
Σ t.EE
dp(t,)),
(20)
and
E
dp(t, do (2,
(21)
N
where and T obey the same distribution with the instances in 2. Then have that
3
Σ
(t, 1) + dz( -
Σ
de
N
te (t..))
L
(br (br: f.y) - dup (r. m0)
N
|| - xt|12 - ||tr, Erllz) (rl1, + max (t, E),, 21)
(22)
N
where > is the Lipschitz constant of dip. Meanwhile, we have
Σ
do E
Σ do(t,
N
t
1EEE
de (t. 2) (x, T))
(23)
By Lemma I, we let that for all = 1, ...;.
I, 2p: )
| T1|2 l| l|r,, E. 112
max (t, 27))
Pi
(24)
N (|
that have
L Comp (p:
2112
-11,, E.112)
(t (t, E 27))
Cemplpp (p::)) Cemp (4p; 9)
(n
P
N
2.V
2c-2/1 2/²/ EN
2N 2Nte
2
2c
2c 2N 22 (1nn(2/5))(22N))
2e 1n(2/8)
6.
(25)
L. Cens ( [X] (||=0 /
where
and (lnn2//8)//2N)
N
means that have that with probability at least
Lamllp: 2))
||æce
T la
||t,, fr112) max (dx 21)
In (2/2/5)
(x. 2) - (t,
2.V
(26)
Similarly, for another data pair and obey the same distribution with the instances 2,, we have that with probability least 8.
I, Comp (vp:22)
[|2 E112 -It,,, Full2) + max (dx 21)
In (2/8)
Id (z.
2.V
(27)
20

---

Published as a conference paper at ICLR 2024
By combining the above two equations, we thus have
(dp(z, 2) dp (x, 3)1
1dlp [2, (t, (t, 1) dup ft. i) (x.
$ ddp (z, }) - d (t, f)] + [dp (t. t)
Lumpl (p) )
x 2|12 - ,,, Er + ||= - ø112 || Erl1)) + max 27)
x
in(2/8)
N
2.N
L Camp Lemple 12)))
(1|2: x + ||=
21 max (t, (t, EX))
In(2/5)
N
2NV
{(A) (1|2- 2 + >
2
122 + max (tt 21})
In(2/6)
(28)
N
2N
where E(A) = L 10, A
is monotonically decreasing w.r.t. A. The proof is completed.
B.5 PROOF FOR THEBRRMM 3
Proof. We prove Theorem 4 by analyzing the perturbation (i.e., Pj in the above Eq. (19)) of the loss function L.
We denote that
N N
e doll (20,, x 1//
w = (p:
=
log
e ddd xt
2== 2o
(29)
1=1
and
1
up (mn,, x/)/
e dy ( at/
Wr
log
log
N
e (a: wt 1/7 Ey=ıe do (xe, 2b 0b,1/7²
e (
x 1/7
dhp ( 1/7
(30)
where (Ø. (R In )==)) is an arbitrary mini-batch from the sample space. Then we have that
wr
1
d (5, 1/7
d da 1/7
log
log d ( #1 117 E3 depler Bby)/7
N
de a 1/7 "
dh (8 1/4
e ( 1/7 4 dp (2.,
de (cr: 26 26,117)
1
llog
N
du (0. M/
1=
(C + (/C))(n)log + max (d t 21)
(31)
aN
where w(n) = log
2 + +1) Meanwhile, we have
N
0-ol2t 1)/7
log
E
-log
N
e d (m,, (m:, xt 1/7 1// +i
ddl 23,
e c x 2 2i dop way 3/7
Cemp (4p; 2²) Zomple (4:: ②).
(32)
By Lemma 1, we let that for all
(C 2/C) + max (t, 27)
P: =
(33)
aN
21

---

Published as a conference paper at ICLR 2024
so that have
(C (/C))()())) + max t, 27)) Ex7)
[n(2/6)
Cemp (0 - Complep: 9)
P
a
2.V
2e 2/²²/ E1 ??
2
2e
(C42/C)) (E.
2N (aN(ylln (1b(2/5)//222)
2e
2e ln(2/8)
(34)
(C44/C0hiimlle(11m444444.41111 and (In(2 The proof is com-
where 17 pleted.
B.6 PROOF FOR THEOREM
Proof. Firstly, by using the Lipschitz continuity of F(4) we have that
E| )
- E[F(((())))
< (E (1+1) (1+ - 8/2|gp
(++ -p
-...[(1]VF(( (1) ||2] + (Sn\/2)B(IV (t)
s -n V |1 + /2)02
(35)
where the second incquality follows from the fact that ((+1)) is updated by Algorithm 1. Then, have that
E
[[(+ (1(p))(F((()))) F(y((+1))) (Ima/2)0²
(36)
and thus
|131 < (1/1)) - -((((1)) (Snx/2282²
E
V
M| < (L F({(()) (Sm /2)0²
(37)
E1 (F
||]
L/ (1/mT-) F(y((T))
(S1)=1/2)02
Finally, we combine all inequalities in the above Eq. (37) by letting nuu 171 172-1 17.
Then we have
min 031ST
VFF (1) 11)
V (1)] + (Sm/2)62
T
E[F (0)) [() (Sn/2)82
$
TT
()) -F((")) + +Sn/2282
Tn
1
< _
(F(( (0)) F(O. (Sce))+)
(38)
VT
where c= T... We set = (((((()) F((*)))((83) and we have
min
E (9 S (2 F/100)
(39)
0
which completes the proof.
22


Reviews:
[{"review_id": "oV9cA4rli2", "reviewer": "Reviewer_SsFX", "summary": "This work proposed a new unified regularization technique for both supervised and unsupervised similarity learning. The proposed regularization encourages the feature difference between inter-class samples to be similar. It is claimed to help avoid learning ill-posed clustering in the feature space, which is the often case in normal metric learning.\n\nThe authors did extensive numerical experiments on many tasks and datasets. The authors also provided experiments to show the sensitivity of the proposed method to hyperparameters. Results showed that the proposed regularization is easy to tune. Numerical results on various tasks and datasets are impressive, surpassing SOTA metric learning methods by clear margins.", "questions": "See the weaknesses part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ Interesting idea of constraining the \"second-order\" feature difference. The intuition of the proposed regularization is natural and the authors provided conceptual experiments to corroborate the point.\n\n+ Impressive umerical results and extensive and informative experiments. I also appreciate the authors efforts for showing the stability of the proposed regularization technique in terms of hyperparameters. This is rare among many deep learning methods, making it much more practical to use in real world.\n\n+ This paper is clearly written.", "weaknesses": "- The only concern I have is about the theoretical analysis, especialy thm 3 and 4. The two theorems assumed that $x$, $z$, $\\hat{x}$ and $\\hat{z}$ are from the same distribution. I wonder if the assumption is so strong that makes the result meaningless in real world. The authors are clearly more interested in regularizing the differences of inter-class feature difference. In that case the samples are clearly from different distributions, as the authors assumed in the conceptual experiments in Fig. 2.\n\nI am concerned that the math analysis is only to make the paper more \"mathy\" and of less practical meaning. I would love to raise the score if the authors could help to clarify or tone down the theoretical analysis. I think this work is good even as a pure empirical one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8jurcIwGpU", "reviewer": "Reviewer_Q8tk", "summary": "The paper presents a new approach to representation learning by emphasizing higher-order feature distances. It introduces Difference Alignment Regularization (DAR) to maintain consistent representation differences between inter-class instances, addressing the overfitting issue common in traditional methods. A novel cross-total-variation (CTV) norm is proposed to quantify these differences, and is simplified for optimization. DAR is integrated into Difference-Aligned Similarity Learning (DASL), reducing overfitting and enhancing performance. Theoretical and empirical analyses confirm DAR's effectiveness in tightening error bounds and improving performance in various learning tasks.", "questions": "Please see above.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.     Approach: The paper learns robust and generalized representation by focusing on reducing the the higher-order feature distances rather than the first-order differences, which is hardly explored before.\n2.\tTheoretical Justification: The paper not only proposes a new method but also provides theoretical evidence to support its effectiveness.\n3.\tEmpirical Validation: The superiority of the Difference-Aligned Similarity Learning (DASL) method is backed by experiments conducted on multi-domain data (Image retrieval, face recognition, image classification and NLP classification), showing its effectiveness in both supervised metric learning and unsupervised contrastive learning tasks.", "weaknesses": "1.\tInconsistency between theoretical justification and method. Theorem 4 presents a bound on the generalization gap based on max{d'(t; t_hat)}, a metric akin to first-order measurement of feature distance. Despite the notion that smoothness of discrepancy could minimize the largest max{d'(t; t_hat)} by pulling it towards the majority, this mechanism inherently relies on first-order measurement. Consequently, the theorem does not robustly support the paper's primary claim, casting doubts on the reliability of the proposed solution.\n\n2.\tInconsistency between motivation and method. Figure 2 suggests that a smoother distance between different distributions correlates with better performance. However, this conclusion appears flawed. The generalization in a mixed Gaussian scenario is predominantly governed by the distance of the distributions (a first-order attribute) rather than higher-order terms. For instance, three Gaussians with unit variance located at (0,1), (1, 0), and (1,10000) would exhibit a smaller generalization error compared to them being positioned at (0,1), (1, 0), and (1,1). The implication is that sufficient distance, regardless of higher-order distances, should invariably simplify distinguishability.\n\n3.\tWriting issues. I would suggestion the author revise the introduction, mainly on discuss the motivation and purpose of using higher order feature distance, rather than general discussion on representation learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LwneUEWSsI", "reviewer": "Reviewer_YcLH", "summary": "The authors propose to regularize ERM for similarity learning by enforcing consistency among distances between negative pairs of samples. Forcing the distances between negative pairs to be consistent leads to a more robust representation, and the authors demonstrate this theoretically using a PAC learning style upper bound on the generalization gap.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Results are quite extensive and impressive.\n- I found the method interesting. I like the way it was presented. However, I do have some reservations, as stated in the weaknesses and questions.\n- Figures are very easy to understand.", "weaknesses": "- Regarding the method, could you give some insight as to why it's different from the standard contrastive loss? The standard contrastive loss is: \n\n$L = ReLu(d^+ - \\delta^+) + ReLu(\\delta^- - d^-)$\n\nwhere $d^+$ denotes the distance between positive pairs, $d^-$ denotes the distance between negative pairs, $\\delta^+$ denotes the positive margin and $\\delta^-$ denotes the native margin. This loss repels features of negative pairs until they are $\\delta^-$ away from each-other. In effect, the contrastive loss ensures that all negative pairs are $\\delta^-$ away from each-other. This would also satisfy your loss function. Could your loss function be doing something similar to enforcing a negative margin?\n\n- There are some minor issues I saw with the math and the theorems that I don't think are consequential. For example, the notation is quite complicated; I think Algorithm 1 is obvious given the optimization objective in Eq. (8); Theorem 2 seems to trivially follow from the standard GD convergence result; Theorem 4 ignores the tradeoff between generalization and approximation (you say that higher lambda leads to a smaller upper-bound on the generalization gap, but a higher lambda also means $\\mathcal{L}(\\phi)$ is a bad approximation of what you actually want to minimize.) Obviously, it's not hard to design a loss function that generalizes well if you don't care how well it approximates the actual minimization problem; so I'm not sure how meaningful Theorem 4 is.\n\nIf there were a seven, I would give the paper a 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RoNUjHOcWO", "reviewer": "Reviewer_V3dP", "summary": "This paper addresses similarity-based representation learning, which is important to the machine learning field. In this paper, the authors propose a novel difference alignment regularization (DAR) to encourage all representation differences between inter-class instances to be as close as possible so that the learning algorithm can produce consistent differences to distinguish data points. The proposed method is sound with theoretical proof. The proposed method is extensively verified on various representation learning tasks, including supervised metric learning and unsupervised contrastive learning.", "questions": "Please refer to the weakness part.\n\nOverall, this paper tackles the essential problem with a sound method. The proposed method is theoretically proven to have tight bounds, which is also evaluated with extensive experimental results in different metric learning settings. I am positive about this submission; my initial rating is “8”.\n\n[1] Learning multiple local metrics: Global consideration helps. TPAMI 2019\n\n[2] Deep metric learning with angular loss. ICCV 2017\n\n[3] What Makes Objects Similar: A Unified Multi-Metric Learning Approach. TAPMI 2018", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper addresses similarity-based representation learning, which is of great importance to the machine learning field.\n2. The proposed method is sound with theoretical proof.\n3. The proposed method is extensively verified on various representation learning tasks, including supervised metric learning and unsupervised contrastive learning.", "weaknesses": "1. Calculating the higher-order difference requires complex calculation complexity. It would be better to report the running time comparison of different methods with some specific dataset to show the effectiveness.\n2. There are also other ways to enhance inter-class compactness in the metric learning field, e.g., [1-3]. Please discuss the differences between the current algorithm and these works.\n3. It would be better to report the sensitivity with regard to the regularization parameter lambda. Does the change of it heavily influence the final performance? What is the default choice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)