Volume-Aware Distance for Robust Similarity Learning
Authors: Shuo Chen
Chen Gong
Jun Li
Jian Yang
Venue: ICML 2025 poster (2025)
Keywords: similarity learning
metric learning
contrastive learning
volume-aware
theoretical analysis
Primary area: general_machine_learning->representation_learning
TL;DR: A new volume-aware distance for the better similarity learning

Abstract: Measuring the similarity between data points plays a vital role in lots of popular representation learning tasks such as metric learning and contrastive learning. Most existing approaches utilize point-level distances to learn the point-to-point similarity between pairwise instances. However, since the finite number of training data points cannot fully cover the whole sample space consisting of an infinite number of points, the generalizability of the learned distance is usually limited by the sample size. In this paper, we thus extend the conventional form of data point to the new form of data ball with a predictable volume, so that we can naturally generalize the existing point-level distance to a new volume-aware distance (VAD) which measures the field-to-field geometric similarity. The learned VAD not only takes into account the relationship between observed instances but also uncovers the similarity among those unsampled neighbors surrounding the training data. This practice significantly enriches the coverage of sample space and thus improves the model generalizability. Theoretically, we prove that VAD tightens the error bound of traditional similarity learning and preserves crucial topological properties. Experiments on multi-domain data demonstrate the superiority of VAD over existing approaches in both supervised and unsupervised tasks.

Volume-Aware Distance for Robust Similarity Learning
Shuo Chen Chen Gong? Jun Li Jian Yang?
Abstract
Sampled Data
x Unsampled Data
Pull/Push
Houndary
Measuring the similarity between data points plays vital role in lots of popular representa- tion learning tasks such as metric learning and contrastive learning. Most existing approaches utilize point-level distances to learn the point- to-point similarity between pairwise instances. However, since the finite number training data points cannot fully cover the whole sample space consisting of an infinite number of points, the generalizability of the learned distance is usu- ally limited by the sample size. In this paper, we thus extend the conventional form of data point to the new form of data ball with a predictable volume, so that we can naturally generalize the ex- isting point-level distance distance (VAD) which measures the field-to-field geometric similarity. The learned VAD not only takes into account the relationship between ob- served instances but also uncovers the similarity among those unsampled neighbors surrounding the training data. This practice significantly en- riches the coverage of sample space and thus im- proves the model generalizability. Theoretically, we prove that VAD) tightens the error bound of traditional similarity learning and preserves cru- cial topological properties. Experiments on multi- domain data demonstrate the superiority of VAD over existing approaches in both supervised and unsupervised tasks.
training
humpy
bumpy
(a). Concept illustration of the traditional point-level distance
training
flat
flat
new volume-aware
(b). Concept illustration of volume-aware distance (VAD) Figure I. A quick comparison between the traditional point-level distance and our proposed VAD. By considering the volume aware- ness, our VAD can obtain the flatter decision boundary better generalizability than the traditional distance.
stream recognitions (Weinberger al., 2006; Xing et al., 2002; Liu al., 2018; Yan et al., 2024). Over the past decades, similarity learning has achieved remarkable suc- cess, showing its potency in diverse representative tasks, such as classífication (Kaya & Bilge, 2019), clustering (Zhong et al., 2020), and retrieval (Kou et al., 2022).
Typically, similarity learning algorithms are provided with (pseudo) supervision in the form of pairwise relationships (e.g., similar dissimilar) derived from the training data (Sohn, 2016; Feng et al., 2023). Then the learning algo- rithms pull similar instances closer while pushing dissimilar ones apart, and the overarching goal of similarity learning is to make the learned similarity metric robust and general- izable when applied to the complex test phase with unseen data (Yan et al., 2023b; Kaya Bilge, 2019).
1. Introduction
Similarity learning has been longstanding focus of re- search, aiming to learn to measure pairwise similarity be- tween instances. The learned similarity metric and feature representation can serve as essential components for down-
'School of Intelligence Science and Technology, Nanjing Uni- versity, China SScoool of Computer Science and Technology, Nan- jing University of Science and Technology, China. Correspondence to: Shuo Chen <ssuoochene nju.edu.en>>
Substantial progress has been made toward this goal through the design of various loss functions (Liu & Tsang, 2015; Sohn, 2016; Oh Song et al., 2016; Chen et al., 2020) and plentiful regularization terms (Chen et al., 2021; Yan et al., 2022b; Wang & Qi, 2023; Chen et al., 2024a). These meth- ods refine the sampling process and constrain the hypothesis
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Volume-Aware Distance for Robust Similarity Learning
space to improve learning performance. However, the gen- eralizability of similarity learning remains limited by the sample size, as finite sampled data points (i.e., instances) are unable to sufficiently cover the whole sample space which contains infinite unsampled points. The coverage insufficiency is particularly hard in similarity learning due to considering the Cartesian product (i.e., the pairwise rela- tionship) of the sample space (Feng et al., 2021; Wu al., 2022a). As result, the decision boundary of similarity metric learned from the scattered data points may lack the flatness (see Fig. (aa) needed for generalizable classifica- tion. This practically leads to unsatisfactory performance in some noisy and difficult recognition scenarios for both su- pervised tasks (e.g. metric learning) and unsupervised/self supervised tasks (e.g., contrastive learning).
2.1. Metric Learning& & Contrastive Learning
In supervised scenarios, similarity learning is commonly referred to metric learning (Xing et al., 2002; Yan et al., 2023a; Kaya & Bilge, 2019), where the pairwise relation- ships are provided through human supervision. Metric learning has been extensively studied in both linear models (Davis et al., 2007; Zadeh et al., 2016; Chen et al., 2019) and nonlinear deep neural network based models (Chu et al., 2020; Yan et al., 2023b; Furusawa, 2024). A lot of existing research has focused on designing novel loss functions to enrich sampling results, e.g., triplet loss (Ge, 2018), N N-pairs loss (Sohn, 2016), circle loss (Sun et al., 2020b), etc.
In unsupervised scenarios, self-supervised contrastive learn- ing has garnered significant attention due to its competi- tive performance compared with the fully supervised ap- proach (Chen et al., 2020; Yan al., 2022a). This approach adopts the similarity metric framework from traditional met- ric learning and also trains encoder network in pairwise manner. Contrastive learning builds the positive (similar) data pairs by pulling each instance closer to its data angmen- tation and creates negative (dissimilar) data pairs by pushing each instance away from others (Chuang et al., 2020; Yan et al., 2022a; Tian et al., 2020; Chen et al., 2021).
Given that the entire sample space usually assumes specific volume (e.g., the widely adopted hypercube/hypersphen with feature normalization (Xu et al., 2022a; Yan et al., 2021)). it is inherently challenging to adequately fill in this space by using only volumeless data points. To address this issue, we introduce new measure-head network regular- ized by volume expansion strategy predict the volume of each data point (i.e., extending data balls), thereby effec- tively considering pairwise relationships within the unsam- pled neighborhoods surrounding training data points (see Fig. 1(b)). This allows us to define a novel volume-aware distance (VAD) to measure the geometric proximity between colume-predicttal data balls, successfully learning a reli- able similarity metric with better coverage of the sample space. Theoretically, we establish the geometric soundness of VAD and derive tighter error bound. Extensive experi- ments conducted across multiple domains demonstrate the effectiveness and superiority of our approach. Our main contributions are summarized below: 1) We propose a novel VAD metric for robust similarity learning, supported by comprehensive theoretical analyses that ensure its sound- ness and effectiveness: 2) We build a new similarity learn- ing framework incorporating measure- head network that adaptively predicts instance volumes for reliable similarity determination: 3) The experiments on both the supervised and unsupervised tasks successfully validate the superiority of our method over the state-of-the-aa approaches.
Both supervised and unsupervised methods rely on point-to- point similarity metrics. We propose a novel metric that con- siders field-to-field relationships within the sample space.
2.2. Regularization & Augmentation
The primary goal of regularization is to enhance the general- izability of learning algorithms. Early works minimize the Ezll (Arpit et al., 2016; Yang et al., 201 I), or nuclear norm (Chen et al., 2022a; Dong et al., 2014) of learning parameters to constrain the hypothesis space within inter- pretable regions. Recent works such as dropout (Baldi & Sadowski, 2013), batch normalization (Bjorek et al., 2018), 8),, and mixup (Zhang et al., 2018) focus regularizing the flat decision boundary to achieve smooth generalization.
As an implicit form of regularization, data augmentation is highly effective not only in similarity learning but also across broad spectrum of representation learning (Steiner et al., 2021; You et al., 2020; Zheng et al., 2021). It lever- ages human prior knowledge to quickly expand the training dataset, enabling models to better capture neighborhood re- lationships among sampled data points. Notably, recent ad- vancements, such as set-level similarity (Wang et al., 2022), automatic augmentation (You et al., 2021), and adversar- ial augmentation (Lim et al., 2023), have further enriched original data and extended sample space coverage.
2. Background & Related Work
We briefly review the research related to this paper. Notations.
We write matrices, vectors, and mappings as bold uppercase characters, bold lowercase characters, and calligraphy characters, respectively. We denote the training dataset 2 = (a, E Edd 1=1, 2...., .) where d is the data dimensionality and N is the total number of instances.
In this paper, we consider new metric that expands sample space coverage and regularizes the learning process without relying on additional augmented data.
2

---

Volume-Aware Distance for Robust Similarity Learning
3. Methodology
Sample
Somple Space
In this section, we formulate our proposed new VAD metric and the corresponding learning objective.
Class-J
Class-2
Class-2
3.1. New Metric and New Framework
Dorecti
Intra
Inter
Intra
Inter
The projected Euclidean distance has been a prevalent sim- ilarity metric in existing research (Liu et al., 2018). For instances æ, 3 E Rd, this metric is generally defined as:
Distance
Distance
Volumeless Distance Calculation
Volume-Aware Distance Calculation
(x, 2)- = (l))(x)) (0))
(1)
Figure 2. A comparison between the traditional distance and our proposed VAD, Our metric provides the better coverage to the sample space when calculating distances.
where 4p : Rd + RT represents a feature embedding learned by an encoder network (e.g., ResNet (He et al., 2016) and VIT (Han et al., 2022)), and the corresponding embedding result is sometimes further normalized, namely, for any E R2 (xxx) = ()))) (
and thus there exists (1-) such that (D ( 02) + az))/D p. (x11 03s) < 1, namely D D.p p. x2)+
D D e,)lee D (0 tz)) <D se, p (( erty is friendly to the distance flexibility, because lots of real-workh similarity relations do not actually match the triangle inequality necessarily (Fraigniaud et al., 2008; Xu et al., 2022b), and VAD provides a flexible way to describe those relations as needed. We also allow overlaps among those data balls make VAD able reveal similar features belonging to multiple instances.
Tg). This relaxed triangle prop-
Volume-Aware Distance. As the above conventional metric neglects the important information from the neighborhood fields of x and &tr during its distance caleulation, here we want to extend it to a volume-aware form. To be specific, we build a measure function V: Rd R+ characterize the volume value (xx) of the given instance x Rd. Based on the volumes of two instances x. and @. we can adaptively scale the original distance value de (2r, @) to intuitively con- sider the geometric proximity between the two data balls B(x Vxx)) and (@, ())) with their corresponding vol- umes. As the increased volume makes two data balls closer to each other (see Fig. 2), we naturally use the negative exponential function to confine the effect of volume to (0, 1) and thus have the following VAD metrie:
Learning with Prasure-Head. As shown in Fig. 3, we introduce an additional network H--) ahead of the existing encoder (p()) to measure the volume of the feature embed- ding result (p(x)), i.e., we let H a yp such that
V(z)= = H[p((=) = ReLU/a RRLLUWW <(())), (4)
(I. 2)
where the measure-head H R - R- is implemented pereetoyyerrrrrrrettoo (MLP) (Hastie, 2009). with a classical multi-layer perceptron
=d( T) / (a))
Here W E R a E R² and k is the dimensionality of hidden layer. In this way, the volume prediction effectively guides the training of the encoder network capture useful features. Meanwhile, the reliable features extracted by the encoder &p can also assist the measure-head H. in the faithful volume determination. Both the encoder and the measure- head H. are learned with the conventional empirical loss. For the widely used (n + 1)-tuplet/Npair loss metric learning (Sohn, 2016) and the NCE loss (Chen al., 2020) in contrastive learning, based on the given training set H Rd ji= 1, 1, ...,..... NN. corresponding empirical risk
= -1<x(x) - ( / v()))
(2)
where we can observe that the conventional projected Eu- clidean distance (1r, @) is a specific case of VAD when ((x)=)(()) = => =0. Meanwhile, the significantly large V(x) (or V())) leads to that limy(e))+oc (x,, )) = 0, in which the large-volume data ball will accommodate the other instances, so that they can share the similar discrim- inating features with each other. It is easy to validate that such new VAD metric satisfies the non-negativity and sym- metry, VAD is always semi-metric for any &p and V. and we can obtain that VAD is a strict metric if y is constant mapping. Nevertheless, it is notable that VAD does not satisfy the triangle inequality necessarily, because that!
{x, e RR can be easily summarized as
Lemp(1) (4, 7))
= (b, [1( 7d; (x, Zb, :by: ..... Lbo F)
lim De.v(E1,22) x2) + (22, Ø3). 0, Dip.( 10, 203)
(3)
Da. 1/7
V(
=En (b;)b
log
DDpy (2, 1/7
Ei=1 D Devu(a
'For details, 1o Pm. m2) m2))))))) (
(5)
(ims) (((ll(((())__((())) (lgg)) vie ( llzillla
(V(e2)]lp(an) pppp)))))))pp)))ll
where 7 > 0 a temperature parameter, and the anchor in- stance æ sampled from 2.. Here ar is directly obtained
lim limy Ce V(az)) = 0, with independent of (m2)).
3

---

Volume-Aware Distance for Robust Similarity Learning
Feature Vector
Forward Stop VAD Calculation
Algorithm 1 Solving Eq. (7) via SGD.
Volume Calculation
Input:
training set 2 = { k step size 1l > 0; regu- larization parameter À > 0; batch size n E N+; randomly initialized (0)); maximum iteration number T.
Attract
Measure-Heal wt()
Encoder s()
For t from 1 to T:
Shazed Parameters
Shared Parameiirss
Σ
1). Uniformly pick (n + 1) instances {æa, J from L; 2). Compute Vzhempl (by 1 17-1), VeeRespund(()) ((b 1 1), &eommp ((b) ) and V Rexpand (by 1)=1)) accord-
VAD
Encoder +()
Measure-Heaa. ,n(1)
ing to Eq. (8) and Eq. (9);
Input
Feature
Volume Value
Distance Value
Embedding
3). Update the learning parameter:
Figure 3. The overall framework of our proposed volume aware distance based similarity learning.
[ (p(0) (1(1-1) 77( yp Cemp + X 4 Rexpand); (10)
(H(t) H((t-1) ( H Cump + A AV H Respand);
from the perturbed x: for the unsupervised (contrastive learn- ing) case, or it is chosen randomly from the intra-class set 2 (x)) = {z 8/ = Ya: = E 2 }==}} for the supervised (metric learning) case. Correspondingly, the mini-batch instances {a>b, Fbg.....; :b } are directly selected from 2²)(z) for the unsupervised case, or from the inter-class set 2- (a:) {=13> * Ye: 2 2}} for the supervised case (Vx is the class label of z).
End Output: converged (p(7) and H(
where À >> 0 is a trade-off parameter. Then we are able to optimize our learning objective Eq. (7) in a stochastic way, where we only need to specify the stochastic terms of both Cemp(pp, H) and Rexpan (p. H) for a given mini-batch. is worth pointing out that the learning parameters 4p and H are interdependent, where the stochastic gradient of Rexpand w.r.t. Ht can be calculated as
Volume Expansion Regularizer. Minimizing the above empirical loss can finally learn a VAD metric which pro- vides the consistent prediction with the supervisory in- formation. However, we further need a new regular- ization term to explicitly encourage the volume determi- nation of instances towards a generalizable learning re- sult. Actually, we want the data points in the embed- ding space to capture/cover their intrinsically similar fea- tures as much as possible, so here we simply encourage the volume of each instance V(x) to be as large as pos- sible. Such a volume expansion strategy makes the data balls B( ((x))), B( V(xg)),..... "B))) V(xx)) to accommodate each other as many as they can, and thus we propose the following volume expansion regularizer (VER):
V(as,, (V,,; (8)
V ne Respand ( (b,} E", --aas
0H
and the stochastic gradient Vee ((b, is direct calculation of dE 0E/0H. Based on the gradient result w.r.t H. we can further obtain that
V Veppemp (b1))=11)= = &empl (1b)) (-)) V Rexpand ( ((b)) Vn Rexpand (b,})=))
BH
(9)
ap?
which only requires a single derivative computation of 8H/ /0pp to get the gradient w.r.t. s. The detailed iter- ation steps based on stochastic gradient descent (SGD) (Reddi et al., 2016) are summarized in Algorithm 1. Ex- isting work reveals that the convergence of iteration points (1)(1), H()),, ((()))
Rexpand (gp, H)- = E0,)
V(
(6)
H(2)).... (,( H(T) can naturally inherit from SGD as long as the objective function F(yp H) is Lipsehitz-smooe 2019). We have the following theorem to ensure the gradi- ent boundness and Lipschitz smoothness for our learning objective Fl4p, H) in Eq. (7).
where the mini-bater instances {zbb Tb.y Dbb. are di- rectly selected from 2x, and the natural exponential function is employed again for numerical simplicity.
and gradient bounded (Huang al.,
tradient-bounded
3.2. Optimization & Convergence
Theorem 1. The learning objective Flyp, H) is always gradient-bounderr and Lipschitz-smooth if the encoder (p(.) is gradient- and Lipschitz-smone
Here we elaborate on the optimization process and conver- gence property of our learning algorithm.
Stochastic Gradient & Boundness. By combining the above empirical loss in Eq. (5) and VER in Eq. (6), we finally obtain the objective of our volume-aware distance based similarity learning (VADSL):
This implies that the gradient boundness and Lipschitz smoothness of our learning objective are directly inherited from the original encoder network &p and well preserved in the VADSL. As a result, the practical convergence of our learning algorithm is theoretically guaranteed.
min (F H)- Lemp (4p, H) + & (4,, H0), (7)
s..
4

---

Volume-Aware Distance for Robust Similarity Learning
4. Theoretical Analyses
(a1, b1) t (az, b2) 2.... (dcz, bcz,) ) there always exists H1" IRm - IR+ such that
In this section, we provide in-depth theoretical results to investigate the generalization ability, the distance flexibility, and the sample-space coverage of our proposed method.
P. (Dar Ib,) 2 D (Foz,
Dp.v- (2 TF,z)
(12)
4.1. Generalization Error Bound
where U.e. (aa b,)) = ((A, < B, where A, B = 1, ..2................... NN, V" = =H" o up, and up is independent of H'
Here we would like to prove that our algorithm provides tighter generalization ervor bound (GEB) (Chen et al., 2021) compared with conventional similarity learning ap- proaches. This is achieved by analyzing the convergence rate of the GEB with respect to the sample size N and by demonstrating how the proposed regularization term exppadd contributes to tightening this bound. Specifically, for the underlying data distribution 2, we denote the expected risk Comp( H; 2) = [e,, [Lemp((p Hi; {t, )t and discuss how far it is from the empirical risk Cemp (4, H).
The above interesting result implies that our VAD has flexible adaptability even if the feature embedding & is fixed. The measure-head can always learn an optimal H" to make the finally predicted distances satisfy any given partial ordering relations. This will also be beneficial to the feature extraction of the encoder &p because the measure-head H. successfully separates the distance measurement apart from the feature embedding and explicitly keeps it as white-box module in the overall learning framework.
Theorem For any yp learned from the objective F(4p, H) and any given constant E (0, 1), we have that with proba- bility at least 1 5,
4.3. Finite Covering of Volume-Awarenes:
As our basic motivation, volume- is introduced to extend the traditional data points æ1, Ø2, NN to the data balls B( V(xi), )), B(x2, Vxza)), [[z2))..... B(xN, V(ZN)) )), so that the whole sample space can be covered as much as possible. Therefore, here we investigate if is possible to cover the sample space with any given coverage ratio.
Lemp H)- - Eemp(( (p. H: 2) S [w(n)log(1+ + Dmax) /v [In [1I(2/0))((22B(X))
(11)
where Diaxx = (p. 8) 1, E }. The function (n)) log (o² /n + 1) is monotonically decreasing w.n.t n. The function (AA = CA where the positive constant C is independent of ip H. and Y.
Theorem 4. Suppose that 4p is feature-normalized such that = <p(x)= E [L, U]m, Væ E Rd. Then for any given E (0, 1), there exists sufficiently large N such that See[t, 1mm sign|z E U B(x,, (xx)))
The error bound in Eq. (11) is dominated by two factors. First, the generalization error bound decreases as the sample size N and batch size n increase. This behavior aligns with observations in prior research, where larger datasets and batch sizes typically lead to reduced generalization error. More importantly, the error bound tightens the regularization parameter À increases. This is because (()). defined as a monotonically increasing function of X, grows larger, thereby enabling faster convergence of the empirical risk Lemp ( H) to the expected risk Lemp(4p. H; 2). It means that our VER regularizer Rexpand can accelerate the empirical risk convergence to the expected risk.
2 P, (13)
S 1dz
where the integral values SzelL, Umm sign(z
B(x, V(x))) = - ((a)) dz and SzeJz, upm 1dz are the volumes of all data balls and the whole feature space [L, UIm, respectively.
This theorem clearly answers that the can sufficiently cover (with a desired coverage ratio of p) the whole sample space by using only the finite number of (namely N) instances. Intuitively, this is because each instance has been endowed with a volume (i.e., the data ball), and thereby it becomes easy to employ those volume- specified entities to fill in the sample space.
4.2. Distance Flexibility of VAD
Now we further analyze the flexibility of our VAD in order- ing distance sequence. As the most crucial thing in similarity learning is the relative distance (namely the order of dis- tance sequence) but not the absolute distance itself (Liu & Tsang, 2015; Ge, 2018), we want to show that for the given embedding sp, our VAD always has a volume measure V to fil any predefined distance sequence. Specifically, we have the following theorem to reveal the flexibility of our VAD.
5. Experimental Results
We conduct experiments to evaluate the performance of our proposed method using real-world datasets. We first con- duct ablation studies to reveal the usefulness of our newly introduced block/regularizzer Then we compare our pro- posed learning algorithm with existing state-of-the mod- els both the supervised metric learning and unsupervised
Theorem 3. For the given dataset V = {xx E i 1, 2..... N). feature embedding ip, and partial ordering
5

---

Volume-Aware Distance for Robust Similarity Learning
M Resul of VADSL on CIFAR110 (wo ft)
(d). Result VADSL on STI--10 (wio
Table I. Classification accuracy rates of baseline methods and our method on CAR- 196 and CUB-200 datasets (feature embedding sizes are 128 and 512).
30
-50
-100
-120
100
METTOOD
128-dim.
CAR-196
512-dim.
128-dim.
CUB-200
512-dim.
Result VADSI 1IFAR10 J=0
Result of VADSL STL-10 1=0.1)
R@
RGS
RB1
R88
R@1
R@B
R@11
R@B
SpaittBN w/o F2
82,37
58.12
VADSLIN, w/ 94 (A 01]
68.36
86.211
82.481
58.56
79.121
66.881 91.22f
VADSLIN w/ 94 (1=A0_00. 0.13
68.391
88.261
85.301
58.
80344
66.31
91 871
0-
VADSLIN w/ (A 0.50)
70.231
90.391
89.261
96.321
62.07
82.351
69.071
92.551
-50
ProxA.KKSIl) wio H
92.41
VADSL w/ N 01=
69.271
87.771
97.951
69.911
-100
VADSIIP. w/ 0.11-
09.22
58.541
89.38
98
26.
62.26
71,25" 92.6PT
-100
-00
100
40
-20
VADSL[P 76 (A
70.361
91.321
92.421
98.85
63.19
82.211
73,22
94,147
(6). Result of VADSI on CF (wt H. 1=0 E)
(T) Result of VADSL on STL-10 (wl H, 2=0.5)
M.F,R500 W
72.42 89.53
91.76
97.21
69.33
74.42 92.53
VADSLIM. w/ H m DN
72.42
69.77
85.82
90.631
[M w/ PE(A 0. 11]
73.451
91.26/
91.48
97-77
70955
56.481
74.22 92,051
VADSLJM w/ Wt 00.5)
73.59
92.351
92.231
98,451
72.55
58,351
75.74"
93.451
0-
-50
backbone for several contrastive learning methods includ- ing SimCLR (Chen et al., 2020), BYOL (Grill et al., 2020), and HCL (Robinson et al., 2021). In Fig. 5, we record the classification accuracy of all compared methods on CIFAR- 10 (Krizhevsky et al., 2009) and S71.-10 datasets (Coates et al., 2011), 1), where we can observe that our method con- sistently improves the corresponding baseline results in all scenarios. To be more intuitive, we also conduct the t-SNE embedding (Van der Maaten & Hinton, 2008) to obtain the 2-dimensional data points to better understand the useful- ness of our introduced new component. In Fig. 4, VADSL. (w/ measure-head H) can successfully obtain the better sep- arability than the baseline result (w/o H), where the results of X 0.5 achieve very satisfactory separability. These results clearly demonstrate the crucial role of maintaining the measure-head network H along with the corresponding regularizer Rexpend in our approach.
Figure 4. The I-SNE visualizations of our VADSL CIFAR-10 and STL-10 datasets.
contrastive learning tasks. Both the training and test pro- cesses are implemented on Pytorch (Paszke et al., 2019) with TeslaV100 GPUs, where the regularization parameter A is set to 0.5. The dimensionality m and the parameter Y in Eq. (5) are set to 512 and 0.2, respectively. We use a 228-dimensional hidden layer for our H(-) in Eq. (4). The arper-parametes of compared methods are set to the rec- ommended values according to their original papers.
5.1. Ablation Studies & Visualization Results
In this subsection, we explore the effectiveness of our pro- posed VAD and its associated regularizer on different tasks.
5.2. Experiments on Supervised Metric Learning
For the supervised task, we adopt different feature encoders (BN-Inception (Ioffe& & Szegedy, 2015) for Npair (Sohn, 2016), and ResNet-50 (He et al., 2016) for ProxyAnchor (Kim et al., 2020) and MetricFormer (Yan et al., 2022b)) to assess the performance of our method in metric learning. The results are presented in Tab. 1, where we record the test accuracy of all compared methods on CAR-196 (Krause et al., 2013) and CUB-200 (Welinder et al., 2010) datasets (with 500 epochs, learning rate = 10-3, and batch size = 512). We observe that our VADSL performs well across all three baseline methods, providing stable performance in various scenarios with different embedding sizes. Our approach consistently improves upon all baseline methods when equipped with the measure-head H. Furthermore, increasing the regularization parameter À from 0.1 to 0.5 leads to noticeable improvements in recognition accuracy, highlighting the critical importance of our regularization term, the volume expansion.
In this subsection, we evaluate the effectiveness and superi- ority of VADSL on the supervised task of metric learning.
Image Data. We assess the performance of VADSL in general image retrieval tasks across datasets on CAR-196 (Krause et al., 2013). CUB-200 (Welinder et al., 2010), SOP SOP (Oh Song al., 2016), and In-Shop (Liuu al., 2016). The methods we compare include JDR (Chu et al., 2020). IBC (Seidenschwarze et al., 2021), ContextSimilarity (Liao et al., 2023), MetricFormer (Yan et al., 2022b), MFC (Furusawa, 2024), and DASL (Chen et al., 2024a). All compared meth- ods are incorporated into the ResNet-50 backbone, and we refer to the combinations of our approach with Npair loss and Anchor loss as VADSL-NP and VADSL-PA, re- spectively. The NMI and Recall @R R scores of all methods are shown in Tab. 2, where we clearly observe that DASL,, MFC, and our methods obtain higher accuracies than other methods. Compared with those strong baseline methods, our VADSL can further achieve either better or competitive NMI and Recall @ scores.
For the unsupervised task, here we adopt the ResNet-50
6

---

Volume-Aware Distance for Robust Similarity Learning
Table Performance of all compared methods (with ResNet-50 backbone) on CAR-196, CUB-200, SOP, and In-Shop datasets. The best two results are bolded and underlined, respectively.
METHOO
CAR-196
CUB
200
SOP
In-Shop
NMT
R@1
R@4
R@8
NMT
R@1
R@4
R@g
NMI
R@1 I
R 10
R 100
NMI
R@
R 10 R
Sohin 2016)
69.50
82.57
94.97
95.92
69.53
64.52
65.63
91.15
76.21
90.21
98.01
98.72
ProxyA.(Kimcd et at. 2020)
75.72
87.71
95.76
97.86
72.31
69.72
87.01
92.41
91.02
78.39
90.48
96.16
90.22
91.49
98.12
98.83
JDR et al. 2020)
70.56
84,86
94.56
97.21
70.32
69.44
87.01
91.33
92.21
79.21
90.53
96.01
91.69
92.11
97.
63
98. 31
[BCieedesscheag ot ol. 2021)
74,82
86. 11
96.21
98.2
74.01
20.32
87.61
92.72
92.61
81.42
91 12
95.89
91.34
92.5
9% 52
99 99.1
(Xom et al. 2022b)
76.23
91.75 76
96.31
97.21
15.41
74.42
85. 75.75
02.33
02.71
82.22
92.62
96.33
89.32
91 25
97 82
98. 36
ConlextS et 2025)
76.32
91 80
07.1 14
98.41
74,01
71.91
88.82
93.42
92.61
82.63
92 56
96.74
86.89
90.72
97.82
98.51
MICCpeuussss 2024)
75.96
91.61
97.60
98.57
73.37
71.83
88
93.27
91,37
79.59
97.36
96.57
91.78
92
98.86
98.87
DASI, et ol, 20246)
77.32
92.31
97.82
98.90
76.50
73.96
90.54
94.21
93.86
83.32
93 96
97.95
86.32
90.62
97. .5
98.95
VADSL-NP (ours)
93.4
37.651
98.971
76.821
77
77.361
90.62+
94.117
93.85t
92. 381
93.38*
98.21
99 531
VADSL-PA (ours)
78.22
94.13F
98.28;
99.307
77.35
78369
91,45
95.12-
94.967
86.23 95.361 98.92
93.13+
93.82
98,951
99.28-
[a). S10 with 100 epochs
(b). STL-10 with 400 epochs
Table Accuracy rates of all compared methods on AgeDB30, CFPFP, and Megalacce datasets.
Face Verification
Face Ideatif. (McgaFacc)
Accuracy
70-
SinCLA HCL
Accuracy
80
SinCLR
METHOD
HCL
Age.
CFP
M--10%
M.-10
M--10*
BYOL
VADSLIW/SinCLR
BYOL
VADSLIW SIKCLR)
f et al., 2017)
Softmax
93.42
91.30
93.39
94,30
88.38
80.43
92.86
87.11
92.83
95.93
VADSLIW HCL)
VADSLIW BYOL)
60
VADSLOW BYOL)
VADSLOW HCLI
Sph-~SECi et al., 2020)
93.45
94.39
88.42
92.79
95.88
100
200
400
500
600
100
500 600
Arc ne=++y(Deng al., 2019)
93.93
94.77
90.68
94.34
96.83
Batch Size
Batch Size
Arc +SECYZhug al., 2020)
93.82
94,91
90.91
94.56
96,95
(c). CIFAR-10 with 100 epochs
(d). (B).CIFAA CIFAR-10 with 400 epochs
MFCCFurusswa, 2024)
94.98
95.21
90.82
95.68
97.32
95
VADSL fours, VAD)+
94.92*
95,371
89.751
93,771
96.421
VADSL (ours, VAD))
95.21
96.241
91.78"
95.891
97.45t
Accuracy
SIMCLR
Accuracy
SInCLR HCL
HCL
00-
VADSLIW
VADSLIW SinCLR)
VADSLIW HCL)
VADSLIW HCLI
Facial Data. We employ CAsIA-WebFace (Yi et al., 2014) as the training set while using AgeDB30 (Moschoglou et al., 2017), CFP-FP (Sengupta et al., 2016), and MegaFace Keemelmacher--hhizzemman Shlizerman et al., 2016) as the test sets. For all methods, we set the batch size to 256 and the em- bedding size to 512, using the ResNet-50 backbone. The methods compared include various regularized versions of Sphereface (Zhang et al., 2020) and Areface (Deng et al., 2019), and the deep metric learning loss MFC (Furusawa, 2024). As shown in Tab. 3, we can clearly observe that our VAD enhances the performance of both Sphereface and Areface in all cases. For example, on MegaFace with 106 distractors, the accuracies of Sphereface and Arcface are boosted by 1.37% and 1.1%, respectively.
VADSLIW
VADSLIW BYOL)
100
200
500
600
500 500
Batch Size
Batch Size
Figure 5. Classification accuracy of all methods on STL- and CIFAR-10 datasets with different training epachs, where the (neg- ative) batch size is from 32 to 512.
backbone and compare it with three more methods includ- ing DINO (Caron et al., 2021), iBOT (Zhou al., 2022), and MTE (Li et al., 2024). The classification accuracy is evaluated using the linear softmax (i.e., the Top-1 score and Top-5 score of linear probing) and the k-NN classification (here = 8). From the results shown in Tab. 4, it is evi- dent that our method consistently improves both SimCLR and SWAV by at least 3% in most cases. When leveraging the powerful ViT-B/16 encoder, our method consistently improves the baselines and surpasses three state-of-the-art methods (DINO, iBOT, and MTE) across multiple datasets.
5.3. Experiments on Unsupervised Contrastive Learning
In this subsection, use different domains of data eval- uate the effectiveness and superiority of VADSL on the unsupervised task of contrastive learning.
Text Data. In this experiment, we use the STS dataset (Agirre et al., 2016) (including the tasks of STS/2, STS13, STS14, S7S15, and STS16). Following the approach in Sim- CSE (Gao al., 2021), we utilize pre-trained BERT (Devlin et al., 2018) checkpoints and compare our method with InforMin-CL CL (Chen et al., 2022b), misCSE (Klein & Nabi, 2022), PCL (Li et al., 2021), SCL (Wu et al., 2022b), and AD- SCI. NCE (Wu et al., 2024). As we can observe from Tab. 5, our VADSL obtains considerable improvements on the baseline method SimCSE. Meanwhile, our method can outperform the representative methods misCSE and InforMin-CL. in
Image Data. We employ ResNet-50 as the backbone and integrate our method with SimCLR (Chen et al., 2020) and SwAV (Caron et al., 2020). yielding the results labeled as VADSL (cluster-free) and VADSL (cluster-used), respec- tively. We train method on ImageNet- 100 and ImageNet- IK (Russakovsky et al., 2015), and compare it with existing representative approaches including HCL (Robinson et al., 2021), PCL (Li et al., 2021), BYOL (Grill et al., 2020), GCA (Chen et al., 2024b), and INTL ( (Weng et al., 2024). Addition- ally, we implement our method using the popular ViT-B/16
7

---

Volume-Aware Distance for Robust Similarity Learning
Table Classification accuracy (%) of all methods on ImageNet-100 and ImageNet-1K IK datasets. The batch sizes are set to 1024 and 512 for R0 and ViT- B/16 backbones, respectively. Here the best and second-best results are bolded and underlined. respectively.
FCI00
Plant
ImageNet- 100
ImageNer-IK IK
METHOD
100 spochs
400 epochs
300 epochs
800 epachs
S-W 1-8
5-W
5-W
5-W 3-S
3-8
1-$
k-NN
Top-1
Top-5
k NN
Top-
Top-5
K-NN
Top
Top-5
E-NN
Top-1
Top
SimCLR(Cher 2020)
38.45
54.93
67.29
85 29
55.9
61.3
78.6
70.6
25.2
92.1
64.2
67
87.9
66.1
69.3
89.6
Res. R
BYOL/GHill et al. 2020)
35.49
55.91
65.92
85.91
56.3
65.5
77.8
09.2
73.2
90.1
66.9
71.2
90.5
67.
73.2
91.5 5
Res. 50/ 2MMB
PCLiLie al., 20211
39.92
56.12
68.24
87.19
55.9
M1
22.2
21.5
76.1
93.2
59.5
66.5
86.7
62.
70.5
90.5
Bes.50/ 4751
Sa:AA/Camm ot al. 200m
40.19
58
69.29
58.2
61.
79.4
72.1
75.8
92.9
65.4
23.1
91.2
65.7
25.3
91.5
Res.50 23MR
HCL//kabinson al. 2021)
41.39
59.92
70.29
55.9
00.8
79.3
70.2
74.6
642
11.2
91.2
672
71.7
90.7
Res.50/ 22MB
GCAcChea
20340)
60.5
72.8
75.6
91.2
67.4
67.8
76.6
Res.50/ 23MB
INTL Wesg et al. 20144
44.96
62.49
17.05
91.92
60.1
66.5
78.1
69.5
76.3
92.8
67
73.3
91.7
65.8
75.2
91.7
Res.50/ 23MB
VADSL (ours, cluster- free)
45.92
63.59
77.95
92.95
61.6
67.1
79.8
73.9
76.5
93.8
68.9
13.7
92.2
68.3
76.3
92.4
Res.50/ 23MB
VADSL (ours, cluster-used)
45.96
64.19
77.95
93.19
62.T
67.3
80,9
74.3
77.8
94,8
68.8
74,9
923
69.3
77.6
93.8
Bes.50 / 22MB
/ et 2020)
37.01
57.45
67.59
69
57.2
62.8
77.9
72.1
76.9
93.8
66.6
11.4
91.2
68.2
74.2
VIT 16 /8MB
Caron et al.
73.91
90.78
27.8
VIT 48MB
DINOICanon
202))
67.5
818
78.2
05.5
76.1
92.4
76.2
78.2
94.7
(Z et al. 2037;
61.5
68.2
77.5
78.5
95.1
75.0
76.0
VIT 4MM
MIl iBOThLi
2024)
62.3
82.5
94.8
724
15.4
83,9
VIL 48MB
VADSL (ours, cluster-free)
46 46.56
65.
65.34
28.59
65.4
79.5
81
96.5
72.7
77.5
T
79.1
82,1
95.6 6
VIT 48MB
VADSL (ours, cluster used)
47.12
65.45
79.69
93.78
64.6
69.8
83.9
80.7
83.4
97.8
73.8
79.6
93.9
80.5
83.4
96.9
V/L.1/ 4MM
lal. Vialin ploton DC
(bl. Villi on PTC
0.3
Table S. Classification accuracy rates (%) of all compared methods on the STS dataset including five tasks and the corresponding average scores.
CHUCCionppiCL HCL aistiieppe oRO WADSE
0.2
CHoCtGGmpphE HCL intuiinptg OLD VADSE
tsl. Violin pioc on IM WDB
Idl. Viein plet on IN DeM
METHOD
SIS12
STS13
STS14
STS15
SIS16
Aver.
Aocuniyy
SimCSE (Gao et al., 2021)
68.69
82.05
72.91
81.15
79.39
76.84
PCL (Chon et al., 2022h)
72.74
83.36
76.05
83.07
79.26
78.90
Inf.Min (Chen el al., 20226)
70.22
83.48
75.51
81.72
79.88
CHAUCtGnpppi HCL tristinepe ORO WADSE
0.2
CHGCAGGMphhC HCL InluCinephg OLD VADSE
miCSE (Klein & Nabi, 2022)
71.71
83.09
75.46
83.13
80.22
78.72
fex. Violin RDT
07. Violin piot on PROTEINS
SCL (Wu al., 20226)
72.86
84.91
76.79
84.35
81.74
80.13
ADNCE (Wu et al., 2024)
72.83
81 1,88
74.43
85,88
81.88
79.38
Accurayy
VADSL (ours)
72.74
85,54
78,32
87,85
82.44
81,38
CHUcttonppicc HCL aristivepte ORO WADSE
0.4
CHGCAGGephhL HCL IntoGinppig OLD VADSE.
Igt. Vislin plat IC
thi, Vinii plct MUTAG
all eight datasets. Moreover, compared with other graph contrastive learning approaches, JOAO, CI-GCL, and our method can perform relatively better. In most cases, our VADSL surpasses all compared methods with higher accu- racy mean and lower accuracy variance.
Accunay
CAUCIIonppics HCL iniotieppe ORO WADSE
6.8 CHGCHGGMpCCL HCLL Intocrrepig OLD VADSE
Figure 6. Violin plots (with mean values) of compared methods on graph embedding tasks including eight popular datasets.
6. Conclusion
most cases, where our method also achieves the best aver- age score in all compared methods. These results suggest that our method can work well for the text data and VAD leads to enhanced semantic understanding.
In this paper, we introduced an extension to the conven- tional data point by representing each instance as a data ball, endowing it with a volume value. This led to the natural definition of the VAD metric, which computes the geomet- ric proximity between data balls, allowing the relationship among unsampled instances within these data balls to be effectively captured. The incorporation of volume expan- sion regularizer further emphasized the utility of volume- awareness in enhancing the model generalizability. VAD is a general technique that can be easily integrated into both supervised and unsupervised learning tasks with negligi- ble computational overhead. To the best of our knowledge, this is the first work in similarity learning that considers the instance volume. We provided comprehensive theoreti- cal analyses that guarantee the effectiveness of our method. Experiments on real-world data across multiple domains indicated that our learning algorithm acquires more reliable features than state-of-the-ar methods. In the future, we plan to apply VAD in broader paradigms of machine learning.
Graph Data. We further evaluate our method on a chal- lenging graph embedding task using biochemical-moleeule data and social-network data, including DD, PTC, IMDB- B, IMDB-M, RDT-B, PROTEINS, NCI1, and MUTAG (Ya- nardag & Vishwanathan, 2015). We use the representa- tive method InfoGraph (Sun et al., 2020a) as the baseline and perform downstream graph-level classification on these datasets. For evaluation, we fine-tune an SVM (Cortes & Vapnik, 1995) on the learned feature representations using 10-fold valsdaaaiit The dataset is split into training, test, and validation sets in an 8/1/1 ratio. The accuracy results are reported after 10 runs. Our compared methods include HCL, GraphCL (You et al., 2020), JOAO (You et al., 2021), and CI-GCL (Tan et al., 2024). From violin plots in Fig. 6, VADSL consistently improves InfoGraph across
8

---

Volume-Aware Distance for Robust Similarity Learning
Impact Statement
Chen, S., Gong, C., J., Yang. J.. Niu, G., and Sugiyama, M. Learning contrastive embedding in low-dimensional space. Advances in neural information processing sys- tems (NeurlPS), 2022a. 2.2
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Chen, S., Zhou, J., Sun, Y., and He, L. An information minimization based contrastive learning model for un- supervised sentence embeddings learning. In Interna- tional Conference of Computational Linguistics (COL-
Acknowledgment
The authors would like to thank the anonymous reviewers for their critical and constructive comments and suggestions. This work was supported by the National Science Fund of China under Grant Nos. U24A20330, 62361 166670, 62336003, and 12371510.
ING), 2022b. 5.3,5
Chen, S., Niu, G., Gong, C., Koc, O., Yang. J., and Sugiyama, M. Robust similarity learning with difference alignment regularization. In International Conference on Learning Representations (ICLR), 2024a. 1, 5.2, 5.1
References
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International Conference on Machine Learning (ICML), pp. 1597-1607, 2020. 2.1,,,1,, 5.3, 4
Aginre, E., Banca, C., Cer, D.. Diab, M., Gonzalez Agirre, A., Mihalcea, R., Rigau Claramunt, G., and Wiebe, J. Semeval-2016 task I: Semantic textual similarity, mono- lingual and cross-lingual evaluation. In International Workshop on Semantic Evaluation, 2016. 5.3
Chen, Z., Lin, C.-H., Liu, R., Xiao, J., and Dyer, E. L. Your contrastive learning problem is secretly distribu- tion alignment problem. Advances in neural information processing systems (NeurIPS), 2024b. 5.3,
Arpit, D., Zhou, Y., Ngo, H., and Govindaraju, V. Why regularized auto-encoders learn sparse representation? In International Conference on Machine Learning (ICML), pp. 136- 144, 2016. 2.2
Chu, X., Lin, Y., Wang, Y., Wang, X., Yu, H., Gao, X., and Tong, Q. Distance metric learning with joint repre- sentation diversification. In International Conference on Machine Learning (ICML), pp. 1962-1973, 2020. 2.1,
Baldi, P. and Sadowski, J. Understanding dropout. Advances in neural information processing systems (NeurIPS), 2013. 2.2
5.2, 5.1
Chuang, C.-Y., Robinson, J., Yen-Chen, L., Torralba, A., and Jegelka, S. Debiased contrastive learning. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020. 2.1
Bjorek, N., Gomes, C. P., Selman, B., and Weinberger, K. Q. Understanding batch normalization. Advances in neurul information processing systems (NeurIPS), 2018. 2.2
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual fea- tures by contrasting cluster assignments. arXiv preprint arXiv:2006.09880 099882 pp. 1401-1413, 2020. 5.3, 4
Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Interna- tional conference on artificial intelligence and statistics (AISTATS), pp. 215-223, 2011. 5.1
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021. 5.3,4
Cortes, C. and Vapnik, V. Support- vector networks. machine learning. Machine Learning, 1995. 5.3
Davis, J. V., Kulis, B., Jain, Sra. S., and Dhillon, Information coommation-theorttic metric learning. In International Conference Machine learning (ICML), pp. 209-216, 2007. 2.1
Chen, S., Luo, L.., Yang, J., Gong, C., J., and Huang, H. Curvilinear distance metric learning. Advances in neural information processing systems (NeurIPS), 2019. 2.1
Deng, J., Guo, J., Xue, N., and Zafeiriou, S. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 3, 5.2
Chen, S., Niu, G., Gong, C., J., Yang, J., and Sugiyama, M. Large-margin contrastive learning with distance po- larization regularizer. In International Conference on Machine Learning (ICML), pp. 1673-1683, 2021. 2.1, 4.1
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-train neetrriniigg of deep bidirectional transformers for lan- guage understanding. arXiv: orxiv:/8/0.. aet::08000.00 2018. 5.3

---

Volume-Aware Distance for Robust Similarity Learning
Dong, W., Shi, G., Li, X., Ma, Y., and Huang, F. Com- pressive sensing via nonlocal low-rank regularization. IEEE Transactions Image Processing, 23(8):3618- 3632, 2014. 2.2
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learnin (ICML), pp. 1184456,440 2015. 5.1
Feng, L., Shu, S., Lu, N., Han, B., Xu, M., Niu, G., An, B., and Sugiyama, M. Pointwise binary classification with pairwise confidence comparisons. In International Conference on Machine Learning (ICML), 2021.
Kaya, M. and Bilge. Ş. Deep metrie learning: A survey. Symmetry, 11(9))1066, 2019. 1, 1, 2.1
Gemelmacher-Shiizeeman I., Seitz, S. M., Miller, D., and Brossard, E. The megaface benchmark: million faces for recognition at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4873-4882, 2016. 5.2
Feng, L., Shu, S., Cao, Y., Tao, L., Wei, H., Xiang, T., An, B., and Niu, G. Multiple-instarn instance learning from unlabeled bags with pairwise similarity. IEEE Transactions on Knowledge and Data Engineering, 2023.
Kim, S., Kim, D., Cho, M., and Kwak, S. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3238-3247, 2020. 5.1, 5.1
Fraigniaud, P., Lebhar, E., and Viennot, L. The inframetric model for the internet. In IEEE INFOCOM 2008-The 27th Conference on Computer Communications. IEEE, 2008. 3.1
Klein, T. and Nabi, M. micse: Mutual information con- trastive learning for low-shot sentence embeddings. arXiv preprint arXiv:2211. 011221..4922. 2022. 5.3, 5
Furusawa, T. Mean field theory in deep metric learning. In International Conference Learning Representations (ICLR), 2024. 2.1, 5.2, 5.1, 5.2
Kou, X., Xu, C., Yang. X., and Deng, C. Attention-guided contrastive hashing for long-taile anigtailedd image retrieval. In International joint conference on artificial intelligence (IJCAI), 2022.
Gao, T. Yao, X., and Chen, D. Simese: Simple con- trastive learning of sentence embeddings. arXiv preprinl arXiv:2104.0882,, 2021. 5.3,5
Krause, J., Stark, M., Deng. J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In 3dRR, 2013. 5.1.
Ge, W. Deep metric learning with hierarchical triplet loss. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 2.1, 4.2
Krizhevsky, A., Hinton, G., et al. Learning multiple layers features from tiny images. 2009. 5.1
Grill, J.,B., Strub, F., Altche, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 33:
Li, J., Zhou, P., Xiong, C., Socher, R., and Hoi, S. C. Pro- totypical contrastive learning of unsupervised represen- tations. In International Conference on Learning Repre- sentations (ICLR), 2021. 5.3,
21271-21284, 2020. 5.3,4
Li, Z.-Y., Hu, Y.-S., Yin, B.-W., and Cheng, M.-M. Multi- token enhancing for vision representation learning. arXiv preprint arXiv:2411.15772 2024. 5.3, 4
Han, K., Wang. Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., et al. A survey vi- sion transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 3.1
Liao, C., Tsiligkaridss, T., and Kulis, B. Supervised metric
learning to rank for retrieval via contextual similarity optimization. In International Conference on Machine Learnin (ICML), 2023. 5.2, 5.1
Hastie, T. The elements of statistical learning: data mining, inference, and prediction, 2009. 3.1
Hc, K., Zhang, X.. Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 3.1,
Lim, J., Kim, Y., Kim, B., Ahn, Shin, J., Yang, E., and Han, S. Biasadv: Bias-adversarial augmentation for model debiasing. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition (CVPR), 2023. 2.2
Huang, F. Chen, S., and Huang, H. Faster stochastic al- ternating direction method multipliers for nonconvex optimization. In International Conference on Machine Learning (ICML), pp. 2839-2848, 2019. 3.2
Liu, W. and Tsang, I. Large margin metric learning for multi- label prediction. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2015. 4.2
10

---

Volume-Aware Distance for Robust Similarity Learning
Liu, W. Wen, Y., Yu, Z., Li, M., Raj. B., and Song, L. Sphereface: Deep hypersphere embedding for face recog- nition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Sohn, K. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems (NeurIPS), 29:1857-1865,, 2016. 1,
2.1, 3.1, 5.1,5.1
3
Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkor- eit, J., and Beyer, L. How to train your vit? data, augmen- tation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270, 2021. 2.2
Liu, W., Xu, D., Tsang, I. W., and Zhang, W. Metric learning for multi-outpin tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018. 1, 3.1
Liu, Z., Luo, P., Qiu, S., Wang. X., and Tang, X. Deepfash- ion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 5.2
Sun, F.-Y., Hoffmann, J. H., Verma, V., and Tang, J. In- fograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximiza- tion. In International Conference on Learning Represen- tations (ICLR), 2020a. 5.3
Moschoglou, S., Papaioannou, A., Sagonas, C., Deng, J., Kotsia, I., and Zafeiriou, S. Agedb: the first manually collected, in-the-wild age database. In CVPR workshops, 2017. 5.2
Sun, Y., Cheng, C., Zhang, Y., Zhang, C., Zheng, L.., Wang, Z., and Wei, Y. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020b. 2.1
Oh Song, H., Xiang, Y.. Jegelka, S., and Savarese. S. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4004-4012. 2016. 1, 5.2
Tan, S., Li, D., Jiang, R., Zhang, Y., and Okumura, M. community-invariie graph contrastive learning. In Inter- national Conference on Machine Learning (ICML), 2024. 5.3
Tian, Y., Krishnan, D., and Isola, Contrastive multiview coding. In European Conference on Computer Vision (ECCV), pp. 1-18, 2020. 2.1
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems (NeurIPS), 32, 2019.
Van der Maaten, and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 5.1
Reddi, S. J., Hefny, A., Sra, S., Poczos, B., and Smola, A. Stochastic variance reduction for nonconvex optimization. In International Conference Machine Learnin (ICML), pp. 31 314-323, 2016. 3.2
Wang. X. and Qi, G.-J. Contrastive learning with stronger augmentations. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023. 1
Robinson, J., Chuang, C.-Y., Sra, S., and Jegelka, S. Con- trastive learning with hard negative samples. In Interna- tional Conference on Learning Representations (ICLR), 2021. 5.3,4 4
Wang. Z.,, Li, Q., Zhang, G., Wan, P., Zheng, W., Wang, N., Gong, M., and Liu, T. Exploring set similarity for dense self-supervised representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2.2
Russakovsky, O., Deng. J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M.. et al. Imagenet large scale visual recognition chal- lenge. International journal of computer vision, 115(3): 211-252, 2015. 5.3
Weinberger, K. Q., Blitzer, J., and Saul, I.. K. Distance metric learning for large margin nearest neighbor clas- sification. Advances in neural information processing systems (NeurlPS), pp. 1473-14800 2006. 1
Seidenschwarz, J. D., Elezi, I., and Leal-Taixe, L. Learning intra-batch connections for deep metric learning. In Inter- national Conference on Machine Learnin (ICML), 2021. 5.2, 5.1
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F.,
Belongie, S., and Perona, P. Caltech-UCSD Birds 200. Technical Report of Technology, 2010. 5.1,
California Institute
Sengupta, S., Chen, J.-C., Castillo, C., Patel, V. M., Chel- lappa, R., and Jacobs, D. W. Frontal to profile face veri- fication in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV), 2016. 5.2
Weng, X., Ni, Y., Song. T., Luo, J., Anwer, R. M., Khan, S., Khan, F. S., and Huang, L. Modulate your spectrum in self supervupere learning. In International Conference on Learning Representations (ICLR), 2024. 5.3,

---

Volume-Aware Distance for Robust Similarity Learning
Wu, J., Chen, J., Wu, J., Shi, W., Wang, X., and He, X. Understanding contrastive learning via distributionally robust optimization. Advances in neural information processing systems (NeurIPS), 2024. 5.3,5
Yan, J., Deng. C., Huang, H., and Liu, W. Causality- invariant interactive mining for cross-modal similarity learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 1
Wu, S., Liu, T., Han, B., Yu, J., Niu, G., and Sugiyama, M. Learning from noisy pairwise similarity and unlabeled data. Journal of Machine Learning Research, 23(307), 2022a.
Yanardag, P. and Vishwanathan, S. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international con- ference on knowledge discovery and data mining (KDD), 2015. 5.3
Wu, X., Gao, C., Su, Y., Han, J., Wang, Z., and Hu, S. Smoothed contrastive learning for unsupervised sentence embedding. In Proceedings of the 29th International Conference on Computational Linguistics (ACL), 2022b. 5.3,5
Yang, Y., Shen, H. T., Ma, Z., Huang, Z., and Zhou, X. L2, I-norm regularized discriminative feature selection for unsupervised learning. In International joint conference on artificial intelligence (IJCAI), 2011. 2.2
Yi, D., Lei, Z., Liao, S., and Li, S. Z. Learning face repre- sentation from scratch. arXiv preprint arXiv: 1411.7923. 2014. 5.2
Xing, E. P.. Pg, A. Y., Jordan, M. I., and Russell, S. Distance metric learning with application to clustering with side- information. Advances in neural information processing systems (NeurIPS), 15(505-512):12, 512):12, 2002. 1, 2.1
You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y. Graph contrastive learning with augmentations. Advances in neural information processing systems (NeurIPS), 2020. 5.3
Xu. C., Chai, Z., Xu, Z., Yuan, C., Fan, Y., and Wang. Hyp2 loss: Beyond hypersphere metric space for multi- label image retrieval. In Proceedings of the 30th ACM International Conference Multimedia, 2022a.
You, Y., Chen, T., Shen, Y., and Wang, Z. Graph contrastive learning automated. In International Conference on Ma- chine Learning (ICML), 2021. 2.2, 5.3
Xu, X., Yang, Y., Deng, C., and Zheng, F. Deep asymmetric metric learning via rich relationship mining. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b. 3.1
Zadeh, P., Hosseini, R., and Sra, S. Geometric mean met- ric learning. In International Conference on Machine Learnin (ICML), 2016. 2.1
Yan, J., Luo, L., Deng, C., and Huang, H. Unsupervised hy- perbolic metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Zhang, D., Li, Y., and Zhang, Z. Deep metric learning with spherical embedding. Advances in neural information processing systems (NeurlPS), 2020. 3.2
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. 2018. 2.2
Yan, J., Luo, L.,, Xu, C., Deng, C., and Huang, H. Noise is also useful: Negative perrelation-steeree latent con- trastive learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2022a. 2.1
Zheng, M., You, S., Wang, F., Qian, C., Zhang, C., Wang, X., and Xu, C. Ressl: Relational self self-supervised learning with weak augmentation. Advances in Neural Information Processing Systems (NeurIPS), 2021. 2.2
Yan, J.. Yang. E., Deng, C., and Huang. H. Metricformer: A unified perspective of correlation exploring in similarity learning. Advances in neural information processing systems (NeurIPS), 2022b. 1, 1, 5.2, 5.1
Zhong. H., Chen, C., Jin, Z.. and Hua, X.-S. Deep ro- bust clustering by contrastive learning. arXiv preprint arXiv:2008.030% 000000000000, 2020.
Yan, J., Luo, L.., Deng, C., and Huang, H. Adaptive hierar- chical similarity metric learning with noisy labels. IEEE Transactions on Image Processing, 32:1245-122 1256, 2023a. 2.1
Zhou, J., Wei, C., Wang. H., Shen, W., Xic, C., Yuille, A., and Kong, T. ibot: Image bert pre-training with on- line tokenizer. In International Conference on Learning Representations (ICLR), 2022. 5.3, 4
Yan, J., Yin, Z., Yang, E., Yang. Y., and Huang, H. Learn- ing with diversity: Self-expanded equalization for better generalized deep metric learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), 2023b. 1, 2.1
12

---

Appendix for "Volume-Aware Distance for Robust Similarity Learning"
Appendix
This part is the appendix of our manuscript. It includes the additional experiments and the mathematical proofs of theorems.
A Additional Experiments
A.1 Additional Experiments on COCO dataset
We would like to further investigate the transferability of our method on the object detection and instance segmentation tasks. We first pre-train the model (with ResNet-50 backbone) on ImageNet-IK: and then fine-tune the pre-trained backbone on the new dataset. Specifically, select COCO (Lin al., 2014) as our target dataset and follow the common setting (as discussed in MoCo-v3 (Chen al., 2021)) to fine-tune all layers of the pre-trained model while evaluating the performance on the va12017 set. We employ Faster R-CNN (Ren al., 2015) and Mask R-CNN (He et al., 2017) as our backbone for detection and segmentation, respectively. We implement our method based on the loss functions of SimCLR (negative-used) and YOL (negative-free). As listed in Tab. A1, our VADSL shows considerable improvement over MoCo-v3 and DINO on both two recognition tasks. This indicates that our method not only works well on classification-orinnti tasks but also on more natural image-related recognition tasks.
the train2017 set
Table AI. Performance of all methods for transfer learning tasks: object detection and instance segmentation COCO dataset.
Object Detection
Instance Segmentation
METHOD
Aph
AP
AP
Apmk
AP
AP
Supervised
38.2
59.1
41.5
35.4
56.5
38.1
BYOL (Grill et 2020)
39.9
60.2
43.3
36.5
58.4
39.1
SwAV (Caron et al., 2020)
40.3
61.5
44.4
36.3
58.7
39.4
MoCo-v2 (Chen et 202)))
2020)
37.6
57.9
40.8
35.3
55.9
37.9
MoCo-v3 (Chen et al., 2021)
39.9
61.2
43.2
36.5
58.1
38.8
DenseCL (Wang et al., 2021)
40.3
59.9
44.3
36.4
57.0
39.2
DINO (Caron et al., 202.))
40.3
62.0
44.1
36.8
58.8
39.2
INTL (Weng ct al., 2024)
40.7
60.9
43.7
35.4
37.3
37.6
VAD (neg--used)
43.1
43.4
37.6
59.4
40.2
VADSL (neg.frree)
42.1
63.1
45.1
36.5
58.5
39.7
A.2 Additional Experiments on BookCorpus dataset
For the BookCorpus dataset which includes six sub-tasks movie review sentiment product reviews (CR), subjectivity classification (SUBJ), opinion polarity (MPQA), question type classification (TREC), and paraphrase identification (MSRP), we follow the experimental settings in the baseline method quick-thought (QT) (Logeswaran Lee, 2018 8) choose the neighboring sentences as positive pairs. Then, further compare our VDASL with DCL, HCL, CO2 (Wei al., 2021), UnReMix (Tabassum et al., 2022), and ADNCE (Wu et al., 2024), and the corresponding average classification accuracy rates shown in Fig. A1.
A.3 Running Time Comparison
In our learning framework, we have additional measure-hean network as well the corresponding VER regularizer in the learning objective. We would like to investigate if the efficiency of the learning algorithm will be affected by the additional calculations. Here we further provide experiments to record the training time of our method as well as the corresponding baseline method. Specifically, we use two NVIDIA TeslaV100 100 GPUs to train our method based on SimCLR and SwAV with 100 epochs, respectively. For each case, we set the batch size to 512 and 1024.
In Tab. A2, we can find that the proposed new measure-heat and the new regularizer only brings in very little additional

---

Appendix
100
QT
DCL
95
HCL
CD2
90
UnReMix
ADNCE
VADSL tours)
85
Accuracy
80
75
70
65
60
MR
CR
SUBJ
MPQA
TREC
MSRP
Figure AI. Accuracy rates (%) of all methods on BookCorpus dataset including six text classification tasks.
time consumption. This is because the calculations of measure-head and VER are independent to the size of training data, so the training time is completely acceptable in practice use.
A.4 Parametric Sensitivity
As we only introduced one additional hyper-paramann parameter in our method. Here we want to simply investigate the parametric sensitivity of the regularization parameter À in our learning objective. Specifically, we change À in [0.01, 5]. and we record the classification accuracy of our method on STL-10 and CIFAR-10 10 datasets (batch size=256/512/100 epochs= Tab. A3 shows that the accuracy variation of our method is smaller than 1.5%. These results clearly demonstrate that regularization parameter À is relatively stable within a given range. It implies that the easily tuned in practice
of our method can be
Table A2. Training time of the baseline methods and our proposed method (100 epochs, in hours).
CIFAR-10 10
ImageNet- 100
ImageNet- IK
METHOD
512
1024
512
1024
512
1024
SimCLR
2.3
1.3
10.9
5.5
70.1
35.2
SwAV
2.6
1.7
11.5
5.8
71.2
36.7
VADSL (SimCLR+)
2.4
1.5
11.2
5.6
71.5
35.6
(SwAV++
2.7
1.9
11.9
6.0
72.1
36.9
Table A3.. Parametric sensitivity of À the STL-110 and CIFAR-10 10 datasets (%).
Here À is changed within (0.01, 5].
dataset (batchsize)
0.01
0.1
1.5
STI.-10 (256)
76.8
77.8
78.1
77.9
76.9
STI.-10 (512)
78.1
78.8
79.5
78.5
78.2
STL-10 (1024)
81.5
81.9
82.1
81.9
81.5
CIFAR- (256)
87.9
88.5
89.3
88.9
CIFAR-10 (512)
91.5
91.9
92.3
92.5
91.8
CIFAR- (1024)
93.2
93.6
94.5
93.6
93.1
2

---

Appendix
B Proofs
B.1 Proof for Theorem
Proof. For the gradient boundness of let us firstly assume that (2) VV2(()) [m - for any Rdd Then we have that
V "pCemp(P: H)
Ee= Pp.
-e Σj-0e Po.v(a.mu, p..,, x²t)
= 11,,,
c Do.u(a, 20)
En=0e Pu.v(mm,
L Da.v(a, 1/7 2ij=0 -
p,, Tby))
Σラ=0 Po.v(o..mu))))
1 Po. x 1/7 Σ) Po. Do.v 1/7 D x²)) p, 2b,))
Re, (b537
e (( 2/// Ex=0e Pa.v (armm 1/1)
and thus!
V Cemp (,: H)112
Do. -D.0(0000,))1 2
Le ve. 1/7 En 2z=0e
< max
Au D (xo, x+)- V
x 1/7 Ej=oee Del
n
(n 2)² max (iV xt )12: D 1aa )1 ..,, 1V p.vo Dbo
...,,
(2r, 11.)
72
n
(n
+2)2DD
72
= 82.
Meanwhile we have that
V 9 Rexqund (4,, H)
Σ "
Vama (26 ,=
V)p(x).
(0.1)
0H.
and thus
|V 4 Rexpand (4, H)ll
@V([x,) )
Σ 1ー 0H
V x3)
@V((rn))
) ((4(1)) (z:
/=1 0H
n
82.
(0.2)
Forr details, here De, 2) (a.
(D)) ( v(
max([lyp()) (m)) Ex= (z))()) p(m)) E ( + mmxx(())(m)) - vpp() V 4m 28 8 28 =
3

---

Appendix
Similarly, we obtain that
V[ WV Comple. H)ll z 83, and R H)ll2 S 82.
(0.3)
Finally, we have that
I s, H)ll2
V Cemp (4, H) + A V Rexpand ( ( 22)12
II(V o Cemp (40, H) + AV R expanad (40, H), Vn cmp H) xV R (4; H())11
< | v Comp (,; H)ll2 + X²1 V expand (4; H) 12 + VV (4P: H)ll3 + X2 V n R expand (4, H)ll
= 61 + 62 + 53 + 64
(0.4)
which shows that F(P, H) is always gradient-bounder For the Lipschitz smoothness of F, let us also assume that there exists L > 0 such that [[(p(>) vQ(x)]le & Lll G11.. Then we have that
V p, H Cemp (4, V Lemp (p: F0)1
V Comp (8, Hi) - +
/ W H) ( RD]l
(4, (4, H) a < max ( Σ
Vyp Comp
(x)) + (((r)
-(0 V
jml
-max ()))
(x)( + (p))
V ( (2 = + vQ(3)
-
+1-1
12 S {max(2)p(@) 22 Σ 5=1
[x) (x)()) + (2ap)))))22mm((((((m) 2max((/V<(()) -Vp(==) [Vp(()) -
< 44L 1p - 1l12 + AL|p G112 +1- 11 - -i
< 4(n ( H) (G, 1)..
(0.5)
and the proof completed.
B.2 Proof for Theorem 2
We first introduce the following lemma for proving Theorem 2.
Lemma 1. For independent random variables t1, f1,t2....,44 tn E T and a given function w T + R. Vog e
T
(i = 1,,,,...., the function satisfies
lis (n) - w(ln.... (.....,,,)) [6,] S Pi
(0.6)
then for any given p 0, it holds that Pt Pllw(t,....,,,, E[w(t,,.... tn)]l p 2c 24 2///E111
Proof. We prove Theorem 2 by analyzing the perturbation (i.e., Pi in the above Eq. (0.6)) of the loss function Comp-
We denote that
e De..Ni2..
= Gemp H1 2.)
log - 1/7 +Σ=ze m, 7
(0.7)
N
4

---

Appendix
and
Wr N
1
Σ
log
xt)7 Pv. ( xt 1/7 L (o. 20,3/7'
- 1/7
log D-DoNN(2. -z= Do. (2,
Do. Do..x 10,
(0.8)
where (3, (To, 1 is an arbitrary mini-batch from the sample space. Then we have that
w
1
Ds (2, 1/77
Dv. Pp..((r.
log
log
N
De. H a 3/2 j-==
12
De 1., 3/7
De. 1/7 Ei= D 10 B5
P (ā, a 1/7 Dj (
j== D Drin (er, Dp.u(er:2s,)//
1
log
N
Pa, 2t 1/7 D 14. (0,01)/ 1/7
Σ}=ıe - 1/7)
w(n)log(1 + max dup (t, 1) 1), E 227)
(0.9)
2CAN
where (n)) = log ( 11) Meanwhile, we have
e (p,, (w.,
-De x0 1/7
log e De (21, x 1/7 ΣE=
( 20,)/7 -E
De, -log
1/7 wl)
Σ}=z Do D ( m0,117
Lemp ( Hp: 2) - Cemp (40; H: ②)
(0.10
By Lemma 1, we let that for all i = 1, 1, ... N
P: w(n)log(1 + max (t, t) E 27))
(0.11)
2CAN
so that have
P
P { Complyp (emmplyp:00 2)) Complp:: (p: w(n)log(1 max (t,8) 27))
In(2/5)
2CA
2N
1 2c 24²/ EN p
2 2M -2N (mnnr///m///)) 2 1 - 2e max? (u(n]lig(11)) +maxt (d
2N 22 (VIn22/0)
= 1 - 2e
1 2e 1n(2/8)
1 5,
(0.12)
where 7) w(x)log(++mxx do (t) and μ (lI(2)0))/(0())N) The proof completed.
2CA
B.3 Proof for Theorem 3
Proof. We prove the theorem via mathematical induction.
i). We first validate that the inequality holds for (a1, b1) > (az, b2). To be specific, we suppose that D (x Ib,) Dip.y (Dags H.,). Then we let
Vs (Dogg) = (2(
( SV((xa))
(0.13)
Vs Vs(Bo,) V(Eo,),
(Db,) = V((tb,) ):
5

---

Appendix
and we have that
lim Ds.vs vs (Faz Tb22) S-++00 Do. Vi (Bay 2b,))
1yp((ra)) (0(xbb)) (10b2)l222 (1 <0(x0b,) )
e ei J+V(o)))) 0,
lim +
(0.14)
which implies that there exists sufficiently large S such that
De.Vy (Tag, Lbz) s Ds. (F Db,))
(0.15)
Therefore, the case of (a1, 61)= > (az, b2) clearly holds.
ii). By assume that the inequality holds for (a1, (1) > (az, 62) > .... > (ax, bx), show that the inequality can be still satisfied for (a1, b1) b (a2, b2) .... (ax+1, bk+1). Specifically, without loss of generality, we assume that
)bb) 2 ( Lbyy 2 ... (.ra, (b.)
2 ( Bbx+1) 2 D,p,v(Past.. Tbit1 2 ... 2 De, Tbk):
(0.16)
where 0 <is K 1. Then we let
Vs(Faxt1) ) (V(
Vs(xbxr1) = 5V(2bx+1)
(0.17)
Vs( = V(Daz);
Vs( = V(20b)) )
and thus we have
De..us (Fax Lbx
lim
800+00
Do Vs (I Lb,))
((((0a)11) - (((bbxt1)ll
=
8-++00
lim
)){((Za,) - :p(2%,)))
5(V(aox 11 V( 11
=0, 0,
(0.18)
where j = 1,2,.... K, and at most one of the conditions aj E fax+1, bx+1) and by E {ax+1, bx+1) can be satisfied due to the fact that (ak-1, bx+1) & {(as, b1), (02, b2), "..,..,,,,,, (ak, bk)). Therefore, there exists sufficiently large S such that
Vs (#ax+1: :bx+1 min(DDe.ve. (aar Lb, D...v. (Ians Tb2). ......... D Vg- (Fax 3b ))
(0.19)
Then we need to resort (Dp.vs- (Za, Lb, D Vg- (Iaz, Fbb), Dip Us- (æax Tbh)) ). To be specific, we further con- struct that
V/ (Far) VSV(Iex):
V8 (abbR) <IV((xbx)
(0.20)
V (aa,) VVs(Ka,) ) Vs(
V (Ib,) (2x,)
and we have that
se, vys (Fax: Dbx)
lim
$-++00
D
p.yy (Fag: 1hyy)
lim 1( (p(xxxx) )l1 s-++00
||y((xa,) s0(2,, 112
ev ((V(( ++((zgg))
=0, 0,
(0.21)
where 1,2,... K and at most one of the conditions aj E {ax, dxr and by {ax, (ak,bx) can be satisfied due the fact that (ax, bx) ((a1, b1), (az, b2).... (0x-1, bk-1)). Therefore, there exists sufficiently large S₁ such that
G. (Fox
(F Rbxx)
min(D
v.1/ 3 (I Dbb ), D s v. wy (Z Dbo)..... D v,v (Fax Lox Lbx-1)}
(0.22)
6

---

Appendix
By letting S S1, we have that
D
1p, V. vy
(ax+11 Pbx+1)
s Vy. (
(Fax Fbxx
< min D
4, (Dar Lb,), D v.Vg. (Fezi Fbe): 2.V..e (Tox Dbx-1)
(0.23)
By further constructing Eq. (0.20) - times for (ak-1, bk- (ak-2, bK-2), ...., (a1, b1), we finally have that ... S Pv, (Ear Lb,)
D P (Tak41 Lbx+1) S Dop (Iax Tbx) ...
(0.24)
which implies that the inequality holds for (a1, b1) (a2, b2) 7 ... (ax+1, bk+1).
iii). By integrating i) and ii), we have that Dv. v. (Bay Lb,)) 2 D.p (Zaz: Why) 2 ... 2 De..-- (#c3, can be satisfied for (a1, b1) > (az, b2) > (acg, bez The proof is completed.
B.4 Proof for Theorem 4
Proof. We firstly show that for the given dataset L, there exist positive constants Vnnn and Dinns such that the learned encoder yp and measure-head HH" (corresponding to the volume function V*)
(x,, 2 Vmnn Vi = 1.2..... .,,,
(0.25)
and
De v- 22) 2 Dmnn: i
(0.26)
This is because that Fly"","" S F(6, H., namely
D N (a, x 2//
Σ e -
n
1og -D P Eb, 17
AB 1
Σn ==
Do N- (a, 1/7
e
D D (n, x x////"
AIE E -V(aus)
< |m,(b;77-1
-log -D. //1 //
(0.27)
-D P
and thus
V" (m)s))
e D xl. 1/7
DD N: (20
< (b,
llog
+ +oo
e
-D 2 1/7 1/7+5" Ei= Pomm(am,
e Do
1/ 5/ Ez=1e Do N (an (m
nn
Σn -D(ON))]
Ee_
e --r
+ AE (b:))
(0.28)
so that
y" (x(x))
D (0,
e D x (a, x0)x=))//>
> -log
log
+log
Do (2, (xe
- (n. 1/7
-D N- (0, 1/2 Σソ= D'p u (a 1/7
e
-V(on))) 1
Σn= (as,))
+AIE
+AIE(b;) p=1
c
AE (b, 11540
/=1
e D (20, (20 +1/
n. (2,x0 3/7 E==ze Da. (1 1/7
2 log
En, (br)
(20 1/ (0, 1/4
-D' m (m,00,)/7
(0, 1/7
e
1
(0.29)
AE (1517=1
Σ)= e -)(as)))
AE (b; AAE(b, 1
Σラ= v

---

Appendix
So easily have that
V"(ab,
log
X (b,15-1
23 j=1 1 e
-Vles,)
AIE b E e (o
1
2 log
Ae V(
)
log
P(mse))
logl
지 c F(wb,)) )
+ log
V(xbba) )
Vnin-
(0.30)
Meanwhile, have
Det x (0, 1/4 Ei -o--0 - (s, )/7
E (b,)
log
Po x- (0, x0 x07
è D Pu, x
Σ
n
:(Eng))
< Bb,(b;))
logg
AE (hy (b:)
D (e, 1/7 2/
==1
Fr1
(0.31)
which implics that
I e Dor (z, exxp(EE) (e (o "" j$L
(2,
(0.32)
and thus
Dip-. H- (x, 2by})
2 ~ooog
"/.x0 Po- 1/7 jzh DD (20
= ylog
e Pornn 1/7exp(E1) - (e Pe (0mm,a))//440 1, -Dw- x (20 (/7)
2 >log
D (,
2 7log
D a2)
-Do-.'' (-.H-(a; ax = D
(0.33)
8

---

Appendix
Then go ahead show the finite coverage. To be specific, the union volume of all N data balls ( V (x1)), B(æz, (xx))) ..... B(xN, V (xN)) be calculated as
sign(z U B V(x.))]dz i=1
sign(z | E B(a,, V(x,))]dz Σ icj
sign(z E B(x,, v(a:)) B(x,, V(zy)))dz
Σ
sign/z B( V(xx)) 0 B( v(z,))) n B(ox, v(ax))) v(zx))ddz-... .... sign(z N B(x, v(x;)))d
i<jck
N
(f e B(x, v(z,)) 0 Blz,, ((zz)))
sign/z E B(x,, ((xx)))dz= - N max s<j
i=1
NV
Nu(Dmin) V
N(1 (0Dmmnn) min)) mmin.
(0.34)
where mmmm E (0, 1). Finally, we let
L U m
N (1 (((Din))) Vmin
(0.35)
and have that
J2 sign is E EUit B(x,, ((x;)))d
JEEJ ") 1dz
1
2 N(1 p ((Dminn)
[L. Um
p - U
2 (1 pl (Dmmn))))m
(1 Vmin
L U mm
= P.
(0.36)
which completes the proof.
References
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0988 pp. 1401-1413, 2020. Al
Caron, M., Touvron, H., Misra, I., Jégou, H.. Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised
vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. A1
Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.00 04297, 2020. A1
Chen, X., Xie, S., and He, K. An empirical study of training self-supervised vision transformers. In Proceedings the IEEE/CVF International Conference Computer Vision (ICCV), 2021. (document), Al
Grill, J.BB., Strub, F., Altche, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z.. Gheshlaghi Azar, M., al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurlPS), 33:21271-211
2020. Al
He, K., Gkioxari, G., Dollár, P., and Girshick, Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2961-2969 2017. (document)
Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects context. European Conference on Computer Vision (ECCV), 2014. (document)

---

Appendix
Logeswaran, L. and Lee, H. An efficient framework for learning sentence representations. In International Conference on Learning Representations (ICLR), 2018. (document)
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015. (document)
Tabassum, A., Wahed, M., Eldardiry, H., and Lourentzou, I. Hard negative sampling strategies for contrastive representation learning. arXiv preprint arXiv:2206.01 197, 2022. (document)
Wang, X., Zhang, R., Shen, C., Kong. T., and Li, L. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2021. A1
Wei, C., Wang, H., Shen, W., and Yuille, A. Co2: Consistent contrast for unsupervised visual representation learning. In International Conference Learning Representations (ICLR), 2021. (document)
Weng, X., Ni, Y., Song, T., Luo, J., Anwer, R. M., Khan, S., Khan, F. S., and Huang, L. Modulate your spectrum in self-supervised learning. In International Conference on Learning Representations (ICLR), 2024. A1
Wu, J.. Chen, J., Wu, J., Shi, W., Wang, X., and He. X. Understanding contrastive learning via distributionally robust optimization. Advances in neural information processing systems (NeurIPS), 2024. (document)
10


Reviews:
[{"review_id": "wq2BDb097m", "reviewer": "Reviewer_P61C", "summary": "The paper introduces Volume-Aware Distance (VAD), a new similarity metric that generalizes conventional point-wise distance by incorporating volume information. The authors propose a measure-head network to learn instance volumes and a volume expansion regularizer to improve generalization. Theoretical analyses prove a tighter generalization bound, and experimental results show superior performance in metric learning and contrastive learning tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and insightful comments! Our explanation and clarification can be found as follows.\n\n---\n\n**Comment_1:** Slightly higher computational cost due to volume estimation.\n\n**Response_1:** Thanks for your comments! We agree with the reviewer that the volume estimation does indeed introduce additional computation burden due to the necessary feedforward of the head network. However, we want to clarify that such additional computation cost is actually necessary to achieve better performance and is negligible in real-world use. \nSpecifically, as shown in Appendix A.3 of our manuscript, the baseline method SimCLR and our method VADSL spend 2.3 hours and 2.4 hours, respectively, to train 100 epochs on CIFAR-10 (batch size = 512). It implies that the calculation of volume estimation only adds less than **5%** of the time consumption. Similarly, on ImageNet-100, the time cost of the baseline method and our method are 10.9 hours and 11.2 hours, respectively, which means that the volume estimation merely requires an additional **3%** time consumption. In summary, we believe that the additional time cost of our method is relatively small and acceptable in practice.\n\n---\n\n**Comment_2:** Can VAD be applied to NLP tasks such as sentence similarity?\n\n**Response_2:** Thanks for your comments! Yes, VAD can be applied to NLP tasks. In fact, we have already validated the effectiveness of our method on two sentence embedding related datasets including STS and BookCorpus. As the reviewer considered, for each dataset, we construct the positive (similar) pairs by aligning each sentence with its contextual sentences in the same paragraph, while we build the negative (dissimilar) pairs by combining those sentences from different paragraphs. For both datasets, our method can consistently improve the baseline methods and outperform the SOTA methods in most cases (see Tab. 5 and Fig. A1), so we believe that our method is applicable to the sentence embedding tasks.\n\n---\n\n**Comment_3:** How does VAD behave in extremely high-dimensional spaces?\n\n**Response_3:** Thanks for your comments! Here we follow the reviewer’s suggestion to investigate the reliability of VAD for high-dimensional features. Specifically, we increase the output-layer dimensionality of the feature encoder from 512 to 4096, and record the corresponding accuracy of our method on STL-10 (batch size = 512). From the following table, we can clearly observe that the high-dimensional space indeed weakens the feature discriminability, where the accuracy decreases when the feature dimensionality is higher than 1024. However, in these high-dimensional cases, our VAD can still successfully improve the final performance on STL-10, which demonstrates the usefulness of VAD even in the high-dimensional space.\n|Setting|&nbsp; STL_100-eps (w/o VAD) &nbsp;|&nbsp; STL_400-eps (w/o VAD) &nbsp; |&nbsp;  STL_100-eps (w/ VAD) &nbsp;|&nbsp;  STL_400-eps (w/ VAD) &nbsp;|\n| :--- | :----: | :----: | :----: | :----: |\n| 512-dimension | 71.1±1.2 | 78.2±3.3 | &nbsp; &nbsp; **77.1±2.9** ✔| &nbsp; &nbsp; **85.6±3.2** ✔ |\n| 1024-dimension | 70.4±4.2 | 78.2±2.1 | &nbsp; &nbsp; **77.2±3.6** ✔ | &nbsp; &nbsp; **85.9±1.6** ✔ |\n| 2048-dimension | 68.3±2.3 | 76.5±1.3 | &nbsp; &nbsp; **73.3±1.2** ✔ | &nbsp; &nbsp; **82.3±2.3** ✔ |\n| 4096-dimension | 68.9±4.1 | 77.2±1.6 | &nbsp; &nbsp; **75.4±3.2** ✔ | &nbsp; &nbsp; **83.2±2.9** ✔ |", "claims_and_evidence": "The authors claim:VAD provides better generalization (supported by theoretical proofs).VAD outperforms state-of-the-art methods in supervised and unsupervised tasks (validated via extensive experiments). These claims are well-supported by rigorous proofs and strong empirical evidence.", "methods_and_evaluation_criteria": "The method is well-motivated and logically sound. Evaluation is extensive, with results on various domains and datasets, ensuring robustness.", "theoretical_claims": "The proofs for error bounds, flexibility, and sample-space coverage are well-structured and mathematically valid.", "experimental_designs_or_analyses": "Experiments are comprehensive, covering both supervised and unsupervised learning. Ablation studies clearly show the importance of each component. Comparison with strong baselines ensures credibility.", "relation_to_broader_scientific_literature": "The paper relates well to existing literature in metric learning and contrastive learning.", "essential_references_not_discussed": "No major missing references.", "other_strengths_and_weaknesses": "Strengths: 1) Well-motivated and novel approach. 2) Strong theoretical guarantees.\n3) Impressive empirical results. \n\nWeaknesses:\nSlightly higher computational cost due to volume estimation.", "other_comments_or_suggestions": "NA", "questions_for_authors": "(1)\tCan VAD be applied to NLP tasks such as sentence similarity?\n\n(2)\tHow does VAD behave in extremely high-dimensional spaces?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "iNlMlPv9dG", "reviewer": "Reviewer_cnEk", "summary": "This paper introduces a novel approach to similarity learning with the Volume-Aware Distance (VAD) metric. Instead of relying on traditional point-level similarity measures, VAD models data instances as volume-aware data spheres, allowing it to capture both observed and unobserved neighbor relationships. To improve generalization, the method utilizes a measure-head network for estimating instance volumes and incorporates a volume expansion regularizer. Theoretical analysis confirms its enhanced error bounds and topological properties, while extensive experiments across multiple domains demonstrate its effectiveness compared to existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of the novelty, theoretical analyses, and experimental results of our paper! Thanks also for your very insightful and constructive suggestions! Our point-by-point responses are as follows.\n\n---\n\n**Comment_1:** Can VAD be extended beyond similarity learning, e.g., in clustering?\n\n**Response_1:** Thanks for your comments! Yes, VAD is also applicable to clustering task. Actually, in our manuscript, we have already evaluated the clustering performance of VAD by showing the NMI scores on CAR-196, CUB-200, SOP, and In-Shop, where our VAD can successfully improve the baseline methods Npair and ProxyAnchor with at least 3 percentages (see Tab. 2). Here we also want to follow the reviewer’s suggestion to further validate the effectiveness of our VAD on the deep clustering model “contrastive clustering (CC)” (Li et al., Contrastive Clustering, AAAI’21). We simply implement our measure-head network by replacing the contrastive head in CC, so that we can calculate the corresponding VAD similarity values. We record the clustering accuracy and NMI scores of compared approaches on CIFAR-10 and CIFAR-100, and the following table reveals that our method still improves the strong baseline upon CC itself.\n|Setting|&nbsp; CIFAR-10 (Acc) &nbsp;|&nbsp; CIFAR-10 (NMI) &nbsp;|&nbsp;  CIFAR-100 (Acc) &nbsp;|&nbsp; CIFAR-100 (NMI) &nbsp;|\n| :--- | :----: | :----: | :----: | :----: |\n| CC (baseline) | 79.0 | 70.5 | 42.9 | 43.1 |\n| VADSL (ours) | 79.4 | 71.2 | 43.6 | 43.8 |\n\n---\n\n**Comment_2:** How does the measure-head network perform when used with transformers instead of CNNs?\n\n**Response_2:** Thanks for your comments! We agree with the reviewer that transformer has shown very promising reliability in lots of vision tasks. As our Tab. 4 has provided the detailed performance results (i.e., k-NN, Top-1, Top-5), we would like to discuss here again the superiority of our method when it is equipped with the ViT model (ViT.16/48MB). \nSpecifically, for both ImageNet-100 and ImageNet-1K, we implement our method on the cluster-free method BYOL as well as the cluster-based method SwAV, and the encoder networks of all compared methods are ViT.16. It can be observed that our method consistently improves the two baseline methods on all cases of the two datasets.  Furthermore, we also compare our method with SOTA methods including DINO, iBOT, and MTE, and we can find that our VADSL outperforms the SOTA in most cases (see the bottom of Tab. 4). This implies that our method works well with the vision transformer in the feature extraction, and also shows that our VAD is a fairly general technique that can be used in different frameworks.", "claims_and_evidence": "This paper asserts that VAD enhances generalization by encompassing a wider sample space while maintaining essential topological properties, as substantiated by theoretical analysis. Furthermore, VAD surpasses leading methods in both supervised metric learning and unsupervised contrastive learning. These conclusions are reinforced by rigorous mathematical derivations and comprehensive experimental evaluations, highlighting its robustness and broad applicability.", "methods_and_evaluation_criteria": "The experimental setup is thorough, covering multiple datasets (e.g., CIFAR-10, ImageNet, SOP, CUB-200) and different learning paradigms. The proposed method makes sense in similarity learning, particularly in contrastive learning and metric learning.", "theoretical_claims": "The proofs related to generalization error bounds and distance flexibility looks fine. They are mathematically sound, and the theoretical contribution is clear.", "experimental_designs_or_analyses": "The experimental design covers both supervised and unsupervised settings. The baselines are well-selected, and results consistently show the advantage of the proposed VAD. The t-SNE visualizations and ablation studies further strengthen the empirical support.", "relation_to_broader_scientific_literature": "The paper effectively connects its contributions to prior work in metric learning, contrastive learning, and regularization techniques. The citations are comprehensive and demonstrate a good understanding of the field.", "essential_references_not_discussed": "Not obvious", "other_strengths_and_weaknesses": "Strengths:\n+ Novel extension of distance metric learning and comprehensive theoretical guarantees.\n+ Comprehensive empirical evaluation. Well-written and clearly structured.\n\nWeaknesses:\n\nSome details are unclear:\n\na. Can VAD be extended beyond similarity learning, e.g., in clustering?\n\nb. How does the measure-head network perform when used with transformers instead of CNNs?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "a. Can VAD be extended beyond similarity learning, e.g., in clustering?\n\nb. How does the measure-head network perform when used with transformers instead of CNNs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ekccpL2qtE", "reviewer": "Reviewer_EAHq", "summary": "The paper introduces Volume-Aware Distance (VAD), a novel metric for robust similarity learning. Unlike conventional point-level similarity measures, VAD models instances as volume-aware data balls, improving generalization by capturing field-to-field geometric relationships. The paper also proposes a measure-head network to learn instance volumes and a volume expansion regularizer to further enhance generalization. Theoretical analysis shows that VAD tightens generalization bounds and preserves topological properties, while extensive experiments on supervised and unsupervised tasks demonstrate clear advantages over state-of-the-art methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive comments! Our point-by-point responses are provided below.\n\n---\n\n**Comment_1:** Computational overhead due to volume estimation, though the authors argue it remains manageable.\n\n**Response_1:** Thanks for your comments! We agree with the reviewer that the volume estimation does indeed introduce additional computation burden due to the necessary feedforward of the head network. However, we want to clarify that such additional computation cost is actually necessary to achieve better performance and is negligible in real-world use. \nSpecifically, as shown in Appendix A.3 of our manuscript, the baseline method SimCLR and our method VADSL spend 2.3 hours and 2.4 hours, respectively, to train 100 epochs on CIFAR-10 (batch size = 512). It implies that the calculation of volume estimation only adds less than **5%** of the time consumption. Similarly, on ImageNet-100, the time cost of the baseline method and our method are 10.9 hours and 11.2 hours, respectively, which means that the volume estimation merely requires an additional **3%** time consumption. In summary, we believe that the additional time cost of our method is relatively small and acceptable in practice.\n\n---\n\n**Comment_2:** How sensitive is VAD to the choice of the measure-head network’s architecture?\n\n**Response_2:** Thanks for your comments! Since the measure-head network is implemented by the classical MLP with a single hidden layer, the only human-tuned parameter is the dimensionality of the hidden layer (because the input-feature dimensionality is usually fixed and the output-layer dimensionality is always 1). Therefore, here we want to investigate the robustness of our method by changing the hidden-layer dimensionality. \n|Dataset|&nbsp; 32-dim &nbsp;|&nbsp; 64-dim &nbsp;|&nbsp; 128-dim &nbsp;|&nbsp; 256-dim &nbsp;|&nbsp; 512-dim &nbsp;|\n| :--- | :----: | :----: | :----: | :----: | :----: |\n| CIFAR-10 (400-eps, 512-bs) | 86.3 | 93.5 | 94.9 | 93.5 | 91.2 |\n| STL-10 (400-eps, 512-bs) | 79.4 | 85.2 | 85.6 | 85.8 | 81.1 |\n\nIn above table, we vary the hidden-layer dimensionality from 32 to 512 to record the final performance of our method on STL-10 and CIFAR-10. We can easily observe that the hidden-layer dimensionality indeed affects the classification accuracy to some extent, because VAD needs a certain amount of hidden nodes to ensure its nonlinearity. However, it is also obvious that the performance of our method is relatively stable around 128 (e.g., from 64 to 256), which means that such a hyper-parameter can be easily tuned in practical use.\nMeanwhile, we would like to further investigate the sensitivity of VAD by setting different number of hidden layers (including 1-layer, 2-layers, and 3-layer). For each case, we let the dimensionalities of all hidden layers be the same (i.e., 128-dim). The following table records the corresponding accuracy rates of our method on CIFAR-10 and STL-10.\n|Dataset|&nbsp; 1-layer hidden &nbsp;|&nbsp; 2-layers hidden &nbsp;|&nbsp; 3-layers hidden &nbsp;|\n| :--- | :----: | :----: | :----: |\n| CIFAR-10 (400-eps, 512-bs) | 94.9 | 93.6 | 94.5 |\n| STL-10 (400-eps, 512-bs) | 85.6 | 85.6 | 85.5 |\n\nWe can clearly observe that the results on both datasets are relatively stable when the layer number changes, and this means that our method is robust to the choice of network architectures.\n\n---\n\n**Comment_3:** Would VAD work in few-shot learning settings where data volume is extremely limited?\n\n**Response_3:** Thanks for your comments! Actually, VAD is able to work well for few-shot recognition tasks. To be specific, for the four datasets (i.e., CAR-196, CUB-200, SOP, and In-Shop) used in the metric learning task, no classes of the test data are involved in the training phase, and only very few reference examples are provided for classification during the test phase. In such a few-shot setting, our VAD effectively improves the NMI and Recall@K scores of the baseline methods, and our final performance can surpass SOTA methods in most cases (Tab. 2). We believe that the effectiveness of our VAD in few-shot learning settings is well validated.", "claims_and_evidence": "The key claims in the paper include: VAD generalizes better than conventional similarity learning methods, supported by theoretical proofs. VAD outperforms baseline models across multiple tasks, shown via strong empirical results.", "methods_and_evaluation_criteria": "The measure-head network and volume expansion regularizer are well-designed and logically justified. The proposed method is evaluated comprehensively across various datasets and baselines, making the results reliable.", "theoretical_claims": "I examined the proofs for generalization bounds and distance flexibility. The claims are mathematically sound, and the derivations align with existing principles in similarity learning.", "experimental_designs_or_analyses": "Experiments cover multiple learning paradigms (metric learning and contrastive learning). Baseline comparisons are thorough, ensuring fairness. Ablation studies validate the necessity of each proposed component.", "relation_to_broader_scientific_literature": "The paper clearly situates itself within the literature on metric learning, contrastive learning, and regularization methods. The citations are appropriate and comprehensive.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\nWell-motivated and novel approach to similarity learning.\nStrong theoretical backing with comprehensive analysis.\nExtensive empirical validation across diverse datasets.\n\nWeaknesses:\nComputational overhead due to volume estimation, though the authors argue it remains manageable.", "other_comments_or_suggestions": "None", "questions_for_authors": "i). How sensitive is VAD to the choice of the measure-head network’s architecture?\n\nii). Would VAD work in few-shot learning settings where data volume is extremely limited?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OeculjwgAF", "reviewer": "Reviewer_25dP", "summary": "This paper presents Volume-Aware Distance (VAD), a novel metric for similarity learning that extends traditional point-wise distances to field-to-field distances by introducing volume-aware data representations. The authors propose a measure-head network for volume prediction and a volume expansion regularizer to improve generalization. The method is mathematically well-grounded, providing strong theoretical guarantees, and is empirically validated across multiple domains.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive comments! Our responses are given below.\n\n---\n\n**Comment_1:** The computational complexity of the method could be further analyzed in large-scale settings.\n\n**Response_1:** Thank you for your suggestion! Here we would like to provide the training time comparison of our VADSL and the baseline methods (SimCLR and SwAV) on the large-scale ImageNet-1K dataset. Specifically, we use eight NVIDIA TeslaV100 GPUs to train our models based on SimCLR and SwAV with 100 epochs, respectively. For each case, we set the batch size to 512, 1024, and 1536.\n\n| Method | &nbsp; ImageNet-1K (ba. si. = 512) &nbsp; | &nbsp; ImageNet-1K (ba. si. = 1024) &nbsp; | &nbsp; ImageNet-1K (ba. si. = 1536) &nbsp; |\n| :--- | :----: | :----: | :----: |\n| SimCLR | 70.1 | 35.2 | 23.1 |\n| SwAV | 71.2 | 36.7 | 24.2 |\n| VADSL (SimCLR + VAD) | 71.5 | 35.6 | 23.5 |\n| VADSL (SwAV + VAD) | 72.1 | 36.9 | 24.6|\n\n\nThe table above reveals that the introduction of the proposed VAD component causes very little increase in time consumption (less than **2%**). This is because the gradient calculation of VAD is independent of the size of the training data, thereby keeping the training time well within practical limits. We will also add the above time comparison in the camera-ready version if this paper is finally accepted.", "claims_and_evidence": "The main contributions of the paper are: A new similarity metric (VAD) that enhances generalization by incorporating volume-awareness. A theoretical framework demonstrating VAD’s tighter generalization bounds. Empirical results showing VAD’s superiority over state-of-the-art similarity learning methods.", "methods_and_evaluation_criteria": "The proposed VAD framework, including the measure-head network and regularizer, is well-motivated. The evaluation includes multiple supervised and unsupervised tasks, making the results highly convincing.", "theoretical_claims": "The authors present rigorous mathematical proofs for generalization error bounds and sample-space coverage, which validate the effectiveness of VAD.", "experimental_designs_or_analyses": "The experiments are well-structured, covering both metric learning and contrastive learning. The baselines used for comparison are strong, ensuring credibility. Ablation studies demonstrate the necessity of each component.", "relation_to_broader_scientific_literature": "The work is well-grounded in existing literature and makes a meaningful contribution to the field of similarity learning.", "essential_references_not_discussed": "No critical references appear to be missing.", "other_strengths_and_weaknesses": "Strengths:\n\nNovel and impactful contribution to similarity learning.\n\nSolid theoretical underpinnings.\n\nComprehensive empirical evaluation.\n\nWeakness:\n\nThe computational complexity of the method could be further analyzed in large-scale settings.", "other_comments_or_suggestions": "None", "questions_for_authors": "See the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Volume-Aware Distance (VAD), a novel metric for robust similarity learning. Unlike conventional point-level similarity measures, VAD models instances as volume-aware data balls, improving generalization by capturing field-to-field geometric relationships. 

This paper received universal positive scores after the rebuttal period. All reviewers agreed that the approach is novel and that the empirical results were very solid.
Decision: Accept (poster)