Adaptive and Multi-scale Affinity Alignment for Hierarchical Contrastive Learning
Authors: Jiawei Huang
Minming Li
Hu Ding
Venue: NeurIPS 2025 poster ()
Keywords: representation learning
contrastive learning
optimization
Primary area: deep_learning

Abstract: Contrastive self-supervised learning has emerged as a powerful paradigm for extracting meaningful representations without labels. While effective at capturing broad categorical distinctions, current methods often struggle to preserve the fine-grained and hierarchical relationships inherent in real-world data. From the perspective of semantic alignment, conventional contrastive learning aligns representations to semantic structure at a global level, treating the entire embedding space uniformly and frequently overlooking rich local structural information. In this paper, we propose \emph{Adaptive Multi-scale Affinity alignment (AMA-alignment)}, a framework that introduces localized contrastive objectives and a dynamic multi-scale optimization strategy to adaptively identify and refine poorly aligned regions within the embedding space. Although our model is inherently more complex due to its \emph{multi-scale} and \emph{adaptive} design, we provide the theoretical guarantees indicating that its convergence rate remains comparable to that of standard smooth non-convex optimization. We conduct a set of experiments on diverse benchmarks to show that AMA-alignment can effectively preserve hierarchical structure; moreover, AMA-alignment also outperforms existing contrastive methods on a range of downstream tasks.

Adaptive and Multi-scale Affinity Alignment for Hierarchical Contrastive Learning
Jiawei Huang!.2 Minming Li
Hu Ding'
School of Computer Science and Technology. University of Science and Technology of China DDpparmeett of Computer Science, City University of Hong Kong {hjw0330, huding) Email. ustc. edu. en, minming wwt..lit.ctteuu....... edu.
Abstract
Contrastive self-supervised learning has emerged as a powerful paradigm for extracting meaningful representations without labels. While effective at captur- ing broad categorical distinctions, current methods often struggle to preserve the fine- tne-grained and hierarchical relationships inherent in real-world data. From the perspective of semantic alignment, conventional contrastive learning aligns rep- resentations semantic structure global level, treating the entire embedding space uniformly and frequently overlooking rich local structural information. In this paper, we propose Adaptive Multi-scale Affinity alignment (AMA-qll AMMA-aligmment), a framework that introduces localized contrastive objectives and dynamic multi- scale optimization strategy to adaptively identify and refine poorly aligned regions within the embedding space. Although our model is inherently more complex due to its multi-seut scale and adaptive design, we provide the theoretical guarantees indicating that its convergence rate remains comparable to that of standard smooth non-convex optimization. We conduct a of experiments on diverse benchmarks to show that AMA-alignment can effectively preserve hierarchical structure; more- over, AMA-aligmment also outperforms existing contrastive methods on a range of downstream tasks.
Introduction
suwppprrviseddsllf-uurrrviiseeee learning has revolutionized the field of representation learning, achiev- ing remarkable results across various domains with limited labeled data D. These approaches uncover intrinsic data relationships, facilitating the learning of meaningful representations without supervision. Among these methods, contrastive learning (CL) has proved to be a particularly effective framework. By leveraging data augmentations to generate positive and negative sample pairs, CL trains models to distinguish between similar instances (positive pairs) and dissimilar ones (negative pairs) 7 [1). Beyond this discriminative objective, CL can also be interpreted as an "alignment" mechanism, encouraging the learned representations reflect the semantic relationships implicitly defined by the data augmentation process [50 48
Despite its success in capturing high-level categorical distinctions, CL often struggles to preserve the fine-grained and hierarchical semantic relationships inherent in real-world data []. Such relationships are critical for downstream tasks where subtle variations carry significant semantic differences. For instance, while a CL model trained on the DeepFashion dataset [33] may correctly distinguish high- level categories like upper- and lower-body garments, it often struggles with finer-graines distinctions, failing to differentiate between t-shirts, jackets, sweaters, blouses, and vests.
To obtain fine-grained representations, recent advancements in CL mainly focus on designing stronger augmentation strategies. These methods construct more informative positive and negative sample
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Scales
s
S
Affinity Graph
Prob. Coupling
K sem
Pr Pre Ki
Z x(x)
1
1
Goal: Align
cross-entropy
pants
Koep
Space
Pr Pr K)P))
Leggings
Garments
Bottom wear
Figure 1: Illustration of multi-level contrastive learning- An ideal scenario where the learned representations exhibit hierarchical structure that captures both broad categorical distinctions and fine-grained semantic relationships. Our approach identifies and optimizes local worst-case regions (highlighted in red) where representation affinity deviates significantly from the semantic affinity. Here, the affinity matrices Kymm and KTeP represent pairwise semantic and representation relationships, respectively, as defined in (3) and (
For example, Feng et al. [12] leveraged diverse modalities and perspectives of chemical molecular data to construct more effective positive and negative pairs, Additionally, Robinson al. 43 proposed a method for sampling hard negative examples, improving the quality of negative sample selection.
pairs by incorporating task-specific cues leveraging multi-view inputs 32
12
Our intuition for addressing these challenges originates from hierarchical clustering. it also aims to capture structural relationships across multiple levels of granularity. Recent theoretical also reveal fundamental connections between CL and traditional clustering methods. These studies show that contrastive objectives encourage representations to distribute uniformly on the unit sphere 3% and naturally form category-base clusters. In the ideal case, as illustrated in the lower half of Figure such representations would reflect not only coarse categorical structure but also fine-grained semantic hierarchies, thus resembling hicrarchical clustering. For example, sub-categories of lower-bode body clothing would be effectively clustered within the broader lower-body region. However, while existing CL methods perform well in capturing broad distinctions, they typically learn these semantic relationships at a global level, often failing to preserve the fine-grained relationships critical for downstream tasks.
advances [55] 50, 19 [
To overcome this limitation, we seek to incorporate hierarchical structure into the CL process more directly. Hierarchical clustering [38] 39 naturally creates multi-scale region partitions using either top-down (divisive) or bottom-up (agllomerative) strategies. However, several fundamental obstacles hinder the direct application of conventional hierarchical clustering techniques in the CL. context: 1) Classical hierarchical clustering methods rely on distances that are directly accessible, whereas CL operates on an affinity graph Keen (formally introduced in Sec. 2 whose edge weights are determined by a stochastic positive pair sampling process, making them nontrivial to compute directly, 2) Unlike non-parametric clustering algorithms, which do not rely on learned feature mappings [9) CL trains a parameterized encoder go(-) to produce feature embeddings to align semantic structure KTep (see 4 therefore requires to redesign more appropriate optimization strategy for go(-). 3) In fine-graines alignment scenarios, different local regions of the embedding space often exhibit varying degrees of semantic misalignment [3]. Uniformly optimizing over the global affinity structure inefficient; in other words, it should be more effective to prioritize regions that are poorly aligned with the underlying semanties. These limitations call for region-awar approach that retains the hierarchical perspective while adapting to CL's characteristics.

---

Our proposed model and contributions. The above challenges motivate the development of our hierarchical region-aware optimization framework, which dynamically prioritizes poorly aligned regions across multiple semantic scales. We use 2, ..... S) to denote different scales, ranging from coarse to fine-grand finegrained. At each scale 8, we partition the embedding space into set of regions {RT where m² is the number of regions at that scale. For each region Rt, we define corresponding local contrastive loss (0, IR₂) (formally derived in Sec. 3.1) that measures how well the representation preserves semantic relationships within that specific region. Specifically, we formulate new multi-scale optimization objective as follows:
min Σ ==1 rEfmn)
max (0, R.)).
EE
The inner maximization adaptively identifies regions with high loss values, while the outer mini- mization over 0 focuses on reducing the sum of the regional losses. We name our method adaptive multi-scale affinity alignment -MMA-mligmmentt) and highlight its key contributions:
We introduce the new concept of local affinity graph and derive its associated local con- trastive loss, specifically designed to align regional structures. We further propose a new perspective that integrates distributionally robust optimization with local contrastive learn- ing to adaptively refine poorly aligned regions. This framework facilitates multi-scale representation learning and captures both coarse and fine- fnn-grained semantic structures.
We provide theoretical analysis of the proposed optimization objective and algorithm. In particular, even our optimization model is more complicated due to the endowed "multi- scale" and "adaptive" properties, the convergence rate is comparable to the standard smooth non-convex optimization.
We evaluate AMA-alignment on a range benchmark datasets, demonstrating its effective- ness in preserving semantic structure and excelling on challenging downstream benchmarks.
1.1 Other Related Works
Several studies have leveraged multi-level labels 25 66 to learn hierarchical representations. These methods supervised approaches that rely annotated data across multiple levels. In contrast, our work focuses on unsupervised learning, consistent with conventional settings of CL algorithms. Mo et al. [36] introduced method that employs two types of contrastive losses at both shallow and deep network layers, enhancing the quality of learned representations. Additionally, other works 14 59 have explored embedding representations in hyperbolie space to better capture hierarchical relationships. More recently, several studies [32 54 12)
have focused on refining data augmentation strategies more effectively encode semantic information, either explicitly implicitly. Our study demonstrates that by more effectively utilizing the semantic information inherently encoded in augmentations, the model can capture fine-grained semantic structures more accurately through improved optimization techniques. A more detailed discussion of related works is deferred to Appendix A
Preliminaries
2.1 Formulation for Contrastive Learning (CL)
Let X denote the input dataset sampled from the natural data distribution Px, and let A(-) be a randomized data augmentation operator (e.g., cropping, color jittering). For each J X, we define the conditional distribution Prx (-f²) as the probability density over the data space X' induced by applying A--) J.. That is, Pra (x(=) denotes the likelihood of obtaining x as augmented view of F. By applying A(-) to all samples in X. we obtain the augmented data distribution Px, whose support set is denoted as X. Without loss of generality, we assume X is a finite set of size N; if X is infinite, the summations in our analysis can naturally generalize to integrals.
We model the network r h g. where g(-) X Z is the encoder mapping input data to representation space 2 C Rdd and h: Z + ) is task-specific classifier. The encoder g(-) is parameterized by 0 E O where denotes the parameter space. Contrastive Learning (CL.) aims to learn discriminative representations by pulling semantically similar instances closer and pushing
3

---

dissimilar ones apart in the embedding space 58 63). Following standard CL approaches, we adopt the Information Hoise-Contrastivy Estimation (InfNCCE) objective 78:
58
exp (g(2.) ( g(a;)//)
L =
E [La (x)) where (a(z)= -- 2~Px
E It PEA (12)
log
(2)
Enjs exp (g(x.) (( (Txx)/T)
Here T > is the temperature parameter and 2y is negative sample from X. Let Z { C Z denote the set of feature embeddings for the augmented dataset X, where each 2; = g(z;) corresponds to data point 2; X.
2.2 Alignment via Probabilistic Graph Coupling
We reinterpret the contrastive objective in (2) as aligning two graphs- one defined by semantic affinities and the other by representation similarities. This insight motivates our key idea: generalizing the global graph alignment to a hierarchical framework that captures structural correspondences across multiple spatial scales.
Definition 1 (Semantic Affinity Matrix), The semantic affinity matrix K E R over X quantifies the pairwise semantic proximity through augmentation co-occurrence. Specifically, for any Ty Ti Iyj E X, its (i, entry is defined as:
K aen Ex-Px [Pra (xi(z) PrA (x)gl²)
This matrix quantifies the joint probability that x; and Ij are both generated from the same underlying sample क. thereby encoding their semantic relatedness under data augmentation.
Definition 2 (Representation Affinity Matrix), The representation affinity matrix KFep E N over Z measures pairwise similarities in the embedding space. For any Ti Ti Ty E X:
K rep Ker(2,, where 2; =x(,,, 2j = g(1)),
where Ker x 2 R is counslation-nvariinnn kernel function (e.g., Gaussian kernel) [55]
Reen and Keep define two weighted graphs: the former reflects semantic relationships via augmenta- tion, while the latter encodes geometric proximities in representation space. Prior research [55 50 has shown that CL aligns these graphs by learning representations that preserve semantic relationships.
Probabilistic Graph Coupling. To formalize this alignment process, we adopt the Probabilistic Graph Coupling framework [55], which interprets CL as minimizing divergence between distributions over subgraphs sampled from K sen and KTep, These distributions reflect the structural properties of the underlying graphs, and CL aligns them by aligning their subgraph statistics.
We restrict attention binary directed graphs W with exactly one outgoing edge per node mmiroring the pogli--ositive sampling mechanism used in contrastive learning:
SW
(W& E 10,13 | E [N], Σ
N W 1],
(5)
j=1
where Wij =1 denotes directed edge from node x; to xy. We impose this constraint as it mimics the positive pairing mechanism in contrastive learning, where each data point selects exactly one positive sample. Thus the InfoNCE objective can be reinterpreted
minimizing the divergence between subgraph distributions over the constraint ( We summarize the core idea below, and provide a detailed discussion in Appendix
Given an affinity matrix K E {K KTTP), as defined in Def. or 2. the probability of sampling a subgraph W under the topological constraints specified in is given by:
Pr Pr(W x ((W) II (T (K.)) Wo.
where ((W) N 1 (E, W,, ,y 1) enforces the topological constraint, ensuring that each node
has exactly one outgoing edge. The core objective of contrastive learning can thus be reformulated as minimizing the cross-entropy between the subgraph distributions induced by K sen and Krep,
Er [r.)) [log Pr( Pr(wzep = Wree
K**P)]
(7)
As shown in 50 minimizing 7 is equivalent to the InfoNCE loss in 2 This equivalence formally
links CL with probabilistic graph alignment.

---

3 Method
This section introduces our hierarchical contrastive learning framework. We begin with a high-level overview and then detail its two key components.
Overview. Our method performs contrastive alignment across multiple spatial granularities in a unified and synchronous manner, formalized in equation ( (Sec. [ It operates simultancously on: (i) global structure, by preserving high-level semantic consistency through large-scale alignment; and (ii) local structure, by enhancing fine-grained discriminability within local neighborhoods. This multi-scale co-optimization leads to robust representation learning that spans from global categories to subtle intra-class variations. The framework comprises two main technical components:
1. Local Alignment (Sec. 3.1
We define local regions in the embedding space 2 and align corresponding semantic and representation affinity graphs using novel local contrastive loss L (12) which is derived by the Probabilistic Graph Coupling technique.
2.
Adaptive Refinement (.2): We introduce the adaptive multi-scale alignment algorithm that aggregates local losses across scales using distributionally robust optimization (DRO) scheme, which dynamically prioritizes challenging regions.
3.1 Local Contrastive Learning Loss
We first describe how locally align semantic and representation affinities. Extending the affinity- based interpretation of the InfoNCE loss (2). we develop a localized version that aligns affinity graphs within small, spatially defined regions.
Local affinity graph. In contrastive learning, the input space X' is mapped and normalized into unit hypersphere embedding space Z C -- TRd via trainable encoder g(-) (58). To capture structure at multiple spatial resolutions, we introduce S levels of granularity. Each scale is associated with an angular radius 1 > 0 that defines the size of local regions. At scale s, we define m² local regions R (0) (R" 2,)) ]=)=== each centered at an anchor point 2 e Z. For brevity, we denote the r-th region at scale by R Given anchor 2 and radius E", the local region is defined as:
R = (= Aiz, 2+) {(
(8)
where -(: is the angular distance (i..., the arc-cosine of the normalized dot product). This region captures points most similar to the anchor under the current embedding, naturally adapting to local data geometry. To build intuition, a coarse scale (large L²) may group together all clothing items, whereas finer scale (small (*) could isolate specific categories such jeans, skirts, or shorts.
Let XR. S X denote the subset of samples whose embeddings fall inside region RS indexed by TRe. That is, XR: = x[RR;] with Rx = N'. We define the local affinity matrices Keen K
R N' NN submatrices of the global affinity matrices K and T (see Def. and 2
(9) Since Keen and Keep are symmetric, the index set Ie applies to both rows and columns. When the specific region is not of primary interest, we omit the region index for notational simplicity.
K sen
K [Ig,, Iagl
KTop A K [I IRg].
Local contrastive loss. To align KTTh and K. we again employ the probabilistic graph coupling framework. Similar to 77), this alignment is achieved by minimizing the cross-entropy between the distributions of subgraphs W and W sampled from K and K rep respectively. These subgraphs follow the constraint defined in (5), ensuring each node has exactly one outgoing edge. This leads to the following distribution over sampled subgraphs:
Proposition 3.1. Let W. be subgraph sampled from K, (represents either K or K the constraint in Then:
under
Pr K2) x ((W.) II
(J)ENN (K..)) We.s
(10)
where Ω(W ((W.)= 1E, W. N enforces the one-outgoing-ede -edge constraint 5
5

---

serves as the localized counterpart of the global formulation in extending the probabilistic graph coupling to the region- region-specific setting. To further formalize the concept of learning within localized regions, we introduce a restricted augmeniation distribution that limits sampling to examples whose representations lie within specific region:
Equation (T0)
(ra(a X
if g(x) E Re
(x(())
=
0,
otherwise,
(11)
where the normalization constant x; Soxx 1 me (ra(f/#)dz ensures that Pra(-lz) defines valid probability distribution. This formulation constrains the augmentation process to operate only within the local regions Rç.
Similar to the InfoNCE loss, following the probabilistic graph coupling framework, we minimize the cross-entropy between the subgraph distributions over the local affinity matrices K and K and derive the local contrastive loss as shown in
exp (2, ZT/T) Eiyi 1; exp (2, 2
C = EE-PS [T)],
where (z)
log
(12)
N4. (11))
Here, T is the temperature parameter, 2 = g( and 2, ( are embeddings produced by the encoder g. and (*xx) (()))R ensures negatives are drawn from the local region. We have the following lemma:
Lemma 3.2.
Minimizing the cross-entropy between the subgraph distributions induced by keen and K. under the structural constraint in (5), is equivalent to minimizing the (12).
This lemma confirms that the local loss in is theoretically grounded in the same probabilistic
framework as the global contrastive loss 2 but adapted to align affinity structure within localized regions. A full proof is provided in Appendix
Remark 1. The formulation of L is non-trivial. A naive approach might define a local contrastive loss by restricting positive pairs to lie within the local region R,Y, such as:
exp (z, ZJ/T)
2
log E Righ exp (2, =x/T)
(13)
However, the sampling distribution implied by Ene; ex, #x/7)* does not reflect the true
expl= 21/T)
edge sampling probability in the underlying subgraph, ie.., Pr(Wrep 5,1 1) *
exp(x 2//7)
Eeps exp(2, =//r)" Therefore, using (3) does not ensure the alignment between local and global affinities, violating the alignment principle of CL. For example, this formulation would distort the global structure by inappropriately pushing all representations & R. maximally away from representations = Rr
3.2 Adaptive Multi-scale Optimization for Affinity Alignment
we now describe how these losses are integrated into a unified optimization process that emphasizes difficult regions across multiple semantic scales. Our key insight is that not all regions require equal attention during training. In real-world datasets, semantic misalignment often occurs heterogeneouslly may already adequately preserve semantic relationships, while others exhibit significant distortions. For instance, in fashion datasets, the boundary between "formal shirts" and "casual shirts" might be more challenging to delineate than the distinction between "shirts" and "pants", and thus it requires more effort to design an appropriate optimization strategy.
Building on the local contrastive loss 12 in Sec. 3.1
neighborhoods in the embedding space
We formalize this intuition through a novel adaptive multi-scale optimization framework that dynami- cally prioritizes poorly aligned regions while maintaining coordination across scales. Our objective, introduced in equation (T). employs nested min-man structure that adaptively allocates the focus of current computational stage towards the most challenging regions. For analytical clarity and to facilitate tractable optimization, we present the following equivalent reformulation of (T) using a convex combination of region- wise losses:
Σn & -0 R:).
min 8
(14)
q' E EA!!

---

where = (uf..... .... fm-) lies within the probability simplex Am {q and represents the adaptive weighting over regions at scale 8. The inner maximization identifies regions with high loss values, while the outer minimization over A directs learning to reduce regional losses. This formulation is partially inspired by the distributionally robust optimization (DRO) [45 65], and we leave the discussion DRO Sec. 3.3
Sm =1
As introduced in Sec. 3.1 our framework operates across S levels of spatial granularity. Each scale is characterized by its angular radius z", which determines the size of local regions. At each scale, we define m² regions (0) {R partitioning the embedding space 2 according to (8. Finer scales (small (5) yield more focused regions, while coarser scales (large if) group semantically broader patterns. This hierarchical partitioning can be efficiently implemented through spherical k-means clustering on the unit hypersphere. Due to the limited space, we provide our implementation
details in Appendix F
However, directly optimizing this min-max objective (4) brings two key challenges: (I) due the non-convexity of (0, R²) w.r.t and the nested min-max structure, the standard convergence guarantees cannot be easily obtained; (2) the inner maximization over q" may lead to degenerate solutions that excessively concentrate on a single region, resulting in unstable training dynamics. To address these challenges, introduce entropy regularization term and reformulate the objective in (14 as:
F(0) min
max q E'
Σ C LC,, -((,(R)))K((('))
(15)
where KL(q")- = Ere q (m" g is the KL divergence between q' and the uniform distribu- tion, and p > is regularization parameter controlling the strength of the entropic penalty.
Benefits of KL Term in (15). The KL regularization term p E q log "x"") in 15 serves three critical functions: 1) It ensures strong concavity q" which is essential for proving convergence in the non-convex setting for our AMA-alignment algorithm, as established in Theorem D.10 2) It prevents the weight distribution from collapsing the simplex boundary, stabilizing the optimization dynamics and improving convergence behavior. 3) enables a closed-fom update
for qe via exponential weights (see (17) ), avoiding iterative gradient ascent and facilitating efficient implementation.
Algorithm 1 Adaptive Multi-scale Affinity Alignment (AMA-alignment) for Hierarchical CL.
Input: Dataset X, encoder g(-), scale radius {r*}sm iterations T, learning rate Ne,rs regularization P
Output: Optimized parameters 1: Initialize Og from base model
2: Pre-train encoder g(-) with standard contrastive loss C for No epochs
3: Generate partitions
for each scale s (detailed in Appendix
4: For each scale [S]. initialize q" [1/m*, ...., 1/m²'"
5: for = 1 to T do
6: for each scale [S] do
7: Compute region losses (0, R;)
8: Compute optimal weights q (0.) by
9:
end for
10: Compute total gradient g: by 16)
11: Update encoder: 0141 0: 18
12:
end for
13:
return AT
Sketch of Algorithm I Our AMA- AMA-aligmment method outlined in Algorithm The training begins with pre-training phase using the global contrastive loss L for N, epochs, which can be interpreted as optimizing at the global scale (s = 0). After this initialization, the model proceeds to jointly optimize affinity alignment across multiple spatial scales, refining both global and local semantic structures. Each iteration involves two key steps: updating encoder parameters (Algorithm) T lines 7-8) and adaptively adjusting region weights qf (Algorithm line 10):
7

---

Updating Encoder Parameters (0): The encoder parameters are updated via gradient descent at each iteration t. The gradient estimator g computed using the current region weights q::
0.+1 02 10,,
where g:
>>> Li V. R,).
(16)
Here, 17,, is the learning rate for the encoder parameters.
Updating Region Weights (q*): The region weights are then adaptively updated based on the new encoder parameter I This step follows the standard stochastic mirror ascent procedure, which solves a regularized inner optimization problem as formulated in (36) (see Appendix D.I) and admits unique closed-form solution given in (17). For notational simplicity, define time-dependent coefficient "/2 = + 7q The updated weight for each region r is then given by
(qi,,) 2/T exp
Mant L (0c+1 RE))
4
7t
(17)
The" Σド= ( exp
11.11 (%% L= [+ Rg))
where 1/4, controls the adaptation rate of the region weights. This update mechanism automatically prioritizes challenging regions by assigning them higher weights. Regions with higher loss values receive greater attention in subsequent optimization steps, dynamically focusing computational resources on the most difficult regions.
Theorem 3.3 (Convergence Rate (Informal)). Under the smoothness assumption of the objective, suppose the learning rates are set as 1o, weight updates, where o,, 1/0 > 0 are constants. Then Algorithm rate: + ET= B[IV (0:) 1 C constant independent T.
a for parameter updates and lig, = no for region Vt
the following convergence
VT. where F(0) is the overall DRO objective in 15 and C is
We analyze Algorithm in a stochastic optimization setting, where both the region-level loss (lines 7-8) and the gradient g: (line 10) are computed via stochastic approximations. To ensure stable training under this noise, we update the region weights (0,) recursively using a moving-average rule (17), rather than solving the inner maximization xxattly-esppite it admitting closed-form Under mild assumptions (e.g.. bounded gradients/losses and the model smoothness), our algorithm converges rate comparable to standard smooth non-convex methods, despite the multi-scale structure and adaptive updates. The full proof Theorem D.10 is provided Appendix [
3.3 Some More Discussions
Our approach is partly inspired by group DRO method but with following differences: 1) Hierarchical, data-driven grouping: Our method dynamically forms groups based on local neighborhoods in the evolving embedding space rather than fixed, predefined partitions; 2) Region-specific loss: We employ specialized losses derived from the affinity-based interpretation of InfoNCE, aligning representations based on semantic and geometric structure rather than generic empirical risks. 3) Non-convex convergence guarantees: Our analysis establishes convergence in the challenging non-convex optimization settings typical of deep contrastive learning, unlike prior work [44 65] that focused primarily on convex settings.
Discussion of relations to Group DRO methods
45
Analyzing the Optimization Dynamics of AMA-alignment. Previous theoretical work [52 interprets contrastive learning (CL) two-player coordinate-wiss optimization game between model parameters and sample-pair weights-an an interpretation that encompasses classical strategies like hard negative mining[43 (3). A.MA-alignment generalizes this view into a hierarchical three- player optimization framework. In addition to the standard optimization over model parameters and pairwise affinities, we introduce a third axis: region-level importance weights that adaptively prioritize difficult semantic regions across multiple spatial scales. This additional layer of adaptivity enables AMA- alignment to capture both local semantic granularity and global representational structure. We formally analyze this three-level coordinate-wist optimization dynamics in Section B

---

Experiments
We evaluate the proposed Aal-nnignment framework on multiple benchmarks assess its ability to capture fine-grained and hierarchical semantic structures. We compare against state-of-the-art contrastive learning (CL) baselines both unsupervised and supervised settings.
4.1 Experimental Setup
Datasets. We experiment on diverse datasets with inherent hierarchical fine-grained semantics: (1) DeepFashion [33] fine-gre aeeegrained clothing dataset; (2) iNaturalist [56], with taxonomic labels;
(3) CIFAR-100 27 grouped into superclasses; and (4) ModelNet40 [6]. 3D object classification
dataset. (5) Bulmg
breast cancer diagnosis ultrasound images dataset.
Baselines. We compare with popular CL methods including SimCLR 7 MoCo [8 HardNeg 43
and o-CL [52] In supervised settings, we compare with Cross-Entropy SupCon [26]. Guided- Proto (28). and HiMulConE [66]. Unless otherwise specified, we train ResNet-50 [17] as backbone. All models are implemented with PyTorch single NVIDIA RTX 6000 Ada GPU. The model is
optimized using the AdamW W optimizer 34 Additional experimental details are in Appendix
4.2 Experimental Results
Table shows the classification results. AMA-alignment consistently outperforms existing CL methods, with substantial gains on fine-grained benchmarks. On DeepFashion, we observe over 7% improvement, highlighting AMA's ability to capture subtle semantics. Performance on iNaturalist also improves notably, especially in tail classes.
Table 1: Accuracy downstream classification tasks. fine-grained performance.
AMA-alignment significantly improves
Method
DeepFashion
ModelNet
iNat
ImageNet
SimCLR
70.3
79.3
54.0
69.5
MoCo-v2 8
70.8
79.6
55.3
68.1
HardNeg 43
70.9
79.8
55.8
70.4
a-CL 52
71.7
79.6
56.1
70.2
allaalignment (ours)
75.8
80.6
57.2
73.3
Hierarchical Representation Quality To assess the preservation of semantic hierarchies in learned
representations, we employ two metries: (1) Hierarchical Clustering Normalized Mutual Information (HC-NMI) [10), and (2) Intra-class Variance Reduction (IVR). As summarized in Table 2 AMA- alignment achieves superior alignment with ground-truth hierarchies, demonstrating effectiveness in modeling multi-scale structures.
Table Hierarchical alignment results DeepFashion. Higher HC-NMI and lower IVR indicate better semantic structure preservation.
Method
HC-NMI
IVR
SimCLR
0.52
0.134
HardNeg
0.56
0.119
a-CL 52
0.59
0.114
AMA-alignmeent (ours)
0.66
0.091
Ablation Study. We conduct ablation studies to examine the contributions of the key components in AMA-alignment, as summarized in Table 3] Starting from global-only contrastive learning (CL) baseline, we progressively introduce multi-scale partitioning and local contrast loss. The results show that both components essential for achieving robust alignment performance.

---

Specifically, incorporating the multi-scale local contrastive loss (+ Multi-scale) improves accuracy across all benchmarks, indicating that modeling representations different semantic granularities might benefit global consistency. In contrast, if we use an adaptive weighting mechanism (+ Adaptive) but implemented via the heuristic loss given in (T3). the performance degrades. Unlike the proposed local contrastive loss in (12), this formulation does not restrict negatives to local regions, thereby distorting the affinity structure and degrading performance, as discussed in Remark [
In contrast, our AMA-alignment employs the principled local contrastive loss in (12). which rigor-
ously aligns the semantic affinity matrix K*em and the representation affinity matrix K through probabilistic graph coupling. This theoretically grounded formulation ensures that both the global and local alignment objectives work coherently in synergy.
Table 3: Ablation study DeepFashion.
Accuracy (%)
DeepFashion
iNat
ImageNet
Global-only CL
70.2
54.0
69.5
+ Multi-scale (fixed weights)
72.3
55.9
70.2
+ Adaptive (using (33)
67.3
50.4
67.8
AMA -ll-mmmgmmeet
75.8
57.2
73.3
5 Conclusion
We propose novel hierarchical contrastive learning framework that aligns local affinity structures across multiple scales. By introducing local contrastive objectives and applying a distributionally robust optimization strategy, our method dynamically identifies and refines semantically challenging regions in the representation space. The theoretical analysis establishes convergence guarantees, and extensive experiments showcase its superior performance in capturing hierarchical semantics compared existing methods. This work advances contrastive methods beyond global alignment, enabling more interpretable and task-specife representations in complex domains.
Acknowledgments and Disclosure of Funding
This work was supported part by the National Key R&D program of China through grant 2021 YFA 17009000 the NSFC through grants No. 62432016 and No. 62272432, and the Provincial NSF of Anhui through grant 2208085MF163. The authors would also like to thank the anonymous reviewers for their valuable comments and suggestions.
References
[1] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learn- ing recover global and local spectral embedding methods. Advances Neural Information Processing Systems, 35:26671-26688 126655, 2022.
[2] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. Advances in neural information processing systems, 19, 2006.
[3] Deng Cai, Xiaofei He, Xuanhui Wang. Hujun Bao, and Jiawei Han. Locality preserving nonnegative matrix factorization. In IJCAI, volume 9, pages 1010-1015. Citeseer, 2009.
[4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9922 2020. 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pages 9650-96600 2021.
10

---

[6] Jianlong Chang, Lingfeng Wang. Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In Proceedings of the IEEE international conference on computer vision, pages 5879-5887, 2017.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoftrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020.
[9] Sebastian Damrich, Jan Niklas Böhm, Fred Hamprecht, and Dmitry Kobak. From t-SNE to UMAP with contrastive learning. arXiv preprint arXiv:2206.018// 2022.
[10] Kien Do, Truyen Tran, and Svetha Venkatesh. Clustering by maximizing mutual information across views. Proceedings of the IEEE/CVF international conference on computer vision, pages 9928-9999999 9928-9938, 2021.
[11]
Haw-ren Fang and Yousef Saad. Enhanced multilevel manifold learning. Dept. Comput. Sci. Eng., Univ. Minnesota, Minneapolis, MN, USA, Tech. Rep. ys- 20-201,- 2012.
[12] Shikun Feng, Yuyan Ni, Minghao Li, Yanwen Huang. Zhi-Ming Ma, Wei-Ying Ma. and Yanyan Lan. UniCorn: A unified contrastive learning approach for multi-view molecular representation learning. Forty-First International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[13] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu. Rohan Anil, and Chelsea Finn. Effi- ciently identifying task groupings for multi-task learning. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 27503-27516. Curran Associates, Inc., 2021. URL https: e /ttpatpp://prrrcednggss. neurips. cc/paper_ap.. files/ pappr/2021/E/// /ttpppprr..lllsspprr//////////// .pdf
[14] songwei Ge, Shlok Mishra, Simon Kornblith, Chun-Liang Li, and David Jacobs. Hyperbolic contrastive learning for visual representations beyond objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6840-6849, 2023.
[15] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704-67 2022.
[16] K Chidananda Gowda and GJPR Krishna. Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern recognition, 10(2):105-112, 1978.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016
[18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[19] Tianyang Hu, Liu Zhili, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learning is secretly doing stochastic neighbor embedding. In The Eleventh International Conference on Learning Representations.
[20] Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition con- fidence maximisation. In Proceedings of the IEEF/CV conference on computer vision and pattern recognition, pages 8849-88555 2020.
[21] Jiawei Huang and Hu Ding. An effective manifold-basel optimization method for distri- butionally robust classification. In The Thirleenth International Conference Learning Representations, ICLR 2025, Singupore, April 24-28, 2025. OpenReview.net, 2025. URL. https://openrevies net http://wwwww.....ooooooooo///eaa

---

[22] Jiawei Huang, Minming Li, and Hu Ding. Adaptive and multi-scale affinity alignment for hierarchical contrastive learning. In The Thirty-ninth ninth Anmual Conference on Neural Information Processing Systems, 2025. httphhtts:///oorreeeeeee
.ett/forummidd=32221Wia..
[23] Shudong Huang. Ivor Tsang, Zenglin Xu, Jiancheng Lv, and Quan-Hui Liu. Multi-view clustering on topological manifold. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6944-6951, 2022.
[24] Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsuper- vised image classification and segmentation. In Proceedings of the IEEE/CVF international conference computer vision, pages 9865-9874, 2019.
[25] Cheng Jin, Luyang Luo, Huangjing Lin, Jun Hou, and Hao Chen. Hmil: Hierarchical multi- instance learning for fine-graing tne-grained whole slide image classification. IEEE Transactions on Medical Imaging, 2024.
[26] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 18661-111,,,7, 2020.
[27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[28] Loic Landrieu and Vivien Sainte Fare Garnot. Leveraging class hierarchies with metric-guided prototype learning. British Machine Vision Conference (BMVC), 2021.
[29] Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33: 8847-8860, 2020.
[30] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 8547-8555, 2021.
[31] Friedrich Liese and Igor Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10):43444441. 2006.
[32] Shengheng Liu, Tianqi Zhang, Ningning Fu, and Yongming Huang. Fine fine-grained graph representation learning for heterogeneous mobile networks with attentive fusion and contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2025. URL
https: //arpi/ org/ /bs//4122 07809
[33] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018
[35] James MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium Mathematical Statistics and Probability, Volume I: Statistics, volume 5, pages 281 281-298. University of California press, 1967.
[36] Shentong Mo, Zhun Sun, and Chao Li. Multi-level contrastive learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2778-2787, 2023.
[37] Yuhao Mo, Chu Han, Yu Liu, Min Liu, Zhenwei Shi, Jiatai Lin, Bingchao Zhao, Chunwang Huang, Bingjiang Qiu, Yanfen Cui, et al. Hover-trans: Anatomy-aware hover-transformer for roi-free breast cancer diagnosis in ultrasound images. IEEE Transactions on Medical Imaging. 2023.
[38] Fionn Murtagh and Pedro Contreras. Algorithms for hierarchical clustering: an overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(1):86-97, 2012.
12

---

[39] Frank Nielsen. Hierarchical clustering. In Introduction to HPC with MPI Data Science, pages 195-211. Springer, 2016.
[40] Francesco Orabona. A modern introduction to online learning, arXiv preprint arXiv:
2019.
[41] Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, et al. Tripletclip: Improving compositional reasoning of clip via synthetic vision- language negatives. Advances in neural information processing systems, 37:32731-3217 2024.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[43] Joshua David Robinson, Ching-Yan Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021. URLhtttps pttttp://openevviii fttp://peeeeeeeeeeeeeeeeeorrrree mett/foruunid--R1100007
[44] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust
neural networks. In International Conference on Learning Representations, 2019.
[45] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020.
[46] Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 9495-9504, 2021.
[47] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2566 2018
[48] Liangliang Shi, Gu Zhang, Haoyu Zhen, Jintao Fan, and Junchi Yan. Understanding and generalizing contrastive learning from the inverse optimal transport perspective. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 11408-31427 PMLR, 23-29 Jul 2023. URL httpps://proceeedings mlr pttpppeess//////aaaaaa htt.eeddddng..m...preee/////////
.html
[49] Liangliang Shi, Jack Fan, and Junchi Yan. Ot-clip: Understanding and generalizing clip via optimal transport. In Forty-first International Conference on Machine Learning. 2024.
[50] Zhiquan Tan, Yifan Zhang, Jingqin Yang, and Yang Yuan. Contrastive learning is spectral clus- tering on similarity graph. In The Twelfth International Conference on Learning Representations, 2024.
[51] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6827-6839. Curran Associates, Inc., 2020.
[52] Yuandong Tian. Understanding deep contrastive learning via coordinate-wise optimization. Advances in Neural Information Processing Systems, 35:19511-19522 2022.
[53] Christos Tsirigotis, Joao Monteiro, Pau Rodriguez, David Vazquez, and Aaron C Courville. Group robust classification without any group information. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Information Processing Systems, volume 36, pages 56553-56575. Curran Associates, Inc., 2023.
13

---

[54] Yunbin Tu, Liang Li, Li Su, and Qingming Huang. Query-centric audio-visual cognition network for moment retrieval, segmentation and ceepeaationiing In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), 2025. URL https:://axiiii org/ abs/ 2412. 13543
[55] Hugues Van Assel, Thibault Espinasse, Julien Chiquet, and Franck Picard. A probabilistic graph coupling view of dimension reduction. Advances in Neural Information Processing Systems, 35: 10696-10708, 2022.
[56] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 8769-8778, 2018.
[57] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, AntrreAAtoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 1 11(1),, 2010.
[58] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9939-9933, PMLR, 2020.
[59] Rukai Wei, Yu Liu, Jingkuan Song. Yanzhao Xie, and Ke Zhou. Exploring hierarchical information in hyperbolic space for self-supervised image hashing. IEEE Transactions on Image Processing, 2024.
[60] Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering, In Proceedings of the IEEE/CVF international conference on computer vision, pages 8150-8159, 2019.
[61] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[62] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference machine learning. pages 478 487. PMLR, 2016.
[63] Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In International Conference on Machine Learning, pages 24851-24877 PMLR, 2022.
[64] Jianwei Yang. Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5147-5156, 2016.
[65] Lijun Zhang, Peng Zhao, Zhen Hua Zhuang, Tianbao Yang, and Zhi-Hua Zhou. Stochastic approximation approaches to group distributionally robust optimization. Advances in Neural Information Processing Systems, 36, 2024.
[66] Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hierarchical multi-label contrastive learning framework. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16660-16669, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
the abstract and introduction summarize our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: included
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
All assumptions are stated and complete proofs are included in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: included
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: included
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp.ccppulli//guiii wwwiicccguie///eeeeeeeeis CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Anonymized code and data are included in the supplemental material. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[No
Justification: it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: included
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: http::::////iis/.eeeee ctt.rrisscc..c.aececceooooooe httpp///ccc//ccccccccieeoeeeleee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification: we confirm our work complies with all stated principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: does not involve social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model and data do not pose high misuse risks requiring safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: the third-party datasets and code used are properly cited with licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
19

---

Answer:
[NA]
Justification: does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: does not involve human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
20

---

Justification:
No LLMs were used as a part of the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:eae//eerrppoocccc/oooeeeeee for what should or should not be described.
21

---

A Other Related Works
There are two key areas of research closely related to our contribution:
Multi-level Representative Learning Several studies have leveraged multi-level labels [25] 66 to learn hierarchical representations, which supervised learning approaches requiring multi-level labeled data. In contrast, our work focuses on unsupervised learning, aligning with the conventional settings of CL algorithms. Mo et al. [36] proposed to use two types of contrastive losses shallow and deep network layers, enhancing the quality of learned representations. Additionally, other works
14 59 embedded representations into hyperbolic space to better model hierarchical relationships. A
number of recent studies [32] [2) focused on refining augmentation strategies to better encode semantic information, either directly or indirectly. Our study demonstrates that by more effectively utilizing the semantics encoded in augmentations, we can capture fine-grained semantic structures more accurately through improved optimization techniques.
In the field of manifold learning, Fang and Saad [T] developed a multilevel framework that improves computational efficiency through graph coarsening and multi-scale refinement. Huang et al. [23] introduced a multi-view clustering approach that employs topological manifold learning to better capture the intrinsic nonlinear data relationships. Several works [58] 30 19
50 5
in contrastive
clustering have reported high clustering purity. For example, these studies provide various t-SNE or kernel density estimation visualizations confirm that contrastive objectives naturally facilitate semantic grouping.
Distributional Robustness Distributional robustness 29
22 addresses the challenge of model performance under distribution shifts by optimizing for worst-case performance across possible distributions. This concept is extended to group distributional robustness optimization (go)up-DRO) 45), which focuses on maintaining consistent performance across subgroups, including underrepresented ones. Recent work by [53] employs logits adjustment to balance performance across diverse data groups.
46
21
B Probabilistic Graph Coupling and Its Equivalence to InfoNCE
In this section, we first introduce the foundational concepts of the Probabilistic Graph Coupling Framework. primarily discussed in Sec. Scc.2.2] We then demonstrate how the InfoNCE loss can be interpreted within this framework. Although the detailed exposition is provided in [5]], we include summary here for completeness and to lay the groundwork for the theoretical analysis in Sec. 3.1
B.1 Graph-Based Probabilistic Modeling
Directly comparing and aligning the affinity matrices Ke and KTep is computationally challenging due to the potentially infinite number edges in the corresponding graphs. To address this, the probabilistic graph coupling framework (55) reformulates the problem by defining probability distri- butions over unweighted directed subgraphs. Specifically, it introduces posterior distributions over binary adjacency graphs Ween Wrep (0, 1] N each sampled according to the affinities Keea and Krep, respectively. The comparison is then framed as minimizing the cross-entropy (7) between these posterior distributions (see (6)). offering more tractable and principled approach aligning complex graphs.
Properties of the Subgraph Probabilistic Distribution Pr(W|K) Consider a graph with an affinity matrix denoted by K RNN (either K KTep, as defined in Definition and 2 From K, we sample a subgraph W (can refer to the aforementioned Waen or Wrep) under the topological constraints specified in For elarity, we restate the constraint follows:
s L (WE e {0, NNN Vi [M], Σ Wy=13
where Wiy indicates the presence of a directed edge from node i node j. This formulation restricts the subgraphs to binary directed graphs, where each node has exactly one outgoing edge.
22

---

Van Assel et al. [55] demonstrates that the posterior probability of sampling subgraph W is given by the distribution in ( namely:
Pr Pr( S(W)
II (1)) (K)) W.s
where (?())= - II. (,, W., =1) enforces the topological constraint, ensuring that each node exactly one outgoing edge.
Given the unitary out-degree filter described above, let W, and K, denote the i-th rows of W and K, respectively. The distribution Pr (W(W; K) satisfies the following property: for W Pr(::K, we have
W Multinomial (1. (,, Vi [N],
(18)
where the Multinomial(11) distribution produces a one-hot vector by drawing a single sample according the normalized weights in K, In other words, each node selects exactly outgoing edge, with probabilities proportional to its affinities. Moreover, the rows W, and W, are independent for any i, E [.].
B.2 Connecting Probahilistic Graph Coupling with InfoNCE
According to B in Definition T the adjacency matrix Keen represents the semantic relationships between data points in the augmented data space X. For the subgraph Weee sampled from the semantic affinity matrix Keea based on (18), the probability that there an edge from to j given by:
Pr (W = Keen) x Ki = Ex (Ty(())) (ra(z)]))),
(19)
where PRA (x( denotes the augmentation likelihood of x given the natural data point T.
Now we consider the embedding space. The set of feature embeddings corresponding to the aug- mented dataset X is represented as Z C Z, with each embedding
g(x;) representing the data point 2; E X. According to (4) in Definition [ for any 2,, 2j E Z, the matrix element is defined as:
Keep Ker(z;, 2;), where 2; g(v.), 2j = g(Tz).
Here Ker(..-) is kernel function. this work, we use innslatiooninvariiann kernel 555 (e.g.. Gaussian kernel).
For the subgraph Wrop sampled from the representation affinity matrix KTep, based 18 the probability that there is an edge from to j given by:
K
Ker(zi, 23)
Pr (Wzep_ =1 KTeP)
(20)
K
Ensr Ker (Zis 2x)
The cross-entropy between Waar and Wrep is decomposed into individual contributions of edges:
N Pr
W =1
K Kaan) log (Pr (W 1 K KT)P))
(21)
== ip
Ker 21)
EgpP E Pre (-1
log Σ kighe Ker Ziz zx
(22)
The (22 is obtained by substituting
19 and
20 into
If we take the kernel Ker(..)) -) to
Gaussian kernel, namely Ker(zi, 2,) = exp( 2; Zj
we can rewrite the cross-entropy (7)
exp (g(x) g(x))/T)
EawPy Brazy Pra
log Σ exp (g(x;) ( 8(xx)/7)
const,
(23)
which aligns precisely with the InfoNCE loss
(2)
Thus, minimizing (23) matches the InfoNCE objective, thereby establishing a theoretical equivalence between contrastive learning objective and probabilistic graph alignment.
23

---

C Omitted Proofs in Sec. 3.1
In this section, we provide proofs for Proposition 3.1 and Lemma 3.2 in Sec. 3.1 Our goal is to formally establish the connection between (i) the local contrastive objective and the KL divergence over subgraph distributions, and (ii) the subgraph sampling probabilities and row-wise independence, which underpins our localized probabilistic alignment framework.
Proof of Proposition 3.7 This result is direct consequence of the probabilistic graph coupling framework described in Van Assel et al. [55], Tan et al. [50]. Under the constraint Σ; Wi = 1 for all each row of W defines a multinomial distribution over neighbors. We use W. to denote the i-th row of matrix W, then the full sampling distribution factorizes row-wise:
N
N'
Pr(W K.) = Pr k.,)) = II
Multinomial K.,i (1; E3 Ka.s))
(24)
2m1
which corresponds to drawing one outgoing edge from each node independently, proportional to the row-wise kernel similarity.
Lemma C.1 (Restatement of Lemma 3.2 Let Ween and W subgraphs sampled from Keen and K. respectively, subject to the topological constraints in (5). Minimizing the cross-entropy between the distributions of these subgraphs, i.e.,
-Bwje-Pr( (e [og Pr( = W K:PP)
min 9(-)
(25)
is equivalent to minimizing the local contrastive loss L defined in
12
Proof. Recall that the local region (0, Zane) = (= E Zanc S. rs} selects points whose representations lie within hyperspherical cap centered Zanc- Let RR denote the indices of points in X falling within region R. Then the local semantic aftinity matrix K50m is the submatrix of Keen restricted LR IR., and similarly for the representation affinity matrix K
We sample subgraphs W from K and w from K each subject to the topological constraints:
WTOP E SW.O N where SW, & W. (W.
Waen E EWW,,
(26)
W W..y=1111 for W. E (W5t. W
(27)
where SW denotes the set of admissible sampled graphs from K, E KK K. node i has exactly one outgoing edge both sampled graphs W and Wrep respectively.
). That is, each
By Proposition 3.1 the probability of subgraph Ween given Keen factorizes row-by-row:
N!
N
WHo 4,3
N Ween = Vi. 4,6j
Pr(
K &
FeD 8,39
with the constraint
(28)
Specifically, Ween
= 1 means node i connects to node j. The same holds for Wr Pr(. kTP) +,3g
Using the factorized form 28 for both subgraph distributions, we decompose the cross-entropy sum over all (rows) and j (possible targets of node i),
N
N
Pr
WWen 8,55
= K5an) log (Pr (Wrep- = K:T)))
(29)
According to Definition definition of Pr.
we have Pr (Ween
K=en) = EgnPP [IA((T))) (ra(z,|=)]- By
we have
Pr W sen 5,0
= 1 k x Ea-Py (x)) (r)(
(30)
24

---

Further, the probability that the sampled graph W has node i connecting to node j under K2 is given by
K K
Pr (W =1
Kg
(31)
Kit ((W))))
K
Similar with Appendix B.2 if we take the kernel Ker(;;) :) to Gaussian kernel, namely Ker 2z) = exp(-IIz4 2| 2z 11 /T), we can rewrite the above probability (31)
Pr (W Top
=1 kT
exp
(32)
Exjr (0))), (exp(2) 2x/T))
where 1 (n(a,))s(( ERR; is the indicator function ensures that g(x;)) and g(rx)) lie in R. Finally, we can derive the cross-entropy by inserting (30) and (32 into (29
exp (2, 2//T)
E B~P PP
log Σ k 1x(x)) (exp(2, 2x<
+ const,
which exactly matches the local contrastive loss L 12 up to a constant. Thus, minimizing the cross-entropy between local subgraph distributions is equivalent to minimizing the local contrastive
loss.
D Theoretical Analysis for AMA-alignment
In this section, we establish the convergence rate of our proposed method: AMA-alignment (Algo- rithm D
Overview of the analysis. We begin by formalizing the multi-scale max alignment objective and deriving the foosed-form solution for the inner weight maximization. Based on this formulation, we justify the update rules for both the weight vector q and the model parameters @ used in Algorithm (Sec. D.I. Next, we introduce the boundedness and smoothness assumptions required for the theoretical analysis. We prove that the optimal weight q" is Lipschitz continuous w.r.t. 6. Based on these, we further show that the overall objective function is smooth (Sec. [Sec. We then analyze the recursive exponentiated gradient updates (17. demonstrating that they track the optimal weights with an ( error rate under stochastic updates. Finally, by quantifying the gradient approximation error and applying a single-step descent lemma, we telescope the inequalities to establish the O(1/vT) convergence rate (Sec. D.4.
D.1 Problem Formulation and Basic Properties
In this section, we introduce the overall min-max structure of our adaptive multi-scale alignment objective. We first formalize the outer minimization over the encoder parameters and the inner maximization over region-weight distributions qº (35) The algorithmic procedures are based on two technical results: (i) the closed-form solution of the inner maximization (Proposition D.I and (ii) the
gradient formula for the AMA-alignment objective (34) (Lemma D.2)
D.1.1 Restate of Algorithm I and Setup for Analysis
We study the following multi-scale min-max formulation:
min BEB
max EEA
Σ q. C.0, PKL
(33)
where (0, denotes the local contrastive loss at region Rý, and the KL term regularizes the weight distribution qe against the uniform distribution The full objective is
S
F(0) = Σ (*)
(34)
25

---

where for each scale the scale-specific objective is
(0) max (", h (0,0 = E C pKL(q"))u)
(35)
q'E
Although the iner maximizer q² in 35 admits closed-form expression, this optimal solution depends on the current encoder parameters 0, which vary across iterations. Furthermore, in stochastic setting, we only observe noisy estimates of the true loss vector (we denote (0,, Rt)) the estimate L* (0,, RR.)). naive update that directly computes the closed-form solution based on noisy loss estimate for a moving target O, would be unstable.
For both stability and efficiency, we therefore adopt a stochastic mirror ascent scheme. This approach smoothly tracks the moving optimum by regularizing the update step, keeping the new weight distribution qi+1 close to the previous one qi This leads the following regularized optimization problem for updating the weights from iteration to + 1:
4i+1 E argmiax
E (2,) pKL(q" - (LLL())))))
(36)
q'EAm
This step both exploits the new loss signal from the updated parameters 0.+1 and ensures stable cvolution the weights.
The update rules in Algorithm 1 are summarized as follows.
Updating Encoder Parameters (6): The encoder is updated via stochastic gradient descent using the weights from the beginning of the iteration.
0++1 02 To.t Se:
where
gr Σ
Σ Tir (L,
(37)
Updating Region Weights (q*): After updating 0, 0c4-1, the region weights are updated by solving the mirror ascent problem (36). As established in Proposition D.I. this yields the closed-form solution:
(qt,) 2/70 exp 1n (0 R2))
1
(38)
EjE (qi.)) 1/Te exp Mad (0 (29)
where "t = 1 --7gt- These two updates for 0 and {q*} alternate until convergence.
Proposition D.1 (Closed-form solution for regularized mirror ascent update). For the update step from iteration to t + the optimal solution q:+1 the regularized mirror ascent problem defined
in (36) is given by (38). Moreover, for any >> and Nqt > 0, this solution is unique and strictly positive (i.e., q:+1 iinteriorAmm))
Proof. The objective in (36) strictly concave in qa because both -p KL (|u) and 1 KL(q"|lq?) are strictly concave in q" on the simplex, and the remaining term is linear; hence Th. the maximizer is unique.
To find this solution, form the Lagrangian, incorporating the simplex constraint Σm- 9. 1:
Lagrangian(q" (q", A) = Σ ( (B.+1, (1+11,;)) R2)
q log(mm 9..)
log
X
9
9i
Setting the derivative with respect to qe zero yields the KKT condition:
L (0411 RC)- p(log(mm q.) 1)
(log
+ +1)
- ->
q
26

---

We collect terms involving log q and rearrange:
p na. log(g)) (fc+1, R2)) + nu. log(g;,) - plog(m") - C()),
where C(A) p 2+11/gg n + À groups all terms independent of the index r. Let 7/r + pf]q,t. The term on the left is (g.r). Multiplying by ng.t/7 7t isolates log(g)):
log(g = Nou 2 (fe+1, R2)) + log(qi.) -
where (A) is another constant absorbing A, p. and m Exponentiating both sides gives
q. ex=e((((((()) (q )/77 exp Da.i (4 (0.41,,
The term E q = Substituting the expression for q into this constraint and solving for the constant immediately yields the closed form in (38) for E++1.r-
acts as a normalization constant, which is determined by the constraint
Since > 0, Il..t > O, and by induction q > 0, the numerator of (38) is strictly positive for all This ensures that the solution q is unique and lies in the interior the simplex. By strict concavity and Slater's condition, the KKT conditions necessary and sufficient, completing the proof. O
Next, we justify the form of the outer update in T6) as the following lemma.
Lemma D.2 (Gradient of the Objective Function). (0 RC) is continuoully differentiable with respect to for each region and p 0, then F=(0) (35 is differentiable with V F (())
q. V RG,) and F(0) 34 is differentiable and its gradient is given by:
8 (0)
m
=
=
==1r=1 & (OVV
(39)
s=1
where (0) is the unique optimal distribution characterized in Proposition D.I.
Proof. The result follows from a direct application of Danskin's 's theorem for max- max-funntions with regularization. We verify that all necessary conditions satisfied:
(i) Compactness of the constraint ser: The probability simplex Am. compact.
(ii) Continuity and differentiability: The objective function ( (*) defined in (35) is continuous
in the pair (0, q) and continuously differentiable in 0 for each fixed q. (1, R.)) is continuously differentiable in 0 by assumption.
(iii) Uniqueness of maximizer: As established in Proposition D.I for each fixed 0, the KL- regularization term ensures that the maximizer (0) unique and lies in the interior of Am..
Since all conditions are satisfied, Danskin's 's theorem implies that F*)0) is differentiable and its gradient equals the partial gradient of ht (defined in (35)) with respect to d evaluated at the optimal point:
VoF* (0) Vgh (8, q (())
m
V
x (0) - (0]1
m
Ω (O)
(40)
The KL-regularization term vanishes in the gradient computation because it depends on 0 only through q (0), and the envelope theorem accounts for these implicit dependencies.
This elegant formula enables efficient gradient computation without requiring the derivative of q0* with respect to 0, which would be computationally intensive. For the full model with multiple scenarios, the gradient of the overall objective F(0) = E.
34p is simply:
S
8 m
VF(0) VF = Σ Σ q (0) RO,
27

---

D.2 Key Assumptions and Properties of the Adaptive Weight Model
We now state the technical assumptions underlying our convergence analysis, which include bound- edness and smoothness conditions (Assumptions) T.. Under these assumptions, we establish two fundamental properties: (i) the Lipschitz continuity of the optimal weight mapping 0 + q"* (0) (Lemma D.3 and (ii) the smoothness of the composite objective F(0) (Lemma Together, these results guarantee that the interaction between the inner maximization and the outer minimization remains stable and analytically tractable. Unless otherwise specified, all vector norms are Euclidean and all matrix norms are spectral.
Assumption 1 (Gradient Boundedness). For all 0 θ scales s, and regions R. VV (0, RB) S.
Assumption 2 (Loss Boundedness). For all O regions Rý, and scales s, there exists constant Lmxx such that v (6, S Lmax-
Assumption (Smoothness). For each scale & and region r, (", R2) is T-smooth in |VoL (0,, RT))) (oL*(02, Rt)Il S T.||0] - 0211 for all 0,,02 E ④.
Assumption 4 (Two-Batch Sampling). At each iteration 1, we use two independent mini-batches: one for computing the region loss estimates to update the weights q" (0.). and another for computing unbiased stochastic gradients (B,, R²). For constants o2 and og. the stochastic loss satisfy:
(0,, (O,, L (0,, R2); BIIL" (0,, L (0,, R2)10 ozi
(41)
the stochastic gradients satisfy:
El, R2)] v L (0,, R:);
(42)
Remark 2. In practice, single minibatch is often reused for both the weight update and the gradient step. Although this induces a mild statistical dependence between q (0,) and the gradient estimator, experiments show no degradation in convergence. For theoretical rigor, adhere to the two-batch model our theoretical development.
Lemma D.3 (Lipschitz Continuity of Optimal Weights). Under Assumptions 7 the map f -> (0) that gives the unique solution to the inner maximization in 35) is Lipschitz continuous. Specifically, for any 0,, 02 E O:
Gvm (
100 0211
(43)
P
Proof. The function 1t (0, )r) defined in 335 is p-strongly concave with respect to q" due the KL-divergence term. Thus, for any fixed 0, there exists unique maximizer (0). This mapping is differentiable, and we can analyze its Jacobian 0q by applying the implicit function theorem the Karush- Kuhn- Tucker (KKT) conditions.
The KKT condition for the inner maximization problem is Va (0. the Lagrange multiplier for the constraint Σ, q 1. Differentiating this system with respect yields:
(0)) + A1 = 0, where is
aq
ax 00
1"0, + 2 2 "000
+
1 O.
(44)
00
Let P 1 be the projection matrix onto the tangent space of the simplex, fo R | 0). Since (0) = 1 for all 0, its derivative 0g lies in this tangent space, meaning p p0q"r 0q"""
00
Projecting with eliminates the Lagrange multiplier term (as P1 0): 00
0q 0.
PVig (1, -a PV2 h (0.
00
Since 0q""" is already in the tangent space, we can write PV2 230
h² heppa"" qq
do and rearrange to solve for
the Jacobian:
0q
(PV2 ( p) PV
00
28

---

where (-) denotes the pseudoinverse, which acts as the inverse on the tangent space.
We now bound the norms of the matrix terms.
(1/ // ..... 1 1/ Due to p-strong concavity, the projected negative Hessian is positive definite on the tangent space with its smallest eigenvalue being at least p. Thus, the operator norm of its inverse on this subspace is bounded:
(1) Hessian Term: The Hessian of he with respect to q" V qq (
( (PV2 "M
P) = -
(2) Mixed-Derivative Term: The mixed derivative is the Jacobian matrix whose r-th row is V ooL* (0, R.). We can bound its Frobenius norm using Assumption
|
m".
1/2
m
1/2
2 9q
h1
V 6q
=
I1 (0, REM1
Σ
G²
= -Vmm
Combining these bounds and using ||||| S 1, we get:
-qa**
$ (PV2 h pP) ||||| 1V hlll S 1. Gvm Gym
00
By the Mean Value Theorem, for any 01, 0g O there exists some 1 on the line segment between them such that:
0q
Gym
(0:) -0=0(l)) <
100 0211 S
10, 0211-
00
This establishes the Lipschitz continuity of (0) with constant L.g
Gym P
Lemma D.4 (Smoothness of F(0) in (34 Under Assumptions
let M max, m². Each
function F*) is -m-sthoth with a constant LF= S L + MG²
The overall objective F(Ø) =
Σ (0) is L Lm-soooth with LF S S(L + MG²
Proof. For any 01, 02 E O, by Lemma D.2 we have:
(o.F((0)) - VoF
9 (0.) (01, R3) Σ 9: vo' RE)
m
m
=
> Φ Vo) R$)
(0, VoL R3)
(l RC:)
(02)VoL
m
Σ C (B)) R3) VoL RT)|
m
Σ
( - प: (8₂) VoL R:)
(45)
P=1
For the first term, by Assumption 3 and the fact that > (00) 1:
mc
m
(()
V (0, R:)
a. (01) - 021 = L| - 021l (46)
For the second term, by Lemma D.3 and Assumption
29

---

m
Σ (01) (02)) -V (02, R;) S G. (01) (02))
Gv 110 021
S G vm (0)) - 9" (02)l2 SG. m
m G 110 02l1-
p
Combining the two bounds and using m Σ M:
[V Vo (02)l = T.+ [.++ m² (1+*m(()) 110,
(I+ MG²
021
110 - 0211-
This establishes that each (*) is smooth with constant Lp- I L + MG² Since F(0)
Eam F6), the smoothness of the sum is the sum of the smoothness constants (by triangle inequal-
ity), so F(0) is Lm-smooth with Lp E. LF S(L MG²)
D.3 Analysis of Stochastic Weight Updates
We analyze the recursive stochastic exponentiated-gradie
(EG) updates for the region-weight
distributions qe defined in (36). Our goal is to show that, despite stochastic noise and the slow drift of
the encoder parameters Or. the iterates q track the instantancous maximizer qa** (0,) with vanishing
error. particular, we establish an O(1/t) rate for the expected tracking error EE [[LL(q (0:)]l))
For simplicity we fix scale s and drop the superscript for qi. At iteration t, the inner maximization problem is
max m
Σ
(0,, R2)) /KL(allu))
(47)
whose unique solution is the Gibbs distribution qi softmax(C"(0.)))) (Proposition D.I. Algorithm updates qe via the stochastic mirror-ascent step
TT
qe E argmax qEAm
Σ q, (0 PKL(l))u) 99.--1 (I(allq-11)
(48)
where (0,-,1, R is an unbiased estimate of (0,-11 RT) (Assumption 4
We will show that under the usual bounded bounded-gradient and smoothness assumptions, and with 6g itself updated by stochastic gradient descent, the mean squared fi firtracking error El|q: q" (0:) decays as O(1/t).
Theorem D.5 (Tracking Error of Weight Updates). Under Assumptions 73
run the region weight
updates 48) (whose closed-form is (38) as established in Proposition D.I
Then there exists
constant C depending only on nu. G. p. Lmaax T such that
E Elig - q'(0.) < C.
where (0) = arg qEA A. E,, 9r (9, R,) KL(g]]
The proof proceeds three steps: (i) a one-step KL-descent bound for the exponentiated update: (ii) a "thrre-point" decomposition to relate KL(q" (0,-1))(q) q.) to (L(a"(00-)))l (0,-))llq--): and (iii) casting the result as scalar recursion and invoking a standard ((1/t) lemma, followed by Pinsker's inequality.
Lemma D.6 (One-Step Step Contraction). Let qi softmax L (#.-1, Under Assumptions and 4 for any I. such that Iq 1p 5 1/2,, we have
E B[KL(qi-11(:)) i < (1 Tq..-19)) KL(Q--Llqq-- Co 1
(49)
where Co > 0 is an absolute constant.
30

---

Proof. Define the composite objective hq)1 = Σ, 9r (8,-1,
( which is
p-strongly concave w.I.t the KL divergence. The mirror-ascem. step (48) satisfies the following inequality for p-strongly concave objective 40
(01-111))) (1 + 1/q.-11P) KL (. - KL 111 + 1q.111 (6111)) -000-)))))-(0(0-1))))))) 9:
9--1). (50)
Let f denote the zero-mean noise vector. Taking conditional expectation and applying the standard bound El 9: qi_1) Fi-1] S (1a.--1/2) E| Fe-1J, together with Blllk-1 S mo? (Assumption yields
(1,-14)111))) E ( Fi-1] -
+ 21
(51)
Dividing both sides by (1 + 1p) and using 1/(1 x) 1 - x x.² for x: 2 0, we obtain
1 mo?
E[KL(qi-1lQQ:) Fi-1] S (1 -q 11 (1q,t-10)
KLL 2(1 + 1/q ID)
Since both -;-1 and 9r-1 lie in the interior of the m-simplex and each coordinate of q is lower- bounded by 7 > 0 (from the softmax parameterization), we have KL(qi-1lq S log(1 (1/ /?) S log m + A/p, which is a constant independent of t.
Thus, the (1)) 2 (L....) ) term is O( By combining all O ( terms, absolute constant Co > such that for large enough, the bound (49 holds.
find an
Proposition D.7 (Property Bregman Divergences (3I). Let (p(q) Ex qr log gr be the negative entropy function. The KI. divergence Dv(p.g) (T,pp114) is the Bregman divergence induced by r) and satisfies the following three-point identity:
(a D)(a,b) (bb,c)) + (a b. Vv((c) (666)-Velll - Ve(())
(52)
where Vb(()) = llgg + 1.
Proof of Theorem D.5
Let
A, = [[L((q
a = (f-
From Lemma D.6 taking total expectation and using 7q.t-1 =1 7// - gives
0x S (1 m Ar-1 14-1000 0
(53)
Using the Bregman three-point identity,
KL(qillq)) KL (qp-)) (L( + (qî q log q: log q.
(54)
The drift term KL q and the cross-term's dependence on 0 are both bounded by the move-
ment of Or. By Lemma D.3 (Lipschitz continuity of q²) and BI10, 0 ni., - Billg [|
O(1/t),, we have:
E = 0(1/t))
(55)
and for the cross term, by Young's 's inequality:
28
El(gi -i-1 log q: log 1)
<
E
KL (L + 00_111
25
<
22 at + (11(t).
(56)
Taking total expectation of (54) and combining terms gives:
A M (1 (10+2))44 + 0(1
(57)
Substituting
(33) into 57 yields
A S (1 +
26
Dan ) | +((-2).
2
(58)
31

---

Choosing 8 > 0 sufficiently small such that > (which is possible as Mop > 1), we get the recursion:
4 S
A1-1 (1- O(1/t).
(59)
This standard recursion, with contraction factor (1 c/t) and an error term O(1/f), implies that the sequence A, satisfies
A, == E S (1//t).
(60)
Finally, by Pinsker's inequality, this implies bounded [.] error:
B ا.. 2E|KL(a s O(1tt).
(61)
This proves the claim.
D.4 Convergence Guarantees for AMA-alignment (Proof for Theorem D.10)
smoothness (Lemma[D.A) Sec. D.3), weight-tracking error (Theorem [D.5 in Sec. [D.2). and gradient approxima- tion bounds to prove the global convergence rate. We first quantify the error introduced by using
In this section, we bring together all previously established ingrediens-oobjcee
(0,) instead of (0,) in the descent step (Lemma D.S and then derive a single-step descent inequality (Theorem D.9. Summing these inequalities via Abel's lemma yields the final O(1/VT) convergence guarantee for non-convex stochastic optimization (Theorem D.I0P.
E ( V E (0,, I2)) be the gradient used in Algorithm 1 and let (F s=1 == q. VO. (0,, Re) be the true gradient of the objective. Under Assumptions the gradient approximation error satisfies:
Lemma D.8 (Gradient Approximation Error). Let
Bilgg ( S 2G Σ s
(0.) "0*"(00)l
(62)
where og is the variance from stochastic gradient estimation.
Proof. We decompose the error into two components:
S m
S m
g: VF(0,)
V. R
9 (O) R;)
(63)
(0)) (V - +
(0.) (0.)) R2))) (64)
(4)
(B)
Using the inequality ||a bll2 < 2||a|12 2|16112:
Ellgg FF()) 2E[] All?] B|
(65)
For term (A), using Assumption (due to two-batch sampling):
4 and the independence of the stochastic gradients from the weights
VCッ - ΣΣ -
E|| E||
E
.5 m
5 -z Σ 1= So²
I 0 Σ (2)0)))
(66)
For term (B), accounting for non-orthogonal gradients and using Schwarr:
32

---

E
B||
B
Σ
m
(0)) (0.) VL (0,, ((tt) |
m
(0.) (0)) V (0). R2:) ) 2
E
s=1
$ G2
E B" (6.) -t (..'().))l)
(67)
Combining (66)
67 with 665
E g: VF(0))) S = + 2G
Σ ==
5
Blllq" (0.)
Theorem D.9 (Single-Step Progress). Under Assumptions /
with learning rate To,t Lp where
LF = S. (L+ /MG² is the smoothness constant from Lemma D.4 a single step of Algorithm
satisfies:
E [[F(04+))0O) S F(0.) 77.. (1
LF7
|VF(1)))
2
Lr
8
+ 2
(2S03 + 2G
- (O.))) (68)
Proof. By the smosttthsssss of F(0) from Lemma D.4
LF
F(0(+1) < F(O)) + (VF(()), 0.41 - 01) +
1 0ell
(69)
L
F(O)) тe.
8i) +
|
(70)
2
Using g: = VF((:) + (g: V F(())) and taking the conditional expectation:
E[F(O++1)001} < F(0,) TJp, (V))))) VF())) no.. ww. (l. VF(0.))0.)
(71)
(F)))) + (g: (F(
(72)
By the two-batch sampling (Assumption 4). Eig VF(O)) = 0. Expanding the squared norm:
1[F(0++1)00)) < F(O.) TTo.. To.:)1
(73)
LFTiE.
(IVF E -F(())))))
(74)
Applying Lemma D.8 and rearranging:
LF
E F(( & F(O.) n0. (F())))
|VF((1)11
(75)
2
LFnee (2S02 22 Elllg((0)) 9"' (0.1110))))))
s
(76)
= =((O)) 11, (1- LF
(77)
LFTh. (2502 -2 Σ
(lllq((0)) (80((1)1110))
(78)
33

---

Theorem D.10 (Convergence Rate). Under Assumptions with a E (0, 2 and weight step sizes nqtt
using parameter step sizes n8,t
no with 1/o 0, Algorithm achieves:
Σ T 1=1
(79)
C
where C a constant that depends on the initial suboptimality gap F(01) - F*,, the gradient bound G. Here, F inlage F(O), and Co
Lea
0.
Proof. From Theorem D.9 with 17 v.:
(1
Lra
[[F(0++1) & B[F(O.)]
B[IVF(0))1
(80)
Vi
2V1
LFa²
+ 2t
25o + 2G
(l,) q"h (0:)
(81)
Fort 2 to [ Dz [(1-c.)²
1. we have
1
Lea 2
2 Ca- Using this and Theorem D.5
- B1| (0))))
a
LFa? ?
250 + 2
Vi
(82)
21
Rearranging:
SLpa2
LoaR t +
C E <
E[F(
(83)
12
Multiplying by vt and summing from t = to To
BIIVF(O) < -FFF(O)))EEFF001)))
Ca 0
Vl SLpa 2 og 3 t
T
vi
Lya
(84)
22
1=
1=
Using Abel's summation for the first term:
Vi(E[F(O)) ([F(O++)))) (Tol(F((O..)) VTE[F(00+1))
t=fo
Σ t=
-V)
(85)
For the remaining terms in 84 Stp-
we the bounds Σi yi = Σte
Vi 2 and Σi= 1
Eisto 3372
S
de
Vla-i = =(.).
1
2
Since F inf F(0), have 1] F",
F", and Vi 1t-I- 0:
deB
a
((1OE VTF F Σ (Vi Vt- + 21 2T
1=to11
-VTEEF((0.)) VTF* + (VT Vo) 2SL o +0L)
=VT((E[F(0..) - F*) + 2ST 2T
$2SL VT 0(1)
(86)
34

---

Dividing by ca to - 1) and using T - to + = T for large T:
Bl] 2SL VT
0(1)
+
- to +
Cal'
C.al
20L
0(1)
(87)
C
Co
Including the initial to iterations and using the boundedness of gradients (Assumption
to 1
- 10
+1
1
B 1IIFF(1)
G2
T to
t=fo
0(1) T
25L
0(1)
+
(88)
ca CTVT
Ca
For large 7, since dominates Th we have:
EIIVFONN C
Σ
(89)
CaaVT'
where is constant. This establishes convergence rate of O(L/VT) for the average squared gradient norm, which is optimal for non-convex stochastic optimization.
Remark 3. Algorithm 1 the weight step size 174 can be implemented either as a constant (for simple implementation) using decreasing schedule 7q.1 (for theoretical guarantees). Our convergence analysis shows that with appropriate step size schedules, our method achieves the optimal ((//VT) rate for non-convex stochastic optimization, with all error terms vanishing as T increases.
E Gradient Analysis: A Hierarchical Optimization View
In this section, we examine the optimization dynamics of our AMA-aligmment framework by analyz- ing its gradient structure. We extend the coordinate-wise optimization interpretation of contrastive learning [52] and demonstrate how AMA-alignment introduces a novel third level of adaptation through region-wiss weighting. This view reveals the distinct mechanisms by which method emphasizes both fine-grained alignment and difficult semantic regions across scales.
E.1
Why Gradient Analysis?
Gradient- analysis offers valuable insight into how contrastive learning adjusts representations during training. Prior work [22 interprets the InfoNCE loss as two-player game:
A max- max-player updates the encoder parameters to improve representations.
A min-player adaptively reweights negative pairs through soft importance weighting function a.
This perspective explains how methods like hard negative mining operate by emphasizing certain negatives via a.
In AMA-alignment, introduce third optimization regionnegillleeel importance weights which prioritizes regions of semantic misalignment. This leads to a hierarchical three-player game, where the gradient reflects contributions from individual sample pairs, localized regions, and semantic scales.
F.2
Local loss: pair-wise gradient inside a region
We begin with the gradient of the local contrastive loss (0, RE) for region R; scale s. Recall that this loss encourages representations within R2 align with the corresponding semantic affinities (see
35

---

Let (z,, denote a positive sample pair drawn from (-)(- and let represent a negative sample within the same region R,G We define the squared pairwise distances as (2, 2,) = ||, 2, 21
l1g( - g(z,))l II and similarly, df (2,,, 2x) = |||; - =x Then, the gradient of the local loss in 12 can be expressed follows.
Lemma E.1 (Local Contrastive Gradient). The gradient of the local contrastive loss (6, RR)) with respect to parameters 0 is given by:
(0,
E & (a 2~P
E (r
Σ
ai Vad B3 Vod Ved 2,),
(90)
00
keikkEl
where:
explx Ex/T) a Eigie up exp(2, 21/T) 1 ny (2x) is the importance weight for negative pair (x, 24.)
3 EnELKERR 0ly is the aggregate negative importance
I RR (2x) indicates whether sample æu falls within region Rt
Proof. We start with the formulation of local contrastive loss in terms of functions &i and s as defined in [52). Specifically, we take (xx) = T log (e + 2) and vi))) = expxt/T, we can express the local contrastive loss as:
L = L.
E R-Px
E La P1
[(5.))]
E II
Σ
pidyy - dli))] )1]
-d.a)
(91)
R~PR
P
PP
To compute the gradient, we apply the chain rule S. Let Esp En (d di.). then:
dc
0Ev)
]]
E R~Px
('(ss) 00
(92)
00
2
(P
Further applying the chain rule to Eiz:
8Ei
Oldy
[ddy
adl
Σ
v'(dly
db.))
ikk
Σ
d'lc dia)
(93)
00
00
00
00
R
Resf RER;
Substituting back and using the specific form of o and v. we obtain:
acS
0.0
E
E
('(En) Σ
(att dze
(Vod (2), (
(aa))-oodP(a (od +]]]]
(94)
08
B~Px
PT.((1))
kei,
+ and v'(x)) 1 + cxp(xx/T). This yields: With our choice of & and ui we have d' (E(Es) c-Su,
af (EGg)) (dt d2y) .T Eig exp(((lt, - -z)))/)--- exp( l
(95)
c E1AL exp( (d. dz)/T)
As 0 and considering that diy = = 2n 2j + || + ||z can simplify it to:
exp(2,] 2t/T)
0 Σ EIiLER exp(2, 21/T)
1 12 24)
(96)
36

---

Let 5 = Σ
ale. Rearranging terms, we arrive at the final gradient expression:
ke
OL
E 2~P
B (
Σ
0 Vod (2), =k) 33 Vod (z, 2n)
(97)
00
k
This completes the proof.
L
Intuitively, this gradient structure reveals two competing forces within each local region:
1. The first term pushes dissimilar samples apart, with weights og that emphasize hard negatives
2. The second term pulls similar samples together with strength proportional to 3
Unlike traditional contrastive learning, approach restricts these forces to operate within mantically coherent regions Re preserving local structure while preventing inappropriate global repulsion.
E.3 Global objective: aggregating regions and scales
Building on the local gradient structure, we now characterize the overall gradient of our AMA- alignment approach:
Proposition E.2 (AMA-alignment Gradient). The gradient of the AMA-alignment objective F(0) in Equation
15 is:
(gF(0) ΣΣ
E
Σ
of Vid (x,, - m
(98)
8~PR (..2y
s=1==1
KE;
where qh the region importance weights that adaptively emphasize challenging regions.
Proof. By Lemma D.2 we have ==F(0)=Et -= Em T. (0)Vo R²). Then substitute the local loss gradient in Lemma E.1 completes the proof.
L
This gradient formulation reveals how AMA-alignman integrates information across multiple scales and regions. The region weights q act as attention mechanisms, focusing optimization efforts on regions where representation and semantic affinities are most misaligned.
Three-Level Hierarchical Coordinate Optimization. Our AMA-allinment framework extends the traditional two-player game [52 of contrastive learning three-level hierarchical optimization. The proof follows similar analytical principles as established by Tian [2 and is therefore omitted for brevity.
Proposition E.3 (Hierarchical Coordinate Optimization). AMA aligmment implements a three-level coordinate optimization procedure:
m
(Region weights)
(0,) = arg max q E'EAm
Σ
q (0,, - pKL(g')}
(99)
(Pair weights)
(²())) = { Σ Lii,EER exp(2) 21/T)
exp(2 2k/T)
LR (Ix) i.b
(100)
(Parameters)
0ct1 0, 7/8
vo. R2))
(101)
This hierarchical structure involves three coordinating players:
1. Parameter optimizer (0): Updates model parameters optimize representations across all scales
2. Local affinity optimizer (a"): Assigns importance to sample pairs within local regions
3. Region importance optimizer (q*): Adaptively weights regions based on optimization difficulty
37

---

Comparison to Hard Negative Mining Hard negative mining methods 43 modify 0s globally to emphasize difficult negatives. In contrast:
Hard negative mining: Applies globally, adjusts pairwise oue over the full sample space. Hard negative mining methods adjust sample-pair weights Ogk to emphasize challenging samples. For example, methods like [3 modified weights:
HN E
(102)
EN exp( (/
where controls the emphasis hard negatives.
AMA- -ll-aliimmmnt: Operates locally, adds region-level attention mechanism q."" that adaptively prioritizes semantic regions across scales.
The approaches are orthogonal and potentially complementary. While hard negative mining em- phasizes difficult samples, AMA-alignment emphasizes difficult regions-offering new dimension of adaptivity.
F
Implementation Details
Unsupervised Region Construction. In unsupervised settings where no labels are available, we construct hierarchical region partitions via an iterative clustering procedure directly within the representation space Z. In a MoCo v2 implementation [8], for instance, we cluster the embeddings from the momentum encoder's queue, which maintains large and diverse collection of recent representations. We then perform Spherical K-means clustering on this subset to partition the embedding space. To establish the hierarchical structure, this clustering process repeated for each scale E (1....,S) S), employing varying number of clusters, m², for each. Specifically, smaller number clusters (e.g., m² = 5) defines the coarser scales that capture broad categorical divisions, while a larger number (e.g., m 100) constitutes the finer-grained partitions. The resulting cluster centroids are designated as the anchor points (2₃), with each cluster of embeddings forming a distinct region R... Subsequently, any given embedding 2 Z is assigned to the region corresponding to its nearest anchor point in terms of angular distance, consistent with the definition in Sec.B.I] This set of multi-scale partitions is held static for the duration of an epoch, providing stable structural scaffold against which local affinity alignment is optimized.
Supervised Region Construction. In supervised settings where labels are available, we leverage class information to guide the construction of regions. Specifically, we compute the centroid of the embeddings within each class cluster and use these centroids the anchor points 2 E Z. Each sample is then assigned to the closest anchor (in angular distance), forming a set of semantically meaningful regions. When the available label hierarchy is shallow (e.g., only superclass annotations are provided), we follow the same hierarchical random sampling procedure used in the unsupervised setting to further subdivide regions into finer-grained partitions.
G Experimental Results
We evaluate the proposed allgnmitt framework on multiple benchmarks to assess its ability capture fine-grained semantic structures. We compare against state-of-the-an contrastive learning (CL) baselines in both unsupervised and supervised settings. Here is the code.
Datasets and Experimental Setup We conduct experiments mainly five public datasets: Deep- Fashion [33] a clothing dataset with fine-grained annotations suitable for constructing hierarchical category labels; iNaturalist (iNat) [56], long-tailed dataset with taxonomic hierarchies (species - genus > family); CIFAR- 100 [27], consisting of 100 classes grouped into 20 superclasses; Model- Ner40 (61), comprising 3183 CAD models from 40 object categories; and Bulmg (37, ultrasound dataset for breast cancer diagnosis.
We adopt the well-known SimCLR 7. MoCo-v2 g HardNeg 4 and c-CL 52 as unsupervised
CL. baselines. We also consider the supervised scenarios and adopt the popular baselines SupCon 26 Guided-proto (28) and HiMulConE (66).
38

---

The models are trained using ResNet-50 [7] as the backbone and evaluated via linear probing (unless otherwise specified), Training is conducted for 200 epochs using AdamW optimizer [3]], with weight decay 5 x 10-6, and learning rate initialized at 0.01. Part of the implementation details is provided in Appendix F
G.1 Downstream Performance
Table shows classification results (evaluated by linear probing). AMA-alignment consistently outperforms existing CL methods, with substantial gains on fine-grained benchmarks. On Deep- Fashion, we observe over 5% improvement, highlighting AMA's ability to capture subtle semantics. Performance on other datasets also improves notably, especially in tail classes as shown in Table
Table Top-1 accuracy on downstream tasks using linear probing. AMA-all AMA-alignment outperforms over prior methods, highlighting its strength in fine- fine-grained representation learning.
Method
DeepFashion
ModelNet
iNat
ImageNet
SimCLR
70.3
79.3
54.0
69.5
MoCo-v2
g
70.8
79.6
55.3
68.1 1
NardNeg 43
70.9
79.8
55.8
70.4
a-CL
22
71.7
79.6
56.1
70.2
AMA -alignmett (ours)
75.8
80.6
57.2
73.3
We further evaluate AMA-alignne AMA-alignment on Bulmg 337, challenging breast ultrasound dataset for cancer diagnosis. This dataset contains ultrasound images annotated with both binary classification labels (benign/malligantt) and more fine fine-graine grainrdind BI-RADS (Breast Imaging-Reporting Reporting and Data System) categorizations from 1 to 4, where higher numbers indicate greater likelihood of malignancy. Each image was independently labeled by two experienced radiologists, providing a natural multi-scale hierarchy for evaluation coarse binary classification and a finer four-level BI-RADS categorization.
For medical applications, Area Under the ROC Curve (AUC) is more appropriate evaluation metric than accuracy, as it better represents performance across differrnt decision thresholds, which is critical clinical settings where the cost of false negatives and false positives differs significantly. Table T shows that AMA-allgnmeent significantly outperforms existing contrastive learning methods across all BI-RADS categories, with particularly notable improvements in the challenging borderline categories (BI-RADS 3 and 4) where clinical decision decision-making is most difficult.
Table 5: AUC scores for BI-RADS classification the Bulmg breast ultrasound dataset. AMA- alignment shows substantial improvement distinguishing between clinically challenging borderline cases.
Method
BI-RADS
BI-RADS
BI-RADS
BI-RADS 4
SimCLR
0.898
0.811
0.756
0.662
MoCo-v2 8
0.910
0.821
0.768
0.675
a-CL 152
0.912
0.827
0.806
0.681
SupCon 126
0.942
0.877
0.817
0.693
AMA-alignment (ours)
0.956
0.883
0.844
0.735
The substantial improvement in AUC scores demonstrates AMA-alignmen ability to capture clinically relevant features across different levels of diagnostic certainty. This is particularly important in medical imaging applications where hierarchical structures often naturally exist (from normal to definitely abnormal, with gradations between). The multi-scale nature of our approach enables the model to simultaneously learn discriminative features for the overall benign/malignant classification while also capturing the nuanced differences between adjacent BI-RADS categories. Notably, AMA- alignment achieved 0.735 AUC on BI-RAD RDDS 4, representing a 4.2% improvement over supervised contrastive learning, highlighting its effectiveness the challenging "probably benign" category where misdiagnosis risks are highest.
39

---

We further evaluate the downstream performance of CLIP 42 on the CIFAR- 100 dataset using model pretrained on the Conceptual Captions 3M [4 image-caption dataset for 100 epochs. following similar experimental setup to [Я]. We consider two training settings: (i) linear setting, in which only the classifier trained, and (ii) a full setting, in which both the backbone and the classifier are jointly fine-tuned for 10 epochs. The results are reported in Table
Table 6: Top-1 classification accuracy on CIFAR-100.
InfoNCE
HardNeg 43
Triplet [1
CyCLIP
[3
Entropic OT 49
Ours
Linear Full
45.4
46.8
45.5
31.3
46.4 67.9
47.3
68.0
69.2
65.6
70.1
71.6
Hierarchical Representation Quality To assess the preservation of semantic hierarchies in learned representations, we employ two metrics: (1) Hierarchical Clustering Normalized Mutual Information (HC-NMI) [10], and (2) Intra-class Variance Reduction (IVR). summarized Table 7 AMA- alignment achieves superior alignment with ground-truth hierarchies, demonstrating its effectiveness in modeling multi-scale structures.
Table 7: Hierarchical alignment results in DeepFashion. Higher HC-NMI and lower IVR indicate better semantic structure preservation.
Method
HC-NMI 1
IVR
SimCLR
0.52
0.134
HardNeg 4
0.56
0.119
o-CI. 52
0.59
0.114
AMA AMA-alignment (ours)
0.66
0.091
G.2
Comparison with Previous Deep Clustering Methods
To further evaluate the clustering capability AMA- AMAA-alinmmen, we compare it with a comprehensive
set of classical and deep clustering methods three challenging benchmarks: C1FAR-10/100, and
Tiny-ImageNet (27] The baselines include traditional clustering techniques (e.g.. K-means [35
Spectral Clustering (SC) [16). NFM (3), autoencoder-base methods (e.g., AE [2]. DAE [57]
DEC
62 ), and recent deep clustering models (e.g., JULE [64). DAC 66, IIC 24), DCCM [60]]
PICA 20
ConCluster (30)).
Table presents clustering performance terms Normalized Mutual Information (NMI), Accuracy (ACC), and Adjusted Rand Index (ARI). While ConCluster (30 achieves the highest NMI and ART on CIFAR-10 and CIFAR-100, 100, our AMA-alignment surpasses all methods in accuracy (ACC) on every dataset (we evaluate the accuracy of our method by linear probing). This demonstrates that the representations learned by AMM--aligmment are more aligned with semantic ground- -tuud labels, even though our approach is primarily designed for hierarchical contrastive learning rather than clustering-spent specific optimization.
Notably, AMA- -lllamignment exhibits particularly strong performance Tiny-ImageNet, where it achieves a significant gain clustering accuracy, highlighting its potential in handling hierarchical structures, which classical clustering methods often fail to capture. Overall, these results validate the effectiveness of AMA-alignment not only representation learning but also clustering, especially in scenarios where multi-scale semantics crucial.
40

---

Table 8: Clustering performance on various datasets. Best results are shown in bold, and second- best results are underlined. Although AMA-alignment is not specialized for clustering tasks like ConCluster, it still achieves competitive results, ranking second in clustering metrics. AMA-alignment significantly outperforms deep clustering methods in linear probing accuracy.
Metrics
CIFAR-10
CIFAR-100
Iny-ImageNet
NMI
ACC
ART
NMI
ACC
ARI
NMI
ACC
ARI
K-means
0.087
0.229
0.049
0.084
0.130
0.028
0.065
0.025
0.005
SC
0.103
0.247
0.085
0.090
0.136
0.022
0.063
0.022
0.004
NMF
0.081
0.190
0.034
0.079
0.118
0.026
0.072
0.029
0.005
AE
0.239
0.314
0.169
0.100
0.165
0.048
0.131
0.041
0.007
DAE
0.251
0.297
0.163
0.111
0.151
0.046
0.127
0.039
0.007
DEC
0.257
0.301
0.161
0.136
0.185
0.050
0.115
0.037
0.007
JULE
0.192
0.272
0.138
0.103
0.137
0.038
0.102
0.033
0.006
DAC
0.396
0.522
0.306
0.185
0.238
0.088
0,066
0.017
0.006
DCCM
0.496
0.623
0.408
0.285
0.327
0.173
0.224
0.108
0.038
IIC
0.617
0.257
PICA
0.591
0.696
0.512
0.310
0.307
0.171
0.277
0.098
0.040
ConCluster
0.705
0.790
0.637
0.431
0.429
0.266
0.340
0.140
0.071
AMA-aligmment
0.652
0.837
0.572
0.368
0.708
0.244
0.285
0.627
0.065
AMA-alignment is compatible with supervised training. As shown in T it achieves strong performance across datasets, outperforming previous supervised methods such SupCon and Guited-Proto.
Supervised Extension
Table 9: Top-1 accuracy in supervised settings.
Method
ImageNet
DeepFashion
iNat
ModelNet40
Cross Entropy
77.60
72.44 44
56.86
81.31
SupCon 226
78.70
72.82
57.28
81.60
HiMulConE 66
79.14
73.21
59.40
88.46
Guided-Prone
28
76.60
72.61
57.33
83.49
AMA-alignment (ours)
79.39
74.17
59.35
89.26
G.3
Ablation Study
We conduct ablation studies on the key components of AMA-alignment in Table 10 Both multi-scale partitioning and local contrastive loss are essential; removing either results in performance drop.
Specifically, incorporating the multi-scale local contrastive loss (+ Multi-scale) improves accuracy compared with olobal-only CL, indicating that modeling representations at different semantic granularities might benefit global consistency. In contrast, if we use an adaptive weighting mechanism (+ Adaptive) but implemented via the heuristic loss given in (13), the performance degrades. Unlike the proposed local contrastive loss (12), this formulation does not restrict negatives to local regions, thereby distorting the affinity structure and degrading performance, as discussed in Remark
In contrast, our AMA-alignment employs the principled local contrastive loss in (12), which rigor- ously aligns the semantic affinity matrix Ksem and the representation affinity matrix Krep through probabilistic graph coupling. This theoretically grounded formulation ensures that both the global and local alignment objectives work coherently in synergy.
G.4 Additional Experiments
Fine-tuning pre-trained DINO. To further validate the generality of our method, we applied AMA alignment to pre-trained DINO model [5] or SwAV model 4 We fine-tuned the model for 10 epochs using our adaptive multi-scale strategy with negligible additional cost (only ~ 1.05x longer
41

---

Table 10: Ablation study on Various Datasets.
Accuracy (%)
DeepFashion
iNat
ImageNet
Global-only CL
70.2
54.0
69.5
+ Multi-scale (fixed weights)
72.3
55.9
70.2
+ Adaptive (w/o (12)
67.3
50.4
67.8
AMA-alignment (ours)
75.8
57.2
73.3
Table 11: Linear probing accuracy and normalized training time for various configurations on DeepFashion. When (m = =1]==1 our method reduces to the classical contrastive learning frame- work (18).
Configuration {m") fmn) J==1
linear probing accuracy
Normalized Time
{1, 1, (reduced to MoCo) [3, 6, 12) {5, 10, 20}
0.703
0.741
1.31
0.758
1.34
{8, 16, 24)
0.743
1.41
per epoch). This fine-tuning yields consistent improvements on both ResNet-50 and VIT-S backbones, evaluated by ImageNet linear probing and k--N classification:
Table 12: Fine-tuning DINO with AMA alignment on ImageNet.
Method
Linear probe (%)
k-NN (%)
ResNet-50 backbone
SwAV
75.3
65.7
DINO
75.3
67.5
Ours (fine-tuned)
77.4
68.3
ViT-S backbone
SWAV
73.5
66.3
DINO
77.0
74.5
Ours (fine-tuned)
77.8
74.7
The results indicate that AMA effectively enhance pre-trained representations with minimal computational overhead, improving both linear and non-parametric metrics.
Results with VIT-B/16 backbone. We also evaluated our method using the VÝT-B/16 transformer backbone under the same training settings. Table [3 summarizes the results across four datasets, showing that the proposed adaptive multi-scale alignment consistently boosts performance across both natural and geometric domains.
Results on BI-RADS classification. We also extended our evaluation to the BI-RADS mammogra- phy dataset to assess the effectiveness of AMA medical imaging context. As shown in Table 14 our method achieves the highest accuracy across all four BI-RADS categories, demonstrating im- proved discriminative ability in tine-grained and imbalanced medical image scenarios.
These results further demonstrate the robustness and versatility of the proposed alignment strategy across architectures, domains, and scales.
Discussion on Semantic Consistency of Local Regions The premise of our method is that samples within local neighborhoods the embedding space share semantic properties. This assumption well-supported by recent literature. Several works [58 30 of this phenomenon. Rather than merely reporting numerical metries, these studies employ techniques such t-SNE or kernel density estimation reveal the latent structure of learned representations.
19
50
5 9 have provided visual evidence
42

---

Table 13: Comparison using ViT-B/16 backbone on multiple datasets (%).
Method
DeepFashion
ModelNet
iNaturalist
ImageNet
SimCLR
72.1
78.8
56.7
71.8
MoCo-v2
72.5
79.1
57.9
70.4
HardNeg
72.8
79.5
58.4
72.0
a-CL
73.4
79.2
58.8
72.6
HyperbolieCI.
75.2
79.5
58.1
72.9
AMM--algnmment (ours)
77.2
79.5
60.1
74.2
Table 14: Comparison on BI-RADS dataset (%)
Method
BI-RADS 1 BI-RADS 2 BI-RADS
BI-RADS
SimCLR
83.5
75.0
70.7
61.5
MoCo-v2
84.7
76.1
71.2
62.3
a-CL
84.5
76.5
74.5
63.5
SupCon
83.5
80.9
75.5
64.1
HyperbolicCI.
88.1
81.2
70.4
60.1
AMA-alignment (ours)
89.8
84.7
78.4
66.4
In particular, 300 demonstrates evolution of instance features across the training process on ImageNet- 10 over time. The t-SNE visualizations presented therein qualitatively confirm that the representations naturally converge into semantically coherent groups. This empirical evidence supports that the local regions defined in framework (Rt)) are semantically coherent.
G.S Limitation
The multi-scale and adaptive design introduces additional computational complexity compared to standard contrastive learning methods, leading to training time approximately 1.3x-1.4x that of the baseline methods. addition, the method introduces several additional hyperparameters, which may require tuning when adapting to significantly different data distributions.
43


Reviews:
[{"review_id": "Hw1LoUurV0", "reviewer": "Reviewer_iybL", "summary": "The work presents AMA-alignment, a method that evaluates the contrastive objective at multiple scales adaptively using a min-max objective, maximizing over a semantic region specific objective, ensuring that the most challenging regions are addressed adaptively. The motivation stems from coupling of semantic and representation affinities in order to minimize the divergence between the two at varying scales. Subsequently, local sub-graphs are sampled and incorporated into the main objective with theoretical motivation.", "questions": "* Was the network pre-trained over 200 epochs and subsequently fine-tuned using the method?\n* Could the authors present visual illustrations of the semantic relevance of the samples in regions?\n* How was the region radius determined in practice?\n* Might the method be applicable to text-image contrastive learning settings?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Stregths\n* The work is adequately rigorous in theory and the motivations are well founded. \n* The framework with the alignment between semantic and representational matrices is novel.\n* Empirical evaluation shows promise.\n\n## Weaknesses\n* The assumption that the sampled sub-regions (especially in the unsupervised setting) may be semantically related may require empirical evidence.\n* The method may be analogous to hard-negative mining wherein samples semantically similar to the query are chosen from a pre-trained network using similarity based thresholds which contain inherent drawbacks which may be extensible to this method.\n* There may be drastic shifts in the regions if the fine-tuning is conducted over a different data distribution and the initial region specific objective may hinder the performance of the network depending on the strength of pre-training.\n* Computational overhead isn't discussed.\n* Certain implementation details are missing ($\\rho$ and $\\eta_{\\textbf{q}}$) and the robustness to these parameters isn't discussed. The work could benefit from an ablation over these parameters. How many samples per scale were used in practice and what criteria were used to determine \"coarse\" vs \"fine\" scales?\n* Other ablations over the number of scales and choice of radius are missing.\n* The work may be akin to multi-crop augmentation strategy [1] wherein the smaller crops are embedded closely with the main image. If so, is the procedural and computational overhead justified? Might the comparisons be fairer against [1, 2]?\n\n[1] Caron, Mathilde, et al. \"Unsupervised learning of visual features by contrasting cluster assignments.\" Advances in neural information processing systems 33 (2020): 9912-9924.\n\n[2] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "was_revised": true, "final_justification": "Thank you for the opportunity to review this work. The work seems performant and well motivated. A lot of important details and specifics of implementation and parameters at times seem a bit lost within the manuscript and are often clarified in the review phase. Additionally, I am uncertain as to why one cannot simply apply the multi-crop strategy which seems simpler, to an existing work and garner some performance boost instead of the proposed method. A 1-to-1 comparison with that strategy is a comparison that the authors haven't provided, and have instead insisted upon finetuning over the existing checkpoints of the method and it is unclear to me if the multi-crop method method (DINO) when finetuned for the same number of epochs (10 in the response) for a fairer comparison may be better/worse. The baselines are also dated and miss some of the more recent methods. Moreover, the computational overhead is significantly higher with scales of over 1.4x as reported by the authors in this discussion. The work could be potentially accepted for its method based contributions but I am unclear if it may be adapted as an alternative to existing methods. I shall therefore retain my rating.", "rebuttal": "We thank the reviewer for the thoughtful comments, and address your concerns below.\n\n**Responses to Weaknesses**\n\n**W1:** The assumption that the sampled sub-regions (especially in the unsupervised setting) may be semantically related may require empirical evidence. \n\nOur region construction is grounded in the geometric properties of contrastive learning. Prior works have shown that CL naturally organizes semantically similar samples into clusters on the unit hypersphere [3]. Our local region exploits this geometric structure. Samples within local regions in the embedding space $\\mathcal{Z}$ tend to share semantic properties due to the alignment objective of CL. \nRecent theoretical studies further support this, demonstrating that CL is functionally equivalent to Stochastic Neighbor Embedding (SNE) [4] and can be interpreted as performing spectral clustering on the data's similarity graph [5]. Both SNE and spectral clustering are well-established for discovering semantically coherent local neighborhoods. Supporting experimental and visual evidence can also be found in these studies. \n\nMoreover, the quality of these regions improves with training. \nFor instance, on CIFAR-100, superclass purity within local regions improves from 18.3\\% to 85.7\\% at 100 epochs. \nTo further support this claim, we will include an additional analysis in the revised manuscript. This will include a quantitative region purity metric, demonstrating that our sampled regions become increasingly semantically homogeneous over time, as well as t-SNE visualizations that qualitatively confirm our regions align with ground-truth class clusters. \n\n**W2:** The method may be analogous to hard-negative mining... which contain inherent drawbacks which may be extensible to this method.\n\nWe would like to clarify the fundamental differences in the objectives and mechanisms between hard-negative mining (HNM) and our method (AMA-alignment), also highlighting how our approach avoids the inherent drawbacks of HNM.  \nHNM relies on accurately distinguishing between hard negatives and positives at the **sample level**. This distinction is often unclear or noisy, especially in fine-grained settings, and can lead to the well-known ``false negative'' problem, pushing apart semantically similar samples and harming representation quality. \n\nIn contrast, AMA-alignment operates at the **region level**, with the goal of promoting local alignment. The local loss encourages the representation affinities $K_{\\text{rep}}^s$ to match the semantic affinities $K_{\\text{sem}}^s$, preserving the fine-grained relationships among all samples in the region. \nMoreover, this local alignment is performed across multiple scales simultaneously, with the coarsest scale naturally recovering standard global CL. In addition, the adaptive optimization over region weights $q_{s,r}$ is stabilized by the KL-divergence regularizer, which prevents the training dynamics from collapsing onto a single region and ensures a smooth allocation of attention. \n\nWe provide a detailed analysis of the training dynamics in Appendix E, showing that HNM and AMA are orthogonal in nature: HNM adjusts pairwise weights $\\alpha_{ik}$, while AMA introduces a higher-level adaptation by adjusting region-level weights $q_{s,r}$. This makes our approach fundamentally different and sidesteps the need to distinguish hard negatives from positives. \n\n**W3:** ... drastic shifts in the regions if the fine-tuning ...\n\nIn our setting, the goal is to align the  affinity matrices $K^{\\text{sem}}_X$ and $K^{\\text{rep}}_X$; as the dataset X changes, the alignment objective changes accordingly, regardless of the specific data distribution. Pre-training encourages semantic similarity to manifest as geometric proximity in the embedding space. This alignment is further refined during fine-tuning on the target dataset $X'$, allowing the model to adapt effectively to the downstream task.\n\nAlso, the regions in our framework are not static after pre-training, but are dynamically allocated during fine-tuning. Our geometric partitioning operates on the current state of the encoder's embedding space. As the model fine-tunes on a new data distribution $X_B$, the encoder adapts to the embedding space that reflects the structure of $X'$. Consequently, when our algorithm recalculates the regions, they are defined based on the model's current understanding of the new data, not the old one in previous training.\n\nTherefore, our region-specific objective does not hinder performance;  on the contrary, instead of being constrained by an outdated structure from a source domain, the model continuously identifies and targets the most poorly aligned or semantically ambiguous neighborhoods within the new data distribution. \n\n**W4-W6:** \n    \\textit{Computational overhead isn't discussed. Certain implementation details are missing ($\\eta_q$ and $\\rho$) and the robustness to these parameters isn't discussed... Other ablations over the number of scales and choice of radius are missing.}\n\nThe main computational overhead arises from iterating over $S$ scales, with overall training time increasing approximately linearly in the number of scales. Ablation results on the number of scales and regions are presented in Table 7 of the Supplement. For instance, using $S = 3$ scales—where the finest scale contains $m^s = 24$ regions—the training time is approximately 1.41× that of the standard method. \n\n**W7:** \n    \\textit{The work may be akin to multi-crop augmentation strategy [1]... Is the procedural and computational overhead justified? Might the comparisons be fairer against [1, 2]?}\n\nThe core distinction is that multi-crop (used in [1, 2]) is a data augmentation strategy, whereas our AMA-alignment is an optimization strategy. Multi-crop enriches the input signal by creating diverse views (global crop and local crop) of an image, encouraging the model to learn view-invariant representations.  \nIn contrast, AMA-alignment refines how the model learns. It operates in the embedding space by dynamically identifying and prioritizing poorly-aligned regions. This ensures that optimization is focused on where it is most needed.\n\nThese two approaches are orthogonal and potentially synergistic. Our AMA-alignment framework can be applied on top of any augmentation pipeline, including multi-crop. The computational overhead of our method is justified because it is tackling a distinct challenge that augmentation alone does not consider: the adaptive, online refinement of semantically challenging regions in the embedding space. \n\nWe benchmarked AMA-alignment against several baselines (e.g., HardNeg) to demonstrate the effectiveness of our contribution. We agree that combining AMA-alignment with a strong multi-crop baseline is a very promising direction. \n\n\\subsection*{Responses to Questions}\n\n**Q1:** Was the network pre-trained over 200 epochs and subsequently fine-tuned using the method? \n    \nThe total training epoches is $200$. Our model is first trained by standard CL (global alignment) for the first $N_0=100$ epochs to obtain a reasonable base representation. Then the model is trained for the remaining epochs using our objective (14). \n\n**Q2:** Could the authors present visual illustrations of the semantic relevance of the samples in regions?\n    \nWe will add a new section in the Appendix with visualizations as discussed in response to W1. \n\n**Q3:** How was the region radius determined in practice? \n    \nIn experiments, the region radius $\\rho^s$ is not a manually-tuned hyperparameter. Instead, we specify the desired number of regions, $m^s$, for each scale, and the regions themselves are formed via clustering based on the data's current embedding structure. This design allows us to define the number of regions while avoiding manual tuning of the radius. Also, specifying the number of regions is generally more intuitive. \n\nTable 7 in the Supplement summarizes the impact of different configurations. We observe that increasing the number of regions and scales generally improves performance up to a point. The best linear probing accuracy ($0.758$) is achieved with the configuration $\\{5, 10, 20\\}$, while the training time remains reasonable at 1.34× the baseline. These results demonstrate that the method scales effectively and benefits from hierarchical region partitioning. \n\n\n**Q4:** Might the method be applicable to text-image contrastive learning settings? \n\nOur method is applicable to text-image contrastive learning settings.  \n\nWe evaluate the zero-shot classification performance of CLIP with a ResNet-50 image encoder on the CIFAR100 dataset, pretrained on the Conceptual Captions 3M (CC3M) image–caption dataset.  We consider two settings: in the linear setting, only the classifier is trained; in the full setting, both the backbone and classifier are fine-tuned jointly for 10 epochs. \n\n|        | InfoNCE | HNS  | Triplet [8] | CyCLIP [7] | Entropic OT [6] | Ours |\n| ------ | ------- | ---- | ----------- | ---------- | --------------- | ---- |\n| Linear | 45.4    | 46.8 | 45.5        | 31.3       | 46.4            | 47.3 |\n| Full   | 68.0    | 69.2 | 65.6        | 70.1       | 67.9            | 71.6 |\n\n\n\n[1] Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS'20\n\n[2] Emerging properties in self-supervised vision transformers. ICCV'21\n\n[3] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML 2020. \n\n[4] Your Contrastive Learning is Secretly Doing Stochastic Neighbor Embedding. ICLR 2023. \n\n[5] Contrastive Learning is Spectral Clustering on Similarity Graph. ICLR 2024.\n\n[6] Shi et al, OT-CLIP: Understanding and Generalizing CLIP via Optimal Transport. ICML'24\n\n[7] Goel et al, Cyclip: Cyclic contrastive language-image pretraining. NeurIPS'22\n\n[8] Patel et al, TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives. NeurIPS'24"}, {"review_id": "ffXWxv5X4d", "reviewer": "Reviewer_zL25", "summary": "This paper tackles the problem of hierarchical contrastive learning. It introduces localized contrastive objectives and a dynamic multi-scale optimization strategy to adaptively identify and refine poorly aligned regions within the embedding space. Theoretical analysis of convergence is provided. Results on five datasets verify the effectiveness of the method.", "questions": "see weaknesses", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n+ Incorporating hierarchical structure into contrastive learning is an interesting topic.\n+ The proposed method is theoretically proven to converge at a rate comparable to standard smooth non-convex methods.\n+ The method shows promising performance on five datasets.\n\n### Weaknesses\n- The method is evaluated only with the ResNet-50 backbone. Its generalizability to other architectures such as ViT is not studied. \n- The ablation study presented in the main paper is limited and lacks sufficient detail. In particular, it seems that the inclusion of Equation (13) leads to performance degradation. This is not consistent with the statement \"local contrast loss is essential\".\n- The method is only validated on hierarchical image classification tasks. Its applicability to other structured prediction tasks, such as hierarchical semantic segmentation [ref1], should be studied. \n- The batch size used in the experiments is not reported. How does the number of batch size affect learning performance ?\n\n[ref1] Deep Hierarchical Semantic Segmentation. CVPR 2022.", "was_revised": true, "final_justification": "The paper introduces localized contrastive objectives and a dynamic multi-scale optimization strategy to adaptively identify and refine poorly aligned regions within the embedding space for hierarchical contrastive learning. Theoretical analysis of convergence is provided. Results on five datasets verify the effectiveness of the method. The rebuttal addressed most major concerns. I maintain the initial score of borderline accept", "rebuttal": "We thank the reviewer for the thoughtful comments, and address your concerns below.\n\n**Responses to Weaknesses and Questions**\n\n\n**W1**: The method is evaluated only with the ResNet-50 backbone. Its generalizability to other architectures such as ViT is not studied. \n\nWhile ResNet-50 serves as the default visual encoder in our paper, we have also conducted additional experiments using ViT-B/16 under the same settings. \n\n| Method       | DeepFashion | ModelNet | iNat     | ImageNet |\n| ------------ | ----------- | -------- | -------- | -------- |\n| SimCLR       | 72.1        | 78.8     | 56.7     | 71.8     |\n| MoCo-v2      | 72.5        | 79.1     | 57.9     | 70.4     |\n| HardNeg      | 72.8        | **79.5** | 58.4     | 72.0     |\n| α-CL         | 73.4        | 79.2     | 58.8     | 72.6     |\n| HyperbolicCL | 75.2        | 79.5     | 58.1     | 72.9     |\n| **Ours**     | **77.2**    | **79.5** | **60.1** | **74.2** |\n\n\n\n| Method       | BI-RADS 1 | BI-RADS 2 | BI-RADS 3 | BI-RADS 4 |\n| ------------ | --------- | --------- | --------- | --------- |\n| SimCLR       | 0.835     | 0.750     | 0.707     | 0.615     |\n| MoCo-v2      | 0.847     | 0.761     | 0.712     | 0.623     |\n| α-CL         | 0.845     | 0.765     | 0.745     | 0.635     |\n| SupCon       | 0.835     | 0.809     | 0.755     | 0.641     |\n| HyperbolicCL | 0.881     | 0.812     | 0.704     | 0.601     |\n| **Ours**     | **0.898** | **0.847** | **0.784** | **0.664** |\n\n\n\nOur method's core mechanism—adaptive multi-scale alignment in the embedding space—is designed to be architecture-agnostic. It operates on the output representations, not the internal workings of the backbone. Therefore, it is equally applicable to ViTs. \nWe include several experimental results using the ViT architecture. While a comprehensive evaluation on ViT is beyond the scope of the rebuttal period due to computational constraints, we will address this limitation in the revised manuscript.  \n\n\n**W2**: The ablation study presented in the main paper is limited and lacks sufficient detail. In particular, it seems that the inclusion of Equation (13) leads to performance degradation. This is not consistent with the statement \"local contrast loss is essential\".\n\nThe performance degradation you observed is, in fact, an anticipated result that supports our claims. The \"+ Adaptive (using (13))\" row in Table 3 does not use our proposed local contrastive loss (12). Instead, it is a important ablation study using a straightforward local loss formulation, as discussed in Remark 1 (lines 209-215). \n We included this ablation to demonstrate that a straightforward, heuristic approach to local contrastive learning fails, thereby highlighting the necessity of our principled local affinity alignment in loss (12). \nOur work recasts CL as the alignment between a semantic affinity matrix $K^{sem}$ and a representation affinity matrix $K^{rep}$, a process formally modeled by Probabilistic Graph Coupling in Sec 2.2. Crucially, our proposed local contrastive loss in (12) is rigorously derived from this principle, but applied at a finer scale. This ensures that both the global and local objectives are theoretically coherent and work in synergy. \n\nBy contrast, the loss in (13) is a heuristic that breaks this alignment principle. Its key flaw lies in the denominator of the loss function, which incorrectly includes all negatives from the entire batch, rather than restricting them to the local subgraph. As we caution in Remark 1, this formulation distorts the global structure by pushing all representations outside the local region ($z' \\notin R^s_r$) away from those within it ($z \\in R^s_r$). This theoretical misalignment manifests as poor empirical performance (2.9\\% drop on DeepFashion compared to standard CL). \n\n**W3**: The method is only validated on hierarchical image classification tasks. Its applicability to other structured prediction tasks, such as hierarchical semantic segmentation [ref1], should be studied. \n\nWe thank the reviewer for this suggestion, which highlights the potential of our framework. While extending our method to the tasks like hierarchical semantic segmentation is promising, it lies beyond the current scope for several reasons. Our primary goal was to establish the theoretical foundation and validate the effectiveness of adaptive multi-scale alignment in CL. Applying it to dense prediction such as segmentation  would require adaptations (e.g., defining regions over pixel embeddings, ensuring spatial consistency) that constitute a research project in their own right.\n\nWe believe our framework may provide inspiration for such extensions. The adaptive region discovery mechanism and multi-scale alignment are general principles that can be adapted to structured prediction tasks. We plan to explore these applications in future work, where we can dedicate sufficient attention to the unique challenges and requirements of dense prediction scenarios. \n\n\n**W4**: The batch size used in the experiments is not reported. How does the number of batch size affect learning performance? \n\nFollowing the standard MoCo v2 configuration, we use a batch size of 256 across all experiments. While larger batch sizes (e.g., 512) provide marginal gains on some datasets, the improvements are not significant. It is also worth noting that the number of negative samples is crucial for contrastive learning methods. To address this, we leverage MoCo’s momentum-updated queue mechanism (queue size 65,536) to maintain a large pool of negative samples, which is essential for our region-based contrastive learning framework. This queue-based strategy ensures that even with relatively small batch sizes, we have sufficient negative samples for each region."}, {"review_id": "AIYM3SnAx0", "reviewer": "Reviewer_eVoL", "summary": "This paper addresses a key limitation in contrastive self-supervised learning (CL): while effective at capturing broad categorical distinctions, standard CL methods often fail to preserve fine-grained and hierarchical semantic relationships inherent in real-world data. The authors propose Adaptive Multi-scale Affinity alignment (AMA-alignment), which introduces two core innovations: (1) localized contrastive objectives that align semantic structures within adaptively defined regions across multiple scales, enabling hierarchical representation learning; and (2) a distributionally robust optimization strategy that dynamically identifies and prioritizes poorly aligned regions where representation affinity deviates significantly from semantic affinity.", "questions": "Questions\nThe comparative experiments in the current results appear somewhat outdated (e.g., data from 2020). Are there updated comparative studies based on recent relevant literature? If available, could these revised comparative analyses be provided?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n(1) The paper introduces Adaptive Multi-scale Affinity alignment (AMA-alignment), which fundamentally advances contrastive learning by incorporating hierarchical region-aware optimization. This addresses a critical gap in conventional methods that overlook fine-grained semantic relationships. The concept of local affinity graphs and the derived local contrastive loss provide a rigorous mathematical foundation for multi-scale representation learning.\n(2) Despite the complexity of the multi-scale adaptive design, the authors provide strong theoretical guarantees. Theorem D.10 establishes that the convergence rate remains comparable to standard smooth non-convex optimization, which significantly bolsters the method’s credibility. The equivalence between the local contrastive loss and probabilistic graph alignment further solidifies the theoretical contributions.\nWeaknesses:\n(1) Dated baseline selection disproportionately relies on legacy methods rather than contemporary state-of-the-art approaches across both supervised and unsupervised paradigms. While demonstrating gains over established references, this creates an incomplete competitive landscape that fails to reflect current representation learning advancements.", "was_revised": true, "final_justification": "The authors have addressed my primary concerns regarding baseline selection (W1) and experimental currency (Q1). Their clarification on evaluating fundamental contrastive learning settings, commitment to expanding SOTA comparisons, and provision of updated experimental details (including architecture, datasets, and protocols) resolve the core issues raised", "rebuttal": "We thank the reviewer for the thoughtful comments, and address your concerns below.\n\n**Response to Weaknesses and Questions**\n\nW1: Dated baseline selection disproportionately relies on legacy methods rather than contemporary state-of-the-art approaches... this creates an incomplete competitive landscape that fails to reflect current representation learning advancements. \n\nQ1: The comparative experiments in the current results appear somewhat outdated (e.g., data from 2020). Are there updated comparative studies based on recent relevant literature? If available, could these revised comparative analyses be provided?\n\nWe primarily evaluated our method in the most fundamental setting—standard contrastive learning in Euclidean embedding spaces using classic data augmentations and without hierarchical labels. Many hierarchical methods or hyperbolic methods rely on additional supervision (e.g., hierarchical labels) or specialized hierarchical augmentations, which could obscure the impact of our core contribution. \n\nOur approach is orthogonal to these methods—it addresses a foundational limitation that persists across CL variants: the tendency of global contrastive objectives to overlook regions with poor local alignment. This issue arises regardless of the embedding geometry, supervision level, or augmentation strategy. \n\nWe will expand our main results tables to include other SOTA methods. \n\nSpecifically, we evaluate the zero-shot classification performance of CLIP with a ResNet-50 image encoder on the CIFAR100 dataset, pretrained on the Conceptual Captions 3M (CC3M) image–caption dataset. A single-layer linear classifier is attached to the final embedding layer of the image encoder for evaluation. We consider two settings: in the linear setting, only the classifier is trained; in the full setting, both the backbone and classifier are fine-tuned jointly for 10 epochs. \n\n|        | InfoNCE | HNS  | Triplet [3] | CyCLIP [2] | Entropic OT [1] | Ours |\n| ------ | ------- | ---- | ----------- | ---------- | --------------- | ---- |\n| Linear | 45.4    | 46.8 | 45.5        | 31.3       | 46.4            | 47.3 |\n| Full   | 68.0    | 69.2 | 65.6        | 70.1       | 67.9            | 71.6 |\n\n\n\nWhile ResNet-50 is used as the default visual encoder in our paper, we have additionally conducted experiments with ViT-B/16 under the same settings. Besides, we add the HyperbolicCL [4], which also conduct alignment via contrastive learning but embed data to a hyperbolic space. \n\n| Method           | DeepFashion | ModelNet | iNat     | ImageNet |\n| ---------------- | ----------- | -------- | -------- | -------- |\n| SimCLR           | 72.1        | 78.8     | 56.7     | 71.8     |\n| MoCo-v2          | 72.5        | 79.1     | 57.9     | 70.4     |\n| HardNeg          | 72.8        | **79.5** | 58.4     | 72.0     |\n| α-CL             | 73.4        | 79.2     | 58.8     | 72.6     |\n| HyperbolicCL [4] | 75.2        | 79.5     | 58.1     | 72.9     |\n| **Ours**         | **77.2**    | **79.5** | **60.1** | **74.2** |\n\n\n\n| Method           | BI-RADS 1 | BI-RADS 2 | BI-RADS 3 | BI-RADS 4 |\n| ---------------- | --------- | --------- | --------- | --------- |\n| SimCLR           | 0.835     | 0.750     | 0.707     | 0.615     |\n| MoCo-v2          | 0.847     | 0.761     | 0.712     | 0.623     |\n| α-CL             | 0.845     | 0.765     | 0.745     | 0.635     |\n| SupCon           | 0.835     | 0.809     | 0.755     | 0.641     |\n| HyperbolicCL [4] | 0.881     | 0.812     | 0.704     | 0.601     |\n| **Ours**         | **0.898** | **0.847** | **0.784** | **0.664** |\n\n\n\n[1] Shi et al, OT-CLIP: Understanding and Generalizing CLIP via Optimal Transport. ICML'24\n\n[2] Goel et al, Cyclip: Cyclic contrastive language-image pretraining. NeurIPS'22\n\n[3] Patel et al, TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives. NeurIPS'24\n\n[4] Ge et al, Hyperbolic Contrastive Learning for Visual Representations beyond Objects. CVPR'23"}, {"review_id": "0s0ulhPg3K", "reviewer": "Reviewer_VMwq", "summary": "This paper introduces Adaptive Multi-scale Affinity Alignment (AMA-alignment), an innovative contrastive learning framework that significantly enhances representation learning by dynamically integrating hierarchical semantic structures through a novel three-level optimization strategy. The method uniquely combines local affinity alignment within semantically coherent regions, multi-scale decomposition of the embedding space, and adaptive weighting of challenging regions, achieving state-of-the-art performance while maintaining theoretical guarantees comparable to standard contrastive approaches. Through extensive experiments, AMA-alignment demonstrates superior capability in preserving both global categorical distinctions and fine-grained semantic relationships, outperforming existing methods by over 5% on challenging fine-grained classification tasks while remaining computationally efficient during inference. The framework's robustness to noise and fragmented graph structures, along with its compatibility with various graph neural architectures, establishes it as a versatile solution for complex graph representation learning, though its performance remains sensitive to region radius parameters and currently focuses on static graph scenarios.", "questions": "The paper claims AMA-alignment achieves comparable convergence to standard non-convex optimization (Theorem 3.3), yet introduces multi-scale adaptive complexity. How does the convergence rate scale with the number of scales and regions?\n\nWhile AMA-alignment improves fine-grained classification (Table 1), does the local affinity alignment (Eq. 12) risk overfitting to region-specific noise? How does the method balance local semantic preservation with global consistency, especially for ambiguous samples near region boundaries?\n\nThe comparisons focus on contrastive methods (SimCLR, MoCo) and graph techniques. Why not include hierarchical contrastive approaches or hyperbolic embeddings? \n\nThe performance relies on key parameters: region radius, KL penalty, and temperature. How sensitive are results to these choices? Does the adaptive weighting (Eq. 17) mitigate sensitivity, or does it introduce new hyperparameters needing tuning?", "limitations": "Please see Limitations in the Appendix and Supplementary materials.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\nThe manuscript is highly readable.\n\nTheoretical strength is relatively strong.\n\nThe experiments are relatively rich and sufficient.\n\nWeaknesses：\n\nIt lacks comparison with algorithms from the past three years.\n\nIn Table 3, why does adaptive learning perform worse than fixed weights?\n\nHow is the convergence of Equation (15) guaranteed?", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments, and address your concerns below.\n\n**Responses to Weaknesses**\n\n**W1**: It lacks comparison with algorithms from the past three years. \n\nWe primarily evaluated our method in the most fundamental setting—standard contrastive learning in Euclidean embedding spaces using classic data augmentations and without hierarchical labels. Many hierarchical methods rely on additional supervision (e.g., hierarchical labels) or specialized hierarchical augmentations, which could obscure the impact of our core contribution. \n\nCrucially, our approach is orthogonal and complementary to many of these SOTA methods. For instance, a hyperbolic CL method could still suffer from poor local alignment within its hyperbolic space. Our framework could be integrated to enhance such methods by ensuring both global geometric consistency and fine-grained local alignment. \n\nWe will expand our main results tables to include other SOTA methods. \n\n\n**W2**: In Table 3, why does adaptive learning perform worse than fixed weights? \n\nThe performance degradation you observed is, in fact, an anticipated result that supports our claims. The \"+ Adaptive (using (13))\" row in Table 3 does not incorporate our proposed local contrastive loss (12). Instead, it is a crucial ablation study using a straightforward local loss formulation, as discussed in Remark 1 (lines 209-215). \n We included this ablation to demonstrate that a straightforward, heuristic approach to local contrastive learning fails, thereby highlighting the necessity of our principled local affinity alignment in (12). \nOur work recasts CL as the alignment between a semantic affinity matrix $K^{sem}$ and a representation affinity matrix $K^{rep}$, a process formally modeled by Probabilistic Graph Coupling in Sec 2.2. Crucially, our proposed local contrastive loss in (12) is rigorously derived from this principle, but applied at a finer scale. This ensures that both global and local objectives are theoretically coherent and work in synergy. \n\nBy contrast, the loss in (13) is a heuristic that is inconsistent with this alignment principle. Its key issue lies in the denominator of the loss function, which incorrectly includes all negatives from the entire batch, rather than restricting them to the local subgraph. As we caution in Remark 1, this formulation distorts the global structure by pushing all representations outside the local region ($z' \\notin R^s_r$) away from those within it ($z \\in R^s_r$). This theoretical misalignment manifests as poor empirical performance (2.9\\% drop on DeepFashion compared to standard CL). \n\n\n\n**W3**: How is the convergence of Equation (15) guaranteed? \n\nThe convergence of Equation (15) is established through our theoretical analysis in Appendix D, where we provide formal guarantees under standard assumptions for non-convex stochastic optimization. Despite the nested min-max structure, we prove that the overall objective $F(\\theta)$ is $L_F$-smooth, with smoothness constant $L_F = S(L + \\sqrt M G^2/\\rho)$ (Lemma D.4). The KL regularization term is essential for convergence as it ensures the inner maximization over $q^s$ has a unique, closed-form solution, given by exponential weights (Proposition D.1); it prevents weight collapse to the boundaries of the simplex, which would destabilize training dynamics; and it ensures the optimal weights $q^{s,*}$ are Lipschitz continuous with respect to $\\theta$ (Lemma D.3). Our Algorithm 1 achieves the optimal $O(1/\\sqrt{T})$ convergence rate for non-convex stochastic optimization:\n$1/T \\sum_{t=1}^T E[\\| \\nabla F(\\theta_t) \\|^2] \\le C/\\sqrt T$, as formally established in Theorem D.10. The complete convergence analysis, including all technical lemmas, assumptions, and detailed proofs, is provided in Appendix D.\n\n**Responses to Questions**\n\n**Q1**: How does the convergence rate scale with the number of scales and regions? \n\nTheorem D.10 establishes a convergence rate of $O(L_F/\\sqrt{T})$, where $L_F = S \\cdot (L + \\sqrt MG^2/\\beta)$, and $M = \\max_s m^s$ is the maximum number of regions across all scales. The linear dependence on $S$ is expected since we are essentially solving $S$ coupled optimization problems simultaneously. The $\\sqrt{M}$ term reflects the additional complexity of tracking region-wise weights. \nThe experimental results (see Supplement, Table 7) with $S = 3$ scales and up to $M = 24$ regions per scale indicate good practical scalability: the total training time is only about 1.41× that of the baseline method. This demonstrates that the proposed approach remains efficient in realistic settings, incurring only modest overhead despite the increased model structure. \n\n\n\n\n\n**Q2**: Does the local affinity alignment (Eq. 12) risk overfitting to region-specific noise? How does the method balance local semantic preservation with global consistency...? \n\n\nOur method is can mitigate such concerns through several key mechanisms. First, our multi-scale framework inherently prevents overfitting to region-specific noise by simultaneously optimizing across all scales $S$, including the global scale. Notably, when $s = 0$, the local contrastive loss reduces to standard CL loss. As shown in Algorithm 1 (lines 6-10) and (16), the overall gradient is computed as $g_t = \\sum_{s=1}^S \\sum_{r=1}^{m_s} q_r^s(\\theta_t) \\nabla L^s(\\theta_t, R_r^s)$, which aggregates information across all scales and regions. This formulation enables the model to adaptively integrate multi-scale feedback, thereby enhancing robustness and reducing the risk of overfitting to any single region. Second, the local contrastive loss (12) is derived from the same probabilistic graph coupling framework as the global loss, ensuring that local and global objectives are compatible rather than competing. This theoretical foundation prevents the local alignment from distorting the global semantic structure. Third, the KL divergence regularization in (15) promotes smoother optimization and prevents the region weights from collapsing to degenerate solutions (line 249-255). Our experiments confirm that this multi-scale coordination effectively preserves both fine-grained semantics and global structure, as evidenced by improvements in both local discrimination tasks and global classification benchmarks.  \n\n\n\n**Q3**: Why not include hierarchical contrastive approaches or hyperbolic embeddings? \n\nOur method is compatible with existing hierarchical or hyperbolic frameworks. For example, as shown in Table 6 (Appendix G.3), we outperform both HiMulConE and Guided-Proto—two representative hierarchical methods—across multiple datasets. \n\nTo demonstrate its versatility, we also conducted experiments applying our method to hyperbolic contrastive learning. The performance show that our approach offers enhance contrastive learning across architecture or embedding geometry. \n\n| Method           | DeepFashion | ModelNet | iNat     | ImageNet |\n| ---------------- | ----------- | -------- | -------- | -------- |\n| HyperbolicCL [1] | 75.2        | 79.5     | 58.1     | 72.9     |\n| **Ours**         | **77.2**    | **79.5** | **60.1** | **74.2** |\n\n| Method           | BI-RADS 1 | BI-RADS 2 | BI-RADS 3 | BI-RADS 4 |\n| ---------------- | --------- | --------- | --------- | --------- |\n| HyperbolicCL [1] | 0.881     | 0.812     | 0.704     | 0.601     |\n| **Ours**         | **0.898** | **0.847** | **0.784** | **0.664** |\n\n\n\n\n**Q4**: How sensitive are results to these choices? Does the adaptive weighting (Eq. 17) mitigate sensitivity, or does it introduce new hyperparameters needing tuning? \n\nIn our experiments, we do not manually tune the region radius $\\rho^s$. Instead, we specify the desired number of regions, $m^s$, for each scale, and construct the regions via clustering based on the current embedding of the data. This design provides direct control over the number of regions while eliminating the need for radius tuning. \nThe overall performance mainly depends on the number of scales $S$ and the number of regions $m^s$ per scale $s \\in [S]$. Ablation results on both factors are reported in Table 7 of the Supplement. For instance, using $S = 3$ scales with $m^s = 20$ regions at the finest scale yields the best performance on DeepFashion. \nFor the temperature parameter $\\tau$, which is commonly used in contrastive learning, we follow the setting from MoCo v2 and fix it at $\\tau = 0.07$. \n\nThe adaptive weighting scheme in (17) indeed mitigate sensitivity to parameters like the region partition, as we discussed in line 249-255.  \nit prevents weight collapse to the boundaries of the simplex, which would destabilize training dynamics; and it ensures the optimal weights $q^{s,*}$ are Lipschitz continuous with respect to $\\theta$ (Lemma D.3). \n\n\n\n[1] Ge et al, Hyperbolic Contrastive Learning for Visual Representations beyond Objects. CVPR'23"}]

Author rebuttal:

Meta-review: The paper introduces Adaptive Multi-scale Affinity Alignment (AMA-alignment), a contrastive learning framework that incorporates hierarchical, region-aware optimization. By aligning semantic and representational affinities across scales and prioritizing poorly aligned regions with localized objectives and a min-max strategy, the method strengthens fine-grained representation learning. It is theoretically grounded with convergence guarantees comparable to standard contrastive learning and demonstrates consistent gains across benchmarks.

Reviewers praised the novelty and adequate rigor of the framework, particularly its region-level formulation, theoretical grounding, and ability to capture fine-grained semantic structure often missed by standard CL. They found the experiments promising and appreciated clarifications added during discussion: expanded results on ViT backbones, new comparisons with recent SOTA baselines (e.g., HyperbolicCL, Triplet, CyCLIP), and ablations confirming the necessity of the proposed local contrastive loss over heuristic alternatives. The rebuttal also addressed overlap with hard-negative mining, emphasizing AMA’s region-level adaptation that preserves semantic affinity without relying on high-quality initial features.

Remaining concerns centered on experimental breadth and practical clarity. Reviewers noted that initial baselines were dated, with missing sensitivity analyses for key hyperparameters (region radius, KL penalty, temperature), limited backbone diversity, and insufficient discussion of computational overhead. Some worried AMA could resemble multi-crop augmentation or HNM without clear differentiation.  The authors committed to incorporating all feedback -- adding ViT and SOTA baselines such as DINO and SwAV, more ablations, and visualizations -- into the final version. The paper received 3x borderline accepts and 1x borderline reject in final ratings.  Overall, the paper is seen as theoretically strong and innovative, with consensus leaning positive once expanded experiments and clarifications are included.
Decision: Accept (poster)