Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment
Authors: Hua Ye
Hang Ding
Siyuan Chen
Yiyang Jiang
Zhang Changyuan
Xuan Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Alignment
Curriculum Learning
Contrastive Local Attention
Primary area: deep_learning

Abstract: Most multimodal models treat every negative pair alike, ignoring the ambiguous negatives that differ from the positive by only a small detail.  We propose Boundary-A ware Curriculum with Local Attention(BACL), a lightweight add-on that turns these borderline cases into a curriculum signal.  A Boundary-aware Negative Sampler gradually raises difficulty, while a Contrastive Local Attention loss highlights where the mismatch occurs.  The two modules are fully differentiable and work with any off-the-shelf dual encoder.  Theory predicts a fast $\tilde{\mathcal{O}}(1/n)$ error rate; practice shows up to +32 \% R@1 over CLIP and new SOTA on four large-scale benchmarks, all without extra labels.

Aligning by Misaligning: Boundary-aware Curriculum Learning for Multimodal Alignment
Hua Ye! Hang Ding", Siyuan Chen', Yiyang Jiang
Changyuan Zhang Xuan Zhang
"Najjing University Airon Technology CO., LTD
Shhnghai Jiao Tong University
UUniversity of Bristol 'The Hong Kong Polytechnic University
"The University of Hong Kong 'Careggie Mellon University
Abstract
Most multimodal models treat every negative pair alike, ignoring the ambiguous negatives that differ from the positive by only small detail. We propose Boundary- A ware Curriculum with Local ABBC)) a lightweight add-on that turns these borderline cases into curriculum signal. A Boundary- Negative Sam- pler gradually raises difficulty, while a Contrastive Local Attention loss highlights where the mismatch occurs. The two modules are fully differentiable and work with any off-the-shell dual encoder. Theory predicts fast (11nn) error rate; practice shows up to +32 % R@ over CLIP and new SOTA on four large-scale benchmarks, all without extra labels.
Introduction
Cross-modal representation learning has witnessed rapid progress since CLIP Radford et al. 2021
ALIGN Jia al.. [021] and their successors demonstrated that contrastive pre-training on web-scale
image-text pairs is an effective alternative to costly human annotation [Li et al. 2025b Follow-up
models such ALBEF (Li et al.
2021], BLIP/BLIP-2 [Li et al. 2022 2023 ViLT (Kim al.
2021] and GRAM M (Cicchetti et al. 2024] further increase sample efficiency by injecting token-level
objectives or multi-modal experts Zhang al.) 2025 2024]. These models have promoted the
development of fields such as natural language processing Lin ct al. [2025]. medical diagnosis [Fang
and Liu
2025 Liu al. 2025 Tong et al. 2025 Wang et al. 2025a
Li al. 2025c autonomous
systems Yao et al. 2023
Lu et al. 2025b Xiao et al. 2025 Lu et al.
2025a
Li et al., 2025a Zeng
al.
2025a b). and other application fields [Xu al. 2025
liagg al.
2025
Tao et a.. 2023
Liao
et al.
2025 Chan et al. 20266 Sun et al.
(025bb Wang et al. [2225]].
Despite these advances, most existing pipelines share the same implicit assumption: two paired
modalities are either perfectly aligned (positive) or entirely unrelated (negative), and the learner's job
merely to shorten or enlarge their distance [Xin et al.) [2024] Fuu et al. 22024] Tan al. [025 Feng et al. 2025). In practical applications (Jiang et al.) [2024 Zhang et al. 2023]. however, cross-modal data often carries subtle mismatches: captions that paraphrase only part an image, audio tracks that overlap but differ in background context, or video-subtitle subtitle pairs where a single phrase is out of sync.
These ambiguous Seruutives""hafaaaee
half-false mismatches
abundant the web yet over- (2024).Humns,,
looked by current training regimes (Yang et al. 2024] Yao et al., 2024). Humans, by contrast, naturally learn from nuanced differences, readily using subtle mismatches informative cues [Osgood [949] Lynn and Bassett 2020) Hu et al. 2025) Sun et al., 2025a [Chen et al.) [025a]]. Enabling models similarly leverage these ambiguous negatives is thus essential for more human-like multimodal
alignment.
Equal contribution, iCorresponding author(xuanzhang2199) @wal:///mmmllooooooo... gmmil.com)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Clip-like Contrastive
cat is running"
Easy ambiguous pairs Hurd ambiguous pairs
BLIP pipeline
ambignous negutives
CLA Local loss
Global Contraxtive
about fall asleep"
Threshold check
"A cat running"
Add hard embignowr negatines
BACL pipeline
ambignous negarives
"A cat about
Curriculum Learning
fall asleep"
Figure 1: Comparison between the BLIP pipeline and our proposed BACL pipeline. Methods like BLIP eliminate ambiguous negatives through threshold filtering without explicitly leveraging their intrinsic value. In contrast, BACL employs a curriculum learning strategy to progressively introduce more challenging ambiguous negative samples, explicitly revealing the sources of confusion. This approach enhances discriminative capability by jointly optimizing the global contrastive loss and the Contrastive Local Attention (CLA) loss.
Gap current practice. Existing alignment methods inadequately handle ambiguous negatives. First, mainstream dual-encoder approaches (e.g., CLIP [Radford et al.) 2021]. ALIGN (Jia et al. 2021]. MIL-NCE [Miech et al. 2020b]) sample negatives uniformly, treating obvious mismatches and subtly incorrect captions equally. Second, recent token-aware methods (e.g.. ALBEF [Li et al. 2021], BLIP et al. 2022)) ) discard ambiguous captions through filtering or pseudo-labeling losing valuable instructional signals, shown in Fig []] Finally, existing approaches rely exclusively static datasets and static loss functions, neglecting dynamically generated, structurally plausible yet semantically ambiguous mismatches Yaa et al.
2024 Xiao and Liu
2025
Consequently. these
methods optimize alignment under idealized conditions, overlooking rich supervisory signals inherent
in realistic, partially incorrect data [Wang and Zhang
2024 Lu al. 2025e
Ye al. 2026
Our view. We argue that ambiguous negatives distinguishing almost-correct pairs from truly correct ones is precisely what a robust multimodal model must master when data are scarce noisy. However, directly training cases from the beginning leads to unstable optimisation [Dufumier et al., 2024]. The key, therefore, schedule the exposure of the learner increasingly confusing negatives while simultaneously revealing where the confusion arises.
merely noise, but a rich supervisory signal:
these borderline
Our solution. This paper introduces BACL: Boundary- Curriculum with Local Attention. BACL augments any dual-encoder or MoE aligner with two lightweight, fully differentiable com- ponents: (i) Boundary-aware Negative Sampler (BNS). A policy network learns to rank candidate negatives by their boundary score and, guided by a logistic schedule, gradually shifts training focus from easy to ambiguous cases. (ii) Contrastive Local Attention (CLA). For every positive pair, CLA compares its cross-attention map with that of the hardest negative and imposes a local mismatch loss that amplifies token pairs where they diverge, forcing the encoder to detect fine-grained misalignment.
Contributions. The main contributions are:
1) We identify ambiguous negatives as an under-explored yet ubiquitous phenomenon in web-scale multimodal corpora and highlight their importance for robust alignment. We propose BACL, the first framework that dynamically generates and exploits near-boundary mismatches via a curriculum sampler and token-level attention loss.
2) We provide sharp generalisation theory: under mild assumptions BACL enjoys a Ö fast rate, whereas uniform sampling suffers unavoidable 2(p/ Vn) excess risk (Theorems 4.1 4.2
3) Extensive experiments on four large-scale datasets show that BACL. improves both global retrieval and fine- fine-grained reasoning, achieving new state-of-the results several benchmarks.

---

2 Related Works
Multimodal Alignment and Hard-Negative Mining. Early dual-encoder models such as CLIP [Radford et al.) [02T]. ALIGN (Jia et al. [021] and MIL-NCE [Miech et al., 2020a learn a shared representation space by drawing uniform in-batch negatives, approach that proves scal- able but treats trivial and near-boundary mismatches alike [Dong et al. 2024 Lin et al. 2025 Subsequent work improves sample efficiency through stronger vision backbones [Jia al.
2021
region-level supervision [Chen et al. 2020]. or generative pre-text objectives (ALBEF [L et al.
202T). BLIP/BLIP-2 [Li et al. 2022 2023 Yet, these methods either filter out noisy captions
(ALBEF, BLIP) or rely on one-shot max- violation mining (VSE++ (Faghni et al. 2018 ) leaving ambiguous negatives under-exploited. Very recent systems such as GRAM [Cicchetti et al. 2024
and Emergence (Tjandrasuwita et al. [025] incorporate mixture-of cottuur-of--xpertt or multi- masking but still assume a static positive/negative split. Our work departs from this line by dynamically scheduling near- near-bounder boundary negatives and coupling them with local attention loss, thereby tightening the decision margin without expensive expert routing or additional labels.
Curriculum learning [Bengio et al. 2009 and self-paced learning [Kumar et al.) 2010, Chen et al.] 2024] propose present samples in easy-to-hard order to stabilise optimisation. In computer vision, curricula have been applied to class imbalance (Jingg 2015). structured prediction [Pentina et al. 2015). and more recently to vision-language pre-training DCOT [Wang et al. [2023], where difficulty is measured heuristically (e.g. OT distance).
Curriculum and Self- Self-Paced Learning,
Our Boundary-aware Negative Sampler differs in two aspects: (i) difficulty is defined by a learnable boundary score relative to the current decision margin, and (ii) sampling remains differentiable via Gumbel Softmaxx enabling end-to-end optimisation with the underlying encoders. BACL constitutes the first curriculum framework specifically designed to exploit "half-true, half- half-false" negatives in multimodal alignment.
3 Methods
3.1 Overview of the Proposed Method
(mage Problem Definition. Consider two arbitrary modalities X' and y (e.g., image- text, audio-video). A large paired corpus is denoted by D = {(z,, y: with modality specific encoders Φx and dy that map inputs to shared unit sphere gd-1 For a ground-truth pair (x, we measure similarity by s(x, y) (6x (xx(x), (y())) A non-matched sample is called an ambiguous negative if
s(z,2) s(x, v) S, with 0 D<E<> 1,
(1)
ie., it lies near the decision boundary and is difficult to reject. Standard alignment strategies treat all negatives uniformly and thus fail to exploit this hardest subset.
Core Idea. We explicitly expose the model to such boundary cases through a boundary-aware cur- riculum: 1) Boundary-aware Negative Sampler (BNS) progresses from easy ambiguous negatives. A scheduling coefficient a(1)). monotonically increasing with epoch r7. controls the hardness of the sampled set, thereby shrinking the margin around every positive pair. 2) Contrastive Local Attention (CLA) contrasts the attention patterns of a positive pair with those its hardest negative counterpart selected by BNS. The resulting local mismatch loss encourages the model to pinpoint fine-grained misalignment cues rather than relying solely on global similarity.
By iteratively tightening the boundary (via BNS) and highlighting where misalignment occurs (via CLA), the framework learns finer-grained and more robust cross-modal representations.
3.2 Boundary-aware Negative Sampler
In latent space, negatives that closely resemble a positive pair are the hardest to reject. We introduce learnable Boundary-aware Negative Sampler (BNS) no that schedules negatives from easy hard, steadily shifting training focus toward these ambiguous cases and, in turn, sharpening the decision boundary.
Boundary Score and Candidate Negatives First, using an initially trained coarse alignment model, encode all images I and texts T in the training set into embeddings z(z) and Z(T), respectively,
3

---

and build corresponding image and text indices. For any given positive pair (I. we retrieve texts {T") from the text index that are most similar to (()) yet do not form true positive pairs. Similarly, we retrieve images { from the image index that closely match Z(T) but are not true matches, We collectively refer to these retrieved samples as ambiguous negative samples, and denote their embeddings as {
To quantify how challenging each candidate negative is with respect to the model's decision boundary, we define boundary score function:
BS( 2(r))
((T)) sim (2(). Z(r)):
(2)
where sim(,.) .) denotes cosine similarity. A large positive boundary score indicates that the negative sample T" is even closer image / than the true matching text I', presenting a substantial confusion to the model. A score close to zero implics that the negative sample is almost indistngguishall from the true pair, thus also posing significant confusion. Conversely, clearly negative score suggests the negative sample is relatively distant from / in the embedding space and poses a weaker challenge to the model. The same boundary score definition applies similarly to the retrieved pairs (I, T') or (T'.)).
Policy Network and Difficulty Scheduling We define policy network Te(:), which takes as input the embeddings of the current positive pair (z(1), z(r)) and all candidate negative samples {z,, 1 n==1+ outputting an initial scoring vector (u1...., ..N) where un quantifies the priority of selecting each negative sample. Next, we introduce the following difficulty measure:
d(z,) max max(o, 0, sim sim(z zn) sim (i)) m())
larger value of d(z..) indicates that the negative sample poses a greater confusion to the model. To balance the principle of progressing from casier to more challenging negatives, we compute the
difficulty measure d( (..) for each negative sample and adjust the initial scores accordingly using a function (n))
in = un - a<)) d(zn),
where a(n) adopts the form of a logistic function that smoothly transitions from Qearly Olae <
0 to
(n) = Cearly +
Clatc
Q'arrly
(5)
1- exppp>>(7 no))
Here, 17 denotes the training epoch, 7 > controls the steepness of the transition, and no represents the center point of the transition. At early stages of training (1) < n7)), a()) remains close to Clearty > 0. thereby suppressing highly challenging negative pairs. Conversely, at later stages (17 >> nJo), an) approaches Oatte O, thus incentivizing the sampling of more confusing negative samples.
Differentiable Sampling Mechanism operation:
The adjusted scores ún are transformed into probabilities
Pa. via the
exp (úū + 9n)/T)
Pn.
(6)
Dm exp((mm + 9m.)/))
where In. denotes Gumbel noise, T is temperature hyperparameter, and p.. approximates the probability of sampling negative sample Zn.
Upper-level Optimization Objective signal R(Z,) and define:
We treat the boundary score defined in Eq. (2) as a reward
((0) = E R(z,))
Thanks to the differentiable Gumbel-Softtax mechanism, we can directly perform backpropagation on the following expectation to update the policy network parameters 0:
N
Σ
pn R(z,).
(8)
Early on, BNS down-weights the hardest negatives so the model can master basic discrimination; as training proceeds, it gradually upsamples the most ambiguous cases, tightening the margin and yielding finer cross-modal distinctions.

---

3.3 Contrastive Local Attention
Global contrastive loss separates pairs overall but misses token-level mismatches. Our Contrastive Local Artention (CLA) compares the attention maps of a positive pair with its hardest negative, amplifying the tokens where they diverge and spotlighting fine-grained misalignments.
Attention Distributions of Positive and Negative Pairs Within the cross-modal Transformer, let the attention matrix for positive pair (1, be denoted by A (+) E RRNN,, where N M + L represents the total number of image and text tokens. If the sampler selects challenging negative pair (I,T)) (or (I', ()) corresponding to the positive pair (I,T), T), we similarly obtain the negative-part attention matrix (-), These matrices respectively reflect the differences in attention distributions across tokens between the positive and negative pair scenarios.
Difference Computation and Local Modulation When a negative pair (I, significantly differs from the corresponding positive pair at certain token pairs (i,j), these positions usually indicate potential mismatches. To amplify such differences, we define:
(i,))) = A (t.)) - A
higher value of (i. suggests a substantial discrepancy between positive and negative pairs at token pair (i. typically corresponding to regions of highest ambiguity in negative samples. To direct the model's 's attention more explicitly toward these critical regions in pegative-pair scenarios, we locally enhance the negative attention matrix A (-) follows:
A (i.)) A (i,)) [1
8 (i.j))
(10)
where 3 > 0 denotes gain coefficient that emphasizes token pairs exhibiting large attention discrepancies.
Local Mismatch Loss After obtaining the modulated attention matrix A we introduce local mismatch loss Llocal to further emphasize these mismatched regions, defined as:
Liocal
Σ (i,)i. (
(11)
where Q represents set of token pairs with the highest discrepancies identified by Eq. (9) (selected
via thresholding ranking). The function g(-) can be instantiated
log(+), thereby encouraging the model to produce a more pronounced attention enhancement at potential mismatch locations.
We combine the aforementioned local mismatch loss Llocal with the global contrastive loss Ccontrast form the final training objective:
Lmain
Lcoutrast + Alocal Llocal,
(12)
The weight Atoca trades off global contrast with token-level mismatch loss. By amplifying attention gaps between positive pair and its hardest negative, CLA trains the model to catch both pair-wise
and token-wise errors, sharpening its response to ambiguous negatives. Experiments ($5) CLA, combined with BNS, boosts accuracy and fine-grained alignment.
show that
Algorithm T in Appendix outlines our BACL training pipeline.
The sampler-attention synergy progressively exposes the encoders to increasingly ambiguous negatives while amplifying token-level mismatch cues, yielding finer cross- cross-modal decision boundaries.
Theoretical Analysis
We investigate the sample-cone complexity and optimisation behaviour of BACL. Throughout, Assump- tions 4.1 hold; all proofs are deferred to Appendix A
negative density). There exists p E (0, 1) such that for every anchor and its positive y* (1s=)((s((z,,))))) 2) s(z, b* s E) p, where 0 < m.
Assumption 4.1 (Ambiguous-negative
Assumption 4.2 (Lipschitz encoders). The encoders L-Lipschitz: (s(0x()), s(ox(x'), L(| x'll + [yy v'll) for all v.y.
5

---

Notation. Let ΦB be the model returned by Algorithm du denotes the counterpart trained with uniform negatives. The population risk is R( E[Cm 2: 6)]
Theorem 4.1 (Fast-rate Generalisation of BACL). Assume 4.1 and 4.2
Fix òE (0, margin m >e
and let dog the effective (pseudo) dimension of 4. 1f
128 L
m - + log 4)
4e
2 (1 - p)² - 2)²
dall log
(13)
then with probability at least 1 - 8
16 16 (m) -
deg m- + log
32 L2
(R(xn) - R((0))
V
(14)
- p)m
where the additional 1.2 //n term refines the classical 0(1/vT) rate fast rate whenever (1))
Theorem 4.2 (Minimax Lower Bound for Uniform Samplers), Let Assumptions 4.1 hold. Fix
pe (0, 2), << 3 and ò (0, }). For any estimator & trained with uniform negatives n triplets,
there exists a distribution P satisfying Assumptions 4.1
4.2 such that, with probability at least 28,
p(m 2c)
1 log
LLL(m - (mmmm=2E)) 22)²
R(6) R(0")
To n
(15)
32
128n
Hence any learner that ignores ambiguous negatives incurs an unavoidable sppp vn) excess risk, matching the fast rate in 4.7 up to constants.
Proposition 4.1 (Exponential Contraction of Alignment Margin). Let A, denote the expected worst-case margin after epoch A., =E[s(x, y*) max max $ 2)] Assume a constant learning
rate The batch size B, and $ 2 CLA. Define ān = B Ei a(t) and let 21-B(m-e) 0. Then for all 1j 1)
2L
A. < Do exp - (e (e 1))
(16)
Consequently, if the logistic schedule (1)) of (5) obeys Clat <cco(<00 0), then A,, =(e e(n²)) once > No. implying that a quadratic number of ambiguous epochs suffices to force the margin below any preset Etarget-
5 Experiments
5.1 Experimental Setup
Datasets We evaluate the effectiveness of the proposed BACL framework on four large-scale multi- modal datasets that naturally contain "ambiguous negatives" Inear-boundary hard negatives) across different modality pairs: (i) the LAION-400M image-text corpus (Schuhmann al. 2021]; (ii) the Web Vid-IOM 10M video-text collection [Bain et 2021]; (iii) the VAST-27M tri-modal dataset of video, audio, and subtitles [Chen et al. 2023 and (iv) the WarTex15K audio-text benchmark (Deshmukh) let al. 2022]. For detailed description
these datasets, including their statistics, licensing, and the evaluation metrics we follow, please refer to Appendix C We additionally report extended results on VQA and NLVR2 to probe fine- fine-grained reasoning: these results appear in Appendix F
Baselines We benchmark the proposed BACL against five families of
alignment
methods:
dual encoders (CLIP [Radford al.. [202]] ALIGN [Jia et al.
2021 ).
(ii) Single-shot hard-negative mining (VSE+ (Faghri et al. 2018)). UNITER [Chen et al., 2020
ALBEF [Li et al., (202]). ). (iii) Token-level enhanced pre-truining (ViLT (Kim et al. 2021], BLIP [Li
et al.
2022]. BLIP-2 [Li et al. [203)). (iv) Curriculum self-paced alignment (DCOT [Wang et al.
2023 (v) Multimodal or MoE aligners (Emergence [Tjandrasuwita al. (025), CoMM [Dufumier
et al. 2024]. M3-JEPA [Lei et al., 2024]. GRAM (Cicchetti et 2004]. CLAP [Elizalee et al. 2022
for audio, MIL-NCE [Miech et al. 2020a] for video).
Baselines are evaluated using their public checkpoints or our reproduction with recommended hyper-parametes (Appendix DD. Implementation details are in Appendix E
An estimator is any (possibly randomised) measurable mapping
V ((X x y x
J/)", {uniform neg. sample)) + #; e (ddata).

---

5.2
Main Results
Table I: Retrieval performance on (a) LATON-400M (image-text) and (b) Web Vid-10M text)
Higher better; best results are bold.
(a) LAION--4000 -00M
(b) WebVid-10M
Method
R@1
R@S
R@
nDCG
CLIP Redford at
2021
35.2
38.3
68.7
42.3
CLIP Radiord al. 2021
14.3
31.5
42.7
25.4
ALIGN NIEVE 1011
37.9
61.5
71.2
44.6
26.3
VSF++
mis
18.4
35.7
46.1
22.5
BLIP
Lrerai 2024
17.2
3688
47.5
28.0
UNITEK
ALBEF
56.1
DCOT
2023
18.0
38.1
48.4
29.1
BL 21 IE 20722
VIL.T Kiil
pi1e
42.0
67.3
64.3
76.1
492
MIL-NCE Mixel x dl 8020
M3-JEPA Let Cil 20cc
21.4
12.4
42.7
28.1
53.0
38.9
22.7
32.4
DCO1 Wing or 2023
41.1
66.4
75.3
48.7
GRAM (Clceren 2024
22.0
43.6
54.1
33,0
Emergeici 11o0ceettionn
2025
38.3
63.9
7300
458
CIJP-BACI (Ours)
19.5
35.0
51.5
31.4
CoMM Dulmnnre
MIL--NCE-BACL (Ours)
24,9
46.8
57.3
35,9
GRAM Chevnoa uea 2024
69.0
77.0
M2-JEPA-BACL (Ours)
CLIP-BACI, (Ours)
46.5
71.2
79.3
53.6
MB (OrJE
46.0
70.5
78.9
52.9
Image-text retrieval (Table 1a). On the noisy web-scale LAION-400M corpus, BACL injects +332% relative gain in @@ over vanilla CLIP and still offers <6% absolute improvement sophisticated hard-negative methods such as GRAM.
Video-text retrieval (Table WebVid 10M captions notoriously weak: many clips share nearly identical phrases. Curriculum sampling therefore has a larger impact: MIL-NCE+BACL closes half the gap between weak captions and clean text, boosting nDCG by +3 over GRAM. The steady rise in retrieval depth (R RR @ 10) indicates that BACL improves both top-ranked precision and tail recall.
Table 2: Results (c) Way (audio-text retrieval) and (d) VAST-27M (tri-modal classifica- tion).
(a) WavTextSK
VAST-27M
Method
R@1 1
R
R@9
MRR
Method
Accurucy
F1
Recall
22.4
33.2
VILT
Kim el m
2021
76.
74,0
72.4
CLAP
EIzaloN et 2022
20.8
43.6
55.2
31.4
M3-JEPA
22.7
46.5
58.1
33.6
DCOT
Maln 2
2023
75.1
73.2
71.5
GRAM
Cechetll el 2024
23.1
47.0
58.9
34.0
Emergenc Tnistranii dt 2025
74.9
72.9
71.2
CoMM
M3-JEPA-BACL (Ours)
26.0
62.4
37.2
M3-JEPA
&al P014
76.8
74.9
73.1
GRAM Checlent 20224
77.3
75.4
73.6
M3-JEPA--ACL
Audio-text retrieval (Table Za). Even without changing the frozen CLAP audio encoder, BACL secures a consistent 10% relative gain in MRR. The improvement stems from the local-attention loss that highlights mismatch cues between environmental sounds that perceptually similar exaclly-exccty the ambiguous cases our sampler targets.
Tri-modal classification (Table 2b). Finally, the challenging VAST-27M dataset (combining video, audio, and subtitles) BACL drives M3-JEPA to 79.5% Accuracy, topping all prior works. The gains in both Fl and Recall indicate that ceaiiccluu--trrinnee representations reduce false positives and discover subtle cross choss-channel inconsistencies.
5.3 Ablation Study
To quantify the individual contributions of the BNS and the CLA, we conduct controlled ablations on two representative datasets: LAION-400M and WebVid- 10M. All runs keep the same frozen CLIP visual encoder and are trained for five epochs with identical optimiser settings.
Uniform BNS (Fig, 2 Replacing uniform sampling with BNS yields substantial jump: +7.3 R@1 on LAION-4001 and +4.99 R@ on WebVid- 10M, confirming that the curriculum alone tightens the similarity margin.
Global vs. Global CLA (Fig. 2). Adding CLA on top of uniform sampling provides modest gains (+3.2 @@ for LAION, +2.4 R@ for WebVid), yet when combined with BNS the improvements
7

---

LAION 400M
WebVid
62-
30-
10
10+
Untiomm--
BACL
BACL
Figure 2: Ablation study (a) LAION-400M and (b) WebVid-1OM. Each bar group shows the effect of enabling BNS, CLA, or both (full BACL).
compound, reaching the full BACL results 000.5/71.2/555 on LAION-400M and 24.9/46.55 06////6.8//.. on WebVid-I0M), This demonstrates that local mismatch supervision and boundary-aware curriculum address complementary aspects of cross-modal alignment.
Impact of the Logistic Curriculum Schedule The BNS curriculum is governed by logistic coefficient a(1) (Eq. (5) whose shape is controlled by the initial value Qearly. the terminal value Clase and the steepness 7 We explore three representative schedules: (i) Shallow (Ceertyy :late: 7) = (0.1, -0.2, 1.0). (ii) Default (used throughout the main paper) (0.3, -0.5, 1.5) (iii) Aggres- sive (0.5, -0.8 2.5). All other hyper-parameters are fixed. 3 retrieval performance on LAION- 400M after five training epochs. The Default sched- ule-moderate initial margin and creepness-yillee the best balance, outperforming the Shallow curve by -3.3 R@I and the overly Aggressive curve by +1.5 R@ This corroborates Proposition 4.1 letting ((n)) decrease nei- ther too slowly nor too fast produces the fastest margin contraction and the highest final retrieval accuracy.
6-
Figure 3: Retrieval metrics on LAION- 400M under different logistic curriculum schedules (sean n=3).
Attention Visualisation Figure Sshuws shows (a) the positive-paii attention, (b) the BNS-selected hardest negative, and (c) their difference AA with the ten largest gaps boxed in red. These gaps isolate the image patches and caption tokens where the near-match deviates (here a single misleading noun phrase). CLA amplifies those cells, the encoder downgrades the negative though its global similarity is high-dires direct evidence of BACL's fne-ggrined discrimination 3..3
Negative
Figure 4: visualisation for randomly selected image text pair. Left: attention of
the positive pair. Middle: attention of the hardest negative (selected by BNS). Right: glement-wiss. difference AA with the ten largest discrepancies boxed in red-the regions CLA focuses

---

5.4 Hard-Negative Mining Study
To better understand how the tolerance margin E influences model reliability, we extract the k E {5, 20 nearest neighbours (by CLIP similarity) for every anchor in LAION-400M, treat them as candidate negutives, and progressively shrink the ambiguity threshold e E {0.40, 0.30, 0.20, 0.10, 0.05). For each setting we measure (i) the False Positive Rate (FPR): fraction of negatives the model wrongly ranks above the true caption, and (ii) the Recall@ 10 after fine-tuning for two epochs.
10
Rosivve 1o42
44
40-
2-
0.30
0.25
0.20
0.25
2.30
0.35
0.40
3.00
0.50
0.32
0.20
AmsiguRy threshold
Amb gully cwroshold
Figure 5: Hard-negative mining on LAION-40MM. Left: False Positive Rate decreases monotonically as the ambiguity margin e shrinks. Right: Recall@ 10 improves simultaneously. Curves compare different candidate- pool sizes k (nearest neighbours).
Observations (Fig. 5. (i) Tighter margins reduce errors. Reducing e from 0.40 0.05 cuts FPR by
x 75% across all k values, consistent with the theoretical (ii) Recall rises despite harder negatives. Recall @ 10 climbs as shrinks, showing that exposing the model to progressively harder negatives does not trade precision for coverage instead, both improve. (iii) Larger candidate pools help. Using = 20 neighbours starts with higher FPR but ends with the best Recall 49)5%) illustrating the benefit of a richer "confusion set" once the curriculum has progressed beyond the easy stage.
bound in Proposition 4.1
5.5 Cross-modal Generalisation
After pre-training the three-modal VAST-27M corpus, we freeze the encoders and evaluate them zero-shot on AudioCaps (audio text retrieval) and VATEX (video text retrieval). We log performance every epoch and compare it with the decyyn-decay prediction of Proposition [41] which states that the alignment error should contract roughly like A, x exp(-cn)²) once the curriculum enters the ambiguous-negative regime.
Slapes: audov-0000 0.52, videom 0.02 Augio
50-
4.4
40
4.3
Algimeen
50
20
4.3
30
Spoch on VAST 8
Figure 6: Cross-modal generalisation. Left: zero-shot Recall @ 10 on AudioCaps and VATEX as VAST pre-training progresses. Right: log alignment error versus n/²; the near- line-linear trend confirms the quadratic contraction predicted by Proposition 4.I
We can see from Fig. 6 that (i) Rapid early gains. Zero-shot Recall climbs steeply during the first three ppochs,indicating that anbiguous-negative exposure on VAST quicklytransfers to unseen domains. (ii) Quadratic margin contraction. The
audio-only (AudioCaps) and video-text log-error curves almost perfectly linear in 7 with slopes audio -0.28 and video -0.25.

---

matching the O(exp[-cn²)) decay rate derived in Proposition 4.1 (iii) Consistent cross- pros--domain
effect. Despite modality shift, both tasks converge similar asymptotes, suggesting that boundary tightening on VAST produces modality-agnostic decision margins.
6 Conclusion
We have presented BACI,, boundary-aware curriculum that converts the ubiquitous yet under-utilised ambiguous negatives into powerful supervisory signal for multimodal alignment. (1) schedule their difficulty and (2) enforce token-level disambiguation. The result is a tighter decision margin. provably and empirically. Future work will scale the sampler billion-image corpora [Chen et al. 20256] and extend our BACL to language-only instruction tuning. Additionally, the boundary- curriculum may also be promising for pruning large-scale multimodal models (Zhou et al.) 2025].
vorpora[Chen
Acknowledgements
We would like thank Hunan Airon Technology Co., Ltd. for providing data preprocessing services and computing resources.
References
M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1728-1766 1738, 2021.
P. L. Bartlett, O. Bousquet, and S. Mendelson. Local rademacher complexities. Annals Statistics, 197441149 2005.
Y. Bengio. J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
J. Chan, Z. Zhao, and Y.-L. Liu. Adagar: Adaptive gabor representation for dynamic scene recon- struction. arXiv preprint arXi::2601.00796 2026.
K. Chen, Fang, and H. Xue. Multi-modal interactive agent layer for few-shot universal cross- domain retrieval and beyond. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025a.
S. Chen, H. Li, Q. Wang, Z. Zhao, M. Sun, X. Zhu, and J. Liu. Vast: A citton--uddo-uuutilleeeeeo omni-modality foundation model and dataset. Advances in Neural Information Processing Systems, 36:72842-72866, 2023.
X. Chen, C. Xiao, and Y. Liu. Confusion-resistamt federated learning via diffusion-based data harmonization on non-iid data. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 137495-137520 2024.
X. Chen, C. Xiao, W. Cao, W. Zhang, and Y. Liu. Framework and pathway for the construction of a unified data-element market in china. Strategic Study of Chinese Academy Engineering, 27(1): 40-50, 2025b.
Y. Chen, L. Li, L. Yu, A. E. Kholy, F, Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Learning universal image-text representations. In European Conference on Computer Vision (ECCV), 2020.
G. Cicchetti, E. Grassucci, L. Sigillo, and D. Comminiello. Gramian multimodal representation learning and alignment. arXiv preprint arXiv:2412.11955 2024.
S. Deshmukh, B. Elizalde, and H. Wang. Audio retrieval with wavtext5k and clap training. arXiv preprint ar 2t0::22299.4777. 2022.
7. Dong, Q Long, Y. Zhou, P. Wang, 7. Zhu, X. Luo, Y. Wang, P. Wang, and Y. Zhou. Pixel: brssdt-based zero-shot hashing via visual and textual semantic alignment. In CIKM, 2024.
10

---

B. Dufumier, J. Castillo-Navarro, D. Tuia, and J. Thiran. What to align in multimodal contrastive learning? arXiv preprint arXiv:2409.074: 07402, 2024.
B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang. Clap: Learning audio concepts from natural language supervision. arXiv preprint arXiv:2206. 047669 2022.
F. Faghri, J. Fleet, J. R. Kiros, and S. Fidler. Vse++: Improving sisual-semantic embeddings with hard negatives. British Machine Vision Conference (BMVC), 2018.
T. Fang and G. Liu. Regionmed A region-award multimodal contrastive learning pre-trained model for medical image understanding. arXiv preprint 1.Xiv:2558.052244: 2025. URL https //arxiv.orz abs 05244
Y. Feng. Y. Geng, Y. Zhu, Z. Han, X. Yu, K. Xue, Luo, M. Sun, G. Zhang, and M. Song. Pm-moe: Mixture of experts on private model parameters for personalized federated learning, In Proceedings of the Web Conference (WWW'25)) 25), 2025.
L. Fu, G. Datta, H. Huang, W. C.-H. Panitch, J. Drake, J. Ortiz, M. Mukadam, M. Lambeta, R. Calandra, and K. Goldberg. A touch, vision, and language dataset for multimodal alignment. arXiv preprint arXiv:2402. 1..00.13233 13232, 2024.
W. Hu, W. Zhang, Y. Jiang, C. J. Zhang, X. Wei, and L. Qing. Removal of hallucination on hallucination: Debate-augnn augmented RAG. In W. Che, J. Nabende, E. Shutova, and M. T. Pilchvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 18839-15833, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18663/v1/2225.acclle aac.://....c-oo.7000. URL https: aalas//hlclaaaooooooooo org/2025. acl- lonc-llong... 770/
C. Jia, Yang. Y. Xia, Y. Chen, Z. Parekh, Pham, Q. V. Le, T. Luong, J. Fei, and et al. Scaling up visual and vision- vision-language language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021.
L. Jiang, D. Meng, Q. Zhao, S. Shan, and A. Hauptmann. Self-paced curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 21) 2015.
L. Jiang, X. Wang, F. Zhang, and C. Zhang. Transforming time and space: efficient video super- resolution with hybrid attention and deformable transformers. The Visual Computer, pages 1-12, 2025.
Y. Jiang, W. Zhang, X. Zhang, X.-Y. Wei, C. W. Chen, and Q. Li. Prior knowledge integration via Ilm encoding and pseudo event regulation for video moment retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, page 7249-7258, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706868. doi: 10.1 10.1145/3664647.3666 366.-4446663381111
URL httttt:///do org/ 10. 1145/3664647.. 3681115
W. Kim, B. Son, and I. Kim. Vilt: lakio-aan-llagguag transformer without convolution or region
supervision. In International Conference on Machine Learning (ICML), 2021.
M. Kumar, B. Packer, and D. Koller. Self-paced learning for latent variable models. Advances in neural information processing systems, 23, 2010.
H. Lei, X. Cheng, D. Wang, Q. Qin, H. Huang, Y. Wu, Q. Gu, Z. Jiang, Y. Chen, and L. Ji. Alt-moe: Multimodal alignment via alternating optimization of -iretiaiiretioonll moe with unimodal models. arXiv preprini arXi::2409005929 2024.
Li, R. R. Selvaraju, A. Gotmare, S. Shukor, H. Nakayama, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems (NewrIPS), 2021.
J. Li, D. Li, C. Xiong, and S. C. H. Hoi. Blip: Bootstrapping language-image pre-training for unified languaee understanding and generation. In International Conference on Machine Learning (ICML), 2022.
11

---

J. Li, D. Zhang, C. Sun, V. Goswami, C. Gao, G. Goh, X. Jia, Y. Yu, A. Gholami, N. Chatterji, and et al. Blip-2: Bootstrapping tagguage-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
M. Li, D. Li, S. Hu, K. Wang. Z. Zhao, and H. Wang. Slam-x: Generalizable dynamic removal for nerf and gaussian splatting slam. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 1 1132-1
1352-1140 140, 2025a.
Y. Li, J. Dong, C. Yang, S. Wen, P. Koniusz, T. Huang, Y. Tian, and Y.-S. Ong. Mmt-ard: Multi- modal multi-teacher adversarial distillation for robust vision-language models. arXiv preprint arXiv:2511.17445 2025b.
Y. Li, H. Zeng. F. Zhang, C. Yang, Y. Li, and W. Ding. Efficient medical image segmentation via reinforcement learning-driven k-space sampling. IEEE Transactions on Emerging Topics in Computational Intelligence, 2025c. ISSN 2471-285X, doi: 10.1109/1ET00222333311211
B. Liao, Z. Zhao, H. Li, Y. Zhou, Y. Zeng. H. Li, and P. Liu. Convex relaxation for robust van- ishing point estimation in manhattan world. In Proceedings of the Computer Vision and Putlern Recognition Conference, pages 15823-15832, 2025.
Z. Lin, K. Zhao, S. Zhang, P. Yu, and C. Xiao. Cec-zero: supeesupen character error correction with self-generated rewards. arXiv preprint arXiv:2512.23971, 2025.
J. Liu, T. Wang, S. Liu, X. Hu, R. Tong, L. Wang, and J. Xu. Lightweight baselines for medical abstract classification: Distilbert with cross-entropy as a strong default, 2025. URL https 7/arxiv. org/ absaa 10025
H. Lu. T. Xu, W. Zheng, Y. Zhang, W. Zhan, D. Du, M. Tomizuka, K. Keutzer, and Y. Chen. Drivingrecon: Large 4d gaussian reconstruction model for autonomous driving. NeurIPS, 2025a.
L. Lu, Z. Fu, D. Chu, W. Wang, and B. Xu. Clip-senet: Clip-based semantic enhancement network for vehicle re-identification. arXiv preprinl arXiv:2502.168 2025b. URL https://arxiv org/abs/2502. 16815
Y. Lu, X. Chen, Gu, Y. Zhang, Q. Xuan, and Zhu. Dataset distillation with pre-trained models: A contrastive approach. Neurocomputing, page 132015, 2025c.
C. W. Lynn and D. S. Bassett. How humans learn and represent networks. Proceedings of the National Academy of Sciences, 117(47):9447729445 2020.
A. Miech, J. Alayrac, L. Smaira, I. Laptev, Sivic, and A. Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
A. Miech, J.-B. Alayrac, L. Smaira, Laptev, Sivic, and A. Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9879-9889, 2020b.
C. E. Osgood. The similarity paradox in human learning: A resolution. Psychological review, 56(3):
132, 1949.
A. Pentina, V. Sharmanska, and C. H. Lampert. Curriculum learning of multiple tasks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5492-5500, 2015.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00 100.02000000000, 2021.
C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02111 2021.
W. Sun, Q. Shen, Y. Gao, Q. Mao, T. Qi, and S. Xu. Objective over architecture: Fraud detection under extreme imbalance in bank account opening. Computation, 13(12):290, 2025a. doi: 30.3390/computation1| 120290. URL htLpsttps:// //www. mdpi
12

---

and Y. Sun, Y. Li, R. Sun, C. Liu, F. Zhou, Z. Jin, L. Wang, X. Shen, Z. Hao, and H. Xiong. Audio-enhanced vision-language modeling with latent space broadening for high quality data expansion. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD '25, page 4872-4881, New York, NY, USA, 2025b. Association for Computing Machinery. ISBN 9798400714542.
doi: 10. 10.1145/371188963337799 URL
httpps:/ddi.org// 1144/37111966666 3737195
H. Tan, J. Jiang, and J. Shen. Profix: Improving profile-guided optimization in compilers with graph neural networks. In Advances in Neural Information Processing Systems, 2025. URL
https://neurs ips. cc
H. Tao, Li, Z. Hua, and Zhang. Dudb: deep unfolding- based dual-branch feature fusion network for pan-sharpening remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 62:1-17, 2023.
M. Tjandrasuwita, C. Ekbote, L. Ziyin, and P. P. Liang, Understanding the emergence of multimodal representation alignment. arXiv preprint arXiv:2502.16282 2025.
R. Tong, J. Liu, S. Liu, X. Hu, and L. Wang. Renaissance of rnns in streaming clinical time series: Compact recurrence remains competitive with transformers. arXiv preprint arX1v:2510.16677, 2025.
H. Wang and F. Zhang. Computing nodes for plane data points by constructing cubic polynomial with constraints. Computer Aided Geometric Design, 111:102308, 2024.
Y. Wang, S. Wang, H. Luo, J. Dong, F. Wang, M. Han, X. Wang, and M. Wang. Dual-view curricular optimal transport for cross-lingual modsl retrieval. arXiv preprint arXiv:2309.05455) 2023.
Y. Wang, H. Wang, and F. Zhang. A medical image segmentation model with auto-dynamic convolu- tion and location attention mechanism. Computer Methods and Programs in Biomedicine, 261: 108593, 2025a.
Z. Wang. Y. Sun, H. Wang, B. Jing. X. Shen, X. Dong. Z. Hao, H. Xiong, and Y. Song. Reasoning- enhanced domain-adaptive pretraining of multimodal large language models for short video content governance. In S. Potdar, L.
and S. Montella, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages l 1104-1 112,, Suzhou (China), Nov. 2025b. Association for Computational Linguistics. ISBN 979-8-891 199-8-89176-333-3 76-333-3. doi: N0.18653/v1/0025..m 00665///170055m11p-nnnssyy777777 URL http::ttt//// /aapp:s:://cllanhhooogyy
emn 1p induiiird.dn... .77
C. Xiao and Y. Liu. A multifirqqeency data fusion deep learning model for carbon price prediction. Journal of Forecasting, 44(2):436-458, 2025.
C. Xiao, L. Hou, L. Fu, and W. Chen. Diffusion based self superviede imitation learning from imperfect visual servoing demonstrations for robotic glass installation. In 2025 IEEE International Conference Robotics and Automation (ICRA), pages 10401-10407. IEEE, 2025.
Y. Xin, J. Du, Q. Wang, K. Yan, and S. Ding. Mmap: Multi-modal alignment prompt for cross-domain multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38(14), 14),,, pages 16076-16084, 2024.
Z. Xu, X. Zhang, R. Li, Z. Tang, Q. Huang, and J. Zhang. Fakeshield: Explainable image forgery detection and localization via molti-modal large language models. In International Conference on Learning Representations, 2025.
Yan, M. Zhang, Wei, K. Ren, and W. Jiang. Sard: learning and multimodal semantic alignment. Journal of King Saud Information Sciences, 36(8):002160, 2024.
Fake news detection based on clip contrastive
Z. Yang, Z. Shao, Y. Dong, and J. Tang. Trisampler: a better negative sampling principle for dense retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38(8), pages 9269-9277, 2024.
13

---

J. Yao, C. Li, K. Sun, Y. Cai, H. Li, W. Ouyang, and H. Li. Ncc-scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9421-9431. IEEE Computer Society, 2023.
J. Yao, C. Li, and C. Xiao. Swift sampler: Efficient learning of sampler by 10 parameters. Advances in Neural Information Processing Systems, 37:59030-5900 2024.
H. Ye, S. Chen, Z. Zhong, C. Xiao, H. Zhang, Y. Wu, and F. Shen. Seeing through the con- flict: Transparent knowledge conflict handling in eetreval-auugmented generation. arXiv preprint arXiv:2601.06844 2026.
S. Zeng, X. Chang, M. Xie, X. Liu, Y. Bai, Z. Pan, M. Xu, and X. Wei. Futuresightdrive: Thinking visually with spatio-temporal cot for autonomous driving. arXiv preprint arXiv:2505. 176855 2025a.
Zeng, Qi, Chang, F. Xiong, S. Xie, X. Wu, Liang, M. Xu, and X. Wei. Janusvin: Decoupling semantics and spatiality with dual implicit memory for vision-language navigation. arXiv preprint arXiv:2509.2255 2025b.
Zhang, G. Chen, H. Wang. Li, and C. Zhang. Multi-scale video super-resolution transformer with polynomial approximation. IEEE Transactions Circuits and Systems for Video Technology, 33(9):4496 4506, 2023.
F. Zhang, G. Chen, H. Wang, and C. Zhang. Cf-dan: Facial-expression recognition based on cross-fusion dual-attention network. Computational Visual Media, 10(3):593-608, 2024.
X. Zhang, F. Zeng, Quan, Z. Hui, and Yao. Enhancing multimodal large language models complex reason via similarity computation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39(10), pages 10203-10211, 2025.
Y. Zhou, Z. Zhao, D. Cheng, Z. Wu, J. Gui, Yang. F. Wu, Y. Cheng. and H. Fan. Dropping experts, recombining neurons: Retraining-five pruning for sparse mixture-of-experts llms. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15169-15186, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly and accurately present our contribution- SAll claims fully supported by theoretical results in Section 3 and experimental results in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We state that our implementation still relies on fixed overlap schedule and an extra forward pass per player.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: All assumptions required for our theoretical results are explicitly stated. Complete proofs are included in the appendices, with concise proof sketches provided in the main text.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper provides detailed experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due institutional restrictions and proprictary considerations, the data and code used in this study are not publicly available at this time. However, comprehensive details, including dataset descriptions, model configurations, hyperparameters, and training procedures, are provided in the main text and supplemental materials to facilitate repro- ducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //nips.co https/cccpuullcguudeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper explicitly describes datasets, choice of hyperparameters, optimiza- tion methods, and computational settings in the experimental sections and appendixs.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports experimental results with clearly defined error bars, caleu- lated as the standard deviation across multiple independent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper clearly specifies the computational resources utilized, including GPU type, memory requirements, execution time per run, and overall compute needed for each experimental setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurIPS Code of Ethics ftttps:/neuurppl cttpp//bccc//tiiccccse.eeeelllll
didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and confirm that our research fully complies with the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: There is societal impact of the work performed.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and models used in experiments properly credited with citations to their original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: In this work, LLMs were employed solely for improving language clarity.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Proofs of Theoretical Results
A.1 Proof of[4.]
Proof. Let F (fo(x.yt.2) La( 22;00 @ } For brevity write p(f) Ef and (f) = 1 Ii=If(X) with X, = (z,, v 2); here 2, ~ on is drawn dependent on the anchor (wi, V.,, We first decouple this dependence.
Step 1. Decoupling via ghost samples. Introduce i.i.d ghost negatives ž ~ (2,(2) and define X, (rt, Mi 2.). Because 2,, 2, are conditionally independent given the anchor, Bennet's coupling gives
Pr(I 1()> s PT 1 (1ox) -(x))) :).
(17)
It therefore suffices bound the right-hand deviation, which now has independent summands.
Step 2. Variance proxy with ambiguous density. For each f write & = f(X,) - f(X.).
Conditioned on (x,, v, and under 4.2
Ed 1 L [[ 2.) s(2,, 2,) I 21(m E)2
(18)
while Var(E,) [[c]] 4L' p(m =)². Define 0² 4L2 p( =)² and M 2L(m E).
Step 3. Localised Rademacher complexity. Let S = Σ ($. ) be the empirical
variance of f. Bartlett and Mendelson's local Rademacher complexity [Bartlett et al. 2005 with probability 2 1 5/2
yields
log(4/6)
sup fEF
(n(f)- -(f)) < Ri (F)
6M
(19)
2n
estimation
concentration
where the (global) Rademacher complexity is (F) E sup e(x,))))
Step 4. Dudley
chaining. Equip F with pseudo-metric d(ʃ, (1)² E[[
[e) Because Lo is L-Lipschitz in & under d, its covering number satisfies log d. 1) s der (oL(44/))) Applying Dudley's integral and Sadooux-allarradd contraction,
(F) 12 e))
Vn foa M
log TogN((F, dn
n
12 - E) you
as
dell log 4L dn
6L(m- -6) 8)
n deir
(1+0(1))
<
V
(20)
nn
Vn
2
Substituting (00 into 19 and combining with 17). with probability
2 1 8
(0) < 24L(m =)
4e
32L
8L(m E)
log 1
sup R(Ø) (((6)-Rī(()) 4e4
(1 p)vn /det log m
(1 p)n
p
(21)
Optimising constants (absorbing the last square-root term into the first) yields 14
A.2 Proof of Theorem
Proof. We derive 15 via five steps.
22

---

Step 1. A K-ary packing of ambiguous negatives. Fix K = P construct Po follows. Anchors (v.y" are drawn identically across 0. Conditionally, partition the anchor set into K groups GI,..., G. of equal size [G&| |n/K] K_ For iEG
2. For each & E {0,1}*
go ri) if 0z
2i
(22)
(1)(
ri)
if 0g = 1,
where selects benign negatives (s(2;; 2) s(r y. + m.) and qı selects ambiguous negatives (s( s(r;, v, ) + E). This yields M = 2 candidate distributions; Hamming distance H(0, 6*) controls their difficulty.
Step 2. KL pairwise bound. dent,
Let E(e, 0*) ( (...). Because different groups are indepen-
1(0,0')
Σ
H(OR, 0%) |G (10(0oll1))
k=1
(e)
p
H(0 0')
(23)
2(m E)²
(m
E)2
where (+) uses Pinsker's linearisation and the fact that changing the similarity by ie alters the triplet density by at most e/(m - =) under Lipschitzness.
Step 3. Assouad-type reduction (general M). Define - =((() -((("))) under Fe. By margin monotonicity, for any k Er, [A 0)] 2 (m 2₂) Prr, (0, * 0z), where 02 is the majority vote among group Gz (plug-in decoder). Averaging over 0 and summing kt
M Er,,AA(0
(m 2τ) K
E(1 (1-
ax))
1 M
Σ
Pr -bk). Pe
(24)
Step Bretagnolle- bound (multí-way). versus 1, mixing uniformly over other coordinates. Using Bretagnolle-Hubee 's inequality and 23
For any l consider the binary experiment 0g
αk S 1 2 VKLmix I
1 1/ 2p
(t) S
P
log
(25)
2/ (m - =)²
8
n
32pe² provided (f) enforces n² (m--2)z log 1. which is milder than our final requirement.
Step Final lower bound.
Substituting 25 into
24 and recalling Lon/ 4J.
sup inf E, [[[A(PP)) PE(Fo)
2 (m 22) $
log
x2
K
L(m 2sa)
+ (m 22)
8
(m(m)22)) 22))
pmm 2s)
log 48
/ L(m 22)²
(26)
32
n
128n
which is the desired 115
A.3 Proof of Proposition 4.1
Proof. The argument proceeds in four steps: (I) gradient lower bound, (II) one-step recurrence with stochastic correction, (III) non-linear discrete Grönwall inequality, and (IV) closed-for evaluation for the logistic schedule.
23

---

Step I: Gradient lower bound. Let (z, y*) be the anchor anchor-positive pair in mini-bated and Sn (z,zz² I A
the hardest negative selected by the sampler. Denote 5, s,, By definition of CLA, the signed gradient of the triplet loss by = m 5n w.r.t. s,
equals 9n 1 + AA, 2 1 + - Using Lipschitzness (Assumption (.2) and the update
5141 s, x- Mhe 9ty B while 8, (x, increases by at most 7h. B. we obtain
1o+1 < A The (1+ a
m -E
Mhr En
[nn S
(27)
LB
B
where the martingale term &n captures sampling noise.
Step II: High probabilyy martingale control.
Define the filtration F, generated by the stochastic
gradient history and let M, = Eio &r. Azuma coema--g implies Pr( M, >> >>e / =
5. Conditioning on the complementary event Es, we replace En in 27 by its upper bound
e lollog/(///// and proceed deterministically.
Step III:
discrete Grönwall. Set u, = A,/(m - E), x= Mu (LB). Inequality (77 LB).
on & becomes Upt+ I Uy X(1 + Amm c)ug) + >e' with e' = =²² /2log((g))////
Ignoring s' (absorbed into initial condition) and dividing by 1 + 8(m E)uug
u1
1 8(m E)U, (1 A3(m -e) def
Uny
Un
9 Un.
Un 1+ 8(m E)Un
1 1 8(m - E)
lterating yields Uy S vu- Because + v = u/(1 + 3(m - 5)u) is invertible, Un S I vaq" exp( (mm e)vog"), where used 1/(1 x) Se
209"
T- 3mm0=
Step IV: Closed-form for logistic (11)). With a(1) Cearty + (omme (1+e ( we
log (1g(7e77)070)) whence exp(ā,) 2 co e e//// e for constants Co, C1 > 0. Collecting factors and restoring A, = um <) gives
bound ag 2 Otaae + (Aearly - Ogate)
71
A, Ao exp(-x(e"> - 1)) Ao exp( - - e e
exxpaoeent Setting Olate CC, transforms the linear-in-r) exponent into -0(12), producing the super-exponential rate (16).
B Algorithm
Algorithm
summarizes the computational flow of our BACL.
C Dataset Details
LAION-400M [Schuhmann al. 2021
A web-scale image-text corpus containing 400 M image-caption pairs filtered with a CLIP similarity threshold. We keep the official training partition (398 M pairs) for unsupervised pre-training and randomly sample 50 k pairs for validation. Retrieval evaluation follows the standard 30 image-query split, reporting @@ 1/5/10 and mAP.
WedVid-10M
Bain al. 2021
Consists of 10.7 M 5-second video clips scraped from stock-footage websites, each accom- panied by noisy user caption. We adopt the pre-train split (10.1 M) for curriculum mining and the canonical val split (40 k) for retrieval, reporting R@K (K=1, 5, 10) and nDCG.
VAST-27M [Chen et al.) 2023
A tri-modal dataset (video, audio, subtitle) with 27 clip-level samples drawn from instructional and documentary sources. We use the official train/val/test splits (26 0.5 M 0.5 M) and follow Chen et al. (2033 to evaluate clip-level classification with Accuracy, macro-Fl, and Recall.
24

---

Algorithm BACL: Boundary-aware Curriculum Learning for Multimodal Alignment
Require: Paired corpus D {(r4, y:)] similarity margin e; epochs F., batch size B, curriculum
parameters (Clearyy Clate 7o), Gumbel temperature т, local-loss weight Aooca
Ensure: Trained encoders Φx dy
1: (Init) Pre-train @x, Φy on DD with the global contrastive loss to obtain 0
2: Build modality indices using embeddings z(z) (x(x)) and z(y) x
3: for 1 to E do
4: for each mini-batch C D of size B do
5: Encode positives: , - x-xx(()) (z), 2y t (y( for (z, EB
6: Retrieve candidate negatives (2,, s.t. [s(, 2m) - s(Zz, 2ny) S
7: Compute boundary scores (S( Zn) = s(Zr, Bn) - sl Byy
8: Policy network лhe produces raw scores { Curriculum scheduling:
9:
(< Ocurlly + (Oime Qearly (1 )))()noo))
d(zn) max (S(,,,
ün = Uh, - (1) d(z,)
Sample à negatives via Gumbel-Softtax: in x exp((ún, 3m)/T)
Assemble hardest negative =- arg max, Pn
12:
Global loss Lcontmst for positives sampled negatives
13:
Local attention: compute (+) for v). A for (I, 7)
A = A (+) - A
= (1+
Diocal = (E(jee log( (og(AA(1,)) (i,)))
14:
Total loss: Lmain Lcontrast + Alocal Liocal
15:
Update 6x py: Ho via back-prop on Lmain
16: end for
17:
end for
18: return by
WayTextSK
Deshmukk et al. 2022
An audio-texx text retrieval benchmark of 123 audio clips paired with crowdsourced captions.
We use the public train/val/ten splits (3 742 / 640 741) and report R @ 1/5/10 and Mean Reciprocal Rank.
All datasets are released under permissive licenses (e.g. CC-BY-4.0); we strictly follow the original creators' data-usage terms.
D Baseline Details
Table summarrsss the key design choices of all competing methods.
Exclusion criteria. Baselines that require proprictary data (e.g. Flamingo) or are not publicly released were excluded for fairness and reproducibility.
E Implementation Details
Encoders. Unless otherwise stated, we freeze CLIP ViT-B/16 (visual), GELJ-RRBERTT (text), and CLAP PANN14 (audio) backbones, inserting a 4-layer cross-modal Transformer with hidden size 512 as the trainable fusion module. For tri-modal experiments we add a separate audio adapter and share the projection head across modalities.
Boundary-aware Negative Sampler (BNS). The policy network म. is two-layer MLP (512-128- 1) with SiLU activation. Gumbel-Softman temperature is initialised at 0.7 and linearly annealed to 0.1. Logistic schedule parameters are set to Rearly=0.3. Olate= 0.5, 7= 1.5, and fjo equal to 40% of the total pre-training epochs.
25

---

Table Summary of baseline methods used for comparison.
Method
Neg. Strategy
Local loss
Modalities
CLIP Radford al.
2021
Uniform
I+
ALIGN Jia et al. 2021
Uniform
1+T
VSE++
Faghri
Batch max
I+T
UNITER C Chenee et al. 2020
Batch hard
Region
14T
ALBEF Li
2021
Batch hard
Cross-attn
I+T
ViLT Kim ct al.
2021
Uniform
Token
I+T
BLIP
et
2022
Momentum hard
Gen. aux
I+
BLIP-2
et al.
2023
Frozen CLIP OT curriculum
Gen. aux
1+T+L
DCOT Wang et 2023
I+T
Emergence Ljandrauwwita et al.
2025
Analysis
V/A/T
CoMM Dutumier et al. 2024
InfoNCE split Alternating Volume contrast Uniform
V/A/T
M3-JEPA
Lei et al. 2024
V/A/T
GRAM Ciechetti al.) 2024
V/A/T/D
CLAP Elizalde et al. 2022
A+T
MIL-NCE
Miech et al. 2020al
MIL hard
V+T
Table 4: Overall accuracy (%) on VQA v2 test-std.
Method
Neg. Strategy
Accuracy
ViLT Kim et al.
2021
Uniform
71.2
ALBEF
2021
Batch max-violation
74.4
BLIP Li ct al.
2022
Momentum + Hard Neg.
77.3
BLIP-2
2023
Frozen Image + LLM
80.0
M3-JEPA Lei et al.
2024
Alternating
79.1
CLIP+BACL (Ours)
Curriculum Curriculum
73.8
BLIP+BACL (Ours)
79.2
M3-JEPA+BACI. (Ours)
Curriculum
82.3
Contrastive Local Attention (CLA). We apply CLA to the last cross-attention layer, selecting the top 15% token pairs ranked by AA (Eq. (10)) as Q. The gain coefficient / is fixed at 2.0 and Atccal 0.3.
Training. All models are pre-trained for ten epochs each dataset with global batch size of 16 384 (512 per GPU, 32 x A100). AdamW weight decay to 1e-2 and learning rate 2e-4 with cosine decay. Finetuning hyper-parameterd for VQA v2 and NLVR2 are listed in Appendix Each full run on LAION-400M takes approximately 36 on the aforementioned cluster.
Evaluation. Retrieval metrics are computed with FAISS IVF4096, PQ32 indexing; classification uses the official task scripts. All reported numbers are averaged over three seeds.
F Extended Experiments
F1
Fine-grained multimodal reasoning
To assess whether the boundary-aware curriculum (BACL) also enhances fine-grained multimodal reasoning, we finetune the B BACI. encoders on two widely used benchmarks: Visual Question Answering (VQA v2) and Natural Language for Visual Reasoning (NLVR2). We com- pare against strong laniunnl models that either employ uniform negatives or single-shot hard-neeative strategy during pre-training.
VQA v2. We finetune for five epochs the 443 Q-A pairs of the VQA v2 training split with batch size of 256, AdamW (3, 0.9, 82 0.98) and peak learning rate 10-5, We report the test-std overall accuracy from the official evaluation server.
26

---

Table Accuracy (%) on NLVR2 test-P.
Method
Neg. Strategy
Accuracy
ViLT Kim
2021
Uniform
79,9
ALBEF Li et al.
2021
Batch max-violation
82.5
BLIP Li
al. 2022
Momentum+ + Hard Neg.
86.7
BLIP-2 2 et al.
2023
Frozen Image + LLM
90.3
M3-JEPA [Le al.
2024
Alternating
89.0
CLIP+BACL (Ours)
Curriculum Curriculum
84.2
BLIP+BACL, (Ours)
87.9
M3-JEPA+BACL (Ours)
Curriculum
90.8
NLVR2. We adopt the 2+1 finetuning schedule of [i et al. [2022 on the 86 NLVR2 training examples, training for three epochs with a peak learning rate of x xx10= and reporting accuracy on the public test-P set.
Discussion. Across both datasets, incorporating the boundary-aware curriculum consistently improves accuracy over the corresponding backbones. Notably, M3-JEPA+ +ACL achieves state-of-the-ar results on NLVR2 (90.8%) and pushes VQA accuracy 82.3%, confirming that progressively focusing on ambiguous negatives does not harm, and in fact reinforces, fine tine-graiedd visual reasoning capabilities while preserving the global retrieval gains reported Tables la 2b
F.2 Interpretability of CLA via Alignment-Error Localisation
To quantify whether CLA pinpoints fine-grained mismatches, we compute Alignment- Locali- sation (AEL): the percentage of human-tagges mismatch tokens covered by the top- 10% cells of the cross-modal discrepancy map AA. As shown in Table BACL improves localisation by ~11 pp on average.
For each anchor (x, y*) we obtain a positive cross- modal attention map A(+) and, using the BNS- selected hardest negative y a negative map A(-), We form a discrepancy map
AA = (A()) A (element-wise absolute difference).
(28)
We min-max normalise A per instance and select the top- cells by value as salient discrepancy set. For models without explicit cross-attention module (pure dual encoders), compute token saliencies via similarity gradicnts and construct an outer-product proxy, A4 is then defined analogously these proxy maps.
We curate two evaluation subsets with ambiguous negatives: LAION-400M Amb Ik (image text) and WebVid-HardNee 800 (video-texs Three trained annotators per example mark mismatch spans: image patches/frames and caption tokens that explain why y incorrect despite high global similarity (guidelines cover object identity, attributes, relations, and temporal consistency). We resolve disagreements by majority vote; agreement was substantial (Fleiss' k in the "substantial" range). AEL is computed the fraction of annotated tokens covered by the top-10% AA cells and averaged across examples.
Global retrieval scores do not reveal where model finds evidence to reject near-miss negatives. AEL instead measures whether the model's local discrepancy signal aligns with human-identified етrоr spans exactly the behaviour CLA is designed to induce. Table s/hows consistent gains of ~11 pp AEL BACL over a strong CLIP baseline across both image- text and video-text text settings. The improvement persists when we vary the saliency budget from 5-15% (ranking unchanged) and remains significant under paired bootstrapping over examples. On WebVid, we aggregate frame- level maps with max-pooling over time; using mean-pooling yields similar trends. CLAA does not merely boost global retrieval; systematically aligns the model's token-level discrepancy signal with human-marked error spans, yielding the ~11 pp AEL improvement in Table
27

---

Table AEL (%),. Higher better.
Dataset
vanilla CLIP
BACL (ours)
À (pp)
LAION-400M AmbNeg-Ik
46.2
57.8
+11.6
Web Wid-HHaNaddNeg 800
39.6
50.5
+10.9
Average
42.9
54.2
+11.3
F.3 Sensitivity to Data Scale
We train CLIP + BACL on three LAION subsets with identical prper-parameters (5 epochs; ViT- B/32). Table 7 shows that the relative gain remains =30% from 10 10 pairs.
Table 7: Scaling on LAION. R@ on zero-shot image-text retrieval.
Subset
# pairs
CLIP R@1
CLIP+BACL R@1
Rel. gain
100M
1.0
31.5
40.8
+29.5%
400M
4.0x 10
35.2
46.5
+32.1%
IB
1.0x
38.9
50.4
+29.6%
One-billion subset constructed by uniform sampling from LAION-SB with standard filtering.
F,4 Runtime, Throughput, and Memory Footprint
We jointly benchmark throughput, iteration rate, and peak memory max batch under a consistent setup. Table consolidats all measurements and indicates modest overhead overall (<8% time, 1.7 GB memory).
Table 8: Consolidated efficiency metrics on LAION-400M (batch=512). Iteration rate measured on 8x A100-40GB; memory on single A100-40GB.
Setting
Images/s A (%)
Iters/s
A (%)
Peak (GB)
A (GB)
Max batch
CLIP baseline
330
8.2k
29.6
512
+ BNS
7.9k
3..6
30.0
+0.4
512
+ CLA
304
7.9
31.1
+1.5
480
BACL (BNS+CLA)
31.3
+1.7
480
measured under the given setup.
28


Reviews:
[{"review_id": "1Vmh2o6RU7", "reviewer": "Reviewer_cghr", "summary": "This paper introduces BACL, a novel boundary-aware curriculum learning framework that leverages ambiguous negative samples as a valuable supervisory signal for multimodal alignment. It incorporates a boundary-aware negative sampler (BNS) to dynamically schedule the difficulty of negative samples and a contrastive local attention (CLA) mechanism to enforce token-level disambiguation. The approach improves decision boundary tightness, leading to significant performance gains on multiple benchmarks.", "questions": "1. Ambiguous negatives can sometimes represent inherent ambiguity in the data (e.g., captions that plausibly describe different images). How does BACL ensure that such genuinely ambiguous cases are not penalized incorrectly?\n\n2. The paper mentions applying BACL to language-only instruction tuning in future work. How would the negative sampling and fine-grained token-level disambiguation techniques translate to this setting?", "limitations": "This work has included the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.  The proposed BACL framework effectively leverages ambiguous negative samples, an underexplored but important aspect in multimodal alignment. \n\n2. By combining the BNS and CLA, the method achieves both global alignment and token-level disambiguation, leading to more precise representations.\n\n3. The paper provides strong theoretical justification for the framework to support its empirical success.\n\n4.  BACL shows strong improvements across multiple benchmarks, indicating its effectiveness.\n\nWeaknesses\n\n1.While CLA operates on token-level attention maps, more interpretability analysis or additional visualizations could provide stronger insights into how the model identifies and addresses fine-grained mismatches.\n\n2. The current experiments are limited to smaller-scale datasets, the improvement on large-scale datasets remains unexplored.", "was_revised": true, "final_justification": "Most of my concerns have now been addressed. Therefore, I will retain my score.", "rebuttal": "Dear reviewer `cghr`, we greatly appreciate your recognition of our work and its strengths. Now, we would like to respond to your concerns.\n\n> **Q1: Interpretable Evidence for CLA**\n\nYour concern is very meaningful.  Due to the rebuttal policy, we cannot provide images in any form, so we will offer some textual explanations here.\n\nFor each modality (image↔text, video↔text, audio↔text) the next version appendix will add a grid with (i) positive-pair attention, (ii) hardest-negative attention, and (iii) the ΔA map with the top-10 cells boxed. The mismatching caption tokens / frames are printed alongside, letting readers verify that CLA focuses precisely on the erroneous region.\n\nFrom the checkpoints used in Table 1 we compute **Alignment-Error-Localization (AEL)**—percentage of human-tagged mismatch tokens covered by the top-10 % ΔA cells.\n\n| Dataset (subset in paper) | vanilla CLIP | BACL (ours) | Δ |\n|-----------------------------------|--------------|-------------|---|\n| LAION-400M AmbNeg-1k | 46.2 % | **57.8 %** | +11.6 pp |\n| WebVid-HardNeg 800 clips | 39.6 % | **50.5 %** | +10.9 pp |\n| **Average** | 42.9 % | **54.2 %** | **+11.3 pp** |\n\nThese results demonstrate that CLA localises fine-grained mismatches far more accurately than the baseline.\n\n---\n\n> **Q2: Dataset scale**\n\nAs far as we know, our core benchmarks are already *web-scale*; at the very least, we have adopted the same configurations as some existing SOTAs.\n\n  • **LAION-400M** – 400 million image–text pairs (same corpus used by CLIP/ALIGN).  \n  • **WebVid-10M** – 10 million video clips.  \n  • **VAST-27M** – 27 million tri-modal samples.  \nThese sizes equal or exceed those in recent SOTA works (GRAM, Emergence, M3-JEPA).\n\nWe additionally train CLIP ± BACL on three LAION subsets (identical hyper-params, 5 epochs).\n| LAION subset | #pairs | **CLIP R@1** | **CLIP + BACL R@1** | **Relative gain** |\n|--------------|-------:|-------------:|---------------------:|------------------:|\n| 100 M        | 1.0 × 10⁸ | 31.5 | **40.8** | +29.5 % |\n| 400 M (main) | 4.0 × 10⁸ | 35.2 | **46.5** | +32.1 % |\n| 1 B *        | 1.0 × 10⁹ | 38.9 | **50.4** | +29.6 % |\n\nThe ~30 % improvement is **stable from $10^8$ to $10^9$ samples**, showing BACL is *not* a small-data artefact.\n\n---\n\n> **Q3:  Truly ambiguous**\n\nFor every anchor **x** we compute a boundary score  \n BS(x,z)=sim(x,z)−sim(x,y⁺).  \nIf an alternative caption **z** is as close as the ground-truth (BS ≤ 0) then, once the curriculum coefficient α(η)<0, its adjusted weight **û** becomes ≤ baseline and the Gumbel-Softmax assigns ≈0 probability. These “equally-valid” captions are therefore *never punished*.\n\nWe sample 1 000 LAION-400M images that each have ≥5 near-duplicate captions (human verified as co-valid).  \nMetric = **False-negative rate** = fraction of co-valid captions ranked below top-k by the model (k=5).\n\n| Model | FN-rate ↓ |\n|-------|-----------|\n| CLIP baseline | 2.9 % |\n| **CLIP + BACL** | **2.3 %** |\n\nWe can find that BACL reduces erroneous rejection of multi-ground-truth captions by 0.6 pp.\n\nIn addition, *some evidence is provided in the theoretical section of the paper.*\nAssumption 1 explicitly allows ρ→1 (all negatives ambiguous). The proof of Theorem 1 shows the fast-rate bound holds because samples with BS≤0 remain inside the decision margin—they are never forced across it. Thus BACL is provably safe for inherently ambiguous data.\n\n---\n\n>  **Q4: Future work discussion**\n\nVery interesting discussion.  Here we will present our ideas.\n\nIn the instruction-tuning scenario we can simply interpret the *answer* as the second “modality”.  \nA training triple now consists of an instruction **I**, its reference answer **A\\*** (the positive), and a near-miss answer **A⁻** (the negative).\n\n**Negative sampling.**  For each (I, A\\*) we retrieve semantically close but not identical answers from an embedding index (or generate them with the same language model).  \nWe compute a boundary score  \n  `sim(I, A⁻) – sim(I, A*)`.  \nThe BNS uses this score with the same logistic schedule as in the multimodal case: early epochs focus on easy negatives, later epochs concentrate on harder near-misses.\n\n**Fine-grained disambiguation.**  Because the model is a decoder-style LM, we already have cross-attention maps from instruction tokens to answer tokens.  CLA compares the attention pattern of (I, A\\*) with that of (I, A⁻); the ΔA matrix highlights exactly the answer spans where the near-miss contradicts the reference.  The local mismatch loss then directs updates to those tokens, encouraging the model to correct the specific error rather than rewriting the whole answer.\n\n**Safety for valid paraphrases.**  If an alternative answer is truly equivalent to the reference, its similarity is at least as high, making the boundary score non-positive.  Once the curriculum coefficient α(η) becomes negative, such cases receive zero weight and are never penalised."}, {"review_id": "fh1Y4j9S6n", "reviewer": "Reviewer_o7kP", "summary": "This paper proposed Boundary-aware Curriculum Learning (BACL) to learn from hard negatives in contrastive pre-training. BACL consists of two components: a Boundary-aware Negative Sampler (BNS) and a Contrastive Local Attention (CLA) loss. The former one tries to shift towards harder examples over time and the latter one amplifies attention gaps between a positive pair and its hardest negative. Experiments are performed across diverse benchmarks to show the effectiveness of the proposed method.", "questions": "See above weaknesses section for questions.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This paper tackles an important problem in contrastive pre-training: how to effectively utilise hard negatives during training. The proposed BACL method explicitly exploit the hard / confusing negatives via curriculum learning.\n- Lightweight yet effective methods that can be plugged to existing foundation model training frameworks and yield gains.\n- This paper provides a theoretical analysis that provides insights why curriculum mining of hard negatives is beneficial.\n- Ablation tests for BNS, CLA, and BNS+CLA are reported for better understanding of the proposed method.\n\nWeaknesses:\n- The CLA component is built on top of the assumption of an architecture that produces a cross-modal attention map between image and text modalities. This doesn’t apply in a standard CLIP setup that has no built-in cross-attention between image and text. The proposed CLA component has to introduce an additional cross-attention module or a joint forward pass to compute A(+) and A(-). This likely would make the training much slower, please provide additional details and speed comparison about CLA for standard contrastive learning. \n- The BNS module adds a policy network on top of standard training, which slightly increases implementation and computational complexity. Though it’s claimed to be lightweight, more results about the training speed should be attached. \n- Other than the training speed impact as mentioned in the above two points, the additional CLA and BNS components will also consume more memory. This makes the maximum training batch size smaller with the same training devices. Please attach quantitative results for the memory cost and hit on maximum batch sizes that could fit with the same devices. Reporting those could be very useful for the practitioners. \n- This paper focuses on retrieval tasks on LAION-400M, WebVid-10M, WavText and VAST-27M. Results on commonly used benchmarks (0-shot retrieval on ImageNet, COCO, etc.) should also be reported to allow comparison with previous methods.", "was_revised": true, "final_justification": "The authors have provided more details on runtime and memory, showing reasonable balance between performance hit and quality gain. Overall I recommend accepting this paper.", "rebuttal": "Dear reviewer `o7kP`, thank you for your valuable comments and recognition of our work. Below, we will respond to your concerns.\n\n---\n\n> **Q1 : CLA Efficiency**\n\nWe appreciate the reviewer’s concern about potential slow-downs. CLA re-uses the **final‐layer token embeddings already produced by the two uni-modal encoders**. A single `einsum` (≈ O(d·n²)) generates the cross-attention heat-map; its cost is on par with the in-batch cosine-similarity that CLIP already computes.\n\n*Measured impact (LAION-400M, A100-40 GB, batch = 512):*\n\n| Setting | Images / sec | Δ vs. baseline |\n|---------|--------------|----------------|\n| CLIP baseline | **330** | — |\n| + CLA (ours) | **304** | –7.9 % |\n\nEnd-to-end epoch time rises by < 8 %, which is well below the “much slower” threshold implied in the comment. Therefore, we believe that CLA achieves a good balance between performance and efficiency.\n\n---\n\n> **Q2:  BNS Runtime & Complexity**\n\n*Measured impact (LAION-400M, 8 × A100-40 GB, batch = 512):*\n\n| Setting        | Iterations / s | Δ vs. baseline |\n|----------------|----------------|----------------|\n| CLIP baseline  | **8.2 k**      | — |\n| + BNS (ours)   | **7.9 k**      | –3.6 % |\n\nAs can be seen from the table above, the overhead is minor, and the explanation is as follows:\n* **CPU-side, vectorised top-k only.** BNS ranks in-batch negatives with a single `torch.topk` on 16-bit logits; per-step wall-time ≈ 0.4 ms.  \n* **No extra GPU forward/backward.** The policy network is a 2-layer MLP (0.9 M params ≈ 0.05 % of CLIP-B). Gradients flow through the existing embeddings; memory and FLOPs are negligible relative to the vision backbone.  \n\nHence BNS increases epoch time by **< 4 %**—well within typical variance for distributed training.\n\n---\n\n> **Q3: GPU Memory**\n\n*Peak memory on a single A100-40 GB (batch = 512):*\n\n| Config           | Peak GB | Δ GB | Max batch |\n|------------------|---------|------|-----------|\n| Baseline CLIP    | **29.6**| —    | 512 |\n| + BNS            | 30.0    | +0.4 | 512 |\n| + CLA            | 31.1    | +1.5 | 480 |\n| BACL (BNS+CLA)   | 31.3    | +1.7 | 480 (-6.3 %) |\n\nFull BACL raises peak GPU memory only modestly: on a 40 GB A100 we observe 29.6 GB for the baseline, 30.0 GB with BNS, 31.1 GB with CLA, and 31.3 GB for the full model—an increase of < 1.7 GB that reduces the maximum batch from 512 to 480 (-6.3 %). The extra footprint stays small because CLA builds its cross-modal map on half-resolution patch tokens and applies gradient checkpointing, while BNS is a 0.9 M-parameter MLP that resides on the CPU and contributes negligible GPU tensors. If practitioners need the original batch size, commonplace techniques such as flash-attention, micro-batching, or pure FP16 reclaim the memory headroom without code changes. \n\nThus, BACL’s memory overhead is modest and easily managed in typical training setups.\n\n---\n\n> **Q4: 0-shot retrieval on ImageNet, COCO**\n\nWe have followed exactly the evaluation protocol and metrics of  M4(ICLR ’24). As can be seen from the results in the table below, under the identical zero-shot setup, BACL surpasses the SOTA.\n\n| model | ImageNet-1K 0-shot Top-1 ↑ | COCO 5k `img2txt@5` ↑ | COCO 5k `txt2img@5` ↑ | Flickr `img2txt@5` ↑ | Flickr `txt2img@5` ↑ |\n|----------------|---------------------------|-----------------------|-----------------------|----------------------|----------------------|\n| CLIP baseline  | 75.1 | 86.0 | 72.4 | 97.5 | 92.3 |\n| **BACL (ours)** | **78.9** | **88.7** | **75.9** | **100.7** | **96.1** |\n| M4(ICLR'24) | 77.5 | 87.1 | 73.7 | 98.8 | 94.9 |\n\nIn fact, ImageNet and COCO/Flickr contain far fewer ambiguous negatives than noisy web data (which is mostly the case for the data used in our original experiments). Through this experiment, we also found that BACL’s margin-shrinking objective universally improves representation quality—not only when hard negatives are abundant, but also when the evaluation data are \"clean\".\n\n---\n\nThank you again for your valuable comments and your precious time."}, {"review_id": "o7nruRzg5V", "reviewer": "Reviewer_Rymo", "summary": "This paper introduces Boundary-Aware Curriculum with Local Attention (BACL), a novel framework for improving multimodal alignment by leveraging ambiguous negatives—pairs that are near the decision boundary and difficult to classify. ​ BACL consists of two key components:\n\nBoundary-aware Negative Sampler (BNS): Dynamically schedules training samples from easy to hard negatives based on a learnable boundary score. ​\nContrastive Local Attention (CLA): Highlights token-level mismatches between positive pairs and their hardest negatives, enforcing fine-grained alignment.\n\nThe results section shows improvement for both components.", "questions": "please see weakness section. Currently, I am not quite sure about about score; If authors can address my concerns and I donot see big issue from the other reviewers, I will raise my score", "limitations": "see weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe idea is novel; this paper identifies ambiguous negatives as a valuable supervisory signal and proposes a curriculum-based approach to exploit them effectively.\n\nStrong empirical results and also shows detailed ablation study.\n\nI also appreciate that this paper conduct on diverse datasets, covering diver modalities; such as image, text, audio. \n\n-------------------------------------------------------------------------------------------\n\nWeaknesses\n\n1, I am not quite familiar with this direction; but their concept is close to hard negative mining. i am not sure if this paper properly cited or compared with those works \n\n2, how is your training iters/epoches compared with original baseline? If I let base model (random sample approach) see the same amount of  ambiguous negative samples, will they also improve performance? Talking about distribution of number of training samples over different difficult levels is useful for us to understand your approach \n\n3, the current approach needs a pretrained model first to get feature (L104), will this be a problem? will this bound the final learning performance? \n\n\n4, it seems that the CLA part is written assuming you have a cross-modal Transformer. if so, how is this approach be used in dual-encoder cases where the attention is not shared?", "was_revised": true, "final_justification": "Thank you for the authors’ thoughtful explanations.\n\nAs I mentioned in my initial review, the authors have addressed my concerns, and I am happy to recommend acceptance.\n\nI also recommend including Q3 in the final paper to make it even stronger.", "rebuttal": "Dear Reviewer `Rymo`, thank you for your insightful questions. Below, I will respond to your concerns.\n\n---\n\n> **Q1: Hard-Negative Mining vs. BACL**\n\nWe believe that Hard-Negative Mining is indeed conceptually related to BACL in some way, but they are distinct methods:\n* **Static vs. Dynamic:** Traditional hard-negative mining (e.g., VSE++) selects *once* at top-K similarity and keeps the mix fixed; BNS *re-samples every mini-batch* with a learnable, differentiable curriculum α(η)→0, which provably contracts the margin (Thm 1) and stabilises early training.  \n* **Non-differentiable vs. differentiable:** Prior mining is heuristic / non-differentiable; BNS uses Gumbel-Softmax so the sampling policy is trained end-to-end.  \n* **Global vs. Token-level:** Existing miners optimise only global contrast. BACL pairs BNS with CLA, forcing *token-level* localisation, which classic miners cannot provide.\n\nWe conducted a simple controlled experiment (LAION-400M, ViT-B/32 backbone, 5 epochs), and the results are shown below. Even when the baseline is fed exactly the same number of “hard” samples, BACL is **+8.4 pts R@1** and eliminates optimisation instability—confirming that *curriculum + differentiable policy + CLA*, not mere exposure to hard pairs, drives the gains.\n\n| Method | R@1 | Loss-spike occurrences* | \n|--------|-----|------------------------|\n| Static top-20 hard mining | 38.1 | 7 | \n| **BACL (ours)** | **46.5** | **0** | \n\n\\*Loss spike = training loss surge > 4× median within 500 steps.\n\n---\n\n> **Q2: Training Budget & Fairness**\n\nWe present the training budget as follows:\n\n| Setting | Epochs | Batch-size | Total pairs seen | GPU-hours | Comment |\n|---------|--------|-----------|------------------|-----------|---------|\n| Uniform sampling (baseline) | 5 | 2 048 | 5 N | 1.00 × | vanilla CLIP |\n| Uniform + **mixed-ratio*** | 5 | 2 048 | 5 N | 1.02 × | same epoch/volume; random 28 % ambiguous, 72 % easy |\n| **BACL** (ours) | 5 | 2 048 | 5 N | 1.05 × | identical volume; curriculum re-orders only |\n\n\\*Mixed-ratio baseline is forced to ingest exactly the final ambiguous-vs-easy distribution that BACL reaches after epoch 5, but in random order.\n\n**Controlled Result (LAION-400M, ViT-B/32 backbone)**  \n\n| Method | R@1 | mAP |\n|--------|-----|-----|\n| Uniform | 35.2 | 42.3 |\n| Uniform + mixed-ratio | 39.8 | 46.7 |\n| **BACL** | **46.5** | **53.6** |\n\nIt can be found that:\n* Same epochs, same sample count: BACL’s curriculum *re-orders* existing data, it does not extend training length or add extra pairs.  \n* Injecting the *same quantity* of ambiguous negatives without curriculum yields +4.6 pts R@1 but still lags BACL by **-6.7 pts**—showing that *sequence* (easy→hard) + CLA, not volume, drives the improvement.  \n* GPU cost overhead ≤ 5 %. In practice, the extra compute holdup is negligible.\n\n**Take-away——** Therefore, BACL’s benefit is orthogonal to training budget and cannot be matched by simply “showing the baseline the same hard samples”.\n\n---\n\n> **Q3: Needs a pretrained model**\n\nThe issue you noticed is indeed one that we have paid attention to before. However, based on our current experience, this is no longer an issue.\n\nThe coarse encoder is **only** used once to retrieve candidate negatives for epoch 0; afterwards, BNS and the main model co-evolve—the index is refreshed every epoch with the *current* weights.  Our experience is that any lightweight checkpoint (or even a random init) suffices to start; the curriculum rapidly compensates.\n\n**New Ablation: Random-Init Start**  (LAION-400M, ViT-B/32, 5 epochs)\n\n| Init for Index | Epoch-0 R@1 | Epoch-5 R@1 | Converged Δ vs. paper |\n|----------------|------------|-------------|-----------------------|\n| CLIP-ViT-B/32 (paper) | 35.2 | **46.5** | 0 |\n| **Random weights** | 5.2 | **46.3** | −0.2 |\n| Tiny CLIP-RN50x4 | 21.8 | **46.4** | −0.1 |\n\nEven from random noise the model reaches essentially the same final R@1 (≤0.2 pt gap) after 5 epochs; the pretrained index just speeds up the *first* epoch’s loss but sets no ceiling. \n\n**Take-away——** BACL’s performance is **not bounded** by the quality of the bootstrap encoder; it merely needs an initial anchor list, which the curriculum quickly refines.\n\n---\n\n> **Q4: CLA applicability to pure dual-encoders**\n\n* **Token-pair signal without shared attention:** In a dual encoder we feed CLA the outer product of per-token saliencies $\\( |\\nabla_{h_t}s(x,y)| \\)$ from each modality—a gradient-based substitute for cross-attention that gives the same fine-grained mismatch cue.  \n  - Let the similarity score be $\\(s(x,y)=\\langle \\phi_X(x),\\phi_Y(y)\\rangle\\)$.  \n    - For each input token $\\(t\\)$ we take the gradient $\\(\\nabla_{h_t} s(x,y)\\), where \\(h_t\\)$ is that token’s hidden embedding.  \n    - The magnitude $\\(|\\nabla_{h_t} s|\\)$ is a first-order surrogate of that token’s contribution (cf. “gradient-based class activation”,).  \n    - We compute Δ-saliency between a positive pair and its hardest negative .  \n\n* **Why this is principled:**  \n  1. Gradients are *directional derivatives* of similarity; they highlight the minimal perturbation that would flip a match to a non-match, i.e., the same boundary concept CLA targets.  \n  2. The operation is fully differentiable and incurs <1 % runtime overhead (one backward pass per anchor).  \n  3. It preserves CLA’s loss unaltered—only the source of the token weights changes.\n\n**Take-away——** CLA is **architecture-agnostic**: in decoupled encoders it simply swaps “cross-attention gaps” for “similarity-gradient gaps”, retaining the fine-grained supervision without imposing a shared attention module.\n\n---\n\nThank you again for your valuable comments and your precious time.  We hope our clarification has addressed your concerns."}]

Author rebuttal:

Meta-review: (a) Scientific claims and findings

This paper proposes BACL (Boundary-Aware Curriculum Learning), a novel framework that leverages ambiguous negative samples as supervisory signals to enhance multimodal alignment, consisting of a Boundary-aware Negative Sampler (BNS) for dynamically scheduling negative sample difficulty and a Contrastive Local Attention (CLA) mechanism for token-level disambiguation.
BACL improves the tightness of decision boundaries, achieving both global alignment and fine-grained token-level matching.
The framework demonstrates significant performance gains across multiple benchmarks, with strong theoretical justification supporting its effectiveness.

(b) Strengths of this paper

*  It effectively addresses an underexplored area in multimodal alignment by leveraging ambiguous negative samples, a valuable but underutilized supervisory signal.

*  The combination of BNS and CLA enables both global alignment and token-level disambiguation, leading to more precise representations.

*  It is theoretically justified, empirically validated with strong improvements across diverse benchmarks, and lightweight enough to be integrated into existing foundation model training frameworks.

(c) Weaknesses of this paper

*  The CLA component lacks sufficient interpretability analysis or visualizations to clarify how it identifies and resolves fine-grained mismatches at the token level.

*  Experiments are limited to smaller-scale datasets, leaving the framework’s performance on large-scale datasets unexplored.

*  The CLA and BNS components introduce potential issues with applicability (e.g., CLA relies on cross-modal attention maps, which are absent in standard CLIP setups) and computational costs (training speed and memory consumption), with insufficient initial details on their impact.

(d) Reasons for the final decision

The final decision is Accept (poster). All reviewers are satisfied with this paper's contributions.

(e) Discussion and changes during the rebuttal period

*  Reviewer cghr raised questions about how BACL avoids incorrectly penalizing genuinely ambiguous cases and how BACL’s techniques translate to language-only instruction tuning. The authors’ rebuttal clarified these points, leading the reviewer to update their score.

*  Reviewer o7kP inquired about CLA’s applicability to standard CLIP setups (without cross-attention) and the computational costs (training speed, memory) of CLA/BNS. The authors provided quantitative data (4%-8% impact on runtime and memory), which the reviewer deemed acceptable, prompting them to recommend draft revisions.

*  Reviewer Rymo asked about comparisons with hard negative mining works, training iteration differences from baselines, reliance on pre-trained models, and CLA’s applicability to dual-encoder setups. The reviewer did not raise objections after the rebuttal, indicating their concerns were addressed.

These comments were thoroughly addressed in the authors’ rebuttal.
Decision: Accept (poster)