Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval
Authors: Jian Xiao
Zijie Song
Jialong Hu
Hao Cheng
Zhenzhen Hu
Jia Li
Richang Hong
Venue: NeurIPS 2025 poster ()
Keywords: cross-modal retrieval
text-video retrieval
video understanding
Primary area: applications
TL;DR: We introduce GARE, a gap-aware retrieval framework that learns pair-specific increments to alleviate optimization tension and false-negative noise in cross-modal alignment, achieving better uniformity and semantic structure.

Abstract: Recent progress in text–video retrieval has been largely driven by contrastive learning.
However, existing methods often overlook the effect of the modality gap, which causes anchor representations to undergo in-place optimization (i.e., optimization tension) that limits their alignment capacity.
Moreover, noisy hard negatives further distort the semantics of anchors.
To address these issues, we propose GARE, a Gap-Aware Retrieval framework that introduces a learnable, pair-specific increment $\Delta_{ij}$ between text $t_i$ and video $v_j$, redistributing gradients to relieve optimization tension and absorb noise. We derive $\Delta_{ij}$ via a multivariate first-order Taylor expansion of the InfoNCE loss under a trust-region constraint, showing that it guides updates along locally consistent descent directions. A lightweight neural module conditioned on the semantic gap couples increments across batches for structure-aware correction. Furthermore, we regularize $\Delta$ through a variational information bottleneck with relaxed compression, enhancing stability and semantic consistency. Experiments on four benchmarks demonstrate that GARE consistently improves alignment accuracy and robustness, validating the effectiveness of gap-aware tension mitigation.

Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval
Jian Xiao*, Zijie Song². Jialong Hu-, Hao Cheng'. Zhenzhen Hu :u Li Richang Hong'
SSchoo of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China
2 Schooo of Big Data and Statistics, Anhui University, Hefei, China
[j xiao hfut, chenghao]@mail hfut. edu. cn, zjsongSahu edu. 2dszds5340gmail.com com, (zzhu, lijiaa @hfut. edu, en, hongre. wwwprcchfuttgmal.c.omm com
Abstract
Recent progress in text-video retrieval has been largely driven by contrastive learning. However, existing methods often overlook the effect of the modality gap, which causes anchor representations undergo in-place optimization (i.e., optimization tension) that limits their alignment capacity. Moreover, noisy hard negatives further distort the semantics of anchors. To address these issues, we propose GARE, a Gap-Aware Retrieval framework that introduces learnable, pair-specific increment Aij between text 4; and video Uj, redistributing gradi- ents to relieve optimization tension and absorb noise. We derive Aij via mul- tivariate first-order Taylor expansion of the InfoNCE loss under trust-region constraint, showing that it guides updates along locally consistent descent direc- tions. lightweight neural module conditioned on the semantic gap couples increments across batches for structure-awarr correction. Furthermore, we regu- larize through variational information bottleneck with relaxed compression, enhancing stability and semantic consistency. Experiments on four benchmarks demonstrate that GARE consistently improves alignment accuracy and robustness, validating the effectiveness of gap-aware tension mitigation. Code is available at https:/giihub.com/mmsiccaaa 17/GARE-text- veve-e-eeeviio--errivvlllll
1 Introduction
Text-video retrieval (TVR) [55] a fundamental task in video understanding, aiming to retrieve relevant videos given text query [34, 33, 45, 17]. With the proliferation of short video platforms, this task has attracted growing research interest. In recent years, vision- languaee pretraining models such CLIP [38] have shown great success in cross-modal representation alignment, demonstrating strong performance on various retrieval benchmarks. These models typically learn a shared embedding space by aligning visual and textual modalities through large-scale contrastive learning [50, 20, 10, 11, 19, 46, 22, 31], and have thus become a popular backbone in TVR systems.
Despite the empirical success of contrastive learning in text-video retrieval, two critical problems persist. First, the most challenge is optimization tension, arising from the modality gap: text and video embeddings typically occupy disjoint regions of the representation space [31, 54], with markedly different semantic structures and high distributional divergence (e.g., large KL divergence [28] between modality-wiss, feature distributions). As shown in Figure la, this separation creates structural conflict for a text anchor t,: the gradient from its positive video v; attracts t, toward the video manifold, while gradients from all negatives By repel it in the opposite direction, yielding nearly collinear but reversed forces. The second is the prevalence of false negatives: semantically similar
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Test embedding
Video embedding
Fals negative
Tension
Offload
wrreg pusk fron
roellan Σ
Vi
casc wrong prahi
Pacch Poin
V;
Paiti mae
part Rroo
Text Modality
Text Modelity
Video Modality
Video Modality
(a)
(b)
Figure 1: Tension and false-negative challenge vs. our offloading strategy. (a) Owing to the modality gap [31], gradients from negative samples overlap with the positive direction, creating optimization tension around the anchor t, and limiting its update freedom. (b) GARE offloads part of this optimization pressure from t; to learnable increments A.j. relaxing the gradient field and absorbing false-negative noise. Each Ai encodes a semantically meaningful correction of the text-video gap.
yet unlabeled pairs are incorrectly treated as hard negatives [12, 9], introducing noisy gradients and aggravating misalignment. These issues jointly limit the ceiling of semantic alignment, leading models to converge to suboptimal solutions.
We further analyze the optimization tension through aggregated gradient statistics across batches (Figure 2). In dimensions exhibiting significant gradient behavior, both positive and neg- ative components reach values on the order of 40-60 (bottom of Figure 2), yet their sum-the actual gradient applied to ts #g-remains close to 2-3 (top of Figure 2). This indicates that positive and negative signals not only oppose each other in direction but also nearly cancel in magnitude. As result, text anchors undergo an in-place optimization behavior: their rep- resentations remain close to their initial positions throughout training, thereby limiting the attainable alignment performance of contrastive learning.
To address both issues, we introduce pair-specific increment Ai that acts as a learnable adjustment between L and vy. As shown in Figure 1b, unlike the anchor embedding t,. which aggregates gradients from all video pairs (i, k). the increment Ag only absorbs gradients transmitted from its corresponding pair (i,j). This ensures that each Ai captures localized component of the optimization signal, while retains global supervision. As result, gradients acting on f; are partially redistributed A.j, effectively diluting the optimization tension and preventing anchors from being trapped in conflicting descent directions. Beyond relieving tension, this design also buffers the local gradient noise introduced by false negatives, since noisy repulsion from mislabeled pairs is absorbed at the pair level rather than directly perturbing anchor representations.
Mean and variance of summed gradient (top) and negative gradients (bottom) across 512 dimen- sions, showing collinear opposite forces that largely cancel out.
Figure/ 2:
To guide Asj toward constructive updates, we derive its local rule from a multivariate first-order Taylor expansion of the contrastive loss under an l trust-region constraint. This lincarization defines descent space in which, once a constraint radius is specified, the steepest-descent direction the coupled state becomes unique and preserves the local relative-ranking structure of InfoNCE [37]. We implement this process on the semantic gap (vy - tz). The module is optimized through backpropagation across batches, while a norm-based prior regularizes the magnitude of A, serving as an implicit trust-region radius regularization. To ensure semantically stable and generalizable updates, we formulate the learning of A.y as a deterministic variational information bottleneck: the module & outputs Aijs and Jensen-relaxed KL bottleneck term balances informativeness and compression.
pair- specific, amortized update using lightweight module & conditioned

---

On the MSR-VTT Ik-A validation set, we observe that the overall cosine similarity of positive pairs decreases relative the baseline. This indicates that releasing optimization tension via A encourages samples to spread with larger angular separations, thereby improving uniformity on the unit hypersphere [46]. addition, we find that the Fuclidean distance between paired embeddings increases after applying A.j. suggesting that pair-level adjustments help alleviate tension and provide finer control over semantic positioning.
Our contributions are threefold: 1) We analyze the gradient structure of InfoNCE and reveal its inherent multi-variabee coupling by introducing pairwise increments Aty multivariate first-order Taylor expansion within trust region yields update rule for each A; consistent with the InfoNCE descent direction. We propose Cap-Aware Retrieval (GARE) framework, where learnable network predicts pair-specific increments Aij and integrates them into the forward pass to offload optimization tension while mitigating noise from false negatives. We also introduce a relaxed variational information bottleneck (VIB) objective that regularizes Aij. balancing informativeness and compression. 3) Experiments on four video retrieval benchmarks, i.e., MSR-VTT [52], DiDeMo [2], ActivityNet Captions [27], and MSVD [8], showing consistent improvements, and further analyses confirm that the learned increments are semantically meaningful and geometrically structured.
2 Approach
2.1 Preliminaries
Task Definition.
Given dataset of N paired video-text examples ((1s, 44) the goal of text- video retrieval learn a pair of encoders: visual encoder (e(-) and text encoder (e).), that map inputs into shared embedding space. The similarity between any pair (uz, 44) is computed using scoring function, typically the cosine similarity:
(t,),) TO)
$1j (t)( (6.))-llo.0;;:] (0;()))
where (t,) E IR' and D is dimension. During training, contrastive learning is applied to increase the similarity matched pairs while decreasing that mismatched ones. At inference, retrieval is performed by ranking all candidate texts (or videos) for given query video (or text) based on similarity scores. For brevity, we henceforth refer (t) as t, &r as v.
Optimization Tension and First-order Increment Modeling. As discussed in the introduction, contrastive optimization in video-text retrieval suffers from gradient tension caused by modality gap and noise from false negatives. These factors hinder stable optimization of the anchor representation t, which must simultaneously align with the positive v, and repel all negatives Uj Formally, the per-anchor InfoNCE loss is given by:
cos(ts,)))/
C. log B
ES= cco((t))))
and its gradient with respect to t is:
V.C. Σ
(pij Vij)
Dy |
cos(ti, 0z)
(3)
| 12
weight
acs ER Josy
Dess ERD
gradient basis
where Pij (oftmma(coos(t:;: Wz)/T), Vij = [==+ is temperature parameter and B is batch size. JC,
Here, the weight term dsiy controls the relative strength of each pair, while the gradient basis
Bag E R specifies the update direction. Under a strong modality gap, the positive and negative dt,
gradient bases are nearly collinear but reversed, and the positive weight |pa - 1 is of comparable magnitude the aggregate negative weight Ejø; Pij [44]. Together, these conditions cause the gradients largely cancel along nearly the same axis, leaving only small residual update for 4;-
To alleviate both challenges, we introduce a pair-specific increment Asj that locally adjusts the relative positioning of each text-video pair. This increment serves two purposes: offloads optimization

---

tension by redistributing part of the gradients originally acting on E; to the pair level, and buffers false- negative noise before propagates the anchor embedding. For clarity, we first consider applying Aij to the text side, yielding an adjusted representation lAis 1 + Aij. The same mechanism can also applied the video side (e.g., vA,; = 1j + Asj), depending on dataset characteristics. This flexibility raises a central question: how should each Aij be optimized to effectively reduce the contrastive loss?
A naive approach would linearize the loss around the original anchor & (i.e., A.. - 0), treating each Ay as an independent perturbation. However, under this formulation, the gradient V VA., C, depends only the static similarity cos( U2), failing to account for the influence of other Au in the same batch. This leads to decoupled gradients that ignore the softmax coupling intrinsic to InfoNCE. More importantly, expanding the loss only at 4 results in univariate approximation, which does not reflect how different Á.j collectively reshape the similarity ranking across all uj. Since our goal is to adjust the relative structure of the entire pair set for each anchor, we require multivariate formulation where all Aij are coupled and optimized under shared comparison scale.
To this end, we treat all Aij as jointly optimized variables, and reinterpret the per-anchor contrastive loss C multivariate function over the full set {Ai )ー=
eau/ Σ etult"
((A AIB) = - logg
8i cos(1, + Aeg, m2).
(4)
Unlike standard per-sample optimization, the softmax structure couples all A. meaning the gradient of any single increment depends on the values of the others. To capture this dependency, we perform multivariate first-order Taylor expansion around prior coupled state (0) ( } where all increments are nonzero. In practice, we treat this prior as the current state at iteration (i.e., we let
(t) := ( and analyze the local descent behavior from this reference point. This converts our optimization into an iterative process, where each step refines the current set of coupled increments. The resulting linear approximation is:
L((A.)) LIAA + Σ C(AA) (Aig - -
Crucially, this expansion preserves the softmax-inducee coupling: each gradient term VA,, C, is evaluated assuming that all other increments Askyy remain fixed at their nonzero states. The resulting linearized loss defines a local descent landscape for each Aij. To ensure reliable updates within this approximation, we impose trust-region constraint that bounds the update magnitude: | S which reflects our belief that meaningful corrections should occur within a localized region around 4. Under this constraint, the optimal update direction for each Aij corresponds steepest descent along its local gradient:
V VA, CA!!!)
A (++1)
A
a
(t)
(6)
CA4
where step size oij is analytically determined to ensure Cauchy-Schwarz inequality to project onto the trust-region boundary (see Appendix H for de- tails). This yields the steepest descent direction of the linearized loss, evaluated at coupled batch state (!)) where all increments are fixed but nonzero. However, this update rule has two main limitations: (i) it only guarantees descent within the current batch, lacking cross-batch generalization; and (ii) it suffers from scale ambiguity, as the optimal trust-region radius & should vary across pairs with semantic difficulty and training stage. To overcome these issues, we adopt learnable network
|| I and derived via the
z that directly predicts the coupled increment state A from the semantic gap By t, (or tz - 2.)). thereby establishing cross- cross-batch coupling among increments. The subsequent update to A [t+1) is implicitly performed via backpropagation on the InfoNCE loss. Since the loss gradient naturally aligns with the steepest descent direction, this formulation amortizes the iterative update process into a trainable prediction problem, enabling structure-aware and generalizable optimization over the full set of pairwise increments.
i-

---

2.2 Gap-Aware Increment Modeling via Pair-Specific As
To make this optimization tractable, we replace the explicit iterative updates with a learnable function & that directly predicts each increment (t) Specifically, we amortize the descent process by training a parameterized network & to generate the current coupled increment state based on pairwise semantic difference and modality-dependeen context:
D = + (nlo, - C.).;;
where 7 E {-1, +1} controls whether the increment encodes video-specific or text-specific residual semantics. Here, t; denotes the [CLS] embedding of the text sequence, and V is the mean- meea--poled representation of the frame features V frame of the j-th video. We implement v Cross-Attention module, with the semantic gap ;j t; as the Query and the feature sequence V frame E RR,, x (or word E I xxxx) xD as the context C (i.e., the Key and Value). Intuitively, the query encodes what semantics semantics are assigned higher weights, so that the aggregated increment Diy acts as a semantic patch correcting t,. Rather than computing explicit descent steps of Eq. (6), 3/ learns to output increments that approximate the coupled descent direction while incorporating structure-aware priors from the context. Since A.j directly enters the InfoNCE loss, &i is optimized end-to-end via backpropagation, enabling its outputs to align with the true gradient field. We now consider the case where = frame and the increment Ag is added to the text side. In this setting, the two variables 4 and Diy share the same gradient flow: Vi,C, = =ドド Via.g L; and VA. = V Viais Li where the gradient with respect to each perturbed anchor is given by:
present in 2; but missing in f (or vice versa); sequence features carrying such
(pij Vig)
Dj
LAs
Via. C.
cos(ts.),, U/)
(8)
M |05
Ita.,
This formulation introduces gradient redistribution effect: unlike standard InfoNCE where all gradients act directly on the shared anchor f,, our pair-specific design allows each negative video U influence only its associated increment Aij. The positive pair (t,, 0z) contributes attraction gradients to both f; and As, facilitating alignment; meanwhile, each negative pair (f,, Uy), * i, applies repulsion primarily to A.y. This leads two key benefits: 1) tension relief: Ag can absorb the gradient from (f,, U2). reducing the burden on t; to decrease loss in single step; 2) false- negative suppression: repulsion from semantically similar negatives is redirected into their respective increments, reducing the semantic bias of the anchor representation. We apply this strategy under symmetric InfoNCE loss:
eaulr
B
Linto
log Σe=
log
(9)
2
B
easst
B j=1
B E ea/lTT
video
video to text
This loss function maximizes the similarity of positive pairs s(t, + Aus U)) and minimizes the similarity of negative pairs.
Norm-Based Regularization of Trust-Region Radii. In our formulation, the increment Au predicted by & is directly constrained within a trust region, thus its norm Eij || | lla serves as the trust-region radius. This radius controls how far the corrected representation ب3م is allowed to deviate from t; in order to release optimization tension and adjust semanties. Intuitively, semantically similar pairs should yield smaller radii, while dissimilar pairs should allow larger ones. To encourage such structured variability, we regularize the intra-anchor distribution of radii by promoting norm diversity:
C. = L- [Var ((=s)ffi)]
(10)
where B, denotes the batch of text anchors. A lower bound max --,,--) with À > 0 is applied to prevent instability. This regularization sharpens the implicit trust-region structure learned by v/, guiding Ai to reflect pairwise semantic variability in a stable manner.
5

---

Directional Diversity Regularization. To enhance the expressiveness of the learned increments Aij. we introduce a directional regularization that encourages the directions of [ررر} under each anchor I to be diverse. This helps the model assign distinct update directions to different candidate videos, improving the generalization of representations and mitigating mode collapse. Specifically, we normalize each increment to obtain unit vectors 2ij
A. and define the regularization loss as
the expected angular similarity across all anchor-specific direction sets:
Lair B, log Eg [exp (-a (1 (zig, 21x)))1
(11)
where a is a scale factor to control uniformity. This loss softly penalizes directional concentration, while still allowing nearby directions for semantically similar negatives preserving flexibility under uncertainty. Combined with the norm-basee regularization, this term enables fine- greeneaaind control over both the magnitude and direction of each increment Aij, leading to more stable and structure-aware optimization.
2.3 Variational Information Bottleneck (VIB) for Semantic Increments
We motivate our regularization from the Information Bottleneck (IB) principle [41, 1J. which seeks to maximize predictive information while suppressing nuisance factors. In our case, the increment A is optimized only by the gradient from its paired sample (4,, r2) and thus lacks contrastive behavior, often collapsing into trivial solutions. We therefore treat A as an information bottleneck that extracts semantic signals from (t,, fÿ) while discarding noise, effectively constraining it within a prior structure before allowing to reduce the InfoNCE objective. Formally, this corresponds to maximizing the mutual information between A and the target semantics while minimizing its dependence on the input pair:
max T(A;Y) - 3 T(A:X)
(12)
p(A(X)
where x= (t,, vÿ) denotes input pair, Y the label indicating whether it is positive match, A the pair-specific increment (i.e, the latent variable 2), and (3 trade-off between task term 7(A; Y) and compression term I(L; X). Following the standard variational derivation (details in Appendix D), we obtain the objective
Lvm - ((,,,,)) EA (A log (v A)] +B. ((,,))
(13)
contrastive term Lini
compression term Ci
where gu (A)t, v) denotes the variational encoder that parameterizes the posterior of A given v). (o(y) is the variational classifier instantiated softmax predictor, and r(A) = ( I) is the Gaussian prior used in the upper-bound regularization of the latent space.
In practice, we adopt deterministic instantiation where the variational distribution qu (Alt, v) collapses to a Dirac posterior centered at the network output Ay (i..., μ = Aig). without uncertainty- aware sampling. Since the Dirac measure is singular with respect any continuous prior, the KL divergence is ill-defined. To obtain a tractable and stable regularizer, we aggregate posteriors along the text dimension, leveraging the asymmetric nature of video text data (i.e., videos typically contain higher redundancy and correspond to multiple semantically related texts). By the convexity of KL(-Ilr) and Jensen's 's inequality, this yields relaxation that also circumvents the singularity of the deterministic posterior:
((t.o))
[L (g (Alt, u) (A)) E. Eqp KL (Alt, [(])]]
(14)
2 Eg KL (A (A))),
where 7. (A/v) := Eej (g v)] is the aggregated increment posterior for video e. This relaxation effect while reducing sensitivity to text-side variability. The
preserves the precise relationship between this relaxed KL term and the original mutual information [(A; X) derived in Appendix E. Putting the relaxation into practice, we approximate (A)r)) with a Gaussian fitted to the set of increments { 1 associated with each video anchor Vy. This yields the following relaxed compression loss:
(15) where Hy and of denote the mean and variance of increments ( { over the text batch for each video dy- This relaxed KI. term operationalizes the bottleneck by regularizing increments at the video level, enforcing centered and isotropic corrections while avoiding over-penalization of text-side variability. We provide the overall training objective and inference details in Appendix F.
CT = Eny-B [KL (UNN((,,)))11 (0,, I))

---

Table 1: Comparison results MSR-VTT dataset Text-to- Video Retrieval and Video-to-Teds Retrieval. DiCoSA [24] utilizes QB-Norm [6] for inference and grayed out for a fair comparison. Note that T2VLA [47] is non-CLIP method.
Methods
Text-to- Video Retrieval
Vidco-to-Tex Retrieval
R@ If
R@5t
R@ 1OT
MdR,
MnR
R@ 11
R@5t
R@ 101 MdRj MuR
T2VLA [47] CVPR21
29.5
59.0
70.1
4.0
31.8
60.0
71.1
3.0
CLIP4Clip [34] Neurocoppiing22
44.5
71.4
81.6
2.0
15.3
42.7
70.9
80.6
2.0
11.6
X-Pool [17] CVPR22
46.9
72.8
82.2
2.0
14.3
44.4
73.3
84.0
2.0
9.0
TS2-Nct [36] BCCV22
47.0
74.5
83.8
2.0
13.0
45.3
74.1
83.7
2.0
9.2
EMCL-Net [22] NeurIP222
46.8
73.1
83.1
2.0
12.8
46.5
73.5
83.5
2.0
8.8
UATVR [16] JCCV23
47.5
73.9
2.0
12.3
73.8
83.8
8.6
DiCoSA [24] UCA22
47.5
74.7
83.8
2.0
13.2
46.7
75.2
84.3
2.0
8.9
ProST [30] ICCV23
48.2
74.6
83.4
2.0
12,4
46.3
74.2
83.2
2.0
8.7
HBI [23] CVPR22
48.6
74.6
83.4
2.0
12.0
46.8
74.3
84.3
2.0
8.9
DiffusionRet [25] JOCV2
49.0
75.2
2.0
12.1
47.7
73.8
84.5
8.8
EERCF [40]
47.8
74.1
84.1
44.7
74.2
83.9
MPT [56] ACM MM24
48.3
72.0
81.7
14.9
46.5
74.1
82.6
11.8
Bascline
46.6
73.4
82.2
2.0
12.6
73.4
82.4
9.6
GARE (Ours)
49.1
74.7
83.6
2.0
12.0
48.6
75.3
85.3
2.0
8.5
Table 2: Comparison results on DiDeMo, ActivityNet Captions, and MSVD datasets on Text-to- video is a non-CLIP method.
Retrieval. Note that FROZEN [3]
DiDeMo
ActivityNet Captions
MSVD
Methods
R@ R@
RØ 10 MnR
Methods
R@ 1 R@ 5 R@ 10 MnK
Methods
R@ 1
R@5 5 RØ 10 MnR
TS2-Net
41.8
71.6
82.0
14.8
CLIP4Clip
40.5
72,4
83.6
7.5
FROZEN [3]
33.7
64.7
76.3
CLIP4Clip
428
68.5
79.2
18.9
TS2-Net
41.0
73.6
84.5
8.4
CLIP4Clip
45.2
75.5
84.3
10.3
DiCoSA
45.7
74.6
83.5
11.7
DiCoSA
42.1
73.6
84.6
6.8
EMCL-Net .Nct
42.1
71.3
81.1
17.6
DiffusionRet
46.7
74.7
82.7
14.3
MPT
41.4
70.9
82.9
7.8
UATVR
46.0
76.3
85.1
10.4
HBI
46.9
74.9
82.7
12.1
HBI
42.2
73.0
84.6
6.6
Diffusion
46.6
75.9
84.1
15.7
Baseline
45.4
74.3
82.0
12.3
Baseline
40.2
72.5
7.5
Baseline
45.0
75.5
84.5
10.7
GARE (Ours)
47.6
75.4
83.1
12.0
GARE (Ours)
42.6
73.2
84.8
6.6
GARE (Ours)
46.4
26.1
84.5
10.6
3 Experiment
3.1 Experiment settings
Datasets and Metrics. We evaluate our method on four standard text-video retrieval benchmarks:
MSR-VTT [52], DiDeMo [2], MSVD [8], and ActivityNet Captions [27]. MSR-VTT contains 10K
videos with 20 captions each; we follow the IK- validation split. DiDeMo includes 10K videos
segmented into 5-second clips, each annotated with multiple sentences. MSVD consists of 1.9K short video clips with English captions. ActivityNet Captions provides dense annotations for 20K long-form videos with multiple temporally grounded descriptions. We choose Recall rank K=(1, 10) (R @ K), Median Rank (MdR), and Mean Rank (MnR) to evaluate the retrieval performance.
Implementation Details. We adopt CLIP (VIT-B/32) [38] as the base dual-encoder, equipped with a 4-layer Temporal Transformer [42] following the CLIP vision encoder for video encoding. Following prior works [34, 17, 30, 45], we use 32-word captions and 12 video frames for MSR-VTT and MSVD, and 64-word captions with 64 frames for DiDeMo and ActivityNet Captions due their longer video durations. We use the Adam optimizer [14] with linear warm-up. as in prior works. The learning rate set to set B = 0.07, train the model for epochs on MSR-VTT, MSVD, and DiDeMo, and epochs on ActivityNet Captions. All experiments are conducted on 4 to 8 GPUs including RTX 4090, A 100 and V100.
for CLIP's text and visual encoders, and 1 for all other modules. We 0.01, a = 2, and À = 0.5 for MSR-VTI ETT. All experiments use a batch size of 128.
3.2 Comparison with Other Methods
Table and Table 2 shows the performance of our method across four standard text-video retrieval benchmarks. As seen, our approach consistently outperforms recent oth methods on MSR-VTI ActivityNet, DiDeMo and MSVD.

---

Table 3: Ablation on losses combination on Text-to- Video Retrieval results on MTR-VTT Ik-A. First row denotes the baseline.
Table Ablation on Context Modality Choice of v. Text-to- video retrieval results on three datasets under different context modalities.
A Creaa Le Cai
R@It
R@st
R@ 10T
MnR
Dataset
|Context C
||@11 11
R@ST
10T
MnR
Baseline
46.6
73.4
82.2
12.6
MSR-VTT
Twoed
47.4
73,5
82.1
12.9
47.4
73.8
82.8
12.4
fimme
49.1
73.3
82.2
12.4
/
47.2
47.0
47.4
73.3
73.1
73.7
82.2
82.3
82.8
12.3
12.4
12.6
ActivityNet
T Tword
frume
42.6
40.2
72.2
73.6
83.6
84,4
8.1
6.8
48.3
74.2
83.2
12.4
DiDeMo
T word
46.5
74.3
82.6
12.3
49.1
74.7
83.6
12.0
frame
47.6
75.4
83.1
12.0
Table 5: Ablation on the interaction mode of da on Text-to-Vide Retrieval results on MSR-VT VTT Ik-A. The variant removes the relative gap modeling by using & as the query and V Viame as the key-value, producing ti and 4 = Bi t Our gap-aware design preserves pair-specific structure and yields superior alignment.
Table 6: Ablation on the IB prior r(A) on MSR-VTT Ik-A. Comparison between nor- malized and unnormalized A. distributions with different Gaussian priors.
R@If
R@51
R@ 101
MnR.
Normalized 1.0
47.8
74.5
82.1
12.9
Interaction Mode of
R@ I1
R@5t
R@ 101
MnR
Unnormalized A
0.1
47.7
73.4
82.2
12.9
Query to (no gap)
46.1
73.2
81.9
13.7
1.0
49.1
74.7
83.6
12.0
Query 2/ - f
49.1
74.7
83.6
12.0
10.0
48.1
74.6
83.5
12.0
100.0
48.6
74.7
83.2
11.8
3.3 Ablative Analysis
All ablation studies are conducted on the MSR-VTV Ik-A validation set. addition to the ablations presented below, we further provide results on the lower bound coefficient À of Le, the interaction between the context modality type C v and the direction indicator 77, the scale factor a of Carr. the choice of anchor in C and hard negative methods comparison in Appendix C.
Losses Combination. We conduct ablation studies on the MSR-VT VTT Ik-A validation set to assess effectiveness of proposed increment A and its associated regularizers. As shown in right of Table 3, directly injecting A into the InfoNCE flow improves performance from 46.6 to 47.4, validating the benefit of gradient tension release via pairwise adjustment. Introducing the relaxed information bottleneck (IB) loss further boosts performance to 48.3, highlighting its role in guiding A toward semantically meaningful corrections. In contrast, adding the norm constraint or the directional diversity regularizer alone yields no gain, since each Aij only optimized with respect to its corresponding sample pair and thus lacks inherent contrastive behavior; simply minimizing InfoNCE drive A toward trivial or collapsed solutions. The IB loss imposes a prior structure by restricting the optimization freedom of A, and within this semantically grounded structure, the norm and diversity regularizers become truly effective; without such semantic constraints, applying them to trivial solutions of would be meaningless. When combined, relaxed IB with norm and diversity regularization achieves the best performance (49.1), demonstrating that semantic grounding and structured regularization must work together to fully exploit the potential of A.
Impact of Cruss-attention Interaction Design. As shown in Table 5, we examine the effect of replacing d pair-wise gap-aware interaction with simplified query-key setting, where t, serves as the query and V frame as the key-value. This variant yields a fused representation t'y and residual
Aij 2 t'y. thereby removing explicit gap modeling between & and ey. Although it captures frame-level semanties aligned with f;, directly updating t; via Aiy leads to severe performance degradation, gradients near the loss side propagate to tA. = t, + - t'y. disturbing the anchor semanties. To mitigate this, we normalize A., to unit direction Anom and compute a similarity-ban -ased magnitude R. from frame-wise similarities between l and Uj. obtained through linear projection followed by exponentiation. The final correction la. = 1, + R Anom partially alleviates gradient interference but still lacks explicit semantic- gap awareness, preventing d from leveraging CLIP's

---

Irity
147
1.DE
122
23-
23
13
12
k2mm
(a) Cosine similarity distribution of positive pairs.
(b) Norm distribution on negative pairs.
Posiiiee sarplle va Originll
Detarce Change nea -1 40
Norm
(c) Norms distribution on positive pairs.
(d) Distance shift: to vs. f.
Figure 3: Qualitative analysis on the MSR-VTT Ik-A validation set. tatha denotes t.. Our method induces greater angular separation between positive pairs (a), redistributes la norms to release gradient tension (b, c); and pushes la outward from "y (d), promoting uniformity.
prior alignment- that explicit pair-wise gap modeling is crucial for robust cross-modal alignment.
Prior r(A) of Information Bottleneck. We further examine the effect of the prior distribution in the information bottleneck objective. A standard Gaussian prior is commonly used to regularize normalized embeddings, implicitly encouraging an isotropic distribution concentrated around hyperspherical shell in high-dimensional space. In contrast, our best performance is achieved when the unnormalized A. distribution is regularized against the same standard Gaussian prior. As shown in Table 6, normalizing Aij before KL regularization accelerates convergence but reduces final accuracy, as it constrains all increments the unit sphere and removes one degree of each A., from being regularized along its own radial axis. Moreover, since Ai operates in the unnormatized space to correct the anchor t4. preserving magnitude information is crucial for effective aligmment. We further tested priors r(A) NO0, o²I) with different standard deviation E (0.1, 10, 100} under the unnormalized setting, but none outperformed the standard Gaussian (a = 1), suggesting that the optimal prior variance is dete-dee,,,,n,, while the standard Gaussian provides a balanced regularization strength in our case.
preventing
Context Modality Choice in vh. We evaluate the effect of applying A.j to either modality across datasets with contrasting characteristics. As shown in the left of Table 4, on MSR-VTT, injecting A on the text side (i.e.. l + Aij) achieves the best performance, while applying it to the video side degrades results. This aligns with the fact that MSR- -TT contains many visually similar short clips, making it more effective to adjust the concise text representation for fine- fineggrained distinctions. Conversely, on ActivityNet, applying A to the video side leads to a notable performance boost, whereas modifying text harms results (R@ 1 ~ 40). This is likely because the long but redundant videos are paired with rich, structured captions- -making video-side adaptation more beneficial. These trends highlight the importance of aligning yts modality choice with dataset structure.
3.4 Qualitative Analysis
Geometric Properties of A. To understand the effect of the learned increment A, we conduct qualitative analysis on the MSR-VTT Ik-A validation set, focusing on its impact on representation geometry and alignment behavior at inference. As shown in Figure 3a, our method GARE yields

---

lower cosine similarities between positive pairs than the baseline, indicating larger angular separation and improved uniformity on the unit hypersphere [46].
Figures 3b and 3c show that the norm of the adjusted text embedding la consistently exceeds that of the original i, implying that A expands text representations onto a series of spheres of larger radius. Positive embeddings to., also have greater norms, consistent with our analysis of A A's iterative update behavior. Performing the first-order Taylor expansion around nonzero Á states (i.e., with nonzero initialization per iteration) mitigates logit-ranking distortion that occurs when expanding solely at zero. As negative increments Ay (j + i) are pushed outward, the positive Ais also expands to preserve relative belief masses [39] among logits, lowering overall cosine similarity while maintaining relative softmax probabilities.
Figure 3d further shows that la., lies farther from than 4, implying that the model does not simply reduce inter-modal distance but expands the text representation onto larger spherical shell for finer alignment. This suggests that, in cosine-based contrastive learning, encouraging greater dispersion of samples in the unnormalized feature space (i.e., higher pre-projection uniformity) may facilitate more effective alignment on the unit hypersphere.
Overall, these results indicate that optimization tension is effectively released: representation learning is no longer confined to the narrow region induced by the modality gap but occurs within a broader geometric space, thereby raising the upper bound of achievable alignment performance. Additional training- taniing-imeeaae analysis is provided in Appendix G.
Gradient Analysis. To further analyze how A redistributes optimization tension during training, we visualize the per- dimension gradient statisties of ta., at a representative training step (Figure 4). In dimensions with significant optimization activity, both positive and negative gradients reach magnitudes around x 2.5 and appear as approximate opposites, corre- sponding to the pair-specific gradient form in Eq. (8).
When aggregated over all pairs, these opposite gradients largely cancel out in the anchor's 's update Vr, Cr. leading to the near- zero gradient state described carlier. However, unlike 4, which aggregates signals from all pairs, each Aij only receives the gradient transmitted from its corresponding pair. Consequently. the positive increment AH receives an effective gradient of approximately +g. while each negative Ais receives around -9/ B. where B denotes the batch size. Since these gradients act independently and are not mutually canceled, the total non- vanishing optimization strength per anchor f, is approximately + + 1 B| = 2|gl. indicating that A components remain actively optimized rather than stagnant.
Figure 4: Mcan and variance of per- dimension gradients, indicating the pos- itive gradients (top) acting on ta and A; and the sum of all negative gradi- ents (hottom) for ta., and Dij-
This reveals that the trajectory of Aij reflects how l; explores the representation space. By distributing gradients across A, our framework effectively offloads optimization tension from the anchor, expanding its reachable region and breaking the locality constraint imposed by the modality gap.
Conclusion
We revisited contrastive learning for text-video retrieval from an optimization perspective, identifying two key challenges: optimization tension from the modality gap and gradient noise from false negatives. Through first-order Taylor expansion of the InfoNCE loss under trust-region constraint, we derived function space of increments Aiy that approximate descent directions. A learnable gap- aware network predicts Ai to redistribute gradients across pairs, expanding the optimization range beyond the modality gap and mitigating false-negatin noise. To ensure Aij encodes compact yet informative corrections, we employ variational information bottleneck with relaxed compression objective for stable training. Experiments on four benchmarks demonstrate consistent improvements, and further analyses show that the learned increments form structured, semantically meaningful geometry that releases optimization tension and enhances alignment capacity.
10

---

5 Acknowlegements
This work was supported by the NSFC 62172138 and No. 62202139. This work was also partially supported by the Fundamental Research Funds for the Central Universities NO. 00 and No. JZ2025HGTB0226.
References
[1] Alexander A Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy, Deep variational information bottleneck. arXiv preprint 161.v:16/..00410, 2016.
[2] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivie, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEFE international conference on computer vision, pages 5803-5812, 2017.
[3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to- end retrieval. In Proceedings of the IEEE/CVE international conference on computer vision, pages
2021.
[4] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Conditioned and composed image retrieval combining and partially fine-tuning clip-based features. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR) Workshops, pages 4959-4968, June 2022.
[5] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip- features. In 2022 IEEE/CVF Conference Computer Vision and Patiern Recognition (CVPR), pages 21434-21442, 2022.
[6] Simion-Ve Vlad Bogolin. Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. Cross modal retrieval with querybank normalisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5194-5205, 2022.
[7] Simion-Ve Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. Cross modal retrieval with querybank normalisation, 2022.
[8] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting the association for computational linguistics: human language technologies, pages 190- 200-200,220 2011.
[9] Tsai-Shien Chen, Wei-Chih Hung, Hung- Yu Tseng, Shao-Yr Yi Chien, and Ming- Yang. Incremental false negative detection for contrastive learning. arXiv preprint (Xxv:2/06.033799 2021.
[10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 042907, 2020.
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640-9649, 2021.
[12] Ching Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning, Advances in neural information processing systems, 33:8765-8775, 2020.
[13] Arthur P Dempster. Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (merhodologicall, 39-1):1-22, 1977.
[14] Kingma Diederik. Adam: A method for stochastic optimization. (No Title), 2014.
[15] Jianfeng Dong, Xianke Chen, Minsong Zhang, Xun Yang. Shujie Chen, Xirong Li, and Xun Wang. Partially relevant video retrieval. In Proceedingy of the 30th ACM International Conference on Multimedia, MM '22, page 246 257, ACM, October 2022.
[16] Bo Fang, Wenhao Wu, Chang Liu, Zhou, Yuxin Song, Weiping Wang, Xiangbo Shu, Xiangyang Ji, and Jingdong Wang. Uatvr: Uncertainty-adappive text-video retrieval. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13723-13733 2023.
[17] Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool, Cross-moda language- video attention for text-video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1006-5015, 2022.
11

---

[18] Bo He, Jun Wang, Jielin Qiu, Trung Bui, Abhinav Shrivastava, and Zhaowen Wang. Align and attend: Multimodal summarization with dual contrastive losses, 2023.
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceddings of the IEEF/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[20] R Devon Hjelm, Alex Fedorov, Samuel Swtwwii-Marrchldooo Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 06670, 2018.
[21] Chen Jiang, Hong Liu, Xuzheng Yu, Qing Wang, Yuan Cheng, Jia Xu, Zhongyi Liu, Qingpei Guo, Wei Chu, Ming Yang, al. Dual-modal attention-enhaneed text-video retrieval with triplct partial margin contrastive learning. In Proceedings of the 31 ACM International Conference on Multimedia, pages 4626-4636, 2023.
[22] Peng Jin, Jinfa Huang. Fenglin Liu, Xian Wu, Shen Ge, Guoli Song. David Clifton, and Jie Chen. inseetation-mmxximzaann contrastive learning for compact aadeo-anddlanguage representations. Advances 2022.
in neural information processing systems, 35:30291
[23] Peng Jin, Jinfa Huang. Pengfei Xiong, Shangxuan Tian, Chang L Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2472-2482, 2023.
[24] Peng Jin, Hao Li, Zesen Cheng, Jinfa Huang, Zhennan Wang, Li Yuan, Chang Liu, and Jie Chen. Text-video retrieval with disentangled conceptualization and set-to-set alignment. arXiv preprint arXiv:2305.12218, 2023.
[25] Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji. Chang Liu, Li Yuan, and Jie Chen. Diffusionret: Generative text-video retrieval with diffusion model. In Proceedings the IEEE/CVF international conference on computer vision, pages 2470-2481, 2023.
[26] Weike Jin, Zhou Zhao, Xiaochun Cao, Jieming Zhu, Xiugiang He. and Yueting Zhuang, Adaptive spatio- temporal graph enhanced vision-language representation for video qa. IEEE Transuctions on Image Processing, 30:5477-5455 2021.
[27] Ranjay Krishna, Kenji Hata, Frederie Ren, Li Fei-Fei, and Juan Carlos Nichles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706-715, 2017.
[28] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):99 86, 1951.
[29] Kuang-Huch Lec, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked eross attention for image-text matching, In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[30] Pandeng Li, Chen- Wei Xie, Liming Zhao, Hongtao Xie, Jiannan Ge, Yun Zheng, Deli Zhao, and Yongdong Zhang. Progressive spatio-temporal prototype matching for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4100-4110, 2023.
[31] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:1 7612-11765, 2022.
[32] Zengrong Lin, Zheng Wang. Tianwen Qian, Pan Mu, Sixian Chan, and Cong Bai, Neighborretr: Balancing hub centrality in cross-modal retrieval. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9263 9273, 2025.
[33] Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. In European conference on computer vision, pages 319-335. Springer, 2022.
[34] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lci, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing. 508:293-304, 2022.
[35] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to-end multi- grained contrastive learning for video-text retrieval, 2022.
12

---

[36] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic. and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos, 2020.
[37] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint #xxiv/807.037466 2018.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[39] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems, 31, 2018.
[40] Kaibin Tian, Yanhua Cheng, Yi Liu, Xinglin Hou, Quan Chen, and Han Li. Towards efficient and effective text-to-video retrieval with coarse-to-fine fine visual representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5207-5214, 2024.
[41] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0000057, 2000.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[43] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
[44] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2495-2504, 2021.
[45] Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, and Xian-Sheng Hua. Disentangled representation learning for text-video retrieval. arXiv:2203.07111, 2022.
[46] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pages 9929-9939. PMLR, 2020.
[47] Xiaohan Wang, Linchao Zhu, and Yi Yang. T2vlad: global-local sequence alignment for text-video retrieval. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7799-508, 2021.
[48] Yimu Wang, Xiangru Jian, and Bo Xue. Balance act: Mitigating hubness in cross-modal retrieval with query and gallery banks, 2023.
[49] Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarsc-to- to-nne alignment for video-text retrieval, 2023.
[50] Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018
[51] Hu Xu, Gargi Ghosh, Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre- me-training for zero-shot video-text understanding. 2021.
[52] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288 5296, 2016.
[53] Xiangpeng Yang. Linchao Zhu, Xiaohan Wang, and Yi Yang. Dgl: Dynamic global-local prompt tuning for text-video retrieval, 2024.
[54] Can Yaras, Siyi Chen, Peng Wang, and Qing Qu. Explaining and mitigating the modality gap in contrastive multimodal learning. arXiv preprint arXiv:2412.07909 2024.
[55] Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. End-to-end concept word detection for video captioning, retrieval, and question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3165-3173, 2017.
[56] Haonan Zhang, Pengpeng Zeng. Lianli Gao, Jingkuan Song. and Heng Tao Shen. Mpt Multi guatied prompt tuning for text-video retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1206- 1214, 2024.
13

---

Appendix
This appendix provides additional discussions, derivations, and analyses that complement the main paper. We first discuss the limitations of our current framework and provide related works for broader context. Then, we present extended ablation studies and comparisons, followed by detailed mathematical derivations, including the relaxed form of our variational information bottleneck (VIB) objective and the rationale for using a provide implementation and efficiency details of the GARE model to support reproducibility and deployment.
state in the multivariate Taylor expansion. Finally,
A Limitations
While our method successfully reduces the tension between text and video embeddings by optimizing pair-wise Aij based on the direction that minimizes the InfoNCE loss, several limitations remain:
Lack of modality alignment. The fundamental issue of modality gap persists. Despite reducing the tension between the embeddings of text and video, the two modalities still reside in completely disjoint regions of the representation space. Our approach alleviates this problem by relcasing the optimization tension exerted on anchor representations, thereby expanding the effective optimization range of each text anchor through multiple pair-specific increments Dij- As result, the method mitigates the effects of the modality gap and reduces some noise, but it does not address the root cause of the misalignment between the modalities.
Lack of generalized supervision for A.j. Our approach relies heavily on adjusting AÚ through gradient-based optimization, but Aiş lacks a more generalizable supervision signal. Currently, we optimizing each pair-wise Aij based solely on the gradient of the InfoNCE loss, which only loosely guides the optimization direction. While this helps alleviate the tension between positive and negative pairs, it does not provide a stronger supervisory signal to guide the model toward better generalization across unseen data.
B Related Work
Contrastive learning and modality gap. Contrastive learning has become a foundational paradigm in multimodal representation learning. Wang and Isola [46] formalize contrastive learning via the principles of alignment and uniformity on the hypersphere, offering a geometric perspective on representation quality. Wang and Liu [44] show that contrastive loss is hardness-aware and cemperaatue--sensttvve, and reveal a trade-off between representation uniformity and semantic toler- ance, highlighting the need to preserve meaningful structure among semantically similar samples. Liang et al.[31] investigate the modality gap in multimodal contrastive learning and attribute it to initialization imbalance and cone effects. False negatives have also been recognized as a key challenge in contrastive learning [12, 9], with solutions ranging from reweighting and elimination to dynamic detection and correction.
Text-video retrieval. Text-video retrieval is one of the prominent tasks [15, 26, 43, 4, 18, 51, 36, 5, 24, 29, 48] in cross-modal learning. The majority of existing research [30, 16, 25, 35, 49, 733 in this area utilizes mapping technique that aligns both text and video inputs within shared latent space to facilitate direct similarity assessment. CLIP4Clip [34] is the first to adapt CLIP [38] for video-text retrieval via temporal frame aggregation. TS2-Net [33] improves temporal modeling through token shift and selection. X-Pool [17] introduces text-guided pooling to highlight salient video tokens. HBI [22] values possible correspondences between frames and words using Banzhaf interaction for sensitive and explainable cross-modal modal alignment. EMCL [22] introduces an expectation-maxxiizztioo [13] framework learn a compact latent space where video and text features are represented as linear combinations of shared bases. This decomposition reduces the rank of the latent space, mitigating the modality gap and enhancing semantic alignment. Unlike prior works that refine matching structures, our method analyzes the gradient form of InfoNCE and introduces a learnable gap-aware increment A.j to offload optimization tension, enabling structured optimization in trust-region-aweee aware formulation.
14

---

C More Ablation and Comparison Experiments
In the next four experiments conducted on the MSR-VTT IK-A dataset, we investigated the design of network inputs, the impact of the scale coefficient a and the orthogonality variant in the directional diversity regularization, the impact of the lower bound À of Norm-Based Regularization of Trust- Region Radii, and the impact of the anchor choice of Crelax We also provide hard negative methods comparison.
C.1 Ablation on the Design of vb Network Inputs
We analyze the design of the v network from three com-
Table Ablation on context C and se- mantic gap under different correction modes. 7- denotes v-t.
plementary aspects: 1) the context used for conditioning, 2) the semantic gap direction 77, and 3) the corrected an- chor (either i or v). When correcting the text anchor (!), using the video-side sequence (v) as context yields bet- ter performance; conversely, when correcting the video anchor (v), using the text-side sequence (t) as context per- forms better. In both cases, the semantic gap is defined v - t, suggesting that the query direction asymmetric. Overall, performance is mainly determined by the inter- play between the corrected anchor and the chosen context: cross-modal conditioning (e.g., correcting with as con- text) consistently outperforms uni-modal configurations, highlighting the benefit of cross-modal fusion.
7 |R@If R@51 R@ 101 MnR,
I-corrected
Ward
46.6
74.2
83.8
12.9
wond
48.2
74.2
83.2
12.9
frame
47.8
74.1
82.8
12.4
frame
49.1
74.7
83.6
12.0
v-corrected
frame
47.7
73.2
82.9
13.1
Trame
+1
47.5
73.5
81.8
13.0
Moreover, defining the semantic gap as v - t achieves better results than t fact that video features tend to capture more shared and general concepts, while text features encode instance speciffi information. Removing text-specific components from the video representation thus encourages more generalized and robust semantic alignment.
Ward
48.2
73.6
82.8
12.7
This can be attributed the
word
48.8
73.2
83.6
12.9
C.2 Ablation of the Scale Coefficient and the Orthogonality Variant in Directional Diversity
To further analyze the impact of the scale coefficient a in the directional diversity regularization, conduct an ablation study as shown in Table 8. Recall that the loss adopts a log- mean-exp form:
Table 8: Ablation scale coeffi- cient a for directional diversity.
R 11
R@5t
R@ 101
MnR
0.5
47.6
74.5
83.4
12.1
Lai E log E,, [exp (-a (1 (Zag. =as 101
1.0
48.5
74.4
83,9
12.1
2.0
49.1
74.7
83.6
12.0
3.0
46.2
74.4
82.9
12.1
where Σig = Au denotes normalized directions under the 2
47.0
74.3
83.4
12.1
5.0
47.2
74.1
82.8
12.4
same anchor f;. The term (1 (#iy, 2xk) lies in 2), and the log-mean-exp operator favors pairs with higher cosine simi-
larity (i.e., smaller angular distance), encouraging local angular diversity without enforcing strict orthogonality.
The coefficient adjusts the strength of uniformity: 1) When a is small, the exponential term emphasizes pairs with higher similarity, focusing optimization on local clusters of Ai directions. 2) As increases, the optimization becomes more uniform across pairs, driving the distribution of Ag directions toward isotropy.
Empirically, shown in Table 8, performance peaks when a 2.0, achieving the best R@ R@S and MnR. When a grows larger, R@ I and R@5 exhibit reduced variance, indicating that the model becomes less sensitive to angular differences. This aligns with our interpretation that large values saturate the exponential cerm-nuusing exp(-ax) diminishing the discriminative effect among diverse directions. Overall, moderate achieves the best trade-off between directional diversity and training stability, effectively preventing directional collapse while maintaining meaningful variation across A.j.
(zxj 25x))) to approach zero uniformly-thus
15

---

To further enhance the distinction among Ai directions, we additionally experimented with strict orthogonalization objective:
B (2i 2ik)
Fott
which enforces all direction pairs under each anchor t; to be mutually orthogonal. Unlike the log-mean-exp formulation, this loss treats both high- and low-similarity pairs equally and aims to drive all pairwise cosine similarities toward zero. However, this uniform orthogonal- ity constraint yields inferior performance (R@ 1=4.77 R@5=74.3, R@ 10=83.1, MnR=11.1). We attribute this degradation to the excessive suppression of negatively correlated direc- tions-pairs that are semantically opposite or far apart are also pushed toward zero similarity. This rigid treatment prevents Aij from encoding opposing semantic relations, indicating that directional diversity should remain seman- tically flexible rather than uniformly orthogonal.
Abtet on & Norm Regularization Strength [AI
Nevertheless, the orthogonalization approach exhibits stronger performance on video-to-text retrieval (R@ 1=49.4, R@5=75.4, @@ 10=83.6, Mean R=8.2). We conjecture that this improvement stems from the fact that the directional diversity regularization is ap- plied with text as the anchor over candidate videos, where enforcing orthogonality among Aij encourages the model to better separate diverse visual counterparts for the same textual query.
48.
43.
05
Figure 5: R@@ score with vary- ing À for A norm regularization.
C.3 blltion on Norm Regularization Strength
We investigate the effect of varying the lower bound factor A, which controls the target margin of separation within each anchor fę. As shown in Figure 5, increasing À from 0.1 to 0.5 improves R@ 1, with the best performance at À 0.5. This suggests that moderate diversity in eij is beneficial for enhancing semantic discrimination. However, as increases further, performance degrades, likely due to over-regularizztion.
To mitigate the instability introduced by hard truncation (i.e., directly thresholding Eij). we also experiment with a smooth approximation using a log-sum-eyt formulation:
LE-ISE log 1
1 B
Σ
exp - (eig - -
Although this variant imposes a natural lower bound and provides vanishing gradients near con- vergence, it does not significantly improve retrieval. We hypothesize that this is due to gradient saturation when the variance among Eiy becomes too large, which in turn weakens the ability to further enforce directional separation.
C.4 Ablation on the Anchor Choice in Relaxed VIB Regularization
We further investigate the effect of the anchor choice in the relaxed information bottleneck term Crelax In the main formu- lation, the KL. regularization is anchored on videos, i.e.,
Table 9: Effect of anchor choice in KL regularization Crelax Anchor- ing on yields better performance.
() (A ([r)) 2 (KTT)
Anchor R@@ R@5 R@
MdR MnR
where q. (A) E.,, Igw (Alt. v)]. This relaxation aggregates increments A.j i= under each video anchor v and preserves the information information-bootlenees bottleneck effect while reducing sensitivity to the variability of short text inputs.
47.9
74,9
83.8
2.0
12.5
49.1
74.7
83.6
2.0
12.0
To contrast this design, we also test the reversed relaxation that anchors on text:
Ee, (KL qu A v) [|r())]] > (A t) [[])))
where To (A)t) = F. ( (A v)). Practically, this means that the set [A:))_ associated with
each text anchor t, is modeled Gaussian posterior, yielding the corresponding compression loss.
16

---

Querya
Query: person putting the vegetable
Baseline: 0.2400
GARE: 0.2281
Baseline: 0,2729
GARE: 0.2488
Baseline: 0.2349
GARE: 0.2276
X
Baseline: 0.2554
GARE: 0.2410
Caption: woman talking to man interview
Caption: cooking recipe chicken vegetables
(a)
(b)
Figure Comparison hard negative aligmment before and after applying Ai optimization. Compared with the baseline, GARE produces smaller similarity gaps among semantically related videos Vj. This indicates that GARE effectively mitigates the noise from hard negatives and reduces the semantic deviation of the anchor 1,, leading more stable and consistent alignment across similar samples.
Notably, our relaxation introduces stochasticity directly from the dataset pairing itself, rather than by sampling additional uncertainty variables.
As shown in Table 9, anchoring the KL term on the video side leads to superior retrieval performance (R @ 1=49.1 vs. 47.9). We attribute this to the inherent data characteristics: the textual descriptions in the dataset are typically short and less informative, while videos are semantically richer and often correspond to multiple captions. Consequently, conditioning A captures more shared, modality-invand invariant structure, aligning better with the underlying multimodal distribution and yielding stronger retrieval results.
C.5 Hard Negative Comparison
We compare GARE with two recent methods that explicitly or implicitly handle hard negatives:
DMAE [21] (R@ 1: 46.9, +1.6% over base. ACM MM 2023): DMAE enhances fine-grained alignment by mining hard positives- for instance, text queries associated with specific frames-whicl which implicitly improves the model's capability separate hard negatives. Conceptually, this shares similarity with our variational information bottleneck (IB) loss, where A is compressed through bottleneck to retain critical alignment signals while filtering noisy gradients.
NeighborRetr [32] (R@ 49.5, +2.3% over base. CVPR 2025): This method employs a memory bank to compute k-neighbor co-occurrence statisties, identifying "good hubs" that encourage local consistency and reduce over pee-pennaizztton of hard negatives. Although not explicitly formulated hard- herd-negative mining, the top-k co-occurrence selection serves similar role by adaptively reweighting difficult negatives.
GARE (R @ 1: 49.1, +2.5% over base): Unlike the above methods, GARE does not rely on explicit mining or external memory struc- tures. Each increment Aij absorbs loss gradients locally from its paired (f,, U2) sample, mitigating the reliance on global contrastive comparisons when encountering hard negatives. This local realloca- tion of gradients softens noisy updates and improves generalization. As illustrated in Figure 6, the text anchor in GARE exhibits en- hanced semantic stability: for semantically related videos Uy, the similarity values 8a and 8ij vary smoothly, indicating that Aij en- ables more accurate fine-grained alignment across similar samples. This aligns with the gradient visualization in Figure 7, where the gradients on f,'s dimensions remain centered around zero after introducing A, demonstrating its role as a stable semantic prototype.
Figure 7: Mean and variance of total gradients acting each dimension.
Efficiency comparison. GARE maintains superior efficiency despite comparable or higher perfor- mance. It uses only single cross-attention layer, whereas NeighborRetr includes 8 MLP modules and
17

---

multiple transformer or convolutional blocks. NeighborRetr also requires large-scale memory banks (10,240 samples per modality) and ~4.5h training time, while GARE achieves similar accuracy with 1 h 34 min of training and negligible additional memory cost compared to the baseline. Together, these results demonstrate that GARE achieves comparable hard-negative robustness with substantially lower computational overhead.
D Derivation of the VIB Objective for Pair-Specific Increments
We model each pair-specific increment as latent variable Z = Aij for the input X = (4,, 2/) and the match label Y E {0,1} Our objective maximizes predictive information under compression constraint:
max I(Z;Y)- -II(Z;),, B> O.
(16)
p(=
Below derive tractable variational lower bound of Eq. (16).
Lower bound for [(Ayy: Y). For convenience, in the following, we will refer to Vij, Aij. 4, and Vj as y, A, i, and v respectively. We make the standard assumption that the match label does not depend on A once v)) is given:
p()A,1,0) = p(ul(.))
(17)
By definition,
I(A:)) // p(A, v) log
A
dAdy
p(y)
/| p(A, v) log
(o(v)
A(ylA)
dAdy
p(y)
qe (y
// p(A, v) log
(0(3
dAdy + KT. KT. (p.l
p(v)
2 /ʃ p(A, v) log
(0() dAdy (by non-negativity of KL)
ply)
(18)
// p(A
v) v(A,v) log 9o dAd H(Y)
2 [/ plu, A, 1, log go A) dddyd((,
/Iʃ plyl,, v) w) (V, v) pl,, v) log go (y/ ()) dAdyd(I,
Barqu log go (v)
where A)) is variational classifier, the negativviy entropy term H(Y) constant w.r.t. model parameters, and the last second equality uses assumption Eq. (17). Let qu (A]t, u) be the variational encoder. Then,
I(A:v) 2 E(t,o Ba log go (v)
(19)
The two inequalities above arise from the non-negatyvity of KL(-II.) and from rewriting log p()) as the entropy H (y).
18

---

Upper bound for ((y t, ey). Using r(A; t, I (L.p), with any auxil- iary prior r(z),
4(); = /| ((4,4,) log
p t,
dAd(L,
((A) p(t,u)
/| log
p(A 1, v)
dad(t,u)
P(A)
/] p(A,t,u)
p(Alt, 2) r(A)
ddd(t,
r(A) p(A)
(20)
// ((A,4,))og log
p() ddd(L,0) - (T.,K))(())))((
r(A)
|/ p(),t,
p() v) ddd(t,
r(A)
(e.))) (TT
where the inequality follows from (L(p(A) 2 and p(Alt, v) denotes the variational encoder qu (Alt,, v) that parameterizes the model's conditional distribution of the pair-specific increment (deterministic in our implementation). We use a diagonal Gaussian prior r(z) = (),
VIB lower bound. Combining Eq. (18) and Eq. (20) yields the variational lower bound of Eq. (16), i.., the upper bound of the negative of Eq. (16):
LvBs (t. E. (A log go (y) A)] + + 8- BBe..) (A)t,
(21)
E
Relation between the Relaxed VIB Compression Term and the Original IB Objective
We start from the standard information bottleneck (IB) formulation, which regularizes the mutual information between the learned increment A and the input pair (t, v):
I(A:T,V) = Ep (K t, t, ll go (A))).
(22)
Since the marginal ao (A) is intractable, it is commonly replaced by fixed prior r(A), leading to the following upper bound:
(1.)) (KT v,, || [|]) I(A;T, (T((+(A) [|r(A)).
(23)
This term penalizes the total amount of information A retains about both modalities and serves the compression loss in the original VIB objective.
Relaxation anchored on the video side. To mitigate sensitivity to text-side variability, we anchor the expectation on videos and aggregate increments over all textual pairs associated with the same v. By convexity of the KL divergence (Jensen's inequality), we have:
(1,) (A 4, [[])]] E. E.j (T (A (t,u) [|(()
(24)
E, KK go AA v) [[](]))],
where the aggregated posterior 7.(4 v) E, ( t, r) represents the mixture distribution of increments under the same video anchor.
Applying the decomposition in Eq. (23) to the right-hand side of Eq. (24) yields
E, (LL(K (A v) r(A)) = I,(A;)) KL (A) r(A)),
(25)
where I,(A; V' denotes the mutual information defined under the aggregated distribution qu. Com- bining the two equations gives
V(A;T.V) > V))
(26)
indicating that the relaxation effectively removes the conditional term T(A:7) V) from the full IB regularization.
19

---

Interpretation. The relaxed KL term therefore optimizes
Ctelux = E. (A U) || 4 + KL(g (( [[(())),
(27)
which serves as lower bound of the original compression term. While the standard IB loss penalizes all information in A about (T, V), the relaxed form only constrains information shared with V and ignores the conditional mutual information I(A: V). Intuitively, this relaxation preserves the bottleneck effect but allows A to encode text-specific variations within each video cluster, yielding more stable optimization when multiple captions correspond the same visual content.
F
Overall Objective and Inference
The overall training objective is formulated as
Liotal
Linfo + B CTB the Le + Auir Lait,
(28)
Lvs
where the first two terms constitute the VIB optimization objective, and λE and Aur are the weights of the structural regularizers. In practice, we set 3 = 0.07 and A₂ = Adi = 0.01 for MSR-VTT VTT. During encodes the semantic consistency
inference, we retain the learned increment to assist retrieval, between 4 and Uz. We further discuss the efficiency of using A at both training and inference stages in the Appendix as well as its distinction from traditional dual-eneoder retrieval paradigms and deployment strategies for large-scale retrieval.
G Analysis of A Behavior During Training
To better understand the role and dynamics of the learned semantic-gap vector A throughout training, we visualize four key aspects of A's behavior, presented in Figure 8a to 8d. These analyses reveal the underlying geometric transformations and provide further insight into how A facilitates optimization under InfoNCE loss. We will continue conducting ablation studies on the Text-to- retrieval task using the MSR- VTT [52] dataset.
Angle between A and (u, t.). As shown in Figure 8a, track the angle between A and the initial cross-modal gap vector f throughout training, for both positive and negative pairs. At initialization, this angle is close to 5 indicating that A is nearly orthogonal to uy vy-t,, t, and thus carries no meaningful alignment with the cross- modal semantic gap. This implies that the early A vectors do not differentiate between positive and negative pairs, acting more like isotropic perturbations in space. As training proceeds, however, this angle gradually increases into the obtuse region for both positive and negative samples. This reflects significant shift in behavior-ratie
than attempting to directly bridge the semantic gap vy - 4, the model learns to push 4 away from Uj. effectively offloading the gradient tension induced by the modality gap and false negative interference. This offloading allows contrastive learning to take place in an expanded embedding space, where A modulates the representation geometry ease the optimization burden. This trend corroborated by Figure 8d, where the Euclidean distances between the updated text embeddings tA.; f4 + Aij and vy become significantly larger than the original distances ||t ey |l. for both positive and negative samples. That is, A introduces a global scaling effect in the representation space, and contrastive optimization is carried out under a larger geometric regime.
Interestingly, the new positive pair distances || w are also larger than their original counterparts ||t, - vill- Though this seems counterintuitive- since the gradient of InfoNCE with respect to A pushes toward tù it actually reflects the relative nature of the InfoNCE loss. The network does not aim to minimize absolute distances, but rather to increase the similarity margin between matched and mismatched pairs. Thus, pushing all embeddings outward in norm (as further validated in Figure 8c) gives the model more room to maneuver in angular space, while the cosine similarity objective remains stable under such rescaling. This aligns with our design intent: to leverage A as structural carrier for modality-awee aware tension redistribution, providing optimization flexibility on a normalized manifold.
Moreover, early training stages, the angles between A and v,- - 1, are nearly identical for positive and negative samples, showing that the model has not yet learned to encode fine-grained pairwise
20

---

Angle between and (v)
Angle between and ti
SOON Stngs
1.80
1.60
2.64
1.4
1.02
2.60
1.3
2.50
1.2
1.50
raining
(a) Angle between A and (0) 4)
(b) Angle between A and &
Norms t new
Pair Distances: tolddttt_ooee new <->
26
(000
Unow (neg)
Mo ManlMannn Myw Mammammnn Mawn
25
Budidian Distance
bm wmm ww 9
22
23
3000
4000
Trairing 5000
Thaiiing
(c) Norms of and fa.s
(d) Pair distances: t/tta f2
Figure 8: Training dynamics of the learned modality-gap vector A. fnew denotes to (a) A grows increasingly orthogonal the initial modality gap, indicating embedding space expansion. (b) A initially aligns with the anchor f, then deviates to encode semantic distinctions. (c) Updated embeddings faew operate under larger norms, enlarging the contrastive space. (d) Pairwise distances increase and then stabilize, reflecting semantic separation and convergence.
differences. But training advances, slight but consistent gap emerges- A for positive samples tends to have slightly smaller angles than that of negatives. This subtle divergence signals that A has begun capture semantically meaningful pair-level distinctions, enabling more discriminative alignment in later stages.
Angle between A and ty. As shown in Figure 8b, the angle between A and f; exhibits a distinct pattern: it decreases rapidly in the early training phase, indicating that A is initially aligned with the anchor text embedding. This suggests that the model's default behavior is to trust the prior structure of the text space and apply similar A directions across different Uy. especially when pairwise semantic differences are not yet learned.
However, over time, this alignment loosens-4 A deviates further from the model begins adapt to modality-speciin differences. This marks a transition where the network ú no longer treats the anchor as default direction and instead generates based on nuanced distinctions across the visual features Uj. thus leading to more expressive and semantically grounded vectors.
Pair distances between text and video features. Figure 8d visualizes the Euclidean distances between text and video pairs over the course of training, comparing both the original pairs f, < Uy and the updated pairs ta., ++ Uj. across positive and negative samples. Several important patterns emerge: First, we observe that positive pair distances remain consistently smaller than negative pair distances, both in the original and updated forms. This confirms that the learned A not only preserves the basic alignment structure of contrastive learning but also enhances semantic discrimination, effectively encoding fine-grained pairwise structure through the transformation. Second, for both positive and
21

---

negative pairs, the distances between ta and v are larger than those between the original t and demonstrating that the model pushes the updated text embeddings into a larger-scale embedding regime. This is precisely in line with our gradient tension offloading design: A introduces a controlled displacement that alleviates the direct optimization pressure on 4,, allowing contrastive comparisons to operate within a higher-norm, more expressive space, also supported by Figure 8c. The evolution of the distance curves over time also reflects meaningful training dynamics. Initially, all distances decrease, which we interpret noisy and unstructured, and the model first compresses them into a tighter, more consistent geometric configuration. This is followed by steady increase in distances, the model begins to explicitly separate positive and negative samples to satisfy the InfoNCE objective. Finally, all curves converge and stabilize into bounded range, suggesting that the semantic configuration of the embedding space has reached a relatively converged structural state.
normalization phase embedding distributions at initialization are
Taken together, these observations highlight the critical role of Á not only in scaling the embedding geometry but also encoding structurally-awe aawaee, semantically-aligne displacements that facilitate tension redistribution and enable robust representation learning.
H Why We Don't Use an Initial Aix =0 Expansion and the Need for a Non-Zero Initial A
H.1 Introduction to the Issue
In the previous approach, we considered using multiple variables Aij to distribute the gradient impact of each video V/y on the text embedding 4, However, the specific form of A.j was unclear. The idea was to update each Aij in the direction that would reduce the InfoNCE loss. A natural first step was to expand the loss function L, as Taylor expansion around the initial state A,, 0, which corresponds to expansion solely based on 4;.
However, this expansion introduces key issue: the gradient terms in each Aij expansion are computed under the assumption that all other As remain zero. This neglects the mutual interactions among (Ary 1 and decouples the updates for different pairs. Since the InfoNCE loss fundamentally depends on the relative ranking of cosine similarities between &; and all candidate videos {ey}, such decoupled formulation disrupts the prior ordering structure already established by the CLIP encoder's logits. In other words, it breaks the model's s inherent relational prior among candidates, which is essential for maintaining consistent contrastive comparisons.
H.2
Why Expanding with A,, 0 Is Inadequate
When i = 0 for all j. the gradient expansion for each pair Aij looks like:
(A())) (,( + VA, (.) Asy
(29)
where VA., (00) is the gradient of the loss evaluated at Aij = 0. In this case, the gradient Pij Vij computed as:
VA., (,( (p:;(0) - Vij) VA., (0);
(30)
where Ptj (0) is the softmax term evaluated at the cosine similarity cos(4,, v2). This expansion does not account for the interdependence between different Auk, and the gradients are computed as if each Ai were independent of all other At. which violates the relative comparison required for contrastive learning.
H.3 The Need for a Non-Zero Initial A;,
To avoid this issue, we recognize that each A,y should be updated in way that reflects the interde- pendence between different pairs, not just based on an independent text-video pair. The gradient of the InfoNCE loss must be evaluated with respect to the current state of all Aijs not just the initial zero state.
22

---

Thus, we need to initialize A5x in a way that reflects the current state of all other pairs rather than assuming they are zero. The update for each A, should respect the relationship between all the pairwise similarities, as the optimization of one pair affects the relative similarity between all other pairs.
In this case, the correct approach is to perform the first-order Taylor expansion around a non-zero state A that has already been optimized for several steps. This allows to incorporate the effect of each pair Ai on the entire set of pairwise comparisons, thus preserving the relative relationships that are critical for InfoNCE loss.
H.4 Optimization Objective and Trust-Region Constrained Solution
Having established the need for coupled, non-zero initialization of Ag to preserve the relative struc- ture required by InfoNCE, we now formulate the optimization problem for finding the approximate update direction under a trust region constraint. Specifically, we assume trust region radius e that bounds the magnitude of each Dij and analyze the solutions in both the single-step (non-iterative) and iterative update settings.
Non-iterative first-order update. We begin by considering the first-order Taylor expansion of the per-anchor loss L; with respect to single variable Aij evaluated at the origin:
(A (0) VA,, (i)) Agj.
(31)
Since the constant term (0) does not affect optimization, minimizing the approximated loss is equivalent to minimizing the linear term under a trust-region constraint:
min |A
VA. (00)
(32)
where denotes the radius limiting the update magnitude of A.y. By the Cauchy-Schwarz Schwarz inequality, for 9ij VA., (0) we have
2- الروا || 2 - 199,,11,
(33)
which provides tight lower bound for the objective within the trust region. The bound is attainable when A. is colinear with -gy and satisfies
|| | = E, yielding the feasible update
A = J&
(34)
This closed-form solution achieves the maximal decrease in the first-order approximation of C,, corresponding to the steepest descent direction constrained by the trust region. However, this derivation treats L; as a function of a single A.j and neglects the coupling among { } thus breaking the relative similarity structure essential to InfoNCE. A more faithful formulation must therefore incorporate all Aij jointly, as discussed in the following subsection.
Iterative update with coupled expansion. We next analyze the update rule when Aiy is expanded around a coupled, non-zero state (t) {
LB Jz=1, where each gradient VA,,, Li is computed under the influence of all other nonzero A ik We seek the next update step within a trust region:
(t)
((-1) A (4)
+ 8,
|| (e+1) Se.
(35)
The first-order Taylor expansion of the per-anchor loss L, at this coupled state is:
B = ( Σ ((A)))
(A
- -
(36)
This multivariate linearization serves to reveal that each partial gradient VA, (A() conditioned on the current coupled state, and thus inherently encodes the interactions among all pairs. We do not minimize the linearized approximation itself; rather, it clarifies that the correct descent direction for each Ay should be taken from the true gradient of L3 evaluated at the coupled state.
23

---

Following the principle of steepest descent, we update each component by
VA Ci VA
b = ) oif Dis
1++1)
9ij
(37)
To satisfy the trust-region constraint | from
(1+1) S E, we determine the maximal feasible step size aij
(0)
(t)
Dij Dij
(4) (2)
1
(38)
which expands to the quadratic inequality:
oig 20
(1) (1)
T
Dij
1| 2 S.
(39)
The maximal feasible solution is given by:
= 1 (t)
Dis (A (D) 0 +
(40)
This update ensures that each A moves along its steepest descent direction while keeping its magnitude within the trust-region radius E. Importantly, because all gradients Va., (4 are computed under the coupled multivariate state, the optimization preserves the relative ranking structure among all candidate pairs required by the InfoNCE objective.
H.S
Error Analysis
The iterative update of A. is derived from a multivariate first-order Taylor expansion of L, under a trust-region constraint. This formulation defines a descent function space that guarantees loss reduction under the coupled gradient field, rather than prescribing unique update direction for each A.j. Once the trust-region radius Ety is specified (assuming an f2-ball constraint), the steepest- descent direction becomes unique, and the theoretical update moves Aij onto the boundary where the Bauchy-Schward inequality reaches equality. However, in practice, we do not explicitly assign radius Eij each pair; instead, the effective trust-region constraint emerges implicitly from the model's 's prior, which we regularize through a norm-based penalty that controls the magnitude of A. Specifically, the Aij produced by the v network during the forward pass corresponds the current
state A bj and the optimizer update during backpropagation embeds the next-step state A (++1) into the parameters of vb. When the model performs the next forward pass, the generated Aij naturally reflects this updated state, while the norm-based regularization acts as the trust-region constraint on the previous update step. Therefore, the theoretical and practical procedures share the same steepest-descent direction, but their update magnitudes may differ due to the implicit learning the trust-region radius through model priors. Consequently, our error analysis focuses on the consistency of update magnitudes between the theoretically derived trust-region step and the actual updates produced by the training framework.
Analytic step length. At the equality boundary of the trust-region constraint, the maximal feasible (t+1)
step size (t) satisfying |
S e given by:
1j
i
11 12 =2,
(41)
which projects the update exactly to the boundary of the feasible region and enforces norm-bounded motion of Aij-
Learned step length. During training, the neural module sé is optimized end-to-end via AdamW, implicitly learning how to adjust Ai through gradient descent. After one step, the actual update can be expressed as:
(++1) A ( = (1
(t)
VO0 C.
(42)
V
1
where (0) denotes the effective step size produced by the optimizer. Although A, is updated through the training framework rather than by explicitly solving the analytic trust-region problem, apply
24

---

Eath-etr Hoat brx boteer Ador A
Tath-roe Nowr foe bdbwen Adsedd Sealdo
(a) Batch-wise mean ertor for positive pairs (i =
(b) Batch-wise mean emoor for negative pairs (i of j), exhibiting divergence that promotes embedding uniformity.
j). showing convergence and bounded deviation (1.0, 1.5)).
Figure 9: Comparison between the analytic step length au and the effective Adam step length Ty-- Although Dij learned through backpropagation, the norm-based trust-region constraint ensures that its magnitude remains bounded and consistent with the analytic formulation. Positive pairs show convergence and bounded errors, while negative pairs exhibit divergence that aids global uniformity.
Table 10: Comparison of computational efficiency between CLIP4Clip and GARE.
Metric
CLIP4Clip
GARE
Training time
1h 30m
Ih 34m
Inference time
7.6s
6.9s
Training memory (reserved)
4x 12175 MB
4x 12561 MB
Inference memory (reserved)
4136 MB
4216 MB
Training FLOPss (per hatch)
39,167.58 GFLOPs
39,287.55 GFLOPs (+0.3%)
Inference FI.OPs (per batch)
11,868.70 GFLOPs
11,905.05 GFLOPss (+0.3%)
norm-based trust-region regularization L. that restricts ||A, within a trust-region radius. This guarantees that the update magnitudes remain consistent with the theoretical constraint, even when learned implicitly.
To measure the deviation between the theoretical and learned step magniudes, we compute the scalar error:
E
(43)
This metric quantifies how closely the effective update magnitude in training adheres to the analytic trust-region step size.
Empirical analysis. Figure 9a and Figure 9b visualize the batch-wise mean error for positive and negative pairs, respectively. For positive pairs (i = j). the error || - asall remains within a stable range of [1.0, 4.5) and shows clear convergence trend. Given the 512-dimensional embedding space, this corresponds to a per-dimension deviation of roughly 4.5/512, which negligible. This indicates that the learned As updates remain consistent with the analytic trust-region dynamics and are well regularized by the norm constraint.
In contrast, for negative pairs (i * j). the error |n, auyl shows a divergent trend, which is theoretically expected. Each negative Ag is influenced by repulsive gradients with (512 1 degrees of freedom, while the positive pair has a single dominant alignment direction. Moreover, since norm-based regularization enlarges the updates whose magnitudes exceed the batch-wise mean ||A the dispersion of negative A; naturally increases. Such divergence. however, plays a constructive role: by expanding negative (Dir}e J. (j # i) into a larger representational space, the model enhances embedding uniformity, allowing positive A to achieve more stable alignment under broader geometric margin.
Summary. In summary, our analysis demonstrates that while Ai is optimized implicitly through the training framework, the norm-based trust-region constraint effectively maintains the magnitude of A within the theoretical bound. The learned v module reproduces the analytic trust-region
25

---

Table 11: Module-wise GFLOPs breakdown during forward propagation.
Module
GFLOPs (%)
Main computation source
CLIP visual encoder
11,673.60 (98.05%)
VIT image processing
Video temporal transfomerr
38.81 (0.33%)
FFN + self-attention
V (cross-attention)
36.35 (0.31%)
linear projection
CL.P text. encoder
156.29 (1.31%)
text sequence processing
Total
11,905.05 (100%)
Inference Procedure (Pseudocode)
Precompute all text and video embeddings offiine using CLIP encoders for each text batch T:
for each video batch
Compute pair-specific delta via di and adjust text embeddings logits model. gett_ get__eet_______ee
simi simiimmmiiiii lariit.rtty... to (T, v)
Append cosine- similarity block to the global similarity matrix
L.m_matrix.... www.ama...ppeeeddaeemam.. (logits)
Discard the delta tensor immediately (transient variable)
# Concatenate all similarity blocks for final retrieval
Figure 10: Inference pseudocode of GARE. A are computed on-the-fly and discarded immediately to ensure constant memory usage during large-scale scale retrieval.
Cross-Attention Pairwise Parallelization (Simplified Code)
def cross_ iestentioonquerry key, value): query: (a, b, dim) for (v_j t_i), text batch size, video
batch size
# key/value: (b, f, dim) for video frames
query query. permute purmute
(1,2,0)
(a,b,dim) -> (b, (b,dim,a)
Q Q_proj (query)
K K_proj (key)
V V_proj (value)
# This op is sarrise-parallellzaahl cross all pairs in the batch logits matmul (K, # (b,f,dim) (b,dim,a) -> (b,f,a)
scores softmax (logits sqrt(dim), dim=frame_dim,
scores permute (0,2,1) # (b,f,a) (b,a,f)
out = matmul (scores, V) # (b,a,f) x (b, (b,f,dim) -> (b,a,dim) (b,a,dim) (a,b,dim)
out. out.permuuee (1, 2))
return out
Figure 11: Simplified implementation of s's cross-attention operation. The computation is pairwise- parallelizable across all text-video pairs, ensuring linear scaling with batch size and high GPU utilization.
behavior for positive pairs, while the controlled divergence of negative pairs improves embedding uniformity- consistent with the objectives of contrastive learning.
I Model Efficiency and Implementation Details
GARE remains fully compatible with the standard dual-tower CLIP architecture while introducing only lightweight cross-modal adjustment module, vh. Implemented single-layer cross-attention transformer without FFN expansion, vē adds merely 1.58M parameters to the 354M parameters of the CLIP encoders. Its computational cost negligible- only 36.35 GFL.Pss per batch compared to CLIP's 11,868.70 GFLOPs.
26

---

During inference. 1 operates after both text and video embeddings have been precomputed. applying transient. pair-specific deltas before cosine similarity is calculated. This design preserves the pre- computability and scalability of the dual-tower framework: all embeddings can be cached offline, performs only on-the-fly adjustments, and no delta tensors are stored only similarity scores are retained.
Efficiency comparisons on MSR-VTT (4xRTX 4090 GPUs, batch size 128) show that GARE introduces minimal overhead (Table 10). A module-wise breakdown (Table 11) further confirms that over 98% of total FLOPs come from the visual encoder, while g contributes less than 0.4%, demonstrating computational parity with the original CLIP.
As illustrated in Figure 10, GARE performs inference in batch aitch-parallel streaming manner, generating and discarding deltas on-the-fly to maintain constant memory usage. The internal cross-attention pro- cess (Figure 11) is pairs and ensuring high GPU utilization.
enabling simultaneous computation across all text-video
Overall, GARE retains the efficiency of dual-tower architectures while introducing transient, pair- specific adjustment that enhances line-grained cross-modal alignment without increasing latency or memory cost.
Distinction from traditional Qual-Tower models. Traditional dual-tower architectures define a consistent metric space where the triangle inequality holds, e.g., for any text-video pair (t,, 22) and another video Uj.-
d(t,, UU) S d(t;, vk) + d(vk,
This property allows efficient large-scale retrieval via approximate nearest neighbor (ANN) search, since all embeddings share unified metric geometry.
In contrast, GARE introduces pair-specific adjustments through Aij. yielding a conditional distance
dGARE(14, v2) = d(44 + Aig, Uy):
which depends on the paired video uy. Substituting 4 + Aig and + D. into the above inequal- ity breaks the shared metric assumption, and thus the triangle inequality no longer strictly holds. Consequently, while GARE improves fine-grained alignment, it cannot directly support ANN-based retrieval relying on fixed metric consistency.
From efficiency standpoint, GARE computes A in pair wair-wise parallel manner within each batch, discarding them immediately after computing cosine similarities. This design keeps latency low but introduces additional FLOOS that make direct large-scale application challenging. Nevertheless, find that in practice the retrieved candidates from the unadjusted text embeddings t, already cover most of the top-ranked results produced by GARE. For instance, on the MSR-VTT Ik-A validation set, the top-256 candidates retrieved using t include all of GARE's top-10 matches.
Therefore, GARE naturally supports two-stage retrieval pipeline: (1) use the dual-tower model with t; for large-scale ANN retrieval to obtain a compact candidate set, (2) re-rank the top-k candidates using GARE's pair-specifir adjustments. This strategy preserves the scalability of dual-tower models while benefiting from GARE's 's fine-grained alignment refinement.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract accurately describes our contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have raised two limitations regarding the current work in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
We did not propose a theorem.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: This includes experimental details and our hyper-parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification: The official version of the code will be released immediately upon receipt of the paper (within one day).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides/ Stwpe///oooSSuusiionnnee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/eddeS/Seuub ssi wwwinnssssnniiiiiiio sesionnyocc) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The specific experimental details have been presented in the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We did not provide an error bars experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the type of GPU we used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https :ttp:p:://eeuure http///eeerrpppeeeeee https///e/pppiitiiiiiiiiiiiieeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with NeurIPS Code of Ethic.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification; The method proposed in this paper belongs the application category of short video platforms and has direct impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no risk of misuse in the method we proposed.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this paper are under the license of CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce a new training framework including a custom loss function and optimization strategy. Full code will be released with detailed documentation and instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
33

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : /ttp///erss////// theurips.cc/ooffeeee /20e 2LLM/LLM for what should or should be described.
34


Reviews:
[{"review_id": "dzUlaMtxic", "reviewer": "Reviewer_ztES", "summary": "The paper proposes GARE, a framework addressing optimization challenges in text-video retrieval by introducing learnable pair-specific increments $\\Delta_{i,j}$ to mitigate gradient tension from modality gaps and false negatives. The paper presents a theoretically grounded approach to addressing gradient tension in text-video retrieval, with strong experimental support for its effectiveness. However, its departure from the dual-tower paradigm and potential inefficiencies raise critical questions about its practical utility.", "questions": "please see weaknesses above", "limitations": "please see weaknesses above", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. **Valid Core Motivation**: The work effectively targets a critical and well-recognized issue in cross-modal learning: the domain (modality) gap between text and video representations. By framing this gap as a source of optimization tension under contrastive learning, the paper identifies a meaningful problem with clear practical relevance, laying a solid foundation for the proposed solution.\n2. **Rigorous Experimental Design**: The experiments are comprehensive and well-structured. The evaluation spans four standard benchmarks (MSR-VTT, DiDeMo, ActivityNet Captions, MSVD), with consistent comparisons to state-of-the-art methods. Ablation studies systematically validate the contributions of key components ($\\Delta_{i,j}$, regularization terms, modality-specific application), ensuring the observed improvements are attributable to the proposed mechanisms rather than confounding factors.\n3. **Sound Theoretical and Design Contributions**: The derivation of $\\Delta_{i,j}$ via multivariate Taylor expansion of the InfoNCE loss under a trust-region constraint is mathematically rigorous, providing a clear rationale for pair-specific increments. The analysis of $\\Delta_{i,j}$’s geometric properties (e.g., angular distributions) and the design of complementary regularizers (norm constraints, directional diversity, information bottleneck) demonstrate careful consideration of how to stabilize and interpret the learned increments. This integration of theory and design strengthens the framework’s validity.\n\n## Weaknesses\n1. **Violation of Dual-Tower Architecture Principles**: A central drawback is that the introduction of pair-specific $\\Delta_{i,j}$ disrupts the independence of text and video towers in cross-modal retrieval. Traditional dual-tower models enable efficient retrieval via precomputed embeddings and dot-product similarity, but GARE’s $\\Delta_{i,j}$ introduces dynamic, pair-dependent interactions $t_i + \\Delta_{i,j}$ that cannot be precomputed. This undermines the core advantage of dual-tower designs—scalable, offline retrieval—and limits practical deployment in real-world systems where low latency is critical. The paper does not address this trade-off or propose mitigations (e.g., approximation strategies for faster inference).\n2. **Efficiency Concerns**: The training and inference efficiency of GARE is inadequately discussed. The learnable $\\Delta_{i,j}$ is pair-specific, meaning its computational cost scales with the number of text-video pairs in a batch, which could become prohibitive for large datasets or high-throughput applications. If $\\Delta_{i,j}$ requires real-time computation for each query-target pair during inference, this would drastically slow down retrieval compared to static embedding-based methods. Without details on latency or scalability, the practical feasibility of GARE remains uncertain. Additionally, the paper does not clarify how $\\Delta_{i,j}$ is computed for unseen test pairs.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for raising two critical concerns:\n(1) whether the introduction of pair-specific deltas violates the principles of dual-tower architectures;\n(2) whether the proposed `ψ` module could impact training and inference efficiency.\n\nWe address both points in detail below.\n\n---\n\n# **Q1: Dual-Tower Compatibility**\n\nGARE augments a standard **dual-tower CLIP** architecture with a **lightweight cross-modal adjustment module** `ψ`.\n\n* `ψ` is implemented as a **single-layer cross-attention Transformer** with FFN hidden dimension not expanded, adding only **1.58M parameters** compared to the **354M parameters** in the CLIP encoder. At the same time, during the inference phase, the GFLOPs per batch introduced by `ψ` is **36.35 GFLOPs**, much smaller than the baseline encoder’s **11,868.70 GFLOPs** per batch.\n* It operates **after embeddings are precomputed** by the dual towers, applying pair-specific deltas to adjust text embeddings before computing cosine similarity.\n\nImportantly, this design **does not compromise dual-tower precomputability**:\n\n1. **All text and video embeddings** are computed and cached offline by the CLIP encoder (4-layer video temporary Transformer equipped after CLIP vision encoder).\n2. `ψ` only introduces a **transient delta computation** for each text–video batch pair.\n3. **No delta embedding is stored**, and only the resulting cosine similarity block is retained.\n\nThus, GARE **preserves the key advantages of dual-tower architectures**—offline caching and scalable batch-parallel retrieval—while enabling **fine-grained pairwise alignment** through a minimal adjustment step.\n\n---\n\n# **Q2: Training and Inference Efficiency**\n\nThank you for raising concerns about computational efficiency. We will address them by comparing FLOPs, memory usage, how efficient batch-pair level similarity calculation is implemented during inference, and how the $\\psi$ module ensures negligible latency by computing (text, video) pair-wise parallelizable cross-attention.\n\nWe provide a detailed comparison of GARE and CLIP4Clip on **4×RTX 4090 GPUs (batch size 128)** and MSR-VTT:\n\n| **Metric**                   | **CLIP4Clip**    | **GARE**                 |\n| ---------------------------- | ---------------- | ------------------------ |\n| Training Time                | 1h 30min         | 1h 34min                 |\n| Inference Time               | 7.6s             | 6.9s                     |\n| Training Memory (reserved)   | 4 × 12175MB      | 4 × 12561MB              |\n| Inference Memory (reserved)  | 4136MB           | 4216MB                   |\n| Training FLOPs (batch-wise)  | 39,167.58 GFLOPs | 39,287.55 GFLOPs (+0.3%) |\n| Inference FLOPs (batch-wise) | 11,868.70 GFLOPs | 11,905.05 GFLOPs (+0.3%) |\n\n---\n\n### **Module-wise GFLOPs Breakdown**\n\nHere is a summary of the **GFLOPs per module** during forward propagation:\n\n| Module                      | GFLOPs   | Percentage | Main Calculation Source            |\n|-----------------------------|----------|------------|------------------------------------|\n| CLIP Visual Encoder          | 11,673.60 | 98.05%     | ViT Image Processing               |\n| Video Temporal Transformer   | 38.81    | 0.33%      | FFN (66.4%) + Self-Attention (33.3%) |\n| `psi` (Cross-Attn Transformer)  | 36.35    | 0.31%      | Linear Projection (73.6%)          |\n| CLIP Text Encoder            | 156.29   | 1.31%      | Text Sequence Processing           |\n| **Total**                    | **11,905.05** | **100%**  | -                                  |\n\n---\n\n**Key observations:**\n\n* Training memory increases by **\\~3%**, and inference memory increases by **\\~1%**, which is modest.\n* FLOPs increase in inference is **\\~0.3% per batch pair** and remains dominated by the encoder.\n* End-to-end inference is **slightly faster** (6.9s vs 7.6s) due to efficient batch-wise similarity computation. **Delta computation is identical for training and inference and remains transient (e.g., 128×128×512, \\~32MB) during the forward pass.** Only the cosine similarity block is retained for further computation.\n\nCrucially, **deltas are computed on-the-fly and immediately discarded**, with a typical tensor (128×128×512, float32) occupying only **≈32MB** transiently. This **streaming computation** ensures that memory usage remains minimal even for large-scale datasets.\n\n---\n\n### **Inference Procedure (Pseudocode)**\n\nThe inference pipeline preserves dual-tower efficiency via **batch-parallel streaming computation**:\n\n```python\n# Precompute all text and video embeddings offline using CLIP encoders\n\nfor each text batch T:\n    for each video batch V:\n        # 1. Compute pair-specific delta via ψ and adjust text embeddings\n        logits = model.get_similarity_logits(T, V)  \n        \n        # 2. Append cosine similarity block to the global similarity matrix\n        sim_matrix.append(logits)\n        \n        # 3. Discard the delta tensor immediately (transient variable)\n        \n# Concatenate all similarity blocks for final retrieval\n```\n\n**Key points:**\n\n* Deltas are **never stored**, only used to produce logits;\n* Memory footprint remains low, and computation is **fully GPU-parallel**;\n* The design is **deployable for real-world retrieval** with minimal latency.\n\n---\n\n### **Cross-Attention Pair-Wise Parallelization**\n\nTo further clarify how the cross-attention mechanism in \\$\\psi\\$ operates efficiently, here is the **pseudocode** for the pair-wise parallelized cross-attention computation:\n\n```python\ndef CrossAttentionLayer(query, key, value):\n    \"\"\"\n    Compute Q, K, V of Cross-Attention (omitting multi-head).\n    \n    Args:\n        query: ti - vj with shape (a, b, dim)\n        key & value: video context sequence with shape (b, f, dim), where f is the number of frames\n    \n    Returns:\n        Attention output\n    \"\"\"\n    \n    a, b, dim = query.shape\n    _, f, _ = key.shape\n    \n    query = query.permute(1, 2, 0)  # -> (b, dim, a)\n    \n    q_proj = Q_proj(query)\n    k_proj = K_proj(key)\n    v_proj = V_proj(value)\n    \n    # This operation can be pair-wise parallelized\n    attn_logits = k_proj @ q_proj  # (b, f, dim) x (b, dim, a) -> (b, f, a)\n    attn_scores = softmax(attn_logits / sqrt(d), dim=1).permute(0, 2, 1)  # -> (b, a, f)\n    \n    attn_out = attn_scores @ v_proj  # (b, a, f) x (b, f, dim) -> (b, a, dim)\n    attn_out = Out_proj(attn_out).permute(1, 0, 2)  # -> (a, b, dim)\n    \n    return attn_out\n```\n\n**Key points:**\n\n* This computation is **pair-wise parallelizable**, meaning it processes text-video pairs within a batch efficiently, leveraging parallelization across the batch dimension.\n* This design ensures that the cross-attention operation remains **scalable and efficient**, preserving the advantages of dual-tower retrieval while introducing fine-grained alignment.\n\n---\n\n# **Conclusion**\n\nBy embedding `ψ` in a **streaming, transient, and batch-parallel** workflow, GARE:\n\n1. **Preserves dual-tower scalability and offline precomputability**;\n2. **Adds negligible memory and FLOPs overhead**;\n3. **Maintains low-latency retrieval**, compatible with large-scale deployments;\n4. **Provides expressive pair-specific alignment** that significantly improves retrieval performance.\n\nWe appreciate the reviewer’s feedback, which helped us clarify GARE’s **practical efficiency and deployability**."}, {"review_id": "85aA4K1ARe", "reviewer": "Reviewer_FbVo", "summary": "The paper proposes GARE, a new gap-aware retrieval framework that introduces learnable pair-specific increments (Δij) to alleviate optimization tension in text-video contrastive learning. Through first-order Taylor approximation of InfoNCE loss, the work derives Δij as an optimal adjustment direction that resolves gradient conflicts caused by modality gaps and false negatives. The framework employs a lightweight neural module to predict Δij based on semantic gaps between pairs, enabling structure-aware corrections while maintaining training stability. Three key regularizations (trust-region constraint, directional diversity, and information bottleneck) are introduced to enhance the interpretability and geometric properties of the learned increments. Extensive experiments across four benchmarks demonstrate consistent improvements in alignment accuracy and robustness, with qualitative analysis showing Δij effectively redistributes gradient tension while preserving semantic structure.", "questions": "See Weaknesses section. I look forward to the author's response to my concerns during the rebuttal stage.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**[Strengths]**\n\n**+ Novel solution.** The paper introduces a novel approach to address the inherent optimization tension in contrastive learning by proposing pair-specific increments (Δij). This mechanism dynamically redistributes gradient conflicts caused by modality gaps and false negatives, offering a more stable and interpretable optimization landscape.\n\n**+ Theoretical support.** The work derives the ideal form of Δij through a first-order Taylor expansion under trust-region constraints, providing a solid theoretical foundation.\n\n**+ Extensive experiments.** The paper demonstrates consistent improvements across four diverse benchmarks (MSR-VTT, DiDeMo, ActivityNet, MSVD), outperforming recent state-of-the-art methods. The ablation studies (e.g., modality choice, regularization strength) provide actionable insights for adapting the framework to different data characteristics.\n\n**+ Qualitative analysis.** The visualization reveals that Δij learns geometrically structured corrections—pushing embeddings outward while maintaining angular separation. This aligns with the uniformity principle of contrastive learning.\n\n**[Weaknesses]**\n\n**- Lack of efficiency analysis.** The application of learnable per-pair increments inevitably introduces additional training and inference overhead. However, the article only provides qualitative and quantitative analysis of this method on the Text-Video Retrieval task. Does the performance improvement solely come from adding more learnable parameters?  \n\n**- The method lacks adaptability.** According to the left side of Table 3 in the ablation experiments section, the authors extensively explored the Context Modality Choice and concluded that the decision on which modality to inject Δij into should be based on the characteristics of the dataset. Moreover, the reported results for each dataset in the article require selecting the best-performing setting through trial and error. This raises concerns about the method's poor generalizability and universality, as it necessitates dataset-specific configurations. Can this process be made adaptive?  \n\n**- More tasks are needed to verify generalizability.** Due to the presence of annotation noise, the issue of optimization tension is not limited to contrastive learning in text-video retrieval tasks but also exists in various cross-modal contrastive learning tasks. It is recommended that the authors conduct additional experiments on tasks such as text-image retrieval to verify whether the proposed method possesses sufficiently strong generalizability.  \n\nWriting standards and typos:  \n   * Equations 2 and 3 exhibit obvious subscript and superscript errors. For example, \"$i$\" should be a subscript, and \"$B$\" should not be a superscript of \"$i=1$\" but rather should be written as $\\sum_{j=1}^{B}$.  \n   * Line 268: \"MST-VTT\" should be corrected to \"MSR-VTT.\"  \n   * The authors are advised to carefully review the article to correct these typos and formula formatting issues.", "was_revised": true, "final_justification": "The rebuttal effectively addressed my concerns regarding efficiency and generalizability:\n\n* Due to the use of a batch-wise streaming strategy, GARE is comparable in efficiency to CLIP4Clip.\n* GARE also demonstrated performance improvements over the baseline in the Image-Text Retrieval task.\n\nMy primary concern remains with the method's adaptivity. The author provided experimental results for bidirectional delta generation along with a detailed analysis, concluding that the two-path module will weaken its alignment behavior, and it is indeed necessary to decide in which modality to inject Δij based on the characteristics of the dataset.\n\nIn summary, this paper proposes an innovative method to address the inherent optimization tension in contrastive learning, supported by thorough theoretical analysis and extensive experiments that validate the method's effectiveness. From my perspective, it is a work worthy of acceptance. Since my initial score was positive, I have decided not to change the score.", "rebuttal": "Thank you for raising the questions and carefully pointing out the incorrect writings! We will correct them and check the spelling again. Below is my response to the questions.\n\n---\n\n## **Q1: Impact of Efficiency and Parameter Quantity**\nWe thank the reviewer for the insightful observation and concerns about the efficiency of the method. While our method introduces pairwise $\\Delta_{ij}$ with quadratic complexity, we **do not store all deltas**. Instead, we use a **batch-wise streaming strategy** that keeps memory usage practical.\n\nAt inference time, we precompute all CLIP encoder outputs and evaluate cosine similarities between text–video batches (e.g., 128×128 pairs) using the lightweight delta module $\\psi$. For each batch pair:\n\n- $\\Delta_{ij}$ (shape: 128×128×512, ≈32MB in float32) is computed on-the-fly;\n- It is immediately discarded after computing the 128×128 similarity matrix;\n- Only the similarity block is retained and later concatenated into the full similarity matrix.\n\nThus, **only one delta tensor is active at any time, and is Pair-Wise parallelization**.\n\nBelow is a runtime and memory comparison (4×RTX 4090 GPUs) on MSR-VTT:\n\n|**Metric**|**CLIP4Clip**|**GARE**|\n|-|-|-|\n|Training Time|1h 30min|1h 34min|\n|Inference Time|7.6s| 6.9s|\n|Training Memory|4 × 12175MB|4 × 12561MB|\n|Inference Memory|4136MB|4216MB|\n|Training FLOPs (batch-wise)|39,167.58 GFLOPs|39,287.55 GFLOPs|\n|Inference FLOPs (batch-wise)|11,868.70 GFLOPs|11,905.05 GFLOPs|\n\n**Key takeaway**:\n\n- $\\Delta_{ij}$ is a **transient variable**, not stored;\n- Only the similarity matrix is retained;\n- $\\Delta$ is computed via **pair-wise paralleled** cross attention (details in ztES).\n\n*(Due to space constraints, more details about pair-wise parallelization computing can be found in my response to Reviewer ztES. I apologize again for this.)*\n\n---\n\n### **Impact of parameter quantity**\n\nThank you for raising this concern. To clarify that the improvements in GARE are not merely due to the increase in parameters, we conducted several experiments comparing GARE with the baseline (**while the performance of GARE: R@1=49.1,R@5=74.7,R@10=83.6**):\n\n1. **Baseline comparison with additional layers**: We increased the **video temporary Transformer** from 4 to 5 layers in the baseline, aligning its parameters with GARE. The performance was **R@1 = 47.3**, **R@5 = 73.2**, **R@10 = 82.5**, showing that the increase in parameters led to only a modest improvement, indicating GARE’s better performance with the same parameter budget.\n2. **Parameter alignment with baseline**: Reducing the **video temporary Transformer** layers in GARE to 3, we aligned the parameter count with the baseline. The results were **R@1 = 48.1**, **R@5 = 73.8**, **R@10 = 83.6**, confirming that GARE has a structural advantage.\n3. **Interaction modification**: We modified GARE’s interaction by setting the **query** for $\\psi$ to be $t_i$ and the **key-value** to be frames from $v_j$, outputting $t_i'$. To match the scale of the delta in GARE, we computed $\\Delta = v_j - t_i'$ (or $t_i' - v_j$), effectively removing the relative structure information between $t_i$ and $v_j$. This modification resulted in **R@1 = 47.6**, **R@5 = 73.1**, **R@10 = 82.0**, **MdR = 2.0**, and **MnR = 12.6**, showing that while GARE uses a pair-wise interaction, its **interaction logic** is superior. The results demonstrate that **$\\psi$'s awareness of the relative structure between $v_j$ and $t_i$** is indeed crucial for improving performance.\n\nIn conclusion, GARE’s performance gains come from **structural advantages** and **improved interaction logic**, not just the addition of parameters (only **1.5M**, far less than CLIP’s **151M**).\n\n# **Q2: Adaptivity of the Method**\n\nThank you for raising the question about adaptivity. To explore this, we intuitively used two $\\psi$ modules for bidirectional delta generation, resulting in $(t_i + \\Delta^{t}\\_{ij}, v_j + \\Delta^{v}\\_{ij})$. This led to performance degradation, with R@1 dropping from 42.6 to 41.2 on ActivityNet Captions and from 49.1 to 48.4 on MSR-VTT, though still outperforming the baseline, indicating the delta-based approach maintains adaptivity. Additionally, we observed performance improvement in an image-text retrieval task on MSCOCO, further supporting GARE’s adaptivity across different domains (details in the final answer).\n\nWe hypothesize that this decline results from **progressive weakening of $\\Delta^{t}_{ii}$'s alignment role** due to **semantic ambiguity introduced by $\\Delta^{v}\\_{ii}$**. As training progresses, $v_i$ evolves into a **prototype-style anchor**, similar to Fig. 2(b). When $v_i$ is used as the context in $\\psi$ to compute $\\Delta^t_{ii}$, its **over-generalized nature** hinders accurate computation of $\\Delta^t_{ii}$, reducing alignment effectiveness and compromising training.\n\nTo validate that $\\Delta_{ii}$ performs the alignment task, we **compare the learned delta update magnitude with the theoretically derived update magnitude**, measuring the error to assess whether $\\Delta_{ii}$'s behavior aligns with the theoretical expectation. By expanding $\\mathcal{L}\\_i$ w.r.t. $\\Delta_{ij}^{(t)}$, we obtain the **analytic update step size** $\\alpha_{ij}^{(t)}$, ensuring $\\Delta_{ij}^{(t+1)}$ stays within a trust region:\n\n$\\left\\| \\Delta_{ij}^{(t)} - \\alpha_{ij}^{(t)} \\hat{g}\\_{ij}^{(t)} \\right\\|^2 \\le \\varepsilon^2, \\quad  \\hat{g}\\_{ij}^{(t)} = \\frac{\\nabla_{\\Delta_{ij}} \\mathcal{L}\\_i^{(t)}}{\\|\\nabla_{\\Delta_{ij}} \\mathcal{L}\\_i^{(t)}\\|}, \\varepsilon = \\|\\Delta_{ij}^{(t+1)}\\|.$\n\nSimplifying for the boundary case gives the **analytic step size**:\n\n$\\alpha_{ij}^{(t)} = \\Delta_{ij}^{(t)\\top} \\hat{g}\\_{ij}^{(t)} + \\sqrt{ \\left( \\Delta_{ij}^{(t)\\top} \\hat{g}\\_{ij}^{(t)} \\right)^2 - \\|\\Delta_{ij}^{(t)}\\|^2 + \\varepsilon^2 }.$\n\nIn contrast, the actual update induced by $\\psi$ after one optimizer step is:\n\n$\\Delta_{ij}^{(t+1)} = \\Delta_{ij}^{(t)} - \\eta_{ij}^{(t)} \\cdot \\frac{\\nabla_{\\Theta^{(t)}} \\mathcal{L}\\_i^{(t)}}{\\|\\nabla_{\\Theta^{(t)}} \\mathcal{L}\\_i^{(t)}\\|},$\n\nwhere $\\eta_{ij}^{(t)}$ is the implicit learned step size. To evaluate their consistency, we compute the error between the two step sizes:\n\n$|\\eta_{ij}^{(t)} - \\alpha_{ij}^{(t)}|.$\n\n----\n\n- Empirical results show that for **positive pairs** ($i = j$), the error remains within a **moderate range** ([1.0, 4.5] in 512-D space) and **decreases steadily** as training progresses, indicating that $\\psi$ produces updates **consistent with the theoretical rule**. This validates that $\\Delta_{ii}$ plays a **precise alignment role**.\n\n- For **negative pairs** ($i \\neq j$), the error is larger and does not converge, as $\\Delta_{ij}$ receives only **push gradients from $v_j$** without alignment supervision. This optimization occurs in a **$(D-1)$-dimensional space**, leading to more variable updates compared to the **1-dimensional alignment** of $\\Delta_{ii}$. Consequently, $\\Delta_{ij}$ primarily enhances **uniformity** by increasing scale and dispersing representations.\n\n- **Comparison with Dual $\\psi$ Modules**:\n When using dual $\\psi$ modules (i.e., $(t_i + \\Delta^t_{ij}, v_j + \\Delta^v_{ij})$), both $\\Delta^t_{ii}$ and $\\Delta^v_{ii}$ show **slower convergence** to the theoretical update step size $\\eta^{(t)}$. Surprisingly, after some initial convergence, the update magnitude error for $\\Delta^t_{ii}$ and $\\Delta^v_{ii}$ **starts to diverge**, which suggests that their **alignment capabilities are weakened**, preventing accurate update directions from being learned.\n\n---\n\nThis validates our analysis: when $\\Delta_{ii}$ performs the alignment function, using the two-path $\\psi$ module will weaken its alignment behavior. In future work, we will focus on alleviating the optimization tension while seeking a more adaptive dual-path method.\n\n*(We will include the error plots in the appendix and public codebase if the paper is accepted, as current rebuttal guidelines do not permit figures.)*\n\n# **Q3: GARE' performance on Image-Text Retrieval**\nThank you for your suggestion to evaluate GARE’s generalization in image-text retrieval. We used CLIP as the baseline and conducted ITM tasks on MS-COCO 1K and 5K, comparing it with methods like VSE∞, PCME++, and PAU. Due to time constraints, we did not fine-tune GARE on MS-COCO, resulting in limited improvement on the ITM task. Currently, we use CLIP’s image patch tokens as the context input for the module, but these often introduce noise and weaker semantic information compared to the image’s `[CLS]` embedding. In the future, we plan to aggregate the patches to extract cleaner features, enabling better validation of GARE’s effectiveness on the ITM task. Below are the results of GARE on MS-COCO with ViT-B/32 as the backbone:\n\n|Method||1k|i2t||1k|t2i||5k|i2t||5k|t2i| \n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n||R@1|R@5|R@10|R@1|R@5|R@10|R@1|R@5|R@10|R@1|R@5|R@10|\n|VSE∞|**82.0**|**97.2**|98.9|_69.0_|_92.6_|_96.8_|62.3|**87.1**|**93.3**|**48.2**|**76.7**|**85.5**|\n|PCME++|_81.6_|**97.2**|_99.0_|**69.2**|**92.8**|**97.1**|62.1|_86.8_|**93.3**|_48.1_|**76.7**|**85.5**|\n|PAU|80.4|96.2|98.5|67.7|91.8|96.6|_63.6_|85.2|92.2|46.8|74.4|*83.7*|\n|**Baseline**|80.1|95.7|98.2|67.1|91.4|96.6|62.9|84.9|91.6|46.5|73.8|82.9|\n|**ours**|80.7|_96.8_|**99.1**|68.2|91.9|96.6|**63.9**|85.5|*92.4*|47.2|*74.8*|83.5|\n\nAt the same time, we also performed **noise correspondences** training on MS-COCO, which demonstrates significantly better robustness compared to the baseline:\n\n|Noise ratio|Method||1k|i2t||1k|t2i||5k|i2t||5k| t2i| \n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|||R@1|R@5|R@10|R@1|R@5|R@10|R@1|R@5|R@10|R@1|R@5|R@10|\n|20%|Baseline|76.0|94.3|97.5|63.4|89.0|94.8|55.3|79.1|86.9|41.0|68.8 |79.3| \n||ours|**79.4**|**96.0**|**98.8**|**65.2**|**90.7**|**96.4**|**58.9**|**83.7**|**91.2**|**43.7**|**70.7**|**80.8**|\n|50%|Baseline|73.9|93.0|97.2|60.1|87.3|94.0|54.1|78.5|86.6|39.7|67.2|77.5|\n||ours|**77.2**|**93.6**|**97.2**|**62.5**|**89.1**|**94.2**|**56.1**|**80.8**|**87.9**|**41.2**|**68.6**|**79.2**|"}, {"review_id": "Wm7G3pCRUb", "reviewer": "Reviewer_3HTR", "summary": "This paper proposes GARE, a method for improving text-video retrieval by addressing issues in contrastive learning. It introduces a learnable pair-specific adjustment that helps reduce gradient conflicts caused by modality gaps and false negatives. The approach shows consistent performance gains across multiple benchmarks.", "questions": "please see the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. Introduces a novel mechanism (∆_{ij}) to locally adjust representation gaps between text and video, rather than forcing global alignment.\n2. Provides a solid theoretical foundation using first-order Taylor expansion under trust-region constraints.\n3. Effectively addresses both modality gap and false negatives, which are common challenges in contrastive learning.\n\nCons:\n1. While the authors derive ∆_{ij} updates from a multivariate first-order Taylor approximation under a trust-region constraint, the actual implementation relies on a learned approximation using a neural module. The paper doesn’t sufficiently quantify how closely this learned update matches the ideal gradient direction. Including some error analysis between the true and learned update directions would strengthen the claim of \"structure-aware\" correction.\n2. The method introduces a pairwise increment ∆_{ij} for each text-video pair, which potentially scales quadratically with batch size. However, the paper does not address the memory or computational overhead introduced by this component, nor how it compares to other methods like hard negative mining or momentum encoders.\n3. From the ablation results, norm and direction regularization only help when combined with the IB loss. This suggests some over-regularization may occur if these components are applied individually. The paper might benefit from a discussion of when to apply these losses and how to tune their trade-off hyperparameters.", "was_revised": true, "final_justification": "The authors' clarifications and empirical evidence have largely addressed my concerns. I will keep my original score unchanged.", "rebuttal": "We sincerely thank Reviewer for the insightful and constructive comment. We apologize for the potential ambiguity in our original manuscript regarding the interpretation of the delta update mechanism.\n\n---\n\n## **Q1: Error Analysis**\n\nWe clarify that the update rule for each $\\Delta_{ij}^{(t+1)}$, derived from a multivariate first-order Taylor expansion under a trust-region constraint, is not meant as a unique or ideal direction, but rather as a principled descent direction compatible with InfoNCE optimization. The Taylor expansion offers a general update framework, not a fixed solution.\n\nTo address the batch-local limitation, we introduce a neural module $\\psi$ to generate $\\Delta_{i*}^{(t)}$ and embed the update into end-to-end training. Backpropagation allows deltas to evolve implicitly, capturing transferable and structure-aware patterns across batches while remaining consistent with the theoretical formulation.\n\nSince the descent direction is not unique, a strict vector-wise comparison is infeasible. Instead, we assess consistency via step size, focusing on the analytic length $\\alpha_{ij}^{(t)}$ that enforces the trust-region radius $\\varepsilon$:\n\n$\\left\\| \\Delta_{ij}^{(t)} - \\alpha_{ij}^{(t)} \\hat{g}\\_{ij}^{(t)} \\right\\|^2 \\le \\varepsilon^2, \\quad \\text{where} \\quad \\hat{g}\\_{ij}^{(t)} = \\frac{ \\nabla_{\\Delta_{ij}} \\mathcal{L}\\_i^{(t)} }{ \\left\\| \\nabla_{\\Delta_{ij}} \\mathcal{L}\\_i^{(t)} \\right\\| }, \\varepsilon=\\|\\Delta_{ij}^{(t+1)}\\|.$\n\nWe take the **equality case** to project the update precisely to the **boundary of the feasible region**, and expand the squared norm leads to a quadratic equation in $\\alpha_{ij}^{(t)}$:\n\n$\\alpha_{ij}^{(t)^2} - 2\\alpha_{ij}^{(t)} \\Delta_{ij}^{(t)\\top} \\hat{g}\\_{ij}^{(t)} + \\left\\| \\Delta_{ij}^{(t)} \\right\\|^2 - \\varepsilon^2 = 0,$\n\nwhose positive root gives the **closed-form solution**:\n\n$\\alpha_{ij}^{(t)} = \\Delta_{ij}^{(t)\\top} \\hat{g}\\_{ij}^{(t)} + \\sqrt{ \\left( \\Delta_{ij}^{(t)\\top} \\hat{g}\\_{ij}^{(t)} \\right)^2 - \\left\\| \\Delta_{ij}^{(t)} \\right\\|^2 + \\varepsilon^2 }.$\n\nIn contrast, the neural module $\\psi$ is updated through AdamW, and after one optimization step, the actual update can be written as:\n\n$\\Delta_{ij}^{(t+1)} = \\Delta_{ij}^{(t)} - \\eta_{ij}^{(t)} \\cdot \\frac{ \\nabla_{\\Theta^{(t)}} \\mathcal{L}\\_i^{(t)} }{ \\left\\| \\nabla_{\\Theta^{(t)}} \\mathcal{L}\\_i^{(t)} \\right\\| },$\n\nwhere $\\eta_{ij}^{(t)}$ denotes the **effective step size** resulting from the AdamW update. To quantify the closeness between the learned and analytic updates, we compute the scalar error:\n\n$\\left| \\eta_{ij}^{(t)} - \\alpha_{ij}^{(t)} \\right|.$\n\n------\n\n**Empirical results:**\n\n- For **positive pairs** ($i = j$):\n  We observe that the error $\\left| \\eta_{ii} - \\alpha_{ii} \\right|$ remains within a bounded range of **[1.0, 4.5]** during training and shows a clear trend of **convergence**. Considering the 512-dimensional embedding space, this deviation is moderate and acceptable.\n\n- For **negative pairs** ($i \\ne j$):\nthe error is larger and non-convergent, reflecting their inherently repulsive role without consistent alignment supervision.\nThese fluctuations are meaningful: by expanding negative $\\Delta_{ij}$ norms and lowering their cosine similarity, they enhance embedding uniformity, a key property in contrastive learning.\n\n------\n\nIn summary, the scalar step-size analysis shows that $\\psi$ produces updates consistent with the analytic trust-region rule for positive pairs, validating its structure-aware behavior, while the divergence on negative pairs naturally serves to promote embedding uniformity, in line with contrastive learning objectives.\n\n*(We will include the error plots in the appendix and public codebase if the paper is accepted, as current rebuttal guidelines do not permit figures.)*\n\n---\n\n## **Q2: Effeciency and Hard Negative**\nWe thank the reviewer for the observation. Although our method introduces pairwise $\\Delta_{ij}$ (quadratic in batch size), we **never store all deltas**. Instead, a **batch-streaming strategy** keeps memory practical:\n\n- For each 128×128 text–video block, $\\Delta_{ij}$ (≈32MB in float32) is computed on-the-fly by $\\psi$, used to form the similarity matrix, and immediately discarded;\n- Only one delta tensor exists at a time, and only the **similarity block** is retained for later concatenation.\n\nThis design keeps memory low and preserves **dual-tower efficiency**. Runtime and memory (4×RTX 4090) on MSR-VTT are:\n\n|**Metric**|**CLIP4Clip**|**GARE**|\n|-|-|-|\n|Training Time|1h 30min| 1h 34min|\n|Inference Time|7.6s| 6.9s|\n|Training Memory|4 × 12175MB|4 × 12561MB|\n|Inference Memory|4136MB | 4216MB|\n|Training FLOPs (batch-wise)|39,167.58 GFLOPs|39,287.55 GFLOPs|\n|Inference FLOPs (batch-wise)|11,868.70 GFLOPs|11,905.05 GFLOPs|\n\n**Takeaway:** \n- $\\Delta_{ij}$ is **transient**, only the similarity matrix is stored, and overhead remains minimal.\n- Training memory increases by **~3%**, and inference memory increases by ~1%, which is modest.\n- $\\Delta$ is computed via **pair-wise paralleled** cross attention (details in ztES).\n\n*(Due to space constraints, the detailed logic of the inference stage can be found in my response to Reviewer ztES. I apologize again for this.)*\n\n-----\n\n### **Hard Negative Comparison**\n\n We compare GARE with two recent methods related to hard negative handling:\n\n**DMAE** (ACM MM 2023, R@1: 46.9, +1.6% over base, our reproduction: 47.1):\n DMAE improves fine-grained alignment by mining hard positives (e.g., queries tied to specific frames), which implicitly enhances the model’s ability to push away hard negatives. This is conceptually similar to our *IB loss*, which compresses $\\Delta$ via a variational bottleneck to retain critical alignment signals.\n\n**NeighborRetr** (CVPR 2025, R@1: 49.5, +2.3% over base, our reproduction: 49.2):\n Uses a memory bank to compute k-neighbor co-occurrence and identify “good hubs.” This promotes local consistency and reduces over-penalization of hard negatives. While not explicitly framed as hard negative mining, selecting top-k co-occurring samples achieves similar effects.\n\n**GARE** (R@1: 49.1, +2.5% over base):\n GARE does **not require explicit hard negative mining**. Each $\\Delta_{ij}$ absorbs loss gradients locally from the corresponding pair, reducing reliance on $t_i$ when encountering hard negatives. This softens noisy updates and improves generalization.\n\n**Efficiency Comparison**:\n\n- GARE uses only one cross-attention layer; NeighborRetr includes 8 MLPs and multiple transformer/conv layers.\n- NeighborRetr requires 10240-sample memory banks per modality and ~4.5h training time; GARE achieves similar performance with **1h34min** training and minimal memory.\n\nEmpirically, we observe that $t_i$ in GARE becomes semantically more stable: for semantically similar $v_j$, the similarities $s(t_i, v_i)$ and $s(t_i, v_j)$ are smoother than in the baseline. This supports more accurate delta-based alignment and aligns with Fig. 2, where $t_i$ functions as a stable prototype. Ultimately, the enhanced semantic stability of $t_i$ allows $\\Delta$ to carry out more precise fine-grained alignment across similar samples. (Due to rebuttal guidelines, we regret that we cannot include figures here.)\n\n## **Q3: Role of IB loss**\n\nOur method follows the **variational information bottleneck (VIB)** principle.\n The standard VIB objective is\n\n$\\mathcal{L}\\_{\\text{VIB}} = -\\mathbb{E}_{z\\sim p(z|x)}[\\log q(m|z)] + \\beta \\cdot \\mathrm{KL}\\big(p(z|x) || r(z)\\big),$\n\nwhere $x$ is the input, $z$ is the bottleneck variable, and $m$ is the match label.\n In our cross-modal retrieval setting, we set $x = (t_i, v_j)$ and $z = \\Delta_{ij}$.\n The first term naturally corresponds to the **InfoNCE loss** on positive/negative text–video pairs, and the second term corresponds to our **IB loss (KL divergence)** on $\\Delta_{ij}$.\n\nIn principle, the compression term should be **pair-wise**:\n\n$\\mathrm{KL}\\big(p(\\Delta_{ij}\\mid t_i,v_j) || r(\\Delta)\\big),$\n\nwhich would require modeling each $\\Delta_{ij}$ as a distribution and sampling $k$ times, leading to $(k, B_t, B_v, \\text{dim})$ complexity.\n To make this tractable, we adopt a **video-conditioned approximation**:\n\n$p(\\Delta_{ij}\\mid t_i,v_j) \\approx p(\\Delta_{ij}\\mid v_j),$\n\nand compute the KL term by taking the mean and standard deviation along the text dimension:\n\n$\\mu_j = \\mathrm{mean}\\_i(\\Delta_{ij}), \\quad \\sigma_j = \\mathrm{std}\\_i(\\Delta_{ij}),$\n\napproximating $p(\\Delta_{ij}\\mid v_j) \\approx \\mathcal{N}(\\mu_j, \\sigma_j^2)$.\n This is consistent with the **many-to-many nature** of video–text datasets (e.g., MSRVTT has ~20 captions per video) and still satisfies the Monte Carlo property by treating text samples as stochastic draws. Empirically, video-conditioning outperforms text-conditioning because videos provide more stable, modality-level common information that forms an effective bottleneck.\n\nOur **overall training objective** is therefore:\n\n$\\mathcal{L}\\_{\\text{total}} = \\underbrace{\\mathcal{L}\\_{\\text{Info}} + \\beta \\cdot \\mathcal{L}\\_{\\text{IB}}}\\_{\\text{VIB objective}} + \\lambda_{\\varepsilon} \\cdot \\mathcal{L}\\_{\\varepsilon} + \\lambda_{\\text{dir}} \\cdot \\mathcal{L}\\_{\\text{dir}},$\n\nwhere we set $\\beta = 0.07$, and $\\lambda_{\\varepsilon} = \\lambda_{\\text{dir}} = 0.01$.\n The small weights for norm/direction regularization are due to the naturally large vector magnitudes in a 512-D embedding space, where a small coefficient is sufficient for stable adjustment. \n\nFinally, this also explains **why norm and direction regularization only take effect when combined with IB loss**.\n Without the IB loss, $\\Delta_{ij}$ has excessive freedom under InfoNCE and can trivially vary in magnitude or orientation, making these penalties ineffective.  The IB loss compresses $\\Delta_{ij}$ into a **low-entropy, structured bottleneck**, which stabilizes its distribution; under this constraint, the norm and direction terms provide meaningful geometric guidance to improve alignment and uniformity."}, {"review_id": "Gy0sXkSmkf", "reviewer": "Reviewer_SveN", "summary": "This paper presents GARE, a video-text retrieval framework that adjusts each pair's similarity by a learnable increment $\\Delta_{ij}$. $\\Delta_{ij}$ can be imposed on the embedding of any modality to reduce the modality gap and mitigate the impact of false negatives. Experiments on four popular text-video retrieval datasets show the effectiveness of GARE.", "questions": "Please see weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The studied problem, i.e., modality gap and false negatives, is practical in cross-modal retrieval.\n- The proposed method that refines the embedding by an increment is novel and instructive.\n- The paper presents good qualitative analysis.\n\nWeaknesses:\n- This paper is not easy to follow. Many technical details are unclear. ①Is the final training goal the sum of various loss functions? Or does it need to be adjusted through trade-off parameters? ②What is the model structure of $\\psi$? And how does it handle input from both the modality gap and the context vector?\n- In general, the modality gap is a global metric to measure inherent differences, i.e., $|| \\bar{v} -\\bar{t}||$. But the proposed method tackles it in a pair-specific way, which requires some reasonable explanation.\n- Missing related works in contrastive learning/cross-modal retrieval with false negatives.\n- Compared methods are not strong, and many of them are worse than the basic CLIP model. The proposed method should be compared with noise-robust (designed for false negatives) methods.\n- Does the learned network $\\psi$ also generalize to inference time to refine the representation?", "was_revised": true, "final_justification": "I think this is a technically good paper, with its main weakness being clarity. Since the authors have committed to adding the missing technical details in the revised manuscript, I have decided to increase my rating.", "rebuttal": "Thank you very much for your positive feedback on our method. We also apologize for the unclear explanations in the manuscript that caused difficulties in reading. We will address your questions one by one.\n\n---\n\n# **Q1: Clarification on Technical Details**\n\n- **On the training objective**: We use weight parameters **$\\beta$, $\\lambda_{\\varepsilon}$, and $\\lambda_{\\text{dir}}$** to adjust the overall training objective, which can be divided into two main parts: one is the **VIB optimization objective** composed of **$\\mathcal{L}\\_{\\text{info}} + \\beta \\cdot \\mathcal{L}\\_{\\text{IB}}$**, and the other is the **delta structural regularization** composed of **$\\lambda_{\\varepsilon} \\cdot \\mathcal{L}\\_{\\varepsilon} + \\lambda\\_{\\text{dir}} \\cdot \\mathcal{L}\\_{\\text{dir}}$**. The full training objective is:\n\n  $\\mathcal{L}\\_{\\text{total}} = \\underbrace{\\mathcal{L}\\_{\\text{Info}} + \\beta \\cdot \\mathcal{L}\\_{\\text{IB}}}\\_{\\text{VIB objective}} + \\lambda_{\\varepsilon} \\cdot \\mathcal{L}\\_{\\varepsilon} + \\lambda_{\\text{dir}} \\cdot \\mathcal{L}\\_{\\text{dir}}.$\n\n  For MSR-VTT, we set **$\\beta = 0.07$**, **$\\lambda_{\\varepsilon} = \\lambda_{\\text{dir}} = 0.01$**. Since the adjustments are based on the geometric properties of the non-normalized delta embeddings, the two **$\\lambda$** parameters need to be kept small (around **1e-2**) to ensure training stability.\n\n - **On the $\\psi$ structure**: We apologize for the unclear explanation of $\\psi$ in the manuscript. **$\\psi$** is implemented as a **single-layer cross-attention Transformer**, where the cross-attention structure is combined with an FFN that does not have the 4× expansion. In our method, we use **$v_j - t_i$** as the query input, where **$t_i$** is the `[CLS]` embedding output from the CLIP text encoder (tensor shape: **(a, dim)**, where **a** is the batch size of text dimension), and **$v_j$** is the mean of **f** frames from the j-th video sample (the frame emb with the shape of **(b, f, dim)** are obtained by the 4 layers of the video temporal Transformer after CLIP vision encoder, where **b** is the batch size of video dimension). So, the input for the pair-wise computation is of shape **(a, b, dim)**.\n   - For the cross-attention part of the $\\psi$ module, since we perform pair-wise delta generation, the corresponding attention score calculation is also parallelized in a pair-wise manner. Specifically, given that the query has a shape of $(a, b, \\text{dim})$, we first permute it to $(b, \\text{dim}, a)$, and then perform a batch matrix multiplication with the key:\n$\\text{logits}=\\text{key} \\times \\text{query} \\quad \\text{i.e.,} \\quad (b, f, \\text{dim}) \\times (b, \\text{dim}, a) \\to (b, f, a).$\nNext, we apply softmax to the logits and permute them again to $(b, a, f)$. Finally, the attention output is computed as: \n\n     $\\text{attn-out} = \\text{attn-scores} \\times \\text{value}, \\quad \\text{i.e.,} \\quad (b, a, f) \\times (b, f, \\text{dim}) \\to (b, a, \\text{dim}).$\n\n     This achieves pair-wise parallel computation, with virtually no delay introduced in practice.\n\n---\n\n# **Q2: Explanation Regarding the Modality Gap**\n\nThank you for raising the question about the modality gap. In fact, the motivation of our work is not to directly address the modality gap itself (i.e., our method does not aim to reduce $|| \\overline{v} - \\overline{t} ||$), but rather to resolve the optimization tension caused by the modality gap. By releasing this tension, we achieve larger $\\text{Var}(t)$ or $\\text{Var}(v)$, allowing us to optimize in a broader space. At the same time, in retrieval tasks, reducing the modality gap is a sufficient but not necessary condition for improving retrieval performance. Even with a large gap, we can still achieve high retrieval performance. By optimization tension, we refer to the issue that arises when optimizing an anchor $t_i$ with respect to its InfoNCE sub-loss:\n\n$\\mathcal{L}\\_i = -\\log \\frac{e^{\\cos(t_i, v_i)/\\tau}}{\\sum_{j=1}^B e^{\\cos(t_i, v_j)/\\tau}}.$\n\nThe gradient of this loss can be decomposed into the form:\n\n$\\nabla_{t_i} \\mathcal{L}\\_i = \\sum_{j=1}^B \\underbrace{(p_{ij} - y_{ij})}\\_{\\nabla_{\\cos(t_i,v_j)}\\mathcal{L}\\_i} \\cdot \\underbrace{\\left[ \\frac{v_j}{\\|t_i\\|\\|v_j\\|} - \\cos(t_i, v_j) \\cdot \\frac{t_i}{\\|t_i\\|^2} \\right]}\\_{\\nabla_{t_i}\\cos(t_i, v_j)}.$\n\nwhere $y_{ij}$ is the matching label.\n\nIn this optimization process, \\$(p\\_{ii}-1) \\cdot \\nabla\\_{t\\_i} \\cos(t\\_i, v\\_i)\\$ pulls \\$t\\_i\\$ towards \\$v\\_i\\$, while \\$p\\_{ij} \\cdot \\nabla\\_{t\\_i} \\cos(t\\_i, v\\_j)\\$ pushes \\$t\\_i\\$ away from \\$v\\_j\\$. However, the modality gap forces the optimization of \\$t\\_i\\$ in two directions: 1) towards \\$\\overline{v}\\$, and 2) away from \\$\\overline{v}\\$. We collected gradients of \\$t\\_i\\$ during training on the CLIP4Clip model, including the positive sample gradient \\$(p\\_{ii}-1) \\cdot \\nabla\\_{t\\_i} \\cos(t\\_i, v\\_i)\\$ and the sum of the negative sample gradients \\$\\sum\\_{j=1, j \\neq i}^B p\\_{ij} \\cdot \\nabla\\_{t\\_i} \\cos(t\\_i, v\\_j)\\$. Summing these gradients and calculating the mean and variance across the 512 dimensions, we found that for the dimensions mainly involved in optimization, the positive gradient is around 40-60, while the negative gradient sum is between -40 and -60. When both of them are summed, the resulting value for these dimensions is around -3 to -4. This strongly suggests that the modality gap induces optimization tension, as the gradient error is concentrated near zero, causing stagnation in \\$t\\_i\\$ optimization and severely limiting the model's alignment ability.\n\n*(As the rebuttal guidelines prevent the inclusion of plots, I am unable to provide the statistical analysis visuals. If the reviewer's  interested, I can provide a detailed explanation of how we observe this phenomenon in CLIP4Clip.)*\n\nTo address the optimization tension on $t_i$, we reallocate the gradient to $\\Delta$, where each $\\Delta$ corresponds to a specific pair and only receives gradients from the corresponding pair. Essentially, this approach lifts the ceiling on cosine similarity computation because, in the presence of optimization tension, the optimization of $t_i$ is confined to a small region near it, but the introduction of $\\Delta$ expands the optimization space. As shown in Figure 2 in the paper, qualitative analysis demonstrates that the introduction of $\\Delta$ alleviates the tension (enabling matching in a larger scale space) while also improving the uniformity characteristics (lower similarity distribution for positive pairs).\n\n---\n\n# **Q3&Q4: Cross-modal Retrieval and False Negatives**\n\nThank you for your suggestion regarding the missing related works on false negatives and the comparison with methods designed to handle false negatives. The methods compared in the current paper are all based on CLIP, typically involving structural modifications and fine-tuning on the well-known CLIP4Clip model used in TVR tasks. In the revised version, we will include additional methods related to false negatives for comparison, including the following cross-modal approaches:\n\n- **CUSA** (AAAI 2024) addresses false negatives by using soft-label alignment from pre-trained models, which reduces the impact of hard labels and enhances sample recognition.\n- **VSE++** (BMVC 2018) improves performance by focusing on the hardest negatives within each batch via Max-Hinge Loss, making the model more robust to false negatives.\n- **L2RM** (CVPR 2024) utilizes an Optimal Transport framework to realign mismatched pairs, improving robustness against hard negatives by refining sample alignment.\n\nThese works address false negatives through various techniques, such as soft-label alignment, hard negative mining, and improved robustness metrics. \n\n### **Comparison with Stronger Methods Designed for False Negatives**\n\nWe now compare **GARE** with two recent methods designed to handle hard negatives in text-video retrieval:\n\n- **DMAE** (ACM MM 2023, R@1: 46.9, +1.6% over base, our reproduction: 47.1) improves fine-grained alignment by mining hard positives, which implicitly helps push away hard negatives. This is conceptually similar to our **IB loss**, which uses a variational bottleneck to compress $\\Delta$, retaining critical alignment signals for robust optimization.\n- **NeighborRetr** (CVPR 2025, R@1: 49.5, +2.3% over base, our reproduction: 49.2) uses a memory bank to compute k-neighbor co-occurrence and identify \"good hubs,\" promoting local consistency and reducing over-penalization of hard negatives. However, **GARE** does not require explicit hard negative mining. Instead, when encountering hard negative sample $v_j$, it shifts part of the loss gradient responsibility to the corresponding $\\Delta_{ij}$, allowing $\\Delta_{ij}$ to absorb the noisy gradient acting on $t_i$. This approach helps alleviate the erroneous optimization of $t_i$, thereby mitigating the noise introduced by hard negatives.\n\n### **Efficiency Comparison**\n\n- **GARE** uses only **one cross-attention layer**, while **NeighborRetr** includes 8 MLPs and multiple transformer/conv layers.\n- **NeighborRetr** requires **10240-sample memory banks** and about **4.5 hours of training**, whereas **GARE** achieves similar performance with **1 hour and 34 minutes** of training and minimal memory.\n\n### **Empirical Observations**\n\nWe observe that $t_i$ in GARE becomes semantically more stable: for semantically similar $v_j$, the similarities $s(t_i, v_i)$ and $s(t_i, v_j)$ are smoother than in the baseline. This supports more accurate $\\Delta$-based alignment and aligns with Fig. 2, where $t_i$ functions as a stable prototype. Ultimately, the enhanced semantic stability of $t_i$ allows $\\Delta$ to carry out more precise fine-grained alignment across similar samples.\n\n---\n\n# **Q5: $\\psi$ in inference stage**\n\nSorry for not making this clear in the paper. The $\\psi$ module also participates in the inference phase during forward propagation, contributing to refining the representation for consistent matching responsibility."}]

Author rebuttal:

Meta-review: Considering that most of the reviewers' concerns have been addressed in the post-rebuttal process, the paper received three BA and one BR. The paper offers a novel mechanism and solution with a solid theoretical foundation, and features rigorous experimental design alongside extensive and qualitative analysis. It is recommended to accept this paper.
Decision: Accept (poster)