Mitigating Semantic Collapse in Partially Relevant Video Retrieval
Authors: WonJun Moon
MinSeok Jung
Gilhan Park
Tae-Young Kim
Cheol-Ho Cho
Woojin Jun
Jae-Pil Heo
Venue: NeurIPS 2025 poster ()
Keywords: Partially Relevant Video Retrieval; Text-Video Retrieval
Primary area: deep_learning
TL;DR: Addresses the problem of semantic collapse in PRVR.

Abstract: Partially Relevant Video Retrieval (PRVR) seeks videos where only part of the content matches a text query. 
Existing methods treat every annotated text–video pair as a positive and all others as negatives, ignoring the rich semantic variation both within a single video and across different videos. 
Consequently, embeddings of both queries and their corresponding video‐clip segments for distinct events within the same video collapse together, while embeddings of semantically similar queries and segments from different videos are driven apart.
This limits retrieval performance when videos contain multiple, diverse events.
This paper addresses the aforementioned problems, termed as semantic collapse, in both the text and video embedding spaces. 
We first introduce Text Correlation Preservation Learning, which preserves the semantic relationships encoded by the foundation model across text queries.
To address collapse in video embeddings, we propose Cross-Branch Video Alignment (CBVA), a contrastive alignment method that disentangles hierarchical video representations across temporal scales.
Subsequently, we introduce order-preserving token merging and adaptive CBVA to enhance alignment by producing video segments that are internally coherent yet mutually distinctive.
Extensive experiments on PRVR benchmarks demonstrate that our framework effectively prevents semantic collapse and substantially improves retrieval accuracy.

Mitigating Semantic Collapse in Partially Relevant Video Retrieval
WonJun Moon', MinSeok Jung', Gilhan Park, Tae-Young Kim, Cheol-Ho Cho, Woojin Jun, Jae-Pil Heo"
Sungkyunkwan University
{wjun0830, alstjr88, a01152a, jackdawson, gersys, junwoojin, jaepilheo)
Abstract
Partially Relevant Video Retrieval (PRVR) seeks videos where only part of the content matches a text query. Existing methods treat every annotated text-video pair positive and all others negatives, ignoring the rich semantic variation both within a single video and across different videos. Consequently, embeddings of both queries and their corresponding video-clip segments for distinct events within the same video collapse together, while embeddings of semantically similar queries and segments from different videos are driven apart. This limits retrieval performance when videos contain multiple, diverse events. This paper addresses the aforementioned problems, termed as semantic collapse, in both the text and video embedding spaces. We first introduce Text Correlation Preservation Learning, which preserves the semantic relationships encoded by the foundation model across text queries. To address collapse in video embeddings, we propose -rosscBranchh Video Alignment (CBVA), contrastive alignment method that disentangles hierar- chical video representations across temporal scales. Subsequently, we introduce order-preserving token merging and adaptive CBVA to enhance alignment by producing video segments that are internally coherent yet mutually distinctive. Extensive experiments on PRVR benchmarks demonstrate that our framework ef- fectively prevents semantic collapse and substantially improves retrieval accuracy.
Introduction
Recently, Partially Relevant Video Retrieval (PRVR) 6. 46 has emerged as significant research challenge computer vision. PRVR shares the same objective traditional Text-to-Vide Re- trieval [26 36 30,
13 16, 31, retrieving the video that best aligns with given text query. However, the key difference lies in PRVR's assumption that target videos may be only partially relevant the query rather than requiring a perfect semantic match. The primary challenge in PRVR lies in learning from text-video pairwise annotations. single video is often associated with multiple distinct text queries labeled positive pairs; however, the semantic relationships among these text queries are not explicitly defined, and fine-grained temporal annotations that indicate their precise alignment within the video are typically unavailable.
As result, conventional training for retrieval based on the InfoNCE loss induces semantic collapse problem in PRVR. Semantic collapse refers to the phenomenon where paired text queries and visual segments are excessively attracted to each other while being indiscriminately repelled from features of other pairs, regardless of their actual semantic similarity. Fig. (a) illustrates this issue within the text embedding space; text queries associated with the same video tend cluster
Conresponning author 'Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Q1: Boy in gray top is playing with action figure."
Q2: "Woman in pink sweater is holding a dog her arms."
Video A
QI: "Blonde woman holds dog up to her chest."
Q2: "A woman gets some carrots to feed to her white horse." "
Video B
Query Representation
Video Representation
Positive/Negative Semantic Relation
qZ
Q2
42
QI
Q1
(a) Text Semantic Collapse
(b) Video Semantic Collapse
Figure 1: Illustration of semantic collapse. (Up) Untrimmed videos in PRVR encompass diverse semantics that be described by different texts. As result, semantic segments (both text and video clips) from the same video may convey very different meanings, while segments from different videos can nonetheless be closely related. For example, Q2 of Video A and QI of Video both depict "holding a dog". (Down) Since all queries tied to a given video are treated as positives and negative queries drawn from other videos, the model pulls together all text embeddings (and their corresponding video segments) for that video, regardless of true meaning, and pushes apart semantically similar queries (and segments) from different videos. (a) illustrates that queries of the same video are pulled together regardless of their semantic relationships (left), while queries with similar context (holding dog) are pushed apart (right). (b) shows that video segments also suffer from the same phenomenon.
together even when they are semantically unrelated, while semantically similar queries are pulled apart when they are paired with different videos. In addition, the same phenomenon occurs in video embeddings; video segments drawn from the same video collapse together regardless of their true semantic differences, shown in Fig. [ (b). This is because the training guidance is provided by video ID, not by their individual semantic content. In short, every segment in a video shares the identical set of paired text queries as positives.
Previous works, e.g., GMMFormer [47 and GMMFormer-vz v2 46 have attempted address the semantic collapse within text embeddings. Specifically, these methods explicitly reduce the similarity between text queries paired with the same video. However, the semantic relationships between text queries are often overlooked, and the issue of semantic collapse within video embeddings remains underexplored, leading to sub-optimal performance.
In this paper, we aim to mitigate the semantic collapse in both text and video embeddings for PRVR. First, we introduce Text Correlation Preservation Learning (TCPL), which leverages CLIP (37). vision-language foundation model with a well-structured semantic space. By distilling the seman- tic relationships encoded in CLIP, TCPL effectively regularizes the semantic collapse within text embeddings. While TCPL leverages CLIP's s rich text-semantic structure to regularize collapse in the textual embedding space, we point out that the same approach cannot be directly applied to video embeddings. This is because CLIP's pretraining operates on static images, thereby lacking the capacity to model temporal dynamics 277.
To this end, we introduce Cross-Branch Video Alignment (CBVA), a dedicated objective to pre- serve context diversity in the video modality. CBVA utilizes dual-brande architecture commonly adopted in PRVR to encode hicrarchical video representations and employs contrastive objective to differentiate distinct events within video. Concretely, frame- and clip-level embeddings from the same timestamp are encouraged to align closely, while those from different timestamps are driven apart. Then, we further leverage the token merging strategy in two ways to enhance adaptivity within CBVA; (1) aggregation, and (2) bipartite token merging [1] is leveraged to organize representative contexts within each video. By encoding clips in a context-awand manner, we encourage videos to be represented in line with their true semantic content. Consequently, with TCPL. and CBVA combined, our method achieves state-of-Ne the-art performances in all tested benchmarks.
token merging is introduced for semantically consistent video clip

---

In summary, our contributions are (I) We propose Text Correlation Preservation Learning, which leverages the semantic relationships within the foundation model to address semantic collapse within text embeddings, (2) We propose Cross-Branch Video Alignment to mitigate the semantic collapse in video modality by distinguishing distinet events within a video, (3) We leverage token merging strategies to encourage the precise video alignment, and (4) Our method achieves superior performances across all datasets in PRVR.
2 Related Work
Partially Relevant Video Retrieval. PRVR aims to retrieve untrimmed videos that are partially relevant to a given query 6
MS-SL [6] addresses this challenge by proposing a dual encoding strategy that explicitly separates features for frame and clip segments, capturing different temporal scales within untrimmed videos. Subsequently, DL-DKD 7 leverages CLIIP [37 to enhance PRVR performance by distilling text-frame similarity. GMMFormer [47] introduces Gaussian Mixture Model-based Transformer that enables efficient retrieval with reduced set of video features. It also identifies semantic collapse key challenge and proposes query-diverse loss to enforce separation among multiple text queries linked to the same video. Building on this, GMMFormer further addresses semantic collapse by explicitly controlling the degree of semantic separation between queries associated with the same video. Unlike these methods that only enforce separation among a small set of queries, our approach aims leverage their true semantic relationships and additionally mitigates semantic collapse in the video embedding space.
19
20 5T
v2 [6]
Knowledge Distillation. The aim of knowledge distillation is to train student model with fewer parameters to achieve performance comparable to larger teacher model [15]. For classification tasks, Kullback-Leible divergence loss is widely applied to align the student's output distribution with that of the teacher after the softmax layer, allowing the student model to learn from the teacher's s predictions. Subsequently, transferring knowledge at the intermediate feature level has been the next stream [5 18 4 However, as they fail to effectively capture the relationships between individual features, Relational Knowledge Distillation (RKD) [35 29 41 was proposed to distill the relationships within the semantic space of the teacher model to that of the student. In PRVR, the problem of semantic collapse occurs due the lack of consideration for relationships among queries paired with the same video, as well as across queries from different videos. Therefore, we leverage RKD transfer structured semantic relationships within the foundational model to typical PRVR that often suffer from semantic collapse.
network designs 66 47 46)
Token Merging, Token merging 2 34 has been proposed to improve the efficiency of Trans- former 42) by reducing token redundancy. A representative method, ToMe M. uses bipartite matching token similarities to merge spatial tokens in the vision transformer. Recently, token merging strategies have been extended to the video domain. For example, LearnableVTM (23) learns per per-patch saliency scores and applies for merging across long vidcos. TempMe [38 sequentially merges tokens within progressively larger fixed-window clips, addressing both spatial and temporal redundancy for retrieval. In contrast, our work applies token merging for two purposes: adjacent video frames to assemble coherent contexts in each video clip, and
merge
leverage token merging to determine the representative context within each video. These facilitate precise alignment between hierarchical video representations.
3 Method
3.1 Preliminary
Our architectural design is illustrated in Fig. 2 Similar to prior works, we employ pretrained encoders to extract tokens, which are processed through trainable layers.
Text encoder. Given a batch of text inputs, we utilize the pre-trained text encoder to extract text tokens T R B, L,, xd4, where By. Tag and dy denote the number of text queries, the number of words per query, and the dimension of query representation, respectively. The sequence of word tokens includes [SOS] (start of sequence) at the beginning and [EOS] (end of sequence) the end, making the total number of tokens Ly. These tokens are forwarded through projection layers and transformer layers to produce text representations f g L,, for downstream text-video retrieval, where d denotes
3

---

Sec. 3.2
Sec. 3.3
01: Flondo woman holds dog up to ber chest
Text Correlation Preservation Leaming
Order- Preserving Token Merging
BDB
Vision Encoder
Text Encoder
Taken Merging
M
T-ES
CLIP
Clip Features
Freme Features
Transtormer
Transfomer
Finetined
Transformer
Video A Ouery Tokes
Pociive Relstionn
Quary Tokan
Negative Redation
Cross- Branch Video Aignment
thn. Pooling
base
CBVA
VE
Figure 2: Method overview. We extract text and visual tokens with pretrained backbones, which are then processed via transformer layers. Text tokens are aggregated via attention pooling to produce single query token I for each text query. Also, following prior works, dual-branch visual tokens are encoded (both frame- and clip-level), producing a sequence V of video tokens for each level. baseline retrieval loss ghese aligns T with the most similar video token at each level. To mitigate text-side semantic collapse, Text Correlation Preservation Learning transfers CLIP's query relationships. On the other hand, Cross-Brande Video Alignment aligns hierarchical segments by timestamping to mitigate collapse and preserve visual details. Furthermore, CB VA is precisely enhanced by constructing coherent clips with Sadde--Preservee Token Merging and improving adaptivity (illustrated in Sec. 3.3
the projected dimension. Finally, attention pooling is applied to f to derive a single aggregated token E R n xd that represents the final representation of the text query.
Video encoder. For batch of B, videos with I. frames each, we utilize the pre-trained image or video encoder to extract a visual token (e.g. [CLS] token from CLIP) for each frame, generating frame tokens V E B Additionally, to represent moments of varying temporal lengths, the frame tokens V are aggregated into video clips in the clip branch, to generate clip-level tokens V. E BRB.,L,,.e where L., denotes the number of clips per video. Note that our clip construction process is performed with order-preserving token merging, which is discussed in Sec. 3.3 Then, each frame and clip token is encoded independently through the transformer layers to capture contextual relationships. Consequently, Vf BR x Xd and Ve IR B, x are produced for final video representations.
Training objective. To retrieve a video with the given text query, we perform similarity matching between the representations from two modalities. Specifically, during training, we first select one video token per video that yields the highest similarity to the given text query in both frame and clip branches. Then, these video tokens (one from each video representation) are used to conduct retrieval for training using InfoNCE loss 3 21 and triplet ranking loss [ Accordingly, the final training objective is formulated as follows.
Chase
L + Chip
L
trip
where Lese and Ltip indicate the InfoNCE loss and triplet ranking loss, respectively, and E and Lj represent the clip-level loss and frame-level loss, respectively.
Problem definition: semantic collapse. Existing PRVR approaches suffer from semantic collapse which indicates that the general relationships among queries and videos are disrupted. This phe- nomenon occurs because pairwise text-video annotations (which only specify positive relationships) are used for learning PRVR. Specifically, in PRVR, each video is associated with multiple distinct text queries, which triggers the typical contrastive learning to encourage the queries paired with the same video to cluster together, while text queries paired with different videos are separated as they are attracted to different videos. In this work, we attempt to alleviate the semantic collapse within the text embedding in Sec. 3.2 and video embedding in Sec. 3.3

---

3.2 Semantic Collapse in Text Embeddings: Text Correlation Preservation Learning
Previously, GMMFormer 47 and GMMFormer-v2 46 have attempted to address semantic collapse in that they enforced separation between text queries paired with the same video. However, we argue that they only partially alleviate the semantic collapse since all text queries paired with the same video are pushed apart without considering their actual semantic relationship.
To mitigate this issue, we propose Text Correlation Preservation Learning (TCPL., which leverages the well-structured semantic space of CLIP. Specifically, TCPL explores the semantic relationships between text queries within the CLIP semantic space and distills the relationships toward the retrieval space. In this work, we measure the relationships with two metrics: Euclidean distance and angular distance. These two metrics are defined with the pair (x, y) and triplet (x, y, 2), where x, y, and denote text embeddings, respectively, as follows:
fx y)
||x - y ((x,y,2) /
y x y
z y
2
y112
S and l denote Euclidean and angular distance functions, respectively, represents the average distance among all tokens in the mini-batch and (x.y) y) denotes the dot product of x and y.
To measure the semantic relationships within the text embedding space of CLIP, we first gather [EOS] tokens of CLIP in the mini-batch. We define the set of [EOS] tokens in a mini-batch as follows:
7:8OS : T2..,...... TBB.L. E R Bg Xdeur
where T, represents the [EOS] token of the first text query within the mini-batch. Note that [EOS] is used for the distillation since [EOS] conveys more informative clues than other tokens in CLIP 49 and using [EOS] reduces computational overhead compared token-wise distillation. Then, the knowledge of CLIP is distilled towards the encoded text tokens, T. Specifically, we distill the pairwise Fuclidean distance relationships and triplet angular distance relationships from the CLIP text embeddings into the text-vion text-video joint embedding space. The distillation process is expressed as:
Σ C" (f
BB 1)
L.EE $43
Ca
Σ (" ThOs " (T,,,TT,T 3 B3 LJR.EB,
(5)
where B, = 2,, B,} stands for a set of indices such that |B,l = By and L denotes Huber loss [14], which leads stable training by behaving L2 loss for small errors and LI loss for large errors. Finally, the objective for TCPL is defined as follows:
CTCPL = _ XAC,
(6)
where F and X are weights for L and LA respectively, By preserving the well-structured semantic relationships within the foundation model, TCPL. mitigates semantic collapse within text embeddings.
3.3 Semantic Collapse in Video Embeddings: Cross-Branch Video Alignment
Semantic collapse also occurs within the video modality. While the conventional text-video retrieval loss effectively pushes apart videos with different semantics, does not explicitly preserve the multi-contextual nature of events within single video. As result, contextually distinct segments within the same vidco may collapse into similar embeddings, limiting intra-video discriminability.
Therefore, we introduce Cross- Brasch Video Alignment (CBVA) that aims to disentangle the repre- sentations of distinct events within video, thereby mitigating semantic collapse. Specifically, we leverage the representations from the typical dual-branch architecture used in PRVR frameworks, with separate encoders for clip- and frame-level branches 47) In CBVA, timestamp correspondence leveraged to align each video frame with its matching clip segment while repelling it from segments at other timestamps. However, simply aligning different levels of video representation proves ineffective. This issue stems from the common practice of generating clip segments by uniformly verage-poollngg fixed-length segments [ 46), which causes each clip to cover multiple contexts that can overlap across adjacent segments.
5

---

Order-Preserving Token Merging, To address the fragmentation of temporally adjacent content in untrimmed videos, we first introduce Order- Seder-Preserving Token Merging (OP-ToMe) to construct consistent clip segments Ve, as shown in Fig. 2 Unlike general token-merging schemes that may fuse tokens from arbitrary spatial or temporal locations [ 38. OP-ToMe restricts all merging operations to pairs of tokens drawn from successive frames, thereby preserving the original playback order (for stable temporal modeling). Concretely, given a sequence of per-frame tokens, we first compute cosine similarities between disjoint adjacent-frame pairs. We then select the approximately top-N% of most similar adjacent-frame pairs and merge each into a single clip token. This merging procedure is repeated for M iterations until the frames are aggregated into the standard 32 clips used in prior work. At each merge, the two tokens are fused via a weighted average of their feature vectors. Note that the proportional attention mechanism [ is integrated in our framework account for each token's 's size (the number of raw frames it represents). By repeating this process, OP-ToMe produces condensed sequence of clip segments that (1) maintain strict temporal order, (2) retain coherent contextual semantics, and (3) reduce redundant information across llames-propert for robust performance in PRVR. We provide the algorithm for OP-ToMe in the Appendix.
that are crucial
Cross-Branch Video Alignment. Once the context-consistet clips are constructed via OP-ToMe, we perform cross-branch contrastive learning to encourage fine-grained temporal discriminability within each video. Specifically, each clip token and its corresponding frame tokens are treated as positive pairs, while frame tokens from other temporal moments in the same video regarded as negatives. This facilitates the model in learning to distinguish between different contextual segments of single video. Formally, given that V. = {oe ),= frame-level video tokens respectively, we also define the set of associated frames of each clip i as:
Jimi and V {b))) 3 denote the clip-level and
F, = (+/10)) = i}, X, = 1FrlF
where 8(.) returns the clip index of frame among the Le clips. Then, the objective of CBVA is formulated with frame-to-clip and clip-to-frame NCE as:
1 Lg L
exp(sim( viy, tic
8(1)
exp(sim(F.(zt)) 1 log Ly Σ exp((im(()) (r) o)))
C
log
(8)
j== exp(sim oy. of)) L (=1
i=1
where sim((s) .) denotes cosine similarity and F1[x]] is the æ-th frame token in the set F,
Adaptive CBVA. Although CBVA disentangles different contexts within a single video, real-world footage often contains an unknown (potentially variable) number of distinct contexts. Consequently, applying the contrastive objective in Eq. with a fixed clip length Le may introduce noise: for example, an interview video composed of largely homogeneous frames will nonetheless be split into Tue segments, unnecessarily fragmenting coherent content. To address this, we first estimate the number of contexts in each video and then adaptively aggregate LE representative clips to guide precise CBVA. We employ bipartite token merging [ to extract representative clip segments, since semantically similar content may occur intermittently or across non-contiguous intervals within video. However, optimizing the number of semantics per video is costly during the token merging process. Therefore, we instead pre-define discrete set of clip numbers based fixed merge rate, and then match each video to the level that best reflects its internal similarity structure (number of different semantics). To initially establish discrete set of clip levels, we define N% to denote the merge rate and Cmin to represent the minimum number of semantically different clips in each video. Then, generate K levels of clip number candidates {LO)L by recording clip number after each merge step as:
L: -1:((//2 (N/100) + +11, Ciiia)
Le L = max(2 x
(9)
2
and let K be the largest index for which T,K 2 Cmin- Next, we compute high-similarity ratio w for each video by measuring the fraction of clip-pair cosine similarities (using frozen features from the backbone Vo) that exceed a threshold r. A low indicates many distinct contexts, so we retain the full original clip set (L: = L.). Otherwise, we select the smallest E {l..... , K satisfying w >>> K and perform - iterations of bipartite merging at rate N%, yielding L: = L final clips. We remark that, for simplicity, we use the same merge rate N% as OP-ToMe. Consequently, in Fq. 8 the original clip segments are replaced with these merged clips to further enhance video adaptivity. Detailed algorithm for both merging processes are provided in the Appendix.

---

Table 1: Ablation study on QVHighlights dataset. Table 2: Performance when using variants of
video correlation preservation learning instead of Cross-Branch Video Alignment.
Model
R1
RS
RIO
R100
SumR
Baseline
21.8
48.1
60.6
95.0
225.5
Method
R1 R5 RTO R100 SumR
(b)
TCPL
22.8
49.5
63.3
95.0
230.6
(a) TCPL. baseline
22.8 49.5 63.3 95.0 230.6
Naive CBVA
22.8
49.4
63.7
95.0
231.0
(a)+ Retrieved segment
23.4 50.4 63.4 94.6
231.7
OP-ToMe
24.2
50.4
63.0
94.9
232.5
(a)+ Uniform Sampling
22.5 50.8 64.1
94.9
232.3
Adaptive CBVA
23,9
51.5
63.7
95.5
234.6
Ours
23.9 51.5 63.7
95.5
234.6
3.4
Total Training Objective
Finally, our total objective with retrieval, TCPL, and CBVA losses expressed as:
Coverall
- + X (CBVACCBW)
(10)
Experiments
Datasets & Metrics. We evaluated our method on four PRVR datasets: OVHighlights
TVR 25
ActivityNet Captions [22]. and Charades-STI STA [12]. OVHighlights 24] is a collection of news and vlog-style videos, recently reorganized for PRVR [2). Each video paired with an average of 3.3 text queries describing semantically diverse segments. TVR (25 is built from scenes across six popular TV shows, with each video annotated by five text queries targeting different segments. The training set contains 17,435 videos and 87,175 queries, while the evaluation set includes 2,179 videos and 10,895 queries. ActivityNet Captions [22] is sourced from YouTube videos, with an average of 3.7 text queries per video. The dataset includes 10,009 videos for training and 4,917 for evaluation. Charades-STA [12] extends the original Charades dataset by adding sentence-level annotations for specific temporal segments. It consists of 13,898 video-sentence pairs for training and 4,233 for evaluation. For evaluation, we use recall-based metrics, which are commonly used retrieval 44]. We denote this metric as R R@Q Q, where Q represents the proportion of queries for which the correct video appears within the top-Q ranked results. Additionally, SumR is the sum of all R@Q used for evaluation, assessing the overall retrieval performance.
tasks [43 11 48 17
Implementation Details. For feature extraction, we follow recent works 15] 33 32 : we extract video features with CLIP-B/32 (37) and Slowfast [10), and use CLIP-B for text embeddings for QVHighlights, and use CLIP-L (37 for encoding both modalities in other datasets. Hyperparameter configurations are adopted from GMMFormer-v2 [46] (e.g., learning rate, batch size, epochs, and optimizer settings) except for the fusing ratio between the frame and clip branches. We assign a frame score weight of 0.6 and clip score weight of 0.4. All loss coefficients are fixed across datasets: X = 15, A = 30, and CBVA = 0.1. To construct consistent clips with OP-ToMe, we set N to 75% (Note that M is then computed automatically from N to match the number of clips used in prior works [46 6)) Finally, we set the minimum clip count per video to Cmin = 5, and set a similarity threshold 7 to 0.7 for QVHighlights, 0.8 for TVR and CetivittNee-Caaptiin The reason behind using varying 7 is that the internal segment-to te-eemetto-seemeet similarity distributions differ: QVHighlights exhibits the lowest similarities, TVR and ActivityNet-Captionn are intermediate, and Charades shows the highest. All experiments are conducted on single RTX A6000 GPU and an Intel Xeon Gold 6338 CPU (2.00GHz) for all datasets.
and 0.85 for Charades.
4.1 Ablation Study
Studies are conducted on QVHighlights, which includes numerous events in each untrimmed video.
The default configuration used to generate the reported results is highlighted in grey.
Component ablation. To quantify the contribution of each module, we report a component-wisg
ablation in Tab. [ Our baseline is built upon GMMFormer-v2 architecture [46], only trained with the standard retrieval loss Chase, Then, we sequentially add Text Correlation Preservation learning (TCPL) and Cross-Branch Video Alignment (CBVA), which are introduced in Sec. 3.2 and Sec. [.3] Initially, in row (b), incorporating TCPL mitigates semantic collapse in the text embedding space, yielding notable gain over the baseline. From row (c) to (e), we subdivide the CBVA into (c) Naive CBVA, (d) adding Oo-ToMe, and (c) applying adaptive CBVA. Specifically, the basic CBVA objective
7

---

Table 3: Ablation studies of various components on QVHighlights. 'Coef" denotes coefficient.
(a) TCPL ratio.
(b) TCPL coef.
(c) TCPL Source.
(d) CBVA coef. (c) Merge rate.
(f) Threshold
A x
SumR
X
X
SumR
Model
Sunk
X X
SumR
N%
Sumk
SumR
229.7
5
10
231.5
CLIPB
234.6
u.1
234.6
50
2326
0.5
234_3
2:1 (30.15)
231.8
10
20
233.5
CLIP-I
235.6
0.15
234.9
75
234.6
0.6
233.5
(15,33))
234.6
30
234.6
OpenCLIP-B
235,4
0.2
232.9
0.7
234.6
20
40
232.5
OpenCLIP-L
232.6
produces only marginal increase in performance since fixed-length clip segments may encompass multiple overlapping contexts. However, we find that augmenting CBVA with Oo-ToMe to construct semantically consistent clip segments drives performance boost by reducing spurious alignments across events. Finally, dynamically adjusting each video's 's clip count according to the estimated number of video contexts further refines the alignment, producing substantial gain. These results confirm that addressing both the text- and video-side semantic collapse is significant for PRVR.
Video Correlation Preservation Learning (VCPL). Similar to TCPL. one can assume that we can apply the identical approach to video embeddings to mitigate semantic collapse. However, this direct adaptation is suboptimal since CLIP's video embeddings cannot model temporal dynamics. To substantiate this, Tab. 2 compares VCPL against our CBVA. 'Retrieved segment" is conducted similarly to TCPL; we first select the representative video token for every text query by identifying the token with the highest similarity within the paired videos (using ground-trum truth pair) and distill the relationships between representative video segments. Also, we study the variant of VCPL where uniformly sample 4 segments per video and conduct relation learning between all sampled segments Although these approaches yield modest improvement, we find that these
from the variants lag behind VA by 2.3 points in SumR. VCPL is applied to both clip and frame branches.
Loss coefficients. For our training objective, we control the TCPL loss with X and XA, and the CBVA loss with A XBVA In Tab. 3a we first studied the X X4 over {1 1,2 : 1, 2). Then, in Tab. 3b with 1:2 ratio, which yields the best performance, increasing both weights to (15, 30) improved performance; beyond that, gains plateaued. For CBVA, in Tab. 3d performance rose as CBVA increased up to 0.15, but for simplicity across datasets, we fixed it at 0.1.
TCPL source model. By default, we use the pretrained text encoder as the source model for TCPL to provide semantic relationships (CLIP-B for HVHigligghts and CLIP-L for other datasets). To assess sensitivity to the source model, we replaced CLIP-B with alternative lision-language encoders and measured SumR on the QVHighlights dataset in Tab. 3c As observed, swapping in the larger models (e.g., CLIPPL and OpenCLiP-L.) increased SumR by up to 1.8 points. These results indicate that TCPL's effectiveness scales with the quality of the source model's is semantic structure.
Token-Merging Ratio. We use a single merge rate N % for both OP-ToMe and adaptive CBVA. Em- pirically, setting N to approximately 75% reduces 128 frames to 32 clips in only a few steps (matching the standard PRVR frame/clip counts), while keeping computational overhead minimal. As Tab. 3e shows. increasing the number of merge iterations while lowering the per-step ratio to 50% actually degraded accuracy. Thus, we fix N 75% across all datasets.
Adaptively measuring video context number. We determine the optimal number of contexts for each video by thresholding the pairwise similarity among its clips at a value In this work, we vary T to evaluate how sensitive our context-count estimation is to this threshold. As shown in Tab. [i the adaptive CB VA method exhibits only minor fluctuations across different values, indicating that is robust to the choice of similarity threshold between 0.5 and 0.8.
4.2 Comparison with the State-of-the-Art
Table 4: Results on QVHighlights. f denotes reproduced results.
HVHighlights. In Tab.
report results on QVHighlights [24], a recently introduced benchmark for PRVR. To illustrate, our method outperforms the previous state of the art by up to 8 points in SumR. We attribute these gains to method's is capability to
Methods
R1
R5
RI0
R100 SumR
MS-SL
6
20.4
46.7
60.7
94.6
222.5
GMMF
47
18.2
43.7
56.7
92.5
211.1
AMDNe1
39
17.4 40.8 55.0
93.4
206.6
mitigate semantic collapse, especially when videos exhibit frequent and rapid event transitions.
BGMNet
50
20.6
46.3 58.8
94.0
219.7
GMMF-v2
[46
21.7
48.0
60.5
95.0
225.2
TVR & ActivityNet-Captionn
& Charades. Tab. 5 reports results on these three datasets. Specifically,
ProtoPRVR
32
22.6
48.8
61.3
93.9
226.6
Ours
23.9
51.5
63.7
95.5
234.6
8

---

Table 5: Performances on TVR, ActivityNet Captions, and Charades-STA using CLIP-L/14 backbone. f are reproduced results, and all results on Charades are reproduced with official codes.
TVR
ActivityNet Captions
Tharades-STi
Method
R1
R5
R10
R100
SumR
R1
R5
RIO
R100
SumR
R1
R5
R10 R100
SumR
CLIP zero-shot
16.2 33.5 41.8
75.7
167.2
15.1
33.9
45.1
78.9
172.9
2.0
8.1
13.6
49.4
73.1
MS-SL [6]
31.9 57.6 67.7
93.8
251.0
14.7
37.1
50.4
84.6
186.7
3,4
11.5
18.7
62.5
96.0
GMMF 47
29.8 54.2 64.6
92.5
241.1
15.2
37.7
50.5
83.7
187.1
2.7
10.5
16.7
59.4
89.3
AMDNel
39
27.7 52.3 63.3
92.3
235.6
14.0
36.3
49.9
84.2
184.5
2.1
7.8
13.9
57.2
81.1
BGM-Net
50
31.1
56.3 66.5
93.8
247.7
15.6
37.9
51.3
85.4
190.3
3.0
11.8
18.2
63.7
96.7
GMMF-v2
46
34.0
59.7 69.8
94.5
258.1
17.1
40.6
53.7
85.5
196.9
3.1
11.6
18.2
61.4
94.2
ProtoPRVR
32
34.7
60.0
70.1
94.4
259.2
16.0
38.8
52.4
85.1
192.3
ARL [S]
34.6
60.4
70.7
94.4
260.1
15.3
38.4
51.5
85.2
190.4
Ours
35.1 61.6 71.5
94.9
263.1
17.7
42.0
55.6
86.8
202.1
3.2
12.6 20.1
63.8
99.7
Table 6: Inference time (ms) and memory (MB) across varying size of video database.
Number of Videos
Method
Metric
100
200
300
400
474
Time (ms)
3.09
3.85
4.66
5.14
5.58
MSSL
Memory (MB)
717.47
796.15
874.83
954.14
1010.89
Time (ms)
1.97
1.98
1.99
2.02
2.05
GMMF
Memory (MB)
243.11
248.95
254.78
260.62
264.10
Time (ms)
2.31
2.38
2.40
2.61
2.78
GMMF-v2
Memory (MB)
419.75
440.18
459.62
480.55
493.46
Time (ms)
2.32
2.37
2.40
2.60
2.70
Ours
Memory (MB)
419.75
440.18
459.62
480.55
493.46
our method achieves state-of-the-art results on all datasets. The performance gains on these datasets are relatively modest compared to QVHighlights, primarily because QVHighlights exhibits very little overlap between different queries and video segments for the same video, making it especially susceptible to semantic collapse. Despite this, our method maintains state-of-the-an performance across all benchmarks, underscoring its generalizability and effectiveness.
Efficiency. In Tab. 6 we report infer- ence/training time and memory, along with model parameters and FLOPs on QVHigh- lights. Reported times are averaged over 5 runs. For the inference, we measure the infer- time and memory across database sizes from 100 to 474 videos. As shown, our method attains the second-lowest inference latency and memory footprint while achieving substantially higher retrieval accuracy. Note that inference time refers to query time since video features are precomputed and cached in practical deployments. Training statistics in Tab. show higher time and memory due to learning fine-grained video context, but this cost is paid offline, whereas inference efficiency governs wal-world deployment where latency and memory are critical.
Table 7: Training efficiency and model complexity.
Training details
MSSL
GMMF GMMF-v2
Ours
Time / epoch (ms)
10,934
12,828
17,223
62,641
Memory (MB)
2,375
3,333
7,826
9.755
Model params (M)
4.57
12.72
32.14
32.14
FLOPS (G)
0.37
0.99
2.78
2.78
4.3 Analysis
Table 8: Semantic similarity comparison between and video instances per video. Intra Sim is the
average similarity among instances of the same video, Total Sim is the average pairwise similarity
across all instances, and Diff. Norm is computed (Intra Sim Total Sim)/ (Intra Sim + Total Sim)
to represent the normalized gap between Intra Sim and Total Sim.
Method
Modality
Intra Sim Total Sim Diff. Norm Modality Intra Sim Total Sim Diff. Norm
GMMF [7
0.1175
0.0113
0.8245
0.6419
0.0623
0.8230
G2 46
Text
0.1646
0.0196
0.7872
Video
0.6041
0.0387
0.8796
Ours
0,2198
0.0813
0.4600
0.5531
0.0812
0.7440
Similarity Structure. We compare the pairwise similarity between queries (video segments) asso- ciated with the same video (Intra Sim) and between all instances across videos (Total Sim). the relationship between contexts and their descriptive queries within each video were indistinguishable

---

from that observed across different videos. Diff. Norm would equal O: if every context within video were identical, Diff. Norm would equal 1. For the analysis, we leverage QVHighlight assess semantic collapse via similarity structure, it exhibits relatively minimal semantic overlap among queries within the same video. As shown in Tab. g our method substantially reduces Diff. Norm to a point where we claim that our method preserves an appropriate level of relative coherence within each video (not too low) while also mitigating semantic collapse (not too high).
Spearman rank correlation with CLIP. We assess whether our method effectively preserves the semantic structure compared baseline approaches. Specifically, we measure how each method preserves the semantic structure of CLIP using Spearman's 's rank
Table 9: Spearman's rank cor-
relation with CLIP.
Method
CLIP
Baseline
35,40
correlation [4]]. For the evaluation. we use the pooled text to-
MS-SL 66
37.17
kens T from each PRVR model to compare with the [FOS] tokens within CLIP query embeddings. Tab. | demonsttattss how our pro- posed method well preserves the semantic relationships between text queries, thereby mitigating semantic collapse.
GMMF 47
36,06
GMMF- +2 46
35.74
Ours
68.18
Qualitative results. Fig. shows qualitative re-
Query: "The camera is submerged in the water filming the occan and divers." "
trieval results for text query. Our method cor- rectly retrieves and localizes the video token that
Rank 1
Rank 2
Rank
overlaps the query's 's target moment (within addi-
tional temporal margin [22 28 32 whereas the
baseline models are distracted by superficially similar content (depicting generic ocean scenes). This failure stems from their embedding collapse, which blurs distinct events with similar global semantics. In contrast, by preserving fine-grained semantic structure, our approach disambiguates these contextually similar contexts and retrieves the exact segment corresponding to the query.
GTAvideo
(a) GMMForeer
GT Video
(b) GMMFomer-v2
GT Video
5 Conclusion & Limitation
(c) Ours
Figure 3: Retrieval example. 'GT Video' denotes the ground-truth paired video the query. v, A, and X indicate whether the retrieved video token is semantically aligned not, regardless of its origin from the ground-truth video.
Conclusion. In this paper, address semantic collapse in PRVR, where semantically diverse text queries and video segments are undesirably attracted or repelled due to pairwise annotation schemes. To mitigate this, we propose a unified framework consisting of Text Correlation Preservation Learning (TCPL) and Cross- BrascBBranch Video Alignment (CBVA). TCPL distills the relational structure from CLIP preserve semantic consistency across text queries, while CBVA aims to structure video embeddings according to their inherent semanties, supported by our token merging strategies. Extensive evaluations highlight the importance of addressing semantic collapse for effective PRVR.
Limitation. Our method has two limitations. First, our method builds upon the pretrained CLIP model, it can inherit weaknesses; it may struggle with fine-grained spatial/directional queries (e.g., distinguishing "left of" from "right of"). However, we emphasize that this limitation does not extend to compositional understanding. As we demonstrate in the Appendix, our method actively corrects CLIP's common failure modes where the queries involve multi-entity contexts and multi-event temporal compositions (recovering 28% of CLIP's R@I 1 failure cases and 57% of its R@ 10 failure cases). Second, our framework incurs an increased training cost. However, for deployment, our model architecture does not introduce any new modules that increase inference time, incurring no additional latency compared to standard retrieval baselines.
Acknowledgements
This work was supported in part by MSIT/ITT (No. RS-2022-11220680, RS-2020-1120182) RS-2211119 RS-2019- I1190421, RS-2024-00459618. RS--2224000360227, RS-2024-00437633, RS42024-00437102, RS- 254-2554,,,,9,. MSIT/NRF (No. R0-2024-00335722), and KNPA/KIPoT (No. 2533-223332
10

---

References
[1] D. Bolya, C.-Y. Fu, X. Dai, Zhang, C. Feichtenhofer, and J. Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations, 2023.
[2] D. Bolya and J. Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4603, 2023.
[3] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[4] W.-C. Chen, C.-C. Chang, and C.-R. Lec. Knowledge distillation with feature maps for image classification. In Computer Vision-ACCY 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2- 6, 2018, Revised Selected Papers, Part III 14, pages 200-215. Springer, 2019.
[5] C.-H. Cho, W. Moon, W. Jun, M. Jung, and J.-P. Heo. Ambiguity-restrainee text-video representation learning for partially relevant video retrieval. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 2500-2500 2025.
[6] Dong, X. Chen, M. Zhang, X. Yang, S. Chen, X. Li, and X. Wang. Partially relevant video retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 246-257, 2022.
[7] J. Dong, M. Zhang, Z. Zhang, X. Chen, D. Liu, X. Qu, X. Wang, and B. Liu. Dual learning with dynamic knowledge distillation for partially relevant video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11302- 11002-11311 11312, 2023.
[8] F. Faghri, D. J. Fleet, R. Kiros, and Fidler. Vse++; Improving visual-semantic embeddings with hard negatives. 2018.
[9] B. Fang, W. Wu, C. Liu, Y. Zhou, Y. Song, W. Wang. X. Shu, X. Ji, and J. Wang. Uatvr: Uncertainty- adaptive video retrieval. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13723-13733, 2023.
[10] C. Feichtenhofer, Fan. Malik, and He. Slowfast networks for video recognition. Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211, 2019.
[11] V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In European Conference on Computer Vision, pages 214-224 Springer, 2020.
[12] J. Gao, C. Sun, Z. Yang, and R. Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267-52777 5275, 2017.
[13] P. Guan, R. Pei, B. Shao, Liu, W. Li, J. Gu, H. Xu, S. Xu, Y. Yan, and E. Y. Lam. Pidro: Parallel isomeric attention with dynamic routing for text-video retricval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[14] T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[15] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge a neural network. arXiv preprint arXiv: 1503.0033, 2015.
[16] S. Huang, B. Gong, Y. Pan, J. Jiang, Y. Lv, Y. Li, and D. Wang. Vop: Text-video co-operative prompt tuning for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6565-6574 2023.
[17] Y. K. Jang, D. Kim, Z. Meng, D. Huynh, and S.-N. Lim. Visual delta generator with large multi-modal models for semi-supervised composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16805-16814, 2024.
[18] M. Ji, B. Heo, and S. Park. Show, attend and distill: Knowledge distillation via attention-base based feature matching. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 7945-7952, 2021.
[19] X. Jiang, Z. Chen, X. Xu, F. Shen, Z. Cao, and X. Cai. Progressive event alignment network for partial relevant video retrieval. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 1973-1978. IEEE, 2023.

---

[20] W. Jun, W. Moon, C.-H. Cho. M. Jung, and J.-P. Heo. Bridging the semantic granularity gap between text and frame representations for partially relevant video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39. pages 4166-4174, 2025.
[21] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-1881 2020.
[22] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Nichles. Dense-captioning events in videos, In Proceedings of the IEEE international conference on computer vision, pages 706-715, 2017.
[23] S.-H. Lec, J. Wang. Zhang, D. Fan, and X. Li. Video token merging for long video understanding. Advances in Neural Information Processing Systems, 37:13851 2024.
[24] J. Lei, T. L. Berg, and M. Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:11846-11 11858, 2021.
[25] J. Lei, L. Yu, T. L. Berg, and M. Bansal. Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings. Part XXI 16, pages 447 463. Springer, 2020.
[26] H. Li, Song, Gao, X. Zhu, and H. Shen. Prototype-based aleatoric uncertainty quantification for cross-modal retrieval. Advances in Neural Information Processing Systems, 36, 2024.
[27] W. Li, R. Zhou, J. Zhou, Y. Song, J. Herter, M. Qin, G. Huang, and H. Pfister. 4d langsplat: 4d language gaussian splatting via multimodal large language models. arXiv preprint arXiv:2503. 10437, 2025.
[28] T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[29] Y. Liu, J. Cao, B. Li, C. Yuan, W. Hu, Y. Li, and Y. Duan. Knowledge distillation via instance relationship graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7096-7104, 2019.
[30] W. Ma. Q Chen, T. Zhou, S. Zhao, and Z. Cai. Using multimodal contrastive knowledge distillation for video-text retrieval, IEFE. Transactions on Circuits and Systems for Video Technology, 2023.
[31] Y. Ma, G. Xu, X. Sun, M. Yan. J. Zhang, and R. Ji. X-clip: End-to-end multi- qulti-grained contrastive learning for vidco-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 638 1116647,22 2022.
[32] W. Moon, C.-H. Cho. W. Jun, M. Shim, T. Kim, I. Lee, D. Wee, and J.-P. Heo. Prototypes are balanced units for efficient and effective partially relevant video retrieval. arXiv preprint arXiv:2504.1300 13035, 2025.
[33] T. Nishimura, S. Nakada, and M. Kondo. Vision-language models learn super images for efficient partially relevant video retrieval. ACM Transactions on Multimedia Computing, Communications and Applications. 2023.
[34] N. Norouzi, S. Orlova, D. De Geus, and G. Dubbelman. Algm: Adaptive local-then-glohal token merging for efficient semantic segmentation with plain vision transformers. In Proceedings of the IEFF/CVF Conference on Computer Vision and Patiern Recognition. pages 15773-15782. 2024.
[35] W. Park, D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3967-3976, 2019.
[36] R. Pei. J. Liu, W. Li, B. Shao, S. Xu, P. Dai, Lu, and Y. Yan. Clipping: Distilling clip-based models with student base for video-language retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19983-18992 2023.
[37] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[38] L. Shen, T. Hao, T. He, S. Zhao, Y. Zhang, pengzhang liu, Y. Bao, and G. Ding. Tempme: Video temporal token merging for efficient text- retrieval. In The Thirteenth International Conference on Learning Representations, 2025.
12

---

[39] P. Song, L. Zhang, L. Lan. W. Chen. D. Guo, X. Yang, and M. Wang. Towards efficient partially relevant video retrieval with active moment discovering. arXiv preprint arXiv:2504. 10920, 2025.
[40] C. Spearman. The proof and measurement of association between two things. 1961.
[41] Y. Tian, D. Krishnan, and P. Isola. Contrastive representation distillation. arXiv preprint arXiv: 2019.
[42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I.. Jones, A. N. Gomez, F.,. Kaiser, and I. Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[43] J. Wan, D. Wang. S. C. H. Hoi, P. Wu, J. Zhu, Y. Zhang, and J. Li. Deep learning for content-based image retrieval: A comprchensive study. In Proceedings of the 22nd ACM international conference Multimedia, pages 157-166, 2014.
[44] J. Wang, G. Sun, P. Wang. D. Liu, S. Dianat, M. Rabbani, R. Rao, and Z. Tao. Text is mass: Modeling as stochastic embedding for text-video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16551-16560, 2024.
[45] K. Wang, X. Gao, Y. Zhan, X. Li, D. Dou, and C.-Z. Xu. Pay attention features, transfer learn faster cnns. In International conference on learning representations, 2019.
[46] Y. Wang, Wang, Chen, Dai, R. Luo, and S.-T. Xia. Gmiffommer v2: An uncertainty for partially relevant video retrieval. arXiv preprint Xii::2405.13824 2024.
awerr aware framework
[47] Wang. J. Wang, Chen, Z. Zeng, and S.-T. Xia. Gmmformer: Caussian-mixture-mee mixture-model based transformer for efficient partially relevant video retrieval, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5767-5775, 2024.
[48] Y. Xie, Y. Lin, W. Cai, X. Xu, H. Zhang. Y. Du, and S. He. D3still: Decoupled differential distillation for asymmetric image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17181-17190, 2024.
[49] M. Yi, A. Li, Y. Xin, and Z. Li. Towards understanding the working mechanism of text-to- text-to-image diffusion model. Advances in Neural Information Processing Systems, 37:55342-5555 2024.
[50] S. Yin, S. Zhao, H. Wang. Xu, and E. Chen. Exploiting instance- relationships in weakly supervised text-to-video retrieval. ACM Tronsactions Multimedia Computing, Communications and Applications, 20(10):1-21, 2024.
[51] Q Zhang, C. Yang. B. Jiang, and B. Zhang. Multi-grainee alignment with knowledge distillation for partially relevant video retrieval. ACM Transactions on Multimedia Computing. Communications and Applications, 2025.
[52] Y. Zhao, Y. Xiong, Wang, Wu, X. Tang, and D. Lin. Temporal action detection with structured segment networks. In Proceedings of the IEEE international conference on computer vision, pages 2914 4--2233, 2017.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Claims including contributions are included in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
13

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitation is included in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
14

---

Justification:
Details are in implementation details. Code will be released with data links.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code will be released (tttp///ggttabbcccm//mmmin///)/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
sslicioyconno) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nsps.cc/publiccguu http:///gggideeeo/oeeubiisissss http//oooeeubmieeec wwwdddeuumisssiooPPoooo ssionoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
15

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes. Check implementation details and code. The intuition behind choosing the value of hyperparameters is in implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate
information about the statistical significance of the experiments?
Answer: [No]
Justification: We used fixed seeds for all experiments, ensuring identical results across and reproducibility of the findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Check implementation detail for GPU and CPU.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We checked the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: Societal impacts are included in the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
17

---

Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited every asset we used.
Guidelines:
The answer NA means that the paper does not existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are relcased, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
18

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Only used for grammar check and editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to LLM policy fattps://neuripe: http::/pss..eeooeeeeeene///////s for what should or should not be described.
19

---

Table A1: Sensitivity to temperature 7 across datasets. Rows marked with gray indicate the default configuration used in the main results.
Dataset
T
R@D
R@5
R@
10
R@ 100
SumR
0.70
35.6
61.0
70.8
95.0
262.4
0.75
35.5
61.2
71.1
94.9
262.6
TVR
0.80
35.1
61.6
71.5
94.9
263.1
0,85
35.1
61.2
71.2
95.0
262.5
0.90
35.1
61.1
71.1
94.9
262.2
0.70
17.6
41.9
55,4
86.8
201.7
0.75
17.8
41.9
55,4
86.7
201.8
ANet
0.80
17.7
42.0
55.6
86.8
202.1
0.85
17.7
42.1
55.3
86.8
201.9
0.90
17.2
41.9
55.5
86.8
201.4
0.70
3.3
11.6
19.8
63.9
98.6
0.75
3,4
12.7
19.4
64.8
100.3
CHA
0.80
3.4
12.0
18.7
64.5
98.6
0.85
3.2
12.6
20.1
63.8
99.7
0.90
3.3
12.4
19.1
64.0
98.9
A Further Analysis on Hyperparameter Sensitivity
We noted that all hyperparameters are unified across datasets except the similarity threshold т. which we set per dataset to account for different internal segment-to-segment similarity distribu- tions [32]. Beyond the QVHighlights ablation, Table AI evaluates T sensitivity on TVR, ActivityNet- Captions (ANet), and Charades well. Empirically, HiHighlights exhibits the lowest similarity levels, TVR and ANet are intermediate, and CHA shows the highest. Accordingly, we adopt 7=0.70 for QVHighlights, = for TVR and ANet, and ~00.85 for CHA. As shown, varying within moderate range causes only minor fluctuations in each dataset, indicating that performance is not overly sensitive this hyperparameter once set near the optimum.
B Impact of CLIP's Failure Rate on TCPL
In this section, we evaluate whether TCPL inherits or corrects CLIP's semantic errors in the PRVR setting. We conduct this study on the TVR dataset since most text queries in TVR involve multiple named entities sequential actions that require the capability to comprehend complex temporal and contextual cues. On the test set TVR (10,895 queries), we mark success when the ground-truth video appears within the top-Q retrieved results (Q {1, 10}}) and compare model (with TCPL) to zero-shot CLIP via a 2x2 outcome matrix. Specifically, for each text query, we record (i) both correct, (ii) ours correct & CLIP wrong, (iii) ours wrong & CLIP correct, and (iv) both wrong. Tab. reports the counts (and proportions).
To illustrate, when Q=1, our model corrects 2,551 of CLIP's failures (while the reverse occurs in 500 cases); at Q=10, the corresponding counts are 3,627 vs. 386. Our proposed framework also retains CLIP's strengths, answering correctly together on 1,277 (R@) 1) and 4, 4,162 (R @ 10) queries.
We further analyze the instances where one model succeeds and the other fails. When CLIP fails, the correct item is, on average, ranked 56th, indicating severe confusion. These failures consistently involve queries with multi-entity contexts and temporal compositions. For example, CLIP ranked the correct video at 237 for "Sebastian grabs his folder and stands up from the table" and at 418 for "George pulls back on Meredith's 's rolling chair and drags her". By contrast, when our model fails but CLIP succeeds, the ground- ground-truth -ruth video is still ranked highly. with an average position of 6.7. These cases are typically simple and object-centric queries requiring little compositional or temporal reasoning. For instance, CLIP correctly retrieved the videos for "House takes sip of soda from the bottle" and "Joey folding his coat in the kitchen". while our model placed them rank Taken together, these outcomes demonstrate that the retrieval objective reshapes the representation toward lask-specific temporal and compositional semantics, with TCPL preserving robust high-level alignment while correcting CLIP's fine-grained failure modes.

---

Table A2: Comparative analysis of retrieval correctness between our model and zero-shot CLIP on the TVR test set (10,895 queries), evaluated using (a) Recall @ and (b) Recall @ 10 as success criteria. Values are raw counts with percentages in parentheses.
(a) R
(b) Recall @ 10.
CLIP correct
CLIP wrong
CI.IP correct
CLIP wrong
Ours correct
1277 (11.7%)
2551 (23.4%)
Ours correct
4162 (38.2%)
3627 (33.3%)
Ours wrong
500 (4.6%)
6567 (60.3%)
Ours wrong
386 (3.5%)
2720 (24.9%)
Algorithm Preer-Preservvng Token Merging (OP-ToMe)
Require: Frame tokens V E RRBxxL,d,, Merge rate N%, Number of iterations M
Ensure: Clip tokens V. TR. Bexxx. where T.e 32
1: Initialize token sizes # + 1,, E TR4
D Each token represents frame
2: for m 1 M do
3: Compute cosine similarity between disjoint adjacent- pairs:
Sli] f cos(VV Li 1J) for 1,3,5, ...., Ls
4: Select top-N %% most similar adjacent pairs based on S
5:
for each selected pair (i, do
6:
Compute size-weighted average:
Vinerged - +[s[++[i+1]
7:
Replace [t[t] with Vieereed remove Vili+ 1]
8:
Update size: s[i] s(i] + sli 1), remove s(i
9:
end for
10:
Update Ly <- new token length
11: if L < 32 then
break
13:
end if
end for
15:
return V
C Algorithms for
Video Alignment
In this section, we provide detailed algorithm for sub-components of our Cross-Branch Video Alignment (CB.A). Particularly, we illustrate sedte-Preserving Token Merging (OP-ToMe), the process of pre-computing discrete of different levels clip number (number semantics), and the process of ver-video merging for Adaptive CBVA in Algorithm. [ Algorithm, and Algorithm. respectively.
D Positive and Negative Societal Impacts
Positive Impact. Our work improves the text-video retrieval based partial content descriptions within long, untrimmed videos. We expect that the proposed method will enhance the user experience in video search and navigation. This is particularly valuable in domains such education, where lengthy untrimmed videos are commonly utilized.
Negative Impact. However, the ability to isolate specific video contexts and retrieve segments based on partial descriptions could be misused surveillance settings (e.g., CCTV), enabling the tracking of individuals the extraction of sensitive behaviors without consent. Such misuse may raise potential concerns regarding privacy and ethical deployment.

---

Algorithm 2 Pre-computing the different levels of clip number (Eq. 9
Require: Initial clip length L La (e.g., 32), merge-rate N%, minimum clips Cmin Ensure: Candidate list 1, L + [
L L2 ...... L
2: while L > Cmin do
L. (L,/2) (NV100) +
3: Le - max 2
Cuin
2
4: if L Li then break
5: end
6: Append - to L
7: iti
8:
end while
9: K+ |L]
D number discrete clip levels
10: return L.
Algorithm 3 Constructing merged clips for Adaptive CBVA
Require: Clip tokens Ve RB xd. Global candidate list L of length K, Merge rate N%, Similarity threshold Projected Clip tokens E. B.X.B.x.ox Baxx Ensure: Adapted clip tokens V. with length L: Stage Estimate internal similarity
1: Compute cosine- cosine-similarity matrix S from frozen Ve
S(, >T, it1)
2:
> high-similarity ratio
(Te - 1)
Stage 2. Select merging depth
3:
then if w < 1 K
P if diverse, keep all clips
4:
k +-1
5: else
6:
k- t min (2... (2W > K-k
K
7: end if
Stage 3. Merge clips 1 times
8: V. + V.
9: for = to 1- 1 do
10: Apply bipartite token merging (TOME) 1 to V. at rate N%
end for
12: L- + [V
13: return V.
3


Reviews:
[{"review_id": "XT5kzk3l8A", "reviewer": "Reviewer_fskT", "summary": "A regularization method is proposed to mitigate the semantic collapse issue in partial video retrieval. Compared with the previous methods, the author propose to mitigate the issue in both the text domain and vision domain. Specifically, in text domain, Text Correlation Preservation Learning (TCPL) is proposed. The text semantic relationship among text queries in a batch is preserved by maintaining the distances among the queries in CLIP feature space. In vision domain, Order-Preserving Token Merging (OP-ToMe) is proposed. The visual token is iteratively merged by refering the cosine similarities between each adjacent frames until pre-defined number of clips is achieved. Further, Cross-Branch Video Alignment is proposed to force the frame-level visual feature and clip-level visual feature to align with each other. The proposed method is evaluated on four benchmark text-to-video retrieval datasets to show its effectiveness.", "questions": "1.\tFor the semantic collapse issue, can directly aligning the text queries with their corresponding annotated video segment also help mitigate the issue? \n2.\tWhy not using text queries relationship in the feature space constructed by a pure NLP encoder as the training supervision guidance?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\n1.\tThe proposed method is extensively evaluated to show its effectiveness;\n\n2.\tThe proposed method achieves the state-of-the-arts performance;\n\n3.\tThe manuscript is well-organized;\n\n\n**Weakness:**\n \n\n1.\tIn TCPL, the training objective is to mimic the relationships among text queries constructed in the feature space of CLIP model. However, the sematic collapse issue may also occur in the CLIP feature space, considering that the training objective of CLIP is to align the text feature with the image feature. Why not using the feature space constructed by a pure NLP encoder?\n\n2.\tThe description of the proposed method is not clear. For example, there is no equation or figures to illustrate how the OP-ToMe works, and it is not easy to re-implement it;", "was_revised": true, "final_justification": "Considering the good performance and the insights the authors share with the community, I tend to rate borderline accept. And I suggest the authors to repolish the paper to make it easy to understand.", "rebuttal": "# (TCPL) CLIP may also be vulnerable to semantic collapse. Why not distilled by NLP encoder?\n\nSemantic collapse in PRVR primarily arises from the deterministic learning with partially labeled data, where unannotated pairs are treated as negatives, even though they may share positive semantic contexts. \nWhile CLIP also employs contrastive learning over pairwise data, it is trained on a significantly larger and more diverse set of image-text pairs. \nThis diversity helps mitigate the risk of semantic collapse, making CLIP more robust compared to models trained solely on narrow or task-specific datasets.\n\nMoreover, despite potential vulnerabilities, CLIP’s vision-language aligned space offers strong semantic unification across modalities [A], which is critical for PRVR, where text and video need to be embedded into a common space effectively.\nIn contrast, using an NLP encoder for distillation may reduce the risk of collapse due to its inherently relational training, but it lacks alignment with the visual backbone. \nThis misalignment limits its effectiveness for vision-language tasks like PRVR, as evidenced by the empirical performance gap shown in the table below.\nWe observe the same performance patterns when a RoBERTa encoder is used as both the base text backbone and the TCPL source model.\nExperiments are conducted on ANET and CHA due to their readily available GT and fast training time.\n\n[A] Unified Visual Relationship Detection with Vision and Language Models. ICCV. 2023.\n\n### AcitivityNet\n| Model($\\lambda^E, \\lambda^A$) | R1 | R5 | R10| R100 | SumR |\n|--|--|--|--|--|--|\n| **CLIP base** | **15.7** | **39.2** | **52.9** | **85.9** | **193.8** |\n| CLIP+ Rob TCPL (10,20) | 13.2 | 34.1 | 47.9 | 83.3 | 178.5 |\n| CLIP+ Rob TCPL (15,30) | 12.3 | 33.3 | 46.7 | 82.5 | 174.8 |\n| CLIP+ Rob TCPL (20,40) | 12.0 | 32.8 | 45.7 | 81.8 | 172.3 |\n\n### Charades\n| Model($\\lambda^E, \\lambda^A$)  | R1 | R5 | R10| R100 | SumR |\n|--|--|--|--|--|--|\n| **CLIP base**| **2.6** | **11.2** | **18.3** | **62.6** | **94.8** |\n| CLIP+ Rob TCPL (10,20) | 3.4 | 11.2 | 18.2 | 62.4 | 95.1 |\n| CLIP+ Rob TCPL (15,30) | 3.1 | 10.6 | 17.4 | 61.6 | 92.7 |\n| CLIP+ Rob TCPL (20,40) | 3.0 | 11.0 | 17.6 | 60.1 | 91.7 |\n\n# Unclear description, especially OP-ToMe.\nWe sincerely apologize that re-implementation based on our current description may be difficult.\nTo provide a detailed description for OP-ToMe, we provide an algorithm below.\n\n────────────────────────────────────────────────────────────────────────────\n\nAlgorithm 1. Order‑Preserving Token Merging (OP‑ToMe)\n```\nInput : Frame tokens  V_f ∈ ℝ^{B_v × L_f × d_v}\n        Merge rate    N %      (percentage of pairs to merge per iteration)\n        Iterations    M\nOutput: Clip tokens   V_c ∈ ℝ^{B_v × L_c × d_v}   where L_c = 32\n```\n```\n1.  Initialize token‑size vector  s ← 1_{L_f}        # each token covers 1 frame\n\n2.  for m = 1 … M do\n3.      # 1. Compute similarity of disjoint adjacent pairs (odd indices 1,3,5,…)\n4.      for i ∈ {1, 3, 5, … , L_f‒1}:\n5.          S[i] ← cos( V_f[i], V_f[i+1] )\n6.      end for\n7.\n8.      # 2. Pick the top‑N% most‑similar adjacent pairs\n9.      P ← indices of top‑N pairs in S\n10.\n11.     # 3. Merge each selected pair\n12.     for (i, i+1) ∈ P (processed left→right):\n13.         V_merge ← ( s[i]·V_f[i]  +  s[i+1]·V_f[i+1] ) / ( s[i] + s[i+1] )\n14.         V_f[i]← V_merge                 # replace left token by the merge\n15.         delete V_f[i+1]                   # remove right token\n16.         s[i] ← s[i] + s[i+1]            # update size\n17.         delete s[i+1]\n18.     end for\n19.\n20.     L_f ← |V_f|                           # new token length\n21.     if L_f ≤ 32 then break\n22. end for\n23.\n24. return V_c ← V_f                          # 32 order‑preserving clip tokens\n```\n\nIn addition, to improve the readability of adaptive CBVA, we will rewrite the opening sentence of L214-218 as:\n\"However, optimizing the number of semantics per video is costly during the token merging process.\nTherefore, we instead pre-define a discrete set of clip numbers based on a fixed merge rate, and then match each video to the level that best reflects its internal similarity structure (number of different semantics).\nTo initially establish a discrete set of clip levels, we define $N\\%$ to denote the merge rate and $C_\\text{min}$ to represent the minimum number of semantically different clips in each video.\nThen, we generate $K$ levels of clip number candidates $\\{L_c^i\\}_{i=1}^{K}$ by recording clip number after each merge step as:\", and also provide an algorithm for preparing adaptive lengths of clips for Adaptive CBVA (shown below).\nSpecifically, the process of pre-computing a discrete set of different levels of clip number (number of semantics) is shown in Algorithm 2.\nAlgorithm 3 shows the process of per-video merging for Adaptive CBVA.\n\n────────────────────────────────────────────────────────────────────────────\n\n### Algorithm 2. Pre-computing the discrete clip-count levels before Adaptive CBVA\n```\nInput : Initial clip length   L_c^1 = L_c      (e.g., 32)\n        Merge‑rate            N %              (percentage applied per step)\n        Minimum clip count    C_min\nOutput: Candidate list        L = [L_c^1 , L_c^2 , … , L_c^K]\n```\n```\n1.  i ← 1                                   ;  L ← [ L_c^1 ]\n2.  while  L_c^i > C_min  do\n3.      # Compute next coarser clip length\n4.      L_new ← max( 2 × floor( ( L_c^i − (L_c^i /2) × (N/100) + 1 ) / 2 ),\n5.                     C_min )\n6.      if  L_new = L_c^i   then  break      # no further reduction possible\n7.      append  L_new  to  L\n8.      i ← i + 1\n9.  end while\n10. K ← |L|                                  # number of discrete levels\n11. return  L\n```\n────────────────────────────────────────────────────────────────────────────\n\n### Algorithm 3. Constructing Merged Clips for Adaptive CBVA\n```\nInput : Clip tokens        V_c  ∈ ℝ^{B_v × L_c × d_v}\n        Global candidate   L    = [L_c^1 … L_c^K]   # pre‑computed clip counts\n        Merge‑rate         N%                      # applied at each merge\n        Similarity thresh. τ\nOutput: Adapted clip tokens  Ṽ_c   (length  L_c* )\n```\n```\n# Stage 1 – Estimate internal similarity\n1.  S ← cosine‑similarity matrix of frozen V_c              # L_c × L_c\n2.  ω ← |{(i,j) : S_ij > τ , i≠j}| / [L_c (L_c − 1)]        # high‑sim ratio\n\n# Stage 2 – Select merging depth k*\n3.  if  ω ≤ 1 − 1/K                     then   k* ← 1        # keep all clips\n4.  else\n5.       k* ← min{ k ∈ {2 … K} |  ω > (K − k)/K }           # pick coarser level\n6.  end if\n\n# Stage 3 – Merge clips (k* − 1 times)\n7.  Ṽ_c ← V_c\n8.  for  m = 1 … (k* − 1)  do\n9.        Apply bipartite token‑merging (ToMe [B]) to Ṽ_c at rate N %\n10. end for\n11. L_c* ← |Ṽ_c|\n12. return  Ṽ_c\n```\n\n[A] Token Merging: Your ViT But Faster. ICLR 2023.\n\nWe claim that this operation (pre-computing levels of different clip numbers) offers two benefits over determining whether to merge the paired clips based on their similarity.\n\n(1) Efficiency: \nClip count candidates are pre-computed once, and each video is matched to the most appropriate level, thereby no recursive per-pair merge decision is needed.\nBy contrast, deciding at every pair whether to merge would insert an extra similarity-threshold branch into each pairing step, increasing the computational overhead.\n\n(2) Robustness: \nVisual similarity is not always accurate in measuring the contextual similarity in text descriptions.\nWhile thresholding the pairwise similarity introduces brittle sensitivity to the chosen threshold value, we claim that a coarsely discretised set of clip count candidates alleviates such sensitivity and yields more stable performance across diverse video domains.\n\nWe will integrate this clarification, as well as the Algorithms in the appendix.\nAlso, we will carefully proofread the entire paper to improve clarity.\n\n# Does the use of annotated video segments mitigate Semantic collapse?\nAs mentioned by the reviewer, using the annotated video segment may partially mitigate the semantic collapse by encouraging the model to pull each query toward its exact visual span.\nHowever, this strategy (i) depends on dense temporal annotations that are costly and often impractical in real-world PRVR settings, (ii) still leaves unlabeled portions of the video collapsed together, and (iii) does not address semantic collapse among different videos that share similar events. \n\nTo verify whether the annotated video segment can only partially mitigate the semantic collapse, we trained a model to align each query with its annotated span using the GT‑segment.\nWe tested this model on ANET, CHA, and TVR datasets where GT timestamps are readily available.\nTo illustrate, the use of GT timestamps results in a performance boost in ActivityNet-Captions (+4.3), but has little effect on TVR or Charades.\nMoreover, our framework boosts the baseline by +7.9 on ANET, surpassing the gains achieved with ground-truth segments and indicating that it tackles semantic collapse more effectively than simply injecting GT boundaries.\nTherefore, we argue that ground-truth timestamps alone are insufficient; additional labels that capture the semantic relationships between video segments are also necessary to effectively address semantic collapse."}, {"review_id": "yJPEvEOGLk", "reviewer": "Reviewer_aktj", "summary": "This paper addresses semantic collapse in Partially Relevant Video Retrieval (PRVR), categorized into: (i) text semantic collapse, where queries from the same video cluster despite semantic irrelevance, while similar queries across videos are separated; and (ii) video semantic collapse, where segments from the same video converge regardless of semantic disparity. To mitigate (i), the authors propose TCPL to distill semantic relations from CLIP. For (ii), they introduce CBVA to align embeddings of temporally matched frames and clips while separating others. Additionally, OP-ToMe is proposed for various clip embeddings. Extensive experiments on four datasets validate the model’s superior retrieval performance.", "questions": "1. Why are the ablation studies conducted only on QVHighlights? How about the ablation results on other datasets?\n\n2. Could the authors elaborate on possible failure scenarios, particularly in the context of highly complex videos—such as those involving frequent context shifts, significant visual ambiguity, or high levels of noise—where token merging or adaptive CBVA may struggle to produce accurate segmentation? Additionally, are there any mitigation strategies that could be employed in such cases?\n\n3. TCPL heavily relies on the capabilities of the CLIP model, which is known to exhibit biases in fine-grained semantics (e.g., “picking up a puppy” vs. “petting a puppy”) and rare concepts. If CLIP itself incorrectly models the relationship between certain query pairs—such as misclassifying semantically related pairs as unrelated—would TCPL amplify such errors? Could the authors provide ablation studies to assess the impact of CLIP’s error rate on TCPL’s performance?\n\n4. Could the authors explain in more detail what the \"baseline\" model is used in this paper? I noticed that the results reported in Table 1 and Table 4 show that the baseline already achieves strong performance, with marginal differences (1.1 SumR, ~0.3 average) behind ProtoPRVR. Are the performance gains reported in the paper primarily attributed to the strong baseline itself?\n\n5. In Equations 4 and 5, why is the [EOS] token chosen for distillation? Additionally, at which stage is OP-ToMe applied—before feeding inputs into the clip-level branch, or at the output layer?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper is well-formatted, and Figure 1’s design and color scheme are visually appealing and conceptually clear.\n2. The definitions of semantic collapse and the corresponding proposed method are clear and easy to understand.\n3. The method performs well on PRVR retrieval benchmarks.\n\n\n**Weaknesses**\n\n1. The technical contributions are somewhat incremental and combinational. The core components, such as CLIP-based structural distillation and token merging, themselves are adaptations or applications of existing techniques rather than novelties.\n\n2. The proposed method introduces more than 7 additional hyperparameters to tune, raising concerns about the method’s generalizability and applicability in deployment.  It remains unclear whether the reported performance gains are due to overfitting to the test set. The paper did not mention whether the experiments have independent validation and test sets.\n\n3. Efficiency analysis is missing. The proposed method leverages a heavier backbone and employs multi-iteration aggregation to generate clip embeddings, which raises efficiency concerns. Therefore, it is necessary to report model size and provide any efficiency analysis, such as training-time cost, inference-time retrieval speed, or memory overhead, to justify the trade-off.\n\n4. Missing comparison with baselines [A-D].\n\n----\n\n[A] Yin, Shukang, et al. \"Exploiting Instance-level Relationships in Weakly Supervised Text-to-Video Retrieval.\" ACM Transactions on Multimedia Computing, Communications and Applications 20.10 (2024): 1-21.\n\n[B] Zhang, Qun, et al. \"Multi-Grained Alignment with Knowledge Distillation for Partially Relevant Video Retrieval.\" ACM Transactions on Multimedia Computing, Communications and Applications (2025).\n\n[C] Jiang, Xun, et al. \"Progressive Event Alignment Network for Partial Relevant Video Retrieval.\" 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023.\n\n[D] Song, Peipei, et al. \"Towards Efficient Partially Relevant Video Retrieval with Active Moment Discovering.\" arXiv preprint arXiv:2504.10920 (2025). Accepted by TMM 2025.", "was_revised": true, "final_justification": "While minor points remain, the authors' revisions have sufficiently improved the manuscript to a borderline level.", "rebuttal": "# Technical Contributions incremental.\nWe respectfully disagree with the characterization that our contributions are “merely incremental”.\nOur work introduces a comprehensive formulation of semantic collapse in PRVR (spanning both text & video modalities), quantifies it (Tab 6,7), and mitigates it with purpose-built objectives. \nThese elements go beyond simply combining existing tools and address facets that prior art has either overlooked or only partially addressed.\n\nTo address text semantic collapse incurred by determining relationships between text queries with matched video IDs:\n\n- Proposed TCPL, which distills relational structure from CLIP to preserve semantic distinctiveness across related queries.\nUnlike prior methods that simply push every same‑video query apart, TCPL achieves strong performance gain by preserving meaningful structure of the shared embedding space.\n\nTo address video semantic collapse caused by overlapping events in untrimmed videos:\n\n- Proposed novel CBVA, which aligns hierarchical frame and clip-level features to resolve intra-video ambiguity.\nOP‑ToMe is tailored for temporal coherence in untrimmed videos: unlike generic token‑merging schemes, it merges only consecutive frames, thereby respecting playback order.\nAdaptive CBVA dynamically estimates video context granularity to better align hierarchical representations based on video-specific semantic density.\n\n# Hyperparams & Overfitting.\nWe acknowledge the use of several hyperparams (e.g., TCPL weights ($\\lambda^{\\text{E}}$, $\\lambda^{\\text{A}}$), CBVA weight ($\\lambda^{\\text{CBVA}}$), merge rate ($N\\%$), similarity threshold ($\\tau$), and the minimum clip count ($C_{\\text{min}}$)). \nYet, we emphasize that our hyperparams(except $\\tau$) were held constant across all datasets to avoid overfitting and ensure fair comparisons.\nTo clarify our hyperparam selection strategy, we provide the following justifications based on empirical observations:\n\n$\\lambda^{\\text{E}}$, $\\lambda^{\\text{A}}$: As in Tab.3(a,b) in paper, TCPL consistently outperforms baseline (225.5 SumR) across a range of settings, as long as the values are within a reasonable scale. \n\n$\\lambda^{\\text{CBVA}}$, $N\\%$: In Tab.3(d,e), we report consistent performance gains with intuitive values. \nAlso, the same hyperparams are used for other datasets, demonstrating that the CBVA hyperparams generalize well and are not overfitted to any specific dataset.\n\n$\\tau$: As shown in Tab.3(f), varying $\\tau$ in the range of 0.5 to 0.8 yields stable and effective results, again indicating generalizability across datasets.\n\n$C_{\\text{min}}$: This was heuristically set to 5 across all datasets, reflecting the typical upper bound of text queries per video observed in PRVR benchmarks.\n\n- These findings suggest that ours **does not rely on hyperparam tuning and avoids overfitting to specific datasets**.\nIn contrast, we note that **our baselines** GMM&GMMv2 use **dataset-specific hyperparams while our method consistently outperforms them**.\n\n# Baseline & Efficiency & Complexity\nWe apologize for any confusion caused regarding our baseline.\nTo clarify, our baseline is derived from GMM v2, excluding its two key contributions: (1) revamped query diversity loss and (2) optimal matching loss; in terms of model size, our backbone remains on the same scale as GMM v2, and we adopt its hyperparams without additional tuning.\nThus, we claim that our gains are not from the strong baseline but our proposed components that yield 9.1 SumR gain on QVHighlights, while objectives in GMMv2 are not as effective with CLIP backbone.\n\nAlso, for efficiency, we provide time and memory for both inference and training along with # of params, and FLOPs on QVH.\nInference and training time are averaged over 5 runs.\n\n(1) Inference time&mem: Across DB sizes from 100 to 474 videos, our method delivers the second‑best inference latency and memory footprint, while achieving substantially higher retrieval accuracy.\nNote that video features are pre-computed and stored offline in real-world text-video retrieval systems; when a user inputs a text query, the system operates on these cached features. \nAs a result, our token-merging step imposes no additional cost at inference time.\n\n(2) Train time&mem: We acknowledge our training incurs more time and memory costs to learn fine-grained video contexts.\nYet, we argue that training is performed offline; inference efficiency drives real-world deployment.\n\n|Inference time(ms)|100 (vid DB size)|200|300|400|474|\n|--|--|--|--|--|--|\n|MSSL|3.09|3.85|4.66|5.14|5.58|\n|GMM|1.97|1.98|1.99|2.02|2.05|\n|GMMv2|2.31|2.38|2.40|2.61|2.78|\n|Ours|2.32|2.37|2.40|2.60|2.70|\n\n|Inference Mem(MB)|100|200|300|400|474|\n|--|--|--|--|--|--|\n|MSSL|717.47|796.15|874.83|954.14|1010.89|\n|GMM|243.11|248.95|254.78|260.62|264.10|\n|GMMv2|419.75|440.18|459.62|480.55|493.46|\n|Ours|419.75|440.18|459.62|480.55|493.46|\n\n|Others|MSSL|GMM|GMMv2|Ours|\n|--|--|--|--|--|\n|Train Time 1 epoch(ms)|10934|12828|17223|62641|\n|Train Mem(MB)|2375|3333|7826|9755|\n|Param(m)|4.57|12.72|32.14|32.14|\n|FLOPs(G)|0.37|0.99|2.78|2.78|\n\n# Missing baseline\nWe thank the reviewer for noting the missing baselines (will be referenced).\nBelow shows reproduced scores with official codes when available; our method outperforms these by large margins(will be added to paper).\n\n| |QVH|ANET|CHA|TVR|\n|--|--|--|--|--|\n|BGM-Net|20.6 46.3 58.8 94.0 219.7|15.6 37.9 51.3 85.4 190.3|3.0 11.8 18.2 63.7 96.7|31.1 56.3 66.5 93.8 247.7|\n|MGAKD|X|X|X|X| \n|PEAN|X|X|X|X| \n|AMDNet|17.4 40.8  55.0  93.4  206.6|14.0 36.3 49.9 84.2 184.5|2.1 7.8 13.9 57.2 81.1|27.7 52.3 63.3 92.3 235.6|\n|Ours|23.9 51.5 63.7 95.5 234.6|17.7 42.0 55.6 86.8 202.1|3.2 12.6 20.1 63.8 99.7|35.1 61.6 71.5 94.9 263.1|\n\n# Why ablation on QVH?\nWe concentrated our ablations on QVH since (1) its untrimmed videos exhibit frequent and diverse context changes, making it the most demanding benchmark, and (2) it is built from a recent crawl of YouTube content, so its distribution closely mirrors real-world video-retrieval scenarios.\n\nTo confirm that the gains are not dataset‑specific, we conducted the ablation on ANET. \nImprovement pattern is almost identical, showing that our components generalize well across datasets. \nNotably, since videos in ANET contain fewer intra‑video scene changes than QVH, the text‑side (TCPL) contribution is bigger than video‑side gains (as can be expected when visual context diversity is lower).\n\n|Method|R1|R5|R10|R100|SumR|\n|--|--|--|--|--|--|\n|Base|16.1|39.5|52.7|85.9|194.2|\n|+TCPL|17.2|41.2|54.7|86.5|199.6|\n|+Naive CBVA|17.4|41.5|54.9|86.5|200.3|\n|+OP‑ToMe|17.5|41.9||86.9|201.7|\n|+Adaptive CBVA|17.7|42.0|55.6|86.8|202.1|\n\n# Scenarios token merging or adaptive CBVA struggle?\n\nOne potential failure mode we have already mitigated is the brittleness of bipartite token‑merging since CLIP visual similarity does not accurately represent the degree of semantic overlap (fails to detect fine-grained action changes occurring within the same background). \nAs stated in L214, we avoid per‑video threshold tuning by approximating the context count with a small, pre‑defined set of coarse clip‑count levels. \nThis design choice not only (1) eliminates the expensive search for an exact context count for every video, but also (2) makes our method far less sensitive to noisy or ambiguous pairwise similarities.\nSimply put, our design in defining a discrete set of clip levels mitigates the sensitivity to specific similarity thresholding hyperparam.\nWe will emphasise this rationale in the revised paper.\nAs future work, we plan to incorporate temporal action cues and motion‑aware features into the merging criterion so that subtle action transitions can be recognized.\n\n# Does TCPL Amplify CLIP’s Semantic Errors?\n\nWe agree that CLIP exhibits a high error rate for certain fine-grained concepts, such as directional cues or specific actions.\nYet, in many practical scenarios (including PRVR) where coarse-grained semantic understanding is essential, the benefits of maintaining alignment with vision-language embedding spaces often outweigh the drawbacks stemming from their limitations.\nIndeed, we only use CLIP's knowledge to prevent overfitting; TCPL simply regularizes the student so it does not drift too far from CLIP’s global structure while retrieval losses and CBVA adapt the embeddings to PRVR. \nConsequently, any fine‑grained mistakes inherited from CLIP are free to be corrected during fine‑tuning, while TCPL preserves the robustness that CLIP’s coarse semantics provide.\n\nLastly, while CLIP has limitations in fine-grained understanding, this is an active area of research [A], and we expect such issues to diminish as more capable models are developed (our method is backbone‑agnostic, so any stronger model can be integrated with minimal effort).\nThus, we believe this concern, while valid, falls outside the scope of our work.\n\n[A] FG-CLIP: Fine-Grained Visual and Textual Alignment. ICML 2025.\n\n# In Eq. 4,5, why [EOS] for distillation?\n\n[EOS] is used for distillation for 3 reasons.\n\n1. Global Summary: \n[EOS] token in CLIP text encoder serves as a global summary of the input.\nSince PRVR is typically performed at sentence level, we use [EOS] to capture the overall semantics of each query.\nAlso, [EOS] is shown to convey more information than other tokens in CLIP [B].\n\n[B] Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model. NeurIPS 2024.\n\n2. Efficiency: \nDistilling only from [EOS] reduces computational overhead compared to token-wise distillation, enabling more efficient training.\n\n3. T-V Alignment:\nIn CLIP, text and visual features are aligned in a shared space through global features. \nThus, we used [EOS] to follow this convention for stabilizing the training.\n\n# At which stage OP-ToMe is applied?\nWe apologize for ambiguity. \nOP-ToMe is applied immediately after extracting video features from CLIP, and before passing them into any learnable modules.\nWe will state this in the revised manuscript."}, {"review_id": "Cxi8mL5Zaf", "reviewer": "Reviewer_jkfJ", "summary": "This paper focuses an interesting problem in partially relevant video retrieval, where existing loss functions may inadvertently push apart semantically similar features while pulling together dissimilar ones, i.e., \"semantic collapse\" called by the authors. To mitigate this, the authors propose two constraints: TCPL (Section 3.2) and CBVA (Section 3.3) to preserve the learned feature representation consistent with the original CLIP semantic space.\nAlthough this is an interesting question, the proposed method does not solve this problem well. The performance improvement may be achieved at the expense of the model's computational efficiency and inference speed.", "questions": "1. The paper needs to discuss computational efficiency or inference latency, especially compared to exiting works.  The merging operation in Section 3.3 appears computationally slow, yet no runtime analysis is provided.\n2.  Training objective in Section 3.3, that pushes apart features from different timestamps of the same video, seems unreasonable.\n Cause videos typically maintain semantic consistency across timestamps.", "limitations": "Authors should make the writing and figures clearer. Figure 2 makes the simple distillation too complicated and difficult to read. Similarly, Figure 1 is also too complicated. More critically, as the core chapter of this article, the process of adaptive CBVA in section 3.3 is too complicated, making it difficult to understand how the method is actually implemented.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. The paper makes a valuable observation about current partially relevant video retrieval methods: existing loss functions may inadvertently push apart semantically similar features while pulling together dissimilar ones (semantic collapse). This problem formulation is clearly presented.\n2. Section 3.1 is well-written. However, the key Section 3.3 lacks sufficient methodological clarity.", "was_revised": true, "final_justification": "Thanks for your response, and most of my concerns are addressed. I tend to raise my score.", "rebuttal": "# Computational efficiency \\& Inference latency\nTo first clarify efficiency concerns, we first note that our baseline is built upon GMMFormer V2, but excludes the two objectives proposed in that work: (1) revamped query diversity loss and (2) optimal matching loss.\nThus, our baseline maintains the same model scale as GMMFormer V2.\nIn addition, we report inference time, inference memory, training time, training memory, the number of parameters, and FLOPs on the QVHighlights benchmark.\nInference time and training time are averaged over five independent runs.\n\n(1) Inference time & memory : Across database sizes from 100 to 474 videos in QVHighlights, ours offers the second‑best inference latency and memory footprint, while achieving substantially higher retrieval accuracy.\nNote that video features are pre-computed and stored offline in real-world text-video retrieval systems; when a user inputs a text query, the system operates on these cached features. As a result, our token-merging step imposes no additional cost at inference time.\n\n(2) Training time & memory : Although our training incurs greater time and memory to learn fine-grained video contexts, we argue that the training cost is paid offline.\nInstead, real-world deployment is driven by efficiency during inference.\n\n\n|Inference time(ms)|100 (vid DB size)|200|300|400|474|\n|--|--|--|--|--|--|\n|MSSL|3.09|3.85|4.66|5.14|5.58|\n|GMM|1.97|1.98|1.99|2.02|2.05|\n|GMMv2|2.31|2.38|2.40|2.61|2.78|\n|Ours|2.32|2.37|2.40|2.60|2.70|\n\n|Inference Mem(MB)|100|200|300|400|474|\n|--|--|--|--|--|--|\n|MSSL|717.47|796.15|874.83|954.14|1010.89|\n|GMM|243.11|248.95|254.78|260.62|264.10|\n|GMMv2|419.75|440.18|459.62|480.55|493.46|\n|ours|419.75|440.18|459.62|480.55|493.46|\n\n|Others|MSSL|GMM|GMMv2|Ours|\n|--|--|--|--|--|\n|Train Time 1 epoch(ms)|10934|12828|17223|62641|\n|Train Mem(MB)|2375|3333|7826|9755|\n|Model Param(m)|4.57|12.72|32.14|32.14|\n|Inference FLOPs(G)|0.37|0.99|2.78|2.78|\n\n# Justification for Cross-Branch Video Alignment\nWe completely agree that blindly pushing apart features from different timestamps within the same video is unreasonable. \nIn fact, as shown in Tab. 1(c), the naive CBVA (which repels all cross-timestamp pairs), yields little to no improvement even on PRVR benchmarks that involve multiple semantics per video.\n\nTo address this limitation, our adaptive CBVA explicitly avoids such indiscriminate repulsion. \nBefore applying the CBVA loss, we first perform a token merging step that semantically groups relevant segments, even if they are temporally distant (L212-214).\nConsequently, our final adaptive CBVA only pushes apart frame–clip pairs that (1) are from distinct timestamps and (2) exhibit low semantic similarity. \nThis ensures the model learns to discriminate between different events while preserving semantically coherent ones.\n\nThis ensures that our final CBVA promotes meaningful separation between distinct events, while preserving and aligning semantically coherent segments regardless of their timestamp.\n\nFurthermore, to enhance the readability of Adaptive CBVA, we will revise L214-218 as \"However, optimizing the number of semantics per video is costly during the token merging process.\nTherefore, we instead pre-define a discrete set of clip numbers based on a fixed merge rate, and then match each video to the level that best reflects its internal similarity structure (number of different semantics).\nTo initially establish a discrete set of clip levels, we define $N\\%$ to denote the merge rate and $C_\\text{min}$ to represent the minimum number of semantically different clips in each video.\nThen, we generate $K$ levels of clip number candidates {${L_c^i}$}$_{i=1}^{K}$ by recording clip number after each merge step as:\".\n\n# Methodological / Figure Clarity\nWe appreciate the reviewer’s positive feedback on Section 3.1 and acknowledge the need for clearer exposition in Section 3.3. \nTo address this, we provide step-by-step algorithms below describing the OP-ToMe and adaptive CBVA process, and simplify the paragraph structure for better readability. \nThese will be added to Appendix.\n────────────────────────────────────────────────────────────────────────────\n\n### Algorithm 1. Order‑Preserving Token Merging (OP‑ToMe)\n```\nInput : Frame tokens  V_f ∈ ℝ^{B_v × L_f × d_v}\n        Merge rate    N %      (percentage of pairs to merge per iteration)\n        Iterations    M\nOutput: Clip tokens   V_c ∈ ℝ^{B_v × L_c × d_v}   where L_c = 32\n```\n```\n1.  Initialize token‑size vector  s ← 1_{L_f}        # each token covers 1 frame\n\n2.  for m = 1 … M do\n3.      # 1. Compute similarity of disjoint adjacent pairs (odd indices 1,3,5,…)\n4.      for i ∈ {1, 3, 5, … , L_f‒1}:\n5.          S[i] ← cos( V_f[i], V_f[i+1] )\n6.      end for\n7.\n8.      # 2. Pick the top‑N% most‑similar adjacent pairs\n9.      P ← indices of top‑N pairs in S\n10.\n11.     # 3. Merge each selected pair\n12.     for (i, i+1) ∈ P (processed left→right):\n13.         V_merge ← ( s[i]·V_f[i]  +  s[i+1]·V_f[i+1] ) / ( s[i] + s[i+1] )\n14.         V_f[i]← V_merge                 # replace left token by the merge\n15.         delete V_f[i+1]                   # remove right token\n16.         s[i] ← s[i] + s[i+1]            # update size\n17.         delete s[i+1]\n18.     end for\n19.\n20.     L_f ← |V_f|                           # new token length\n21.     if L_f ≤ 32 then break\n22. end for\n23.\n24. return V_c ← V_f                          # 32 order‑preserving clip tokens\n```\n────────────────────────────────────────────────────────────────────────────\n\n### Algorithm 2. Pre-computing the discrete clip-count levels before Adaptive CBVA\n```\nInput : Initial clip length   L_c^1 = L_c      (e.g., 32)\n        Merge‑rate            N %              (percentage applied per step)\n        Minimum clip count    C_min\nOutput: Candidate list        L = [L_c^1 , L_c^2 , … , L_c^K]\n```\n```\n1.  i ← 1                                   ;  L ← [ L_c^1 ]\n2.  while  L_c^i > C_min  do\n3.      # Compute next coarser clip length\n4.      L_new ← max( 2 × floor( ( L_c^i − (L_c^i /2) × (N/100) + 1 ) / 2 ),\n5.                     C_min )\n6.      if  L_new = L_c^i   then  break      # no further reduction possible\n7.      append  L_new  to  L\n8.      i ← i + 1\n9.  end while\n10. K ← |L|                                  # number of discrete levels\n11. return  L\n```\n────────────────────────────────────────────────────────────────────────────\n\n### Algorithm 3. Constructing Merged Clips for Adaptive CBVA\n```\nInput : Clip tokens        V_c  ∈ ℝ^{B_v × L_c × d_v}\n        Global candidate   L    = [L_c^1 … L_c^K]   # pre‑computed clip counts\n        Merge‑rate         N%                      # applied at each merge\n        Similarity thresh. τ\nOutput: Adapted clip tokens  Ṽ_c   (length  L_c* )\n```\n```\n# Stage 1 – Estimate internal similarity\n1.  S ← cosine‑similarity matrix of frozen V_c              # L_c × L_c\n2.  ω ← |{(i,j) : S_ij > τ , i≠j}| / [L_c (L_c − 1)]        # high‑sim ratio\n\n# Stage 2 – Select merging depth k*\n3.  if  ω ≤ 1 − 1/K                     then   k* ← 1        # keep all clips\n4.  else\n5.       k* ← min{ k ∈ {2 … K} |  ω > (K − k)/K }           # pick coarser level\n6.  end if\n\n# Stage 3 – Merge clips (k* − 1 times)\n7.  Ṽ_c ← V_c\n8.  for  m = 1 … (k* − 1)  do\n9.        Apply bipartite token‑merging (ToMe) to Ṽ_c at rate N %\n10. end for\n11. L_c* ← |Ṽ_c|\n12. return  Ṽ_c\n```\n\nWe also appreciate the feedback on visual clarity.\nFor Fig.1, we agree that the figure may seem too complicated and may obscure the core message. \nTo improve clarity, we will revise the color coding of the circles with our core message; instead of depicting generic positive/negative relations, we will use green to indicate text–video pairs that share similar semantics but belong to different videos (thus being pushed apart), and red to indicate segments with different semantics within the same video (thus mistakenly pulled together). \nThis change will better illustrate the misalignment issues addressed by our method.\nFor Fig.2, we will simplify the illustration by removing the explicit annotations of relation types (e.g., positive/negative edges) and focus solely on conveying the core idea of relation distillation. \nWe will also add short description such as “Distill the relation” to emphasize the key message.\nAdditionally, we will include a new illustrative diagram specifically for adaptive CBVA and annotate each merging step. \nOverall, these changes will significantly enhance readability and understanding."}, {"review_id": "rMoXQUgaQM", "reviewer": "Reviewer_3bLp", "summary": "In this paper, authors propose a novel method for PRVR, which addresses the semantic collapse problem on the text and video side. \nThe proposed method contains two modules, *i.e.*, CBVA and TCPL. Extensive experiments show the superiority of the proposed method.", "questions": "See weakness.", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is novel to me.\n2. The proposed model consistently outperforms state-of-the-art methods on four PRVR datasets with up to 8-point gains in SumR.\n3. Demonstrates the necessity of each component (TCPL, OP-ToMe, adaptive CBVA) with clear metrics and diagnostic experiments.\n\nWeakness:\n1. Line 126 - Line 133. Confirm that the frame-level loss only use one visual token per video for l_f^* right?\n2. What is the \\delta for huber loss (Line 163)?\n3. How do \\lambda^E and \\lambda^A affect performance?\n4. Line 192. How to get the size-weighted average?\n5. What backbones do methods in Tables 4 and 5 use?\n6. While L^base explore frame-level loss, how about work-level loss?\n7. What if the benchmark dataset does only have one-to-one video-text data pair?\n8. For now, the proposed methods try to address the problem on the uni-modal side, text and video sides. How about cross-modal side? For example, frame-to-word?", "was_revised": true, "final_justification": "Authors' response addressed most of my concerns. However, this paper to me is an okay/good enough paper. So I maintain my rating at 4 (Leaning to accept but not a clear accept, borderline accept).", "rebuttal": "# Only one visual token per video is used to calculate $\\mathcal{L}_{f}^{*}$?\nYes, the reviewer is correct.\nTo compute $\\mathcal{L}_{f}^{*}$, we select only the video token (from frame-level encodings) that yields the highest similarity to the given text query.\nThis design choice reflects the nature of long untrimmed videos, where only a specific temporal region typically aligns with the query.\nBy focusing on the most relevant token, we encourage the model to learn precise and semantically meaningful associations, rather than diffusing supervision across irrelevant video segments.\n\n# What is delta for Huber loss?\nWe apologize for the confusion. \nIn our custom implementation, we do not use the Huber‑loss parameter $\\delta$; it is fixed at the standard default value of 1.\n\n# How do $\\lambda^{E}$ and $\\lambda^{A}$ affect performance?\nAs shown in Tab. 3 (a,b), the TCPL loss consistently improves performance across a range of values. \nThis demonstrates that our method is robust to the specific values of $\\lambda^{E}$ and $\\lambda^{A}$ as long as they are within a reasonable scale. \nWe selected these weights using general heuristics without tuning them per dataset to avoid overfitting. \nWe will clarify this robustness in the revised manuscript.\n\n# L192. How to get the size-weighted average?\nWe follow the original Token Merging (ToMe) paper [A] and adopt a size-weighted averaging procedure during token merging.\nSpecifically, each visual token is assigned a size vector that is initially set to 1, representing a single frame.\nWhen two tokens $\\mathbf{a}$ and $\\mathbf{b}$ are merged, their corresponding size values $n_a$ and $n_b$ are summed to reflect how many original tokens they now represent.\nThe merged token $\\mathbf{m}$ is then computed as the weighted average of the two token embeddings:\n\\begin{equation}\n    \\mathbf{m} = \\frac{n_a \\times \\mathbf{a} + n_b \\times \\mathbf{b}}{n_a + n_b},\n\\end{equation}\nand its size is updated to the sum of their sizes, i.e., $n_m = n_a + n_b$.\nThis operation ensures that the resulting token faithfully aggregates semantic information from the constituent frames in proportion to their temporal extent.\n\n[A] Bolya et al. Token Merging: Your ViT But Faster. ICLR 2023.\n\n# What backbones do methods in Tab. 4 and Tab. 5 use?\n\nAll methods compared in Tab. 4 and Tab. 5 use identical backbones to ensure fair comparisons across all baselines.\nThe list of backbones used for all methods is provided below.\n\n| Dataset | Visual | Text |\n|--|--|--|\n| QVHighlights | CLIP-B/16 + Slowfast | CLIP-B/16 |\n| TVR | CLIP-L/14 | CLIP-L/14 |\n| ANet-Captions | CLIP-L/14 | CLIP-L/14 |\n| Charades-STA | CLIP-L/14 | CLIP-L/14 |\n\n# While $L^\\text{base}$ explores frame-level loss, how about work-level loss?\n\nWe interpret this as a suggestion to explore word-level (i.e., token-wise) loss for finer-grained text supervision. \n\nHowever, there are two primary reasons we opted for sentence-level loss:\n\n1) Computational Efficiency: Word-level alignment significantly increases training cost since our Text Correlation Preservation Learning (TCPL) involves both pairwise similarity calculation for Euclidean distance and triplet similarity calculation for angular distance.\n\n2) Robustness to Domain Variance: Sentence-level embeddings are more stable across domains, whereas word-level tokens can vary significantly depending on the context. \nFor instance, in real-world video datasets like TVR used in Partially Relevant Video Retrieval (PRVR), text often includes domain-specific proper nouns (e.g., person names or locations). \nSuch variability can make word-level alignment less reliable and prone to overfitting. By focusing on [EOS] token supervision, we encourage the model to capture more domain-invariant semantics.\n\n# What if the benchmark dataset does only have one-to-one video-text data pair?\n\nThis situation can be interpreted in two ways:\n\n1) When each video is still long and untrimmed, so that videos contain multiple semantic contexts internally, but only one query is provided per video:\n\nIn this case, although assigning only one query per video removes intra-video semantic confusion to some extent, semantic collapse can still occur due to semantic overlaps across videos.\nTo verify the effectiveness in this scenario, we trained our baseline and proposed method using one query per video and evaluated it on the full QVHighlights test set.\nResults are as below:\n\n| QVHighlights | R1 | R5 | R10 | R100 | SumR |\n|--|--|--|--|--|--|\n| Base | 13.1 | 32.1 | 44.7 | 86.7 | 176.6 | \n| Ours | 15.6 | 34.6 | 47.5 | 90.1 | 187.8 |\n\nAs shown, even in the one-to-one setting, our method demonstrates clear improvements, highlighting its robustness and general applicability to PRVR-style scenarios.\nThe overall relative performance drop compared to Tab. 4 in the main paper is primarily due to the drastic reduction in training pairs (only about 30\\% of the original text-video pairs are retained under the one-to-one pairing constraint).\n\n2) When each video corresponds to a single semantic concept and is fully matched by one query:\n\nIn this case, the setup reduces to standard video-text retrieval where videos are short and semantically coherent (i.e., aligned one-to-one with a query).\nThus, the risk of semantic collapse is significantly reduced.\nAlthough our method would still function properly in such settings, we expect less contribution from CBVA or TCPL since our method is specifically designed for PRVR scenarios, where videos are long, untrimmed, and associated with multiple semantics.\n\nTo illustrate, Cross‑Branch Video Alignment (CBVA) assumes that a single video may contain several distinct semantic segments (PRVR scenario) and learns to explicitly distinguish different semantics within the long untrimmed videos.\nOn the other hand, TCPL tackles the PRVR-specific problem where multiple queries tied to the same video are forced together (while semantically similar queries from other videos are pushed apart). \nTherefore, our gains are expected to be naturally smaller in the one-to-one setting, where videos are matched fully with the paired query.\nAgain, this is not because the method fails, but because the risk of semantic collapse is significantly reduced.\n\n\n# For now, the proposed methods try to address the problem on the uni-modal side, text and video sides. How about cross-modal side? For example, frame-to-word?\n\nAs the reviewer mentioned, our work primarily focuses on addressing semantic collapse on uni-modal side.\nParticularly, we leveraged underexplored yet reliable forms of supervision (the semantic understanding capability of the vision-language backbone and video timestamps to construct self-supervised objectives that capture intra-video semantic differences).\n\nWe thank the reviewer for the insightful suggestion.\nWe agree that addressing semantic collapse from a cross-modal perspective is a promising future direction.\nIn fact, we experimented with applying TCPL to cross-modal pairs (e.g., sentence-to-frame and word-to-frame alignment), but it led to suboptimal results.\nWe believe this is primarily due to domain shifts between the visual content and the text vocabulary, which makes CLIP's text-video aligned space suboptimal for PRVR (as can be inferred from CLIP's low zero-shot performance in Tab. 5). \nTherefore, designing an effective cross-modal strategy requires more careful formulation than merely distilling CLIP's knowledge and remains an important avenue for future work.\nAs future work, we intend to develop more robust cross-modal alignment techniques that can better adapt to these domain discrepancies."}]

Author rebuttal:

Meta-review: This paper addresses the challenge of semantic collapse in Partially Relevant Video Retrieval (PRVR) and proposes a novel framework combining Text Correlation Preservation Learning and Cross-Branch Video Alignment (CBVA), enhanced with order-preserving token merging and adaptive CBVA. The approach is well-motivated and effectively mitigates embedding collapse, leading to substantial improvements in retrieval accuracy on PRVR benchmarks. The reviewers appreciated the clear problem formulation, strong empirical results, and practical relevance of the proposed solution. While some concerns were raised about limited ablation studies and generalizability analysis, these do not significantly undermine the overall contributions or validity of the findings. And the authors respond to these minor concerns in the discussion period. Given its novelty, strong experimental performance, and potential impact on text-video retrieval, I recommend acceptance.
Decision: Accept (poster)