Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach
Authors: Zechen Bai
Tianjun Xiao
Tong He
Pichao WANG
Zheng Zhang
Thomas Brox
Mike Zheng Shou
Venue: ICLR 2025 Poster (2025)
Keywords: Text-video Retrieval
Vision-Language Model
Multimodal
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce a data-centric approach that bridges the inherent information asymmetry in the text-video video task and achieves state-of-the-art performance.

Abstract: As online video content rapidly grows, the task of text-video retrieval (TVR) becomes increasingly important. A key challenge in TVR is the information asymmetry between video and text: videos are inherently richer in information, while their textual descriptions often capture only fragments of this complexity. This paper introduces a novel, data-centric framework to bridge this gap by enriching textual representations to better match the richness of video content. During training, videos are segmented into event-level clips and captioned to ensure comprehensive coverage. During retrieval, a large language model (LLM) generates semantically diverse queries to capture a broader range of possible matches. To enhance retrieval efficiency, we propose a query selection mechanism that identifies the most relevant and diverse queries, reducing computational cost while improving accuracy. Our method achieves state-of-the-art results across multiple benchmarks, demonstrating the power of data-centric approaches in addressing information asymmetry in TVR. This work paves the way for new research focused on leveraging data to improve cross-modal retrieval.

Published as a conference paper at ICLR 2025
BRIDGING INFORMATION ASYMMETRY IN TEXT-VIDEO RETRIEVAL: A DATA-CENTRIC APPROACH
Zechen Bai', Tianjun Xiao?, Tong He², Pichao Wang", Zheng Zhang?:
Thomas Brox Mike Zheng Shou 1+ Show Lab, National University of Singapore
2 Amazon
University of Freiburg
ABSTRACT
As online video content rapidly grows, the task of text-video retrieval (TVR) becomes increasingly important. key challenge in TVR is the information asym- metry between video and text: videos are inherently richer in information, while their textual descriptions often capture only fragments of this complexity. This paper introduces a novel, data-centric framework to bridge this gap by enriching tex- tual representations to better match the richness of video content. During training, videos are segmented into event-level clips and captioned to ensure comprehensive coverage. During retrieval, a large language model (LLM) generates semantically diverse queries to capture broader range of possible matches. To enhance retrieval efficiency, we propose query selection mechanism that identifies the most relevant and diverse queries, reducing computational cost while improving accuracy. Our method achieves state-of-ther the power of data-centric approaches in addressing information asymmetry in TVR. This work paves the way for new research focused on leveraging data to improve cross- rross-modal retrieval.
results across multiple benchmarks, demonstrating
INTRODUCTION
The explosion of online vidco content has generated an unprecedented demand for efficient and accurate text-video retrieval systems. As key task in vision-language learning, text-video retrieval enables users to find semantically relevant videos based on natural language queries, making indispensable for video search and recommendation systems. Typically, this task approached by encoding both text and video into a joint latent space and calculating their similarity using metric like cosine similarity. Yet, despite advances in the field, text-video retrieval remains fundamentally constrained by an often- ovdeloovvrlooked problem: information asymmetry.
Information asymmetry refers the inherent imbalance between the rich, multi-layered content of videos and their more concise textual descriptions. While videos can capture visual scenes, actions, and interactions in great detail, their associated text queries or captions often capture only fragments of this complexity. For example, in datasets like MSR-VTT (Xu al. (0016b), some key moments in a video may under-described or even omitted altogether, as illustrated in Fig. I This asymmetry complicates both the training and inference processes, as retrieval models may learn to focus only specific parts of the video while disregarding other potentially relevant aspects. Moreover, when users issue text queries that fail to capture the full semantics of a target video, retrieval accuracy diminishes. These challenges raise an important question: can we mitigate information asymmetry by enriching textual representations better match the richness of video content?
Most of-te--ffthhe-at approaches to text- vede-video retrieval (Gorti et al.] 2022: Wu et al.) 2023b Xue et al. 2022), especially those built on the CLIP architecture (Radford et al. 2001), assume a symmetrical relationship between text and video by projecting both into a shared latent space. However, this assumption fails to account for the vast disparity information density between these two modalities. Some recent methods, such as T-MASS (Wang et al. (024a) and UAVTR (Fang et al.) (003).. have sought address this imbalance by enriching text embeddings through stochastic or distribution- based mechanisms, yet they focus on model-centric solutions and largely overlook the underlying
"Corresponding Author

---

Published as a conference paper at ICLR 2025
One Video
Multiple Queries
Absent
Two people fighting each other.
Absent
Absent
woman smiling at looking something
Cage holding a gun.
Figure I: Videos contain much richer information than text. video can be described by numerous possible text queries, while some of them missing in the data.
data asymmetry. In contrast, we propose a novel, data-centric approach that directly addresses this issue by enhancing the textual representations both the training and retrieval stages.
In this paper, we introduce unified text enrichment framework designed to bridge the information asymmetry between text and video. Our method operates multiple levels to comprehensively enrich textual representations. During training, we generate event-level captions for videos using pre-trained Vision-Language Models (VLMs) and event-aware segmentation mechanism, ensuring that all semantically significant moments in the video are captured. This process results in a richer, more diverse set of textual representations that more accurately reflect the video's content. During retrieval, we harness the generative power of Large Language Models (LLMs) to expand and diversify text queries, allowing for better coverage of the semantic complexity of the target video. This two-stage approach ensures that the textual modality can more fully represent the richness of the video, significantly narrowing the information gap.
However, simply generating multiple enriched queries introduces its own challenge: not all queries contribute equally to retrieval success. This realization led us to explore how effectively selecting the most relevant queries can further enhance retrieval performance. A key insight from our work is the introduction of the "Oracle Query" concept, which sheds light on the potential of query selection to significantly boost retrieval performance. By selecting the optimal query from set of enriched queries - the so-called "Oracle Query" - we observed a dramatic improvement in retrieval accuracy. For example, in the MSR-VTT dataset, the Rank-1 metric increased from 46.7% 61.4%. While this "oracle" approach requires access to ground truth during query selection (which impractical real applications), it reveals critical opportunity: effectively selecting queries can simultaneously reduce computational cost and enhance retrieval accuracy. To capitalize on this finding, we introduce a novel query selection mechanism, which selects the most relevant and diverse queries without requiring oracle-level information. This mechanism evaluates queries based on their semantic relevance and diversity, ensuring that only the most effective queries are used. By doing so, we achieve a further performance improvement with the reduction in computational cost, further narrowing the gap between text and video representations. Our contributions can summarized as follows:
We address the problem of information asymmetry in text-video retrieval from novel data-centric perspective, enriching text representations to better align with video content.
We propose unified text enrichment framework that operates during both the training and retrieval stages, leveraging VLMs for event-level captioning and LLMs for query diversification.
We introduce the concept of "Oracle Query", demonstrating the performance potential of optimal query selection in narrowing the gap between text and video representations.
We design query selection mechanism that optimizes the use of enriched queries, reducing computational costs and improving retrieval accuracy.
Our method achieves state-of-ther -ae-oo---t-hr---eee art performance on standard benchmarks such as MSR-VTT (Xu 2016b), MSVD (Wu et al. (2017)), LSMDC (Rohrbach et al. (0155)). and VATEX (Wang et al. demonstrating its robustness and effectiveness across diverse datasets.
ct al.
2019
RELATED WORK
Text- Video Retrieval Early text-video retrieval methods primarily relied on pre-trained unimodal models, leveraging hand-crafted features and modeling cross-modal alignment (Gabeur et al. 2020
Wang et al.] 2022 Lict 2022) The introduction lasoun-angguag models (VLMs), represented
by CI.IP et al. 2021 have inspired works such CL CLIP4Clip (Luo et al. 2022) and Straight-
2

---

Published as a conference paper at ICLR 2025
CLIP (Portilo-Quiitero et al. 2021) to build retrieval models by leveraging pre-trained shared latent spaces. Subsequent studies have tried to finetune CLIP on video-text datasets with specialized designs for cross cross-modal attention mechanisms (Gorti et al. 2022 Wang al.
2022 Fang et al.
2021 lbrahimi et al. 2023 Li et al., 2024). VLMs have also inspired works utilizing synthetic data for this task. Cap4video (Wu 20033)) and CIIP-VIP (Xue al. 2022) focus novel architectures using global video captions or ine-grained frame captions, while Intern Vid (Wang et al.
2023b) and HAVTR (Wang et al. 2024b) generate more larger datasets improve performance. Our approach differs by focusing on data-centric solutions to address information asymmetry problem in tixt-video retrieval. By comprehensively considering both training and retrieval phase, our approach demonstrates significant improvements without the need for architectural changes or large-scale data augmentation. addressing information asymmetry, T-MASS (Wang et al.. 2024a) model the text embedding in the latent space as a stochastic embedding, enriching it with flexible semantic range. UAVTR (Fang et al., 2023) proposes a umcertainty-adaptii approach that models each look-up as a distribution matching procedure. However, they do not fully resolve the asymmetry issue in terms of data. Our work takes a novel data-centric perspective to tackle this problem, providing a complementary approach to these models.
Foundation Models The increasing data and compute give birth to large foundation models (FMs) (Bommasani et al. 202T), including unimodal FMs and multimodal FMs. The most represen- tative unimodal FMs is Large Language Models (LLMs) (Touvron et al. 2023 Brown et al. 2020). By training on huge amount of text data via next-token prediction, LLMs demonstrate exceptional capabilities in language understanding and generation, instruction following, and emergent ability.
The development of LLMs has also boosted the lasion-language domain in various ways. On the model side, a series of Multimodal Large Language Models (MLLMs) (Wu et al.
2023a Bai et al.
2024), such as BLIP-2 (Li et al. 2023a). LLaVA (Liu et al.] 2023), have been built based on LLMs. demonstrating remarkable performance on vision-language tasks (aag al., 2020 Bai al. 2021 and video tasks (Tang et al. 2023 Bai et al. 2025 Lin et al. 2024 Fan et al. (023). On the data side. there are some attempts on "re- "re-writing" (Fan et al.) (024) training text data of CLIP (Radford et al.) 2021) using LLMs to upgrade the original CLIP model. In contrast to previous methods that focus solely on text generation, our text enrichment framework utilize both VLMs and LLMs to enrich the textual representations, providing a more diverse and contextually rich set of text-video matches.
Query expansion has long been used in document retrieval to bridge the gap between sparse queries and rich document content (Formal et al. (202T) Traditional approaches use lexical knowledge bases [Voorhees] 1994) or pseudo-relevance feedback (Lavrenko & Croft. 2017). while modern methods leverage deep generative models (Zheng et al.) 2020). Some recent studies explore prompting LLMSs for query expansion (Wang et 20233a, aiming to diversify the query to mitigate vocabulary mismatch. Our method diverges from traditional query expansion in a key way: we tackle the more challenging cross-modal scenario by using LLMs not just to mitigate lexical gaps, but to enrich the query's s semantic depth and diversity, ensuring the generated queries better align with the rich content of video data. This approach allows to cover broader range of possible video descriptions while maintaining high retricval precision.
Query Expansion
3 METHOD
3.1 MODEL
In text-video retrieval, the goal is to learn a similarity function s(Q, V) for a given text query Q and video V. It aims to maximize the similarity score of positive text-video samples and assign
lower similarity for irrelevant pairs. Consistent with previous studies Luo et al. (2022); ) Portillo
Quintero et al. (202T): Gorti et al. (2022); : Wu et al. (2023b), we adopt pre-trained CLIP Radford
et al. (021)) as the backbone. In the process of adapting pre-trained CLIP from image to the video
domain, a commonly employed technique is mean-pooling, which aggregates frame embeddings into a global video embedding. However, since videos are much more expressive than texts, typical text-agnostic aggregation methods that encompass the entire video may encode irrelevant information not accounted for in the input text. Hence, we utilize text-conditioned pooling from the previous work, X-Pool Gorti et al. (2022).
To elaborate, when provided with the text query Q and the video V, we first use CLIP backbone to obtain the text embedding e, E R. and frame embeddings e, E R RxD where D is the dimension of the latent space, F is the number of frames. Then we employ learnable cross-attention module
3

---

Published as a conference paper at ICLR 2025
Video
Quer glaue guonlii beieoldas bost
Evenil-sware Temporal Sey.
Video Encoder
0
Query Selection
Pooting
pameciig a gooe onu ritualie lot clive rort sirovcatic iitoratto apweling g
Captioner
Apgregation
gro sling it aucurre are coutted lakingg
aclding aiting moos dig thst Aups tuee besinu regltc cililoo
Retriesal Model Wew kaw
Text Encoder
Text Enrichment in Training
Model Training
Text Enrichment in Retrieval
Illustration of the unified text enrichment framework. The left part shows the process of enriching text representations of training data via comprchensive video captioning approach, including an event-aware video temporal segmentation module and a pre- pre-trained captioner. In the middle, we adopt a dual-encoder model with a text-conditioned video pooling design. The right part illustrates text enrichment during retrieval phase, where a query generation module, a query selection module, and aggregation module work together enhance the retrieval performance.
Figure 2:
that learns to highlight the most semantically similar video frames as described in the given text:
evlt CrossAttnne;; ev, e.);
(I)
where the query the cross-attention module is the text embedding er, the key and value are both frame embeddings e.. The resulting output enje aggregated video embedding conditioned on the text. For the sake of simplicity, we omit certain implementation details, such as layer normalization. For more details, please refer to X-Pool Gorti et al. (2022).
For the training objective, we follow the common practice of optimizing the bidirectional learning objective. A symmetric cross-entropy loss is employed to maximize the similarity between matched text-video pairs and and minimize the similarity for other pairs:
B
exp(s(e), (/T))
1
Lozv
log
(2)
B
j=l exp(s(e), e (1/T))
1=1
e
1
Lv2q
log د= exp(s(e) et \/T))
(3)
B
[ (CQ2v Lv29).
where s(s.) ) is cosine similarity function, B is the batch size and is learnable scaling parameter.
3.2
UNIFIED TEXT ENRICHMENT
3.2.1
TRAINING PHASE
The text-conditioned pooling architecture effectively addresses the information asymmetry between the text query and the video by capturing video frames relevant to the text. As result, irrelevant video information still limits the model performance. For instance, information considered irrelevant for text query might be highly relevant text query information is unlikely to be captured during training. In such scenario, the video embedding will be biased toward the content described by query in the latent space. During retrieval, query B will struggle to effectively retrieve the video. If the text queries could comprehensively capture all scenes of a video, then the information of the video can be sufficiently and evenly mined. To tackle this problem, propose enriching the text queries in the training set using comprehensive video captioning approach with two key modules: a video temporal segmentation module and an image captioning module.
mentioned in the text will be eliminated. However, asymmetry the dataset
query B is not present in the training dataset, this
4

---

Published as a conference paper at ICLR 2025
As the core of the video temporal segmentation module, Kernel Temporal Segmentation (KTS) PPotpovv et al. (2014); Lin & Lei (2023) algorithm is adopted. KTS divides the video into non-overlapping temporal segments by analyzing sequence of frame descriptors and identifying the change points. In our implementation, we use a pre-trained CLIP to extract the frame features. Given video V E RRFxxxX3
e, = CLIPimg img (L)P),g(V)) (V), e, ER
(5)
(11, bm (T))
(6)
where t, is the change-ponee point position, m² is the optimal number of change points selected by KTS. For the sake of simplicity, we omit the details of the KTS algorithm. With the detected change-poing the video can be split into multiple segments. Since CLIP features contain semantically rich information, the obtained segments are semantically-consiseeee being regarded as 'atomic' scenes.
Next, employ an image captioning module to generate descriptions for each scene. Specifically, we select the middle frame of each scene as the representative frame as Eq. 7 Compared with video captioning, an image captioning model is more efficient as takes only one frame input. We assume single frame is sufficient for representing the 'atomic' scene.
Qtaain ImageCaptioner( V 4 + li+1 1)-
2
With the above modules, we can generate a set of text queries that captures each semantically- consistent 'atomic' scene for a given video. By running on all the videos of the training set, video-text dataset with complete text queries is obtained. Inevitably, even using state-of-the-an image captioning models, there will be some noise being introduced. In order to make our approach compatible with a wider range of model-centric methods and reveal the importance of data, we have not explored model architecture designs specifically tailored address the noise issue. Instead, we show that simply introducing a pre-training stage on the noisy expanded dataset is effective. More study on the data will be elaborated in Sec. 4.2.1
3.2.2 RETRIEVAL PHASE
In traditional text-video retrieval, one text query is used to retrieve the target video. However, under the information asymmetry condition, a video can be successfully retrieved by multiple possible text queries. In this section, we introduce text query enrichment in the retrieval phase.
In this phase, there no access to the video content or other potential texts related to the video. We must initiate the process from the single existing text query. Large Language Models (LLMs) powerful text generation models, especially suited for zero-shot, few-shot, or prompt- based learning. Therefore, we design prompts to instruct an L.MM to understand the given text description, reason the visual scene, and describe the scene in diverse way to cover various semantic concepts. The detailed prompts will be provided in the Appendix. Ablations (see Tab. (6) suggest that LLMs preferable over manual or rule based methods (e.g. NLTK Bird et al. (2009)) thanks to its language modeling and reasoning capability. Formally, given a text query Q,
{Q1 Q = LLM(prompt, Q).
(8)
is the number of generated queries, specified in the prompts. In this paper, we set n 10.
Next, we explain how the generated queries aid in the retrieval process. One straightforward approach create new, longer text description by appending additional information to the original text. The advantage is that it does not bring additional computation since it only requires one- one-time similarity computation. However, in our experiments (see Tab. (6). we discovered that concutenating text queries or pverage-pooling the text embeddings had a detrimental effect on retrieval performance. We hypothesize that the text encoder struggles to understand the long description. Another option is to run the retrieval with each query independently and then aggregate the results (similarity ranking). By experimenting a series of aggregation strategies, our empirical study shows that aggregating the ranking results using a majority voting strategy yielded the best performance.
3.3 QUERY SELECTION MECHANISM
Despite the performance improvement, several drawbacks remain. First, not all enriched text queries contribute equally to retrieval success. Some contain noisy information due to LLM hallucination, while others are nearly identical to the original query, bringing no novel information. Additionally, more queries increase computational costs. For example, using original and 10 enriched queries
5

---

Published as a conference paper at ICLR 2025
Dap
Step Foom Query AND Dlary
Seg ALL prrviuu Inct
Sonngi Eartee hoo Quarg
Figure 3: Illustration of Farthest Query Sampling (FQS) algorithm. The queries are distributed within certain range of relevance. The blue point (user query) is set as the root query. At each step, FQS samples the query that is farthest from all previous sampled queries.
requires 11 times more pairwise similarity computations than the previous approach. These challenges motivate us to explore effective query selection.
We first introduce the concept of "Oracle Query". From the set of {Q. Qt ..... Qk=*t j. we always select the one that yields the best retrieval performance. Since this process requires access to the ground truth video-text matching, which is impractical in application, demonstrate the potential of text query enrichment and query selection. It can be regarded as an upper bound performance. We found that this oracle query selection improvement even on a baseline model. In MSR- VTT dataset, by simply replacing the original query with oracle query, the Rank-1 metric is boosted from 46. 46.7% 61.4% (see Tab. 6). This surprising result reveals two key insight: i) text enrichment in text phase has significant potential for improving text-video retrieval; ii) selecting an effective query can simultaneously reduce computational costs and enhance retrieval performance.
only use this approach to
achieve dramatic performance
The result motivates us to investigate effective query selection. When defining effective query selection, our primary considerations are two aspects: relevance and diversity. The former aspect demands that enriched queries should accurately convey semantic meaning with minimal illusions. The latter emphasizes that the text queries should be expressed in variety of ways, such that more novel views can be included. Often, these two aspects exhibit some degree contradiction. Relevance may limit diversity, whereas increased diversity also increases the risk of introducing illusions. In this work, the relevance controlled in two folds. First, by designing an appropriate prompt. we enforce the LLM to generate enriched text queries strictly adheres to the factual information in the original query, limiting the generated ones in a certain range. Secondly, in the following query selection process, we always primarily include the user query the initial one, serving the most relevant and reliable anchor. As shown in Fig. [ the output queries are constrained within a range of relevance. After that, we sample a certain number of queries from this pool. The goal of this step is to maximize the diversity within this range. In the process, we design a novel Farthest Query Sampling (FQS) algorithm inspired by point sampling in point cloud research Qi al. (2017).
The main idea is to iteratively select queries in a way that maximizes the minimum distance between the selected queries. The algorithm starts with an initial query and repeatedly selects the query that is farthest from all the previously selected queries. This process continues until the desired number of queries selected. Note that the result of this algorithm affected by the initial query. We set the original user query the initial query provided by the human user to stabilize the algorithm and provide double insurance for relevance. The formal expression is Eq. 9 where Q is the original query, {Qi, Qi) are the enriched queries, k is the number of query selection (k < n). The distance is computed at an embedding space.
(Q,Qi.... Q FQS((Q, geett
Qiest) },k),
EXPERIMENTS
Setups. We employ MSR- VTT Xu et al. (016 MSVD Wu et al. (0077a), Rohrbahh Rohrbach et al. (2015a), and VATEX Wang et al. 2019) to evaluate our method. We use standard retrieval metrics: recall at rank K (R@) @ K, higher is better), median rank (MdR, lower is better), and mean rank (MnR, lower is better) evaluate the performance. The model backbone is initialized from pre- traineddee CLIP ViT-B/32 and undergo end-to-end training on each dataset. More details of experiment setup and implementation elaborated in the appendix.

---

Published as a conference paper at ICLR 2025
Table 1: Text-to-video retrieval results the IK-A test of MSR-VTT 1K Xu al. (016b).
*Denotes that the method uses DSL Cheng et al. (2021) post-processing.
Rank@1
CLIP-WT-BB22
CLo el 2022
Neurccmmp
22
44.5
71.4
31.6
15.3
X-Poo 1a Eu2E
CVPR
QB-Norm Bogoli
2022
CVPR'22
47.2
73.0
$3.0
2,0
ECCV"22
47.0
74.5
83.8
2,0
13.0
LAT Viiierieeeee
ICCV
47.5
73.9
83.5
2.0
12.9
Cap4Viten Wir 2174b
CVPR'25
49.3
74.3
83.8
2.0
12.0
TEFAL
ICCV'23
ProST Sr A7230
ICCY223
48.2
83A
12A
L'COFLA Wane el ML
2023c
ICCV'23
49.4
72.1
12.9
UMI
ICCV'23
51.0
76.5
84.2
CLIPVIP
ICLR'21
30.1
74.8
84.6
T-MANS Want BI 211/41
CVPR
50.2
75.3
85.1
1.0
11.9
TeadiCL
CVPR
46.8
74,3
ECCV'24
DITS Whrsy de
NeurlPS 24
51.9
75.7
84.6
1.0
11.5
X-Pool Gorlt
2022
+Orss
52.1
76.8
86.3
9.7
X-Pool
Oure"
CLIP-VIP et 2t
2022
Ours
53.2
79.3
83.2
10.2
CLIP-VIP XIP-VIPXIEEUU et
2022
Ours*
$8.9
83.4
89.5
1.0
8.3
CLIP-WI-R/16
CLIP2TV Gap et 2021
arXiv
18.3
71.6
$2.8
14,9
CenterC1. Zran
SIGIR 22
48.4
73.8
82.0
2.0
13.8
TS2-Nei
ECCV
49.4
Walyy
UAT VKi et 20745]
IOCV'23
50.8
76.3
85.5
1.0
12.4
CVp
HUPS
CVPR'25
51.4
75.7
$3.9
12.4
CLIP Vippiiiieee erat
T-MASS Wink eral.
2020h
CVPR/24
52.7
77.1
35.6
10.5
X-Pool Garn at
2022
Ours
54.6
79.5
86.7
1.0
9.1
Gorli
+urs*
Table 2: Comparisons with SOTA of 12v retrieval on MSVD Wu et al. (2017b). Grey text denotes using ViT-B/I6 backbone.
Table 3: Comparisons with SOTA of (2v retrieval on VATEX Wang et al. (2019). Grey text denotes using ViT-B/I6 backbone.
Method
RG1
Ri
Ri
MdR]
Method
Rd
Rd15t
R60 10*
MdR
CE Liu et
2019
19.8
49.0
63.8
6.0
HGG
2000
35.1
73.5
83.5
2.0
SUPPORT
28.4
72.3
82.2
2.0
SUPPORT
82.1
89.7
1.0
33.7
CLIPAC
TMVM D.DE4 Poree
36.7
81.3
QB Norm MENSOEE BIN
88.3
93.8
1.0
CLINC my Jein Aineoall Di
43.2
75.3
$4.3
2.0
TS2 Net CIEESIN PUZZY
59.1
90.0
95.2
1.0
ITATVR Fihiruess bil8
46.0
76.3
83.1
200
TEFAT MA Bili PIRS
61.0
90.4
95.1
10
X-Pool
47.2
77.4
$6DD
20
PreST ST-PIR PLASTS
60.6
90.5
95.4
10
DRI Wannersun
48.3
79.1
87.3
20
UATV
61.3
91.0
95.6
1.0
UCUFI
47.4
UCOFTA
90.5
TeachT
T-MASS
10
Cxp4Vides
51.8
Cap4Viate
93.1
1.0
X-Pool Gorti al.
2022
OOuss
51.1
81.1
88.6
X-Pool Gorti
2022
Ours
63.3
92.4
96.8
1.0
4.1
COMPARISON WITH THE STATE-OF-THE-ARE
In this section, we compare our method with recent state-of-the-art methods four ben- chamarks. Tab. shows the comparisons on MSR-VTT. Our method sets art performance in text-to-video retrieval for both ViT-B/32 and ViT-B/16 backbones, sur- passing previous methods significantly. For in- stance, we achieve remarkable +7.6% higher R@! compared to CLIP4Clip when using the same ViT-B/32 backbone for text-to-video re- trieval, without any post-processing techniques. In comparison to methods that also utilize syn-
Table Comparisons with SOTA of 12v retrieval on LSMDC Rohrbach et al. (2015b).
Methed
RBI
R85
R@:00
MdR
MnR.
state-of-the
CF at 2019
11.2
26.9
34.8
25.3
MBNT
2120
12.9
20.9
40.1
19.3
75.0
Nocee Zonna F-E11 2011
6.4
19.5
25,4
3900
Stright MupparCuni intuli Aiiiieee pt
2021
11.3
22.7
29.2
36.5
MDMM Mhloaree 2001
18.8
38.5
47.9
12.3
$8.0
Prnser (khh pxte
15.0
30.8
98.8
300
Teachfeurce cta
2021
17.2
36.5
26.3
13.7
CLIPPSCIP Limiiii F0r4
22.6
41.0
29.1
11.0
61.0
2522
43.7
53.5
8.0
552
CLIP Cip--m 2022
25.6
45.1
S4.6
80
PiuST
24.1
TEPAL
2688
thetic text captions, such as Cap4Video Wu et al. (2023b) and CLIP-VIP Xue et al. (0022), approach consistently achieves higher performance with both ViT-B/32 and VIT-B/16 backbones. It's worth noting that CLIP-VID VIP uses the large-scale scale video-text dataset How To Miech et al.) (2019) for pre-training Despite this, our method still outperforms it, providing further evidence of effec- tiveness of our approach. We further show that, data-centric method, method is compatible
7

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study text enrichment of training phase MSR-VTT Xu et al. 2016b). We
compare our method with other captioning- based text enrichment strategies.
Text Enrichment
R@ 11
R@5t
R@ 101
MdR
MnR,
1 X (Baseline)
46.7
73.0
83.3
2.0
14.2
2 Global Video Caption
46.9
73.0
81.9
2.0
13.9
3 Fixed Time Interval Caption
47.0
72.2
82.3
2.0
14.2
4 Ours (Training Enrich. Only)
47.7
71.9
82.1
2.0
13.9
5 Ours (Retrieval Enrich. Only)
50.2
78.0
86.9
1.0
10.1
Ours (Unified Enrichment)
52.1
76.8
86.3
1.0
9.7
Table 6: Ablation study on text enrichment of the retrieval phase, including query generation and result aggregation strategies.
Query Generation
Result Aggregation
R@11
R@51
R 101
MdR
MnR
X
X
46.7
73.0
83.3
2.0
14.2
Concat.
40.2
78.3
Avg. Text
45.1
71.8
81.6
2.0
14.7
NLTK
Avg.
81.6
Viting
48.3
73.4
84.3
2.0
10,2
Oracle
Concat.
45.4
70.5
80.3
2.0
15.7
Avg. Text
45.9
73.6
82.3
2.0
13.7
LIM
Avg. Sim.
46.1
73.7
82.2
2.0
13.7
Voting
86.0
9.2
Orccle
61.4
82.4
90.0
1.0
7.4
with stronger models, such as CLIP-VIP et al. (022)) and post-processing techniques, such as DSL Cheng et al. (202T), yielding significantly higher performance.
Furthermore, we conduct evaluations on other benchmarks, including MSVD (Tab. 2}. VATEX (Tab. 3. and LSMDC (Tab. 4). To ensure a fair comparison, all experiments in these three datasets conducted using the ViT-B/32 backbone. Our method consistently improves performance across different datasets, ultimately achieving new ofate-of-the-ar results. Overall, our method's consistent performance improvements across these four benchmarks demonstrate the its effectiveness.
4.2
ABLATION STUDY
4.2.1
TEXT ENRICHMENT OF TRAINING PHASE
In Tab. we study the impact of text enrichment of training phase. Row the baseline model trained on the original MSR-VTT Xu et al. (2016b), without query enrichment in training. Row shows results with the model pre-trained on video captions provided by a previous work Wu et al. (2023b), which uses zero-shot video captioning model ZeroCap Tewel et al. (2022). In Row 3, we design a strawman setting, in which we uniformly break a video fixed intervals and do captioning for each chunk. The text-video retrieval model is then pre- pre-trained this enriched caption set. In Rows 4 and Row 6, the model is pre-trained on comprehensive, event-aware video captions generated by approach. To ensure that the scale of data does not lead unfair comparison, generate comparable number of captions for Rows 2 to 6, resulting in approximately 45K additional queries for the 9K training videos in the MSR-VTT dataset. Comparing Rows extra text queries in the training data is generally beneficial. Particularly, Our (Training Enrich. Only) is shown to be more effective, demonstrating 1.0% performance increase. We attribute this improvement to the proposed approach, which effectively reduces the information gap by covering more video scenes. To further validate effectiveness, we compare Row Ours (Testing Enrich. Only) and Row 6 Ours (Unified Enrichment). The 1.9% performance gap demonstrate the text enrichment of training again.
suggests that incorporating
4.2.2 TEXT ENRICHMENT OF RETRIEVAL PHASE
We use original query and 10 enriched queries on the MSR-VTT test set to evaluate and compare different types of query generation and result aggregation methods.
Query Generation. We first compare the two types of query generation methods. For the NLTK- based method, we utilize the NLTK Bird et al. (2099) tool to identify representative terms (nouns
8

---

Published as a conference paper at ICLR 2025
Table 7: Ablation study on query selection, including the selection algorithm and the choice of parameter k, i.e., number of enriched text queries. We report only R@ metric for simplicity. The bold number denotes the highest number for each row.
Model
Query Selection
k= 0
==
E
E=
10
16.7
49.6
18.8
49.6
Baseline
Randoen
46.7
46.7
58.2
49.0
48.9
49.6
49.6
49.6
16.7
A-DP Chen al: 2018
46.7
50.2
49.9
49.6
FQS
16.7
50.2
50.1
49.5
Unitedd Earichment
K-DPS Chen
46.7
50.9
49.6
PQS
47.7
49.6
and verbs) in the query and generate new queries by replacing these terms with their synonyms obtained from the NLTK tool. This strategy can be regarded as traditional query expansion. For the LLM-based method, we instruct the L.L. (GPT-4) to generate text descriptions by reasoning the visual scene. As shown in Tab. [6] under all the result aggregation methods, L.M always outperforms the NLTK-based method. Notably, the oracle result, which is achieve by using one single text query and removes the influence of aggregation methods, provide clearer distinction between the two query generation methods. We will elaborate the detail of how we achieve the oracle performance in Appendix. This performance gap demonstrates the effectiveness of using LLM tool for text query enrichment, thanks to its exceptional capabilities on high-level language understanding, reasoning, and generation.
Result Aggregation. Next, we investigate different methods for aggregating the retrieval results from multiple queries. The results in Tab. 6 show that certain methods, such as concatenating queries and mean-pooling of text embeddings, can lead to even worse performance than the baseline. Among these methods, majority voting is simple yet effective one, outperforming other methods. The most interesting part is the Oracle result. Firstly, it shows the huge potential of text enrichment in retrieval phase with the significant performance improvement. Secondly, the gap between the Oracle result and other methods reveals that there are still a large room for improvement. Finally, the Oracle result is achieved without compromising on computational cost, while majority voting with all the 11 queries appears to be inefficient. These findings inspire us to develop dedicated query selection module to further improve retrieval performance as well as reduce computational cost.
4.2.3 IMPACT OF QUERY SELECTION MECHANISM
Query selection aims to enhance retrieval performance to approach the Oracle performance and improve inference efficiency. In Tab. 7 we report the R@ I retrieval performance under various query selection strategies and selection number k. Firstly, we employ a random selection strategy on the baseline model and run the retrieval with 4 different random seeds, resulting the first 4 rows of the table. It can be observed that randomly selecting a specific number of queries can lead to promising performance. For example, when & = 2, the @@ accuracy performance variation among different & indicates that not all the enriched queries contribute equally.
achieve 49.6 or even 50.2. The
The primary drawback of random query selection is its lack of stability, making the retrieval result of each round different. This characteristic should be avoided in retrieval systems. We demonstrate that the our proposed Farthest Query Sampling achieves competitive or better performance under the same setting of k. The performance improvement can be attributed to that our query selection method considers both relevance and diversity of text queries, being able to cover larger area in the embedding space and naturally increasing successful retrieval rate.
We also compare our FQS with another deterministic method, named determinantal point processes (DPPs)JKleszaa
Kulesza et al. (012).. In particular, we utilize MAP inference of DPPs introduced in Chen al. 2018) for fast and deterministic inference. DPP is similar to our FQS in implementation. returns subset of points of pre-specified size (i.e. k) that are maximally diverse. The results show that kDDPP can be a viable implementation of query selection. However, it exhibits a slightly worse performance than FQS. We analyze that the main difference lies in the initial text query. In FQS, we always take the user query as the starting point and probe enriched queries to maximize the diversity. The involvement of ground truth user query essentially ensures the relevance of text query and establishes a solid retrieval lower bound, allowing FQS to achieve better retrieval performance. In contrast, k-DPP treat the ground-trutt user query and the enriched queries equally. It maximizes

---

Published as a conference paper at ICLR 2025
(2066b) dataset, assessing the impact of Table 8: Experiments conducted MSR-VTT Xu et al.
LLM employed in retrieval text query enrichment.
L.M
Param.
Text Query
R@1
R@5
R@:10
GT User Query
47.7
71.9
82.1
GPT-4
Unknown 3.8
Farthest Enriched Query
41.6
66.2
76.3
Phi-3.5
Farthest Enriched Query
GPT-
Unknown 3.8
FQS (A=2)
52.1
76.8
86.3
Phi-3.5
FQS (k=2)
51.5
77.7
86.5
the diversity aspect from global view, while neglecting the relevance aspect. Thus, it is possible the user query would not be selected. making the retrieval be sensitive to some noisy enriched queries.
In the context of query selection, the hyper- hyper-paramenn parameter k plays a crucial role. Our results indicate that the optimal value for & is 2. It achieves the highest @@ score among most of the tested settings as shown in Tab. 7 Notably, when compared with k 10 (using all the queries), setting i 2 results in higher performance, with an R@l score of 50.2 compared to 49.6, while also substantially reducing computational costs. We choose an even number for l to ensure there is odd number of queries when including the original query, which avoids potential ties in the majority voting process. both theoretical and experimental scenarios, k 2 proves to be the optimal choice.
4.2.4 IMPACT OF LLM CHOICE IN RETRIEVAL. PHASE
In the retrieval phase, the text queries are enriched by an LLM, which plays a crucial role. Throughout the paper, we use GPT-4 API to achieve the text enrichment. this section, we use an open-source, light- LLM,, Phi-3.5 1i-din Abdin et al. (2024) to do the text enrichment task in retrieval phase. The experiment results are shown in Table
Firstly, when using the farthest one of the enriched queries, we observe that both GPT-4 and Phi-3.5 exhibit inferior retrieval performance compared to using GT user query. The performance gap indicates that naively using LLM enrich the text query would introduce noise. In the two LLMs, Phi-3.5 shows much worse performance than GPT-4. This aligns well with our expectation, as the Phi-3.5 model is known to be weaker than GPT-4, introducing more noise in the enriched text queries.
Next, we apply our proposed Farthest Query Sample algorithm to the enriched text queries and conduct retrieval. Surprisingly, we observe only a small performance gap between the two LLMs, 0.5% in 1ank-1. This experiment proves that our method is robust to the LLM. The FQS algorithm and the majority voting aggregation can mitigate large noise. Moreover, the exceptional performance of Phi-3.5 unveil of opportunity of using small language model, further improving inference efficiency.
4.3 MORE RESULTS
We provide more results in the appendix, including I. Experiment setups and implementation
details: A.3 Details and more interesting findings of the oracle query selection; A.4 Analysis of
computational efficiency: a.5]anddA.6. and [A.6] experiments on model architectures beyond X-Pool showing that data-centric approach is compatible with wide range of model-centric approaches; A.7 prompt LLM; [A.9] qualitative examples that demonstrate how the enriched text queries in retrivaal aid in text-video retrieval. We also thoroughly discuss the limitation and future work.
CONCLUSION
In this work, we address the information asymmetry problem between video and text in the text-video retrieval task. We take novel data-centric perspective address this issue at multiple stages. During training. we enrich text descriptions through an event-aware, comprehensive video captioning approach, ensuring full video scene coverage. In the retrieval process, we use large language models to enrich the text queries by reasoning the semantic concepts. We introduce the concept of Oracle Query to reveal the significant potential of data-centric investigation in retrieval. Moreover, we design farthest query sampling algorithm for effective query selection, improving retrieval performance and reducing computational costs. Extensive experiments prove that our method has consistent improvements over four benchmarks, surpassing current state-of-the methods. Our work serves as an inspiration for future research efforts, encompassing both model-centric methods and data- centric enhancements.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG3-RP-2022-030)
ETHICS STATEMENT
This research does not involve human subjects or the collection of personally identifiable information. The datasets used in this work are publicly available and have been previously vetted for ethical use in academic research. We ensured compliance with relevant data licensing agreements and conducted all experiments accordance with the terms of use of the datasets, The proposed framework that involves the usage of large language model (LLM), was designed to improve the fairness and accuracy of cross-modal retrieval systems. However, we acknowledge that LLMs may inadvertently encode and propagate biases present in the training data. To mitigate this, we adopted data-centric approach aimed at generating diverse and representative queries. We encourage future work to continue monitoring and addressing bias in these models to prevent unintended societal harm. Additionally, our method was designed with computational efficiency in mind to reduce resource consumption and ensure more environmentally sustainable research practice. No conflicts of interest or external sponsorship influenced the outcomes of this research.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that our results are fully reproducible. Detailed descriptions of the proposed framework, including the data processing pipeline, model architecture, and training procedures, provided in the main paper well as the comprehensive appendix. To facilitate reproducibility, all datasets used in our experiments are publicly datasets, and the specific processing steps sufficiently described. We will make the code, data, and pre-trained model public.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 19,19, 2024.
Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. Noise estimation using density estimation for sulf-super multimodal learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6644-6652, 2021.
Zechen Bai, Yuta Nakashima, and Noa Garcia. Explain me the painting: Multi-topic knowledgeable art description generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5422-5432, 2021.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint 10XV:22404.18930 2024.
Zechen Bai. Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language instructed reasoning segmentation in videos. Advances in Neural Information Processing Systems, 37:6833-68557 2025.
Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In ICCV, pp. 1738,173,, 2021.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O' O'Reilly Media. Inc." ", 2009.
Simion- Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. Cross modal retrieval with querybank normalisation. In CVPR, pp. 5194-5205, 2022.
11

---

Published as a conference paper at ICLR 2025
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint (tX.::2208..0255000 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in nearal information processing systems, 33:1877-1901, 2020.
Laming Chen, Guoxin Zhang, and Eric Zhou. Fast greedy map inference for determinantal point process to improve recommendation diversity. Advances Neural Information Processing Systems, 31, 2018.
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR, pp. 11888-1067,,200 2020.
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. Improving video-text retrieval by multi- strtii corpus alignment and dual softmax loss. arXiv preprint arXiv:2109.04290 04090, 2021.
Ioana Croitoru, Simion-Vias Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 1583--15933, 2021.
Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, and Aleksandr Petiushko. Mdmmt: Mul- tidomain multimodal transformer for video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3354-3363, 2021.
Ke Fan, Zechen Bai, Tianjun Xiao, Dominik Zietlow, Max Horn, Zixu Zhao, Carl-Johann Simon- Gabriel, Mike Zheng Shou, Francesco Locatello, Bernt Schiele, et al. Unsupervised open- vocabulary object localization in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13747-13755. 2023.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024.
Bo Fang, Chang Liu, Yu Zhou, Min Yang, Yuxin Song, Fu Li, Weiping Wang, Xiangyang Ji, Wanli Ouyang, al. Uatvr: Uncertainty-adaptie text-video retrieval. arXiv preprint arXiv:2301.06309 2023.
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106. 11097, 2021.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.
2021.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, pp. 214-229. Springer, 2020.
Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. Clip2tv: An empir- ical study on transformer-based methods for video-text retrieval. arXiv preprint arXiv:2/11.0555 2021.
Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-vides attention for text-video retrieval. In CVPR, pp. 5006-5015, 2022.
Sarah Ibrahimi, Xiaohang Sun, Pichao Wang, Amanmeet Garg, Ashutosh Sanan, and Mohamed Omar. Audio-enhancee text-to-vided retrieval using text-conditions feature alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12054-12064, 2023.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trendy in Machine Learning, 5(2-3):123-286, 2012.
12

---

Published as a conference paper at ICLR 2025
Victor Lavrenko and W Bruce Croft. Relevance- language models. In ACM SIGIR Forum, volume 51, pp. 260-267. ACM New York, NY, USA, 2017.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- pre- training with frozen image encoders and large language models. arXiv preprini 1rX1: 2023a.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang. Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. Proceedings of the IEEE/CVF 2023b.
International Conference Computer Vision, pp.
Pandeng Li, Yan Li, Hongtao Xie, and Lei Zhang. Neighborhood-adage structure augmented metric learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1367-1375, 2022.
Pandeng Li, Chen-Wei Xie, Liming Zhao, Hongtao Xie, Jiannan Ge, Yun Zheng, Deli Zhao, and Yongdong Zhang. Progressive spatio-temporal prototype matching for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4100-4110, 2023c.
Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random real. Advances in neural information processing systems, 36, 2024.
Chengzhi Lin, Ancong Wu, Junwei Liang, Jun Zhang, Wenhang Ge, Wei-Shi Zheng, and Chunhua Shen. Text-adaptive multiple visual prototype matching for vidco-text retrieval. In NewrIPS, 2022.
Kevin Qinghong Lin and Stan Weixian Lei. Vlog: Video as long document, 2023.
Kevin Qinghong Lin, Pengehuan Zhang, Difei Gao, Xide Xia, Joya Chen, Ziteng Gao, Jinheng Xie, Xuhong Xiao, and Mike Zheng Shou. Learning video context as interleaved multimodal sequences. In European Conference on Computer Vision, pp. 375-395--0 -966..96.00 Springer, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 000583, 2023.
Yang Liu, Samuel Albanic, Arsha Nagrami, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv:1907.13487, 2019.
Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. In ECCV, pp. 319-335. Springer, 2022.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
Antoine Micch, Dimitri Zhukov, Scan-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto Learning text-video embedding by watching hundred million narrated video clips. In ICCV, pp. 2630-2640, 2019.
Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander G Hauptmann, Joao F Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR, 2021.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-624, 2023.
Jesús Andrés Portillo-Quinters José Carlos Ortiz-Bayliss and Hugo Maashima-Marinn A straightfor- ward framework for video retrieval using clip. In Mexican Conference on Pattern Recognition, pp.
3-12. Springer, 2021.
13

---

Published as a conference paper at ICLR 2025
Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid. Category speccfff video summarization. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part VI 13, pp. 540-555. Springer, 2014.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. pp. 8748-8763 8763. PMLR, 2021.
Anna Rohrbach, Marcus Rohrbach, and Bernt Schiele. The long-short story of movie description. In Pattern Recognition: 37th German Conference. GCPR 2015, Aachen, Cermany, October 7-10, 2015, Proceedings 37, pp. 209 2211.2211122 Springer, 2015a.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. dataset for movie description. In CVPR. pp.
2015b.
Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: survey. arXiv preprini arXiv:2312.17432. 2023.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In CVPR, pp. 17918-17928, 2022.
Kaibin Tian, Ruixiang Zhao, Zijie Xin, Bangxiang Lan, and Xirong Li. Holistic features are almost sufficient for text-to-video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17138 17147, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ellen M Voorhees. Query expansion using rexical-semantcr relations. In SIGIR '94: Proceedings of the Seventeenth Annual International ACM-SIGIL Conference on Research and Development in Information Retrieval, organised by Dublin City University, pp. 61-69. Springer, 1994.
Jiamian Wang, WANG Pichao, Dongfang Liu, Qiang Guan, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Diffusion-inspires truncated sampler for text-video retrieval. In The eiihth- Annual Conference on Neural Information Processing Systems.
Jiamian Wang, Guohao Sun, Pichao Wang. Dongfang Liu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Text is mass: Modeling as stochastic embedding for text-video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16551-16560 2024a.
Li Wang, Zechen Bai, Yonghua Zhang, and Hongtao Lu. Show, recall, and tell: Image captioning with recall mechanism. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 12176-12183, 2020.
Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models. arXiv preprint T.Vv:233330067.. 2023a.
Qiang Wang. Yanhao Zhang, Yun Zheng, Pan Pan, and Xian-Sheng Hua. Disentangled representation learning for text-video retrieval. arXiv preprint arXiv:2203.0711 2022.
Wenzhe Wang, Mengdan Zhang, Runnan Chen, Guanyu Cai, Penghao Zhou, Pai Peng. Xiaowei Guo, Jian Wu, and Xing Sun. Dig into multi-modal cues for video retrieval with hierarchical alignment. In IJCAL, pp. 1113-1121, 2021a.
Xiaohan Wang, Linchao Zhu, and Yi Yang. T2vlad: global-local sequence alignment for text-video retrieval. In CVPR, pp. 5079-5088, 2021b.
14

---

Published as a conference paper at ICLR 2025
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-ane aetoo-nddlanuuage research. In ICCV, pp. 4581-4591, 2019.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma. Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.0604 2023b.
Yimu Wang, Shuai Yuan, Xiangru Jian, Wei Pang, Mushi Wang, and Ning Yu. Havtr: Improv- ing video-text retrieval through augmentation using large foundation models. arXiv preprint arXiv:2404.00500 05083, 2024b.
Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarse-to-fine alignment for video-text retrieval. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2816-2827, 2023c.
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. Multimodal large language models: survey. In 2023 IEEE International Conference on Big Data (BigData), pp. 2247-2256 IEEE, 2023a.
Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023b.
Zuxuan Wu, Ting Yao, Yanwei Fu, and Yu-Gang Jiang. Deep learning for video classification and captioning, In Frontiers of multimedia research, pp. 3-29. 2017a.
Zuxuan Wu, Ting Yao, Yanwei Fu, and Yu-Gang Jiang. Deep learning for video classification and captioning. Frontiers of multimedia research, pp. 3-29. 2017b.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5288-528999 5296, 2016a.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pp. 5288-5296, 2016b.
Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. arXiv preprint (tt::220090043330 2022.
Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. Centerclip: Token clustering for efficient text-video retrieval. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 970-981, 2022.
Zhi Zheng, Kai Hui. Ben He, Xianpei Han, Le Sun, and Andrew Yates. Bert-qe: contextualized query expansion for document re-ranking. arXiv preprint arXiv:2009.07258. 2020.
Xianwei Zhuang, Hongxiang Li, Xuxin Cheng, Zhihong Zhu, Yuxin Xie, and Yuexian Zou, Kdpror: A knowledge-decoupling probabilistic framework for video-text retrieval. In European Conference on Computer Vision. Springer. Springer, 2024.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.I EXPERIMENT SETUPS AND IMPLEMENTATION DETAILS
Datasets. MSR- VTT Xu et al. (20 contains total of 10K video clips, each having 20 cap- tions. We utilize the training in subset for training and the test-1K-A subset for evaluation. MSVD Wu et al. (2017)) contains 1,970 videos with 80K captions, with 40 captions per video. There are 1200, 100, and 670 videos in the train, validation, and test sets, respectively. LSMDC[Rorbbae et al. (0155) consists of 1 118,081 video clips sourced from 202 movies with one caption corresponding to each clip. Evaluation is conducted on a test set 1,000 videos from movies disjoint from the train and validation sets. VATEX Wang et al. (2019) collects around 35K videos with multiple text annotations in both English and Chinese for each video. There are around 26K videos for training, 1,500 for validation, and 1, 1,500 for testing.
average
Evaluation Metrics. We use standard retrieval metrics: recall at rank K (R@K, higher is better), median rank (MdR, lower better), and mean rank (MnR, lower is better) evaluate the performance. R@K K (Recall at K) calculates the percentage of test samples for which the correct result is found in the top-K retrieved points the query. We report results for R @ 1, @@5, and @@ 10. Median Rank calculates the median of the ground-truth results in the ranking. Mean Rank calculates the mean rank of all correct results.
Implementation Details. We initialize our backbone from the pre-trained CLIP ViT-B/32. Following previous work Gorti al. (2022). the text-conditioned pooling module randomly initialized. Our models undergo end-to-oee to--to-en- en- training on each dataset. For the training hyyper-araaeters we also follow X-Pool Gorti et al. (2022). The model is trained for 3 epochs on the enriched training set and 5 epochs on the standard training set. A cosine scheduler CrntuulerLosschiile Loshchilov & Hutter (2016) is employed to decay the learning rate. For all experiments, we follow previous works by uniformly sampling 12 frames from each video and resizing them to 224x224. For text enrichment in training, use bl blip2-opt-2 -2. . 7b-coco as the captioner. For text enrichment retrieval, we utilize GPT-4 model through the API to generate queries. We train our model using NVIDIA A10 24G GPU. The training takes around 12 hours.
A.2 ADDITIONAL ABLATION OF FQS
Table 9: Comparing FQS with other variants.
Query Selection (k=2)
Rank-1
Rank-5
Rank-10
Random
49.6
76.1
85.9
Nearest Neighhor
50.4
76.0
85.7
Farthest Neighbor
51.4
76.5
85,8
k-DPP
50.9
76.7
FQS (s=0.5)
52.1
76.8
86.3
FQS (s=0.75)
52.1
76.8
86.3
FQS (s=0.85)
51.6
76.7
86.2
FQS (s=0.95)
49.8
76.1
86.3
NQS FQS
49.9
75.8
52.1
76.8
86.3
The Farthest Query Sampling (FQS) algorithm proposed in the main paper plays an important role on selecting relevant yet diverse queries, In this section, we compare this algorithm with wide range of settings to further demonstrate its effectiveness. The results are shown in Tab. 9
Random serves as a bascline, which randomly select & queries from the enriched queries.
Nearest Neighbor selects k queries that are closest to the user query from the enriched queries.
Farthest Neighbor selects & queries that are farthest to the user query from the enriched queries.
https://openai. httpa/aaa...aamaoo///////eee/
16

---

Published as a conference paper at ICLR 2025
k-DPP is classical algorithm selecting subset with maximal diversity from set of data points.
FQS(s=) is an variant of FQS suggested by one reviewer during the review process of the paper. The value of "s" is minimum similarity threshold. In implementation, when iteratively selecting queries in FQS, we add an additional condition that requíres the selected queries to meet the minimum similarity. If no query meets this condition, we will use the original user query instead.
NQS is another variant of FQS suggested by the reviewer, denoting Nearest Query Sampling. which minimizes the minimum distance between selected queries.
The comparison in Tab. 9 can further verify the effectiveness of the proposed FQS algorithm.
A.3 UNDERSTANDING ORACLE QUERY SELECTION
The oracle query selection is implemented follows: for one specific instance, among GT query and & enriched queries, we always select the query that yields the best ranking, where the best ranking refers to that the target video is ranked highest. If there are multiple such queries, we prioritize the GT query. We call it 'Oracle' because we need to access the ground-truth matching during the query selection, which is impractical in real application. Following the rule above, we stats the percentage of GT query test samples of MSR-VTT dataset, & 10. The result shows that 59.5% of selected queries come from the GT query set, while 40.5% of selected queries are from the enriched queries. This actually provides an explanation of the huge performance gap between baseline and oracle, suggesting that the text queries can be specially investigated to enhance the retrieval.
enriched query in the selected oracle query. The statistic is conducted on 1, 000
A.4 ANALYSIS OF COMPUTATIONAL COST
Table 10: Analysis of computational cost.
Model
Term
N₂
N.
N.
Memory
Infer. time
Txt Enc.
Otar
1
0
602 MB
6.30 ms
Txt Enc. (Text Enrich.)
(1 I N₂) & Ctat
10
618 MB
8.65 ms
Similarity
Catm
0
100,000
10096 MB
96.71
Similarity (Text Enrich.)
(1 No) @0 N₂ & Csim
10
100,000
16258 MB
233.80 ms
Similarity (Query Select.)
(1 I N.) 8 N, 8 Czim
I
100,000 2
11490 MB
139.27 ms
Full Infer. (Baseline)
Cuattt CLEMS
0
100,000
103.01 ms
Full Infer. (Text Enrich.)
Crasa CLEM
1
10
100,000
242.45 ms
Full Infe. (Query Select.)
Ctotal CLLM
1
2
100,000
147.92 ms
We decompose the computational cost of a CLIP-based retrieval model into several parts. Formally, given N, text queries and N, candidate videos, supposing each text query is enriched with N. additional text queries, the computational cost can be expressed as follows:
Ctotal = (N, + N₂ x No) @ Cist_ene + No @ Goid e (N, + N, N.) @ N1 @ Csim + N2 @ CLIMM,
(10)
where Ctxe ene and Coid enc the inference cost of the text encoder and video encoder, respectively. Csim is the cost of computing the cross- modal similarity. CLEM is the inference cost of LLM. The operation of @ indicates that the the cost C is influenced by the factors N. Note that @ is different from traditional multiplication operation x, as it can be parallelized and accelerated by GPU.
In practical applications, we usually one query to search among large volume of candidate videos. So, we first set N₂ = 1 for simplicity. Besides, the video embeddings are usually pre-extracted and stored in database. Thus, the Cuid_enc enc be omitted. For each query, the total cost can be simplified as:
Ciotal =
-1 (1+ (N) @ Crnt_ene + (1+ Ne) @ No @ Csim + CLLM.
(11)
Based on this, we suggest two ways to reduce the computational cost. First, our query selection mechanism can effectively reduce N, without sacrificing performance. In fact, it even further boost
17

---

Published as a conference paper at ICLR 2025
the performance by large margin. It worth mentioning that in real world applications, No can be super large. Thus, reducing Ne would be effective on improving the overall efficiency. Second, our approach exhibits strong robustness the selection of LLM, as discussed in Sec (4...4 of the paper. implies that we are allowed to employ lightweight LLM to reduce CLEM and improve efficiency.
To help understand the inference practical retrieval scenario, we may have one user query and large volume of candidate videos, say N; = 1 and N, = 100, 000. The inference time are computed by averaging 1,000 randomly generated examples. We observe that, with the help of GPU parallelization, increasing Ne does not lead to significant inference burden text encoder. However, the similarity computation cost a long inference time. This is because X-Pool requires a text-conditioned pooling, as introduced in the main paper. Finally, we show that vanilla text enrichment strategy increase the per query inference time from m0 into 242.45ms. With the help of query selection, the inference time decrease to 147.92ma validating the effectiveness of query selection. Not that this inference time does not take CLLM into account, it varies depends on different LLMs. We acknowledgement that the inference cost of LLM could be large. However, can be reduced by selecting lightweight model or more advanced techniques, e.g., KV Cache Pope al. 2023). Our main goal is to unveil the power of data in text-video retrieval. We leave this optimization as a future work.
more intuitively, we provide example in Table [O. In
A.5 EXPERIMENT ON VANILLA MODEL
In main paper, utilize the pre-trained CLIP and the text-conditioned pooling module the retrieval model. The text-conditioned pooling is designed to capture video frames relevant to the text query, effectively alleviating the information asymmetry. In this section, we demonstrate that our data-centric approach is not coupled with a specific architecture design, such as the text-conditioned pooling. We conduct experiments on the vanilla CLIP model with a mean mrean-pooling strategy obtain the video embedding. The results are reported in Tab.
Table 11: Text-to-video retrieval results with vanilla CLIP mean-pooling model on MSRVTT Xu et al. (2016b).
Text Enrich. Training
Text Enrich. Retrieval
R @
R@5
R@
MdR
MnR
1
x
42.2
70.4
79.2
2.0
15.7
x
42.9
70.2
79.8
2.0
16.3
3
X
/
46.5
75.9
85.1
2.0
11.6
4 /
/
48.8
75.3
84.6
2.0
11.4
The comparison between Row and Row 2, well as between Row 3 and Row 4, illustrate the effectiveness of Text Enrichment in Training. In the baseline model (Row 1 vs. Row 2). Text Enrichment in Training achieves improvement of 0.7% on R@1 1. The performance gain becomes more obvious when combined with Text Enrichment in Retrieval (+2.3% on R@ 1, Row 3 vs. Row 4). Finally, the 6.6% performance gap between the results in Row and Row 4 further validates the benefits of our unified framework.
A.6 TEXT ENRICHMENT IN RETRIEVAL AS A TRAINING-FREE TOOL
In this section, we demonstrate that the proposed Text Enrichment in Retrieval can be applied to a wide range of text-video retrieval models as a training free tool for performance improvement. We conduct evaluations on the MSR- -TT dataset Xu al. (2016b) using four widely recognized methods. 1) For CLIP Radford et al. (2021) zero-shot, we utilize the pre-trained CLIP model and employ mean-pooling to obtain the video emhedding in a zero-shot manner; 2) CLIP4Clip Luo et al.) (2022). classical method in the text-video retrieval domain; 3) X-Pool Gorti et al. (2022) adopted as the base model in our work; 4) CLIP- VIP Xue et al. (2022). recognized as one of the current ofate-of-the-an methods. To maintain consistency, all experiments utilize the VIT-B/32 the retrieval backbone and CLIP text encoder for query selection. The baseline results are reproduced from their released codebases.
The text-to-video retrieval results are reported in Tab. 12
We observe that Text Enrichment in Retrieval consistently improves the retrieval performance across the four methods. For the zero-
18

---

Published as a conference paper at ICLR 2025
Table 12: Text-to-vider retrieval results with various methods the MSR-VTT Xu et al. (2016b).
The result are reproduced based the released codebase of each method respectively.
Model
R@1
R@5
R@
MdR
MnR
CLIP sese shor Radford
2021
31.2
52.4
63.6
5.0
42.8
CLI slo Kaaffrd of ah 2021
Text Earich. Retrieval
36.4 (+5.2)
62.6
71.6
3.0
27.7
J.IPCCiipaaxppoiin
J00
(Reprduced)
42.3
2.0
16.5
CLIP4Clipnangodi
Text Enrich. Retrieval
46.3 (+4.0)
76.3
85.3
2.0
11.9
X-Pool Gorti al. 2022
(Reproduced)
46.7
73.0
83.3
2.0
14.2
X-Pool Goru
Text Enrich. Retrivaal
50.6 (+3.9)
76.3
86.5
1,0
10.4
CLIP VIP Xue et
2022
(Repraduced)
49.3
74.8
84,9
2.0
13.4
CLIP-VIP Xue
Text Fnrich. Retrieval
53.2 (+3.9)
79.3
88.2
1.0
10.2
CLIP-VIP Xue et 2022
DSL Cheng et all 2021 (Reproduced)
55.1
78.9
86.6
1.0
11.1
CLIP-VIP XB2
DSL
Chenng
Text Enrich. Retrieval 58.9 (+3.8)
83.4
89.5
1.0
8.357
shot CLIP. our method boosts the R@I by 5.2% and R@S by over 10%, which is a significant improvement. Notably, for the state-of-the-art CLIP- VIP Xue al. (2022), despite the remarkable results achieved, our method still improves the performance by a large margin, further validating its effectiveness.
We note that several popular methods, including DhenCheng et al. [202]) and QB-Norm Bogolin et al. (2022), enhance pre-trained text-video retrieval performance using a similar training-free approach. In the final experiment of Tab. 12] we demonstrate that our Text Enrichment in Retrieval is compatible with these training- -free methods. The combination of our approach and DSL significantly elevates the performance to much higher level. Moreover, it is important to note that DSL [Cheng et al. (202T) requires access to all queries during testing, which is an impractical assumption for real- woaldwrldd retrieval systems. In contrast, our approach does not require access to other queries during testing, making it more practical in real-world applications.
Prompts
You are given caption describing visual scene. Your task rewrite the caption into 10 different sentences following the rules:
1. You can diversify the sentence structure and word usage, but you should strictly keep the same semantic meaning.
2.
Do not add uncertain details that do not associate with the visual scene. The rewriting should strietly follow the factual information in the original caption.
The rewritten captions should be diverse in number of words.
The rewritten captions should be no more than 10 words longer than the original caption.
The input caption is: 0
Figure 4: Prompts used for text enrichment in retrieval.
A.7 PROMPT FOR TEXT ENRICHMENT IN RETRIEVAL
We provide the prompt used for enriching testing retrieval queries in Fig.
A.8 LIMITATION AND FURTHER WORK
Despite the significant performance achieved by our method, it exhibits certain limitations that serve as inspiration for future research. Firstly, in the text enrichment in training. our approach focuses on data-centric approach that enriches the dataset. However, there is a potential for further improvement by designing dedicated model architectures tailored to the enriched dataset, such as multi-instance learning. Secondly, in the text enrichment of testing, we employ LLM as the tool to generate diverse queries. Given the vast prompting space of LLM, we have not invested significant efforts in prompt engineering. As a future research avenue, more dedicated prompts, such as in-context learning, can be harnessed to further enhance the quality of text queries. Thirdly, although query selection mechanism has effectively reduced the computational cost in retrieval, the computation of LLM still exists. Optimization this part further benefit more practical applications. Fourthly, the
19

---

Published as a conference paper at ICLR 2025
involvement of VLMs and LLMs would inevitably introduce hallucinated content, which may limit the performance of the method. Although the mechanisms the paper are shown to be resilient to noise, reducing hallucination has the potential to further improve the performance in the future. Finally, we acknowledge that there is still a noticeable gap in terms of retrieval performance and computational cost between method and the upper bound. This diserepancy encourages further research to bridge the gap.
A.9 QUALITATIVE EXAMPLES
In this section, we present a series of text-video retrieval examples from the MSR-VT VTR dataset Xu al. (2016b). These examples not only qualitatively validate the effectiveness of our approach but also demonstrate how the enriched queries aid in text-video retrieval. The examples from training set shown in Fig. The examples from the testing phase shown in Fig. 3 Fig. Fig. and Fig. 8 In each example, we show that the original query fails to retrieve the target video, while the enriched query does. Detailed explanations are presented in the caption of each figure, respectively.
20

---

Published as a conference paper at ICLR 2025
Original query: soccer team walking out the field. Incorrect video.
Enriched query: Players from soccer leam are stepping onto the playing field.
/ Correct video.
JPN
NED
Figure 5: Rephrasing concepts. The original query focuses on the concept 'team'. Although the concept of 'team" may imply the involvement of 'multiple players', it is not explicitly stated. The enriched query provides a more explicit view elaborate the query, correcting the retrieval.
Original query: woman is cooking food a man setting table. x Incorrect video.
Enriched query: The man is setting table, and at the same time, woman is cooking- / Correct video.
Figure 6: Altering word order. The initially recalled video successfully covers the concept of "woman', 'man'. and 'cooking food'. While it does contain content corresponding to 'setting table'. Our hypothesis is that this is because 'setting table' is positioned far back in the original query, limiting its impact. The enriched query recalls the video that captures the most terms the query. This is achieved by altering the word order and emphasis of the original query.
21

---

Published as a conference paper at ICLR 2025
Original query: opening nest rate coming out searching something eats something on a human hand x Incorrect video.
Enriched query: From open rat comes searches, and consumes something from human hand.
v Correct video.
Figure 7: Typo correction. We notice that there a typo in the original query ('rate' vs. 'rat'). Even a very small typo like this can lead to problematic retrieval result. The incorrect video has captured the concept of 'hand' but fails on other aspects. With the corrected query, the target video. This capability stems from strong language understanding ability of the large language model, which automatically recognizes the incoherent word in the query and corrects it. This case is challenging to correct for rule-based methods, such a NLTK, since 'rate' is not a wrong word.
successfully find
Original query: a woman dances in the background while guy doesn't move. Incorrect video.
Enriched query: is standing still while woman in the background is dancing. Correct video.
Figure 8: Robustness. In this example, the original query is clear while the initially recalled video appears be not strongly relevant the original query. This might be due to the inherent weakness of the retrieval model. However, the enriched query can still successfully retrieve the target video. This can seen as an indication that text enrichment improves the robustness of the retrieval model, making it resilient to corner failure cases.
22

---

Published as a conference paper at ICLR 2025
Two men sitting on red chairs in a room.
A man sitting in chair with microphone and woman standing behind him. woman green jacket holding a microphone.
The voice judges are sitting in chairs on stage.
close up owl with yellow eyes room.
A close of an owl looking at the camera.
A person petting small bird pink surface.
person is petting a small bird on pink surface.
man is walking down a city street at night.
A group of people standing outside of store at night.
man is standing in front of pizza shop window.
A man is standing in front of vending machine.
A man standing behind counter in restaurant with sign.
person walking down a hallway with a flashlight.
Two people sitting in chairs talking to each other.
A woman looking at something the dark.
woman looking off into the distance a forest.
A man and woman sitting at a table with laptop.
person stirring a pot stew with a wooden spoon.
close up of a bowl of meat on table.
A person is adding seasoning to a pile of meat. bowl food with a spoon in it.
Figure 9:
Examples from MSR-VTT training dataset with text enrichment.
23


Reviews:
[{"review_id": "dqIS3pJUfS", "reviewer": "Reviewer_Tkyv", "summary": "This work aims to solve the problem of information asymmetry between text descriptions and videos in text-video retrieval,i.e., text often contains only part of the video content. The author uses VLM to generate more captions from videos during training; and uses LLM to expand queries and add more content during inference.", "questions": "1. Line 290 proposed to ensure the diversity in the supplemented queries. Why are the supplemented queries should be expressed in as many ways?\n\n2. To verify the effectiveness of the proposed method, the authors should provide some examples of the generated captions during training and the expanded queries during inference. It can help to understand the contribution of the work.\n\n3. The approach proposed in Figure 3 requires more illustration on design idea and details; the current version is a bit confusing.\n\n4. The authors need to explain the training efficiency and inference efficiency of the proposed method.\n\nOther questions seen in the Weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed problem of information asymmetry in the text-video retrieval is important and needs to be solved in the development of this task.\n2. The paper is well written and easy to follow.", "weaknesses": "1. The proposed method uses VLM and LLM to generate details to the original caption. The captions and queries generated in this way are unreliable and contain a lot of information that is not in the video. In the inference phase, without the addition of oracle information, the newly generated captions and querys are hallucinated by the large language model. The authors need to provide more evidence to support their idea.\n\n2. The work involves utilizing large visual language models and large language models. The required training cost and inference computation are much higher than compared works. It`s not fair to compare directly in experiments. The authors need to prove the performance gain is not from the accumulation of more computation.\n\n3. Line 238 mentions that this work used an image captioner. The designed method does not consider the temporal cues in the video neither. The proposed method is essentially designed for an image-text retrieval task rather than a video-text retrieval task.\n\n4. In line 278, the fact that oracle query can improve model performance does not mean that supplementing the query is useful. Introducing the ground truth query can greatly enhance model performance, regardless of whether the query is supplemented or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jnRvKfqzJG", "reviewer": "Reviewer_38bZ", "summary": "Due to the information asymmetry between video and text in text-video retrieval (TVR), this paper introduces a data-centric framework aimed at enriching textual representations to better align with the rich information contained in video content. However, despite efforts to leverage vision-language models (VLMs) and large language models (LLMs), the method lacks significant innovation.", "questions": "Q1.  Could you include experiments comparing the model’s performance with more recent methods beyond those published in 2022? This would enhance the context of the model's strengths and limitations relative to current advancements.\nQ2. How might the proposed framework perform in real-world, large-scale text-video retrieval systems？\nQ3.  Could you clarify how this work distinguishes itself from existing methods beyond leveraging VLMs and LLMs? Are there any unique components or methodological innovations that specifically contribute to the improvements in performance?\n\nBelow are a few suggestions which could help the authors to refine a better version of this work.\n\n1. This paper addressed the issue of information asymmetry in text-video retrieval from a data-centric perspective, therefore leveraging VLMs for event-level captioning and LLMs for query diversification. While this approach primarily combined existing methods to enhance textual representations, emphasizing the uniqueness of the model structure would better highlight its innovative contributions.\n \n2. As the paper mentioned, relevant and diverse queries are expected to be retrieved. After initially constraining the relevance of the generated queries, FQS is applied to iteratively select queries by maximizing the minimum distance between them.\nIt may be worth considering adding relevance constraints to FQS, such as setting a minimum similarity threshold between query embeddings. Similarity metrics, such as dot product or cosine similarity, could be used for this purpose. Incorporating both distance and similarity into FQS might enhance its performance.\nAdditionally, FQS could be also compared with methods that minimize the minimum distance between selected queries to further demonstrate its ability to select diverse queries effectively.\nOverall, designing alternative algorithms as comparisons would provide a stronger demonstration of FQS’s effectiveness.\n \n3. In this paper, Table 1 presents the performance of the latest models. However, Tables 2 and 3 do not include these methods, such as Cap4Video. Including these comparisons would provide a more comprehensive evaluation.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1.  This paper shows a method of utilizing VLMs and LLMs to enrich textual representations in training and inference stage.\nS2. The data-centric method proposed by this paper maybe useful in practice, as this paper provides evidence of its effectiveness.\nS3.  The writing is well-structured and clear, making the paper easy to follow.", "weaknesses": "W1. The method does not present substantial innovation. The improvement in model performance appears to be primarily attributed to the inherent capabilities of VLMs and LLMs in visual and textual understanding.\nW2. While the paper introduces a query selection mechanism and designs a Farthest Query Sampling (FQS) algorithm, it would benefit from exploring additional query selection algorithms to further validate FQS’s effectiveness.\nW3. The comparative experiments with the latest methods are somewhat lacking. Most of the related work cited in the experiments on state-of-the-art (SOTA) methods is limited to publications from 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "863AklBP8W", "reviewer": "Reviewer_8p6G", "summary": "This paper investigates the problem of information asymmetry in text video retrieval from a data-centric perspective. Existing works usually use model-centric approaches of carefully designing advanced text-video interaction modules, e.g., text-conditioned video representations, stochastic embedding, etc. This paper proposes a unified text enrichment framework to enhance the textual representation during both training and testing phase. The results consistently show promising improvements over existing methods. Interestingly, the concept of “oracle query” clearly demonstrate the potential of the query generation and selection, possibly opening up a new venue in this domain.", "questions": "- Will the author open-source the enriched dataset and code?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper investigates an under-explored area in the field of text-video retrieval, a data-centric approach to improving the performance by identifying pitfalls in the current training dataset captions and test retrieval process.\n- The full method demonstrate remarkable performance in the text-video retrieval task across several benchmarks.\n- The experiments with the oracle queries show a huge gap in the current methods, which is novel and interesting. It opens up future works to consider such data-centric approaches for improving performance.\n- The paper is well written and easy to follow.", "weaknesses": "- There are some existing works on augmenting the data, e.g., [1], what is the main difference between this work and existing works?\n- The oracle query experiment shows huge performance gap. Although the result is inspiring, it is better to provide more explanations and/or investigations about this phenomenon. What contribute to the final performance?\n- In the ablation study, the performance gain seems mainly come from the Retrieval Phase Enrichment, while the gain Training Phase Enrichment seems to be marginal.\n- The method will inevitably increase the computational cost, which should better be investigated.\n- It is mentioned that majority voting over the enriched queries yields the best performance. However, it is not clear how this is implemented. I would suggest include such details, as least in the appendix.\n\n[1] HAVTR: Improving Video-Text Retrieval Through Augmentation Using Large Foundation Models. ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)