Diffusion-Inspired Truncated Sampler for Text-Video Retrieval
Authors: Jiamian Wang
Pichao WANG
Dongfang Liu
Qiang Guan
Sohail Dianat
MAJID RABBANI
Raghuveer Rao
ZHIQIANG TAO
Venue: NeurIPS 2024 poster (2024)
Keywords: text-video retrieval
diffusion model
multimodal
Primary area: machine_vision

Abstract: Prevalent text-to-video retrieval methods represent multimodal text-video data in a joint embedding space, aiming at bridging the relevant text-video pairs and pulling away irrelevant ones. One main challenge in state-of-the-art retrieval methods lies in the modality gap, which stems from the substantial disparities between text and video and can persist in the joint space. In this work, we leverage the potential of Diffusion models to address the text-video modality gap by progressively aligning text and video embeddings in a unified space. However, we identify two key limitations of existing Diffusion models in retrieval tasks: The L2 loss does not fit the ranking problem inherent in text-video retrieval, and the generation quality heavily depends on the varied initial point drawn from the isotropic Gaussian, causing inaccurate retrieval. To this end, we introduce a new Diffusion-Inspired Truncated Sampler (DITS) that jointly performs progressive alignment and modality gap modeling in the joint embedding space. The key innovation of DITS is to leverage the inherent proximity of text and video embeddings, defining a truncated diffusion flow from the fixed text embedding to the video embedding, enhancing controllability compared to adopting the isotropic Gaussian. Moreover, DITS adopts the contrastive loss to jointly consider the relevant and irrelevant pairs, not only facilitating alignment but also yielding a discriminatively structured embedding. Experiments on five benchmark datasets suggest the state-of-the-art performance of DITS. We empirically find that DITS can also improve the structure of the CLIP embedding space. Code is available at https://github.com/Jiamian- Wang/DITS-text-video-retrieval

Diffusion-Inspired Truncated Sampler for Text- Video Retrieval
Jiamian Wang"s i Pichao Dongfanng Liu', Qiang Guan 3 Sohail Dianat',
Majid Rabbani' Raghuveer Rao*, Zhiqiang Tao
"oohhsser Institute of Technology. 2 AAmazon,
3 Kent State University, *DECCMM Army Research Laboratory
Abstract
Prevalent text-to-video retrieval methods represent multimodal text-video data in joint embedding space, aiming at bridging the relevant text-video pairs and pulling away irrelevant ones. One main challenge state-of-the-an retrieval methods lies in the modality gap, which stems from the substantial disparities between text and video and can persist in the joint space. In this work, we leverage the potential of Diffusion models to address the text-video modality gap by progressively aligning text and video embeddings in a unified space. However, we identify two key limitations of existing Diffusion models in retrieval tasks: The L2 loss does not fit the ranking problem inherent in text-video retrieval, and the generation quality heavily depends on the varied initial point drawn from the isotropic Gaussian, causing inaccurate retrieval. To this end, we introduce new Diffusion-Inspired Truncated Sampler (DITS) that jointly performs progressive alignment and modality gap modeling in the joint embedding space. The key innovation of DITS is to leverage the inherent proximity of text and video embeddings, defining truncated diffusion flow from the fixed text embedding to the video embedding, enhancing controllability compared to adopting the isotropic Gaussian. Moreover, DITS adopts the contrastive loss to jointly consider the relevant and irrelevant pairs, not only facilitating alignment but also yielding discriminatively structured embedding. Experiments on five benchmark datasets suggest the state-of-the-ee performance of DITS. We empirically find that DITS can also improve the structure the CL.IP embedding space. Code is available at https://gitubb.co/Jjana Wang/DDTSSeextvideo--ettt
Introduction
Text-video retrieval aims match textual descriptions with relevant video content and vice versa, utilizing both modalities to improve ranking accuracy [Zhu et al., 2023]. This task is challenging due to the multi-modality gap, which represents the inherent differences between textual and visual data representations [Yang et al., 2021, Huang et al., 2023b, Sun et al., 2024, Wang et al., 2021]. Bridging this gap effectively requires advanced methodologies to extract and align semantic information from both modalities into a joint embedding space [Cheng et al., 2023, Xu et al., 2021, Pei et al., 2023].
Current kate-of-the-ae art approaches have explored various strategies to narrow this gap. Advanced feature extraction techniques focus on capturing fine- mneeggrannd details through temporal modeling [Liu al., 2022b] and muult-granulariig matching strategies, such frame-level, patch-level, werd-level, and token-level representations [Li et al., 2023b, Guan et al., 2023]. Additionally, leveraging large- scale data augmentation has been shown enhance the capabilities of visual-language models [Wu
'The work does not relate to author's position at Amazon. t Corresponding authors: Jiamian Wang (ju49050rit edu) & Zhiqiang Tao tehhiiang.taobrit.cco taoßrit.edu)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Relevant testivideo embedding
jAigned embeediing
Inrelevant embediing
sometry Gaussian an
Denoiseralgnment not Gap approximation
o
Diffusion
T,
Contrastive loss
8+8 1
ttt B
L2 loss
P
(a) Modality Gap
(b) Fixed prior
(c) Vanilla Diffusion Model
(d) Truncated sampling (Ours)
Figure 1: Multi-modality gap alignment for the text-video retrieval. (a) Considering the many-to-one correspondence of text-to-video, we define the modality gap stemming from the text (8 v t) for uniqueness. (b) Fixed prior alignment posits Gaussian distribution, which is inflexible. (c) Diffusion- based alignment upon L2 loss, which does not fit the retrieval task. The generation heavily affected by the random samples from (O. I). (d) The proposed diffusion-inspires truncated sampler (DITS) aligns from to v and gradually models the gap, guided by the contrastive loss.
et al., 2023, Wang et al., 2024b], while incorporating additional modalities like audio has also improved retrieval performance [lbrahimi et al., 2023]. Despite the progress, the modality gap (Fig. 1 (a)) remains a persistent issue, significantly impacting the performance of retrieval systems.
A critical observation in text-video retrieval is that existing methods often struggle with the variability and complexity of the semantic alignment between text and video. This work addresses this chal- lenge by investigating Diffusion models [Ho et al., 2019, Song et al., 2021] to bridge the modality truncated sampler (DITS) for
gap. Our main contribution is developing a new Diffusion-inspire models to iteratively align video and text embeddings in a progressive manner. Unlike traditional approaches [Qiu et al., 2024, Nukrai et al., 2022] that impose fixed priors for the alignment (Fig. 1 (b)). our method starts from the text embedding and employs truncated diffusion process to generate aligned video embeddings, addressing the inflexibility and inadequacy of fixed prior methods.
We first explore the generative capability of Diffusion models to mitigate the modality gap. Initially, we employ pre-trained text and video embeddings and model the gap using a diffusion process, where the initial state (isotropic Gaussian) represents fixed prior, progressively denoised to align the embedding (Fig. I (c)). However, we found that the vanilla Diffusion model's L2 loss does not the ranking problem inherent in text-video retrieval. The Le loss assumes Gaussian distribution and minimizes the mean squared error, treating all errors equally and failing to prioritize the alignment of semantically relevant pairs over irrelevant ones. Moreover, the reverse process introduces diversity through initial Gaussian noise sampling, which impacts the generation quality and causes the mis- alignment. Therefore, it non-trivial and necessary to tailor the Diffusion modeling from generation (where diversity is beneficial) to ranking (where precise alignment is crucial), catering to text-video retrieval. The focus of diffusion learning should be on closing relevant embeddings while pushing apart irrelevant pairs. This requires a different loss function and sampler for better alignment.
To address the above challenges, DITS leverages the inherent proximity of text and video embeddings in the joint space [Liang et al., 2022, Zhou et al., 2023] and incorporates a truncated diffusion process into the contrastive loss (depicted in Fig. (d)). Different from the vanilla diffusion- -based alignment, DITS adopts text embedding as a more meaningful intermediate latent state to initiate sampling, reducing the underlying variability and improving the alignment accuracy over Gaussian noises. The proposed DITS reformulates each diffusion step as modeling the modality gap and gradually controls the text-to-video alignment through the contrastive loss, acting on both aligned and video embeddings, which thus intrinsically promotes the alignment across timestamps in a more fine-grained manner. Empirical evidence shows that DITS can also enhance the structure of the CLIP embedding space. We summarize the contributions of this work as follows.
This work studies the Diffusion model to bridge the modality gap of text-video retrieval, identifying two key limitations of the vanilla Diffusion model. A new sampler, namely DITS, is also proposed enhance the multi- modality alignment and benefit the retrieval.
DITS enables a new truncated diffusion process to conduct the video-text alignment progres- sively, starting from the intermediate latent state given by text embedding to approaching the relevant video embedding. Governed by the contrastive loss, the proposed DITS gradually models the gap distribution along each timestamp by learning from relevant/ireelvant pairs.

---

Extensive experiments on five datasets (MSRVTT. LSMDC, DiDeMo, VATEX, and Cha- rades) suggest that DITS achieves ofate-of-the-art performance. Empirical evidence shows that DITS also improves the structure of the CLIP embedding space.
Related Work
Text-video Retrieval. State-of-the-ae text-video retrieval methods [Huang et al., 2023a, Gao et al., 2021, Yu et al., 2018, Deng et al., 2023, Jin et al., 2023, Croitoru et al., 2021] develop different strategies to facilitate the multi-modality alignment. Some adopt the temporal modeling [Li et al., 2023b, Liu al., 2023, Li et al., 2023c, Han et al., 2022b] on word sequence or video frames, better extracting the semantic clues and enhancing the retrieval. Others extract multi-modality features at different granularity [Liu et al., 2022b, Chen al., 2020b, Wang al., 2023, Ma al., 2022, Han et al., 2022a, Wang et al., 2021], uncovering hierarchical relationships between word sequences and video clips. Apart from the intricate model designs [Guan et al., 2023, Wu et al., 2021, Bain et al., 2021, Liu et al., 2021, Miech et al., 2021], incorporating the large-scale text-video data via augmentation emerges as promising approach [Luo et al., 2022, Wu et al., 2023, Falcon al., 2022], which effectively amplifies the potential of the visual-language foundation models [Chen et al., 2024, Ko et al.. 2023] and benefiting the retrieval. Additionally, harnessing new modality data, such as the audio [lbrahimi et al., 2023, Akbari et al., 2021, Lin et al., 2022, Miech et al., 2018, Liu et al., 2022a, Shvetsova al., 2022], paves a new direction to bridge the text-video modalities, with abundant semantics clues. Recent advances [Fang et al., 2023, Wang et al., 2024a, Gao et al., 2024, Ji et al., 2023, Chun et al., 2021, Nukrai et al., 2022] study different representation forms of the multi-modality data, among which T-MASS [Wang et al., 2024a] achieves the outstanding performance. Despite the prosperity, the modality gap persists the joint space. Distinguished from existing methods, this work studies how to learn the Diffusion model to model the modality gap in the joint embedding space, considering the substantial gap between the generation task and the ranking nature inherent to text-video retrieval.
Diffusion models. Diffusion models [Croitoru et al., 2023, Ho et al., 2019] define a transition from the isotropic Gaussian to the clean data with a learned diffusion process, emerging as powerful generative models for learning complex distributions. This approach has been extensively applied in image generation tasks (Yang et al., 2023], with models such as Denoising Diffusion Probabilistic Models (DDPM) [Ho et al., 2019], DDIM [Song et al., 2021], I]. and LDM [Rombach et al., 2022] demonstrating impressive capabilities in generating high-fidelity images. Building upon these foundations, recent research [Zhang et al., 2023] has extended Diffusion models to the multimodal domain, not only enabling the generation of diverse and realistic data with distinct type (such image, audio, etc.) [Yang et al., 2023], but also achieving the transition across varied modalities (e.g., text-to- image, text-to-video, etc.) [Ghosh et al., 2024]. Nevertheless, leveraging the Diffusion model to the ranking tasks such as the text-video retrieval non-trivial due to the significant difference in data, evaluation metrics, and the inherent challenges (e.g., diversity, coherence, and realism for generation v.s. relevance scores in retrieval). To our best knowledge, previous works of DiffusionRet [Jin et al., 2023] and MomentDiff [Li et al., 2023a] combines Diffusion model with retrieval. Albeit the encouraging performance, their method adopts the Diffusion model to learn a mapping from random noise to the signal (e.g., similarity score real moments) using the diffusion L, La loss, treating the ranking problem as a generation task without addressing the problem studied in this work.
3 Method
3.1 Preliminaries
Vide-Video Retrieval. We have multi-modality dataset consisting of N,, video clips and N, text descriptions, e.g., D (o(0), {G)), where 1,..., = 1..., N₂. Each video clip of0) may correspond to and text. The retrieval system adopts a multi-modality model, such as upon CLIP [Radford et al., 2021], to abstract the video and text features into a joint embedding space, Le.. t Rd, where d denotes the embedding dimension of the joint space.
more text samples {(), For simplicity, we let and to denote arbitrary video
== Fo(u); t= (1)
(I)
3

---

where (-) denotes the advanced visual encoder and (-)-) denotes the text encoder. Existing works
mainly adopt the symmetric-fone formed cross-entropy [Oord al., 2018] loss to train the model
B 2
ea(t()),
Lece = B log
Ej ga(t(0 + log Σ, (t
(2)
where the cosine similarity s(t, [[].v| is employed as distance measuring function and B denotes the batch size. A learnable temperature scaling factor 7 is employed control the smoothness of the loss. The first term in Eq. (2) guides the text-to-video (t + v) retrieval and the second term focuses on the video-to-text (v + retrieval. Notably, the scenario of Lece = 0 presents a perfect alignment for all video and text embedding in the joint space, where the relevant Y and I overlaps |y]
[ty]]
(s(t, v) = 1), and the irrelevant text-video orthogonal, i.e., s(t, v) O. While in practice, it is hard to achieve a perfect alignment considering the substantial disparities hetween the and v data. To this end, modality gap inevitably exists between the relevant text and video embedding as shown by Fig. 1 (a), which poses challenges in determining the relevancy. Nevertheless, how to model the modality gap 6 and perform alignment accordingly can benefit the retrieval but does not seem to have been fully explored. Note that there are two ways to define the ó by either stemming from or V. Considering the one-to-many correspondence of fideo-to-text, as exampled in Fig. (v(1) v.s. (1(), ((2) (,) each t associates a unique ö but corresponds to multiple 8 vectors. Accordingly, we consider 8 stermiing from and consistently adopt 5 = =V - in this work.
Fixed Prior Alignment. We first posit fixed prior for the multi-modality gap, similar to existing multi-modality aligmment practices [Qiu et al., 2024, Nukrai et al., 2022], and have
8 00,,
(3)
where o² determines the scale of the modality gap, i.e., to what extant v deviates from the relevant t. Given the arbitrary text embedding& t, we draw 8 from the fixed prior as shown above and impose onto the embedding as the alignment, e.g., t 8. Then this aligned embedding will take effect in the retrieval system as substitution of Unfortunately, there is a decrease in retrieval performances for variant priors (see Table 4). To elaborate, the fixed prior is far from enough to capture the delicate distribution of given the complicated structure of the CLIP embedding space [Paszke et al., 201 9I. Besides, the fixed prior is inflexible to adapt to varied text and video inputs, causing misalignment. These limitations naturally prompt us to resort to more advanced technologies for the gap modeling. As known, Diffusion models [Yang et al., 2023, Croitoru et al., 2023] approximate the sophisticated distributions without explicitly presuming a prior and has been proven to excel in bridging multi-modality representations [Zhou et al., 2023]. In this work, we investigate the potential of the Diffusion models for the gap modeling under text-video retrieval.
3.2 Proposed Diffusion-based Alignment
We define the target distribution of the Diffusion model upon the modality gap 8. Following the previous works [Gorti et al., 2022, Guan al., 2023], our proposed Diffusson model operates in the embedding space of the CLIP model and defines a Markov chain of the latent variables for 8, i.e., 81,... δy. where the initial state of the Diffusion model remains an isotropic Gaussian, i.e., 57 ~ ~NO. I), which in this case, can also be treated as a fixed prior for the modality gap 8, and do corresponds to the ground truth modality gap. is non-trivial incorporating the proposed Diffusion model into existing retrieval pipeline due to different natures of the generation and retrieval. To achieve this, we reimplement the denoising network, develop a pretraining stage and a reverse process-engaged retrieval pipeline.
Denoising Network Design. We first design a denoising network by referring to the prevalent DiT [Peebles and Xie, 2023]. The denoising network, denoted as e, (-). consists of N DiT blocks and let 7 denote the learnable parameters for the network. As shown in Fig. 2, the denoising network takes three inputs, including the timestamp t, the latent embedding 8, E Rdd and the condition e E Rd Since we define the modality gap stems from as in Section 3.1, we let the text embedding as the condition to guide the generation, i.e., t, which substitutes the vanilla classifier-fr free guidance [Ho and Salimans, 2021] in DIT. Finally, since we still perform the retrieval in the same latent space, there is no need to perform a reshape operation at the end of the denoising network. As shown in Fig. I (c), the denoising network (.). approximates the noise imposed on 5; during the diffusion process. To facilitate the learning of (-), we introduce pretraining stage follows.

---

clll beautiful sceneary is shown and the sight seeing through the train is lovely "The train is passing through forest" "Outiide the train window, there lakes, mountains, and a small town."
DiffusionInsppred Truncated Sampler (DITS)
amazing and
Timestepat
Madality Gap Madeling
(
P
5y(
"the girl shows the boys her medol'
(i) cartoon shows two dogs talking to
6 PP-
Σ Pmpiee
e "Two dogs feel confused in the carteon.""
Computa similarity or contrastive leerning
Video Encoder Fo())
Text Encoder Gp(-)
Embediing Space
Fe(:)
For C = T", 1.,,1o0 do
Embedding Space
Aligmmet network e
V/-1 (Ve- (x, + A₂z
DITS w/ Ly(
DIT
XM
Layer Nor
Rolevant textividoo embedding pairs
Modality gap distribution
Latent ambedding
Embed
Lincar
Allgned embedding
modules 2-N0 I olso -0
Figure 2: Overview of DITS. Given the video embedding v from Fo(:) and the text embedding t from (o(.), DITS performs T -steps sampling by starting from gradually approach v (as exampled by a relevant pair (1), v()) with the aligned embedding vy² at the timestamp t = O. Mcanwhile the progressive alignment, DITS intrinsically approximates the distribution of the modality gap 8 ~ Pt
at each timestamp t T', ..... 1. The contrastive loss is adopted to guide the alignment and modeling,
with parameters: θ {0, Φ. 7) We devise c, upon DiT [Peebles and Xie, 2023].
Diffusion model Pretraining. The target distribution of å can be highly adaptive to v and f given by Eq. (1). To stabilize the learning, we adopt the pretrained retrieval model and fix (- and G during the pretraining. The learning objective of (-) derives from the diffusion process [Qiu et al., 2024, Nukrai al., 2022], which gradually adds the Gaussian noise to óg for T steps, where 51 .... 57 are the latent variables of the same dimension, i.e., 8, E Rd
((61.T160) = II T q(6r)61-1), (66.16,11) = -666 V 1 1,6,-11, 8r1),
where $1,..,,/ fr can simply be held as constant [Ho et al., 2019] to form a variance schedule. By adopting the reparameterization [Kingma and Welling, 2013] upon Eq. (4), 5, can be analytically represented in a closed form
(6e (ōo) = =N(sr; Vārōo, (1 a.))), where at := 3,, ā.
II 0gg-
(5)
To approximate (6--1(66)) we use the variational lower bound to optimize the negative log- likelihood - log (ō0), which can be parameterized as L2 loss [Ho al., 2019]
Es llle e, (Vāāu + VI- L, c)1l3].
(6)
where e ~ (O, Uniform(1,.... .1,, T) and denotes the condition. Specifically, e, and c are three inputs of the denoising network e,, (.). .. By minimizing the L,, the Diffusion model learns to transition from the isotropic Gaussian to the target distribution of do through a reverse process, based on which this method aligns the input text embedding t (given as condition) as t+ do, to bridge its relevant counterpart v. We introduce the reverse process-engaged retrieval pipeline in the next.
Reverse Engage Retrieval. We perform the vanilla DDPM [Ho et al., 2019] sampling process in this work. Starting from the isotropic Gaussian dy (O, I), the reverse process samples the gap dy through a T-step Markov chain
1
1-04 a4
8
(b.
VT -1-ax (,,,t, o.z where z ~ NO,
(7)
Va.
where t decreases from T 1, Off 1, and of B,. The reverse process of the Diffusson
model will be plugged into the retrieval pipeline as the fest-feature extraction alignment, collectively forming a reverse process-engager retrieval pipeline. To this end, one can substitute t with the aligned embedding t + du and perform retrieval with regard to arbitrary v. Since the embedding space structure has been edited by the Diffusion model, might be better to further rearrange the multi-modality embeddings with the contrastive loss. We opt to jointly tune all of the learnable parameters θ = (0, 7, Φ) upon this reverse process-engaged retrieval pipeline with the symmetric cross-entropy given in Eq. (2). We provide more details and discussion in Section 4.1.
5

---

3.3 Proposed Alignment by DITS
To sum up. this work first implements a feasible approach to leverage the generative power of the Diffusion model in the retrieval pipeline. We firstly pretrain the Diffusion model with Eq. (6) and then incorporate the reverse process in Eq. (7) for the aligmment. Notably, we find that despite that the Diffusion model effectively bridging the relevant text and video embedding, either applying the pretrained model for the alignment or adopting the reverse process-engaged retrieval causes obvious performance declines (shown in Table 4). This prompt us to rethink Diffusion models in the context of our problem. Based on the designs in Section 3.2, we develop DITS in the following.
Rethinking the Vanilla Difflusion model. We find two-fold limitations of the Diffusion model defective for the task of re- trieval. First, the L2 loss (C-,) does not fit the ranking problem inherent text-video retrieval. On one hand, C, shown in Eq. (6) minimizes a mean squared eπror, which inherently lacks an prioritization effect for the relevant pairs over irrelevant ones. On the other, it is not clear what the desired alignment should be for the irrelevant pairs, making it hard to define a target distribution accordingly and implement L. for the modeling. Nevertheless, there is no guarantee that the L2 loss (e.g., L-) can effectively pull away irrelevant pairs, even though the gap mitigated for the relevant pairs as shown in Fig. 3. Second, the reverse process either fails to benefit the retrieval (Table 4) the initial starting point of the reverse, i.e., of öT ~ (0, can significantly impact the diversity and the generation quality [Zhou al., 2023, Chung et al., 2022, Zheng et al., 2023]. This can incur inaccurate alignment and retrieval. How to determine the 5r without affecting the performance remains a challenge.
wi Protralaed Doffucion Model
15
Count
10
1.10
1.15 1y-narm the Modality Gap
1.20
1.25
1:30
1.35
1.40
Figure 3: Modality gap distribution for the relevant pairs before and af- ter the Diffusion model alignment.
Diffusion-Inspired Truncated Sampler. Bearing the above findings, in this work, we introduce the diffusion-inspired truncated sampler (DITS) as effective alignment tool to benefit the retrieval. Considering the neighboring nature the CLIP embedding space - relevant v and have already been within arm's 's length of the distance) [Liang et al., 2022, Nukrai et al., 2022] -there is no need to start sampling from the isotropic Gaussian as in the vanilla Diffusion models. Instead, DITS proposes progressive alignment procedure that directly starts from the t to shown in Fig. 2 upper right, forming truncation. Given v by (-) and t by G-)-)
Vt-1 (v: (ve, t, c)) + Azz, For t == T',..,, 1,
(8)
where (.). serves as an alignment network and T' denotes the number of the truncated timestamps.
1-ay The starting point VT (taking (12) as an example in Fig. 2) and Xt- ya, Va,-a,a; denotes a
pre-defined schedule. Besides, we let z ~ NO if 1, else z = O. We set X, = dg Compared with vanilla sampling process given by Eq. (7), the initial latent of DITS becomes a text embedding point (e.g.. (() rather than a random sample from the isotropic Gaussian, which brings following advantages: (1) DITS gets rid of the effect of starting point random sampling. (2) the query text
(such as for + v) initializes the alignment in a more straightforward way, rather than solely being a
condition to guide the learning. Considering this, the condition с is not used in c.()) In addition, we adopt the contrastive loss upon Eq. (2) to guide the learning, during which we substitute with the aligned embedding Vo. We jointly train all of the learnable parameters θ {0, Φ. 7} as denoted in red in Fig. 2. By observing and considering both relevant and irrelevant pairs, DITS learns the alignment that potentially benefits the overall retrieval performance.
It turns out DITS intrinsically involves the modality gap modeling along the timestamps, (i.e., 5 Pt) as shown in Fig. 2. One can derive the distribution of the modality gap ó from Eq. (8), which can be represented by using the reparameterizatton
P = E, Xie, (Ve, (, + AzZ,
where (Ve, t, c) can be regarded as a bias of the modality gap and X controls the scale of the gap modeling at the timestamp I. Interestingly, DITS yields a remarkable performance boost over the baseline and achieves the state-of-the-an performance (Section 4.2). We also empirically find that DITS also serves as an effective tool to help improve the CLIP embedding space. See more details in Section 4.4.

---

Table I: Text-to-videe retrieval performance MSRVTT [Xu al., 2016] and LSMDC [Rohrbach et al., 2015]. Bold denotes the best performance. " denotes that the result unavailable.
MSRVTT Retrivaal
LSMDC Retrieval
Methods
R65 T
R
Mal
MnR
R&1
RWS
R
MaR 1 MnR
CLIP-NVT-B/32
X. Pool [Gorti et ol., 2022]
46.9
72.8
82.2
2.0
14.3
25.2
43.7
53.5
8.0
53.2
[L.iu et nl,, 2023]
46.9
23.7
ProST [Li 20230)
48.2
74.6
83.4
2.0
12.4
24.1
42.5
51.6
9.0
54.6
DiffusionRet [Jin et 2023]
49.0
75.2
82.7
2.0
12.1
24.4
43.1
54.3
40.7
UATVR [Fang 2023]
47.5
73.9
83.5
2.0
12.3
UCOFIA [Wang et 2023]
49.4
72.1
12.9
TEFAL [Ibrahimi
2023]
49.4
75.9
83.9
12.0
26.8
44.4
CLIP-ViP [Xue al., 2023]
50.1
74.8
84.6
1.0
25.6
45.3
54.4
8.0
T-MASS [Wang al., 2024a]
50.2
75.3
28.9
43.3
DITS (Ours)
51,9
75.7
84.6
1.0
11.6
28.2
47.3
56.6
6.0
43.7
CLIP-VIT-R/16
X-Pool [Gorti
2022]
48.2
26.1
46.8
47.3
UAIVR [Fang et al., 2023]
50.8
76.3
85.5
1.0
12.4
CLIP ViP [Nue
2023]
54.2
T-MASS [Wang ol,, 2024a]
52.7
77.1
85.6
1.0
10.5
30.3
52.2
61.3
5.0
40.1
DITS (Ours)
55.0
79.8
10.0
31.0
52.4
38.4
Table 2: Text-to-video retrieval performance on DiDeMo [Anne Hendricks et al., 2017] and VA- TEX [Wang et al., 2019]. Bold denotes the best performance. denotes that the result unavailable.
Methods
DiDeMo Retrieval
VATEX Retrivall
Rwi5
CLIP-ViT-B/32
X-Pool [Gorti al., 2023]
44.6
73.2
82.0
2.0
15.4
60.0
90.0
95.0
1.0
3.8
ProST at. 2023b]
44.9
72.7
82.7
2.0
13.7
60.6
90.5
95.4
3.4
STAN [Liu a02,
46.5
71.5
80.9
2.0
DiffusionRet [Jin 2023]
46.7
74.7
82.7
2.0
UATVR [Fang et allll,2.0000 2023]
43.1
71.8
82.3
2.0
15.1
61.3
91.0
95.6
1.0
3.3
UCOFIA [Wang et 2023]
46.5
74.8
13.4
61.1
90.5
CLIP-Vi [Xue al., 2023]
48.6
77.1
84.4
2.0
T-MASS [Wang et al., 20244]
50.9
77.2
85.3
1.0
12.1
63.0
92.3
96.4
1.0
3.2
DITS (Ours)
51.1
77.9
85.8
1.0
12.1
64.1
92.7
97.0
2.9
CI.P--VITB/16
[Gorti 2022]
47.3
74.8
2.0
62.6
UATVR [Fang et al., 2023]
45.8
73.7
83.3
2.0
13.5
64.5
92.6
96.8
1.0
2.8
CLIP-ViP [Xue al., 2023]
50.5
78.4
87.1
1.0
T-MASS [Wang al., 2024a]
53.3
80.1
87.7
65.6
DITS (Ouns
55.8
80,5
87.5
1.0
11.0
66.4
94.3
97.5
1.0
2.7
4 Experiment
4.1 Experimental Settings
Datasets. We employ five benchmark datasets for evaluation. Firstly, we utilize MSRVTT [Xu et al., 2016], comprising 10, 000 YouTube video clips (cach having 20 captions) and follow the IK-A testing split in Liu et al. [2019]. Secondly, LSMDC [Rohrbach et al., 2015] includes 118, 081 text-video pairs, providing videos with longer duration. The testing set contains 1000 videos, as per Gabeur al. [2020], Gorti et al. [2022]. Thirdly, DiDeMo [Anne Hendricks et al., 2017] contains ~ 40, 000 captions and ~ 10, 000 video clips. We adhere to the data splits detailed in Luo et al. [2022], Jin al. [2023]. Fourthly, VATEX [Wang et al., 2019] comprises 41, 250 video clips, where each is paired with ten English and ten Chinese descriptions. We follow the split in Chen et al. [2020a]. Lastly, Charades [Sigurdsson et al., 2016) contains 9848 video clips, each with multiple text descriptions detailing daily activities and actions. We adopt the split protocol of Lin et al. [2022].
Implementation Details. Following previous works [Gorti et al., 2022, Li al., 2023b], we resize the video to be the spatial size of 224 x 224 and uniformly sample 12 frames from the video for all datasets. For the retrieval model, adopt X-Pool [Gorti al., 2022] as the baseline. where both LITP-ViT/B33 and CLLI-VIT/B--I 16 are employed as the feature extractor. The dropout is set to 0.3. Different from DiT [Peebles and Xie, 2023], we set our denoising network (also used as the alignment network in DITS) with N = 4 blocks, with 16 heads and an MLP ratio of 4.0. We let the dimension 512 for the whole model. We find that a timestamp T alignment. For DITS, we set the truncated timestamp 7' = 5 for DiDeMo and T' 10 for others. A linear variance schedule with 3 0.1 and B = 0.99 adopted. All of the parameters trained with an AdamW [Loshchilov and Hutter, 2017] optimizer with weight decay of 0.2 and
is enough for diffusion-
(0, 7)

---

Table 3: Text-to-video comparisons on Charades [Sigurdsson al., 2016).
Bold denotes the best.
Methods
R@1
R@5
R@ 10
MdR
MnR
CLIP-Vi ViT ViT-B/32
ClipBERT [Lei ct al., 2021]
6.7
17.3
25.2
32,0
149.7
CLIP44Iip [[ et al., 2022]
9.9
27.1
36.8
21.0
85.4
X-Pool [Gorti et al., 2022]
11.2
28.3
38.8
20.0
82.7
T-MASS [Wang et 2024a]
14.2
36.2
48,3
12.0
54.8
DITS (Ours)
15.5
36.5
47.8
12.0
55.0
CLIP-ViT-B/16
CLIP4Clip [[uo et al., 2022]
16.0
38.2
48.5
12,0
54.1
X-Pool [Gorti et al., 2022]
20.7
42.5
53.5
9.0
47.4
T-MASS [Wang et al., 2024a]
26.7
51.7
63.9
5.0
30.0
DITS (Ours)
27.4
52.0
64.3
5.0
29.7
Table 4: Ablation study of the proposed DITS MSRVTT-IK. We adopt X-Pool [Gorti al., 2022] as the "Baseline". "Diffusion" denotes the Diffusion-based alignment method Section 3.2.
Methods
Alignment
L
Leco
Truncation
R@l I
R@5
R@
MdR
MnR
Baseline
X
X
/
X
46.9
72.8
82.2
2.0
14.3
0 = 1.0 a² 0.1
/
X
/
X
28.0
61.9
74.7
3.0
19.7
Fixed Prior
/
x
/
x
45.1
73.3
82.6
2.0
13.8
Pretrain
/
/
X
35.4
66.7
78.3
3.0
14.4
Diffusion
Fine tume
/
x
X
46.5
73.9
83.0
2.0
13.3
DITS
x
51.9
75.7
84.6
1.0
11.6
warmup rate of 0.1. We set the training epochs to 5 for all datasets and adopt the same seed of 24. We perform contrastive learning with batch size of B 32 for all datasets and backbones. Same as X-Pool [Gorti et al., 2022]. the learning rate of the CLIP model is initialized as 1 x 10-5, (0-5, The learning rate for non-CL.IP modules is 10-5 for MSRVTT [Xu al., 2016] and the other datasets. For a fair comparison, no post-processing techniques [Bogolin et al., 2022, Cheng et al., 2021] are employed in this work. We implement DITS with Py TyTorhh [Paszke et al., 2019] and perform experiments
10-5 5 for all
NVIDIA A100 GPU.
Compared Methods. We compare DITS with X-Pool [Gorti al., 2022], STAN [Liu al., 2023], ProST [Li et al., 2023b), DiffusionRet [Jin et al., 2023], UATVR [Fang et al., 2023], UCOFIA [Wang al., 2023], TEFAL [lbrahimi et al., 2023], CLIP-ViP [Xue et al., 2023] and T-MASS [Wang et al., 2024a]. Among them, TEFAL adopts additional audio data to train the model. CLIP-ViL ViP further adopts WebVid-2.5M [Bain et al., 2021] and LED-VIL--10NT [Xue et al., 2022] as the augmentation. Besides, some methods take larger batch sizes for the training (e.g.. 64 in DiffusionRet [Jin et al., 2023] and UATVR, 128 for ProST, STAN, and UCOFIA except DiDeMo). We perform evaluation using metrics such as Recall at ranks (1, 5, 10), Median Rank (MdR), and Mean Rank (MnR).
4.2 Performance Comparison
We compare the text-to-video retrieval performance of different methods in Table 1, 2, and 3. The proposed DITS can achieve the best performance on different CLIP backbones, datasets, and metrics. For example, on MSRVTT, DITS outperforms the baseline X-Pool by 5% R@ and outperforms T-MASS by 1.7%/2.39 at R @ 1 on different backbones. On DiDeMo, DITS outperforms CLIP- ViT by 2.2% with CLIP- VYT-B/32 and 5.3% with VTTB-VVT-B/6, achieving a consistent boost on different metrics. Moreover, on the challenging dataset of Charades, DITS also brings remarkable performance boost over X-Pool on both backbones, e.g., 4.3% and 6.7% at R@ I, respectively. There is one scenario that T-MASS enables better retrieval than DITS on LSMDC with CLIP-ViT-B/33 However, our method gains an advantage with a larger CLIP backbone on all datasets, indicating better scalability on the retrieval model. We also note that DITS excels in retrieval top-ranked results, such as on MSRVTT, DiDeMo a progressive alignment and gap modeling facilitates a narrower retrieval scope, encouraging retrieval precision. We provide in-depth analysis of this tendency in Section 4.4. Overall, extensive results on benchmark datasets demonstrate that DITS effectively aligns text-video embedding in the joint space, delivering promising retrieval results.
4.3 Ablation Study
Table provides an ablation study of the proposed method. Specifically, "Baseline" denotes the baseline method of X-Pool [Gorti al., 2022], based on which we provide three types of the
8

---

Table 5: Discussion DITS. Highlighted settings adopted for the benchmark comparison.
(a)
Discussion on the timestamps (T") MSRVTT. (b) The effect of DITS on CLIP. We study the DITS
alignment in the fixed CI.I space and upon learn-
T'
R@11
1
R@5T 1
R 1@10T
MdR
MnR
able CLIP model. By aligning the embedding, DITS
0
46.9
72.8
82.2
2.0
14.3
guides the CLIP learning and improves the space.
1
49.2
75.0
85.1
2.0
11.3
10
51.9
75.7
84.6
1.0
11.6
Methods
R@1
R@5
R@ 10
MdR
MnR
20
47.0
73.9
83.7
12.2
Baseline w/o DITS
46.9
72.8
82.2
2.0
14.3
30
32.5
63.5
76.0
3.0
16.2
DITS (Fix CLIP)
39.2
65.4
77.1
2.0
18.0
40
0.2
0.5
1.0
493.5
497.9
DITS (Joint train)
51.9
75.7
84.6
1,0
11.6
alignment methods, corresponding to Fig. 1. First, we show the naive modeling methods using
two different fixed priors, such as o² 0.1, 1.0. Both of them bring performance descent. Fixed priors are inflexible, hardly benefiting the retrieval expected. We notice that the sensitivity of the performance toward the prior. This also prompts us to leverage advanced methods of Diffusion model below. We provide the retrieval performance corresponding to both training stages of pretraining and fine-tuning as in Section 3.2. In pretraining, the reverse process is adopted for the generation. As shown, the pretraining with Le loss undermines the retrieval performance (e.g.. 35.4% Despite the fact that L2 loss effectively attracts relevant pairs (Fig. 3), it fails to align the irrelevant ones, leading to the performance descent. By comparison, when we fine-tune the model with the contrastive loss, we observe a remarkable boost of 11.1% 1 at R@ over pretraining and better results than baseline of X-Pool on metries except R @1 1. This indicates that such method, although enables a good retrieval scope (e.g., being favorable for R@ 5/10), but fails to performs accurate alignment and retrieval, underlying which, one main flaw lies in the random sampling in isotropic Gaussian. Drawing inspiration, the proposed DITS in the bottom line of Table 4 proposes a truncation process to alleviate the above concern, and adopts contrastive learning to guide the alignment. Encouragingly, we find an evident boost of implementing such a pipeline by following the variance schedule of the vanilla diffusion flow (e.g., at 3t). This prompt to study the behavior of DITS in the following.
46.9%).
4.4 Discussion on DITS
Discussion on Truncated timestamps. The number of the timestamps T" plays an important role for an accurate retrieval. We provide the performance of DITS at different timestamps in Table 5a. The 7" O denotes the baseline of X-Pool. As shown, the performance of DITS first increases and then drops when we gradually enlarging T" from to 40. DITS achieves the best performance at T 10. Specifically, one-step operation (T'= 1) enables determining rough retrieval scope, benefiting Recall over a larger range, e.g., R @ 10. Based on this, DITS requires more steps to narrow the scope for more accurate alignment and retrieval (c.g., T = 10). However, continually enlarging 7" enforces DITS to learn an extremely precise result, sacrificing the flexibility to accommodate the testing data and in the end, resulting the loss of the generalization ability.
DITS (Joins Train) DITS CLIP)
20-
Count
10
1.20 1.25
1.30
1.35 1.40 1.45
Ly-narm Modality Gap
Figure 4: DITS reduces the modal- ity gap for the relevant pairs, serv- ing as a tool to improve the space by guiding the CLIP's learning.
Improving the CLIP Embedding Space. We also study the effect of DITS to the CLIP in Table 5b. There is large performance gap when we fix CLIP or perform joint train for DITS. This indicates that solely performing the alignment in the original CLIP space is not enough. To this end, DITS not only plays the role of text-video embedding alignment, but also serves as a tool to guide the learning of CLIP, thus improving the CLIP space. As shown in Fig. 4, the distribution of the modality gap' between two spaces deviates from each other, characterizing the aligning effect of DITS.
Diffusion Model Conditions. We perform different conditions of the (-) in DITS, including with text condition t, with video condition v, and with both conditions and v, as well as without any conditions (see discussions in Section 3.3). As shown in Table 6a, we find that there is no need to introduce the conditions onto the network. The diffusion process for this task emphasizes "accurate alignment". unlike the general Diffusion models highlighting diversity in creative works. The previous intuition of the diffusion condition may not be applicable in this work: (1) Due to the one-to-many
Measured by L2-norm (non- boundodnd,d), corresponding to Euclidean distance. We also provide mean absolute distance version (using L1 Li-norm of the same data in Fig. 6).

---

Table 6: Discussion DITS. Highlighted settings adopted for the benchmark comparison.
(a) Different model conditions for DITS on MSRVTT. (b) Different types of the modality gap 8 on MSRVTT.
Condition
R@1
R@5
R@
MdR
MnR
8
Rwi
RossT
R9e10
MdR
Mnk
w/o alignment
46.9
72.8
82.2
2.0
14.3
aliymmeni
42.3
70.6
80.9
2.0
14.3
1=1-
39.2
65.4
77.1
2.0
18.0
23.1
51.9
65.9
5.0
24.8
v=v-t
75.5
12.2
71.3
80.8
2.0
w/o condition
50.1
75.5
84,9
1,0
12.2
mapping of video-to-text, one video can map to multiple gap vectors. Taking the video as a condition can guide the model learn the undesired modality gap, empirically dropping the performance. (2) DITS start from the text, ensuring the alignment starts from a semantically meaningful initial point. The text condition acts the constraint guideline for the learning. Simultaneously applying the text embedding as the starting point and the condition can cause conflicting instructions, empirically decreasing the performance.
(3) Joint usage of text and video conditions can inherent both limitations, empirically leading to an unsatisfactory performance.
Modality Gap Stemming from video embedding. We also provide experiments when let 5 == t-1 Table 6b. As shown, such setting brings sub-optimal performance. Imposing =t v to the works like approximating t for the retrieval, conducting the "text-to-text" retrieval, considering less semantic clues within the text, might be hard to retrieve accurately. By comparison, the proposed method DITS implements 5 = v t to the t works like approximating for the retrieval, performing the "video-to-video" retrieval. Since video contains much abundant clues, might be easier to get better performance. Interesting, we notice that the previous work of Cap4video [Wu al., 2023] also demonstrates similar tendencies Table 5, denoted as Seery-Caption/Viie Only case.
Conclusion
This work studied the task of text-video retrieval by proposing inffusion-Inspiree Truncated Sampler (DITS) for the multi-modalityy alignment. The primary contribution of this work was to offer insights on tailoring the Diffusion model for the ranking task inherent in retrieval. We uncovered two-fold limitations of the vanilla Diffusion models: the vanilla L2 loss was inadequate for alignment by overlooking the irrelevant pairs, the the random initial sampling in isotropic Gaussian introduces variability, causing misalignment. We proposed DITS, which leveraged the inherent proximity of text and video embedding and directly started sampling from text to alleviate the sampling variability, and proposed to adopt the contrastive learning not only to guide the iterative alignment steps over time, but also facilitate gap modeling. Extensive experiments demonstrated the state-of state-of-the-and --the-art performance of DITS. We found DITS could encourage an improved embedding space by guiding the CLIP's learning. We hope DITS can inspire future explorations in studying Diffusion models in the task of text-video retrieval.
6 Acknowledgment
This research work was supported by the DEVCOM Army Research Laboratory under contract W911QX-21-D-0001.
10

---

References
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang. Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In NeurlPS, 2021. 3
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell, Localizing moments in video with natural language. In ICCV, 2017.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 3,8
Simion-VIas Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanic. Cross modal retrieval with querybank normalisation. In CVPR, 2022. 8
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine- line-grained video-text retrieval with hierarchical graph reasoning. In CVPR, 2020a. 7
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine- line-grained video-text retrieval with hierarchical graph reasoning. In CVPR, 2020b. 3
Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, and Jing Liu. Cosa: Concatenated sample pretrained vision-language foundation model. In ICLR, 2024.
Feng Cheng, Xizi Wang. Jie Lei, David Crandall, Mohit Bansal, and Gedas Bertasius. Vindlu: A recipe for effective video-and-language pretraining. CVPR, 2023.
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. arXiv, 2021. 8
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus.
Probabilistic embeddings for cross-modal retrieval. In CVPR, 2021. 3
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come--llser-diifuse--ff conditional diffusion models for inverse problems through stochastic contraction. In CVPR, 2022.
Accelerating
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, 2023. 3,
Ioana Croitoru, Simion- Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In CVPR, 2021.
Chaorui Deng, Qi Chen, Pengda Qin, Da Chen, and Qi Wu. Prompt switch: Efficient clip adaptation for text-video retrieval. In ICCV, 2023. 3
Alex Falcon, Giuseppe Serra, and Oswald Lanz. feature-space multimodal data augmentation technique for text-video retrieval. In ACM MM, 2022. 3
Bo Fang, Chang Liu, Yu Zhou, Min Yang, Yuxin Song, Fu Li, Weiping Wang, Xiangyang Ji, Wanli Ouyang, et al. Uatvr: Uncertainty-adapiiee text-video retrieval. In ICCV, 2023. 3, 7,8
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, 2020.
Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. CLIP2TV: an empirical study on transformer-basee methods for video-text retrieval. CORR, 2021.
Zixian Gao, Xun Jiang, Xing Xu, Fumin Shen, Yujie Li, and Heng Tao Shen. Embracing unimodal aleatoric uncertainty for robust multimodal fusion. In CVPR, 2024. 3
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. In NewrIPS, 2024.

---

Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg. and Guangwei Yu. X-pool: Cross-modal language-video attention for text-video retrieval. In CVPR, 2022. 4, 7,
Peiyan Guan, Renjing Pei, Bin Shao, Jianzhuang Liu, Weimian Li, Jiaxi Gu, Hang Xu, Songeen Xu, Youliang Yan, and Edmund Lam. Pidro: Parallel isomeric attention with dynamic routing for text-video retrieval. In ICCV, 2023. 3,3,
Ning Han, Jingjing Chen, Hao Zhang, Huanwen Wang, and Hao Chen. Adversarial multi multi-grained em- bedding network for cross-modal text-video retrieval. ACM Trans. Multimedia Comput. Commun. Appl., 2022a.
Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long-term video. In CVPR, 2022b.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021. 4
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2019. 3,5
Huang, Y. Li, J. Feng, X. Wu, X. Sun, and R. Ji. Clover: Towards a unified video-language alignment and fusion model. In CVPR, 2023a. 3
Jingjia Huang, Yinan Li, Jiashi Feng, Xinglong Wu, Xiaoshuai Sun, and Rongrong Ji. Clover: Towards a unified video-language alignment and fusion model. In CVPR, 2023b.
Sarah Ibrahimi, Xiaohang Sun, Pichao Wang, Amanmeet Garg, Ashutosh Sanan, and Mohamed Omar. Audio-enhancee text-to-video retrieval using rext-conditioned feature alignment. In ICCV, 2023. 2, 3, 7,8 8
Yatai Ji, Junjie Wang. Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, and Yujiu Yang. Map: Multimodal uncertainty-aware vinion-language pre-training model. In CVPR, 2023. 3
Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, and Jie Chen. Diffusionret: Generative text-video retrieval with diffusion model. In ICCV, 2023. 3, 7, 8, 17
Diederik P Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
vrXiv:1312.00 14,61111400000 2013.
Dohwan Ko, Joonmyung Choi, Hyeong Kyu Choi, Kyoung- Woon On, Byungseok Roh, and Hyun- woo J Kim. Meltr: Meta loss transformer for learning to fine-tune video foundation models. In CVPR, 2023.
Kroung-Woon
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for
learning via sparse sampling. In CVPR, 2021. 8
Pandeng Li, Chen-Wa Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. In NeurlPS, 2023a.
Pandeng Li, Chen Wei Xie, Liming Zhao, Hongtao Xie, Jiannan Ge, Yun Zheng, Deli Zhao, and Yongdong Zhang. Progressive spatio-temporal prototype matching for text-video retrieval. In ICCV, 2023b. 1, 7,8
Yi Li, Kyle Min, Subarna Tripathi, and Nuno Vasconcelos. Svitt: Temporal learning of sparse video-text transformers. In CVPR, 2023c.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. In NewrIPS, 2022. 2, 6
Yan-Bo Lin, Lei, Mohit Bansal, and Gedas Bertasius. Eelipse: Efficient long-range video retrieval using sight and sound. In ECCV, 2022. 3,7
12

---

Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, and Thomas H Li. Revisiting temporal modeling for clip-based image-to-video knowledge transferring. In ICCV, 2023. 7,
Yang Liu, Samuel Albanie, Arsha Nagrami, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. In BMVC, 2019. 7
Yang Liu, Qingchao Chen, and Samuel Albanie. Adaptive cross-modal prototypes for cross-domain visual-language retrieval. In CVPR, 2021. 3
Yu Liu, Huai Chen, Lianghua Huang, Di Chen, Bin Wang, Pan Pan, and Lisheng Wang, Animating images to transfer clip for video-text retrieval. In ACM SIGIR, 2022a. 3
Yuqi Liu, Pengfei Xiong, Lubui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. In ECCV, 2022b. 1,3, 17
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017." 7
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. ClipAclip: An empirical study of clip for end to end video clip retrieval and captioning. In Neurocomputting, 2022. 7.8
Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to-end multi-grained contrastive learning for video-text retrieval. In ACM MM, 2022. 3
Antoine Miech, Ivan Laptev, and Josef Sivic. Learning a text-video embedding from incomplete and heterogeneous data. arXiv, 2018. 3
Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Thinking fast and slow: Efficient text-to-visual retrieval with transformers. CVPR, 2021. 3
David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected clip. In EMNLP, 2022. 3, 4, 6
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arX0v:1807.03748, 2018. 4
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019, 4, 8
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 4, 5, 7
Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songeen Xu, Peng Dai, Juwei Lu, and Youliang Yan. Clipping: Distilling clip-based models with a student base for video-language retrieval. In CVPR, 2023.
Longtian Qiu, Shan Ning, and Xuming He. Mining fine-grained image-text alignment for zero-shot captioning via text-only training. In AAAI, 2024. 2, 5,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. dataset for movie description. In CVPR, 2015. 7
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 3
Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio S Feris, David Harwath, James Glass, and Hilde Kuehne. Everything at sacc--ulti-moda. fusion transformer for video retrieval. In CVPR, 2022. 3
13

---

Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev. and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In ECCV, 2016. 7.8
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. 2021. 2,
Guohao Sun, Can Qin, Jiamian Wang, Zeyuan Chen, Ran Xu, and Zhiqiang Tao. Sq-llava: Self- questioning for large vision-language assistant. In ECCV, 2024. 1
Jiamian Wang, Guohao Sun, Pichao Wang. Dongfang Liu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Text is mass: Modeling as stochastic embedding for text-video retrieval. In CVPR, 2024a. 3, 8,8,11
Xiaohan Wang, Linchao Zhu, and Yi Yang. T2vlad: global-local sequence alignment for text-video retrieval. In CVPR, 2021. 1,3
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, 2019. 7
Yimu Wang, Shuai Yuan, Xiangru Jian, Wei Pang, Mushi Wang, and Ning Yu. Havtr: Improv- ing video-text retrieval through augmentation using large foundation models. arXiv preprint arXiv:2404.05082 2024b. 2
Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarse-to-fin alignment for video-text retrieval. In ICCV, 2023. 3, 7,
Peng Wu, Xiangteng He, Mingqian Tang, Yiliang Lv, and Jing Liu. Hanet: Hierarchical alignment networks for video-text retrieval. In ACM MM, 2021. 3
Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In CVPR, 2023. 1, 3, 10
Hu Xu, Gargi Ghosh, Pa-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vit: A large video description dataset for bridging video and language. In CVPR, 2016. 7.8
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing nigh-resolution video-language representation with large-scale video transcriptions. CVPR, 2022.
Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. In ICLR, 2023. 7,8
Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token-aware cascade contrastive learning for video-text alignment. In ICCV, 2021.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong. Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 2023. 3,4
Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question
answering and retrieval. In ECCV, 2018. 3
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang. and In So Kweon. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07900 2023. 3
Huangjie Zheng, Pengeheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion- based ad adversarial auto-encoders. In ICLR, 2023. 6
Yufan Zhou, Bingchen Liu, Yizhe Zhu, Xiao Yang. Changyou Chen, and Jinhui Xu. Shifted diffusion for text-to-ine eox-to-image generation. In CVPR, 2023. 2, 4, 6
Cunjuan Zhu, Qi Jia, Wei Chen, Yanming Guo, and Yu Liu. Deep learning for video-text retrieval: a review. International Journal of Multimedia Information Retrieval, 12(1):3, 2023.
14

---

A
Appendix supplemental material
We provide in-depth discussions, more results of the proposed DITS follows
Limitations discussion. (Section A.1).
Broader impacts on the proposed method. (Section A.2).
More discussions and results on Diffusion-based alignment (Section A.3)
More discussions on computational cost and efficiency. (Section A.4).
A.1 Limitations
Despite the advancements brought by the proposed Diffusion-Inspired Truncated Sampler (DITS), there might be some limitations can be considered for the future research.
DITS, like other multi molli-moddllty alignment methods using contrastive learning, requires redundant amounts of data to train effectively. In scenarios where labeled text-video pairs are scarce, the performance of DITS remains unexplored. This raises a question on studying DITS on challenging text-video retrieval scenarios or zero-shot, few-shot retrieval tasks.
In addition, the robustness of DITS to noisy or incomplete data may be potential concern. In real-world scenarios, text descriptions and video content can be noisy, incomplete, or of poor quality, While DITS aims bridge the modality gap, its performance may still degrade in the presence of such imperfect data. This highlights the need for further research into deploying DITS to real-worke retrieval systems with data imperfections.
A.2 Broader Impacts
The proposed Diffusion-Inspiree Truncated Sampler (DITS) presents a new alignment solution the realm of text-video retrieval. By effectively modeling and aligning the modality gap, DITS elevates the performance of retrieval systems and may contribute to the broader landscape of multi-modality learning and the diffusion models.
In essence, DITS offers a novel methodology for mitigating the disparities between textual and visual data, potentially inspiring diverse applications, for example, image captioning models that can accurately describe the contents of images, image question answering models that requires precise semantics understanding capabilities of the model toward the visual information.
Beyond its impact on the multi motaliyyy learning, DITS presents a new diffusion flow that is guided by the contrastive learning and considers nature of the ranking task. In light of this, DITS can pave the way for future innovations in the diffusion model design for learning the relationships of the multi-modality data. Besides, traditional diffusion models often struggle with maintaining alignment accuracy due to their reliance on fixed priors and isotropic Gaussian initial states. DITS introduces truncated diffusion process that shifts from Gaussian noise start a meaningful intermediate state, showcasing a new way to handle complex alignment tasks in diffusion models.
Despite the encouraging performance of the proposed method. There exists possibility that the proposed method can be utilized in the real- world applications with the illegitimate purpose.
Table Discussion on L1 Loss. We compare the retrieval performance with the pretrained Diffusion model as an alignment. We perform experiments on MSRVTT. All the other settings are kept the same for a fair comparison.
Methods
R@L
R@5
R@ 10
MdR
MnR
Baseline w/o alignment
46.9
72.8
82,2
2.0
14.3
Pretrained Diffusion w/ L
30.9
59.5
68.4
4.0
40.6
Pretrained Diffusion w/ Ca
35.4
66.7
78.3
3.0
14.4
A.3 More Details the Proposed Method
Discussion on J Loss. We perform experiments on different diffusion loss of L1 and L2 losses for the diffsuion-barse alignment pretrainig. As shown in Table 7, the La provides better performance.
15

---

Table 8: Discussion the timestamps (T") MSRVTT.
T
R@
1 f
R@51
e@ 1
MdR
MnR
0
46.9
72.8
82.2
2.0
14.3
1
49.2
75.0
85.1
2.0
11.3
5
49.6
75.1
84.9
2.0
11.2
10
51.9
75.7
84.6
1.0
11.6
15
51.1
76.1
85.8
1.0
11.8
20
47.0
73.9
83.7
2.0
12.2
30
32.5
63.5
76.0
3.0
16.2
40
0.2
0.5
1.0
493.5
497.9
Table 9: Modality gap (measured by normm,, similarity, and performance change discussion.
Methods
Averaged Modality Gap
Averaged Similanity (f) RTI
DITS fix CLIP
23.76
0.122
39.2
DITS (Ours)
18.13
0.168
$1.9
Le loss penalizes larger errors more heavily than smaller ones because the error term is squared. This can be beneficial in tasks where it is crucial reduce large deviations, as it forces the model to pay more attention to outliers or significant errors. By comparison, L2 loss treats all errors equally, which might not be ideal if large errors need be minimized more aggressively. In the context of our problem, L2 ensures that these outliers are brought into alignment, reducing their impact on the overall model performance, which might be reason for this observation.
More Discussions on Timestamp. In Table 8, we provide a more thorough comparison of different timestamps. As we gradually enlarge the total number timestamp T" from 40, all metrics will first get better, and then decrease. This is because DITS performs a progressive alignment. gradually narrows down the retrieval scope when we initially increase T", enabling generally improved performances (R@) 1, R 5, MdR). However, the model loses the generalization ability if we continually inlarge T", causing performance degradation (e.g., from 15 to 40). We also notice there are some inconsistent tendencies, such as @ 10 decreases from T 1 to T' = 15, contrary to the tendency of R @1 and R @@5 This because different recall precision (e.g.. 105,10) can be regarded as different retrieval tasks with different focus. When the model continually narrows down the retrieval scope for the given query, it can easily affect @@ 10 that requires larger scope. To balance between different Recall precision, we choose T' = 10 for MSRVTT. This experiment shows the potential of the progressive alignment in adjusting the retrieval scope. Note that devising an automated method for timestamp selection may improve performance. However, our experimental results suggest that setting the timestamp to a small range, c.g., ~ 15 yields reasonable performances and within this small range, general and consistent setting (I" 10) enables leading performances across diverse datasets, e.g, MSRVTT, LSMDC, Charades, and VATEX. Thus, we adopted the empirical timestamp setting to retain computational efficiency and method extensibility.
Effectiveness on Learning the Modality Gap In Table 9, we study the effect of DITS on CLIP embedding space, to uncover how DITS bridge the modality gap. We statistically compare DITS with a "DITS fix CLIP" baseline, terms of the averaged modality gap, averaged similarity, and the performance. Specifically, we compute the -oom of the modality gap (cosine similarity) for each relevant pair and compute the averaged value. Overall, DITS can bridge the modality gap by effectively aligning the CLIP embedding space, yielding better performance. In Fig. 5, we further provide the distribution of the similarity change. Note that Fig. 4 computes the L2-norm of the modality gap distribution, corresponding to the Euclidean distance. The values (such as mean around 1.28) are scaled by Lg-norm and are non-bounded. We also provide the mean absolute distance version (using -norm)) of the same data in Fig. 6 (left). As shown, L1 -orm enables a different scale and is also non- bounded. The difference between "Joint Train" and "Fix CLIP" is more remarkable under the L, -orm scale. We also provide the cosine similarities version of the same data in Fig. 6 (right). As shown, cosine similarities are bounded (i.e.) ). The similarity values are generally enlarged with joint train, being consistent with Fig. 4 and Fig. 6 (left).
A.4 Computational Cost and Efficiency
In this section, we discuss the computational efficiency for both training (Table 10) and inference (Table 11). The proposed DITS does not need much additional resources for training and reports comparable inference efficiency. We specifically compare with both non-diffusion methods (e.g.. X-
16

---

Table 10: Training resource usage comparison with MSRVTT dataset.
Methods
GPU Memory (MB)
GPU Request
Training Time (h)
X X-Pool [Gorti et al,, 2022]
18986
1x RTX3090
14.67
Diffusionket (Stage I)
27255
105.50
DiffusionRet (Stage 2)
8654
106.85
DiffusionRet (Total) [Jin et al., 2023]
27255
2x RTX3090
212.35
T-MASS [Wang et al., 2024a]
20390
1x RTX3090
14.74
DITS (Ours)
20950
1x RTX3090
14.90
Table 11: Inference time efficiency and GPU usage comparison on MSRVTT dataset.
Methods
GPU Memory (MB)
Inference Time (S)
R@j
X-Pool [Gorti et al., 2022]
5452
63.35
46.9
TS2-Net [Liu et ul., 20226]
2835
119.50
47.0
DiffusionRet [Jin et al., 20231
3375
64.23
49.0
T-MASS [Wang ct al., 2024a]
5452
76.41
50.2
DITS (Ours)
5464
70.17
51.9
Pool, TS2-Net, T-MASS) and diffusion-based method (e.g., DiffusionRet). All training and inference costs are measured with the same computational platform (2 NVIDIA RTX3090 GPU-24GB, Intel 19-10900X CPU).
For the training, since the proposed truncated sampler DITS starts from an pre-aligned text embedding and empirically needs small numbers of iterations, it requires much smaller training time compared with conventional diffusion model-based method. Besides, DITS enjoys comparable GPU memory usage, being easily deployed single GPU.
For inference, DITS requires comparable GPU memory usage and inference efficiency with previous methods. Since there is need to refer to the video embedding at each iteration of the alignment, we can compute and cache all the aligned embeddings beforehand. Thus we can skip the iterative sampling when performing retrieval. We find this method makes DITS faster than non-diffum con-diffusion methods such as TS2-Net and T-MASS.
17

---

Similarity Change
25
Similarity Change Max:0.1322
Count
20
10
-0.025
0,000 0.025
0.050 0.075 0.100
Distribution of the Similarity Change
Figure 5: Similarly value subtraction between DITS and "DITS fix CLIP" (relevant pairs). DITS enables higher similarity values, yielding positive-valued histogram shown above. By comparison, DITS can effectively bridge the modality gap by aligning the CLIP embedding space.
DITS (Jinnt Train) DITS (Fix CLIP)
20.0
DITS (Joint Train)
DITS (Fix CLIP)
0.8
17.5
15.0
Frequency
0.6
Count
12.5
10.0
0.4
5.0
2.5
0.0
18
20
26
0.0
0.00
0.05
0.10
0.15
0.20 0.25 0.30
Ly-norm of the Modality Gap
Distribution of the cosine similarity
Figure 6: DITS reduce the modality gap for the relevant pairs. Left: The modality gap comparison upon Li-morm (Mean absolute distance, unbounded). Right: The modality gap comparison measured by the cosine similarity (bounded, i.e., [0, 1)).
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of the work performed by the authors in the supplementary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of the paper in the supplementary. We provided the code in the supplementary and claimed to further release the code and pretrained models in the abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The manuscript and the supplementary provides detailed information in reproduce the results. We provided the code in the supplementary and claimed to further release the code and pretrained models in the abstract.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public/uuides/// Sttpeee//ooeettiiiiiiooooool.o.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cctpublic/guii httpa//gguiieee/eeeuuuiiiieeeee wwwesSubmisiiooPoo...
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The manuscript and the supplementary provides detailed information about the experimental setting/details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification: Error bar is not applicable to this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources has been reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics ptttts://neurippi cc hetoolii//tiissuiiideeoo
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "GYSgsvoDkl", "reviewer": "Reviewer_CwRd", "summary": "The paper introduces a novel method to address the challenge of bridging the modality gap between text and video data in retrieval tasks. The authors propose the Diffusion-Inspired Truncated Sampler (DITS), leveraging diffusion models to model the text-video modality gap.\nDITS performs progressive alignment and modality gap modeling, starting from text embeddings and using a truncated diffusion process to generate aligned video embeddings.", "questions": "1. The author claims that the vanilla Diffusion model's L2 loss does not fit the ranking problem in text-video retrieval. However, in the method part (Line 183), the author still uses the conventional L2 loss of diffusion model. I want to know the detail of that how the author addresses the problem of L2 loss.\n\n2. As far as I know, it is unrealistic to directly use the diffusion model to learn the modality gap or the joint distribution probability of cross-modal alignment. For example, after I ran through the code of [1], I found that its main contribution is not in the diffusion model. Like [2], using the diffusion model to solve the impact of time distribution on moment retrieval is a reasonable and effective motivation. Therefore, the author needs to provide more quantitative experiments to prove that the proposed diffusion model can learn the accurate modality gap, such as similarity change or other metrics.\n\n3. The paper could provide more details on the computational efficiency of the proposed method, including runtime and resource usage, which are important considerations for practical deployment. According to my experience of using diffusion models to solve retrieval tasks, the diffusion module will seriously slow down the retrieval speed, and has limited improvement in optimizing alignment quality and improving retrieval accuracy. The experimental results of this paper also show that the proposed method has limited improvement over the SOTA method. The author should provide experimental data on time efficiency. I think this improvement is not enough compared to the obvious lack of time efficiency.\n\n4. The author assumes that the modality gap is Gaussian distributed. Is there any corresponding proof or basis for this assumption?\n\n[1] Diffusionret: Generative text-video retrieval with diffusion model. In ICCV, 2023.\n[2] MomentDiff: Generative Video Moment Retrieval from Random to Real", "limitations": "See Weaknesses part.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel method, DITS, which leverages diffusion models to address the modality gap in text-video retrieval. The authors have identified and addressed the limitations of existing diffusion models when applied to ranking tasks, which is a significant contribution to the field.\n\n2. The authors have provided a theoretical foundation for their method, including a discussion on the limitations of L2 loss and the benefits of truncated diffusion processes.", "weaknesses": "1. The author claims that the vanilla Diffusion model's L2 loss does not fit the ranking problem in text-video retrieval. However, in the method part (Line 183), the author still uses the conventional L2 loss of diffusion model. I want to know the detail of that how the author addresses the problem of L2 loss.\n\n2. As far as I know, it is unrealistic to directly use the diffusion model to learn the modality gap or the joint distribution probability of cross-modal alignment. For example, after I ran through the code of [1], I found that its main contribution is not in the diffusion model. Like [2], using the diffusion model to solve the impact of time distribution on moment retrieval is a reasonable and effective motivation. Therefore, the author needs to provide more quantitative experiments to prove that the proposed diffusion model can learn the accurate modality gap, such as similarity change or other metrics.\n\n3. The paper could provide more details on the computational efficiency of the proposed method, including runtime and resource usage, which are important considerations for practical deployment. According to my experience of using diffusion models to solve retrieval tasks, the diffusion module will seriously slow down the retrieval speed, and has limited improvement in optimizing alignment quality and improving retrieval accuracy. The experimental results of this paper also show that the proposed method has limited improvement over the SOTA method. The author should provide experimental data on time efficiency. I think this improvement is not enough compared to the obvious lack of time efficiency.\n\n4. The author assumes that the modality gap is Gaussian distributed. Is there any corresponding proof or basis for this assumption?\n\n[1] Diffusionret: Generative text-video retrieval with diffusion model. In ICCV, 2023.\n[2] MomentDiff: Generative Video Moment Retrieval from Random to Real", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that Reviewer `CwRd` provides valuable comments and finds the proposed method novel. Since the weakness and questions are the same, we answer the weakness below. Due to the limited rebuttal space, we will not be able to copy all questions but summarize the weakness statements below.   \n\n**`R1.1`**: The author claims that the vanilla Diffusion model's L2 loss does not fit the ranking problem in text-video retrieval. However, in the method part (Line 183), the author still uses the conventional L2 loss of diffusion model. I want to know the detail of that how the author addresses the problem of L2 loss.\n\n**`A1.1`**: We use the contrastive loss ($\\mathcal{L}\\_\\text{sce}$ in Eq.2)  but not $\\mathcal{L}\\_2$ loss to train DITS (see L239). The method part (L183, Section 3.2) is a baseline method, not our full model DITS. The statement of “conventional $\\mathcal{L}\\_2$ loss does not fit the ranking problem in text-video retrieval” is based on this baseline that uses the $\\mathcal{L}\\_2$ loss of the diffusion model. To elaborate, $\\mathcal{L}\\_2$ loss only minimizes the modality gap between the relevant pairs (as shown in Fig.3 in the manuscript), failing to handle irrelevant pairs. To address the $\\mathcal{L}\\_2$ loss issue, we adopt the contrastive loss $\\mathcal{L}\\_{sce}$ to incorporate with a proposed truncated sampler. The resulting method DITS models the gap in an iterative manner and achieves promising retrieval performance.\n\n**`R1.2`**: Provide more quantitative experiments to prove that the proposed diffusion model can learn the accurate modality gap, such as similarity change or other metrics.\n\n| Methods | Averaged modality gap ($\\downarrow$) | Averaged Similarity ($\\uparrow$) | R@1 ($\\uparrow$) | \n|----------|----------|----------|----------|\n| DITS fix CLIP | 23.76 | 0.122 | 39.2 | \n| DITS (Ours) | **18.13** | **0.168** | **51.9** | \n\n**Table T4. Modality gap (measured by $L\\_1$-norm), similarity, and performance change discussion on DITS.** \n\n**`A1.2`**: Previous methods [DiffusionRet (ICCV 2023), MomentDiff (NeurIPS 2023)]  learn a mapping from random noise to the signal (e.g., real moments) using the diffusion  $\\mathcal{L}\\_1$  loss. Differently, DITS adopts the contrastive loss $\\mathcal{L}\\_\\text{sce}$ instead of $\\mathcal{L}\\_1$ (or $\\mathcal{L}\\_2$) loss. DITS starts from the text embedding with the proposed truncated sampler instead of from the random noise. These differences enable DITS to iteratively learn the modality gap and distinguish from previous diffusion-based designs.  We will include and discuss MomentDiff [2] in the related work. \n\nIn `Table T4` above,  we study the effect of DITS on CLIP embedding space, to uncover how DITS bridge the modality gap. We statistically compare DITS with a ``DITS fix CLIP'' baseline, in terms of the averaged modality gap, averaged similarity, and the performance. Specifically, we compute the  $L\\_1$-norm of the modality gap (cosine similarity) for each relevant pair and compute the averaged value. Overall, DITS can bridge the modality gap by effectively aligning the CLIP embedding space, yielding better performance. In `Fig.F1` in the attached PDF, we further provide the distribution of the similarity change per reviewer's request.  \n\n**`R1.3`**: More details on the computational efficiency including runtime and resource usage is expected. \n\n**`A1.3`**: Thanks for the valuable comment. We provide a thorough discussion on the computational efficiency comparison for both training (`Table T1`) and inference (`Table T2`) in the `Global Rebuttal`. The proposed DITS does not need much additional resources for training (e.g., DITS: 14.90h *v.s.* DiffusionRet: 212.35h) and reports comparable inference efficiency (e.g., DITS: 70.17s *v.s.* TS2-Net: 119.50s). For the performance comparison, state-of-the-art methods can use larger settings than DITS (L283\\~286, e.g., larger batch sizes such as 64, 128) to improve their performance. By comparison, DITS adopts smaller batch size (i.e., 32, L274) and achieves a remarkable boost, especially compared with DiffusionRet (+2.9% at R@1 on MSRVTT, +4.4% at R@1 on DiDeMo). We will add the discussions on computational resources and efficiency into the final version. \n\n**`R1.4`**:  The author assumes that the modality gap is Gaussian distributed. Is there any corresponding proof or basis for this assumption?\n\n**`A1.4`**: Thanks for the valuable comment. We posit Gaussian distribution based on its mathematical properties and previous works in multimodal learning [R1\\~R6]. Gaussian distributions offer (a) parametric convenience -- capturing the central tendency (mean) and the variability (variance) of the modality gap and (b) mathematical tractability -- linear transformation and additivity, ensuring the modality gap retains the Gaussian after the processing pipeline.  Previous works benefit from positing the multimodality gap as the Gaussian, such as (1) quantifying the multimodal uncertainty [R1, R3, R4, R6], (2) enriching the text embedding with flexible and resilient semantics [R2], and (3) enabling data augmentation via Gaussian noise injection [R5]. We also empirically observe the Gaussian-like multimodality gap in our experiment (e.g., Fig. 2, 4).  We will discuss the references in the related work.   \n\n[R1] Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion. In CVPR 2024.\n\n[R2] Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval. In CVPR 2024.\n\n[R3] Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations. In CVPR 2023.\n\n[R4] MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model. In CVPR 2023.\n\n[R5] Text-Only Training for Image Captioning using Noise-Injected CLIP. In EMNLP 2022. \n\n[R6] Probabilistic Embeddings for Cross-Modal Retrieval. In CVPR 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsxbQDzA6j", "reviewer": "Reviewer_fBRP", "summary": "The paper tackles the task of text-video retrieval. It aims to address the modality gap between text and video that usually stems in state-of-the-art models. In order to do this, it leverages Diffusion models. So, it introduces DITS that jointly performs progressive alignment and modality gap modeling in the joint embedding space in order to improve the performance. Finally, the authors test the performance of their method on five benchmarks.", "questions": "Is there any additional cost in terms of running time for the proposed method as opposed to other sota methods?", "limitations": "The limitations are discussed in the appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper tackles an important task and achieves good results. I find the idea interesting and the paper is fairly well written.", "weaknesses": "While I think that the paper is fairly well written, some parts can be a bit confusing at the first read, though they become clear if you read twice. For example, abstract lines 6-8 first states that Diffusion is used to mitigate the problem, but then there is immediately the claim that there are \"flaws\" in diffusion. So, I think a rephrasing and introducing a bit later that changes are needed to diffusion in order to make that work would be better.\n\nline 145 \"Unfortuntely, we find descent retrieval performances\" -> \"Unfortunately, there is a decrease in retrieval performance\"\n\nThe main concern that I have is related to understanding the limitations. I think there should have been a section that discusses the need for additional computational resources, if applicable and how does the running time compare agains non diffusion methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that Reviewer `fBRP` provides valuable comments and finds the idea is novel and interesting, with good results. \n\n**`R2.1`**: While I think that the paper is fairly well written, some parts can be a bit confusing at the first read, though they become clear if you read twice. For example, abstract lines 6-8 first states that Diffusion is used to mitigate the problem, but then there is immediately the claim that there are \"flaws\" in diffusion. So, I think a rephrasing and introducing a bit later that changes are needed to diffusion in order to make that work would be better.\n\n**`A2.1`**: Thanks for the reviewer’s commitment in helping improve the manuscript. We rephrase the abstract lines according to the suggestion as follows: ``In this work, we leverage the potential of Diffusion models to address the text-video modality gap by progressively aligning text and video embeddings in a unified space. However, we identify two key limitations of existing Diffusion models in retrieval tasks.’’. We will accordingly revise the introduction to ensure better clarity and coherence. We are committed to making further modifications if any parts of the manuscript remain unclear.\n\n**`R2.2`**: line 145 \"Unfortunately, we find descent retrieval performances\" -> \"Unfortunately, there is a decrease in retrieval performance\"\n\n**`A2.2`**: Thanks for the detailed proofreading. We will change this sentence in the manuscript as suggested by the reviewer. \n\n**`R2.3`**: The main concern that I have is related to understanding the limitations. I think there should have been a section that discusses the need for additional computational resources, if applicable and how does the running time compare against non diffusion methods.\n\n**`A2.3`**: Thanks for the valuable suggestion. We provide a thorough discussion on the computational efficiency including runtime, resource usage for both training (`Table T1`) and inference (`Table T2`) in the `Global Rebuttal`. Overall, DITS does not need much additional resources for training and reports comparable efficiency especially with non-diffusion methods. \n\n(1) As DITS starts from the text embedding pre-aligned by CLIP and empirically needs small numbers of iterations,  it does not need much additional cost for the training. Specifically, DITS requires comparable GPU usage and time usage compared with non-diffusion methods, including X-Pool, TS2-Net, and T-MASS. (2) For inference, DITS requires comparable GPU memory usage and inference runtime -- as there is no need to refer to the video embedding at each iteration of the alignment, we can compute and cache all the aligned embeddings beforehand and skip the iterative sampling when performing retrieval. We find this implementation strategy makes DITS faster than non-diffusion methods such as TS2-Net (DITS: 70.17s *v.s.* TS2-Net: 119.50s) and T-MASS (DITS: 70.17s *v.s.* T-MASS: 76.41s). We will add the above discussions into the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CIZZC6DCql", "reviewer": "Reviewer_QeV9", "summary": "The authors introduce Diffusion-Inspired Truncated Sampler  (DITS) that jointly performs progressive alignment and modality gap modeling in the joint embedding space.  Experiments on five benchmark datasets suggest the state19 of-the-art performance of DITS.", "questions": "1. In Table 4, do Diffusion and Fine tune parts not need L2 loss? Why does Pretrain have L2 loss, while Fine tune does not need L2 loss and the result is high?\n2. In Table 4, does the last row DITS not need L2 loss?", "limitations": "See the weaknesses section", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is clearly described and easy to understand.\n2. This work studies the Diffusion model to bridge the modality gap of text-video retrieval, identifying two key limitations of the vanilla Diffusion model.\n3. Extensive experiments on five datasets (MSRVTT, LSMDC, DiDeMo, VATEX, and Cha76 rades) suggest that DITS achieves state-of-the-art performance.", "weaknesses": "1. In Table 1, under the CLIP-ViT-B/32 feature extractor, the author's method has limited performance improvement compared to the comparison method.\n2. In Formula 2, these two terms are added, but why are these two the same? They should be different terms added.\n3. What does it mean to multiply ϵ, t, c in the norm of Formula 6? This is not standard.\n4. The writing of the methods section needs further standardization and improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that Reviewer `QeV9` finds the motivation is clear and easy to follow, and the proposed method achieves the state-of-the-art performance. \n\n**`R3.1`**: In Table 1, under the CLIP-ViT-B/32 feature extractor, the author's method has limited performance improvement compared to the comparison method.\n\n**`A3.1`**: (1) For the performance comparison, other methods can use larger settings than DITS (L283\\~286), e.g., larger batch sizes such as 64 or 128 to help improve their performance. Due to limited computational resources, DITS adopts batch size of 32 \n (L274) and achieves a remarkable boost, especially compared with the previous diffusion-based method, DiffusionRet (+2.9\\% at R@1 on MSRVTT, +4.4\\% at R@1 on DiDeMo). (2) DITS enables better scalability. The boost over T-MASS is enhanced (MSRVTT: 2.3\\% at R@1, LSMDC: 0.7\\% at R@1) when changing CLIP-ViT-B/32 to CLIP-ViT-B/16. \n \n**`R3.2`**: In Formula 2, these two terms are added, but why are these two the same? They should be different terms added.\n\n**`A3.2`**: The two terms in the Eq.2 are different. We follow the standard notations of symmetric-formed cross entropy loss (specifically, InfoNCE loss [R1]) that is used in text-video retrieval domain (including both text-to-video and video-to-text) [R2, R3, R4, etc.]. To clarify, we directly copied it from the manuscript and highlight the difference between two terms below:\n\n$\\mathcal{L}\\_{\\texttt{sce}} = - \\frac{1}{B}\\sum\\limits^{B}\\_{i=1}\\log\\frac{e^{s(\\mathbf{t}^{(i)}, \\mathbf{v}^{(i)})\\cdot \\tau}}{\\sum\\nolimits\\_{\\textcolor{red}{j}}e^{s(\\mathbf{t}^{(i)}, \\mathbf{\\textcolor{red}{v}}^{\\textcolor{red}{(j)}})\\cdot \\tau}} + \\log\\frac{e^{s(\\mathbf{t}^{(i)}, \\mathbf{v}^{(i)})\\cdot \\tau}}{\\sum\\nolimits\\_{\\textcolor{red}{j}}e^{s(\\mathbf{\\textcolor{red}{t}}^{\\textcolor{red}{(j)}}, \\mathbf{v}^{(i)})\\cdot \\tau}}$\n\n* The first term makes a summation over $v^{(j)}$ in the denominator, representing the sum of similarities between the i-th query text $\\mathbf{t}^{(i)}$ and all key videos $\\mathbf{v}^{(j)}$\n* The second term makes a summation over $t^{(j)}$ in the denominator, representing the sum of similarities between the i-th query video $\\mathbf{v}^{(i)}$ and all texts $\\mathbf{t}^{(j)}$. \n\n We will put more descriptions into the manuscript and are willing to illustrate any point that remains unclear to the reviewer. \n\n[R1] Representation Learning with Contrastive Predictive Coding. \n\n[R2]  X-pool: Cross-modal language-video attention for text-video retrieval. In CVPR 2022.\n\n[R3] Unified Coarse-to-Fine Alignment for Video-Text Retrieval. In ICCV 2023. \n\n[R4] Text is mass: Modeling as stochastic embedding for text-video retrieval. In CVPR 2024.\n\n**`R3.3`**: What does it mean to multiply ϵ, t, c in the norm of Formula 6? This is not standard.\n\n**`A3.3`**: We kindly remind the reviewer that $\\textcolor{red}{\\epsilon}$, $\\textcolor{red}{t}$, and $\\textcolor{red}{c}$ are not multiplied in Eq.6. $\\textcolor{red}{\\epsilon}$, $\\textcolor{red}{t}$, and $\\textcolor{red}{c}$ are three inputs of the denoising network $\\textcolor{blue}{\\epsilon\\_\\gamma(\\cdot)}$ (L165, L172). As we directly copied from the manuscript and highlight below,\n\n$\\mathcal{L}\\_\\gamma = \\mathbb{E}\\_{\\mathbf{\\delta}\\_0, t, \\epsilon} [||\\epsilon - \\textcolor{blue}{\\epsilon\\_\\gamma(}\\sqrt{\\bar{\\alpha}}\\mathbf{\\delta}\\_0 + \\sqrt{1-\\bar{\\alpha}\\_t}\\textcolor{red}{\\epsilon}, \\textcolor{red}{t}, \\textbf{\\textcolor{red}{c}}\\textcolor{blue}{)}||^2]$\n\nTo clarify, (1) using $\\epsilon$ to denote the noise (or the corresponding denoising network), $t$ to denote the timestamp, and $c$ to denote the condition are standard representation forms in Diffusion model literatures [R5, R6, R7]. We will add more descriptions toward Eq.6 in the manuscript. We are willing to provide more illustrations to help address further concerns of the reviewer. \n\n[R5] Denoising diffusion probabilistic models. In NeurIPS 2020.\n\n[R6] Adding conditional control to text-to-image diffusion models.\" In ICCV 2023.\n\n[R7] Noise2Music: Text-conditioned Music Generation with Diffusion Models. Google Research.\n\n**`R3.4`**: The writing of the methods section needs further standardization and improvement.\n\n**`A3.4`**: We appreciate the reviewer’s commitment in helping improve the manuscript. We will consider all comments of the reviewer and carefully revise the method section accordingly. \n\n**`R3.5`**: In Table 4, do Diffusion and Fine tune parts not need L2 loss? Why does Pretrain have L2 loss, while Fine tune does not need L2 loss and the result is high? In Table 4, does the last row DITS not need L2 loss? \n\n**`A3.5`**: Thanks for the valuable comment. “Diffusion pretrain” adopts conventional $L\\_2$ loss. “Diffusion Fine tune” adopts the contrastive loss $L\\_{sce}$ (L315) instead of $L\\_2$ loss.  We make it more clear  about the loss for different baselines in Table 4 in the manuscript, as copied in `Table T3` in the attached rebuttal PDF. We provide comments below to help the reviewer better locate the key points concerning the ablation study. \n\n* **Diffusion Pretrain**: We firstly study the effectiveness of vanilla Diffusion model with $L\\_2$ loss in text-video retrieval, identifying the limitation of $L\\_2$ loss (abstract L8\\~L10, introduction L48\\~L51, method L210\\~L218, and experiment L312\\~L314). \n* **Diffusion Fine tune**: Based on the above observation, we fine tune the Diffusion model with contrastive loss ($L\\_{sce}$ in  Eq.2), obtaining a performance boost (L315). Contrastive loss jointly considers relevant and irrelevant pairs, calibrating the embedding that is misaligned by the $L\\_2$ loss.  \n* **DITS**: Based on the above observation, DITS adopts contrastive loss and does not need $L\\_2$ loss (abstract: L16, introduction L62\\~L64, method L238\\~L240). \n\nWe are willing to provide more illustrations in solving the reviewer’s further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HxxPj7R0Ux", "reviewer": "Reviewer_UKTF", "summary": "This paper smartly leverage the diffusion model to solve a famous modality gap problem in the a CLIP retrieval problem. The proposed method called DITS set the text embedding as the initial point in the 1D diffusion model and try to generate the video embedding. Finally, by using the contrastive loss to align the generated video embedding and ground truth video embedding to finish the training of the diffusion model. The work show convincible result comparing to existing SOTA methods and the ablation study in Table 4 and 5 show the effectiveness of the method comparing to raw CLIP and existing work DiffusionRet.", "questions": "check the weakness", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The idea to leverage the diffusion model to solve modality gap is highly novel.\nIt is a nice viewpoint to the famous modality gap problem.\nAfter this paper, the problem will highly to be reexamined in the community. \n1. The gap should be some intrinsic problem in the multi-modality model training. \n2. With this paper, the gap is kind of filled and most of old retrieval methods and work will be re-actived in this research area.", "weaknesses": "The figure is too small to read.(miner issue)\nFigure 4, it looks like the gap is still large as the mean absolute distance is ~1.28. why it is not some small value likes 0.5?\nIs there any investigation on cases, which video retrieval cases got improved and which video cases become worse?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that Reviewer `UKTF` provides valuable comments and finds the idea is novel, showing effectiveness in  filling the gap. \n\n**`R4.1`**: The figure is too small to read.(miner issue) Figure 4, it looks like the gap is still large as the mean absolute distance is ~1.28. why it is not some small value likes 0.5? \n\n**`A4.1`**: Thanks for the valuable comment. We will revise the layout of the figure to make it more clear. In the manuscript, Fig. 4 computes the $L\\_2$-norm of the modality gap distribution, corresponding to the Euclidean distance. The values (such as mean around 1.28) are scaled by $L\\_2$ norm and are non-bounded. \n\nWe also provide the mean absolute distance version (using $L\\_1$-norm) of the same data in `Fig.F2` (left) in the attached rebuttal PDF.  As shown, $L\\_1$-norm enables a different scale and is also non-bounded. The difference between “Joint Train” and “Fix CLIP” is more remarkable under the $L\\_1$-norm scale. \n\nWe also provide the cosine similarities version of the same data in `Fig. F2` (right) in the attached rebuttal PDF. As shown, cosine similarities are bounded (i.e., $[0,1]$). The similarity values are generally enlarged with joint train, being consistent with Fig. 4 and `Fig. F2` (left). We will put above evidence and analysis into the manuscript and provide more illustrations to the values in Fig. 4. \n\n**`R4.2`**: Is there any investigation on cases, which video retrieval cases got improved and which video cases became worse?\n\n**`A4.2`**: Thanks for the valuable comment. We provide investigations and potential intuitions on both retrieval cases when comparing DITS with the raw CLIP baseline in Fig. 4. (1) Since DITS performs the iterative alignment, it can better avoid drastic changes and provide more chances to adjust the misalignment step. We find when the text and video data is more challenging (such as vague texts or blurry videos), DITS can identify the relevant pairs and get the performance improved. (2) Conversely, on the simpler cases where text is rich and informative, or the video is temporally consistent, we did not see a large difference between DITS and the raw CLIP baseline. However, we observe when the caption data is incorrectly annotated, DITS can have a chance to miss the relevant video. We will add the above discussions into the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f7QvjLaW6r", "reviewer": "Reviewer_vaB9", "summary": "The paper introduces a new method, Diffusion-Inspired Truncated Sampler (DITS), designed for text-video retrieval tasks. It addresses the primary challenge of bridging the modality gap between text and video data, a problem that existing retrieval methods often fail to solve effectively. The authors propose DITS to harness the strengths of diffusion models in reducing this gap and enhancing the alignment between text and video embeddings within a joint embedding space.", "questions": "* The authors would do well to make a comparison of inference times. \n* Since the timestamps for the truncated diffusion process have a huge impact on performance, could an automated method be devised to determine time steps?\n* The performance of unconditional diffusion surpasses that of text or video diffusion, which is counter-intuitive. Could the authors provide further analysis to explain this phenomenon?", "limitations": "The limitations of the method have been discussed in detail.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The authors investigate the use of Diffusion models for addressing the text-video modality gap and identify limitations in applying standard Diffusion models to retrieval tasks. Specifically, they highlight that the L2 loss is not well-suited for the ranking problems inherent in retrieval, and that there is a dependency on varied initial points from an isotropic Gaussian distribution.\n* Extensive experiments on five benchmark datasets demonstrate DITS's state-of-the-art performance. The method shows flexibility in adjusting the retrieval scope over time and improves the structure of the CLIP embedding space.\n* The authors have committed to releasing the code, which will facilitate further research and application of the proposed method.", "weaknesses": "* Generalization ability. The authors highlight the importance of selecting an optimal number of timestamps for the truncated diffusion process. If the number of timestamps is too large, the model might lose its generalization ability.\n* Computational efficiency. Computational efficiency is a common concern with Diffusion models, which typically require substantial computational resources for training and inference.\n* There are some confusions in the writing. In Section 3.2, it is mentioned that the condition for diffusion is the textual feature. However, in Section 3.3, the condition is not specified. Additionally, in Appendix A.3, the authors state that an empty condition works best.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that Reviewer `vaB9` provides valuable comments and finds the proposed method shows state-of-the-art  performance. We are committed to release the training and inference code, as well as the pretrained models. \n\n**`R5.1`**: Generalization ability. The authors highlight the importance of selecting an optimal number of timestamps for the truncated diffusion process. If the number of timestamps is too large, the model might lose its generalization ability.\n\n**`A5.1`**: Thanks for the valuable comment. Since CLIP embedding space offers a good proximity between text and video (L57\\~L58) as shown in prior works [R1, R2], the timestamp is not supposed to be too large. We also empirically find that DITS \n achieves good performance with a small timestamp, e.g., $T’=10$ on MSRVTT, LSMDC, Charades, and VATEX (L270). Both the previous study [R1, R2] and empirical results suggest that DITS retains a good generalization ability using a relatively small and consistent timestamp setting, being free from the elaborate tuning. \n\n[R1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. In NeurIPS 2022.\n\n[R2] Shifted diffusion for text-to-image generation. In CVPR 2023.\n\n**`R5.2`**: Computational efficiency. Computational efficiency is a common concern with Diffusion models, which typically require substantial computational resources for training and inference.\n\n**`A5.2`**: Thanks for the valuable suggestion. We provide a thorough discussion on the computational efficiency including runtime, resource usage for both training (`Table T1`) and inference (`Table T2`) in the `Global Rebuttal`. Overall, DITS requires comparable GPU resources and time for training (e.g., DITS: 14.90h v.s. DiffusionRet: 212.35h). DITS also reports comparable inference efficiency, such as compared to previous methods of TS2-Net (DITS: 70.17s v.s. TS2-Net: 119.50s) and T-MASS (DITS: 70.17s v.s. T-MASS: 76.41s). \n\n**`R5.3`**: There are some confusions in the writing. In Section 3.2, it is mentioned that the condition for diffusion is the textual feature. However, in Section 3.3, the condition is not specified. Additionally, in Appendix A.3, the authors state that an empty condition works best.\n\n**`A5.3`**: Thanks for the detailed proofreading. For the Diffusion model baseline (Section 3.2), the condition is the textual feature. For the proposed DITS (Section 3.3, Appendix A.3), no condition is needed (empty condition). To clarify, the conventional Diffusion model (Section 3.2, proposed baseline) has to adopt the text condition to guide the learning. Differently, based on the experiment in Appendix A.3, we find an empty condition for DITS works best, thus does not specify condition for DITS in Section 3.3. Notably,  DITS and conventional Diffusion models adopt the same text embedding in different ways, DITS  takes the text embedding as the starting point, and the diffusion model takes text embedding as the condition. We will make it more clear by pointing out in Section 3.3 that the condition is not used in DITS. \n\n**`R5.4`**: Since the timestamps for the truncated diffusion process have a huge impact on performance, could an automated method be devised to determine time steps?\n\n**`A5.4`**: Thanks for the valuable comment. Devising an automated method for timestamp selection can be an attractive direction and may improve performance. However, our experimental results suggest that setting the timestamp to a small range, e.g., $1\\sim15$ yields reasonable performances (Table 7) and within this small range, a general and consistent setting ($T’=10$) enables leading performances across diverse datasets, e,g, MSRVTT, LSMDC, Charades, and VATEX (L270). Thus, we adopted the empirical timestamp setting to retain computational efficiency and method expansibility.   \n\nWe could provide a potential solution with a bi-level optimization framework. The upper-level can optimize the timestamp hyperparameter and the lower-level can take the timestamp to guide the iterative alignment. One can potentially perform an alternating training strategy with contrastive loss. However, a series of challenges can persist: timestamps may affect the iterative diffusion process non-linearly, introducing the local minima or saddle points. The search space for timestamps can be high-dimensional, considering the broad context on which the timestamp operates. Evaluating different timestamps can be computationally intensive. \n\nOverall, DITS provides a new viewpoint and foundation for future research and we will continue exploring per reviewer’s inspiration. We will add this in the manuscript. \n\n**`R5.5`**: The performance of unconditional diffusion surpasses that of text or video diffusion, which is counter-intuitive. Could the authors provide further analysis to explain this phenomenon?\n\n**`A5.5`**:  Thanks for the valuable suggestion. The diffusion process for this task emphasizes ``accurate alignment’’, unlike the general Diffusion models highlighting diversity in creative works. The previous intuition of the diffusion condition may not be applicable in this work. We provide further analysis to elaborate this phenomenon:\n* Due to the one-to-many mapping of video-to-text, one video can map to multiple gap vectors. Taking the video as a condition can guide the model learn the undesired modality gap (L599\\~L601), empirically dropping the performance. \n* DITS start from the text, ensuring the alignment starts from a semantically meaningful initial point. The text condition acts as the constraint or guideline for the learning. Simultaneously applying the text embedding as the starting point and the condition can cause conflicting instructions, empirically decreasing the performance. \n* Joint use of text and video conditions can inherent both limitations, empirically leading to an unsatisfactory performance. \n\nWe will incorporate the above analysis in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal 

We would like to express our sincere gratitude to all reviewers for the time and effort in reviewing our manuscript. We much appreciate  that reviewers find the proposed method novel and achieve good results.   We are committed to release the training and inference code, as well as the pretrained models. We will address each reviewer's concern in each rebuttal. Due to the limited space, we provide extra results below. 

* We provide an overall discussion on the computational cost and efficiency below.
* We provide `Figure F1` in the attached PDF to address the concern of the similarity change for Reviewer `CwRd`.
* We provide `Table T3` in the attached PDF to make it clear for Table 4 for Reviewer `QeV9`. 
* We provide `Figure F2` in the attached PDF to address the concern of Fig. 4 for Reviewer `UKTF`.

**Discussion on the Computational Cost and Efficiency**

We discuss the computational efficiency for both training (`Table T1`) and inference (`Table T2`) to solve reviewers’ concerns.  The proposed DITS does not need much additional resources for training and reports comparable inference efficiency. We specifically compare with both non-diffusion methods (e.g., X-Pool, TS2-Net, T-MASS) and diffusion-based method (e.g., DiffusionRet). All training and inference costs are measured with the same computational platform (2$\times$NVIDIA RTX3090 GPU-24GB, Intel i9-10900X CPU). We will add these discussions into the final version.

| Methods | GPU memory (MB)    | GPU Request    | Training Time (h)    |
|----------|-------------|-------------|-------------|
|X-Pool | 18986          | 1 x RTX3090           | 14.67           |
|DiffusionRet (Stage 1) | 27255           | --           | 105.50          |
|DiffusionRet (Stage 2) | 8654           | --           | 106.85           |
|DiffusionRet (Total) | 27255           | 2 x RTX3090           | 212.35           |
|T-MASS| 20390           | 1 x RTX3090           | 14.74           |
|DITS (Ours)| 20950           | 1 x RTX3090           | 14.90           |

**Table T1. Training resource usage comparison with  on MSRVTT dataset.**

| Methods | GPU memory (MB) | Inference Time (s) | R@1 |
|----------|----------|----------|----------|
| X-Pool (CVPR 2022)       | 5452        | 65.35        | 46.9        |
| TS2-Net (ECCV 2022)       | 2835        | 119.50        | 47.0        |
| DiffusionRet (ICCV 2023)       | 3375        | 64.23        | 49.0        |
| T-MASS (CVPR 2024)        | 5452        | 76.41        | 50.2        |
| DITS (Ours)        | 5464        | 70.17        | 51.9        |

**Table T2. Inference time efficiency and GPU usage comparison on MSRVTT dataset.**

For the training, since the proposed truncated sampler DITS (1) starts from an pre-aligned text embedding and (2) empirically needs small numbers of iterations (L270, $T’=10$ on four benchmark datasets),  it requires much smaller training time compared with conventional diffusion model-based method. For example, DiffusionRet requires in total over one week to finish the two stage training despite having a large batch size (i.e., 128 in DiffusionRet paper). Besides, DITS enjoys a comparable GPU memory usage, being easily deployed on a single GPU. 

For inference, DITS requires comparable GPU memory usage and inference efficiency with previous methods. Since there is no need to refer to the video embedding at each iteration of the alignment, we can compute and cache all the aligned embeddings beforehand. Thus we can skip the iterative sampling when performing retrieval. We find this method makes DITS faster than non-diffusion methods such as TS2-Net and T-MASS.

Meta-review: This paper received mixed reviews, with three strongly positive reviews (scores 6,7,7) and two mildly negative ones (scores 4,4). 

The paper introduces a novel method to address the challenge of bridging the modality gap between text and video data in retrieval tasks. The authors propose the Diffusion-Inspired Truncated Sampler (DITS), leveraging diffusion models to model the text-video modality gap. DITS performs progressive alignment and modality gap modeling, starting from text embeddings and using a truncated diffusion process to generate aligned video embeddings.

The reviewers agreed that the problem is important, the solution innovative, the theoretical motivation solid, and that the method achieved good results. The main concerns were computational efficiency, some writing issues, and various questions of detail. The authors provided a rebuttal that address these issues. Most of the positive reviewers confirmed that the rebuttal solved all their concerns. Unfortunately, the negative reviewers chose not to participate in the post review phase. This, the fact that many of the concerns they raised seem more like requests for clarification that real negative flags, and that the explanations provided by the authors seem sensible, have led the AC to discount the negative scores. 

As a note to the authors. I would say that it is not OK to accuse the reviewers of bias just because they did not review your work positively. These type of remarks do not help your cause. If the reviewer did not fully understood your paper, it is likely because the paper could better written. In fact, other reviewers have mentioned that the paper could be more clear. In the future, please do not engage in this type of behavior again.
Decision: Accept (poster)