Diffusion-Inspired Truncated Sampler for Text-Video Retrieval

Authors: jiamian  wang
pichao  wang
dongfang  liu
qiang  guan
sohail  dianat
majid  rabbani
raghuveer  rao
zhiqiang  tao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 02:21:03

Abstract: Prevalent text-to-video retrieval methods represent multimodal text-video data in a joint embedding space, aiming at bridging the relevant text-video pairs and pulling away irrelevant ones. One main challenge in state-of-the-art retrieval methods lies in the modality gap, which stems from the substantial disparities between text and video and can persist in the joint space. In this work, we leverage the potential of Diffusion models to address the text-video modality gap by progressively aligning text and video embeddings in a unified space. However, we identify two key limitations of existing Diffusion models in retrieval tasks: The L2 loss does not fit the ranking problem inherent in text-video retrieval, and the generation quality heavily depends on the varied initial point drawn from the isotropic Gaussian, causing inaccurate retrieval. To this end, we introduce a new Diffusion-Inspired Truncated Sampler (DITS) that jointly performs progressive alignment and modality gap modeling in the joint embedding space. The key innovation of DITS is to leverage the inherent proximity of text and video embeddings, defining a truncated diffusion flow from the fixed text embedding to the video embedding, enhancing controllability compared to adopting the isotropic Gaussian. Moreover, DITS adopts the contrastive loss to jointly consider the relevant and irrelevant pairs, not only facilitating alignment but also yielding a discriminatively structured embedding. Experiments on five benchmark datasets suggest the state-of-the-art performance of DITS. We empirically find that DITS can also improve the structure of the CLIP embedding space. Code is available at https://github.com/Jiamian- Wang/DITS-text-video-retrieval

Main paper: 
 introduction: 
 Text-video retrieval aims to match textual descriptions with relevant video content and vice versa, utilizing both modalities to improve ranking accuracy [Zhu et al., 2023] . This task is challenging due to the multi-modality gap, which represents the inherent differences between textual and visual data representations [Yang et al., 2021 , Huang et al., 2023b , Sun et al., 2024 , Wang et al., 2021] . Bridging this gap effectively requires advanced methodologies to extract and align semantic information from both modalities into a joint embedding space [Cheng et al., 2023 , Xu et al., 2021 , Pei et al., 2023] .Current state-of-the-art approaches have explored various strategies to narrow this gap. Advanced feature extraction techniques focus on capturing fine-grained details through temporal modeling [Liu et al., 2022b] and multi-granularity matching strategies, such as frame-level, patch-level, word-level, and token-level representations [Li et al., 2023b , Guan et al., 2023] . Additionally, leveraging largescale data augmentation has been shown to enhance the capabilities of visual-language models [Wu et al., 2023 et al., , Wang et al., 2024b]] , while incorporating additional modalities like audio has also improved retrieval performance [Ibrahimi et al., 2023] . Despite the progress, the modality gap (Fig. 1 (a)) remains a persistent issue, significantly impacting the performance of retrieval systems.A critical observation in text-video retrieval is that existing methods often struggle with the variability and complexity of the semantic alignment between text and video. This work addresses this challenge by investigating Diffusion models [Ho et al., 2019 , Song et al., 2021] to bridge the modality gap. Our main contribution is developing a new Diffusion-inspired truncated sampler (DITS) for models to iteratively align video and text embeddings in a progressive manner. Unlike traditional approaches [Qiu et al., 2024 , Nukrai et al., 2022] that impose fixed priors for the alignment (Fig. 1 (b)), our method starts from the text embedding and employs a truncated diffusion process to generate aligned video embeddings, addressing the inflexibility and inadequacy of fixed prior methods.We first explore the generative capability of Diffusion models to mitigate the modality gap. Initially, we employ pre-trained text and video embeddings and model the gap using a diffusion process, where the initial state (isotropic Gaussian) represents a fixed prior, progressively denoised to align the embedding (Fig. 1 (c) ). However, we found that the vanilla Diffusion model's L 2 loss does not fit the ranking problem inherent in text-video retrieval. The L 2 loss assumes a Gaussian distribution and minimizes the mean squared error, treating all errors equally and failing to prioritize the alignment of semantically relevant pairs over irrelevant ones. Moreover, the reverse process introduces diversity through initial Gaussian noise sampling, which impacts the generation quality and causes the misalignment. Therefore, it is non-trivial and necessary to tailor the Diffusion modeling from generation (where diversity is beneficial) to ranking (where precise alignment is crucial), catering to text-video retrieval. The focus of diffusion learning should be on closing relevant embeddings while pushing apart irrelevant pairs. This requires a different loss function and sampler for better alignment.To address the above challenges, DITS leverages the inherent proximity of text and video embeddings in the joint space [Liang et al., 2022 , Zhou et al., 2023] and incorporates a truncated diffusion process into the contrastive loss (depicted in Fig. 1 (d) ). Different from the vanilla diffusion-based alignment, DITS adopts text embedding as a more meaningful intermediate latent state to initiate sampling, reducing the underlying variability and improving the alignment accuracy over Gaussian noises. The proposed DITS reformulates each diffusion step as modeling the modality gap and gradually controls the text-to-video alignment through the contrastive loss, acting on both aligned and video embeddings, which thus intrinsically promotes the alignment across timestamps in a more fine-grained manner.Empirical evidence shows that DITS can also enhance the structure of the CLIP embedding space.We summarize the contributions of this work as follows.• This work studies the Diffusion model to bridge the modality gap of text-video retrieval, identifying two key limitations of the vanilla Diffusion model. A new sampler, namely DITS, is also proposed to enhance the multi-modality alignment and benefit the retrieval.• DITS enables a new truncated diffusion process to conduct the video-text alignment progressively, starting from the intermediate latent state given by text embedding to approaching the relevant video embedding. Governed by the contrastive loss, the proposed DITS gradually models the gap distribution along each timestamp by learning from relevant/irrelevant pairs.• Extensive experiments on five datasets (MSRVTT, LSMDC, DiDeMo, VATEX, and Charades) suggest that DITS achieves state-of-the-art performance. Empirical evidence shows that DITS also improves the structure of the CLIP embedding space.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Text-video Retrieval. State-of-the-art text-video retrieval methods [Huang et al., 2023a , Gao et al., 2021 , Yu et al., 2018 , Deng et al., 2023 , Jin et al., 2023 , Croitoru et al., 2021] develop different strategies to facilitate the multi-modality alignment. Some adopt the temporal modeling [Li et al., 2023b , Liu et al., 2023 , Li et al., 2023c , Han et al., 2022b] on word sequence or video frames, better extracting the semantic clues and enhancing the retrieval. Others extract multi-modality features at different granularity [Liu et al., 2022b , Chen et al., 2020b , Wang et al., 2023 , Ma et al., 2022 , Han et al., 2022a , Wang et al., 2021] , uncovering hierarchical relationships between word sequences and video clips. Apart from the intricate model designs [Guan et al., 2023 , Wu et al., 2021 , Bain et al., 2021 , Liu et al., 2021 , Miech et al., 2021] , incorporating the large-scale text-video data via augmentation emerges as a promising approach [Luo et al., 2022 , Wu et al., 2023 , Falcon et al., 2022] , which effectively amplifies the potential of the visual-language foundation models [Chen et al., 2024 , Ko et al., 2023] and benefiting the retrieval. Additionally, harnessing new modality data, such as the audio [Ibrahimi et al., 2023 , Akbari et al., 2021 , Lin et al., 2022 , Miech et al., 2018 , Liu et al., 2022a , Shvetsova et al., 2022] , paves a new direction to bridge the text-video modalities, with abundant semantics clues. Recent advances [Fang et al., 2023 , Wang et al., 2024a , Gao et al., 2024 , Ji et al., 2023 , Chun et al., 2021 , Nukrai et al., 2022] study different representation forms of the multi-modality data, among which T-MASS [Wang et al., 2024a] achieves the outstanding performance. Despite the prosperity, the modality gap persists in the joint space. Distinguished from existing methods, this work studies how to learn the Diffusion model to model the modality gap in the joint embedding space, considering the substantial gap between the generation task and the ranking nature inherent to text-video retrieval.Diffusion models. Diffusion models [Croitoru et al., 2023 , Ho et al., 2019] define a transition from the isotropic Gaussian to the clean data with a learned diffusion process, emerging as powerful generative models for learning complex distributions. This approach has been extensively applied in image generation tasks [Yang et al., 2023] , with models such as Denoising Diffusion Probabilistic Models (DDPM) [Ho et al., 2019] , DDIM [Song et al., 2021] , and LDM [Rombach et al., 2022] demonstrating impressive capabilities in generating high-fidelity images. Building upon these foundations, recent research [Zhang et al., 2023] has extended Diffusion models to the multimodal domain, not only enabling the generation of diverse and realistic data with distinct type (such as image, audio, etc.) [Yang et al., 2023] , but also achieving the transition across varied modalities (e.g., text-toimage, text-to-video, etc.) [Ghosh et al., 2024] . Nevertheless, leveraging the Diffusion model to the ranking tasks such as the text-video retrieval is non-trivial due to the significant difference in data, evaluation metrics, and the inherent challenges (e.g., diversity, coherence, and realism for generation v.s. relevance scores in retrieval). To our best knowledge, previous works of DiffusionRet [Jin et al., 2023] and MomentDiff [Li et al., 2023a ] combines Diffusion model with retrieval. Albeit the encouraging performance, their method adopts the Diffusion model to learn a mapping from random noise to the signal (e.g., similarity score or real moments) using the diffusion L 1 or L 2 loss, treating the ranking problem as a generation task without addressing the problem studied in this work.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discussed the limitations of the work performed by the authors in the supplementary.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Text-Video Retrieval. We have a multi-modality dataset consisting of N v video clips and N t text descriptions, e.g., D = {v (i) , t (j) }, where i = 1, ..., N v , j = 1, ..., N t . Each video clip v (i) may correspond to one or more text samples t (j) . For simplicity, we let v and t to denote arbitrary video and text. The retrieval system adopts a multi-modality model, such as upon CLIP [Radford et al., 2021] , to abstract the video and text features into a joint embedding space, i.e., v, t ∈ R d , where d denotes the embedding dimension of the joint space.EQUATIONwhere F θ (•) denotes the advanced visual encoder and G ϕ (•) denotes the text encoder. Existing works mainly adopt the symmetric-formed cross-entropy [Oord et al., 2018] loss to train the modelEQUATIONwhere the cosine similarity s(t, v) = |t•v| |t|•|v| is employed as a distance measuring function and B denotes the batch size. A learnable temperature scaling factor τ is employed to control the smoothness of the loss. The first term in Eq. ( 2) guides the text-to-video (t → v) retrieval and the second term focuses on the video-to-text (v → t) retrieval. Notably, the scenario of L sce = 0 presents a perfect alignment for all video and text embedding in the joint space, where the relevant v |v| and t |t| overlaps (s(t, v) = 1), and the irrelevant text-video are orthogonal, i.e., s(t, v) = 0. While in practice, it is hard to achieve a perfect alignment considering the substantial disparities between the t and v data. To this end, a modality gap δ inevitably exists between the relevant text and video embedding as shown by Fig. 1 (a), which poses challenges in determining the relevancy. Nevertheless, how to model the modality gap δ and perform alignment accordingly can benefit the retrieval but does not seem to have been fully explored. Note that there are two ways to define the δ by either stemming from t or v. Considering the one-to-many correspondence of video-to-text, as exampled in Fig. 2 (v (1) v.s. {t (1) , t (2) , t (3) }), each t associates to a unique δ but v corresponds to multiple δ vectors. Accordingly, we consider δ stemming from t and consistently adopt δ = vt in this work.Fixed Prior Alignment. We first posit a fixed prior for the multi-modality gap, similar to existing multi-modality alignment practices [Qiu et al., 2024 , Nukrai et al., 2022] , and haveEQUATIONwhere σ 2 determines the scale of the modality gap, i.e., to what extant v deviates from the relevant t.Given the arbitrary text embedding t, we draw δ from the fixed prior as shown above and impose it onto the embedding as the alignment, e.g., t + δ. Then this aligned embedding will take effect in the retrieval system as a substitution of t. Unfortunately, there is a decrease in retrieval performances for variant priors (see Table 4 ). To elaborate, the fixed prior is far from enough to capture the delicate distribution of δ given the complicated structure of the CLIP embedding space [Paszke et al., 2019] . Besides, the fixed prior is inflexible to adapt to varied text and video inputs, causing misalignment. These limitations naturally prompt us to resort to more advanced technologies for the gap modeling. As known, Diffusion models [Yang et al., 2023 , Croitoru et al., 2023] approximate the sophisticated distributions without explicitly presuming a prior and has been proven to excel in bridging multi-modality representations [Zhou et al., 2023] . In this work, we investigate the potential of the Diffusion models for the gap modeling under text-video retrieval.We define the target distribution of the Diffusion model upon the modality gap δ. Following the previous works [Gorti et al., 2022 , Guan et al., 2023] , our proposed Diffusion model operates in the embedding space of the CLIP model and defines a Markov chain of the latent variables for δ, i.e., δ 1 , ..., δ T , where the initial state of the Diffusion model remains an isotropic Gaussian, i.e., δ T ∼ N (0, I), which in this case, can also be treated as a fixed prior for the modality gap δ, and δ 0 corresponds to the ground truth modality gap. It is non-trivial incorporating the proposed Diffusion model into existing retrieval pipeline due to different natures of the generation and retrieval. To achieve this, we reimplement the denoising network, develop a pretraining stage and a reverse process-engaged retrieval pipeline.Denoising Network Design. We first design a denoising network by referring to the prevalent DiT [Peebles and Xie, 2023] . The denoising network, denoted as ϵ γ (•), consists of N DiT blocks and let γ denote the learnable parameters for the network. As shown in Fig. 2 , the denoising network takes three inputs, including the timestamp t, the latent embedding δ t ∈ R d , and the condition c ∈ R d . Since we define the modality gap δ stems from t as in Section 3.1, we let the text embedding t as the condition to guide the generation, i.e., c = t, which substitutes the vanilla classifier-free guidance [Ho and Salimans, 2021] in DiT. Finally, since we still perform the retrieval in the same latent space, there is no need to perform a reshape operation at the end of the denoising network. As shown in Fig. 1 (c ), the denoising network ϵ γ (•) approximates the noise imposed on δ t during the diffusion process. To facilitate the learning of ϵ γ (•), we introduce a pretraining stage as follows."a beautiful sceneary is shown and the sight seeing through the train is amazing and place is so lovely to watch"Video Encoder ) $ (⋅) Text Encoder * % (⋅)"The train is passing through a forest" "Outside the train window, there are lakes, mountains, and a small town.""the girl shows the boys her medal""a cartoon shows two dogs talking to a bird." "Two dogs feel confused in the cartoon."× (′ 2 " ~4" 4 " = 6 " 7 " 8 * 9 " , ;, , + = " > Embedding Space / (%) / (') / (() 0 (%) 0 (') 0 (() 0 ()) 0 (*) 0 (+)Modality Gap Modeling Timestep=;8 * (⋅) 2 "+, ~4"+, Embedding Space * % (⋅) ) $ (⋅)Compute similarity or contrastive learning-* (") -(") -(%)- (&) . (") . (%) . (&) . (') . (() . ( ))-(") -(%) -(&). (") -* (") and the text embedding t from G ϕ (•), DITS performs a T ′ -steps sampling by starting from t, to gradually approach v (as exampled by a relevant pair t (1) , v (1) ) with the aligned embedding v? .+/ ? . M 24+ = (M 2 -C 2 1 5 M 2 , E, F ) + N 2 z Relevant text/(1) 0 at the timestamp t = 0. Meanwhile the progressive alignment, DITS intrinsically approximates the distribution of the modality gap δ ∼ P t at each timestamp t = T ′ , ..., 1. The contrastive loss is adopted to guide the alignment and modeling, with parameters: Θ = {θ, ϕ, γ}. We devise ϵ γ (•) upon DiT [Peebles and Xie, 2023] .Diffusion model Pretraining. The target distribution of δ can be highly adaptive to v and t given by Eq. (1). To stabilize the learning, we adopt the pretrained retrieval model and fix F θ (•) and G ϕ (•) during the pretraining. The learning objective of ϵ γ (•) derives from the diffusion process [Qiu et al., 2024 , Nukrai et al., 2022] , which gradually adds the Gaussian noise to δ 0 for T steps, where δ 1 , ..., δ T are the latent variables of the same dimension, i.e.,EQUATIONwhere β 1 , ..., β T can simply be held as constant [Ho et al., 2019] to form a variance schedule. By adopting the reparameterization [Kingma and Welling, 2013] upon Eq. ( 4) , δ t can be analytically represented in a closed formEQUATIONTo approximate P γ (δ t-1 |δ t ), we use the variational lower bound to optimize the negative loglikelihood -log P γ (δ 0 ), which can be parameterized as a L 2 loss [Ho et al., 2019 ]EQUATIONwhere ϵ ∼ N (0, I), t ∼ Uniform(1, ..., T ) and c denotes the condition. Specifically, ϵ, t, and c are three inputs of the denoising network ϵ γ (•). By minimizing the L γ , the Diffusion model learns to transition from the isotropic Gaussian to the target distribution of δ 0 through a reverse process, based on which this method aligns the input text embedding t (given as condition) as t + δ 0 , to bridge its relevant counterpart v. We introduce the reverse process-engaged retrieval pipeline in the next.Reverse Process-Engaged Retrieval. We perform the vanilla DDPM [Ho et al., 2019] sampling process in this work. Starting from the isotropic Gaussian δ T ∼ N (0, I), the reverse process samples the gap δ 0 through a T -step Markov chainEQUATIONwhere t decreases from T to 1, z = 0 if t = 1, and σ 2 t = β t . The reverse process of the Diffusion model will be plugged into the retrieval pipeline as the post-feature extraction alignment, collectively forming a reverse process-engaged retrieval pipeline. To this end, one can substitute t with the aligned embedding t + δ 0 and perform retrieval with regard to arbitrary v. Since the embedding space structure has been edited by the Diffusion model, it might be better to further rearrange the multi-modality embeddings with the contrastive loss. We opt to jointly tune all of the learnable parameters Θ = {θ, γ, ϕ} upon this reverse process-engaged retrieval pipeline with the symmetric cross-entropy given in Eq. ( 2). We provide more details and discussion in Section 4.1.To sum up, this work first implements a feasible approach to leverage the generative power of the Diffusion model in the retrieval pipeline. We firstly pretrain the Diffusion model with Eq. ( 6) and then incorporate the reverse process in Eq. ( 7) for the alignment. Notably, we find that despite that the Diffusion model effectively bridging the relevant text and video embedding, either applying the pretrained model for the alignment or adopting the reverse process-engaged retrieval causes obvious performance declines (shown in Table 4 ). This prompt us to rethink Diffusion models in the context of our problem. Based on the designs in Section 3.2, we develop DITS in the following. Rethinking the Vanilla Diffusion model. We find two-fold limitations of the Diffusion model defective for the task of retrieval. First, the L 2 loss (L γ ) does not fit the ranking problem inherent in text-video retrieval. On one hand, L γ shown in Eq. ( 6) minimizes a mean squared error, which inherently lacks an prioritization effect for the relevant pairs over irrelevant ones. On the other, it is not clear what the desired alignment should be for the irrelevant pairs, making it hard to define a target distribution accordingly and implement L γ for the modeling. Nevertheless, there is no guarantee that the L 2 loss (e.g., L γ ) can effectively pull away irrelevant pairs, even though the gap is mitigated for the relevant pairs as shown in Fig. 3 . Second, the reverse process either fails to benefit the retrieval (Table 4 ) -the initial starting point of the reverse, i.e., of δ T ∼ N (0, I) can significantly impact the diversity and the generation quality [Zhou et al., 2023 , Chung et al., 2022 , Zheng et al., 2023] . This can incur inaccurate alignment and retrieval. How to determine the δ T without affecting the performance remains a challenge.Diffusion-Inspired Truncated Sampler. Bearing the above findings, in this work, we introduce the diffusion-inspired truncated sampler (DITS) as an effective alignment tool to benefit the retrieval. Considering the neighboring nature of the CLIP embedding space -relevant v and t have already been within arm's length of the distance) [Liang et al., 2022 , Nukrai et al., 2022] -there is no need to start sampling from the isotropic Gaussian as in the vanilla Diffusion models. Instead, DITS proposes a progressive alignment procedure that directly starts from the t to v as shown in Fig. 2 upper right, forming truncation. Given v by F θ (•) and t by G ϕ (•)EQUATIONwhere ϵ γ (•) serves as an alignment network and T ′ denotes the number of the truncated timestamps. The starting point v T ′ = t (taking t (1) as an example in Fig. 2 ) and χ t = 1-αt √ αt-αt ᾱt denotes a pre-defined schedule. Besides, we let z ∼ N (0, I) if t > 1, else z = 0. We set λ t = σ t . Compared with vanilla sampling process given by Eq. ( 7), the initial latent of DITS becomes a text embedding point (e.g., t (i) ) rather than a random sample from the isotropic Gaussian, which brings following advantages: (1) DITS gets rid of the effect of starting point random sampling. (2) the query text (such as for t → v) initializes the alignment in a more straightforward way, rather than solely being a condition to guide the learning. Considering this, the condition c is not used in ϵ γ (•). In addition, we adopt the contrastive loss upon Eq. ( 2) to guide the learning, during which we substitute t with the aligned embedding v 0 . We jointly train all of the learnable parameters Θ = {θ, ϕ, γ} as denoted in red in Fig. 2 . By observing and considering both relevant and irrelevant pairs, DITS learns the alignment that potentially benefits the overall retrieval performance.It turns out DITS intrinsically involves the modality gap modeling along the timestamps, (i.e., δ ∼ P t ) as shown in Fig. 2 . One can derive the distribution of the modality gap δ from Eq. ( 8), which can be represented by using the reparameterizationEQUATIONwhere ϵ γ (v t , t, c) can be regarded as a bias of the modality gap and λ t controls the scale of the gap modeling at the timestamp t. Interestingly, DITS yields a remarkable performance boost over the baseline and achieves the state-of-the-art performance (Section 4.2). We also empirically find that DITS also serves as an effective tool to help improve the CLIP embedding space. See more details in Section 4.4.Table 1 : Text-to-video retrieval performance on MSRVTT [Xu et al., 2016] and LSMDC [Rohrbach et al., 2015] . Bold denotes the best performance. "-" denotes that the result is unavailable. [Fang et al., 2023] 47.5 73.9 83.5 2.0 12.3 -----UCOFIA [Wang et al., 2023] 49.4 72.1 --12.9 -----TEFAL [Ibrahimi et al., 2023] 49.4 75.9 83.9 [Wang et al., 2019] . Bold denotes the best performance. "-" denotes that the result is unavailable. Implementation Details. Following previous works [Gorti et al., 2022 , Li et al., 2023b] , we resize the video to be the spatial size of 224 × 224 and uniformly sample 12 frames from the video for all datasets. For the retrieval model, we adopt X-Pool [Gorti et al., 2022] as the baseline, where both CLIP-ViT/B-32 and CLIP-VIT/B-16 are employed as the feature extractor. The dropout is set to 0.3. Different from DiT [Peebles and Xie, 2023] , we set our denoising network (also used as the alignment network in DITS) with N = 4 blocks, with 16 heads and an MLP ratio of 4.0. We let the dimension d = 512 for the whole model. We find that a timestamp of T = 10 is enough for diffusion-based alignment. For DITS, we set the truncated timestamp T ′ = 5 for DiDeMo and T ′ = 10 for others. A linear variance schedule with β = 0.1 and β = 0.99 is adopted. All of the parameters Θ = {θ, ϕ, γ} are trained with an AdamW [Loshchilov and Hutter, 2017] optimizer with weight decay of 0.2 and warmup rate of 0.1. We set the training epochs to 5 for all datasets and adopt the same seed of 24.Methods MSRVTT Retrieval LSMDC Retrieval R@1 ↑ R@5 ↑ R@10 ↑ MdR ↓ MnR ↓ R@1 ↑ R@5 ↑ R@10 ↑ MdR ↓ MnR ↓ CLIP-ViT-B/32 X-Methods DiDeMo Retrieval VATEX Retrieval R@1 ↑ R@5 ↑ R@10 ↑ MdR ↓ MnR ↓ R@1 ↑ R@5 ↑ R@10 ↑ MdR ↓ MnR ↓ CLIP-ViT-B/32 X-We perform contrastive learning with a batch size of B = 32 for all datasets and backbones. Same as X-Pool [Gorti et al., 2022] , the learning rate of the CLIP model is initialized as 1 × 10 -5 . The learning rate for non-CLIP modules is 3 × 10 -5 for MSRVTT [Xu et al., 2016] and 1 × 10 -5 for all the other datasets. For a fair comparison, no post-processing techniques [Bogolin et al., 2022 , Cheng et al., 2021] are employed in this work. We implement DITS with PyTorch [Paszke et al., 2019] and perform experiments on an NVIDIA A100 GPU.Compared Methods. We compare DITS with X-Pool [Gorti et al., 2022] , STAN [Liu et al., 2023] ,ProST [Li et al., 2023b] , DiffusionRet [Jin et al., 2023] , UATVR [Fang et al., 2023] , UCOFIA [Wang et al., 2023] , TEFAL [Ibrahimi et al., 2023 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
performance comparison: 
 We compare the text-to-video retrieval performance of different methods in Table 1 , 2, and 3. The proposed DITS can achieve the best performance on different CLIP backbones, datasets, and metrics. For example, on MSRVTT, DITS outperforms the baseline X-Pool by 5% at R@1 and outperforms T-MASS by 1.7%/2.3% at R@1 on different backbones. On DiDeMo, DITS outperforms CLIP-ViT by 2.2% with CLIP-ViT-B/32 and 5.3% with CLIP-ViT-B/16, achieving a consistent boost on different metrics. Moreover, on the challenging dataset of Charades, DITS also brings a remarkable performance boost over X-Pool on both backbones, e.g., 4.3% and 6.7% at R@1, respectively. There is one scenario that T-MASS enables better retrieval than DITS on LSMDC with CLIP-ViT-B/32. However, our method gains an advantage with a larger CLIP backbone on all datasets, indicating better scalability on the retrieval model. We also note that DITS excels in retrieval top-ranked results, such as on MSRVTT, DiDeMo -a progressive alignment and gap modeling facilitates a narrower retrieval scope, encouraging retrieval precision. We provide in-depth analysis of this tendency in Section 4.4. Overall, extensive results on benchmark datasets demonstrate that DITS effectively aligns text-video embedding in the joint space, delivering promising retrieval results.Table 4 provides an ablation study of the proposed method. Specifically, "Baseline" denotes the baseline method of X-Pool [Gorti et al., 2022] , based on which we provide three types of the Table 5 : Discussion on DITS. Highlighted settings are adopted for the benchmark comparison.(a) Discussion on the timestamps (T ′ ) on MSRVTT. alignment methods, corresponding to Fig. 1 . First, we show the naive modeling methods using two different fixed priors, such as σ 2 = 0.1, 1.0. Both of them bring performance descent. Fixed priors are inflexible, hardly benefiting the retrieval as expected. We notice that the sensitivity of the performance toward the prior. This also prompts us to leverage advanced methods of Diffusion model below. We provide the retrieval performance corresponding to both training stages of pretraining and fine-tuning as in Section 3.2. In pretraining, the reverse process is adopted for the generation. As shown, the pretraining with L 2 loss undermines the retrieval performance (e.g., 35.4% v.s. 46.9%).T ′ R@1 ↑ R@5 ↑ R@10 ↑ MdR ↓Despite the fact that L 2 loss effectively attracts relevant pairs (Fig. 3 ), it fails to align the irrelevant ones, leading to the performance descent. By comparison, when we fine-tune the model with the contrastive loss, we observe a remarkable boost of 11.1% at R@1 over pretraining and better results than baseline of X-Pool on metrics except R@1. This indicates that such a method, although enables a good retrieval scope (e.g., being favorable for R@5/10), but fails to performs accurate alignment and retrieval, underlying which, one main flaw lies in the random sampling in isotropic Gaussian. Drawing inspiration, the proposed DITS in the bottom line of Table 4 proposes a truncation process to alleviate the above concern, and adopts contrastive learning to guide the alignment. Encouragingly, we find an evident boost of implementing such a pipeline by following the variance schedule of the vanilla diffusion flow (e.g., α t , β t ). This prompt us to study the behavior of DITS in the following. Discussion on Truncated timestamps. The number of the timestamps T ′ plays an important role for an accurate retrieval.We provide the performance of DITS at different timestamps in Table 5a . The T ′ = 0 denotes the baseline of X-Pool. As shown, the performance of DITS first increases and then drops when we gradually enlarging T ′ from 1 to 40. DITS achieves the best performance at T ′ = 10. Specifically, one-step operation (T ′ = 1) enables determining a rough retrieval scope, benefiting Recall over a larger range, e.g., R@10. Based on this, DITS requires more steps to narrow the scope for more accurate alignment and retrieval (e.g., T ′ = 10). However, continually enlarging T ′ enforces DITS to learn an extremely precise result, sacrificing the flexibility to accommodate the testing data and in the end, resulting the loss of the generalization ability.Improving the CLIP Embedding Space. We also study the effect of DITS to the CLIP in Table 5b .There is a large performance gap when we fix CLIP or perform joint train for DITS. This indicates that solely performing the alignment in the original CLIP space is not enough. To this end, DITS not only plays the role of text-video embedding alignment, but also serves as a tool to guide the learning of CLIP, thus improving the CLIP space. As shown in Fig. 4 , the distribution of the modality gap3 between two spaces deviates from each other, characterizing the aligning effect of DITS.Diffusion Model Conditions. We perform different conditions of the ϵ γ (•) in DITS, including with text condition t, with video condition v, and with both conditions t and v, as well as without any conditions (see discussions in Section 3.3). As shown in Table 6a , we find that there is no need to introduce the conditions onto the network. The diffusion process for this task emphasizes "accurate alignment", unlike the general Diffusion models highlighting diversity in creative works. The previous intuition of the diffusion condition may not be applicable in this work: (1) Due to the one-to-many Modality Gap Stemming from video embedding. We also provide experiments when let δ = tv in Table 6b . As shown, such a setting brings sub-optimal performance. Imposing δ = tv to the v works like approximating t for the retrieval, conducting the "text-to-text" retrieval, considering less semantic clues within the text, it might be hard to retrieve accurately. By comparison, the proposed method DITS implements δ = vt to the t works like approximating v for the retrieval, performing the "video-to-video" retrieval. Since video contains much abundant clues, it might be easier to get better performance. Interesting, we notice that the previous work of Cap4video [Wu et al., 2023 ] also demonstrates similar tendencies in Table 5 , denoted as Query-Caption/Video Only case. 
conclusion: 
 This work studied the task of text-video retrieval by proposing a Diffusion-Inspired Truncated Sampler (DITS) for the multi-modality alignment. The primary contribution of this work was to offer insights on tailoring the Diffusion model for the ranking task inherent in retrieval. We uncovered two-fold limitations of the vanilla Diffusion models: the vanilla L 2 loss was inadequate for alignment by overlooking the irrelevant pairs, the the random initial sampling in isotropic Gaussian introduces variability, causing misalignment. We proposed DITS, which leveraged the inherent proximity of text and video embedding and directly started sampling from text to alleviate the sampling variability, and proposed to adopt the contrastive learning not only to guide the iterative alignment steps over time, but also to facilitate gap modeling. Extensive experiments demonstrated the state-of-the-art performance of DITS. We found DITS could encourage an improved embedding space by guiding the CLIP's learning. We hope DITS can inspire future explorations in studying Diffusion models in the task of text-video retrieval. 
acknowledgment: 
 This research work was supported by the DEVCOM Army Research Laboratory under contract W911QX-21-D-0001.Table 8 : Discussion on the timestamps (T ′ ) on MSRVTT. L 2 loss penalizes larger errors more heavily than smaller ones because the error term is squared. This can be beneficial in tasks where it is crucial to reduce large deviations, as it forces the model to pay more attention to outliers or significant errors. By comparison, L 2 loss treats all errors equally, which might not be ideal if large errors need to be minimized more aggressively. In the context of our problem, L 2 ensures that these outliers are brought into alignment, reducing their impact on the overall model performance, which might be a reason for this observation.T ′ R@1 ↑ R@5 ↑ R@10 ↑ MdR ↓More Discussions on Timestamp. In Table 8 , we provide a more thorough comparison of different timestamps. As we gradually enlarge the total number timestamp T ′ from 1 to 40, all metrics will first get better, and then decrease. This is because DITS performs a progressive alignment. It gradually narrows down the retrieval scope when we initially increase T ′ , enabling generally improved performances (R@1, R@5, MdR). However, the model loses the generalization ability if we continually inlarge T ′ , causing a performance degradation (e.g., from 15 to 40). We also notice there are some inconsistent tendencies, such as R@10 decreases from T ′ = 1 to T ′ = 15, contrary to the tendency of R@1 and R@5. This is because different recall precision (e.g., 1,5,10) can be regarded as different retrieval tasks with different focus. When the model continually narrows down the retrieval scope for the given query, it can easily affect R@10 that requires a larger scope. To balance between different Recall precision, we choose T ′ = 10 for MSRVTT. This experiment shows the potential of the progressive alignment in adjusting the retrieval scope. Note that devising an automated method for timestamp selection may improve performance. However, our experimental results suggest that setting the timestamp to a small range, e.g., 1 ∼ 15 yields reasonable performances and within this small range, a general and consistent setting (T ′ = 10) enables leading performances across diverse datasets, e,g, MSRVTT, LSMDC, Charades, and VATEX. Thus, we adopted the empirical timestamp setting to retain computational efficiency and method extensibility.Effectiveness on Learning the Modality Gap In Table 9 , we study the effect of DITS on CLIP embedding space, to uncover how DITS bridge the modality gap. We statistically compare DITS with a "DITS fix CLIP" baseline, in terms of the averaged modality gap, averaged similarity, and the performance. Specifically, we compute the L 1 -norm of the modality gap (cosine similarity) for each relevant pair and compute the averaged value. Overall, DITS can bridge the modality gap by effectively aligning the CLIP embedding space, yielding better performance. In Fig. 5 , we further provide the distribution of the similarity change. Note that Fig. 4 computes the L 2 -norm of the modality gap distribution, corresponding to the Euclidean distance. The values (such as mean around 1.28) are scaled by L 2 -norm and are non-bounded. We also provide the mean absolute distance version (using L 1 -norm) of the same data in Fig. 6 (left). As shown, L 1 -norm enables a different scale and is also non-bounded. The difference between "Joint Train" and "Fix CLIP" is more remarkable under the L 1 -norm scale. We also provide the cosine similarities version of the same data in Fig. 6 (right). As shown, cosine similarities are bounded (i.e., ). The similarity values are generally enlarged with joint train, being consistent with Fig. 4 and Fig. 6 (left). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we discuss the computational efficiency for both training (Table 10 ) and inference (Table 11 ). The proposed DITS does not need much additional resources for training and reports comparable inference efficiency. We specifically compare with both non-diffusion methods (e.g., X- NeurIPS Paper ChecklistMeasured by L2-norm (non-bounded), corresponding to Euclidean distance. We also provide mean absolute distance version (using L1-norm of the same data in Fig.6). 


Decision: This paper received mixed reviews, with three strongly positive reviews (scores 6,7,7) and two mildly negative ones (scores 4,4). 

The paper introduces a novel method to address the challenge of bridging the modality gap between text and video data in retrieval tasks. The authors propose the Diffusion-Inspired Truncated Sampler (DITS), leveraging diffusion models to model the text-video modality gap. DITS performs progressive alignment and modality gap modeling, starting from text embeddings and using a truncated diffusion process to generate aligned video embeddings.

The reviewers agreed that the problem is important, the solution innovative, the theoretical motivation solid, and that the method achieved good results. The main concerns were computational efficiency, some writing issues, and various questions of detail. The authors provided a rebuttal that address these issues. Most of the positive reviewers confirmed that the rebuttal solved all their concerns. Unfortunately, the negative reviewers chose not to participate in the post review phase. This, the fact that many of the concerns they raised seem more like requests for clarification that real negative flags, and that the explanations provided by the authors seem sensible, have led the AC to discount the negative scores. 

As a note to the authors. I would say that it is not OK to accuse the reviewers of bias just because they did not review your work positively. These type of remarks do not help your cause. If the reviewer did not fully understood your paper, it is likely because the paper could better written. In fact, other reviewers have mentioned that the paper could be more clear. In the future, please do not engage in this type of behavior again.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel method to address the challenge of bridging the modality gap between text and video data in retrieval tasks. The authors propose the Diffusion-Inspired Truncated Sampler (DITS), leveraging diffusion models to model the text-video modality gap.
DITS performs progressive alignment and modality gap modeling, starting from text embeddings and using a truncated diffusion process to generate aligned video embeddings.
strength_weakness: strength_weakness: 1. The author claims that the vanilla Diffusion model's L2 loss does not fit the ranking problem in text-video retrieval. However, in the method part (Line 183), the author still uses the conventional L2 loss of diffusion model. I want to know the detail of that how the author addresses the problem of L2 loss.

2. As far as I know, it is unrealistic to directly use the diffusion model to learn the modality gap or the joint distribution probability of cross-modal alignment. For example, after I ran through the code of [1], I found that its main contribution is not in the diffusion model. Like [2], using the diffusion model to solve the impact of time distribution on moment retrieval is a reasonable and effective motivation. Therefore, the author needs to provide more quantitative experiments to prove that the proposed diffusion model can learn the accurate modality gap, such as similarity change or other metrics.

3. The paper could provide more details on the computational efficiency of the proposed method, including runtime and resource usage, which are important considerations for practical deployment. According to my experience of using diffusion models to solve retrieval tasks, the diffusion module will seriously slow down the retrieval speed, and has limited improvement in optimizing alignment quality and improving retrieval accuracy. The experimental results of this paper also show that the proposed method has limited improvement over the SOTA method. The author should provide experimental data on time efficiency. I think this improvement is not enough compared to the obvious lack of time efficiency.

4. The author assumes that the modality gap is Gaussian distributed. Is there any corresponding proof or basis for this assumption?

[1] Diffusionret: Generative text-video retrieval with diffusion model. In ICCV, 2023.
[2] MomentDiff: Generative Video Moment Retrieval from Random to Real
questions: questions: 1. The author claims that the vanilla Diffusion model's L2 loss does not fit the ranking problem in text-video retrieval. However, in the method part (Line 183), the author still uses the conventional L2 loss of diffusion model. I want to know the detail of that how the author addresses the problem of L2 loss.

2. As far as I know, it is unrealistic to directly use the diffusion model to learn the modality gap or the joint distribution probability of cross-modal alignment. For example, after I ran through the code of [1], I found that its main contribution is not in the diffusion model. Like [2], using the diffusion model to solve the impact of time distribution on moment retrieval is a reasonable and effective motivation. Therefore, the author needs to provide more quantitative experiments to prove that the proposed diffusion model can learn the accurate modality gap, such as similarity change or other metrics.

3. The paper could provide more details on the computational efficiency of the proposed method, including runtime and resource usage, which are important considerations for practical deployment. According to my experience of using diffusion models to solve retrieval tasks, the diffusion module will seriously slow down the retrieval speed, and has limited improvement in optimizing alignment quality and improving retrieval accuracy. The experimental results of this paper also show that the proposed method has limited improvement over the SOTA method. The author should provide experimental data on time efficiency. I think this improvement is not enough compared to the obvious lack of time efficiency.

4. The author assumes that the modality gap is Gaussian distributed. Is there any corresponding proof or basis for this assumption?

[1] Diffusionret: Generative text-video retrieval with diffusion model. In ICCV, 2023.
[2] MomentDiff: Generative Video Moment Retrieval from Random to Real
limitations: limitations: See Weaknesses part.
paper_summary: paper_summary: The paper tackles the task of text-video retrieval. It aims to address the modality gap between text and video that usually stems in state-of-the-art models. In order to do this, it leverages Diffusion models. So, it introduces DITS that jointly performs progressive alignment and modality gap modeling in the joint embedding space in order to improve the performance. Finally, the authors test the performance of their method on five benchmarks.
strength_weakness: strength_weakness: While I think that the paper is fairly well written, some parts can be a bit confusing at the first read, though they become clear if you read twice. For example, abstract lines 6-8 first states that Diffusion is used to mitigate the problem, but then there is immediately the claim that there are "flaws" in diffusion. So, I think a rephrasing and introducing a bit later that changes are needed to diffusion in order to make that work would be better.

line 145 "Unfortuntely, we find descent retrieval performances" -> "Unfortunately, there is a decrease in retrieval performance"

The main concern that I have is related to understanding the limitations. I think there should have been a section that discusses the need for additional computational resources, if applicable and how does the running time compare agains non diffusion methods.
questions: questions: Is there any additional cost in terms of running time for the proposed method as opposed to other sota methods?
limitations: limitations: The limitations are discussed in the appendix.
paper_summary: paper_summary: The authors introduce Diffusion-Inspired Truncated Sampler  (DITS) that jointly performs progressive alignment and modality gap modeling in the joint embedding space.  Experiments on five benchmark datasets suggest the state19 of-the-art performance of DITS.
strength_weakness: strength_weakness: 1. In Table 1, under the CLIP-ViT-B/32 feature extractor, the author's method has limited performance improvement compared to the comparison method.
2. In Formula 2, these two terms are added, but why are these two the same? They should be different terms added.
3. What does it mean to multiply ϵ, t, c in the norm of Formula 6? This is not standard.
4. The writing of the methods section needs further standardization and improvement.
questions: questions: 1. In Table 4, do Diffusion and Fine tune parts not need L2 loss? Why does Pretrain have L2 loss, while Fine tune does not need L2 loss and the result is high?
2. In Table 4, does the last row DITS not need L2 loss?
limitations: limitations: See the weaknesses section
paper_summary: paper_summary: This paper smartly leverage the diffusion model to solve a famous modality gap problem in the a CLIP retrieval problem. The proposed method called DITS set the text embedding as the initial point in the 1D diffusion model and try to generate the video embedding. Finally, by using the contrastive loss to align the generated video embedding and ground truth video embedding to finish the training of the diffusion model. The work show convincible result comparing to existing SOTA methods and the ablation study in Table 4 and 5 show the effectiveness of the method comparing to raw CLIP and existing work DiffusionRet.
strength_weakness: strength_weakness: The figure is too small to read.(miner issue)
Figure 4, it looks like the gap is still large as the mean absolute distance is ~1.28. why it is not some small value likes 0.5?
Is there any investigation on cases, which video retrieval cases got improved and which video cases become worse?
questions: questions: check the weakness
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces a new method, Diffusion-Inspired Truncated Sampler (DITS), designed for text-video retrieval tasks. It addresses the primary challenge of bridging the modality gap between text and video data, a problem that existing retrieval methods often fail to solve effectively. The authors propose DITS to harness the strengths of diffusion models in reducing this gap and enhancing the alignment between text and video embeddings within a joint embedding space.
strength_weakness: strength_weakness: * Generalization ability. The authors highlight the importance of selecting an optimal number of timestamps for the truncated diffusion process. If the number of timestamps is too large, the model might lose its generalization ability.
* Computational efficiency. Computational efficiency is a common concern with Diffusion models, which typically require substantial computational resources for training and inference.
* There are some confusions in the writing. In Section 3.2, it is mentioned that the condition for diffusion is the textual feature. However, in Section 3.3, the condition is not specified. Additionally, in Appendix A.3, the authors state that an empty condition works best.
questions: questions: * The authors would do well to make a comparison of inference times. 
* Since the timestamps for the truncated diffusion process have a huge impact on performance, could an automated method be devised to determine time steps?
* The performance of unconditional diffusion surpasses that of text or video diffusion, which is counter-intuitive. Could the authors provide further analysis to explain this phenomenon?
limitations: limitations: The limitations of the method have been discussed in detail.

Comments: title: Response To Reviewer vaB9
Response To Reviewer fBRP
Rebuttal answer
comment: We appreciate the reviewer's recognition of our response and support for our work!
We appreciate the reviewer's valuable comments. We thank the reviewer's recognition of our rebuttal!
Thank you for your detailed reply. My concerns have been addressed. I have also read the comments from the other reviewers and the authors' responses. I think the paper as a whole is interesting, though it would benefit from more detailed descriptions and discussions. Therefore, I will keep my rating.
Thank you for providing additional details! I confirm that I read the rebuttal and I don't currently have other questions