SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pandeng Li $^{1*}$ , Chen-Wei Xie $^{2}$ , Hongtao Xie $^{1}$ , Liming Zhao $^{2}$ , Lei Zhang $^{1}$ , Yun Zheng $^{2}$ , Deli Zhao $^{2}$ , Yongdong Zhang $^{1}$
<sup>1</sup> University of Science and Technology of China, Hefei, China  
<sup>2</sup> Alibaba Group
lpd@mail.ustc.edu.cn,{htxie,leizh23,zhyd73}  $@$  ustc.edu.cn
{eniac.xcw, lingchen.zlm, zhengyun.zy}@alibaba-inc.com
zhaodeli@gmail.com
Video moment retrieval pursues an efficient and generalized solution to identify the specific temporal segments within an untrimmed video that correspond to a given language description. To achieve this goal, we provide a generative diffusion-based framework called MomentDiff, which simulates a typical human retrieval process from random browsing to gradual localization. Specifically, we first diffuse the real span to random noise, and learn to denoise the random noise to the original span with the guidance of similarity between text and video. This allows the model to learn a mapping from arbitrary random locations to real moments, enabling the ability to locate segments from random initialization. Once trained, MomentDiff could sample random temporal segments as initial guesses and iteratively refine them to generate an accurate temporal boundary. Different from discriminative works (e.g., based on learnable proposals or queries), MomentDiff with random initialized spans could resist the temporal location biases from datasets. To evaluate the influence of the temporal location biases, we propose two "anti-bias" datasets with location distribution shifts, named Charades-STA-Len and Charades-STA-Mom. The experimental results demonstrate that our efficient framework consistently outperforms state-of-the-art methods on three public benchmarks, and exhibits better generalization and robustness on the proposed anti-bias datasets. The code, model, and anti-bias evaluation datasets are available at https://github.com/IMCCretrieval/MomentDiff.
Video understanding [1-9] is a crucial problem in machine learning [10-16], which covers various video analysis tasks, such as video classification and action detection. But both tasks above are limited to predicting predefined action categories. A more natural and elaborate video understanding process is the ability for machines to match human language descriptions to specific activity segments in a complex video. Hence, a series of studies [17-22] are conducted on Video Moment Retrieval (VMR), with the aim of identifying the moment boundaries ( i.e., the start and end time) within a given video that best semantically correspond to the text query.
As shown in Fig. 1(a), early works address the VMR task by designing predefined dense video proposals ( i.e., sliding windows [23-25], anchors [26] and 2D map [27]). Then, the prediction segment is determined based on the maximum similarity score between dense proposals and the
*Interns at Alibaba Group
(a) Dense: 2DTAN
(b) Sparse: MomentDETR
(c) Random: MomentDiff
Figure 1: (a) Proposal-predefined methods. Yellow highlights in the timeline represent frequently occurring moments in the dataset. (b) Proposal-learnable methods. (c) Our generative method. (d) We model the VMR task as a process of gradually generating real temporal span from random noise.
query text. However, these methods have a large redundancy of proposals and the numbers of positive and negative proposals are unbalanced, which limits the learning efficiency [28]. To deal with this problem, a series of VMR works [29-36] have recently emerged, mainly discussing how to reduce the number of proposals and improve the quality of proposals. Among them, a promising scheme (Fig. 1(b)) is to use sparse and learnable proposals [34-36] or queries [37, 38] (i.e., soft proposals) to model the statistics of the entire dataset and adaptively predict video segments. However, these proposal-learnable methods rely on a few specific proposals or queries to fit the location distribution of ground truth moments. For example, these proposals or queries may tend to focus on video segments where locations in the dataset occur more often (i.e., yellow highlights in Fig. 1(b)). Thus, these methods potentially disregard significant events that transpire in out-of-sample situations. Recent studies [39, 40] indicate that VMR models [35, 27, 32] may exploit the location biases present in dataset annotations [23], while downplaying multimodal interaction content. This leads to the limited generalization of the model, especially in real-world scenarios with location distribution shifts.
To tackle the above issues, we propose a generative perspective for the VMR task. As shown in Fig. 1 (c) and (d), given an untrimmed video and the corresponding text query, we first introduce several random spans as the initial prediction, then employ a diffusion-based denoiser to iteratively refine the random spans by conditioning on similarity relations between the text query and video frames. A heuristic explanation of our method is that, it can be viewed as a way for humans to quickly retrieve moments of interest in a video. Specifically, given an unseen video, instead of watching the entire video from beginning to end (which is too slow), humans may first glance through random contents to identify a rough location, and finally iteratively focus on key semantic moments and generate temporal coordinates. In this way, we do not rely on distribution-specific proposals or queries (as mentioned in the above discriminative approaches) and exhibit more generalization and robustness (Tab. 5 and Fig. 4) when the ground truth location distributions of training and test sets are different.
To implement our idea, we introduce a generative diffusion-based framework, named MomentDiff. Firstly, MomentDiff extracts feature embeddings for both the input text query and video frames. Subsequently, these text and video embeddings are fed into a similarity-aware condition generator. This generator modulates the video embeddings with text embeddings to produce text-video fusion embeddings. The fusion embeddings contain rich semantic information about the similarity relations between the text query and each video frame, so we can use them as a guide to help us generate predictions. Finally, we develop a Video Moment Denoiser (VMD) that enhances noise perception and enables efficient generation with only a small number of random spans and flexible embedding learning. Specifically, VMD directly maps randomly initialized spans into the multimodal space,
taking them as input together with noise intensities. Then, VMD iteratively refines spans according to the similarity relations of fusion embeddings, thereby generating true spans from random to real.
Our main contributions are summarized as follows. 1) To the best of our knowledge, we are the first to tackle video moment retrieval from a generative perspective, which does not rely on predefined or learnable proposals and mitigates temporal location biases from datasets. 2) We propose a new framework, MomentDiff, which utilizes diffusion models to iteratively denoise random spans to the correct results. 3) We propose two "anti-bias" datasets with location distribution shifts to evaluate the influence of location biases, named Charades-STA-Len and Charades-STA-Mom. Extensive experiments demonstrate that MomentDiff is more efficient and transferable than state-of-the-art methods on three public datasets and two anti-bias datasets.
Video Moment Retrieval. Video moment retrieval [32, 31, 41, 42, 33, 43-46] is a newly researched subject that emphasizes retrieving correlated moments in a video, given a natural language query. Pioneering works are proposal-based approaches, which employ a "proposal-rank" two-stage pipeline. Early methods [23, 25-27, 32] usually use handcrafted predefined proposals to retrieve moments. For example, CTRL [23] and MCN [25] aim to generate video proposals by using sliding windows of different scales. TGN [26] emphasizes temporal information and develops multi-scale candidate solutions through predefined anchors. 2DTAN [27] designs a 2D temporal map to enumerate proposals. However, these dense proposals introduce redundant computation with a large number of negative samples [28, 36]. Therefore, two types of methods are proposed: 1) Proposal-free methods [17, 31, 47] do not use any proposals and are developed to directly regress start and end boundary values or probabilities based on ground-truth segments. These methods are usually much faster than proposal-based methods. 2) Proposal-learnable methods that use proposal prediction networks [34-36] or learnable queries [37, 38] to model dataset statistics and adaptively predict video segments. QSPN [35] and APGN [34] adaptively obtain discriminative proposals without handcrafted design. LPNet [36] uses learnable proposals to alleviate the redundant calculations in dense proposals. MomentDETR [37] can predict multiple segments using learnable queries. Since proposal-learnable methods adopt a two-stage prediction [34-36] or implicit iterative [37] design, the performance is often better than that of proposal-free methods. However, proposal-learnable methods explicitly fit the location distribution of target moments. Thus, models are likely to be inclined to learn location bias in datasets [39, 40], resulting in limited generalization. We make no assumptions about the location and instead use random inputs to alleviate this problem.
Diffusion models. Diffusion Models [48-51] are inspired by stochastic diffusion processes in nonequilibrium thermodynamics. The model first defines a Markov chain of diffusion steps to slowly add random noise to the data, and then learns the reverse diffusion process to construct the desired data samples from the noise. The diffusion-based generation has achieved disruptive achievements in tasks such as vision generation [52-58] and text generation [59]. Motivated by their great success in generative tasks [60], diffusion models have been used in image perception tasks such as object detection [61] and image segmentation [62]. However, diffusion models are less explored for videotext perception tasks. This paper models similarity-aware multimodal information as coarse-grained cues, which can guide the diffusion model to generate the correct moment boundary from random noise in a gradual manner. Unlike DiffusionDet [61], we avoid a large number of Region of Interest (ROI) features and do not require additional post-processing techniques. To our knowledge, this is the first study to adapt the diffusion model for video moment retrieval.
In this section, we first define the problem in Sec. 3.1, introduce our framework in Sec. 3.2, and describe the inference process in Sec. 3.3.
Suppose an untrimmed video  $\mathcal{V} = \{\pmb{v}_i\}_{i=1}^{N_v}$  is associated with a natural text description  $\mathcal{T} = \{\pmb{t}_i\}_{i=1}^{N_t}$ , where  $N_v$  and  $N_t$  represent the frame number and word number, respectively. Under this notation definition, Video Moment Retrieval (VMR) aims to learn a model  $\Omega$  to effectively predict the moment  $\hat{\pmb{x}}_0 = (\hat{\pmb{c}}_0, \hat{\pmb{w}}_0)$  that is most relevant to the given text description:  $\hat{\pmb{x}}_0 = \Omega(\mathcal{T}, \mathcal{V})$ , where  $\hat{\pmb{c}}_0$  and  $\hat{\pmb{w}}_0$  represent the center time and duration length of the temporal moments, i.e., predicted spans.
Figure 2: Our MomentDiff framework, which includes a Similarity-aware Condition Generator (SCG) and a Video Moment Denoiser (VMD). The diffusion process is conducted progressively in VMD.
3.2 The MomentDiff Framework
Fig. 2 sheds light on the generation modeling architecture of our proposed MomentDiff. Concretely, we first extract frame-level and word-level features by utilizing pre-trained video and text backbone networks. Afterward, we employ a similarity-aware condition generator to interact text and visual features into fusion embeddings. Finally, combined with the fusion embeddings, the video moment denoiser can progressively produce accurate temporal targets from random noise.
3.2.1 Visual and Textual Representations.
Before performing multimodal interaction, we should convert the raw data into a continuous feature space. To demonstrate the generality of our model, we use three distinct visual extractors [32, 44] to obtain video features  $\mathcal{V}$ : 1) 2D visual encoder, the VGG model [63]. 2) 3D visual encoder, the C3D model [64]. 3) Cross-modal pre-train encoder, the CLIP visual model [65]. However, due to the absence of temporal information in CLIP global features, we additionally employ the SlowFast model [66] to extract features, which concatenate CLIP features. Besides, to take full advantage of the video information [38], we try to incorporate audio features, which are extracted using a pre-trained PANN model [67]. To obtain text features, we try two feature extractors: the Glove model [68] and the CLIP textual model to extract 300-d and 512-d text features  $\mathcal{T}$ , respectively.
3.2.2 Similarity-aware Condition Generator
Unlike generation tasks [69] that focus on the veracity and diversity of results, the key to the VMR task is to fully understand the video and sentence information and to mine the similarities between text queries and video segments. To this end, we need to provide multimodal information to cue the denoising network to learn the implicit relationships in the multimodal space.
A natural idea is to interact and aggregate information between video and text sequences with a multilayer Transformer [70]. Specifically, we first use two multilayer perceptron (MLP) networks to map feature sequences into the common multimodal space:  $\mathbf{V} \in \mathbb{R}^{N_v \times D}$  and  $\mathbf{T} \in \mathbb{R}^{N_t \times D}$ , where  $D$  is the embedding dimension. Then, we employ two cross-attention layers to perform interactions between multiple modalities, where video embeddings  $\mathbf{V}$  are projected as the query  $\mathbf{Q}_v$ , text embeddings  $\mathbf{T}$  are projected as key  $\mathbf{K}_t$  and value  $\mathbf{V}_t$ :  $\hat{\mathbf{V}} = \mathrm{softmax}\left(\mathbf{Q}_v\mathbf{K}_t^{\mathrm{T}}\right)\mathbf{V}_t + \mathbf{Q}_v$ , where  $\hat{\mathbf{V}} \in \mathbb{R}^{N_v \times D}$ . To help the model better understand the video sequence relations, we feed  $\hat{\mathbf{V}}$  into a 2-layer self-attention network, and the final similarity-aware fusion embedding is  $\mathbf{F} = \mathrm{softmax}\left(\mathbf{Q}_{\hat{\mathbf{v}}}\mathbf{K}_{\hat{\mathbf{v}}}^{\mathrm{T}}\right)\mathbf{V}_{\hat{\mathbf{v}}} + \mathbf{Q}_{\hat{\mathbf{v}}}$ , where  $\mathbf{Q}_{\hat{\mathbf{v}}},\mathbf{K}_{\hat{\mathbf{v}}},\mathbf{V}_{\hat{\mathbf{v}}}$  is the matrix obtained from  $\hat{\mathbf{V}}$  after three different projections respectively.
In the span generation process, even for the same video, the correct video segments corresponding to different text queries are very different. Since the fusion embedding  $\mathbf{F}$  serves as the input condition of the denoiser, the quality of  $\mathbf{F}$  directly affects the denoising process. To learn similarity relations for  $\mathbf{F}$  in the multimodal space, we design the similarity loss  $\mathcal{L}_{sim}$ , which contains the pointwise cross entropy loss and the pairwise margin loss:
$$
\mathcal {L} _ {s i m} = - \frac {1}{N _ {v}} \sum_ {i = 1} \boldsymbol {y} _ {i} * \log (\boldsymbol {s} _ {i}) + (1 - \boldsymbol {y} _ {i}) * \log (1 - \boldsymbol {s} _ {i}) + \frac {1}{N _ {s}} \sum_ {j = 1} \max  \left(0, \beta + \boldsymbol {s} _ {n _ {j}} - \boldsymbol {s} _ {p _ {j}}\right), \tag {1}
$$
where  $s \in \mathbb{R}^{N_v}$  is the similarity score, which is obtained by predicting the fusion embedding  $F$  through the MLP network.  $y \in \mathbb{R}^{N_v}$  is the similarity label, where  $y_i = 1$  if the  $i$ -th frame is within the ground truth temporal moment and  $y_i = 0$  otherwise.  $s_{p_j}$  and  $s_{n_j}$  are the randomly sampled positive and negative frames.  $N_s$  is the number of samples and the margin  $\beta = 0.2$ . Although  $\mathcal{L}_{sim}$  may only help the fusion embedding retain some coarse-grained similarity semantics, this still provides indispensable multimodal information for the denoiser.
3.2.3 Video Moment Denoiser
Recent works [39, 40] have revealed that previous models [35, 27, 32] may rely on the presence of location bias in annotations to achieve seemingly good predictions. To alleviate this problem, instead of improving distribution-specific proposals or queries, we use random location spans to iteratively obtain real spans from a generative perspective. In this section, we first introduce the principle of the forward and reverse processes in diffusion models. Then, we build the diffusion generation process in the video moment denoiser with model distribution  $p_{\theta}(\pmb{x}_0)$  to learn the data distribution  $q(\pmb{x}_0)$ .
Forward process. During training, we first construct a forward process that corrupts real segment spans  $\pmb{x}_0 \sim q(\pmb{x}_0)$  to noisy data  $\pmb{x}_m$ , where  $m$  is the noisy intensity. Specifically, the Gaussian noise process of any two consecutive intensities [49] can be defined as:  $q(\pmb{x}_m \mid \pmb{x}_{m-1}) = \mathcal{N}(\pmb{x}_m; \sqrt{1 - \beta_m} \pmb{x}_{m-1}, \beta_m \mathrm{I})$ , where  $\beta$  is the variance schedule. In this way,  $\pmb{x}_m$  can be constructed by  $\pmb{x}_0$ :  $q(\pmb{x}_{1:m} \mid \pmb{x}_0) = \prod_{i=1}^{m} q(\pmb{x}_i \mid \pmb{x}_{i-1})$ . Benefiting from the reparameterization technique, the final forward process is simplified to:
$$
\boldsymbol {x} _ {m} = \sqrt {\bar {\alpha} _ {m}} \boldsymbol {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {m}} \boldsymbol {\epsilon} _ {m}, \tag {2}
$$
where the noise  $\epsilon_{m}\sim \mathcal{N}(0,\mathrm{I})$  and  $\bar{\alpha}_{m} = \prod_{i = 1}^{m}(1 - \beta_{i})$
Reverse process. The denoising process is learning to remove noise asymptotically from  $\pmb{x}_m$  to  $\pmb{x}_0$ , and its traditional single-step process can be defined as:
Figure 3: Video moment denoiser. For simplicity, we only draw the intensity-aware attention structure that is different from the general Transformer.
$$
p _ {\theta} \left(\boldsymbol {x} _ {m - 1} \mid \boldsymbol {x} _ {m}\right) = \mathcal {N} \left(\boldsymbol {x} _ {m - 1}; \boldsymbol {\mu} _ {\theta} \left(\boldsymbol {x} _ {m}, m\right), \hat {\sigma} _ {m} ^ {2} \boldsymbol {I}\right) \tag {3}
$$
where  $\sigma_{m}^{2}$  is associated with  $\beta_{m}$  and  $\pmb{\mu}_{\theta}(\pmb{x}_{m}, m)$  is the predicted mean. In this paper, we train the Video Moment Denoiser (VMD) to reverse this process. The difference is that we predict spans from the VMD network  $f_{\theta}(\pmb{x}_{m}, m, \pmb{F})$  instead of  $\pmb{\mu}_{\theta}(\pmb{x}_{m}, m)$ .
Denoiser network. As shown in Fig. 3, the VMD network mainly consists of 2-layer cross-attention Transformer layers. Next, we walk through how VMD works step by step. For clarity, the input span and output prediction presented below are a single vector.
$\bullet$  Span normalization. Unlike generation tasks, our ground-truth temporal span  $\pmb{x}_0$  is defined by two parameters  $c_{0}$  and  $\pmb{w}_{0}$  that have been normalized to [0, 1], where  $c_{0}$  and  $\pmb{w}_{0}$  are the center and length of the span  $\pmb{x}_0$ . Therefore, in the above forward process, we need to extend its scale to  $[- \lambda, \lambda]$  to stay close to the Gaussian distribution [71, 61]. After the noise addition is completed, we need to clamp  $\pmb{x}_{m}$  to  $[- \lambda, \lambda]$  and then transform the range to  $[0, 1]$ :  $\pmb{x}_{m} = (\pmb{x}_{m} / \lambda + 1) / 2$ , where  $\lambda = 2$ .
$\Theta$  Span embedding. To model the data distribution in multimodal space, we directly project the discrete span to the embedding space through the Fully Connected (FC) layer:  $\pmb{x}_m' = \mathrm{FC}(\pmb{x}_m) \in \mathbb{R}^D$ . Compared to constructing ROI features in DiffusionDet [61], linear projection is very flexible and decoupled from conditional information (i.e., fusion embeddings), avoiding more redundancy.
$\Theta$  Intensity-aware attention. The denoiser needs to understand the added noise intensity  $m$  during denoising, so we design the intensity-aware attention to perceive the intensity magnitude explicitly. In Fig. 3, we use sinusoidal mapping for the noise intensity  $m$  to obtain  $\pmb{e}_m \in \mathbb{R}^D$  in the multimodal space and add it to the span embedding. We project  $\pmb{x}_m' + \pmb{e}_m$  as query embedding and the positional embedding  $\pmb{pos}_m \in \mathbb{R}^D$  is obtained by sinusoidal mapping of  $\pmb{x}_m$ . We can obtain the input query:  $Q_m = \text{Concat}(\text{Proj}(\pmb{x}_m' + \pmb{e}_m), \pmb{pos}_m)$ . Similarly, the input key is  $\pmb{K}_f = \text{Concat}(\text{Proj}(\pmb{F}), \pmb{pos}_f)$  and the input value is  $\pmb{V}_f = \text{Proj}(\pmb{F})$ , where  $\text{Proj}(\cdot)$  is the projection function and  $\text{pos}_f \in \mathbb{R}^D$  is the standard position embedding in Transformer [70]. Thus, the intensity-aware attention is:
$$
\boldsymbol {Q} _ {m} = \operatorname {s o f t m a x} \left(\boldsymbol {Q} _ {m} \boldsymbol {K} _ {f} ^ {\mathrm {T}}\right) \boldsymbol {V} _ {f} + \boldsymbol {Q} _ {m}. \tag {4}
$$
$\clubsuit$  Denoising training. Finally, the generated transformer output is transformed into predicted spans  $\hat{\pmb{x}}_{m - 1} = (\hat{c}_{m - 1},\hat{w}_{m - 1})$  and confidence scores  $\hat{z}_{m - 1}$ , which are implemented through a simple FC layer, respectively. Following [71], the network prediction should be as close to ground truth  $\pmb{x}_0$  as possible. In addition, inspired by [37, 72], we define the denoising loss as:
$$
\mathcal {L} _ {\mathrm {v m r}} \left(\boldsymbol {x} _ {0}, f _ {\theta} \left(\boldsymbol {x} _ {m}, m, \boldsymbol {F}\right)\right) = \lambda_ {\mathrm {L 1}} \left\| \boldsymbol {x} _ {0} - \hat {\boldsymbol {x}} _ {m - 1} \right\| + \lambda_ {\mathrm {i o u}} \mathcal {L} _ {\mathrm {i o u}} \left(\boldsymbol {x} _ {0}, \hat {\boldsymbol {x}} _ {m - 1}\right) + \lambda_ {\mathrm {c e}} \mathcal {L} _ {c e} \left(\hat {\boldsymbol {z}} _ {m - 1}\right), \tag {5}
$$
where  $\lambda_{\mathrm{L1}}$ ,  $\lambda_{\mathrm{iou}}$  and  $\lambda_{\mathrm{ce}}$  are hyperparameters,  $\mathcal{L}_{\mathrm{iou}}$  is a generalized IoU loss [73],  $\mathcal{L}_{\mathrm{ce}}$  is a cross-entropy loss. Note that the above procedure is a simplification of training. Considering that there may be more than one ground truth span in the dataset [37], we set the number of input and output spans to  $N_r$ . For the input, apart from the ground truth, the extra spans are padded with random noise. For the output, we calculate the matching cost of each predicted span and ground truth according to  $\mathcal{L}_{vmr}$  (i.e., the Hungarian match [72]), and find the span with the smallest cost to calculate the loss. In  $\mathcal{L}_{\mathrm{ce}}$ , we set the confidence label to 1 for the best predicted span and 0 for the remaining spans.
After training, MomentDiff can be applied to generate temporal moments for video-text pairs including unseen pairs during training. Specifically, we randomly sample noise  $\hat{x}_m$  from a Gaussian distribution  $\mathcal{N}(0,1)$ , the model can remove noise according to the update rule of diffusion models [69]:
$$
\hat {\boldsymbol {x}} _ {m - 1} = \sqrt {\bar {\alpha} _ {m - 1}} f _ {\theta} (\hat {\boldsymbol {x}} _ {m}, m, \boldsymbol {F}) + \sqrt {1 - \bar {\alpha} _ {m - 1} - \sigma_ {m} ^ {2}} \frac {\hat {\boldsymbol {x}} _ {m} - \sqrt {\bar {\alpha} _ {m}} f _ {\theta} (\hat {\boldsymbol {x}} _ {m} , m , \boldsymbol {F})}{\sqrt {1 - \bar {\alpha} _ {m}}} + \sigma_ {m} \boldsymbol {\epsilon} _ {m}. \tag {6}
$$