SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[55] Liu, Z., R. Feng, K. Zhu, et al. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05125, 2023.
[56] Chen, X., L. Huang, Y. Liu, et al. Anydoor: Zero-shot object-level image customization. arXiv preprint arXiv:2307.09481, 2023.
[57] Zhao, S., D. Chen, Y.-C. Chen, et al. Uni-ControlNet: All-in-one control to text-to-image diffusion models. In NeurIPS. 2023.
[58] Wang, X., H. Yuan, S. Zhang, et al. VideoComposer: Compositional video synthesis with motion controllability. In NeurIPS. 2023.
[59] Li, X. L., J. Thickstun, I. Gulrajani, et al. Diffusion-lm improves controllable text generation. In NeurIPS. 2022.
[60] Popov, V., I. Vovk, V. Gogoryan, et al. Grad-tts: A diffusion probabilistic model for text-to-speech. In ICML, vol. 139, pages 8599-8608. 2021.
[61] Chen, S., P. Sun, Y. Song, et al. DiffusionDet: Diffusion model for object detection. In ICCV, pages 19830-19843. 2023.
[62] Liang, C., W. Wang, J. Miao, et al. GMMSeg: Gaussian mixture based generative semantic segmentation models. In NeurIPS. 2022.
[63] Simonyan, K., A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR. 2015.
[64] Tran, D., L. D. Bourdev, R. Fergus, et al. Learning spatiotemporal features with 3d convolutional networks. In ICCV, pages 4489-4497. 2015.
[65] Radford, A., J. W. Kim, C. Hallacy, et al. Learning transferable visual models from natural language supervision. In ICML, vol. 139, pages 8748-8763. 2021.
[66] Feichtenhofer, C., H. Fan, J. Malik, et al. Slowfast networks for video recognition. In ICCV, pages 6201-6210. IEEE, 2019.
[67] Kong, Q., Y. Cao, T. Iqbal, et al. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE Trans Audio Speech Lang Process, 2020.
[68] Pennington, J., R. Socher, C. D. Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543. 2014.
[69] Song, J., C. Meng, S. Ermon. Denoising diffusion implicit models. In ICLR. 2021.
[70] Vaswani, A., N. Shazeer, N. Parmar, et al. Attention is all you need. In NeurIPS. 2017.
[71] Chen, T., R. Zhang, G. E. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In ICLR. 2023.
[72] Carion, N., F. Massa, G. Synnaeve, et al. End-to-end object detection with transformers. In ECCV, vol. 12346, pages 213-229. 2020.
[73] Rezatofighi, H., N. Tsoi, J. Gwak, et al. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, pages 658-666. 2019.
[74] Regneri, M., M. Rohrbach, D. Wetzel, et al. Grounding action descriptions in videos. TACL, 1:25-36, 2013.
[75] Gao, J., X. Sun, M. Xu, et al. Relation-aware video reading comprehension for temporal language grounding. In EMNLP, pages 3978-3988. 2021.
[76] Chen, T., S. Kornblith, M. Norouzi, et al. A simple framework for contrastive learning of visual representations. In ICML. 2020.
[77] Kingma, D. P., J. Ba. Adam: A method for stochastic optimization. In ICLR. 2015.
[78] Rodriguez-Opazo, C., E. Marrese-Taylor, B. Fernando, et al. DORi: Discovering object relationships for moment localization of a natural language query in a video. In WACV. 2021.
[79] Lu, C., L. Chen, C. Tan, et al. Debug: A dense bottom-up grounding approach for natural language video localization. In EMNLP. 2019.
[80] Escorcia, V., M. Soldan, J. Sivic, et al. Temporal localization of moments in video collections with natural language. arXiv preprint arXiv:1907.12763, 2019.
[81] Lei, J., L. Yu, T. L. Berg, et al. TVR: A large-scale dataset for video-subtitle moment retrieval. In ECCV, pages 447-463. 2020.
[82] Yuan, Y., X. Lan, X. Wang, et al. A closer look at temporal sentence grounding in videos: Dataset and metric. In HUMA. 2021.
[83] Krishna, R., K. Hata, F. Ren, et al. Dense-captioning events in videos. In ICCV, pages 706-715. 2017.
[84] Paszke, A., S. Gross, F. Massa, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS. 2019.
This supplementary material provides more details of our MomentDiff framework:
1. Implementation details.
2. Inference efficiency of MomentDiff.
3. More experiment results.
4. Broader impacts.
Public datasets. Charades-STA [23] serves as a benchmark dataset for the video moment retrieval task and is built upon the Charades dataset, originally collected for video action recognition and video captioning. The Charades-STA dataset comprises 6,672 videos and 16,128 video-query pairs, allocated for training (12,408 pairs) and testing (3,720 pairs). On average, the videos in this dataset have a duration of 29.76 seconds. Each video is annotated with an average of 2.4 moments, with each moment lasting approximately 8.2 seconds. QVHighlights [37] contains 10,148 videos, each 150 seconds long and annotated with at least one text query describing its relevant content. These videos are from three main categories, daily vlogs, travel vlogs, and news events. On average, there are approximately 1.8 non-overlapping moments per query, annotated on 2s non-overlapping clips. The dataset contains a total of 10,310 queries with 18,367 annotated moments. The training set, validation set and test set include 7,218, 1,550 and 1,542 video-text pairs, respectively. TACoS [74] is compiled specifically for video moment retrieval and dense video captioning tasks. It is comprised of 127 videos that depict cooking activities, with an average duration of 4.79 minutes. TACoS contains a total of 18,818 video-query pairs. In comparison to the Charades-STA dataset, TACoS has more video segments that are temporally annotated with queries per video. On average, each video contains 148 queries. Additionally, the TACoS dataset is known for its difficulty, as the queries it contains are limited to only a few seconds or even just a few frames. To ensure impartial comparisons, we use the same dataset split [31], which consists of 10,146, 4,589, and 4,083 video-query pairs for the training, validation, and testing sets, respectively. ActivityNet-Captions [83] comprises of 20,000 videos and 100,000 descriptions that encompass a wide range of contexts. Similar to [27], we designate val 1 as the validation set and val 2 as the testing set. The dataset contains 37,417, 17,505, and 17,031 pairs of moments and corresponding sentences for training, validation, and testing respectively.
Table 7: Training and test sets on two anti-bias datasets.
Dataset	Charades-STA-Len	Charades-STA-Mom
w0≤10s	w0>10s	Total	c0+w0/2≤15s	c0-w0/2>15s	Total
Training	9307	2326	11633	5330	1332	6662
Test	197	788	985	259	1038	1297
Anti-bias datasets. To explore the location bias problem, we construct two anti-bias datasets with location distribution shifts based on the Charades-STA dataset. In video moment retrieval, length and position are important parameters of spans.
Therefore, we first investigate the effect of span length  $w_0$ . As shown in Tab. 7, in the training set of Charades-STA-Len, we collect 9,307 video-text pairs with span length  $w_0 \leq 10s$  and 2,326 video-text pairs with  $w_0 > 10s$ , accounting for  $80\%$  and  $20\%$  of the total training set. In contrast, in the test set, we select 197 video-text pairs with  $w_0 \leq 10s$  and 788 video-text pairs with  $w_0 > 10s$ , accounting for  $20\%$  and  $80\%$  of the total test set.
Then, we design the dataset Charades-STA-Mom based on the span's end time  $c_{0} + w_{0}/2$  and start time  $c_{0} - w_{0}/2$ . In the training set of Charades-STA-Mom, we collect 5,330 video-text pairs with  $c_{0} + w_{0}/2 \leq 15s$  and 1,332 video-text pairs with  $c_{0} - w_{0}/2 > 15s$ , accounting for  $80\%$  and  $20\%$  of the total training set. In contrast, in the test set, we select 259 video-text pairs with  $c_{0} + w_{0}/2 \leq 15s$  and 788 video-text pairs with  $c_{0} - w_{0}/2 > 15s$ , accounting for  $20\%$  and  $80\%$  of the total test set.
Algorithm 1: MomentDiff Training in a PyTorch-like style.
Video features: v_feats  $\in \mathbb{R}^{N_v\times D}$    
# Text features: t_feats  $\in \mathbb{R}^{N_t\times D}$    
# Ground truth spans: gt_spans:  $[\ast ,2]$    
# alpha_cumprod(m): cumulative product of  $\alpha_{i}$    
# Fully-connected layer: FC()   
# Video Moment Denoiser: VMD()   
def train(v_feats, t_feats, gt_spans): # Similarity-aware Condition Generator f_feats  $=$  SCG(v_feats, t_feats) #  $N_{v}\times D$    
# Span normalization ps  $=$  pad_spans(gt_spans) # Pad gt_spans to  $[N_r,2]$  ps  $=$  (ps  $*2 - 1)*\lambda$  # Signal scaling m  $=$  randint(0,M) # Noise intensity noisy normal(mean=0, std=1) # Noise ps_m  $=$  sqrt(alpha_cumprod(m)) \* ps + sqrt(1 - alpha_cumprod(m)) \* noisy span ps_m  $=$  (ps_m/λ +1)/2 # Normalization   
# Span embedding ps_emb_m  $=$  FC(ps_m)   
# Intensity-aware attention output_m  $=$  VMD(ps_emb_m, m, f_feats) # Output embedding   
# Denoising training hat_ps_m  $=$  Span_pred(output_m) # Predicted span hat_cs_m  $=$  Score_pred(output_m) # Confidence score   
# Computing loss loss  $=$  loss_sim(f_feats)  $+$  loss_vmr(hat_ps_m, hat_cs_m, gt_spans) return loss
A.2 Pseudo Code of MomentDiff
Algorithm 1 provides the pseudo-code of MomentDiff Training in a PyTorch-like style.
The inference procedure of MomentDiff is a denoising sampling process from noise to temporal spans. Starting from spans sampled in Gaussian distribution, the model progressively refines its predictions, as shown in Algorithm 2.
B Inference Efficiency of MomentDiff
Inference time. Inference efficiency is critical for machine learning models. We test 2DTAN [27], MMN [28], MomentDETR [37] and MomentDiff on the Pytorch framework [84] in Tab. 8. We test all models with one NVIDIA Tesla A100 GPU.
Compared with 2DTAN [27] and MMN [28], MomentDiff (Step=1) not only achieves the best results on recall, but also improves the inference speed by 5-7 times. This is because 2DTAN and MMN predefine a large number of proposals, which may be redundant and increase computational overhead. Compared to MomentDETR [37], MomentDiff (Step=10) achieves better results with similar inference time. The possible reasons are that we adopt fewer random spans, very simple network structures, and avoid post-processing.
Algorithm 2: MomentDiff inference in a PyTorch-like style.
Table 8: The inference time of 2DTAN [27], MMN [28], MomentDETR [37] and MomentDiff on Charades-STA with VGG video features and Glove text features. We report R1@0.5, R1@0.7 and  $\mathrm{MAP}_{avg}$ . Default settings are marked in blue.
# Video features: v_feats  $\in \mathbb{R}^{N_v\times D}$    
# Text features: t_feats  $\in \mathbb{R}^{N_t\times D}$    
# Video Moment Denoiser: VMD()   
def test(v_feats, t_feats, sampling_num): # Similarity-aware Condition Generator f_feats  $=$  SCG(v_feats, t_feats) #  $N_{v}\times D$    
# Noisy span:  $[N_r,2]$  ps_m  $=$  normal(mean=0, std=1)   
# uniform sample intensity  $=$  reversedlinespace(-1,M,sampling_num)) # [(M-1,M-2)，(M-2，M-3)，...，(1，0)，(0，-1)] intensity_pairs  $=$  list(zip(intensity[: -1], intensity[1:])   
for intensity_now, intensity_next in zip(intensity_pairs): # predict ps_0 from ps_m output_m  $=$  VMD(ps_m,f_feats, intensity_now) # Predicted span hat_ps_m  $=$  Span_pred(output_m) # Update ps_m ps_m  $=$  ddim_update(hat_ps_m,ps_m,intensity_now, intensity_next)   
# Confidence score hat_cs_m  $=$  Score_pred(output_m)   
return hat_ps_m, hat_cs_m
Method	Charades-STA	Inference time (second)
R1@0.5	R1@0.7	MAPavg
2DTAN [27]	41.34	23.91	29.26	42.18
MMN [28]	46.93	27.07	31.58	53.42
MomentDETR [37]	50.54	28.01	29.87	12.42
MomentDiff (Step=1)	49.17	26.39	29.12	7.56
MomentDiff (Step=2)	50.81	27.84	31.27	8.23
MomentDiff (Step=10)	52.36	28.08	31.75	11.01
MomentDiff (Step=50)	51.94	28.25	31.66	20.74
MomentDiff (Step=100)	52.21	28.84	31.01	34.35
C.1 Experiment on ActivityNet-Captions
The results of our method on ActivityNet-Captions are shown in Tab. 9. We record the 1 epoch training time and the inference time for all test samples in one NVIDIA Tesla A100 GPU. Our model has been improved compared with baseline (MomentDETR). Compared to SOTAs, we still have competitive results. Compared with MMN, our method is 25 times faster in training time and 7.24 times faster in testing time.
Table 9: Performance comparisons (%) on ActivityNet-Captions with C3D video features and Glove text features.
Method	R1@0.3	R1@0.5	R1@0.7	MAPavg	Training time	Inference time
2DTAN [27]	59.92	44.63	27.53	27.26	1.1h	523.74s
MMN [28]	65.21	48.26	28.95	28.74	1.5h	662.12s
MomentDETR [37]	61.87	43.19	25.74	25.63	0.05h	52.72s
MomentDiff	62.79	46.52	28.43	28.19	0.06h	91.43s
Figure 6: Performance fluctuations (\%) corresponding to different features and multiple random seeds on the Charades-STA dataset.
C.2 Error bars
Fig. 6 shows the performance fluctuation of the model on the Charades-STA dataset. We use different random seeds (seed= 2023, 2022, 2021, 2020, 2019) and different features (VGG, Glove; C3D, Glove; SF+C, C;) to organize experiments. This shows that the model always converges and achieves stable results for different initializations. This phenomenon demonstrates the ability of the model to learn to generate real spans from arbitrary random spans.
C.3 Ablation study
Different sampling strategies. Denoising Diffusion Probabilistic Models (DDPM) [49] and Denoising Diffusion Implicit Models (DDIM) [69] are popular and classic diffusion models. We show the results of both strategies in Tab. 10(a). We find that DDIM and DDPM perform similarly, but DDIM samples faster. Therefore we adopt DDIM as the default technology.
Effect of the schedule of  $\beta$ . The schedule of  $\beta$  determines the weighting ratio of different intensity noises. In Tab. 10(b), we find that the cosine schedule works better in our experiment. The cosine schedule makes the noisy spans change slowly at the beginning and end during the diffusion process, and the generation effect is more stable. So we set the cosine schedule as the default.
Effect of the Box Renewal. Box Renewal is a post-processing technique in DiffusionDet [61]. Tab. 10(c) shows that Box Renewal can indeed slightly improve the results. To keep the inference process as simple as possible, we do not use Box Renewal by default.
Effect of the batch size. As shown in Tab. 10(d), we set the batch size to 32 to achieve the best results.
Effect of the layer number. In Tab. 10(e), we show the effect of the layer number of Similarity-aware Condition Generator (SCG) and Video Moment Denoiser (VMD). The default setting  $(2 + 2:2)$ : SCG contains 2 cross-attention and 2 self-attention layers and VMD contains 2 cross-attention layers. The default setting works best.
Effect of the weight value on  $\mathcal{L}_{sim}$ . We set the weight values of  $\mathcal{L}_{sim}$  and  $\mathcal{L}_{vmr}$  to 4 and 1, respectively. Keeping the weight value of  $\mathcal{L}_{vmr}$  unchanged, we organize the weight influence experiment of  $\mathcal{L}_{sim}$ , as shown in Tab. 10(f). The default setting works best.
Effect of the number of spans on R1@0.5. Our training and inference are decoupled. Our simple framework allows us to input any number of random noises. Tab. 10(g) shows that there is a slight improvement when testing with more noise boxes.
Table 10: Ablation study (%) on the Charades-STA dataset with SF+C video features and CLIP text features. We report R1@0.5, R1@0.7 and  $\mathrm{MAP}_{avg}$ . Default settings are marked in blue.
(a) Different sampling strategies.
Sampling	R1@0.5	R1@0.7	MAPavg
DDPM [49]	55.62	32.37	32.48
DDIM [69]	55.57	32.42	32.85
(c) Effect of the Box Renewal [61].
Schedule	R1@0.5	R1@0.7	MAPavg
w/o Box Renewal	55.57	32.42	32.85
w/ Box Renewal	56.03	32.64	33.18
(e) Effect of the layer number.
SCG:VMD	R1@0.5	R1@0.7	MAPavg
1+1:1	51.74	28.97	29.82
2+2:2	55.57	32.42	32.85
3+3:3	54.39	32.16	32.54
(b) Effect of the schedule of  $\beta$
Schedule	R1@0.5	R1@0.7	MAPavg
Linear	54.76	31.43	31.59
Cosine	55.57	32.42	32.85
(d) Effect of the batch size.
batch size	R1@0.5	R1@0.7	MAPavg
16	52.42	30.81	30.14
32	55.57	32.42	32.85
64	53.78	32.25	31.93
(f) Effect of the weight value on  $\mathcal{L}_{sim}$ .
Weight	R1@0.5	R1@0.7	MAPavg
1	52.34	29.63	29.98
2	53.74	30.82	31.15
4	55.57	32.42	32.85
(g) Effect of the number of spans on R1@0.5.
train\test	1	3	5	10	20
1	50.83	50.91	50.97	50.87	50.82
3	53.98	54.12	54.19	54.23	54.21
5	55.36	55.41	55.57	55.69	55.51
10	53.71	53.84	53.86	53.89	53.93
20	53.16	53.38	53.40	53.44	53.42
First, our work does not involve private data. Second, we believe that AI is a double-edged sword, and our model is no exception. For example, when users or websites use our model, only natural language is needed to locate video moments and collect desired video material, which improves the productivity of society. However, this may have a negative impact if the natural language entered by the user contains words related to violence, pornography, etc. We will consider these scenarios and implement a more secure VMR model.