SlowGuess/ACL_2020_OCR huggingface dataset:
2019b) and DEBUG (Lu et al., 2019a). In all result tables, the scores of compared methods are reported in the corresponding works. Best results are in bold and second best underlined.
The results on Charades-STA are summarized in Table 2. For fair comparison with ExCL, we follow the same setting in ExCL to use the C3D model fine-tuned on Charades dataset as visual feature extractor. Observed that VSLNet significantly outperforms all baselines by a large margin over all metrics. It is worth noting that the performance improvements of VSLNet are more significant under more strict metrics. For instance, VSLNet achieves  $7.47\%$  improvement in IoU = 0.7 versus
Table 3: Results  $(\%)$  of  $\mathrm{R}@\mathfrak{n},\mathrm{IoU} = \mu$  and "mIoU" compared with the state-of-the-art on ActivityNet Caption.
Model	IoU = 0.3	IoU = 0.5	IoU = 0.7	mIoU
CTRL	18.32	13.30	-	-
TGN	21.77	18.90	-	-
ACRN	19.52	14.62	-	-
ABLR	19.50	9.40	-	13.40
ACL-K	24.17	20.01	-	-
L-Net	-	-	-	13.41
SAP	-	18.24	-	-
SM-RL	20.25	15.95	-	-
DEBUG	23.45	11.72	-	16.03
VSLBase	23.59	20.40	16.65	20.10
VSLNet	29.61	24.27	20.03	24.11
Table 4: Results  $(\%)$  of  $\mathrm{R}@\mathfrak{n},\mathrm{IoU} = \mu$  and "mIoU compared with the state-of-the-art on TACoS.
Module	IoU = 0.3	IoU = 0.5	IoU = 0.7	mIoU
BiLSTM + CAT	61.18	43.04	26.42	42.83
CMF + CAT	63.49	44.87	27.07	44.01
BiLSTM + CQA	65.08	46.94	28.55	45.18
CMF + CQA	68.06	50.23	30.16	47.15
Table 5: Comparison between models with alternative modules in VSLBase on Charades-STA.
$0.78\%$  in IoU  $= 0.5$ , compared to MAN. Without query-guided highlighting, VSLBase outperforms all compared baselines over IoU  $= 0.7$ , which shows adopting span-based QA framework is promising for NLVL. Moreover, VSLNet benefits from visual feature fine-tuning, and achieves state-of-the-art results on this dataset.
Table 3 summarizes the results on ActivityNet Caption dataset. Note that this dataset requires YouTube clips to be downloaded online. We have 1,309 missing videos, while ExCL reports 3,370 missing videos. Strictly speaking, the results reported in this table are not directly comparable. Despite that, VSLNet is superior to ExCL with  $2.06\%$  and  $0.16\%$  absolute improvements over IoU = 0.7 and IoU = 0.3, respectively. Meanwhile, VSLNet surpasses other baselines.
Similar observations hold on TACoS dataset. Reported in Table 4, VSLNet achieves new state-of-the-art performance over all evaluation metrics. Without QGH, VSLBase shows comparable per
6548
Module	CAT	CQA	Δ
BiLSTM	26.42	28.55	+2.13
CMF	27.07	30.16	+3.09
Δ	+0.65	+1.61	-
Table 6: Performance gains (\%) of different modules over "R@1, IoU = 0.7" on Charades-STA.
Figure 5: Similarity scores,  $S$ , between visual and language features in the context-query attention.  $a^s / a^e$  denote the start/end boundaries of ground truth video moment,  $\hat{a}^s / \hat{a}^e$  denote the start/end boundaries of predicted target moment.
formance with baselines.
We conduct ablative experiments to analyze the importance of feature encoder and context-query attention in our approach. We also investigate the impact of extension ratio  $\alpha$  (see Figure 3) in query-guided highlighting (QGH). Finally we visually show the effectiveness of QGH in VSLNet, and also discuss the weaknesses of VSLBase and VSLNet.
4.4.1 Module Analysis
We study the effectiveness of our feature encoder and context-query attention (CQA) by replacing them with other modules. Specifically, we use bidirectional LSTM (BiLSTM) as an alternative feature encoder. For context-query attention, we replace it by a simple method (named CAT) which concatenates each visual feature with max-pooled query feature.
Recall that our feature encoder consists of Convolution + Multi-head attention + Feed-forward layers (see Section 3.2), we name it CMF. With the alternatives, we now have 4 combinations, listed in Table 5. Observe from the results, CMF shows stable superiority over CAT on all metrics regardless of other modules; CQA surpasses CAT whichever feature encoder is used. This study indicates that CMF and CQA are more effective.
Table 6 reports performance gains of different
(a)  $\mathrm{R}@\mathrm{I},\mathrm{IoU} = 0.3$
(b)  $\mathrm{R}@\mathrm{I},\mathrm{IoU} = 0.5$
(c)  $\mathrm{R}@\mathrm{I},\mathrm{IoU} = 0.7$
(d) mIoU
Figure 6: Analysis of the impact of extension ratio  $\alpha$  in Query-Guided Highlighting on Charades-STA.
(a) Charades-STA
Figure 7: Histograms of the number of predicted results on test set under different IoUs, on two datasets.
(b) ActivityNet Caption
modules over “R@1, IoU = 0.7” metric. The results show that replacing CAT with CQA leads to larger improvements, compared to replacing BiLSTM by CMF. This observation suggests CQA plays a more important role in our model. Specifically, keeping CQA, the absolute gain is  $1.61\%$  by replacing encoder module. Keeping CMF, the gain of replacing attention module is  $3.09\%$ .
Figure 5 visualizes the matrix of similarity score between visual and language features in the context-query attention (CQA) module  $(S\in \mathbb{R}^{n\times m}$  in Section 3.3). This figure shows visual features are more relevant to the verbs and their objects in the query sentence. For example, the similarity scores between visual features and "eating" (or "sandwich") are higher than that of other words. We believe that verbs and their objects are more likely to be used to describe video activities. Our observation is consistent with Ge et al. (2019), where verb-object pairs are extracted as semantic activity concepts. In contrast, these concepts are automatically captured by the CQA module in our method.
6549
(a) Two example cases on the Charades-STA dataset
(b) Two example cases on the ActivityNet Caption dataset
Figure 8: Visualization of predictions by VSLBase and VSLNet. Figures on the left depict the localized results by the two models. Figures on the right show probability distributions of start/end boundaries and highlighting scores.
(a) Charades-STA
Figure 9: Plots of moment length errors in seconds between ground truths and results predicted by VSLBase and VSLNet, respectively.
4.4.2 The Impact of Extension Ratio in QGH
We now study the impact of extension ratio  $\alpha$  in query-guided highlighting module on Charades-STA dataset. We evaluated 12 different values of  $\alpha$  from 0.0 to  $\infty$  in experiments. 0.0 represents no answer span extension, and  $\infty$  means that the entire video is regarded as foreground.
The results for various  $\alpha$ 's are plotted in Figure 6. It shows that query-guided highlighting consistently contributes to performance improvements, regardless of  $\alpha$  values, i.e., from 0 to  $\infty$ .
Along with  $\alpha$  raises, the performance of VSLNet first increases and then gradually decreases. The optimal performance appears between  $\alpha = 0.05$  and 0.2 over all metrics.
Note that, when  $\alpha = \infty$ , which is equivalent to no region is highlighted as a coarse region to locate target moment, VSLNet remains better than VSLBase. Shown in Figure 4, when  $\alpha = \infty$ , QGH effectively becomes a straightforward concatenation of sentence representation with each of visual features. The resultant feature remains helpful for capturing semantic correlations between vision and language. In this sense, this function can be regarded as an approximation or simulation of the traditional multimodal matching strategy (Hendricks et al., 2017; Gao et al., 2017; Liu et al., 2018a).
4.4.3 Qualitative Analysis
Figure 7 shows the histograms of predicted results on test sets of Charades-STA and ActivityNet Caption datasets. Results show that VSLNet beats VSLBase by having more samples in the high IoU ranges, e.g., IoU  $\geq 0.7$  on Charades-STA dataset. More predicted results of VSLNet are distributed in the high IoU ranges for ActivityNet Caption dataset. This result demonstrates the effectiveness
(a) A failure case on the Charades-STA dataset with IoU = 0.11.
Language Query: After, the man grabs the girl's arm, then the girl pushes the man over the wall.
(b) A failure case on the ActivityNet Caption dataset with IoU = 0.17.
Figure 10: Two failure examples predicted by VSLNet,  $a^s / a^e$  denote the start/end boundaries of ground truth video moment,  $\hat{a}^s / \hat{a}^e$  denote the start/end boundaries of predicted target moment.
of the query-guided highlighting (QGH) strategy.
We show two examples in Figures 8(a) and 8(b) from Charades-STA and ActivityNet Caption datasets, respectively. From the two figures, the localized moments by VSLNet are closer to ground truth than that by VSLBase. Meanwhile, the start and end boundaries predicted by VSLNet are roughly constrained in the highlighted regions  $S_{\mathrm{h}}$ , computed by QGH.
We further study the error patterns of predicted moment lengths, as shown in Figure 9. The differences between moment lengths of ground truths and predicted results are measured. A positive length difference means the predicted moment is longer than the corresponding ground truth, while a negative means shorter. Figure 9 shows that VSLBase tends to predict longer moments, e.g., more samples with length error larger than 4 seconds in Charades-STA or 30 seconds in ActivityNet. On the contrary, constrained by QGH, VSLNet tends to predict shorter moments, e.g., more samples with length error smaller than -4 seconds in Charades-STA or -20 seconds in ActivityNet Caption. This observation is helpful for future research on adopting span-based QA framework for NLVL.
In addition, we also exam failure cases (with IoU predicted by VSLNet lower than 0.2) shown in Figure 10. In the first case, as illustrated by Figure 10(a), we observe an action that a person turns towards to the lamp and places an item there. The QGH falsely predicts the action as the beginning
of the moment "turns off the light". The second failure case involves multiple actions in a query, as shown in Figure 10(b). QGH successfully highlights the correct region by capturing the temporal information of two different action descriptions in the given query. However, it assigns "pushes" with higher confidence score than "grabs". Thus, VSLNet only captures the region corresponding to the "pushes" action, due to its confidence score.
By considering a video as a text passage, we solve the NLVL task with a multimodal span-based QA framework. Through experiments, we show that adopting a standard span-based QA framework, VSLBase, effectively addresses NLVL problem. However, there are two major differences between video and text. We further propose VSLNet, which introduces a simple and effective strategy named query-guided highlighting, on top of VSLBase. With QGH, VSLNet is guided to search for answers within a predicted coarse region. The effectiveness of VSLNet (and even VSLBase) suggest that it is promising to explore span-based QA framework to address NLVL problems.
This research is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (Project #A18A1b0045 and #A18A2b0046).
Language Query: The person turns off the light.
6551
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
João Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4724-4733.
Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. 2018. Temporally grounding natural sentence in video. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 162-171. Association for Computational Linguistics.
Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. 2019. Localizing natural language in videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8175-8182.
Shaoxiang Chen and Yu-Gang Jiang. 2019. Semantic proposal for activity localization in videos via sentence query. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8199-8206.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186. Association for Computational Linguistics.
Jiyang Gao, Runzhou Ge, Kan Chen, and Ramakant Nevatia. 2018. Motion-appearance co-memory networks for video question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6576-6585.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ramakant Nevatia. 2017. Tall: Temporal activity localization via language query. In IEEE International Conference on Computer Vision, pages 5277-5285.
Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia. 2019. Mac: Mining activity concepts for language-based temporal localization. In IEEE Winter Conference on Applications of Computer Vision, pages 245-253.
Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann. 2019. ExCL: Extractive Clip Localization Using Natural Language Descriptions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1984-1990. Association for Computational Linguistics.
Dongliang He, Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, and Shilei Wen. 2019. Read, watch, and move: Reinforcement learning for temporally grounding natural language descriptions in videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8393-8400.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
F. C. Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2018. Localizing moments in video with temporal language. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1380-1390. Association for Computational Linguistics.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell. 2017. Localizing moments in video with natural language. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5804-5813.
Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, and Weizhu Chen. 2018. Fusionnet: Fusing via fully-aware attention with application to machine comprehension. In International Conference on Learning Representations.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles. 2017. Dense-captioning events in videos. In IEEE International Conference on Computer Vision, pages 706-715.
Hung Le, Doyen Sahoo, Nancy Chen, and Steven Hoi. 2019. Multimodal transformer networks for end-to-end video-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5612-5623. Association for Computational Linguistics.
6552
Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng Chua. 2018a. Attentive moment retrieval in videos. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 15-24. Association for Computing Machinery.
Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua. 2018b. Cross-modal moment localization in videos. In Proceedings of the 26th ACM International Conference on Multimedia, pages 843-851. Association for Computing Machinery.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Chujie Lu, Long Chen, Chilie Tan, Xiaolin Li, and Jun Xiao. 2019a. DEBUG: A dense bottom-up grounding approach for natural language video localization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5147-5156. Association for Computational Linguistics.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019b. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, pages 13-23. Curran Associates, Inc.
Duy-Kien Nguyen and Takayuki Okatani. 2019. Multitask learning of hierarchical vision-language representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10492-10501.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543. Association for Computational Linguistics.
Wasifur Rahman, Md Kamrul Hasan, Amir Zadeh, Louis-Philippe Morency, and Mohammed Ehsan Hoque. 2019. M-bert: Injecting multimodal information in the bert structure. arXiv preprint arXiv:1908.05787.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392. Association for Computational Linguistics.
Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. 2013. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1:25-36.
Marcus Rohrbach, Michaela Regneri, Mykhaylo Andriluka, Sikandar Amin, Manfred Pinkal, and Bernt Schiele. 2012. Script data for attribute-based recognition of composite activities. In Proceedings of the 12th European Conference on Computer Vision. Springer Berlin Heidelberg.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In International Conference on Learning Representations.
Gunnar A. Sigurdsson, Gúl Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision (ECCV), pages 510-526. Springer International Publishing.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. 2019a. Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019b. Videobert: A joint model for video and language representation learning. In 2019 IEEE International Conference on Computer Vision (ICCV).
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5099-5110. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008. Curran Associates, Inc.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692-2700. Curran Associates, Inc.
Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with LSTM. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1442-1451. Association for Computational Linguistics.
6553
Shuohang Wang and Jing Jiang. 2017. Machine comprehension using match-lstm and answer pointer. In International Conference on Learning Representations.
Weining Wang, Yan Huang, and Liang Wang. 2019. Language-driven temporal activity localization: A semantic matching reinforcement learning model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 334-343.
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 189-198. Association for Computational Linguistics.
Among Wu and Yahong Han. 2018. Multi-modal circulant fusion for video-to-language and backward. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 1029-1035. International Joint Conferences on Artificial Intelligence Organization.
Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. In International Conference on Learning Representations.
Huijuan Xu, Kun He, Bryan A. Plummer, L. Sigal, Stan Sclaroff, and Kate Saenko. 2019. Multilevel language and vision integration for text-to-clip retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9062-9069.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32, pages 5754-5764. Curran Associates, Inc.
Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations.
Jianfei Yu and Jing Jiang. 2019. Adapting bert for target-oriented multimodal sentiment classification. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5408-5414. International Joint Conferences on Artificial Intelligence Organization.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134.
Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu. 2019a. Semantic conditioned dynamic modulation for temporal sentence grounding in videos. In Advances in Neural Information Processing Systems, pages 536-546. Curran Associates, Inc.
Yitian Yuan, Tao Mei, and Wenwu Zhu. 2019b. To find where you talk: Temporal sentence localization in video with attention based location regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9159-9166.
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. 2019. Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1247-1257.