SlowGuess/NAACL_2022_OCR huggingface dataset:
In our work, for the multilingual video corpus moment retrieval (mVCMR), we introduce a simple and effective Cross-Lingual and Cross-Modal Consolidation (i.e.,  $\mathbf{C}^3$ ) strategy. It enhances the reliability of the similarity score from a single modality through the knowledge distillation from the similarity score with access to the multi-modal information. Meanwhile, it exploits the complementary information across languages by cross-lingual knowledge distillation for both video retrieval and moment localization. Extensive experimental results demonstrate the effectiveness of our method.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812.
Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan A Plummer. 2020. Learning to scale multilingual representations for vision-language tasks. In European Conference on Computer Vision, pages 197-213. Springer.
Joao Carreira and Andrew Zisserman. 2017. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657-668.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan Russell. 2019. Temporal localization of moments in video collections with natural language. arXiv preprint arXiv:1907.12763.
Hongliang Fei, Tan Yu, and Ping Li. 2021. Cross-lingual cross-modal pretraining for multimodal retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3644-3650.
Hao Fu, Shaojun Zhou, Qihong Yang, Junjie Tang, Guiquan Liu, Kaikui Liu, and Xiaolong Li. 2021. Lrcbert: Latent-representation contrastive knowledge distillation for natural language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12830-12838.
Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander G Hauptmann. 2019. Excl: Extractive clip localization using natural language descriptions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1984-1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. 2017. Attention-based multimodal fusion for video description. In Proceedings of the IEEE international conference on computer vision, pages 4193-4202.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33.
Hengtong Hu, Lingxi Xie, Richang Hong, and Qi Tian. 2020. Creating something from nothing: Unsupervised knowledge distillation for cross-modal hashing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3123-3132.
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. 2019. Knowledge distillation via route constrained optimization. In ICCV.
Donghyun Kim, Kuniaki Saito, Kate Saenko, Stan Sclaroff, and Bryan Plummer. 2020. Mule: Multimodal universal language embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11254-11261.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706-715.
Jie Lei, Tamara L Berg, and Mohit Bansal. 2021a. mtvr: Multilingual moment retrieval in videos. ACL.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021b. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331-7341.
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020. Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 447-463. Springer.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. Hero: Hierarchical encoder for video+ language omni-representation pretraining. In EMNLP.
Haoliang Liu, Tan Yu, and Ping Li. 2021. Inflate and shrink: Enriching and reducing interactions for fast text-image retrieval. In Proceedings of the 2021
Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9796-9809, Virtual Event / Punta Cana, Dominican Republic.
Jiaheng Liu, Haoyu Qin, Yichao Wu, Jinyang Guo, Ding Liang, and Ke Xu. 2022. Coupleface: Relation matters for face recognition distillation. arXiv preprint arXiv:2204.05502.
Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Ken Chen, Wanli Ouyang, and Dong Xu. 2020. Block proposal neural architecture search. IEEE TIP, 30:15-25.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353.
Antoine Miech, Ivan Laptev, and Josef Sivic. 2018. Learning a text-video embedding from incomplete and heterogeneous data. arXiv preprint arXiv:1804.02516.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640.
Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. 2016. Jointly modeling embedding and translation to bridge video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4594-4602.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. 2019. Correlation congruence for knowledge distillation. In ICCV.
Bryan A Plummer, Matthew Brown, and Svetlana Lazebnik. 2017. Enhancing video summarization via vision-language embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5781-5789.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015. A dataset for movie description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3202-3212.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. 2019a. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv:1906.05743.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019b. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019c. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4323-4332.
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and TieYan Liu. 2018. Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations.
Atousa Torabi, Niket Tandon, and Leonid Sigal. 2016. Learning language-visual embedding for movie understanding with natural-language. arXiv preprint arXiv:1609.08124.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Fei Huang, and Kewei Tu. 2020. Structure-level knowledge distillation for multilingual sequence labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3317-3330.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Ran Xu, Caiming Xiong, Wei Chen, and Jason Corso. 2015. Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In Proceedings of the AAAI Conference on Artificial Intelligence.
Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the European Conference on Computer Vision (ECCV).
Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gun-hee Kim. 2017. End-to-end concept word detection for video captioning, retrieval, and question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
A More detailed preliminary on mXML
The current multilingual moment retrieval model mXML (Lei et al., 2021a) is built upon the Crossmodal Moment Localization (XML) model (Lei et al., 2020), which performs efficient video-level retrieval at its shallow layers and accurate moment-level localization at its deep layers. To adjust to the multilingual settings in the video corpus moment retrieval (VCMR) task and improve the efficiency and effectiveness, mXML employs two strategies (i.e., encoder parameter sharing and language neighborhood constraint loss) to better utilize the multilingual data while maintaining smaller model size. In this section, we briefly review the mXML model, which is also illustrated in Figure 2.
A.1 Input Feature
ResNet-152 (He et al., 2016) and I3D (Carreira and Zisserman, 2017) extract the visual features of each video moment. The generated video moment visual features are denoted by  $\mathbf{E}^v = [\mathbf{e}_1^v,\dots ,\mathbf{e}_L^v ]\in \mathbb{R}^{d\times L}$ , where  $d$  is the feature dimension. The language features are extracted by RoBERTa-base (Liu et al., 2019) for English (Liu et al., 2019) and Chinese (Cui et al., 2020), respectively. For queries, token-level features are used. The query features are denoted by  $\mathbf{E}_g^q = [\mathbf{e}_{g,1}^q,\dots ,\mathbf{e}_{g,L_q}^q ]\in \mathbb{R}^{d\times L_q}$ , where  $L_{q}$  is the number of tokens and  $g\in \{\mathrm{en,zh}\}$ . For subtitles, token-level features in a video moment are max-pooled into a single vector. The subtitle features are denoted by  $\mathbf{E}_g^s = [\mathbf{e}_{g,1}^s,\dots ,\mathbf{e}_{g,L}^s ]\in \mathbb{R}^{d\times L}$ .
A.2 Encoding
mXML uses Self-Encoder (SE) implemented by Transformer (Vaswani et al., 2017) to further encode the query's token features:
$$
\mathbf {H} _ {g} ^ {q} = \operatorname {S E} (\mathbf {E} _ {g} ^ {q}) = [ \mathbf {h} _ {g, 1} ^ {q}, \dots , \mathbf {h} _ {g, L _ {q}} ^ {q} ].
$$
A modular attention (Lei et al., 2020) is conducted on the query token features  $\mathbf{H}_g^q$ , generating two modularized query vectors  $\mathbf{q}_g^v$ ,  $\mathbf{q}_g^s \in \mathbb{R}^d$ . In parallel, mXML encodes the moment subtitle features  $\mathbf{E}_g^s$  and moment visual features through a stack of two Self-Encoders:
$$
\mathbf {H} _ {g, 0} ^ {s} = \operatorname {S E} (\mathbf {E} _ {g} ^ {s}), \mathbf {H} _ {g, 1} ^ {s} = \operatorname {S E} (\mathbf {H} _ {g, 0} ^ {s}),
$$
$$
\mathbf {H} _ {0} ^ {v} = \operatorname {S E} (\mathbf {E} ^ {v}), \mathbf {H} _ {1} ^ {v} = \operatorname {S E} (\mathbf {H} _ {0} ^ {v}).
$$
Among them, the output of the first Self-Encoder,  $\mathbf{H}_{g,0}^{s}$  and  $\mathbf{H}_0^v$ , are used for video retrieval. The
output of the second Self-Encoder,  $\mathbf{H}_{g,1}^{s}$  and  $\mathbf{H}_1^v$  are used for moment localization.
A.3 Video Retrieval
Given the modularized queries  $\mathbf{q}_g^v, \mathbf{q}_g^s$  and the encoded contexts  $\mathbf{H}_0^v, \mathbf{H}_{g,0}^s$ , the video-level retrieval (VR) scores  $S_{s}(q_{g}, v)$  and  $S_{v}(q_{g}, v)$  using the subtitle context and the visual context are computed as follows, respectively:
$$
\begin{array}{l} S _ {s} \left(q _ {g}, v\right) = \max  _ {l \in [ 1, L ]} \cos \left(\mathbf {q} _ {g} ^ {s}, \mathbf {H} _ {g, 0} ^ {s} [:, l]\right), \\ S _ {s} (v) = \left( \begin{array}{l l l} & & \\ & \left(l ^ {2}, \mathbf {I} _ {2} ^ {2 1} [ - 1 ]\right) \end{array} \right) \end{array} \tag {12}
$$
$$
S _ {v} (q _ {g}, v) = \max  _ {l \in [ 1, L ]} \cos (\mathbf {q} _ {g} ^ {v}, \mathbf {H} _ {0} ^ {v} [:, l]),
$$
where  $\mathbf{H}_{g,0}^s [ \cdot ,l ]$  denotes the  $l$ -th column vector in  $\mathbf{H}_{g,0}^s$  and  $\cos (\cdot ,\cdot)$  measures the cosine similarity between two vectors. The score essentially computes the cosine similarity between each clip and query and picks the maximum. Then, the final video-level retrieval (VR) score  $S(q_{g},v)$  using both subtitle and visual contexts is defined as follows:
$$
S \left(q _ {g}, v\right) = S _ {s} \left(q _ {g}, v\right) + S _ {v} \left(q _ {g}, v\right). \tag {13}
$$
A.4 Moment Localization
Given the modularized queries  $\mathbf{q}_g^v, \mathbf{q}_g^s$  and the encoded contexts  $\mathbf{H}_1^v, \mathbf{H}_{g,1}^s$ , mXML computes the query-clip similarity scores  $S_{s}(q_{g},c_{l})$  and  $S_{v}(q_{g},c_{l})$  using the subtitle and visual contexts as follows, respectively:
$$
S _ {s} \left(q _ {g}, c _ {l}\right) = \left\langle \mathbf {H} _ {1} ^ {v} [:, l ], \mathbf {q} _ {g} ^ {v} \right\rangle , \tag {14}
$$
$$
S _ {v} (q _ {g}, c _ {l}) = \langle \mathbf {H} _ {g, 1} ^ {s} [:, l ], \mathbf {q} _ {g} ^ {s} \rangle ,
$$
where  $\langle \cdot ,\cdot \rangle$  denotes the inner product between two vectors. Similarly, the final query-clip similarity score using both contexts is defined as follows:
$$
S \left(q _ {g}, c _ {l}\right) = S _ {s} \left(q _ {g}, c _ {l}\right) + S _ {v} \left(q _ {g}, c _ {l}\right), \tag {15}
$$
which is also the summation of these query-clip similarity scores. Then, to produce moment localization predictions from the final query-clip score  $S(q_{g},c_{l})$ , mXML adopts the Convolutional StartEnd detector (ConvSE) with two 1D convolution filters for learning to detect start (up) and end (down) edges in the score curves and generate the start (st)  $\mathbf{s}_g^{\mathrm{st}}$  and end (ed) scores  $\mathbf{s}_g^{\mathrm{ed}}$ , which are also shown as follows, respectively:
$$
\mathbf {s} _ {g} ^ {\mathrm {s t}} = \operatorname {C o n v S E} _ {\mathrm {s t}} \left(S \left(q _ {g}, c _ {l}\right)\right), \tag {16}
$$
$$
\mathbf {s} _ {g} ^ {\mathrm {e d}} = \operatorname {C o n v S E} _ {\mathrm {e d}} (S (q _ {g}, c _ {l})).
$$
Then, these scores are normalized with the softmax function to output the start and end probabilities  $\mathbf{P}_g^{\mathrm{st}},\mathbf{P}_g^{\mathrm{ed}}\in \mathbb{R}^L$  for each query.
Figure 2: Illustration of inference process for multilingual video corpus moment retrieval (i.e., mVCMR) task. Here, we take the language  $g \in \{\mathrm{en}, \mathrm{zh}\}$  as an example to show the processes of video retrieval and moment localization. For the mVCMR setting, all encoders are shared for different languages, and language neighborhood constraint is used on both query and subtitle embeddings. "ConvSE" and "Aggregation Function" operations are proposed in (Lei et al., 2020). "SVMR" denotes single video moment retrieval.
A.5 Training and Inference
In Figure 2, in the training process, mLX optimizes the video retrieval score and the moment localization probabilities based on the triplet loss and the cross-entropy loss, respectively. Besides, to facilitate more stronger multilingual learning, mLX also utilizes language neighborhood constraint loss for both query and subtitle embeddings based on the triplet loss. Below we introduce these loss functions in detail.
Video Retrieval Loss. For each positive pair  $(q_{g}^{i}, v^{i})$ , mXML samples two negative pairs  $(q_{g}^{i}, v^{j})$  and  $(q_{g}^{k}, v^{i})$  from the same mini-batch to calculate the combined hinge loss as follows:
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {v r}} = \sum_ {g \in \{\mathrm {e n , z h} \}} \sum_ {i = 1} ^ {n} \frac {- 1}{n} \{[ S (q _ {g} ^ {i}, v ^ {i}) - S (q _ {g} ^ {i}, v ^ {j})) + m _ {\mathrm {v r}} ] _ {+} \\ + [ S (q _ {g} ^ {i}, v ^ {i}) - S (q _ {g} ^ {k}, v ^ {i})) + m _ {\mathrm {v r}} ] _ {+} \}, \\ \end{array}
$$
where  $[x]_{+} = \max (x,0)$ ,  $m_{\mathrm{vr}}$  is the margin and  $n$  is the number of samples for each mini-batch.
Moment Localization Loss. Given the start and end probabilities  $\mathbf{p}_g^{\mathrm{st}},\mathbf{p}_g^{\mathrm{ed}}\in \mathbb{R}^L$  , the moment localization loss is defined as follows:
$$
\mathcal {L} ^ {\mathrm {l o c}} = \sum_ {g \in \{\mathrm {e n}, \mathrm {z h} \}} \sum_ {i = 1} ^ {n} \frac {- 1}{n} \left[ \log \left(\mathbf {p} _ {g} ^ {\mathrm {s t}} \left(t _ {\mathrm {s t}} ^ {i}\right)\right) + \log \left(\mathbf {p} _ {g} ^ {\mathrm {e d}} \left(t _ {\mathrm {e d}} ^ {i}\right)\right) \right],
$$
where  $t_{\mathrm{st}}^i$  and  $t_{\mathrm{ed}}^i$  are the ground-truth indices of the start and the end, respectively.
Language Neighborhood Constraint Loss. Following (Kim et al., 2020; Burns et al., 2020), mXML additionally adopts language neighborhood
constraint loss for multilingual learning. It encourages sentences that express the same or similar meanings to be close to each other in the embedding space via a triplet loss. Given the  $i$ -th paired sentence embeddings  $\mathbf{e}_{\mathrm{en}}^i \in \mathbb{R}^d$  and  $\mathbf{e}_{\mathrm{zh}}^i \in \mathbb{R}^d$  from each mini-batch, mXML samples the  $j$ -th and the  $k$ -th negative sentence embeddings  $\mathbf{e}_{\mathrm{en}}^j$  and  $\mathbf{e}_{\mathrm{zh}}^k$  from this mini-batch, where  $i \neq j$  and  $i \neq k$ . The language neighborhood constraint loss  $\mathcal{L}^{\mathrm{nc}}$  can be formulated as follows:
$$
\begin{array}{l} \mathcal {L} ^ {\mathrm {n c}} = \sum_ {i = 1} ^ {n} \frac {- 1}{n} \left\{\left[ \cos \left(\mathbf {e} _ {\mathrm {e n}} ^ {i}, \mathbf {e} _ {\mathrm {z h}} ^ {k}\right) - \cos \left(\mathbf {e} _ {\mathrm {e n}} ^ {i}, \mathbf {e} _ {\mathrm {z h}} ^ {i}\right) + m _ {\mathrm {n c}} \right] _ {+} \right. \\ + \left[ \cos \left(\mathbf {e} _ {\mathrm {e n}} ^ {j}, \mathbf {e} _ {\mathrm {z h}} ^ {i}\right) - \cos \left(\mathbf {e} _ {\mathrm {e n}}, \mathbf {e} _ {\mathrm {z h}} ^ {i}\right) + m _ {\mathrm {n c}} \right] _ {+} \}, \\ \end{array}
$$
where  $m_{\mathrm{nc}}$  is the margin. The language neighborhood constraint loss is applied on both query and subtitle embeddings.
Overall, the final loss function of mXML  $\mathcal{L}_{\mathrm{mXML}}$  is defined as follows:
$$
\mathcal {L} _ {\mathrm {m X M L}} = \mathcal {L} ^ {\mathrm {v r}} + \lambda_ {1} \mathcal {L} ^ {\mathrm {l o c}} + \lambda_ {2} \mathcal {L} ^ {\mathrm {n c}}, \tag {17}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are the loss weights of the moment localization loss and the language neighborhood constraint loss, respectively.
Inference. At inference, in Figure 2, for the video corpus moment retrieval task, the predicted start and end probabilities are employed to generate the single video moment retrieval (SVMR) score, where SVMR is to localize a video segment from a video under the language query. Then, the video retrieval score and the SVMR score are used to produce the final VCMR score using the aggregation function proposed in (Lei et al., 2020).