SlowGuess/NAACL_2021_OCR huggingface dataset:
We fine-tune our model on VTT, VATEX, and Multi30K to evaluate on text  $\rightarrow$  video search tasks. In the zero-shot cross-lingual transfer experiments, we use only English-video data and fine-tune with Eq. 1-3. We then test the model with non-English queries. When annotations in additional languages are available (by humans in VATEX and Multi30K;
Text-B	Video-B	R@1↑	R@5↑	R@10↑
XLM-R	S3D	19.5	49.0	62.8
XLM-R	R(2+1)D	19.0	49.5	63.2
XLM-R	R+S	21.0	50.6	63.6
mBERT	R+S	19.9	49.8	62.5
Table 1: Text and Video (B)ackbone comparison.
T layers	V layers	R@1↑	R@5↑	R@10↑
1	1	20.0	50.3	63.2
2	1	20.1	50.5	63.8
2	2	21.0	50.6	63.6
2*	2*	20.7	50.5	63.3
4	4	20.8	50.4	63.8
Table 2: Architecture comparison. Number of multilingual multimodal transformer layers. *:Weight sharing between video and text transformers.
Objective	Inter	Intra	Cross	R@1↑	R@5↑	R@10↑
Triplet	✓	13.3	36.0	55.2
Triplet	✓	✓	20.9	49.3	63.0
NCE	✓	21.4	49.3	61.1
NCE	✓	✓	21.0	50.6	63.6
NCE*	✓	✓	21.3	50.7	63.5
NCE*	✓	✓	✓	21.5	51.0	63.8
Table 3: Objective comparison. *Training with additional machine translated de-video and  $fr$ -video pairs.
by MT models (i.e., translate-train) in VTT), we utilize all available multilingual annotations (i.e., fully supervised) and iterate over all possible  $(x, v, y)$  pairs to train with Eq. 5-7 to demonstrate the strong performance target for evaluating zero-shot cross-lingual transfer on VTT and to compare fairly with other fully-supervised baselines in multilingual text  $\rightarrow$  video search on VATEX and Multi30K. We report the standard recall at  $k$  ( $R@k$ ) metrics (higher is better).
5.4 Comparison Experiments and Ablations
In this section, we ablate and compare different text/video encoders, Transformer model architectures, and learning objectives for English  $\rightarrow$  video search on VTT.
Text and Video Encoders. Table 1 compares different text and video encoder backbones. For the visual encoders, while R(2+1)D outperforms S3D, the simple concatenation (i.e., early-fusion) of their output features provides a  $1.5 \sim 2.0$  improvement in R@1. For the text encoder, XLM-R significantly outperforms mBERT.
Transformer Pooling. Table 2 compares various configurations of the proposed Transformer pooling module. We observe that a simple 2-layer Transformer achieves the best performance. Weight
Model	en	de	fr	cs	zh	ru	vi	sw	es	Avg↑
mBERT	19.9	11.1	11.6	8.2	6.9	7.9	2.7	1.4	12.0	9.1
mBERT-MP	20.6	11.3	11.9	8.0	7.1	7.7	2.5	1.1	12.5	9.2
mBERT-MMP	21.8	15.0	15.8	11.2	8.4	11.0	3.7	3.4	15.1	11.7
XLM-R	21.0	16.3	17.4	16.0	14.9	15.4	7.7	5.7	17.3	14.7
XLM-R-MP	23.3	17.4	18.5	17.1	16.3	17.0	8.1	6.2	18.5	15.8
XLM-R-MMP	23.8	19.4	20.7	19.3	18.2	19.1	8.2	8.4	20.4	17.5
mBERT + translated VTT	19.6	18.2	18.0	16.9	16.2	16.5	8.4	13.0	18.5	16.1
mBERT-MMP + translated VTT	21.5	19.1	19.8	18.3	17.3	18.3	8.9	14.1	20.0	17.4
XLM-R + translated VTT	21.5	19.6	20.1	19.3	18.9	19.1	10.3	12.5	18.9	17.8
XLM-R-MMP + translated VTT	23.1	21.1	21.8	20.7	20.0	20.5	10.9	14.4	21.9	19.4
Table 4: Recall@1 of multilingual text→video search on VTT. Upper: Zero-shot cross-lingual transfer. Lower: Performance with synthesized pseudo-multilingual annotations for training. MMP: multilingual multimodal pretraining on Multi-HowTo100M. MP: Multimodal (English-Video) pre-training on HowTo100M.
(a) English  $\rightarrow$  Video
(b) Zero-shot German  $\rightarrow$  Video
Figure 3: R@1 trends in languages used for multilingual multimodal pre-training. Left: English  $\rightarrow$  video search. Right: Zero-shot German  $\rightarrow$  video search.
sharing of the video and text Transformer slightly degrades the performance. Therefore, we choose to separate them.
Learning Objective. From Table 3, the intramodal contrastive objective is important for both NCE and Triplet loss. In general, the NCE loss outperforms the Triplet loss. The proposed intermodal and intra-modal NCE objective achieves the best performance. When captions in multiple languages are available, cross-lingual NCE additionally provides a consistent improvement.
5.5 VTT Zero-Shot Cross-Linguual Transfer
Table 4 shows the multilingual text  $\rightarrow$  video search results on VTT. With the best English-video models at hand (with either mBERT or XLM-R as the text backbone), we first investigate how well these models transfer to other non-English languages under the zero-shot setting. We then analyze the benefit of the proposed multilingual multimodal pre-training.
The upper section shows the zero-shot results. Unlike cross-lingual transfer in NLP tasks, employing multilingual Transformers in vision-language tasks apparently does not generalize well across languages. For example, there is a significant drop in  $\mathbb{R}@\mathbb{1}$ $(19.9\rightarrow 11.1$ $(-44\%)$  with mBERT,
$21.0 \rightarrow 16.3$  (-24%) with XLM-R) when directly applying English-finetuned model to German  $\rightarrow$  video search. For comparison, there is only a -10% degradation for XLM-R on  $en \rightarrow de$  cross-lingual transfer in XNLI (Conneau et al., 2018). Multimodal (English-video) pre-training (MP) on HowTo100M only improves average R@1 (+0.1 or mBERT and +1.1 for XLM-R) compared to model-from-scratch. In contrast, our proposed multilingual multimodal pre-training (MMP) is shown to be the key facilitator for zero-shot cross-lingual transfer. MMP improves German  $\rightarrow$  Video search ( $11.1 \rightarrow 15.0$ , +35% for mBERT, and  $16.3 \rightarrow 19.4$ , +20% for XLM-R) and achieves 2.6 ~ 2.8 improvement in average R@1. We attribute the effectiveness of MMP to learning improved alignments between multilingual textual and visual context in the shared embedding space, as relatively balanced improvements between English  $\rightarrow$  video and non-English  $\rightarrow$  video is observed with fine-tuning.
Fig. 3 demonstrates the trend of R@1 while incrementally incorporating additional languages for MMP. For XLM-R, the improvement in R@1 asymptotically converges when pre-training with more multilingual text-video pairs. On the other hand, for zero-shot German→video search, pretraining with more languages keeps improving the search performance, even though the additional language (e.g., French) is different from the target language (i.e., German).
The lower section of Table 4 shows the results of models fine-tuned with (synthesized) pseudomultilingual annotations. It can be regarded as the translate-train scenario, which serves as a strong performance target for evaluating zero-shot cross-lingual transfer, as discussed in (Lample and Conneau, 2019; Hu et al., 2020). Both mBERT and XLM-R yield better performance across non
Figure 4: Qualitative multilingual (en, ru, vi, zh) text→video search results on VTT.
English languages with the in-domain translated pseudo-multilingual annotations. However, for English  $\rightarrow$  video search, a 0.7 degradation is observed compared to the zero-shot setting. It is likely due to the noise in the translated captions. Notably, there is still a performance gap between zero-shot and translate-train settings for models with mBERT. In contrast, the gap is much smaller for models with XLM-R. In the following sections, we refer Ours-MMP as our best model with XLM-R as the text backbone and compare it with other state-of-the-art methods.
Qualitative Results Fig. 4 shows the multilingual text  $\rightarrow$  video search results with Ours-MMP (VTT:en-only) on VTT under the zero-shot setup. Note that only one shared English-finetuned model is used for text  $\rightarrow$  video search in all languages. As demonstrated, the proposed model successfully retrieves the correct videos with English (en) and Russian (ru) queries. The other top-ranked videos also share similar visual appearance to the correct one. For zero-shot transferring of the English-finetuned model to distant languages such as Vietnamese (vi) and Chinese (zh), we observe that there is still limitation for our zero-shot models to understand abstract concepts (e.g., "space project") and associate small objects (e.g., "microphone") with the text queries in distant languages.
5.6 Comparison to Supervised State of the Art
English  $\rightarrow$  Video Search on VTT. Table 5 shows the comparison of English  $\rightarrow$  video models on VTT. For a fair comparison to other baselines, our model fine-tunes only with the original English annotations on VTT. The results show that our model outperforms other baselines by a large margin. Specifically, our model achieves  $8.9\mathrm{R}@\mathrm{l}$  improvement over the original HowTo100M model (Miech et al., 2019) and other recent baselines with pre-training on HowTo100M. Using a smaller set of visual fea
Model	R@1↑	R@5↑	R@10↑
JSFusion (Yu et al., 2018)	10.2	31.2	43.2
JPoSE (Wray et al., 2019)	14.3	38.1	53.0
VidTrans† (Korbar et al., 2020)	14.7	-	52.8
HT100M† (Miech et al., 2019)	14.9	40.2	52.8
Noise† (Amrani et al., 2020)	17.4	41.6	53.6
CE2 (Liu et al., 2019)	20.9	48.8	62.4
Ours(VTT:en-only)	21.0	50.6	63.6
Ours-MMP (VTT:en-only)	23.8	52.6	65.0
Table 5: English→video search performance on VTT. †: Models with pre-training on HowTo100M.
Model	English to Video	Chinese to Video
R@1↑	R@5↑	R10↑	R@1↑	R@5↑	R@10↑
VSE (Kiros et al., 2014)	28.0	64.3	76.9	-	-	-
VSE++ (Faghri et al., 2018)	33.7	70.1	81.0	-	-	-
Dual (Dong et al., 2019)	31.1	67.4	78.9	-	-	-
HGR (Chen et al., 2020a)	35.1	73.5	83.5	-	-	-
Ours (VATEX:en-only)	43.5	79.8	88.1	23.9	55.1	67.8
Ours-MMP (VATEX:en-only)	44.4	80.5	88.7	29.7	63.2	75.5
Ours-MMP (VATEX:en, zh)	44.3	80.7	88.9	40.5	76.4	85.9
Table 6: Multilingual text  $\rightarrow$  video search on VATEX.
tures and training on a smaller (6,513 vs 9,000) training set2, our model also outperforms CE (Liu et al., 2019) with or without pre-training.
Multilingual Text  $\rightarrow$  Video Search on VATEX. Table 6 summarizes English  $\rightarrow$  video and Chinese  $\rightarrow$  video search performance on the VATEX dataset. Under the zero-shot setting where we train with only English-video pairs, our model already outperforms other baselines. However, a clear performance gap between English  $\rightarrow$  video and Chinese  $\rightarrow$  video search is observed, indicating that cross-lingual transfer to a distant language remains challenging even with XLM-R. With the proposed MMP, the gap is significantly closed by 5.8/8.1/7.7 in R@1/5/10. When in-domain human-annotated Chinese captions are available, the performance of our model can further be improved for both languages and our model yields new state-of-the-art performance.
$^{2}$ CE uses 9,000 videos (VTT training and part of exclusive testing set) for training, while other baselines and our model in Table 5 are trained on the official VTT training set which contains 6,513 videos.
Model	M30K
# lang.	English to Image	German to Image	Czech to Image
R@1↑	R@5↑	R10↑	R@1↑	R@5↑	R@10↑	R@1↑	R@5↑	R@10↑
OE (Vendrov et al., 2015)	2	25.8	56.5	67.8	21.0	48.5	60.4	-	-	-
VSE++ (Faghri et al., 2018)	2	39.6	69.1	79.8	31.3	62.2	70.9	-	-	-
Pivot (Gella et al., 2017)	2	26.2	56.4	68.4	22.5	49.3	61.7	-	-	-
FB-NMT (Huang et al., 2020a)	2	47.3	75.4	83.5	37.0	64.0	73.1	-	-	-
MULE (Kim et al., 2020)	4	42.2	72.2	81.8	35.1	64.6	75.3	37.5	64.6	74.8
SMALR (Burns et al., 2020)	10	41.8	72.4	82.1	36.9	65.4	75.4	36.7	68.0	78.2
MHA-D (Huang et al., 2019b)	2	50.1	78.1	85.7	40.3	70.1	79.0	-	-	-
Ours (M30K:en-only)	1	48.4	78.3	85.9	31.4	61.1	72.6	33.2	65.2	76.1
Ours-MMP (M30K:en-only)	1	50.0	79.2	86.8	33.8	63.3	74.7	37.9	68.8	78.2
Ours-MMP (M30K:en, de, cs, fr)	4	51.6	80.1	87.3	45.1	75.6	85.0	46.6	75.9	83.4
Table 7: Multilingual text→image search on Multi30K. MMP: Multilingual multimodal pre-training.
Cross-Modality Transfer to Multi30K: From Video-Text to Image-Text. To extend our study on zero-shot cross-lingual transfer for image-text tasks, we investigate the feasibility of transferring our video-text model across modalities. We replace the 3D-CNN in the original video-text model with a 2D-CNN to encode the image. In practice, following MHA-D (Huang et al., 2019b), we utilize the Faster-RCNN (Ren et al., 2015) pre-trained in Visual Genome (Krishna et al., 2016) to extract regional visual features. Essentially, an image is encoded as  $e_v = \mathbb{R}^{M \times H}$  where  $M = 36$  is the maximum number of visual objects in an image. For models with MMP, we initialize their weights with the model pre-trained on Multi-HowTo100M. To tackle the feature mismatch between 2D-CNN and 3D-CNN, we leverage a linear layer with a doubled learning rate to map 2D-CNN features to the same dimension as 3D-CNN features.
Table 7 shows the results on Multi30K. For zero-shot cross-lingual transfer, when trained from scratch (M30K:en-only), our model achieves comparable performance to MHA-D but lags in German→image search since it only uses English annotations. In Ours-MMP, pre-training improves all recall metrics even with modality gap. The average R@1 improvement is 3.2. A larger gain for (relatively) low-resource language such as Czech is observed. Without using any Czech annotations, our zero-shot model with MMP achieves comparable Czech→image search performance to SMALR (Burns et al., 2020), which uses 10 languages including Czech. However, when transferring across modalities and using only English annotations, there are performance gaps between English→Image and German/Czech→Image search, implying that transferring models across modalities is feasible but remains challenging. We consider zero-shot cross-modal cross-lingual transfer as our future work.
For a fair comparison with other baselines, when trained with annotations in all 4 languages provided by Multi30K, our model greatly outperforms all baselines by large margins in multilingual text  $\rightarrow$  image search.
We have presented a multilingual multimodal pretraining (MMP) strategy, the Multi-HowTo100M dataset, and a Transformer-based text-video model for learning contextual multilingual multimodal representations. The results in this paper have convincingly demonstrated that MMP is an essential ingredient for zero-shot cross-lingual transfer of vision-language models. Meanwhile, there are many remaining challenges, such as resolving the performance gap between zero-shot and training with in-domain non-English annotations; as well as techniques to transfer varieties of vision-language models (e.g., VQA (Goyal et al., 2017), TVQA (Lei et al., 2020)) or visually-enhanced NLP models such as unsupervised multimodal machine translation (Huang et al., 2020b). We believe the proposed methodology, and the corresponding resources we release, will be an important first step towards spurring more research in this direction.
This work is supported by the DARPA grants funded under the AIDA program (FA8750-18-2-0018) and the GAILA program (award HR00111990063) (P.Y.). This work is also supported by EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems [EP/L015897/1] (M.P.). The authors appreciate Prahal Arora, Shengxin Zha, Polina Kuznetsova, Xu Hu, and Geoffrey Zweig for their suggestions of this work. The authors would also like to thank the anonymous reviewers for their feedback.
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Ivan Laptev, Josef Sivic, and Simon Lacoste-Julien. 2016. Unsupervised learning from narrated instruction videos. In Computer Vision and Pattern Recognition (CVPR).
Elad Amrani, Rami Ben-Ari, Daniel Rotman, and Alex Bronstein. 2020. Noise estimation using density estimation for self-supervised multimodal learning. arXiv preprint arXiv:2003.03186.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451-462, Vancouver, Canada. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online. Association for Computational Linguistics.
Jeremy Barnes, Lilja Øvrelid, and Erik Velldal. 2019. Sentiment analysis is not solved! assessing and probing sentiment classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 12-23, Florence, Italy. Association for Computational Linguistics.
Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan A. Plummer. 2020. Learning to scale multilingual representations for vision-language tasks. In The European Conference on Computer Vision (ECCV).
David Chen and William Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 190-200, Portland, Oregon, USA. Association for Computational Linguistics.
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. 2020a. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b. A simple framework for contrastive learning of visual representations. In ICML.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557-570.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Ryan Cotterell and Georg Heigold. 2017. Crosslingual character-level neural morphological tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 748-759, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. 2019. Dual encoding for zero-example video retrieval. In CVPR.
Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70-74. Association for Computational Linguistics.
Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 72-78, Vancouver, Canada. Association for Computational Linguistics.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. Vse++: Improving visual-semantic embeddings with hard negatives. In BMVC.
Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 462-471, Gothenburg, Sweden. Association for Computational Linguistics.
Spandana Gella, Rico Sennrich, Frank Keller, and Mirella Lapata. 2017. Image pivoting for learning multilingual multimodal representations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2839-2845. Association for Computational Linguistics.
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. 2019. Large-scale weakly-supervised pre-training for video action recognition. In CVPR.
Stephan Gouws and Anders Søgaard. 2015. Simple task-specific bilingual word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1386-1390, Denver, Colorado. Association for Computational Linguistics.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR).
Michael Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. CoRR, abs/2003.11080.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019a. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. arXiv preprint arXiv:1909.00964.
Po-Yao Huang, Xiaojun Chang, and Alexander Hauptmann. 2019b. Multi-head attention with diversity for learning grounded multilingual multimodal representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1461-1467, Hong Kong, China. Association for Computational Linguistics.
Po-Yao Huang, Xiaojun Chang, Alexander Hauptmann, and Eduard Hovy. 2020a. Forward and backward multimodal nmt for improved monolingual and multilingual cross-modal retrieval. In Proceedings of the 2020 International Conference on Multimedia Retrieval, ICMR '20, page 53-62, New York, NY, USA. Association for Computing Machinery.
Po-Yao Huang, Junjie Hu, Xiaojun Chang, and Alexander Hauptmann. 2020b. Unsupervised multimodal neural machine translation with pseudo visual pivoting. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8226-8237, Online. Association for Computational Linguistics.