SlowGuess/NAACL_2021_OCR huggingface dataset:
Po-Yao Huang $^{13*}$ , Mandela Patrick $^{23*}$ , Junjie Hu $^{1}$ , Graham Neubig $^{1}$ , Florian Metze $^{3}$ , Alexander Hauptmann $^{1}$
<sup>1</sup>School of Computer Science, Carnegie Mellon University
2Visual Geometry Group, University of Oxford
$^{3}$ Facebook AI
{poyaoh,junjieh,gneubig,alex} @cs.cmu.edu, {mandelapatrick,fmetze} @fb.com
This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextual multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (Multi-HowTo100M) for pre-training. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M is available at http://github.com/berniebear/Multi-HT100M
One of the key challenges at the intersection of computer vision (CV) and natural language processing (NLP) is building versatile vision-language models that not only work in English, but in all of the world's approximately 7,000 languages. Since collecting and annotating task-specific parallel multimodal data in all languages is impractical, a framework that makes vision-language models generalize across languages is highly desirable.
One technique that has shown promise to greatly improve the applicability of NLP models to new languages is zero-shot cross-lingual transfer, where models trained on a source language are applied
as-is to a different language without any additional annotated training data (Täckström et al., 2012; Klementiev et al., 2012; Cotterell and Heigold, 2017; Chen et al., 2018; Neubig and Hu, 2018). In particular, recent techniques for cross-lingual transfer have demonstrated that by performing unsupervised learning of language or translation models on many languages, followed by downstream task fine-tuning using only English annotation, models can nonetheless generalize to a non-English language (Wu and Dredze, 2019a; Lample and Conneau, 2019; Huang et al., 2019a; Artetxe et al., 2020; Hu et al., 2020). This success is attributed to the fact that many languages share a considerable amount of underlying vocabulary or structure. At the vocabulary level, languages often have words that stem from the same origin, for instance, "desk" in English and "Tisch" in German both come from the Latin "discus". At the structural level, all languages have a recursive structure, and many share traits of morphology or word order.
For cross-lingual transfer of vision-language models, the visual information is clearly an essential element. To this end, we make an important yet under-explored step to incorporate visual-textual relationships for improving multilingual models (Devlin et al., 2019; Artetxe et al., 2020). While spoken languages could be different, all humans share similar vision systems, and many visual concepts can be understood universally (Sigurdsson et al., 2020; Zhang et al., 2020). For example, while is termed "cat" for an English speaker and "chat" for a French speaker; they understand similarly. We leverage this observation to learn to associate sentences in different languages with visual concepts for promoting cross-lingual transfer of vision-language models.
In this work, we focus on multilingual text-to-video search tasks and propose a Transformer-based video-text model to learn contextual mul
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2443-2459 June 6-11, 2021. ©2021 Association for Computational Linguistics
tilingual multimodal representations. Our vanilla model yields state-of-the-art performance in multilingual text  $\rightarrow$  video search when trained with multilingual annotations. However, under the zero-shot setting, rather surprisingly, there is a significant performance gap between English and non-English queries (see §5.5 for details). To resolve this problem, motivated by recent advances in large-scale language model (Artetxe et al., 2020) and multimodal pre-training (Lu et al., 2019; Miech et al., 2019; Patrick et al., 2020), we propose a multilingual multimodal pre-training (MMP) strategy to exploit the weak supervision from large-scale multilingual text-video data. We construct the Multilingual-HowTo100M dataset, that extends the English HowTo100M (Miech et al., 2019) dataset to contain subtitles in 9 languages for 1.2 million instructional videos.
Our method has two important benefits. First, compared to pre-training on English-video data only, pre-training on multilingual text-video data exploits the additional supervision from a variety of languages, and therefore, enhances the search performance on an individual language. Second, by exploiting the visual data as an implicit "pivot" at scale, our methods learn better alignments in the multilingual multimodal embedding space (e.g., "chat"-chat"), which leads to improvement in zero-shot cross-lingual transfer (e.g., from "cat"-to "chat"-of vision-language models.
In our experiments on VTT (Xu et al., 2016) and VATEX (Wang et al., 2019), our method yields state-of-the-art English  $\rightarrow$  video search performance. For zero-shot cross-lingual transfer, the proposed multilingual multimodal pre-training improves English-video pre-training by  $2\sim 2.5$  in average R@1 across 9 languages. Additionally, when trained with in-domain multilingual annotations as other baselines, our method outperforms them by a large margin in multilingual text  $\rightarrow$  video search on VATEX and text  $\rightarrow$  image search on Multi30K (Elliott et al., 2016).
To summarize, we make the following contributions: (1) We propose a transformer-based videotext model that learns contextual multilingual multimodal representations (§3.1). (2) We empirically demonstrate that vision-language models, unlike NLP models, have limited zero-shot cross-lingual transferrability. (§5.5). (3) We introduce the multilingual multimodal pre-training strategy and construct a new Multi-HowTo100M dataset (§4) for
pre-training to improve zero-shot cross-lingual capability of vision-language models. (4) We demonstrate the effectiveness of our approach, by achieving state-of-the-art multilingual text  $\rightarrow$  video search performance in both the zero-shot (§5.5) and fully supervised setup (§5.6).
Cross-lingual representations. Early work on learning non-contextual cross-lingual representations used either parallel corpora (Gouws and Søgaard, 2015; Luong et al., 2015) or a bilingual dictionary to learn a transformation (Faruqui and Dyer, 2014; Mikolov et al., 2013). Later approaches reduced the amount of supervision using self-training (Artetxe et al., 2017). With the advances in monolingual transfer learning (McCann et al., 2017; Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2019), multilingual extensions of pre-trained encoders have been proven effective in learning deep contextual cross-lingual representations (Eriguchi et al., 2017; Lample and Conneau, 2019; Wu and Dredze, 2019b; Siddhant et al., 2020; Pires et al., 2019; Pfeiffer et al., 2020). We extend prior work to incorporate visual context.
Video-text representations. The HowTo100M dataset (Miech et al., 2019) has attracted significant interest in leveraging multimodal pre-training for text  $\rightarrow$  video search (Korbar et al., 2020), captioning (Iashin and Rahtu, 2020), and unsupervised translation via image-based (Suris et al., 2020; Huang et al., 2020b) and video-based (Sigurdsson et al., 2020) alignment. This work studies a challenging and unexplored task: Zero-shot cross-lingual transfer of vision-language models. Unlike prior image/video-text work that utilizes RNN (Dong et al., 2019; Chen et al., 2020a; Burns et al., 2020; Kim et al., 2020) and inter-modal contrastive objectives (Sigurdsson et al., 2020; Liu et al., 2019; Huang et al., 2019b; Patrick et al., 2021), we employ Transformers to learn contextual multilingual multimodal representations and uniquely models cross-lingual instances. Moreover, we build Multi-HowTo100M, the largest text-video dataset for multilingual multimodal pre-training.
Cross-lingual Transfer. Cross-lingual transfer has proven effective in many NLP tasks including dependency parsing (Schuster et al., 2019), named entity recognition (Rahimi et al., 2019), sentiment analysis (Barnes et al., 2019), document classification (Schwenk and Li, 2018), and question an
Figure 1: The proposed video-text model for learning contextual multilingual multimodal representations. We utilize intra-modal, inter-modal, and conditional cross-lingual contrastive objectives to align  $(x, v, y)$  where  $x$  and  $y$  are the captions or transcriptions in different languages of a video  $v$ . TP: Transformer pooling head.
swering (Lewis et al., 2020; Artetxe et al., 2020). Recently, XTREME (Hu et al., 2020) was proposed to evaluate the cross-lingual transfer capabilities of multilingual representations across a diverse set of NLP tasks and languages. However, a comprehensive evaluation of multilingual multimodal models on zero-shot cross-lingual transfer capabilities is still missing. To our best knowledge, we are the first work that investigates and improves zero-shot cross-lingual transfer of vision-language models.
We consider the problem of learning multilingual multimodal representations from a corpus  $\mathcal{C}$  of video-text pairs  $\{(x_i, v_i)\}_{i=1}^C$ , where  $v_i$  is a video clip and  $x_i$  is its corresponding text (caption or transcription) that is written in one of  $K$  languages. Our goal is to learn a shared multilingual text encoder  $c_x = \Phi(x)$  and a video encoder  $c_v = \Psi(v)$  both of which project the input to a shared  $D$ -dimensional embedding space  $c_v, c_t \in \mathbb{R}^D$ , where semantically similar instances (i.e., paired  $(x_i, v_i)$ ) are closer to each other than the dissimilar ones (i.e.,  $(x_i, v_j), i \neq j$ ). In the following, we denote a batch of multilingual text-video samples as  $\mathcal{B} = \{(x_i, v_i)\}_{i=1}^B$ , where  $\mathcal{B} \subset \mathcal{C}$ .
3.1 Multilingual Multimodal Transformers
Figure 1 gives an overview of the proposed method. Our text encoder consists of a multilingual Transformer (e.g. multilingual BERT (Devlin et al., 2019)) and a text Transformer pooling head (explained below). Similarly, our video encoder consists of a 3D-CNN (e.g. R(2+1)D network (Tran et al., 2018)) and a video Transformer pooling head. We use these multilingual multimodal Transformers to encode text and video for alignment.
Unlike prior multilingual text-image models (Gella et al., 2017; Kim et al., 2020; Huang
et al., 2019b) that utilize word embeddings and RNNs, our multilingual text encoder is built on a multilingual Transformer that generates contextual multilingual representations  $e_{x} \in \mathbb{R}^{N \times D}$  to encode a sentence  $x$  containing  $N$  words. We employ an additional 2-layer Transformer which we will call a "Transformer pooling head (TP)" as it serves as a pooling function to selectively encode variable-length sentences and aligns them with the corresponding visual content. We use the first output token of the second Transformer layer as the final sentence representation. Precisely, we set  $c_{x} = \mathrm{Trans}_{x}^{(2)}(\mathrm{query} = \mathrm{key} = \mathrm{value} = e_{x})[0]$  where  $\mathrm{Trans}_{x}^{(2)}$  is a 2-layer stack of Transformers (Vaswani et al., 2017) with  $e_{x}$  as the (query,key, value) in the multi-head attention. Note that we use the same text encoder to encode sentences in all languages.
For encoding videos, our model uses pre-trained 3D-CNNs that encode spatial-temporal context in a video. For a  $M$ -second video  $v$ , we apply  $\mathrm{R}(2 + 1)\mathrm{D}$  (Tran et al., 2018) and S3D (Miech et al., 2020) networks to its frames, concatenate network outputs, and apply a linear layer to encode the visual input,  $e_v \in \mathbb{R}^{M \times D}$ , to our model. Similarly to the text part, we employ a two-layer Transformer as the pooling head to encode videos with different lengths into fixed-length representations. Formally, we set  $c_v = \mathrm{Trans}_v^{(2)}$  (query=key=value=e_v)[0]. Since videos are typically long and have a high frame rate (e.g., 30 fps), it is infeasible to update 3D-CNNs simultaneously and therefore, we use pre-extracted video features. Our model is parameterized by  $\theta = \theta_{\mathrm{mBERT}} \cup \theta_{\mathrm{Trans}_x} \cup \theta_{\mathrm{Trans}_v}$ .
3.2 Multilingual Text-Video Alignment
For learning multimodal representations, the common practice is to minimize a contrastive objective to map the associated (video, text) embeddings
to be near to each other in a shared embedding space. The inter-modal max-margin triplet loss has been widely studied in video-text (Yu et al., 2018; Liu et al., 2019) and image-text (Kim et al., 2020; Burns et al., 2020; Huang et al., 2019b) research. In this work, we generalize and model all inter-modal, intra-modal, and cross-lingual instances with a noise contrastive estimation objective (NCE) (Gutmann and Hyvarinen, 2010; van den Oord et al., 2018; Chen et al., 2020b).
Inter-modal NCE. Let  $\mathcal{X}$  and  $\mathcal{V}$  denote the subsets of the sampled sentences in multiple languages and videos in  $\mathcal{B}$ , respectively. And let  $s(a, b) = \frac{a^T b}{\|a\| \|b\|}$  be the cosine similarity measure. We use an (inter-modal) NCE objective defined as:
$$
\mathcal {L} (\mathcal {X}, \mathcal {V}) = - \frac {1}{B} \sum_ {i = 1} ^ {B} \log \ell^ {\mathrm {N C E}} \left(\Phi \left(x _ {i}\right), \Psi \left(v _ {i}\right)\right), (1)
$$
$$
\ell^ {\mathrm {N C E}} \left(c _ {x}, c _ {v}\right) = \frac {e ^ {s \left(c _ {x} , c _ {v}\right)}}{e ^ {s \left(c _ {x} , c _ {v}\right)} + \sum_ {\left(x ^ {\prime} , v ^ {\prime}\right) \sim \mathcal {N}} e ^ {s \left(c _ {x ^ {\prime}} , c _ {v ^ {\prime}}\right)}} \tag {2}
$$
In inter-modal NCE,  $\mathcal{L}^{\mathrm{inter}} = \mathcal{L}(\mathcal{X},\mathcal{V})$ , the noise  $\mathcal{N}$  is a set of "negative" video-text pairs sampled to enforce the similarity of paired ones are high and and those do not are low. Following Miech et al. (2020), we set the negatives of  $(x_{i},v_{i})$  as other  $x_{j}$  and  $v_{j},j\neq i$  in  $\mathcal{B}$ .
Intuitively, inter-modal NCE draws paired (semantically similar) instances closer and pushes apart non-paired (dissimilar) instances. Note that we do not distinguish language types in  $\mathcal{X}$  and the sentences in all possible languages will be drawn towards their corresponding videos in the shared multilingual text-video embedding space.
Intra-modal NCE. Beyond cross-modality matching, we leverage the intra-modal contrastive objective to learn and preserve the underlying structure within the video and text modality. For example, Corgi should be closer to Husky than Balinese. Prior image-text work (Gella et al., 2017; Huang et al., 2019c) utilizes a triplet loss to maintain such neighborhood relationships. Inspired by recent success in self-supervised image and video representation learning (Yalniz et al., 2019; Ghadiyaram et al., 2019), our model leverages intra-modal NCE that constrains the learned representations to be invariant against noise and to maintain the within-modality structure simultaneously. We minimize
the following intra-modal NCE loss:
$$
\mathcal {L} ^ {\text {i n t r a}} = \mathcal {L} (\mathcal {X}, \mathcal {X} ^ {m}) + \mathcal {L} (\mathcal {V}, \mathcal {V} ^ {m}), \tag {3}
$$
where  $\mathcal{X}^m$  and  $\mathcal{V}^m$  are the noised version of the original sentences and videos. For noising, we randomly mask  $5\%$  of the multilingual text tokens and video clips. We optimize our model by
$$
\min  _ {\theta} \mathcal {L} ^ {\text {i n t e r}} + \mathcal {L} ^ {\text {i n t r a}} \tag {4}
$$
3.3 When Visually-Pivoted Multilingual Annotations Are Available
In many multilingual multimodal datasets, there are sentences in different languages that describe a shared visual context. For example, 10 English and 10 Chinese descriptions are available for each video in VATEX. With these visually-pivoted (weakly paralleled) sentences  $(x,y)$ , we further revise the contrastive objectives to leverage this additional supervisory signal. Given a visually-pivoted corpus  $\mathcal{C}^p$  that contains all possible combination of visually-pivoted pairs  $\{(x_i,v_i,y_i)\}_{i = 0}^{C_p}$ , we sample batches  $\mathcal{B}^p = \{(x_i,v_i,y_i)\}_{i = 1}^{\mathcal{B}^p},\mathcal{B}^p\subset \mathcal{C}^p$  and revise the contrastive objective as:
$$
\mathcal {L} ^ {\text {i n t e r}} = \mathcal {L} (\mathcal {X}, \mathcal {V}) + \mathcal {L} (\mathcal {Y}, \mathcal {V}) \tag {5}
$$
$$
\mathcal {L} ^ {\text {i n t r a}} = \mathcal {L} (\mathcal {X}, \mathcal {X} ^ {m}) + \mathcal {L} (\mathcal {Y}, \mathcal {Y} ^ {m}) + \mathcal {L} (\mathcal {V}, \mathcal {V} ^ {m}) \tag {6}
$$
Visual-pivoted Cross-lingual NCE. Inspired by Translation Language Modeling (TLM) in XLM (Lample and Conneau, 2019), we propose a multimodal TLM-like contrastive objective which promotes alignments of descriptions in different languages that describe the same video. We use the intuition that conditioned on a video, the descriptions (need not to be translation pairs) in different languages would likely be semantically similar. To this end, we set the cross-lingual NCE as:
$$
\mathcal {L} ^ {\text {c r o s s}} = \mathcal {L} (\mathcal {X} | \mathcal {V}, \mathcal {Y} | \mathcal {V}) \tag {7}
$$
For visually-pivoted sentences, as shown in Fig. 1, we generate their representations conditioned on the video they describe. We extend the key and value of multihead attention with the additional visual content  $e_v$  and generate new  $c_{x|v}$  and  $c_{y|v}$  for matching. Specifically, our model employs  $c_{x|v} = \mathrm{Trans}_x^{(2)}(\mathrm{query} = e_x, \mathrm{key} = \mathrm{value} = e_x||e_v)[0]$ . With the access to (visually-pivoted) multilingual annotations, we optimize our model by
$$
\min  _ {\theta} \mathcal {L} ^ {\text {i n t e r}} + \mathcal {L} ^ {\text {i n t r a}} + \mathcal {L} ^ {\text {c r o s s}} \tag {8}
$$
Figure 2: Video clips and the corresponding multilingual subtitles in Multi-HowTo100M.
At the inference time, we simply apply  $c_{x} = \Phi(x)$  and  $c_{v} = \Psi(v)$  to encode multilingual text queries and videos. For text-to-video search, we sort videos according to their cosine similarity scores to the text query.
4 The Multilingual HowTo100M Dataset
As large-scale pre-training has been shown important in recent NLP and vision-language models, we construct the Multilingual HowTo100M dataset (Multi-HowTo100M) to facilitate research in multilingual multimodal learning. The original HowTo100M (Miech et al., 2019) dataset is a large-scale video collection of 1.2 million instructional videos (around 138 million clips/segments) on YouTube, along with their automatic speech recognition (ASR) transcriptions as the subtitles. For each video in HowTo100M, we crawl and collect the multilingual subtitles provided by YouTube, which either consist of user-generated subtitles or those generated by Google ASR and Translate in the absence of user-generated ones. Essentially, we collect video subtitles in 9 languages: English (en), German (de), French (fr), Russian (ru), Spanish (es), Czech (cz), Swahili (sw), Chinese (zh), Vietnamese (vi).
At the time of dataset collection (May 2020), there are 1.1 million videos available, each with subtitles in 7-9 languages. The video length ranges from 1 minute to more than 20 minutes. We utilize Multi-HowTo100M for multilingual multimodal pre-training to exploit the weak supervision from large-scale multilingual text-video data. In Fig. 2, we provide a visualization of few instances sampled in Multi-HowTo100M with the corresponding video frame, timestamp, and transcriptions in different languages. Please refer to Appendix for more details and dataset statistics.
In this section, we first describe our experimental setup (§5.1-5.3). In §5.4, we conduct ablation studies to validate the effectiveness of proposed multilingual text-video model. With the best models at hand, we investigate their zero-shot cross-lingual transferability in §5.5, where we showcase that the proposed multilingual multimodal pre-training serves as the key facilitator. We then verify the superior text→video search performance of our method under the monolingual, multilingual, and cross-modality settings in §5.6.
5.1 Evaluation Datasets
MSR-VTT (VTT) (Xu et al., 2016) contains 10K videos, where each video is annotated with 20 captions. Additionally, we created pseudomultilingual data by translating the English captions into 8 languages with off-the-shelf machine translation models. We use the official training set (6.5K videos) and validation set (497 videos). We follow the protocol in Miech et al. (2019); Liu et al. (2019) which evaluates on text→video search with the 1K testing set defined by Yu et al. (2018).
VATEX (Wang et al., 2019) is a multilingual (Chinese and English) video-text dataset with 35K videos. Five  $(en,zh)$  translation pairs and five non-paired  $en$  and  $zh$  descriptions are available for each video. We use the official training split (26K videos) and follow the testing protocol in Chen et al. (2020a) to split the validation set equally into 1.5K validation and 1.5K testing videos.
Multi30K (Elliott et al., 2016) is a multilingual extension of Flickr30K (Young et al., 2014). For each image, there are two types of annotations available: (1) One parallel (English, German, French, Czech) translation pair and (2) five English and five Ger-
<sup>1</sup>https://marian-nmt.github.io/
man descriptions collected independently. The training, validation, and testing splits contain 29K, 1K, and 1K images respectively.
For the video backbone, we use a 34-layer,  $\mathrm{R}(2 + 1)$  -D (Tran et al., 2018) network pre-trained on IG65M (Ghadiyaram et al., 2019) and a S3D (Miech et al., 2020) network pre-trained on HowTo100M. We pre-extract video features and concatenate the two 3D-CNN outputs to form  $e_x\in \mathbb{R}^{M\times 1024}$  as a video input.
For the text backbone, we use multilingual BERT (mBERT) (Devlin et al., 2019) or XLM-Roberta-large (XLM-R) (Artetxe et al., 2020), where the latter achieves near SoTA zero-shot cross-lingual transfer performance for NLP tasks. Following Hu et al. (2020), instead of using the top layer, we output the 12-th layer in XLM-R and mBERT. For vision-language tasks, we freeze layers below 9 as this setup empirically performs the best.
Our model employs a 2-layer Transformer with 4-head attention for the text and video transformer pooling (TP) modules. The embedding dimension  $D$  is set to 1024. We use the Adam (Kingma and Ba, 2015) optimizer and a 0.0002 learning rate to train our model for 16 (pre-training) and 10 (finetuning) epochs. The softmax temperature in all noise contrastive objectives is set to 0.1.
We use Multi-HowTo100M for multilingual multimodal pre-training (MMP). For each video, we randomly sample the start and end time to construct a video clip. For a video clip, we randomly sample one language type each time from 9 languages and use the consecutive ASR transcriptions that are closest in time to compose (text-video) pairs for training. For simplicity and speed purposes, we follow the training protocol of XLM-R to pre-train on a multilingual corpus wihout using translation pairs, i.e., we use multilingual text-video pairs  $(x,v)$  but no translation pairs from Multi-HowTo100M and utilize only inter- and intramodal NCE (Eq. 1-3) for MMP.