SlowGuess/NAACL_2021_OCR huggingface dataset:
Choice of Layers to Freeze Similar to output layers, the choice of frozen layers greatly affects cross-lingual transferability. For both mBERT and XLM-R, it is desirable to freeze part of the lower layers and make the top-3 layers trainable for video-text models. We observe that when freezing all layers (i.e., using the pre-extracted contextual multilingual embeddings) does not lead to satisfactory results. For mBERT, R@1 drops from 19.9 to 18.9 in English→video search and 11.1 to 9.8 in German→video search. For XLM-R, R@1 drops from 21.0 to 18.9 in English→video search and 16.3 to 14.1 in German→video search. These results imply that text-only contextual multilingual embeddings along are likely to be infeasible to be applied to vision-language tasks without proper fine-tuning.
An important observation is that the best English  $\rightarrow$  video search performance corresponds to the best German  $\rightarrow$  video performance. This trend implies that for model selection, the configuration for the best English  $\rightarrow$  video model usually translates to the best configuration for (zero-shot) cross-lingual model. This shared trend justifies the English  $\rightarrow$  video ablation studies in the original paper. Note that we utilize the best English  $\rightarrow$  video for all (zero-shot) cross-lingual experiment in our experiment section.
For multilingual text  $\rightarrow$  video search, the best configuration we found in our experiments is to output the 12-th layer and freeze the layers below 9 for both mBERT and XLM-R.
The coverage of our text  $\rightarrow$  video search experiments is summarized in Table 11. Our experiments cover the following scenarios:
In-domain, English: Table 5 (VTT) and Table 6 (VATEX) in the original paper.
In-domain, non-English: Table 4 (VTT, 9 languages) and Table 6 (VATEX, Chinese).
Out-of-domain, English: Additional (zero-shot) generalization results across datasets are in §E.1.
Out-of-domain, non-English: We consider this as our future work.
Table 11: Coverage of our experiments
Model	R@1	R@5	R@10
VSE (Kiros et al., 2014)	10.1	29.4	41.5
VSE++ (Faghri et al., 2018)	14.4	35.7	46.9
Dual (Dong et al., 2019)	13.7	36.1	48.2
HGR (Chen et al., 2020a)	16.4	38.3	49.8
Ours-Full	24.0	50.5	62.1
Table 12: Zero-shot generalization on YouTube2Text with VTT-finetuned model.
E.1 Generalizability across English-Video Datasets
In this section, we provide additional experiment results regarding zero-shot generalization of the VTT-finetuned model on out-of-domain dataset. Specifically, we test on YouTube2Text (Chen and Dolan, 2011). The aim of this experiment is to test the cross-dataset generalizability of our model without using domain-specific training data.
Table 12 shows the comparison of English  $\rightarrow$  video search results on the YouTube2Text testing set. Models in this table are only fine-tuned on VTT and use no YouTube2Text training data. As can be observed, our model with MMP generalizes well on YouTube2Text, outperforming HGR (Chen et al., 2020a) by 7.6 and DualEncoder (Dong et al., 2019) by 10.3 in R@1.