SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 1: Evaluation results on the multi-modal downstream tasks. Indomain denotes MSCOCO and Visual Genome corpus while outdomain stands for Conceptual Caption and SBU corpus. Text corpus includes BookWiki and OpenWebText while image corpus contains OpenImages and unlabeled COCO. AT means using adversarial training during both pretraining and finetuning stages. Blue number denotes experiments with additional text premise input. - denotes the result is not available
Models	Pretraining corpus	Backbone	AT	Flickr30k-IR R@1 / R@5/ R@10	Flickr30k-TR R@1 / R@5/ R@10	SNLI-VE val / test	\( \mathrm{N}\mathrm{L}\mathrm{V}\mathrm{R}^{2} \) dev / test-p	VQA2.0 test-dev / -std
two-step pretraining
ViLBert (Lu et al., 2019)	Conceptual Cap.	ResNet101	x	58.20 / 84.90 / 91.52	-	-	-	70.55 / 70.92
VL-Bert (Su et al., 2020)	Conceptual Cap.	ResNet101	x	-	-	-	-	71.79 / 72.91
VisualBert (Li et al., 2019)	MSCOCO	ResNet152	x	71.33 / 84.98 / 86.51	-	-	67.40 / 67.00	70.80 / 71.00
Unicoder-VL(Li et al., 2020a)	outdomain	ResNet101	x	71.50 / 90.90 / 94.90	86.20 / 96.30 /99.00	-	-	-
LXMERT (Tan et al., 2019)	indomain	ResNet101	x	-	-	-	74.90 / 74.50	72.42 / 72.54
VLP (Zhou et al., 2021)	outdomain	ResNext101	x	-	-	-	-	70.50 / 70.70
UNITER (Chen et al., 2020)	indomain+outdomain	ResNet101	x	72.52 / 92.36 / 96.08	85.90 / 97.10 / 98.80	78.59 / 78.28	75.85/75.80	72.70 / 72.91
OSCAR (Li et al., 2020b)	indomain+outdomain	ResNet101	x	-	-	-	78.07 / 78.36	72.16 / 73.44
VILLA (Gan et al., 2020)	indomain+outdomain	ResNet101	✓	74.74 / 92.86 / 95.82	86.60 / 97.70 / 99.20	79.47 / 79.03	78.39 / 79.30	73.59 / 73.67
Ernie-ViL (Yu et al., 2020)	outdomain	ResNet101	✓	74.44 / 92.72 / 95.94	86.70 / 97.80 / 99.00	-	-	72.62 / 72.85
UNIMO (Li et al., 2021)	indomain+outdomain+ text-corpus+ image-corpus	ResNet101	✓	74.66 / 93.40 / 96.08	89.70 / 98.40 / 99.10	80.00 / 79.10	-	73.79 / 74.02
end-to-end pretraining
Pixel-Bert (Huang et al., 2020)	indomain	ResNet50	x	59.80 / 85.50 / 91.60	87.00 / 98.90 / 99.50	-	71.70 / 72.40	71.35 / 71.42
E2E-VLP (Xu et al., 2021)	indomain	ResNet101	x	-	-	-	75.23/-	72.43/-
ViLT (Kim et al., 2021)	indomain+outdomain	ViT-B	x	64.40 / 88.70 / 93.80	83.50 / 96.70 / 98.60	-	75.70 / 76.13	71.26/-
SOHO (Huang et al., 2021)	indomain	ResNet101	x	72.50 / 92.70 / 96.10	86.50 / 98.10 / 99.30	85.00 / 84.95	76.37 / 77.32	73.25 / 73.47
KD-VLP (ours)	indomain	ResNet101	x	78.20 / 94.56 / 97.02	91.40 / 98.90 / 99.40	78.21(88.18) / 77.87(88.21)	77.36 / 77.78	74.20 / 74.31
Table 2: Evaluation of image retrieval (IR) and text retrieval (TR) task on MSCOCO dataset and the performance of VCR task.
Models	Backbone	MSCOCO-IR(1K)R@1 / R@5 / R@10	MSCOCO-TR(1K)R@1 / R@5 / R@10	MSCOCO-IR(5K)R@1 / R@5 / R@10	MSCOCO-TR(5K)R@1 / R@5 / R@10	Q→A	VCRQA→R	Q→AR
two-step pretraining
Unicoder-VL(Li et al., 2020a)	ResNet101	69.70 / 93.50 / 97.20	84.30 / 97.30 / 99.30	46.70 / 76.00 / 85.30	62.30 / 87.10 / 92.80	72.60	74.50	54.40
UNITER (Chen et al., 2020)	ResNet101	-	-	50.30 / 78.50 / 87.20	64.40 / 87.40 / 93.10	74.56	77.03	57.76
OSCAR (Li et al., 2020b)	ResNet101	-	-	54.00 / 80.80 / 88.50	70.00 / 91.10 / 95.50	-	-	-
VILLA (Gan et al., 2020)	ResNet101	-	-	-	-	75.54	78.78	59.75
VL-Bert (Su et al., 2020)	ResNet101	-	-	-	-	73.80	74.40	55.20
end-to-end pretraining
Pixel-Bert (Huang et al., 2020)	ResNet50	64.10 / 91.00 /96.20	77.80 /95.40 / 98.20	41.10 / 69.70 / 80.50	53.40 / 80.40 / 88.50	-	-	-
ViLT (Kim et al., 2021)	ViT-B	-	-	42.70 / 72.90 / 83.10	61.50 / 86.30 / 92.70	-	-	-
SOHO (Huang et al., 2021)	ResNet101	73.50 / 94.50 / 97.50	85.10 / 97.40 / 99.40	50.60 / 78.00 / 86.70	66.40 / 88.20 / 93.80	-	-	-
KD-VLP (ours)	ResNet101	75.21 / 94.89 / 97.99	88.62 / 98.18 / 99.44	56.64 / 82.17 / 89.49	74.28 / 92.86 / 96.28	76.70	78.63	60.54
Implementation Details: We follow BERT to tokenize caption into word tokens by using Word-Piece, and resize the image into (800, 1333) as prior works. For model architecture, a widely-used ResNet101 for visual encoding and 12-layer Transformer for multi-modal fusion are adopted for a fair comparison. Both networks are initialized with ImageNet and BERT pretrained parameters. Besides, following the majority of two-step methods, we apply the widely-used object detector BUTD (Anderson et al., 2018) to generate object proposals as well as their RoI embeddings as our supervision.
For model learning, we optimize the entire network by using SGD for CNNs with a learning rate of 1e-2 and AdamW for Transformer with a learning rate of 1e-4, as suggested in SOHO. The training iterations are up to 100K with batch-size 512 in each. The learning rate decays 10 times at 20K, 40K respectively. All experiments are conducted on 16 NVIDIA V100 GPUs with mixed-precision training to reduce memory cost about 7 days.
As in prior works, we evaluate our approach by finetuning it over a set of well-established VL understanding tasks, including image-text retrieval, visual entailment (VE), natural language visual rea
soning  $(\mathrm{NLVR}^2)$ , VQA, and VCR. During finetuning, we compound a specific learnable head with the pretrained visual backbone and Transformer, then finetune the entire network with downstream task-specific loss in an E2E fashion. In this work, we mainly compare performance with SOHO, Pixel-Bert, E2E-VLP, and ViLT since they are the E2E pretraining as ours. Besides, several representative two-step pretraining approaches are also selected to compare without loss of generality. Next, we will depict results analysis for each task and leave finetuning experiment setups in Suppl.
Image-Text Retrieval aims retrieval an image when give a specific caption, or vice versa. As in Tab.1&2, we achieve superior performances in all evaluation settings on both datasets, especially outperforming SOHO by  $5.65\%$  and  $4.90\%$  R@1 in Flickr30k-IR/-TR,  $1.71\%$  and  $3.52\%$  R@1 in MSCOCO-IR/-TR 1K test set as well as  $6.04\%$  and  $7.88\%$  in the 5K test set. It is worth noting that we outperform the two-step pretraining SOTA approach UNIMO by a moderate margin, despite that they use additional outdomain datasets, text corpus, image collections, and adversarial training.
Visual Entailment (VE) predicts whether an image semantically entails the text and requires
Table 3: Ablation study of various proposed pretext tasks. Image-text Retrieval task is conducted on MSCOCO 1K test set.
Models	Pretext Tasks	MSCOCO-TR(1K)R@1 / R@5 / R@10	MSCOCO-IR(1K)R@1 / R@5 / R@10	SNLI-VEval / test	\( \mathrm{N L V R}^2 \)dev / test-p	VQA2.0test-dev / -std
baseline	ITM+MLM	57.99 / 87.80 / 94.66	73.10 / 93.42 / 97.32	73.44 / 73.40	62.13 / 62.08	66.62 / 66.68
-	ITM+MLM+StandardMVM	58.22 / 87.59 / 94.60	73.58 / 93.66 / 97.63	74.00 / 73.46	63.26 / 62.75	66.66 / 66.86
-	ITM+MLM+RandomMVM	58.18 / 87.12 / 94.68	73.60 / 94.80 / 97.50	73.99 / 74.58	64.02 / 64.68	66.90 / 66.05
-	ITM+MLM+OMVM	60.32 / 88.65 / 95.15	74.83 / 94.34 / 97.74	74.54 / 75.12	66.23 / 66.76	67.95 / 68.21
KD-VLP (ours)	ITM+MLM+OMVM+PRA	61.10 / 89.40 / 95.50	76.70 / 95.00 / 98.00	74.62 / 75.22	66.71 / 67.59	68.19 / 68.43
Figure 2: (a) demonstrates the comparison of different masking vision strategies where the first row presents the  $15\%$  Bert-like masking strategy adopted by all previous works and the second row shows our knowledge-guided masking strategy. Red masks denote the masked regions. (b) demonstrates a comparison of word-to-image attention maps. The bright region denotes higher attention scores between word and visual tokens.
fine-grained reasoning ability in a model. In Tab.1, we achieve we achieve  $78.21\%$  accuracy in val set and  $77.87\%$  in test set. It is worth noting that SOHO takes additional text premise as input, which leads to large improvements. For a fair comparison, we also implement that setting and outperform SOHO by a sizeable margin.
$\mathbf{NLVR}^2$  aims to determine whether a natural caption is true about a pair of photographs, which is full of semantic diversity, compositionality challenges. We outperform SOHO, Pixel-bert, ViLT and E2E-VLP by a clear margin as in Tab.1, and performs comparably with two-step pretraining.
VQA requires a richer multi-modal understanding to solve the free-form and open-ended questions. In Tab.1, the results present a clear improvement compared with E2E pretraining methods while surprisingly outperform the strong two-step pretraining methods by a slight margin.
VCR requires higher-order cognition and commonsense reasoning about the world. We achieve superior accuracy, specifically  $76.70\% / 78.63\% / 60.53\%$  in three different problem setting. It is worth noting that we set up the first end-to-end benchmark for the challenging VCR task without relying on detection during inference. Besides, we output form VL-BERT and OSCAR by a clear margin and work comparably with VILLA, which adopts advanced adversarial training and
more outdomain corpus.
Overall, our approach outperforms the previous E2E pretraining by a sizeable margin, which indicates the superiority of our object-aware E2E multimodal representation. In addition, we also performs better or comparably with previous state-of-the-art two-step pretraining, like UNIMO, VILLA, Ernie-ViL, which even adopt more outdomain corpus, sophisticated adversarial training.
4.3 Ablation Study & Visualization Analysis
In this section, we validate the effectiveness of each pretext task and provide qualitative visualization analysis. To save experimental cost, we adopt a light-weighted ResNet-18 and 3-layer Transformer network to conduct the ablation study.
Baseline: The baseline takes standard ITM and MLM to train the entire model. In Tab.3, it still achieves decent results over various VL tasks.
Object-guided masked vision modeling: As in Tab.3, compared with baseline, OMVM presents a clearly consistent improvement over all downstream tasks. It suggests that OMVM can enhance the end-to-end multi-modal representations with explicit object concepts learning. In addition, the knowledge-guided masking strategy further helps establish cross-modal correspondence.
To further investigate the OMVM task, we randomly mask a box region with  $15\%$  probability
Figure 3: Performance gains in different model size
rather than sampling a region based on the normalized similarity score  $\alpha_{z,n}$ , denoted as RandomMVM. The other pretraining details are the same as in OMVM. We observe a significant performance drop over all downstream tasks, especially in image-text retrieval and NLVR $^2$ . It indicates that simple RandomMVM will result in inefficient multi-modal representation learning because there is a high probability that the selected region has no relationship with the associated description.
In addition, we also explore the similar masked feature regression task as in UNITER by randomly masking out the image grid features as in BERT and then requiring the Transformer to reconstruct its original features rather than the external object RoI embeddings, denoted as StandardMVM. The results show that such StandardMVM fails to facilitate multi-modal representation learning in the E2E framework.
Phrase-region alignment: The OMVM above mainly focuses on instance-level knowledge distillation by absorbing external object RoI features and semantic labels. Different from that, PRA aims to establish positive object-phrase correspondence while suppressing the negative ones under the guidance of similarities between noun phrases and object labels in linguistic space. As in Tab. 3, we significantly improve  $0.78\%$  R@1 of MSCOCOTR and  $1.87\%$  in MSCOCO-IR. In addition, PRA shows slight improvements for more challenging fine-grained reasoning tasks, like VE,  $\mathrm{NLVR}^2$ , and VQA. The results indicate that PRA is beneficial to multi-modal representation learning.
Visualization analysis: In Fig.2(a), our knowledge-guided masking strategy always masks out the phrase-related image regions, which can facilitate multi-modal learning. On the contrary, previous works, like SOHO, VILLA ..., mask out background regions or part of the object region with a high probability, which have no relationship with the corresponding description
Table 4: Performance with different object detectors
Models	Detectors	Categories	NVLR-dev	VQA-test dev
SOHO	-	-	64.62	66.69
KD-VLP (ours)	FRCNN on COCO	80	65.86	67.14
KD-VLP (ours)	BUTD	1600	66.71	68.19
Table 5: Individual contribution of each pretext task
Models	Pretext Tasks	NVLR-dev	VQA-test dev
baseline	ITM+MLM	62.13	66.62
-	ITM+MLM+MRC	64.44	67.27
-	ITM+MLM+MRFR	64.23	67.36
-	ITM+MLM+PRA	63.78	67.17
KD-VLP (ours)	ITM+MLM+MRC+MRFR+PRA	66.71	68.19
and result in inefficient cross-modal alignment. Fig.2(b) demonstrates the word-to-image attention maps. Compared to SOHO, our method can attend more accurately to image regions for the corresponding word. Surprisingly, even the word "smiling" can locate the baby's face correctly, which suggests that our approach not only learns better noun-region alignment but also helps establish high-order correspondence, like actions. (see Suppl. for more visualization.)
Influence of object detector: We adopt the default BUTD detector in a typical 2-step pretraining method for a largely fair comparison. To investigate the influence of object detectors, we also conduct pretraining with objects knowledge extracted from FRCNN-RN101 pretrained on COCO. In Tab.4, we observe a performance drop compared with the model pretrained with BUTD, which suggests large object knowledge space will facilitate multimodal pretraining. Besides, although with COCO detector, we still outperform SOHO by a clear margin, indicating the superiority of object knowledge in E2E pretraining framework.
Contribution of each pretext task: In Tab.5, we show the individual contributions of our proposed tasks. MRC, MRFR, PRA pretext tasks all help facilitate multi-modal representation learning and improve the performance compared with the baseline model as a result.
Impact of object knowledge distillation in different model sizes: We take SOHO as a strong baseline and compare it at different model sizes (ResNet18 + 3-layer Transformer, ResNet101 + 12-layer Transformer) to investigate the impact of object knowledge distillation. Fig.3 demonstrates the performance gains over some representative vision-language tasks. It shows that object concepts learning always helps multi-modal representation learning no matter what model size it is. In
VE and text-retrieval, the larger model even improves significantly than the light-weighted model and shows more capacities to learn external object semantics knowledge.
In this paper, we have proposed a novel self-supervised VLP method that promotes learning object-aware multi-modal representations in an end-to-end framework. Our key idea is to perform object knowledge distillation in both semantic and feature space from external detectors in the pretraining stage. In particular, we develop an object-guided masked vision modeling task for distilling external object knowledge, and a phrase-region alignment task for learning better alignment of linguistic entities and visual concepts. Compared with prior works, we achieve competitive or superior performance without relying on sophisticated object detectors during model finetuning and test in downstream tasks.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In CVPR2015.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In ECCV2020, pages 213-229.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Learning universal image-text representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adversarial training for vision-and-language representation learning. NeuIPS2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR2016.
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. 2021. Seeing out of the box: End-to-end pre-training for vision-language representation learning. CVPR2021.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. ICML2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. In ECCV2018.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020a. Unicoder-vl: A universal encoder for vision and language by cross-modal pretraining. In AAAI2020.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. 2021. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. ACL2021.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV2020.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV2014.
Yongfei Liu, Bo Wan, Lin Ma, and Xuming He. 2021. Relation-aware instance refinement for weakly supervised visual grounding. In CVPR2021.
Yongfei Liu, Bo Wan, Xiaodan Zhu, and Xuming He. 2020. Learning cross-modal context graph for visual grounding. In AAAI.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. NeuIPS2019.
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. 2020. Visualcomet: Reasoning about the dynamic context of a still image. In ECCV2020.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer imaged-to-sentence models. In ICCV2015.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. Vl-bert: Pretraining of generic visual-linguistic representations. ICLR2020.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706.
Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, and Fei Huang. 2021. E2e-vlp: End-to-end vision-language pre-training enhanced by visual learning. ACL2021.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernievil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006.16934.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In CVPR2021.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2021. Unified vision-language pre-training for image captioning and vqa. In AAAI2020.
In this supplementary material, we firstly discuss the limitations of work, then give the detailed dataset statistics of pretraining and each downstream task, and depict more advanced implementation details of the pretraining. In addition, we
also demonstrate how to generate a binary mask for each object proposal, followed by detailed experimental setups and finetuning strategies of downstream tasks. Besides, we also discuss the influence of image size during pretraining stage. Finally, we provide more qualitative visualization results for better understanding.
In this paper, we only pretrain our proposed KDVLP framework on indomain datasets, including MSCOCO and Visual Genome caption datasets. In the future, we need to scale up our model pretrained on more noisy web image-text pairs to make it to learn more general knowledge.
As shown in Tab.6 we summarize the dataset statistics of pretraining and each downstream task, including the number of image-text pairs and number of images for each dataset split. It is worth mentioning that we select the MSCOCO & Visual Genome image-text data as our pretraining datasets since they are typical indomain datasets for many downstream tasks and are widely adopted by prior works.
Table 6: Dataset statistics of pretraining and downstream tasks. The number in brackets indicates the number of images
task	data sources	training	val	test
pretraining	MSCOCO Visual Genome	5.1M(207K)	131K(7.1K)	-
VCR	MovieClips LSMDC	213K(80K)	26.5K(9.9K)	25.2K(9.5K)
Image-text Matching	Flickr30k	145K(29K)	5K(1K)	5K(1K)
MSCOCO	567K(113.2K)	25K(5K)	25K(5K)
Visual Entailment	Flickr30k SNLI	52.9K(29.7K)	17.8K(1K)	17.9K(1K)
VQA	MSCOCO Abstract Scenes	443.8(82.8K)	214.4K(40.5K)	447.8K(81.4K)
NLVR2	Flickr30k	529.5K(29.8K)	17.9K(1K)	17.9(1K)
B.2 More Pretraining Details
In pretraining stage, we also adopt gradient accumulation<sup>3</sup> and gradient checkpointing<sup>4</sup> techniques to further reduce the GPU memory footprint and increase the batch-size. In our experiments, the gradient accumulation step size is set as 4.
B.3 Binary mask for each proposal
As shown in Fig.4, we generate a binary mask of the same size of feature map for each proposal
<sup>3</sup>https://nvidia.github.io/apex/advanced.html
4https://pytorch.org/docs/stable/checkpoint.html
Figure 4: Illustration of generating binary mask for each proposal.
where locations within the bounding box fill 1 and others fill 0.
B.4 Detailed experiment setup for each downstream task
Image-Text Retrieval: The image-text retrieval typically includes two sub-tasks: image-retrieval (IR) aims to retrieve an image when given a specific caption and text-retrieval (TR) is on the contrary. We perform experiments on both Flickr30k (Plummer et al., 2015) and MSCOCO dataset. As in UNITER, we construct a mini-batch for each GPU of a matched image-text pair,  $t - 1$  negative images, and  $t - 1$  negative texts where  $t$  is set as 32. Besides, we take a fully-connected network on top of  $\mathbf{h}_{\mathrm{cls}}$  and adopt the binary cross-entropy loss as supervision signal. The finetuning iterations are up to 10K by following linear decay scheduling with initial lr 7e-5 for Transformer, 1e-4 for CNNs. Top-K (R@K,  $K \in \{1,5,10\}$ ) recall is the evaluation metric.
Visual Entailment (VE): VE task aims to predict whether an image semantically entails the text and requires fine-grained reasoning ability in a model. VE dataset is built upon SNLI (Bowman et al., 2015) and Flickr30k. Each image-text pair is assigned with one of three classes: entailment, neutral, contradiction. As in UNITER, we formulate it as 3-way classification problem based on  $\mathbf{h}_{\mathrm{cls}}$ . The batch size is 32 per GPU while other finetuning strategies are the same.