UNITER: Learning UNiversal Image-TExt Representations
Authors: Yen-Chun Chen
Linjie Li
Licheng Yu
Ahmed El Kholy
Faisal Ahmed
Zhe Gan
Yu Cheng
Jingjing Liu
Venue:  (2020)
Keywords: Self-supervised Representation Learning
Large-scale Pre-training
Vision and Language

Abstract: Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design three pre-training tasks: Masked Language Modeling (MLM), Image-Text Matching (ITM), and Masked Region Modeling (MRM, with three variants). Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use Conditioned Masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking. We also conduct a thorough ablation study to find an optimal combination of pre-training tasks for UNITER. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks over nine datasets, including Visual Question Answering, Image-Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR2. 

Under review as a conference paper at ICLR 2020
UNITER: LEARNING UNIVERSAL IMAGE-TEX REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual un- derstanding. In this paper, we introduce UNITER, UNiversal Image-TExa Rep- resentation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design three pre-training tasks: Masked Language Modeling (MLM), Image- Text Matching (TTM), and Masked Region Modeling (MRM, with three variants), Different from concurrent work on multimodal pre-training that apply joint ran- dom masking to both modalities, we use conditioned masking on pre-traie pre-training tasks (i.e., masked language/region modeling is conditioned full observation of image/text). Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking. We also conduct thorough ab- lation study to find an optimal setting for the combination of pre-training tasks. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks (over nine datasets), including Visual Question Answering, Image-Ter Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reason- ing, Visual Entailment, and NLVR?.
INTRODUCTION
Most Vision-and-I Language tasks rely on joint multimodel embeddings to bridge the semantic gap between visual and textual clues in images and text, although such representations are usually tailored for specific tasks. For example, MCB (Fukui et al. 2017). BAN (Kim et al. 2018))
DFAF (Gao et al. 2019) proposed advanced multimodal fusion methods for Visual Question An- swering (VQA) (Antol et al. 2015). SCAN (Lee et al. 2018) and MAtNet (Yu et al. 2018) studied learning latent alignment between words and image regions for Image- Text Retrieval (Wang et al. 2016) and Referring Expression Comprehension (Kazemzadeh et al.) (2014) tasks. While each of these proposed models has pushed the state of the art on respective benchmarks, their architectures diverse and the learned representations are highly task-specific, preventing them from being gen- eralized to other tasks. This raises a million-dollar question: can we learn a universal image-text representation for all V+L tasks?
To answer this question, we introduce UNiversal Image- Representations (UNITER). large- scale pre-trained model for multimodal embedding. We adopt Transformer Vaswani et al. 2017) as the core of our model, to leverage its elegant self-attention mechanism designed for learning con- textualized representations. Inspired by BERT (Devlin et al., 2019)) which has successfully applied Transformer to NL.P tasks through large-scale language modeling, we pre-train UNITER through three pre-training tasks: (4) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iit) joint Image-Text Matching (ITM). To fur- ther investigate the effectiveness of MRM, we propose three MRM variants: (i) Masked Region Classification (MRC); (ii) Masked Region Feature Regression (MRFR); and (Mii Masked Region Classification with KL-divergence (MRC-kI).
As shown in Figure D UNITER first encodes image regions (visual features and bounding box features) and textual words (tokens and positions) into a common embedding space with Image Embedder and Text Embedder, then applies Transformer module to learn generalizable contex-

---

Under review as a conference paper at ICLR 2020
UNITER Model
Image Embedder
Text Embedder
mage Feature
Transformer
FC
Emb
Embb
R.CNN
Location
Token
Postion
man
with
his dog
couch
UNITER
UNITER
UNITER
man with [MASK]
man with dog
[CLS] the
Masked Language Modeling (MLM) Masked Region Modeling (MRM)
Image-Text Matching (ITM)
Figure 1: Overview of the proposed UNITER model (best viewed in color), consisting of an Image Embedder, a Text Embedder and multi-layer self attention Transformer, learned through three pre-training tasks.
tualized embeddings for each region and word through aforementioned pre-training tasks. Com- pared with LXMERT (Tan & Bansal 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized represen- tations for image regions and textual words through single Transformer. Besides, our masked language/region modeling is conditioned on full observation image/text, different from other con- current pre-trained models that apply joint random masking to both modalities. We show that the conditional masking strategy can successfully case the cüising--alignnen between images and text, and obtain better joint embeddings for downstream tasks. Detailed ablation study also demonstrates yields the best pre-training performance.
that the combination of MLM=ITM-MRCC---MRE
To demonstrate the power of UNITER, evaluate on six V+L tasks across nine datasets, including: (i) VOA; (ii) Visual Commonsense Reasoning (VCR) (Zellers et al. 2019); (iti) NLVR (Suhr
al., 2019): (fv) Visual Entailment (Xie al. 2019); (v) Image-Text Retrieval (including zero-shot
setting) (Lee et al. 2018); and (vi) Referring Expression Comprehension. Our UNITER model is trained on large-scale V+L dataset composed of four subsets: (i) COCO (Lin al., 2014): (ii)
Visual Genome (VG) (Krishna et (017)2 (iii) Conceptual Captions (CC) (Sharma et al. (218);; and (ir) SBU Captions (Ordonez et al. 201T). Experiments show that UNITER achieves new state of the art with significant performance boost across all six downstream tasks. Moreover, training on additional CC and SBU data (containing unseen images/text in downstream tasks) further boosts model performance over training on COCO and VG only.
(i) We introduce UNITER, a powerful UNiversal Image-TExt Representations for lision-and-Langgag tasks. (ii) We achieve new state of the art (SOTA) on multiple V+L benchmarks, outperforming existing SOTA and concurrent multimodal pre-training methods by large margin. (iii) We present extensive experiments and analysis to provide useful insights on the effectiveness of each pre-training task/dataset for multimodal encoder training.
Our contributions can be summarized as follows:
RELATED WORK
Self-supervised learning utilizes original data its own source of supervision, which has been applied to many Computer Vision tasks, such as image colorization (Zhang et al., 2016). solving jigsaw puzzles (Noroozi Favaro 2016 Trinh et al. 2019). inpainting (Pathak et (0166). ro- tation prediction (Gidaris et al.] 2013 8), and relative location prediction (Doersch et al. 2015). Re- cently, pre-trained language models such as EL.Mo (Peters et al.) 2018), BERT (Devlin al. 2019). GPT2 (Radford et al. 2019), and XLNet (Yang et al.) (019) have shown great advances for NLP tasks. There are two keys to their success: effective pre-training tasks over large language corpus, and the use of Transformer (Vaswani et al. 2017) for learning contextualized text representations.
More recently, there has been some concurrent work on self-supervised learning for multimodal tasks, by pre-training large-scale image/video and text pairs, then finetuming downstream tasks. For example, VideoBERT (Sun et al.) (2019) applied BERT to learn a bidrectional joint distribution over quantized video frame features and linguistic tokens from video-text pairs. ViLBERT (Lu
2

---

Under review as a conference paper at ICLR 2020
et al. 2019) and LXMERT (Tan & Bansal 2019) introduced the two-stream architecture, where two Transformers are applied to images and text independently. which will be fused by a third Transformer in later stage. On the other hand, VisualBERT (Li et al. 2019b), Unicoder- VL (Li et al. 2019a). VL-BERTT (Su et al., 2019) and B2T2 (Alberti et al. 2019) proposed the single- stream architecture, where single Transformer is applied to both image and text. Specifically, LXMERT model was pre-trained with downstream tasks such as VQA (Antol et al.] 2015) and
GQA (Hudson & Manning, 2019), while the others were pre-trained on image-text pairs only. Our UNITER model belongs to the second family. One key difference between UNITER and the other methods the masking approach on pre-training tasks. Instead of randomly masking both image regions and sentence words, we use conditional masking, i.e., masking only one modality while keeping the other untainted. In addition, we examine the best combination of pre-training tasks through a thorough ablation study on the effects of each pre-training task and dataset on downstream tasks.
Another related work is DDAF (Gao al. 2019 which proposed novel architecture of inter- modality and intra-modality attention modules to learn the latent alignment between two modalities for VQA. Compared with Gao et al. (2019), UNITER learns a relatively more generic V+L repre- sentation via pre-training.
3 UNIVERSAL IMAGE-TENT REPRESENTATIONS
In this section, we first introduce the model architecture of UNITER (Section 3.1, J. then describe the designed pre-training tasks and V+L datasets used for pre-training (Section 3.2 and 3.3
3.1 MODEL OVERVIEW
The model architecture of UNITER is illustrated in Figure Given a pair of image and sentence, UNITER takes the visual regions of the image and textual tokens of the sentence as the input. We design Image Embedder and Text Embedder extract their respective embeddings. These embeddings are then feed it into a multi-layer self-attention Transformer to learn a cross-modality contextualized embedding between visual regions and textual tokens. Note that the self-attention mechanism in Transformer is order-less, thus it is necessary to explicitly encode positions/locations of tokens/regions as additional inputs.
Specifically, in Image Embedder, we first use Faster R- R-CNN to extract the visual features (pooled ROI features) for each region. We also encode the location features for each region via 7- dimensional v Both visual and location features are then fed through fully-connected (FC) layer, be projected into the same embedding space. The final visual embedding for each region obtained by summing up the two FC outputs and then passing through a layer normalization (LN) layer. For Text Embedder, we follow BERT (Devlin et al., 2019) and tokenize the input sentence into WordPieces (Wu et al. (2016). The final representation for each sub-word toker is obtained via summing up its word embedding and position embedding, followed by another LN layer
We introduce three main tasks pre-train model: Masked Language Modeling conditioned on image regions (MLM), Masked Region Modeling conditioned on input text (with three variants) (MRM), and Image-Test maxte-Text Matching (ITM). As shown in Figure our MRM and MLM are in analogy to BERT, where we randomly mask some words or regions from the input and learn to recover the words regions the output of Transformer. Specifically, word masking is realized by replacing the token with a special token [MASK], and region masking is implemented by replacing the visual feature vector with all zeros. Note that each time we only mask one modality while keeping the other modality intact, instead of randomly masking both modalities like ViLBERT and LXMERT. This prevents potential aliggmenn when a masked region happens to be described by a masked word. Empirically, we show that with conditional masking, our model is able to learn better embeddings
'Our Faster R-CNN was pre-trained Visual Genome object+atrribte data Anderson et al. 2018 2 (pr,, yı, 22, V2, w. h w * h] (normalized top/left/bottoom/rgg coordinates, width, height, and arca.)
We use word/sub-word and token interchangeably throughout the rest of the paper.
"We also use a special modality embedding to help the model distinguish between textual and visual input, which is similar to the 'segment embedding' in BERT. This embedding is also summed before the LN layer in each emhedder. For simplicity, this modality embedding omitted in Figure T
3

---

Under review as a conference paper at ICLR 2020
(in Section Lastly, we also learn instance-level alignment (rather than token/region-level) between the whole image and the sentence via ITM. During training, we sample both positive and negative image-sentence pairs and learn their matching scores.
To pre-train UNITER with the aforementioned different tasks, we randomly sample one pre-training task for each mini-batch and train on only one objective per SGD update.
3.2 PRE-TRAINING TASKS
Masked Language Modeling (MLM) We denote the image regions as V = {ui,.., uk} }. the input words w {w1, ..., WT), }. and the mask indices as m N [ out the input words with probability of 15%, and replace the masked ones Wm with special token [MASK]] D The goal is to predict these masked words based on the observation of their surrounding words Wim and all image regions v. by minimizing the negative loglliklihhodd:
In MLM, we randomly mask
MLMM (0) = - (w, log Po Po(Wml [ v).
(1)
where 0 is the trainable parameters. Each pair (w, is sampled from the whole training set D.
Image-Text Matching (ITM) In ITM, an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities. The inputs ITM are a sentence and a set of image regions, and the output is a binary label (0 for negative match, and for positive match). We extract the representation of [CLS] token as the joint representation of the input text and image, then fed into fully connected layer and a sigmoid function to predict score between O and 1. We denote the output score as sp(w, v). The ITM supervision is over the [CLS] ] token During training, we sample positive negative pair (w, v) from the dataset D at each step. The negative pair is created by replacing the image or text in a paired sample with a randomly- one from other samples. We denote the label y E {0, 1}. indicating if the sampled pair is match. Then we apply a binary cross cross-entropp entropy loss for optimization:
(ITM(O) - -((,y)>Dly log so(v, w) + (1 y) log(1 - so(v, w))l).
Masked Region Modeling (MRM) Similar to MLM, we also sample image regions and mask their visual features with a probability of 15%. The model is trained to reconstruct the masked regions Vm given the remaining regions Vam and all the words w. The visual features Um of the masked region are replaced by zeros. Unlike textual tokens that are represented as discrete labels, visual features are high nigh-dimensional and continuous. thus cannot be supervised via class likelihood. Instead, we propose three variants for Masked Region Modeling, which share the same objective base:
LMRM (0) = (((..)>>DDe fo Vim, w).
1) Masked Region Feature Regression (MRFR) MRFR learns to regress the Transformer output of each masked region vm to its visual features. Specifically, we apply an FC layer to convert its Transformer output into vector ha ho("))) of same dimension the input ROI pooled feature r(vi)) (+)
Then we apply L2 regression between the two: (6(vm/V..m w) = ΣM =1 [l/a(v - r(v (1) II2.
2) Masked Region Classification (MRC) MRC learns predict the object semantic class for each masked region. We first feed the Transformer output of the masked region v(i into an FC layer to predict the scores of K object classes, which further goes through a softmax function to be transformed into normalized distribution go (V(()) R.K Note that there is no ground-truie truth label, as the object categories are not provided. Thus, we use the object detection output from Faster R- CNN, and take the detected object category (with the highest confidence score) as the label of the
masked region, which will be converted into a one-hot vector c(vi))) (t) E RRK. The final objective minimizes the cross-entropy (CE) loss:
w) = EM CE(c(vl8) (8) (o (v))).
'N is the natural numbers, M is the number of masked tokens, and m is the set of masked indices. FFllowing BERT, we decompose this 15% into 10% random word, 10% unchanged, and 80% [MASK]. 'The supervision over the [CLS] token in pretraining also alleviates the input mismatch problem between pretraining tasks and downstream finetuning tasks, since most of the downstream tasks take the representation of [CLS] token as the joint representation.
4

---

Under review as a conference paper at ICLR 2020
In-domain
Out-of-domain
Split
COCO Captions
VG Dense Captions
Conceptual Captions
SBU Captions
train
533K (106K)
5.06M (TOTK)
3.0M (3.0M)
990K (990K)
val
25K (5K)
106K (2.1K)
14K (14K)
10K (10K)
Table 1: Statisties on datasets used for pre-training. Each cell shows #image-text pairs (#images).
Task
Datasets
Image Src.
#lmages
#Text
Metric
VQA
VQA
COCO
204K
1.IM
VQA-score
2
VCR
VCR
Movie Clips
110K
290K
Accuracy
3
NLVR
NLVR
Web Crawled
214K
107K
Accuracy
4
Visual Entailment
SNLI-VE
Flickr30K
31K
507K
Accuracy
COCO
COCO
92K
460K
5
Image-Text Retrieval
Recall @ 1,5,10
Flickr30K
Flickr30K
32K
160K
RefCOCo
20K
142K
6 RE Comprehension
RefCOCO+
COCO
20K
142K
Accuracy
RefCOCOg
26K
95K
Table 2: Statistics on the datasets of downstrearn tasks.
3) Masked Region Classification with KL-Divergence (MRC-kI) MRC takes the most likely object class from the object detection model as the hard label (w.p. 0 or 1), which assumes the detected object class is the ground-truth label for the region. However, this may not be true, as no ground-truth label provided for the detected region. Thus, MRC-kI, we avoid this assumption by using soft label as supervision signal, which is the raw output from the detec- tor (i.e., distribution of object classes o(v(!)) )) MRC-kl aims distill such knowledge into UNITER as Hinton et al.] (2015). by minimizing the KL divergence between two distributions: fe Mym w) Σn DKL (o( m (lgg)llo )v))))
3.3 PRE-TRAINING DATASETS
We construct our pre-training dataset based on four existing V+L datasets: COCO (Lin al.) 2014). Visual Genome (VG) Krishna et al. 2017), Conceptual Captions (CC) (Sharma et al. 2018), and SBU Captions (Oroneez et al. 2011). D. Only image and sentence pairs are used for pre-trainee pre-training purpose, which makes the model framework more scalable, as additional image-sentence pairs are easy to harvest for further pre-training.
To study the effects of different datasets on pre-training, we divide the four datasets into two cat- egories. The first one consists of image captioning data from COCO and dense captioning data from VG. We call it "In-domain" data, as most V+L tasks are built on top of these two datasets. To obtain a 'air'' data split, we merge the raw training and validation splits from COCO, and exclude all validation and test images that appear in downstream tasks. We also exclude all co-occurring Flickr30K (Plummer al. 2015) images via URL matching, as both COCO and Flickr30K images were crawled from Flickr and may have o The same rule was applied to Visual Genome as well. this way, we obtain 5.6M image-text pairs for training and 131K image-text pairs for our internal validation, which is half the size of the dataset used in LXMERT (Tan & Bansal 2019), due to the filtering of overlapping images and the use of image-text pairs only. We also use additional Oft-of-domain data from Conceptual Captions (Sharma et al. 2018) and SBU Captions (Ordonez ct al. 2011) for model training The statistics on the cleaned splits are provided in Table
EXPERIMENTS
We evaluate UNITER on six V+L tasks (listed in Table 2) by transferring the pre-trained model to each target task and finetuning through end-to-end training. We report experimental results on two model sizes: UNITER-base with 12 layers and LargeR-large with 24 layers 10
'A total of 222 images were eliminated through this process.
WW apply the same URL matching method, excluding 109 images from the training set. JUNITTER-bae base: L=12, H=768, A=12, Total Parameters=86M UNITER- large: L-24, H=1024. A=16, Total 3AMMMeters=303M (I.: number of stacked Transformer blocks; H: hidden activation dimension; A: number of attention heads). 882 and 3645 V100 GPU hours were used for pre-training UNITER-base and UNITER-large.
5

---

Under review as a conference paper at ICLR 2020
TR (Plickr)
Ref-
Pre-training Data
Pre-training Tasks
Meta-Sum
VQA
NLVR
(Flckr)
COCO+
lest-dev
val
val
dev
val
None
1
None
314.34
67.03
61.74
65.55
51.02
68.73
Wikipedia
MLM only)
346.24
69.39
73.92
83.27
50.86
68.80
BookCorpus
3
MRFR
344.66
69.02
72.10
82.91
52.16
68.47
4
ITM
385.29
70.04
78.93
89.91
74.08
72.33
MLM
386.10
71.29
77.88
89.25
74.79
72.89
393.04
71.55
81.64
91.12
75.98
72.75
In-domain
393.97
71.46
81.39
91.45
76.18
73.49
(COCO+VG)
396,24
71.73
81.76
92.31
76.21
74.23
MLM ITM MRCAT
397.09
71.63
82.10
92.57
76.28
74.51
MLM ITM MRC.I MRER
399,97
71.92
83.73
92.87
76.93
74.52
MLM ITM MRC-K MRER
396.51
71.68
82.31
92.08
76.15
74.29
(w<o cned. mask) MRFR
Out-af-domaal
395.45
71.47
83.10
92.21
75.58
73.09
(SBU+CC)
In-domain
MIM ITM MRCAT MRFR
402.50
72.27
84.68
93.69
77.14
74.72
Out-of domain
Table 3:
Evaluation pre-training tasks and datasets using VQA, Image-Text Retrieval on Flickr30K, NLVR2, and RefCOCO+ benchmarks. All results obtained from UNITER-bass Averages of R@ R@5 and R@ on Flickr30K for Image Retrieval (IR) Text Retrieval (TR) are reported. Dark and light grey colors highlight the top and second best results across all the tasks trained with In-domain data.
4.1 DOWNSTREAM TASKS
In VQA, VCR and NLVR² tasks, given an input image (or a pair of images) and natural language question (or description), the model predicts answer (or judges the correctness of the descrip- tion) based on the visual content in the image. For Visual Entailment, we evaluate on the SNLI-VE dataset. The goal is to predict whether given image semantically entails an input sentence. Classifi- cation accuracy over three classes (" ("Entailment", "Neutral" and "Contradiction") is used to measure model performance. For Image-Text Retrieval, we consider
datasets (COCO and Flickr30K) and evaluate the model in two settings: Image Retrieval (IR) and Text Retrieval (TR). Referring Expression (RE) Comprehension requires the model select the target from a set of image re- gion proposals given the query description. Models are evaluated on both ground-truth objects and detected proposal [ (MAuNet Yu al. 2018)).
For VQA, VCR, NLVR?, Visual Entailment and Image-Tex: Retrieval, we extract the joint embed- ding of the input image-text pairs via a multi-layer perceptron (MLP) from the representation of the [CLS] token. For RE Comprehension, we use the MLP to compute the region-wise alignment scores. These MLP layers are learned during the finetuning stage. Specifically, we formulate VQA, VCR, NLVR? Visual Entailment and RE Comprehension as classification problems and minimize the cross-entropy loss over the ground-truth answers/responses. For Image- Text Retrieval, we for- mulate it ranking problem. During finetuning, we sample three pairs of image and text, one positive pair from the dataset and two negative pairs by randomly replacing its sentence/image with others. We compute the similarity scores (based on the joint embedding) for both positive and neg- ative pairs, and maximize the margin between them through triplet loss.
4.2 EVALUATION ON PRE-TRAINING TASKS
We analyze the effectiveness of different pre-training settings through ablation studies over VQA, NLVR², Flickr30K and RefCOCO+ as representative V+L benchmarks. addition to standard metrics for each benchmark (listed in Table 2). we also use Meta-Sum (sum of all the scores across all the benchmarks) as a global metric.
Firstly, we establish two baselines: Line (L1) in Table g indicates no pre-training is involved, and L2 shows the results from MLM initialized with pre-traindd weights from Devlin et al. (2019). Although MLM trained text only did not absorb any image information during pre-training, see a gain of approximately +30 on Meta-Sum over L.1. Hence, we use the pre-trained weights in 1.2 to initialize model for the following experiments.
The evaluation splits of RE comprehension using detected proposals denoted as val". test". etc.

---

Under review as a conference paper at ICLR 2020
Unicoder
UNITER
Tasks
SOTA
ViLBERT
VLBERT
VisuaBBERT
LXMERT
-VL
BASE
LARGE
lest-dev
70.63
70.35
70.50
70.80
72.42
72.27
73.24
VQA
test-std
70.90
70.92
70.83
71.00
72.54
72.46
73.40
Q->A
72.60
73.30
74.00
71.60
75.00
77.30
VCR
QA-RR
75.70
74.60
74.80
73.20
77.20
80.80
Q >R
55.00
54.80
55.50
52.40
58.20
62.80
NIVR
dev
54.80
67.40
74,50
77.14
78.40
test-P
53.50
67.00
74.50
77.87
79.50
SNLI
val
71.56
78.56
79.28
VE
test
71.16
78.02
78.98
R@
31.86
42.40
62.34
65.82
7S IR
R@5
61.12
71.80
(Plickr)
R@
81.50
93.52
R
48.60
58,20
68.30
71.50
73.66
IR
R@5
77.70
84.90
90.30
91.16
93.06
(Rlckr)
R@ 10
85.20
91.52
94.60
95.20
95.98
R@
38 60
44,50
48, 48.42
51.72
IR (COCO)
R@5
69.30
74.40
76.68
78.41
R@ 10
80:40
84.00
85, 90
86.93
R
61.60
75.
77.50
ZS TR (Flickr)
96.30
R@
90.10
95.50
98.50
R@l
67.90
82.30
84.70
88.20
TR (Flick))
R@s
90.30
95.10
97.10
98,40
R@ 10
95 80
97.80
99.00
99.00
R
50.40
59.60
63.28
66.60
R@5
82,20
85. 15.10
87.04
89.42
(COCO)
R @
90.00
91.80
93.08
94.26
87.51
91.64
91.84
89.02
92.26
Ref-
testB
87.05
90.46
91.19
COCO
val"
81.24
testA" 4
83.37
86.48
87.04
testB
70.32
73.94
74.17
val
75.38
78.44
82.84
84.04
testA
80.04
81.30
85.70
85.87
Ref-
71.18
COCO+
val
68.19 18.19
72.34
71.84
74.72
74.94
testA" o
75.97
78.52
77.59
80.65
81.37
te
57.52
62.61
60.57
65.15
65.35
val
81.76
86.52
87.85
Ref-
test
81.75
86.52
87.73
COCOg
val" 68.22
74.31
74.86
test"
69.46
74.51
75.77
Table 4: Results on downstream V+L tasks from UNITER model, compared with task specifcc saate-of-the-ane (SOTA) and concurrent pre-trained models. ZS: Zero-Shot, IR: Image Retrieval and TR: Text Retrieval.
Secondly, validate the effectiveness of each pre pre-training task through thorough ablation study. Comparing 1.2 and L3, MRFR (L3) achieves better results than MLM (L2) only on NLVR On the other hand, when pre-trained on ITM (L4) MIM (L.5) only, observe significant improve- ment across all the tasks over L1 and L2 baselines. When combining different pre-training tasks, MLM + ITM (L6) improves over single ITM (L4) or MLM (L5). When MLM, ITM and MRM are jointly trained (L7-L10), we observe consistent performance gain across all the benchmarks, Among the three variants of MRM (L7-L99), we observe that MRC-kl (L99 achieves the best performance (397.09) when combined with MLM + ITM, while MRC (L7) the worst (393.97). When combin- ing MRC-kl MRFR together with MLM and ITM (L10), we find that they are complimentary to each other, which leads to the highest Meta-Sum score. We use this as the optimal pre-training setting for further experiments.
Additionally, validate the contributions of conditional masking through comparison study. When we perform random masking on both modalities simultancously during pre-training, i.c., w/o conditional masking (L11), observe decrease in Meta-Sum score (396.51) compared to that with conditional masking (399.97). This indicates that the conditional masking strategy enables the model learn better joint image-text representations effectively.
Lastly, we study the effects pre- te-irriinng datasets. Our experiments so far have been focused on In-domain data. In this study, pre-train model Out-of-domain data (Conceptual Captions

---

Under review as a conference paper at ICLR 2020
SBU Captions). A performance drop (395.45 in L12) from the model trained on In-domain data (COCO + Visual Genome) (399.97 in L10) shows that although Out-of-domain data contain more images, the model still benefits more from being exposed to similar downstream images during pre- training. We further pre-train our model on both In-domain and Out-of-domain data. With doubled data size, the model continues to improve (402.50 in L.13).
4.3 RESULTS ON DOWNSTREAM TASKS
Table presents the results of UNITER on all downstream tasks. Both our base and large mod- els are pre-trained on HLM-IIM-MRC-L-MREE
Ooat-n--ttt-o--o-oooee datasets, with the optimal pre-training setting: The implementation details of each task are provided in Appendix A.2 We compare with both task-specific models and concurrent pre- treitri models on each down- stream task. SOTA task-specific models include: MCAN (Yu et al. (019) for VQA, MaxEnt (Suhr) ct al. 2019) for NLVR², B2T2 (Alberti et al. 2019) for VCR, SCAN (Lee et al. 2018) for Image- Text Retrieval, EVE-Image (Xie et al. 2019) for SNLI-VE, and MAttNet for RE Comprehension (ReíCOCO, RefCOCO+ and ReiCOCOg
Concurrent pre-trained models include: ViLBERT, LXMERT, Unicoder- VL, VisualBERT and VLBERT.
Results show that our UNITER- -lrge model achieves new state of the art across all the benchmarks. UNITER- base model also outperforms the others by a large margin across all tasks except VQA. model outperforms SOTA by approximately +2.8% for VCR on Q-AR, +2.5% for NLVR +7% for SNLI-VE, +4% on R@ 1 for Image-Text Retrieval (+15% for zero-shot setting), and -2% for RE Comprehension.
Specifically, our UNITER-bar
Note that LXMERT pre-trains with downstream VQA (+VG+GQA) data, which may help adapt the model VQA task. However, when evaluated on unseen tasks such as NLVR², UNITER-base achieves 3% gain over LXMERT. In addition, among all the models pre-trained on image-text pairs only, our UNITER- outperforms the others by >1.5% on VQA.
It is also worth mentioning that both VilBERT and LXMERT observed two-stream model out- performs single-stream model, while our results show empirically that with our pre-training set- ting. single-stream model can achieve new state-of-the-- art results, with much fewer parameters (UNITER- base: 86M, LXMERT: 183M, VilBERT: 221M
For VCR, we propose two-stage pre-training approach: (i) pre-train on standard pre-training datasets; and then (ii) pre-train on downstream VCR dataset. Interestingly, while VLBERT and B2T2 observed that pre-training is not very helpful on VCR, we find that the second-stage pre- training can significantly boost model performance, while the first-stage pre-training still helps but with limited effects (results shown in Table This indicates that the proposed two-stage approach is highly effective in our pre-trained model over new data that are unseen in pre- pre-training datasets.
Different from other tasks, NLVR² takes two images as input. Thus, directly finetuning UNITER pre-trained with image-sentence pairs might not lead to optimal performance, as the interactions between paired images not learned during the pre-training stage. Thus, experimented with three modified settings on 2LVR²: (f) Triplet: joint embedding of images pairs and query captions; (ii) Pair: individual embedding of each image and each query caption; and (iii) Pair-biattn: bidirectional attention is added to the Pair model to learn the interactions between the paired images.
Comparison results are presented in Table 6 Triplel setting even without cross-attention between the image pairs. We hypothesize that it is due to the fact that our UNITER is pre-trained with image-text pairs. Thus, it is difficult to finetune a pair- based pre-trained model on triplet input. The bidirectional attention mechanism in the Pair-biatth setting, however, compensates the lack of cross-attention between images, hence yielding the best performance with large margin. This show that with minimal surgery on the top layer of UNITER, our pre-trained model can adapt to new tasks that are very different from pre-training tasks.
The Pair setting achieves better performance than the
12 MAttNett results are updated using the same features as the others. More details are provided in Appendix. The word embedding layer contains excessive rare words, thus excluded from the parameter counts.
8

---

Under review as a conference paper at ICLR 2020
Stage
Stage
Q-+A
QA-
Q - AR
Setting
dev
test-P
N
N
72.44
73.71
53.52
Triplct
72.76
73.55
N
Y
73.52
75.34
55.6
Pair
75.37
75.97
Y
N
72.83
75.25
54.94
Pair-bialta
77.14
77.87
74.56
77.03
57.76
Table 6: Experiments on three modified settings for NLVR?. All models use pre-trained UNITER-base.
Table 5: Experiments two-stage pre- pre-training for VCR. Results are from UNITER-bar on VCR split. Stage I and Stage denote first-stage and second-stage pre-training.
5 CONCLUSION
In this paper, we present UNITER, large-scale pre-trained model providing UNiversal Image- TExt Representations for Vision-and-Languaal tasks. Three main pre-training tasks are proposed and evaluated through extensive ablation studies. Trained with both in-domain and out-of-domain datasets, UNITER outperforms ofatc-of-the-art models over multiple V+L tasks by a significant margin. Future work includes studying early interaction between raw image pixels and sentence tokens, as well as developing more effective pre-training tasks.
REFERENCES
Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. In EMNLP, 2019.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In CVPR, 2018.
14
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In ICCV, 2015. 1
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-traine Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019,
3
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In EMNLP, 2017. [
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In CVPR, 2019.
13
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018. 2
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint MXXi:155030023/1 2015.
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 3
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In CVPR, 2015. 13
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014. !
Jin Hwa Kim, Jachyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. In NeurIPS, 2018.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. In EMNLP, 2019. 14

---

Under review as a conference paper at ICLR 2020
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
Kuang-Huee Hueei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, 2018.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder
for vision and language by cross-modal pre-training. arXiv preprint arXiv: (e0.11988000666 2019a. 3
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint urXiv:I908.08:: 2019b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 5
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
12
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin-
guistic representations for ssion-and-languaal tasks. In NeurlPS, 2019.
2 16 17
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw
puzzles. In ECCV, 2016. 2
Vicente Ordonez, Girish Kulkarni, and Tamara L Berg. Im2text: Describing images using million captioned photographs. In NeurlPS, 2011. 2
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation.
WMT, 2018. 12
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in
pytorch. 2017. [2]
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016. 2
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACI, 2018. 2
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to-phree correspondences for richer image- to-sentence models. In ICCV, 2015. 5
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic linguisin representations. arXiv preprint arXiv: 1etp::90800... 2019, 3
Alane Suhr and Yoav Artzi. NIvr2 visual bias analysis. arXiv preprint arXiv: 104/1, 2019,
Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. ACL, 2019.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid, Videobert: A joint model for video and language representation learning. In ICCV, 2019. _
Hao Tan and Mohit Bansal. Lxmert: Learning modallty encoder representations from trans- formers. In EMNLP, 2019.
16
17
10

---

Under review as a conference paper at ICLR 2020
Trieu H Trinh, Minh-Thang Luong, and Quoc V Le. Selfie: Sepf-supe pretraining for image embedding. arXiv preprint 19X.::9906002440,, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
2 13
Liwei Wang, Yin Li, and Svetlana Lazebnik, Learning deep structure- image-text embed dings. In CVPR, 2016. [
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao. Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint
160.v: 2016. 3
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint 1xXx::1901006766 2019,
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:/906.0083 2019. 2
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- met: Modular attention network for referring expression comprehension. In CVPR, 2018. T
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In CVPR, 2019.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, 2019, 2
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 DATASET COLLECTION
As introduced, our full dataset composed of four existing V+L datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions. The dataset collection is not simply combining them, as we need to make sure none of the downstream evaluation images are seen during pre-training. Among them, COCO is the most tricky one to clean, as several downstream tasks are built based on it. Figure 2 CoCo/eefcocO+/Reca
lists the splits from VQA, Image-Text Retrieval, COCO Captioning, Ref- and the bottom-up top-down (BUTD) detection (Anderson et al. 2018). all from COCO images.
As observed, the validation and test splits of different tasks are scattered across the raw COCO splits. Therefore, we exclude all those evaluation images that appeared in the downstream tasks. In addition, we also exclude co-oceurring Flickr30K images via URL matching, making sure the zero-shot image-text retrieval evaluation on Flickr is fair. The remaining images become the COCO subset within our full dataset, as shown in Figure bottom row. We apply the same rules to Visual Genome, Conceptual Captions, and SBU Captions.
(raw)
VOA
train
lest
Img-Txt Retrieval
train
train
vol
test
Img Captioning
train
train
val
test
test
Re()
BUTD
Irain
train
val
test
UNITER
train
train
val
Figure 2: Different data splits from downstream tasks based on COCO images. Our UNITER pre- training avoids seeing any downstream evaluation images.
A.2 IMPLEMENTATION DETAILS
Our models are implemented based on PyToret use Nvidia Apex T3 for mixed precision training. All pre-training experiments are run on Nvidia V100 GPUs (16GB VRAM; PCIe connection). Finetuning experiments are implemented on the same hardware Titan RTX GPUs (24GB VRAM). To further speed up training, we implement dynamic sequence length to reduce padding and batch examples by number input units (text tokens + image regions). For large pre-training experiments, multi-node communications (on TCP connections through ethernet) with up to nodes of 4x V100 server. Gradient accumulation (Ott et al. 2018) is also applied to reduce multi-GPU communication overheads.
14 Paszke al. 2017). speed up training, we
use Horovod + NCCL for
Visual Question Answering (VQA) We follow Yu et al. (011)) to take 3129 most frequent an- swers as answer candidates, and assign a soft target score each candidate based its relevancy to the 10 human responses. To finetune on VQA dataset, we use binary cross-entropy loss to train a multi-label classifier using batch size of 102400 input units over maximum 5K steps. We use AdamW optimizer (Loshchilov & Hutter 2019) with learning rate of 3e 4 and weight decay of 0.01. At inference time, the max-probable answer is selected as the predicted answer. For results on test-dev and test-std splits, both training and validation sets are used for training, and additional question-answer pairs from Visual Genome are used for data augmentation as Yu et al. 2019).
14 "http://pytorchhoor//
hhttps//gthuub.cmm/////DAAA
"hatps//gihuub.co///oooooodd 17 "ban:://gghhuhccom///AAAAAAAeel
12

---

triplet) with a batch size of 4096 input units over Under review as a conference paper at ICLR 2020
Visual Commonsense Reasoning (VCR) VCR can be decomposed into two multiple-choice sub- tasks: question-answent task (Q - A) and answer ciaseerjustfficttion task (QA - R). In the holistic setting (Q + AR), model needs to first choose an answer from the answer choices, then select supporting rationale from rationale choices if the chosen answer is correct. We train our model in two settings simultaneously. When testing in the holistic setting, we first apply the model predict an answer, then obtain the rationale from the same model based on the given question and the predicted answer. To finetune on VCR dataset, we concatenate the question (the qeustion and the ground truth answer) and each answer (rationale) choice from the four possible answer (rationale) candidates. The 'modality embedding"
extended help distinguish question, answer and rationale. Cross- entropy loss is used to train classifier over two classes ( 'right'' ' or 'wrong' ') for each question-answer pair (question-answer-raae maximum 5K steps. We use Adam W optimizer with learning rate of le 4 and weight decay of 0.01.
Since the images and text in VCR dataset are very different from our pre-training dataset, we further pre-train our model on VCR, using MLM, MRFR and MRC-kI as the pre-training tasks. ITM is discarded because the text in VCR does not explicitly deseribe the image. The results of both pre- trainings on VCR are reported in Table and and discussed in the main text. In conclusion, for down- stream tasks that contain new data which is very different from the pre-training datasets, second- stage pre-training helps further boost the performance.
In our implementation, the second-stage pre-training is implemented with batch size of 4096 intput units, a learning rate of 3e - 4 and weight decay of 0.01 over maximum 60K steps. After second- stage pre-traing, we finetune our model with a learning rate of 6e 5 over maximum 8K steps.
Natural Language for Visual Reasoning for Real (NLVR2) NLVR2 is new challenging task for visual reasoning. The goal is to determine whether a natural language statement is true about the given image pair. Here we discuss the three architecture variants of NLVR² finetuning in detail. Since UNITER only handles one image and one text input at pre-training, the 'modality embedding' extended to help distinguish the additional image presented in the NLVR² task. For the Triplet setup, we concatenate the image regions and then feed into the UNITER model. An MLP transform applied on the [CLS] output for binary classification. For the Pair setup, we treat one input example as two text-image pairs by repeating the text. The two [CLS] outputs from UNITER are then depth concatenated as the joint embedding for the example, Another MLP further transform this embedding for the final classification. For the Rair-biattn setup, the input format is the same as the Pair setup. As for the joint representation. instead of rely on only two [CLS] J outputs, we apply multi-head attention layer (Vaswani et al. (017) one sequence of joint image-text embeddings to attend to the other sequence of embeddings, and vice versa. After this 'bidirectional" attention interactions, a simple attentional pooling is applied on each output sequences and then final con- cat+MLP layer transforms the cross-attended joint representation for true/false classification.
We finetune UNITER on NLVR for 8K steps with a batch size of 10K input units. AdamW opti- mizer is used with learning rate of le - 4 and weight decay of 0.01.
Image-Texx Text Retrieval Two datasets are considered for this task: COCO and Flickr30K. COCO consists of 123K images, each accompanied with five human-written captions. We follow Karpa- thy & Fei-Fei (2015) to split the data into 82K/5K/5K training/validttionnees images. Additional 30K images from MSCOCO validation set are also included to improve training as in Lee et al. 2018). Flickr30K dataset contains 31K images collected from the Flickr website, with five textual descriptions per image. We follow Karpathy & Fei-Feil (2015) to split the data into 30K/ training/validatiioneest splits. During finetuning, we sample two negative image-text pairs per posi- tive sample from image and text sides, respectively. For COCO, we use batch size of 60 examples, learning rate of 2e- and finetune our model for 20K steps. For Flickr30K, we finetune model with a batch size of 120 examples and a learning rate of 5e - 5 over maximum 16K steps.
To obtain the final results in Table 4 we further sample hard negatives to facilitate the finetuning. For every N steps, we randomly sample 128 negative images per text input and obtain a sparse scoring matrix for the whole training set. For each image, sentences as hard negative samples. Similarly, we get 20 hard negative images for each sentence according to their scores. The hard negatives are sent to the model additional negative samples.
choose the top 20 ranked negative
13

---

Under review as a conference paper at ICLR 2020
I0sons
regions
takens
regions
(a) Vertical
(b) Diagonal
(c) Vertical + Diagonal
(d) Block
(c) Heterogeneous
(f) Reversed Block
Figure 3: Attention heatmaps UNITER
In the end, we have two randomly sampled negatives and two hard negative samples per positive sample. N is set to 4000 for COCO and 2500 for Flickr30K.
Visual Entailment (SNLI-VE) Visual Entailment is task derived from Flickr30K images and Stanford Natural Language Inference (SNLI) dataset, where the goal is to determine the logical relationship between natural language statement and an image. Similar to BERT for Natural Language Inference (NLI), we treat SNLI-VE as a three-way classification problem and apply an MLP Transform on [CLS] output. The UNITER model finetuned using cross-entropy loss. The batch size is set to 10K input units and we use A with learning rate of Se 5 to train for 3K steps.
Referring Expression Comprehension We use three referring expression datasets: RefCOCO, RefCOCO+, and RefCOCOg for the evaluation, all collected on COCO images. To finetune UNITER on this task, we add MI.P layer on top of the region outputs from Transformer, com- pute the alignment score between the query phrasc/sentence and each region. Since only object is paired with the query phrase/sentence, we apply cross-entropy loss on the normalized alignment scores. The finetuning is efficient we train the model with batch size of 64 examples and learning rate of 5 for only 5 epochs, and achieve state-of-the-as performance.
Note all works including Genome) to extract the visual features. While this does not affect other downstream tasks, it raises an issue for RE comprehension, are a subset of COCO's training split. Strictly, our object detector is not allowed to train with these val/test images. However, just for "fair" comparison with concurrent works, we ignore this is- sue and use the same features (Anderson et al. 2018) as the others. We also update the results of MAttNet using this "contaminated" features, whose accuracy is 1.5% higher than the original one. As aforementioned, the interaction between sentence and image could start from tokens and pixels instead of the extracted features. We leave this study and RE comprehension with strictly correct features to future work.
off-the-shelf object detectors trained COCO (and Visual
the val/test images of RefCOCO, RefCOCO+, and RefCOCOg
A.3 VISUALIZATION
Similar to Kovaleva et al. 2019), we observe several patterns in the attention maps of the UNITER model, shown in Fig. Note that different from Kovaleva et al. (2019), our attention mechanism
14

---

Under review as a conference paper at ICLR 2020
A girl with a striped shirt is sitting on a bicycle
girl
striped
shirt
sitting
bicycle
Figure Text-to-image attention visualization example.
A man and child working a puzzle
puzzle
Figure 5: Text-to-image attention visualization example.
An army officer is inspect ##lingg something
army
something
Figure Text-to-image attention visualization example.
operates both inter- and intra-modalitiy manners. For completeness, we briefly discuss each
pattern here:
Vertical: attention special tokens [CLS] or [SEP]: ]:
Diagonal: attention to the token/region itself or freceding/followingg tokens/regions;
Vertical + Diagonal: mixture of vertical and diagonal;
Block: modra-modality attention, i.e., textual self-attention and visual self-attention;
Heterogeneous: diverse attentions that cannot be categorized and is highly dependent actual input:
Reversed Block: inter-modality attention, i.e., text-to-image and image-to-text attention.
shows cross cross-modee modality alignment between tokens and regions. In Fig. 5 and 6 we visualize several examples of text-to-image attention to demonstrate the local cross-modality alignment between regions and tokens.
Note that Reversed Block (Fig. 31
15

---

Under review as a conference paper at ICLR 2020
(a) Conditional Masking
(b) Joint Random Masking
man with his <MAKK> and sitting on the sofa
Figure 7: Example showing difference between conditional masking and joint random masking.
MLM on COCO
MLM on VG
0.80
0.70
0.79
0.68
Val. Accuracy
Val. Accuracy
0.78
0.66
0.77
0.76
pretrain condi. masking
pretrain
0.62
condi. masking
0.75
joint rand. masking
joint rand. masking
0.60
100K
200K
300K
400K
500K
103K
200K
300K
400K
500K
Iter
Iter
(a) Validation accuracy of ML.M COCO and VG datasets
MRC-kl on COCO
MRC-kl on VG
0.725
0.700
0,700
Val. Accuracy
0.675
Val. Accuracy
0.675
0.650
0.625
0.625
0.600
pretrain condi. masking
0.600
pretrain
condi. masking
0.575
0.575
joint masking
joint masking
0.550
0.550
100K
200K
300K
400K
500K
0
100K
200K
300K
400K
500K
Iter
Iter
(b) Validation accuracy MRC-kl on COCO datasets.
Figure& 8: Comparison of MLM and MRC-kl validation accuracy using joint masking and our pro- posed conditional masking.
A.4 CONDITIONAL MASKING VS. JOINT RANDOM MASKING
We further discuss the advantage of our proposed conditional masking over joint random masking used [Tan & Bansal 2019 Lu et al., 2019). Intuitively, our conditional masking learns better latent alignment of entities (regions and words) across two modalities. Fig. 7 shows example image with "man with his dog and cat sitting on a sofa". With conditional masking, when the region model should be able infer that the region is dog. based the context
of dog is masked, of both surrounding regions and the full sentence (Fig. Πa)), and vice versa. However, for the joint masking implementation, it could happen when both the region of dog and the word dog
16

---

(Su et al. 2019). and our Under review as a conference paper at ICLR 2020
Model
Q-+A
QA-R R
Q AR
VLBERT- large (single)
75.8
78.4
59.7
ViLBERT (10 ensemble)
76.4
78.0
59.8
UNITER-large (single)
77.3
80.8
62.8
UNITER- large (10 ensemble)
79.8
83.4
66.8
Table 7: VCR results from VLBERT (Su et al.
2019
ViLBERT Lu 2019), and UNITER.
Model VisualBERT 67.3
Balanced
Unbalanced
Overall
Consistency
68.2
67.3
26.9
LXMERT
76.6
76.5
76.2
42.1
UNITER- -arge
80.0
81.2
80.4
50.8
Table 8: NLVR results test-U split from VisualBERT Li et al. UNITER.
2019b
LXMERT
Tan Bansal 2019
masked (Fig. (b)).. (Fi.. In such case, the model has to make the prediction blindly, which might lead to mis-alignment.
To verify this intuition, show the validation curves during pre-training of MLM and MRC-kl in Fig. 8 Each sub-figure shows a comparison between applying conditional masking and joint random masking during the pre-training of UNITER. The MLM accuracy measures how well UNITER can reconstruct the masked words, and MRC-kl accuracy
18 measures how well UNITER can classify the masked regions. In both cases, shown in Fig. 8 our conditional masking converges faster and achieves higher final accuracy than joint random masking. In addition, Table 3 (row 10 & 11) shows our conditional masking also performs better on fine-tuned downstream tasks.
A.5 MORE RESULTS ON VCR AND NLVR2
Following the VCR setup in Table. we further construct an ensemble model using 10 UNITER- large. Table. shows the comparison between VLBERT, ViLBERT and UNITER on VCR. The
Q - AR accuracy of our ensemble model outperforms ViLBERT (Lu et al. 2019) ensemble by a large margin of 7.0%. Note even single UNITER- already outperforms ViLBERT ensemble and VLBERT-large by 3.0%.
Besides, we also compare our UNITER-large with LXMERT (Tan & Bansal 2019) and Visual-
BERT (Li et al. 2019b) on an additional testing split of NLVR in Table. Our results consistently outperform the previous SOTA all metricy!900 by large margin of ~4.0%%
A.6 DIRECT COMPARISON TO VLBERT AND VILBERT
VQA
RefCOCO+ (det)
test-dev
val
testA
testB
ViLBERT 70.55
72.34
78.52
62.61
VLBERT-bass
71.16
71.60
77.72
60.99
UNITER- base
71.22
72.49
79.36
63.65
Table 9: A direct comparison between ViLBERT (Lu et al.
2019
VLBERT
UNITER, all trained on Conceptual Captions (Sharma et al. 2018) only.
To further demonstrate our idea, we conduct a direct comparison to ViLBERT [u al. 2019) and
VLBERT (Su et al.] 2019). trained on Conceptual Captions (Sharma et al.) 2018). We pre-train
UNITER on Conceptual Captions only (instead of datasets (.3) using our proposed conditional masking and the best pre-training tasks (MLM ITM + MRC-KI + MRFR). Table. shows that
Whhen validating on MRC-kl accuracy, we simply pick the most confident category from the predicted probability and measure its correctness. The balanced and unbalanced evaluations were introduced in Suhr & Artzi 2019
17

---

Under review as a conference paper at ICLR 2020
UNITER still consistently outperforms the other models by a visible margin VQA and Ref- COCO+
18


Reviews:
[{"review_id": "SygtFAYaKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# 1. Summary\nThe authors introduce a new pre-training procedure for image-text representations. The idea is to train the model on a huge collection of different image-text datasets and the use the model for downstream tasks. The difference between the proposal wrt the concurrent work is that conditioned masking is used: (i) Masked Language Modeling (MLM) conditioned on image; (ii) Masked Region Modeling (MRM) conditioned on text; and (iii) joint Image-Text Matching (ITM).\n\nI am on the fence for this paper given the balance between strengths and  weaknesses listed below. I am conservative here and decide for weak reject; but I am open for discussion, if the authors answer to my concerns detailed below. \n\nStrengths:\n* State-of-the-art results on several downstream vision-language tasks\n* Empirical work to investigate different ways to perform conditioned masking \n      \nWeaknesses:\n* Some parts of the method needs clarification (see point 2 below) to better understand the details and practical advantages of the method. \n* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)\n     \n      \n# 2. Clarity and Motivation\nThe paper reads quite well, although some points need to be improved:\n* \"Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized ...\", why is this an advantage? Using two streams might also lead to learning context? Maybe an example can clarify my question.\n* End of Sec. 3.1 (and paragraph in Sec. 3.2): not clear how the model is training for ITM. What's the input and output? Why do you need a new symbol [CLS]?\n* Sec. 3.2 ITM: \"an additional special token [CLS] is fed into our model, which indicates the fused representation of both modalities\" - This is not clear. Why this special token is needed? Why is not needed in the MLM and MRM?\n* \"The scoring function is denoted as s\" -> please indicate in the main text what function you used\n* MRFM and MRC are clear, however the intuition of MRC-kl is missing. Why is this needed? What does it mean in practice to minimize such divergence (provide practical example)?\n* Combination of tasks (MLM + ITM + MRC-kl + MRFR) -> it is not clear how this is done in practice. Is the loss function composed (summed)? Within the mini-batch, the method randomly chooses which operation to do (e.g., MLM) for each sample? This should be clarified in the main text of the paper.\n  \n\n# 3. Novelty\nThe novelty of the paper is quite limited since it is an extension of BERT to the visual domain. The authors propose an empirical analysis of different ways to mask the visual input, however this might not be a substantial extension of previous work. In fact, recently there are many other papers (ViLBERT, VisualBERT, LXBERT, ...) working on similar topic with small differences. What it is missing in this paper is an understanding and intuition on the reasons why the conditioned masking idea should be better than the other visual masking ideas proposed in previous work.\n\n\n# 4. Experimentation\nThe main advantage of this paper relies on the extensive experimental analysis done on many challenging datasets reaching the state of the art on several downstream tasks.\nThe evaluation on both pre-training tasks and downstream tasks show that the method is working well in practice.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1e6rHd6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel method for image-text representations called UNITER. The proposed method has been subsequently tested in many downstream tasks. A detailed ablation study helps to understand the role of each pretrained task in the proposed model.\n\nAlthough the empirical results are nice, performing the intensive set of experiments on many different tasks is definitely time-consuming and needs a lot of engineering efforts, the technical contribution does not seem significant to me. The paper modifies an existing pre-training procedure by conditional masking (Section 2). I agree this is well-motivated but it has little novelty and a similar idea is there in VQA (See “Dynamic fusion with intra and inter-modality attention flow for visual question answering”). MLM and MRM are not new training procedure either, they are basically extending the BERT’s training procedure with the consideration of multiple modalities.\n\nI have some questions for the authors:\n(1) What are the advantages of using single-stream transformer over two-stream transformer (page 2). I guess it leads to fewer parameters but I don’t think this is a big problem.\n(2) Some visualization of attention weights would be helpful. \nMinor\n•\tIn “m \\e N^M” (equation 1), what is N and M? \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkeMz3WRFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an impressive paper. LIke BERT, it proposes a tranformer based approach to derive a pre-trained network for representing images and texts. The resulting pre-trained network, used in 9 different tasks, advances the SOTA on all the tasks. \nThe major limitation of this paper is why. Why does it happen? How this results can be achieved? What is exactly represented in this pre-trained network. Why the tasks used for pre-training build a network that is so informative?\nThis is really the major obscure point of this impressive paper.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes an approach to pre-train general-purpose image and text representations that can be effective on target tasks requiring embeddings for both modes. The authors propose several pre-training tasks beyond masked language modelling that are more suitable for the cross-modal context being addressed, and also investigate which dataset/pretraining task combinations are effective for given target tasks.

All reviewers agree that the empirical results that were achieved were impressive.

Shared points of concern were:
- the novelty of the proposed pre-training schemes.
- the lack of insight into the results that were obtained.

These concerns were insufficiently addressed after the discussion period, particularly the limited novelty. Given the remaining concerns and the number of strong submissions to ICLR, this submission, while promising, does not meet the bar for acceptance.

Decision: Reject