VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Authors: Weijie Su
Xizhou Zhu
Yue Cao
Bin Li
Lewei Lu
Furu Wei
Jifeng Dai
Venue:  (2020)
Keywords: Visual-Linguistic
Generic Representation
Pre-training

Abstract: We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark.

Published as a conference paper at ICLR 2020
VL-BERT: PRE-TRAINING OF GENERIC VISUAL- LINGUISTIC REPRESENTATIONS
Weijie Su Xizhou Zhu Zhu Yue Cao², Bin Li', Lewei Lu², Furu Wei', Jifeng Dai 'University of Science and Technology of China
Microsoft Research Asia
{jackroos, ezra0408 08) @mmail 1. ustc. edu. on, binl SeiinniQuee weutttustcee.. edu. en (yuecao, lewlu, fuwei, , ji fdai /emicrosoft.com
ABSTRACT
We introduce new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VI-BERT for short). VL.-BERT adopts the sim- ple yet powerful Transformer model the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the in- put is either of word from the input sentence, or a region-of-interest (Rol) from the input image. It is designed to fit for most of the risual-linguistie downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Ex- tensive empirical analysis demonstrates that the pre-training procedure can bet- ter align the risual-linguise clues and benefit the downstream tasks, such visual commonsense reasoning, visual question answering and referring expres- sion comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at hattps://gitee com fack/ /VL-BERT
INTRODUCTION
Pre-training of generic feature representations applicable to a variety of tasks in a domain is a hall- mark of the success of deep networks. Firstly in computer vision, backbone networks designed for
and pre-trained ImageNet (Deng et al., (009) classification are found to be effective for improv- ing numerous image recognition tasks. Recently in natural language processing (NLP), Transformer networks (Vaswani et al.] 2017) pre-trained with "masked language model" (MLM) objective (De- vlin et al. 2018) on large language corpus excel at a variety of NLP tasks.
Mcanwhile, for tasks at the intersection of vision and language, such as image captioning (Young ct al. 2014|Chee et al. 2015 [Sharma et al., (2018), visual question answering (VQA) (Antol et al,
2015 Johnson et al. 2017; (ioyal et al. 2017 Hudson & Manning. 2019), visual commonsense reasoning (VCR) (Zellers et al.] 2 Gao et al. 2019), there lacks such pre- generic feature representations. The previous practice to combine base networks pre-trained for image recognition and NLP respectively in a task-specific way. The task-specific model is directly finetuned for the specific target task, without any generic visual-linguistic pre-training. The task-specific model may well suffer from overfitting when the data for the target task is scarce. Also, due to the task- sssc-specffic model design, it is difficult benefit from pre- pre-training, where the pre-training task may well be different from the target. There lacks a common ground for studying the feature design and pre- training of visual-linguistic tasks in general.
In the various network architectures designed for different visual-linguise tasks, key goal is to effectively aggregate the multi-modal information both the visual and linguistic domains. For ex- ample, to pick the right answer in the VQA task, the network should empower integrating linguistic information from the question and the answers, and aggregating visual information from the input image, together with aligning the linguistic meanings with the visual clues. Thus, we seek to derive generic representations that can effectively aggregate and align visual and linguistic information.
* Eguul contribution. This work is done when Weijie Su and Xizhou Zhu are interns at Microsoft Research Asia. Corresponding author.

---

Published as a conference paper at ICLR 2020
In the meantime, we see the successful application of Transformer attention (Vaswani et al. 2017) in NLP, together with its MLM- based pre-training technique in BERT (Devlin et al. 2018). The attention module is powerful and flexible in aggregating and aligning word embedded features in sentences, while the pre-training in BERT further enhances the capability.
Inspired by that, we developed VL-BERT, tre-trriinbll generic representation for visual- visual-linguistic tasks, as shown in Figure ! The backbone of VL-BERT is of (multí-modal) Transformer attention module taking both visual and linguistic embedded features as input. In it, each element is either of a word from the input sentence, or a region-offinterest (Rol) from the input image, together with certain special elements to disambiguate different imput formats. Each element can adaptively aggregate information from all the other elements according to the compatibility defined on their contents, positions, categories, and etc. The content features of a word/ an Rol are domain specific (WordPiece embeddings (Wu et al. 2016) as word features, Fast R-CNN (Girshick] 2015) features for Rols). By stacking multiple layers of multi-mode modiil Transformer attention modules, the derived representation is of rich capability in aggregating and aligning disual-linguise clues. And task- specific branches can be added above for specific visual-linguisti tasks.
To better exploit the generic representation, we pre-train VL-BERT at both large visual-linguistic corpus and text-only dataset=" [ The pre-tra pre-training loss on the -ingul-inguiitic corpus is incurred via predicting randomly masked words or Rols. Such pre-training sharpens the capability of VL-BERT in aggregating and aligning visual-linguistic clues. While the loss on the text-only corpus is of the standard MLM loss in BERT, improving the generalization on long and complex sentences.
Comprehensive empirical evidence demonstrates that the proposed VL-BERT achieves state-of-ther art performance on various downstream visual-linguitic tasks, such visual commonsense reason- ing, visual question answering and referring expression comprehension. In particular, we achieved the first place of single model on the leaderboard of visual commonsense reasoning.
RELATED WORK
Pre-training for Computer Vision Prior to the era of deep networks, is far from mature share features among different tasks and to improve the features via pre-training. The models for various computer vision tasks are of too diverse design choices to derive a generic representation. With the success of AlexNet (Krizhevsky et al. 2012) in ImageNet (Deng et al. 2009) classification, the renaissance of convolutional neural networks (CNNs) the vision community. Soon after that, researchers found that ImageNet pre-trained CNNs serve well as generic feature representation for various downstream tasks (Donahue et al. 2014). such as object detection (Girshick et al. 2014). semantic segmentation (Long et al. (015). instance segmentation (Hariharan et al.) 2014). The improvement in backbone networks for ImageNet classification further improves the downstream tasks. Recently there are research works on directly training CNNs from scratch on massive-scale target datasets. without ImageNet pre-training (He et al., 2018). They achieved performance on par with those with ImageNet pre-training. While they also note that pre-training on a proper massive dataset is vital for improving performance on target tasks with scarce data.
see
Pre-training for Natural Language Processing (NLP) It interesting to note that the development of pre-training techniques in NLP lags quite behind computer vision. There are previous research works on improving word embedding (Mikolov et al. 2013 [Pennington et al. 2014 Kiros et al.) 2015), which is low-level linguistic feature representation. On top of that, numerous diverse ar- chitectures are designed for various NLP tasks. In the milestone work of Transformers (Vaswani et al., 2017), the Transformer attention module is proposed generic building block for various NLP tasks. After that, a serious of approaches are proposed for pre-training the generic represen- tation, mainly based on Transformers, such GPT (Radford et al.] 2018), BERT (Devlin et al.) 2018). GPT-2 (Radford et al. 2019). XLNet ( (Yang et al. 2019). XLM (Lample & Conneau [2019), and RoBERTa (Liu et al. 2019). Among them, BERT is perhaps the most popular one due to its simplicity and superior performance.
Pre-training for Visual-Linguistie Tasks. The development course of models for visual-linguistic tasks is also quite similar to those in the computer vision and NLP communities. Previously, task-
Herr exploit the Conceptual Captions dataset (hhrmaa al. 2018 the visual-linguistic corpus, and the Cooks h7hu al. 2015) & English Wikipedia the text-only corpus.
2

---

Published as a conference paper at ICLR 2020
specific models designed, wherein the features derived from off-the- computer vision and NLP models are combined in an ad-hoc way for specific tasks. Model training is performed on the dataset for the specific task only.
VideoBERT (Sun et al., 2019b) is the first work secking to conduct pre-training for visual- -inguisii tasks. In it, video clips processed by off-the-shelf networks for action recognition, and are signed to different clusters (visual words) based on the derived features. The pre-training loss is incurred via predicting the cluster ids of masked video clips. Due the abrupt clustering of the video clips, it losses considerable visual content information and hinders updating visual network parameters. In the following work of CBT (Sun et al. (019a)), such clustering mechanism re- moved. Both works are applied on videos, which are of linear structure in the time dimension, same as sentences. It is highly desired to study at the well-established image-based lisuus-linguistic tasks.
Concurrent to our work, multiple works released on Arxiv very recently also seek to derive pre- trainable generic representation for visual-linguistic tasks. Table 3 in Appendix compares among them. We briefly discuss some of these works here.
In ViLBERT (Lu et al. 2019) and LXMERT (Tan & Bansal 2019). which are under review or just got accepted, the network architectures are of two single-modal networks applied on input sentences and images respectively, followed by a cross-modal Transformer combining information from the two sources. The attention pattern in the cross-moda Transformer is restricted, where the authors believe to improve the performance. The authors of ViLBERT claim that such two-stream design is superior than single-strean unified model. Meanwhile, in the proposed VL-BERT, it is of a unified architecture based on Transformers without any restriction on the attention patterns. The visual and linguistic contents are fed input VL-BERT, wherein they interact early and freely. We found that our unified model of VL-BERT outperforms such two-stream designs.
VisualBert (Li al. 2019b), B2T2 (Alberti al. 2019), and Unicoder-V VI. (Li al. 2019a). which are of work in progress or under review, are also of unified single-stream architecture. The differ- ences of these works are compared in Table 5 The concurrent emergency of these research works indicates the importance of deriving a generic pre-trainable representation for visual- -ssual-inguustic tasks.
In addition, there are three noticeable differences between VL-BERT and other concurrent works in pre-training. Their effects are validated in Section 4.3 (1) We found the task of Sentence- Image Re- lationship Prediction used in all of the other concurrent works (e.g.. ViLBERT (Lu et al., 2019) and
LXMERT (Tan & Bansal 2019)) is of no help in pre-training visual-linguistic representations. Thus such a task is not incorporated in VI-BBETT. (2) We pre-train VI.-BERT on both visual-linguistic and text-only datasets. We found such joint pre-training improves the generalization over long and complex sentences. (3) Improved tuning of the visual representation. In VL-BERT, the parame- also updated. To avoid visual clue leakage in
ters of Fast R-CNN, deriving the visual features, the pre- pre-training task of Masked Rol Classification with Linguistic Clues, the masking operation is conducted on the input raw pixels, other than the feature maps produced by layers of convolution.
3 VL-BERT
3.1 REVISIT BERT MODEL
Let x (.1...., NN be the input elements in BERT (Devlin et al. 2018), which are of embed- ded features encoding sentence words. They are processed by multi-layer bidirectional Trans- former (Vaswani et al., 2017). where the embedding features of each element are transformed layyrr in the fashion of aggregating features from the other elements with adaptive atten- tion weights. Let x' = (af.... ...... xyy be the features of the 1-th layer (x is set as the input æ). The features of the (1 + 1)-th layer, ++1 computed by
M
N
7,41 Σ W!- Σ
AN v v
Multi-head Attention,
(1)
m=1
4 [4| LayerNorm (o} k(
Residual Connection,
(2)
$/+1 w GELU(WW(+1))) 42/ b/+12 6
Feed-forward,
(3)
[+1 LayerNorm 1!11)
Residual Connection,
(4)
3

---

Published as a conference paper at ICLR 2020
where m in Eq. indexes over the attention heads, and A & exp " (Ktt'>) 2 de- notes the attention weights between elements and in the m-th head, which is normalized by Σ A 1. W Q K and V!-1 are learnable weights for m attention head, W WW-t. and of+1 b in Eq. 3 are learnable weights and biases, respectively. Note that, the operations in Eq. [~_ irrelevant to the order of input sequence, i.e. the final BERT representa- tion of permuted input is same as the final BERT representation of the original input after the same permutation. The position of an element in BERT is encoded in its own embedding features by se- quence positional embedding. Thanks to such decoupled representation, the BERT model is flexible enough to be pre-trained and finetuned for variety of NLP tasks.
In BERT pre-training, the masked language modeling (MLM) task introduced. The embedded features of certain input word would be randomly masked out (the token embedding channels capturing the word content is replaced by a special [MASK] token). The BERT model is trained predict the masked word from linguistic clues of all the other unmasked elements. As explained in Wang & Cho (2019), the overall MLM-based training of BERT is equivalent to optimizing the following joint probability distribution
N
log P(x(0) (())
log (z10).
(5)
where (x(()) is the potential function for the i-th input element, with parameters 0, and Z(0) is the partition function. Each log-potential term log (x) is defined
log oe (x = x.. f((m),));..
(6)
where (0()) denotes the final output feature of BERT corresponding to the i-th element for input Ty where 2yy is defined as vi { -1 [MASK], Ii.l. NN.. The incurred MLMM-based loss is
(0) P..... log (ar),
where is randomly sampled sentence from the training set D, and is a randomly sampled location for masking words.
The second pre-training task, Next Sentence Prediction, focuses on modeling the relationship be- tween two sentences. Two sentences are sampled from the input document, and the model should predict whether the second sentence is the direct successor of the first. In BERT, the sampled two sentences are concatenated into one input sequence, with special elements [CLS] and [SEP] inserted prior to the first and the second sentences, respectively. A Sigmoid classifier is appended final output feature corresponding to the [CLS] element to make the prediction. Let x be the input sequence, t E {0, 1} indicates the relationship between the two sentences. The loss function is defined as
the
LNsP(0) E(,,)) log(g(=))) + (1 - t) log(1 g(ab))),
(8)
where æh is the final output feature of the [CLS] element (at the L-th layer), and g(x()) is the classifier output.
3.2 MODEL ARCHITECTURE
Figure illustrates the architecture of VL-BERT. Basically, it modifies the original BERT (Devlin et al. 2018) model by adding new elements to accommodate the visual contents, and new type of visual feature embedding to the input feature embeddings. Similar to BERT, the backbone is of multi-layer bidirectional Transformer encoder (Vaswani et al., 2017). enabling dependency mod- eling among all the input elements. Different to BERT processing sentence words only, VL-BERT takes both visual and linguistic elements input, which are of features defined regions-of-inters (RoIs) in images and sub-words from input sentences, respectively. The RoIs can either be bounding boxes produced by object detectors, be annotated ones in certain tasks.
worth noting that the input formats vary for different visual- -inguisii tasks (e.g., <Caption, Image>> for image captioning, and < <Question, Answer, Image> for VQA (Antol al. 2015 John- son et al. 2017, Goyal et al. 2017 Hudson & Manning (011)) and VCR (Zellers et al. 2019 Gao al. 2019)). But thanks the unordered representation nature of Transformer attention (e.g., the
4

---

Published as a conference paper at ICLR 2020
Masked Language Modeling with Visual Clues
Masked Roi Classification with Linguistic Clues
Fully Connected
Appearance
bottle
[Cat]
Feature
Geometry
Visual- Lingulsii BERT
Embedding
[CLS]
[MASK]
[SEP]
[IMG]
[MMG]
[END]
Fast(er) R-CNN
Embedding
Regions of Interest
Visual Feature
Embedding
Segment
Embedding
A
A
Sequence
+
+
Zero Out
Position
4
5
8
Embedding
Caption
Image Regions
Figure 1: Architecture for pre-training VL-BERT. All the parameters in this architecture including VL-BERT and Fast R-CNN are jointly trained in both pre-training and fine-tuning phases.
position of word in sentence encoded by the positional embedding only, other than the order in the input sequence), generic representation can be derived long as the input elements and embedding features are properly designed. Three types of input elements are involved, namely, vi- sual, linguistic, and special elements for disambiguating different input formats. The input sequence always starts with a special classification element ([CLS)), then goes on with linguistic elements, then follows up with visual elements, and ends with special ending element ([END]). A special separation element ([SEP]) is inserted in between different sentences in the linguistic elements, and between the linguistic and visual elements. For each input element, its embedding feature is the summation of four types of embedding, namely, token embedding, visual feature embedding, seg- ment embedding, and sequence position embedding. Among them, the visual feature embedding is newly introduced for capturing visual clues, while the other three embeddings follow the design in the original BERT paper.
Token Embedding Following the practice in BERT, the linguistic words are embedded with Word- Piece embeddings (Wu al. 2016) with 30,000 vocabulary. A special token is assigned each special element. For the visual elements, special [IMG] token is assigned for each one of them.
Visual Feature Embedding We firstly describe visual appearance feature and visual geometry em- bedding separately, and then how to combine them to form the visual feature embedding.
For the visual element corresponding an Rol, the visual appearance feature is extracted by apply- ing a Fast R-CNN (Girshick 2013) detector (i.c., the detection branch in Faster R-CNN (Ren et al., 2015), where the feature vector prior to the output layer of each Rol utilized as the visual feature embedding (of 2048-d in paper). For the nin-visual elements, the corresponding visual appearance features are of features extracted on the whole input image. They are obtained by applying Faster R-CNN on an Rol covering the whole input image.
The visual geometry embedding is designed inform VI.-BERT the geometry location of each in- (fr
ML zua put visual element in image. Each Rol is characterized by 4-d vector, as W W ). where
(ILr, MEr) and (æRB, VRB) denote the coordinate of the top-left and bottom-right corner respectively, and W, H are of the width and height of input image. Following the practice Relation Net- works (Hu et al.) 2018), the 4-d vector is embedded into high-dimensional representation (of 2048-d in paper) by computing sine and cosine functions of different wavelengths.
The visual feature embedding is attached to each of the input elements, which is the output of fully connected layer taking the concatenation of visual appearance feature and visual geometry embedding as input.
Segment Embedding Three types of segment, A, B, C, are defined to separate input elements from different sources, namely, A and B for the words from the first and second input sentence respec- tively, and C' for the RoIs from the imput image. For example, for input format of <Question, Answer, Image>, A denotes Question, B denotes Answer, and C' denotes Image. For input format
5

---

Published as a conference paper at ICLR 2020
of <Caption, Image>, A denotes Caption, and C' denotes Image. A learned segment embedding is added to every input element for indicating which segment it belongs to.
Sequence Position Embedding A learnable sequence position embedding is added to every imput element indicating its order in the input sequence, same as BERT. Because there is no natural order among input visual elements, any permutation of them in the input sequence should achieve the same result. Thus the sequence position embedding for all visual elements are the same.
3.3 PRE-TRAINING VL-BERT
The generic feature representation of VL--BERT enables us to pre-train it on massive-scale datasets, with properly designed pre-training tasks. We pre-train VL-BERT on both only datasets. Here we utilize the Conceptual Captions dataset (Sharma et al. 2018) as the visual- linguistic corpus. It contains around 3.3 million images annotated with captions, which are harvested from web data and processed through an automatic pipeline. The issue with the Conceptual Captions dataset is that the captions are mainly simple clauses, which are too short and simple for many down- stream tasks. To avoid overfitting on such short and simple text scenario, we also pre-train VL-BERT on text-only corpus with long and complex sentences. We utilize the BooksCorpus (Zhu et al. 2015) and the English Wikipedia datasets, which are also utilized in pre- pre-training BERT.
and text-
In SGD training, in each mini-batch samples are randomly drawn from both Conceptual Captions and BooksCorpus & English Wikipedia (at ratio of 1:1). For sample drawn from Conceptual Captions, the input format to VL-BERT is of <Caption, Image>, where the Rols in the image are localized and categorized by pre-trained Faster R-CNN object detector. Two pre-training tasks are exploited to incur loss, which are as follows.
Task #1: Masked Language Modeling with Visual Clues This task is very similar to the Masked Language Modeling (MLM) task utilized in BERT. The key difference is that visual clues are incor- porated in VL-BERT for capturing the dependencies among visual and linguistic contents. During pre-training, each word in the input sentence(s) is randomly masked (at a probability of 15%). For the masked word, its token replaced with a special token of [MASK]. The model is trained to predict the masked words, based on the unmasked words and the visual features. The task drives the network to not only model the dependencies in sentence words, but also to align the visual and lin- guistic contents. For example, in Figure "kitten drinking from IMASK]", without the input image, the masked word could be any containers, such as "bowl", "spoon" and "bottle". The representation should capture the correspondence of the word "bottle" and the corresponding Rols in the image to make the right guess. During pre-training the final output feature corresponding to the masked word is fed into classifier over the whole vocabulary, driven by Softmax cross-entropy loss.
Task #2: Masked Rol Classification with Linguistic Clues This is a dual task of Task #1. Each Rol in image is randomly masked out (with 15% probability), and the pre- pre-training task is predict the category label of the masked Rol from the other clues. To avoid any visual clue leakage from the visual feature embedding of other elements, the pixels laid in the masked Rol are set zeros before applying Fast R-CNN. During pre-training, the final output feature corresponding to the masked Rol is fed into a classifier with Softmax cross-entropy loss for object category classification. The category label predicted by pre-trained Faster R-CNN is set as the ground-truth. An example is shown in Figure I The Rol corresponding to cat in image is masked out, and the corresponding category cannot be predicted from any visual clues. But with the input caption of "kitten drinking from bottle", the model can infer the category by exploiting the linguistic clues.
For sample drawn from the BooksCorpus & English Wikipedia datasets, the input format to VL-- BERT degenerates to be <Text, Ø >, where no visual information is involved. The "visual feature embedding" term in Figure D is a learnable embedding shared for all words. The training loss is from the standard task of Masked Language Modeling (MLM) as in BERT.
In summary, the pre-training visual-linguisti corpus improves the detailed alignment between visual and linguistic contents. Such detailed alignment is vital for many downstream tasks (for example, in Visual Grounding (Kazemzadeh et al.. 22014) the model locates the most relevant object or region in an image based on a natural language query). While the pre-training on text-only corpus facilitates downstream tasks involving understanding of long and complex sentences.

---

Published as a conference paper at ICLR 2020
3.4 FINE-TUNING VL-BERT
VL-BERT is designed to be a generic feature representation for various visual-linguistic tasks. is relatively simple to finetune VL-BERT for various downstream tasks. We simply need feed VL- BERT with properly formatted input and output, and finetune all the network parameters end-to-end. For the imput. the typical formats of <Caption, Image> and <Question, Answer, Image> cover the majority visual-linguisti tasks. VL-BERT also supports more sentences and more images as long as appropriate segment embeddings are introduced identify different input sources. At the output, typically, the final output feature of the [CLS] element is used for contenc--imag--elaan prediction. The final output features of words or Rols are for word-level or Rol-level prediction. In addition to the input and output format, task-specific loss functions and training strategies also need to be tuned. See Section for the detailed design choices and settings.
level
EXPERIMENT
4.1 PRE-TRAINING
As described in Section 3.3 pre-train VL-BERT jointly Conceptual Captions (Sharma al. 2018) as visual- lingull-iiissticcccc.o corpus, and BooksCorpus (Zhu et al. (2055) & English Wikipedia as text- only corpus. As VL.-BETT is developed via adding new inputs capturing visual information the original BERT model, we initialize the parameters to be the same as the original BERT described in (Devlin et al. 2018 VL-BERT BASE and VL-BERT denote models developed from the origi- nal BERT BASE and LERT models, respectively. The newly added parameters in VL-BERT are randomly initialized from a Gaussian distribution with mean of and standard deviation of 0.02. Visual content embedding is produced by Faster R-CNN + ResNet- 101, initialized from parame- ters pre-trained on Visual Genome (Krishna al. 2017 for object detection (see BUTD (Anderson) et al. 2018)).
Prior to pre-training on Conceptual Captions, the pre-trained Faster R-CNN is applied to extract Rols. Specifically, at most 100 Rols with detection scores higher than 0.5 are selected for each im- age. At minimum, 10 Rols are selected from one image, regardless the detection score threshold. The detailed parameter settings are in Appendix.
4.2 FINE-TUNING ON DOWNSTREAM TASKS
The pre-trained VL-BBETT model can be fine-tuned for various downstream visual-linguistic tasks, with simple modifications on the input format, output prediction, loss function and training strategy.
4.2.1
VISUAL COMMONSENSE REASONING (VCR)
Model
Q
-
A
QA
+
R
Q +
AR
val
test
val
test
val
test
R2C (Zellers et al. 2019)
63.8
65.1
67.2
67.3
43.1
44.0
ViLBERT (Lu ct al. 2019
72.4
73.3
74.5
74.6
54.0
54.8
VisualBERT et al. 2019b
70.8
71.6
73.2
73.2
52.2
52.4
B2T2 (Albrtti et al. 2019
71.9
72.6
76.0
75.7
54.9
55.0
VL-BERT BASE w/o pre-training
73.1
73.8
54.2
VL-BERTRASE RASE.
74.4
55.2
VL-BERTLARGE
75.5
75.8
77.9
78.4
58.9
59.7
Table 1: Comparison to the state-of-the-an methods with single model on the VCR dataset. i indicates concurrent works.
Visual Commonsense Reasoning (VCR) focuses on higher-order cognitive and commonsense un- derstanding of the given image. In the dataset of Zellers et al. (2019), given image and a list of categorized Rols, a question at cognition level raised. The model should pick the right answer to the question and provide the rationale explanation. For each question, there 4 candidate swers and 4 candidate rationales. This holistic task (Q - AR) is decomposed into two tub-tasks wherein researchers can train specific individual models: question answering (Q - A) and answer

---

Published as a conference paper at ICLR 2020
Answer Classification
Object Classification
Visual-Linguistic BERT
[CLS]
Tok
Tok N
[SEP)
Tok 1
Tok N
[SEP]
[IMG]
[IMG]
[END]
Question
Answer
Image Regions
(a) Input and output format for Visual Commonsense Reasoning (VCR) dataset
Answer Prediction
Visual-Linguistic BERT
[CLS]
Tok 1
Tok
[SEP]
[MASK]
[SEP]
[IMG]
[IMG)
(END)
Question
Unknown Answer
Image Regions
(b) Input and output format for Visual Question Answering (VQA) dataset
Region Classification
Visual-Linguistic BERT
[CLS]
Tok
Tok N
[SEP)
[IMG]
[IMG]
[END]
Query
Image Regions
(c) Input and output format for Referring Expression task on ReiCOCO+ dataset
Figure 2: Input and output formats for fine-tuning different visual-linguistic downstream tasks.
justification (QA R). The released VCR dataset consists of 265k pairs of questions, answers, and rationales, over 100k unique movie scenes (100k images). They are split into training, validation, and test sets consisting of 213k questions and 80k images, 27k questions and 10k images, and 25k questions and 10k images, respectively.
Our experimental protocol for VCR follows that in R2C (Zellers al. 2019). The model trained on the train split, and is evaluated at the val and test sets. In the original work R2C, task-specific "Grounding". "CContxtuullization" and "Reasoning" modules are designed. Here we simply adopt the generic representation of VL-BERT for the task. Figure _ (a) illustrates the input format, <Question, Answer, Image>.. For the sub-task of Q - A, 'Q" and are filled the Question section and Answer section respectively. For the sub-task of QA + R. the concatenation of 'Q' and 'A' is filled to the Question section, and 'R' is filled to the Answer section. The input Rols to VL-BERT are the ground-truth annotations in the dataset. The final output feature of [CLS] element fed to Softmax classifier for predicting whether the given Answer is the correct choice. During fine-tuning, adopt two losses, the classification classification with linguistic clues. The detailed parameter settings are in Appendix.
the correctness of the answers and the Rol
Table D presents the experiment results. Pre-training VL-BERT improves the performance by 1.0% in the final Q - AR task, which validates the effectiveness of pre-training. Compared with R2C, do not use ad-hoc task-specific modules. Instead, we simply adopt the generic representation of VL-BERT and jointly train the whole model end-to-end. Despite the same input, output and experimental protocol as R2C, VL-BERT outperforms R2C by large margins, indicating the power of our simple cross-modal architecture. Compared with other concurrent works, i.e., ViLBERT, VisualBERT and B2T2, our VL-BERT achieves the ofte-oofthhe-rt performance.
4.2.2 VISUAL QUESTION ANSWERING (VQA)
In the VQA task, given a natural image, a question at the perceptual level is asked, and the algorithm should generate / choose the correct answer. Here conduct experiments on the widely-used VQA v2.0 dataset (Goyal et al., 2017), which is built based on the COCO (Lin et al., 2014) images. The VQA v2.0 dataset is split into train (83k images and 444k questions), validation (41k images and
8

---

Published as a conference paper at ICLR 2020
Model
test-dev
lest-std
BUTD (Anderson et al.
2018
65.32
65.67
ViLBERT (Lu al. 2019)
70.55
70.92
VisualBERT (Li et 2019b)
70.80
71.00
LXMERT (Tan & Bansal 2019
72.42
72.54
VL-BERTRAS RASE w/o pre-traiming
69.58
VL-BERT BASE
71.16
VL-BERT LAKGE
71.79
72.22
Table Comparison the State-of-the-ee methods with single model on the VQA dataset. i indicates concurrent works.
214 14 questions), and test (81k images and 448k questions) sets. Following the experimental protocol in BUTD (Anderson et al., 2018), for each question, the algorithm should pick the corresponding answer from a shared set consisting of 3,129 answers.
Figure (b) illustrates the input format for the VQA task, which is of <Question, Answer, Image>>.
As the possible answers are from shared pool independent the question, we only fill [MASK] element to the Answer section. As in BUTD (Anderson et al. 2018). the input Rols in VL-BERT are generated by a Faster R-CNN detector pre-trained on Visual Genome (Krishna al., 2017). The answer prediction is made from multi-class classifier based upon the output feature of the [MASK] element. During fine-tuning, the network training is driven by the multi-class cross-entropy loss over the possible answers. The detailed parameter settings are in Appendix.
Table presents our experimental results. Pre-training VL-BERT improves the performance by 1.6%, which validates the importance of pre-training. VL-BERT shares the same input (i.e., ques- tion, image, and Rols), output and experimental protocol with BUTD, a prevalent model specifi- cally designed for the task. Still, VL-BERT surpasses BUTD by over 5% in accuracy. Except for LXMERT, our VL-BERT achieves better performance than the other concurrent works. This is be- cause LXMERT is pre-trained on massive visual question answering data (aggregating almost all the VQA datasets based on COCO and Visual Genome). While our model is only pre-trained on captioning and text-only dataset, where there is still gap with the VQA task.
4.2.3 REFERRING EXPRESSION COMPREHENSTON
Ground-truth Regions
Detected Regions
Model
testA
testB
val
lestA
testB
MAttNet Yuu et al. 2018)
71.01
75.13
66.17
65.33
71.62
56.02
ViLBERT (Lu et al. 2019]]
72.34
78.52
VL-BERT w/o pre- ve-trrining
74,41
77.28
67.52
66.03
71.87
56.
"L-BERTBASE
79.88
82.40
75.01
71.60
77.72
60.99
VL-BERT
80.31
83.62
75.45
72.59
78.57
62.30
Table Comparison the state-of-the-arr methods with single model the RefCOCO+ dataset. i indicates concurrent work.
A referring expression is natural language phrase that refers to an object in an image. The referring expression comprehension task localize the object in an image with the given referring expres- sion. We adopt the RefCOCO+ (Kazemzadeh et al. 2014) dataset for evaluation, consisting of 141k expressions for 50k referred objects in 20k images in the COCO dataset (Lin al. 2014). The referring expressions in RefCOCO+ are forbidden from using absolute location words, e.g. left dog. Therefore the referring expressions focus on purely appearance- based descriptions. RefCOCO+ are split into four sets, training set (train), validation set (val), and two testing sets (testA and testB), Im- ages containing multiple people are in testA set, while images containing multiple objects other categories are in testB set. There no overlap between the training, validation and testing images.
Figure (c) illustrates the input format for referring expression comprehension, where the input format is of <Query, Image>. Model training and evaluation conducted either the ground- truth Rols or on the detected boxes MAtNeet (Yu et al., 2018). And the results are reported either in the track of ground-truth regions that of detected regions, respectively. During training,

---

Published as a conference paper at ICLR 2020
compute the classification scores for all the input Rols. For each Rol. binary classification loss applied. During inference, we directly choose the Rol with the highest classification score as the referred object of the imput referring expression. The detailed parameter settings in Appendix.
Table presents our experimental results. Pre-trained VL-BERT significantly improves the per- formance. Compared with MAttNet, VL-BERT is much simpler without task-specific architecture designs, yet much better. VL-BERT achieves comparable performance with the concurrent work of ViLBERT.
4.3 ABLATION STUDY
Settings
Masked Language
Modeling with
Rol Classification with Linguistic Clues
Sentence-Image Relationship Prediction
with Text-only Corpus
Tuning Fast R.CNN
Q-+
VCR
QA-R
VQA
Detected Regions
RefCOCO+
Visual Clues
val
w/o pre truining
72.9
73.0
69.5
62.7
72.9
73.1
71.0
69.1
(h)
73.0
73.1
71.1
70.7
72.2
72.4
70.3
69.5
(d)
73.4
73.8
71.1
70.7
VL-BERT
73.8
73.9
71.2
71.1
Table 4: Ablation study for VL-BERTRASE with 0.5x fine-tuning epochs.
Table abbates key design choices in pre-training VL-BERT. For experimental efficiency, the fine- tuning epoches of VL-BERT are of 0.5x of those in Section with only "L-BERTBASH model.
Overall, the pre-training of VL-BERT improves the performance tasks (by comparing setting "w/o pre-training" and VL-BERTBASE). The improvement amplitude varies for different tasks. By comparing setting (a) that of "w/o pre-training", we see the benefits of Task #1, Masked Language Modeling with Visual Clues. By further incorporating Task #2, Masked Rol Classification with Linguistic Clues, the accuracy further improves on RefCOCO+, but gets stuck at VCR and VQA. This might be because only RefCOCO+ utilizes the final output feature corresponding to [IMG] tokens for prediction. Thus the pre-training of such features is beneficial. Setting (c) incorporates the task of Sentence- Relationship Prediction as in ViLBERT (Lu ctt al. 2019) and LXMERT (Tan & Bansal 2019). would hurt accuracy on all the three down- stream tasks. We guess the reason is because the task of Sentence- Image Relationship Prediction would introduce unmatched image and caption pairs as negative examples. Such unmatched samples would hamper the training of other tasks. Setting (d) adds text-only corpus during pre-training. Compared with setting (b), it improves the performance over all three down-stream tasks, and is most significant on VCR. This is because the task of VCR involves more complex and longer sentences than those in VQA and RefCOCO-E By further finetuning the network parameters of Fast R R-CNN, which generates the visual features, we get the final setting of VL-BERT BASE Such end-to-end training of the entire network is helpful for all the downstream tasks.
all the three down-stream
CONCLUSION
In this paper, developed VL-BERT, linguistic tasks. Instead of using ad-hoc task-specific modules, VL-BERT adopts the simple yet powerful Transformer model the backbone. It is pre-trained on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the linual-linguistic clues, and thus benefit the downstream tasks. In the future, we would like to seek better pre- trainininn tasks, which could beneficial more downstream tasks (e.g., Image Caption Generation).
pre-trainable generic representation for visual-
ACKNOWLEDGMENTY
The work is partially supported by the National Natural Science Foundation of China under grand No.U19B2044 and No.61836011.
II VCR, there are 16.0 and 33.7 words per sample on average for Q A and QA - R sub-tasks, respec- tively. While the words per sample for VQA and RefCOCO+ are of 7.2 and 3.5, respectively.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. arXiv preprint arXiv: /900:/990..00 000500 2019.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077-6086, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 10te:::504003335 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang. Eric Tzeng, and Trevor Darrell, Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference machine learning, pp. 647-655, 2014.
Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. From two graphs to n ques- tions: A vqa dataset for compositional reasoning on vision and commonsense. arXiv preprint (rXiv:190.002961 2019.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 1440-1448, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904-6913, 2017.
Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultancous detection and segmentation. In European Conference on Computer Vision, pp. 297-312. Springer, 2014.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. arXiv preprint 1rXiv:/811.0888 2018.
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3588-3597, 2018
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6700-6709, 2019.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
11

---

Published as a conference paper at ICLR 2020
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294-3302, 2015.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, pp. 1097-1105, 2012.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining, arXiv preprint 1XXV:09001.02991, 2019.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vl: universal encoder for vision and language by cross-modal pre-training, 2019a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 190X:v::00.03300 2019b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692,
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 31.333334400 2015.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. arXiv preprint arXiv: 190:: 02265, 2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 1301.37,, 2013.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99,9999999 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2556-2565, 2018
12

---

Published as a conference paper at ICLR 2020
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Contrastive bidirectional trans- former for temporal representation learning. arXiv preprint Xxxv:/906.00573,, 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. arXiv preprint
2019b.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp- 5998-6008, 2017.
Jesse Vig. A multiscale visualization of attention in the transformer model. N5fi::9906.57/11 2019. URLttss:: org/ 0000//9906.0777
arXiv preprint
Alex Wang and Kyunghyun Cho. Bert has a mouth, and must speak: Bert as a markov random field language model. arXiv preprint arXiv: 1902.04094, 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: wew.16609084444 144, 2016.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 1.K6:0990608233,, 2019.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78. 2014.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- tnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1307-1315, 2018.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition cognition: Visual commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6720-6731, 2019.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answer- ing in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995-5004, 2016.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19 27, 2015.
13

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1
COMPARISON AMONG VL-BERT AND OTHER WORKS
Table 5 compares among VL-BERT and other concurrent works for pre-training generic visual-
linguistic representations.
Method
Archteeture
Vissal Token Pre-trin Donseets
Pre-tran Tisks
Dowastean Tasks
Pablidhal Warie
StkkottERT San all j0nihh
single cineormdal "oondomnrr
Video frome
Oacking Sun al 23196
21 makked languaze aodeling
I) serreece-inne inngg aligrment
zero-dhrt deit oction clasitbcttion video captioning
3i makked Vissal words prediction
CRT San dtall 2019
1wo drgll-mmohll Iranelomee
(vision & languae roppativey))
valoo frame
Coekine11K Sui et al 2019
2) mekell languag modding
[] Jizrizecz-ingge slgrmmm
sasioon antiopation vadeo aaptaning
+ one crnsormii) TTaneborre
3) makked vinsal-feetre regrssiins
I] srrrrcc-iing alggnmrr
1) visaal queeiun avooring
Vil RERT Lectdl 2019
a ase crisi-arodal TeansSomter (with meruzed sturtion paturn))
sirglo-motal Travdomer (bingagg)
image Rol
Conerpaaa
Stanna et 2018
3) masked viseal fectuee clasilication
2) mekial langugg modking
51 groending refercing expressions
visual ormmunennee steuning
innge retieval
Domo-dan imego nerieval
Alheni oi 2015
single cress-modal Transtomerr
image Rol
Stamm at 2011
1) jjariiien-ingg algyiieer
21 masked languuge moddlnng
T Tiveial crmmonmee stouning
10000
I] sesteenc imige aligtneer.
Ti Vitaal qresion aniwering
Weeks
Unter
LXVEX! ban & Ismal 201v (019
(véinn & language pestivic) oze crosi ciosi-coddl TeansSomerr
two single medi Transformer
image Rol
.G QA
VG Caption
3) maked visual featue clasificttio
2) masked language modeing
Dinuual larguage visual reesoting
Roviows
YQA GQ4
4) makked visual feutue regresions
Jas Get
5) visui quesiie asswering
Acripea
I] serieras--nneee alggrma
vive queeists aeewening
VERTSEKT 1 al 2019h
sizgle csosi-anodll Transfomrr
image Rol
Chenetal 2001
COCO Castion
21 maked language modking
nauzal lazguzze visua reesoning
visaal commrrennre stauning
pounding plaaee
UniratenV) Li cl E0192
single crono-madal Toandommerr
image Bul
Conceemnal Contuns
Shanma et 2018
31 maked vissl-feattre festure clamifictioo
2) makeul language mokking
I) Jjseieetssinggee aligrmare
I) ninical
amn-dol imngo-iod natioval
Calccetoa Cucina Tinreeonn
I) makked languag medkling
viral mowriin
2] makiol vineal-rottm clanifiction
veral oommonnnn sesoming
Oue VL-NERT
cronomadal Tonwlomeer
irage Rol
gronding reformig exprasiime
Zhm dl B.15
peoottanndd Spaion
p0n5] Cattin
[013]
Marning Saly
Table 5: Comparison among our VL-BERT and other works seeking to derive pre-trainable generic representations for disual-linguiste tasks.
A.2 DETAILED EXPERIMENT SETTINGS
Pre-training is conducted on 16 Tesla V100 GPUs for 250k iterations by SGD. each mini-batch, 256 samples are drawn. Among them, 128 samples are of <Caption, Image>> pairs from Conceptual Captions, and the rest 128 samples are sequential tokens (at most 64 tokens for each sequence) from BooksCorpus & English Wikipedia. In SGD, Adam optimizer (Kingma & Ba 2014) is applied, with base learning rate of x 10-6, 81 = 0.9, 32 = 0.999, weight decay of 10 learning rate warmed up over the first 8,000 steps, and linear decay of the learning rate. All the parameters in VL-BERT and Fast R-CNN are jointly trained in both pre-training and fine-tuning phase. The visual feature input for textual corpus is a learnable embedding shared for all words. In the task of Masked Rol Classification with Linguistic Clues, the pixels lying in all the masked Rols are set as zeros in the image. A box covering the whole image is added as a Rol and would not be masked.
For VCR, the fine-tuning is conducted on 16 Tesla V100 GPUs for 20 epochs. In each 256 triplets of <Question, Answer, Image> are sampled. In SGD, the basic mini-batch gradient descent is conducted, with base learning rate of 5 x 10-3, momentum of 0.9, and weight decay of 10-4, The learning rate is linearly warmed up in the first 1,000 steps from an initial learning rate of 0. and is decayed by 0.1 at the 14-th and the 18-th epochs.
For VQA, the fine-tuning is conducted on 16 Tesla V100 GPUs for 20 epochs. In each mini-batch, 256 triplets of <Question, Answer, Image> are sampled. In SGD, Adam optimizer is applied, with base learning rate of x 10-4, 31 = 0.9, Bg = 0.999, weight decay of 10 -4, learning rate warmed up over the first 2,000 steps, and linear decay of the learning rate.
For RefCOCO+, the fine-tuning is conducted on 16 Tesla V100 GPUs for 20 epochs. In each mini- batch, 256 pairs of <Query, Image>> sampled. In SGD, Adam optimizer is applied, with base learning rate of 1 x 10 31 = 0.9, B2 = 0.999, weight decay of 10 learning rate warmed up over the first 500 steps, and linear decay of the learning rate.
14

---

Published as a conference paper at ICLR 2020
U2g
(a)
(b)
Layw Head
Layar
(d)
Layer
Lape
(e)
(f)
Figure 3: Visualization of attention maps in pre-trained VL-BERT BASE- Line intensity indicates the magnitude of attention probability with the text token query and the image Rol key. The intensity affinely rescaled to set the maximum value as and the minimum as 0, across different heads in each layer. The index of network layer and attention head is counted from 0.
15

---

Published as a conference paper at ICLR 2020
A.3 VISUALIZATION ATTENTION MAPS IN VL-BERT
To better understand what VL-BERT learns from pre-training, we visualized the attention maps of pre-trained VL-BERT (without fine-tuning downstream tasks) using Bert Vig Vig 2019).
Some visualization results on COCO (Lin et al., 2014, Chen et al., 2015) val2017 set are shown in Figure We can see different attention patterns across attention heads. For some attention heads, text tokens attend more on the associated image Rols. While in some other heads, text tokens attend uniformly all Rols. It demonstrates the ability of VL-BERT in aggregating and aligning visual- linguistic contents.
16


Reviews:
[{"review_id": "SyxJtUoRtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\n\nThis paper propose a new model for learning generic feature representations for visual-linguistic tasks by pretraining on large-scale vision and language datasets like Conceptual Captions and language-only datasets like BookCorpus and English Wikipedia. They demonstrate that the pre-training procedure can help improve performance on down-streaming tasks like visual question answering, visual commonsense reasoning. \n\nOverall I liked the design choices made in the presentation. Although the paper doesn't provide insights around what the representations have learned and how they differ from representations learned / used by existing methods, they have provided substantial evidence to suggest that pre-training helps in a lot of downstream tasks. \n\nAlthough it's hard to evaluate the paper without putting it in context with other concurrent works that have come out recently, I tried my best to evaluate the merits of the paper in isolation. \n\n### Strengths:\n\n- The paper explores an interesting direction of learning generic feature representations for visual-linguistic tasks for down-streaming tasks. Traditionally, people learn feature representations from scratch for each downstream task which might not always be possible if the training data is limited.\n- The paper does a decent job mentioning all the concurrent work in the space of learning multi-modal representations that have come out very recently. They distinguish the proposed method from existing work and also compare the performance of the proposed approach with concurrent work on downstream tasks showing performance on-par or better than existing methods.\n- I liked some of the design choices made in the paper. (1)  Instead of training a separate  transformer network for each type of input segments (question, ans, caption, image, etc). This makes the model easily extensible to other tasks as long as the correct segment embeddings are used to identify different input sources. (2) They also use a separate embedding for visual features instead of a common embedding  for both language tokens and visual tokens.\n- Unlike the pre-training task in concurrent work, the model was pre-trained not just on multi-modal datasets like conceptual captions but also on text-only corpus like BookCorpus and English Wikipedia. The authors claim that this leads them to learn better representations for longer sentences which they found useful for VCR task.\n\n### Weaknesses:\n\n- The authors claim that attention mechanism in cross-modal transformer by concurrent approaches is restrictive but doesn't give substantial evidence that this is true. What are the limitations for cross-modal attention mechanisms compared to a single transformer model as described in this paper.\n- On the contrary, by having a cross modal architecture, they can pre-train each modality separately on unaligned data. For instance, the text only transformer can be trained using large text corpora similar to BERT while the image only transformer can be trained on big datasets like OpenImages, ImageNet etc\n- While the paper gives a lot of empirical evidence that pre-training helps, it would have been interesting to develop an understanding of what the model is actually learning and how are these representations better than learning representations from scratch for each task. For instance maybe the authors can visualize attention similar to [1].\n\n### Other questions:\n\n- When training on text-only datasets, what is the input on Visual Feature Embedding since there are no associated images. The authors mention that for non-visual elements, the features are extracted on the whole image. It's still unclear what the associated visual features are for text-only datasets.\n- One of the pre-training tasks is masked ROI classification but it assigns a hard label to each ROI feature. It might be interesting to instead try learn the entire probability distribution (the output of a pre-trained classifier) by either minimizing the KL-divergence or by using softmax with soft-targets.\n- While the model was learnt on text-only data, as mentioned in the above section, will the model help from image-only datasets such as large-scale classification datasets?\n- While the models are tested on vision-and-language datasets, will these generic representations also be useful for unimodal tasks?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1ludY4GqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a pre-trainable generic representation for visual-linguistic tasks call VL-BERT. VL-BERT extend BERT by changing the input from subsequent sentence to image regions and modify the caption words has the additional visual feature embedding. The authors pre-train the VL-BERT on the conceptual caption dataset and Wikipedia and book corpus dataset, empirical results show that the VL-BERT achieve the SOTA performance on the VCR, VQA and refer expression tasks. \n\nAs the authors mentioned in Table 5, pre-training the visolinguistic representation for vision and language tasks is very popular recently, and 5~6 similar works have appeared recently. One of the nice features I found on this work is it's joint train with text-only corpus and faster RCNN weight. While ViLBERT designs for easier extendable for other modalities, VLBERT is more focus on the representation learning on the vision and language, since the caption input also combines with the visual feature embedding. \n\nOverall the paper is well written and performs extensive experiments/ablations. There is some specific point that is not clear to me or needs further clarifications from the authors. \n\n1: The authors mentioned the improvement over tuning the visual parameters, I wonder what is the details on that? is the region proposal network's weight fixed? if not, how to avoid the shift on the proposal layer? Is the model still has the visual genome target or objective? Which layer is fixed/updated? and what is the optimizer and learning rate scheduler? \n\n2: I notice there is a change in the textual input which take visual feature embeddings. I wonder what is the performance without these features? What is the visual feature input for textual corpus? \n\n3: For the Masked RoI classification with Linguistic Clues, what if there are overlapped regions? what if the detection label from faster rcnn is incorrect? will this introduce any noise? \n\n4: For VCR tasks, it seems the VL-BERT_base w/o pre-training is performed similar compare to the with pre-training (only 0.7% lower on val of Q->A) I wonder what is the reason of this? Is this show the pre-training is not important for the VCR tasks? \n\n5: The VCR tasks also have the object bounding box correspondence, is VL-BERT take any of this supervision for input? If not, how does the VL-BERT learn the correspondence? \n\n6: For refer expression tasks, the VL-BERT_base is actually worse than ViLBERT on the detected regions. It's not a fair comparison since other models use bert-base model. \n\nOverall, I think this paper is well written and has solid experiment results. It will be great if the authors can further clarify the above questions. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByxhvaYpYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# 1. Summary\nThe paper introduces a pre-training procedure for visual-linguistic representations. The model is an extension of BERT (with transformer backbone) to deal with visual input. Images are encoded using object detectors which regions are masked at pixel level. Experiments show state-of-the-art results on different downstream tasks.    \n\nStrengths of the paper:\n* State-of-the-art results on 3 vision-language tasks\n      \nThe weak reject decision was mainly guided by the following two weaknesses of the paper:\n* Clarity of the paper needs to be improved to make the readers understanding the details of the model (see point 2 below)\n* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)\n\n      \n# 2. Clarity\nThe paper reads quite well, although some points need to be improved:\n* How were words split in sub-words (Sec 3.2)?      \n* \"For each input element, its embedding feature is the summation of four types of embedding, ...\": it is not clear how you sum embeddings. E.g., token embedding has 30k dimensions while image one has 2048 dimensions.\n* \"It is attached to each of the input elements, which is the output of a fully connected layer taking the concatenation of visual appearance feature and visual geometry embedding as input\" -> this is not clear; what output are we talking about? What is the geometry embedding? I suggest to describe the two features first and then say at the end of the paragraph that the representation is the concatenation.\n* \"For the non-visual elements, the corresponding visual appearance features are of features extracted on the whole input image\" -> what is the intuition of having the full image here? Some terms do not need to have an image associated (e.g., verbs or articles). Do you take care somehow of that?\n* Once textual embeddings are masked by [MASK], the related visual embedding (whole image) is also masked? To my understanding the answer is no: what's the intuition of this?\n* Segment embedding: is this important? This should be easy to show with an experiment in the ablation study of Table 4?\n* It seems that there is a semantic asymmetry of input to the loss during training when considering only the text information (bookscorpus) and the image-text information (conceptual captions): how is training coping with this? Doesn't it make more sense to have 2 pre-training phases: first on text information only and then on image-text information?\n\n\n# 3. Novelty and Motivation\nThe novelty of the paper is quite limited. It strongly relies on transformer networks and then recent success of BERT in the NLP domain. The proposal is an extension of these two ideas to visual domain.\n\nMoreover, there is a body of concurrent work that is very similar to the proposed idea with slight differences (ViLBERT, VisualBERT, LXBERT, UNITER, B2T2), i.e., using transformers with masking operation on the RoIs. It is not clear what is the intuition related to the differences between the methods, i.e.\n* Why one is better than the other; why should someone prefer this pre-training technique wrt others?\n* Why a unified network (this work) is preferred wrt a two-stream one (ViLBERT, LXMERT)?\nIt seems that everything heavily depends on the experiments and empirical results obtained by trying many variants during the prototyping phase. It is missing a bit of understanding and intuition on the reasons why this technique should be used.\n\n\n# 4. Experimentation\nExperiments are the strength of the paper showing state-of-the-art results on 3 vision-language tasks. Some additional analysis is missing:\n* If masking is conducted on the raw pixel, this makes training much slower since you need to perform inference many times. What is the impact in terms of accuracy? Did you carried out an experiment showing that it is better to mask raw pixels instead of conv maps?\n* How long is the model trained for?\n* What is the performance/accuracy on the pre-training tasks?\n* How important is the segment embedding?\n* Footnote 2 should be in the main text (Sec 4.1). It is too hidden, but very important to let the reader knowing about it.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new pretrained language model which can take visual information into the embeddings. Experiments showed state-of-the-art results on three downstream tasks. The paper is well written and detailed comparisons with related work are given. There are some concerns about the clarity and novelty raised by the reviewers which is answered in details and I think the paper is acceptable.
Decision: Accept (Poster)