Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training
Authors: Hongwei Xue
Yupan Huang
Bei Liu
Houwen Peng
Jianlong Fu
Houqiang Li
Jiebo Luo
Venue: NeurIPS 2021 Poster (2021)
Keywords: vision-language pre-training
Transformer
inter-modality

Abstract: Vision-Language Pre-training (VLP) aims to learn multi-modal representations from image-text pairs and serves for downstream vision-language tasks in a fine-tuning fashion. The dominant VLP models adopt a CNN-Transformer architecture, which embeds images with a CNN, and then aligns images and text with a Transformer.  Visual relationship between visual contents plays an important role in image understanding and is the basic for inter-modal alignment learning. However, CNNs have limitations in visual relation learning due to local receptive field's weakness in modeling long-range dependencies. Thus the two objectives of learning visual relation and inter-modal alignment are encapsulated in the same Transformer network. Such design might restrict the inter-modal alignment learning in the Transformer by ignoring the specialized characteristic of each objective. To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment. Specifically, we propose a metric named Inter-Modality Flow (IMF) to measure the interaction between vision and language modalities (i.e., inter-modality). We also design a novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer to further promote the inter-modality learning. To the best of our knowledge, this is the first study to explore the benefit of Transformer for visual feature learning in VLP.  We verify our method on a wide range of vision-language tasks, including Visual Question Answering (VQA), Visual Entailment and Visual Reasoning. Our approach not only outperforms the state-of-the-art VLP performance, but also shows benefits on the IMF metric.

Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
Hongwei Xue Yupan Huang? : Bei Liu', Houwen Peng", Jianlong Fu Houqiang Li", Jiebo Luo
UUiiersityy of Science and Technology of China, Hefei, China,
Sun Yat-sen University, Guangzhou, China, Microsoff Research, Beijing, China,
UUniversity of Rochester, Rochester, NY "ggh01112000 120mma11. ustc. edu. cn, "huangyp280mal12 sysu, sysu.edu.cmm edu.
{[bei. 1iu, houwen, nouven-peng, jianf Camiffmmmmmc #mmimmmiiooooore ammicrossf..com com, 'lihqQustc.ed edu. cn, "jluoêcs rochester. edu
Abstract
Vision-Language Pre-training (VLP) aims to learn multi-modal representations from image-text pairs and serves for downstream vision- langunlanguage tasks in fine- tuning fashion. The dominant VLP models adopt EXN-Transforme architecture, which embeds images with a CNN, and then aligns images and texts with a Transformer. Visual relationship between visual contents plays an important role in image understanding and is crucial for inter-modal alignment learning VLP. However, CNNs have limitations in visual relation learning due to local receptive field's weakness in modeling long-range dependencies. Thus the two objectives learning visual relation and inter-modal alignment are encapsulated in the same Transformer network. Such design might restrict the inter-modal alignment learning in the Transformer by neglecting the specialized characteristic of each objective. To tackle this challenge, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter- motel alignment. Specifically, we propose a metric named Inter-Moddlle Modality Flow (IMF) to measure the interaction between vision and language (i.e., inter-modality). We also design novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer further promote the inter-modality learning. To the best of our knowledge, this is the first study to explore the benefit of Transformer for visual feature learning in wide range of vision-language tasks, including
VLP. We verify our method Image-Tex: Text Retrieval, Visual Question Answering (VQA), Visual Entailment and Visual Reasoning. The result shows that our approach not only outperforms the state-of-the the-art art VLP models, but also exhibits superiority on the IMF metric.
Introduction
Vision-Language Pre-training (VLP) has shown great benefits for Visual-Language (VL) tasks such as Visual Question Answering (VQA), Visual Entailment, etc., in many recent works (9] 21 B1
32
51. VLP is designed to learn vision and language (VL) joint representation and alignment with a huge number of image-text pairs. It provides powerful initialization of multi- modal representation for downstream VL tasks in fine-tuning way. In this paper, we explore how to improve the multi-mon modal representation which is the key challenge in VLP from the perspective of intra-modal and inter-modal learning.
36
40
"This work was performed when Hongwei Xue and Yupan Huang were visiting Microsoft Research as research interns.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Existing works use image features as visual tokens and learn their alignments with language tokens using a multi-modal Transformer. Three main types of image representation are commonly used in VLP: region feature, grid feature and patch projection. Most VI.P models g
42
40)
extract
region-based image features with an off-the-shelff object detector (34, Each visual token in VLP corresponds to pre-defined region feature. Some recent works directly learn grid features from images by CNNs to lift restrictions of bounding boxes and pre-defined object categories [22] 21]. They train CNN and multi-modal Transformer in an end-to-end fashion enable visual features optimized for pre-training objectives. addition to use CNNs for visual feature learning, a recent work ViLT [28] directly input image patch projections into the multi-moda Transformer to achieve the fastest inference speed with the lightest VLP architecture.
In vision-language (VL) tasks, the relation between visual concepts is crucial. For example, given a question "what is the man doing?" for an image of surfing man, VL model is expected to infer the relation of "surfing" from object "man" and object "surfboard". However, existing three types of image representations fail to model the intra-vision relation. For both region feature and patch projection, each unit (i.e., bounding box patch) is independent while global relations are not encoded in the visual embedding. For grid feature learned by CNNs, the local receptive field in the convolution layer results in local features of neighbor regions. Thus the two objectives of VLP, i.e., learning visual relation and inter- model alignment, are encapsulated in the multi-modas Transformer network. Such design might restrict the inter-mode alignment learning in the Transformer by ignoring the specialized characteristic of each objective. On the other hand, for language, texts are highly structured and relations are explicitly provided by grammar. This inconsistent representation of different modalities distracts the multi-modal Transformer from the inter-modal alignment.
To better learn visual relation and further promote inter-modal aligmment, we propose fully Trans- former VLP model which adopts self-attention for visual feature learning. Self-attention breaks the spatial inductive bias and enables long-range global relation learning of visual features. Thus, multi-modal Transformer can be specialized for multi-modal joint learning. In self-attention mech- anism, each visual token is approximate weighted mixture of all tokens. The higher weight indicates higher dependency. We name this way of image feature learning as visual parsing. As visual parsing provides dependencies of each visual token pair, moder-modality learning can be further promoted by masking visual tokens with high dependency, forcing the multi-modal Transformer to have more attention the language side. To the best of our knowledge, our proposed Masked Feature Regression (MFR) is first ranking-based masking mechanism in VLP that leverages the intrinsic properties of visual extractor. To promote the inter-modality learning, we propose the Inter-Modality Flow (IMF) metric based on Attention Flow [1] to measure the fusion of inter-modality in VLP. IMF aims quantify the information flow between the two modalities.
To verify the effectiveness of our approach, we conduct experiments on wide range of vision- language tasks that are highly related to visual relation understanding and inter-modal reasoning. Our approach not only outperforms the tfate-of-the-an VLP models, but also shows superiority on the pro- posed IMF metric. We also conduct extensive ablation studies to demonstrate the effectiveness of our proposed self-attention visual parsing and parsing-basee masking mechanism. We thoroughly probe the moter-modality learning in VLP from the perspective of information flow and data distribution. Our probing reveals how vision and language fuse with each other.
Our contributions in this paper are summarized as follows:
1 We are the first to adopt self-attention learn visual features for VLP, aiming to promote inter- modality learning in multi-modal Transformer. Our model outperforms existing works on a wide range of language tasks.
2 We propose a novel Inter-Modality Flow (IMF) metric to measure and reveal vision and language fusion in VLP.
3 We design the first ranking- based masking mechanism for visual self- module to further promote inter-modality learning, verified by well-designed ablation studies.
2 Related Work
Multi-Modal Pre-training. Pioneering works of Vision-Language Pre-training like ViLBERT (3 and LXMERT [22 adopt two-stream architecture, which utilizes two separate Transformers to encode image and text modalities, and a third Transformer for multi-modal fusion. Recent works using

---

single-stream architecture directly fuse the two modalities in a Transformer to automatically learn
inter-and-intr modality fusion 40 32 31 The majority of existing works advocate the single- stream designs, which show stronger performance and include fewer parameters than two-stream models. For above reasons, we mainly focus on one-stream architecture in this paper.
Single-stream architectures typically employ Transformer to learn multi-modal contextualized fea- tures based on the singular embedding of each modality. For vision part, most VLP works use Bottom-Up and Top-Down attention [2] to extract region-level visual features by a Faster R-CNN 39 detector pre-trained on Visual Genome dataset [29]. Region-level features may miss the contex- tual information out of the bounding boxes and often suffer from low quality, noisy, and over-sampled boxes. To overcome the restrictions of region- -bsee image features, SOHO [2i] designs an end-to- end VLP pipeline which directly learns image embedding at pixel-level by a CNN. This end-to-end learning paradigm does not require bounding box annotations and enables inference 10 times faster than region-based approaches. A recent work ViLT (28) directly inputs patch projections into multi-modal Transformer to achieve fast inference speed. While ViLT has performance gaps with existing methods on some downstream tasks like VQA.
Vision Transformer. As convolutional network only captures information in a local window, it will miss correlations of long-range features. To improve the capability of encoding distant dependencies heterogeneous interactions, some works complement CNNs by extending self-attention modules The augmentation by self-attention also benefits general visual feature extraction. This kind
of method has been applied for the object detection task g 52 47.
The pioneering work of Vision Transformer (ViT) [3] totally abandons convolution and applies a Transformer architecture image patch projections for image classification. Inspired by VIT, there are some recent works studying vision Transformer for a broad range of vision tasks such as image classification [35] 43). object detection [4, semantic segmentation [50]. In this paper adopt [35] as our visual backbone. Since CNN and Transformer favor different kinds of optimizer [48], our fully Transformer structure makes it easier to train and fine-tune. We utilize unified optimizer for vision Transformer and multi-modal Transformer.
3 Approach
3.1
Self-Attention Visual Parser
Despite grid features overcome the restrictions of region- bbsood image features and keep all visual information in images (21). CNN-based image feature learning remains challenging when utilized to vision-language pre-training. Specifically, convolution network tends to focus on local regions in images, adding the learning of global visual relation to the multi-modal Transformer. Instead of concentrating on inter-modal interaction learning, multi-modal Transformer has to learn intra-model interactions in visual modality well. For the language side, texts highly structured and they explicitly provide some relations from grammar. For example, prepositions and verbs often indicate the relations of objects. Thus, the two modalities are fed into the multi-modal Transformer at different information levels. This inconsistency makes Transformer tend to Iearn stand-alons representation of each modality. As a result, the multi-modal Transformer is distracted from focusing on learning model alignment.
Inspired by Vision Transformer (ViT) 13 we apply self-attention for visual embedding in vision- language pre-training. To learn visual features, we adopt a vision Transformer VT with parameters
V VT(1 E ERR. RMX
(1)
where I is the input image. m denotes the number of visual features, and e is the dimension of hidden states. To reduce computational complexity, we adopt Swin Transformer [35), which has hierarchical structure. Although shifted windows makes the receptive field smaller than the whole image, the large window size and gradual down-sampling mechanism make the spatial inductive bias much lower than CNN. We also explicitly get the correlations of visual tokens. In the last layer of vision Transformer, the correlation of each token pair can be modeled by attention weights which track how features get mixed.
3

---

Self- solf-Attention Visual Parsing for Vision-Language Pre-training
"The men bet readies to swing pitch while the umpire looks on.'
Text Embedding
(b) Mask Generation
Pre-training Objectives
Concatenation
Language folkens
Multi-modal Transformer
(a) Visual Parsing
(c) Inter-modality Probing
Visual tokens
Visual weights
(a) Visual Parsing
(b) Mask Generation
[c] Inter-modality Probing
Visual
Random
Feeed Formurr
Julil Hormalow
weightt
Masking
1
Atti
Layer ++1
Layer !
Visual
Vision Transformer
weightt
Figure 1: Pipeline of our Self-Attention Visual Parsing for Vision- Vision-Language Language Pre-training, (a) The Visual Parsing module applies a vision Transformer to learn visual representations. (b) The Mask Generation module generates masks for visual and language tokens, which are concatenated and passed a Multi-modal Transformer for joint embedding learning. (c) Inter-modality Probing measures the vision-language alignment with proposed IMF metric.
3.2 Inter-modality Probing
In single-stream structure for Vision-Language Pre-training, multi-modal Transformer is used to fuse the vision and language modality. To quantify the interactions between two modalities, we propose a metric named Inter-Modality Flow (IMF) to probe how vision and language fuse together.
Transformer is a stack of self-attention modules and feed forward neural networks. The interactions of image-and-text learnt in self-attention modules. Following Attention Flow [1] first compute values in layer after excluding shared operations on each tokens as:
Ti+1 = Ti + W Wauli,
where Ti and Ti+1 are the input and output of layer L. Wt layer I's attention weight. To account for residual connections, weights of mixture can be represented as:
A' 0.5(T + W
To track down the information propagating in Transformer layers, we first calculate attention flows of arbitrary two layers, e.g., layer and layer j where j )2 i:
A
II A a=)
This calculation sums over all possible paths between two layers, and A is the attention weight matrix of layer i's input to layer j's output. Specifically, when A represents attention weights in layer i. To measure the interactions between two modalities, we propose an Inter-Modality Flow inter and it is computed as the proportion of inter-modal attention among all attentions:
pdd inter
A Ant + A
(A
(5)
where Ald A is the summation over all attentions of visual and language inter-modality features, while A3d intra sums over one modality, vision or language:
A Anter
Σ Aij.
A
Fnntra
Σ
A
(6)
Ointer

---

where Pinter f E V. or o VJ and Pintra = f V or I, L}. L and V token index set of vision and text. We calculate F modalities interact with each other.
inter pre-training data to probe how the two
3.3 Pre-training Pipeline
Model Overview.
Figure shows the overview of our vision-language pre-training framework. Our model is composed of a vision Transformer and a multi-modal Transformer. The vision Transformer takes image as input and outputs the visual tokens
{uı, 02 12;,..; Um. To encode spatial information of images, we utilize 2-D position embedding computed by sine function following We apply a Linear layer and Layer Normalization 3 to embed the vision tokens. For the input sentence, we follow BERT I to tokenize and get word embeddings W. We concatenate vision and language tokens (V and W) to form an input sequence for multi-modal learning. Similar to other VLP models, we add two special tokens [CLS] and [SEP] into the input sequence to indicate classification position and the separation of two modalities, respectively. A multi- layer Transformer is adopted to take the joint vision- language input, and outputs the attended features. In order to distinguish from the vision Transformer VT, we call it multi-modal Transformer MT.
other works 88 [3]
We adopt three pre-training tasks in our model: Masked I. Language Modeling (MLM), Image-Text Matching (ITM) and Masked Feature Regression (MFR). Among them, MLM and ITM two commonly used pre-training task and MFR is novel task which is proposed to mask visual tokens with similar or correlated semanties our framework.
Masked Language Modeling.
We adopt Masked Language Modeling (MLM) following most
vision-language pre-training works from the contextualized tokens:
9
21 28 to predict the ground-truth labels of masked text tokens
CVLM -E log (w; W1, 1))
We adopt the same masking strategy as in BERT, and use Linear layer as the MLM head to output logits over vocabulary, which are then computed as the negative log likelihood loss for the masked tokens.
Image- Text Matching. To learn the inter-modal alignment, we adopt Image-Ters Text Matching (TTM)
task for pre-training as in most vision-languas pre- vre-training works (9 21 28). With the probability of 0.5, we randomly replace the aligned image to a different image. We use a single linear layer as ITM head to predict logits y over binary class (match not), and we compute negative log likelihood loss as our ITM loss:
LIM V logp(y W)
(8)
We also design vision-language token alignment (VLA) task inspired by the word region alignment objective in 9 (88). Our VLA loss optimize the Optimal Transport (OT) distance approximated by
IPOT 46 Following 9 28
add the VLA loss multiplied by 0.1 the ITM loss.
Masked Feature Regression. Without bounding boxes, random masking for visual feature regres- sion loses its effectiveness the model will directly copy from neighbor features [2I]. By visual parsing introduced in Section 3.1 correlations of each visual token are explicitly modeled by attention weights. Similar to A in Equation that visual tokens of high attentions share similar semantics or correlations. We first randomly pick visual token to mask, and then mask tokens with top-k attention weights. We apply I.2 regression between masked tokens and regressed features:
the last layer's attention weight for masking. We assume
LMFR
Σ
v
v
(8) 2
1=1
The vision Transformer and multi-modal Transformer are trained end-to-end with above objectives:
Spre -MLM + CITM + A2eMER
(10)
5

---

Table 1: Evaluation of image-to-text retrieval (TR) and text-to-image retrieval (IR) on FKickr30K
dataset. "." indicates the detail is not reported.
Method
VSF++114]
SCAN[30
ViLBERT [3
Unicoder 31
U
ViLT [2
SOHO
Ours
@@1 $2.9
67.4
86.2
85.9
83.7
86.5
87.0
TR R@5
80.5
90.3
96.3
97.1
97.2
98.1
98.4
R@10
87.2
95.8
99.0
98.8
98.1
99.3
99.5
R@I
39.6
48.6
58.2
71.5
72.5
62.2
72.5
73.5
R@5
70.1
77.7
84.9
90.9
92.4
87.6
92.7
93.1
R@10 10
79.5
85.2
91.5
94.9
96.1
93.2
96.1
96.4
4
Experiments
4.1 Pre-training Details
We follow the dataset settings of SOHO 21 for pre-training. We focus on in-domain datasets: MSCOCO Captions (MSCOCO) 35 and Visual Genome Dense Captions (VG) (29], which are typical in-domain datasets for many VL downstream tasks. When comparing with UNITER (9) we use its in-domain pre-training results for fairness.
We follow BERT to adopt the WordPiece tokenizer [49 to split a sentence into word tokens. We adopt Swin Transformer for vision Transformer and 12-layer Transformer for multi-modal Transformer. If not specified, we use 384 x 384 as input resolution. This resolution is much lower than the resolution of 600 x 1000 or 800 1333 adopted by most previous works 336
21 28 Increasing the resolution
will likely further improve our performance as per the findings by 125 Our models are initialized based on ImageNet [10] and BERT I.] We use AdamW optimizers for vision Transformer with learning rate 5e-6 and multi-modal Transformer with learning rate 5e-5 empirically. Empirically, we set the Equation as 7. We set the X and X2 in Equation 10 as and 0.01 respectively. Our model is pre-trained on 8 NVIDIA Tesla V100 GPUs with a batch size of 2048. Following SOHO [11, the learning rate is warmed up for the first 500 iterations. The training process takes 40 epochs until convergence and the learning rate decays by 10 times at 25th and 35th epoch. To reduce memory cost, we pair an image with four texts in each batch, including two matched pairs and two unmatched pairs. MLM and MFR tasks are only calculated with the matched image-text pairs.
4.2 Downstream Tasks
We evaluate our method by fine-tuning the pre-trained model on downstream vision- languaee tasks. As we particularly focuses on visual relation and modal learning, we choose four tasks related visual relation understanding and inter-modal reasoning: Image-Texx Retrieval, Visual Question Answering (VQA), natural language for visual reasoning (NLVR), and fine- fine-grained visual reasoning (Visual Entailment). We compare our model with several task-specific and pre-training models. Among them, SOHO (1 and ViLT (28 are end-to-end VI.P models like ours. SOHO, Vil.T and our model adopt CNN, Linear projection and Transformer for visual embedding learning, respectively.
Image-Text Retrieval. Image-text retrieval aims to retrieve the most relevant text from candidate images, vice versa. Image-text retrieval includes two sub-tasks: image-to-ter retrieval (TR) and text-to-image retrieval (IR). We follow the same practice SOHO to conduct image-text retrieval for fair comparisons. During training, we construct image-text pairs mini-batch by sampling aligned pairs from ground-truth annotations, and unmatched pairs from other captions within the mini-batch. To predict whether an image-text pair is aligned or not, we use the joint embedding representation of the [CLS] token from Transformers perform binary classification. Since the binary classification objective of image-text retrieval model consistent with the image-text matching (ITM) task in pre-training stage, we initialize the task-specific head from the pre-trained ITM head for better initialization. We adopt AdamW optimizer with learning rate of 5e-5. The mini-bate size is set to 32. We train 10 epochs until convergence and decay the learning rate by half at 5th epoch empirically.
Experiment results on Flickr30k 38 are shown in Table T Our model outperforms ViLT and SOHO under all metrics on Flickr30k. The promising results of our model on image-text retrieval indicate the advantage of our fully Transformer architecture for learning cross- mooss-modal alignment.

---

Table 2: Evaluation of VQA on VQA 2.0 dataset. indicates the detail is not reported.
Table 3:
Evaluation of Visual Reasoning on
NLVR² dataset.
Model
test-dev
test-std
Model
dev
test-P
MUTANES
60.17
MAC123
50.80
51.40
BUTD2
65.32
65.67
Text Only
50,90
51.10
Unified VLP
70.50
70.70
Image Only 41
51.60
51.90
ViLBERT
70.55
70.92
CNN+RNN
41
53.50
52.40
VisualBERT
70.80
71.00
MaxEnt 41
54.10
54.80
VLBERT 40
71.79
72.22
VisualBERT
67.40
67.00
LXMERT 42
72.42
72.54
LXMERT
74.90
74,50
UNITER
72.70
72.91
UNITER 9
75.85
75.80
ViLT-B/32-Aug 28
70.94
ViLT-B/32-Aug/23
75.24
76.21
SOHO
73.25
73.47
SOHO
76.37
77.32
Ours
74.00
74.17
Ours
77.61
78,05
Table 4: Evaluation of Visual Entailment on SNLI-VE. "-" indicates the detail is not reported.
Method
EVE-Image[451
e-SNLI-VE-20012
UNITER142
SOHO
Ours
val
71.56
73.02
78.59
85,00 84.95
84.75
test 71.16
78.28
85.08
Visual Question Answering. Visual Question Answering (VQA) aims to give answer for question and an image. VQA task is typically formulated as a classification problem according to the majority of existing works. We adopt multi- perception head to output logits from the [CLS] token. Following [27]. we optimize the model by a binary cross-entropy loss on 3,192-way classifications. We fine-tur mee-tune for 100 epochs with batch size of 512 until convergence. We keep the optimizer the same as in the pre-training stage, and we decay the learning rate by 10 at the 65th and 75th epochs empirically.
We present the experimental results on VQA v2.0 dataset in Table 2 Our model obtains 0.75% and 0.79 0.7% absolute gains on test-dev and test-std splits over SOHO respectively. The results validate the effectiveness of modeling the intra-modality visual information with self-attention mechanism to facilitate intelligent visual question answering.
Visual Reasoning. Visual Reasoning with Natural Language (NLVR) aims to predict whether a image-text pairs, text is related given pair of images. With the requirement of comparing comparisons, and NLVR further focuses on the compositional visual reasoning abilities of relations, quantities. For this task, we evaluate our model on NLVR dataset [41]. As there two input images instead of one input image used in the pre-training setup, we follow the pair method introduced in UNITER [9] input
image-text pairs to Transformer and get two embedding vectors from [CLS) tokens. Then we learn a classifier that takes the concatenation of the embedding vectors to infer "true" "false" by cross-entropy loss. We fine-tune for 80 epochs with batch size of 512 until convergence. We decay the learning rate by 2 at the 60, 65, 70, 75, 78, 79th epochs empirically.
Results shown in Table We observe clear improvement of model over previous state-of- the-art model SOHO, where ..24% and 0.73% absolute accuracy improvements are obtained on the val and test splits respectively. Over 2% improvements made over ViLT. Our promising results demonstrate our advantage in compositional visual reasoning by promoting inter-modal learning with fully Transformer architecture.
Visual Entailment. Visual Entailment (VE) is a fine-grained visual reasoning task to infer whether an image semantically entails text. To classify the more tnn-ggriied relationship than NLVR be- tween an image and a text pair, VE aims to infer the mageetto-eexy relationship to be true (entailment), false (contradiction) or neutral. For this task, we evaluate our model on SNLI-VE dataset [5] which is constructed based on Stanford Natural Language Inference (SNLI) 6 and Flickr30K 38 datasets. We follow 9 121 to perform the VE task as three-way classification problem. The model predicts the scores of each class by Linear layer on the representation output by the Transformer from the [CL.S] token. We fine-tune the model for 25 epochs with batch size of 512 until convergence. The learning rate is initialized 5c-5, and decayed by 10 at the 15th, 20th and 23th epochs empirically.
7

---

Table Ablation Study on the effectiveness of self-attention visual parsing and the proposed Masked Feature Regression (MFR) objective. A 3-layer Transformer is adopted for multi-modal Transformer. For the backbone visual embedding, R50, R101, Swin-T and Taan indicate ResNet 50, ResNet 101, Swin-tiny Transformer and language-spectnn Transformer respectively. FLOPS and Parameter numbers of different visual embedding backbones are listed.
Backbone
FLOPs (G)
Params (M)
Objectives
intter
F.3
VQA dev
VQA std
R50
25.6
MLM+ITM
0,437
64.65
65.12
R50
25.6
MLM-ITM++MRRNN
0.439
64.89
65.46
R101
7.8
44.5
MLM+ITM
0.441
65.44
65.71
Swin-T
4.5
28.0
MLM+ITM
0,461
67.13
67.40
Swin-T
4.5
28.0
MIM-TTMMMMRRRaa
0.463
67.24
67.41
Swin-T
28.0
MMM-IIM--MR
0.474
67.74
67.90
Swin-T+TZan
4.7
48.0
MIM-ITMMMER
0.485
67.96
68.23
Parsen
Random
"A person is surfing"
"A plane coming in for landing"
"Boy laying with frisbee"
"Giraffe running through field"
Visualization of our visual parsing. The first is masking based visual attention weights. while the second row is random masking. We randomly sample and show a related caption of each image.
Figure 2:
Table compares models on SNLI-VE dataset. Here see that our model and SOHO are significantly better than UNITER with about 6% absolute gains on accuracy due to our end-to-end training architecture. Our method is comparable with SOHO on the val split and is slightly better than SOHO on the test split.
Ablation Study
To validate the effectiveness of our visual parsing for vision-language pre-training, we conduct ablation studies the VQA task. To compare CNN and Transformer on visual embedding, we adopt ResNet 50 and 101 [19] which respectively have comparable and much larger parameters and FLOPs compared with Swin-Ting Transformer. For multi-modal Transformer, we use 3-layer Transformer. We use the same pre-training and VQA settings in Section 4.1 except for image resolution. We set resolution as 224 224 in all ablation studies for fair comparison. We also study the proposed Masked Feature Regression (MFR) objective to verify the effectiveness of masking based on the parsing attention weights. Specifically, we design four settings: without MFR, random masking (MFRRand). masking based on parsing weights (MFR), and variant (MFRcxx) designed for CNN which measures correlations by cosine distance. We set & Equation 9 as visual tokens number m times the probability of random masking p. k p.

---

0.6
0.3
0.2
0.1
10
12
10
11
12
VITT22)
Figure 3: Inter-Modality Flow (IMF) from the input to each layer of the multi-modal Trans- former Fl. J [1, 12] calculated ViLT 288
Figure 4: Inter-Modality Flow (IMF) within each
layer of the multi-modal Transformer P
inter: i E
[1, 12] calculated on ViLT (28). SOHO 21
SOHO 21 UNITER 9 and our model.
UNITER 9 and our model.
06
04
02
10
Figure 5: NMI scores calculated on Visual Genome for multi-modal fusion probing from the view of data distribution. A smaller NMI value indicates higher fusion degree. We compare NMI scores of ViLT (28). SOHO [2I]], UNITER [ and our model.
Results are presented in 3 The Inter-Modality Flows of all Swin-Tiny backbones are higher than ResNet 50 and ResNet 101. This shows that vision Transformer provides better visual embedding for inter-modality learning compared with CNNs. Although the parameters and FL.OPs of the ResNet 101 backbone more than 1.5 times of our Swin- Tiny Transformer backbone. the latter achieves better result by about 2% on the VQA score. Inter Modality Flow verifies the superiority of self-attention visual embedding in promoting inter-modality learning. From this view, we further try add Lunguage-speciff Transformer Lan which process the input language before feeding into cross- modal Transformer. From the results, Laa can also promote performances. For fair comparison. we do not include Tan in our final model.
To study the proposed Masked Feature Regression (MFR), the last lines show that the masking based on the attention weights further promotes the interaction of two modalities. MFRCNN also shows improvement yet less significant than MFR. To intuitively demonstrate the attention-based masking, we sample some results and visualize the mask of visual tokens. Compared with random masking, iitention-based masking tends to mask regions with similar semantics. In Figure 2 random masking lacks the ability of preventing the multi-modal Transformer from directly copying from neighbor regions. While attention- based masking mechanism forces the model to refer the language side for inferring masked features.
4.4 Inter-modality Probing
In this section, we aim to verify our model's effectiveness of promoting inter-modality learning and better study how the two modalities fuse together. We choose UNITER (9] SOHO [21] ViLT 22 and our model models that use region features, grid features, patch projections, features learned by self-attention, respectively. These four models all adopt 12-layer Transformer for fusion. The fusion of two modalities can be demonstrated from two perspectives: the inter-modal interaction and feature distance of two modalities the joint space. Thus we the inter-modality probing by our proposed Inter-Modalle Flow (IMF) and distance of data distribution measurement. We will discuss the difference between these two perspectives in this section.
Inter-Modality Flow. Inder-Modality Flow is proposed to quantify interaction between vision and language modalities. Majority of existing works VLP fuse the vision and language modality

---

by multi-moda Transformer. Our proposed Inter-Modality Flow measures the information flow between two modalities from layer i's imput to layer j's oulput. larger information flow implies more interactions between image and text. We design two Inter-Modality Flow probing tasks layer by layer in the multi-modal Transformer. We calculate the F F E [1, 12] to track the process of inter-modality learning and F inter: E
[1, 12] to probe each layer's impact modality fusion.
From the result shown in Figure 3 our model has larger IMF from the input tokens to every layers consistently. This indicates that there more interactions between image and text in our model. SOHO and UNITER have comparable IMF. ViLT has much smaller IMF than other methods, as patch projections learn little about visual relationships. The result shown in Figure indicates that the interactions gradually increase in deeper layers. It is worth noting that the interactions decrease a little in the last few layers as each modality optimizes by different pre-training tasks such MLM and MFR. For the models in these two probing tasks, the IMF is positively correlated with the performance of downstream tasks, which testify to the importance of inter-modality learning in VLP.
4
Data Distribution. Following 77. we study the multi-modal fusion degree of model from the view of data distribution. In every layer, we extract all visual and language features and apply the k-means algorithm (with k = 2) on features to partition them into two clusters. We measure the difference between the k-means clusters and ground- truth visual/textual clusters via Normalized Mutual Information (NMI) which is an unsupervised metric for evaluating differences between clusters. A larger NMI value implies more significant distinction between clusters, indicating lower fusion degree (44]. We average all NMI scores on data samples from Visual Genome.
From Figure 5 SOHO and our model has lower NMI scores than UNITER in every layer consistently. One explanation that end-to-end training of visual backbone reduces the distinction between two modalities' data distributions. The NMI scores of UNITER gradually decrease while SOHO and our model's NMI score fluctuate within small values. ViLT, layers with large and small NMI both exist. NMI of ViLT is overall larger than the other three models as it lacks visual backbone to optimize visual features towards alignment to word embeddings. As NMI only evaluates differences between data distributions, it lacks the capacity of measuring inter-modal interactions. Thus, our proposed Inter-Modality Flow is better correlated to inter-modal alignment.
5 Conclusion
In this paper, we explore the benefit of self-attention for visual feature learning in Vision- Language Pre-training (VLP). We explore CNN's limitation in intra-vision modality learning by introducing fully Transformer for visual embedding in VLP. We also design novel masking optimization mechanism named Masked Feature Regression (MFR) in Transformer to further promote the inter- modality learning. To verify MFR and probe the mechanism of inter-modal alignment, we propose Inter-Modality Flow (IMF) to measure the interaction between vision and language modalities. In the future, we will further explore the different properties of various modalities and design a unified framework for inter- modaaiyy interaction 20) Another potential direction is to involve more than two modalities in inter- modality learning to enable more powerful understanding and interactions, such as image tags [16 [5].
Acknowledgments and Disclosure of Funding
Funding in direct support of this work: NSF of China under Grant 61836011 I and 62021001, GPUs provided by Microsoft Research Asia. Additional revenues related to this work: Internship at Microsoft Research Asia.
References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, 2020.
[2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question
10

---

answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086, 2018.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton, Layer normalization. arXiv preprint arXiv: 16060066500 2016.
[4] Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer-basee object detection. arXiv preprint arXiv:2012.09958, 2020.
[5] Hedi Ben- Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE international conference
on computer vision, pages 2612-2620, 2017.
[6] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. EMNLP, 2015.
[7] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained iision-and-language models. In European Conference on Computer Vision, pages 565-580. Springer, 2020.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. and--t--end object detection with transformers. In European Conference Computer Vision. pages 13-2299. Springer, 2020.
(9] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pages 104-120. Springer, 2020.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
Ieee, 2009.
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.
Proceedings the the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
[12] Virginie Do, Dana-Maria Camburu, Zeynep Akata, and Thomas Lukasiewicz. e-snli-ve- 2.0: Corrected risual-textual entailment with natural language explanations. arXiv preprint arXiv:2004.0377 037444. 2020.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 19999, 2020.
[14] Fartash Faghri, David Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05611 2017.
[15] Jianlong Fu, Tao Mei, Kuiyuan Yang. Hanqing Lu, and Yong Rui. Tagging personal photos with transfer deep learning. Proceedings of the 24th International Conference on World Wide Web, pages 344-354, 2015.
[16] Jianlong Fu, Jinqiao Wang, Yong Rui, Xin-Jing Wang, Tao Mei. and Hanqing Lu. Image tag refinement with view-dependent concept representations. IEEE Transactions on Circuits and Systems for Video Technology, 1409-144091122 2014.
[17] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3146-3154, 2019.
[18] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-69 6913, 2017.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
11

---

[20] Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi-directional image and text generation. In Proceedings of the 29th ACM International Conference Multimedia, pages 1138-11444 2021.
[21] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision- langugg representation learning. arXiv preprint WIXfv:2104.03135, 2021.
[22] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.008: 00049, 2020.
[23] Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. In International Conference on Learning Representations, 2018.
[24] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL 2019-57th 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[25] Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Leamed-Miller,, and Xinlei Chen. In defense of grid features for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10267-10276, 2020.
[26] Andrej Karpathy and Li Fei-Fei. Deep visual- alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2233137, 2015.
[27] Jin-Hwa Kim, Jachyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 1571 1581, 2018
[28] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision languuuge transformer without convolution or region supervision. arXiv preprint arXiv:2102.033344 2021.
[29] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:3)3. 2017.
[30] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European Conference Computer Vision (ECCV), pages 201-216, 2018
[31] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13336-13444 2020.
[32] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXh::1908.03557, 2019.
[33] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
[34] Bei Liu, Zhicheng Huang, Zhaoyang Zeng, Zheyu Chen, and Jianlong Fu. Learning rich image region representation for visual question answering. arXiv preprint urXiv:/9100 /3077, 2019.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
[36] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for International Conference on Neural Information Processing Systems, 2019.
langugg tasks. In Proceedings of the 32nd
[37] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018
12

---

[38] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting segion-to-phrasee correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[39] Shaoqing Ren, Kaiming He, Ross B Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
[40] Weijie Su, Xizhou Zhu, Yue Cao,, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vi-bert: Pre- training of generic visual-linguisic representations. In International Conference on Learning Representations, 2019.
[41] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019.
[42] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IICNLP), pages 5103-5114, 2019.
[43] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 2020.
[44] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7794-7803, 2018
[45] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment task for visually- grounded language learning. arXiv preprint 1rXiv:181 1000:881..000 10588, 2018.
[46] Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact wasserstein distance. In Uncertainty in Artificial Intelligence, pages 433-453. PMLR, 2020.
[47] Zhaoyang Zeng, Bei Liu, Jianlong Fu, Hongyang Chao, and Lei Zhang. Wsod2: Learning bottom-up and top-down objectness distillation for weakly-supervised object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8292-8300, 2019.
[48] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank J Reddi, Sanjiv Kumar, and Suvrit Sra. Why adam beats sgd for attention models. arXiv e-prints, pages arXiv-1912, 2019.
[49] Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334 4343, 2019.
[50] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang. Philip HS Torr, al. Rethinking semantic segmentation from sruuencc-to-sequenee perspective with transformers. arXiv preprint arXiv:2012.15840, 2020.
[51] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041-13049, 2020.
[52] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04155 2020.
A Experiment Statisties
We summarizes the statistics of all our pre-training and downstream tasks in Table 7
13

---

Table 6: Statistics of different tasks. Notation "*" denotes Karpathy split 26 not applicable.
Notation "_" denotes
Task
Dataset
Train Split
Test Split
Metric
VG 29
train
Pre-training
MSOCO 33
train+restval"
Image- Text Retrieval
Flickr30K
38
train
test
Recall@ 1,5,10
Visual Question Answering
VQAZ.0 [18
train+val
lest-dev/test-std
VQA-score [[8]
Visual Reasoning
NLVR 44
train
dev/test-P
Top-1 Accuracy
Visual Entailment
SNLI-VE 45
train
val/test
Top-1 Accuracy
Model Size
As performances strongly related to image size and Transformer token number which affect FLOPs, we consider FLOPs as model size, we list FLOPs of models in detail:
Table FLOPs of Vision- LaiguaL pre-training models.
Visual Feature Type
Model
FLOPs (G)
ViLBERT
958.1
VisualBERT
425.0
Region
LXMERT
952.0
UNITER
949.9
UnicoderVL
419.7
CNN
SOHO
125.2
Linear
ViLT
55.9
Self Attention
Ours
84.28
From the results of above table, we can see that our model has much fewer FLOPss than SOHO and region-based models.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] our supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In our supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
14

---

(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "Zb5A4WUTbUq", "reviewer": "Reviewer_Wohx", "summary": "This paper proposed to use the vision transformer as the visual backbone for vision-language pre-training. To quantize the information flow transferred between vision and language modalities, the authors proposed a metric called Inter-modality flow that aggregates the attention weight across different layers. Experiments on VQA, visual entailment, and visual reasoning validates that the proposed method achieves state-of-the-art performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n\n- Using visual transformers as the backbone to extract visual features is a neat and intuitive idea. The whole architecture is more consistent with the multi-modal transformers. The proposed IMF seems to have correlations with the downstream task performance, which might be a good indicator to measure the pre-trained vision language model without downstream tasks. \n\n[Weakness]\n\n- My major concern of this paper is the lack of novelty. Visual parsing model is basically the vision transformer backbone, Masked feature regression is very similar to other pre-training objectives and has been used in previous work. The inter-modality flow can only measure the interaction between different modalities, \n\n- For masked feature regression, the proposed mask the token with top-k attention weight, is it possible to mask based on the proposed IMF metric? \n\n- The paper only tests with 3 downstream tasks (VQA, NLVR, Visual Entailment). Some other tasks which might require more vision-language interaction (VCR) are not tested in the paper. What is the IMF across different tasks? For example, is the IMF required in VQA different from NLVR? It will be good to add these ablation studies. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper didn't provide texts about limitations and societal Impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "Sk2YCfnIwbP", "reviewer": "Reviewer_9tzA", "summary": "This paper provides a novel way of applying masked feature regression (MFR) to transformer-based visual encoders. In addition to this new way of MRM, the paper also provides a quantifiable measurement of how two modalities (image and text) interchange their information with the aids from the attention flow method. This quantity, the inter-modality flow (IMF), is then used to analyze the characteristics of different vision-and-language pretraining models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a novel way of applying masked feature regression (MFR) to transformer-based visual encoders. In addition to this new way of MRM, the paper also provides a quantifiable measurement of how two modalities (image and text) interchange their information with the aids from the attention flow method. This quantity, the inter-modality flow (IMF), is then used to analyze the characteristics of different vision-and-language pretraining models.\n\nQuestion 1. I wonder if MFR can be applied to the features of non-transformer visual encoders (CNNs) by adopting correlations between spatial features other than attention weights. If so, I am eager to see how it can boost the performance of Pixel-BERT, which does not have a visual-related masked modeling objective.\n\nComment 1. The main model uses a Swin transformer as the visual encoder. Since Swin uses local window attention to build hierarchical structures out of vision transformer, it also has a local receptive field like CNN. Thus, the CNNs' limitations in visual relation learning mentioned in the abstract can also be applied to Swin. I believe this delineation should be revised.\n\nComment 2. The authors model each layer's inter-modality flow by *indexing* the *contextualized vector sequence* with the indices assigned to each modality before the first attention layer. However, the information of each index is mixed all together by processed through layers after layers. Thus, the vectors can not be considered a hundred percent text modality or a hundred percent image modality. The near-zero NMI at layers 8 to 11 of ViLT (figure 5) makes this speculation more concrete since the near-zero NMI means that two modalities are heavily mingled.\n\nQuestion 2. To further analyze Comment 2, I am curious about how the indices are assigned to each cluster changes over layer in figure 5. Will the text and image indices be assigned to the same cluster for every layer?\n\n### Remarks\n[+] (Clarity) Overall, the paper is well written and easy to understand.  \n[+] (Originality) I find originality out of both (1) using the attention weight to group the patches to mask and (2) employing the attention flow to analyze the inner interactions of the modalities.  \n[+] (Significance) The proposed MFR is significant because it liberates vision-and-language pretraining methods' visual masked modeling from its pre-trained object detectors, which are used to obtain the target distributions of masked regions. However, MFR is only applicable to transformer-based visual encoders, limiting its usage for CNN visual encoders. (Question 1)  \n[-] (Quality) The limitation of CNNs is too exaggerated. (Comment 1)    \n[-] (Quality) The inter-modality flow should be further analyzed. (Comment 2 and Question 2)  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors fairly well addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "Iq1-AdmN75f", "reviewer": "Reviewer_1TrJ", "summary": "This paper proposes a transformer based visual backbone for vision and language understanding. It uses a vision transformer backbone (Swin Transformer) to extract visual representation, which is then passed to a mask generation module. The visual tokens(including masked tokens) and text tokens embeddings are fed to a multimodal transformer model, where different pretraining objectives are applied. This architecture achieves state-of-the-art results in VQA, SNLI-VE, NLVR^2, Image-Text Retrieval. Authors propose an Inter Modality Flow metric that measure interaction between the visual and textual modalities and correlates well with the downstream results.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\n- It is impressive that the model achieves state-of-the-art results with a smaller input image dimension : 384x384 compared to previous works that use much larger image input dimensions like 600×1000 or 800×1333 which were difficult to train end to end due to huge computational cost. \n- Results on downstream tasks shows that with limited pretraining data but better visual representations, good accuracy can be achieved.\n- The Inter Modality Flow metric seems to correlate well with the downstream results. This can act as a strong validation metric to monitor during pretraining on multimodal datasets. \n\nWEAKNESSES and REMARKS\n\n- It is not clear to me from the ablation studies where the performance improvement is coming from. From Table 4. ablations, switching the backbone to SWIN-T resulted in the biggest improvement in accuracy. Is it because Swin Transformer has already learnt better visual features to start with and hence provides better results when multimodal pretraining is performed with visual features extracted from that? In general, in the vision and language community it has been observed that improved visual backbone helps to achieve superior downstream results on tasks like VQA etc. For example, the recent work [1], shows that a very powerful image backbone can generate representations of a richer collection of visual objects and concepts and hence get impressive downstream results in V&L tasks.\n\n      [1] Zhang, Pengchuan, et al. \"Vinvl: Revisiting visual representations in vision-language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n- For fair comparison with other end to end models like SOHO, authors should provide comparative results with SOHO model where it’s CNN encoder is replaced with a Swin Transformer of similar capacity as used in this work.\n- It is not clear which Swin Transformer(Swin-T, Swin-S, Swin-B, Swin-L) is used for different experiments reported in Table 1, Table 2, Table 3. It will be better if parameter counts are also reported for different approaches in Table 1, Table 2, Table 3 for fair comparison and understanding whether the improvements correlate to param count. \n- MFR(Masked Feature Regression) metric in different forms has already been explored in previous works like VilBERT, UNITER etc. The only difference in this work as I understand is that the visual features are masked by masking all top-k attention weights for a particular randomly picked token. Given that, I am not sure this is a novel contribution. \n- The improvement from adding the MFR task is relatively very low (+0.5) improvement over no MFR. Is 0.5 a statistically significant improvement? It is usually better to take average of multiple runs when the gap is too small. \n- IMF metric is used for evaluating different models. It is not clear how IMF is used during the training process or how it helps to improve the model : whether it is used during pretraining as a validation metric to pick the best checkpoint or just used was a framework to evaluate different models after they are pretrained? If it is the former, then for fair comparison, IMF metric should also be used during pretraining of UNITER, SOHO etc to understand the value of the metric to give a better pertained model. However, IMF metric is not shown in ablation studies in Table 4, which makes me assume it was not used as a metric to monitor the model during pretraining.\n- I am unsure about the added novelty of this work, it seems that majority of the performance improvements come from a better visual encoder backbone, and although the IMF metric shows the model to be better than SOHO/UNITER, it seems much of it is due to using a better vision encoder (Swin Transformer vs R/X-101). \n\n\nI would be interested to see the discussion and authors' replies to the concerns during the response stage. I am overall leaning towards improving the rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please consider adding the Broader Impact Statement to the main paper rather than in the Appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "73yQD-okFte", "reviewer": "Reviewer_mRRK", "summary": "This paper introduces Swin Transformer in the visual domain for vision-language pretraining. Previous VL pretraining models feed image tokens and text tokens together into the cross-modal transformer, and the visual relationship learning and inter-modal alignment are encapsulated in the same transformer network. The authors propose that visual relationship between different visual objects or concepts is important for inter-modal alignment learning, and there should be independent processing of visual relationship in the framework. Motivated by this hypothesis, the authors add a visual-only transformer to learn visual relationship before feeding the image and text tokens together to the cross-modal transformer. In addition, the authors propose a metric named inter-modality flow (IMF) to measure the cross-modal interactions. They also propose masked feature regression as a new pretraining task. Pretraining is conducted on in-domain datasets COCO and Visual Genome. Evaluation is conducted on three downstream tasks VQA, visual reasoning, and visual entailment.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The motivation is clear and the method is clearly written.\n\n2. The inter-modality flow is a novel metric to reflect the cross-modal interactions of different vision-language frameworks. The authors have deep discussion into this metric, and compare the IMF of the proposed approach and previous appraoches. The data distribution analysis in page 9 is also interesting and indicates that the proposed model learns better aligned vision-language features.\n\n3. The vision-only structure for learning visual relationship of different objects has not been explored before. This paper makes the first attempt on introducing transformer-based visual relationship reasoning before the cross-modal transformer. I think it might inspire future work, discussions, and analysis.\n\nWeakness:\n\n1. The effectiveness of the swin transformer is not fully validated in the ablation study. To make fair comparisons, at least two experiments should be conducted. (1) Compare swin transformer with other vision transformers, such as ViT, to validate the superior of the swin transformer structure. (2) Compare with a baseline model that does not have the visual parsing module, i.e., the image patches are directly projected into visual tokens and then feed into the cross-modal transformer. This experiment can validate how well the visual parsing module works.\n\n2. Why there is a vision-only transformer, but not a language-only transformer, before the cross-modal transformer. I would expect an ablation study experiment on adding a language parsing module which uses a transformer to process the input sentences. I am curious whether adding the language transformer will help or harm the performance.\n\n3. The pretrained model is not evaluated on the downstream tasks of image captioning and text-image retrieval, which I think are two important downstream tasks for vision-langauge pretraining.\n\n4. In Lines 298-299, \"ViLT has much smaller IMF than other methods, as patch projections learn little about visual relationships.\" I do not totally understand it. I understand that smaller IMF indicates less inter-modality interactions, but how does it related to the visual relationships?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not discuss much on limitations and societal impact. I encourage the authors to add some discussions on the potential limitations, problems, or failure cases of the proposed method.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially received divergent recommendations, but after the detailed author response all reviewers lean to accept the paper.

I recommend accept under the expectation that authors will revise the paper according to the detailed author response, addressing the reviewers' concerns. This includes, but is not limited to the following points
1) clarifications & delineations, including to prior work.
2) Additionally results and ablations (tables) provided in author response
3) details on model size in FLOPs
[some can go to supplement if they don't fit in the page limit]
Decision: Accept (Poster)