VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
Authors: Hangbo Bao
Wenhui Wang
Li Dong
Qiang Liu
Owais Khan Mohammed
Kriti Aggarwal
Subhojit Som
Songhao Piao
Furu Wei
Venue: NeurIPS 2022 Accept (2022)

Abstract: We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Specifically, we introduce Multiway Transformer, where each block contains a pool of modality-specific experts and a shared self-attention layer. Because of the modeling flexibility of Multiway Transformer, pretrained VLMo can be fine-tuned as a fusion encoder for vision-language classification tasks, or used as a dual encoder for efficient image-text retrieval. Moreover, we propose a stagewise pre-training strategy, which effectively leverages large-scale image-only and text-only data besides image-text pairs. Experimental results show that VLMo achieves state-of-the-art results on various vision-language tasks, including VQA, NLVR2 and image-text retrieval. 

VLMo: Unified Vision-Language Pre-Training with Misturre-f-Moodaii--PPPpp
Hangbo Bao'; Wenhui Wang", Li Dong", Qiang Liu", Owais Khan Mohammed',
Kriti Aggarwal", Subhojit Som*, Songhao Piao*, Furu Wei
Harbin Institute of Technology. Microsoff Corporation https: //aka. Bacikk..smmsszee ms/msragi
Abstract
We present unified Vision- Lannua-La pretrained Model (VLMo) that jointly lears dual encoder and fusion encoder with modular Transformer network. Specifically, introduce Multiway Transformer, where each block contains a pool of modality-specific experts and shared self-attention layer. Because of the modeling flexibility of Multiway Transformer, pretrained VLMo can be fine-tuned as a fusion encoder for vision-language classification tasks, or used dual encoder for efficient image-text retrieval. Moreover, we propose a stagewise pre-training strategy, which effectively leverages large-scale image-only and text-only data besides image-text pairs. Experimental results show that VLMo achieves state-of-the the-art results various vision- vision-language tasks, including VQA, NLVR2 and image-text retrieval. The code and pretrained models are available http: h/p::///aa ms/ ms/mlmo
Introduction
Vision-Language (VL) pre-training [31 36
24 learns generic cross-modal representations from large-scale image-text pairs. Previous models usually employ image-text matching, image-text contrastive learning, masked region classification/feauue regression, saandregiin/patchh alignment and masked language modeling to aggregate and align visual and linguistic information. Then the pretrained models can be directly fine-tuned on downstream vision- vision-language tasks, such as VL retrieval and classification (visual question answering, visual reasoning, etc.).
27
21
Two mainstream architectures widely used in previous work. CLIP [6]] and ALIGN 19 adopt dual-encoder architecture to encode images and text separately. Modality interaction is handled by the cosine similarity of the image and text feature vectors. The dual-encoder architecture effective for retrieval tasks, especially for masses of images and text. Feature vectors of images and text can be pre-computed and stored. However, the shallow interaction between images and text not enough to handle complex VL classification tasks. ViLT [2] finds that CLIP gives relatively low accuracy on visual reasoning task. Another line of work [31 42 44 4 21 24 relies on fusion encoder with cross-modal attention to model image-text pairs. Multi-layer Transformer [46) networks are usually employed to fuse image and text representations. The fusioe- architecture achieves superior performance on VI. classification tasks. But it requires to jointly encode all possible image-text pairs compute similarity scores for retrieval tasks. The quadratic time complexity leads to a much slower inference speed than the dual-encoder models whose time complexity is linear.
In order take advantage of the types of architectures, we propose unified Vision-Lane Language pretrained Model (VLMo) that can be used as either a dual encoder to separately encode images and text for retrieval tasks, or used as a fusion encoder model the deep interaction of image-text pairs for
"Contribution during internship at Microsoft.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

classification tasks. This is achieved by introducing Multiway Transformer that can encode various modalities (images, text, and image-text pairs) within Transformer block. Multiway Transformer employs a pool of modality experts to replace the feed-forward network in standard Transformer. It captures modality-specitfe information by switching to different modality experts, and uses the shared self-attention across modalities to align visual and linguistic information. Specifically, Multiway Transformer consists of three modality experts, namely vision expert for image encoding, language expert for text encoding, and vision- vision-language expert for image-text fusion. Thanks to the modeling flexibility, we can reuse Multiway Transformer with the shared parameters for different purposes, i.e., text-only encoder, image-only encoder, and image-tex text fusion encoder.
VLMO jointly learned with three pre-training tasks, namely image-text contrastive learning, image- text matching. and masked language modeling. In addition, we propose stagewise pre-training strategy to effectively leverage large-scale image-only and text-only corpus besides image-text pairs in VLMO pre-training. We first pretrain vision experts and self-attention modules of Multiway Transformer on image-only data using masked image modeling proposed in BETT [ We then pretrain language experts on text-only data using masked language modeling [ЛЛ. Finally, the model is used to initialize vision-language pre-training. By getting rid of the limited size of image-text pairs and their simple and short captions, stagewise pre-training on large amounts of image-only and text-only data helps VLMO to learn more generalizable representations.
Experimental results demonstrate that VLMO achieves state-of ofute-of-the-art results on vision- vision-language retrieval and classification tasks. Our model, used as a dual encoder, outperforms fusion-encoderr- based models 4 15 21 24 while enjoying much faster inference speed on retrieval tasks. Moreover, our model also achieves state-of-the-arr results on visual question answering (VQA) and natural language for visual reasoning (NLVR2), where VLMo is used as a fusion encoder.
Our main contributions are summarized as follows:
We propose a unified vision-language pretrained model VLMO that can be used as a fusion encoder for classification tasks, or fine-tuned as a dual encoder for retrieval tasks.
We introduce a ceneral-purposs multimodal Transformer for vision- vision-language tasks, namely Multiway Transformer, to encode different modalities. It captures by modality experts, and aligns contents of different modalities by the self-attention module shared across modalities.
information
We show that stagewise pre-training using large amounts of image-only and text-only data greatly improves our lision-language pretrained model.
2
Related Work
Pre-training with Transformer [46] backbone networks has substantially advanced the state of the art
across natural language processing 35
11
29
23
12
37 2
| (2). computer vision [3 45
and languael [44 42
51
36
19
21
24
tasks.
The approaches of linguaelanguage pre-training can be divided into two categories. The first category utilizes dual encoder to encode images and text separately, and uses cosine similarity linear projection layer to model the interaction between images and text [36] 19.. Image-text contrastive learning is usually employed optimize the model. Dual-encoder models are effective for vision- language retrieval tasks. However, the simple interaction is not enough to handle tasks that require complex reasoning, such as visual reasoning and visual question answering (VL classifiicat tasks). The second category models the interaction of images and text using a deep fusion encoder with 48). Image-text matching.
cross-modal attention [4 31 25 53 27 26 15 51 17
18 21
masked language modeling, word-region/ rotin/pcgin alignment, masked region classification and feature models. These models achieve better
regression are widely used to train dusion-encoder-bassee performance for liniunelanguage classification tasks, while the joint encoding of all image-text pairs leads to a slow inference speed for retrieval tasks. large portion of tusion-encoder-basee models rely on an off-the-shelf object detector like Faster R-CNN features. Generating region features slows down the inference speed and renders the approach less scalable. Recently, Pixel-BERT by convolutional neural networks. ALBEF 24 employs image Transformer [3 45 to obtain the representations of images, and uses text Transformer |||| to learn the contextualized representations
68 to obtain image region
[17 removes object detector and encodes images into grid features

---

Image-Text Contrastive Learning
V-FFN
L-FFN
Lx
Lx
Mult-Hoad Salf-Attantion
Muti-Hood SoF-Alltantion
Poich Embeddings
Word Er brbuddinng
Image- Text Contrast
baseball throwing
Butidhing Modirity Expart
Yes No
V-FFN
LFFN
VL-FFN
Meiin Expert
Larguage Fapwe
Hard
Expert
Negative
VL-FFN
Mining
Mutt-Heed Self Attention
Lx
Norm
Image-Tex Text Matching
V-FFN
L-FFN
(1-FFX
Self Solf-Attantion
Hull
Multi-Head Self-Altention
Paxch
Emooddings
Nor
bosetall player throiing ball
Masked LM
Muttimodal
player
Multiway Transformer with Shared Parameters
Multi Head Selt Attention
VIFFN
(L.F)X
Mult-Hoadd Stlt-Attation
Puich Emboddings
Word Embodd
[MASK] throwing [MASK]]
Figure 1: Overview of VLMO pre-training. We introduce Multiway Transformer to encode different modality input by modality-specifee experts. The model parameters shared across image-text contrastive learning, masked language modeling, and image-text matching pre-training tasks. During fine-tuning, the flexible modeling enables use VLMO either dual encoder (i.e., separately encode images and text for retrieval tasks) or a fusion encoder (i.e., jointly encode image-text pairs for better interaction across modalities).
of text. These representations are then fused by cross-modal attention. ViLT [21] encodes images into patch embeddings, and then feed the concatenation of image patch embeddings and word embeddings into a Transformer network to learn contextualized representations and model the interaction of images and text.
Different from previous work, our unified pre-training using shared Multiway Transformer enables the model perform separate encoding for retrieval tasks, and jointly encode image-text pairs to capture deeper interaction for classification tasks. Our model achieves competitive performance, while enjoying a faster inference speed for both retrieval and classification tasks.
3 Methods
Given image-text pairs, VLMo obtains image-only, text-only and image-text pair representations by the Multiway Transformer network. shown in Figure the unified pre-training optimizes shared Multiway Transformer with image-text contrastive learning on image-only and text-only rep- resentations, image-text matching and masked language modeling on image-text pair representations. Thanks to the modeling flexibility, the model can be used as dual encoder for retrieval tasks to encode images and text separately during fine-tuning. can also be fine-tuned as a fusion encoder to model deeper modality interaction of images and text for classification tasks.
3

---

Vsion-Language Pre-Training
Vision Pre-Training
Language Pre-Training
Switthing Moda Expert
V-FEN
L-FFN
VL-FFN
Ex
Lilli Self-A ref-AAtentinn
Mutsitinn 400
Mulli
Image-Only Data
Text-Only Data
Image-Text Pair Data
Stagewise pre-training using image-only and text-only corpora. We first pretrain the vision expert (V-FFN) and self-attention module on large-scale image-only data as in BErT 3] Then the parameters of vision expert and self-attention module are frozen, and we train the language expert (L-FFN) by masked language modeling on large amounts of text-only data. Finally, we train the whole model with vision- lainuge pre-traee pre-training.
Figure 2:
3.1 Input Representations
Given image-text pair, encode the pair into image, text and image-text vector representa- tions. These representations are then fed into the Multiway Transformer to learn contextualized representations and align image and text feature vectors.
Image Representations Following vision Transformers [3 45 31. the 2D image v E "XXXC
is split and reshaped into N = HW HW/P2 patches v² E R RN((PC) where is the number of channels, (H, W) is the resolution of the input image, and (P, P) is the patch resolution. The image patches are then flattened into vectors and are linearly projected to obtain patch embeddings. We also prepend a learnable special token [I_CLS] to the sequence. Finally, image input representations obtained via summing patch embeddings, learnable ID position embeddings yos R(N+1)xD
and image type embedding Vayee R D: Hi [Vtr_cus), Vo,, v..,.. Vo Vpos Hö E TR (N+1))D linear projection V E (PPC))D
Viype> where
Text Representations Following BERT 111, tokenize the text to subword units by Word- Piece [49]. A start-of-sequence token ([T_ ([T_CLS]) and a special boundary token ([T_SEP]) are added to the text sequence. Text input representations Hü' E R(M+2)x are computed via summing the corresponding word embedding, text position embedding and text type embedding H. [Wr_cCs) Wi, WMs [[TSSEP] + Tpos + Tiype- indicates the length of tokenized subword units.
Image-Text Representations
We concatenate image and text input vectors to form the image-text input representations H :H
3.2 Multiway Transformer
Inspired by mixture-of-expert networks 41 we propose general-purpont multimodal Trans- former for vision-language tasks, namely Multiway Transformer, to encode different modalities. Multiway Transformer introduces mixture modality experts as a substitute the feed forward network of standard Transformer. Each modality expert is also the feed forward network which consists of two linear transformations and an activation. Given previous layer's output vectors H E L]. each Multiway Transformer block captures modality-specifie information by switching to different modality expert, and employs multi-head self-attention (MSA) shared across modalities to align visual and linguistic contents. LN is short for layer normalization.
H MSA(LN + Hi-1
(1)
H = Multiway-FFN(LN)
+ H
(2)
Multiway-FFN selects an expert among multiple modality experts to process the input according to the modality of the input vectors H and the index of the Transformer layer. Specifically, there

---

Task- pecific Classification Layer
Image-Test Contrasive Learning
VI-FF4
VFN
L FFN
SelfAse
Head Set Atterition
Head Abtertoo
V.FFN
FPN
F Erbending
Head
bevetall trining
Farch mrieddings
f Eibeddiggs
basebell shyyr trowingg
(8) Fine Tuning on VL Retreval Tasks
(b) Fine-Tunmin Tuning on VL Classiicctionn Tasks
Figure 3: Fine-tuning VLMo on vision-language retrieval and classification tasks. The model can be fine-tuned dual encoder to separately encode image and text for retrieval tasks. VLMO can also be used as a fusion encoder to handle interaction of image-text pairs for classification tasks.
three modality experts: vision expert (V-FFN), language expert (L-FFN) and vision-language expert (VL-FFN). If the input is image-only or text-only vectors, we use vision expert for encoding images and language expert for encoding text. the input consists of vectors of multiple modalities, such as the vectors of image-text pair, we employ vision expert and language expert to encode the respective modality vectors the bottom Transformer layers. lisiua-language expert is then used at the top layers to capture more modality interaction. Compared with conventional miixture-ff-expee networks 14), Multiway Transformer conducts hard routing according the input modality. Given the three types of input vectors, we obtain image-only, text-only and image-text contextualized representations.
3.3 Pre-Training Tasks
VLMO is jointly pretrained by image-text contrastive learning on the image and text representations, masked language modeling and image-text matching on the image-text pair representations with shared parameters.
Texte-Text Contrast Given a batch of N image-text pairs, image-text contrastive learning aims to predict the matched pairs from N x N possible image-text pairs. There are N N negative image-text pairs within training batch.
The final output vectors of [I_CLS] token and [T_ [T_CLS] token are used the aggregated representa- tion of the image and text, respectively. Followed by linear projection and normalization, we obtain ) in training batch to compute image-to-text and image vectors ( ) and text vectors {hr) J==1 text-to-image similarities:
$
321 hy..
121 h
h
(3)
exp( 5.1
exp
425 /0)
22
22
(4)
Σ -ز=1 exp( (x)
Σ exp(s
Where sf. represents image-to-text similarity of image of i-th pair and of j-th pair, st the text-to-image similarity. h." E T and hy TR. indicate the normalized vectors of i-th text and j-th image, is learned temperature parameter. pf and 2 the sofmaiex-nrmalized similarities. Cross-entropy losses over image-to-text and text-to-image similarities are used to train the model.
Masked Language Modeling Following BERT [II,, randomly choose tokens in the text sequence, and replace them with the [MASK] token. The model trained to predict these masked tokens from all the other unmasked tokens and vision clues. We 15% masking probability as BERT. The final output vectors of masked tokens are fed into a classifier over the whole text vocabulary with cross-entropy loss.
Image- Text Matching Image-text matching aims to predict whether the image and text matched. We use the final hidden vector of the [T_CLS] token to represent the image-text pair, and feed the
5

---

vector into a classifier with cross-entropy loss for binary classification. Inspired by ALBEF [24]. we sample hard negative image-text pairs based on the contrastive image-to-text and text-to-image similarities.
3.4 Stagewise Pre- Pre-Training
We introduce stagewise pre-training strategy, which leverages large-scale image-only and text-only corpus to improve the vision-language model. As present in Figure we first perform vision pre- training image-only data, and then perform language pre-training on text-only data to learn general image and text representations. The model is used to initialize the vision-language pre-training to learn the alignment of visual and linguistic information. For vision pre-training, we train the attention module and vision expert of Multiway Transformer as in BEIT [3] on image-only data. We directly utilize the pretrained parameters of BEIT initialize the attention module and vision expert. For language pre- pre-training, we freeze parameters of the attention module and vision expert to avoid catastrophic forgetting of vision knowledge learned in the first stage, and utilize masked language modeling [1]] to optimize the language expert on text-only data. Compared with image-tex pairs, image-only and text-only data are easier to collect. addition, text data of image-text pairs usually short and simple. Pre-training on image-only and text-only corpus improves the generalization on complex pairs.
3.5 Fine-Tuning VLMO on Downstream Tasks
As present in Figure i classification tasks.
our model can be fine-tuned to adapt to various vision-language retrieval and
Vision-Language Classification For classification tasks such as visual question answering and visual reasoning, VIMMO is used fusion encoder model modality interaction of images and text. We use the final encoding vector of the token [T_CLS] as the representation of the image-text pair, and feed it task-specific classifier layer predict the label.
Vision-Language Retrieval For retrieval tasks, VLMO can be used as a dual encoder to encode images and text separately. During fine-tuning, our model is optimized for the image-tes contrastive loss. During inference, we compute representations of all images and text, and then use dot product to obtain image-to-ter toggetto-textee and text-to-image similarity scores of all possible image-text pairs. Separate encoding enables a much faster inference speed than fusion-encoder- based models.
Experiments
We pretrain our model using large-scale image-text pairs and evaluate the model visual-linguisti linguistic classification and retrieval tasks.
4.1 Pre-Traie Pre-Training Setup
Following previous work 4,22 our pre-training data consists of four image captioning datasets:
Conceptual Captions (CC) 40), SBU Captions [33], COCO 28 and Visual Genome (VG) 22
datasets. There are about 4M images and 10M image-text pairs in the pre-training data.
Our models adopt the same network configuration as ViT [13 and BEIT 3. VLMo- consists of 12-layer Transformer blocks with 768 hidden size and 12 attention heads. VIMo--Lare is a
24-layer Transformer network with 1024 hidden size and 16 attention heads. VLMo-Base uses vision-language expert on the top two Transformer layers, and VLMo-Large introduces vision- language expert on the top three layers. VLMo-Base consists of 175M parameters and VLMo-Large contains 562M parameters. For images, the input resolution is 224 x 224 and the patch size is 16 x 16 during pre-train vraiirnn We apply RandAugment [00] to the input images. The tokenizer of the uncased version of BERT is employed to tokenize the text. The maximum text sequence length is set to 40. We also employ whole word masking for the masked language modeling pre-training task. We pretrain the models for 200k steps with 1024 batch size. We utilize AdamW 30 optimizer with 31 = 0.9, 32 = 0.98. The peak learning 2e-4 for the base-size model, 5e-5 for the large-size model. Weight decay is set to 0.0.1. We use linear warmup over the first 2.5k steps and linear decay.

---

# Pretrain Images
VQA
NLVR2
Model
test-dev
test-std
dev
test-P
Base-Size Models Pretrained COCO, VG,
SBU and CC datasets
UNITER -
4M
72.70
72.91
77.18
77.85
'ILLA-Base 15
4M
73.59
73.67
78.39
79.30
UNIMO-Base 26
4M
73.79
74.02
ViLT-Base 21
4M
71.26
75.70
76.
ALBEF-Base
4M
74.54
74.70
80.24
80,50
VLMo-Base
4M
76.64
76.89
82.77
83.34
Large-Size Models Pretrained on COCO, VG,
SBU and CC datasets
UNITER-Large
4M
73.82
74.02
79.12
79.98
VILLA-L Large 15
4M
74.69
74.87
79.76
81.47
UNIMO- Large 26
4M
75.06
75.27
VLMo-Large
79.94
79,98
85.64
86.86
Models Pretrained More Data
VLnLLee (51)
5.7M
76.52
76.60
82.67
83,98
48
1.8B
79.32
79.56
84.13
84.84
Sim
48
1.8B
80.03
80.34
84.53
85.15
Florence-Huu
[50
900M
80.16
80.36
Flamingo
2.3B
82.00
82.10
VLMo-Large++
1.0B
82.88
82.78
88.62
89.54
Table 1: Fine-tuning results of base-size and large-size VLMo vision-language classification datasets. VLMo-Large++ is the model trained on one billion noisy image-texl pairs with a larger batch size. We report vqa-score VQA test-dev and test-standard split, and report accuracy for NLVR2 development and public test set (test-P).
4.2 Training on Larger-scale Datasets
We scale up vision- -laiuue representation learning by training VLMo-Large on one billion noisy web image-text pairs with a larger batch size. We first pretrain the model for 200k steps with 16k batch size, and then continue train the model for 100k steps with 32k batch size. The other hyper- parameters are the same as the training 4M data. Please refer to the supplementary material for more details hyper- parameters used for pre-training and fine-tuning.
4.3 Evaluation on LaniuaLa Classification Tasks
We first conduct fine-tuning experiments on two widely used classification datasets: visual question answering [16] and natural language for visual reasoning 43 encoder to model deeper interaction.
The model fine-tuned as fusion
Visual Question Answering (VQA) For VQA, a natural image and a question are given, the task to generate/choose the correct answer. We train and evaluate the model on VQA 2.0 dataset [16]. Following common practices, we convert VQA 2.0 from a shared set consists of 3, 129 answers. We use the final encoding vector of the [T [T_CLS] token as the representation of the image-question pair and feed it to a classifier layer to predict the answer.
classification task, and choose the answer
Natural Language for Visual Reasoning (NLVR2) The NLVR2 [3 dataset requires the model to predict whether text description true about a pair of images. Following OSCAR [27] and VinVL [51), convert the triplet input two image-text pairs, each containing the text description and one image. We concatenate the final output vectors the [T_CLS] token of the two input pairs. The concatenated vector is then fed into classification layer to predict the label.
We present the results of VL classification tasks in Table [ VLMO achieves state-of-tt of-the-art per- formance and substantially outperforms previous methods. Our large-size model even outperforms Sim MonVLM-Hugg [48) and Florence-Huge Huge [50] by large margin, which consists of more parameters and are also trained on larger-scale image-text pairs. Our model uses simple linear projection to em- bed images as in ViLT (1I] This leads a significant speedup compared with previous models using
image region features, which extracted by off-the object detector
31 42
L5
26 51]

---

MSCOCO (5K test set)
Flickr30K (IK test set)
# Pretrain Images
Model
IR
TR
IR
R@l 1 R@S R 10 R@I R@S R 10 R
R@5
R@10 R@1
R@S R@ 10
Base-Sizz Models Pretrained on COCO, VG, SBU and CC datasets
UNITER-Bar
4M
64.4
87.4
93.1
50.3
78.5
87.2
85.9
97.1
98.8
72.5
92.4
96.1
VILLA-
4M
86.6
97.9
99.2
74.7
92.9
95.8
ViLT-Base
4M
61.5
86.3
92.7
42.7
72.9
83.1
83.5
96.7
98.6
64.4
88.7
93.8
ALBEF-Base
4M
73.1
91.4
96.0
56.8
81.5
89.2
94.3
99.4
99.8
82.8
96.7
98.4
VLMo-Basef
4M
74.8
93.1
96.9
57.2
82.6
89.8
92.3
99.4
99.9
79.3
95.7
97.8
Large-Size Models Pretrained on COCO, VG, SBU and CC datasets
UNITER-Large
4M
65.7 88.6
93.8
52.9
79.9
88.0
87.3
98.0
99.2
75.6
94.1
96.8
VII.LA-Larg
4M
87.9
97.5
98.8
76.3
94.2
96.8
VLMo-Larget 4M
78.2 94.4
97.4
60.6
84.4
91.0
95.3
99.9
100.0
84.5
97.3
98.6
Models Pretrained on More Data
Vinre-L
5.7M
75.4
92.9
96.2
58.8
83.5
90.3
ALrGN-Largef 1.8B
77.0
93.5
96.9
59.9
83.3
89.8
95.3
99.8
100.0
84.9
97.4
98.6
Florence-Huget
900M
81.8
95.2
63.2
85.7
97.2
99.9
87.9
98.1
VLMo-Large+**
1.0B
83.1
96.0
98.2
65.2
86.5
92.2
96.8
100.0
100.0
88.1
98.4
99.3
Table 2: Fine-tuning results of text-retrieval (TR) and image-retrieval (IR) on COCO and Flickr30K. t: ALIGN, Florence and our model encode images and text separately, and then employ a shallow interaction (dot product) to obtain the similarity scores. I: ALBEF first encodes images and text separately to obtain the top-k candidates, and then feed these representations into a fusion encoder to rerank the candidates. The others require to encode all image-text combinations by a fusion encoder. VLMo-Larges represents the model trained on one billion noisy image-text pairs with a larger batch size.
NLVR2
Flickr30k TR
Stagewise Pre-Training
dev
test-P
IR
Image-Only Pre- Tre-Trinnig
80.33
81.06
95.60
87.69
Image-Only Text-Only Pre-Training
82.09
82.49
95.67
88.52
Table Ablation studies of stagewise pre-training i.e., different initialization for vision- language pre-training. We report the average of R @ R@5 and R@ 10 for Flickr30k. Results of NLVR2 are averaged over three runs.
4.4 Evaluation on Vision-Language Retrieval Tasks
The retrieval tasks contain image-to-tex retrieval and text-to-image retrieval. We evaluate the model on the widely used COCO (28] and Flickr30K (34] datasets, and use the Karpathy split [20] for both datasets. The model used dual encoder for retrieval tasks. We encode images and text separately and compute their similarity scores by the dot product of image and text vectors.
As present in Table 2 VLMo achieves competitive performance with previous fusion-encoder- based models while having a much faster speed. Fusion-encoder-basee models need to jointly encode all possible image-text pairs to compute their similarity scores, which requires quadratic time complexity. Moreover, our large-size model even outperforms the huge-size model of Florence (50), which also trained on massive image-tex pairs using larger batch size. VLMo pre-training effectively leverage larger-scale noisy pairs and benefit from large batch training.
4.5 Evaluation on Vision Tasks
As shown in Table we use VLMO an image-only encoder and evaluate it on image classification
(ImageNet (39 and semantic segmentation (ADE20K (52) tasks. The model also achieves competi- tive performance, even slightly better than the BEIT model used for the initialization of VLMo. The image resolution is 224x224 for ImageNet, and 512x512 for ADE20K. We perform intermediate fine-tuning (3] ImageNet-21k for all three models.

---

Models
ImageNet (acc@
ADE20K (mloU)
ViT-Base
83.6
BErT-Basc
85.2
52.8
VLMo-Base
85.5
53.4
Table 4: Results on image classification and semantic segmentation.
Pre- Fre-Training Tasks
Transformer
NLVR2
Flickr30k
ITC
ITM
MLM
Std TRM
Multiway
Multiway VLExp
dev
test-P
TR
IR
X
x
X
58.51
58.83
92.23
84.24
X
V
X
73.91
73.75
94.07
85.82
/
X
X
76.46
76.19
94.37
85.67
/
V
X
78.81
79.27
93.37
85.73
V V
X
X
79.58
80.11
94.50
86.69
V
X
80.13
80.31
95.17
87.25
Table 5: Ablation studies Multiway Transformer and vision-language pre-training tasks. "TTC" is short for image-text contrastive loss, "TTM" is image-text matching, and "MLM" is masked language modeling. "Std TRM" is short for standard Transformer, and "Multiway VVLxx" Multiway Transformer without VL experts. The average of @@) 1, @ and R @ 10 is reported for Flickr30k. Results of NLVR2 are averaged over three runs.
4.6 Ablation Studies
Stagewise Pre-Tree Pre-Training We first conduct ablation experiments of stagewise pre-training. ViLT 21 shows that using the ViT (13) model pretrained on image-only data the initialization achieves better performance than the BERT model pretrained on text-only data. Therefore we start experiments with image-only pre-training. We compare using image-only pre-training and image-only pre-training plus text-only pre-training as the initialization. For image-only pre-training, we directly use the parameters of BErT-Bass initialize the self-attention module and all modality experts. For image- only pre-training plus text-only pre-training, we use pretrained parameters of BErT-Base to initialize the vision expert and self-attention module of Multiway Transformer, and then pretrain language expert on text corpora. As shown in Table image-only pre-training plus text-only pre-training improves our vision-language model. We also have tried to perform vision-language pre-training with random initialization but obtain a relatively low accuracy downstream tasks. Stagewise pre-training effectively leverages large-scale image-only and text-only corpus, and improves our vision-language language pre-training. Morcover, given the limited size of image- pairs used during pre- training, stage-wise pre-traine pre-training on image-only and text-only data alleviates the need for image-text pair data. We have tried perform multitask training image-only and text-only data to combine the first two stages and observe similar performance. Stage-wise pre-training effectively leverages the pretrained weights to reduce the computation cost.
Multiway Transformer We also conduct ablation experiments of Multiway Transformer. We employ ViT-Base to initialize the models for the ablation experiments. As present in Table using Multiway Transformer achieves better performance than standard Transformer for both retrieval and classification tasks. In addition, we also analyse the contribution of vision-language expert (VL-FFN) used in Multiway Transformer. We remove the vision-language expert used in the top Transformer layers. Experimental results demonstrate that the introduction of vision-language expert improves the model. Using vision-language expert captures more modality interaction.
Pre- Tre-Training Tasks We perform ablation studies analyse the contribution of different pre-training tasks, and the results are presented [3] Compared with the model trained only using image-text contrastive loss, unified training performs much better across classification and retrieval tasks. Introducing image-text matching with hard negative mining also greatly improves the model. This demonstrates the effectiveness of our unified-training framework with Multiway Transformer. In addition, experimental results show that masked language modeling positively contribute to our model. Please refer the supplementary material for more ablation studies.

---

NLVR2
Models
dev
test-P
Local hard negative mining
2
77.70
77.95
Global hard negative mining (ours)
79.54
79.48
Table 6: Global hard negative mining improves the model. We perform experiments using 32 V100 GPUs for base-size model. The batch size per GPU is 32, and the total batch size is 1024. Local hard negative mining samples hard negatives from training examples of the single GPU (32 examples), while global hard negative mining uses training examples gathered from all GPUs as the candidates (1024 examples).
Global Hard Negative Mining Different from ALBEF 24 which samples hard negatives from training examples of the single GPU (named as local hard negative mining). We perform hard negative mining from more candidates by gathering training examples of all GPUs (named as global hard negative mining). As shown Table 6 our global hard negative mining brings significant improvements.
Conclusion
In this work, we propose unified vision-language pretrained model VLMo, which jointly learns dual encoder and fusion encoder with a shared Multiway Transformer backbone. Multiway Transformer introduces a pool of modality experts to encode modality-specifie information, and aligns different modalities using the shared self-attention module. The unified pre-training with Multiway Transformer enables the model to used as a dual encoder for efficient vision-language retrieval, fusion encoder to model crosllmodal interactions for classification tasks. We also show that stagewise pre-training that leverages large-scale image-only and text-only corpus greatly improves vision- vision-language language pre-training. Experimental results demonstrate that VLMo outperforms previous state-of-the-an models on various vision-language classification and retrieval benchmarks.
In the future, we would like to work on improving VLMo from the following perspectives:
We will scale up the model size used in VLMO pre-training.
We are also interested in fine-tuning VLMO for vision-language generation tasks, such as image captioning. following the method proposed in UniLM 112.
We are going to explore to what extent vision-language pre-training can help each other modality, especially as the shared Multiway Transformer backbone naturally blends text and image representations.
We extend the proposed model to integrate more modalities (e.g.. speech, video, and structured knowledge), supporting general- -prrpsse multimodal pre-training.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Ruther- ford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro. Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Miko- laj Binkowski. Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. CoRR, Abs/2204.14199. 2022. doi: 00.48550/arXiv.2200 14198. URL URL httpps//:///o: org/10, 10. 48550/8555// arXiv 2204. 14198
[2] Hangbo Bao, Dong, Furu Wei, Wenhui Wang, Nan Yang. Xiaodong Liu, Yu Wang. Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao-Wun Hon. UniLMv2: Pseudo-masked language models for unified language model pre-training.
Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Evení, volume 119 of Proceedings of Machine Learning Research, pages 642-652, PMLR, 2020. URL http! //proceedings.ro mlr, wwwps/eeeseeepp. https///19bbbbo....mm html
10

---

[3] Hangbo Bao. Li Dong, and Furu Wei. BEĪT; BERT pre-training of image transformers. CoRR, abs/2106.08254, 2021. URL httpt::t/trtiii..... org. wttp:///ww..rgr//////....// 08254
[4] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and KanMMichael Frahm, editors, Computer Vision ECCV 2020- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 104- 120. Springer, 2020. doi: 10. 1007/ 978-3-030-58577-88
htt.httss://doo.. org/ 10. 1007 000000/977-330 00000/99788-3-000-877-------
[5] Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xianling Mao, and Heyan Huang. Cross-lingual natural language generation via pre-training. CoRR, abs/1909 10481, 2019.
[6] Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang. Xia Song, Xian- Ling Mao, Heyan Huang, and Ming Zhou. InfoXLM: An information-thcoree framework for cross-lingual language model pre-training In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576 3588, Online, June 2021. Association for Computational Linguistics.
/tttss/t::////t aclweh we....e..oo org/ anthtlogyotaaaaaeaaaaaaatt 22221.0111111 maacl-main 280
[7] Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. XLM-E: Cross-lingual language model pre-training via ELECTRA. ArXiv, 161//106.16138, 2021.
[8] Alexis Comneau and Guillaume Lample. Cross-lingual language model pretrain- ing. In Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL https: wtw.ooceeddnnn.nnuisss....... seeurr/ps.ccppppe//11/////11 /atpp/ppa/r////llee//
pdf
[9] Alexis Conneau, Kartikay Khandelwal, Naman Goyal. Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised gross-lingual representation learning scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440 8451, Online, July 2020. Association for Computational Linguistics. URL http::t//////. //www. aclweb. org/ anthology/ 2020. acl-main..... 747.
[10] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In 2020 IEEF/CVE Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pages 3008-3017. Computer Vision Foundation IEEE, 2020.
[11] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171- 4186. Association for Computational Linguistics, 2019. doi: 118653/v1/n199 1423. URL httpp:://don http://oooo.oo///./o/// 18653/v1/119- 1423.
[12) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao Wuen Hon. Unified language model pre-training for natural language under- standing and generation. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 13042-13054, 2019.
[13) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. preprint urXiv:2010.11922 2020.
[14] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101 abs/2101.03961, 2021. URL https:// //apxii///xvvv.. org/ absaabas/110 03961

---

[15] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and and-language representation learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[16] Yash Goyal, Tejas Khot, Douglas Summers-Stayy Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA. July 21-26. 2017, pages 6325-63333 IEEE Computer Society, 2017. doi: 10.1109/
CVPR.2017.670 URI. https: //dpi.:///oo..o org/10 10. 100.1119R.//.001111111 2017 660
[17] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep mudti-modal transformers, CoRR, abs/2004.00849, 2020. URL
. org/ absaa/s///// 2000/2004.000 00849
(18) Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 12976- 12985. Computer Vision Foundation IEEE, 2021.
[19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun- Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and lasion-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 4904-4916. PMLRR, 2021. URI.http:: pttp://d/noccceees.omooo .ml...mr....... pttpss/vee/ jia21b. html
[20] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 1112-31177 IEEE Computer Society, 2015.
[21] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languagee transformer without convolution or region supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5583-5594, PMLR, 2021. httt:h//////oooee
mlr ptwps//v3eeee/ppppppppppp 1139//mmmmkk.. html
[22] Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Comput. Vis., 123(2):3 2017.
Int.
[23] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising wowpueenc--t---queeee pre-training for natural language generation, translation, and comprehension. arXiv preprint 1910 1 13461, 2019.
[24] Junnan Li. Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. CoRR, abs/2107.07651, 2021. URL http::t/axxxii...t wwwpexivv.orrg..oo /bs/
2107.07651
[25] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. CORR, abs/1908 005507355, 2019. URL http://arxiv. http:///aaxiv..g/abbb/////s/ http///bb///99.0035553 03557
[26] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. UNIMO: towards unified-mon modal understanding and generation via cross-modal con- trastive learning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computeaional Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, pages 2592-2607. Association for Computational Linguistics, 2021.
12

---

[27] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 121- Springer, 2020. doi: 10. (0.10077978-3300--5557777 https://doi. org/10. 1007 0-178-8---00---57---818
URL
[28] Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014.
[29] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http:///axxvvv.roo org/ wwt./rg////////////e///////// 1907
11692
[30] llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[31] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-land -nnoo--n--a tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Anmual Conference on Neu- ral Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, wwt.ggs..urrsp...
BC, Canada, pages 23-23, 2019. UtL/tttss//prrrceednng
hwtpiippsccpppper/2000 wttpccccpppppppee wtw.://aaadeeetaaaaabaaatrtaa.. Abstracttttttttac.........aaa.. html
[32] Shuming Ma, Li Dong, Shaohan Huang. Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. Deltalm: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders. CoRR, 0bs/2106.13736, 2021. URL /uttps://arxx:
[33] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images us- ing million captioned photographs. In John Shawe- Shawe-Taylor, Richard S. Zemel, Peter Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Process- ing Systems 2011. Proceedings of meeting held 12-14 December 2011, Granada, Spain, pages 1143-1151, 2011. URL httptt::tpppceeeeeeeeeeeee http:///pooceesg..././ $cd8abb6e033aa9566655555 wbw.:ecffobbbbccccc / --eae- www.bbrtttttttaa
weapeeeddinnnnn.eeiispeeee wttpipp..ccpappp//0000 201/hash//hhhh////
[34] Bryan A. Plummer, Liwei Wang. Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-pian -o-parasse correspondences for richer amage-to-sentence models. In 2015 IEEE International Conference Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2641-2649 IEEE Computer Society, 2015. doi: 10.1 10.1100//CC222005..333 URL htt.::ttp///// //doi. org/ 10. 1109/ ICCV 2015.303
[35] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Im- proving language understanding by generative pre-training. 2018
URL
https://s3-us-wett. amazonaww.aaaaaaaaammmmm http://wwww.....o..ooppaaaaeeeee hagggggeisssrrrrrs//aggggguuuuuu
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and 1lya Sutskever. Learning transferable visual models from natural language supervi- sion. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML, 2021, 18-24 July 2021, Virtual Eveni, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 2021. URL. http: / http://///cceeee.../// mlr. wttppp/s///////affoooooo html
13

---

[37] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140: 1-446667 2020. URI. htlphttp://mmm
org/paapers/v21/20-0744....
[38] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell., 39(6): 1137-1149, 2017. doi: 10. 1..1109/PAM.220025500033
URI.httss://// //oii org 1 1109/
TPAMI 2577031
[39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein. Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
[40] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemyyedd image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2556-25655 Association for Computational Linguistics, 2018. URL sttps://aallanthology org wwwp//g........... 1238//
[41]
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le. Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open 2017. URL
https://openrevieet net/ forum? Batee///ooounnndddldddppppp ckMDggg
[42] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: pre- training of generic visual-linguistie representations. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL WRK.httpps///opnreeviee http://wwwwweetttttteeeeeeeeeeee
[43] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Anna Korhonen, David R. Traum, and Lluís Mårquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 6418-6428. Association for Computational Linguistics, 2019. doi: 10. 10.18653/11/19-1166
URL httphhtttttsi..... http://ooo.orgg1//// 1tt.1/1111111111111111
[44] Hao Tan and Mohit Bansal. LXMERT: learning mooas- encoder representations from transformers. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5099-51100 50. Association for Computational Linguistics, 2019. doi: 1111665/11/D1115555 URL http::ttt////c.. org/10 100011//5////11191500 ottt::///ooa.rrr/00...66.///1//1
[45] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. preprint arXiv:2012. 12877, 2020.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, UI- rike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1998-6008, 2017. URL http:/ttpp/:prrrtte nips cc/paperr 7181-attention-ie iaa-iss-lll all-
[47] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequeeee learning framework. CoRR, 0ttp/2222.00055 2022. URL.
https: htops://arxivv org/ abs//202. 03052
14

---

[48]
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov. and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. CoRR, abs/2108. 10904, 2021.
2108. 10904
[49] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Mclvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08 144, 2016. httlhhtttp//rrttie org/ abbb/609 08144]
[50] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella. Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li. Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang. Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. org/ abs/ / 2111, 11432
CoRR, abs/2111.11444 2021.
[51] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5579-558$ Computer Vision Foundation IEEE, 2021. URL https //openaccess. thecvf com/ Cowteatt/CVPR220//aattaaaa/////e
VinVL_ VinWL__eevssiiinnn...n...o...ntL html
[52] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. Comput. Vis., 127(3):302-322 321, 2019. doi: 10. 20.1007/111233-0183 111---0001111100,,111 1140-0. URL https::/too..i. org /000000000 1007/ 811263-018-1140 1140-0
[53] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and VQA. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pages 13041-13049. AAAI Press, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please refer to the conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e-g., with respect to the random seed after running experi- ments multiple times)? [Yes] We report average results of multiple runs.
15

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're ro using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "XNLPhmN6iq", "reviewer": "Reviewer_SVes", "summary": "This paper proposes a new method called VLMo for multimodal pretraining. VLMo consists of a dual encoder and a fusion encoder with Mixture-of-Experts, where experts of different modalities process information of specified modality or modality combination. This method achieves state-of-the-art performance in several multimodal downstream tasks. ", "questions": "1. Why does this work uses such a high resolution of 768 * 768 only for VQA, but uses 224*224 or 384 * 384 for the other tasks?\n2. Different from the common practice, VLMo adds image-text contrastive learning to the pretraining tasks, which use two towers to process information from different modalities separately, incompatible with the fusion, and I believe that there should be 4 forward operations （2 forwards in the image and text encoder for contrastive learning, 1 for ITM, and 1 for MLM） for a single batch. Does this cause increase in training costs? Is it possible to provide an ablation study on the pretraining tasks? \n", "limitations": "This paper did not point out limitations. It is suggested that the paper explain about lack of analysis in the expert design, lack of capabilities in generation tasks, and lack of further analysis about scaling, especially data scaling. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method achieves state-of-the-art performance in several multimodal downstream tasks. \n2. The method creatively incorporates mixture-of-experts with multimodal representation learning, which bring significant improvements in downstream tasks. \n\nWeaknesses:\n1. Lack of analysis of the mixture-of-experts. As this is the key innovation of the paper, which is somewhat novel and to the my best of my knowledge the first time to see modality expert in multimodal pretraining, the paper should demonstrate clearly why it is an important design, how it contributes to the performance, how different experts function and cooperate to reach a better effect than the conventional fusion methods. \n2. Unclear introduction to the model details: a. The MoE is essentially not the conventional MoE [1] with routing, so there should be clarification to avoid misunderstanding. b. There should be a clear demonstration of the number of parameters. As there are 3 FFN experts in the model, the transformer layers of VLMo-large should consist of at least 700M parameters, let alone the embedding and other parameters. Directly stating that it is a large-size model without clarification misleads readers. c. Based on my understanding, though the model is much larger, it did not bring much increase in computation costs. This should be illustrated clearly. Also, the paper should clarify which layers to use vision and text experts for different modalities, and which layers to process them with the VL experts, instead of using words like \"bottom layers\" and \"top layers\". \n3. The design of stagewise pretraining requires further explanation as there are many details, e.g., why is such stage design? how about training them jointly? why is the self attention fixed in the second stage? The ablation study should be comprehensive, and make more fair comparisons with the baselines. \n4. VLMo achieves stunning performance in VQA, which surpasses even all the current methods as far as I know, but I find from the supplementary details that the VLMO-large++ for VQA uses a high resolution of 768 * 768. This requires explanation or ablation study. From the presented results, readers cannot figure out where the improvements over VLMo-Large come from, scaling data or scaling resolution. \n5. This work should implement the model on more downstream tasks, including visual entailment, referring expression comprehension, etc. They are essential benchmarks for multimodal pretrained models. Also, this work should compare VLMo with the state-of-the-art methods (3 months before the NeurIPS submission), including BLIP [2] and OFA [3]. Especially for the cross-modal retrieval, VLMo should be compared with BLIP. Furthermore, as VLMo is pretrained with image-text contrastive learning, it is encouraged to conduct zero-shot cross-modal retrieval experiments. \n\n[1]. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.\n[2]. Li, J., Li, D., Xiong, C., & Hoi, S. (2022). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086.\n[3]. Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., ... & Yang, H. (2022). Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DX4McEtPZBu", "reviewer": "Reviewer_oGKY", "summary": "This paper focuses on Vision-Language Pre-training (VLP), in which they propose a unified vision-language pretrained model VLMO that jointly learns a dual encoder (i.e., the text and image encoders) and a fusion encoder (i.e., the multimodal encoder).\n\nSpecifically, the authors introduce the Mixture-of-Modality-Experts (MOME) Transformer to encode different modalities,  where each block contains a pool of modality-specific experts to  capture modality-specific information and a shared self-attention layer to learn the alignments between different modalities. Moreover, it can utilize large amounts of image-only and text-only data to perform stagewise pre-training. The pre-trained model can be used as a fusion encoder for classification tasks, or fine-tuned as a dual encoder for retrieval tasks.\n\n", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation of the proposed VLMO model makes sense to me, which can alleviate the limitations of existing single-stream and two-stream VLP models.\n\n2. The paper is generally well organized, and mostly easy to follow. Although the pre-training tasks are generally not new, the proposed Mixture-of-Modality-Experts (MOME) Transformer is relatively novel.\n\n3. Experimental results demonstrate the effectiveness of the proposed pre-training framework and stagewise pre-training strategies. \n\n\nWeaknesses:\n\nI have several concerns as follows:\n\n1.  In the pre-training stage, the proposed model is first trained with a MRM with the image-only data, followed by a MLM with the text-only data. The motivation is not clear. Is it possible to perform MLM first followed by MRM? Or is it better to perform alternative training?\n\n2. The downstream tasks only include image-text classification/retrieval tasks, but fail to consider many other representative VL generation tasks such as image captioning.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BZFov31lqJp", "reviewer": "Reviewer_Z9fk", "summary": "The paper introduces an architecture which supports multiple modalities (text, image and text-image), via  a novel transformer module (MoME), in which blocks consist of modality specific experts with shared self-attention.\nThe method achieves new SOTA on vision language classification tasks (VQA and NLVR2), as well as text retrieval and image retrieval, on MS-COCO 5k and Flickr 30K.\nThe model can be pre-trained stage-wise, using text-only and image-only large scale databases.\n", "questions": "How does this model compare to  the Flamingo model of Alayrac et al?\nHow would the model perform on Meta-Dataset of Triantafillou et al? Would the retrieval benefit from text pre-training? Would image pre-training improve results, due to the different data distributions and increased number of classes?\nAre the Vision / Language and Vision-Language experts trained independently? Would it benefit the subsequent tasks if these experts were pre-trained upfront?", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* support for text-only, image-only and text-image pretraining, which allows leveraging existing datasets, reducing the need  for large datasets with both modalities labeled.\n* image-text contrastive learning used for pre-training;\n* selection via multi-headed self attention of which expert to use for subsequent processing.\n* detailed ablation study.\n\nWeaknesses:\n* 4M to 1B images for pre-training leads to ~6% improvement. It would be interesting to understand if similar accuracy could be achieved with less data, or, conversely, get a higher improvement from 250x data.\n* (minor) No details about the architecture of the experts (it seems these are feed forward networks).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept