Write and Paint: Generative Vision-Language Models are Unified Modal Learners
Authors: Shizhe Diao
Wangchunshu Zhou
Xinsong Zhang
Jiawei Wang
Venue: ICLR 2023 poster (2023)
Keywords: Foundation model
Multi-modal learning
Vision-language pre-training

Abstract: Recent advances in vision-language pre-training have pushed the state-of-the-art on various vision-language tasks, making machines more capable of multi-modal writing (image-to-text generation) and painting (text-to-image generation). However, few studies investigate if these two essential capabilities can be learned together and boost each other, making a versatile and powerful multi-modal foundation model. In this work, we disclose the potential of symmetric generative vision-language pre-training in learning to write and paint concurrently, and propose a new unified modal model, named DaVinci, trained with prefix language modeling and prefix image modeling, a simple generative self-supervised objective on image-text pairs. Thanks to the proposed prefix multi-modal modeling framework, DaVinci is simple to train, scalable to huge data, adaptable to both writing and painting tasks, and also strong on other vision, text, and multi-modal understanding tasks. DaVinci achieves competitive performance on a wide range of 27 generation/understanding tasks and demonstrates the superiority of combining vision/language generative pre-training. Furthermore, we carefully benchmark the performance of different vision-language pre-training objectives on different scales of pre-training datasets on a heterogeneous and broad distribution coverage. Our results demonstrate the potential of exploiting self-supervision in both language and vision inputs, and establish new, stronger baselines for future comparisons at different data scales. The code and pre-trained models are available at https://github.com/shizhediao/DaVinci.

Published as a conference paper at ICLR 2023
WRITE AND PAINT: GENERATIVE VISION-LANGUAGE MODELS ARE UNIFIED MODAL LEARNERS
Shizhe Diao
Wangchunshu Zhou ByteDance Al Lab
The Hong Kong University of Science and Technology sdi aoaa@connect.. hk
wangchunshu www.tinf.uoooooo
Xinsong Zhang
Jiawei Wang
ByteDance Al Lab
Shanghai Jiao Tong University w wjw_sitts
zhangxinsong 0320ebytedance. com
ABSTRACT
Recent advances in vision-language pre-training have pushed the state-of-the-art on various vision-language tasks, making machines more capable of multi-modal writ- ing (image-to-tex generation) and painting (text-toiimage generation). However, few studies investigate if these two essential capabilities can be learned together and boost each other, making a versatile and powerful multi-modal foundation model. In this work, we disclose the potential of symmetric generative vision-language pre-training in learning to write and paint concurrently, and propose new unified modal model, named DAVINCI, trained with prefix language modeling and prefix image modeling, a simple generative self-supervised objective on image-text pairs. Thanks the proposed prefix multi-mode modeling framework, DAVINCI is sim- ple to train, scalable to huge data, adaptable to both writing and painting tasks, and also strong other vision, text, and multi-modal understanding tasks. DAVINCI achieves competitive performance on a wide range of 27 generation//ndderttaniii tasks and demonstrates the superiority of combining vision/language generative pre-training. Furthermore, we carefully benchmark the performance of different vision-language pre-training objectives on different scales of pre-training datasets on heterogeneous and broad distribution coverage. Our results demonstrate the potential of exploiting self-supervision in both language and vision inputs, and establish new, stronger baselines for future comparisons at different data scales T
INTRODUCTION
Self-supervised language model pre-training (Peters et al. 2018 Radford et al. 2018 Devlin et al.
2019 Liu et al. 2019] Lewis et al. 2020 Raffel et al. 2020; Brown al. 2020 Iu et al. 2022 Zhou
ct al. 202 lb Diao et al. 2020 2021 Zhou et al. 202 Xu et al. 2020 Zhou et al. 2020 2022a Pan et al. 2022 Diao et al. 2023) has reshaped the landscape of modern natural language processing (NLP) research, pushing the state-of-the-an of a wide range of NL.P tasks. Recently, this success has been transferred to the multi modal context and resulted in number of vision-languag pre-
trained models (VLMs) (Lu et al.] 2019 Tan & Bansal (019a). achieving state-of-the-ant art results on various vision-language tasks. Most existing VLMs are BERT-like Transformer (Vaswani al. 2017)
encoders pre-trained with a combination of different vision-language pre-training (VLP) objectives:
masked multi-modal modeling (Lu al. 2019 Tan & Bansal 2019b Chen al. 2020 Li al.
2020), multi-modal alignment prediction (Lu et al. |019| Tan & Bansal 2019b Chen et al. 2020
al. 2020). region of interest feature regression (Tan & Bansal 2019b), | image-text matching
et al., 2021 Zeng et al. 2021). name a few. However, the roadmap towards large language models
reveals a transition pattern from encoder-only models like BERT (Devlin et al. 2019) / RoBERTa LLu
al.) 2019) to sequence-tor -e--ueeee---seeeeeeeeee models like T5 (Raffel et al. 2020) BART (Lewis al. 2020
and autoregressive models like GPT-3 (Brown al. 2020) PaLM (Chowdhery et al. 2022) 10 tackle
* "Wor done during the internship at ByteDance AI Lab.
1 "ooresponning author
'The code and pre-trained models available
coo//h/hii zhedi ao/ ao/DaVinn

---

Published as a conference paper at ICLR 2023
more tasks in a unified way, and from complicated objectives like masked language modeling / next sentence prediction/ / replace token detection to a simple language modeling objective to improve the scalability of pre-training. This suggests that the generative pre-training paradigm with simple targets shows great potential for pre-training more scalable and general VLMs.
To this end, several recent studies (Cho et al. 2021 Zhang et al. 202 Wang et al. 2021b 2022)
investigated sequence-to-sequeee sequence (seq2seq) vision-language pre-training and achieved state-of- the-art results a range of vision-language understanding and generation tasks. For example, VL-T5 (Cho et al., 2021), ,, OFA (Wang et al. 2022) and Pal.I (Chen et al., 2022) formulate various vision-and-language problems into seq2seg tasks and pre-train seq2seq VLM by multi-tasking on these tasks. In addition, ERNIE- Vil.G (Zhang et al. (021a) and VLM (Wang et al.) (022b) pre-train seq2seq VLMs with simple language modeling or prefix language modeling objective on a large number of image-caption pairs. While achieving promising results, these objectives are not
versatile enough, resulting VLMs that are only capable of subset of tasks in image-ter modalities. On the other hand, the recent success of generative language pre- pre-training (Brown et al.) (220) and
generative vision pre-training (He et al. 2022 Bao et al.) (021) motivates us to explore generative vision-language pre-training to learn more versatile and scalable vision-language models.
In this work, we introduce prefix multi-modal modeling, a unified generative pre-training framework that extends prefix language modeling to the multi-modal context and learns a multi-modal foundation model by learning to write and paint simultaneously. As illustrated in Figure ! given an image- caption pair, we split the image and caption into two parts denoted as prefix and suffix. To make prefix image modeling compatible with the seq2seq formulation of conventional prefix language modeling, we follow DALLE (Ramesh et al.) (222]) and convert images into discrete sequences of image tokens (van den Oord et al. 2017). We then train the model to generate the suffix in one modality based on the prefix in the same modality and the complete input in the other modality. In this way, prefix multi-modal modeling can fully exploit self-supervision from large-scale segee-scal image-caption pairs by learning to write and paint simultaneously. We pre- preitrain DAVINCI [ vision-language foundation model, with the proposed prefix multi-modal modeling framework on large-scale image-text pairs. DAVINCI is the first self-supervised vision-language foundation model that is versatile for all kinds of tasks in vision-and -nd--aandddlggeee modalities, including image-to-text generation, text-to- text-to-image generation, vision-languaae understanding, and single-modal language / vision tasks. DAVINCI consistently outperforms FLAVA (Singh et al. 2021). , an existing vision-language foundation model, on both language, vision, and multi-modal tasks, and performs competitively with state-of-the th-the-art models across wide range of tasks and modalities. Moreover, DAVINCI also shows strong few-shot and zero-shot image/text generation capability.
In addition, most existing VLMs are pre-trained with mixed pre-training objectives and different data sources varying in size, making difficult to disentangle the impact of pre-training objectives and data sources on the downstream tasks. To this end, we conduct systematic analysis of the performance of generative language pre-training by carefully ablating different pre- pre-training objectives, such as prefix language/ / image modeling, and the amount of pre-training data with different qualities, revealing the impact of different objectives and data sources to facilitating future research.
To summarize, our contribution is three-fold. (1) We introduce prefix multi-modal modeling, simple unified generative vision-languaal pre-training framework that is scalable for large-scale pre-training and versatile for image-to-text generation, text-to-image generation and various multi-modal / single- modal understanding tasks. (2) We pre-train DAVINCI, a vision-languee
foundation model, with the proposed approach, demonstrating competitive performance on a wide range of 27 downstream tasks and the superiority of combining vision/languaae generative pre-training. (3) We conduct an analysis about the impact of different pre-training data sources and pre-training objectives the performance of seq2seq VLMS.
2 RELATED WORK
Inspired by the success of language model pre-training, several studies investigated vision-language pre-training large-scale image-caption pairs. ViI.BRTT (Lu et al.; 2019) and LXMERT (Tan) & Bansal 2019b) first propose to extract visual object features with an external object detection model like FaNI Girshick 2015), feed the image features together with texts into Transformer
Named after the Italian polymath Leonardo da Vinci, who displayed infinite grace in everything. We noticed that this name is used in GPT-3 versioning. However, we think there is no conflict because it is only suffix for specific checkpoint of the GPT-3 family.
2

---

Published as a conference paper at ICLR 2023
Prefie image Modeling
Prefix Language Modeling
Laum
Lpm
The Last Supper Jease with the "oeaive by
Leonando Vind
Cross- modal Encoder
Cross- modal Decoder
00 40'#1 1om
Come
Image Doooder
Encoder Input
Decoder Input
Jenus with the Twekie Apceline, paarang by Laomardo de
123 234 345
Image
321
989 868 777
Figure 1: Illustration of the overall architecture and pre-training procedures of DAVINCI, Transformer-har seuueecce-tosequenee model. Given an image-text pair, DAVINCI first splits either the word sequence or image token sequence into prefix and suffix. It then concatenates the prefix with the complete sequence in the other modality as imput. DAVINCI is trained to recover the suffix with maximum likelihood estimation.
models, and train the model align vision and language representations with masked multi-modal modeling and multi-modal alignment prediction objectives. Many following works (Li et al. 2020
Zhang al., 1b Chen et al. 2020: Li al. 2022a 2021 Zeng al. 2021 Wang et al. 2021a)
propose several new objectives to improve object detection based VLP and explored using vision Transformer (Dosovitskiy et al.) 2021 Touvron al.) 2021) as visual feature extractor.
More recently, FLAVA (Singh et al. 2021). a new vision-languas foundation model, is pre-trained with masked multi-modal modeling objective. Performing competitively language, vision, and vision-language understanding tasks, FLAVA is designed for understanding tasks without text and image generation abilities.
While achieving promising results on multi-modal understanding tasks, most VLMs are based on encoder-only architectures with bidirectional attention, making them non-trivial to adapt multi- modal generation tasks such as image captioning and text-to-image generation. Inspired by the success of seq2seq pre-trained language models such T5 (Raffel al. 2020) and BART (Lewis) it al., (220), VL-T5 (Cho et al. (021) and OFA (Wang et al.) 2022) propose to formulate both vision- vision-language pre- gre-training objectives and various downstream vision-language tasks as seq2seq tasks and pre-train seq2seq VLM by multi-tasking on these tasks. However, the scalability and the zero-shot transfer capability of this approach are limited by the availability of large-scale and diverse lision-language tasks. To this end, SimVLM (Wang et al.) (2021b), the most related work to our approach, instead pre-trains a seq2seq VLM with a simple prefix language modeling objective on text generation. It easily scales to very large and potentially noisy pre-training data and achieves competitive results. However, SimVLM only exploits language self-supervision, and thus it does not perform well image understanding tasks and is unable to tackle image generation tasks. Another recent study is CM3 (Aghajanyan et al. 2022). which proposes a causal masked multi-modal model learned from large web data and differs from our work in pre-training objectives and target tasks.
As for the text-to-image generation task, Ramess et al. (202T); Ding et al. (202T) ; Yu et al. (2022) achieved promising performance by learning an auto- regressive target with Transformer and VQ- V-VEE / VQ-GAN tokenizer. Most recently, Ramesh et al. (2022); Saharia et al. (2022) advanced the image generation capability by using diffusion models and high-quality text embeddings (e.g., CLIP, T5). Therefore, is natural to explore boosting image generation via stronger multi-modal understanding.
Previous studies are good at either image-to-text text-to-image generation, but few studies investi- gate whether these two important capabilities can be learned together and boost each other. this paper, explore making versatile and powerful nulti-modal foundation model that is good at text-to-image generation, image-to-text generation, and multi-modal understanding tasks.
3 DAVINCI
Given the superior performance of regresseeessive language models (LM) (Bronn et al. 2020 Chowdhery al. 2022 Rae et al. 2021) zero-shot and few-shot transfer abilities, decided to
3

---

Published as a conference paper at ICLR 2023
adopt a decoder optimized by language modeling loss to retain the generalization capabillities and an encoder to represent the prefix input. Unlike using a causal mask in the decoder, the encoder employs fully-visible attention for the prefix imput. This architecture resembles prefix language modeling, which shows effectiveness a wide range of language tasks (Dong et al. 2019 Raffel et al. 2020) and enables zero-shot generalization abilities. Contrary to the previous multi-stage approaches (Wang] iet al.] 202 Singh et al. 202I), our model is trained from scratch in an end-to-end manner thanks to the model's simplicity. In this section, we introduce the proposed prefix multi-modal modeling framework and the DAVINCI model. The overall architecture of DAVINCI is depicted in Figure [ We first explain our model architecture in detail in and then introduce pre-training objectives and procedures in 3.2
3.1 MODEL ARCHITECTURE
Textual Feature Embedding Given an input sentence S, we first use WordPiece (Wu et al. 2016 to tokenize it to a sequence of tokens W = {wi, u2,.... Wp). To obtain text features T," for each token w. a token embedding e, and position embedding p; are computed by two separate embedding matrices. Finally, the textual feature embedding T = {ti, t2,..., L4 .... fn is calculated by t = Layer N Nerm(e; + p.). where i indicates the i-th position, and Layer (Ba al. 2016 is a layer normalization function.
Visual Feature Embedding Given an input image L. we first use a CNN backbone to extract and learn the image features. Following (Dai et al., 202T Wang et al. 2021b), we use the first three blocks of ResNet (He et al., 2016) obtain the feature maps. The feature maps are then flattened to F = f2. .... Jm J along the spatial dimension, where m denotes the number of features. To keep the position information of visual features, we inject absolute learned positional embeddings p and the final visual embeddings V {v1, ..., Vi,, Um) are calculated by U; ʃi + Pt. where i indicates the i-th position.
To fuse the textual and visual feature embeddings into common space, we adopt simple canonical Transformer architecture as the fusion module. The input is the combi- nation of visual embedding V and textual embedding T, namely X = {zr1, 42,
Cross-Modas Transformer
.2....... ... {v1, U22 -... Um, (1, l2, (n), The input embedding vectors X are then fed into a cross-modal Trans- former encoder to obtain hidden state vectors H = {h1, h2, ..., h. Finally, a Transformer decoder is applied to generate visual or textual tokens with H and decoder input as illustrated in Figure [ Image Tokenizer and Decoder Because Transformer is modeling on discrete tokens. to unify the text tokens and image tokens, we discretize an image into tokens by an image tokenizer and reconstruct the raw image by an image decoder. The image tokenizer and decoder are implemented with a discrete variational autoencoder (dVAE) (Ramesh et al., 2021). After training of the image tokenizer, it could tokenize an image into sequence of discrete visual tokens Z = {21, 22, .... &m } according to a learned vocabulary. Visual tokens Z serve as the ground-truth labels for the prefix image modeling objective. In work, directly use an off-the-shee image tokenizer and decoder from VQGAN (Esser et al.) (202)), with a vocabulary size of 1024 and a compression rate of 16, which means 256 256 image will be tokenized into 16 x 16 grid of tokens and then flattened to sequence of 256 tokens.
V T]
3.2 PRE-TRAINIGG OBJECTIVES
Our major motivation is to conduct language modeling with image information and image modeling with text information simultaneously, which only requires image and text pairs that are easy to collect, model to have
making our approach easy scale. The interaction would force the vision-langual a deeper understanding of both text and image. Learning from this interaction connects the visual representation with textual representation, enabling zero-shot transfer.
Prefix Language Modeling (PLM) The core idea of prefix language modeling is "given a full image X Xymage and a prefix caption Xtezt, recover the masked textual tokens (i.e., suffix caption Ylez)".. Given an input caption, first randomly mask some continuous words at the end (we call it suffix caption hereafter) and recover the masked textual tokens with full image by optimizing the cross-entropy loss,
LPLM
Σ (T,S)D
log Y text X Ximages X
Xtoxt);
(1)
where and S are images and captions from the pre-training corpus D.
4

---

Published as a conference paper at ICLR 2023
Because of the lack of textual information, recovering the suffix caption requires the model to understand both the image and prefix caption. The full image is rich in semantic information that would help language modeling. The prefix length is randomly decided during training, and especially when prefix caption none, this task will degenerate into "image captioning" task, which forces the model generate caption with the input image.
PLMM =
Σ (1
log p( Y text (maage)
(2)
Prefix Image Modeling (PIM) The core idea of prefix image modeling is "given a full caption and corrupted image (we call it prefix image hereafter), recover the masked visual tokens". Given an input image, we first randomly mask some continuous image patches at the end (we call it suffix image hereafter). The prefix image and full caption will be fed into the model and try to recover the original visual tokens obtained from the image tokenizer by optimizing the cross-entropy loss.
LPIM
Σ
log (Y imnage
X textty Ximage)
(1,S)E
Similar to PLM. when prefix image is none, this task will degenerate into "text-to-image generation" task, forcing the model to generate an image with the input caption:
PPI
Σ
log Y (Yinage
iimage
Xtext)
(4)
(I.S)ED
Unified Learning Objective Our model is learned by optimizing the combination of PLM and PIM.
= LPLM + CPIM
EXPERIMENTS
4.1 PRE-TRAINING DATASETS
Since existing studies pre-trained their models on different corpora, making the fair comparison difficult. Considering results only on state-of-the-art performance would underestimate the potential of this line of research. Therefore, we propose several practical settings including small-scale and large-scale, and then conduct detailed comparisons them in Section 5.1 More details about the datasets are shown Appendix A.3
Type
Datasct
Image Domain
#Total
(TD)
Genome
scale Web Data (SWD)
SBU, CC.3M, CC-12M
Web
14.9M
Regiinn Data (ORD)
VG regions, VG objects, COCO objects, Refenen, Open Image, Ohj365 ImageNet-21K
COCO, Flickr
17.0M
Vision Data (VD)
ImageNet
13.2M
Large-scale Web Data (LWD) (TD)
LAION-400M, DAVINCI-200M
Web
601.3M
800GB
Table 1: Statistics of the pre-training datasets. #Total denotes the total number of image-text pairs.
4.2
DOWNSTREAM TASKS
We test our models' ability and versatility on five dimensions: language understanding on GLUE tasks (Wang et al. 2019). vision understanding ImageNet fine-tuning and 12 popular vision datasets for linear evaluation, multi-modal understanding on VQAv2 (Goyal et al. 2017b), SNLI- VE (Xic et al. 2019) and NLVR2 (Suhr al. 2019). text-to-image generation COCO (Chen et al.) 2015). and image-to-text generation on COCO, NoCaps (Agrawal et al. 2019), and VI.E (Zhou al. 2022b). Details of downstream tasks and fine-tuning process are described in Appendix A.2
4.3
IMPLEMENTATION DETAILS
Our model is base-size Transformer implemented with a 6-layer encoder and a 6-layer decoder, 768 dimensions for hidden states, 512 for maximum input length, and 3072 for intermediate size. We train our model from scratch without initializing the Transformer encoder and decoder. However, the image encoder is initialized from ResNet-101 (He al. 2016) with ImageNet weights since find
5

---

Published as a conference paper at ICLR 2023
BERT
RoDERTa
VIT
MLM
MIM
FLAVA
CLIP
SimVLM
DAVISCI
SimVLM
DAVINCE
Fual.
16C35
18.2
70M
70M
70M
46.4M
647.7M
647.7M
MNLI
FT
64.20
50.33
32.85
83.27
FT
MRPC
FT
84.75
90.20
73.24
84.16
63.74
$2.70
83.14
81.26
84,54
QOF
FT
89.00
91.30
86.08
58.74
59.17
53.39
58.15
89.05
58.92
SST-2
FT
87.96
90.48
91.12
91.37
QNLI
88.28
87.00
RIE
FT
62.50
78.70
50.54
57.76
53.07
59.80
60.72
63.34
64,22
SIS B
PT 88.20
91,20
78.89
$5.67
13.70
50.62
96.27
87.24
87.05
NLP Avg
90.84
86.35
71.55
78.19
46.44
78.79
78.79
80.10
80.23
ImageNet
LE
30.90
41.79
75.54
72.95
74.31
75.87
76.04
77.65
Food101
LE
36.70
53.30
$8.51
$5.49
83.41
89.33
85.52
90.12
CIFAR1O
92.41
93.96
CIFARIOO
LE
Cars
LE
34.20
14.71
70.87
02.84
61.44
72.09
68.83
74.57
Aireraft
LE
46.00
13.83
47.31
40.02
41.28
47.42
47.75
49.55
LE
78.33
86.12
Flowers102
LE
09.20
67.23
96.37
94.94
93.24
96.12
95.41
96.88
MINSST
LE
97.40
96.40
98.42 42
97.338
90.66
98.67
98.45
99.01
Coumey211
27.81
Vision
78.10
77.34
79,50
VQAY
FT
72.49
59.81
73.89
75.03
76.44
73.53
78.74
NLVR2
FT
77.45
77.91
79.72
80.25
12T
FT
58,00
38.50
38, 38.10
39.20
128.1
130.44
T2
22.41
TZI
FT
23.58
19,82
VQAV
PS
$4.69
54.85
51.88
$4.90
SNLE-VE
15
68.04
12T0B44
FS
35,90
36.40
32.70
37,00
RTOC
FS
117.75
120.43
112.30
122.56
12
75
25
VLUEB84
75
9.70
9.40
10:40
11.60
VLUERC
25
33,92
$4.80
39.75
40.83
C
25
48.6
58.58
THERS
75
25
20.83
24.21
Multi-modal Ang.
$7.89
58.30
59.13
62.50
Table 2: Experimental results on vision, language and multi-modal downstream tasks. @B4, @C denote BLEU@4, CIDEr, respectively. 12T and T2I denote imago-to-teys and text-to-image tasks. Multi-modaa Avg. the average score of all multi-modal tasks. FT: fine-tur tuming. LE: lincar evaluation, FS: fow-shot, ZS: zcro-shot. Under few-shot setting, fine-tune pre-trained model for 3 epochs on 1% training data. Results for BERT are obtained from Iki & Aizawa (02TT. Results for RoBERTa
from its corresponding paper (Liu al. 2019) and they use the mid-training (Phang el al. 2018) on MNLI RTE, MRPC and STS-B while other models (e.g., BERT, Sim VLM, daVInCI) do not apply this trick. Results for ViT are from ViT-Based Base/16 model (Radford et al.
2021 We list the reported performance of text-only and image-only models in grey for reference.
a warm start provides reliable visual representation and helps the convergence. All pre-training experiments conducted on 32GB NVIDIA V100 GPUs. The model trained on the largest data takes around 10 days on 1024 V100 GPUs. We adopt dynamic masking in our experiments, where the masking ratio is randomly sampled from uniform distribution U(0, 1). More details of the fine-tuning, network architectures, and byper-parameten setups are given in Appendix A.1
4.4 EXPERIMENTAL RESULTS
We extensively compare the performance DAVINCI with state-of-the-art unified foundation models and vision-language models across vision, language, and multi-modal tasks, accessing five different abilities: (1) text understanding. (2) image understanding, (3) text-to-image generation, (4) image-to- text generation, (5) multi-modal understanding.
Overall Performance We report the overall performance on language tasks from GLUE, 12 vision tasks, 3 multi-modal tasks, 3 image-to-text tasks and text-to-image task. We compare our model with FLAVA and SimVLM two the most recent and best performing vision-language
Since ViM is not noun-ee and uses 1.8B in-house data without telling the exact size of its base model, we replicate it on our data with the same size as DAVINCI. Experiments on Sim VL.M small ensure successful reproduction (see Appendix A.4

---

Published as a conference paper at ICLR 2023
Text MNLI Ace
Visioss
ImageZText
Text2lmage COCO /
Multi-mddal
Model
4Params
ImageNet
COCO R04/C
VQA
NLVR2
LF/FT
tost-dev / sost-4id
dev test-P
Encesier-nnly Malt-modaa Models
ViaVL Zheng et al
2021b
157M
38.27 129.3
75:95 / /76.12
82.05
83.08
VILT
8SM
70.85
74,91 /75.57
ALBB
210M
75 76.04
83.14
2A0 [02]
84.41/ /8.76
VLMO
Wang el mi 202TR
76.64/ 11.64/76.88
82.77/ 83.34
Encoder Ceeoode-Decoderrr Multi modol Modele
UNICORN
Yeap al 2021
35.87 199.1
69. / 00 49
-/-
Uni BRi-ENNN
PREU FuMeU
110M
72.20 72. 50
-/-
Pixel-FFF
Wile 2020
144M
74.45 / 7T4 55
76.50/ 77.30
E2E VLP
9IM
36.27 117.3
73:25 73.67
17.25 / 77.96
VL-T5
11.5/166.5 35.17 116.6
70.30
/3.60
VL-BAKT
220M
-771.0
71.70/ 10.30
TentZrnuge Madels
DM-GAN
2hu el al 2019
32.20/250
DALLE [Rmmerr 2e 10 mid
(250M)
12B
17.90/27..5
DALLE
[Kmmnsh 2f
(640M)
82M
15.79 / 29.22
Cog View
18.30/27.10
Uniiied Models
Unifying
[Iuann al
2021
37.3/ 122.6
-/29.90
FLAVA
Sinch ot al.
2X41
240M
80.33
75.54/-
72,80/ 72,
VinVLNT
Wamee el
(640M)
153M
83.27
76.04 /-
38.5/ 128.7
75.04/7500
78.82/ 79.72
SimVI M
(1S8)
83.40
80.60/
39.00 134.8
77.87 78. 14
81.72/81.7
OFA
182M
84.30
-782.20
41.07 138.2
21.50" / /20.00
78.00/ 78, 10
Hloreize
790.05
80.16 730.96
DAVINCI
83.13
78.81
39.2: 130.4 17.44 (22.41"//44211 (19.82")
76.32/76.46
80.03/ 90.25
Table 3: Comparison with sfate-of-the-art vision-language models on vision, language, and mudti-modal downstream tasks. All results are from base-size models. LE and FT denote linear evaluation and tne-tunnng performance, respectively. Image2Text results are reported without CIDEr optimization. 1 are our reproduced models." are the results after finc-tuning. SimVLM (1.8B) and OFA are pre-trained with much larger corpus or human-labeled data of many downstream lasks, and thus they are not comparable and are labeled in gray. Florence (Yuan et al.) (02T) is pre-trained with much larger model size pre-training data (900M), so the numbers are in grey. bold denotes the best across unified models.
637M) and more
foundation models. We also include comparisons with some baseline models (e.g., MIM, MLM, CLIP). There are several observations. First, DAVINCI (column 8) outperforms FLAVA (column 3) and SiM (column 7) across almost all tasks, providing new and stronger unified foundation model. Compared with FLAVA, DAVINCI improves an average of 2.04%. 1.68% on language and vision tasks, respectively. Compared with Sim,LM, DAVINCI achieves comparable results on language tasks (+0.13%) while performing much better on vision tasks (+2.46%). To make a fair comparison in terms of similar data size, compare FLAVA (70M data, column 3) with DAVINCI (46.4M data, column 6). It is observed that DAVINCI still outperforms FLAVA even with much less data. Considering the multi-modal tasks, DAVINCI consistently outperforms FLAVA and Sim VLM on VQA and VE. Note that FLAVA is incapable of generation and SimVLM cannot generate images; only DAVINCI is competent to all tasks and demonstrates stronger capability of unifying vision and language tasks.
Zero-shot and Few-shot Transfer One of the critical benefits of generative pre-trained vision- language models is the good generalization ability on zero-shot and few-shot tasks, For zero-shot transfer, two out-of-domain distribution datasets are considered (NoCaps and VLUE), with results shown in Table 2 First, DAVINCI outperforms Sim VLM on both zero-shot and few-shot settings, demonstrating its better transfer capabilities. It also shows the effectivenesss and robustness of the synergy of our proposed language supervision and image supervision. Second, it is observed that the performance improvement is bigger on 647.7M data (column 7 v.s. column 8) than 46.4M data (column 5 v.s. column 6). This shows DAVINCI generalizes well with the increase of large-scale data. We even observe some performance drops on small data (46.4M) but excellent performance improvements on large data (647.7M) It is consistent with the recent observation that zero-shot ability could only be triggered with large pre-training data (Wei et al.. 2022) and scaling to large data and keeping simple training objectives benefit generalization performance (Wang et al. (0211).
Comparison with state-of-the-art vision-language models In addition to unified lision-language foundation models, we compare DAVINCI with state-of-the-art lasion-language models as well. The results are shown in Table 2 DAVINCI demonstrates its superiority in vision understanding and text-to-image generation. Compared with current popular auto-regressie
image generation models like DALLE and View, our model achieves comparable IS and better FID scores with significantly fewer model parameters than DALLE and Cog View. Note that the original DALLE is implemented based VQVAE, so here, compare our model with reproduced VQGAN-based DALLE with
7

---

Published as a conference paper at ICLR 2023
Settings
Pre-training Data
#lmage
&Caphion
Models
COCO
VQA
SNLLVE
NLVR2
ID SWD
ORD
VD
I.WD
B044/C
Acc
Acc
Acc
SILVLM
35.2/115.06
76.10
71.21
0.2M
1.3M
DAVINCI
35.8/ 117.30
69.25
76.22
72.55
2
/
15.1M
16.2M
SImVLM
37.07 122.63
71.54
78.36
75.50
DAVINCI
37.4/123.11
71.88
78.62
77.46
3
v
/
2.7M
18.3M
SIEVLM
38.27 123.85
69.37
76.65
70.50
DAVINCI
38.0/ 124.20
70.02
76.92
72.01
SimVLM
36.2/ 119.73
70.53
76.90
73.25
13.4M
14.5M
DAVINCI
36.6/ 121.27
71.23
77.40
74.62
5
d
d
30.5M
46.4M
SimVLM
38.5/ 128.12
71.84
76.75
DAVINCI
38,6/ 128.73
73.53
79.24
77.55
6
601.3M
601.3M
SimVLM
37.3/ 123.81
73.73
78.79
77.69
DAVINCI
37.6/ 124.42
73.95
79.29
78.54
SimVLM
37.9/ 125.50
74.64
79.05
77.68
7
601.5M
602.6M
DAVINCI
38.1 125.91
74.91
79.22
78.12
SiLVLM
38.5 128.25
75.04
79.32
78.82
6318M
647.7M
DAVINCI
39.1 130.21
76.32
80.04
80.03
Table 4: Evaluation on downstream tasks using COCO Captions, VQA, SNLI-VE, and NLVR2. #Image and #Caption denote the numbers of images and image-text pairs that used in the pre-training.
similar model sizes, and find DAVINCI still achieves significant improvement over it. Generated images are presented in Appendix[A.Il for further qualitative comparison.
On multi-modal tasks such as VQA, DAVINCI not only outperforms unified models (e.g.. SimVLM (640M)) and other encoder-decoder multi-modas models (c.g.. E2E- VLP, VL-T5), but also achieves competitive performance with many conventional encoder-only multi-model models (e.g., VL, ALBEF, VLMO). Note that VIM (1.8B) and OFA are not directly comparable because Sim VLM uses 1.8B in-house image-text pairs, and OFA uses human-labeled data of many downstream tasks during pre-training. Even though, we still report their results for reference and observe a better performance on ImageNet fine-tuning and text-to-image generation than OFA.
The advantages of image generation over DALLE/ / Cog the superiority of image-to-texx over Sim VLM, and the competitive performance with conventional synergistic effect of our proposed PLM (language supervision) and PIM (image supervision).
models demonstrate the
ANALYSIS
5.1 IMPACT OF PRE-TRAINING DATASETS
In this section, we disclose the impact of various multi- modtl data sources for VLMs. We choose Sim VLM and DAVINCI our baseline models for their competitive performance, the capability of training from scratch, and the scalability of extending to the noisy large-scale corpus. We use the same text corpus, C4, for all the variations. The results are shown in Table 4 In general, the performance is increased along with the data size, and DAVINCI consistently outperforms SimVLM on almost all the data settings and all the downstream tasks. Both object-region data and vision data are clearly helpful in vision language pre-training (refer to settings and 4). We surprisingly observe that models pre-traie trained object-region data with much fewer images performs models pre-trained with small-scale web data on the COCO Caption task (refer to settings 2 and 3). Although large-scale web data is usually noisier than small datasets (e.g., ID, ORD, VD, and SWD), it is powerful for multi- modal pre- pre-training (refer to settings 5 and 8). We believe our analysis has broader impacts on the research of VLMs in the community. First, this enables fair comparisons for pre-trained models in the same data settings. Second, one can focus on the model designs at part or all of the data settings according to available computation resources. Third, we reveal that object-region and vision data, normally overlooked in VLM pre-training, also play significant role.
better than
5.2 ABLATION STUDY
To verify the contributions of different modules in our framework, we ablate them and evaluate DAVINCI on five kinds of downstream tasks: language understanding (MNLI, SST-2), vision under- standing (ImageNet, Food101. CIFARIO). multi-modaa understanding (VQAv2, SNLI-VE, NLVR2), image-to-text generation (COCO Captions), and text-to-image generation. Experiments are conducted with the same model architecture on in-domain data (ID). The results are shown in Table
Effects of Objectives First, all three objectives (PLM, PIM, and Text2Text) bring improvement and the combination confirms a synergistic effect. Second, it is observed that without PLM, the performance decreases significantly multi-modal understanding and image-to-text generation,
8

---

Published as a conference paper at ICLR 2023
COCO
VQA
SNLE-E
SLYRE
ImageNet
Foodlol
CIFARIO Acc
MNLI
SS1-2
T21
R84/C
Ace
Ace
Aoc
Ace
training
32.1/96.71
52.73
66.32
DAVINEL
35.8/117.30
69.25
76.22
72.55
48.88
75.32
73,82
81.76
90.25
12.35 53.14
PLM
23.6/
63.15
73.91
72.98
91.42
89.97
10.26/ 59.64
- PIM
34.5/116.58
68.89
75.79
69.78
45.54
71.18
70.11
81.94
90.53
Texx2Test
34.1/ 115.21
68.14
75.38
70.34
48.07
74.26
73.23
76.48
88, 14
12.07 / 54.77
PI:-0
154//11700
66,90
75.52
71.05
48.45
68.1 18
73.73
78.69
00
11 55.76/55.36
PL=155%
35.7/116.53
69.16
75.09
70.44
41.58
52.15
68.55
79.02
89.46
PL=50%
35.1//15553
68.55
37,69
70.15
78.59
MIM In-gainting
34.7/113.4
68.18
75.34
69.06
48.46
56.95
72.79
81.72
59.84
/4.13
34.5/112.5
67.46
75.41
47.50
71.20
81.55
89.84
9.97 16815
Token Projection
17.7/49.2
52.13
25.63
90.25
11
Patch Projectivn
25.7/79.5
$7.69
71.92
57.45
36.23
69.40
81.73
90.05
11.4/66..8 /61.87
Table 5: Ablation study on COCO Captions, VQA, SNLI-VE, NLVR2, ImageNet, Food101, CIFARIO, MNLI, SST-2, and text-to-image (T2I) generation. "-" denotes removing the corresponding objective. PL denotes the prefix length under fixed masking ratio settings. Because the linear probe requires a pre-trained model frozen. "No Pre-traine traininnnn" results on ImageNet, Food101, and CIFARIO are not reported and labeled by ", For T2I, we report the zero-shot results. Note that the following four variants cannot perform zero-shot text-to-image generation (labeled by "): (1) No Pre-training, (2) DAVINCI - PIM, (3) PL=15%, and (4) PL=50%.
indicating the importance of language supervision. Third, PIM brings more gains than PLM and text2text on vision understanding, which is expected because it enhances the vision encoding ability with image supervision. In addition, the text2text objective is important to text understanding. Last, on the text-to-image generation task, it observed that PLM is also helpful, confirming the synergistic effect of PIM and PLM again. Intuitively, PIM and PLM can help each other learn the alignments of visual and textual features, which will benefit both image generation and other multi-modal tasks.
Effects of Masking Ratios Our model adopts dynamic masking ratios as described in Section 3.2 We also conduct experiments with static masking ratios with the prefix length fixed to 0, 15%, and 50%. The comparison between dynamic masking ratios and static masking ratios (PL=0, 15% and 50%) reveals that dynamic masking is better. We attribute this improvement to the smoothing effects of dynamic masking ratios. We also find that the standard language model (PL=0) performs worse on VQA, Food101, and -maaee--mmae generation, which is consistent with the observation in VLM. In our experiments, the masking ratio is sampled from a uniform distribution U(0, 1).
Effects of Masking Strategies Here we also compared three different masking strategies: 1) masked image modeling (randomly masking some patches), 2) in in-painting (randomly masking some continuous spans in the middle of the image), and 3) tuffix-painting (ours). The results are shown in Table[5] Both masked image modeling and in paintinnnng are effective and competitive. It is observed that suffix-paintiin is better than masked image modeling and in in-painting across all tasks, demonstrating that suffix-panc suffix-painting works well.
Effects of Image Feature Extraction There are several different ways to extract image features. We compare three different image representation methods: 1) token projection (projecting the prefix tokens to the hidden dimension of the backbone network on the token-level), 2) patch projection (similar to ViT embedding, we split an image into fixed-size patches, embed each of them by trainable linear projection on the pixel-level), and 3) ResNet feature extraction (ours). From the results in Table we observed that ResNet feature extraction outperforms token projection and patch projection by a large margin. Therefore, we decided to adopt ResNet to extract image features.
We provide more details and discussions about the effects of compute (A.5). masking strategies (A.6). image feature extraction methods (A.7p. and scaling effects of data size (A.8) the Appendix.
CONCLUSION AND DISCUSSION
In this work, we first benchmark several settings on sequence-to-sequeme vision-language pre- training in terms of pre-training dataset size, aligning SimVLM and our model on them. We propose a simple and unified generative pre-training model, DAVINCI, to simultaneously leverage the language supervision and image supervision through two objectives under unified framework: prefix language modeling and prefix image modeling. DAVINCI is simple yet effective, demonstrating strong capabilities in both multi-modal writing and painting tasks. Experimental results explicitly imply that combining suffix caption generation and suffix image generation offers large gains on all benchmark settings. We also discussed limitations and future work in Appendix A.10

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their valuable suggestions. We would like to acknowledge Yan Zeng, Wenguan Huang. and Zhi Zhang at ByteDance, and Zhiling Zhang at Shanghai Jiao Tong University for their generous assistance in data collection and helpful discussions. We also wish to thank Hang li ByteDance, and Tong Zhang at HKUST for inspiring feedback, valuable comments, and great support to this work.
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201007520, 2022.
Eneko Agirre, Lluis Mårquez, and Richard Wicentowski (eds.). Proceedings of the Fourth Inter- national Workshop on Semantic Evaluations (SemEval-2007), Prague, Czech Republic, 2007. wwt.
Association for Computational Linguistics. URL http::://ccaatttttttttee/
Harsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, and Stefan Lee. nocaps: novel object captioning at scale. 2019 IEEF/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pp. 8947-8956 IEEE, 2019. doi: 10. 109/ICCV.2019.00990
URL httpp:://aeotm org/10gg//oooooo 20019/ICCCS 2019. 00904
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv preprint,
2016. URL https::/aaxxi..orrrr abs////////// /1600.607.00000 06450
Hangbo Bao, Li Dong, and Furu Wei. BETT: Bert pre-training of image transformers. arXiv preprint, 2021.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-nn 10ld--nm- discriminative compo- nents with random forests. In European conference computer vision, pp. 446-461. Springer, 2014.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL htt.:s://///ooseoeec 1457 L457c0ddbicb4961444 5tw.110bbbbbaaa112222222220
.er.nneuripp wwwrips.rpp... wwwpppppppppppppapaaa.aaaaeeeaaa http://appr/222//aahh /2020/hash/ html
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.067.. 06794, 2022.
Xinlei Chen, Hao Fang, Tsung- Yi Lin. Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO Captions: Data collection and evaluation server. arXiv preprint, 2015.
10

---

Published as a conference paper at ICLR 2023
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In European Conference on Computer Vision (ECCV), 2020.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and- -nd--annudg-aee tasks via text generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 1931-1942. PMLR, 2021. URL http:
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311 2022. URL https
7/arxiv. abs/ 2204. 02311
Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp. 3606-3613. IEEE Computer Society, 2014. doi: 10.1 10.1009//VP 4,461, URL http:ttts/// /oii wwraadi..ooooooooo orgg 111.111.11111/110 /t//CVPR 201 2014.4055 .461
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourleenth international conference on artificial intelligence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005.
Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing $ystems,34:3965-20 34:3965-3977, 2021.
Jacob Devlin, Ming-Wey Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-4186, Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199 1423. URL /ttt:://eclanthboooo.oo///////
Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. Zen Pre-training chinese text encoder enhanced by n-gram representations. In Findings of the Association for Computational Linguistics:: EMNLP 2020, pp. 4729 4740, 2020.
Shizhe Diao, Ruijia Xu, Hongjin Su, Yilei Jiang, Yan Song, and Tong Zhang. Taming pre-trained language models with n-gram representations for low-resource domain adaptation. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 2021
Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, Yong Lin, and Tong Zhang. Black-box prompt learning for pre-trained language models. Transactions on Machine Learning Research, 2023.
URL htttps::/
not wwwp:///ooommmmddIsoodds www.vvsGPPPPPP. 7xRvm
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin. Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering ext-te--maae generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL
htttss://accaantooooooo www.ooooogooooooooooooooooo 200-5002
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre- pre-training for natural language understanding and generation. In NewrlPS, pp. 13042-13054, 2019.

---

Published as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May net/ forum? d dNTTY
3-7, 2021. OpenReview.net, 2021. URL htttt::t//ppprreeeeeees
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution im- age synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12873-12883, 2021.
Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, and Lei Li. Contextual representation learning beyond masked language modeling. In ACL (1), pp. 2701-2714. Association for Computational Linguistics, 2022.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan, The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1-9, Prague, 2007. Association for Computational Linguistics. URL
https://aclantee
org/ wwg///7-0000 W07- 1401
Ross B. Girshick. Fast R-CNN. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1440-1448. IEEE Computer Society, 2015. doi: 10.1 N0.1109/1CC.22015.1166 URL
wttp//io....ooooo//oo/eeeoe 1100/11111110110 ICCV 2015. 169
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. ArXiv preprint, abs/1706.02677, 2017a. URL
02677
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 pp. 6325 6334. IEEE Computer Society, 2017b. doi: D..109/CVPPR.2017..77 URL httpt::///// //t/i org / 10. 11t.1111.//..2111111011
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiceolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10. 10./VR/222.
URL|attss:/ //cci wwg/ee.../...oo. wwt.111199//...111100
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 7772--9735. IEEE, 2020. doi: 10. 1091CC94422000.22000000 2020. 00975
httphtts/://// /oi org 101111199////42200
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pp. 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6626 6637, 2017. URL https: http:////ccccccpooeee/eee wwwrrcceedinss.nnuiiip..... www.rnss...... http:ppp...ppppp/p//////
N. Vishwanathan,
Abstract.hen.
Kashmir Hill and Jeremy White. Designed to deceive: Do these people look real to you. The New York Times, 11, 2020.
12

---

Published as a conference paper at ICLR 2023
Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi- directional image and text generation. In Proceedings of the 29th ACM International Conference Multimedia, pp. 1138-1144 2021.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-BERT: Aligning image pixels with text by deep multi-moda transformers. arXiv preprins, 2020.
Taichi Iki and Akiko Aizawa. Effect of visual extensions on natural language understanding in assion-and-langage models. Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 2189-2196, 2021.
Shankar Iyer, Nikhil Dandekar, Kornél Csernai, et al. First quora dataset release: Question pairs. data quora. com, 2017.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Lc, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- lsigun- representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Reseurch, pp. 4904-4916, PMLR, 2021. URL http: pttp://pccceeding.. mlr. wtt...ppeesssv///./ 5aa2bb....
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Aision-add-aanee
transformer without convo- lution or region supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5583-5594, PMLR, 2021. URL http://proceedingg mir. wtt../.9es////////..
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization, Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoye. BART: Denoising sequence-to- -equueece pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 787 1-7880, Online, 2020. Association for Computational Linguistics. doi: 10. (0.18653/v1/2020.es main. URL
https://ache httpss//aclaatthoooo.ooooo aca.mmmmi..... 703
Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint, 2019.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq- Jenq Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. European Conference on Computer Vision (ECCV), 2020.
Yehao Li, Jiahao Fan, Yingwei Pan, Ting Yao. Weiyao Lin, and Tao Mei. Uni-eden: multi-granular
Universal encoder-
decoder network by multi granular vision-language pre- pre-training. ArXiv preprint, 2022b. URL htRtshttss://aat. www.aaawwwwwvroraaaaaaaaaaaase .04026
13

---

Published as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeytlemoyer. and Veselin Stoyamov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:// net
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agoostic vi- siolinguistic representations for vision-and-language tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13-23 2019, URL http::ppppp/pppcp.siiiiioooo c74 2ee.:lleee222/44aa5aaae -aaaaaadtttteelaaaa--aattttt.... html
neurips. wwwpppppppppppeeeeeeeeceeeeaaeet haw.aperr//99//aiaiaa
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David García, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver; BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL htttps://pppeeeee
ex. net. Forummiddrrgsseer
Maria-Elent Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Rui Pan, Shizhe Diao, Jianlin Chen, and Tong Zhang. Extremebert: A toolkit for accelerating pretraining of customized bert. arXiv preprint arXiv:2211.17201, 2022.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pp. 3498-3505. IEEE Computer Society, 2012. doi: 2.110//CVRR.22122624800 URL
https://del crg. 1119//1P2. 6248092
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pupers), pp. 2227-2237, New Orleans, Louisiana,
2018. Association for Computational Linguisties. doi: 18.18653/v 1/N18- 1202. / hocogntooooogyoropg
URL attps
1202
Jason Phang. Thibault Févry, and Samuel R. Bowman. Sentence encoders stilts: Supplementary training on intermediate labeled-data tasks. ArXiv, abs/1811.00 1.01088, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021. URL httpttt:ppppocceeinng.. ml www..pppssssse... peress/v139/e http:////1/aaaaffrr//// radfor221 html
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 12. 2021.
14

---

Published as a conference paper at ICLR 2023
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/ 1264. URL
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8821-8831 PMLR, 2021. URL http::ttppoccceee///eeoee mlr. press v1 v939/1119// htt./rammsh2lll html
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 0rXiv:2204.06//5, 2022.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1179-1195, IEEE Computer Society, 2017. doi: N0.1109/CVPR.2017..35 URL https:/ httpps://doo..csco org/ 110.11109999000 1ttp://.....10/...220000
131
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 1-22, 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.oomm 04671, 2016.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.1 14877, 2022.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2226 2234-2234, 2016. - html
2016 2016/hash/Sa33 Satteshhha33333mm /ttp://33abbbee/22/aaa22 299 /1116//6664466000
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, (3(11)):54-6 2020.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision aligmment model. ArXiv preprint, abs/21 12.04482, 2021. URLppsts:///axx wwtparwwwwwwwi.rr.oooeeeeeeoeeee wtt/:be///////eoeee /ttpa//b//ll..... 04482
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, 2013. Association for Computational Linguistics. URL htt https::/aclaathholll
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: pre- training of generic visual- linguisic representations. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htt.hhttps://pennee/r
15

---

Published as a conference paper at ICLR 2023
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6418 6428,, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/119-1666 1644. URL https://acaanntholoooooor/r/////
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818-2826 IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.308 URL https://oan org/ 10. 1109/C122. 308
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 5100-5111, 5111, Hong Kong, China, 2019a. Association for Computational Linguistics. doi: 10. 1/D19-1514444 URL
Hao Tan and Mohit Bansal, LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- (JCNLP). pp. $100-5111 Hong Kong, China, 2019b. Association for Computational Linguistics. doi: 10. I/D 1514. URL httpttp:://///aaaboooooo..ooe/e
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR, 2021. URL touvren2la.. h
mll. colr.prees pprsss/v333
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. h httpss//eBBaaaeeeeeeeeaaaeeeeeea
2017. URL httpptts/ppocceeeiiiingg htw.eeiiiiinnnnnnninneeee wwwrrips.cc/ppee paper/
Abst httll
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 5998 6008, 2017. URL URL http::ppppe//c///gg//s////e www.eneeuriee wwtpurip..c/paaeeee. wwtpppppppp///////////p /hash/ 3f5ee243547dees1 http://aaaee1aa000333c3444443433 Aberaaaa-bbbsacccc html
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL.Atttss//opeereviiee
net /est/fooo forum? 2id= d= 7
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple scquence-to-seqqeence learning framework. ArXiv preprint, abs/2202.03052, 2022. URL https
//axxiv.org/abbt 22wp://222222 2202003052
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vasion-language pre-training with of experts. ArXiv preprint, abs/2111.02358, 2021a. bRtttss:///axxt
org Btg/abs/21111 0ttpb///211..223.3.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint, 2021b.
16

---

Published as a conference paper at ICLR 2023
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computationnll Linguistics, 7:625 :,625661, 2019. doi: 10.1162/ tacl a_ 00290. URL httttp:://acaattttaeeee
wwtp:/9-19/00999/0/00 Q14-1040
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions Machine Learning Research, 2022. URL httpp::/ wtw.ppppwwre......eeeoeoeeoooeee net//orumm id=yzkSUSzdwD Survey Certification.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo- gies, Volume (Long Papers), pp. 1112-1122, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/11/N18- 1101. UtL.psssss//accaaatco.ooo
N18-1101
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv preprint, abs/ 609.080 (.09.081444 144, 2016. hatt:s:///r/aattta
/ttp//////////aaaae//////eaaeeea /109. 0066699.800 08144
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint, 2019.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bf Com- pressing BERT by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7859-7869, Online, 2020. Association for Computational Linguistics. doi: 10.18653/w1/2020..com wmw.ppmmmmppp..mm. main.633. URL
org/2200 mainpp-main... 633
Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, and Fei Huang. E2E- VLP: and-to-end vision-language pre- pre-training enhanced by visual learning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 503-513, Online, URL
202 Association for Computational Linguistics. doi:
.42
Jingjing Xu, Wangchunshu Zhou, Zhiyi Fu, Hao Zhou, and Lei Li. A survey green deep learning. ArXiv preprint, abs/2111.05111 2021b. URL /tttps://axx/ttti wwtpeiiiiirrggaaaaa wttpo/////gg..o////////
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In 2018 IEEE Conference on Computer Vision and Pultern Recognition, CVPR 2018, Salt Lake City, UT, USA. June 18-22, 2018, pp. 1316-1324. IEEE Computer Society. 2018. doi: 10. 2w1.1099/VVR..22.8.... 2018/htm1/ http//ttmm//xAAttnAAA Fine Gra ined Tww.__Ceeteeteeeeeeeee 2018 2wwp_p0peer______
URL /ppp:://peeccccosssss http///ppcccccceeeeeee twecti con/ http://ottettttooe cvpr
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Crossing the format boundary of text and boxes: Towards unified vasion-language modeling. ArXiv, abs/211 11. 12085, 2021.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. ArXiv preprint, abs/1708. 003888 2017. URL httthhhtt:///xxtiti wttpaaawwwwaaaaaaaaaaaaaaaaeseee wwt./oor/////////oo 1708. 03888
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint 2tti::200.1078880 2022.
Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz. Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14448-14457, 2021.
17

---

Published as a conference paper at ICLR 2023
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang. Jianfeng Wang. Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. arXiv preprint, 2021.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. ArXiv preprint, abs/2111.08276, 2021. URL btttts:////aiill org/ /bs/
08276
Han Zhang, Weichong Yin, Yewei Fang, Lanxin Li, Boqiang Duan, Zhihua Wu, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Erii-vilg: Unified generative pre-training for bidirectional vision- language generation. ArXiv preprint, abs/21 12. 15283, 2021a. URL bttps:://///
/
2112. 15283
Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinVL: Revisiting visual representations in vision- sssoon-anguuge models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. BERT loses patience: Fast and robust inference with early exit. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020. /tt.itttp://proceeedi F4 (ct.://.........ccccceeeeeeeaeae
neuri ps
cC/
Wangchunshu Zhou, Tao Ge, Canwen Xu, Ke Xu, and Furu Wei. Improving sequence-to-ssqqueeee pre- training via sequence span rewriting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 571-582, Online and Punta Cana, Dominican Republic, 2021a. Association for Computational Linguistics. doi: 10. 11151//1/2022.eemm wenpleemmllppamme m5.n.45. URL httess///cclaatcooooog.orr////// 1ptp:///......rr22m...mmpmpppppp wwwmpppmminn...
Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee, and Xiang Ren. Pre-training text-to-text transformers for concept-centric common sense. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL http:ttttt////eeeeviieae wwt.://.eeeeeeeeeeeeeeeeeeeeeeee fow.foorumioo (tt.ormmmmid-32201111111
Wangchunshu Zhou, Canwen Xu, and Julian McAuley. BERT learns to teach: Knowledge distillation with meta learning. In ACL (1), pp. 7037-7049. Association for Computational Linguistics, 2022a.
Wangchunshu Zhou, Yan Zeng, Shizhe Diao, and Xinsong Zhang. Vlue: A multi-task benchmark for evaluating vision-language models. CoRR, abs/2205. 1..2200...22.... 15237, 2022b.
Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang. Efficient neural network training via forward and backward propagation sparsification. Advances in Neural Information Processing Systems, 34, 2021c
Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3599, 2021d.
Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, and Tong Zhang. Probabilistic bilevel coreset selection. In International Conference on Machine Learning, PMLR, 2022c.
Minfeng Zhu, Pingho Pan, Wei Chen, and Yi Yang. DM-GAN: dynamic memory generative adver- sarial networks for text-to-image synthesis. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 5802-5810. Computer Vision Foundation IEEE, 2019. doi: 10. 1009/PR. 20w..099.000 00595. URL http:://ppnaccee//
thecvf. ccm/ content CVPR 2019/atm1/ Zhu DM- GAN Dynamic Memory Generative
Adversarial Networks for
Image Synthesis CVPR 2019 paper.htm
18

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DETAILS OF PAPEE-PARAMETERS
Pre-training Our model is a base-size Transformer implemented with 6-layer encoder and 6-layer decoder, 768 dimensions for hidden states, 512 for maximum input length, and 3072 for intermediate size. We train our model from scratch without initializing the Transformer encoder and decoder. The image encoder is initialized from ResNet-101 (He et al. 2016) with ImageNet weights since we find a warm start provides reliable visual representation and helps the convergence. For models pre-trate pre-training on large-scale data, we optimize 10 epochs while for other small-scale datasets, optimize 40 epochs with the AdamW optimizer. The weight decay is set to 0.01 with 31 = 0.9, 32 = 0.999. The learning rate is 2e-4 with a warm-up period for the first 2% steps and linearly decayed to 0 after 2% of the total training steps. In each batch, there are 8,192 image-text pairs for eett-to-mmage generation and image-to-ter generation with 8, 19. text-only documents for text-to-text generation. We use center-crop to resize each image the size of 256x256, which is the only data augmentation used during training. All pre-training experiments are conducted on 32GB NVIDIA V100 GPUs. We adopt mixed- -eexxee-reisioonn (Micikevicius et al. 2018) to accelerate training and save memory. The model trained on the largest data takes around 10 days on 1024 V100 GPUs. The default settings are shown in T We adopt dynamic masking in our experiments, where the masking ratio is randomly sampled from uniform distribution U(0, 1).
Fine-tuning The learning rate is [le-5, 5e-5] and our model is optimized by AdamW. Because the image resolution differs between pre- tre-training and fine-tuning, the position parameters are adapted using linear interpolation. For all downstream tasks, we apply random resize crops and horizontal flips augmentation during training. All fine-tuning experiments are conducted on 32GB NVIDIA V100 GPUs. The default settings for text classification, image classification, multi-modal understanding
and image-to-text generation are shown in Tables 7 8 and 9 respectively.
config optimizer
value
Adam W
Loshchilov & Hutter 2019
learning rate weight decay
2e-4
0.01 31, 009, 0.999 8192 linear decay 0.02
optimizer momentum batch size learning rate schedule warmup ratio (Goyal et al. 2017a) training epochs augmentation
{10, 40)
Random
Table 6: Pre-training setting.
config
value
optimizer learning rate weight decay optimizer momentum batch size learning rate schedule warmup ratio training epochs
Adam W (1c-5, 2e-5, 5c=5)
0.01
31, 0.9 0.999 [16. 32. 64] linear decay
0.1
(5. 10)
Table 7: Text classification: GLUE setting.
A.2 DETAILS OF DOWNSTREAM TASKS
Language Understanding We conduct experiments
GLUE benchmark including
MNLI Williams et al. 2018). CoLA (Warstadt et al. 2019). MRPC (Dolan & Brockett
2005).
QQP (Iyer al. 2017 SST-2 (Socher et al. 2013). QNLI (Rajpurkar al. (2166).
19

---

Published as a conference paper at ICLR 2023
config optimizer base learning rate weight decay optimizer momentum batch size learning rate schedule
value
LARS
You ct al. 0.1
2017)
0
0.9
16384
cosine decay
warmup epochs
training epochs augmentation
90
RandomResizedCrop
Table Image classification: Linear probing setting.
config
value Adam W
optimizer
learning rate
[le-5, 5c-5] 0.02 S1, =0.9. 0.999 1024
weight decay optimizer momentum
batch size learning rate schedule
linear decay
warmup epochs
[2. 5]
training epochs
[5. 15]
label smoothing (Szegedy al. 2016)
0.1
augmentation
RandomResiiz:dCrop. HorizontalFlips
Table 9: Multi-modal understanding and image-to-text generation: fine-tuning setting.
RTE (Dagan et al. 2005 Haim et al. 2006 Giampiccolo et al.
2007 Bentivogli al. 2009). and
STS-B (Agirre et al. 2007). ) We follow the practice of BART (Lewis et al. 2020) and feed the same input to the encoder and decoder, and the hidden state of the final decoder token is fed into a new multi-class linear classifier regression head. MNLI results
average of MNLI-m and MNLI-mm. MRPC and QQP results are average accuracy and F1. Matthews correlation coefficient (MCC) is reported for CoLA and Pearson correlation coefficient (PCC) is reported for STS-B.
Vision Understanding We conduct vision experiments in both fine-tuning and linear evaluation (linear eval). The linear evaluation follows common practice (Caron al. 2021 He et al.) 2020; (Singh et al. (021) in self-supervieed learning to evaluate the representation quality, where the pre-te backbone model is frozen and a new linear classifier is appended on top of We choose 12 popular datasets: ImageNet (Russakovsky et al., (2015), Food101 (Bossard al.
2014), CIFAR1O 10 (Krizhevsky et al. 2009). CIFARTOO (Krizhevsky et al. 2009). Cars (Krause al. 2013), Aireraft (Maji al.) 2013). DTD (Cimpoi al. 2014)) Pets (Parkhi et al.) (2012).
Flowers 102 (Nilsback & Zisserman, (0088), MNIST (LeCun & Cortes 2010), STLIO (Coates et al.
2011 and Country211 (Radford et al. 2021).
Multi-modal Understanding We consider three popular multi-modal tasks: VQAv2 (Goyal et al.) 2017b), SNLI-VE (Xie et al., (2119) and NLVR2 (Suhr et al., 2019) to evaluate our model's 's multi- modal understanding ability. For VQAv2, following ALBEF (Li al., 2021), the image and question are fed to the encoder and the decoder generates answers based on the multi-modal embeddings. For SNLI-VE, we follow SimVLN (Wang al. 202Tb) feed the image the encoder and the text to the decoder. A classifier is appended on top of our pre-trained model, and it is trained to predict the result based on the last hidden states of the decoder. For NLVR2, two input pairs are constructed, each of them including one image and the textual description. The prediction is made based on the concatenation of these two embeddings following VLM (Wang et al., 2021b). The resolutions for VQAv2, SNLI-VE, NLVR2 are 480, 384, 384, respectively.
Text-to- Text-to-Image Generation The texi-to-image task requires the model to understand the textual instruction first and then draw the image according to the input's 's intention. The input text is fed to encoder, and our decoder will generate visual tokens one by one. After obtaining visual tokens, they are decoded into a raw image by an image decoder. We directly use an off-the-shelf image decoder from VQGAN (Esser al. 2021). Following (Ramesh al. 2021) directly evaluate
20

---

Published as a conference paper at ICLR 2023
Data Type
Datasct
Image Domain
#lmages
#Captions
#Total
COCO
COCO
110.3K
551.7K
In-Domain Data (ID)
1.3M
Visual Genome
COCO
108.2K
759.0K
SBU
Web
859.7K
859.7K
Small-scale Web Data (SWD)
CC-3M
Web
2.9M
2.9M
14.9M
CC-12M
Web
11.1M
11.1M
VG regions
COCO
108.2K
3.6M
VG objects
COCO
108.2K
925.6K
COCO objects
COCO
110.3K
736.6K
Object-Regien Data (ORD)
Refcoco
COCO
27.9K
589.9K
17.0M
Open Image
Flickr
1.7M
7.5M
Obj365
Flickr
577.6K
3.6M
Vision Data (VD)
ImageNet-211
ImageNet
13.2M
13.2M
13.2M
DA DAVINCI-200M
Web
205.6M
205.6M
Large-scale Web Data (LWD)
601 .3M
LAION-400M
Web
395.7M
395.7M
Text Data (TD)
C4
Web
800GB
Table 10: Statistics of the pre-training datasets. #lmages, #Captions, and #Total denote the number of images, the number of image-text pairs, and the total numher of image-text pairs, respectively.
pre-trained model on 30, 000 images randomly sampled from COCO (Chen et al. 2013) validation split. Both Fréchet Inception Distance (FID) (Heusel al. 2017) and Inception Score (IS) (Salimans)
et al. 2016 are reported. The image resolution is 256.
Image-to- Text Generation For image-to-text generation (also called image captioning), the image is given to encoder and the decoder will generate the corresponding caption. Our experiments are conducted on COCO dataset (Chen et al. 2015) with cross-cntropy optimization. Other task-specific techniques such CIDEr optimization (Rennie et al. 2017) are not introduced. The image resolution is 480. We also conduct zero-shot captioning experiments on NoCaps (Agrawal et al. 2019) and VLUE (Zhou al. 2022b).
A.3 PRE-TRAINING DATASETS
Since existing studies pre-trained their models on different corpora, some of which are publicly available (e.g., CC-3M, CC-12M) while some are in-house datasets (e.g.. ALIGN (Jia et al. (021)), making the fair comparison difficult. Considering results only the state-of-the-ae thr-te- performance would underestimate the potential of this line of research. Therefore, we propose several practical settings, including small-scale and large-scale, and then conduct detailed comparisons on them in section 5.1
We collect large set of datasets with diverse distributions for pre-training. According its source, we divide them into in-domain, small-scale web data, object-region data, vision data, and large-scale web data. The statistics and details shown in Table 10 Most of them are naturally image-text pairs, while to enrich our corpus, we leverage object descriptions, region descriptions, and vision data (i.e.. ImageNet). For objects and regions, we crop them from the original image according their bounding box. The text part is composed according to a human- written template and objects. For example, the prompt template "This image contains [OBJ_A A] and [OBJ_ [OBJ_]]* where [OBJ [A] and [OBJ_B] are two object names from the data. For vision data, because they are usually labeled with single word or short phrase, we compose description with prompt templates such as "A picture of [LABEL]" or "The image contains [LABEL].. For example, "A picture of cat" or "The image contains cat". We curated dataset containing about 205.6M image-text pairs, which are available publicly the internet. The data distribution is similar to LAION-400M. Because both are from web images, we merge them into large-scale web data (LWD).
A.4 REPRODUCTION OF SIMVLM
Since VLM not open-sourced, we need to reproduce it by ourselves. There are two main difficulties in the reproduction: 1. it uses 1.8 billion in-house data 2. the configurations (e.g., parameter size, number of layers) of its base model are not clearly stated. However, there are still some clues in Section 4.4 of the Sim VLM paper, where they propose Sim VLM model with
21

---

Published as a conference paper at ICLR 2023
layers, 512 embedding dimensions, and trained about 200M web data. To demonstrate the success of our replication, we train a Sim SimVLMMmal model with the exact same configurations on about 200M web data. We obtain a VQA score of 68.50, surpassing the reported score of 67.43 in the original paper. We argue this result verifies our successful replication.
A.5 EFFECTS OF COMPUTE
Our model trained with large compute. To reveal the effects of compute, we visualize the performance improvement trends of SimVLM and DAVINCI There are two goals: 1) to compare better with prior work, as well as to 2) to show if that level of pre-training compute was necessary. We conduct experiments on the image-to-text generation task under both zero-shot and fine-tuning settings. The results are shown in Figure 2 is observed that with the increase in compute, both models improved significantly and converged at 40% of compute (zero-shot), and 80% of compute fine-tuning), respectively. Large compute is especially helpful for fine-tuning settings. After convergence, our model outperforms Sim VLM consistently in these two settings.
function of the compute spent.
(a) COCO Captioning (Zero-shot)
(b) COCO Captioning (Fine-tuning)
Usell
Vaoel
130
05
129
3300
00
134
100
100
Compute
Compute (4)
Figure 2: The effects of compute. X-axis is the percentage of compute and Y-axis is the CIDEr score on COCO captioning task.
A.6 EFFECTS OF MASKING STRATEGIES
In our experiments, we adopt dynamic masking, where the masking ratio sampled from uniform distribution U(0, 1). The prefix ratio could be 0, where the prefix image is none, and the model is forced to predict the whole image with the input caption. There are other designs to mask images. Here compared three different masking strategies: 1) masked image modeling (randomly masking some patches), 2) in- in-painting (randomly masking some continuous spans in the middle of the image), and 3) suffix-painting (ours). The results are shown in Table [1] Both masked image modeling and in-painting are effective and competitive. It is observed that suffi--panningg is better than masked image modeling and in in-painting across all tasks, demonstrating that suffix- suffix-paintiin painting works well.
Mathed
COCO BB44/C
VQA Ase
SNLL-VE Ace
NIVR2 Aze
ImageNet Ace
Food10l Aze
CIFARID
MNLJI
SSE22 Acc
Tear21mage
Acc
Acc
IS /ID
No Pre-training
32.1/9677
52.73
54.23
$1.08
66.32
79.84
MIM
113.4
68.18
75.34
69.66
48.46
56.95
72.79
81.72
89.84
9.50/ 74.13
ta-painting
34.5/ 112.5
67.46
75.41
68.66
47.50
54.38
71.20
31.55
89.84
9.97 68.15
Sufit painting
317.
69.25
76.22
72.55
73,82
81.76
90.25
12.35 53.14
Token Projection
17.7/49.2
$2.13
71.11
52.01
15.11
25.62
61.01
82.01
90.25
11.89 60.96
Projection
25.7
11.41/6 /611161..6
ResNat Festuuee (purs)
35.8/ 117.3
69.25
76.22
72.56
48.58
75.32
73.82
81.76
90.25
12.35 53.14
The effects of masking strategies and image feature extraction on COCO Captions, VQA, SNLI-VE, NLVR2, ImageNet, Food101, CIFARIO, MNLI, SST-2, and text-to-image generation. MIM denotes masked image modeling, where some patches are randomly sampled and masked. Because lincar probe and zero-shot text-to-image generation require a pre-trained model to frozen, the "No Pre-training" results on ImageNet, Food101, CIFAR 10, and Text2lmage are not reported and labeled by
Table 1I:
22

---

Published as a conference paper at ICLR 2023
A.7 EFFECTS OF IMAGE FEATURE EXTRACTION
There are several different ways to extract image features. We compare three different image representation methods: 1) token projection (projecting the prefix tokens the hidden dimension of the backbone network on the token-level), 2) patch projection (similar to ViT embedding, we split an image into fixed-size patches, embed each of them by a trainable linear projection and 3) ResNet feature extraction (ours). The comparison is shown in Table []] From the results, we observed that ResNet feature extraction outperforms token projection and patch projection by large margin. Therefore, we decided to adopt ResNet to extract image features.
the pixel-level),
A.8 SCALING EFFECTS OF DATA SIZE
In this section, we explore the scaling effects of our model. We plot the trends with the increase in data size on four tasks: COCO captioning, VQA, SNLI-VE, and NLVR2. The performance improvement shown in demonttaeesssssrttts that both Sim VLM and DAVINCI are scaling well with pre-training data size. In addition, DAVINCI consistently outperforms Sim VLM on different data sizes across these tasks.
(a) COCO Captioning
(b) VQA
SNLI-VE
(d) NLVR2
00/0
130
10200
1300
10.00
61720
1.50
$0700
Figure 3: The scaling effects of data size.
A.9 FULL COMPARISON EXISTING METHOSS
In Table [2] we display comprehensive comparison with state-of-the-art vision-language models on vision, language, and multi-modal downstream tasks.
A. LIMITATION AND SOCIETAL IMPACTS
Limitation. Like most of the previous pre-training studies, the entire project consumed 40 V100 GPU years on in-house computing cluster with large electricity costs. We tried keep our model size small enough, but there is still potential for efficiency improvements such as sparse training (Zhou al. 2021d c). dataset distillation (Zhou al. 2022c and progressive training (Rusu et 2016).
We will explore those techniques to improve the training efficiency and reduce the carbon footprint so that it can adhere to proposals on "green" deep learning (Schwartz al., 2020 Xu al. 2021b).
Furthermore, although we have tried our best to include as many tasks as we can to demonstrate the versatility of DAVINCI, we believe our method can be expanded more tasks (e.g., machine translation, summarization, object detection, etc.), modalities (c.g., video and speech). We leave these investigations to future work.
Potential Societal Impacts. Our model has image generation ability with risk of abuse, like fake portraits social media (Hill & White 2020). which is common potential risk in image generation research. Viable solutions are watermarking (Yu ct al. 202I) and introducing a strict user license.
A.11 VISUALIZATION OF IMAGE GENERATION
In this section, we conduct qualitative analysis by visualizing the generation samples. Figure shows the comparison with DALLE and OFA with the same query. More generated samples are shown in Figures 5
23

---

Published as a conference paper at ICLR 2023
Text MNLI
Vision
ImageZTest COCO
TextZimage COCO 1ST /FID
Mulli-madal
Model
PPaamm
ImageNet
VQA
NLVR2
Ace
LE/FT
B84/C
tost-dex fest-sul
dev text-P
Encoder-anly Malti-woodal Models
VisaaBEKT di
70.80/ 71.00
67.40/67.00
70.35
71.16
LXMEXL
240M
80.40
72/42/32 72.54
74,90/20 /4.50 74.50
OSCAR
155M
36.57 123.7
73.16 / 73.44
78.07 /
ViaVL
Zmsy Ec3l: 2021b
157M
38.2/ 129.3
75.95 770.
82.05 83.08
VILT
70.85
74.91 75.57
210M
75. 76.04
(Kang 07 m0
76
Encader-Dreader Multi Modele
UNICORN
Yaug et 2021
35.81 199.1
69. 69.20 40
---
Uni ENDN
11 P02600
110M
72.20 / 72.50
Pixc-BBERT
Hinee 1o 20020
144M
74/45 / 74455
76.50/ 71.30
E2G VLP
94M
36.27 117.3
73.25 73.67
17.25 77.96
14.5/116.5
71.70/
Teut2mmage Madels
AIaGAN
Au 201
23.30/35.20
DM-GAN
Ze 2000
32.20 1220/2244
DALLE
204
(250M)
12B
17.90 /7.50
(640M)
18.30/27.10
Uniiied
Unifying
Mommeeal
2021
228M
37.3/ 122.6
-/29.90
-/-
FLAVA
Smprre 21 min 2014
240M
80.33
73.54/
72.80/72.
SimVLM M
Wany
1921h
(640M)
153M
83.27
76.04 f-
38.57 126.7
75,04/1 /7.0775.08
78.82/7977
SimVi
M
(188)
83:40
30.60/
39.07 134.8
77.87 / 14
81.72/81.77
138.2
DAVINCI
(22.41))24421 (19.821)
76.32/76.4
Table 12: Comparison with state-of-the-art vision-language models vision, language, and multi-modal downstream tasks. All results from base-size models. LE and FT denote linear evaluation and fine tuning performance, respectively. Image2Texx results are reported without CIDER optimization. models." are the results after fine-tuning. SimVI.M (1.8B) and OFA are pre-trained or human-labeled data of many downstream tasks, and thus they are not comparable and are labeled in gray. win--i. H. 637M) and more
reproduced much larger corpus
Florence (Yuan et al. 2021) is pre-trained with much larger model size (Florence- CoSwin- pre-training data (900M), so the numbers in grey. bold denotes the best across unified models.
DALLE
OFA
Ours
large flower sillnng the vase the shelf
flowers
maat sitting on a piate with groen vegotables on the side
a fire hydrant sitting front yord naxt a sign
Figure 4: Comparison with DALLE and OFA text-to-image generation.
24

---

Published as a conference paper at ICLR 2023
purple f.ll
park sunny day
fire front
beach on sunny day
one carl train comingg down the ralioadd tracks
rod and white boat docked on shore
picture of mountain
STOP
W
building in froni a roundabout the
bathroom with marble walls and counter surrounds mirror
many the plate on
bunch shop
set sandlwich
noodles and broccol on a plate
Figure 5: Generation samples by DAVINCI.
25


Reviews:
[{"review_id": "_8uRdci6Wf", "reviewer": "Reviewer_wium", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a vision-and-language transformer that is multi-tasked to both generate an image completion, given language (\"Prefix Image Modeling\") and to both generate language given images (\"Prefix Language Modeling\"). The key claim of this paper is that by multitasking these two settings, the resulting model is able to do well on a wider range of vision-and-language tasks.\n\nDespite not being a very large model, (it's BERT-Base sized), it performs favorably to SimVLM and FLAVA (two recent very large-scale approaches) in a finetuned setting, on both language-only and vision-language tasks. An ablation study (Table 5) suggests that these objectives are complementary.", "summary_of_the_review": "I'm leaning positive on this paper but I'd like some of my concerns to be addressed in the author response first :)\n\n---\n\nUpdate post-review: I think the authors addressed my main concerns, so I'm bumping up my score to 8.", "strength_and_weaknesses": "Strengths:\n* The proposed model does well while being quite simple, which in turn suggests it has potential to be widely adopted (especially considering how many transformer implementations are already very well-optimized). The paper is easy to read as well.\n* I appreciate the comparison with a variety of past models covering NLP, Image Classification, and vision-and-language. \n\nWeaknesses:\n* I'd like to see a comparison with this paper if possible CM3 A Causal Masked Multimodal Model of the Internet https://arxiv.org/abs/2201.07520 - because it (like this proposed approach) uses a Discrete VAE token-based approach.\n* Most of the tasks specified in the vision-and-language space seem focused on literal recognition of what's in images. I'd be curious as to how this model stacks up on Visual Commonsense Reasoning (visualcommonsense.com) for instance.\n\nPerhaps less important but still important: I'm a fan of this paper overall, but I do wonder why it takes 10 days to train on 1024 V100 GPU's. That's quite a lot for a BERT-Base model -- and in turn, it makes me think that one of the main reasons this paper does so well is because it was just trained with more effective compute than FLAVA or SimVLM. That doesn't mean that this paper is bad by any means, but I would be happy to increase my score if there was some experiment showing e.g. the performance of DaVinci over the course of pretraining -- e.g. as a function of the compute spent, both a) to compare better with prior work, as well as to b) show if that level of pretraining compute was necessary.\n\nAnother related question relates to the ablations in Table 5. For PLM only vs. PIM only, vs. the full model -- are the ablations only trained with half the effective compute as the full model? (as they're trained on half as many sequences) I'd be curious as to how they do given double the pretraining compute.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YZnSoFawKTZ", "reviewer": "Reviewer_9zCh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, a generative vision-language model that can perform both image-to-text generation and painting text-to-image generation, called DAVINCI, is proposed. In DAVINCI, prefix language modeling and prefix image modeling are adopted as the training objective, which make the model simple and scalable. On 27 generation/understanding tasks, DAVINCI achieves highly competitive results.", "summary_of_the_review": "This work is overall a good contribution to the research community. I would recommend acceptance of this paper.", "strength_and_weaknesses": "Strengths:\n\n1.A simple and unified foundation model for vision-language pre-training is proposed.\n2.Compared with previous unified vision-language foundation models and other vision-language models, the proposed model shows highly competitive results.\n3.Various ablation studies have confirmed the advantages of the proposed model.\n\nWeaknesses:\n\n1.The authors should conduct more rigorous experiments and comparisons. For example, in Tab. 3, the authors only listed the results of SimVLM (1.8B), OFA and Florence, and claimed that these results should not be directly compared. However, to clearly demonstrate the effectiveness of the proposed method, the authors could change the training settings (e.g., train on the same data of OFA or use a model with the same size as Florence) to make fair comparisons with these models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IYeOrK_XXF", "reviewer": "Reviewer_qYLj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new unified model named Davinci, which is trained with prefix language modeling and prefix image modeling. Davinci is able to do images and text generation as well as other vision and language understanding tasks. The authors carefully benchmark the performance of different vision and language pre-training objectives on different scales of pre-training datasets and achieve competitive performance. ", "summary_of_the_review": "This paper proposed a new unified model named Davinci, which is trained with prefix language modeling and prefix image modeling. The proposed method is simple and shows some potential for training a unified model. However, there is multiple weakness in the paper that require the authors furhur clarify. (check the weakness section for details). ", "strength_and_weaknesses": "[Strength]\n\n- The proposed method combines prefix language modeling with prefix image modeling, which is a natural and simple extension. \n\n-  Experiments ablating different objectives, as well as different dataset sources and showing the effectiveness of the different types of pre-training data. \n\n- Similar to OFA, the proposed method can do text-to-image generation tasks as well as vision and language understanding tasks. \n\n[Weakness]\n\n- From the introduction, the proposed method is motivated by auto-regressive models like GPT3 and PaLM. GPT3 has few shot abilities, and PaLM can multi-task and even do zero-shot tasks. One key property of these two models is they can use the same set of parameters for many tasks. However, the proposed method still falls into the first pretrain and then finetune formula. \n\n- The authors claim that VL-T5 and OFA are hard to scale up since it is non-trivial to collect a large number of vision and language datasets for pre-training. I hold a different opinion, the pre-training tasks (mask language modeling and mask image modeling) can be one of the tasks, and thus the model still benefits from a large training corpus. \n\n- Another big concern of the proposed method is the model size. The paper only shows models with 152M parameters. Thus it is not clear whether the proposed method can scale to a large model. \n\n- The pre-training datasets also contain supervised datasets such as COCO, refer coco, VG, etc. Given the claim of OFA and VL-T5, the proposed method suffers a similar scaling problem. \n\n- In Table 3, the author didn't compare OFA and simVLM (1.8B). Although OFA uses some additional tasks, its pre-training corpus is substantially smaller compared to the proposed approach. \n\n- for prefix language modeling, the author randomly samples the mask ratio,  is it from (0, 1)? or some other span? Table 5 shows the results of the ablation study. I am more interested in the effect of image masking strategies: such as random masking vs. masking at the end. Different ratios of masking etc. However, there is no experiment on that. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2Y4jeQD4pYA", "reviewer": "Reviewer_NJt6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new way to pre-train VLMs (vision-language-models) using both text generation and image generation as the VLPs (pre-training objectives).\n\nThey then ablate the objectives on their model (and show that each objective’s inductive bias is actually helping) and the datasets used for training (smaller more curated datasets being better in general) both their model and a reimplementation of SimVLM.\n", "summary_of_the_review": "Leaning towards acceptance, since this seems a good enough empirical contribution and a “proof of existence” of image generation working as a pre-training objective to make the model stronger.\n", "strength_and_weaknesses": "Strengths:\n - All required ablations of the VLPs\n - All required ablations of the dataset sources\n - Comparisons with appropriate re-implementations of baselines.\n - Comparisons are over a varied class of task suites (NLU, Image Understanding, Multimodal Understanding, Image Generation, NL Generation) and actual benchmarks on 27 tasks.\n\nWeaknesses:\n - Not necessarily a weakness: Since the “prefix” of the image is more seen than the suffix, I wonder if this leads to an inductive bias that the image will be less likely to paint the prefix of the image itself. i.e. say if the most important aspect of the image falls in the top-left block of the image. I wonder if the authors considered other ways of masking the image, by either just in-painting (i.e. “missing tiles”) instead of suffix-painting. This could be left for future work.\n - Also not necessarily a weakness: Right now images are treated in a hybrid manner, image input to the model is treated in pixel space (first three blocks of resnet extracts the feature maps), however the suffix that needs to be generated is in the token space (VQGAN tokens). One could have resorted to also taking the whole image as a sequence of VQGAN tokens, projecting a prefix of them to the hidden dimension of the backbone network and that prefix would be the input, while the rest of the suffix became the output. This could be left for future work.\n - Re: SimVLM base size, did the authors contact the SimVLM authors to get the exact specifics of the SimVLM base size and architecture. Although SimVLM_small seems to be replicated exactly, it is also possible that some bugs remain in the reimplementation and SMALL was small enough for these not to matter.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster