CoBIT: A Contrastive Bi-directional Image-Text Generation Model
Authors: Haoxuan You
Mandy Guo
Zhecan Wang
Kai-Wei Chang
Jason Michael Baldridge
Jiahui Yu
Venue: ICLR 2024 poster (2024)
Keywords: Contrastive Pre-training
Vision-Language
Text-to-Image Generation
Auto-regressive Model.
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A contrastive bi-directional image-text generation model that unify contrastive learning, image-to-text generation and text-to-image generation.

Abstract: The field of Vision-and-Language (VL) has witnessed a proliferation of pretrained foundation models. Current techniques typically employ only one type of training objective, whether it's (1) contrastive objectives (like CLIP), (2) image-to-text generative objectives (like PaLI), or (3) text-to-image generative objectives (like Parti). However, all these three objectives are mutually relevant and are all based on image-text pairs. Intuitively, the first two objectives can be considered as complementary projections between two modalities, and contrastive learning can preserve global alignment and generations facilitate fine-grained understanding. Inspired by this, we present a Contrastive Bi-directional Image-Text generation model (CoBIT) to first time unify the three pre-training objectives in one framework. Specifically, CoBIT employs a novel unicoder-decoder structure consisting of an image unicoder, a text unicoder, and a cross-modal decoder. The image/text unicoders can switch between encoding and decoding in different tasks, enabling flexibility and shared knowledge that benefits both image-to-text and text-to-image generations. CoBIT achieves superior performance in image understanding, image-text understanding (Retrieval, Captioning, VQA, SNLI-VE), and text-based content creation, particularly in zero-shot scenarios.

Published as a conference paper at ICLR 2024
COBIT: A CONTRASTIVE BI-DIRECTIONAL IMAGE- TEXT GENERATION MODEL
Haoxuan You', Mandy Guo?, Zhecan Wang', Kai-Wei Chang Jason Jason Baldridge', Jiahui Yu² 'Columbia University, Gooole Research, 'UCLA hacxuan, youncs. columbia a edu, (xyguo, jasonbaldridge, jiahuiyu]2googlee
Zero-shot Bi-directional Generation (T21 and 12T) Results:
Inpué Ailvering leturs dbor ouutioaiinn ratest enycking dour
Aspur A/Ccoon wwsering of ctait
Generaled: namronse
(iput An allos
Gonerated asicous under the ona
Image
cchopu reading newegaaee
1840 pholograph
Text
CoBIT
Gererald
Multinadal Resr
Text Image
CoBIT
&
Multinddal Understanding
Image
Understanding
Captioning
Figure 1: CoBIT can address a variety of vision and vision-language tasks in zero-shot and fine- tuning settings. The right-side displays the zero-shot generated images by CoBIT given novel prompts, and the zero-shot generated captions by CoBIT given the previously generated images as input.
ABSTRACT
The field of Vision-and-Languuae (VL.) has witnessed proliferation of pretrained foundation models. Current techniques typically employ only one type train- ing objective, whether it's (I) contrastive objectives (like CLIP), (2) image-to- text generative objectives (like PaLI), or (3) text-to-image generative objectives (like Parti). However, all these three objectives based on image-text pairs. Intuitively, the first two objectives can be considered as complementary projections between two modalities, and contrastive learning can preserve global alignment and generations facilitate fine-grained understanding. Inspired by this, we present a Contrastive Bi-directional Image-Text generation model (CoBIT) to first time unify the three pre-training objectives in one frame- work. Specifically, CoBIT employs a novel unicoder-decoder structure consist- ing of image unicoder, text unicoder, and a cross-modal decoder. The im- age/text unicoders can switch between encoding and decoding in different tasks, enabling flexibility and shared knowledge that benefits both image-to-tex, and text-to-image generations. CoBIT achieves superior performance in image un- derstanding, image-text understanding (Retrieval, Captioning, VQA, SNLI-VE), and text- content creation, particularly in zero-shot scenarios.
mutually relevant and are all
INTRODUCTION
Recently, there has been rising interest in developing multimodal foundation models for vision-
language tasks. By mapping text and image representation in the same space. the models can (1) generate images from text (Ramesh et al., 2021 Yu et al. 2022b; Chang al. 2022 2023), , (2) generate captions from images (Wang et al., 2022a Chen et al. 2022 Wang et al. 2021: Alayrac
et al. 2022), and (3) retrieve images from text and vice verse (Radford al. 2021 Yao et al. 2021 Mu et al. 2022) You et af.) (022). Although these tasks are highly relevant and can be operationalized on the same set image-text pairs. They are often considered separately, and the corresponding foundation models are trained with different pre-training losses designed for the corresponding task.
This work done when Haoxuan was intern Google.

---

Published as a conference paper at ICLR 2024
Specifically, there three pre- pre-training losses that are widely used in the literature: (1) contrastive objectives, (2) image-to-text generative objectives, and (3) text-to-image generative objectives. Most models are trained with only one of these objectives, while some are trained with two. For example, CoCa Yu et al. 2022a) combines contrastive learning and image-to-text generation. OFA (Wang al. 2022b) and UnifiedOO (Lu al. 2022) integret image-to-text and text-to-image generation. However, none of the approaches has considered using all these three losses although they are highly relevant and can be trained with the same set of image-text pairs.
Intuitively, these pre-training objectives complement each other. Specifically, contrastive learning drives high-level image-text matching, whereas image/text generation encourages the model to learn fine-grained image and text representations. Therefore, it is intuitive to utilize them in the same framework. is worth noting that these three pre-training losses can share part of the computational graphs. Therefore, optimizing them jointly does not increase much overhead compared to only optimizing one.
In this paper, we propose to unify the three commonly used pre-training VL objectives: cross-modal contrastive learning, image-to-text generation, and text-to-image generation, and consolidate their strengths in one framework. Our key innovation is a simple and unified Contrastive Bi-directional Image-Text Text generation model (CoBIT), which consists of an image unicoder and a text unicoder, well as cross-attention decoder. The proposed image/text unicoder uses the Transformer chitecture. It alternates between two modes: unimodal image/text encoding and decoding depend- ing the pre-training tasks. Importantly, the same set of Transformer parameters both encoding and decoding, with only the input embedding and attention masks differing, As shown in Fig. 2 when optimizing contrastive objective, image unicoder, and text unicoder work as two encoders. When optimizing text/image generation loss, image/text unicoder extracts fea- tures in encoding mode, and the text/image unicoder works in autoregressive decoding mode, then the cross-attention decoder will let autoregressive text/image features cross-attend to encoded im- age/text feature, serving as fuser and generator. Each unicoder efficiently shares the knowledge between encoding and decoding and, therefore, can jointly improve both T21 and I2T generation without increasing the number of parameters, exhibiting excellent parameter efficiency. In such a way, all three pre-training paradigms are unified in our framework.
used for
Our extensive experiments demonstrate CoBIT's superior performance, and more importantly, first time verifies the compatibility of the three objectives. Benefiting from the compatible objectives, CoBIT subsumes strong zero-shot and transferable capacities of unimodal visual understanding, image-text matching, image-text understanding, and text-to-image generation. For example, CoBIT achieves 82.7% 7% accuracy in zero-shot ImageNet classification, 9.37 FID in zero-shot text-to-image generation, 44.8 CIDEr score in zero-shot image-to-tar to-text captioning. After fine-tuning, CoBIT fur- ther achieves 86.44% linear probing accuracy on ImageNet, 4.62 FID on text-to-image generation, and 78.3 VQA score.
RELATED WORK
Learning Visual Representation from Text, Recent works studied pre-training a visual backbone supervised by paired text data and produced transferable visual representations. CLIP (Radford ct al. 202T) and ALIGN (Jia et al.] 2021) are prominent examples of global contrasting between image-fext pairs. Florence (Yuan al. (02T), BASIC (Pham al. 202T), and LiT (Zhai et al. 20226) further scale both datasets and models. FILIP (Yao et al.) 2021) proposes to employ local token features from images and text for fine-grained contrastive learning. MS-CLIP (You et al.,
2022 and CLIPPO (Tschannen et al. 2022) study sharing the model parameters between vision and text.
VL. Pre-training. Another line of research focuses on learning solid joint multimodal embedding through pre-training. Some pre-train with mask-reconstruution loss (Li et al., 2019 Wang et al. 2022c Li et al., 2022 Chen et al. 2019: Shen et al. 2021 Li et al. 2021 i.e., mask partial image
and text tokens in input and require the model to predict the masked tokens. Others pre- train models
by generating text autoregressively (Wang et al. 2021 Chen et al. 2022 Wang et al. 2022a Alayrac
et al. 2022). Both perform strongly in downstream VL understanding tasks, such as VQA Antol
al. 2015 and captioning.
2

---

Published as a conference paper at ICLR 2024
CoBIT:
Img Generation loss
Text Generation loss
13
B77
Taa
degs
MT-VOGAN
Cross-Modal Decoder
Image Detokenizer
Image Tokenizor
Image Unicoder
Contrastive
loss
Text Unicoder
10c
doga
Lug
Riild
Image
Tiai Tokaans
(4) Pre-truining Pipeline of CoBIT
Text Generation loss
Image Generation loss
Contrastive
loss
Cross Mossa- Decoder
Modal Decoder
Cross-Altentien
Image Unicoder [Encoding Moda)
Text Unicoder (Encoding Modiu)
Cross Attention
Image Unicoder (lincoding Modn)
Text Unicoder
Image Unicoder (Decoding Modo)
Text Unicoder
(Decoding Mode)
(Encoding Modie)
Tob
Tixt Tolons
Downstrsam: Retneval;
Tokeee
Tolette
Downstream:: Image Captioning:
Understanding.
Downstreen earr. Text-lo-lmage Creetion
LanginnL Linderslanding
(b) Contrastive Objecthre
Onjective
(d) Objective
Figure 2: (a): Overview of CoBIT pre-training pipeline; (b): When optimizing contrastive objective, image unicoder and text unicoder work as two encoders; (c) and (d): When optimizing image/text generation loss, text/image unicoder extracts features in encoding mode and image/text unicoder works in autoregressive decoding mode, then the cross-attention decoder will let autoregressive image/text features cross-attend to encoded text/image feature.
Text-to-Imase Image Generation. Text-guided image creation is challenging problem that has attracted intense interest in the past two years. Two families of methods are widely studied: diffusion-baal based and token-based. Diffusion-based models (Rombach et al. 2022 Saharia et al. 2022 Ramesh al.) 2022) are based on a process that iteratively adds noise to images and then learns to reverse the nois- ing process while conditioning raw images are quantized into image tokens by an image tokenizer; then, given a text input, Trans- former models predict image tokens autoregressively like machine translation (Ramesh al. 2021
textual descriptions of the image. With token- methods,
Yu et al. 2022bb or by iteratively predicting image tokens in parallel[Chang et al. [022 2023).
As these three broad lines of research have demonstrated great transferable ability to various down- stream tasks, there have been many efforts to unify some of them (Yu et al. 2202aa] Wang et al.,
20226 Lu al. 2022; Zhang al., 2021a Kim et al., 2022] Huang et al. [2021] Our work, CoBIT, serves as the first effort to integrate contrastive loss, image-to to-text generation, and text-to-image loss under one unified pre-tree gre-training framework.
COBIT
We begin with describing the input processing and then present the model architecture, which in- cludes proposed unicoder module that shares the merit of both unimodal encoding and decoding. Finally, we explain the pre-training of CoBIT and discuss a comparison with other unified models.
3.1 INPUT
To cover various tasks, CoBIT supports three inputs: text tokens, discrete image tokens, and raw images.
Text Tokens. Following the default process in past works (Raffel et al. 2020 Jia al. 2021 Yu al. 2022a), we tokenize text inputs using SentencePiece model with 64k vocabulary trained on the sampled pre-training datasets. The maximum text token length is 64.
Discrete Image Tokens. CoBIT generates images in autoregressive manner, which requires tokenizing 2D images into a sequence of image tokens (Ramesh et al. [202T] Ding et al. 2021 2022
Gafni et al.] 2022: Yu et al. 2022b). Following Parti (Yu et al. 2022b). employ pre- trained and frozen VIT VOGAN (Yu et al. 2021) as the tokenizer. Specifically, each 256 256 image is tokenized into 32x32 grid of image tokens, with 8192 image token classes in the codebook.
3

---

Published as a conference paper at ICLR 2024
Image Unicoder
Text Unicoder
Cross- modal Decoder
Model
Total Params
Layers
Dims
Layers
Dims
Layers
Dims
CoBIT-Base CoBIT-Large
12
768
12
768
18
1024
626M 1082M
20
1024
12
1024
30
1024
Table I: Size variants of CoBIT.
We append the codebook to the text vocabulary as additional tokens. In inference, to generate images, we decode the image tokens one-by-one and feed them into the decoder in ViT-VQGAN to reconstruct the raw image.
Raw Image. For image and image-text understanding tasks, input raw images, and each image divided into non-overlapped patches following the de facto process in ViTs. In default, unless specified, the image resolution is 288x288, and the patch size is 18x18.
3.2 ARCHITECTURE
As shown in Fig. 2 CoBIT comprises one image unicoder, one text unicoder, and cross-attention decoder. We term them unicoders because they can act as either encoders or decoders, depending on the role they play for each task. The incorporation of text/image unicoder is inspired by [Dong it al. (2019);Bao et (2020); /hoo et al. (2020). which demonstrated that one Transformer model can perform both bidirectional encoding for understanding tasks and autoregressive decoding for generation tasks. In our scenario, compared with plain image/text encoders, unicoders in decoding mode can take advantage of the common knowledge shared with encoding to produce unimodal autoregressive features decent prior for cross-modal generative objective. Experimental ablation also validates that unicoders boost both T2I generation and multimodal understanding.
Image Unicoder. Recently, Vision Transformers (ViT) (Dosovitskiy et al.) 2020; Touvron et al. 2021 Liu et al., 202I) has been established as the strongest approach for image feature encoding. As decoders, Transformers are used in autoregressive image token generation (Ramesh et al. 1021
Gafni et al. 2022; Yu et al. 20226). We combine these two functionalities into a single image unicoder. The image unicoder has two working modes: (1) In the encoding mode, following ViT, each 2D patch in the raw image is projected into feature vector by trainable linear projection layer. Then, the projected feature sequence is input into cascaded Transformer layers obtaín the encoded (2) In the decoding mode, firstly, the input processing is different. As described in Sec. [.I]] we tokenize the raw image into image tokens and initialize an embedding layer where token embeddings are indexed. Then, the same Transformer layers in encoding mode are reused decoding mode to process the features; however, to guarantee the causal decoding ability, we use causal conv-shaped attention mask (Ramesh et al. [022]] Yu et al.) 2022b] Child et al. 2019) instead. Overall, the two modes share the Transformer layers' parameters, and only differ in input processing and attention masks. We assume that, compared with the design of plain image encoders as in previous works Yu et al. 2022a Wang et al.) 2022a), the additional decoding mode exploit the common knowledge learned in image encoding to generate image autoregressive features, which we hypothesize should boost the feext-to-jimage generation capacity.
image features, where the attention mask is bi bi-directional.
Text Unicoder. Similar the image unieoder mentioned above, the text unicoder also has both encoding and decoding modes, which reuse the Transformer parameters. In both modes, the same tokenizer and embedding layer are utilized to obtain token features, given that they share the same input formats. A causal attention mask is applied in decoding mode. During encoding of text, there are two options in previous works: bi-directional mask (Devlin et al. 2018 Raffel
al., 2020 Yu et al. 2022b), or causal mask (Brown et al., 20200) Radford et al.) 2021 Yao et al.) 22021). We empirically found that two masks make no difference in performance and use causal masking as the default in the reported experiments.
Cross-modal Decoder The cross-modal decoder performs as a dusion-and-genedee which structure-wise follows the cross-attention decoder (Vaswani et al.] 2017; Yu et al. 2022a). When generating text, the input is the text autoregressive feature from the text unicoder decoding mode; encoded image features will be treated as cross-altention information, i.e., key and value in cross-attention layers. When generating the image, symmetrically, the image token autoregressive feature from the image unicoder in decoding mode is input and cross-attends to encoded text fea- tures. Also, different from text generation, where the plain causal (autoregressive) mask is used in the cross-modal modal decoder, image generation employs a conv-shaped masked sparse attention (Rameshl
module,
4

---

Published as a conference paper at ICLR 2024
et al. 2021 Yu et al. 2022b; Child et al. 2019). which can memory and computation brought
by long sequences of image tokens.
3.3 PRE-TRAINING
The pre-training of CoBIT subsumes three fundamental objectives: image-text contrastive loss, 12T generation loss and T2I generation loss. Here, we provide details on the losses and also clarify the scaling and initialization strategy.
Contrastive Loss. We input raw image and text into the image unicoder and the text unicoder, respectively (both in encoding mode), to get encoded image and text features. For text, as with CLIP (Radford et al. 2019) and ALIGN (Jia et al. 2021), we take the feature vector of the CLS token appended at the end of the input sequence as the global representation. For images, however, the unicoder outputs a sequence of features. To aggregate them, following (Yu et al. (022a), we apply an attention pooler, which is single multi-head attention layer with one learnable query and unicoder output features as key and value. After obtaining two global features of image and text, a contrastive loss is applied optimize the paired image-text against others in the same batch:
Lcin Σ log
1
explx 9./T) 1 exp(=? Ws/T)
Σ log Σ exp(y! mz/T)
exply x./T)
where r; and y, denote the normalized global embeddings of i-th image and j-th text. is learnable temperature for adjusting the scale of the loss.
12T and T21 Generation Loss. We formulate two generation tasks as token generation problems. As shown in Fig. 2 by cascading the image unicoder, text unicoder, and cross-modal decoder, can perform two tasks seamlessly by only switching the working modes of unicoders. A cross- entropy loss is applied on top of the cross-modal decoder maximize the conditional likelihood of the ground-truth token under the forward autoregressive factorization.
log 1), Lra LITT Σ
Σ log logPo(ra
(2)
where y and x denote text and image tokens respectively.
Classifier-Free Guidance for T2I. Following Yu et al. (202bb); Chang et al. 2023): Ramesh et al. (202T). we employ classifier-free guidance (CFG) (Ho & Salimans) 2022) in text-to-image generation. To be more specific, training, we randomly mask conditioning vectors, i.e., input text tokens, by certain possibility (10% in implementation). In inference,
compute two
predictions: conditional one I'z and unconditional one I(2), which only differ text input: conditional prediction I(2, T) has original text tokens as input while the input text of unconditional prediction [(2) is fully masked. Then we linearly interpolate the I (22, c) and I (2) to obtain the final generated image:
I= [(2) + (([[2,T)) - I(x)),
(3)
where is hyperparameter to adjust the scale of classifier-free guidance, and we set a=2.0 in default.
Final Loss. In the end, we add those three losses up to optimize the model end-to-end.
LCaam = Acon LCon + Air + AraCru
(4)
where Acon. Azr, ÅrZ denote corresponding scalar coefficients for contrastive, I2T and T2I loss. In default, we set Xra Azr Aco 0.2 0.1.
Scaling, As shown in Tab we start from CoBIT-Base and scale it up, w.r.t. both number of layers and model dimension, to obtain CoBIT-Large with around IB parameters.
Initialization. In previous text-to-image generation models Yu et al. 2022b; Rombach et al. 2022 Saharia al. 2022), the text feature extractor is usually initialized by pre-trained text model. Correspondinglyy in CoBIT, we also initialize the text unicoder with another pre-trained text uni- modal decoder from CoCa (Yu al. (022a)), while leaving the image unicoder and cross-modal decoder trained from scratch. In Sec. 4.4 we compare it with training all from scratch and find the initialization indeed helps.
5

---

Published as a conference paper at ICLR 2024
3.4
COMPARISON WITH OTHER UNIFIED WORKS
For clearer comparison with other unified works, we provide detailed explanation of CoBIT R.S. recent unified diffusion-based and auto-regressive models, such as Versatile Diffusion (Xu et al.)
2022).
CoDi (Tang al., 2023). Hu et al. (2022), UniDiffuser (Bao et al., 2023), Unified-IO LLu
ct al. 2022). OFA (Wang et al. 2022b), BETT-3 (Wang et al. 2022c) and CM3Lcon (Yu et al. 2023)
in Appendix 6.5
4 EXPERIMENTS
In this section, we first describe the pre- pre-training details (Sec.
4.2 and Sec. introduce the primary results of zero-shot and fine-tuning evaluation, respectively. Both eval- uations examine three capacities: (1) visual understanding, (2) image captioning and multimodal
4.1 Following, Sec.
understanding, (3) text-to-image content creation. Lastly, Sec. 4.4 brings ablation.
4.1 PRE-TRAINING DETAILS
Data. CoBIT is designed to be pre-trained with image-text data. For contrastive loss and 12T loss, we use a mixture of ALIGN dataset (Jia et al., 202I), and JFT-4B dataset (Zhai et al. 2022a) where category names are transformed into texts by prompts in Pham et al. (202I). Differently, for T2I generation, we found that the short text in JFT-4B is less informative for generating the image extensive descriptions of visual details are important. Instead, replace JFT with WebLI dataset (Chen et al. 20222) ) and mix it with ALIGN for T2I generation loss. We further perform de- duplication, as in Jia et al. (202T); Zhai et al. (2022b), to remove the examples close downstream tasks. In the end, we obtain 1.IB pairs from ALIGN dataset, 162M pairs from WebL. dataset, and 4B pairs from JFT-4B dataset.
Optimization. CoBIT is implemented using Pax (Team 2023), Jax-based framework. Within each batch, for optimizing T2I loss, we sample 1,024 image-text pairs from a mixture of ALIGN and WebLI datasets, and for optimizing contrastive and 12T losses, we sample 30,720 image-text pairs from a mixture of ALIGN and JFT datasets. In total, the batch size is 31.744. We use the Adafactor (Shazeer & Stern] 2018) optimizer with 81 = 0.9, 82 = 0.96 and weight decay of 0.045. As for the learning rate schedule, we warm it up to 4.5e-5 in the first 5,000 steps and then use an exponential decay starting from the step of 85,000. In total, models are pre-trained for 1M steps and takes around 12 days on 256/5 12 CloudTPUv4 chips. Then, following Radford et al. (2021): Jia et al. (2021). we further pre-train models for 50k steps with 576x576 high-resolution raw images as input in image encoding. The image input to ViT-VQGAN, iee. image for decoding, is kept at 256x256 resolution.
BeSDT--ase/CoBI-LLate
4.2 ZERO-SHOT EVALUATION ON DOWNSTREAM TASKS
We evaluate CoBIT on representative tasks, and compare CoBIT against CLIP (Radford et al,,
2021I. ALIGN (Jia al.) 2021). FILIP Yao et al. 2021). Florence (Yuan et al. 202 YYu al.
2022a). ZeroCap (Tewel et al. 2021), D. SiM (Wang el al. 2021), VLKD (Dai et al. 2022). Parti
Yu al.] 2022b), LDM (Stable Diffusion)-1.445 (Rombach et al., 2022). Flamingo-BB (Alayrac
et al.
2022). DALL-E 2 (Ramesh et al. 2022), Versatile Diffusion Xu et al. 2022 CoDi (Tang
al.
2023) and CM3Leon (Yu et al. | (002))
Zero-shot Image Classificattion. We follow the standard evaluation protocols as in CLIP, ALIGN, etc (details in Appendix (6.4.2p.
shown in Tab. 2 compared models with similar scales, in ImageNet (Russakovsky et al. 2015). CoBIT-Large can achieve 82.7%, outperforming strong baselines such as CLIP and ALIGN. practice, we find batch size difference also profoundly affects models' performance, which may contribute to the 2% discrepancy between the performance of CoBIT-Large and CoCa-Large as CoCa's 's batch size is 64k while ours is appendixonly.
Zero-shot Image-Text Retrieval. The image and text feature extraction process is the same as zero-shot image classification. Flick (Plummer et al. 2015) and MS-COCO (Lin et al. 2014) are used for evaluation. In Tab. 2 within comparable scales, CoBIT-Large
outperform the previous best model CoCa-Large in 5 out of 8 metrics and is ranked the second best in another two metrics.
Zero-shot Image Captioning. Since CoBIT is already pre-trained with image-to-ten, to-text generation loss on noisy image-text data, we directly evaluate on zero-shot image captioning. As in Tab. 2 in can achieve 43. 33.0/44.8 CIDEr score, surpassing SimVLM by
MS-COCO, CoBIT-Base/CoBrT

---

Published as a conference paper at ICLR 2024
Table 2: Zeroshot Evaluation of CoBIT against previous image-text models. The models in gray
background have > 1B parameters while others in white background have 1B parameters. For models with S IB parameters, we highlight the best score in bold+underline and the second-best score in underline. Understd. is the abbreviation of Understanding.
Image Understd.
Image-Text Text Understd.
Content Creation
Model
ImageNet Classification
Flickr Retrieval
MS-COCO Retrieval
MS- AS-COCO
MS-COCO
Captioning
T21 Generation
Image- TTxx
Tmxe-
Image-TText Text-
Acc(SE)
R@1
R@5
R@ 1
R@s
R@l
R@s
R@1
R@5
CIDEr
FID (L)
CLIP
76.2
98.7
68.7
90.6
37.8
62.4
ALIGN
76.4
88.6
98.7
75.7
93.8
58.6
83.0
45.6
69.8
FILIP
78.3
89.8
99.2
75.0
93.4
61.3
84.3
45,9
70.6
Florence
83.7
90.9
99.1
76.7
93.6
64.7
85.9
47.2
71.4
CaCa-Large
84.8
91.4
99.2
79.0
95.1
65.4
85.6
50.1
73.8
ZeroCap
SimVLM
32.2
VLKD
58.3
Parti-350M
14.10
Parti-750M
10.71
LDM (SD)-14B
12.63
Coca-2B
86.3
92.5
99.5
80.4
95.7
66.3
86.2
51.2
74.2
Make-A-Scema
11.84
Versatile Diffusion
11.10
CoDi
11.26
DALL-E
10.39
CM3Lcon-7B
10.82
Parti-20B
7.23
CoBIT-BAA Basc
79.4
89.5
98.4
76.5
94.3
62.1
83.5
47.3
72.3
43.0
10.35
CoBrT-L.rre
82.7
91.5
99.1
79.9
95.3
65.1
85.5
50.3
74.2
44.8
9.37
10.8/12.6. It's noted that the models with 1, egg., Flamingo, VLKD, have much higher scores than others because they reuse pre-trained large language model as decoder that inherits strong text generation ability.
Zero-shot Imaae Generation. We follow the standard evaluation process as in Parti and DALL-E (detail Appendix 6..43). As we can see in T CoBITs can beat specialized models with comparable scales, and CoBIT-Large can achieve impressive FID of 9.37 which outperforms some models with larger scales by a significant margin, e.g., DALL-E 2/Make-A -A-ecee with 3.5B/4B parameters.
4.3 FINE-TUNING ON DOWNSTREAM TASKS
To demonstrate the transfcrability of CoBIT, we further conduct linear probing or fine-tuning on multiple downstream tasks. Besides the existing methods we mentioned in the previous section,
also compare UNITER (Chen et al. 2019). VinVL (Zhang et al. 2021b), CLIP-ViL (Shen et al.
2021), OFA (Wang al. (2022b). X- X-LMMERT (Huang al. 2021) and PALI (Chen et al. 2022)
The detailed hyperparameters of these tasks are shown in Tab.
Linear Probing on ImageNet. Following CLIP, we linear probe CoBIT by fixing parameters of the image unicoder and only training a linear classifier on top for image recognition. CoBIT-Large can outperform CLIP and ALIGN by around 1%.
Image-Texx Text Understanding. We categorize VQA (Antol et al. 2015), SNLI-VE (Xie et al. 2019) and image captioning into tasks requires image-text understanding. We fine-tune all parameters of CoBIT and evaluate it on the val/test set.
Captioning. In fine-tuning, CoBIT computes caption predictions in the same way as zero-shot image captioning in Sec. 4.2 In Tab. 3 we can the CoBIT can achieve a competitive CIDEr score against other models. It's noted that some works (Wang et al., (022b) additionally apply task- specific tricks such CIDEr optimization, but for fair comparison, only present their results with plain cross-entropy loss.
VQA. We follow prior works (Wang al. in Appendix 5.4.4). As shown in Tab. CoBII can achieve satisfactory performance compared with other VL. models.
2021 1: Yu et al. 2022a to setup VQA fine-tuning (Detail
7

---

Published as a conference paper at ICLR 2024
Table 3: Fine-tuning Evaluation of CoBIT against previous image-text models. PT. denotes pre- trained, and Scratch denotes trained from scratch. *OFA incorporates images and text in its input, while others only use image one.
Image Understd.
-mate-Text Understd.
Content Creation
Visual
Model
Backbone
ImageNet Linear Probing
VQA
SNLI-VE
MS-COCO Captioning (CIDRR)
MS-COCO T21 Generation (FID.)
test-dev test-std
dev
test
CLIP
Scratch
85.4
ALIGN
Scratch
85.5
UNITER
FasterRRCNN
73.8
74.0
79.4
79.4
VinVL
FasterRRCNN
76.5
76.6
130.8
CLIP-VIL
CLIP
76.5
76.7
80.6
80.2
134.2
ALBEF
PT. VIT
75.8
76.0
80.8
80.9
BLIP
PT. VIT
78.3
78.3
136.7
SimVLM
PT. ResNet
80.0
80.3
86.2
86.3
143,3
OFA
PT. ResNet
82.0
82.0
91.07
91.2
145.3
10.5
X-IXMERT
Faster-RCNN
122.6
29.9
Scratch
77.9
91.1
122.3
CoDi
CLIP
149.9
3.22
CoCa-2.IB
Scratch
80.0
80.3
87.0
87.1
143.3
BEIT3-1.9B
Scratch
84.2
84.0
147.6
PALI-17B
PT. VIT
84.3
84.3
149.1
Parti-20B
3.22
CoBIT-Buse CoBIT-Large
Scratch
83.48
76.3
76.6
85.4
85.4
135.4
5.06
Scratch
86.44
77.9
78.3
86.2
86.0
139.5
4.62
SNLI-VE. Similar to fine-tuning VQA, we extract the final token output feature of the cross-modal decoder and apply linear classifier top to predict the three relations. As shown in Tab, 3 CoBIT can outperform strong VLP models and achieve superior performance. Note that other models, including CoBIT only use image premises as inputs, but OFA incorporates both image and text premises in its input.
Text-to-Image Generation. Following Parti and DALL-E, we fine-tune CoBIT on MS-COCO training set and evaluate the FID score on the sampled appendixtest set. Compared with zero-shot performance, fine-tuning on CoBIT-Base/CoBrLe Large further reduces the FID from 10.35/9.37 to 5.06/4.62, outperforming models of comparable scales.
4.4 ABLATION
This section comprehensively ablates the design choices in CoBIT. Most ablation experiments are conducted CoBIT-Base with reduced batch size and shrunken training schedule (See detail in Appendix 6.4.5). We select the following representative tasks: zero-shot ImageNet Classifica- tion (ZS IN.) for image understanding, VQA (fine-tuned) zero-shot Captioning multimodal understanding, and zero-shot text-to-image generation (ZS IG.) for image generation.
(ZS Cap.) for
Training Objectives. We ablate the existence of three training objectives: contrastive loss, 12T loss, and T2I loss, and study how they affect each other. The result is shown in Tab. + We obtain several interesting observations: (1). By comparing the first and last rows, it is found cross-modal generation objectives can improve image understanding a bit on top of contrastive loss. The zero-shot ImageNet accuracy is improved by 0.3%. (2) Comparing the second, third, and fourth rows, we see two generations losses, i.e., 12T loss and T2I loss, contradict each other little bit. But it's also promising because joint training essentially saves half of the parameters compared with using ensemble of two separate models. (3) From the fourth row and fifth row, we can see contrastive loss improves vision-language understunding while doesn't It influence image generation. Overall, we demonstrate the feasibility of harmoniously unifying three fundamental objectives in one framework.
Loss Weight. Given three objectives, we ablate different weights for them and select the best one as the default configuration for all experiments. Please see Appendix 6.2
Unicoder vs. Encoder. In previous Vision-Language works (Wang et al. 2021 Yu et al. 2022a
Chen et al.) 2022), encoder-decoder has been a de facto pipeline, where encoder encodes image/text propose uni-
features and cross cross-modal decoder fuses them and perform generation. Differently, coder to replace encoder, which can both encode and decode unimodal representations with shared parameters. Here, ablate image and text unicoders against image and text encoders. In the Ap-
8

---

Published as a conference paper at ICLR 2024
Table 4: Ablation on three objec- tives. Con. contrastive loss.
Table 5: Ablation unicoder encoder.
Ablation on initialization.
Table 6:
Objectives
Evaluation
Module
Evaluation
Con. T21 12T ZS IN. VQA ZS IG. (1)
Init. Text from CoCa
From Scratch
Image
Text
VQA
7S Cap. 7$ IG. ())
70.8
4
12.6
Encoder Encoder 65.9
32.9
13.8
7S IN.
75.35
75.02
Unicoder Encoder
66.5
36.9
13.38
VQA
68.48
68.55
Encoder Uniender 67.8
35.0
13.67
4
13.2
75 TG. (1)
11.42
11.63
Unicoder Unicoder 66.9
37,9
13.31
4
71.1
66.9
13.3
Appaiiaa AaPAiin aisking HOT UNNRRON aysthaare dIUAA on
A Aualy dOg MOBRRO sunglasses
cale calicee Suatiored budy bandrg
PARSRO bowf raage
bedddy havkip CAZZO dl nig Aroo dquavo
INTO made OffOR
A Higg tuat akya deap Boarnings
DEEP LEARNING
Figure Qualitative results of zero-shot text-to-image generation from CoBIT-Large with both good and failed cases.
pendix 6.6 put a diagram Fig. 4 to illustrate how the compared encoder-only models work, and explain the number of parameters of those two designs. As shown in Tab. [5) either image unicoder or text unicoder can improve over encoders, and applying them together brings the best trade-off for both image generation and multimodal understanding.
Pre-training Data. The ablation of three different pre-training datasets is detailed in Appendix 6.3
Train From Scratch. As mentioned in Sec. 3.3 we initialize the text unicoder with a pre- trained unimodal text decoder from CoCa. Here, we also attempt to train all from scratch. In this comparison, all models are trained with non-shrunken batch size to mitigate the possible gap due to the much larger batch size of CoCa. In Tab. 6 loading pre-trained weight from CoCa improves zero- shot Imagenet recognition and text-to-image generation by 0.3% and 0.2, which is a small margin. Also, doesn't 't even improve VQA. This comparison verifies the do-ability of training CoBIT all from scratch without hurting much performance.
4.5 VISUALIZATION
We visualized good and failed generated images of CoBIT-Large using the prompts from Par- tiPrompt (Yu et al., 2022b). As in Fig. 3 CoBIT can generate high-quality, broadly capable, open- domain images based on text. As for failed cases, we can see CoBIT misunderstands "A car made out of Sushi" as "A car with Sushi on top", also CoBIT fails to generate the reflection of mountains in the bowl of soup. More visualization and analysis are Fig.
5 CONCLUSION
We present VL foundation model, CoBIT, which unifies three objectives: cross-modal contrastive learning, image-to-tey generation, and text-to-image generation. The model is trained on large- scale noisy web-crawled image-text and image annotation data. CoBIT achieves strong zero-shot and transferable capacities of unimodal visual understanding, image-text matching, image-text un- derstanding, and text-to-image content creation.

---

Published as a conference paper at ICLR 2024
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 10.0004.449900 14198, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2223,, 2015.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint ar arXiv:2303.06555, 2023.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang. Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et Unilmv2: Pseudo-masked language models for unified language model pre-training. In International conference on machine learning, pp. 642-652. PMLR, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-115 11325, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- cration via masked generative transformers. arXiv preprint arXiv:2301.00704 2023.
Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski. Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0677 2022.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. 2019.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: /904: 10509, 2019.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation clip via vision-language knowledge distillation. arXiv preprint arXiv:2203.06:: 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering ext-to-imagee generation via transformers. Advances in Neural Information Processing Systems, 34: 14:21922-20 018822-1935, 19835, 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. arXiv preprint arXiv:2204. 14217, 2022.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2024
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In Computer Vision-ECo ECCV 2022: 17th European Conference. Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XV, pp. 89-106. Springer, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffuuson guidance.
arXiv preprint
Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for simultaneous vision- language generation. arXiv preprint arXiv:2211.14842, 2022.
Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi- directional image and text generation. In Proceedings of the 29th ACM International Conference Multimedia, pp. 1138-1147, 2021.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904-996. PMLR, 2021.
Taehoon Kim, Gwangmo Song, Sihaeng Lee, Sangyun Kim, Yewon Seo, Soonyoung Lee, Se- ung Hwan Kim, Honglak Lee, and Kyunghoon Bae. L-verse: Bidirectional generation between image and text. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16526-16536, 2022.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- training for unified vision- language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022.
Liunian Harold Li, Mark Yatskar Da Yin, Cho-Jui Hsieh, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: (55i::/900.035577 2019.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755, Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022. 2021.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022.
Norman MU, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Sulf-super meets language-image pre-training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Isruel, October 23-27, 2022, Proceedings, Part XXVI, pp. 529-544. Springer, 2022.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Ting Chen, Minh- Saah-Thang Luong, Yonghui Wu, et al. Combined scaling for open-vocabulary image classification. arXiv e-prints, pp. arXiv-2111, 2021.

---

Published as a conference paper at ICLR 2024
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region to- to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference computer vision, pp- 2641-2649, 2015.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Dla Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Alec Radford, Jong Wook Kim, Chris Hallaey, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-551,, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pp.
PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.061125 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 15:211-252, 2015.
Hshmat Sahak, Daniel Watson, Chitwan Saharia, and David Fleet. Denoising diffusion probabilistic models for robust image super-resolution in the wild. arXiv preprint arXiv:2302.078664 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 10X1::2205.11487. 2022.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp.
PMLR, 2018.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei-Wei Chang, Zhewei Yao. and Kurt Keutzer. How much can clip benefit arXiv:2107.06383, 2021.
tasks? arXiv preprint
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. arXiv preprint arXiv:2305.11846 2023.
Pax Team. Paxml, aka pax, https://ithubbcom/goollle/aa 2023. URL https://gihhbb. com/ gwwgg/oooggllnnemee
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zero-shot image-to-text generation for visual-semantic arithmetic. arXiv preprint arX14:2111.14447, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Michael Tschannen, Basil Mustafa, and Neil Houlsby.
Image-and-languuee understanding from
pixels only. arXiv preprint 2xXi:222/2.08045 2022.
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100, 2022a.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple ggquence-t--squenne learning framework. In International Conference on Machine Learning, pp. 23318-23340 PMLR, 2022b.
Wenhui Wang. Hangbo Bao, Li Dong. Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks, arXiv preprini 220i 10442, 2022c.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 2021.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint wwXiiv1901..6770 2019.
Xingqian Xu, Zhangyang Wang, Erie Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. arXiv preprint (1XXxx2211.0833333 2022.
Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training, arXiv preprint ar 1rX1v:2111.07783, 2021.
Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-shared contrastive language-image pre- training. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII, pp. 69-87, Springer, 2022.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arX1v:2110.0467,, 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917. 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfci Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022b.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang. Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive mudti-modal models: Pretraining and instruction tuiigg. arXiv preprint arXiv:2309. 02591, 2023.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2 1432, 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12104-12113, 2022a.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 18123-18133, 2022b.
13

---

Published as a conference paper at ICLR 2024
Han Zhang, Weichong Yin, Yewei Fang, Lanxin Li, Boqiang Duan, Zhihua Wu, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vilg: Unified generative pre-training for bidirectional vision- language generation. arXiv preprint arXi:22112.12883, 2021a.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588, 2021b.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 13041-13049 2020.
14

---

Published as a conference paper at ICLR 2024
ImageNet Linear Probe
MS-COCO Captioning
MS-COCO
VQA
SNLI-VE
T21 Generation
Optimizer
SGD
Adafacter
Gradient Clip
1.0
LR decay schedule Cosine Schedule to zero
Linear Schedule to zero
Exponential Schedule to zero
RandAugment
2,
1. 10
None
Training Step
225k
100k
50k
15k
100k
Warm-Up Up Step
0
1000
1000
500
1000
Batch Size
512
32
128
128
256
Learning Rate
3.2
1c-5
5c-5
5c-6
le-5
Weight Decay
0.0
0.1
0.1
0.01
0.045
Table 7: Hyper-parametter used in the multimodal experiments.
APPENDIX
ACKNOWLEDGEMENT
We would like to thank Prof. Shih-Fu Chang, Luowei Zhou, Long Chen for constructive discussion, Zirui Wang for help with downstream fine-tuning, Huiwen Chang for proofreading, and Laurent El Shafey for infra support.
6.1 LIMITATIONS BROADER IMPACT
Limitations. Although CoBIT unifies contrastive loss, text-to-image generation (T2I) loss, and image-to-texs (I2T) generation loss, from ablation experiments, we can find that T2I and 12T objec- tives contradict each other little bit. We hypothesize that it's because two generations require some fine-grained knowledge that is specific to each modality. We will leave it as a future direction.
Broader Impact. Models such as Stable Diffusion, DALL-E, Parti, CoBIT are trained on large and noisy image-text datasets that inevitable include potential biases toward people of different backgrounds. The bias can arise because the dataset's image and text samples might not be repre- sentative of the real-world population and could inadvertently promote certain stereotypes. Also, because the generated images of CoBIT are high quality, concern abusing the model to create and spread manipulated data (deepfake, political manipulation, etc.) does exist.
6.2 ABLATION LOss WEIGHT
Table 8: Ablation on weights of three losses. Con. denotes contrastive loss. T2I denotes text- to-image generation loss. 12T denotes image-to-text generation loss. ZS IN. denotes zero-shot ImageNet classification. ZS IG. denotes zero-shot text-to-image generation on MS-COCO, which is evaluated by FID and lower FID is better.
Weights
Evaluation
Con.
T2I
12T
ZS IN.
VQA
ZS IG. (L)
0.1
63.2
13.17
0.2
65.4
13.24
67.8
16.33
0.1
0.2
71.1
66.9
13.31
0.4
0.2
71.2
66.5
13.92
Given three objectives, we ablate different weights for them and select the best one as the default configuration for all experiments. We start with T2I and I2T first: given the weight of T2I fixed to
15

---

Published as a conference paper at ICLR 2024
Image Encoder + Text Encoder:
Generation loss
Text Generation loss
Contrastive loss
Cross-Modal Dacoder
Modal
Gress-Artention
Crose-Attetionn
Image Encoder
Text Encoder
Text Encoder
Image Encoder
Eribendiing
HR
PR
Image Tolens
Text Toiles
Objjective
(c) T21 Objective
Objectivv
Image Unicoder + Text Encoder:
Generation
Image Generation loes
Contrastive
Cross-Modal Decoder
Croes-Modal
Image Mode)
Encoder
Cross-Attantion
Image
Mode)
Encoder
(Enooding Mode)
Emstending
Toxt Talavis
Jeat Tokiena
Anagge Tolinn
Rost Tolioni
(b) Contrustive Objectiive
(4) T21 Objective
(d) 12T Objective
Image Encoder + Text Unicoder:
Text Generation loes
Image Generation loss
Contrastive loss
Cross-Modal Decoder
Gross-Modal Decoder
Cross-Attontinn
Image Encoder
Text Unicoder [Encoding Mode)
Croam-Antention
Text Unicoder (El (Encodling Mode)
Image Encoder
Text Unicoder (Dacoding Moda)
Vmage Talots
Tea! Thilenod
(b) Contrastive Objectiiv
(4) 121 Onjective
(d) 12T Objective
Figure 4: Diagram of three compared models in the ablation of unicoder vs. encoder. Top: Re- placing both image unicoder and text unicoder with image encoder and text encoder respectively. Middle: Replacing text unicoder with text encoder while keeping image unicoder. Bottom: Replac- ing image unicoder with image encoder while keeping text unicoder.
we ablate the loss of 12T. Then given T2I and 12T loss both fixed, the weight of contrastive loss is ablated. As we can see in Tab. a high weight of 12T such as will hurt the image generation heavily but also improve VQA. On the other hand, high weight of contrastive loss like 0.4 will not essentially improve image recognition and hurts both VQA and image generation. Overall, we chose Con.:221:12T 0.1:0.2:1 as our default setting, as it achieves good trade-off between three losses.
6.3 ABLATION ON PRE-TRAINING DATA
Table 9: Ablation on weights of three pre-training datasets. ZS IN. denotes zero-shot ImageNet classification. LP. means linear probing. ZS IG. denotes zero-shot text-to-image generation on MS-COCO, which is evaluated by FID and lower FID is better.
Datasets
Evaluation
ZS IN.
LP. IN.
VQA
ZS IG. (4)
JFT
71.6
81.4
64.8
14.6
ALIGN
70.9
81
67.2
13.8
WebLI
70.0
80.2
66.2
13.4
16

---

Published as a conference paper at ICLR 2024
Here ablate the three pre- pre-training datasets. To make fair comparison, the batch size is kept the same, the training is conducted for 200k steps on base model. As we can see in Tab. JFT is beneficial to classification tasks, focusing basic and precise semanties of image: WebLI has higher-quality image data and is specifically heneficial to text-to-image generation; ALIGN rela- tively noisy but covers broad semantics. That's the reason why we mixed them for a more balanced training.
6.4
DETAILS OF EXPERIMENTS
6.4.1 HYPERPARAMETERS IN FINE-TUNING
In Tab.
we present the hyperparameters we used in -hentuunng/liner probing of CoBIT.
6.4.2 ZERO-SHOT IMAGE CLASSIFICATION
we apply the same set of prompts transfer labels into sentences, such "a photo of (class}". Sim- ilar to the contrastive loss computed in Sec. 3.3, we input raw image/text into image/text unicoders in encoding mode obtain the global image and text features. Then, we compute their similarity to match images and labels.
6.4.3 ZERO-SHOT TEXT-TO-IMAGE GENERATION
In decoding, we employ Top-K sampling to sample 16 images for each text and use reranker to select the best image for evaluation. Following the de facto process, we compute FID score (Heusel et al.
2017) on MS-COCO appendixdata (lower FID is better).
6.4.4
VQA FINE-TUNING
we use the VQA v2, and the task formulated as a classification problem over 3, 129 most frequent answers in the training set. To accomplish this, the raw image is fed into the image unicoder using encoding mode, while the question is processed by the text unicoder in decoding mode. Subse- quently, the modal decoder utilizes the text decoding features input and cross-attends the encoded image features. The final token output feature of the cross- cross-modal decoder is considered the fused global feature. To predict the answer, linear classifier is trained on top of this feature.
6.4.5 SETUP OF ABLATION TRAINING
Specifically, the total batch size is 4,352, containing 4,096 for contrastive and I2T loss and 256 for T2I loss, and the total training step is 200k without figh-resslutton pre-training.
6.5 DETAILED COMPARISON WITH OTHER UNIFIED WORKS.
Unified Diffusion- Models. Some recent works utilize diffusion models jointly learn text-to-image and image-to-text learning, such as Versatile Diffusion (Xu et al.) 2022), CoDi (Tang et al. 2023). Hu al. (2022), UniDiffuser (Bao et al. 2023). Although they work well in image generation, they tend to perform worse in text generation and fail to handle image-text understanding tasks such as VQA, retrieval, etc. Moreover, they all initialize from Stable Diffusion, while CoBIT is mostly trained from scratch and learns superior image understanding capability.
V.S. Unified Auto-Regressiv Models. Unified-IO (Lu et al. 2022), OFA (Wang et 2022b) and CM3Leon Yuu et al.] 2023) also train text-to-imase and image-to to-text jointly. However, they use plain decoder or decoder model without considering the contrastive alignment as in CoBIT. BEIT-3 /Wang et al. (2022c) conducts the image-to-text generation by mask-and-reconstruutioo with an encoder model but could not handle the text-to-image generation task.
6.6 ILLUSTRATION OF REPLACING UNICODERS WITH ENCODERS IN COBIT
In Sec.4.4, ablate Unicoder vs. Encoder and demonstrate the effectiveness of proposed uni- coders. In Fig. 4 we show the diagram of using image and text encoders, image encoder+text uni- coder, and image unicoder+text encoder. As we can see, encoders can only encode visual textual
17

---

Published as a conference paper at ICLR 2024
Space cruise
White sports shoes made with real honeycomb
An eiephant under the sea
AAn pairiting of a robot hanging from hot air balloon and snowy mountain range in the background
Flying car logo
giraffe looking airplane window from the outside
FLYNG.CAR
PPoto a living corgi mede of stained glass. the corgi is walking outside in park
very beautiul painting of silhouette of a moose looking over a dramatic alaska mountain landscape
A leddy bear shaped like an
avocado
Pallure porfrait of briish shörthair wearing sunglesses holliing a sign says
Failure: red cubg beside a smaller yellow sphere
Failure: geico that looks like cat
GEICO LOOK
URAW MI ?
Figure 5: Qualitative results of zero-shot text-to-image generation from CoBIT-Large with both good and failed cases.
features while unicoders can perform both encoding and decoding, which shares the knowledge and boosts the generation result shown in previous ablation. It's noted that in pre-training, unicoder doesn't tt add extra parameters compared to encoders because encoding and decoding in unicoder reuse the same set of parameters; In the finetuning of text-to-image and image-to-text tasks, the uni-
18

---

Published as a conference paper at ICLR 2024
coder design indeed brings more parameters than encoder. Therefore, mainly evaluate zero-shot captioning and zero-shot image generation this ablation (Tab. [ to eliminate the difference of parameter numbers.
6.7
MORE VISUALIZZTIOON
In Fig. S We attach more visualization of CoBiT-Large on zero-shot text-to- tomee generation with novel prompts PartiPrompts Yu al. (2022). For better visualization when zoom-in, em- ploy [Shaa et al. (2023) as the super-resolution module to upsample generated 256x256 images to 1024x1024 images. It's noted that when computing FID,, we still use 256x256 images and the high- resolution ones are only used for visualization. In failed cases, messes up the size attributes two objects. For example, in the last example, yellow sphere ought to be smaller. (2) CoBIT sometimes couldn' 't render the details of words in text very well. In the second last example, "DRAWIT" is rendered as "DRAWMI?". (3) CoBIT occasionally misunder- stands the text. In the third last example, we expect geico that looks like cat whereas CoBIT first renders "GEICO THAT LOOK" then generates a cat. It's indeed a new way to interpret the text but not the desired way of humans.
find that: (1) CoBIT sometimes
19


Reviews:
[{"review_id": "XuFLZ1LCHu", "reviewer": "Reviewer_PXu6", "summary": "This paper introduces a complex training framework that combines contrastive learning, image-to-text, text-to-image, and image-to-image approaches. In addition to the intricate model, they also establish a large-scale database. Through the use of these complex models and extensive data, the trained model performs well on various test tasks.", "questions": "Most of the questions that I would like the authors to address have been raised in the \"Weaknesses\" section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The primary contribution of this paper lies in the integration of several models and the creation of a database tailored for training such models. Building upon this foundation, they conducted parameter optimization to fine-tune the loss, ultimately yielding favorable results.\n- This paper is easy to read, with coherent writing throughout. The experimental section offers a rich set of comparisons, although there is room for improvement in the analysis and conclusions.", "weaknesses": "While this paper primarily focuses on pre-training and achieves promising results through additional data collection, it still faces the following issues:\n\n- The most significant concern regarding model design in this paper is the limited novelty of the model's contribution. Each module of the model presented in Fig. 2 is already established, and the authors simply combined and fused their losses for joint training. While the training results appear favorable on the extensive data they collected, the paper's drawback remains the lack of model novelty.\n\n- Data is also a concern here. While it's reasonable to leverage more data, it introduces an element of unfairness because the data for the compared models is not consistent. This makes it challenging to draw valid conclusions, and I believe this aspect may also impact the overall findings.\n\n- The optimization process by the authors appears rather intricate, involving the design of numerous loss functions and hyperparameters. This complexity in model design makes replication challenging because the network and parameters are closely tied to the specific dataset they chose. I believe this is also a weakness of the study.\n\n- In the Model Initialization section, the authors mention using CoCa but do not provide a specific explanation for this choice. They do not clarify why they didn't opt for pre-trained BERT models, sentence transformers, or CLIP-based sentence encoders. It would be beneficial if the authors could offer more insight into the rationale behind their selection of COCA as the initialization method and why they didn't consider other pre-trained models for this purpose.\n\n- In Fig. 2, the fact that ViT-VGGAN is frozen to some extent can be considered a weakness, as the existing CoBIT framework assumes the availability of a well-pretrained ViT-VQGAN. This assumption is partially valid, but if the domain changes, it implies that the entire model would need to be re-pretrained. Moreover, if there's a new VQGAN model, the entire model would also require retraining. I am curious about how the authors address this issue, as this paper primarily focuses on pre-training and should ideally rely on existing model checkpoints as much as possible.\n\n- The results of the Linear Probing experiment show a significant improvement, yet the authors have not provided a more substantial explanation. If Linear Probing is the reason for the 1% improvement in performance, how would fine-tuning more parameters affect the outcome?\n\n- The authors trained image captioning using only cross-entropy loss and overlooked RL-based rewards, which are an important component. The authors should have at least discussed how to integrate RL-based methods into this model because RL-based methods, such as self-critical sequence training (SCST), also impose specific requirements on model design.\n\n- Furthermore, when reporting captioning results, the authors should present them on the MSCOCO online test set, as it provides a fair evaluation benchmark, rather than reporting them on a local set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L7yJDIvTUE", "reviewer": "Reviewer_Ui8A", "summary": "This paper introduces a new model known as the Contrastive Bi-directional Image-Text generation model (CoBIT), which is the first to combine three pre-training objectives in one framework: contrastive objectives, image-to-text generative objectives, and text-to-image generative objectives. CoBIT is composed of an image unicoder, a text unicoder, and a cross-modal decoder. The unicoders can perform both encoding and decoding tasks, allowing them to share knowledge and improve both image-to-text and text-to-image generations. The CoBIT model demonstrated superior performance in multiple areas, including image understanding, image-text understanding and text-based content creation, particularly in zero-shot scenarios. The paper concludes that CoBIT's unification of the three objectives has led to strong zero-shot and transferable capacities in unimodal visual understanding, image-text matching, image-text understanding, and text-to-image content creation.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. It is the first to unify three pre-training objectives in one framework, effectively combining contrastive objectives, image-to-text generative objectives, and text-to-image generative objectives. This innovative approach sets it apart from existing models and makes a significant contribution to the field of Vision-and-Language (VL).\n\n2. The performance of CoBIT is outstanding. The authors provide a thorough and detailed explanation of the CoBIT model, including its novel unicoder-decoder structure. They also present extensive experimental results to demonstrate its superior performance in various tasks, including image understanding, image-text understanding, text2image and image2text generation.\n\n3. By allowing the image and text unicoders to switch between encoding and decoding in different tasks, the CoBIT model demonstrates a novel approach to handling both text-to-image and image-to-text generation within a single framework. This not only improves the flexibility of the model but also has the potential to inspire future research in multimodal generation tasks.", "weaknesses": "1. Computational Efficiency: The use of Unicoder variants in cross-modal generation scenarios, as mentioned in the appendix of the paper, appears to increase the computational load and the number of parameters used in downstream fine-tuning tasks. However, this aspect is not well elaborated in the main text of the paper. A more thorough discussion on the computational efficiency and trade-offs of using Unicoder compared to Encoder would provide clearer insights into the practicality of the model.\n\n2. Impact of Pre-training Data: The paper shows good ImageNet linear probe performance for CoBIT in Table 3. It would be interesting to see a more detailed analysis of how the pre-training data impacts the model's performance. This could include experiments with different sizes or types of pre-training datasets.\n\n3. Ablation Study: The ablation study in the paper could be strengthened. \n\n4. From Table 1, it can be observed that the proposed base and large sizes of the model have more parameters compared to other multimodal models. This increase in model complexity might have implications for computational resources and efficiency. The authors should provide more explanation or justification for this design choice, and possibly discuss the trade-off between model complexity and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z1yNMdGyJs", "reviewer": "Reviewer_wSdD", "summary": "This paper first propose a multi-modal transformer trained using image-to-text, text-to-image, and contrastive loss, which are three main types of loss function in multimodal area of research. Specially designed architecture allows training the model with three different loss functions all together.\n\nTrained in bidirectional manner, the proposed CoBiT shows great performance on both image-to-text and text-to-image tasks, along with comparable performance on image-text retrieval and other downstream tasks.", "questions": "Is there any experimental results of CoBiT model trained with more general and smaller datasets? (CC3M, CC12M, yfcc15m, LAION400M, LAION2B)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "This paper further advance the research on bidirectional image-text generation task. Shown from previous works, bidirectional image-text training stablize the training while showing comparable performance to other unidirectional models. This paper further add contrastive loss to this bidirectional concept to let the model efficiently learn the latent of multimodal domain.\n\nTrained on large scale dataset (>4B), CoBiT shows great performance on image-to-text and text-to-image generation tasks.", "weaknesses": "The problem is that concept of combining two or more different multimodal losses has existed before. The proposed CoBiT architecture seems to be a mixture of L-Verse ( Kim et al. 2022) and CoCa (Yu et al. 2022). While using three different training losses and successfully train a model is a hard work, The experiment section still lacks justification on how it can improve the model's performance.\n\nSince model is trained with more than 4B image-text pairs which is not easily accessible to other researchers (JFT-4B), training CoBiT from scratch with smaller or more general datasets will help readers compare the performance of CoBiT with other works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bYErwRHMCD", "reviewer": "Reviewer_GAkS", "summary": "This paper proposes to unify three commonly used objectives in VL pre-training, namely cross-modal contrastive learning, image-to-text generation, and text-to-image generation, into one pretraining framework. In particular, the authors share the same parameters for Transformer in encoding and decoding. After pretraining on ALIGN, JFT-4B and WebLI datasets, the model shows superiority on image understanding, image-text understanding (retrieval, VQA, captioning) and content creation (text-to-image) tasks among models smaller than 1B parameters.", "questions": "•\tCan you list the exact parameter sizes of each model in Table1,2,3? That should be more fair comparison.\n•\tWhy are the tasks shown in ablation studies (Table 4,5,6) not consistent? What is the reason for lacking some tasks in each table?\n•\tPlease check the weakness for more questions.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper is well-written and easy to follow. \n2.\tExperimental results show good performance in several benchmarks.", "weaknesses": "1.\tThe whole pipeline is not novel and is more like an assembling of existing method/components. As mentioned by the authors, the unicoder is already used in other works  (Sec. 3.2). Three objectives for cross-modal learning are commonly used. Even though it is first joint learned, I cannot see the advantage of combining them. Although the authors claim that they should benefit from each other, it is intuitive that they have different focuses. For example, contrastive loss will benefit retrieval tasks. It could be better if the authors could have more insights on each objective’s advantages from their results.\n2.\tThe advantages of the model over other models are not clear. There is no performance that shows fair comparison with other models. It could be the dataset used for pre-training, batch-size, pretraining epochs, etc. \n3.\tFrom the ablation studies in Table 4, 5 and 6, the gain of proposed joint training of three objectives and shared Transformer (namely unicoder) is rather small and not steady over all tasks. This could not convince me of the advantage of proposed method.\nOverall, I think the insight and technical contribution of this paper are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vfD9LHkTJN", "reviewer": "Reviewer_vaSs", "summary": "The paper presents a novel model named Contrastive Bi-directional Image-Text generation model (CoBIT) aimed at unifying three pre-training objectives, namely, cross-modal contrastive learning, image-to-text generation, and text-to-image generation, under a singular framework. This unification is achieved through a unique unicoder-decoder architecture that houses an image unicoder, a text unicoder, and a cross-modal decoder. The unicoders are capable of toggling between encoding and decoding roles based on the task at hand, thereby promoting shared knowledge which is advantageous for both image-to-text and text-to-image generations. The authors claim that this model architecture allows for superior performance in various tasks like image understanding, image-text understanding, and text-based content creation, with a significant highlight on its effectiveness in zero-shot scenarios. The model's efficiency is validated through a series of comprehensive experiments, demonstrating an impressive performance against existing models in the field of Vision-and-Language (VL).", "questions": "- Theoretical Justification: A deeper theoretical analysis of the assumed synergy between image-to-text and text-to-image generation tasks could strengthen the premise of the unified framework.\n\n- Conflict Mitigation Strategies: Developing and incorporating strategies to mitigate the potential conflicts between different objectives could help in achieving a more balanced optimization across all tasks.\n\n- Failure Analysis: Conducting a thorough failure analysis to identify the model's weaknesses and understanding its behavior under different conditions or limitations could provide a clearer path for future improvements.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unified Framework: CoBIT effectively brings together three prevalent pre-training objectives under a single framework, which could potentially lead to a more holistic understanding and representation of image-text pairs.\n\n- Flexible Architecture: The unicoder-decoder structure is innovative and allows for flexibility and shared knowledge, which is beneficial for multiple generation tasks.\n\n- Zero-shot Performance: The model demonstrates high accuracy and superior performance in zero-shot scenarios across various tasks like image understanding, image-text retrieval, image captioning, and text-to-image generation.\n\n- Parameter Efficiency: The paper highlights excellent parameter efficiency, as the same set of Transformer parameters are utilized for both encoding and decoding tasks, which is a crucial factor considering the computational resources.", "weaknesses": "- Assumed Synergy: The core premise of CoBIT hinges on the idea of a symbiotic relationship between image-to-text and text-to-image generation tasks. However, the paper doesn't thoroughly investigate or justify the assumed synergy. It's crucial to establish a theoretical foundation for this assumption, or the unified framework might not hold in different contexts or datasets.\n\n- Objective Conflicts: While unifying different objectives under a single framework is innovative, it poses a risk of conflicting objectives that might detract from optimizing each task individually. The paper acknowledges this to some extent but doesn't provide a robust solution to mitigate potential conflicts.\n\n- Evaluation Scope: The evaluation primarily focuses on showcasing the model's strengths, with a less thorough investigation into the model’s weaknesses or failure modes. A more balanced evaluation, including a deeper exploration of where the model falls short, would provide a more comprehensive understanding of the model's capabilities and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)