CoCa: Contrastive Captioners are Image-Text Foundation Models
Authors: Jiahui Yu
Zirui Wang
Vijay Vasudevan
Legg Yeung
Mojtaba Seyedhosseini
Yonghui Wu
Venue: Accepted by TMLR ()

Abstract: Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and 91.0% with a finetuned encoder.

Published in Transactions on Machine Learning Research (08/2022)
CoCa: Contrastive Captioners are Image-Texx Foundation Models
Jiahui Yu"
jiahuigu@goooge.ccm ziruiu@google.com
Zirui Wang*
Vijay Vasudevan
Legg Yeung
Mojtaba Seyedhosseini
Yonghui Wu
Google Research
Equal contribution.
Reviewed on OpenReview: https: // openreview. net/ formm? id= E2277P3AYC
Abstract
Exploring large-scale pretrained foundation models is of significant interest computer vision because these models can be quickly transferred to many downst tasks. This paper presents Contrastive Captioner (CoCa), minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generalive methods like SimVLM. In contrast to standard eecoderd transformers where all decoder layers attend to encoder outputs, CoCa omits cross-altention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply contrastive loss between unimodal image and text embeddings, in addition a captioning loss the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web- scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves stato-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNett Kinetics- 400/600/700, Moments-in-Timee) crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with frozen encoder and learned classification head, and 91.0% with fineluned encoder.
1 Introduction
Deep learning has recently witnessed the rise of foundation language models (Bommasani et al., 2021) such as BERT (Devlin et al., 2018), T5 (Raffel et al., 2019), GPT-3 (Brown et al., 2020), where models are pretrainod on web-scale data and demonstrate generic multi-tasking capabilities through zero-shot, few-shot or transfer learning. Compared with specialized individual models, pretraining foundation models for massive downstream

---

Published in Transactions on Machine Learning Research (08/2022)
Captioning Loss
image captioning & multimodal representation
Crow Attention
Multimodal Text Decoder
classification
alignment
Multimodal Text Decoder
Contrastive Loss
Image Encoder
Unimodal
Image Encoder
Image Encoder
Unimodal Text Decoder
Image
Unimodal Text Decoder
Decoder
Encoder
1 image
image
text
image
text
image
text
Image Captioning & Multimodial Understanding (encoder decoder models)
CoCa
Visual Recognition (single-encoder models)
Crossmodal Alignment (dusl-encoder models)
Pretraining
Zero-shot, frozen-feature or finetuning
Figure Overview of Contrastive Captioners (CoCa) pretraining as image- text foundation models. The pretrained CoCa can be used for downstream tasks including visual recognition, vision-language alignment, image captioning and multimodal understanding with zero-shot transfer, frozen-feature evaluation or end-to- end finetuning.
tasks can amortize training costs, providing opportunities to push the limits of model scale (Barham et al., 2022) for human-level intelligence.
For vision and vision-language problems, several foundation model candidates have been explored: (1) Pioneering works (Girshick et al., 2014; Long et al., 2015; Simonyan & Zisserman, 2014) have shown the effectiveness of single-encoder models pretrained with cross-entropy loss image classification datasets such as ImageNet (Deng et al., 2009). The image encoder provides generic visual representations that can be adapted for various downstream tasks including image and video understanding (Dai et al., 2021; Zhang et al., 2021a). However, these models rely heavily on image annotations as labeled vectors and do not bake in knowledge of free-form human natural language, hindering their application to downstream tasks that involving both vision and language modalities. (2) Recently, a line of research (Radford et al., 2021; Jia et al., 2021; Yuan al., 2021) has shown the fcasibility of imago-text foundation model candidates by pretraining two parallel encoders with a contrastive loss on web-scale noisy image-text pairs. In addition to the visual embeddings for vision-only tasks, the resulting dual-encoder models can additionally encode textual embeddings to the same latent space, enabling new crossmodal alignment capabilities such as zero-shot image classification and image-text retrieval. Nonetheless, these models are not dircctly applicable for joint vision-language understanding lasks such as visual question answering (VQA), due to missing joint. components to learn fused image and text representatious. (3) Another line of research (Vinyals al., 2015; Wang et al., 2021b; 2022) has explored generative pretraining with encoder-decoder models to learn generic vision and multimodal representations. During pretraining, the model takes images the encoder side and applies Language Modeling (LM) loss (or PrefixI.M (Raffel et al., 2019; Wang et al., 2021b)) on the decoder outputs. For downstream tasks, the decoder outputs can then be used as joint representations for multimodal understanding tasks. While superior Aisionllnguage results (Wang et al., 2021b) have been attained with pretrained encoder-decoder models, they do not produce text-only representations aligned with image embeddings, thereby being less feasible and efficient for crossmodal alignment tasks.
In this work, unify single-encoder, dual-encoder and encoder-decoder paradigms, and train one image-text foundation model that subsumes the capabilities of all three approaches. We propose a simple model family named Contrastive Captioners (CoCa) with a modified encoder-decoder architecture trained with both rastive loss and captioning (generative) loss. As shown in Figure 1, we decouple the decoder transformer into two parts, a unimodal decoder and a multimodal decoder. We omit cross-attention in unimodal decoder layers to encode text-only representations, and cascade multimodal decoder layers cross-attending to image encoder outputs to learn maltimodal image-text representations. We apply both the contrastive objective between outputs of the image encoder and unimodal text decoder, and the captioning objective at. the
2

---

Published in Transactions on Machine Learning Research (08/2022)
output of the multimodal decoder. Furthermore, CoCa is trained on both image annotation data and noisy image-text data by treating all labels simply as text. The generative loss on image annotation text provides a fine-grained training signal similar to the single-encoder cross-entropy loss approach, effectively subsuming all three pretraining paradigms into a single unified method.
The design of CoCa leverages contrastive learning for learning global representations and captioning for fine-grained region-level features, thereby benefiting tasks across all three categories shown in Figure 1. CoCa shows that single pretrained model can outperform many specialized models using zero-shot transfer minimal task-specific adaptation. For example, CoCa obtains 86.3% zero-shot accuracy on ImageNet and better zero-shot crossmodal retrieval on MSCOCO and Flickr30k. With a frozen-encoder, CoCa achieves 90.6% on ImageNet classification, 88. 08.0%//8..5% 11...%%/8.111. on Kinetics-40/600 /00000000000//// /700 and 47.4% on Moments-in-Tinee After lightweight finetuning, CoCa further achieves 91.0% on ImageNet, 82.3% on VQA and 120.6 CIDEr score on NoCaps.
2 Related Work
Vision Pretraining. Pretraining ConvNets (Krizhevsky et al., 2012) or Transformers (Vaswani et al., 2017) large-scale annotated data such as ImageNet (Girshick et al., 2014; Long et al., 2015; Simonyan & Zisserman, 2014), Instagram (Mahajan et al., 2018) or JFT (Zhai et al., 2021a) has become a popular strategy towards solving visual recognition problems including classification, localization, segmentation, video recognilion, tracking and many other problems. Recently, self-supervised pretraining approaches have also been explored. BETT (Bao et al., 2021) proposes a masked image modeling task following BERT (Devlin et al., 2018) in natural language processing, and uses quantized visual token ids as prediction targets. MAE (He et al., 2021) and SimMIM (Xie et al., 2021) remove the need for an image tokenizer and directly use a light-weight decoder or projection layer to regress pixel values. Nonetheless, these methods only learn models for the vision modality and thus they are not applicable to tasks that require joint reasoning over both image and text inputs.
Vision-Language Pretraining. In recent years, rapid progress has been made in vision-language pretraining (VLP), which aims to jointly encode vision and language in a fusion model. Early work (e.g. LXMERT (Tan & Bansal, 2019), UNITER (Chen et al., 2020), VinVL (Zhang et al., 2021b), VL-T5 (Cho et al., 2021)) in this direction relies on pretrained object detection modules such as Fast(er) R-CNN (Ren et al., 2015) to extract visual representations. Later efforts such as ViLT (Kim et al., 2021) and VLMo (Wang et al., 2021a) unify vision and language transformers, and train multinodal transformer from scratch. More recently, a line of work has also explored zero-shot /fwws//ffft learning for vision- language tasks by re-using pretrained large language models (Yang et al., 2022b; Jin et al., 2021; Tsimpoukelli et al., 2021). Compared to prior methods, this paper focuses on training a unified model from scratch subsuming the capability of multinodal understanding and generation.
Image-Text Foundation Models. Recent work has proposed image-text foundation models that can subsume both vision and vision-language pretraining. CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) demonstrate that dual-encoder models pretrained with contrastive objectives on noisy image-text pairs can learn strong image and text representations for crossmodal alignment tasks and zero-shot image classification. Florence (Yuan et al., 2021) further develops this method with unified contrastive objective (Yang et al., 2022a), training foundation models that can be adapted for a wide range of vision and image-text benchmarks. To further improve zero-shot image classification accuracy, LiT (Zhai et al., 2021b) and BASIC (Pham et al., 2021a) first pretrain model on an large-scale image annotation dataset with cross-entropy and further finetune with contrastive loss on an noisy alt-text image dataset. Another line of research (Wang et al., 2021b; 2022; Piergiovanni et al., 2022) proposes encoder-decoder models trained with generative losses and shows strong results in vision- benchmarks while the visual encoder still performs competitively on image classification. In this work, we focus on training an image-texi foundation model from scratch in a single pretraining stage to unify these approaches. While recent works (Singh et al., 2021; Li et al., 2021; 2022) have also explored imago-text unification, they require multiple pretraining stages of unimodal and multimodal modules to attain good performance. For example, ALBEF (Li et al., 2021) combines contrastive loss with masked language modelling (MLM) with dual-encoder design. However, our approach is simpler
3

---

Published in Transactions on Machine Learning Research (08/2022)
and more efficient to train while also enables more model capabilities: (1) CoCa only performs one forward and backward propagation for batch of image-text pairs while ALBEF requires two (one on corrupted inputs and another without corruption), (2) CoCa is trained from scratch on the two objectives only while ALBEF is initialized from pretrained visual and textual encoders with additional training signals including momentum modules. (3) The decoder architecture with generative loss preferred for natural language generation and thus directly enables image captioning.
3
Approach
We begin with a review of three foundation model families that utilize natural language supervision differently: single-encoder classification pretraining, dual-encoder contrastive learning, and encoder-decoder image captioning. We then introduce Contrastive Captioners (CoCa) that share the merits of both contrastive learning and image-to-caption generation under a simple architecture. We further discuss how CoCa models can quickly transfer to downstream tasks with zcro-shot transfer or minimal task adaptation.
3.1 Natural Language Supervision
Single-Encoder Classification. The classic single-encoder approach pretrains visual encoder through image classification large crowd-sourced image annotation dataset (c.g., ImageNet (Deng et al., 2009), Instagram jMahajan et al., 2018) JFT (Zhai et al., 2021a)), where the vocabulary of annotation texts is usually fixed. These image annotations are usually mapped into discrete class vectors to learn with a cross-entropy loss as
Lcs = -ply) log (o()),
(1)
where p(y) is a one-hot, multi-hot or smoothed label distribution from ground truth label y. The learned image encoder is then used as generic visual representation extractor for downstream tasks.
Dual-Encoder Contrastive Learning. Compared to pretraining with singlo-encoder classification, which requires hunan-annolated labels and data cleaning, the dual-encoder approach exploits noisy web-scale text descriptions and introduces a learnable text tower to encode free-form texts. The two encoders are jointly optimized by contrasting the paired text against others in the sampled batch:
N
exp(x.ys/o)
log explyT m//o) Σn= exp(u xz/o)"
LCou
log Σn exp(x] exp(xTuy/o)
(2)
image-to-text
text-to-image
where n, and Y5 are normalized embeddings of the image in the i-th pair and that of the text in the j-th pair. N is the batch size, and a is the temperature to scale the logits. In addition to the image encoder, the dual-encoder approach also learns aligned text encoder that enables crossmodal aligmment applications such as image-text retrieval and zero-shot image classification. Empirical evidence shows zcro-shot classification is more robust (Radford et al., 2021; Jia et al., 2021; Andreassen et al., 2021) on corrupted or out-of-distribution images.
Encoder-Decoder Captioning. While the dual-encoder approach encodes the text as a whole, the generative approach (a.k.a. captioner) aims for detailed granularity and requires the model to predict the exact tokenized texts of y autoregressively. Following a standard encoder-decoder architecture, the image encoder provides latent encoded features (e.g., using a Vision Transformer (Dosovitskiy et al., 2021) or ConvNets (He et al., 2016)) and the text decoder learns to maximize the conditional likelihood of the paired text under the forward autoregressive factorization:
Caap Σ log Po(uelvet,.).
(3)
The encoder-decoder is trained with teacher- feacher-forcing (Williams & Zipser, 1989) to parallelize computation and maximize learning efficiency. Unlike prior methods, the captioner approach yiclds a joint imago-text representation that can be used for vasion-language understanding, and is also capable of image captioning applications with natural language generation.
4

---

Published in Transactions on Machine Learning Research (08/2022)
Captioning Loss
two running fiold Dsl
Algorithm 1 Pseudocode Contrastive Captioners architecture.
ixage, soxt.ido, text. tabellabela, seat, sask: paired (inage, text) dasa * eonguwry: query token for contraative sxbedding cap ap--quecy1 query votens for captioning exbedding
Multimodal
Cross Attertion
Text Decoder
clo_token_id: apecial cla cla_tokun, 1d vocabulury
def atteucional pooling(featuree sultibeed, Sastsibad___ttttee stt_aatttntiinnnnn (featuure, quary) retura noyor_nom(ours)
Contrastive Loss
attentional pooling
cls-token
Image Encoder
Unimodal Text Decoder
con festure astentional pooling festure, con quezy) Datch,
tng- tng-fexture vit__encader (tzage)
Daatch reg_len, dix]
cap_feature attestioaal.polllie www.inggonoggg..... (inturuurrra cap-quezy) [batch,
ido concut(text. ida, slo_tokon, id)
mask - concat (tast rast, secoe, seros (telccl,.tokon_id)) a unped ela,telon_id tat, cate arbedding lodup(
aooaoaoooaca
[<] two dags runningin in
field [CLS]
unixodal, tn.tramneersarr (txt. mask, cruss, cross_sstordeon)) milstnodal_oot in.transtomerr www.noommuninddl.ccom
attarcap. attarcap_feaatre) feature)
cla, feature
Dbatess dix]
con, Josa contrastive contrattivellaaaee (con featuze, cla_tolen_feature
"two dogs running in a field"
pairs
cap Jose softusr. croes, entropy- Jote( cultinodol, ous, labela text labelo, text.eaak))
transtoemer based encoder. trmnsfossr: language model transformers.
image
text
Figure 2: Detailed illustration of CoCa archilecture and training objectives.
3.2
Contrastive Captioners Pretraining
Figure depicts the proposed contrastive captioner (CoCa): a simple encoder-decoder approach that seamlessly combines the three training paradigms. Similar to standard image-text encoder-decoder models, CoCa encodes images to latent representations by neural network encoder, for example, vision transformer (ViT) (Dosovitskiy et al., 2021) (used by default; it can also be other image encoders like ConvNets (He et al., 2016)), and decodes texts with a cansal masking transformer decoder. Unlike standard decoder transformers, CoCa omits tention in the first half of the decoder layers to encode unimodal text representations and cascades the rest of the decoder layers, cross- cross-attending to the image encoder for multimodal image-tex: representations. As a result, the CoCa decoder simultaneously produces both unimodal and multimodal text representations that allow us to apply both contrastive and generative objectives as
LcoCa ACoa LCon
ACap CCap:
(4)
where Aoon and ACap are loss weighting hyper-parameters. We note that the single-encoder cross-entropy classification objective can be interpreted as a special case of the generative approach applied on image annotation data, when the vocabulary is the set of all label names.
Decoupled Text Decoder and CoCa Architecture. The captioning approach optimizes the conditional likelihood of text while the contrastive approach uses an unconditional text representation. To address this dilemma and combine these two methods into single model, we propose simple decoupled decoder design where split the decoder into unimodal and multimodal components, by skipping the cross-attention mechanism in the unimodal decoder layers. That is, the bottom Nuui unimodal decoder layers encode the input text as latent vectors with causally-masked self-attention, and the top Amulti multimodal layers further apply cansally- mansally-masked masked self-attention and together with cross-attention to the output of the visual encoder. All decoder layers prohibit tokens from attending to future tokens, and it is straightforwant to use the multimodal text decoder output for the captioning objective CCap- For the contrastive objective LCons we append a learnable [CLS] token al the end of the input sentence and use its corresponding output of unimodal decoder the text embedding. We split the decoder in half such that Nuui Ammulti- Following ALIGN (Jia al., 2021), we pretraín with image resolution of 288 288 and patch size 18x 18, resulting in a total of 256 image tokens. Our largest CoCa model ('CoCa' in short) follows the ViT-giint setup in Zhai et al. (2021a) with IB-parameters in the image encoder and 2. 2.1 2.1B-parameten altogether with the text decoder. We also explore two smaller variants of "CoCa-Base and "CoCa-Large detailed in Table 1.
Attentional Poolers. It is noteworthy that the contrastive loss uses a single embedding for each image while the decoder usually attends to sequence of image output tokens in encoder-decoder captioner
5

---

Published in Transactions on Machine Learning Research (08/2022)
(Wang et al., 2021b). Our preliminary experiments show that a single pooled image embedding helps visual recognition tasks as a global representation, while more visual tokens (thus more fine-grained) are beneficial for multimodal understanding tasks which require region-level features. Hence, CoCa adopts task-specific attentional pooling (Lee et al., 2019) to customize visual representations to be used for different types of training objectives and ream lasks. Here, pooler is a single multi-head attention layer with Nquerry learnable queries, with the encoder output as both keys and values. Through this, the model can learn pool embeddings with different lengths for the two training objectives, as shown in Figure 2. The use of task-specific pooling not only addresses different needs for different tasks but also introduces the pooler natural task adapter. We use attentional poolers in pretraining for generative loss Qquery 256 and contrastive loss Rquery- = 1.
Pretraining Efficiency, A key benefit of the decoupled autoregressive decoder design is that it can compute two training losses considered efficiently. Since unidirectional language models are trained with causal masking on complete sentenees, the decoder can efficiently generate outputs for both contrastive and generative losses with a single forward propagation (compared to two passes for a bidirectional approach (Li et al., 2021)). Therefore, the majority of the compute is shared between the two losses and CoCa only induces minimal overhead compared to standard encoder-decoder models. On the other hand, while many existing methods (Zhai et al., 2021b; Pham et al., 2021a; Singh et al., 2021; Wang et al., 2021a; Li et al., 2021; 2022) train model components with multiple stages on various data sources and/or modalities, CoCa is pretrained end-to-end from scratch directly with various data sources (i.e., annotated images and noisy alt-text images) by treating all labels as texts for both contrastive and generative objectives.
3.3 Contrastive Captioners for Downstream Tasks
Zero-shot Transfer, A pretrained CoCa model performs many tasks in a zero-shot manner by leveraging both image and text inputs, including zcro-shot image classification, zero-shot image-text cross-retrieval, zero-shot video-text cross-retrieval Following previous practices (Radford et al., 2021; Zhai et al., 2021b), "zero-shot" here is different from classical zero-shot learning in that during pretraining, the model may see relevant supervised information, but no supervised examples are used during the transfer protocol. For the pretraining data, we follow strict de-duplication procedures introduced in Jia al. (2021); Zhai et al. (2021b) to filter all near-domain examples to our downstream tasks.
Frozen-feature Evaluation. As discussed in the previous section, CoCa adopts task-specific attentional pooling (Lee et al., 2019) (pooler for brevity) to customize visual representations for different types downstream tasks while sharing the backbone encoder. This enables the model to obtain strong performance as frozen encoder where we only learn a new pooler to aggregate features. It can also benefit to multi-task problems that share the same frazen image encoder compulation but different task-specific heads. As also discussed in He et al. (2021), linear-evalaatin struggles to accurately measure learned representations and we find the attentional poolers more practical for real-world applications.
CoCa for Video Action Recognition. We use a simple approach to enable a learned CoCa model for video action recognition tasks. We first take multiple frames of a video and feed each frame into the shared image encoder individually as shown in Figure For frozen-feature evaluation or finetuning,
eertmop
attantiono pooking
we learn an additional pooler on top of the spatial and temporal feature tokens with softmax tross-entropy loss. Note the pooler has single query token thus the computation of pooling over all spatial and temporal tokens is not expensive. For zero-shot video-texi retrieval, we use an even simpler approach by computing the mean embedding of 16 frames of the video (frames are uniformly sampled from video). We also encode the captions of each video as target embeddings when computing rieval metrics (similar to the image-text case).
Image Encoder
Image Encader
Image Encoder
Image Encodar
video fromm
video froms
video framn
video framme
2
3
4
5
6
Figure: 3: CoCa for video recognition.
6

---

Published in Transactions on Machine Learning Research (08/2022)
Image Encoder
Text Decoder
Image
Text
Model
Layers
ML.P
Params
Tluii
Pmuatii
MLP
Params
Hidden
Heads
Total Params
CoCa-Base
12
3072
86M
12
12
3072
297M
768
12
383M
CoCa-Large
24
4096
303M
12
12
4096
484M
1024
787M
CoCa
40
6144
1B
18
18
5632
1.18
1408
16
2.1B
Size variants of CoCa. Both image encoder and text decoder are Transformers (Dosovitskly et al., 2021; Vaswani et al., 2017).
Table 1:
4 Experiments
this section, we first describe the details of our experimental setup. The main results are presented next organized as visual recognition tasks, crossmodal alignment tasks, image captioning and multimodal understanding tasks. Our main results are conducted under three categories for downstream tasks: zero-shot transfor, frozen-feature evaluation and finetuning. We also present ablation experiments including training objectives and architecture designs.
4.1 Training Setup
Data. As discussed in Section 3.2, CoCa is pretrained from scratch in a single stage on both web-scale alt-text data and annotated images by treating all labels simply as texts. We use the JFT-3B dataset (Zhai et al., 2021a) with label names as the paired texts, and the ALIGN dataset (Jia et al., 2021) with noisy alt-texts. Similar to Pham et al. (2021a), we randomly shuffle and concatenate label names of each image in JFT together with prompt sampled from Radford et al. (2021). An example of the resulting text label of a JFT image would look like "a photo of the cat, animal". Unlike prior models (Zhai et al., 2021b; Pham el al., 2021a) that also use the combination of these two datasets, we train all model parameters from scratch at the same time without pretraining an image encoder with supervised cross-entropy loss for simplicity and pretraining efficiency. To ensure fair ovaluation, we follow the strict de-duplication procedures introducod in (Zhai et al., 2021b; Jia al., 2021) to filter all near-domain examples (3.6M images are removed in total) to our downstream tasks. To tokenize text input, we use a sentence piece model (Sennrich al., 2015; Kudo, 2018) with vocabulary size of 64k trained the sampled pretraining datasct.
Optimization. Our models are implemented in the Lingvo framework (Shen et al., 2019) with GSPMD (Huang et al., 2019; Xu et al., 2020; Lepikhin et al., 2020; Xu et al., 2021) for scaling perfor- mance. Following (Pham et al., 2021a), we use a batch size of 65,536 image-text pairs, where half of each batch comes from JFT and ALIGN, respectively. All models are trained on the combined contrastive and captioning objectives Eq.(4) for 500k steps, roughly corresponding ALIGN. As shown later in our studies, we find a larger captioning loss weight is better and thus λCap = 2.0 and Acon 1.0. Following Jia et al. (2021), apply contrastive loss with trainable temperature with initial value of 0.07. For memory efficiency,
epochs on JFT and 10 epochs on
use the Adafactor (Shazeer & Stern, 2018) optimizer with 31 = 0.9, 32 = 0.999 and decoupled weight decay (Loshchilov & Hutter, 2017) ratio of 0.01. We warm up the learning rale for the first 2% of training steps to a peak value of 8 x 10-1, and linearly decay it afterwards. Pretraining CoCa takes about days on 2,048 CloudTPUv4 chips. Following Radford et al. (2021); Jia et al. (2021); Yuan et al. (2021), we continue pretraining for one epoch on a higher resolution of 576x576. For finetuning evaluation, we mainly follow simple protocols and directly train CoCa on downstream tasks without further metric-specific tuning like CIDEr scores (details in Appendix A and B).
4.2 Main Results
We extensively evaluate the capabilities of CoCa models on wide range of downstream tasks as a pretrained foundation model. We mainly consider core tasks of three categories that examine (1) visual recoguition, (2) crossmodal alignment, and (3) image captioning and multimodal understanding capabilities. Since CoCa produces both aligned unimodal representations and fused multimodal embeddings at the same time, is easily transferable all three task groups with minimal adaption. Figure 4 summarizes the performance key benchmarks of CoCa compared to other dual-encoder and encoder-decoder foundation models and
7

---

Published in Transactions on Machine Learning Research (08/2022)
Imogelett
NoCape
Kinetics- 400
T22
118
NE
Kinetice 600
20.5
SNU-VE
646
Zero-shot
ImagoNet
78
49
87.
VQA
Zero zero-shot MSCOCD((1)
SOTA
(speciolized
Florence
(dual-encader)
SImVIM
Zeto-shot
(encoder-decode)
(121)
MSCOCO(12)
CoCa
Zero-shot
Flickr30K (127)
Figure 4: Comparison of CoCa with other image-text foundation models (without task-specific customization) and multiple state-of- the-arr task-specialized models.
Model
ImageNet
Model
K-400
K-600
K-700
Moments-in-Time
ALIGN
88.6
VIVITS
84.8
84.3
38.0
Florence
90.1
MoViNet
81.5
84.8
79.4
40.2
LataeseudoLabels
90.2
VATT
83.6
CoAtNet"
90.9
Florence
86.8
88.0
VIT-G"
90.5
MaskFeat
87.0
88.3
80.4
Model Soups
90.9
CoVeR!
87.2
87.9
78.5
46.1
CoCa (Irozen)
90.6
CoCa (frozen)
88.0
88.5
81.11
47.4
CoCa (finetuned)
91.0
CoCa (finetuned)
88.9
89.4
82.7
49.0
Table 2: Image classification and video action recognition with frozen encoder finetuned encoder. Model reference: *(Jia et al., 2021) "(Yuan et al., 2021) "(Pham et al., 2021b) 4(Dai et al., 2021) "(Zhai et al., 2021a) 5(Wortsman et al., 2022) 8(Arnab et al., 2021) "(Kondratyuk et al., 2021) '(Akbari et al., 2021) k(Wei et al., 2021) (Zhang et al., 2021a).
hato-of-tho-art task-spocialized methods. CoCa sets new statc-of-tho-art results on tasks of all three categorics with single pretrained checkpoint.
4.2.1 Visual Recognition Tasks
Our visual recognition experiments are conducted on ImageNet (Deng et al., 2009) as image recognition benchmark, and multiple vidco datascts including Kinetics-400 (Kay et al., 2017), Kinctics-600 (Carreira al., 2018), Kinetics-700 (Carreira et al., 2019), Moments-in- in-eettt----mee Time (Monfort al., 2019) as test-beds for video action recognition; noteworthy that CoCa pretrains image data only, without accessing any extra
8

---

Published in Transactions on Machine Learning Research (08/2022)
pOCa
coca
91
SwI12
Forence
Iropetet Top Accurscy (98)
Imaegetae Top-1 Aecurnry 199
82
80
FuP
AUGG
76
87
74-
1000
2500
Panamis (14llamm))
Parama (Millions)
(a) Finetuned ImageNet Top-1 Accuracy.
(b) Zero-Shot ImageNet Top-1 Accuracy.
Figure 5: Image classification scaling performance of model sizes.
Flickr30K (IK test set)
MSCOCO (5K
test set) Text
Image
Text
Text
Image
Image + Text
Image
Model
R@l
R
R@10
R@1
R@5
R@10
R@1
R@5
R@00
R@1
R@5
R@
CLIP (Radford et al., 2021)
88.0
98.7
99.1
68.7
90.6
95.2
58.4
81.5
88.1
37.8
62.4
72.2
ALIGN (Jia et al., 2021)
88.6
98.7
99.7
75.7
93.8
96.8
58.6
83.0
89.7
45.6
69.8
78.6
FLAVA (Singh et al., 2021)
67.7
94.0
65.2
89.4
42.7
76.8
38.4
67.5
FILIP (Yao et al., 2021)
89.8
99.2
99.8
75.0
93.4
96.3
61.3
84.3
90.4
45.9
70.6
79.3
Florence (Yuan et al., 2021)
90.9
99.1
76.7
93.6
64.7
85.9
47.2
71.4
CoCa- Base
89.8
98.8
99.8
76.8
93.7
96.8
84.7
90.7
47.5
72.4
80.9
CoCa-Large
91.4
99.2
99.9
79.0
95.1
97.1
65.4
85.6
91.4
50.1
73.8
81.8
CoCa
92.5
99.5
99.9
80.4
95.7
97.7
66.3
86.2
91.8
51.2
74.2
82.0
Table 3: Zero-shot image-text retrieval results on Flickr30K (Plummer et al., 2015) and MSCOCO (Chen
et al., 2015) datasets.
video datasots. We apply the CoCa encoder video frames individually (Section 3.3) without carly fusion of temporal information, yet the resulting CoCa-for-Videe model performs better than many spalio-temporal early-fused video models.
Frozen-feature. We apply a pretrained frozen CoCa model on both image classification and video action recognition. The encoder used for both tasks while the decoder is discarded. discussed Section 3.3, an attentional pooling learned together with a softmax cross-entropy loss layer on top of the embedding outputs from CoCa encoder. For vidco classification, single query-token is learned weight outputs of all tokens of spatial patches X temporal frames. We set learning rate of 5 x 10 both attentional pooler and softmax, batch size of 128, and a casine learning rate schedule (details in Appendix A). For video action recognition, we compare CoCa with other approaches on the same setup (i.e., without extra supervised video data and without andio signals as model inputs). As shown in Table 2, without finetuning full encoder, CoCa already achieves competitive Top-1 classification accuracies compared to specialized image and outperforms prior berf-performing specializod methods on vidco tasks.
Finetuning. Based on the architecture of frozen-feature evaluation, further finetune CoCa encoders image and video datasets individually with smaller learning rate of 1 10-4, More experimental details are summarized in the Appendix A. The finctuned CoCa has improved performance across these tasks. Notably, CoCa obtains 91.0% Top-1 accuracy on ImageNet, as well as belter video action recognition results compared with recent video approaches, More importantly, CoCa models use much less parameters than other methods in the visual encoder as shown in Figure 5a. These results suggest the proposod framework efficiently combines text training signals and thus able to learn high-quality visual representation better than the classical single-encoder approach.
9

---

Published in Transactions on Machine Learning Research (08/2022)
Model
ImtgeNet
ImageNet-A
ImageNet-R
ImageNNt-VV2
ImageNet-Skeich
ObjuctNel
Average
CLIP (Radford ct al., 2021)
76.2
77.2
88.9
70.1
60.2
72.3
74.3
ALIGN (Jia et al., 2021)
76.4
75.8
92.2
70.1
64.8
72.2
74.5
FILIP (Yso et al., 2021)
78.3
Florence (Yuan al., 2021)
83.7
LIT (Zhai al., 2021b]
84.5
79.4
78.7
81.1
BASIC (Pham et al., 2021a)
85.7
85.6
95.7
80.6
76.1
78.9
83.7
CoCa-Base
82.6
76.4
93.2
76.5
71.7
71.6
78.7
CoCgeLarge
84.8
85.7
95.6
79.6
75.7
78.6
83.3
CoCa
90.2
80.7
77.6
82.7
85.7
Table 4: Zero-shot image classification results on ImageNet (Deng et al., 2009), ImageNet-A (Hendrycks el al., 2021b), ImageNet-R, (Hendrycks et al., 2021a), ImageNet-V2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019) and ObjectNet (Barbu et al., 2019).
4.2.2 Crossmodal Alignment Tasks
Unlike other fusion-based foundation methods (Wang et al., 2021b; Singh al., 2021; Wang al., 2022), CoCa is naturally applicable to crossmodal alignment tasks since it generates aligned image and text unimodal embeddings (see Appendix C for results on video-text retrieval). In particular, zero-shot setting where all parameters are frozen after pretraining and directly used to extract embeddings. Here, we use the same embeddings used for contrastive loss during pretraining, and thus the multimodal text decoder is not used.
are interested in the
Zero-Shot Image- Text Retrieval. We evaluate CoCa on the two standard image-text retrieval benchmarks: MSCOCO (Chen al., 2015) and Flickr30K (Plummer et al., 2015). Following the CLIP setting (Radford et al., 2021), first independently feed each image/text to the corresponding encoder and obtain embeddings for all image/text in the test set. We then retrieve based on cosine similarity scores over the whole test set. As shown in Table 3, CoCa significantly improves over prior methods on both image-to-text and text-to-image retrievals on all metrics. In addition, our model is parameter--ffieennt, with CoCa-Base already outperforming strong baselines (CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021)) and CoCa-Large outperforming Florence (Yuan et al., 2021) (which contains a parameter count comparable to ViT-Huge). This shows that CoCa learns good unimodal representations and aligns them well across modalities.
Zero-Shot Image Classification. Following prior work (Radford al., 2021; Jia et al., 2021), aligned image/text embeddings to perform zero-shot image classification by matching images with label names without finetuning. We follow the exact setup in Radford et al. (2021) and apply the same set of prompts used for label class names. As shown in Table 4, CoCa. sets new state-of-the-art zero-shot classification results on ImageNet. Notably, CoCa uses fewer parameters than prior best model (Pham et al., 2021a) while smaller CoCa variants already outperform strong basclines (Radford al., 2021; Yuan et al., 2021), as shown in Figure 5b. In addition, our model demonstrates effective generalization under zero-shot evaluation, consistent with prior findings (Radford et al., 2021; Jía et al., 2021), with CoCa improving on all six datascts considered. Lastly, while prior models (Zhai et al., 2021b; Pham et al., 2021a) found sequentially pretraining with single-encoder and dual-encoder methods in multiple stages is crucial to performance gains, our results show it is possible to attain strong performance by unifying training objectives and datasets in a single-stage framework.
the
4.2.3 Image Captioning and Multimodal Understanding Tasks
Another key advantage of CoCa is its ability to process multimodal embeddings an encoder-decoder model trained with the generative objective. Therefore, CoCa can perform both image captioning and multimodal understanding downstream tasks without any further fusion adaptation (Shen et al., 2021; Dou et al., 2021). Overall, experimental results suggest CoCa reaps the benefit of a encoder-dccodr model obtain strong multimodal understanding and generation capabilities, in addition the vision and retrieval capabilities as a dual-encoder method.
10

---

Published in Transactions on Machine Learning Research (08/2022)
VQA
SNLI-VE
NIVR2
Model
test-dev
test-std
dev
test
dev
test-p
UNITER (Chen et al., 2020)
73.8
74.0
79.4
79.4
79.1
80.0
VinVI. (Zhang et al., 2021b)
76.6
76.6
82.7
84.0
CLIP-ViT. (Shen et al., 2021)
76.5
76.7
80.6
80.2
ALBEF (Ti et al., 2021)
75.8
76.0
80.8
80.9
82.6
83.1
BLIP (Li et al., 2022)
78.3
78.3
82.2
82.2
OFA (Wang et al., 2022)
79.9
80.0
90.31
90.2f
VLMo (Wang et al., 2021a)
79.9
80.0
85.6
86.9
SinVLM (Wang al., 2021b)
80.0
80.3
86.2
86.3
84.5
85.2
Florence (Yuan et al., 2021)
80.2
80.4
METER (Dou et al., 2021)
80.3
80.5
CoCa
82.3
82.3
87.0
87.1
86.1
87.0
Table 5: Multimodal understanding results comparing vision-language pretraining methods, 1OFA uses both image and text premises as inputs while other models utilize the image only.
MSCOCO
NoCaps
Valid
Test
B@4
M
C
S
S
CLIP-ViL (Shen et al., 2021)
40.2
29.7
134.2
23.8
BLIP (Li et al., 2022)
40.4
136.7
113.2
14.8
VinVL(Zhang et al., 2021b)
41.0
31.1
140.9
25.4
105.1
14.4
103.7
14.4
SimVLM (Wang et al., 2021b)
40.6
33.7
143.3
25.4
112.2
110.3
14.5
LEMON (Hu al., 2021)
41.5
30.8
139.1
24.1
117.3
15.0
114.3
14.9
LEMONscST (Hu et al., 2021)
42.6
31.4
145.5
25.5
OFA (Wang et al., 2022)
43.5
31.9
149.6
26.1
CoCa
40.9
33.9
143.6
24.7
122.4
15.5
120.6
15.5
Table Image captioning results on MSCOCO and NoCaps (B04: BLEU@4, M: METEOR, C: CTDEr, S: SPICE). "Models finetuned with CIDEr optimization.
Multimodal Understanding, As shown in Wang et al. (2021b), the output of encoder-decoder models can jointly encode image and text inputs, and can be used for tasks that require reasoning over both modalities. We consider three popular multimodal understaning benchmarks: visual question answering (VQA v2 (Goyal et al., 2017)), visual entailment. (SNLI-VE (Xie el. al., 2019)), and visual reasoning (NLVR2 (Suhr al., 2018)). We mainly follow the settings in Wang al. (2021b) and train linear classifiers on top of the decoder outputs to predict answers (more details in Appendix B). Our results in Table 5 suggest that CoCa outperforms strong vision-language pretraining (VLP) baselines and obtains the best performance three tasks. While prior dual-encoder models (Radford et al., 2021; Yuan et al., 2021) do not contain fusion layers and thus require an additional VL pretraining stage for downstream multimodal understanding tasks, CoCa subsumes the three pretraining paradigms and obtains better performance on VL tasks with lightweight finetuning.
all
Image Captioning. In addition multimodal classification tasks, CoCa is also directly applicable image captioning tasks as encoder-decoder model. We finetune CoCa with the captioning loss LCap only on MSCOCO (Chen et al., 2015) captioning task and evaluate both MSCOCO Karpathy-test split and NoCaps (Agrawal et al., 2019) online evaluation. As shown by experiments in Table 6, CoCa outperforms strong baselines trained with cross-entropy loss on MSCOCO, and achieves results comparable to methods with CIDEr metric-specific optimization (Rennie et al., 2017). is noteworthy that we do not use CIDEr-specific optimization (Rennic et al., 2017) for simplicity. On the challenging NoCaps benchmark, CoCa obtains better results both validation and test splits (generated examples shown in Figure 6). These results showcase the generative capability of CoCa as image-text foundation model.
11

---

Published in Transactions on Machine Learning Research (08/2022)
NEW
hand holding a sen francisco 49ers football
row of cerrons with the eiffel tower the background
a while van with license plate that flynn
person sitting on wooden bridge holding umbrella
truck is reflected the side mirror of car
Figure 6: Curated samples of text captions generated by CoCa with NoCaps images as input.
loss
LE
FT
loss
VQA
TPU cost
Acap
ACon
ZS
VQA
Coss
81.0
85.1
Con
70.7
59.2
1:1
71.5 68.6
CCap
82.1
84.9
CCap
68.9
1.17x
1:2
71.0 68.1
LcoCa
71.6
69.0
1.18x
2:1
71.6 69.0
(a) Encoder-decoder vs. singlo-cncoder models (trained on JFT).
(b) Training objectives ablation.
(c) Training objectives weights.
Pani
ZS
VQA
variant
AE
MSCOCO
variant
ZS
VQA
70.2
69.0
[CLS]
80.7
41.4
parallel
71.2
68.7
6
71.6
69.0
+ text tokens 80.3
40.2
cascade
71.6
69.0
71.4
68.8
[CLSS]
80.3
36.9
Aquuery 0
71.5
69.0
text tokens 80.4
40.3
Aquuayy
69.3
64.4
Aquuery
32
71.2
68.2
(d) Unimodal decoder layers.
(c) Coutrastive text embodding design ablation.
(f) Attentional pooler design ablation.
Table 7: CoCa ablation experiments. On ImageNet classification, report top-1 accuracy for: zero-shot (7S), linear evaluation (LE.), attentional evaluation (AE) using pooler on frozen feature, and finetuning (FT). On MSCOCO retrieval, we report the average of image-to-text and text-to-image R@l. On VQA, we report
the dev-set vqa score. The default CoCa setting is bold.
4.3 Ablation Analysis
We extensively ablate the properties of CoCa on a smaller model variant. Specifically, we train CoCa- CoCa-Base with a reduced 12 decoder layers and a total batch size of 4,096. We mainly evaluate using zero-shot image classification and VQA, since the former covers both visual representation quality and crossmodal alignment, while the later representative for multimodal reasoning.
Captioning us. Classification. We first examine the effectiveness of captioning loss image annotation datasets. To do this, we train naive mccder-decoder model using LCap the JFT-3B dataset, and compare with a standard ViT-Base single-encoder model trained with LCss in Table 7a. We find encoder-decoder models to perform on par with single-encoder pretraining on both linear evaluation and finetuned results. This suggests that the generative pretraining subsumes classification pretraining, consistent with our intuition that Lcia a special case of CCap when text. vocabulary the set of all possible class names. Thus, our CoCa model can be interpreted as an effective unification of the thrce paradigms. This explains why CoCa does not need a pretrained visual encoder to perform well.
Training Objectives. We study the effects of the two training objectives and compare CoCa with single- objective variants in Table 7b. Compared to the contrastivo-only model, CoCa significantly improves both zero-shol alignment and VQA (notice that the contrastive-only model requires additional fusion for VQA). CoCa performs on par with the captioning-only model on VQA while additionally enables retrieval-style tasks such as zero-shot classification. Table 7c further studies loss ratios and suggests that the captioning loss not only improves VQA but also zero-shot alignment between modalities. We hypothesize that generative objectives learn fine-grained text representations that further improve text understanding. Finally, we compare
12

---

Published in Transactions on Machine Learning Research (08/2022)
training costs in Table 7b (measured in TPUva-core-days; larger is slower) and find CoCa to be as efficient the captioning-only model na.v...nave encoder-decoder with same architecture as CoCa) due to the sharing of compute between two objectives. These suggest combining the two losses induces new capabilities and better performance with minimal extra cost.
Unimodal Textual Representation. CoCa introduces a novel decoder design and we ablate its components. In Table 7d, we vary the number of unimodal decoder layers (while keeping the total number of layers the same). Intuitively, fewer unimodal text layers leads to worse zero-shot classification due to lack of capacity for good unimodal text understanding, while fewer multimodal layers reduces the model's power to reason over multimodal inputs such as VQA. Overall, we find decoupling the decoder in half maintains good balance. One possibility is that global text representation for retrieval doesn't i require deep modules (Pham et al., 2021a) while early fusion for shallow layers may also be unnecessary for multimodal understanding, Another key design of unimodal textual representation the application of [CLS] tokens. In particular, we experiment with the number of learnable [CLS] tokens as well as the aggregation design. For the later, we aggregate over either the [CLS] lokens only (denoted as N [CLS]) the concatenation of [CLS] and the original input. sentence (denoted as N [CLS] text tokens). Interestingly, in Table 7e we find training single [CLS] token without the original input is preferred for both vision-only and crossmodal retrieval tasks. This indicates that learning an additional simple sentence representation mitigates interference between contrastive and captioning loss, and is powerful enough for strong generalization.
Attentional Poolers. CoCa exploits attentional poolers in its design both for different pretraining objectives and objective-ppecifee downstream task adaptations. In pretraining, we compare a few design variants on using poolers for contrastive loss and generative loss: (1) the "parallel" desiggn which extracts both contrastive and generative losses at the same time on Vision Transformer encoder outputs shown in Figure 2, and (2) the "cascade" design which applics the contrastive pooler on top of the outputs of the generative pooler. Table 7f shows the results of these variants. Empirically, we find at small scale the "cascade" version (contrastive pooler on top of the generative poolcr) performs better and is used by default in all CoCa models. We also study the effect of number of queries where Aquery = 0 means no generative pooler is used (thus all VIT output tokens are used for decoder cross- atss-attention). Results show that both tasks prefer longer sequences of detailed image tokens at cost of slightly more computation and parameters. As result, we use generative pooler of length 256 to improve multimodal understanding benchmarks while still maintaining the strong frazen-feature capability.
5 Conclusion
In this work we present Contrastive Captioners (CoCa), a new image-text foundation model family that subsumes existing vision pretraining paradigms with natural language supervision. Pretrained on image-text pairs from various data sources in a single stage, CoCa efficiently combines contrastive and captioning objectives in an encoder-decoder model. CoCa obtains series of state-of-the-art, performance with single checkpoint on wide spectrum of vision and vision-language problems. Our work bridges the gap among various pretraining approaches and we hope it motivates new directions for image-text foundation models.
Broader Impact Statement
This work presents an image-text pretraining approach on web-scale datasets that is capable of transferring to wide range of downstream tasks in a zero-shot manner or with lightweight finetuning. While the pretrained models are capable of many vision and vision-language tasks, note that our models use the same pretraining data as previous methods (Jia et al., 2021; Zhai et al., 2021a;b; Pham et al., 2021a) and additional analysis of the data and the resulting model is necessary before the use of the models in practice. We show CoCa models are more robust on corrupted images, but it could still be vulnerable to other image corruptions that are not yet captured by current evaluation sets or in rcal-world scenarios. For both the data and model, further community exploration is required to understand the broader impacts including but not limited to fairness, social bias and potential misuse.
13

---

Published in Transactions on Machine Learning Research (08/2022)
References
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1948-8951 2019.
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong.
Valt: Transformers for multimodal sell-supervisee learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34, 2021.
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of- distribution robustness throughout fine-tuning. arXiv preprint arXiv:2106. 0et-2106.188911 15831, 2021.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6836-6846, 2021.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08255 2021.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Paul Barham, Aakanksha Chowdhery, Jelf Dean, Sanjay Ghemawat, Steven Hand, Dan Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, et al. Pathways: Asynchronous distributed dataflow for ml. Xiv preprint arxiv:2209. 12533, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiu preprint arXim:2188. 07258, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Joao Carreira, Eric Noland, Andras Banki-Horvall Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint 1rxi::1808 01340, 2018
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 700 human action dataset. arXiv preprint arXiv:1907.069877 2019.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint ar X50: 00325, 2015.
Yen-Chur Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying viston-and-language tasks via text generation. In International Conference on Machine Learning, pp. 1931 1942. PMLR, 2021.
Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on compuler vision and pallern recognition, pp. 248 255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint ar arxiv:18100 0810.04805, 2018.
14

---

Published in Transactions on Machine Learning Research (08/2022)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Usækoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference
on Learning Representations, 2021. URL https://openrevien wttp://wwwe..etttttuuu/t////o//. net//foruriiceV.ccaaaTYYy
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang, Lijuan Wang, Chenguang Zhu, Zicheng Liu, Michael Zeng, et al. An empirical study of training end-to-end vision-and-languaage transformers. arviv preprint arxiv:2111. 11.02887 2021.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6903, 2017.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In Proceedings of the IEEE conference on compuler vision and pallern recognilion, pp. 770 778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXie preprinl or xxx:22111.003777, 2021.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340 8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. ar preprint ar xrxiv:21 1 11. 12233, 2021.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Woojcong Jin, Yu Cheng, Yclong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of pa- rameters? low-resource prompt-based learning for vision-language models. arXiv preprint arxiv:2110 trXiv:21000844.. 08.4844, 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arxio: 1705006950 2017.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution region supervision, 2021.
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong, Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 16020-160330 16030, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. Xiv preprint or arXiu:1804... 10959, 2018.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invarian neural networks. In International Conference on Machine Learning, pp. 3744 3753. PMLR, 2019.
Dmitry Lepikhin, HyoukJoong Lec, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16663 2020.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty; Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified lision-language understanding and generation. arXiv preprint arXiv:2201.1080. 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation ation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431 3431-3440, 2015.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1xxi::1111.051 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming Hc, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions paltern analysis and machine indelligence, 42(2):502 508, 2019.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoe V. Le. Combined scaling for open-vocabulary image classification, 2021a. URL https: htttp:://axiiv wwwg attpaab//..1 10050.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. In Proceedings of the IEEE/CV Conference on Computer Vision and Pottern Recognition, pp. 11557 11568, 2021b.
AJ Picrgiovanni, Wei Li, Weicheng Kuo, Mohammad Saffar, Fred Bertsch, and Anclia Angelova. Answer-me: Multi-lask open-vocabulary visual question answering. Xiv preprint xxxi:2205500949. 2022.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641 2641-2649, 2015.
Jesús Andrés Portillo-Quintero, José Carlos Ortiz-Bayliss, and Hugo Terashima-Marn. A straightforward framework for video retrieval using clip. ar preprint arXiv:2102. 12443, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 9748-8763 PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint
2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 3389-5400.. PMI.R, 2019.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal wetworks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: pdf.
/ //proceedings...... wwtpeeedinnn...uuriieee
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava. Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- -0204 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation rare words with subword units. XrXiv preprint orXi::1008. 500.000980790, 2015.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604. PMLR, 2018.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, et al. Lingvo: a modular and scalable framework for sequence-too-eequence modeling, 2019.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit aabio--addlangguag tasks? arXiv preprint 1xXt:2207.06388, 2021.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in newral informalion processing systems, 27, 2014.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment. model. arXiv preprint wrXiu:2112.04498 2021.
Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. A short note the 700-2200 human action dataset. arXiv preprint arXi::200011088 10864, 2020.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded photographs. arXiv preprint arXiv:1811.00499 2018.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. 1 1908.008.00 07490, 2019.
Xiviv preprint Xiv:
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34: 200 212, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3156 3164, 2015.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust. global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to-scquenee learning framework. orXiv preprinl ar arXiv:2202.0000 2022.
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vasion-language pre-training with mixture-o- XiXio preprint or Xiv:211 1111101588, 2021a.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. aix preprint orXiu:2108. 10904, 2021b.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training- arXiv preprinl arXiv:2112.0900 091333, 2021.
Ronald J Williams and David Zipser. A learning algorithm for contimually running fully recurrent neural networks. Newral computation, 1(2)) 280, 1989.
Mitchell Wortsman, Gabriel Iharro, Samir Yitzhak Gadre, Rebecca Roclofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korublith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2203.00 05440488 2022.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arx preprint arXiv:1901.06700, 2019.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:2111. 111.0088,, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging vidco and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-588 5296,, 2016.
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Hongjun Choi, Blake Hechtman, and Shibo Wang. Automatic cross-replica sharding of weight update data-parallel training. arXiv preprint arx::20004. 19336, 2020.
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et al. Gspmd: general and scalable parallelization for ml computation graphs. arXiv preprint arxiv:2105.0 05664663, 2021.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-labe space, 2022a.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp. 3081 3089, 2022b.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fino-grained interactive language-image pro-training. arXiv preprint or 211.::2111.07783, 2021.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprinl ar Xrxiv:2111 11432, 2021.
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint or Xii:2204.00559 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2021a. URL https: http://xxxvv..grge wwwpp/wwww...///////....// 200.b./0000000000.... 04560.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning, arXiv preprint arXiv:2111.07991, 2021b.
Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. arXiv preprinl ftu:2112.07177 2021a.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in languuel models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5579 0579-5588. June 2021b.
18

---

Published in Transactions on Machine Learning Research (08/2022)
A
Visual Recognition Finetuning Details
ImageNet
(inettcc-4400600/// Frozen-feature Finetuning
Moments-in-Time
Hyper-parameter Frozen-feature Finetuning
Finetuning
Optimizer
Adafacter with Decoupled Weight Decay
Gradient clip
1.0
EMA decay rate
0.9999
LR decay schodule
Cosine Schedule Decaying to Zero
Loss
Soflmax
MixUp
None
CutMix
None
AutoAngment
None
RegmatedAAgment
None
Ranmengment
2,
2,
None
None
None
None
Label smoothing
0.2
0.5
0.1
0.1
0.0
0.0
Train steps
200k
200k
120k
120k
120k
120k
Train batch size
512
512
128
128
128
128
Pooler LR
5e-4
5e-4
5e-4
5e-4
5e-4
5e-4
Encoder LR
0.0
5e-4
0.0
5e-4
0.0
5e-4
Warm-up steps
0
0
1000
1000
1000
1000
Weight decay rate
0.01
0.01
0.0
0.0
0.0
0.0
Table 8: Hyper-parameters used in the visual recognition experiments.
In addition to zero-shot transfer, evaluate frozen-feature and finetuning performance of CoCa on visual recognition tasks. For frozen-feature evaluation, add attentional pooling layer (pooler) top of the output sequence of visual features and additional softmax cross entropy loss layer to learn classification of images and videos. For finetuning, we adapt the same architecture as frozen-feature evaluation (thus also with poolers) and finetune both encoder and pooler. All learning hyperparameters are listed in Table 8.
B Multimodal Understanding Finetuning Details
Harareper
VQA
SNLI-VE NLVR2 MSCOCO NoCaps
Optimizer
Adafacter with Decoupled Weight Decay
Gradient clip
1.0
LR decay schedule
Cosine Schedule Decaying Zero
RandAugment
1, 10
1, 10
None
None
None
Train steps
100k
50k
50k
50k
10k
Train batch size
64
128
64
128
128
Pooler LR.
5e-4
le-3
5e-4
NA
NA
Encoder LR
2c-5
5e-5
2e-5
le-5
le-5
Warm-up steps
1000
1000
1000
1000
1000
Weight decay rate
0.1
0.1
0.1
0.1
0.1
Table 9: Hyper-parameterr used in the multimodal experiments.
CoCa is encoder-decoder model and the final decoder outputs be used for multimodal understanding/ generation. Thus, we evaluate on popular vision-language benchmarks. We mainly follow the same setup introduced in Wang et al. (2021b). All haperepsr are listed in Table 9.
For multimodal classification, we feed the image into the encoder and the corresponding text to the decoder. We then apply another attentional pooler with single query extract embodding from the decoder output, and train linear classifier on top of the pooled embedding. For VQA v2 (Goyal al., 2017), we follow prior work and formulate the task classification problem over 3, 2,129 most. frequent answers in the training set.
19

---

Published in Transactions on Machine Learning Research (08/2022)
MSR-VTT Full
Text
Video
Video
Text
Method
R@1
R
R@10
R&1
R
R:100
CLIP (Portillo-
Quintero PPotiilo-Quiatero al,, 2021)
21.4
41.1
50.4
40.3
09.7
79.2
Socratic Models (Zeng al., 2022)
44.7
71.2
80.0
CLIP (PortilloQuuiatero al, 2021] (subset)
23.3
44.2
53.6
43.3
73.3
81.8
Socratic Models (Zeng et al., 2022) (subset)
46.9
73.5
81.3
CoCa (subset)
30.0
52.4
61.6
49.9
73.4
81.4
Table 10: Zero-shot Video- Text Retrieval on MSR-VTT Full test set.
We additionally enable coiraining with the generative loss on the concatenated pairs of textual questions and answers to improve model robustness. Similarly for SNLI-VE, the image and the textual hypothesis are fed to encoder and decoder separately, and the classifier is trained to predict the relation between them as entailment, neutral contradiction. For NLVR2, we create two input pairs of each image and the text description, and concatenate them input to the classifier. We do not use image augmentation for NLVR2.
For image captioning, apply simple cross-entropy loss (same as the captioning loss used in pretraining) and finetune the model on the training split of MSCOCO to predict for MSCOCO test split and NoCaps online evaluation. We use beam search with beam size of 4 for our experiments.
C Zero-Shot Video Retrieval
We evaluate video-text retrieval using CoCa on MSR-VTT (Xu et al., 2016) using the full split. Table 10 shows that CoCa produces the highest retrieval metrics for both texl-lo-video and video-lo-text retrieval. It is important to note that MSR-VTT videos are sourced from YouTube, and we require the original videos to compute our embeddings. Many of the vidcos have been made explicitly unavailable (Smaira et al., 2020), hence compute retrieval over the subset of data that is publicly available at the time of evaluation. Using code provided by the authors of Socratic Models (Zeng et al., 2022), re-computed metrics on the available subset for those methods, indicated by "(subset)" for fairest comparison.
httpp:///occaaticoodlsss git.hu......g.t..i......
20


Reviews:
[{"review_id": "uPKTPbsmu1", "reviewer": "Reviewer_gXWo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nOverall, this was a good read for me. The paper is clearly written and easy to follow. The presentation of method and results are also in a good flow. The evaluation of the proposed method is also quite sufficient. The model has been empirically shown to achieve strong performance over many downstream tasks, with abundant ablation studies to verify the design choices of the final model.\n\nWeakness:\n\nNo major weakness, please refer to section below for some minor questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a simple combination of contrastive loss and captioning loss to pre-train an image-text foundation model, with a ender-decoder architecture, dubbed CoCa. To make contrastive loss to work along with captioning loss,  CoCa keeps the first half of the decoder for unimodal language encoding and the later half for multimodal fusion. CoCa can be viewed as an unification of previous work CLIP and SimVLM in both model architecture and pre-training losses. This simple unification enables CoCa to achieve superior performance on a wide range of downstream tasks with zero-shot transfer or minimal task-specific adaptation, including image/video classification, multimodal retrieval/understanding and image captioning. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- For the comparison of training costs in table 7b, does the captioning only model (referred as naive encoder-decoder architecture in the discussion paragraph) contain a unimodal text decoder? If not, how may layers are used in multimodal decoder for this captioning model? \n", "broader_impact_concerns": "Discussions about some potential societal impact are included in the paragraph following the conclusion section, which I believe is sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qddf3noOYD", "reviewer": "Reviewer_u8gt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The new architecture omits cross-attention in the first half of decoder layers to encode unimodal text representations and cascades the remaining decoder layers that cross-attend to the image encoder for multimodal image-text representations, which is efficient for training. \n\n2. The training objectives jointly combine contrastive loss and captioning loss, taking both benefits from contrastive approaches and generative approaches. \n\n3. The empirical results are good. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a new method for image-text pretraining. Particularly, the proposed CoCa exploits an encoder-decoder architecture and combines contrastive and captioning objectives. On several downstream tasks, CoCa achieved state-of-the-art performances with zero-shot transfer or minimal task-specific adaptation. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. The ablation studies show that the contrastive loss helps little for the VQA task and has not shown it can help ZS. The paper should add more experiments to demonstrate the effectiveness of the contrastive loss. \n\n2. The paper missed some related and important literatures. They have to be cited and compared. \na. An empirical study of gpt-3 for few-shot knowledge-based vqa. A\nb. A Good Prompt Is Worth Millions of Parameters? Low-resource Prompt-based Learning for Vision-Language Models\nc. Multimodal Few-Shot Learning with Frozen Language Models\nd. Unifying Vision-and-Language Tasks via Text Generation", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "40fKc2cWI9", "reviewer": "Reviewer_VEHW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper proposes the contrastive captioning method that naturally merges two pre-training frameworks for multimodal research. The split-decoder design can bring the contrastive learning and generative pre-training together without corrupting the input data. Another benefit of this specific design is that it can be easily adapted to different kinds of tasks, e.g., it does not require to repurpose the models for captioning tasks (OSCAR needs to shift the tokens for generative tasks, CLIP needs another training round).\n\n2. The paper shows a very strong results on diverse types of tasks, such as image recognition, video recognition, multimodal retreival, multimodal text generation, and multimodal understanding. The paper also present results under different experimental setups, e.g., zero-shot, frozen-feature adaptation, and full fine-tuning. This quantitive evaluation on downstream tasks is comprehensive. Btw, I like the figure style of Fig.6, which is clear and tidy.  \n\n3. The paper shows the benefit of multitasking, which is not obvious according to previous works. For examples, in VilBERT 12-in-1, the separate trained models still show stronger performance than the multitask model. Thus CoCa provides a good examples that a careful design of task combination can bring significant results over original models.\n\n4. Although the ablation studies of the paper is not exhaustive, they include most of the bullet that I care about. Some results are also inspiring and worth reading. For example, the paper shows that the captioning loss with comma-separated labels can be an alternative to the standard softmax classification loss. To the best of my knowledge, I did not see such results in previous literature. \n\n5. The paper provides a comprehensive list of hyper-parameters used in the paper, which would help reproduction of the results in the paper. \n\n\n## Weaknesses\n\nThe work is mostly complete to me and here are some improvements that I can think of. \n\n0. No code/model releasing plan is specified in the paper, which I considered as a loss to the community. It would be great if a smaller model trained on public dataset can be released. \n\n1. The pre-training of the model takes both generative tasks and contrastive tasks in a single run. However, in the downstream evaluation, only one of these tasks are utilized for most of the cases. It might be better to consider both tasks in downstream as well. For example, the image classification tasks can be formulated as a text-matching problem (as demonstrated in this paper) but can also be reformulated as a class-name generation problem, thus the later formulation can effectively leverage the. The same to the VQA task, where only discriminative approach is considered. Utilizing the generative modules may also extend the work to (semi-) open-domain VQAs, such as text-vqa, OK-VQA. \n\n2. The visual tasks with spatial localization is not evaluated in this paper. I would think that the model would have the capacity to handle the tasks like object detection/segmentation since the cross-attention layer is included. A plain ViT adaption (like [1]) to detection tasks would be a great improvement to this work. \n\n3. As a multimodal model, the model is proposed in an asymmetrical way where the specific designs are on the language sides. I would think that these designs can also be brought to the vision side by utilizing a similar framework.  \n\n[1] Li, Yanghao, Hanzi Mao, Ross Girshick, and Kaiming He. \"Exploring plain vision transformer backbones for object detection.\" arXiv preprint arXiv:2203.16527 (2022).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In general, the paper brings a new way to combine the two existing methods: generative pre-training and contrastive pre-training. I am impressed by the ways that the method showing good qualitative results on downstream tasks and better training efficiency. Here is a summary of the contributions:\n\n1. Proposing the contrastive captioner method, which combines generative pre-training and contrastive pre-training. The model design splits the language model into two parts: a unimodal text decoder and a multimodal text decoder. Although splitting decoder has been seen in previous works (by specifying the layers with cross-attentions), the exact way of using it to utilize a single unimodal feature (for contrastive learning) and next-word logits (for conditional language modeling) is novel to me. \n\n2. As a unified model, the paper shows strong performance on different types of tasks, including visual recognition (classification), multimodal retrieval, and multimodal understanding. The adaptation to these downstream tasks are natural and achieve sota results on them.\n\n3. The model is proved at large scale. Usually, the combination of two kinds of training paradigm is hard to survive in large scale since one paradigm might overshadow the other. Thus, showing the success at large scale can entail that these two tasks are complementary to each other. \n\n4. A detailed ablation studies are provided regarding some key designs of the model, such as whether the captioning less can used to replace the classification loss. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I think that the paper looks good to me and I have several questions regarding the details. All these questions are considered as minor to me and they would not affect my decision to this paper.\n\n1. The description regarding the \"learnable CLS tokens\" (about Table 7 (e)) is not clear enough to me. I am not sure what the paper means by the four variations in Table 7 (e). The connection from the \"unimodal decoder layers\" to \"text embedding design\" is also not obvious given the text in \"Unimodal and multimodal decoders\" paragraph. I would suggest a rewriting here.\n\n2. In Sec 3.3, \"we also encode the captions of each video as target embeddings\" is not clear to me. Is it the standard retrieval pipeline that encodes the text to embeddings and do nearest neighbor search?\n\n3. There are several questions regarding the attention pooler. (a) What is the number of queries when applying to the image classification task? Is it a feature_map --> attention pooler with n_cls queries --> softmax NLL loss design or a feature_map --> attention pooler with 1 query --> FC to n_cls --> softmax NLL loss design? (b) In Figure 4.a, there is a comparison between CoCa and other works on ImageNet top-1 accuracy. Since CoCa uses attention pooler in its classification head, I would like to know the classification head for other works such as SwinV2, ViT, CoAttNet. (c) At the end of sec 4.3 and Table 7.f, it shows that using attention pooling of 256 queries shows similar results as 0 quires (thus directly using the original visual features). Sec 3.2 shows that the visual feature numbers are 16x16=256, thus I would like to see more explanation about the need of an additional attentional pooler to 256. \n\n4. Does the image captioning evaluation use beam search or just greedy decoding? What the number of beams if so.\n\n5. Missing citations for the decoupled weight decay paper, \"Decoupled Weight Decay Regularization\" in Sec 4.1, \"optimization\".\n\n\n", "broader_impact_concerns": "I do not see any. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
This paper introduces Contrastive Captioners (CoCa), a new family of multi-modal foundation models. CoCa enables large scale multi-modal pre-training and combines generative pre-training and contrastive pre-training. In terms of architecture, Coca subsumes single-encoder, dual-encoder and encoder-decoder paradigms, and uses a decoupled decoder without any cross-attention in the unimodal text decoder. 

The submission received positive reviews, with two accept recommendations and one leaning accept recommendation. The reviewers pointed out the relevance of the combination of the generative and contrastive pre-trainings, as well as the decoupled decoder architecture. The claims are also strongly experimentally validated, since the approach reaches very good performances on several downstream tasks, *e.g.* visual recognition (classification), cross-modal alignment, image captioning and multimodal understanding. Numerous ablation studies also validate the proposed design choices. 
The authors successfully answered to the main reviewers concerns during the discussion period.

The AE carefully reads the submission and agrees that the decoupled decoder architecture is novel and that the experimental validation is strong. 
The AE thus recommends paper acceptance under the two minor changes/answers: 
- The references provided by Ru8gt (Yang et al., 2022b; Jin et al., 2021; Tsimpoukelli et al., 2021) have been included in the related work section ; it would be nice to position Coca more precisely with respect to these recent baselines. 
- RVEHW raised the question of the diffusion of open source code and of model release. The AE indeed considers that it would be an important added valued to the community. 

Decision: Accept (poster)