Image Captioners Are Scalable Vision Learners Too
Authors: Michael Tschannen
Manoj Kumar
Andreas Peter Steiner
Xiaohua Zhai
Neil Houlsby
Lucas Beyer
Venue: NeurIPS 2023 oral (2023)
Keywords: contrastive learning
CLIP
CapPa
Cap
vision-language
image captioning
visual representation learning
weakly supervised learning
VLM
multimodal learning
VQA
image classification
TL;DR: We present an extensive comparison of contrastive representation learning and representation learning via image captioning from large image-text data sets.

Abstract: Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones, especially in the context of large multimodal models. At the same time, image captioning on this type of data is commonly considered an inferior pretraining strategy. In this paper, we perform a fair comparison of these two pretraining strategies, carefully matching training data, compute, and model capacity. Using a standard encoder-decoder transformer, we find that captioning alone is surprisingly effective: on classification tasks, captioning produces vision encoders competitive with contrastively pretrained encoders, while surpassing them on vision & language tasks. We further analyze the effect of the model architecture and scale, as well as the pretraining data on the representation quality, and find that captioning exhibits the same or better scaling behavior along these axes. Overall our results show that plain image captioning is a more powerful pretraining strategy than was previously believed. Code is available at [https://github.com/google-research/big_vision](https://github.com/google-research/big_vision).

Image Captioners Are Scalable Vision Learners Too
Michael Tschannen Xiaohua Zhai
Manoj Kumar Andreas Steiner Lucas Beyer"
Neil Houlsby Google DeepMind
Abstract
Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones, especially in the context of large multimodal models. At the same time, image captioning on this type of data is commonly considered inferior pretraining strategy. In this paper, we perform a fair comparison of these two pretraining strategies, carefully matching training data, compute, and model capacity. Using a standard encoder-decoder transformer, we find that captioning alone is surprisingly effective: on classification tasks, captioning produces vision encoders competitive with contrastively pretrained encoders, while surpassing them vision & language tasks. We further analyze the effect of the model architecture and scale, as well as the pretraining data on the representation quality, and find that captioning exhibits the same better scaling behavior along these axes. Overall our results show that plain image captioning is more powerful pretraining strategy than was previously believed.
Introduction
Contrastive language image pretraining (CLIP) [50] has recently become a very popular pretraining strategy, enabling scalable vision-language pretraining billions of image-text pairs collected from the web. CLIP not only enables zero-shot transfer to arbitrary image classification and retrieval tasks, but also produces high-quality vision backbones rivaling the best label-supervised ones [14]. The corresponding checkpoints [50, 9, 58] are among the most powerful publicly available vision backbones, enjoying wide adoption in the context of multi-modal vision-language models, where often pretrained vision backbone is combined with a pretrained language model [1, 6, 62, 23, 17, 5].
Before contrastive image-text pretraining popularized by CLIP [50] and ALIGN [26], number of works explored generative image captioning [13, 54] and n-gram prediction [27, approaches for representation learning small scale. However, [50] showed that the zero-shot classification performance contrastive pretraining scales much better than captioning as a function of the number of training examples seen (see [50, Fig. 2]). Since then, follow-up works of [50] which focus on pretraining from scratch do usually not rely on captioning alone, but augment it with contrastive loss [66, 34, 40]. Those follow-up works based generative captioning alone typically rely on cross-modal encoders with early fusion of image and text, targeting visual question answering (VQA) and/or captioning tasks with the full encoder-decoder system [63, 22, 10], and do not study the properties of the vision backbone alone.
We revisit image captioning as pretraining task for learning general vision encoders from web image-text pairs. We compare CLIP-style models with image captioning ones consisting of Vision Transformer (ViT) [15] encoder and standard Transformer decoder, henceforth simply referred to as Captioner (Cap). In our experiments, we carefully match pretraining compute and model capacity, and train both models for the same number of examples.
"Significant technical contributions. 'TT led the project. Code is available at htpps//gituub.coo/googgoo howpa//boooooooooooooeeaaahhiiaa vision.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Contractive lods
D]
Crass- cass--ntropyy Loss
Crass-entropy Loss
Cross atin
(causal) Self-altn maak
Cross attn
N2
Self-atin mack
Visian Transfomer
Transformer Encoder
Visien Transformer
Transtormer Decodar
Vision Transforer
Transtormer Decodar
cow
"A small
Figure Contrastive models (left) use two separate Transformer encoders extract vector repre- sentations from image-text pairs, which are then matched across a potentially large batch [50]. Cap (middle) uses a Transformer encoder-decoder architecture [60] and predicts text tokens autoregres- sively. During training, all tokens are predicted in parallel by shifting the expected output by one token and applying a causal self-attention mask (teacher forcing). parallel decoding (right) the Transformer decoder has to predict all tokens at once, conditioned only the image. CapPa trains single model switching between autoregressive and parallel decoding.
D: model width, M: number of image tokens, N: number of text tokens, V: vocabulary size.
While our results confirm the findings of [50] that Cap models generally lag contrastive ones in zero-shot classification accuracy, the gap becomes smaller with increasing scale. More importantly, the Cap vision backbone matches or outperforms comparable CLIP models in few-shot classification and obtains competitive performance when transferring to classification tasks with large labeled data sets. When combining the vision encoder with randomly initialized transformer decoder, the Cap pretrained encoder outperforms the CLIP one on captioning, OCR, and VQA tasks. This indicates that Cap vision backbone might be better suited for multimodal downstream tasks. These benefits can be even more pronounced when combining the vision encoder with a pretrained, partially frozen language decoder similar to [1, 6, 62, 23, 17, 5].
We further propose the CapPa pretraining procedure: the decoder training alternates between standard autoregressive prediction (Cap) and parallel prediction (Pa), where the entire caption is predicted a single decoder forward pass. This only changes the decoder text input (all input tokens are set to [MASK] tokens) and self-attention mask (no mask instead of a causal mask) while requiring no change in the loss or architecture. This mixed training, termed CapPa, significantly improves classification accuracy of the vision backbone.
We ablate how various training and architecture design choices impact performance, and discover promising scaling properties of captioners. Finally, we show that Cap achieves state-of-the-ar perfor- mance in zero-shot classification tasks which require careful handling of word order and attributes, rather than treating the query as a bag-of of mag-of-wordd words, as measured by ARO [67] and SugarCrepe [21].
Overall, results show that pretraining simple encoder-decoder architecture via image captioning alone can produce vision encoders competitive with CL.P and presents an interesting alternative to contrastive pretraining- particular for multimodal vision-lanee saiig-la models built form pretrained vision and language modeling components.
2 Related work
Large scale contrastive vision-languse pretraining was popularized by CLIP [50] and ALIGN [26]. Several works investigated scaling beyond these works along several relevant axes [49, 69] using pretrained vision and language backbones with contrastive training [70].
Recent works investigating plain image captioning for pretraining are [13, 54]; [27, 37] study gram prediction from images, which can considered precursor captioning. However, image captioning as pretraining task to learn general vision representations did not attract much attention as contrastive training, possibly because of inferior and less efficient zero-shot transfer capabilities. Related to captioning, Sim VLM [63] uses prefix language modeling to pretrain multimodal encoder-

---

decoder model with early vision-languee fusion and hybrid convolutional/rrasffoomm vision encoder, targeting transfer to VQA and captioning. Further, encoder-decoder models for document, website, and UI understanding are often pretrained to generate captions from rendered websites/documents which trains the model to extract features from those data types and perform OCR [36, 29, 38]. Focusing on image captioning only, LEMON [22] investigates scaling of an encoder-decoder model.
Several works have combined contrastive and captioning losses [66, 40, 34], optionally using a separate text encoder in addition to the decoder. While obtaining excellent results on range of vision and vision- langun-e tasks, the impact of the loss terms are not well disentangled, nor are comparisons of individual losses provided.
Image captioning is popular ingredient to build large multimodal models from separately pretrained vision and language models [1. 6, 62, 23, 17, 5]. Some of these models freeze large parts of vision and language components, sometimes only learning small adapter between the two [1, 17, 39]. It is interesting to see how the pretraining strategy for the vision model affects this type of adaptation.
Finally, masked image-text modeling, often combined with image-text matching, is popular pre- training strategy for encoder-only vision-language models at small data scale [24, 30, 41, 64, 56, 16].
3 A simple recipe to pretrain vision encoders via image captioning
Our goal is to establish pretraining recipe based on image captioning that is comparable to CLIP in terms of simplicity, scalability, and efficiency. Therefore, we adopt Vision Transformer (ViT) [15] backbones vision encoders, and use a standard Transformer decoder architecture [60] to predict image captions, feeding the ViT-encoded sequence to the decoder via cross-attention As is common in recent literature [52, 11], we remove biases from attention layers, MLP blocks, and LayerNorms and we replace ReLU by GELU The decoder input and output embeddings are not shared. Ablations for these choices are in Section 4.4. The decoder has the same width, attention- heads, and MLP dimension as the encoder, but has half the depth. This leads to captioning models which have slightly lower parameter count than corresponding CLIP models, but require about the same pretraining compute in terms of accelerator hours per epoch (see Table 1). We rely on standard next-token- prediction language modeling and train our captioning models with causal attention mask and teacher forcing (Fig. 1, middle), henceforth referring to this variant simply as Captioner (Cap).
Parallel prediction We also experiment with a slightly modified training recipe (Fig. 1, right): Instead of training the model only for autoregressively, we train it to predict all tokens in parallel for a fraction of the training examples instead (sampling the prediction type for every example at random throughout training). To this end, we replace the (shifted) decoder input sequence with a sequence of all [MASK] tokens, and drop the causal decoder attention mask. We emphasize that this kind of parallel prediction does not modify the training objective or decoder architecture, and does not incur any extra training cost, but simply modifies the decoder input sequence and attention mask for a fraction of training examples. Moreover, this is different from bag-of-wort word prediction as not only the presence but also the position of each token has to be predicted. We perform parallel prediction for 75% of the training examples by default and term this variant Cap with parallel prediction (CapPa).
Intuitively, captioning via next token prediction induces an implicit weighting on the supervisory signal of the caption tokens: To predict the first few tokens of a caption, the decoder can benefit a lot from using the image information, while to predict later tokens it can rely more and more on already predicted tokens. Consequently, early tokens might provide a stronger supervisory signal to the encoder than later tokens. By contrast, when predicting all the caption tokens independently in parallel, the decoder can only rely on the image information to predict each token.
Implementation aspects We emphasize that our approach closely follows a standard en- coder/decoder transformer architecture [60], with the only fundamental difference being the input data format and patch embedding, as well as the modification of the decoder input sequence and attention mask for parallel prediction. This means that our approach is easy to implement in existing transformer code bases [52, 47, 57]. We do not rely on image-specific preprocessing operations other than resizing, see Section 4.1. As result, existing infrastructure for model-paralle, and dis- tributed training can readily be used to scale our approach. In particular, our loss does not require computations across devices the way CLIP does.
3

---

Table 2: Performance of frozen visual representations trained via image captioning (Cap/CapPa) and contrastively (CLIP"), when combined with a single transformer decoder trained from scratch for image classification, captioning and VQA (we CIDEr for captioning, the VQAv2 weighted accuracy for VQAv2, and exact matching accuracy for all other tasks). Bold marks results where two standard-deviation interval overlaps with the two standard-deviation interval of the best result.
Classification
Captioning
OCR
Question Ans
ilk
sun
food
res
pet
COCO
Flickr
VQA
VQAv2
GQA
Cap
80.2.0.1
82.3.02
90.3
03.620.1
93.1
117.5
78.6
62.2.101
68.2
55.0=0.1
CapPa
81.3=01
82.4+a1
90.9
94.2+a2
94.4
117.9 #0.6
80.5
62.2.
68.3+a0
55.7=002
CLIP (8k)
81.1=00
83.2
91 2002
94.8=a0
93,4202
115.8202
74.5
56.020.1
66,5
54.3203
CLIP (16k)
81.4#01
83.3
92.0
95.2+002
93.6+02
116.3
77.1 +007
56.5 +o.1
66.7+
44.8
CLIP
81.6.0.1
82.5
92.6.0.1
94.9.0..
93.61:01
118,4 78.7
60.0
67.8
57.5
CapPa L/14
84.4:00.
84.9.0.1
93.8.100
96.0
95.6:100
125.8.0.1 89.3
65.61101
70.9:000
58.3::022
CLIP# L/14
84.7=0.1
85.7
94.6=001
96,4
95.2+01
123.2#0o
85.5
61.3
68.5
55.3=01
CLIP L/14
84.8=0.0
84,8
95. 1 96.3 95.4203
124.420. 87.1
64. 14.0
70.420. 58.7.0.
Zero-shot classification via scoring Image captioning models allow for zero-shot classification simply by scoring the class name. Unlike with contrastive models, where the text (class) embeddings can be computed once and reused for new images, with captioning models all class names have to be scored again for every new image. Cap/CapPa are hence less efficient zero-shot classifiers than CLIP. We emphasize that we focus on learning vision encoders here and zero-shot transfer is not our focus.
4 Experiments
4.1 Experiment setup
Pretraining data We use a subset of the WebLI data set
Table 1: Parameter count and
[6] which contains 10B images and 12B multilingual alt-texts. Specifically, we rely the WebLI subset corresponding to
TPrv4-hrs. per bn. examples seen.
English websites and apply text-based filtering similar to [26, Sec. 31, [4, Sec 2.2] obtain IB image/alt-text pairs, not using any image-text similarity- based filtering. Importantly. WebLI was de de-duplicated w.r.t. the images in the evaluation data sets we use in this paper. Please refer to [6, See 2.2] for more details on the I data set and to 16, Appendix B] for datasheet.
Model
Params
Trr-
B/16 Cap
192 M
454
B/16 CLIP#
197 M
444
L/14 Cap
570 M
1570
1/14 CLIPP
640
1596
Pretraining details and baselines' We use batch size of 8k for our captioning models (larger batch size did not lead to improved performance). and both 8k and 16k for our retrained CLIP baselines (henceforth denoted by CLIP* to avoid confusion with the model checkpoints released by [50], which we reference by CLIP in our results). We explicitly note the batch size for CLIP* models when relevant; CLIP* without modifier refers to the variant trained with batch size 16k. Models are trained on up to 9B image/alt-text pairs (corresponding to 9 epochs on our subset of WebLI). We use the AdaFactor variant from [68] with cosine schedule (with 10k warmup steps), and set learning rate and decay factor to 10 3 and 10-4, respectively. Previous work [70, 69, 59] established these optimizer settings for contrastive pretraining and we adopt them here for captioning. Images are resized to a resolution of 224 x 224, and alt-texts are tokenized to a 32k-sized vocabulary with a sentence piece model trained on the English portion of C4 [52], with maximum sequence length of 64. Following [70, 69, 59] for CLIP* we use identically sized image and text towers, and use global average pooling (GAP) compute the image representation.
4.2 Evaluation protocols and data sets
We focus on properties of the frozen representations and also present some fine-tuning results.
'Code is available at
vision.

---

Table 3: 10-shot linear evaluation accuracy on the pre-logit representation. CapPa outperforms Cap and achieves overall comparable results with CLIP trained with a batch size of 16k.
Table 4: Frozen transfer for zero-shot classifica-
tion and retrieval via LiT [70]. Especially for the larger model, CapPa is competitve with CLIP"
with comparable or fewer LiT examples seen.
INet
CIFARI00
Pets
Cars
INet Oshot
COCO 12i
COCO i2t
Cap
57.2.
58.6208
83.7=06
84.2 1.01
LiT pairs:
3B
12B
3B
12B
3B
12B
CapPa
59.1+ea
62.4.0.
86.5 0.1
86.6+02
Cap
67.8
69.0
37.5
39.1
53.9
54,8
CLIP (8k)
58.5
64.9
77.7=1.5 .
80.8
CapPa
68.8
70.2
37.3
38.6
53,9
55.1
CLIP* (16k)
59.7
66.3
80.6-1.2
82. 10.1
CLIP
69.0
70.0
38.9
40.1
55.1
57.0
CLIP
59.0102
68 68.6.10.
82.1
70.2
CLIP
68.3
32.3
52.8
CapPa L/14
70.6.
72.9
92.6_0.5
92.2:002
CapPa L/14
76,4
77.5
43.9
45.4
60.3
62.6
CLIP# L/14
69.8
74.1
87.7
89.2+ao
CLIP* L/14
75.8
76.6
44.7
46.3
60.7
62.3
CLIP L/14 69.7.al
79.4
90.4=08 81.1
CLIP L/14 75.1
36.5
56.6
Probing the frozen representation As an inexpensive way to assess classification performance we use the linear adaptation protocol (based on the pre-logits layer for CLIP and GAP of the encoder output sequence for our captioning models) and eval sets from [68, 70], reporting the 10-shot classification accuracy. We also evaluate the classification accuracy when using the full ImageNetlk training set learn a dense layer, an MLP, and a multihead attention pooling (MAP)-based [35] classifier. To assess the amenability of the different the CLIP* and CapPa vision encoders to fine- grained classification, we train MAP-based classifiers on a range of specialized data sets which can be divided in two groups. The first group requires fine-grained classification of animal or plant breed [45, 61, 48, 28], or product variant [32, 31, whereas data sets in the second group covers a range of distinct objects [18, 12, 8, 71, 46, 19].
inference We use LiT [70] to learn a text embedding matched to the embedding of our pretrained vision encoders. Generally, LiT is an efficient way to equip any pretrained vision backbone with zero-shot classification and retrieval capabilities, here particularly for Cap whose pretrained decoder does in principle have these capabilities but incurs significant inference cost. We follow the setup from [69] and assess the zero-shot classification accuracy on ImageNetlk [53] and retrieval recall @1 COCO captions [7]. We choose the LiT text encoder to mirror the architecture of the vision encoder at hand and attach a randomly initialized MAP head to the vision encoder map into the shared image-text embedding space. For comparison, also apply LiT to our CLIP* models; this is not necessarily meant to be practically useful (continuing training the CLIP* model might be a better investment of compute).
Text encoder/decoderr-berl
Motivated by recent work combining pretrained vision backbones and language models [1, 6, 62, 23, 17, 5]. we investigate the amenability of the learned representations to interface with a text decoder. Concretely, we use the "LiT decoder" setup [2] which trains a transformer decoder from scratch on top of frozen vision representation to solve captioning [7, 65]. VQA [20, 25, 43] and classification [53, 71, 8, 48] tasks in a multitask fashion (we use the default hyperparameters from [2] except for the data mixing strategy set to "concat image-question pairs" [2, Sec. 5.3) )
In addition, we explore combining our representations with a pretrained T5 decoder [52]. We rely on the previously described multitask LiT decoder setup and tune the most important hyper parameters (see supplementary material for details). For the T5 decoder we keep all the parameters frozen but reinitialize and train the cross-attention layers. Finally, we also leverage a frozen, pretrained 12-layer GPT-2 decoder [51] ] for image captioning by combining it with the frozen vision encoders via an adapter, similar to ClipCap [44].
Fine-tuning We fine-tune our vision encoders on the full ImageNet1k data set, attaching a randomly initialized MAP head to the pretrained representation (see supplementary material for details).
Using the pretrained text decoder Finally, we also evaluate our models (with the pretrained text decoder) on the SugarCrepe [21] and the Attribute, Relation and Order (ARO) [67] benchmarks (which are derived from different captioning data sets [33, 65, 7D. Specifically, SugarCrepe and ARO shuffle the attributes, relations and order of captions and measures the sensitivity of rision-language
5

---

ImageNet
Caption
VQAv2
70
70-
120
CapPa
66
CLIP
40
70
110
M
B
1/14
M
L/14
5
M
L/14
5
M
1/14
F1.OPs (enooder)
(encoder)
(encoder)
(encoder)
ImageNet 10-shot
Pets 10-shot
COCO Caption
VQAv2
90
125
70
CapPa
CLIP
68
80
115
Arch
66
B/16
L/14
39 training examples scon
900M
training examples seen
training examples soza
training oxamples soon
Figure 2: 10-shot classification accuracy on the frozen pre-logit representation (left two columns); captioning and VQA performance with decoder (right two columns). Top row: Performance of vision backbones pretrained with captioning (Cap/CapPa) and contrastively (CLIP*) ) as a function of the model size/FLOPs (we compare ViT-S/16, M/16, B/16, and L/14). CapPa exhibits favorable scaling behavior on captioning and VQA tasks. Bottom row: Performance of CapPa and CLIP" as function of the number of training examples seen. The behavior similar as for model scale.
models to these manipulations, As shown by [67], contrastive models are not very sensitive to precise relational and attributional information and behave closer to bag-of-words models. Intuitively, since captioning- based models model the joint distribution over all tokens, it is interesting to see how they compare contrastive models on ARO. For each example, we use log-liielhooo to score both the true caption and the shuffled captions. The model prediction is the caption with the highest score.
4.3 Main results
Tables 2, 3, 4, and Fig. show the LiT decoder results, the 10-shot classification accuracy, the LiT tuning results, and the full linear probing accuracy for our models trained for 9B examples seen.
Table 5: Finetuning on INet.
B/16 B/16
L/14336
CLIP* 84.9
86.0
88.1
CapPa outperforms Cap across almost all evaluations, and often also outperforms CLIP* trained with the same batch size (8k), while being competitive with CLIP* trained with a batch size of 16k. This
CapPa 84.4
85.7
87.7
Cap
83.9
85.3
remarkable since CLIP" benefits substantially from a large batch size [69]. The trend becomes
more pronounced when increasing the model size: CapPa clearly outperforms CLIP* in 10-shot classification accuracy for ViT-L/14 encoder (Table 3).
When transferred to ImageNetl k classification with linear probe (Fig. 3), the frozen Cap and CapPa encoders lag somewhat behind CLIP" but the gap almost vanishes when using MAP head instead of a linear probe. This is not very surprising, given CLIP* models trained with a linear head, which might induce linear sep- arability the average pooled encoder output representation. In contrast, the Cap models feed into a decoder via cross-attention which might not impose linear separability as strongly. For fine-tuning the full model on ImageNetlk (Ta- ble 5) we only observe minor gap between CapPa and CLIP". As for text encoders learned via LiT (Table 4) CapPa outperforms CLIP* for long LiT training and large model size in zero-shot classification, but outperformed by CLIP* in retrieval. Notably, for short LiT tuning with examples models outperform CLIP [50] in zero-shot classification and retrieval while matching
CLIPs better Cap
-1
-2
Linear probe
MLP probe
MAP probe
B/16 Caplaa
L/11 CapPa.
Figure 3: Linear probing makes cap pre-trained image encoders seem worse, but when learning the pooling (MAP probe), the gap is essentially closed.

---

Attrib.
Rel.
Order (F) Order (C)
Training
Arch
Repl.
Swap
Add
Blind dec.
83.7
86.2
98.8
98.7
Cap
B/16
88.21
84.00
98.94
Cap
88.9
86.6
99.1
99.0
CapPa
B/16
87.67
83.11
99.13
CapPa
85.7
86.7
99.2
98.8
CLIP"
B/16
81.95
63.22
81.91
CLIP"
53.2
39.7
45.5
37.0
NegCLIP
B/32
85.36
75.33
87.29
CLIP
62.7
58.7
57.9
49.5
OpenCLIP
G/14
86.50
68.56
88.36
ARO Best
88.0
73.0
60.0
46.0
Table 7: Results the SugarCrepe [21] bench- mark, which fixes known issues previous image-text benchmarks like ARO. Full results are in Table 18 the appendix. Small Cap and CapPa models outperform even large hard- negative trained CLIP models in all categories, and essentially solve the category of tests which "Add"s plausible but wrong details to the caption.
NegCLIP 71.0
81.0
91.0
86.0
Table 6: Results on the Attribute, Relation and Order (ARO) benchmark [67]. Cap and CapPa clearly outperform all CLIP* and CLIP variants across all data sets. They also outperform Neg- CLIP [67] which was explicitly trained to be sensitive to word ordering and attribution.
the number of examples seen by CLIP (12.8B) when summing over pretraining (9B) and LiT tuning (3B), despite the vision encoder being frozen during LiT tuning (which saves compute). Matching the number examples seen by CLIP during LiT tuning (12B) leads a clear additional improvement.
Combining our models with a fresh (LiT) decoder (Table 2), we observe that CapPa performs better than CLIP" trained with batch size 16k captioning and VQA, while obtaining competitive accuracy on classification tasks. Again, this pattern becomes more pronounced with increased models size. Indeed, for ViT-L/14 encoder CapPa outperforms CLIP in the majority of LiT decoder tasks.
Scaling properties In Fig. we present an analysis of our encoders as a function of the model size and the number of training examples seen for a selection of 10-shot classification and vision language evaluations using fresh decoder (plots for all evaluations and Cap can be found in the supplementary material). It can be seen that CLIP" and CapPa models exhibit similar scaling behavior, with CapPa showing somewhat steeper increase captioning and VQA performance as function of model size and examples seen, in particular for a ViT-L/14 backbone. This indicates that benefits CapPa models might become more pronounced with further model and data scaling.
Attribution, relation, ordering Table presents our results on the ARO Benchmark [67]. Cap and CapPa models achieve close to perfect score on the ordering subsets. They outperform the best contrastive variants by around 30% and 40% on Flickr Order and COCO Order, respectively.
We also compare with NegCLIP [67]. which employs additional supervision to make contrastive models sensitive to word ordering and attribution. Cap and CapPa exceed NegCLIP by 8% and 13% out-of-the-box COCO Order and Flickr Order. The same be observed for the attribution, and relation subsets of ARO. So we are facing a clear trade-off: CLIP-style models outperform captioning- based models in terms of standard zero-shot classification accuracy, whereas the latter are much better at processing fine- seee-gaaiedd descriptions of visual content. Finally, we also trained
CLIP= - better - CapPa
+10
LiT-deeooder (scratch)
Fine-grained
T5-decoder (frozen)
Coarse-grained
+-5
ImagoNet
Food10)
Flickr VQAv2
Birds
STL-10
SUN397
Cultech
SUN397
RESISCAS
OCR-VQA
GQA
Cars
CarDog
RESISC
Figure Absolute improvement of CapPa over CLIP* in various settings. Left: CapPa pairs significantly better with decoders in image-language tasks, especially when the decoder is a pre- trained and frozen language model. Right: CapPa seems be noticeably better frozen feature extractor for ane-grained classification tasks (we show L/14 results, see appendix for B/16).
7

---

Cap model with no image input (Blind dec.). which is just language model for alt-text. This model overall performs only slightly worse than Cap and CapPa, which suggests that the sentence manipulation in ARO can to a large extent be detected by language modeling alone. This was also observed in concurrent work [42,21].
SugarCrepe The SugarCrepe benchmark [21], introduced concurrently to our work, promises to fix the issues in ARO and similar benchmarks; for instance, blind model no better than chance. Still, even our small captioning pretrained B/16 model outperforms the largest G/14 contrastive model as well as the specifically trained NegCLIP. The "Swap" category is especially sensitive the relation between multiple things in the image, something that is fundamentally hard for contrastive pretraining to learn. The "Add" category. where highly plausible but wrong things are added to the text is essentially solved by captioners. The full breakdown, more baselines, and qualitative examples are provided in Tables 18 and 22-24 in the appendix. This result is strong evidence that captioning as pretraining objective imbues capabilities to the model that contrastively trained models are blind to.
Frozen T5 decoder We also trained models with frozen encoder and frozen decoder (initialized with a T5 checkpoint). For these experiments, only the cross attention weights were trained (28M trainable parameters, compared to the 248M trainable parameters when the decoder trained from scratch). The relative improvements of CapPa over CLIP" are shown in Fig. 4 (left). Even though the absolute performance of the decoder trained from scratch (Table 2) is better than when only training the cross attention weights (Table 12), CapPa with frozen decoder closes the gap three classification tasks, reverts the trend on ImageNet, and improves the performance by large margins on Pets, as well as all captioning and VQA tasks. This result suggests that the captioning objective is better suited to train an encoder that is later combined with a pretrained language decoder.
Frozen GPTT-2 decoder We combined our frozen vision encoders with frozen GPT-2 model from [51] using an adapter as proposed by ClipCap [44]. We found that this setup is less well suited for VQA and multitask conditioning, so we only evaluate it on captioning in single-task fashion as done by ClipCap. We tried the MLP and trans- former adapters from [44], but obtained better results for both CLIP* and CapPa with "resampler", LayerNorm followed by single cross-attention layer with learnable query embeddings, generating prefix of 32 soft tokens for GPT-2. This resampler has only 7.1M parameters, about 4 4x less than the MIP adapter from [44]. CapPa modest
CLIPs better - CapPa
8-
LiT-d.
6-
T5-d.
GPT2
COCO Caption
Flickr Caption
Figure 5: Absolute improve- ment (single-task) of CapPa over CLIP* for a decoder trained from scratch (LiT-d.), a frozen T5 decoder, and a frozen GPT-2 similar to [44].
still outperforms CLIP" in this setup, but the gains than for the T5 decoder and a decoder trained from scratch (both single-task, see Fig. 5). We emphasize that both encoder and GPT-2 decoder are frozen and we only use small adapter.
Fine-grained classification Fig. 4 (right) shows the improvement when using CapPa instead of CLIP* for a range of specialized data sets. CapPa outperforms CLIP* on the majority of fine-grained tasks which suggests that captioning as a pretraining task leads to better features for fine-grained classification than contrastive pretraining.
4.4 Ablations
All models discussed in this section are trained for 900M training examples seen.
Parallel prediction Recall that for parallel prediction we replace all text input tokens with [MASK] tokens. An alternative would be to only replace a random subset, as done e.g. in [22], to provide a partial context for the prediction. However, we did not observe improvements of the vision encoder when only masking a fraction of the tokens, so we focused on fully masked input sequences. For fully masked input sequence Fig. 6 (left) shows the improvement in 10-shot classification accuracy over training for pure autoregressive decoding as a function of the fraction of examples for which the decoder is trained for parallel prediction instead. A fraction of 0.75 leads to a balanced improvement across all considered data sets. Finally, alternating between parallel and autoregressive prediction for all examples in a batch, rather than performing parallel prediction with mixed batches, led to significantly worse results.
8

---

Cap - better - CapPa
+2
CLIP better
5
ResNet-50 (p)
NexNet-50
ImageNet CIFARI00
Pets Cars
VIT-R/32
-15
25%
50%
75%
909
ImageNet
ImageNet
Fraction of batches with parallel prediction
10-shot
zero-shot
Figure Left: Improvement in 10-shot classification accuracy as function of the fraction of training examples for which parallel prediction is performed in CapPa. A fraction of 0.75 leads to balanced improvement. Right: Improvement of 10-shot/zero-she shot (without prompts) classification accuracy when using Cap instead of CLIP". For ResNet-50 (p) the decoder consumes 4 averaged image tokens prefix (no cross-attention). Cap competitive in 10-shot accuracy for ViT-B/16.
Encoder architecture Next, we investigate the effect of the encoder architecture on the repre- sentation quality, comparing CLIP" with Cap when using BiT ResNet50 [31], ViT-B/32, and ViT-B/16 vision encoder. We use B-sized text encoder and B-sized 6-layer decoder for CLIP and Cap, respectively, for all encoders, except for ResNet50 for which we also train with a prefix decoder following [50, Sec. A2]. According to [15] the BiT ResNet50 obtains performance roughly comparable to ViT-B/32 when trained and evaluated on image classification. Further, ResNet50 and ViT-B/32 both produce sequence of length 49 before pooling for 224 224 images, which we feed to the decoder. Fig. (right) shows the improvement obtained when using Cap instead of CLIP as function of the encoder architecture. The improvement (regression) in ImageNet zero-shot accuracy is smallest (largest) for the ResNet50 architecture (this is the architecture used to create [50, Fig. 2] that compares contrastive with bag-of-words and captioning approaches) and is significantly improved when using a ViT architecture and when reducing the patch size (which does not increase model capacity but the sequence length and hence FLOPs). Also recall that these models were only trained on 900M examples. Interestingly, the difference between CLIP" and Cap on 10-shot metrics generally smaller, and for ViT-B/16 encoder the two approaches lead to similar performance.
Captioning task variants We train Cap while randomly reversing the caption with probability 0.5. This maintains model capacity and pretraining compute unchanged. We do not observe improved performance (see Table 8, left). While [13] ablates backwards captioning and shows improvements, they use separate decoder for backwards captioning, so the ablated model has fewer parameters and FLOPs (here we control for both factors). Additionally, decoders, one for autoregressive prediction and another one for parallel prediction, each with 3 layers (instead of 6). This model matches the pretraining compute of the default CapPa but underperforms in the majority 10-shot tasks.
train CapPa variant with two parallel
Training with trained T5-Base decoder (which has 12 decoder layers). To obtain a stable training setup we adapt the optimizer apper-parameters (learning rate 0.0005, 82 = 0.95) and unfreeze the cross attention. Optionally, we re- re-initialize the cosssattention parameters and unfreeze the decoder. None of these variants performs better than Cap overall (see Table 8, right), and the more allow the decoder to deviate from its
decoder Finally, we train Cap ViT-B/16 with frozen pre-
weights the better the vision performance gets.
Table 8: Left: Comparing Cap and CapPa with other captioner variants: Forward and backward captioning with the same decoder, autoregressive and parallel prediction with 3-layer decoders. Right: Training Cap with a frozen T5 decoder does not help, even when unfreezing parts of it.
ImageNet
CIFAR100
Pets
Cars
ImageNet CIFAR100
Pets
Cars
Cap
49.7
56.0
72.6
74.7
Cap (12 layers)
48.7
54.8
74.4
73.8
Cap (fw.+bw.)
49,2
56.1
71.7
73.0
+ frozen T5 dec.
42,8
44.9
69.3
62.3
CapPa dec.
49.5
54.9
75.8
79.0
reinit. xatt.
43.7
45.7
68.3
66.9
CapPa
50.4
57.4
76.2
78.5
unfreeze dec.
48.6
55.2
72.0
75.6

---

Decoder architecture While following the original transformer decoder architecture [60] closely, we adopt the now common change of removing the biases the decoder [52, 11] to improve stability without affecting performance, see Table 19 (left) in the appendix. We use separate embeddings for the decoder's input and output, Table 19 (left) shows that this works slightly better. We use six decoder layers (see Table 19, right) which simultaneously leads to overall good results while also matching the total parameter count the corresponding CLIP* model.
Inspired by experiments from [68], where applying a stronger weight decay to the classification head of ViTs trained for image classification led accuracy improvements, we also experimented with increased weight decay applied to the decoder or cross-attention layers, but we did not observe any benefits from this. Further, we explored using tokenizer trained on 300M randomly sampled WebLI alt-texts instead of the one pretrained on C4, which did not improve accuracy.
Effect of pretraining data based on models pretrained on a variant of WebLI, and one might wonder whether our findings transfer to pre- training on other data sets. We therefore train some of our models and baselines on the smaller, publicly available LAION-400M dataset [55] which was collected and fil- tered following a different protocol. For instance, was filtered using existing CLIP model to score image-text pairs, which might induce a significantly different bias in the training data and our conclusions. However, the 10-shot linear classification results in Fig. show that the conclusions remain the same: CapPa achieves accuracy comparable with CLIP* and outperforms Cap.
So far all our results were
100
CLIPP (8k)
10-shot Accuracy [%]
Cap
CapPa
75
I CIFARICO
Pets
Cars
Figure 7: Pretraining LAION-400M.
Discussion and conclusion
We presented extensive comparison of vision encoders pretrained with a contrastive and generative (captioning) objective and found that the generatively pretrained encoders obtain better performance when used for captioning, VQA, fine-grained and few-shot classification tasks, while achieving competitive performance in classification overall. This is in contrast to previous work [50] which argued that predicting alt-text from images explicitly and exactly an overly challenging pretraining task and may lead to sub-par sar-par vision capabilities. Moreover, our results show that captioning pretraining task might exhibit favorable scaling properties with increasing model and data scale, and we hope future work will explore this avenue.
One downside of our approach is that the Cap/CapPa text decoder is of limited use. While it achieves excellent results when word order and object relationships matter-a scenario where CLIP-style models exhibit a strong bag-of-word bias zei--zerosiie shott classification and retrieval are computationally expensive. We showed that this can be mitigated relatively cheaply by training a text encoder matched to the frozen Cap/CapPa representation via LiT tuning. Note that such procedure is cheaper than training the text encoder with the image encoder and text decoder throughout the whole pretraining procedure as done e-g. by CoCa [66].2 Another promising avenue to explore parallel prediction for inference task, as it would allow for efficient scoring e.g. for zero-shot classification. Indeed, parallel prediction produces a joint distribution over all tokens, which can be used to score an arbitrary number of classes or queries.
In conclusion, we established plain image captioning as a competitive pretraining strategy for vision backbones from image-text data. We hope to inspire follow up work dedicating more attention the advantages of captioning as pretraining task for vision encoders.
2166, Table 8b)] presents an ablation the contrastive loss CoCa, keeps the unimodal decoder part (without rros-attennion) which serves as text encoder for the contrastive loss, and hence obtains essentially no gains in accelerator time. The Cap/CapPa decoder. in contrast, does not have unimodal component, and is hence cheaper to evaluate than the CoCa decoder for given architecture size (base, large).
10

---

Acknowledgments We would like to thank Jannis Bulian, Mostafa Dehghani, Alexey Dosovitskiy, Daniel Keysers, Mario Lucic, Basil Mustafa, and Xiao Wang for feedback on this paper and inspiring discussions on captioning- gaased pretraining. We would also like to thank Janko Ferlič from Unsplash for providing the photo used in Fig. I.
References
[1] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katic Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: A visual language model for few-shot learning. In NeurlPS, 2022. 1, 3,5
[2] Lucas Beyer, Bo Wan, Gagan Madan, Filip Pavetic, Andreas Steiner, Alexander Kolesnikov, André Susano Pinto, Emanuele Bugliarello, Xiao Wang, Qihang Yu, Liang-Chieh Chen, and Xiaohua Zhai. A study of autoregressive decoders for multi-tasking in computer vision. 2rXiv:2303.17377 2023. 5, 15, 18, 19, 22
[3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative components with random forests. In ECCV, 2014. 5, 17
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training recognize long-tail tail visual concepts. CVPR, 2021.
[5] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang. Ceslee Montgomery, Paulina Pietrzyk, Marvin Rittrr, AJ Piergiovanni. Matthias Minderer. Filip Pavetic, Austin Waters. Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesmikov, Mojtaba Seycdhosseimi, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut, PaLI-X: On scaling up a multilingual vision and language model. arXiv:2305. 18565, 2023. 1, 3,5
[6] Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Schastian Goodman. Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury. Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner. Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scaled multilingual language-image model. In /CLR, 2023. 1, 2, 3, 5,5
[7] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO Captions: Data collection and evaluation server. arXiv preprint arXiv:1504.0m 00325, 2015. 5
[8] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benehmark and state of the art arXiv preprint 1703001700.0122,, 2017. 17
[9] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, pages 2818-2829, 2023.
[10] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and iision-and-lanne and-language tasks via text generation. In ICML, pages 1991- 2021.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton. Sebastian Gehrmann. Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellster Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. arXiv:2204.00 02311, 2022. 3. 10, 22

---

[12] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, pages 215-223, 2011. 17
[13] Karan Desai and Justin Johnson. VirTex: Learning visual representations from textual annotations. In CVPR, 2021. 2,
[14] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Shuyang Gu, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. CLIP Itself is Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet. arXiv:2212.06/ 2022. 1, 15
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlshy. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021. 1, 3.9, 16
[16] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang. Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. An empirical study of training end-to-end vision-and-language transformers. In CVPR, pages 18145-18155, 2022. 3
[17] Danny Driess, Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. Palm-E: An embodied multimodal language model. arXiv:2303.03376 2023. 1, 3,5
[18] Jeremy Elson, John R. Douceur, Jon Howell, and Jared Saul. Asirra: A CAPTCHA that Exploits Interest- Aligned Manual Image Categorization. In Proc, ACM Conf. Computer and Communications Security (CCS), 2007. 5, 17
[19] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, pages 178 178, 2004. 5, 17
[20] Yash Goyal, Tejas Khot, Douglas Summers-Stay Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 5
[21] Cheng- Yu Hsich, Jieyu Zhang, Zixian Ma. Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for dision-languee compositionality. arXiv:2306. 14610, 2023. 2, 5,
[22] Xiaowei Hu, Zhe Gan. Jianfeng Wang. Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In CVPR, pages 17980-17989, 2022. 1,
[23] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorek, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. 230:2 14045, 2023. 1, 2, 3,5
[24] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-BERT: Aligning image pixels with text by deep multi- modal transformers. orXiv:2004.008/0 2020. 3
[25] Drew A. Hudson and Christopher D. Manning, GQA: a new dataset for compositional question answering over real-world images. In CVPR, 2019. 5
[26] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. In /CML, 2021. 1,2, 4,22
[27] Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data, ECCV. pages 67-84, 2016. 1.2
[28] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- ine-grained image categorization: Stanford dogs. In CVPRW, 2011. 5, 17
[29] Geewook Kim, Teakgyu Hong, Moonbin Yim, lcongYcon Nam, Jinyoung Park, Jinycong Yim, Won- seok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. OCR-Free document understanding
transformer. In ECCV, pages 498-517, 2022. 3
[30] Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-an and language transformer without convolution or region supervision. In ICML, pages 5583 5594. PMLR, 2021. 3
12

---

[31] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver. Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. In ECCV, 2020. 9
[32] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In CVPRW, pages 554-561, 2013. 17
[33] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, pages 73, 2017. 5
[34] Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. Mammut: A simple architecture for joint learning for multimodal tasks. WrXiv23333.66833 2023. 3,3
[35] Juho Lee, Yoonho Lec, Jungtack Kim, Adam R. Kosiorek, Scungjin Choi, and Yoe Whye Teh. Set transformer: A framework for attention-based permutation-invariee neural networks. In ICML, pages 3744-3753 2019.
[36] Kenton Lee. Mandar Joshi, Iulia Turc, Hexiang Hu. Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw. Ming-Wes Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing pretraining for visual language understanding. arXiv:2210.03344 2022,
[37] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data. In ICCV, pages 4183-4192, 2017. 1,2
[38] Gang Li and Yang Li. Spotlight: Mobile UI understanding using vision-language models with focus. In ICLR, 2023. 3
[39] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-
training with frozen image encoders and large language models. arXiv:2301. 12597, 2023. 3
[40] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICMI., pages 12888-12900, 2022. 1, 22
[41] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NewrIPS, 34:9694 9705, 2021. 3
[42] Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. Visualgptscore: Visio-linguistic reasoning with multimodal generative pre-training scores. arXiv:2306.01877 2023. 8
[43] Anand Mishra, Shashank Shekhar. Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA, Visual Question Answering by Reading Text in Images. In ICDAR, 2019. 5
[44] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0973 2021. 5,8
[45] MariaaElena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conf. Computer Vision, Graphics & Image Process., pages 722-729, 2008 5,
[46] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, pages 4004-4012, 2016. 5, 17
[47] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensihle toolkit for sequence modeling. In Proc. NAACL-HI.T: Demonstra- tions, 2019. 3
[48] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
5, 17
[49] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu. Mingxing Tan, and Quoe V. Le. Combined scaling for open-vocabulary image classification. arXiv:2111. 2021.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretehen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 4. 6, 9, 10, 15, 17, 22
[51] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 2019. 5,8
13

---

[52] Colin Raffel, Noam Shazeer. Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Rex., 21:140: 10::46671 2020. 3, 4,5, 5, 10, 22
[53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein. Alexander C. Berg, and Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJV,, 115(3):211-22 2015. 5
[54] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, pages 153- 170, 2020. 1,2
[55] Christoph Schuhmann, Richard Veneu, Romain Bcaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Thco Coombes, Jenia Jitsev, and Aran Komatsuzaki, LAION-400M: Open datasct of CLIP-filtred 400 million image-text pairs. arXiv: 100::2111.02111 2021. 10
[56] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer, How much can CLJP benefit aasion-and-languaae tasks? In ICI.R, 2022. 3
[57] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-LMM. Training multi-billion parameter language models using model parallelism. arXiv: 19099090930 2019. 3
[58] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-CLIP: Improved training techniques for CLIP at scale. 1xXiv:2303.53360 2023.
[59] Michael Tschannen, Basil Mustafa, and Neil Houlsby. CLIPPO: Image-and-language understanding from pixels only. In CVPR, 2023. 4
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017. 2,3, 10, 15, 22
[61] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 2011 Dataset. 2011. S, 17
[62] Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-tedt transformer for vision and language. Trans. Machine Learning Research, 2022. 1, 3.5
[63] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. SimVLM: Simple visual language model pretraining with weak supervision. In ICLR, 2022. 1, 22
[64] Hongwei Xue. Yupan Huang. Bei Liu. Houwen Peng. Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing mnter-modalit: Visual parsing with self-attention for vision-and lan--agg pre-training. NeurIPS, 34:4514- 4528, 2021. 3
[65] Peter Young, Alice Lai. Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2:67-78, 2014. 5
[66] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Scyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Machine Learning Research, 2022. 1, 3, 10, 22
[67] Mert Yuksekgonul, Federico Bianchi. Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bag-of-words models, and what to do about it? In ICLR, 2023. 2, 5, 6, 7, 21
[68] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, pages 12104 12113, 2022. 4, 5. 10, 15, 16
[69] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2xXiv:2303.5344 2023. 2, 5,
[70] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. LiT: Zero-shot transfer with locked-image text tuning. In CVPR, pages 18 1112218112, 2022. 2, 4, 5, 15, 16, 17
[71] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. Trans, PAMI, 2017. 5, 17
14

---

A Transfer and finetuning details
Few-shot evaluation We use the linear adaptation protocol and evaluation sets from [68, 70], reporting the 10-shot classification accuracy. Specifically, we rely on the pre-logits layer for CLIP and GAP the encoder output sequence for our captioning models. For every combination of data set and model we run the 10-shot adaptation three times and report the mean (and standard deviation for key results).
LiT decoder and T5 decoder To train a multi-task decoder from scratch on top of the frozen representation for classification, captioning and VQA, we precisely follow the setup and hyper parameters from [2] except for the data mixing strategy, for which we set to "concat image-question pairs" ([2, Sec. 5.3]). For all encoders, we use the full feature sequence before pooling (including the class token for the evaluation of CLIP). Throughout, we rely on a B-sized transformer decoder [60] with 12 layers.
We also tried tinettunig the image encoder along with the decoder for both CLIP" and Cap/CapPa models and did not obtain an improvement for any of the models. This is consistent with prior work which did not observe an improvement either for CLIP-style models when fine-tuning with the same decoder- based setup, see [2, Sec. 5.7].
For the T5 decoder keep all the parameters frozen but reinitialize and train the cross- atttenttno layers. We perform a small sweep around the default learning rate and weight decay of the setup used for training from scratch, while keeping the other hyperparameters unchanged.
Linear and non-linear ImageNet-Ik Ik probes (frozen transfer) When performing linear and non- linear probes on ImageNet-Ik, Ik, we run a wide hyper-parameter optimization sweep for all types of probes (linear, MLP, MAP) in order to get solid, trustworthy conclusions. Specifically, for each image encoder and probe combination, we sweep the full cross-produce over the following hyper- parameters: epochs: (1, 3, 10, 30, 100); image cropping: resize (256) |random_ crop (224) or inception_ wwwooonnooooooeeeeeoooooeeoeeeoe (224); learning rate: (0.001, 0.0003, 0.0001) plus earlier runs showing 0.003 and 0.01 to perform much worse; weight decay: (0.0001, Ir 0. 0.0); hidden dimension: 0 or 1024; loss: sigmoid or softmax cross-entropy. The head weights are always initialized to 0 and its bias to -6.9 in the sigmoid case.
For each result shown in Fig. 3, we select the best setting using 1% of the training data that was held-out for this purpose, and report its accuracy on the 500000 images in the validation set. For completeness, we further compute various ImageNet test-set variants and report full results in Table 9.
Broad MAP-head transfers (fine-grained) We run the same sweep as described above for each individual dataset and model combination, but only using the MAP-head probe. For each dataset, we either use provided held-out validation set for selecting the best settings, or hold out 20% of the training set if none is provided. Full numeric results are provided in Table 10. Note that we selected and classified the datasets as coarse- or fine-gra ane-grained solely by looking at the datasets and their classes, before running any single experiment on them, and never revisited this selection.
Fine-tuning on the full ImageNet-Ik 1kk data set When fine-tuning on the full ImageNet- 1 dataset, we attach fresh MAP head to the pretrained encoder and run full fine-tuning using the AdaFactor optimizer modified for ViTs in [68]. In each setting (B/16, B/16384, L/1433s), we run the exact same sweep for CLIP* CapPa, and Cap models. Notably, our exploration is significantly smaller than that of [14] and unlike for CLIP [50], ImageNet was fully de-duplicated from our pre-training dataset. In all cases, we select the best model on a held-out 2% of the training data and report that model's performance on the 50 000 image validation set without re-training
For the B/16 models, we sweep over three learning rates: (0.0001, 0.00003, 0,00001); two layer- wise learning-rate decays: (None, 0.8); 2 RandAugment parameters: (10, 15); 3 Mixup: (0.0, 0.2, 0.5); and five Polyak (EMA) averaging factors: (None, 0.9, 0.999, 0.99999, 0.9999999) with batch size of 2048 and 100 epochs. The best setting uses learning rate 0.00001, layer-wise decay 0.8, Mixup 0.5 and no Polyak averaging.
For the l/14 models at 336 px resolution, we sweep over three learning rates: (0.001, 0.0003, 0.0001), three wise learning-rate rate decays: (None, 0.9, 0.8), and five Polyak (EMA) averaging factors:
15

---

(None, 0.9, 0.999, 0.99999, 0. 1.9999999). Note that the latter does not require re- re-training for each setting and hence is cheap. We fix rand-augment (2, 10), Mixup 0.2, and training duration to 50000 steps with batch-size 512, without revisiting these choices. Besides that, we mostly follow [15, 68]. The best setting uses learning rate 0.0001, layer-wise decay 0.9, and Polyak 0.99999 for both models.
B Additional Results
B.1 Probing and LiT tuning results
Table 9 shows the classification accuracy different ImageNet- Ik evaluation sets, when probing the frozen representation with different probes (linear and non-limen -on-linear), extending the numerical results from Fig. 3.
Table 10 presents transfer results of the frozen representation to fine- and coarse-grans graise--rainedd classification tasks (using MAP head). This complements the results from Fig. 4 (Right).
Table 11 expands Table 4 in the main paper and shows frozen transfer for zero-shot classification and retrieval via LiT [70].
Table 12 presents the performance of frozen Cap/Cap and CLIP* encoders when combined via cross- cooss-attention with a frozen T5 decoder. This represents the data from Fig. (Left) in the main paper in tabular form.
Table Extended numerical results for Fig. 3, i.e. linear and non-linear ImageNet- probes on top of the frozen models. While the linear separability of CLIP models is higher, the gap between CLIP and Cap models is mostly closed when the probe also learns how to pool (map).
Model
Head
Top-1
ReaL
-v2
R(endition)
A(dvers.)
ObjectNet
linear
79.8
85.6
69.0
71.9
38.0
49.8
mlp
80.4
86.1
69.6
74.4
39.3
50.9
CLIP# (8k)
map
82.2
87.3
71.5
72.9
34.3
49.0
map+mlp
82,2
87.4
71.7
71.8
34.8
48.3
linear
80.2
85.9
69.2
73.2
40.3
51.3
mlp
80.9
86.1
70.3
71.4
37.3
49.8
CLIP" (16k)
map
82.6
87.5
72.4
73.9
37.2
50.0
map+mlp
82.6
87.5
72.1
73.0
36.3
49.3
linear
77.7
84.1
67.1
68.2
24.1
44.2
mlp
78.5
84.8
68.0
76.0
27.1
45.6
Cap
map
81.6
87.0
71.3
76.2
32.3
45.8
map+mlp
81.5
87.0
71.5
76.2
31.4
45.8
linear
78.3
84.6
66.5
67.7
22.1
43.7
mlp
79.4
85.6
68.6
77.2
25.6
46.2
CapPa
map
82.0
87.5
72.3
80.9
41.5
50.1
map+mlp
82.1
87.3
72.0
79.4
49.5
linear
84,2
88.4
75.0
83.8
59.1
60.2
mlp
84.6
88.5
74.9
83.3
56.6
$8.6
CLIP* L/14
map
85.9
89.3
76.7
84.9
57.4
58.2
map+mlp
85,8
89.2
77.0
83.6
56.1
57.8
linear
83.0
87.7
73.1
81.1
41.6
53.8
CapPa L/14
mlp
88.7
74.6
87.3
47.0
56.8
map
85.8
89.3
76.8
86.1
54.5
56.6
map+mlp
85,8
89.2
76.6
85.5
52.2
56.5
16

---

Table 10: Transfer of the frozen representation to fine- and coarse-grained classification tasks (using a MAP head). This extends the numerical results of Figure (Right).
Dataset
Grain
CLIP* (8k)
CLIP* (16k)
Cap
CapPa
CLIP# L/14
CapPa L/14
Dogs [28]
Fine
77.5
77.9
79.6
81.2
85.0
86.0
Plowers [45]
85.1
89.5
94.0
97.0
96.6
98.9
Birds [61]
Fine
76.9
78.1
76.3
54.2
85.0
86.7
Pets [48]
Fine
91.2
91.2
91.5
94.4
94.4
95.4
Cars [32]
Fine
90.8
91.6
93.4
93.3
94.0
95.8
Food [3]
Fine
91.0
92.1
91.1
91.5
94.9
94.2
RESISC [8]
Coarse
92.5
96.4
90.5
96.1
97.1
96.9
Products [46]
Coarse
88.8
89.0
87.8
88.5
90.7
90.3
SUN397 [71]
Coarse
81.8
82,9
82.0
82.1
85.7
85.2
Caltech [19]
Coarse
93.1
86.5
93.8
93.2
STL-10 [12]
Coarse
98.0
98.5
97.7
98.1
99.2
99.1
Cat/Dog [18]
Coarse
99.9
99.9
99.7
99.7
99.8
99.9
Frozen transfer for zero-shot classification and retrieval via LiT [70] as function of the number of number of training examples seen by the text encoder (the vision encoder is pretrained and frozen, and equipped with MAP head which trained along with the text encoder). The text encoder mirrors the architecture of the vision encoder. Especially for the larger model, CapPa is competitve with CLIP* with comparable or fewer examples seen. The CLIP numbers are obtained by evaluating the image and text encoders released by [50] in our eval setup. We report these numbers for reference, no LiT tuning is done on top of the CLIP vision encoder. This table complements Table 4 in the main paper.
Table 11:
ImageNet Oshot
COCO 12i r@
COCO i2t r@
LiT pairs:
900M
3B
12B
900M
3B
12B
0
900M
3B
12B
Cap
65.9
67.8
69.0
35.3
37.5
39.1
50.3
53.9
54.8
CapPa
66.4
68.8
70.2
34.3
37.3
38.6
49.7
53.9
55.1
CLIP# (8k)
65.6
65.9
67.6
69.0
41.5
36.5
38.2
39.5
56.7
52.0
54.0
56.1
CLIP* (16k)
67.7
66.7
69.0
70.0
43.0
37.0
38.9
40.1
58.2
53.0
55.1
57.0
CLIP
68.3
32.3
52.8
CapPa L/14
74.6
76.4
77.5
40.6
43,9
45.4
56.6
60.3
62.6
CLIP* L/14
74.8
74.5
75.8
76.6
48.1
42.7
44.7
46.3
63.7
57.7
60.7
62.3
CLIP L/14
75.1
36.5
56.6
Table 12: Performance of frozen representations trained via image captioning (Cap/CapPa) and contrastive (CLIP") objective, when combined via cross-attention with a frozen T5 decoder. Only the cross-attention weights updated during the training. See Table 2 for the corresponding models that have the decoder trained from scratch.
Classification
Captioning
OCR
Question Ans.
ilk
sun
food
res
pet
COCO
Flickr
VQA
VQAv2
GQA
Cap
79.0=a.1
81.34a1
89. 92.4+a1 92.34a3
119.7406 72.2
57.7+00
64.6
52.1
CapPa
80.0
81.240. 89 99.9200
1301 93.2403
118.7205
70.0
57.8
63.3
51.9203
CLIP (8k)
79.1
81,5
89.9
92.7+aa
88.5402
110.620s
60.8
50.3
57.2+04
49.5
CLIP* (16k)
29.
81.74a.
00.4
93.7+na
88. 88.6201
110.620s
59.8
50.2404
56.8
49.6203
17

---

B.2 Scaling properties
Fig. 8 and 9 show the performance frozen Cap, CapPa, and CLIP" encoders on variety of tasks as a function of the number of training examples seen and the encoder model size, respectively. Specifically. we evaluate our models on classification, captioning, and VQA when combined with decoder trained from scratch to solve all those tasks jointly (following [2]), and on 10-shot linear classification based on the pre-logit features.
Tables 13, 14 and 15, 16 show the data from Fig. 8 and 9, respectively, tabular form. For completeness, in Table 16 we also present the ImageNet zero-shot accuracy (without prompts) of CapPa and CLIP models obtained with their respective pretrained decoder and encoder. We emphasize that scoring-based zero-shot classification not the focus of this paper, and did optimize the Cap/CapPa models for this.
ImaggNNrr
SUN897
Food101
RESISCA5
Pets
844
95
84-
92
94-
93
NN
78
COCO Caption
Flickr Caption
OCR-VQA
VQAY2
GQA
65
85
70-
170
60
56
55
training examples
ImageNet 10-t
CIFAR100 10-shot
Pots 10-shot
Cars 10sshot
70-
Model
055
K5
Cap
CapPas
600
MJ
CLIP*
35-
75
Arch (butch size)
(8k)
40
1/14
36
38
training examples soot
training examples sOoT
training champles soom
Imining examples seem
B/16 (16k)
Figure 8: Performance of vision backbones pretrained with captioning (Cap/CapPa) and contrastive objective (CLIP*) function of the number of pretraining examples Fig. 2). Top two rows: Classification, captioning, and VQA performance with a decoder trained from scratch in multi-task fashion (see [2] for details). We use CIDEr for captioning, the VQAv2 weighted accuracy for VQAv2, and exact matching accuracy for all other tasks. Bottom row: 10-shot linear classification accuracy on the frozen pre-logit representation.
(expands the results in
18

---

ImageeNet
Pess
36
95.0
864
95
52
90.0
78-
K5.0
Caption
Cuption
OCR-VQA
VQAV2
GQAA
70-
115
110
FL.Ps (encoder)
ImageNNr 10-shot
CIFARIOU 10-shot
10-shor
70-
60
Capla
CLIP"
50-
1214
FL.OPs (encoder)
FL.OPs (enooder)
F1.OPs (ancoder)
F1.OPs (encoder)
Figure Performance of vision backbones pretrained with captioning (CapPa) and contrastive objective (CLIP") as a function of the model size/FL.OPs (we compare ViT-S/16, M/16, B/16, and I/14; this expands the results in Fig. 2). Top two rows: Classification, captioning, and VQA performance with a decoder trained from scratch in multi-task fashion (see [2] for details). We use CIDEr for captioning, the VQAv2 weighted accuracy for VQAv2, and exact matching accuracy for all other tasks. Bottom row: 10-shot linear classification accuracy the frozen pre-logit representation.
Table 13: Data corresponding to Fig. 8 (top two rows) in tabular form. See caption of Fig, 8 for details on the metries.
Classification
Captioning
OCR
Question Ans.
ex. seen model
arch
sun
res
pet
COCO Flickr
VQA
VQAv2 GQA
Cap
B/16 (8k)
77.7
79.8
87.2
93.3
91.9
113.5
72.5
58.6
66.5
54.2
B/16 (8k)
79.1
80.5
88.2
94.2
92.6
112.2
71.7
58.4
66.6
55.0
CapPa
L/14
81.7
82.5
91.1
95.2
94.0
118.7
80.2
61.8
68.6
55.8
900M
B/16 (8k)
78.8
80.9
88.2
94.5
92.0
111.1
70.0
51.7
65.1
53.1
CLIPs
B/16 (16k)
79.2
81.0
88.5
94.5
92.4
111.2
70.5
52.0
65.6
54.1
L/14
82.3
83.6
92.1
95.5
94.4
118.2
78.6
56.8
67.9
55.5
Cap
B/16 (8k)
79.5
92.4
115.6
76.3
60.9
67.6
54.5
B/16 (8k)
80.5
81.5
89.7
94.4
94.1
116.1
75.9
60.9
67.7
54.9
CapPa
L/14
83.3
83.8
92.6
95.5
95.3
122.8
84.7
63.7
70.1
58.4
3B
B/16 (8k)
80.3
82.3
90.1
94.8
93.1
114.2
73.8
55.4
66.0
53.8
CLIPs
B/16 (16k)
80.5
82.5
90.8
94.8
92.9
114.9
74.5
55.2
66.0
53.8
83.6
85.0
93.7
95.8
95.0
121.7
82.1
59.8
68.2
55.6
Cap
B/16 (8k)
80.2
82.3
90.3
93.6
93.1
117.5
78.6
62.2
68.2
55.0
B/16 (8k)
81.3
82.4
90.9
94.2
94.4
117.9
80.5
62.2
68.3
55.7
CapPa
L/14
84.4
84,9
93.8
96.0
95.6
125.8
89.3
65.6
70.9
58.3
9B
B/16 (8k)
81.1
83.2
91.2
94.8
93.4
115.8
74.5
56.0
66.5
54.3
CLIPa
B/16 (16k)
81.4
83.3
92.0
95.2
93.6
116.3
77.1
56.5
66.7
54.8
I//4
84.7
85.7
94.6
96.4
95.2
123.2
85.5
61.3
68.5
55.3
19

---

Table 14: Data corresponding Fig. 8 (bottom row) in tabular form. 10-shot linear classification accuracy on the frozen pre-logit features.
ex. seen
model
arch
ImageNet
CIFARI00
Cars
Cap
B/16 (8k)
49.7
56.0
72.6
74.7
B/16 (8k)
50.4
57.4
76.2
78.5
CapPa
L/14
60.3
68.8
85.6
87.8
900M
B/16 (8k)
50.6
59.2
70.5
CLIP
B/16 (16k)
50.4
59.1
72.6
77.0
L/14
60.0
68.7
80.9
83.8
Cap
B/16 (8k)
55.0
58.0
81.2
81.1
B/16 (8k)
56.0
60.7
83.3
85.2
CapPa
L/14
66.9
70.0
90.8
90.7
3B
B/16 (8k)
55.5
62.3
76.7
78.6
CLIP"
B/16 (16k)
56.7
63.8
77.9
80.7
L/14
66.7
72.8
Cap
B/16 (8k)
57.2
58.6
83.7
84.2
CapPa
B/16 (8k)
59.1
62.4
86.5
86.6
L/14
70.6
92.2
9B
B/16 (8k)
58.5
64.9
77.7
80.8
CLIP"
B/16 (16k)
59.7
66.3
80.6
82.9
L/14
69.8
74.1
87.7
89.2
Table 15: Data corresponding to Fig. (top two rows) in tabular form. See caption of Fig, 9 for details on the metrics
Classification
Captioning
OCR
Question Ans.
arch
FLOPs
model
ilk
sun
food
res
pet
COCO
Flickr
VQA
VQAv2
GQA
CapPa
76.5
78.2
85.4
92.9
91.2
108.4
65.9
58.4
65.0
53.2
S/16
9.2G
CLIP#
76.9
80.1
87.3
92.6
91.0
108.4
68.8
51.7
63.9
52.4
CapPa
79.0
80.7
88.2
93.8
92.7
112.7
71.4
60.1
66.5
54.7
M/16
16.0G
CLIPs
79.1
81.8
92.5
112.2
72.4
54.2
$3.3
B/16
35.1G
CapPa
81.3
82.4
90.9
94.2
94.4
117.9
80.5
62.2
68.3
55.7
CLIPs
81.4
83.3
92.0
95.2
93.6
116.3
77.1
56.5
66.7
54.8
L/14
161.8G
CapPa
84.4
84.9
96.0
95.6
125.8
89.3
65.6
70.9
58.3
CLIPs
84.7
85.7
94.6
96.4
95.2
123.2
85.5
61.3
68.5
55.3
Table 16: Data corresponding to Fig. 9 (bottom row) in tabular form. 10-shot linear classification accuracy the frozen pre-logit features. We also show the ImageNet zero-shot classification accuracy (without prompts) when using the pretrained text encoder (CLIP") or text decoder with scoring (CapPa) for reference (last column).
arch
FLOPs
model
ImageNet
CIFARI00
Pets
Cars
ImageNet zs.
CapPa 40.6
47.1
71.8
71.2
35.1
S/16
9.2G
CLIP*
47.7
52.5
69.0
73.6
52.8
CapPa
49.8
52.4
79.0
80.4
43.0
M/16
16.0G
CLIP*
52.8
58.5
76.9
78.2
58.7
CapPa
59.1
62.4
86.5
86.6
52.7
B/16
35.1G
CLIP*
59.7
66.3
80.6
82.9
64.1
L/14
161.8G
CapPa
70.6
72.9
92.6
92.2
63.8
CLIP*
69.8
87.7
89.2
71.2
20

---

B.3 Attribution, relation, ordering
Table 17 shows extended results for different models on the ARO benchmark [67] (see Table in the main paper). In addition to the clear superiority of Cap/CapPa over CLIP models discussed in the main paper, it can observed that increasing the model capacity form B/16 L/14 leads to an overall improvement for CapPa, while this is not the case for CLIP"
Table 17:
Results on the Attribute, Relation and Order (ARO) benchmark [67]. Cap and CapPa models clearly outperform all CLIP* and CLIP variants across all data sets, even when training the model be sensitive to word ordering and attribution as in NegCLIP [67]. Values for "ARO Best" are taken from [67]. "Blind dec." corresponds to Cap without vision encoder, i.e. the vision encoder features fed to the decoder are replaced with all zeros.
Arch
VG Attribution
VG Relation
Flickr Order
COCO Order
Blind dec.
83.7
86.2
98.8
98.7
Cap
B/16
88.9
99.1
99.0
CapPa
B/16
85.7
86.7
99.2
98.8
CapPa
L/14
89.3
86.0
99.3
99.0
CLIP (8k)
B/16
55.4
39.8
43.7
32.8
CLIP (16k)
B/16
53.2
39.7
45.5
37.0
CLIPP (16k)
I/14
57.8
35.9
40.2
31.5
CL.IP
B/32
63.2
59.1
59.4
47.3
CL.IP
B/16
62.7
58.7
57.9
49.5
ARO Best
88.0
60.0
46.0
NegCLIP
B/32
71.0
81.0
91.0
86.0
B.4 SugarCrepe
We provide the full breakdown of results across all our models on SugarCrepe in Table 18. The numbers for OpenCLIP are taken from [21] and represent the best, largest contrastive model that was benchmarked on SugarCrepe to date. Even the small ViT-B/16 Cap model significantly outperforms of learning well.
it all but the "Replace Object" task, which is task that matches contrastive's "bag of word""style
Table 18: Full results on the SugarCrepe [21] benchmark suite.
Replace
Swap
Add
Training
Arch
Object
Attribute
Relation
Object
Attribute
Object
Attribute
Cap
B/16
91.10
88.32
85.21
79.27
88.74
98.59
99.28
CapPa
B/16
89,95
88.71
84.35
80.49
85.74
98. 84
99.42
CapPa
L/14
92.01
90.10
87.34
82.11
88.44
99.42
CLIP# (8k)
B/16
93.70
82.36
66,29
61.79
67.12
83.46
76.01
CLIP* (16k)
B/16
94.07
84.64
67.14
60.98
65.47
86.37
77.46
CLIP# (16k)
L/14
95.70
84.26
69.06
65.04
68.02
86.76
78.32
OpenCLIP
G/14
96.67
88.07
74.75
62.20
74.92
92.19
84.54
We further show qualitative examples in Tables 22-24. The examples are manually picked to be representative (we show wins and losses), while avoiding uninteresting (i.e. scemingly random), too cluttered, or too verbose examples. Thus, the examples are cherry- picked to be presentable, but are meant to be representative. All images are from the COCO validation set.
Each image comes with positive and (hard) negative caption, and model's prediction deemed correct when it scores the positive caption higher than the negative one. For the CapPa model, we score each caption using the log-likelihood, meaning negative numbers closer to zero correspond to higher score (i.e. score of -20 means the caption fits the image more than a score of -10)) For the CLIP model, we score each caption using the dot-product of normalized embedding similarity as usual, but multiply the resulting score by 100 for readability.
21

---

Table 19: Impact of decoder architecture design choices in Cap 10-shot linear classification accuracy: Left: Effect of sharing the embedding between decoder input and output and removing biases from decoder layers. Right: Effect of the number of decoder layers.
share emb.
dec. bias ImageNet
CIFARIO0
Pets
Cars
dec. layers ImageNet CIFAR100
Pets
Cars
yes yes
47.8
55.8
71.5
71.7
48.7
53.7
73.5
73.7
49.7
56.0
72.6
74.7
49.7
56.0
72.6
74.7
yes
48.3
54.6
74.4
70.2
12
48.7
54.8
74.4
73.8
no
yes
49.3
56.6
72.7
71.9
We noticed that for the Add scenarios, where CapPa performs almost perfectly, the only losses due to typos in the positive caption ("toliet' instead of "toilet" and "bridge" instead of "bride"), so we also provide the score for the corrected caption in the Pos (fixed), which confirms the typos the reason for the model failure.
B.5 Ablations: Decoder architecture
While following the original transformer decoder architecture [60] closely, we investigate several modifications that have become common the literature [52, 11]. Specifically, we ablate the effect of removing biases in decoder layers, well sharing the decoder input and output embeddings. Table 19 (left) shows that not sharing the embeddings leads to overall better 10-shot accuracy than sharing them, and additionally removing the decoder biases does not hurt. Furthermore, observed significantly improved stability across encoder architectures, scales and training schedules when removing the decoder biases.
Table 19 (right) reveals that the overall best 10-shot classification accuracy obtained when using 6 layer decoder. This decoder depth also leads to total parameter count comparable to the corresponding CL.IP* model (Table 1).
B.6 Further Ablations
Table 20 compares the performance of the CapPa and CLIP* vision encoders with ViT-B/16 pretrained supervised fashion on ImageNet-21k when combined with transformer decoder.
Table 21 represents the data from Fig. 6 in tabular form.
Table 20: Comparison of CapPa and CLIP* with a ViT-B/16 pretrained in supervised fashion on ImageNet-21k (we use the checkpoint from Steiner al. 2021) when combined with transformer de- coder trained from scratch for classification, captioning, and VQA [2]. CLIP* and CapPa outperform the model pretrained in supervised fashion.
Classification
Captioning
OCR
Question Ans.
ilk
sun
food
res
pet
COCO
Flickr
VQA
VQA2
GQA
B/16 (i21k)
13. 72.8 81.2.02 86.2 85,6103
95.0 52.310.
39.1101
57.6 37.6103
50.1
CapPa
81.3
82.4+a.1 90.94a1
94.2#a2 94.4
117.9 80.5
62.2+ao
68.3
55.7
CLIP
81,4 83.3 92.040.
95.2=02 93.6202
116.3207 77.1
56,5
66.720.
54,8
C
Societal impact
Our models fit in the broader context of large scale vision-language pretraining and such share many of the benefits and issues of related models such as [50, 26, 66, 40, 63]: They produce versatile vision models which obtain strong performance on natural images, on OCR-related tasks, and also when combined with generative language decoder. These capabilities enable many useful applications (e.g. assistive technologies, medical imaging), but also potentially harmful ones (e.g. surveillance). We generally recommend either employing the CapPa vision encoder with a new, task- lask-specific prediction head, or using the pretrained decoder for scoring only. We do not recommend the pretrained decoder for downstream image captioning applications without further refinement, as
22

---

Table 21: Ablation results representing Fig. 6 in tabular form. Left: 10-shot linear classification accuracy based on the frozen encoder representation as function of the fraction of training batches for which parallel prediction is performed CapPa. Right: 10-shot linear classification accuracy and zero-shot classification accuracy as function of the vision encoder architecture.
arch
model
INct
C100
Pets
Cars
NNet 7s.
fraction
INet
C100
Pets
Cars
CLIP* (8k)
39.8
33.5
49.2
60.9
43.6
0%
49.7
56.0
72.6
74.7
R50
Cap
37.8
33.3
48.6
52.4
28.5
25%
46.7
52.9
71.9
72.8
CLIP (8k)
44.1
57.7
64.7
68.1
48.3
50%
49.8
57.8
76.8
76.9
VIT-B/32
Cap
41.0
53.7
64.0
58.7
35.4
75%
50.4
57.4
76.2
78.5
CLIP* (8k)
50.6
59.2
70.5
74.4
52.2
90%
49.0
59.5
73.1
79.0
ViT-B/16
Cap
49.7
56.0
72.6
74.7
43.8
it is trained on large number of alt-texts from the web. Harmful biases should be carefully assessed the context of the concrete downstream application and prediction head used. For example, when combining the encoder with (potentially pretrained) decoder for captioning or VQA, an assessment of hallucinations, attribute binding issues and stereotypical attribution should be done.
23

---

Table 22: Representative examples of CapPa L/14 wins (first three) and losses (fourth) over CLIP L/14 on the different replace categories the SugarCrepe hard negatives benchmark suite. See text for example selection criteria. Higher score means better: for CapPa this is the log-likelihood, so closer to 0 is better, while for CLIP this is the matching score (multiplied by 100) so closer to 100 is better.
CapPa wins over CLIP
CLIP wins CapPa
Replace Object Positive
CapPa:-28.6 CLIP:23.6
CapPa:-45.4 CLIP:13.2 CapPa::42.44 CLIP:1688
CapPa:-54.6 CLIP:13.7
Street signs on the corner of Gladys and Detroit
A run down building with A brown bird has small The model toys are two planters outside the yellow head. door
positioned the table.
Negative
CapPa:-53.8 C39
CapPa:559.2 CLIP:14.8 CapPa:-46.2 CLIP:18.1
CapPa:-51.7 CLIP:8.8 A run down building with brown bird has a small The books are positioned on the table.
Street signs on the corner of Gladys and Chicago.
a statue outside the door. yellow beak.
P
K
GLADYS
Replace Attribute Negative Positive
4apPa: CLIP:14.7
C7 CLIP:1311
CapPa--17.3 CLIP:15.7
CapPa:-115.9 CLIP:14.1
A plate food with a fried egg and colorful vegetables.
A bunch of different foods large black truck in a on display counter.
Two large trucks are travelling along a tree-lined madway.
parking lot.
C5p CLIP:15.1 A plate of with a fried egg and monochromatic vegetables.
CapPa:-53.8 CL1P:14.8
C8.3 CLIP:16.1 small black truck in parking lot.
CapPa--61.7 CLIP:12.5
A bunch of similar foods display counter.
Two large trucks are travelling along deserted roadway.
Replace Relation
Positive
Cap0a:-20.0 CLIP
48.2 18.6:18.6
CapPa:299.1 CLIP:24.4
C5p CLIP:17.6
fire hydrant a grassy
A cell phone on top of a calculator near a computer keyboard.
red fire hydrant on a city sidewalk.
A train driving over a small bridge on a green hillside.
field next to a bush
Negative
56p1 CLIP-21.5
CapPa:-56.1 CLIP:19.2
CapPa:-35.6 CLIP:25.6
CapPa:-54.7 CLIP:17.4 A train passing under small bridge on a green hillside.
A fire hydrant a grassy field far from a bush.
A cell phone underneath a calculator near a computer keyboard.
red fire hydrant beside city sidewalk.
24

---

Table 23: Representative examples of CapPa L/14 wins (first three) and losses (fourth) CLIP L/14 on the different add categories of the SugarCrepe hard negatives benchmark suite. See text for example selection criteria. Higher score means better: for CapPa this is the log-likelihood, closer to is better, while for CLIP this is the matching score (multiplied by 100) so closer to 100 is better.
CapPa wins over CLIP
CLIP wins over CapPa
Negative Positive
CapPa:-27.3 CLIP:14.0 an image of a plate of food with meat and CapPa-1500.3 CLIP:13.7 CapPa--649 CLIP:15.5
CapPa:-18.2 CLIP:13.7
CapPa:-30.2 CLIP:14.3
CapPa:-60.3 CLIP:22.5
bathroom with mirror and a sink.
A two layered cake sits on a table top
A bridge and groom culting their wedding
Add Object
veggies
cake that has fruit on top.
CapPa-148.6 CLIP:14.6 An image of plate food with meat. fruit, and cutting their wedding veggies.
CapPa:-53.4 CLIP:21.6
A buthroom with mirror, A two layered cake sits on table top next to a vase of flowers.
A bride and groom
sink, and shower.
cake that has flowers and
fruit on top.
Pos (fixed)
CapPa:-46.1 CLIP:22.7
A bride and groom culting their wedding cake that has fruit on top.
CapPa:-43.8 CLIP:21.0
CapPa:-65.4 CLIP:16.1
CapPa:-49.9 CLIP:17.6
CapPa:-62.1 CLIP:20.4
Positive
A litrle girl smiling for the camera with
A clock fastened to A person frying some
There stuffed animal sitting on the toliet.
Add Attribute Negative
brick store front reads 10 kind food on stove. after 10
umbrella behind her. CapPa:-121.5 CLIP-21.0 A litll girl smiling for the camera with polka-dotted umbrella behind
CapPu:90.77 CLIP:17.0 CapPa:-115.3 CLIP:19.5
CapPa: CLIP:19.1
A clock fastened to a lush A person frying some brick store front reads 10 curry-spiced food stove.
There is a stuffed animal sitring the decorated toilet.
after 10.
Pos (fixed)
CapPa:-36.8 CLIP-21.3 There is a stuffed animal silting the toilet.
25

---

Table 24: Representative examples of CapPa L/14 wins (first three) and losses (fourth) over CLIP L/14 on the different swap categories of the SugarCrepe hard negatives benchmark suite. See text for example selection criteria. Higher score means better: for CapPa this is the log-likelihood, closer to is better, while for CLIP this is the matching score (multiplied by 100) so closer to 100 is better.
CapPa wins over CLIP
CLIP wins over CapPa
CapPa:333.5 CLIP:22.5
CapPa:-549 CLIP:22.1
CapPa:338.1 CLIP:15.6 CapPa--1114 CLIP:20.5
Positive
Swap Object
a bright kitchen with tulips on the table and plants by the window
A person cutting a pizza next to salad and bottles of wine on wooden table.
close up of a sandwich
Statues on the second floor of a building, sitting below clock.
with a drink in the back.
Negative
CapPa:556.8 CLIP-22.8 A bright kitchen with plants on the table and tulips by the window.
CapPa:-57.5 CLIP:22.5
CapPa:-45.6 CLIP:16.2 CapPa:-11083 CLIP:19.4
A person cutting a salad next to pizza and bottles of wine on wooden table.
close up of a drink with A clock on the second floor of a building, sitring below statues.
sandwich in the back.
Swap Attribute Positive
Cap6a:332.6 CLIP:15A
4ap9a:-45.9 CLIP:19A CapPa:-28.4 CLIP:16.3
CapPa:-108.6 CLIP:19.3
white cake is by bunch of flowers
A blue tennis racket has black bike rests against a yellow tennis ball on it. brown bed
All the cows are poking their heads out, eating some hay. CapPa--107.11 CLIP:18.2
Negative
CapPa:-64.1 CLIP:17.3 CapPa-54.9 19.5
Cap CLIP:16.9
A bunch of cakes are by a A yellow tennis racket has blue tennis holl on against black bed.
brown bike rests
Some cows are poking their heads out, eating all of the hay.
white flower
26


Reviews:
[{"review_id": "86NBVwFHht", "reviewer": "Reviewer_J9Jb", "summary": "In this paper, the authors more a controlled comparison between two pretraining approaches to learning visual representations from language supervision:\ncontrastive (CLIP-style) and generative (= image captioning, e.g. VirTex, SimVLM, BLIP, CoCa), etc.\nThe goal of this paper is not to advance the state-of-the-art, but rather to observe the model/data scaling behavior of these pretraining tasks.\nExperiments show that captioning-pretrained models can match or outperform CLIP-style contrastive models on several multi-modal tasks.\n", "questions": "Minor suggestion:\n\n- This paper references CLIP in many places in the text. However, it is sometimes awkward to read as \"[40] showed...\" or \"released by [40] ...\".\n  Ultimately, it is personal preference, but I may recommend the authors use `citet` format like \"Radford et al. [40] showed that ...\"\n  for a better reading experience if they do not have a preference.\n- `Line 84`: GeLU -> GELU. ReLU (\"Re\" = \"Rectified\") and GELU (\"GE\" -> \"Gaussian Error\") :-)\n- What is the \"(ok:as)\" in section 4.2 title? Seems like a latex macro :-)", "limitations": "The authors have included a reasonable discussion about the limitations of this study (and their trained captioning models)\nin the final section of the paper.\n", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**I think this paper, in its current form, already matches the quality of a typical publication at the NeurIPS conference.**\nI strongly recommend acceptance; it is relevant to the conference audience and will spur interesting discussion in the community.\nBelow I highlight the main strengths of the paper to substantially support my assessment:\n\n1. **Paper presents contrary evidence to existing results:**\nThe vision community is making rapid progress with vision-language models as they enable new transfer applications that can be specified using natural language.\nMuch of this progress in the last 2-3 years was catalyzed by the development of CLIP (and concurrent works like ALIGN).\nEver since, the vision community has largely gravitated towards pushing progress on multi-modal contrastive models,\nfollowing the \"image captioning models converge slower on web data\" result from the CLIP paper.\nThis paper presents a piece of contrary evidence that simple captioning-only models can match or outperform their contrastive counterparts.\n\n2. **Promising alternative to poor language understanding of contrastive models:**\nImage captioning as a pretraining task has been studied in previous models (e.g. VirTex, SimVLM, BLIP, CoCa, etc.).\nThe main novelty of this work is a direct comparison of captioning with the contrastive objective at scale,\nwith controlled model capacity and training dataset size.\nHence, I think this is a timely contribution that will force practitioners and researchers to rethink the relevance of text-generative models\nand side-step the embarrassing failures of contrastive models, e.g. their inability to distinguish \"man eating a sandwich\" from \"sandwich eating a man\".\n\n3. **New evaluations with captioning models show practical runtime solutions for classification/retrieval:**\nCaptioning models are known for their slow image/text retrieval runtime since they cannot \"cache\" the text classifier weights once like contrastive models.\nHowever, they are better at language understanding than contrastive models which are known to behave as bag-of-words models.\nAuthors show evaluations related to \"LiT tuning\" to convert a captioning-pretrained image encoder to a contrastive model --\nthis helps overcome the runtime overhead of image captioning-only models.\n\n4. **Experiments are thorough and well presented:**\nThe paper studies a targeted comparison between captioning and contrastive pretraining approaches.\nThe authors present a series of experiments and evaluations to support this study.\nAll comparisons seem fair and controlled to the best of my knowledge, with differences specified wherever relevant.\nModeling ablations and experiments with a different dataset (LAION-400M) make the study more self-contained.\nMany evaluations report error bars wherever appropriate.\n\n5. **Excellent clarity in writing and presentation:**\nThe motivation for this study is precisely stated I the abstract and introduction.\nThe coverage of related work is broad and comprehensive.\nAll technical details for empirical analysis are well-stated and easy to follow.\nThe main paper and supplementary material have adequate implementation details to aid reproducibility.\n", "weaknesses": "I have some questions and suggestions that could make the study more comprehensive.\nHave the authors considered the following experiments in their study?\n\n1. **The exact autoregressive language model used in CLIP paper:**\nCLIP paper uses a different autoregressive model, and this particular model is shown to converge slowly.\nHave the authors tried this exact architecture?\nInstead of a transformer decoder with cross-attention to image features (e.g. like VirTex),\nCLIP's autoregressive baseline has a SimVLM-style design wherein image features are pooled into 2x2 grid and passed to the text model as the first four tokens.\nThe model follows the transformer encoder design and predicts caption tokens autoregressively.\n\n2. **Backward captioning or masked language modeling?**\nHave the authors considered auxiliary objectives used by prior works, such as backward captioning (VirTex) or masked language modeling (ICMLM)?\nThese objectives can amortize the cost of forward pass through the image encoder and provide denser gradients to the image encoder.\n\n3. **[Related to above] multiple parallel decoders:**\nThe above suggestion can be extended to enable the use of multiple lightweight text decoders with multiple auxiliary objectives.\nOne can design each decoder head with reduced capacity to make all models have comparable sizes.\nTraining with such multiple objectives should speed up convergence.\n\n4. **Evaluation on dense prediction tasks?**\nIf the goal of this study is to learn high-quality image encoders, then I suggest the authors may include additional evaluations\nwith dense prediction tasks like object detection and segmentation. These tasks are ubiquitous in vision and quite challenging.\nI suggest that authors could train a ViTDet-style model with a frozen/fine-tunable image encoder from Cap/CapPa training.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for suggesting additional experiments, many of which we were able to address.\n\\\n\\\n**Performance of the exact autoregressive language model used in CLIP paper**\n\nWe trained this exact model (with a ResNet-50 encoder as in the CLIP paper) for 900M examples seen and found that it performs somewhat worse than the same encoder model together with the transformer decoder architecture. However, we believe that exploring alternative, potentially simpler decoder architectures at scale is an interesting direction for future work.\n\nComparing prefix decoder with baselines (10-shot linear eval. and scoring)\n| model                         |   ImageNet |   CIFAR100 |   Pets |   Cars |   INet zs. |\n|:------------------------------|-----------:|-----------:|-------:|-------:|-----------:|\n| CLIP* (8k) R50                |       39.8 |       33.5 |   49.2 |   60.9 |       43.6 |\n| Cap R50 (transformer decoder) |       37.8 |       33.3 |   48.6 |   52.4 |       28.5 |\n| Cap R50 (prefix decoder)      |       36.8 |       30.4 |   41.5 |   45.4 |       27.6 |\n\n\\\n**Cap/CapPa with backward captioning or masked language modeling**\n\nWe explored masked language modeling in the context of parallel prediction by masking only a fraction of the tokens, but only observed improvements over pure autoregressive modeling when masking all tokens (see Sec. 4.4, Parallel prediction).\nAs for backward captioning, we trained Cap while randomly reversing the caption with probability 0.5 (with the 900M examples schedule). This ensures that model capacity and pretraining compute remain unchanged. We do not observe improved performance (see below). While VirTex ablates backwards captioning and shows improvements, they use a separate decoder, so the ablated model has fewer parameters and FLOPs (here we control for both factors).\n\nForward vs forward + backward captioning (ViT-B/16 encoder; 10-shot linear eval.)\n| model           |   ImageNet |   CIFAR100 |   Pets |   Cars |\n|:----------------|-----------:|-----------:|-------:|-------:|\n| Cap             |       49.7 |       56.0 |   72.6 |   74.7 |\n| Cap (fwd + bwd) |       49.2 |       56.1 |   71.7 |   73.0 |\n\n\\\n**Using multiple parallel decoders**\n\nTo address this point, we trained a CapPa variant with two parallel decoders, one for autoregressive prediction and another one for parallel prediction, each with 3 layers instead of 6. This model matches the pretraining compute of the default Cap/CapPa models with 6 decoder layers. While this model performs better than Cap with 3 decoder layers in linear 10-shot eval, it does not clearly outperform Cap with 6 decoder layers and performs worse than CapPa on 3 out of 4 eval sets.\n\nComparing separate decoders with baselines (for ViT-B/16 encoder; 10-shot linear eval.)\n| model                  |   ImageNet |   CIFAR100 |   Pets |   Cars |\n|:-----------------------|-----------:|-----------:|-------:|-------:|\n| Cap (3 dec. layers)    |       48.7 |       53.7 |   73.5 |   73.7 |\n| Cap                    |       49.7 |       56.0 |   72.6 |   74.7 |\n| CapPa w/ sep. decoders |       49.5 |       54.9 |   75.8 |   79.0 |\n| CapPa                  |       50.4 |       57.4 |   76.2 |   78.5 |\n\n\\\n**Evaluation on dense prediction tasks**\n\nCLIP and image/text pretrained models more generally seem less popular than supervised/self-supervised vision encoders for such supervised dense prediction tasks. However, vision-language models have become popular recently for open vocabulary semantic segmentation (see e.g. [a, b, c, d]) and the field is evolving rapidly. We believe it might be interesting to explore this direction with captioning models, but we leave this for future work as it would require a substantial extension of the scope.\n\n- [a] Ding et al., Decoupling zero-shot semantic segmentation. CVPR 2022\n- [b] Ma et al., Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models. BMVC 2022\n- [c] Liang et al., Open-vocabulary semantic segmentation with mask-adapted CLIP. CVPR 2023.\n- [d] Mukhoti et al., Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning. CVPR 2023\n\n\n\\\n**Minor issues**\n\nThank you for raising these, we will address them in the next revision of the paper.\n"}, {"review_id": "xmD8GyWH7d", "reviewer": "Reviewer_G8WT", "summary": "This paper revisits image captioning as a pretraining task for learning general vision encoders from web image-text pairs. Surprisingly, the empirical study shows that captioning pre-trained vision encoders is competitive or better than contrastively pre-trained ones on image recognition and vision-language tasks.", "questions": "The paper is well-written and technically flawless. I don't have significant concerns about this paper.", "limitations": "No potential negative societal impact.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper presents a novel and interesting finding: using captioning as a pre-training scheme can also achieve strong results compared with contrastive ones. The empirical study is valuable to the community.\n- The paper presents an in-depth analysis of different design factors, such as the use of decoders, encoders, and pre-training data. One can find many insightful discussions in the experiment section.\n", "weaknesses": "- Although captioning can be a promising scheme for pre-training, it may not be able to replace the existing contrastive pre-trained objective. For many established tasks, such as image-text retrieval or estimating the similarity between a given image-text pair, CLIP-style models are convenient and offer a more efficient computation. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**CLIP-style models are convenient for image-text retrieval/computing image-text similarities and offer a more efficient computation.**\n\nThis is indeed a downside of captioning-based approaches. On the upside, Cap and CapPa are much better than CLIP-style models in zero-shot classification tasks where word order, attribution, and relation are important (see Table 6). Furthermore, we show that LiT-tuning [56], where a text encoder is trained to match a frozen image embedding, represents an efficient way to equip Cap/CapPa with these capabilities (Table 4).\n"}, {"review_id": "RiEZ3XhlyV", "reviewer": "Reviewer_9JDL", "summary": "This paper shows that the captioning loss is a competitive alternative to pretrain image backbones compared with contrastive loss (CLIP) when using the same training budget and training data. The standard next token prediction objective on the full caption sequence (Cap) is complemented with a parallel prediction loss (CapPa) which is used on a quarter of the training batches. Experiments notably show:\n- the benefit of CapPa over Cap in a variety of settings\n- for classification, CLIP models are better than CapPa models at linear probing but the gap is bridged when using MAP probing. \n- with LiT transfer, CapPa models perform competitively in classification and are better at vision and language tasks like VQA or image captioning.\n- CapPa models scale well with model size and training budget.\n- CapPa models are better at attribute / relation / order prediction.\n", "questions": "See weaknesses.", "limitations": "Limitations are discussed in Section 5. Potential negative societal impact is not discussed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- Alternating parallel prediction with autoregressive prediction improves downstream transfer.\n- The experimental setup is well described.\n- Extensive experiments with interesting insights, e.g., the CapPa backbones are most competitive when exploiting all the token embeddings (and not merely average pooling them) which makes sense as they are all used for cross-attention.\n", "weaknesses": "- The layout of the tables and figures is hard to follow: Table 2 appears before Table 1, Table 5 is not discussed anywhere, Figure 3 is placed long after it is discussed, and Table 10 (L238) does not exist. Explaining the CLIP* and 8k/16k meaning in the caption of Table 2 would also help.\n- How do the compute requirements (e.g. gpu memory) of the CLIP* with 8k/16k batch size compare with CapPa?\n- Most experiments use a proprietary dataset (WebLi) for pretraining and the code is not provided, which harms reproducibility.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The layout of the tables and figures is hard to follow**\n\nThank you for bringing this to our attention. We will improve the alignment of tables and figures with the text flow in the next revision.\n\n\\\n**How do the compute requirements (e.g. gpu memory) of the CLIP\\* with 8k/16k batch size compare with CapPa?**\n\nThe default Cap/CapPa model and training setup is chosen such that it closely matches the corresponding CLIP* setup in terms of actual accelerator hours (and examples seen). Table 1 in the paper reports TPUv4 hours per billion examples seen along with parameter count (which is also matched for the two model families). Below are the memory requirements per chip when training the models on 64 TPUv4 chips (the same setup as used to determine TPU hours in Table 1). The memory requirements of CapPa and CLIP* are comparable for the same batch size as well.\n\n| model\t\t\t| TPUv4 mem |\n|:------------------|-----------:|\n| Cap/CapPa B/16\t| 19.62 GiB |\t\t\n| CLIP* (8k) B/16\t| 20.05 GiB |\n| CLIP* (16k) B/16\t| 31.46 GiB |\n\n\\\n**Most experiments use a proprietary dataset (WebLi) for pretraining and the code is not provided, which harms reproducibility.**\n\nThe experiments on the publicly available LAION-400M data set (Sec. 4.4, Fig. 6) show that our most important results transfer to publicly available data. Furthermore, we only perform very simple text-based filtering of the WebLI data (which is from the public web) as done by prior work [4, 25], and we do not use any image-text similarity based filtering or other sophisticated filtering procedures. We are working on a code release and are hopeful to publish code before the conference.\n\n\\\n**Potential negative societal impact is not discussed.**\n\nWe plan to include the following discussion in the paper:\n\nOur models fit in the broader context of large scale vision-language pretraining and as such share many of the benefits and issues of related models such as [40, 21, 52, 33, 50]: They produce versatile vision models which obtain strong performance on natural images, on OCR-related tasks, and also when combined with a generative language decoder. These capabilities enable many useful applications (e.g. assistive technologies, medical imaging), but also potentially harmful ones (e.g. surveillance).\nWe generally recommend either employing the CapPa vision encoder with a new, task-specific prediction head, or using the pretrained decoder for scoring only. We do not recommend the pretrained decoder for downstream image captioning applications without further refinement, as it is trained on a large number of alt-texts from the web. Harmful biases should be carefully assessed in the context of the concrete downstream application and prediction head used. For example, when combining the encoder with a (potentially pretrained) decoder for captioning or VQA, an assessment of hallucinations, attribute binding issues and stereotypical attribution should be done.\n"}, {"review_id": "VUCpa4MntE", "reviewer": "Reviewer_mqQw", "summary": "This paper shows that training a ViT with the image-to-text generation (i.e., image captioning) objective is an effective way to learn good visual representations for downstream tasks. The paper systematically compares between the image captioning and the image-text contrastive learning (CLIP) objectives, where image captioning leads to ViTs that are comparable to the CLIP-ViT. Furthermore, the paper proposes parallel decoding as an additional pre-training objective to complement the conventional autoregressive decoding.", "questions": "- How does the size of the text decoder affect the representation learning performance?\n\n- What if the pre-training is performed using a pre-trained text decoder such as T5, does it improve representation learning?\n\n- Would Cap and CLIP have a complementary effect if they are combined as a multi-task pre-training objective such as in BLIP? ", "limitations": "Yes the authors have addressed the limitations.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper systematically demonstrates a refreshing observation: image-to-text generative learning leads to vision encoders as good as contrastive-learned ones. This opens up more research opportunities for visual representation learning using language supervision.\n\n- The pre-training details are well-controlled to ensure a fair comparison between Cap and CLIP.\n\n- The paper performs a comprehensive evaluation on the pre-trained ViT, including both image classification tasks and vision-language tasks. Table 8 is very nice in particular to compare the frozen ViTs.\n\n- It is interesting to see that Cap/CapPa outperforms CLIP on tasks that require fine-grained language understanding.\n\n- The proposed parallel prediction makes sense intuitively as a way to enforce stronger supervision on the ViT.\n", "weaknesses": "I do not find major weakness from this paper. There is a minor limitation as detailed below. More questions about the paper are listed in the next section.\n\nMinor limitation: Since Cap-ViT is pre-trained using an encoder-decoder paradigm, its representations would be more suited for similar encoder-decoder tasks (image caption, VQA) compared to CLIP-ViT. Therefore, frozen adaption may not justify the advantage of Cap-ViT on such tasks. It would be good to also report fine-tuning performance.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Minor limitation: Frozen adaption may not justify the advantage of Cap-ViT**\n\nWe actually tried fine-tuning the image encoder along with the decoder for both CLIP* and Cap/CapPa models and did not obtain an improvement for any of the models. This is consistent with prior work which did not observe an improvement either for CLIP-style models when fine-tuning with the same decoder-based setup, see [2, Sec.5.7]. Frozen adaptation is also favorable from a computational perspective as no gradients have to be propagated through the encoder.\n\n\\\n**How does the size of the text decoder affect the representation learning performance?**\n\nAn ablation of the decoder depth for Cap can be found in Table 7 (right). 3 and 12 layer decoders obtain a 1-3 points lower 10-shot classification accuracy across the majority of the eval sets compared with the (default) 6 layer decoder.\n\n\\\n**What if the pre-training is performed using a pre-trained text decoder such as T5, does it improve representation learning?**\n\nThis is an interesting question. We investigated this setup with a pretrained T5-Base decoder (which has 12 instead of 6 decoder layers as Cap/CapPa). First, to enable stable training we needed to reduce the learning rate from 1e-3 to 5e-4 and set the optimizer beta2 parameter to 0.95. Furthermore, training was only stable when unfreezing the cross-attention parameters. We also explored variants where we re-initialized the cross-attention parameters, and another variant where additionally all decoder parameters were trained. For all other design choices we follow the Cap-ViT B/16 setup with the 900M example schedule. None of these variants performs better than Cap overall, and the more we allow the decoder to deviate from its language-pretrained weights the better the vision performance gets.\n\nUsing a pretrained T5 decoder (10-shot linear eval.)\n| model                          |   ImageNet |   CIFAR100 |   Pets |   Cars |\n|:-------------------------------|-----------:|-----------:|-------:|-------:|\n| Cap                            |       49.7 |       56.0 |   72.6 |   74.7 |\n| Cap (12 dec. layers)           |       48.7 |       54.8 |   74.4 |   73.8 |\n| Cap T5                         |       42.8 |       44.9 |   69.3 |   62.3 |\n| Cap T5 (rein. xatt.)           |       43.7 |       45.7 |   68.3 |   66.9 |\n| Cap T5 (rein. xatt., unfreeze) |       48.6 |       55.2 |   72.0 |   75.6 |\n\n\\\n**Would Cap and CLIP have a complementary effect if they are combined as a multi-task pre-training objective such as in BLIP?**\n\nWhile there is anecdotal evidence in the literature that these losses can have complementary effects (see e.g. [32, 33, 52]) these experiments do not take pretraining compute into account and/or do not remove unnecessary network components when ablating losses. By contrast, our experiments control for these factors and show that both Cap/CapPa and CLIP lead to competitive vision encoders. It is therefore unclear how much complementary signal they provide when controlling for model capacity and pretraining compute. We leave analysis of this aspect at scale for future work.\n"}]

Author rebuttal:
We thank all the reviewers for carefully reading our paper. We appreciate their thoughtful comments and the overall very favorable assessment. In particular, we liked the many suggestions for additional experiments, some of which we were able to run. We believe these experiments will make the paper stronger.

Please find the detailed responses to each review below.

Meta-review: The work received unanimous positive reviews. The work presented a comparison of contrastive and generative approaches in the context of visual representation learning from image-text pairs and demonstrate that when training data, compute, and model capacity are matched, generative approaches are competitive. Further, results on ARO benchmark provide evidence to suggest that generative approaches might have an advantage when representing relations, attributes, and order. 

AC agrees with the reviewers and finds the work relevant to the vision-language community. Given the enthusiasm from reviewers and high-quality experiments, the AC is pleased to recommend this work for a **Spotlight**! 

The AC also encourages the authors to continue investigating the advantages of generative approaches over contrastive approaches beyond the single ARO benchmark used in this work.
Decision: Accept (oral)