How Much Can CLIP Benefit Vision-and-Language Tasks?
Authors: Sheng Shen
Liunian Harold Li
Hao Tan
Mohit Bansal
Anna Rohrbach
Kai-Wei Chang
Zhewei Yao
Kurt Keutzer
Venue: ICLR 2022 Poster (2022)

Abstract: Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks.


Published as a conference paper at ICLR 2022
How MUCH CAN CLIP BENEFIT VISION-AND- LANGUAGE TASKS?
Sheng Shen"i, Liunian Harold Li Hao Tan", Mohit Bansal', Anna Robrbach' Kai-Wei Chang', Zhewei Yao and Kurt Keutzer TUniversity of California, Berkeley, +University of California, Los Angeles University of North Carolina at Chapel Hill
(sheng. 8, anna. rohrbach, zhewely, keutzer] oberkeley.edue (liunian harold. kwchang) @cs ucla edu, (haotan, mbansal) 8cs unc. edu
ABSTRACT
Most existing Vision-and-Language (V&L) models rely on pre-trained visual en- coders, using relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pre-trate vre-training usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Iman Image Pre- teeiirainin),, trained on a massive amount of image-caption pairs, has shown strong zero-shot capability various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring downstream tasks. We show that CLIP significantly outper- forms widely-used visual encoders trained with in-domain annotated data, such Bpttom TopDown. We achieve competitive better results on diverse V&L tasks, while establishing new state-of-the-an results on Visual Question Answering, Visual Entailment, and V&I. Navigation tasks.
INTRODUCTION
Vision-and-Languune Language (V&L) tasks such VQA Antol al.) 2015) test a system's ability to understand and reason about the semantics of the visual world with the help of natural language. Most V&L models rely visual encoder perceive the visual world, which translates the raw pixels into vectors from a representation space. Recent work (Anderson et al. 2018a Jiang et al. 2020 Zhang ct al. 202T observes that the visual representation has become the performance bottleneck of V&L models and stress the importance of learning a powerful visual encoder. These beggh-eeforming visual encoders are trained on manually-annotated data with class labels (e.g., ImageNet) (Russakovsky et al. 2015) bounding boxes (e.g., Visual Genome) (Krishna et al. (017). However, such detection or image classification data is costly to collect, and the visual representation limited by the pre- defined class labels. Thus, there need for visual encoder that is trained more diverse and large-scale data sources, unbounded by a fixed set of labels, and with generalization ability to unseen objects and concepts.
Recently, CLIP (Radford et al.) 2021) has been proposed to learn visual concepts with language supervision. CLIP consists of visual encoder and a text encoder. It is trained 400M noisy image-text pairs crawled from the Internet. The data collection process is scalable and requires little human annotation. CL.IP has shown strong zero-shot capabilities classification. We hypothesize that it also bears great potential for the V&L tasks. However, directly applying CLIP as a zero-shot model V&L tasks proves be difficult (Section 5 and Kim al. (202T). as many V&L tasks require complex multi-modal reasoning. Thus, we propose to integrate CLIP with existing V&L models by replacing their visual encoder with CLIP's visual encoder]
benchmarks such ImageNet
"The authors contributed equally. 'Without confusion, we use the term CLIP to interchangeably refer to both the whole CLIP model (including the text and visual encoder) and just its visual encoder. We focus studying CLIP as visual encoder and provide analysis CI. CL.IP's text encoder in Appendix A.6

---

Published as a conference paper at ICLR 2022
Masked Language Modeling Text Match Loss
Trick
Contrastive Loss
<>
Transformar
Transtormer
Street Curlibe
Vissll Encoder
X
The [MASK] [MASK]
Why the man jumping?
the pitch while...
Visual Encoder Pro (CLIP)
Veion-and--Lnnuuge Pre-training (Opliona))
Task-spacific Fine-tuning
Figure 1: The training process of V&L model typically consists of three steps: 1) visual encoder pre-training, 2) vision-and-language pre-training (optional), and 3) task-specific fine-tuning. In previous V&L models, visual encoder pre-training requires human annotated vision datasets, which is hard to scale up. Our CLIP-ViL ViL proposes to use CLIP, which is trained on image-text pairs crawled from the Internet, as the visual encoder for V&L models. This reduces the need for human annotated in the pipeline and greatly improves model performance.
We present an empirical study using CLIP as the visual encoder for diverse V&L tasks. We consider two typical scenarios: 1) we use CLIP in direct task-specific fine-tuning (Section 3): 2)
we integrate CLIP with V&L pre-training on image-text pairs and transfer to downstream tasks
(Section 4 For clarity, we denote the models used these two scenarios as CLIP-ViL (without V&L pre-training) and CLIP- (with V&L pre-training).
In direct task-specific fine-tuning, we consider three widely-adopted tasks: Visual Question An- swering (Antol et al. 2015), Image Captioning (Chen al., 2015). and Vision-and- Lasion-and-Languaee Navigation (Anderson et al. 2018b). On all three tasks, CLIP- ViL brings sizable improvement over strong baselines, 1.4% accuracy VQA v2.0, 6.5 CIDEr COCO Captioning, and 4.0% success rate on Room-to-Room navigation.
In V&L pre-training, replace the conventionally used region- based representation (Anderson et al. 2018a) with CLIP. CLIP-ViL performs exceptionally well on three benchmarks, including VQA v2.0, SNLEVE (ie al. 2019), and GQA (Hudson and Manning. 2019), setting (SotA) on VQA (76.70 70% on test-std), and SNLI-VE (80.20% on test). CLIP-ViL, with CLIP-Res50 outperforms models based on the widely used region-based encoder, BottomUp-Toppown (BUTD) ResNet101 (Anderson et al. 20188). Moreover, CLIP-ViL, with CLIP-Res50x4 surpasses VinVL- ResNeXt152 (Zhang et al. 2021), which is an extreme scale-up attempt of the region- encoder with couman-annotate data.
state-of-the-ar
2 BACKGROUND AND MOTIVATION
Disionaadd-Laguuse (V&L) models. V&L tasks require a model to understand the visual world and to ground natural language to the visual observations. Prominent tasks include visual question answering (Antol et al. 2013), image captioning (Chen et al., 2015), vision-language navigation (An derson al. 201 8a), image-text retrieval (Wang et al.) (016) and for these tasks often consist of a visual encoder, a text encoder, and a cross-modal interaction module (Kim al. 202T).
V&L models designed
We illustrate the three typical training stages in Figure 1 1) the visual encoder is trained on annotated vision datasets (Russakovsky al., 20155] Krishna al. 2017) (denoted visual encoder pre- training); 2) (optionally) pre-training on paired image-caption data with a reconstructive objective and an image-text matching objective (denoted vision-and landuaag pre-training) (Lu et 2019: 3) fine-tuning on task-specific data (denoted as task-specific fine-tuning).
We distinguish between V&l. pre-training and CLIP pre-training: V&I. pre-training models LL et al. have deep interactions hetween modalities while CLIP follows shallow-interactionn design (Section
2019
2

---

Published as a conference paper at ICLR 2022
Examples of Visual Encoder Pre-training Data
Visual objects Attributes
ImageNet Images
Curated by OpenAi
400M Images a Notay Text Crawled from the Internet
clissees
Dog
Sughboakts Stock Curlite
Based
Grid
000
Grd
Feature
Reggions Proponaal
Feature
Feature
Extraction
4
Backbone
Backbone
Backbone
Region-Based BaanedBaedddddd
2010)
Grid-Based
CLIP
Figure CLIP versus other visual encoders. Region-based methods (Anderson et al. 201 8a are trained on object detection data. For grid-based methods. previous work use either image classification (He al., 2016) or detection data (Jiang al. 2020). However, CLIP requires only aligned text.
Visual encoders in V&L models. Different models employ different visual encoders, illustrate their architectures and pre-training processes in Figure The encoders can be categorized as follows: region-based models such BUTD object detector (Anderson et al. 2018a Kamath al. 202 2) grid-based models such as Diang et al. (2020) that directly extract gridllik feature maps from the visual backbone (He al. [osovitoss et 2020).
The encoder is first pre-trained on human- annotated vision datasets. Region-based encoders are pre-trained with detection data such Visual Genome (Krishna al. 2017). Grid-based encoders pre-trained with image classification data such ImageNet (Russakovsky et al. 2015) detection data (Jiang et al. 2020). However, these manually labeled datasets are expensive construct and hard to scale up. They only provide supervision for limited number of predetermined visual concepts. This motivates us to use CLIP as the visual encoder.
Pre-training) (Radford et al. 2021 falls into the line of research that learns visual representations from natural language supervision Desai and Johnson 2020 Sariyildiz et al., 2020 Jia et al. 2022). l. CLIP follows "shallow-interacti
CLIP. CLIP (Contrastive Language-Image
design", where Visual encoder and a text encoder encode an input image and text independently, and the dot-product between the encoder's output used the similarity score between the input image and text. pre-trained with a contrastive loss where the model needs to distinguish aligned pairs from randomly sampled pairs. CLIP leverages an abundantly available source of supervision without human annotation: 400M image-text pairs found across the Internet. As a result, CLIP achieves SotA performance range image classification and image-text retrieval tasks zero-shot setting.
2.1 MOTIVATION
Despite the strong zero-shot capability of CLIP vision tasks, CLIP does not exhibit the same level of performance on certain V&l. downstream tasks. For instance, if we cast VQA 2.0 (Goyal et al.
2017) into zero-shot image-to-text retrieval task, we only observe chance performance (Section 5) Thus, we propose to integrate CLIP's visual encoder with previous V&L models (Figure I.. We consider the following CLIP variants with different visual backbones (He et 2016 Dosovitskiy
et al., 2020) (CLIP-ResNet denoted as CLIP-Res): CLIP-Res50, CLIP-ResI01,, Ce.p-Res50x4,
CLIP- VTT-B/16 and CLIP-ViT-B/32. We next describe our methods in two scenarios: 1) direct
task-specific fine-tuning (Section and 2) V&L pre-training (Section
https://gthhh..cm//peemiiCLLLL
3

---

Published as a conference paper at ICLR 2022
3 CLIP-VIL
In this section, we directly use CLIP the visual encoder in task-specific models (referred CLIP- ViL) and fine-tune on three representative tasks including Visual Question Answering (Section 3.1) Image Captioning (Section 3.2 and Vision-Language Navigation (Section 3.3
3.1 VISUAL QUESTION ANSWERING
The task of Visual Question Answering (VQA) (Antol al. 2015) provide the answer given an
image and a related question. Various methods have been introduced (Fukui et al. 2016 Yang et al.
2016; Anderson et al. 2018a Jiang al. 2018 Gao al. 2019 Jiang et al. 2020 Here, select two representative approaches (i.e., Pythia (Jiang et al. 2018) and MCAN Yu et al. 2019) to study impact of the CLIP visual encoders VQA.
Experimental Setup. We evaluate VQA v2.0 (Goyal al. (017) and follow [Tang et al. 2000 for grid feature extraction. Details of Pythia and MCAN as well as full implementa- tion details are included in the Appendix.
Table 1: Results on VQA v2.0. "t" marks results from (Jiang et al., 2020). CLIP visual encoders outperform all baselines, including strong visual encoders pre-trained with in-domain detection data (VG-* and BUTD-*)
Experimental Results. We report results on the VQA v2.0 Test-dev Test-std set in Table We compare with the following visual encoders:
Result Test-dev Test-std
VQA Model Visual Encoder
Standard ImageNet pre-trained visual en- coders (ImageNet-*);
BaggeNet-Res500
63.21
Res50
63.48
63.84
Visual encoders with SotA performance on ImageNet (BiTaa**) (Kolesnikov et al.) 2020);
Res
63.82
64.11
VG-ResNeXt-1011
67.76
Pythia
BUTD-ResNeXI-1011
68.21
Visual encoders pre-trained with detection data (VG.* and BUTD.*) (Anderson al.
59.14
59.56
62.72
62.86
2018a Jiang et al. 2020).
CLIP-Res50
65.55
65.78
Compared the baselines, CLIP visual en- coders demonstrate improvement. We espe- cially note that VG_* * and BUTD.* models are pre-trained on in-domain detection data, Vi- sual Genome, which contain the sames images VQA data. Thus, they significantly out- perform baselines without such detection data (ImageNet-* and -*) However, CLIP_* models without in-domain detection data can outperform VG-* and BUTD_* Detection data are hard to scale up and contain limited object categories, while our results suggest training visual encoders on noisy image-text data as in CLIP is promising and scalable.
CLIP-Resh R0s
66.76
67.14
CLIP-Res50x4
68.37
68.68
67.23
67.46
BUTID-RcNNeX-1101
72.01
VG-ResNeXt-1011
72.59
MCAN
CLIP-VIT-B/32
65.40
65.54
CLIP-Res50
71.49
71.72
CLIP-Res101
72.77
73.19
CLIP-Res5044
74.01
74.17
3.2 IMAGE CAPTIONING
Image captioning aims at generating a natural language description for an image. Various methods have been proposed for image captioning (Karpathy and Fei-Fei 2015 Rennie al. 2017 Anderson et al. 201 8a Luo et al. 2018 Luo 2020). We investigate the effectiveness of the CLIP model for this popular task combined with the method proposed in Luo (2020).
Experimental Setup. We experiment with the basic Transformer model adapted from Vaswani et al. (0077) in Luo 2020). Grid feature maps are extracted for each image. We evaluate our model on COCO dataset (Chen et al.) 2013). We use the standard automatic evaluation metrics including CIDEr (Anderson et al. 2016), BLEU (Papineni et al. (002), METEOR (Lavie and Agarwal 2007).
4

---

Published as a conference paper at ICLR 2022
Table 3: Unseen test results for Room-to-Room (R2R) dataset. 'SR' and 'SPL' are Success Rate and Success rate normalized by Path Length. "Pre-Training" methods are mostly in-domain pre-trained on the Matterport3D (Chang et al.
Table Image Captioning results. B@4, M, C, and S are BLUE-4, METEOR, CIDEr and SPICE metric, respectively. "* marks results from Luo (220)).
2017 environments.
CLIP-Res models outperform ImageNet pre- trained alternatives for both ResNet50 and R0sNet101, as well as the strong in-domain region-based features from BUTD.
Unseen Test
Method
SR
SPL.
No Pre-Training
Model
B@4
S
R2R (Anderson al. 2018b
20
18
BUTD Anderson et al.
2018a
36.3
27.7
120.1
21.4
RPA Wang ct al. 2018
25
23
VLP (Zhou et al. 2020
39.5
29.3
129.8
22.4
S-Follower (Fried al
2018
35
28
AnANct (Huang ct al.
2019b
38.9
29.2
129.8
22.4
RCM WWang ct al. 2019
43
38
Oscur
(Li el al. 2020
40.5
29.7
137.6
22.8
SMNA (Ma ct al. 2019a
48
35
Vin V Leow Zhang el al. 2021
40.9
30.9
140.4
25.1
Regretful (Ma al. 2019
48
40
FAST-Shorf [Kc ct 2019
54
41
BUTDrandumme Transfunner
Luo 2020
127.7
22.5
EnvDrop (Tan al.
47
ImageNet Re550nanntemee
36.2
27.6
118.8
21.2
PRESS ef 201
45
37.4
28.1
122.7
22.1
ALTR (Huang ct al.
2019a
48
45
CLIP-Res550maam Transfeeee
38.6
28.8
127.9
22.7
CG (Anderson el al. 2019
33
30
CLIP-Res Transfommer
39.2
29.1
130.3
23.0
RelGraph (Hong al.
2120
55
52
EnvDrop CLIF-VI
59
53
CLIP-ResS0 Ree-Ree55 Transfenne
40.2
29.7
134.2
23.8
CLIP-ViT-BE WTT-B/32 Transfommer
37.5
28.1
123.1
21.9
Pre-Training
CLIP-ViT-BI WiT-VIT--BIB VTT-B/16gustormer
39.8
29.5
133.2
23.4
AuxRN Zhu
2020
55
50
PREVALENT Hao
2020
54
51
VLN-BERY Hong et aL
HOSCAR
57
53
VLN-BERT
Tong el al. 202T
63
57
and SPICE (Anderson et al. 2016). The scores are obtained on Karparthy test split (Karpathy and Fei-Fei 2015) with beam search of beams. Details are given in Appendix.
Experimental Results. We report Image Captioning results with different models in Table 2 Using the Transformer architecture from (Luo) 2020), we see that CLIP-Res models outperform ImageNet pre-trained alternatives for both ResNet50 (+9.1 +1.5 in CIDEr / SPICE) and ResNet101 (+9.2 +1.5 in CIDEr SPICE). even surpasses the strong in-domain region-bases feature from BUTD and grid-based feature from BiT. As the model size grows in CLIP-ViL, the results also improve and the largest CLIP-Res50x4 achieves the best performance, although there still remains a gap to the pre-trained models that have interactive image-text pre-training phase like Oscar and Vin VLuse- Again, CLIP-ViT VIT variant leads to worse performance compared to other visual modules, that we will discuss in Section 5
3.3
VISION-AND-LANGUAAG NAVIGATION
Vision-and-languaae navigation tests the agent's 's ability to take action according to human instructions, which recently gains popularity in embodied AI (Anderson et al. 2018b Chen al. 2019 Jain et al.
2019 Chen et al. 2019 Qi et al. 2020b Krantz et al. [2200] Nguyen and Daumé III 2019 Ku et al.
2020 Specifically, y. the agent is put at location in the environment (Chang et al. 2017 7) and asked to
reach a target by following the language instructions. Here, we investigate the impact of the CLIP visual encoder on this new task.
Model Architecture. We experiment with the basic attentive neural agent as in Fried et al. (2018) (please refer to the original paper for implementation details). each time step, the agent attends to the panoramic views and the instruction to make an action. We replace the pre-trained visual encoder from ImageNet pre-trained ResNet to the pre-trained CLIP visual encoders. Different from the VQA task that uses feature map to include detailed information, we use single-vector output for the entire image following previous works (Fried et al. 2018). For CLIP-ViT ViT-B/32 models, we take the output of the [CLS] token. For CLIP- models, we take the attentive pooled feature (Radford jet al.) 2021) of the feature map. These features are also linearly projected and L2-normalized as in the CLIP model.
5

---

Published as a conference paper at ICLR 2022
Table 4: Results of Room-to-Room (R2R) and Room-across-Room (RxR) datasets with original ResNet features and CLIP feature variants. 'BT-Agent' is the agent trained with back translation (BT). 'SR' is Success Rate. 'SPL' and 'nDTW" are the main metrics for R2R and RxR, respectively. The best results are bold. CLIP-ViL shows clear improvements over the previous ImageNet-trained ResNet model.
Features
Room-to-Room
Rnom-across-Room
Agent
BT-Agent
English
Hindi
Telugu
Average
SR
SPL
SPL
SR
nDTW
#DTW
nDTW
SR
nDTW
ImageNet-Res152
48.2
44.4
53.5
48.8
35.3
50.6
37.9
51.9
37.1
52.0
36.8
51.5
CLIP-Res50)
52.6
47.4
56.2
49.7
38.8
53.3
44.1
55.7
43.5
55.5
42.1
54.8
LJP--VI-B/32
52.5
47.7
57.4
51.3
40.2
52.5
44.3
55.0
42.1
54.6
42.2
54.0
CLIP-Res MiP-Res101
53.6
47.5
56.7
49.5
41.0
54.6
44.9
56.9
42.2
55.3
42.7
55.6
CLIP- Les5-Re550x4
54.7
48.7
59.2
52.9
40.8
54.7
44.5
56.5
42.4
56.0
42.6
55.7
Experimental Setup. We apply our model to vision-and-language navigation datasets: Room-to- Room (R2R, Anderson et al. al.] (2018b)) and Room-across- Rooosss-oommoooo (RxR, Ku et al. (2020)) R2R is built on the indoor environments from the MatterPort3D dataset (Chang et al.] 2017). The environments are split into training, unseen validation, and unseen test. RxR extends the R2R dataset to multiple languages and follows the environment split. For R2R dataset, follow the hyperparameter of the publicly available implementatior" R2R-En -eR-Epr (Tan et al. 2019) and replace the input feature with the CLIP features. For RxR dataset, we change the path length and instruction length; details are given in Appendix.
Experimental Results. We show the test- unseen results of our best model (CLIP. Res50x4) and the comparison to the previous methods. On R2R dataset (in Table CLIP- ViL reaches 8% higher in SR (success rate) and 6% higher in SPL (Success Rate normal- ized by Path Length) than our baseline, En- virop. CLIP-ViL. outperforms previous non- pre-training agents and shows competitive re- sults to VLN-specific pre-trained models. On RxR dataset (Table 5). CLIP-ViL achieves the best success rate and nDTW (normalized Dy- namic Time Warping) under the mono-lingual setup (Ku al.) 2020) and 4.3% better then the previous results for nDTW.
Table 5: Unseen test results for Coom-across-Room (RxR) dataset under mono-lingual setup. 'SR' and 'nDTW" are Success Rate and normalized Dynamic Time Warping.
Unseen Test
Method
SR
nDTW
Random-Baselian (Ku et al.
2020
7.5
15.4
Mono-Bast Baseline (Kū et al. 2020
25.4
41.1
SAA (Li et al. 2021a
35.4
46.8
EnvDrop + CLIP-VIL
38.3
51.1
In Table we compare different CLIP variants with the previous standard ResNet- 152 feature extractors. These extractors are pre-trained on ImageNet and use the mean-pooled features as the representation for the image. CLIP-Res50 shows a clear improvement over the IN alternative ("ImageNet-Res152") With larger models (i.e., 'CLIP-Res Res101 and CCIP-Res50x44) the agent performance scales well on both R2R and RxR. Lastly, we find that the CLIP ViT model ('CLIP-ViT) B/32') has similar results Ce50- model. ViT also shows relatively better result when back translation (BT) is applied. The success of ViT model in VLN is possibly due to the use of [CLS] feature instead of the feature map.
VISION-AND-LAYGUAGG PRE-TRAINING
Recently, V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al. 2019 Tan and Bansal 2019 Li 2019a Su et al. 2019 Chen
et al., 2020: Zhou et al. 2020 Huang et al. 2020 Li et al. 2020 Zhang et al. 2021 Li et al. 202Tb). Before task-specific tinettunin, the model is pre-trained on aligned image- text data with reconstructive objective and an image-text matching objective. We seek to test the potential of

---

Published as a conference paper at ICLR 2022
Table 6: Evaluation results on three ssiionaand-lagguage tasks, Our model with CLIP-Res50 outperforms most BUTD-based models. Our model with CLIP-Res50x4 sets new state-of-th on VQA and SNLI-VE. It surpasses VinVL, which is scaled-up version of BUTD and undergoes more intensive V&L pre-training than ours.
Model
VisualEncoder
V&l. Pretrain Epoch
VQA
SNLI-VE
GQA Test-Dev
Data
Test-Dev
Test-Std
Dev Test-P
Test-Std
PixelBERT
ImageNet-Resthe
5.5M
40
71.35
71.42
PixelBERT
IeageNet-ResX1522
5.5M
40
74.45
74.55
LXMERT
BUTD-Res101
9.2M
20
72.42
72.54
60.00
60.30
UNITER
R0s11
6.5M
72.70
72,91
78.59
78.28
Oscar
BUTD-Res101
6.5M
118
73.16
73,44
61.19
61.23
VinVL
VinVL-ResX152
8.9M
116
75.95
76.12
65.05
65.65
CLIP-Res50
9.2M
20
73.92
74.09
78.64
78.97
59.79
60.55
CLiP-ViLp
CIJP-Res5004
9.2M
20
76.48
76.70
80.61
80,20
61.42
62.93
combining CLIP pre-training and V&L pre-training. We introduce CLiP- ViLp. a vision vision-and-language model pre-trained on image-text data with CLIP visual encoder as its visual backbone. In the following, we introduce the model architecture and pre-training process of CLiP-Vil in detail.
4.1 CLIP-VIL,
Model Architecture. CLiP-ViLp assumes a text segment T and an image as input. As in BERT, the text is tokenized into a sequence of subwords {wi, 122 .... Wgg }- Every subword is embedded as the sum of its token, position, and segment embeddings (Devlin et al.] (019) and thus the text a sequence of word embeddings {w1, W2, ... wn The image is embedded as a set of visual vectors {V1, v Jm from the grid-like feature map. The text and visual input then concatanated into sequence, {w1, W2: wn. U1, :....., Um. and processed by single Transformer. In most region-basee models, the visual backbone is frozen as fine-tuning the object detector along with the Transformer remains an open problem (Su al.) 2019). In -LiP-ViLL. the CLIP backbone is trained during both V&I. pre-training and task-specific fine-tuning (see discussion in Section 55)
embedded
Pre-training on Image-Text Data. To learn unified representations for both vision and language, we follow prior work and pre-train the model on image-text pairs. We consider three pre-training objectives from LXMERT (Tan and Bansal, 2019): I) grounded masked language modeling, where we randomly mask out 15% of words in the input sentence and train the model to reconstruct the masked words; 2) text-image matching, where the model is provided with mismatched sentence with a probability of 0.5, and trained to classify whether the text corresponds to the image; 3) visual question answering, where train the model to predict the correct answer given question.
4.2 EXPERIMENTS
Setup. We experiment with two variants of CLIP as the visual encoder, CeiP-Res50 and CLIP- Res50x4. Following LXMERT, we use the same corpora aggregated from MS COCO Captions (Chen) al. 2015), Visual Genome Captions (Krishna et al. (017), VQA (Antol et al. 2013). GQA (Hudson and Manning 2019), and VG-QA (Zhu et al.] 2016) for pre-training We follow the same pre- processing procedure and exclude any test data from the pre-training dataset. This results in 9.18M image-text pairs
For computational efficiency, we use relatively small resolution for images. We resize the shorter edges of images to 384 and the longer edges to under 640 with preserved aspect ratios. During pre-training, as the number of image patches is large, we randomly sample 100 image patches for every image following PixeIBERT (Huang et al. 2020). We pre-train the model for 20 epochs and unfreeze the CLIP backbone during pre-training and fine-tuning. For details see the Appendix.
Tasks. For evaluation, fine-tune the pre-trained model three V&L tasks: VQA v2.0 (Goyal (et al. (017)), visual entailment SNLI-VE (Xie et al. 2019), and GQA (Hudson and Manning 2019). We provide more details in the Appendix.
7

---

Published as a conference paper at ICLR 2022
Table 7: Zero-shot performance of CLIP on VQA v2.0 mini-eval. "PE" denotes we follow simi- lar prompt engineering as suggested in CLIP pa- per.
Table 8: The importance of V&L, Prepee--eeee pre-treeattt pre-training (eval- uated VQA test-dev). All three models benefit from V&L pre-traibing significantly.
VQA Question Type
Model
yes/no
number
other
Feature
No Pre-train
Pre-train
Diff
CLiP-Res50
0.037
0.057
0.0
CLIP-Res50
64.66
73.92
+9.26
CLIP-VIT
0,019
0.0
0.0
CLIP-Res50x4
69.91
76.48
+6.57
CLIP-Res50p
0.055
0.057
0.0
BUTD-Res101
66.70
72.42
+5.72
CLIP-Res101es
0.260
0.0
0.0
Pe0P-R 0.446
0.118
0.034
We include previous best pre-trained V&L models and their V&L pre-training data and epochs. As our model is based on BERT we compare only with grouped by their visual encoder type. We first note that
Results. We report the results in Table 6
models based on BERT BASE. The models our two models perform competitively on all metrics. Especially, CLIP-ViL with CesP-Res50x4 establishes a new SotA on VQA and SNLI-VE.
When comparing with the BUTD visual encoder trained on in-domain data (including LXMERT TTan and Bansal, 2019), UNITER (Chen al. 2020), and Oscar (Li et al. 2020)), our two models (CLIP-ViL ViL with CLIP-Res50 and based models. We especially note that LXMERT is trained on the same pre-training dataset and for the same number of epochs as our model, yet our CLiP-ViL, with CLIP-Res50 outperforms LXMERT on VQA by 2.59.
significantly outperform most BUTD- BUTD-Res! 101
VinVI. (Li et al. 2020) is an extreme scale-up of the region-based paradigm, which is pre-trained on multiple object detection datasets, including MS COCO (Lin et al. 2014). OpenImages (uznetsoval ct al. 2020), Object365 (Shao et al., 2019). and Visual Genome (Krishna al. 2017). Yet, our model with Ce55-Res50x4 outperforms VLnVL on VQA, while requiring significantly less steps of V&L pre-training. On GQA, our model under- pederrp VinVL. The potential reason is that GQA automatically constructed from object bounding box data, which may give eegion-based models trained on such object data significant advantage.
Lastly, we compare to Pixel-BERT (Huang et al. 2020), which takes a similar design as our model, but with ImageNet initialized ResNet. CLIP initialization clearly holds advantage over ImageNet initialization, as CLIP-Res50 significantly outperforms Pixcl-BERT with ImageNet-Res50.
5 ANALYSIS
In this section, we provide detailed analyses on few interesting phenomena we observe during our experiments, which may help guide future exploration. Quantitative and qualitative analysis are provided to support our findings.
Zero-Shot Performance of CLIP in VQA. In the original paper, CLIP is intended as a zero-shot model and shows strong performance curious if CLIP can also perform well as a zero-shot model on V&L tasks that may require complex reasoning. To conduct zero-shot image classification, CLIP (Radford et al. (02T) uses the names of all classes in the dataset as the set of candidate text and predict the most probable (image, text) pair. We thus experiment with a similar setting on VQA but modify the candidate text to be the concatenation of question and answer pair for each question. Moreover, [Radford et al. (202T) find a result improvement from prompt engineering. We follow this design by constructing "question: [question text] answer: [answer text]" as the prompt template. The results on VQA v2.0 mini evall are shown in Table All CLIP variants perform at near-chance level in the zero-shot setting while prompt engineering helps only a little. CLIP models also perform worse when the question becomes harder ("other" vs. "yes/no")) All these results suggest the need of a deep interactive model and additional pre-traiinggfiie-tuning.
various vision and image retrieval tasks. We are thus
8

---

Published as a conference paper at ICLR 2022
Benefit of V&L Pre-training. In Table compare the performance of models with without V&L pre-training. We find that V&L pre- veeetraiing brings significant performance improvement for the three models test.
Interestingly, we find that CLIP models benefit more from V&L pre-training. Our conjecture is that the additional benefit could come from unfreezing the visual backbone. Because of technical difficulty in fine-tuning the object detector, most V&L models rely on frozen region-based encoders (Lu et al.) 2019). But for grid-features such CLIP, we can easily fine-tune the visual backbone and could potentially aid CLIP to adapt to the pre-training task. We hope that our finding inspires future work further explore unfreezing the visual backbone V&L models when computational budget allows.
(a) Original
(b) CLIP-ViT-B/P -B/32
(c) CLIP-VIT-B/16
(d) CLIP-RN50
(e) CLIP-RN101 (f) CLIP-RN50x4
Figure 3: Grad-CAM Visualization of CLIP-VIT-BAA B/3- CLIP- VIT- CL1P-Res50. CLIP-Res101 and CLIP-Res50c4 for the question "What color is the woman's 's shirt on the left?""
Qualitative Comparison of CLIP Variants. In our experiments, we find that ViT variants CLIP their ResNet counterparts (Section 3.I.. We perform Gradient- Localization (Grad-CAM) (Selvaraju al. 2017) visualize the salient regions idenfified CLIP variants. We find that qualitatively. ResNet variants of CLIP localize objects better than ViT variants. For example, in Figure 3 CLIP-ResNee variants localizes the sentence "What color is the woman's 's shirt on the left?" better than CLIP-VIT variants without finetuning. This finding is inline with recent studies on vision transformers [Wang et al. 202T; Raghu et al. 2021 examples in the Appendix.
Dai et al. 202T). We provide more qualitative
6 CONCLUSIONS
In this paper, we propose to leverage CLIP as the visual encoder for different V&L models across various tasks. We experiment with two approaches: the first, we directly plug CLIP task-specific fine-tuning; in the second, we integrate CLIP with V&L pre-training and fine-tune on downstream tasks afterwards. A variety of substantial experiments on different V&L tasks demonstrates that CLIP-ViL and CLIP-ViLL ViL, baselines. Analyses from different perspectives explain certain intriguing phenomena and offer new directions for future V&I research.
achieve competitive better performance compared to strong

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We provide the code to reproduce the main results in this paper in the supplementary material, which contains comprehensive instructions reproduce our results. The code and model checkpoints will be made public.
ACKNOWLEDGEMENT
We thank anonymous reviewers for their comments and suggestions. SS and KK were supported by grants from Samsung, Facebook, and the Berkeley Deep Drive Consortium. LL and KC were supported in part by DARPA MCS program under Cooperative Agreement N66001-19-2-4032 We would like to acknowledge DARPA, IARPA, NSF, and ONR for providing partial support of this work. The views expressed those of the authors and do not reflect the official policy position of the Department of Defense or the U.S. Government.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. European conference on computer vision, pages 382-398. Springer.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 201 8a. Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Peter Anderson, Ayush Shrivastava, Devi Parikh, Dhruv Batra, and Stefan Lee. 2019. Chasing ghosts: Instruction following as bayesian state tracking. In NeurlPS.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 20188 8b. Vision-and-languune navigation: Interpreting risually-gruunde navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from rgb-d data in indoor environments. In 2017 International Conference on 3D Vision (3DV), pages 667-676. IEEE.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12538-12547.
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C CLawrence Zitnick, 2015. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.0033.
Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: Universal image-text representation learning. In ECCV. ECCV.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 1rXiv:1412.355
Xiyang Dai. Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang 2021. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 7373-7382.
10

---

Published as a conference paper at ICLR 2022
Karan Desai and Justin Johnson. 2020. Virtex: Learning visual representations from textual annota- tions. arXiv preprint arXiv:2006.0666
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-IILT.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at
scale.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency. Taylor Berg-Kinng Kirkpatrikk,,, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for ssiion-and-language navigation. In NeurlPS.
Akira Fukui, Dong Huk Park, Daylen Yang. Anna Rohrbach, Trevor Darrell, and Mareus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. In EMNLP.
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. 2019. Dynamic fusion with intra-and mnder-modality attention flow for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for dssion-and-languaee navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13137-13146.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(5):1335-1780.
Yicong Hong. Cristian Rodriguez-Opaae Yuankai Qi, Qi Wu, and Stephen Gould, 2020, Language and visual entity relationship graph for agent navigation. In NeurIPS.
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. 2021. Vinæe bert: recurrent vision-and-language bert for navigation. CVPR.
Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, and navigation. In
Eugene le. 2019a. Transferable representation learning in ICCV.
Lun Huang, Wenmin Wang, Jie Chen, and Xiao Xiao-Yong Wei. 2019b. Attention on attention for image captioning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4634 4643.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep modal transformers, arXiv preprint arXiv:2004.0088 000449
Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Fugene Ie, and Jason Baldridge. 2019. Stay on the path: Instruction fidelity in aaiio-aan--language navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1862-1872.
Chao Jia Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and llnguale representation learning with noisy text supervision. arXiv preprint arXiv: (0..200029099 059105
11

---

Published as a conference paper at ICLR 2022
Huaizu Jiang, Ishan Misra, Mareus Rohrbach, Erik Learned- Milleer, and Xinlei Chen. 2020. In defense of grid features for visual question answering. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10267-10276.
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2018. Pythia vO. I: the winning cntry to the vqa challenge 2018. arXiv preprint arXiv: 09956.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. 2021. Mdetr-modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:2104.12763
Andrej Karpathy and Li Fei-Fei. 2015. Deep descriptions. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 3128-3137.
alignments for generating image
Liyiming Ke, Xiujun Li, Yonatan Bisk. Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. 2019. Tactical rewind: Self-correction via backtracking in vision- and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6741-6749.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-langen lan-lag transformer without convolution or region supervision. arXiv preprint a3X33:2102.03334.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. 2020. Big transfer (bit): General visual representation learning. In Computer Eiton-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 491-507. Springer.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. 2020. Beyond the nav-graph: Vision-and and-language navigation in continuous environments. European Conference on Computer Vision, pages 104-120. Springer.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):3
Alexander Ku, Peter Anderson, Roma Patel. Eugene Ic, and Jason Baldridge. 2020. Room-across- room: Multilingual vision-and -nd--auuage navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4392 4412.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. In International Journal of Computer Vision (IJCV).
Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the second workshop on statistical machine translation, pages 228-221 2.1..1.
Jialu Li, Hao Tan, and Mohit Bansal. 2021a. Improving cross-modal alignment in vision language navigation via syntactic information. arXiv preprint arXi::2104.09580.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019a. VisualBERT: A simple and performant baseline for vision and language. In Arxiv.
Liunian Harold Li, Haoxuan You, Zhecan Wang, Alireza Zareian, Shih-Fu Chang, and Kai-Wei Chang. 2021b. Unsupervised vision-and languuge pre-training without parallel images and captions. In NAACI.
12

---

Published as a conference paper at ICLR 2022
Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A Smith, and Yejin Choi. 2019b. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP--ICCN..P, pages 1499.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, al. 2020. Oscar: Object-semanties aligned pre-training for vision- language tasks. In European Conference on Computer Vision, pages 121-137. Springer.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
Jiasen Lu, Batra Dhruv, Parikh Devi, and Lee Lee. 2019. ViLBERT: Pretraining task-agnostic
visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02220
Ruotian Luo. 2020. A better variant of self-critical sequence training. arXiv:2003.0997/.
arXiv preprint
Ruotian Luo, Brian Price. Scott Cohen, and Gregory Shakhnarovich. 2018, Discriminability objective for training descriptive captions. arXiv preprint arXiv:1803.04377
Chih-Yas Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019a. Self melf-monitoiing navigation agent via auxiliary progress estimation. In ICLR.
Chih- Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and 7solt Kira. 2019b. The regretful agent: Heuristic-aided navigation through progress estimation. In CVPR.
Khanh Nguyen and Hal Daumé III. 2019. Help, anna! visual navigation with natural multimodal assistance via retrospective crriossty-encouragiin imitation learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 684-695.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 1-318.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020a. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. 2020b. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9982-9991.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00022
Maithra Raghu, Thomas Unterthiner, Simon Korbblit, Chiyuan Zhang, and Alexey Dosovitskiy. 2021. Do vision transformers like convolutional neural networks? In Thirty- Fifth Conference on Newral Information Processing Systems.
Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. 2017. Self- critical sequence training for image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008-7024.
13

---

Published as a conference paper at ICLR 2022
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision. 115(3)2211255.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. 2020. Learning visual representations with caption annotations, arXiv preprint arXiv:2008.0133 01399..
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626,
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VL-BERT: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: 1rxXiv:908.000 08530.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. Xiviv preprint
Hao Tan and Mohit Bansal. 2020. Vokenization: Improving language understanding with contextual- ized, visual-grounded supervision. In Proc. of the Conference on Empirical Methods for Natural Language Processing (EMNLP).
Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout. In Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pages 2610-2621.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000-6010.
Ramakrishna Vedantam, C Lawrence Zimicck and Devi Parikh. 2015. Cider:
Consensus- based image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4566-4575.
Liwei Wang, Yin Li, and Svetlana Lazebnik. 2016. Learning deep prructure-rreervving image-text embeddings. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 5005-5013.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint 122
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In CVPR.
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. 2018. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and- language navigation. In ECCV.
Ronald J Williams. 1992. Simple statistical gradient- algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-25) -566.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint orXiv:/90/.000
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2016. Stacked attention networks for image question answering. In CVPR.
14

---

Published as a conference paper at ICLR 2022
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6281-6290
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang. Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvi: Revisiting visual representations in vision-language models. arXiv preprint FrXii:2201.003:
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision- -aiiu-la pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041-13049.
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020. Vision- navigation with self- supervised auxiliary reasoning tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 10012-10022.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In CVPR.
A
APPENDIX
A.1 VISUAL QUESTION ANSWERING
Model Architecture Pythia encodes the question with an attention-based GRU (Chung et al. 2014) network and fuse the information with multi-modal factorized bilinear pooling network. MCAN takes a LSTM (Hochreiter and Schmidhuber 1997) as question encoder and an encoder-decoder based modular co-attention network for fusing multiple representations. Both models employ an output classifier on top of the fused representation to predict the final answer. To integrate CLIP for the VQA models, we extract grid features using CLIP. For CLIP-ViT-BAL models, we reshape the patch representation from the final layer into grid features. For CLIP-ResNet models, we simply take the grid features from the last layer before the pooling.
Implementation Details We follow (Jiang et al. 2020) to resize all input images to have maxi- mum shorter side of 600 pixels (longest 1000) when keeping the aspect ratio fixed. For training the detector on the VG dataset, replace the backbone with CLIP visual module using implementation of Faster R-CNN in Detectron2" For training VQA models, we use hyperparameters of the open- source implementation 8 from (Jiang et al. 2020 for the large version of the MCAN and base version of Pythia.
A.2 IMAGE CAPTIONING
Implementation Details available implementation
For training, we follow the 'long epoch' hyperparameter of the publicly During the self-critical stage, sample 5 captions for each image as Luo (2020). For training objective, we experiment with the Self-Critical Sequence Training (SCST)
in Rennie al. (0017). where CIDEr (Vedantam al. 2015) metric is optimized using REINFORCE algorithm (Williams) 1992).
A.3 VISION-AND-LANGUAGE NAVIGATION
Model For the model architecture, we experiment with the basic attentive neural agent as in Fried al. (0018).
The agent model (i.e., another LSTM) then attends to the visual features and the language represen- tations to predict the actions. At each time step the agent attends to the panoramic views {v,,); and the instruction {w,} to make the action. The panoramic view is processed with a pre-trained
15

---

Published as a conference paper at ICLR 2022
Table 9: Comparison between grid features, CLIP features, and ImageNet-trained features on the R2R dataset. 'SR' and 'SPL' are success rate and success rate weighted by path length.
Feature
Dimension
SR
SPL
ImageNet-Res1555
2048
48.2
44.4
CLIP-Res50
1024
52.6
47.4
2048
47.6
44.7
Grid-ResX ResX101
2048
46.5
43.2
Grid-ResX155
2048
47.8
44.6
visual encoder (e.g., ResNet) and the instructions are processed by language L.STM (Hochreiter and Schmidhuber, 1997). denoted LSTM, The agent model, LSTM, then attends to the visual features and the language representations to predict the actions.
gt,i ResNet(v..))
F1,
wr)
(2)
imput, [Attn(h+-1-g.. Attn(
(3)
ho Ct "SSIM,(impui h4-1, 11-1)
(4)
where h2 and o are the hiddens and states of the action LSTM at time step t, respectively. Please refer to FFiied al. (2018) for the implementation details.
Implementation Details We apply our model to two vision aan-llnguage navigation datasets: Room-to-Room (R2R, Anderson et al. (20T8b) and Room-across-Roon (RxR, Ku et al. (2020)). R2R is built on the indoor environments from the MatterPort3D dataset (Chang et al. 2017). The environments are split into training (61 environments), unseen validation (11 environments), and unseen test (18 environments). The agent is trained on the training environments (with 14,025 navigation instructions) and tested on separate sets of environments (2.349 in the mnseen-validation and 4,173 in the unseen-test). RxR extends the R2R dataset with multiple languages and follow the environment split. Besides the multilingual nature, RxR also more diverse in the navigation paths and richer in the present language. For R2R dataset, we follow the hyperparameter (e.g.. batch size, learning rate, optimizer)
the publicly available implementation 10 R2R-EnnDroop (Tan et al.. 2019) and replace the input features [ with the CLIP features. To reduce the computational cost, the features are pre-extracted and frozen during the training of the navigational agent. For RxR dataset, take the processed multilingual data provided in Li et al. (2021a) with Stanza tokenizers (Qi et al. 2020a). Since RxR dataset contains instructions longer than R2R, we change the maximum input length to 160 (from 80) and increase the imitation learning ratio from 0.2 to 0.4 to stabilize the training. Other training hyperparameters of RxR are the same as R2R. The models are trained on one RTX 2080 Ti GPU. takes days converge in R2R and about 1.5 days converge in RxR. We report two significant digits for R2R unseen test results following the leaderboard convention.
Results Comparison Grid Features the main paper, we compare the results regarding the ImageNet-pre-triin ResNet- 152. We also report the comparison to grid features Diang et al. (020) that is trained with detection dataset. Jiang et (2020) showed that the results with these features are comparable to the original bottom-up attention with heavy detection module. The same as the these detection-traine -rrinndd grid features on VLN
VQA task Section 1, we test the performance tasks. Specifically, we use the mean pooling of the feature map the representation of each view following previous works (Anderson et al. 2018b). As shown in Table 9 under the same ResNet50 backbone find that the detection-trained grid features are par with the classification -areccattto---inneddd grid features, still showing a gap to the contrastive- trained grid features. We hypothesize that the grid features inject regional knowledge into the dense feature map thus showing good results with grid-based modules (as shown in Section 3.1]. However, pooling the feature map into a single feature vector (as in previous VLN works) leads loss of this dense information.
http:///hhh...omm//aa/papy///aaa EnvDrop
The CL.IP model uses an attention pooling module and makes modifications over the original ResNet (He 2016) backhone.
16

---

Published as a conference paper at ICLR 2022
A.4 DETAILS OF CLIP- VILP
Pre-training We pre-train with batch size of 512. The Transformer is initialized from BERT BASE and optimized with AdamW (Loshchilov and Hutter 2017) optimizer. We use linearyy-decaying schedule and peak learning rate of 1 x 10 - for the model with CLIP-Res50 and 5 10 55 for the model with CLIP-Res50x4. The ResNet is initialized from CLIP and use SGD with a learning rate of 3 x 10-3. We decay the learning rate of SGD at epochs 12, 17 by factor of 10. Per the suggestion of Tan and Bansal (2019). we only add the visual question answering loss during the later stage of the pre-training (the last 11 epochs) the model is prone to overfit to the visual question answering loss. The model is trained on Nvidia A A00 GPUs and the pre-training takes around days.
(a) Original
(b) CLIP-ViT-B/32
(c) CLIP-ViT-B/16
(d) CLIP-RN50
(e) CLIP-RN101
(f) CLIP-RN50X4
Figure 4: Grad-CAM Visualization of -LIP-VITB//32, CLIP- ViT-B/16, CLIP-Res50, CLIP-Res101
and CLIP- Res5044 for the question "What color are her eyes?".
Fine-tuning We fine-tune CLIP-ViL, three tasks: VQA v2.0, SNLI-VE, and GQA. We introduce the task specifics and fine-tuning hyper-parametery in the following.
Every example in VQA consists of an image and a question, where the task to predict the correct answer. We use the Karpathy split for training and validation (Karpathy and Fei-Fei 2015). We fine-tune the model with the binary cross-entropy loss for epoch with batch size of 256. The Transformer is optimized with Adam W and peak learning rate of 5 x 10-5 The ResNet is optimized with SGD and an initial learning rate of x 10-3 We decay the learning rate ResNet by factor of 10 after epoch 3.
SNLI-VE three-way way classification task, which involves determining the relation between an image and sentence. The three possible relations include entailment, contradiction, and neutral. We fine-tune the model with the negative log-likelihodd loss for 2 epoch with a batch size of 256. The Transformer is optimized with AdamW and peak learning rate of 5 x 10-5 The ResNet is optimized with SGD and an initial learning rate of x 3 We decay the learning rate of ResNet by factor of 10 after epoch 1.
GQA follows the format of VQA but the questions and answers GQA are automatically generated from ground-truth scene graphs. We use the same hyper- parameers in VQA.
17

---

Published as a conference paper at ICLR 2022
(a) Original
(b) CLIP-ViT-B/32
(c) CLIP-ViT-B/16
(d) CLIP-RN50
(e) CLIP-RN101
(f) CLIP-RN50x4
Figure 5: Grad-CAM Visualization of CLIP-VIT-B/32 CLIP- VIT-BV1- CLIP-Res50, CLIP-Res101 and CLIP- 4IP-Re500x4 for the question "What is just above the plate?".
Table 10: The performance of finetuned CLIP text encoder and visual encoder without VLP on VQA.
Image Encoder
Text Encoder
VQAmini eval
BERT-base RoBERTa-base
62.66 62.85
CLIP-Res50
CLIP-Res50-text
62.24
BERT-base
61.51
CLIP-VYT-B/32
RoBERTa- base
61.79
61.12
A.5 MORE QUALITATIVE EXAMPLES
Here present more qualitative examples using (Grad-CAM) (Selvaraju al. 2017) to visualize salient regions CLIP models. Figure 4 and Figure suggest that CLIP-ResNet localizes the sentence better than CLIP-VIT variants.
A.6 ANALYSIS ON THE CLIP TEXT ENCODER
We extended the experiments in Table (VQArest-er)) without pre-training while fine-tuning both visual encoder and text encoder on "QAminii-vall: The results suggest the CLIP text encoder consistently perform worse than BERT/RoBERTa counterparts even though CLIP is pre-trained with "in-domain" image-text pairs.
For directly analyzing the capability of the CLIP text encoder, we add experiments with fine-tuning CLIP text encoder on representative NLU tasks (GLUE). On the largest two tasks (QQP, MNLI), BERT-base (12 layer, 768 width) achieves 87.1 + 0.2 on QQP and 77.9 + 0.3 on MNLL CLIP-Res50 text encoder (12 layer, 512 width) achieves 72.0 + 0.3 and 51.6 + 0.4. CLIP-VIT ViT-B/32 text encoder achieves similar performance CL5P-Res50 with the same architecture. (12 layer, 640 width) achieves 73.8 = 0.3 and 53.8 = 0.3. We conduct the experiments with 32 batch size, epochs, 3 random seeds and search the learning rate in [1 10 6, 1 10 3 x 10 -5 x 10 10-$1. These results may directly reflect the inferior text encoder of CLIP which may be caused by the noisy and short text in the paired image-text data (Tan and Bansal 2020
text encoder
18


Reviews:
[{"review_id": "v8_ecERXcu1", "reviewer": "Reviewer_jbK4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The tendency that more powerful visual encoders yield more performant VL models has been discussed and demonstrated in numerous papers (from the BUTD [^1] to VinVL [^2]), which emphasized the need for powerful visual backbones.  \nAs CLIP showed the great generalization power of their visual encoders compared to the encoders trained with imagenet classification, I feel no surprise they boosted the performance of VL tasks compared to their imagenet pre-trained counterpart.  \nI must credit the paper for solidifying this tendency, but I am doubtful whether this paper opens any new directions to the community.\nAll experiments are replications of existing baselines, just switching the backbone to publicly shared CLIP weights.  \nI think not many insights besides \"switching the visual encoder to CLIP's bring the performance boost\" are given.  \nI was hoping the paper contains some analysis of CLIP's text encoder since the language side of VL models at least take something from what the CLIP's text encoder has learned, but sadly, I found only plug-and-plays of CLIP's visual encoders.  \nTo raise my recommendation, please clarify what directions the paper could suggest to the community other than the general tendency I've mentioned.  \n\n[^1] Anderson, Peter, et al. \"Bottom-up and top-down attention for image captioning and visual question answering.\" _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2018.  \n[^2] Zhang, Pengchuan, et al. \"Vinvl: Revisiting visual representations in vision-language models.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2021.  ", "summary_of_the_paper": "This paper examines how well CLIP's visual encoders are transferred on vision-and-language (VL) tasks.  \nThe paper conducted three without-VLP tasks: VQA, image captioning, and vision-and-language navigation, and three with-VLP tasks: VQA, SNLI-VE, and GQA to show CLIP's visual encoders' transferability,   \nCLIP's resnet-based visual encoders consistently outperformed their Imagenet pre-trained counterpart.  \nThe authors found that CLIP's ViT-based visual encoder performed far worse than the resnets.  \nFrom Grad-CAM and detection fine-tuning experiments, the authors speculated that ViT's features lack localization information.  ", "summary_of_the_review": "Not many insights besides \"switching the visual encoder to CLIP's bring the performance boost\" are given.  \nTo raise my recommendation, please clarify what directions the paper could suggest to the community other than the general tendency I've mentioned.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "goW9aCwyBsi", "reviewer": "Reviewer_EUMT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n1. This work presents experiments on straightforward, but significantly important baselines for the vision-and-language community. The vision features have been shown to be a bottleneck of vision-and-language systems, and a significant part of recent progress can be attributed to better visual encoders. CLIP has been greatly impactful in the field, and the experiments presented in this paper are therefore a must-know to the community, and would be of interest to many.\n\n2. The experiments presented in this work are solid, informative and representative.\n\n3. The paper is clear and well written.\n\n**Weaknesses:**\n\n1. While the the experiments presented in this paper are of great importance to practitioners and researchers interested in vision-and-language, a shortcoming of this paper is the lack of novelty.\n\n2. The authors experiment only with a subset of the publicly released CLIP models, and do not provide numbers for the largest available models. While OpenAI only very recently released more models, it would be great to see their performance in a future version of this paper.\n\n**Other comments:**\n\nCLIP ViT-B could refer to both the ViT-B/32 and ViT-B/16 architectures, and it would be great if authors clarified this in their manuscript.", "summary_of_the_paper": "This paper explores how features from CLIP (Contrastive Language-Image Pre-training, Radford et al., 2021) affect the performance of vision-and-language models across a series of tasks. The authors explore using CLIP as a visual encoder in two settings, plugging its features directly into task-specific fine-tuning; and combining CLIP with intermediary vision-and-language pre-training before fine-tuning on downstream tasks. The experiments suggest that the simple change to CLIP offers significant benefits over commonly used encoders such as BottomUp-TopDown, providing very strong results across a wide range of vision-and-language tasks.", "summary_of_the_review": "This work introduces a must-know baseline for vision-and-language research: using CLIP as a visual encoder. While this paper does not introduce a novel method, it provides important experimental information to many in the community. The results here presented would be of great interest to many, and I recommend it's acceptance to the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2Td7wH2eG1C", "reviewer": "Reviewer_ZXD1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As a paper focusing on empirical studies, it has the following strengths:\n+ The paper conducts extensive experiments to study the generalizability of CLIP pretraining across various tasks. The ablation studies on different types of visual encoders also provide insights into the application of pretraining models.\n\n+ The proposed method is able to establish new state-of-the-art performance in multiple vision-and-language tasks, which can benefit future research in the related fields.\n\nHowever, there are also some notable weaknesses:\n- While I am convinced that CLIP could be beneficial for downstream tasks, it is still unclear to me what are the advantages of CLIP over existing V&L pretraining techniques and how does it achieve good results. For a purely empirical study on an existing technique,  I expect more in-depth analyses than simply showing a collection of results.\n\n- After comparing CLIP with the other V&L pretraining techniques (e.g., Table 2 and Table 6), it seems that it is more advantageous only if used together with V&L pretraining. This is a bit tricky, because the V&L pretraining relies on 9.18M additional samples. It is unclear if the improvements achieved by the method truly result from the advantages of CLIP or are simply due to the use of more external data.\n\n- Relating to the above comment, the analysis on unfreezing encoder is also problematic, as the comparison is between models without pre-training and with V&L pretraining. The higher accuracies of the latter ones could be attributed to the use of additional data, instead of the differences between frozen/fine-tuned visual encoders.\n\n- It appears that CLIP only works well with grid-like features extracted from convolutional neural networks, i.e., no experiment on regional features and worse results when combined with a visual transformer. I am aware of the analyses on ViT-B, however, they only point out the defects of a single model. It does not answer the question of why CLIP can not be applied to different visual transformer models and features.\n", "summary_of_the_paper": "This paper performs empirical analyses of applying the CLIP to various vision-and-language tasks. It demonstrates the potentials of the model in generalizing to different downstream applications, and provides some suggestions for model deployment. Experimental results show that CLIP pretraining leads to competitive performance and further combining it with V&L pretraining can outperform existing methods.", "summary_of_the_review": "While I have no doubt that CLIP could offer opportunities for developing new state-of-the-art models, this paper falls short of explaining the reasons behind the successes of CLIP. I also found several experiments and analyses problematic and do not quite support the claims. Overall, this is a borderline paper to me as it does demonstrate the potential of CLIP with good results. Therefore, I am slightly leaning towards accepting it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1DmrJVOqcOz", "reviewer": "Reviewer_Rnvh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \nThe paper revisits a broad range of vision and language tasks (VQA, VLN, SNLI-VE, Image Captioning) and evaluates the would-be performance for previous state-of-the-art methods by replacing their visual backbones with CLIP-ResNet/ViT. The diversity of the tasks considered and the two scenarios explored in the paper (straightforward fine-tuning and V&L pre-training) illustrate the strengths of the CLIP-based encoders and highlight some of their limitations. \n\nWeaknesses: \nThe motivation for all V&L experiments looks weak because it stems from the following single observation “if cast VQA 2.0 into a zero-shot image-to-text retrieval task, we only observe chance performance. Thus, we propose to integrate CLIP’s visual encoder with previous V&L models.” The prompt engineering considered for VQA is not persuasive and does not imply the same straightforward introduction of CLIP-based backbones to VLN / Image Captioning tasks. Also, I think it is necessary to explain the “chance performance” for VQA, at least for the binary yes/no question type. E.g., does 0.037 mean that one can simply flip the model’s answers?\n\n“Unfreezing” the Visual Backbone helps if done correctly – it is a well-known fact supported by BUTD-Res101 “unfreezing” experiment in the paper. However, CLIP-Res50 benefits more from pre-training than BUTD-Res101. This observation requires further elaboration.\n\nMost of the models considered in their respective tasks seem outdated as of 2021 (e.g., Pythia 2019, MCAN 2019). At the same time, the authors avoid direct comparison to the methods that require large-scale “supervised” or “self-supervised” pre-training, e.g., VLN-BERT, although CLIP itself can be considered as such.\n\nThe authors may want to include findings on localization ability of transformers from the following papers in the later versions of the manuscript:\n-\tDo Vision Transformers See Like Convolutional Neural Networks? arxiv 2021\n-\tDynamic Head: Unifying Object Detection Heads with Attentions, CVPR 2021\n-\tPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions, ICCV 2021\n", "summary_of_the_paper": "The paper utilizes recently proposed CLIP-ResNet/ViT visual encoders instead of the standard backbones to conduct a large-scale empirical study for several vision and language tasks. The experimental setup aims to answer the question in the title of this paper by conducting 1) the task-specific fine-tuning of existing V&L models with CLIP-based visual backbone and 2) performing full-scale V&L pre-training.\nThe resulting performance in VQA, Visual Entailment, V&L Navigation suggests that CLIP is a viable alternative to the existing visual representations (e.g., ResNet-based models pre-trained on ImageNet).", "summary_of_the_review": "It’s not surprising that the CLIP-based vision backbone - after being trained on 400M image-text pairs - can outperform ImageNet-trained baselines in vision and language tasks, especially given its zero-shot performance illustrated in the original work.\nThe presented results can be helpful for the research community, although the answer to the question posed in the title remains largely open. The outcomes of experiments vary from task to task, with no appropriate discussion followed even for the difference between CLIP-ResNet / CLIP-ViT-B features. As it reads in the Conclusion section: “Analyses from different perspectives explain certain intriguing phenomena and offer new directions for future V&L research.”  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are in agreement that this work is a useful, clear, documentary piece of work that shows the utility of CLIP on a number of popular V+L tasks.  There is a somewhat persistent concern that simply demonstrating that a stronger visual encoder leads to improvements downstream is not an insightful result on which the community can build.
Decision: Accept (Poster)