Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning 
Authors: Zaid Khan
Yun Fu
Venue: ICLR 2023 poster (2023)
Keywords: vision-language
CLIP
image-text retrieval
transformers

Abstract: Contrastive vision-language models (e.g. CLIP) are typically created by updating all the parameters of a vision model and language model through contrastive training. Can such models be created by a small number of parameter updates to an already-trained language model and vision model? The literature describes techniques that can create vision-language models by updating a small number of parameters in a language model, but these require already aligned visual representations and are non-contrastive, hence unusable for latency-sensitive applications such as neural search. We explore the feasibility and benefits of parameter-efficient contrastive vision-language alignment through transfer learning: creating a model such as CLIP by minimally updating an already-trained vision and language model. We find that a minimal set of parameter updates ($<$7\%) can achieve the same performance as full-model training, and updating specific components ($<$1\% of parameters) can match 75\% of full-model training. We describe a series of experiments: we show that existing knowledge is conserved more strongly in parameter-efficient training and that parameter-efficient scaling scales with model and dataset size. Where paired-image text data is scarce but strong multilingual language models exist (e.g. low resource languages), parameter-efficient training is even preferable to full-model training. Given a fixed compute budget, parameter-efficient training allows training larger models on the same hardware, achieving equivalent performance in less time. Parameter-efficient training hence constitutes an energy-efficient and effective training strategy for contrastive vision-language models that may be preferable to the full-model training paradigm for common use cases.
Code and weights at https://github.com/codezakh/LilT.

Published as a conference paper at ICLR 2023
CONTRASTIVE ALIGNMENT OF VISION TO LANGUAGE THROUGH GARAMETER-EFFICIEN TRANSFER LEARN-
ING
Zaid Khan, Yun Fu
Northeastern University, Boston, USA [khan. za, y. y.fu/eeorrhhesstem..el za,,
ABSTRACT
Contrastive vision-language models (e.g. CLIP) are typically created by updat- ing all the parameters of vision model and language model through contrastive training. Can such models be created by a small number of parameter updates an already- trained language model and vision model? The literature describes techniques that can create lasion-language models by updating a small number of parameters in a language model, but these require already aligned visual represen- tations and are non contrrasas,,ve hence unusable for aatency-sensitve applications such as neural search. We explore the feasibility and benefits of parameter-efficien contrastive
alignment through transfer learning: creating model such as CLIP by minimally updating an already- trained vision and language model. We find that a minimal set of parameter updates (<7%) can achieve the same per- formance as full-mode training, and updating specific components (<1% % param- eters) can match 75% of full-model training. We describe series of experiments: we show that existing knowledge is conserved more strongly in parameter-efficte training and that parameter-efficien scaling scales with model and dataset size. Where paired-image text data is scarce but strong multilingual language models exist (e.g. low resource languages), parameter-efficient training is even prefer- able to full-model training. Given fixed compute budget, parameter-efficient training allows training larger models on the same hardware, achieving equivalent performance in less time. Parameter-efficien training hence constitutes an energy- efficient and effective training strategy for contrastive vision- Mannuula models that may be preferable to the full-model training paradigm for common use cases. Code and weights https: /ttp://gttttt ccom/codezakh/iil
INTRODUCTION
Advances in transfer learning within the field natural language processing (Houlsby et al. 2019b] Ben Zaken et al.) 2022) have shown that when adapting novel task, updates to small percentage of neurons (< (%) in large, pretrained wransformer-baee language models can achieve nearly equivalent results to finetuning the entire model. (ung al. (2021) showed that given the existence of already-aligned visual representations (e.g. CLIP's visual encoder) only small number (4%) of parameters in pretrained language model need to be updated for the language model complete tasks such as visual question answering using the already -alened visual representations. However, the creation of aligned vision and language representations typically involves updating all the parameters of language model and vision model, often randomly initialized (Radford et al. 2021). Zhai et al. pretrained vision model are used as an initialization, only the
(202T) find that if the weights neurons of the language model need to be updated to align the visual and language representations and match or exceed the performance of full-mode model training, resulting in a 50% reduction in trainable parameters. We take this line of investigation to its natural conclusion, asking given that strong, pretrained vision and language models both exist, can we minimally update both of their parameters align their representations?
Answering this question is valuable for two reasons. From a practical perspective, contrastive vasion-language alignment constitutes form of large-scale pretraining and hence heavy energy

---

Published as a conference paper at ICLR 2023
LiTT: locked image lockeed text
Unirnodal Pretraining
tuning
align
Language Model
Vision Model
lock model parameters insert trainable modules selectively unlock critical parameters
large-scale, unpaired image and language corpora
inage-text pains (patentially small-scale)
Figure 1: A conceptual diagram. After unimodal pretraining, parameter-effcieet transfer to trastive lasion-language alignment is achieved by changing as few as 0.3% of the parameters from initialization, matching the performance of full model training.
expenditure. Methods for parameter-efficien transfer learning result in significantly reduced GPU memory requirements, and can therefore lower energy costs. Second, collecting millions of images with textual annotations is prohibitively expensive when millions of image-text pairs cannot be scraped from the internet, such as in the case of low resource languages or images from domains that require expert descriptions. In these cases, transfer learning by maximally preserving knowledge from strong, unimodal pretraining becomes compelling. Our contributions can be summarized as follows.
We show contrastive vision-language models can be created by updates to a relatively small (<7%) set of parameters in pretrained vision and language models, which dub LilT (Locked image-language tuning) for brevity.
We conduct an detailed empirical study of combinations and interactions of various methods for parameter-efficient transfer learning.
We show that contrastive vision-language models created with parameter-efficien transfer learning conserve useful existing knowledge from their initializations better than full model finetuning, and this has benefits in realistic scenarios.
conduct most of experiments the COCO dataset, and conduct additional scaling experiments with a larger dataset of 1.5M pairs. There is possibility that our conclusions may not hold beyond this range. Second, we choose to focus zero-shot classification and information retrieval tasks. Our conclusions may not hold for other uses of image-text embeddings, such using them input for downstream vision- languae tasks. Finally, contrastive vision-language models. Thus, our conclusions may not apply those based on other architectures. Despite these limitations, we believe our conclusions are useful because there are realistic situations in which there are much fewer than L.5M image-text pairs (e.g. low resource languages) available.
Limitations Similar Desai & Johnson (2021)
we explicitly limit the scope of the study to transformer-basee
Outline First, we cover background material ($2.1). then introduce approach of parameter- efficient transfer learning for contrastive vision-language alignment ($2). We then describe experi- ments and discussion of experimental results ( followed by related work ($4).
METHODS
The basic idea of our approach is to align a vision model and a language model by updating a small percentage of their parameters by gradient descent. This involves four main elements. First, the vision and language model must initialized from strong, pretrained vision and language models, rather than random initialization. Second, we lock all the parameters in each model. Third, selectively unlock critical parameters. Fourth, we insert small trainable modules into each model to aid adaptation. There multiple ways of implementing these strategies, which
in this section.
2

---

Published as a conference paper at ICLR 2023
2.1 BACKGROUND
In this section, we briefty cover the mechanics of contrastive language image alignment as used by (Radford al. 2021), well the common "two-tower" (Zhai et al. 2021), dual transformer encoder architectures employed by CLIP-style models. Contrastive language image alignment pulls representations of matched image-text pairs together, while pushing those of unmatched pairs apart. The goal is to learn an image encoder fe and a text encoder Ie such that given an image-text pair x"), the encoded representations fe (x') and 96 (x") are close under distance metric if they are semantically similar and far apart if not. Let {=/, xT'1 I k=1 be a batch of b image-text pairs. For each image æ in image-text pair { }. the matched text æ is the positive, while all other texts within the batch are used negatives. The image-to-text contrastive loss C/ for at, is then
exp (sk.A)
ch (1k (2) (=]))==)=
log
Ej exp (8b.))
where S/,5 is the similarity of the k-th image the j-th text. The similarity function is usually taken to be the cosine similarity, which can be easily computed as fo (æ') 90 (x") ) if the representations are normalized to unit length. Conversely, the text-to-image contrastive loss for æx is
exp (
C ao (o =
log E, cxp (5T..)
The complete training loss then becomes
C = Σ c + c
(I)
Architectures for contrastive language image alignment must encode both texts and images to vector representations. This is usually implemented using separate text encoder and image encoders. A variety of choices are possible for these encoders, but we restrict ourselves to the popular (Radford
2021: Li al. 2021a Yao al., 2021 Khan et al., 2022 Zhai et al. 202T, Yang al. 2022 Wang et al. (2021) choice of transformer Vaswani et al. 2017) architectures, specifically, the
al.
BERT (Devlin et al.] 2019) family of language models for the text encoder, and the ViT (Dosovitskiy 2021) family for the image encoder. Let t(.) denote an arbitrary architecture from one of the above families. After consuming an input x, the transformer i(-) produces a sequence of vectors
et al.
t(x) {Zela: Zi..... }, where Zels is the embedding of the [CLS] ] token, which is taken to be the representation of the input x following dimensionality reduction by a trainable linear projection.
2.2 ADDING ADAPTERS
Aligning the representations of a language transformer and vision transformer is typically done by updating 100% of the parameters one (Zhai et al. 2021) I or both (Radford et al.) 2021) of the transformers. By freezing the transformers, we exclude full-model training, and must use an alternative strategy to align the image and text representations. A promising approach is inserting a small (relative each transformer), trainable module into the frozen, pretrained transformers that can learn to modify the internal representations of the transformer it is placed within, such that the representation spaces of the frozen vision and language transformers become aligned while leaving the pretrained parameters untouched. We explore two such modules: layerwise adapters (Houlsby et al. 2019a He al. 202T) and "deep" adapters.
Layerwise adapters (Houlsby et al.) 2019a) have been used to adapt pretrained transformer-based language models to new tasks while only updating 2 3% of model parameters. layerwise adapter is inserted before each layer normalization (Ba al. 2016 layer in transformer, and consists of a weight matrix that downsamples the input, followed by an activation function (we use GELU (Hendrycks & Gimpel ) and weight matrix that restores the input to the original dimensionality, and finally, a residual connection. We depict the architecture / placement of layerwise adapters in Fig 3
3

---

Published as a conference paper at ICLR 2023
pretrained
randomly initialize
recieves gradient
N+1
Adapter
N +
transformer stack
XN
XN
<lock>
XN
<lock>
grow stack
freeze pretrained layers
<lock>
<lock>
<lock>
Inputs
Inputs
Inputs
Figure 2: Growing the transformer encoder stack to add a trainable deep adapter to a locked model. The deep adapter is architecturally identical to layer from the encoder stack.
Another solution is to treat the frozen encoders feature extractors, and learn trainable adapters that align the frozen image and text features. Transformer architectures can be seen as stack of identically structured transformer encoder layers, natural solution to the problem of designing a trainable adapter atop stack of frozen transformer encoder layers is to grow the stack, and keep the newly added layers trainable. This yields a generic approach (Fig, to frozen transformer from any of the standardized families (e.g. BERT (Devlin et al. 2019), ViT (Dosovitskiy al. 202T) that only requires small number of parameters to recieve gradients (= 7% for bert-base).
add trainable adapter
2.3 UNLOCKING PARAMETERS
We try two strategies for selectively unlocking
parameters in frozen transformer: unlocking
Outputs
Outputs
the layer normalization (Ba et al. (2016) param-
eters, and BitFit (Ben Zaken et al.
2022). Stan-
LayerNorm
dard transformers (Vaswani et al.
2017 have
two layer normalization (Ba al.
2016 mod-
Dense (upsample)
ules for each transformer encoder layer, and these are known play an important role ($4). Each layer normalization layer has learnable scale y and bias parameters that apply an el- ementwise scale and shift to the input of the layer normalization layer. In the first strategy, allow the layer normalization layers to main unlocked and receive gradient updates. In BitFit (Ben Zaken et al.] 2022), (Bias-term Fine- tuning), the additive bias terms of every module in transformer encoder layer are allowed to remain unlocked and receive gradient updates, Both of these strategies unlock small percent- age (0.24% and 0.31% of the parameters in 12-layer base transformer respectively).
Adapter
Dense
Activation
LaverNorm
Dense (downsample)
Adapter
Self-Attention
Inputs
Frozen
Trainable
Inputs
Figure 3: The architecture and placement of layer- wise adapters combined with layernorm unlock- ing strategy.
2.4 IMPLEMENTATION DETAILS
Datasets We draw 591, 753 image-text pairs
from the training set COCO201 4 et al. (2014), following the split Karpathy Fei-Fei (007)). The weights of the vision encoders initialized from DeiT Touvron et (2021), and the text encoders are initialized from SimCSE (Gao et al., 2021). We train each model with batch size of 512 4x NVIDIA A6000 GPUs for 15 epochs, using the Adam optimizer (Loshchiloy
4

---

Published as a conference paper at ICLR 2023
Table I: An ablation study with
the text encoder and a ViT-B/16 the image encoder. An indicates the component is locked and does not recieve gradient updates, while indicates the opposite. LN T .) indicates the layer normalization weights in the text encoder were locked while those of the image encoder recieved gradient updates, and vice versa for LN(
1). 0 is the trainable linear projection. TR and IR mean text retrieval and image retrieval scores across
Rank-1,5,10. Deep (Fig 3) and Layerwise (Fig. 2) adapters are detailed in 2.2 and BitFit in 223
Components
Flickr
Imtge V2
TE
IE
Unlock Strategy
Adapter
% Trained
IR
Acc-1
(a)
Frozen
LN
0.00 %
1.3
LN Only
LN LN LN
0.04 %
24.3
21.6
4.3
Projection Only
0,20%
38.7
31.8
6.7
LifTZ
0.24%
62.3
51.74
12.5
LI
BilFi
0.31
62.6
52.1
12.6
(
LA w/o LN
LN
Deep
6,96 %
57.5
47.8
9.02
(2)
Li
LN
Deep
6,99 %
68.6
58.5
12.9
(h)
LiT WO
IN
Layerwise
6.97 %
74.8
63.9
12.0
(1 LWT
LNO
Layerwise
7.01 %
75.4
64.4
12.2
(
T (BiiFit)
BilFit
Layerwise
7.09%
75.3
64.4
12.2
(k)
LA (BitFit)
BitFit
Deep
7.06%
68.7
58.4
13.2
(1) LIT
IN
56.01 %
66.1
$3.5
15.0
(m) LIT (reversed)
LN
43.99 %
53.7
46.22
8.8
(n)
LIT DA
LNO
Deep
65.87 %
84.2
75.2
13.6
(0) LIT + LiiT
IN
Layerwise
57.57%
76.7
64.9
13.84
(p) CLIP
LNO
100.0 %
75.8
65.8
12.3
& Hutter, 2017) optimizer with a weight decay of 0.02. The learning rate is warned up to le in the first 10 epochs, and then decayed to le-5, We use random crops of resolution 256 x 256 with RandAugment(Cubuk et al. 2020), with colors transformations removed following Li et al. (2021a).
3 EXPERIMENTS
We conduct experiments zero-shot multimodal classification, image-text retrieval, and multilingual image text retrieval to investigate the following research questions.
1. Can contrastive vision language models be created through parameter-efficient transfer learning?
2. How do different methods for parameter efficient transfer learning interact with each other?
3. Do contrastive vision language models created through parameter-efficiee transfer learning conserve useful knowledge from their initializations better than full-model finetuning?
4. Does parameter-efficient transfer learning scale with respect to model size and dataset size?
We evaluate all models on five tasks: zero-shot natural natural-language guided image classification (Radford al. 202I), image-to tonge-to-texx. to-text retrieval (TR), exxt-to-image retrieval (IR), and O-shot TR/IR For zero-shot classification, we use the ImageNetV2 (Recht et al. 2019) test set. For IR/TR, we use the COC02014 test split of Karpathy & Fei-Feil (2017). containing 5k images and 25k captions. For zero-shot IR/TR, we use the test set of Flickr30kd Plickr30<<Plummrr et al. 2015), containing 1k images and 5k captions.
3.1 ABLATION STUDY
The results of the study are displayed in Table After updating only 0.24% of parameters, parameter unlocking methods achieve equivalent zero-shot classification performance to full-model training: compare (d) & (e) to (p). However, parameter unlocking alone is insufficient to achieve the image- text retrieval abilities of full-model training, but adapter-based methods (f-k) can match full-model training (p) in both zero-shot classification and image-tex retrieval. BitFit and layer normalization unlocking interchangeable parameter unlocking strategies (<c 0.2% difference between (ffj) and (b/i)). LilT LilT L (h), with the layerwise adapters, substantially better (+e 7%) at image text retrieval than LifT (f), and only slightly worse classification. LilT and LiT complimentary (m/n), and
5

---

Published as a conference paper at ICLR 2023
Table 2: Cross-lingual zero-shot retrieval. A multilingual bert-base model is aligned with ViT-B/16 on English image-text pairs from COCO, and evaluated on image-texi pairs in languages unseen during alignment.
RU
PL
TR
ZH
KO
IT
ES
TR
IR
TR
IR
TR
IR
TR
IR
TR
IR
TR
IR
TR
IR
LIT
45.17
40.17
44.0
41.83
24.17
23.33
64.67
61.0
34.17
29.67
60.17
56.0
65.67
62.33
CLIP
59.17
54.83
33.33
79.0
74.0
35.33
65.33
75.67
69.5
La
58.5
51.33
60.33
55.33
42.33
35.0
74.17
67.67
44.67
35.67
74.5
68.83
77.0
74.17
LifT Lwck
61.83
57.0
63.0
56.5
46.5
41.0
79.0
72.83
50.0
43.67
77.67
72.17
79.17
74.5
14.17
13.83
13.83
11.67
113.17
111.17
10.0
1-1.17
17.67
18.33
16.67
16.83
13.5
15.0
is possible to only align only one the encoders in a parameter-efficien manner. While LiT (k)
excels at image classification, it suffers from a similar problem as parameter unlocking strategies: it relatively poor at image text retrieval.
First, it is clear that creating contrastive vision-language models through parameter- efficient transfer learning is feasible, and there are clear differences between model capabilities induced by different parameter-efficient transfer learning methods. Layerwise adapters stand out as the parameter-efficient transfer learning strategy capable of matching or exceeding full-model training. However, in cases where the language distribution is sufficiently simple (e.g. list of singular words), parameter unlocking is sufficient, and casier to implement. Deep adapters stand out for their ability to achieve better performance than full-model training when combined with LiT (m).
Discussion
3.2 CONSERVATION OF KNOWLEDGE FROM INITIALIZATION
We hypothesize that parameter efficient transfer learning preserves more knowledge from initialization than full model finetuning, and this is beneficial in realistic scenarios. Low-resource languages likely do not have large-scale image-text pairs available to train multimodal CLIP-like model for that language. However, unimodal, multilingual language models that have been trained on a dataset containing sentences from a given low-resource language often exist. A possible solution in this situation train a CLIP-like model on available image-text pairs from a high-resource language, while using a multilingual language model as the text encoder. The resulting model may be able to generalize to image-text retrieval tasks in a language unseen during vision- lannuan alignment due to the multilinguality of the pretrained text encoder. We simulate this setting by aligning pretrained multilingual BERT-bas. model with an saaugeNtt-rettaiing ViT-B/16 on English-only image-text pairs, and evaluate on image-tex pairs in six differennt languages that the model never provided paired images for. If parameter-efficien training preserves more knowledge from initialization, and that knowledge is useful, we expect that the retrieval model created through parameter efficient transfer learning should retain more of its multilingual language ability, and hence display greater accuracy on non-English languages.
We reuse the English training data from $2.4 and evaluate each model on the test set of Aggarwal & Kale (2020), which contains 1400 image-fext pairs, split equally between Russian, Polish, Turkish, Chinese, Korean, Italian, and Spanish. We summarize results in Table 2 LilT L outperforms CLIP on 12/14 14 tasks (5.3% absolute improvement), while LilT DA achieves better performance than CLIP on 1 1/14 tasks (1.4% absolute improvement). This suggests that parameter-effiiies transfer learning conserves more information from initialization, and that information useful for multimodal tasks.
3.3 SCALING WITH RESPECT TO DATA AND MODEL SIZE
Can parameter-efficient transfer learning take advantage of larger models and larger amounts of data? We test the the performance of parameter-efficient transfer learning as the amount of image-text pairs is increased to 1500k from 591k (Table 4 and as model size is increased (Table from base (< 200M params) to large (== 700M params). When the amount of training pairs available triples, parameter-efffcient transfer learning continues to match the performance of full-model training: (b) vs (d) in Table I Similarly, the performance of parameter-efficient transfer learning improves as model size increases: (a) (b) & (c) (d) in Table 3

---

Published as a conference paper at ICLR 2023
Table 3: Zero-shot task performance of base/large models after parameter-effiiient training. L/D A indicates adapter types, corresponding to (rows h/f in Tp
Model (591k Training Pairs)
Flickr
ImageNet V2
Configuration
# Trainable
% Trained
TR @
IR@1
TR@5
IR@5
Acc-1
Acc-5
(a)
pabase
14.65 M
7.51%
47.6
34.46
74.1
64.92
12.94
28.39
(b)
Lalge
25.92 M
4.06%
57.6
42.18
82.2
72.38
13.97
30.89
(c)
LilT
14.67 M
7.01%
56.8
41.7
81.1
70.74
12.18
27.78
(d)
Loa llage
51.18 M
7.43%
63.5
50.7
88.5
79.14
14.05
31.31
(e)
Lis--base
109.28 M
56.01% %
44.1
29.64
72.1
59.94
15.0
29.44
(f)
CLIP-base
195.13 M
100.0%
56.1
44.3
81.7
71.98
12.29
28.44
Table Zero-shot performance base models after larger-scale pretraining (1.5M pairs).
Model (1.5M Pairs)
Flickr
ImageNet V2
Configuration
Trainable
Trained
TR@:
IR@1
TR@5
IR@5
Acc-1
Ace-5
(a)
LiT-base
109.28
56.01%
48.8
32.72
78.1
63.02
20.63
38.12
(b)
hasee
195.13 M
100.0%
60.5
43.8
84.7
72.16
16.61
35.14
(c)
LilT DA -base
14.65 14.67 M
7.51%
50.4
35.66
78.2
65.3
16.98
35.53
(d)
LilT Lil -base
7.01%
61.1
44.5
85.6
72.9
15.83
35.31
3.4
WHAT HAPPENS DURING ALIGNMENT?
We attempt understand how alignment changes the language and vision model by studying the layer normalization layers of each model. Let fe an image encoder Io be a text encoder. We initialize fg with weights from DEiT (2021). We then lock all parameters except the layer normalization layers (configuration (c) in Tab. Ip. and train the model following the standard CLIP training procedure, resulting encoders (fo, J.). In total, we have four different models: the unaligned and aligned image encoders fo) and the unaligned and aligned text encoders (go. Fo). Without loss of generality, we describe our procedure for the text encoder pair (ge- Let LN (7, B)) and LN? B), denote the two normalization sublayers of the i-th layer in the transformer encoder stack. For layer E 1. N, we plot the L norm of the difference between the trainable layer normalization parameters 7, of aligned and unaligned encoders. We plot the results Fig Surprisingly, the text and image encoders display clearly opposite patterns (negative Pearson's In the text encoder, the difference between the aligned and unaligned layer normalization parameters decreases with depth - layer normalization parameters in the deeper layers of the text encoder change less result of alignment training. This is the opposite the image encoder. In the image encoder, the layer normalization
al.) (202T). and Is with weights from SimCSE Gao al.
pair of aligned
L2 of (aligned- unaligned)
LN 1. A.ightt (p -0.61) LN1.bias (p=-0.82) weight (p= = -0.69) LN2.bias (p= -0.66)
40
40
40
30
30
20
20
20
20
10
5
10
10
Layer Depth
Layer Depth
Layer Depth
Layer Depth
Figure The depth the layer normalization layers affects how much they are changed by alignment training, and the pattern is reversed between the image and text encoders. is the Pearson correlation coefficient, and the translucent blue/yellow shading indicates 95% % confidence intervals.
7

---

Published as a conference paper at ICLR 2023
Flickr TR@1
Flickr IR@1
Flickr Mean Retrieval
30
11.2
0..
Accuracy
Accuracy
Accuracy
12
10
Baseline
-12
Puttern
Pattern
3.0
4
9
3
9
Layers Fully Frozen
Layers Fully Frozen
Layers Fully Frozen
Figure 5: We freeze all parameters except for the LN parameters, then progressively lock LN parameters by layer. Fig suggests that freezing the LN parameters in the deepest layers of the language model and the shallowest layers of the vision model (Pattern A) should have smaller effect on performance than the opposite pattern (Pattern B), relative to the baseline (LNs in every layer unlocked) which we observe.
parameters which shift the most result of training are the deepest. We conduct another experiment with 50k pairs (Fig 5
test the consequences of this pattern.
Discussion The patterns in the layer normalization layers may indicate that during alignment, the language and image modalities undergo changes at different semantic levels. The shallowest three layer normalization layers of the ViT-B/16 experience a z 70% lower magnitude shift than the deepest three layers. The shallow layers of vision transformer attend more to local information (Raghu al. 2021), while the deeper layers attend more to global context. Intuitively, this makes sense should expect an asymmetry between the amount of information in a short image caption compared to dense image. Simple natural language concepts has already been exploited by certain vision- lssoun-la models - (Khan et al. 2022] Li et al. (2211a) align the lower half of their text encoder to the visual encoder, while using the top half for a different purpose. This makes sense, given that the lower layers of the text encoder seem to change the most during alignment.
often visually complex. Interestingly, this
4 RELATED WORK
VisionLLanguuge Pretraining The dual-encoder CLIP (Radford et al. 2021 (400m pairs) and
ALIGN (Jia et al. 202T) (Ib+ pairs) architectures were the first attempts at large-scale contrastive image-language alignment using the InfoNCE (van den Oord et al. 2018) loss to maximize the
mutual information between matched image and text pairs. Subsequent work (Pham et al. 202
Li et al. 2021b Yao et al. 2021 Cui al.) 2022 Yang et al. 2022; Khan et al. 2022
Li al.
2021a) has improved on the training tasks, dataset, and architecture of CLIP. While systems util utilizing
multimodal encoder and cross attention Li al. (022); Khan al. (2022) Wang et al. (022) y
et al. 2022);; Zhu et (2021) perform better on benchmarks, their multimodal encoder makes them unsuitable for detency-sensitiie search application, because rather than learning separate but aligned image and text embeddings, they learn single multimodal embedding for an image-text pair. Thus, neural search remains the domain of contrastive vision-language models.
Frozen Language Models Tsimpoukelli et al. (222)) demonstrated that pretrained large language models are capable of quickly adapting to image understanding. They use an autoregressive transformer-ben based language model, which is frozen. A trainable ResNet (He et al., 2016) is then trained to transform images into input the frozen transformer can understand, by backpropagating the loss through the frozen transformer. MAGMA |Eichenberg et al. (2021). FROMAGE) Koh et al. (2023) and FLAMINGO Alayrac et al. (2022) scaled the conceptual approach of Tsimppoolllll et al. (2021) to billions of parameters, and recently, Merullo al. (2022) have shown that a simple linear mapping is enough to allow a frozen large language model to (roughly) understand visual input, as long the visual encoder has been trained to represent visual concepts aligned to language (e.g. CLIP). However, emerging approaches such as BLIP-2 ei al. (2023) show that by combining soft prompting with frozen L.M and trainable visual encoder, LLM can achieve state-of-the-art
8

---

Published as a conference paper at ICLR 2023
accuracy visuolinguistic understanding tasks such as visual question answering. Lu et al. 2021 propose the idea that transformers trained on language are capable of form of universal computation. and can adapt to tasks even if they frozen, and do so better than fine-tuned models. However, Rothermel et al. (2021) find the findings may be reversed under certain hyperparameter settings. Interestingly y, both note that the normalization layers seem to play important role in this adaptation.
Parameter-Efffcien Finetuning Many forms of adapters (Houlsby et al. 2019b Karimi Mahabadi
Mahabadi et al., 2021) have been explored in natural language processing. VL-Adapter 2021) investigate adapters in vision-language, but assume aligned visual representations. Lester et al.
al.
[021
(Sung et al.
2021) find that for very large language models, garameter-efficient adaptation approaches such as soft prompting are equivalent to finetuning the large language model. [Luu et al. (2021) extend this finding, showing that combining soft prompting with adapters can often exceed finetuning on a given downstream task. Both prefix (Li & Liang (021) and prompt (Lester et al. (02T)) tuning can also be understood as exploiting the knowledge in frozen transformers, as their optimization loops involve freezing the language model, effectively turning it into part of the loss. Zhang & He (2020) develop a training scheme that progressively unfreezes / freezzes layers of a transformer language model, and significant improvements in training speed. Progressive growth approaches (Gu et al. 2021) slowly increase the depth of a transformer as training proceeds.
Layer Normalization in Transformers Kovaleva et al. (2021) find that the representations of transformers contain outlier dimensions that disrupt the quality of the learned emhedding, and point to High-magnitude parameters in the layer normalization layers. A variety of techniques targeting layer normalization in transformers have been proposed, with various benefits. Xiong et al. (2020) prove that the placement of layer normalization layers relative the residual connection in the transformer block contributes to learning instability under large learning rates, and propose an alternate placement. In contrast, FixUp (Huang et al. 2020) develops a novel initialization scheme for transformers that enables removing the normalization layers entirely. ReZero (Bachlechner et al. 202T) adds learnable gate parameter to each residual connection before layer normalization, and demonstrate training extremely deep transformers quickly.
5 CONCLUSION & FUTURE WORK
We show that the performance of full model training for contrastive vision language alignment can be matched by updating small number of parameters in existing vision models and language models, followed by an insertion of trainable modules. This suggests that the current paradigm of full-model training for contrastive vision language alignment involves significant unnecessary computation, and can be replaced by parameter-efficien transfer learning when the downstream use cases natural- natural-language classification or image-text retrieval. Current aligmment strategies align representations from the top of each encoder stack. We find that in the text encoder, alignment changes the normalization parameters in the shallowest layers the most, while it is the opposite for the image encoder. Investigating and exploiting the asymmetry between vision and language could yield further benefits for multimodal understanding or more efficient training strategies. For future work. it would be interesting to analyze whether CLIP-like models created through parameter-efficctn transfer learning are similar to CLIP ways other than performance - for example, are they more or less biased? Or more or less robust to distribution shift? Another useful line of investigation would be probing dision-lanuuaa the model to understand language. In summary, we believe that existing training methods are not fully exploiting the knowledge that exists in their initializations. Our approach presents one simple but effective way use that knowledge.
models further to understand how alignment training effects the ability of
ACKNOWLEDGMENTS
This work was supported by a faculty award from NEC Laboratories America.
REFERENCES
Pranav Aggarwal and Ajinkya Kale. Towards zero-shot cross-lingual image retrieval, 2020.
Jean- nann-Bapiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan

---

Published as a conference paper at ICLR 2023
Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. ArXiv, abs/2204.14 14198, 2022.
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, 0bs/1607.064333 2016.
Thomas C. Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, G. Cottrell and Julian McAuley. Rezero is all you need: Fast convergence at large depth. In UAI, 2021.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-effiient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pp. 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: (0.18633/1120022.cclshhoo https: pettp:://aclanthol1. httplooyy.org///// wttp:///22.....2oooror....... aal.
URL
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Ekin D. Cubuk, Barret Zoph. Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with reduced search space. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp. 3008 3017. Computer Vision Foundation / IEEE, 2020. doi: 10. 10/10//95W500049 2020.00359 URL
CVcRt 2020/0t22tt /
Cubuk SrndauomennpPaaccc.ca...tttttttt wtw._eaatttaa_eeettttttttooo With_a
Yufeng Cui, Lichen Zhao, Feng Liang Yangguang Li, and Jing Shao. Democratizing con-
trastive language- image pre-training: A clip benchmark of data, model, and supervision. ArXiv, abs/2203.05796, 2022.
Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 141-142, 2012.
Karan Desai and Justin Johnson. VirTex: Learning Visual Representations from Textual Annotations.
In CVPR, 2021.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 1- doi: 10.18653/v1/111-44225 URL
4186. Association for Computational Linguistics, 2019.
https://doi.com htt.://...0......11/110... 1tt..1166////1199//
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
FicbFaNTTy.
Constantin Eichenberg, Sid Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. Magma multimodal augmentation of generative models through adapter-based finetuning. In Conference on Empirical Methods in Natural Language Processing, 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive bert training. In NAACI, 2021.
10

---

Published as a conference paper at ICLR 2023
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. ArXiv, 0ts/2110.04366. 2021.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. ArXiv, 6l6.04006.84155. 2016.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In ICML, 2019a.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019b.
Xiaoshan Huang, Felipe Pérez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In ICML, 2020.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision- lsiion-languag representation learning with noisy text supervision. In ICML, 2021.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. In Annual Meeting of the Association for Computational Linguistics, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell., (9(4):664 676, 2017. doi: 10. 10.100///AMM2221065555333 URL
1109/ TPAMI 2016.2598333
Zaid Khan, B Vijaykumar, Xiang Yu, Samuel Schulter, Manmohan Chandraker, and Yun Raymond Fu. Single-stream multi-level alignment for vision-language pretraining. ArXiv, 2022.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding Language Models to Images for Multimodal Generation, January 2023.
Olga Kovaleva, Saurabh Kulshreshtha. Anna Rogers, and Anna Rumshisky, Bert busters: Outlier dimensions that disrupt transformers. In FINDINGS, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. ArXiv, abs/2104.08691, 2021.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 9694-9705 Curran As- sociates, Inc., 2021a. URI
www.iinn...nnnnueeeeeeeeeee.. newripiippp iirsiis.... ccc/papper/2221//H1e
505259752223337777777778
pdd
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language-image pre-training for unified llaguael understanding and generation. ArXiv, abs/2201.120086. 2022.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip- Bootstrapping pre-training with frozen image encoders and large language models. ArXiv, abs/2301. 12597, 2023.
11

---

Published as a conference paper at ICLR 2023
Xiang Lisa Li and Perey Liang. Prefix-tuning Optimizing continuous prompts for generation. Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), abs/2101 1000900, 2021.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm. ArXiv, abs/2110.05200. 2021b.
Tsung Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs//405.003 405.0312, 2014.
1405. 0312
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning be comparable to fine-tuning universally across scales and tasks. ArXiv, abs22110.06002., 2021.
Iyy Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. ArXiv, abtp//b1.//111110010 2017.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modal tasks. ArXiv, abs/2206.08916, 2022.
Kevin Lu, Aditya Grover, Picter Abbeel, and Igor Mordatch. Pretrained transformers universal computation engines. arXiv preprint arXiv:2103.05247 2021. 2021.
Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In NeurlPS, 2021.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly Mapping from Image to Text Space, September 2022.
Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Thang Luong, Yonghui Wu, Mingxing Tan, and Quoe V. Le. Combined scaling for open- vven-vocabulary image classification. 2021.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2641-264499 2015. doi: 10. 109//CCVV2015.3033.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference
Machine Learning, ICML 2021, , 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021. URL http:::/poocceeeiiiiietet ngs ml wat.asmml..ppprr. wtt.ss/vs333 radford21a a html
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In NeurlPS, 2021.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? ArXiv, Abs/1192.1088 2019.
Dan Rothermel, Margaret Li, Tim Rocktaschel, and Jakob N. Foerster. Don't sweep your learning rate under the rug: A closer look at cross-modal transfer of pretrained transformers. ArXiv, 12460 2021.
12

---

Published as a conference paper at ICLR 2023
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR,
2015.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for lision-and-language tasks. ArXiv, abs/2112.0688/ 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv's e J' J'egou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 200-212. Curran Asso- ciates, Inc., 2021. URL httpps:/ Paper.
neurips cc/papper/021/fille
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1807 (ts//887.00748, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, É ukasz Kaiser, and Illia Polosukhin. Attention is all you need. U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL httpps: www.iig...nr.sp...e http:cccpppppee/.111. BE5ee243547dee wtw.eeeeeeoseeeecaaaaaaaa
Guyon,
pdf
Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. Ufo: A unified transformer for vision-language representation learning. ArXiv,
abs/2111.10022. 2021.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequeece learning framework. In International Conference on Machine Learning, 2022.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On layer normalization in the transformer architecture. ArXiv, abs/2002.04445,, 2020.
Jinyu Yang, Jiali Duan, S. Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul M. Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. ArXiv, abs/2202. 10401, 2022.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. ArXiv, abs/2111.07783, 2021.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. ArXiv, abs/2111.079911 2021.
Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL htttt:s:::/ppcceeiiiiiii
neur neur ips pttpppppppp///2////// html
Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16783-16794, 2021.
13

---

Published as a conference paper at ICLR 2023
6 APPENDIX
6.1 ADDITIONAL DATASETS
We conduct zero-shot classification experiments on three further datasets (Table 5 CIFAR-100 Krizhevsky (2009), SVHNNetzer et al. (201T), and AmageNet-AHHendrccke et al. (2021). As CIFAR- 100 and SVHN are both standard datasets, we only briefly describe them here. The CIFAR-100 dataset consists of 60k 32x32 colour images divided into 100 classes containing 600 images per class. Each class has 500 training and 100 test images, for total of 50k training and 10k test images. We use the CIFAR-100 test set for the evaluations. SVHN is a harder version of MNIST Deng (2012), consisting of natural images of digits cropped from street-level pictures. We use the 26k test images for evaluation. ImageNet-A A consists of natural adversarial examples from the ImageNet distribution, which are natural, correctly labeled images that classifiers incorrectly classify with high confidence. We use the 7k test images.
Table 5: Evaluation additional zero-shot classification tasks. First place is in bold and second place is in red. LilT models are boxed green. Acc-I stands for top-1 accuracy, and Acc-5 is top-5 accuracy. Higher is better.
Model
CIFAR100
SVHN
ImageNet-A
Configuration
# Trainable
% Trained
Acc-1
Acc-5
Acc-1
Acc-5
Acc-1
Acc-5
(8)
LifT-tiny
736.45 K
7.37
16.98
37.49
13.0
$7.39
2,77
9.15
(b)
LiT-tiny
4.45 M
44.57
18.33
39.14
12.47
55.02
3.39
11.03
LilT-small
5.19
10.28
13.8
(d)
CLIP-tiny
9.99 M
100.0
18.74
41.1
14,97
63.18
2.73
10.49
(0)
Liss-bbse
14.65 M
7.51
29.9
53.77
11.84
57.08
5.11
13.8
(0)
LilT-large
25.92
4.06
31.33
57.93
7.39
42,21
7.61
23.44
(g)
LiT-small
28.73
56.98
26.88
47.17
12.3
59.17
5.37
16.01
(h)
CLIP-small
50.42 M
100.0
26.43
49.54
7.18
54.41
4.41
14.45
(1)
LiT-base
109.23 M
56.01
26.15
48. 68.69
11.51
55.75
5,92
18.13
(
CLIP-base
195.13 M
100.0
25.25
50.93
9.47
53.33
4.68
16.41
(k)
VGG-19 Hendrycks al.
2021
143M
100.0
2.72
202T
23 M
100.0
2.17
(m)
ResNet-101 Hendryeks et al.
202
44.7 M
100.0
4.9
(n)
Heendy-s al
2021
60.4 M
100.0
5.2
6.2 NATURAL ADVERSARIAL EXAMPLES
Vision language models display impressive performance on ImageNet-A. ImageNet-A can be con- sidered a "hard slice" of the ImageNet distribution, containing samples which are problematic for supervised classifiers. Suprisingly, the zero-shot classification performance self-supervised vision-language models on ImageNet-A matches and is sometimes greater than the performance of supervised classifiers (ResNet-50 He et al. (2016) and VGG-19 Simonyan Zisserman (2015). This may be partially due to the parameter count there are more total parameters in most of the vision- language models compared to the supervised CNNs. However, considering that the vision-language models facing harder problem (performing zero-shot classification), their performance relative to supervised CNNs is surprising.
6.3 WHERE DO THE MODELS FAIL?
On the SVHN dataset, performance is poor. The large models perform worse than random chance (< 10%). and the smaller the model, the better it performs. One explanation could be that there is no way for the models to leam correspondence between images of digits and the name of each digit, as nothing similar appears in the COCO training distribution, which only contains common objects.
14

---

Published as a conference paper at ICLR 2023
Flickr30K TR@1
Flickr30K IR@1
ImageNetV2 Top-5
Bandom Initialization
Accuracy
40
20
10
10
10a
10
10a
10a
10
Image Text Pairs Seen During Training
Figure 6: The effect of pretraining model performance.
6.4 DOES PRETRAINING MATTER?
6.4.1 PRETRAINING VS. RANDOM INITIALIZATION
We follow the standard training procedure (: 2.4 and train CLIP-base model where both of the encoders are initialized randomly, instead of using weights initialized from unimodally pretrained models (DeIT Touvron et al. (202I) and SimCSE Gao et al. (22II). We train three models, for each dataset size. The results can be seen in Fig 6 Compared to the randomly initialized model, the pretrained model is substantially better
all three datasets and all 3 model sizes. However, it likely that the benefit of unimodal pretraining will be diminished as the number of training pairs available for multimodal vision-language pretraining increases, although do not explore this.
6.4.2
DOES THE KIND UNIMODAL PRETRAINING MATTER?
Flickr30K
COC02014
ImageNet V2
SimCSE-DUE
12.5
50
BERT-DeIT
BERT DINO
10.0
40
TR@1
30
TR@1
40
Top-1
7.5
20
5.0
20
10
2.5
0
30
40
50
30
Top-5 Acc
IR@1
IR@1
30
20
20
10
0
104
105
10a
10a
10a
10a
Image Text Pairs Seen During Training
Figure 7: A comparison of different kinds of pretraining LilT performance. Each model is trained on 591k pairs.
We train LilT-base models with encoders initialized from different kinds of pretraining methods. For the text encoder, we choose between bert seart-baeeuun saee-uncasss ed Devlin et al. (2019) and SimCSE Gao al. 202T). For the image encoder, we choose between Dei'l Touvron al. 2021) and DINO Caron
15

---

Published as a conference paper at ICLR 2023
FKickr30K TR@1
Flickr30K IR@1
ImageNetV2 Top-5
Accuracy
40
betuNN BOX 1O -OT NIE
20
20
10
10
10a
10
10a
10a
10F
Image Text Pairs Seen During Training
Figure 8: CLIP appears be sensitive to the size of the text encoder than the size of the image encoder.
al. 2021). We train all models on 591k pairs following $2.4 The unimodal pretraining methods chosen do have an effect on the performance on the vision-language model. The combination of SimCSE and DeiT appears to be consistently better than other combinations, although on ImageNetV2, BERT-DeiT performs better.
6.5 ZERO-SHIOT PROMPTS
Although CLIFRadford et al. (2021) uses prompt ensemble, we use only a single prompt for all datasets except SVHN: photo of { J. For SVHN, we the prompt photo of the number { J.
6.6 ENCODER SYMMETRY
Which encoder matters more? We train three configurations of CLIP on 5k, 50k, 591k pairs (Fig. 8). One is the symmetric CLIP-base, while the two asymmetric configurations have their text encoder and image encoder respectively replaced with the "tiny" version. Across all three dataset scales, the model with the smaller text encoder performs worse. Zhai et al. (2021) find that on large scale data (10m+ pairs), the opposite holds true - a larger image encoder is better than a larger language model.
6.7 DOES LILT WORK WITH SMALLER MODELS AND LESS DATA?
We test LilT and full-model training on smaller versions of transformers, corresponding to 'bert-base', 'bert-small', 'bert-tiny', and with decreasing amounts of image-text pairs (5k, 50k). The results are depicted in Figuregand Figure TO for LilT A. There are no idiosyncratic results as model size decreased, performance decreases for both full model training and parameter efficient transfer learning. Similarly, as the amount of data decreases, performance also decreases. This holds true for all tested combinations of dataset size and model size.
16

---

Published as a conference paper at ICLR 2023
Flickr30K
COC02014
ImageNer et V2
Top-5 Top-1 Acc
TR@1
50
TR@1
50
10
10
50
IR@1
25
IR@1
20
10
10a
10a
10
108
10a
10a
Image Text Pairs Seen During Training
Figure LilT's performance scales with increasing model size and dataset size not limited to specific model size or dataset size. LilT is pictured.
Flickr30K
COC02014
ImageNet V2
Top-5 Aoe Top-1 Acc
TR@1
50
TR@1
50
25
25
IR@1
IR@1
200 2022
20
20
10
229
0
104
10a
10a
10a
10a
10a
Image Text Pairs Seen During Training
Figure 10: The performance of full-model training smaller models and with less data.
17


Reviews:
[{"review_id": "sqbCTR4J2sg", "reviewer": "Reviewer_YyXF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of efficient fine tuning vision language models. Specifically, the authors seek to answer the following question: \"given that strong, pre-trained vision and language models both exist, can we minimally update both of their parameters to align their representations?\". The authors adopted the following options 1) initializing the vision and language encoders from pre-trained ones, 2) locking their params, 3) unlocking some of the params and 4) insert a trainable modules (adapters). They showed that by fine tuning just 7% of the parameters, they were able to match the performance of the full fine tuning strategy on retrieval (Flickr) and zero-shot classification (ImageNet)", "summary_of_the_review": "This paper had experimental insights to various interesting research questions. However, it has limited technical novelty and the testbed  is narrow.", "strength_and_weaknesses": "[Strength] The authors conducted many experiments to answer various research questions such as multilingual adaptation and alignment effects. They also did extensive zero-shot experiments comparing with corresponding baselines.[Weakness] This paper is not technically novel and the ideas are simple. It is an incremental step from LiT and from fine tuning approaches that use adapters before. In addition, as stated in the papers, it has a limited set up, which include 1) only testing with COCO dataset, and later 1.5M image-text pairs. It might not work for other datasets, 2) only focusing on zero-shot classification and retrieval tasks. Their conclusions might not hold for other downstream problems, 3) only testing on transformer architecture, which in my opinion, is not a main limitation, however. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UIC3ukeIzz", "reviewer": "Reviewer_LWVe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows the good efficiency and effect of parameter-efficient contrastive vision-language alignment through transfer learning. Specifically, contrastive vision-language models can be created by updating a small set of parameters of pretrained vision models(e.g. DeiT) and pretrained language models(e.g. SimCSE). ", "summary_of_the_review": "This work illustrates contrastive vision-language models can be created by updating a small percentage of unimodals by gradient descent. Comprehensive experiment results show adapter and unlocking scheme can create CLIP-like models for natural-language classification or image-text retrieval with little effort. Although there is no explanation for this phenomenon.", "strength_and_weaknesses": "Strength\n\n[+] experiment results demonstrate contrastive vision-language models can be created by updates to a relatively small (<7%) set of parameters in pretrained vision and language models, which is a very practical idea.\n\n[+] empirical study illustrates contrastive vision-language models created with parameter-efficient transfer learning conserve useful existing knowledge from their initializations better than full model finetuning\n\nWeaknesses\n\n[-] As the authors claimed in the Limitation section, the conclusions of this work are limited to the datasets, model structure, and downstream tasks.\n\n[-] It is known that batch size plays a key role in contrastive learning. I am curious about the influence of different batch size on your conclusion.\n\n[-] parameter-efficient transfer learning is a hot but not a new topic. Except for Adapter and Unlocking, there are other methods such as Prompt-tuning. Not all parameter-efficient transfer learning methods work for contrastive alignment may be another limitation.\n\n[-] In the Abstract, \"vision-language models\" and \"vision and language model\" may make readers confused", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IkEFS9M0lc", "reviewer": "Reviewer_soUw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a parameter-efficient contrastive vision-language method by leveraging a pre-trained vision encoder and a text encoder. The key idea is to align a well-trained vision encoder and a text encoder by updating a small percentage of their parameters on a small-scale pre-training dataset. The proposed method can achieve comparable performance as full-model training.  ", "summary_of_the_review": "This paper proposes a simple parameter-efficient training method for contrative vision-language pre-training. Although the problem is important and some interesting finds are observed, the technical novelty and contribution are not significant enough.\n", "strength_and_weaknesses": "Strength:\n1. Vision-Language Pre-training (VLP) requires a huge amount of computational resources, which is not affordable by a lot of researchers. Therefore, it is important to design a parameter-efficient training strategy without sacrificing the model performance.\n2. The proposed method is simple and can be easily applied to existing methods.\n\nWeakness:\n1. The proposed method is quite similar to [1]. Although [1] is designed for fine-tuning CLIP on downstream tasks, these two works share a lot of technical solutions (e.g., layerNorm tuning and adding adapters).\n2. This paper is more like an empirical study (combination of existing techniques). The technical novelty and contribution is limited, and new empirical findings are not significant enough.\n3. One contribution of this paper is “we show that contrastive vision-language models created with parameter-efficient transfer learning conserve useful existing knowledge from their initializations better than full model finetuning”. This conclusion is obvious. Since parameter-efficient training freezes most of the parameters of the well-trained vision/text encoder, this training obviously will keep existing knowledge.\n\n[1] How to Adapt Your Large-Scale Vision-and-Language Model\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster