DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning
Authors: Taku Hasegawa
Kyosuke Nishida
Koki Maeda
Kuniko Saito
Venue: EMNLP 2023 Main (2023)
Keywords: Vision and Language
TL;DR: This paper presents DueT, a transfer learning method for vision and language models. DueT prevents catastrophic forgetting and achieves efficient learning with a reduced number of trainable parameters.

Abstract: This paper presents DueT, a novel transfer learning method for vision and language models built by contrastive learning.
In DueT, adapters are inserted into the image and text encoders, which have been initialized using models pre-trained on uni-modal corpora and then frozen.
By training only these adapters, DueT enables efficient learning with a reduced number of trainable parameters.
Moreover, unlike traditional adapters, those in DueT are equipped with a gating mechanism, enabling effective transfer and connection of knowledge acquired from pre-trained uni-modal encoders while preventing catastrophic forgetting.
We report that DueT outperformed simple fine-tuning, the conventional method fixing only the image encoder and training only the text encoder, and the LoRA-based adapter method in accuracy and parameter efficiency for 0-shot image and text retrieval in both English and Japanese domains.

Pre-training of vision and language for learning the semantic correspondence between the two modal- ities has achieved impressive performance gains wide variety downstream tasks. In partic- ular, contrastive learning methods for image-text dual encoder models such CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) have attracted much attention. These models learn visual concepts directly from raw text about images by jointly train- ing image and text encoders. A crucial aspect of the high performance of such models is that they are trained on massive amount of image-text data, For example, CLIP was trained from scratch on a dataset of 400 million pairs collected from the Web without human annotations, and ALIGN was trained on dataset of over one billion pairs. Thus, build model for new non -nnglihhshh language DueT Image-Teer Text Contrastive Transfer Learning with Dual-adapter Tuning
Taku Hasegawa'; Kyosuke Nishida'; Koki Maeda"! Kuniko Saito'
NTT Human Informaties Laboratories Tokyo Institute of Technology
{taku. hasegawa, kyosuke syosuke.nishiiaa@@tt. nishidajentt.com
Abstract
From-scratch
Locked-image Tuning
Dual-adapter
Fine-tuning
Tuning (DueT)
This paper presents DucT, novel transfer learning method for vision and language mod- els built by contrastive learning. In DueT, adapters are inserted into the image and text encoders, which have been initialized using models pre-trained on uni-modal corpora and then frozen. By training only these adapters, DueT enables efficient learning with reduced number trainable parameters. Moreover, un- like traditional adapters, those in DueT equipped with gating mechanism, enabling effective transfer and connection knowledge acquired from pre- treitrained uni-modal encoders while preventing catastrophic forgetting. We report that DueT outperformed simple fine- tuning, the conventional method fixing only the image encoder and training only the text encoder, and the in accuracy and parameter efficiency for 0-shot image and text retrieval both English and Japanese domains.
R
Adapler
Image
Todl
Image
vou
Image
Dee Choptee
Figure 1: Design choices for transfer learning in image- text contrastive learning with dual encoders. Left: Learn- ing all parameters from scratch or with fine-tuming. Cen- ter: LiT (Zhai al., 2022) trains only the text encoder while keeping the pre-trained image encoder frozen. Right: Our proposal, Due'T (dual-adapter tuning). trains only additional layers (adapters) inserted into each of the frozen pre-trained dual encoders.
adapter method
or a domain not covered by the existing model, it necessitates collecting ample image-text pair data for the target language or domain prior to train- ing. Another aspect is that the models are trained with substantial computational resources because it is important to form very large mini-batches for contrastive learning. Therefore, building top of powerful pre-trained models with more data- and compute-efficiee strategies is highly demanding. However, this issue has not yet been thoroughly vestigated; most prior studies (Radford et al., 2021; Jia et al., 2021; Li et al., 2021; Singh et al., 2022) fo- cused on training models from scratch using large- scale corpora. In an effort to address this challenge, LiT (Zhai al., 2022), parameter-efficient ap- proach for developing vision and language models, has been proposed. involves fixing the parame- ters of pre-trained uni-modal image encoders and exclusively training the text encoders. However. a comprehensive evaluation of the model's perfor- mance remains to be conducted.
1 Introduction
In this study, we propose DueT (Dual-adapter Tuning), method to learn the vision and language models. DueT uses a single-modal pre-trained
"qqual contribution.
Contributed during internship at NTT Lab.

---

model as fixed values for the parameters of each encoder and adds adapters equipped with gating mechanism to both encoders (Figure I, right). In experiments in the English and Japanese domains, Due'T outperformed simple transfer learning and LiT (Zhai et al., 2022). The main contributions of this study are as follows:
et al., 2022), image contrastive learning (Mu al., 2021) and Siamese representation learning (Li et al., 2022b).
Model initialization and freezing. Most of the previous work on pre-training dual encoders falls into two parameter initialization categories: the from-scratch method, as in CLIP and ALIGN, randomizes all parameters the image-text dual encoders (Yao et al., 2022; Yu et al., 2022; Yuan el al., 2021; Li et al.. 2022b), while the fine-tuning method (Li et al., 2021; Singh et al., 2022; Li et al., 2022a) initiates the encoders from pre-trained uni- modal models. Recently, another line of research has appeared in the form of LiT (Zhai al., 2022) and BASIC (Pham al., 2021), which initialize the image encoder from a pre-trained model and fix its parameters during image-text contrastive learning. aligning closely with our study.
Transferability. Our method transfers and connects knowledge acquired by pre-trained uni-modal encoders without catastrophic for- getting.
Paraaetter-fffccienyy. It outperforms con- ventional methods of image-text pre-training while using fewer training parameters.
2 Related Work
2.1 Image-text Contrastive Learning
Focus of this study. This study focuses the model architecture and transfer learning of a pair of uni-modal encoders. The key contribution to in- corporate trainable adapter layers into each frozen pre-trained uni-modal encoder. This represents a new line of research for pre-training image-text dual encoders. While this study does not focus using cross-modal modules other than dual en- coders and pre-training objectives other than con- trastive learning, the recent advances described in this section can be incorporated into our model.
CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) demonstrate that image-text dual encoder models pre-trained with contrastive objectives huge number of image-text pairs can learn strong image and text representations. They simply model the cross-modal interaction via the similarity of the global features of each modality and train the two encoders from scratch. In particular, there has been much research aimed at improving the cross- modal interaction and/or the pre-training strategy including objectives and transfer learning.
Model architectures. Dual encoder models have integrated cross-modal modules fuse image and text features via cross-modal attention, with studies exploring cross-modes modal encoder (Li et al., 2021; Singh et al., 2022; Yuan et al., 2021), image- grounded text encoder-decoder (Li et al., 2022a), and cross-modal text decoder (Yu et al., 2022). Sharing layers between the image and text encoders has also been studied (You et al., 2022). Image- text contrastive learning has also been applied to cross encoders (Zhang et al., 2021) and unified model was developed that can serve as either dual encoder or a fusion encoder (Wang et al., 2021). Pre-training objectives. Besides contrastive learning, pre-training objectives within each modal- ity and across modalities have been also stud- ied. For each modality, objectives inspired by uni- modal self-supervised pre-training schemes have been used for, e.g., masked language modeling (Li et al., 2021; Singh et al., 2022; Li al., 2022b) and causal language modeling (Yu et al., 2022; et al., 2022a) masked image modeling (Singh
2.2 Adaptation of Pre-trained Models
Parameter-efficiet learning in NLP. Efficient adaptation methods of pre-trained language mod- els for downstream tasks, such adapter tun- ing (Houlsby et al., 2019; Pfeiffer al., 2020, 2021; He et al., 2021; Mahabadi et al., 2021), prefix tun- ing (Li and Liang, 2021; Lester et al., 2021), ad- ditive methods (Guo et al., 2021; Hu et al., 2022: Zhang et al., 2020), and sparse-fine-tuning (Sung et al., 2021; Zaken et al., 2022), have been well- studied in the field of NLP. Unified frameworks combining these approaches have also been inves- tigated (Mao al., 2022: He et al., 2022).
A daatatioo of CLIP to downstream tasks. There have been several studies on extending the NLP approaches for adapting CLIP to vari- ous downstream vision-and-language tasks. To adapt CLIP encoders for image recognition tasks, CoOp (Zhou al., 2022b) and CoCoOp (Zhou et al., 2022a) apply prefix tuning to the text en- coder, while visual prompting (Bahng et al., 2022)

---

and VPT (Jia et al., 2022) the image encoder. CLIP-Adapter (Gao et al., 2021) employs a feature adapter layer atop the image and the text encoder CLIP for few-shot image recognition tasks. Tip- Adapter (Zhang et al., 2022) provides a training- free udaption for CLIP in order to conduct few-shot classification. VL-Adapter (Sung et al., 2022) and Flamingo (Alayrac et al., 2022) use the frozen im- age encoder of CLIP for visual features and per- form adapter tuning on pre-trained language mod- els. PAINT (Iharcc et al., 2022) uses interpolations between the model weights before and after finc- tuning on a task, without degrading accuracy on tasks where performance is already adequate.
3.1 Overview
Image-text contrastive learning framework. A dual-encoder model that consists of an image en- coder and text encoder is trained to predict the cor- rect image and text pairings. At run-time, each en- coder maps the image and text to dm -indmmeniional real-valued vectors, x and y, respectively. The sim- ilarity between the input and the text is quantified by taking the dot product of their vectors, $ x
Encoder architecture. Due'T assumes that both the image and text encoders employ the Trans- former architecture (Vaswani et al., 2017). DueT extends the Transformer block by inserting addi- tional modules (Gated Adapter Units: GAUs) be- tween layers, the details of which are described in Section 3.2. Here, we recapitulate the essentials of the Transformer architecture to aid readers in un- derstanding the GAUs. Transformer encoders are composed of L stacked blocks, where each block contains two types of sub-layer: multi-head self- attention and a fully connected feed-forward net- work (FFN). Each sub-layer has residual connec- tion and uses layer normalization (Ba et al., 2016). There are two approaches to applying layer normal- ization: the post-LN type such as BERT (Devlin et al., 2019), performs it after the sub-layer (outside the residual connection), while pre-LN type, such as ViT (Dosovitskiy et al., 2021), performs it be- fore the sub-layer (inside the residual connection). Figure 2 shows pre-LN type Transformer block.
Focus of this study. In contrast to prior stud- ies adapting pre-trained CLIP encoders for down- stream tasks, our focus was on enhancing the image-text contrast learning framework using ro- bust pre-trained uni-modal encoders, avoiding catastrophic forgetting. We developed model with data outside the language or domain of existing image-text models. We also examined the trade-off between performance and efficiency concerning ad- ditional parameters, and the model's performance relative to the number of training data. LilT (Khan and Fu, 2023), targeting efficient pre-training in pa- rameter utilization, adopts a method similar to ours where adapters are inserted into the image and text encoders, which have been initialized using models pre-trained on uni-modal corpora and then frozen. The differences between LilT and our method are detailed in Section 3.2.
Input-output formulation. Each encoder re- peatedly transforms the input sequence of image patches or text tokens in L blocks, resulting in a sequence of d-dimensional embeddings H² hSSEP) E Raxd The [CLS] and
3 Methodology
The goal of this method is to efficiently construct vision and language models for diverse languages and domains, through knowledge transfer from uni- modal encoders such VIT and BERT. To achieve this goal, we propose to train only gated adapter units (GAUs), which are designed for image-text contrastive learning, that are added to each of the frozen pre-trained dual encoders. This method can transfer and extend the excellent representations possessed by the pre-trained uni-modal encoders to the more challenging task of cross-modal under- standing without requiring substantial fine-tuning of the original models. The GAUS learn mod- ulate and adapt the internal features of both the image and text encoders, effectively bridging the gap between the two modalities and enabling them to interact in a more meaningful way.
hi,,.,, h/ [h'is,h [SEP] tokens are special tokens inserted at the be- ginning and end of the input sequence. To cal- culate the similarity of the input and the text, we take the representation heus E Rd as the output of each encoder, and map it into the -m-dimensional multi-modal embedding space (x, with linear projections followed by Lg-normalization
Training objective. We introduce a variant of unified contrastive learning (UniCL.) (Yang et al., 2022). Given the i-th image-text pair, we generate quadruple (æ;, yos Si, (4) via encoders and hash al- gorithms, where 2; Bdd and s; are respectively the embeddings and the hash value of the image and y, E the and f; are those of the text. We treat all image-texts teneettet--piiir pairs associated with the hash values

---

of the pair as positives P; and all other random image-text pairs that can be formed in training batch as negatives. The loss to minimized defined as the sum of two losses:
GAU +
expla. Un/T)
0
Cz.
lug
(1)
B
JBB exp(pp U,/T)"
LayerNorm
FFN
Cafe
explyTaa/11)
LayerNorm
Cim
log Σ jeB exply, 2//T)*
B
or 4 } T is a trainable parameter of the temperature to scale the logits. It is different from Florence (Yuan et al., 2021), which uses UniCI. in that it utilizes the image hash value in addition to the text hash value for constructing the positive set. In this study, the MD5 hash algorithm was used.
where k E P;
[k B
Self-Attention &
Gated Adapter Unit (GAU)
LayerNorm
Transformer Block (Pre-LN)
Figure 2: Gated adapter unit (GAU) and its integration with the Transformer block. Left: GAU is inserted af- ter the feed-forwar network each Transformer block. Right: a GAU is FFN having gating mechanism. the gating mechanism outputs 0, GAU becomes the identity function.
3.2 Gated Adapter Units
Unlike previous adapters (Houlsby et al., 2019; Khan and Fu, 2023) that have only feed-forward network (FFN), the GAUs also have gating mech- anism to facilitate transfer learning of the two pre-trained image and text encoders without catas- trophic forgetting. Gate types can be categorized into two main classifications: adaptive, where gate values depend on the input to the adapter, and static, where they do not; moreover, they can be
W down E Rdxm baown E Rm, W up E Rmxd Dup E Rd and at R are independent trainable parameters for each GAU. Let d and m denote the dimensions of the input-output and intermediate layers, respectively. We use the GeLU activation for Φ and sigmoid function for Ø, respectively. characterized either element-wise, with gate val- ues determined individually for each element, or layer-wise, where values are decided per layer. Pre- liminary experiments found negligible differences among these categories; consequently, for the sake of simplicity, we have chosen to adopt the static and layer-wise approach in this paper. Further de- tails concerning the adaptive type can be found in Appendix G. The FFN, which is independently and identically applied each position, consists of two linear transformations with nonlinear acti- vation & between them. The dimensionality of the input and output d, and the inner-layer has dimensionality m.
With the gated residual connection, the GAU makes use of the gating mechanism to adaptively trans- form or bypass (a = 0) the signals.
Unlike the previous study (Houlsby et al., 2019; Khan and Fu, 2023) in which adapter layers were added both after the self-attention and after the FFN sub-layer within each Transformer block, we place the GAU after the FFN sub-layer, as shown in Figure
3.3 Design choices for dual-adapter tuning
Parameter initialization and freezing. The pa- rameter initialization and freezing strategies for each encoder involve several design choices. We define four potential settings as an extension of the LiT definition (Zhai et al., 2022):
GAU'(HH) = (LN(E((N((H)) + d')H''
(3)
(F) (h.W) down + bloonn) W thp-
(4)
from-scratch (u). Parameters are randomly initialized and trainable (unlocked).
where the input H' is the output after the residual connection of the FFN module of the Transformer.
LN denotes layer normalization (Ba et al., 2016).
fine-tuning (U). Parameters are initialized from pre-trained model and trainable (Un- locked).
'When inserting a GAU into a pre-LN type Trans- former(e.g. ViT (Dosovitskiy et al., 2021 :fig2) that uses an LN within the residual connections of the sub-modules. insert the LN before the FFN, as shown in Equation (1). For the post-L.N N type (c.g. BERT (Devlin et al., 2019)), insert the LN after the FEN.
bocked-tuning (L). Parameters are initialized from a pre-trained model and frozen (Locked).

---

GAU funing (g). The main parameters are ini- tialized from pre-trained model and frozen (except for layer normalization). The extra parameters of the gated adapter units ($3.2) are randomly initialized (except for the gating value a') and trainable.
4.1 Pre-trained Models
We mainly used the architecture and pre-trained weights of ViT-B/16-AR-IN21K (Dosovitskiy et al., 2021) and BERT-base (Devlin et al., 2019) for the image and text encoders, if not stated other- wise. The details are given in Appendix B.
LoRA tuning (1). The main parameters are initialized from pre-trained model and frozen (except for layer normalization). The extra parameters of the LoRA unit (Hu al., 2022) are randomly initialized and trainable.
4.2 Datasets
4.2.1 Pre-Training Datasets
We used the following two pre-training datasets consisting of image and text pairs with different visual concepts and languages.
The settings can changed for each encoder. For example, Lu denotes locked-image tuning (LiT) (Zhai et al., 2022), which trains the text en- coder from scratch while keeping the pre-trained image encoder (LiT-FS); LU denotes an exten- sion of locked-imaan tuning, which keeps the pre- trained image encoder frozen and fine-tunes the text encoder (LiT-FT); gg denotes the setting of our dual-adapter tuning (Due'T): uu denotes the from-scratch setting used in (Radford et al., 2021; Jia et al., 2022) (FS); UU denotes the standard fine- tuning setting used in (Li et al., 2021: Singh et al.. 2022) (FT); and 11 denotes the setting of the LoRA tuning (LoRA). Note that DucT updates all param- eters in the layer normalization of the pre-trained encoders during training, it was done in previ- ous addaptertunnigg studies on NLP (Houlsby et al., 2019; He al., 2021). DueTT initializes the gating value a' to 0.02. Refer to Appendix H.1 for LoRA implementation specifics.
YFCC-CLIP. The YFCC100M (Thomee et al., 2016) contains the metadata on 99.2M images and 0.8M videos from Flickr (2004-2014), (Rad- ford et al., 2021) defines 15M image sub- set, termed YFCC-CLIP, filtered for English ti- tles/descriptions 3, It's split into 14,760,364, 10,000, and 10,000 pairs for training, development, and test sets. Note that the visual concepts of YFCC-CLI are relatively close those acquired by the image encoders pre-trained on ImageNet- 21k, which contains a large number of photos col- lected from Flickr.
JWeb Dataset. We collected private dataset of image and Japanese-text pairs form variety of publicly available sources the Web. We mainly used 5M, 10K, 10K pairs for the trainingJ[Web 5M), development, and test sets. Compared with YFCC-CLIP, the JWeb datasct has different visual concepts and contains various images, including shopping items, slides, PC screens, and Japanese characters. The details are given in Appendix C.
Parameter efficiency. Adapter units can reduce the number of additional parameters with small bottleneck dimension m, with total parameters in a single adapter, inclusive of gating mechanism, as (2d + () + 1). Given equivalent hidden size d and layers I for image and text encoders, total adapter parameters approximately 8Ldm. m is simple trade-off lever between performance and parameter efficiency. See Appendix A for further detail on model size and parameter efficiency.
4.2.2 Evaluation Datasets
Zero-shot image classification. We evaluated zero-shot transfer of models in visual classifi- cation tasks on ImageNet ILSVRR-2212 bench- mark (Deng et al., 2009) (IN). We also used the Japanese version: ImageNet-JP (IN-JP) (translated from ImageNet). We used the top-1 accuracy as the evaluation metric.
Zero-shot image-text retrieval. We also eval- uated the zero-shot transfer performance of mod- els in both text and image retrieval on the MS- COCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2017) and the test set of the pre-training corpus (YFCC-CLIP JWeb5M). We also used the Japanese versions of these datasets: STAIR
4 Experiments
First, we compared TueT with the standard image- text contrastive learning frameworks. 2
WWiile LilT (Khan and Fu, 2023) is contemporaneous study and utilizes different training data, direct comparison through experiments is not conducted in this paper. However, we provide a discussion based on the insights gained from our study in Appendix L..
'https: thub. http///ppeaa///LL/////////aaa// data/yfcc100m.n

---

Captions (Yoshikawa et al., 2017) (translated from MS-COCO) and Flickr30k-JP (Nakayama al., 2020) (translated from Flickr30k). The details are shown in Appendix D. As in (Li et al., 2021), I the evaluation metric was the mean value of Recall @k (R @k; k = 1, 5, 10) defined as the fraction of queries for which the correct item is contained in the closest & items to the query.
eters increases. On the other hand, small num- ber of additional parameters did not cause sig- nificant performance loss: compared with fine- tuning. which updated 210.0M parameters, Due'T updated 13.1M (4.5M) parameters; its performance on STAIR (Flickr30k-JP) was equivalent to that of fine-tuning, and parameter-efficient learning was achieved. DueT with 2.7M parameter up- dates, which is almost the same number that of LoRA (3.2M parameter updates), significantly outperformed LoRA on all of the evaluation sets. These results indicate that DueT is a parameter- efficient method.
4.3 Results
Table I shows the results of the evaluation of the English and Japanese models. DueTT outperformed the baseline evaluation in O-shot image and text retrieval. In particular, it outperformed fine-tuning (from-scratch) on the Flickr30k-JP image retrieval by 4.2 (39.2) points, while reducing the number of training parameters to about 58%. While fine- tuning achieved the highest baseline score and ex- celled on the YFCC-CLIP test set, it underper- formed in O-shot retrievals due to over-fitting in English and Japanese training data. The perfor- mance deficit was particularly noticeable when the image encoder's pre-training and training set do- mains differed in the Japanese model evaluation. LiT-FS and LiT-FT underperformed Due'T in the evaluation of both the Japanese and English models because of their poor adaptability to domains not covered by the pre-training of the image encoder. In the case of from-scratch, which does not perform transfer learning, we found that training data sizes of 5-15M insufficient. While the LoRA tuning scored lower than that of the proposed method, is very parameter-efficient and achieved the same score as LiT-FS with a small number of training parameters. A comparison between the proposed method with a smaller number of training parame- ters and LoRA is shown in 4.4.
How effectively do the GAUs work at each layer? As shown in Table 3, performance de- creased when the number of Transformer blocks into which GAUs were inserted was reduced, es- pecially when adapter insertion was limited to the image encoder only. This result shows that insert- ing adapters in both the image encoder and text en- coder is important for image-text contrastive learn- ing. On the other hand, removal of the adapters from either encoder, with the exception of the re- gion near the output layer of the image encoder. resulted in a minimal performance deterioration. However, there was a notable decrease in perfor- mance when the adapters were removed from the vicinity of the image encoder's s output layer and also when the total number of inserted adapters was substantially reduced.
How do the gates perform at each layer? As shown in Figure 3, the gate values of the image en- coder tended to increase closer to the output layer. This indicates that the effect of the FFN in the GAU increases near the output layer, which further sug- gests that the knowledge of the pre-trained model used for the initial values utilized around the input layer and new knowledge is acquired by the adapter is utilized around the output layer. On the other hand, the gate value for the text encoder was around 0.4 for all layers, which is not as extreme the gate value for the image encoder. This result in- dicates that the text encoder uses the knowledge of the pre-trained model as the initial values to create the embedding of the image-text pairs all layers. The detailed discussion of gate values is provided in Appendix I.
4.4 Ablation studies
We conducted evaluations on GAU, which is the main contribution of our research. In all of the following experiments, the JWeb5M dataset was used the training data.
Is it possible to achieve parameter-efficient learning? The proposed method changes the number of training parameters depending the dimension of the intermediate representation of the FFNS in the inserted GAU. We investigated how the number of training parameters affects the performance of the model. Table shows that accuracy improves the number of training param-
Does the initial gate value and gate learning have an influence? Table 4 contrasts cases with and without gate values (fixed a = 1.0), demonstrating

---

EN (truined YFCC-CLIP)
JP (trained JWeb5M)
YFCC-CLIP MS-COCO'
Flickr30k
IN
JWeb test
STAIR!
Flickr30k-JP IN.JP
Method
EN
1-T
T-11
I-T
T-11
1-T
T-11
1-T
T-11
1-+T
T-+1
1-T
T-II
FS
195.5
210.0
78.88
78.47
43.26
27.2
64.5
42,9
25.29
50.62
50.83
33.31
23.54
40.77
30.01
14.9
FT
195.5
210.0
88.83
88.28
61.37
40.27
83.93
62.77
47.3
72.12
72.95
60.98
50.29
78.7
65.04
37.23
LiT-FS
109.7
124.2
67.39
66.6
52.92
35.47
75.97
55.71
45.18
50.7
48.78
48.59
32.92
66.5
51.23
35.78
LIT-FT
109.7
124.2
71.35
69.71
57.69
38.0
78.87
57.18
48.46
54.77
53.23
52.62
36.39
72.37
54.92
38.49
LoRAs
1.5
1.5
53.14
53.02
52.16
37.41
75.57
59.1
40.51
52.62
52.02
47.34
39.6
65.97
56.1
32.8
DucT
56.8
57.6
86.22
85.33
61.75
42.05
84.5
64.2
55.43
73.17
73.21
61.89
52.53
81.4
69.27
42.85
Table 1: Performance of models trained on YFCC-CLIP and JWeb5M. Text and Image Retrieval (I+T, T-+1) Performance. We used BERT-base and ViT-B/16 and set m 1,536 for DueT and r r= 8 for LoRA. #TP indicates the number of trainable parameters in millions. IN(IN-JP) denotes ImageNet. t Zero-shot retrieval task.
STAIR
JIickr30k-JP IN-JP I-+T
JWeb test
STAIR
Flickr30k-JP I-T
#TP
I-T
T~I
T~>
Qriit
fixed
I-T
T-#1
I-T
T-1
T-+1
LoRAs
1,5
47.34
39.6
65.97
56.1
32.8
L.0
73.18
72.65
60.94
50.86
77.7
68.08
LoRAis
3,2
47.71
39.95
65.93
55.7
33.64
1.0
72.36
72.24
59.79
50.6
77.63
69.23
0,02
68.43
67.83
59.43
50.91
78.7
67.53
48
2.7
55.54
45.88
75.63
62.94
36.25
0.02
73.17
73.21
61.89
52.53
80.4
69.27
96
4.5
57.33
48.0
78.1
65.72
39.62
192
8.0
59.28
49.87
80.3
67.53
40.86
Table 4: Performance of models trained on JWeb5M with different initial gate values. We set m 1,536.
384
15.1
60.41
50.48
80.43
69.33
43.1
768
29.2
62.02
52.27
82.33
69.77
43.45
1536
57.6
61.89
52.53
80.4
69.27
42.85
3072
114.2
62.16
52.38
81.47
70.74
44.43
FT 210.0
60.98
50.29
78.7
65.04
37.23
Text Encoder
0.8
Image Encoder
Table 2:
Zero-shot transfer performance of models trained on JWeb5M with DueT and different inner di- mensions of FFN in adapters. #TP indicates the number of trainable parameters in millions. We used the same value v for adapters both encoders. Light blue cells represent the same level of performance as fine- tuning (FT). Indices 8 and 16 in LoRAs and LoRAin represent LoRA.
Gate coefficient
0.6
0.0
10
12
Layer num ber
STAIR
Flickr30k-Jp JP
Ave.
Figure 3: Gate values for each layer. Orange: represents the gate values of the image encoder. Blue: represents the gate values of the text encoder.
Image
Text
I+
T-I1
I-TT
T-11
1-12
N/A
52.63
45.23
69.9
63.01
$7.69
1-12
8-12
60.78
51.11
79.33
69.66
65.22
1-12
4-12
61.43
52.02
81.8
69.35
66.15
1-12
14
59.71
50.98
80.0
68.71
64.85
the efficacy of gate values introduction. Further- more, performance was improved by updating the gate value through learning. When the initial gate value was set large, the impact of unlearned GAUs was strong in the early stage of learning, making it difficult for the transfer learning progress.
1-12
1-8
60.79
52.17
81.87
68.79
65.91
N/A
1-12
58.83
47.4
75.73
62.43
61.10
8-12
1-12
60.12
50.46
77.83
65.47
63.47
4-12
1-12
61.89
52.01
79.6
69.09
65.65
1-4
1-12
59.0
48.97
77.57
65.88
62.86
1-8
1-12
61.29
50.71
79.23
67.61
64.71
8-12
8-12
57.95
49.4
76.7
66.83
62.72
4-12
4-12
61.45
$1.37
80.6
69.59
65.75
How does the number of training data affect performance? Figure 4 shows the performance of each method trained on various numbers of train- ing data in the JWeb Dataset. The details in the dataset are shown in Appendix C. The proposed method outperformed fine-tuning or LoRA when the number of training data was between 0.5M and 10M. The score of fine-tuning approached that of T as the number of training data increased,
1-4
1-4
56.69
47.13
75.4
63.49
60.68
1-8
1-8
60.03
50.61
79.97
68.78
64.85
1-12
1-12
61.89
52.53
80.4
69.27
66.02
Table 3: Performance of models trained on JWeb15M with adapters inserted in different ranges of layers. Ave. represents the average of STAR and Flickr30k-JP.

---

DueT fhnetune
-2442
gtsrb
Average (Re)) Ress, Rel00)
+16.52
CIFAR100
+99.9
food100
16.39
FGVCAircraf
+1.06
52-
+4.33
+7.08
CIFAR10
+4.30
stanfordcars
+1.05
dtd
14.39
ccuutry211
13,82
sun397
+3.05
caltech101
OxfordilIPP
0.52
+1.40
MNIST
10"
+0.42
+3.30
flowers
training (x10P)
stl10
DueT
Figure 4: Performance of each model versus number of training data. Horizontal: Number of training data in the JWeb dataset (log). Vertical: average of "recaal( @ I, @5, @ 10) of STAIR, Flickr30k-jp, and accuracy of ImageNet.
+0,00
finetuning
sst2
16
A Score(%)
Figure 5: Linear Probe on uni-modal VIT vs. linear probe FT-CLIP and DueT. Bold: Scores within the 99.5% Clopper-Pearson confidence interval show- ing significant difference from uni-modal VIT. Red: Scores where DueT significantly outperformed FT, de- termined by the 99.5% Clopper-Pearson confidence in- terval. Blue: Scores where DucT significantly underper- formed FT, determined by the 99.5% Clopper-Peane Pearson confidence interval.
while LoRA's score was relatively higher than that of fine-tuning when the number of training data was small. This result shows that DueT is effective even when the amount of training data is small.
4.5 Linear Probing
We investigated the effect of Due'T and fine-tuning the image encoders used for initialization using contrastive learning for text and image matching. On the basis of DueT and FT models trained on JWeb-5M in 4.3, linear probing was performed for the downstream task. The details of each task and experiment are presented in Appendix F.I and F.2.
in the pre-training of CLIP (Radford et al., 2021), visual and language model constructed by con- trastive learning. A single-modal pre-trained model used as fixed values for the parameters of each encoder, and adapter GAUs equipped with a gat- ing mechanism extended from the one described (Houlsby et al., 2019) are added to both encoders for training. Transfer learning of uni-modal pre- trained models is important topic in regard to CLIP, which requires large amount of training data. The fact that we were able to devise method that superior performance and parameter effi- ciency to fine-tuning, LiT (Zhai et al., 2022), and LoRA (Hu et al., 2022) is major contribution to re- search on vision and language models. The results of the experimental evaluations also gave us insight into the number of additional parameters, number of training data, and necessity of gates for transfer learning in the construction of Japanese CLIP. The results of this research will contribute the devel- opment of services such as dialogue, navigation, and content generation and retrieval that require integrated visual and linguistic understanding.
Figure 5 depicts the scores of DueT and fine- tuning relative to those of linear probing with ViT- B/16-AR-IN2IK. The results in the figure show that the contrastive learning of DueT lead to higher scores on many tasks compared with the uni-modal ViT. The results the figure show that Due'T had higher scores on almost all tasks (on 13 out of 15 tasks significantly) compared with the uni-modal VIT. Furthermore, compared with FT, Due'I had higher percentage increase in scores seven tasks, With the exception of GTSRB, Due'T performed level equivalent to FT in all other tasks. This sug- gests that DueT prevent catastrophic forgetting of knowledge and successfully combines adaptation to a new domain with knowledge utilization.
5 Conclusion
We propose DueT, which performs transfer learn- ing by using adapter tuning (Houlsby et al., 2019)

---

Limitations
Adam Coates, Andrew Ng. and Honglak Lee. 2011. An analysis of Lingl-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, FI., USA, PMI.R
This study explored only classification and retrieval as zero-shot transfer tasks. It leaves as future work the evaluation of zero-shot transfer for a broader set of tasks such as detection, segmentation, visual
question answering, and image captioning. In par- ticular, have tested the effectiveness of Due'T in the setting of learning image-text matching via contrastive learning, and it will be important to test its effectiveness on the above tasks with LM-based VLMs, such as (Alayrac et al., 2022) and (Yu et al., 2022).
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- Mooss-lingual representation learning at scale. ACLC, pages 8440-844 -451-
This study demonstrates that DueTT is a parameter-efficient method, particularly in in- stances of limited data size and computational re- sources. On the other hand, the fine-tuned model could outperform DucT when there was a large amount of data and sufficient computing resources available. While the results of this study suggest that DucT can save computational costs within a certain budget, it may be useful to consider the fine-tuned setup well, given a sufficient budget.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: large-scale hierarchical image database. In CVPR, pages 248- 255.
Jacob Devlin, Ming-Wac Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre- pre-training of deep bidirectional transformers for language under- standing. In NAACL-HLT, pages 4171-4186.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR.
Due'T is designed to be adaptable to to non- English languages and domains outside the training domain of existing CLIP models. However, this study is limited to validating DueT in Japanese and English; it is not yet clear whether will be ef- fective in other languages. In addition, we leave for future work the analysis of the effects of dif- ferences in encoders that have been pre-trained on uni-modal data used to initialize learning.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2021. Clip-adapter: Better lision-language models with feature adapters. arXiv:2110.04544
Demi Guo, Alexander M. Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with diff prun- ing. In ACL-ICCNLP, pages 4884-4896.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Mal- colm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong. Sina Saman- gooci, Marianne Monteiro, Jacob Menick. Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Bar- reira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: visual language model for fho-shot learning. In Advances Neural Infor- mation Processing Systems. Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hin- ton. 2016. Layer normalization. arXiv:1607.06 1607.064700000
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficien transfer learning. In ICLR.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, and Luo Si. 2021. On the effectiveness of adapter- based tuning for pretrained language model adaptation. In ACI/NCNLLP, pages 2208-2222
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. ICMI., pages 2790-2799.
Hyojin Bahng, Ali Jahanian. Swami Sankaranarayanan, and Phillip Isola. 2022. Exploring visual prompts for adapting large-scale models. arXiv:2203.1 17274.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In ICLR.
Tianqi Chen al. 2016. Training deep nets with sublin- ear memory cost. 1.Xiv:1604.061

---

Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre. Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. 2022. Patching open-vocabulary models by interpolating weights. arXiv; 2208.05592.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, 1tss/997..1662.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V. Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up vi- sual and vision-language representation learning with noisy text supervision. In ICML, volume 139, pages 4916.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Rabeeh Karimi Mahabadi. James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In NeurlPS, pages 1022-1035.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning. In ECCV.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. Unipelt: A unified framework for parameter-efficient language model tuning. In ACL, pages 6253-6264.
Armand Joulin al. 2017. Bag of tricks for efficient text classification. In EACL, pages 427-431.
Paulius Micikevicius et al. 2018. Mixed precision train- ing. In ICLR.
Andrej Karpathy and Li Fei-Fei. 2017. Deep visual- semantic alignments for generating image descrip- tions. IEEE Trans. Puttern Anal. Mach. Intell., 3944)6664-676.
Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. 2021. SLIP: self-superrvsion meets language-iman pre-traing pre-training. arXiv:2112.12250.
Zaid Khan and Yun Fu. 2023. Contrastive alignment of vision to language through parameter-efficient trans- fer learning. In The Eleventh International Confer- ence on Learning Representations.
Samuel G. Müller and Frank Hutter, 2021. Trivial ment: Tuming-free yet statc-of-the-art data augmenta- tion. In ICCV, pages 754-762.
Hideki Nakayama, Akihiro Tamura, and Takashi Ni- nomiya. 2020. A visually-grounded parallel corpus with tarrasetto-rggion linking. In LREC, pages 4204- 4210.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. 2018. Do better imagenet models transfer better? CoRR, abs/1805,0897
Brian Lester, Rami A1-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiet prompt tuning. In EMNLP, pages 3045-3059.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In EACL., pages 487-503.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022a. BLIP: bootstrapping language-iman pre-training for unified vasion-language understand- ing and generation. In ICML., pages 12888-12900.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya
Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun
Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In EMNLP, pages 46-54.
Junnan Li. Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty. Caiming Xiong, and Steven Chu-Hong Hoi. 2021. Align before fuse: Vi- sion and language representation learning with mo- mentum distillation. In NeurlPS, pages 9694-9705.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu. Yi-Ting Chen, Minh-Tham Thang Luong, Yonghui Wu, Mingxing Tan, and Quoe V. Le. 2021. Com- bined scaling for open-vocabulary image classifica- tion. ar1iv:2111.11 10050.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IICNLP, pages 4582-4597.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2017. Flickr30k entities: Collecting region-to-phran correspondences for richer image- to-sentence models. Int. J. Comput. Vis., 123(1):74- 93.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Jun- jie Yan. 2022b. Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm. In ICLR.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In ECCV, volume 8693, pages 740-755.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language

---

supervision. In ICML, volume 139, pages 8748- 8763.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2022. FILIP: fine- grained interactive language-image pre-training, In The Tenth Internaionna Conference on Learning Rep- resentations, ICLR 2022, Viriual Event, April 25-29, 2022. OvenReview.net.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. FLAVA: foun- dational language and vision alignment model. In CVPR, pages 1565-.
Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. 2017. STAIR captions: Constructing large-scale japanese image caption dataset. In ACL. pages 417-4211
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Y. Ng. and Christopher Potts. 2013. Recursive deep mod- els for semantic compositionaliyy over sentiment treeban. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. EMNLP 2013, 18-21 October 2013, Grand Hyatt Seartle, Seattle, Washington, USA, A meeting SIG- DAT, Special Interest Group of the ACL, pages 1631- ACI.
Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. 2022. Learning visual representation from modality-shared contrastive language-image pre- pre-training. In ECCV.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye- ung. Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners image-text founda- tion models. arXiv:2205.01917.
Johannes Stallkamp. Mare Schlipsing, Jan Salmen, and Christian Igel. 2011. The german traffic sign recogni- tion benchmark: A multi-class classification competi- tion. In The 2011 International Joint Conference on Neural Nenworks, IJCNN 2011, San Jose, California, USA, July 31- August 5. 2011, pages 1453-1460) IEEE.
Lu Yuan, Dongdong Chen. Yi-Ling Chen, Noel Codella, Xiyang Dai. Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng, Luowei Zhou, and Pengchuan Zhang. 2021. Florence: A new foundation model for com- puter vision, arXiv:2111.11432.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. 2021. How to train your vit? data, augmen- tation, and regularization in vision transformers. arXiv:2106. 10270.
Yi-Lin Sung, Jaemin Cho. and Mohit Bansal. 2022. VI-adapter: Parameter-efficient transfer learning for ssiionaand-lagguae tasks. In CVPR, pages 5227- 5237.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-models. In ACI, pages 1-9.
Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Train- ing neural networks with fixed sparse masks. In NeurlPS, pages 24193-24205.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. 2022. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, pages 18123- 18133.
Christian Szegedy et al. 2015. Going deeper with con- volutions. In CVPR, pages 1-9.
Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik. 2020. Side- tuning: A baseline for network adaptation via addi- tive side networks. In ECCV, pages 698-714.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. YFCCI00M: the new data in multimedia research. Commun. ACM, 59(2):64-73.
Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang, Yejin Choi, and Jian- feng Gao. 2021. Vinvl: Revisiting visual represen- tations in vision-language models. In CVPR, pages 5579-5588.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In NIPS, pages 5998-6008.
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. 2021. Vlmo: Unified lasion-language pre-training with oo-turr-o--mmdlltyy----
Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao. Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2022. Tip-adapter: Training-free adaption of CLIP for few-shot classification. In ECCV.
arXiv:2111.02355
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. 2022. Uni- fied contrastive learning image-text-label space. In CVPR, pages 19163-19173.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. In CVPR, pages 16816 16825.

---

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Learning to prompt for vision- language models. Int. J. Comput. Vis., 130(9))2337- 2348.
101-
Model size and parameter efficiency
102
As shown in section 3.3, the parameter efficiency of DueTT can be adjusted by setting hyper- pypem--arameter to moderate value. For example, when used ViT-B/16 and BERT (L 12 and d 768) and set m to 96, the number of parameters updated by Due'T was around million, or about 3.64% of the total number of parameters of the two original models (110 million plus 96 million).
10f
107
150
250
350
Number
Figure 6: Distribution of the number of words for each caption in JWeb-5M. Horizontal axis: number of words (bin width 10), Vertical axis: frequency (log)
Note, however, that the addition of GAUs also increases the total number of parameters in the model. Table 5 shows the relationship between the model size and the number of training parameters for each method.
segmentation was performed by goccab-unidi is shown Fig. 6. The mean (median) number characters and words was 26.3 (20.0) characters and 11.9 (9.0) words. Captions consisting of one or two words were included instead of natural sen- tences. When 100 images were sampled for the study, 32 images contained Japanese characters (Kanji, Hiragana, Katakana).
B Models
Image Encoders. We used ViT-B/16-AR-IN11K
which is 12-layer Vision Transformer (Doso- vitskiy et al., 2021) pre-trained on ImageNet- 21k (Deng et al., 2009) (14 million images, 21,843 classes) with AugReg (Steiner et al., 2021) at res- olution 2 The number of hidden states was 768, and images were presented as a sequence of 16x16 patches.
JWeb-k From the JWeb dataset, constructed multiple training sets with different numbers of training data. The training JWeb-k, where k the number of image-text pairs in the training data (e.g., JWeb-500k, WWeb 1m). Note that the training set with the smaller num- ber of data was a subset of the larger training set, and each image-text pair was randomly selected. Table 6 shows the statistics of the training sets.
are denoted
Text Encoders. We used BERT-base as the text encoder, which is 12-layer Transformer pre-trained on lower-cased English Wikipedia and BookCorpus (Devlin al., 2019)s. We also newly constructed its Japanese version pre-trained with lower-cased Japanese Wikipedia and CC-100- ja (Conncau et al., 2020). The number of hidden states d was 768.
JWeb-5M Unless otherwise stated, we utilized JWeb-5M consisting of 5M image-text pairs for training the model. The number of unique im- ages in this dataset is 4,942.737 (confirmed by the md5 hash value) and the number of captions is 4,369, 144. The top-five most frequently used cap- tions were "landscape", "woman," "food," "cat," and "cherry blossom.
C JWeb dataset
The JWeb dataset is the dataset we constructed for this study. based on images and Japanese cap- tions that we collected from a wide range web sites. Images were resized at the time of download- ing to have a short side size of 256 or larger. A trained fastText (Joulin et al., 2017) model was used to determine the language. The distribution of the number of words in the caption when word
D Evaluation dataset
MS-COCO (Lin et al., 2014) contains 123K im- ages, each accompanied with five manually written captions. We followed the data split used in (Karpa- thy and Fei-Fei, 2017) and used the 5K test set. Flickr30K contains 31K images collected from the Flickr website, with five textual descriptions per image. We follow the data split used in (Karpathy and Fei-Fei, 2017) and use the 1K test set.
tgub. wwiithu..cmmgooeeee.. www..///oooleerreeerrhiiiiiiiiia
transformer
'huggingface wtt.c////eete.e ht//berrt-bas-uncased
"https: //tasttetttttteteeeee/eee/aeaeee ca/ cc/docs/en/
language-ide ieeeeeiiintiiiiaaiiii html

---

Method
Not.
wdr)
#araam.
#Train Param.
FS
210.0M (100%)
210.0M (100%)
FT
210.0M (100%)
210.0M (100%)
LIT-FS
210.0M (100%)
124.2M (59.1%)
LIT-FT
210.0M (100%)
124.2M (59.15))
LoRAs
210.6M (100.3%)
1.5M (0.71%)
LoRA1a
16
211.IM (100.5%)
2.0M (0.95%)
LoRAga
32
212.3M (101.1%)
3.2M (1 (1.52%)
LoRAc4
64
214.7M (102.25)
5.6M (2.67%)
LoRAzse
256
228.9M (102.2%)
19.7M (2.67%)
LoRA1024
1024
285.5M (102.25)
76.4M (2.67%)
DueT
48
211.8M (100.9%)
2.7M (1 (1.28%)
DueT
96
213.6M (101.7%)
4.5M (2.3%)
DueT
192
217.3M (103 55%)
8.0M (4.09%)
DueT
384
224.2M (106.8%)
15.1M (7.72%)
DucT
768
238.4M (113.5%%
29.2M (14.9%)
DucT
1536
266.7M
(127.0%)
57.6M (29.5%)
DucT
3072
323.4M (154.0%)
114.2M (58.4%)
Table 5: Number of parameters with VYT-B/16 (d 768) and BERT-base (d 768). #Param.: Number of
parameters in the model at inference. #Train Param.: Number of trainable parameters.
Training set
#lmages
#Captions
E
Experimental Settings
JWeh-50K
49,993
49,073
E.1
Main Experiment
JWeb- 100K
99,973
97,299
JWeb-300K
299,786
286,910
Eight NVIDIA A100 80GB GPUs were used for training. The batch size was 8192, and 16 epochs of training were conducted using mixed precision training (Micikevicius et al., 2018) and gradient check pointing (Chen et al., 2016). The optimizer was AdamW (Loshchilov and Hutter, 2019), with a learning rate of 5e-4. The tempera- fixed at 0.015625 (1/64). The average is the experimentally reported result.
JWOK
499,415
473,266
JWeb-IM
997,637
930,435
JWeb-3M
2.979,041
2,686,802
JWeb-5M
4.942,737
4,369.144
WOM
9,781,186
8,365,458
Table 6: Statistics of each training set. #Images: Num- ber of unique images. #Captions: Number of unique captions.
ture parameter
The training on YFCC-CLIP used Inception- style random cropping (Szegedy et al., 2015), and the resolution was set to 224 x 224. The training on JWeb-5M changed only the lower limit the scale of the crop range to 0.9 from the above set- tings. The lower limit of the scale of the crop range was changed to 0.9 when the model was trained on JWeb-5M. Triminl Wide (Müller and Hut- ter, 2021) was used as the common augmentation method; image normalization CLIP (Radford al., 2021) was used as well. The maximum input to the text encoder was 77 tokens. During testing, only resizing, cropping from the center, and normaliza- tion to a resolution of 224 x 224 were performed. The prompt text in the text encoder was not used for either training testing. For DueT, the hyper- parameter was
The images were collected from Flickr, and natural text captions were created by crowd- worker. nitk-punct's mean (median) word count was 11.3 (eleven) words. For the Japanese version of STAIR Captions (Yoshikawa et al., 2017), five new Japanese captions were created for each im- age by the crowd-worker for the same images as COCO. Flickr30K had 31.5 (31) words.
Flickr30K contains 31K images, each with five captions. The mean (median) number of words is 31.4 (12). The Japanese version, Flickr30k- JP (Nakayama et al., 2020), was created by expertly translating each Flickr30k caption. The mean (me- dian) word count was 18.4 (17) words. For COCO and Flickr30k, the images in the test data were se- lected according the partitioning of the dataset in COCO (Karpathy and Fei-Fei, 2017), as in pre- vious studies.
1536. We committed m 1536 early in our experiments it consis- tently outperformed full fine-tuning. Meanwhile, when conducting multiple ablation tests simultane- ously, it was revealed that for training on JWeb5M,

---

Training set
Warm-up steps
#Epochs
Batch size
We trained a logistic regression classifier following to (Radford et al., 2021).
JWcb-50K
1,000
64
2,048
JWeb-1005
1,000
64
2,048
We train the logistic regression classifier using scikit-learn's L-BFGS implementation with up 1,000 iterations and report the corresponding metric for each dataset. The strength of the L.2 regular- ization was determined using hyperparameter sweep on the validation sets over the range between 10 6 and 10 with 96 logarithmically spaced steps. To save compute required for the sweeps, we perform a parametric binary search that starts with À = [10- ,10- 10-2, 102, 10 109) and iteratively halves the interval around the peak until reaches resolution of steps per decade. The hyperparameter sweeps are performed on vali- dation split of each dataset. For the datasets that contain a validation split addition to test split, we use the provided validation set to perform the hyperparameter search, and for the datasets that do not provide a validation split or have not published labels for the test data, split the training dataset to perform the hyperparameter search. For the final result, we combine the validation split back with the training split and report the performance on the unused split.
JWeb-300K
2,000
64
2,048
JWeb-500K
2,000
4,096
JWeb-IM
1,000
32
4,096
JWeb-aM
2,000
32
4,096
JWeb-5M
2,000
8,192
JWeb-10M
2,000
16
8,192
Table 7: Training setup for each dataset
using m 768 even m 384 proved to be sufficient, as shown Table Determining the optimal setting for m remains a topic for future investigation.
E.2 Other experimental settings
In an experiment to investigate the effect of dif- ferent numbers of training data model perfor- mance, the batch size, number of training epochs, and warm-up steps were set separately for each dataset. Table shows the hyper-paramettrs set individually in the JWeb-k training.
F Linear Probe evaluation
Here, we provide additional details about the linear probe experiments described in this paper, includ- ing the data sets and experimental setup used in the evaluation.
F.3 Results
The individual linear probe scores are provided in Table 9. DueT achieved the best performance on 10 of the 15 datasets, i.e., it was included the Cropper-Pearson 99.5% confidence interval around the top score for each dataset.
F1 Datasets
We used ten datasets from the well-studied eval- uation suite described in (Kornblith al., 2018) as well as five additional datasets in to assess the performance of models on a wide variety of distri- butions and tasks.
On many datasets, DueT and fine-tuning out- performed the uni-modal ViT, demonstrating the superiority of natural language supervision over traditional pre-training approaches based on im- age classification, reported in (Radford et al.. 2021). Furthermore, DueT performed better than fine-tuning: this result demonstrates the improved image recognition capability of. See Section 4.5 for a detailed discussion of the linear probe results.
These datasets included MNIST, STL- 10 (Coates al., 2011), the German Traffic Sign Recognition Benchmark (GTSRB) dataset (Stal- Ikamp et al., 2011), the Country211 (Radford et al., 2021), the Stanford Sentiment Treebanks(SST) dataset (Socher et al., 2013).
The details each dataset and the correspond- ing evaluation metrics provided in Table 8.
G Adaptive gating mechanism
To investigate better configurations of the gating mechanism, implemented token-level and sentence-level input-adaptive gating mechanism using single-layer FFN. The gate value a' of the expression was changed one-layer FFN in order to evaluate the adaptive gate mechanism at the sentence and token levels.
F.2 Experimental settings
We image features taken from the penultimate layer of each model, ignoring the classification layer provided for uni-modal VIT. For the image encoders DueT and FT, we used the features before the linear projection to the embedding space.

---

Dataset
Classes
Training size
Test size
Evaluation metric
Food-101
102
75,750
25,250
accuracy
CIFAR-10
10
50,000
10,000
accuracy
CIFAR-100
100
50,000
10,000
accuracy
SUN397
397
19,850
19,850
accuracy
Stanford Cars
196
8.144
8,041
accuracy
FGVC Aircrafts
100
6,667
3,333
mean mean-per-elass
Deseribable Textures(DTD)
47
3,760
1,880
accuracy
Oxford-IIIT Pets
37
3,680
3,669
mean-per- pere-eer-llss
Caltech-10
101
3,030
5,647
mean- mean-per-clant per-ceess-s
Flowers 102
102
2,040
6,149
mean- mean--er-class
MNIST
10
60,000
10,000
accuracy
STL-10
10
1000
8000
accuracy
GTSRB
43
26,640
12,630
accuracy
Country211
211
43,200
21,100
accuracy
SST2
2
67,349
1,821
accuracy
Table 8: Details on each dataset and the corresponding evaluation metrics
Dataset
VIT
finetune
DueT
JWeb
STAIR
Flickr30k-JP
Gate
I-:T
T~1
I-T
T-1
I-+T T-11
Food- 101d-101
79.05
86.04
88.64
CIFAR-10
89.98
94.11
97.86
N/A
73.18
72.65
60.94
50.86
77.7
68.08
CIFAR-100
73.36
81.36
88.88
FFNeolson
74.0
73.96
61.76
52.16
80.13
70.05
SUN397
69.40
70.38
73.72
FFNest
73.62
73.46
61.38
52.36
80.13
69.27
73.21
52.53
80.4
69.27
StanfordCars
43.15
47.26
49.16
DTD
70.11
74.10
75.00
MNIST
96.52
98.2
97.92
Table 10: Performance of models trained with different
STI.110
98.42
97.96
99.47
gating mechanisms on JWeb-5M. m 1,536.
GTSRB
56.12
80.55
77.21
C
12,10
15.52
16.69
SST2
54.48
54.75
55.35
FGVCAircraft
40.94
45.01
49.25
Table 10 compares the results from models em- ploying different gate mechanisms. The table shows that there is no clear performance improve- ment between the adaptive gating mechanism with the FFN and the gate factor.
OxfordPet
86.95
86.44
90.0
Flowers102
98.05
98.47
99.15
Caltech101
90.93
91.02
94.78
Table 9: Linear probe performance of various pre- trained models on 15 datasets. Scores within the 99.5% Glopper-Pearson confidence interval of each dataset's top score are shown in bold.
H LoRA tuning
This section describes the LoRA tuning implemen- tation used in this paper and presents the unit-size evaluation.
Let GAU'(H)) defined as GAU'(H = FFN' H (LN
(1 - FFN (H
H
H.1 Implementation
(5)
We use WGp We. W., and W, to denote the query/key/value/outuu projection matrices the self-attention module of each transformer block. LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices. We applied LoRA to W and W, for all transformer blocks for both image and text encoders according the (Hu al., 2022). LoRA modules added to all layers of the image and text encoders as in DueT. It also updates all parameters the layer normalization of the pre-trained encoders during training.
Correspondingly, FFN! a,sent
(H))
and
FFN! a.taken (H') are defined as follows:
FFN a,sent (H') a(h'CLs T
"w wgate + bgate)
(6)
FFN! aakken (H') w' gate + bluin) R
(7)
where is sigmoid function. w gate Rd gate E R a trainable parameter and n is the length of the sequences.

---

STAIR
FlickrãOk-IP IN-JP
Text: Encoder
Method
#TP
I
T-+1
I-+T
T-1
Image Encoder
LoRAs
1.5
47.34
39.6
65.97
56.1
32.8
LoRA16
2.0
47.58
39.66
64.8
56,74
33,64
0.8
LoRAua
3.2
47.71
39,95
65.93
55.7
33,64
Gate value
LoRAsa
5.6
47.8
39.89
67.7
55,1
32.66
0.6
LoRA2se
19.7
47.05
39.7
65.2
55.76
33,52
LoRA1u24
76.4
47.01
39.17
66.53
55.88
32.82
Table 11:
Performance of LoRA with varying adapter
size on each test set.
0.0
H.2 Unit-size evaluation on LoRA
Layer number
The effect of the hyper-parameter r was investi- gated for the LoRA used in this study. The results in Table show that there is no significant dif- ference in performance for different values of r. As discussed in section 4.3, while LoRA is effec- tive when the amounts of training data is small and in few-shot tasks with very few additional param- eters, its performance is far below that of Due'T for the range of data used in this experiment. The results also show that increasing the number of ad- ditional parameters did not significantly change the performance, indicating that the method of adapter insertion and gate insertion, rather than the number of additional parameters, had significant impact performance.
Figure 7: Gate values for each layer. Green: represents the gate values of the image encoder. Orange: represents the gate values of the text encoder. Blue: represents the gate values at the layer inserted within the image encoder. Red: represents the gate values at the inserted layer within the text encoder. Gray: represents the gate values of the image (or text) encoder with only some of adapters are inserted
text (or image) encoder.
the case when adapters were inserted into all layers. Given the above discussion, we conclude that this is also result of the image encoder's 's tendency utilize knowledge in the first half of the layer and to acquire knowledge in the latter half.
Lack of adapters around the output layer and lack of units to acquire new knowledge also explains the performance degradation when the adapters in the output layer of the image encoder are removed (Table 3).
I Gates performance at each layer
The values of the gates in the case that only some of the adapters are inserted are plotted as blue and red lines in Figure The gate values tend to be larger when adapters were inserted the text en- coder, and they were particularly large when the number of adapters was small. Considering the dis- cussion in Section 4.4 on knowledge utilization and acquisition in all layers of the text encoder, it can be assumed that the total number of adapters em- bedding new knowledge decreased, and therefore, by increasing the gate value, the impact of the FFN is increased and new knowledge is embedded there. In the image encoder, when adapters were inserted only near the output layer, the gate values were not significantly different from those in the case when adapters are inserted in all layers. On the other hand, different trend was observed when adapters were inserted only near the input layer. Specifically, when adapters were inserted only from the first to fourth layers (or from the first to eighth layers), the values of the gates in the third or fourth layers (or seventh or eighth layers) were larger compared to
Pre-trained models evaluation
We conducted a comparative analysis of different pre-trained models for model initialization. We trained the models on JWeb5M and used the same conditions as Section E for the rest of the experi- mental setup.
J11 Pre-trained models for image encoders
We compared five pre-trained models for initial- ization: AugReg" (Steiner et al., 2021) and CLIP Image Encoder (Radford et al., 2021), in addition to AugReg used in Section 4.3. AugReg" represents model that was pre-trained on ImageNet-21 k and then fine-tuned on ImageNet 2012. We trained the text encoder using BERT-base as initialization in all cases.
The results in Table 12 show that DueT outper- formed fine-tuning in all cases, regardless of the pre-training model employed. These results sug-

---

Model
Data.
Method
IN
I-+T
T-x1
K Effect of varying the amount of training data
DueT
42.85
71.15
60.9
AugReg
IN21K
finetune
37.23
69.84
57.67
Table 14 shows detailed numerical values of the experimental results presented in Figure 4. These values show that DucT had the highest scores all of the test sets when training was conducted with more than 1M data. On the other hand, LoRA outperformed Due'T STAIR, Flickr image-to- text retrieval, and ImageNet classification when the number of training data was relatively small, such as 50k or 100k, confirming that LoRA is effective when the number of training data is small.
DucT
41,86
71.2
60.79
AugReg"
IN
finetune
35.65
69.08
57.69
DucTI
38,42
79.02
69.09
CLIP
WIT
finetune
31.93
70.41 59.38
Table 12:
Zero-shot transfer performance of models trained on JWebSM with DueTT and different image encoders that have the same VřT-B/16 architecture. I-T (T-1l) shows the mean value of Recall@k (R @k; k 1,5, 10) on Flickr30k-JP and STAR.
L Discussion on the Comparison with LilT
Model
Method
I-T
T-1
DueTT
42.85
71.15
60.9
BERT (ours)
finetune
37.23
69.84
57.67
As mentioned Section G, one of the points where DueT differs from LilT is the adoption of the gate mechanism, and the GAU validation results shown in Table 4 indicate that adapters that adopt the gate mechanism effective in transition learning for the vision and language model. Preliminary ex- periments on the number and location of adapter insertions showed that the type adopted this ex- periment, in which one GAU is inserted behind the FFN sub-layer, was the most effective.
DueTT
42.57
72.17
61.47
BERT*
finetune
36.36
67.71
57.94
DueTT
38.52
61.77
53.17
RoBERTa
finetune
34.82
62.26
52.66
Table 13: Performance of models trained on JWeb5M with different text encoders that have the same base architecture. We used AugReg-IN2I1 IN21k ViT-B/16 for the image encoder. We set = 1,536. I I-TT (T-+)) shows the mean value @e (R @k; A = 1, 5, 10) on Flickr30k-JP JP and STAR.
Although the relationship between the insertion point and the gating mechanism, the number of training data and the effect of the training domain not fully investigated, these experimental sults suggest that DueT is expected to be more effective method than LilT in the experimental set- ting we used.
gest that DueT can selectively train new knowledge in GAU without forgetting existing knowledge, re- gardless of the knowledge embedded in the pre- training model.
J.2 Pre-trained models for text encoders
We compared five pre-trained models for initial- ization: B and RoBERTa (Liu et al., 2019)* in addition to BERT (ours) used in Section 4.3. We trained the image encoder using ViT-B/16-AR- IN21k as initialization in all cases.
The results in Table 13 show that DueT outper- forms finetune on all employed pre-training models except for the image to text search with RoBERTa. The results indicate that DueT is effective regard- less of the architecture used in the text encoder or the knowledge embedded in the pre-training model.
"huggingface httpe/efaa..ccccoool/eeee co/cl-tohoku/
bert-base-aa jaaase--apaneee-v "huggingface co/rimma/jgaanee-rrbeeree

---

]Web-test I-T
STAIR
JIickr30k-JP IN-J
Train Data
Method
T-+1
I-T
T-I
I-:T
T-I
DueT
26.93
26.63
23.74
19.61
42.7
35.66
13.01
JWeb-50K
LoRAs
23.36
22.53
26.09
18.06
44.17
32.46
13.22
finetune
25.61
25.0
21.63
17.46
40.2
31.65
12.13
DueT
33.55
33,59
30.25
24.49
47.6
39.27
16.55
JWeb-100K
LoRAs
30.22
29.56
30.74
22.23
49.77
38.55
18.31
finctune
31.85
31,36
27.43
21.34
43.57
37.88
17.13
DueT
44.98
44.69
39.6
31.41
54.8
47.27
23.16
JWeb-300K
LoRAs
40.17
39.64
37.06
27.78
56.97
47.31
24.88
finetune
43.74
43.73
34.07
28.37
51.0
43.5
22.85
DueT
50.79
50.87
44.85
35.69
62.27
54.12
25.24
JWeb-500K
LoRAs
44.64
44.26
39,92
58.87
49.94
27.19
finetune
48.74
48.63
39.76
33.06
56.63
49.03
27.68
DueT
57.77
57.77
48.89
41.21
68.37
59.34
30.76
JWeb-IM
LoRAs
47.85
47.34
42.58
34.74
60.23
52.64
29.4
finetune
54.2
$4.36
44.5
34.99
62.8
52.37
22.61
DueT
68.58
68.87
58.66
48.88
80.27
67.97
39.09
JWeb-3M
LoRAs
52.98
52.33
46.73
39.65
66.93
55.74
33.28
finetune
66.17
66.42
56.19
44.72
73.17
60.81
31.49
DueT
73.17
73.21
61.89
52.53
80.4
69.27
42.85
JWeb-5M
LoRAs
52.62
52.02
47.34
39.6
65.97
56.1
32.8
finetune
72.12
72.95
60.98
50.29
78.7
65.04
37.23
DueT
78.76
78.51
65.02
55.77
84.53
73.04
46.81
JWeb-10M
LoRAs
55.15
79
49.19
41.27
65.93
57.92
34.51
finetune
78.42
79.11
64.81
54.84
80.87
68.84
40.86
Table 14: Performance of each task for different numbers of training data.


Reviews:
[{"review_id": "nyXfKxVH0u", "reviewer": "Reviewer_Tr3S", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, thank you for your review and the constructive feedback that aids in refining our manuscript. \n\n> 1: Using pre-trained image and text encoders for image-text contrastive learning is a common practice, lacking originality. There are many similar works in the literature.\n\nWhile our research utilizes contrastive learning and doesn't delve into introducing new learning theories or fundamental architectures from a technical novelty standpoint, we have incorporated new techniques, including the integration of GAUs, and carried out detailed ablation tests.\nFurthermore, as mentioned in Section 2.2 \"Adaptation of CLIP to downstream tasks\", while a majority of the research focuses on adapting pre-trained CLIP encoders for downstream tasks, our focus was on enhancing the image-text contrast learning framework using robust pre-trained uni-modal encoders.\nThe significance of vision and language foundation models is on the rise. We are confident that our DueT method, which can obtain pre-trained models that achieve high performance with a low learning cost, provides a meaningful contribution to the field.\n\n> 2: Testing on a limited number of languages or training sets is not sufficient to support the claims\n\nAs you rightly pointed out, recent research has been conducted on Multilingual-CLIP. Carlsson et al. (2022) trained a text encoder on machine-translated pairs of English and 67 other languages using teacher learning, thus enabling the encoder to accommodate non-English languages. Zhang et al. (2023) and Chen et al. (2023) adopted machine translation alignments and PEFT to train their text encoder, subsequently evaluating their approach on about 10 languages. Moreover, Khan & Fu (2023) demonstrated that by training with Multilingual-BERT on pairs of images and English text, it becomes feasible to attain low-resource adaptation for seven non-English languages.\n\nOur proposed method delivers a vision and language model that is particularly adept at handling the target language, in contrast to previous studies that aimed to create a single model for a multitude of languages.\nThe following three points are the differences from previous studies and newly obtained findings.\n\n1. Unlike previous studies, our method does not require a parallel corpus (a set of translated texts between two languages). The Duet model is trained simply using pairs of images and text in the target language.\n\n2. The Japanese dataset used in our experiments is more expansive (reaching up to 10M) and encompasses a more diverse range of domains. The previous studies used images from public datasets such as Conceptual Captions, COCO, and Multi30k. Unlike these, our dataset introduces diverse visual concepts and encompasses a range of images, spanning shopping items, slides, PC screens, and Japanese characters, which remain absent from the pre-training domain of the ViT image encoder. We believe that the findings obtained from our experiments are new.\n\n3. In experiments involving Japanese, we utilized a monolingual BERT pre-trained on Japanese text. In contrast, earlier studies often relied on multilingual models, be it Multilingual BERT or XLM-RoBERTa. Supplementary experiments, consistent with the specifications in Table 13, produced the following scores, indicating that the monolingual BERT outperforms the Multilingual-BERT:\n\n|          Model         | ImageNet  |  I -> T  |   T -> I  |  \n|-----------------------|---------------|---------|----------|\n| Duet w/ BERT      |     42.85    |  71.15  |  60.90  |  \n| Duet w/ M-BERT |     38.53    |  63.09  |  56.24  |  \n\nGiven the aforementioned considerations, while our experiments are limited to the linguistic scope of English and Japanese, we believe we have garnered new insights regarding \"deep\" adaptation to Japanese. Furthermore, our methodologies extend beyond language adaptation, encompassing domain adaptation and additional learning paradigms. We will integrate these discussions into both the main body and the appendix of our manuscript.\n\nReferences:\n* Fredrik Carlsson et al.: Cross-lingual and Multilingual CLIP. LREC 2022: 6848-6854\n* Zhen Zhang et al.: Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment. CoRR abs/2305.03510 (2023)\n* Guanhua Chen et al.: mCLIP: Multilingual CLIP via Cross-lingual Transfer. ACL (1) 2023: 13028-13043\n* Zaid Khan and Yun Fu: Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning. ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1: The author suggested a method to use adapters with pre-trained image and text encoders. This reduced the number of parameters that needed to be trained. The adapters had gates that allowed effective transfer and connection of knowledge from the pre-trained encoders for each modality.\n\n2: The author conducted experiments with various text and image encoders, adapter units, parameter initialization methods, and efficiency measures, following the proposed approach. The results demonstrate the effectiveness of the proposed method.\n\n3: The analysis of solid ablation covered various aspects, such as parameter efficiency, the effectiveness of GAUs on different layers, the performance of gates, and so on", "reasons_to_reject": "1: Using pre-trained image and text encoders for image-text contrastive learning is a common practice, lacking originality. There are many similar works in the literature.\n\n2: Testing on a limited number of languages or training sets is not sufficient to support the claims", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work proposes a novel method for transfer learning of vision and language models using adapter tuning with a gating mechanism. The method, called DueT, leverages the knowledge from pre-trained image and text encoders without modifying them and trains only the adapters. The work shows that DueT achieves better performance and parameter efficiency than existing methods for image and text retrieval in both English and Japanese domains. The work also provides insights into the optimal number of parameters, training data, and gates for transfer learning of vision and language models. The work contributes to the research on vision and language models and their applications in various services that require integrated visual and linguistic understanding.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZhYTsthpTl", "reviewer": "Reviewer_pT3q", "summary": "", "questions": "- Can more details be given about the hashing algorithm used?\n\n- In Table 4, was alpha set to 0.2 or 0.02? I can't tell if the last row of this table is supposed to be the same as the base 0.2 setting (L364).\n\n- Why was m=1536 selected? The performance does not seem particularly better than some of the lower values of m in Table 2. This choice of m should be explained in the Experimental setup, before the results.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for emphasizing the contributions of our work and for your comments to further improve the paper.\n> Can more details be given about the hashing algorithm used?\n- We utilized the md5 algorithm to hash the bytestring of both the image and the text. However, other hashing algorithms, such as sha1, can also be used. We will include this information in the paper.\n\n> In Table 4, was alpha set to 0.2 or 0.02? I can't tell if the last row of this table is supposed to be the same as the base 0.2 setting (L364).\n- Thank you for pointing that out. Table 4 is correct, and L364 is an error. DueT initializes the gating value of alpha to 0.02.\n\n> Why was m=1536 selected? The performance does not seem particularly better than some of the lower values of m in Table 2. This choice of m should be explained in the Experimental setup, before the results.\n- As you pointed out, for training on JWeb5M, using m=768 or even 384 proved to be sufficient. We committed to m=1536 early in our experiments as it consistently outperformed full fine-tuning. Because we were concurrently conducting several ablation tests with this setting, we couldn't rerun the tests with the smaller m values this time. We will move the explanation regarding the choice of the m value to the experiment setup section", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper does a good job of applying ideas from parameter-efficient fine-tuning tuning (PEFT) methods to the problem of image-text contrastive learning. The results demonstrate the effectiveness of the proposed solution, evidenced by the strong results on zero-shot retrieval and image classification.\n\n- Experiments are conducted not only on the standard English benchmarks, but also on Japanese datasets. It is rare to come across a multimodal method that shows experiments on languages beyond English, so it's great to see a non-English language represented in the experiments.\n\n- The methodology section is extremely well structured: 3.1 explains all the background well, 3.2 explains the GAU method well, and 3.3 motivates all the design choices very well, effectively explaining how different design choices correspond to different learning settings (L347-L359). Everything was well-written and extremely easy to follow.\n\n- The various ablation studies in Section 4.4 are also very well-motivated, and experiments are designed accordingly -- specifically, how the size of the bottleneck layer affects the model's learning ability, the utility of GAUs in different layers, the analysis of gate values at different layers, the effect of gate initialization, and effect of amount of pre-training data. All of these are very interesting questions that have been addressed in the ablation study.", "reasons_to_reject": "There are no real reasons to reject this work -- some additional details would be useful (mentioned in the Questions for Authors section), but these are not grounds for rejecting the work in my opinion.", "questions_to_authors": "- Can more details be given about the hashing algorithm used?\n\n- In Table 4, was alpha set to 0.2 or 0.02? I can't tell if the last row of this table is supposed to be the same as the base 0.2 setting (L364).\n\n- Why was m=1536 selected? The performance does not seem particularly better than some of the lower values of m in Table 2. This choice of m should be explained in the Experimental setup, before the results.", "paper_topic_and_main_contributions": "This paper introduces DueT, an adapter-based method for aligning pre-trained vision and text encoders (ViT and BERT respectively) by inserting gated Adapter units within the frozen encoders and training them with a contrastive loss. By keeping the base Transformers frozen and only training the GAUs, the authors prevent catastrophic forgetting and preserve the unimodal capabilities of the language models. The authors pre-trained models on both English and Japanese image-text datasets, and evaluate on zero-shot image classification and image/text retrieval in both languages.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "F83HmFp575", "reviewer": "Reviewer_a7vW", "summary": "", "questions": "- The authors should release their code and model to facilitate future research.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate you for highlighting the key aspects of our research. Regarding the issues raised about the augmentation of parameters, we concur with your positive feedback and maintain that the computational overhead is not a significant challenge in our model, particularly since the parameters outside the adapter layers are frozen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is simple and effective. It is technically valid and can enhance the image-text contrast learning framework.\n- The authors describe their network design in detail, including the architecture of the proposed gated adapter unit.\n- The authors considered an array of design choices for the baselines, and provided clear comparisons to demonstrate the effectiveness of DueT.\n- Large-scale pre-training is conducted to validate the idea, including pre-training on public (YFCC) and private (JWeb) dataset, respectively.\n- The paper is well-written and easy to understand. ", "reasons_to_reject": "I don't find serious concerns. The only minor concern is that adding GAUs would also increase training parameters, from 1% to 58%, depending on the hyper-parameter. But considering that all other layers are frozen, the increased num parameters are acceptable.", "questions_to_authors": "- The authors should release their code and model to facilitate future research.\n", "paper_topic_and_main_contributions": "This paper presents a dual-adapter tuning (DueT). The authors add additional layers into both the image and text encoders and then train the added layers. The authors design a gating mechanism to the so called adapters. \n\nThe proposed method outperforms the baselines and LoRA-based adapter method in terms of accuracy and efficiency on image-text retrieval tasks. Linear probing results are encouraging.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an adapter-based method for aligning pre-trained vision and text encoders (ViT and BERT respectively) by inserting gated Adapter units within the frozen encoders and training them with a contrastive loss. 

Two reviewers are positive about the paper, while one reviewer is negative about the paper. Overall, reviewers found that (1) the paper does a good job of applying ideas from parameter-efficient fine-tuning tuning methods to CLIP training; (2) paper is well written and structured; (3) experiments are comprehensive and ablation study is solid. The reviewer who gave a low score mainly complained about (1) lack of novelty; and (2) only test using the proposed method to train CLIP on two languages. 

Generally, the AC is positive about the paper, and agrees that the negative comments did not hold, and that reviewer also did not participate in the discussion to back up his scores.
Decision: Accept-Main