APoLLo : Unified Adapter and Prompt Learning for Vision Language Models
Authors: Sanjoy Chowdhury
Sayan Nag
Dinesh Manocha
Venue: EMNLP 2023 Main (2023)
Keywords: Vision Language Models
Prompt Tuning
Adapter Tuning
Contrastive Learning

Abstract: The choice of input text prompt plays a critical role in the performance of Vision-Language Pretrained (VLP) models such as CLIP. We present APoLLo, a unified multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. Our method is designed to substantially improve the generalization capabilities of VLP models when they are fine-tuned in a few-shot setting. We introduce trainable cross-attention-based adapter layers in conjunction with vision and language encoders to strengthen the alignment between the two modalities. We enforce consistency between the respective encoder branches (receiving augmented inputs) to prevent overfitting in downstream tasks. Our method is evaluated on three representative tasks: generalization to novel classes, cross-dataset evaluation, and unseen domain shifts. In practice, APoLLo achieves a relative gain up to 6.03% over MaPLe (SOTA) on novel classes for 10 diverse image recognition datasets.

APOLLO
Unified Adapter and Prompt Learning for Vision Language Models
Sanjoy Chowdhury I+
Sayan Nag"
Dinesh Manocha
University of Maryland, College Park
University of Toronto
[sanjoyc, wwwnocha)nnocccccmoomoooo
edu
sayan.nag@nan wwwpaannnmaail.ttooottooooo utw.ono...a.ao.ooooomooooo
Project page https: hopps///gant tetts///ammaammme htt.ammaaumd.ddu/preeeeee wtw.:/d.dduuproviiioonnnttaeeeee
Abstract
Eower
Sorraid Cars (Rau)
The choice of input text prompt plays a criti- cal role in the performance of Vision-Language Pretrained (VLP) models such as CLIP. We present APOLO. unified multi-modal ap- proach that combines Adapter and Prompt learning for Languuge models. Our method is designed to substantially improve the generalization capabilities of VLP models when they are fine-tuned a few-shot setting. We introduce trainable cooss-ateention-bat based adapter layers in conjunction with vision and language encoders strengthen the alignment hetween the two modalities. We enforce consis- lency between the respective encoder branches (receiving augmented inputs) prevent over- fitting in downstream tasks. Our method is evaluated on three representative tasks: gener- alization to novel classes, cross-dataset evalu- ation, and unseen domain shifts. In practice, APOLLO achieves a relative gain up 6.03% % over MaPLe (SOTA) novel classes for 10 diverse image recognition datasets.
Octke
heapher
SLS397
15.47
Calccttii
(Baic)
(Novdl
$2.17
0192
94.26
WA.
27.44
56.00
Pastini
Malt (ST0A)
Figure I: APOLLO sets new saate-of-thean across several datasets for the hase-to-novel class generaliza- tion task by learning Asion-language representation using unified adapter and prompt tuning strategy under a few-shot setting. Each axis denotes an accuracy value, either base novel class (specified accordingly) for the corresponding dataset (refer to Table 1).
(Khattak et al., 2023). Recent studies in literature propose fine-tuning of these models by introduc- ing and calibrating the parameters by prompting (Zhou al., 2022a) and adapter (Gao al., 2021a) techniques. While the former injects tunable pa- rameters through learned embeddings one or more modalities, the latter incorporates in-situ ad- justments conventionally near the prediction head.
1 Introduction
Recent years have witnessed tremendous success largescale pre-trained language models (Devlin et al., 2018; Liu et al., 2019; Raffel et al., 2020; Biderman et al., 2023; Scao et al., 2022; Shen et al., 2023; Touvron et al., 2023) well as visual mod- cls (Dosovitskiy et al., 2020; Liu et al., 202 lb; Tan and Le, 2019) leading surge in pre-trained Vision-Language models (Dou al., 2022; Li et al., 2022a; Yang al., 2022; Wang et al., 2023a; Li et al., 2023b,a; Wang et al., 2023b) for multi-modal downstream tasks. Despite being largely successful in terms of generalization capabilities, these VLP models such as CLIP (Radford et al., 2021) are difficult to fine-tune for few-shot learning-bass based downstream tasks (Khattak al., 2023). This is mainly because of the massive scale of these mod- els coupled with the deficiency of training data
The effectiveness of such fine-tuning- based methods notwithstanding recent investigations re- veal (Zhou et al., 2022b) the caveats of these ap- proaches - such as neglecting the useful knowledge gained from the preceding pre-training step and overfitting with the downstream tasks. Although find considerable wealth of studies that in- volve text-based prompt learning (Lu al., 2022; Shu et al., 2022: Huang et al., 2022), the same for the visual pipeline still remains an under-explored area.
We present APOLLO, novel unified adapter and prompt learning approach for VLP models tackle the generalizability prob- lems in few-shot scenarios. We enforce consistency between intra- motal encoders using consistency-
Main Results:
"Equal contribution.

---

guided contrastive loss (Wei al., 2020). This is done teach the association between the query and the semantically similar negative in-batch keys. To further enhance cross-man mrossl alignment, we employ cross-attention in the modality-specifie adapter lay- ers. This leads to better awareness of the multi- modal features. Experimental results over varied range of recognition datasets demonstrate the effi- cacy of our approach. Some novel aspects of our work include:
2021; Gao al., 2021b; Kim et al., 2021) that fare much better compared off-the-shelf VLP models (Radford et al., 2021; Jia al., 2021). In this paper. we present a novel image and text augmentation- based technique for a unified adapter and prompt learning to enable CLIP based model to generalize well under few-shot and zero-shot visual recogni- tion tasks.
2.2 Prompt Tuning
(1) To the best of our knowledge, ours the first method that combines adapter and prompt tun- ing for VLP models (e.g., CLIP) in a unified manner. This facilitates learning new tasks in a few-shot setting without compromising on their zero-shot generalizability.
Prompt tuning (Li and Liang, 2021; Liu al., 2021a; Lester et al., 2021) typically refers prepending language instructions to the input text to facilitate a better understanding of the down- stream tasks. For example, instead of feeding the system with a fixed template 'a photo of <CLASS>', task-specific additional information could be more helpful for the model. More so, tuning- based methods achieve comparable perfor- mance as full finetuning but with ~ 10000x fewer pa- rameters. To this end, Context Optimization(COOp) (Zhou et al., 2022b) proposes a replacement of the hand-crafted prompts with the learnable soft prompts. However, their approach lacks gener- alizability and demonstrates suboptimal perfor- mance under zero-shot setting. CoCoOp (Zhou et al., 2022a) generates an image-conditional con- text along with text conditioning through prompt tuning, while ProDA (Lu al., 2022) incorpo- rates the prompt's prior distribution learning. Pro- Grad (Zhu et al., 2022) performs prompt updates where the gradients are aligned with the original prompt's knowledge. Recent works (Bahng et al., 2022; Khattak et al., 2023) leverage a multi-modal prompting (image text) technique to exploit the natural association across modalities.
(2) We propose novel multi-modal augmen- tation strategy by leveraging LLMs gener- ate descriptive texts as augmented samples in the text branch, and text-conditioned diffusion mod- els to generate image augmentations for the image branch.
(3) Our novel application of attention adapter layers bridges the gap between the two modalities by generating text-guided visual features and vice-versa. This promotes the synergy between the two modalities.
cross-
(4) Extensive evaluation on 10 challenging datasets demonstrates the effectiveness of APOLLO as outperforms existing methods by a significant margin and set a new SOTA (Figure 1) for range of downstream tasks including base-to-novel gen- eralization, cross tross-dataset recognition, and domain generalization.
2
Related Works
2.1
Vision Language Models
2.3 Adapter Tuning
Recent research has indicated that effectively min- ing image-text pairs can enable VLP models to achieve highly satisfactory results on relevant downstream tasks when compared against uni- modal frameworks. For example, models like CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), and Florence (Yuan et al., 2021) demonstrate ex- ceptional performance on a wide range of few-shot and zero-shot visual recognition tasks. However, they are impractical adapt to challenging down- stream tasks. Prior work include specially designed approaches pursuing object detection (Bangalath et al., 2022; Zang et al., 2022; Zhou et al., 2022c) and, fhw-shot image recognition (Zhang et al.,
Adapter-based methods (Houlsby et al., 2019) in- ject additional trainable parameters into pre- trained model to facilitate customized learning for downstream tasks. Yuan et al. (2021); Sung et al. (2022) introduce additional layers near the predic- tion head to enrich pre-trained models with addi- tional parameters. Initial efforts to develop adap- tive methods for computer vision tasks involved in- cremental learning (Rosenfeld and Tsotsos, 2018) and domain adaption methods (Rebuffi et al., 2017. 2018). 8). APOLLO leverages adapter and prompting to boost the performance of the model for down- stream tasks, while contrastive-consistte loss en- sures the generalizability of the model.

---

Image Adapter
Text Adapter
Feed Forward
Feed Forward
Image Encoder
Encoder
Text Encoder
Text Encoder
Cross Atlention Ket Vi*
Cross Affantion
Set Attention
Self Assetton Ky
Embed
2: Ki V2-
small soft votractable claws, and tall
photo of cal
Image Adapter
Text Adapter
Frozon Params
Trainable Params
Diffusion
LLM
Cross Atlention
Figure Overview of the proposed APOLLO framework for unified multi-modal adapter and prompt learning in VLP models. APOILO unifies prompt and adapter tuning for both the image and text branches. The image (blue) and text (yellow) adapter layers are coupled with each other through cross-modal interactions via attention which further improves the alignment between the two modalities. Each modality contains branches receiving augmented versions of input texts and images generated using LLM and text-conditioned diffusion models respectively.
3 Methodology
latent space:
3.1 Brief Review of CLIP
t = TProj (wK) ; E IRd
(4)
Among the existing Vision-Language models, CLIP (Radford et al., 2021) demonstrates strong generalizability for zero-shot image recognition. Being multi-modal framework it's comprised of image and text encoder modules. In our set-up, we deploy transformer-basee image encoder (0) and text encoder (Ø).
3.1.3 Zero-Shot Classification
During zero-shot inference, text prompts are pro- vided with class labels y 21, C).. Class label ÿ with the highest cosine similarity score (sim(-)) w.r.t the input image x is computed, where 7 is the temperature parameter.
3.1.1 Image Encoder
exp (sim (u, Ug)) /T)
p(O x)
(5)
Our patch-based image embedding module is com- prised of IC transformer layers that split an image into M fixed-sized patches which are projected into patch embeddings & E MMxx.. Patch em- bedding & is fed into the next transformer block combined with learnable class token & where (I.. Therefore,
E exp (sim (u, 2;))
This template-based approach was found to pro- duce suboptimal results across domains. To ad- dress this, CoOp (Zhou et al., 2022b) proposed an alternative approach that replaces the hand-crafted prompt with set of learnable context vectors that generate task-specific text embeddings. These are augmented with the previously generated word em- beddings. Likewise, learnable context vectors are concatenated with the patch embeddings. We ex- tend MaPLe (Khattak et al., 2023) to present a novel multi-modal prompt learning approach to en- sure generalizability across different downstream tasks under a zero-shot classification setting.
[G+11 1 A, ((c),
(1)
The final image representation is obtained by projecting class token C of the last transformer layer to a common latent space:
IProo (Cx) u E IR.
(2)
3.1.2 Text Encoder
Text encoder generates feature embedding by tok- enizing the word sequence. At the (i + 1)th step W; is fed as input to the transformer text encoder:
3.2 Our Approach: APOUL
We note that prior methods based on uni-modal prompt tuning approaches to finetune CLIP to per- form downstream image recognition. On the other hand, MaPLe is the only work that uses multi- modal prompt tuning. In a major departure from this approach, we introduce a novel unified adapter
[W+1] (x++1WW) Vi (J. (3)
Like its image counterpart, the final text repre- sentation v is obtained by projecting the text em- bedding of the last transformer block Φx to a shared

---

and prompt learning method with a novel aug- mentation strategy to the text and image inputs to facilitate further regularization. To this end, we supplement our adapter layers with multi-modal cross-attention modules that ensure the text and the image modalities are aligned. The text input is passed through pre-trained LLM to obtain more descriptive expression of itself while the im- age branch comprises text-conditioned diffusion model to obtain augmentation of the input image as shown Figure 2.
sults in enhanced performance (also see Section 4.4 for detailed ablations).
3.2.3 Multi-modal Cross-attention Adapter (MCA):
Lately, adapter layers have played a pivotal role model finetuning for adaptation to new downstream tasks (Yuan et al., 2021; Li et al., 2022b; Chen et al., 2022; Hao et al., 2023). Typically, these adapter layers are trainable parameters that are added on top of encoder to transform the emhedding vec- better adapt new tasks. Inspired by Chen et al. (2022), we maintain trainable cross-attention- based adapter layers, for the respective encoders individually. In a major departure from MaPLe, where the authors used multi-modal prompt cou- pling, we introduce cross-attention modules that enforce inter-modal alignment.
3.2.1 Image and Language Prompting
We facilitate deep image and language prompting across the hierarchies in the image and text trans- former backbones (of CLIP) by introducing learn- able tokens at every transformer layer. It has been shown (Khattak et al., 2023) that systematic prompt sharing in successive stages is more intuitive over independent prompts consecutive transformer blocks ensure densely correlated feature learning.
3.2.4 Loss Formulation
Our training paradigm involves the computation of two types of losses between the multi-modal encoders.
3.2.2 Input Augmentation
Generative data augmentation schemes have been leveraged very recently (Hu et al., 2023; Trabucco et al., 2023; Azizi et al., 2023; Shipard et al., 2023; Yin et al., 2023; Whitehouse et al., 2023), however, not in multi-modal context (Appendix A.4). In our method, we add regularization by using aug- mented versions of respective modality inputs in the corresponding frozen encoder branches. In particular, for the text branch, we employ pre- trained frozen LLM (Brown et al., 2020) gen- crate sentences about the object referred to in the text. These sentences are typically descriptions of the usual characteristics of the object. For example, shown in Figure 2, we provide hand-crafted template ('a photo of <CLASS>') ) as input to the LLM. Considering Cat as the 'CLASS', LLM outputs descriptive text as: small carnivorous mammal with soft fur, retractable claws, and tail'. In this regard, KgCoOp (Yao al., 2023) also introduces textual augmentations. However, they get an average embedding from a pre-defined num- ber of sentences, whereas we restrict our model to generate single sentence (on the fly) which is casier and more diverse. For the image branch introduce text-conditioned diffusion (Rombach al., 2022) based augmentation strategy that gen- erates novel examples of the same class (see Figure 2). Experimental results back our claim that adding these two separate prompting modules indeed re-
Intra-modal Contrastive Consistency: typi- cal contrastive learning setting, the heterogeneous similarities between the query and other in-batch negatives are ignored resulting in suboptimal repre- sentation learning. In order to mitigate this *class- collision' problem and better deal with sampling bias (Wei al., 2020) proposed consistency- based contrastive loss. We employ this loss enforce consistency between the respective intra- modality branches as given below:
(6) In practice we keep 01 02 for all exper- iments. We add more details about Lco, in the Appendix A.2.2.
Loon 01 Cooo, + 02 * Eoo
Inter-modal Similarity Maximization: The inter- modality loss is responsible for maximizing the similarity between the multi-modal branches. The image-text pairs in a mini-batch are passed through the respective encoders followed by the adapter lay- The normalized image and text embeddings are subsequently utilized to compute the inter-modalin Image-Text Contrastive (ITC) loss given as LTT (refer to Appendix A.3 for further details).
The total loss is calculated by the weighted aver- age of the above two losses:
Lintal = LIT
sim + a * Lron-
(7)

---

TetgeNet
Coltech101
OxtordPets
StanfordCars
Flowers
Method
Base
Novel
Base
Novel
Base
Novel
Base
Novel
Base
Novel
CLIP
72.43
68.14
96.84
94.00
91.17
97.26
63.37
74.89
72.08
77.80
CoOp
76.47
67.88
98.00
89.81
93.67
95.29
78.12
60.40
97.60
59.67
Co-CoOp
75.98
70.43
97.96
93.81
97.69
70.49
73.59
71.75
ProGrad
77.02
66.66
98.02
93.89
95.07
97.63
68.63
95.54
71.87
KgCoOp
75.83
69.96
97.72
94.39
94.65
97.76
71.76
75.04
95.00
74.73
MaPLe
76.66
70.54
97.74
94.36
95.43
97.76
72.94
74.00
95.92
72.46
APOUL
78.91
73.43
98.69
95.63
97.11
98.94
78.16
75.66
98.07
77.83
APoOO Ma
+2.25
+2.89
+0.95
+1.27
+1.68
+1.18
+5.22
+2.15
+5.37
Food101
FGVCAircralt
SUN397
DTD
UCF101
Method
Base
Novel
Base
Novel
Base
Novel
Base
Novel
Base
Novel
CLIP
90.10
91.22
27.19
36.29
69.36
75.35
53.24
59.90
70.53
77.50
CnOp
88.33
82.26
40.44
22.30
80.60
65.89
79.44
41.18
84.69
56.05
Co-CoOp
90.70
91.29
33.41
23.71
79.74
76.86
77.01
56.00
82.33
73.45
ProGrad
90.37
89.59
40.54
27.57
81.26
74.17
77.35
52.35
84.33
74.94
KgCoOp
90.50
91.70
36.21
33,55
80,29
76.53
77.55
54.99
82.89
76.67
MaPLe
90.71
92.05
37.44
35.61
80.82
78.70
80.36
59.18
83.00
78.66
APOLLO
91.35
92.58
40.62
39.87
82.97
80.62
83,95
65.21
87.88
80,52
APPOLL MaPt.e
+0.64
+0.53
+3.18
+4.26
+2.15
+1.92
+3.59
+6.03
+4.88
+1.86
Table Comparison of APOLL with SOTA methods Base-to-Novel Class Generalization task.
APOLJO shows strong generalization capabilities across all 10 datasets and outperforms MaPL.e (previous SOTA) in all of them. Best accuracy values are shown bold and the differences with respect MaPLe are given in blue.
4 Experimental Details
4.2 Datasets
To extensively evaluate our model under differ- setting, consider total of 10 image clas- sification datasets covering various recognition tasks including object classification datasets Im- ageNet (Deng et al., 2009), Caltech 101 (Fei- Fei et al., 2004); fine-grained datasets Oxford- Pets (Parkhi et al.. 2012), StanfordCars (Krause et al., 2013), Flowers 102 (Nilsback and Zisser- man, 2008), Food101 (Bossard et al., 2014), FGV- CAAireraft (Maji al., 2013); scene recognition dataset SUN397 (Xiao et al., 2010); action recog- nition dataset UCF101 (Soomro al., 2012); and texture recognition dataset DTD (Cimpoi et al., 2014). For domain generalization, we rigorously evaluate on four ImageNet variants: ImageNetV2 (Recht et al., 2019), ImageNetSketch (Wang et al., 2019), ImageNet-A (Hendryeks et al., 2021b), and
4.1 Downstream Tasks
Base-to-Novel Class Generalization: We follow the standard practice base-to-novel generaliza- tion under zero-shot evaluation with few-shot fine- tuning. Datascts are partitioned into base and novel classes with the model being trained on the base classes and evaluated on both base and novel cate- gories.
Cross-Datased Evaluation: To analyze the zero- shot generalizability of APOLLO, we perform cross-dataset assessment. To facilitate this, the model trained solely on the ImageNet dataset was exposed to other 9 datasets. To be consistent with the prior art, we follow MaPLe and CoCoOp to train our model under a few-shot setting for a fair assessment.
ImageNet-R (Hendrycks al., 2021a).
Domain Generalization: Taking one step further towards a more robust evaluation, the performance of the model was analyzed on its distribuuttonnoo generalization capabilities. Like cross-dataset eval- uation, the performance of the ImageNet trained model was observed under its four other variants: Im2, ImageNetS, ImageNetA, and Ima- geNetR as these are known to contain sufficient domain shifts.
4.3
Main Results
4.3.1
Base-to-Novel Class Generalization
We subdivide the generalization evaluation into the following two categories.
Generalization to Unseen Classes: Table presents the comparison of our method with CLIP, CoOp (Zhou et al., 2022b), CoCoOp (Zhou et al., 2022a), ProGrad (Zhu et al., 2022), KgCoOp (Yao

---

Target
Method
Caltech
Pets
Cars
Flowers
Food
Aircraft
SUN
DID
UCF
CoOp
93,70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
66.55
Co-CoOp
94.43
90.14
65.32
71,88
86.06
22.94
67.36
45.73
68.21
MaPLe
93.53
90.49
65.57
72.23
86,20
24.74
67.01
46.49
68.69
APOILO
95.12
91.56
66.21
73.15
86.82
24.96
67.98
47.63
70.38
performance boost over the latter. The downside is unsatisfactory performance on base classes as it is only effective on 2 / 10 datasets with large drop in average performance. MapLe the other hand betters CoCoOp on of the datasets (Table I). AAPoUO AAPo MaPLe
+1.59
+1.07
+0.64
+0.92
+0.22
+0.97
Table Comparison of APOIJO with SOTA methods on a cross-datasel evaluation task where the model is trained ImageNet and evaluated on the target datasets in zero-shot manner. APOLO obtains the best accuracy among the existing methods suggesting better generalization capabilities. Best accuracies are presented in bold and improvements over MaPL.e (previous SOTA) are shown blue.
Target
Method
ImNetV2
ImNetS
ImNetA
ImNetR
CLIP
60.83
46.15
47.77
73.96
CoOp
64,20
47.99
49.71
75.21
Co-CoOp
64.07
48.75
50.63
76.18
ProGrad
64.73
47.61
49.39
74.58
KgCoOp
64.10
48.97
50.69
76.70
Note that CoOp lacks generalizability over novel classes due to its massive overfitting with the base classes. In contrast, APOUO alleviates this limi- tation and sees large gain over its predecessors proving its effectiveness in learning new classes while maintaining stable performance on base classes. Note that, method outperforms the current benchmark over all 10 datasets to obtain few-shot performance improvements up 5.22% on base categories and 2.66% overall. This sug- gests that the gain under zero-shot setting does not affect its few-shot performance or the other way around.
MaPI.e
64.07
49.15
50,90
76.98
APOμO
64.89
50.17
51.74
78.33
A APPOLO MaPLe +0.82
+1.02
+0.84
+1.35
methods Comparison of APOLLO with SOTA domain generalization task. APOLO demonstrates the best performance across all datasets. Best accuracies are present in bold and improvements over MaPLe (previous SOTA) are shown in blue.
Table 3:
et al., 2023), MaPLe on novel classes. Experimen- tal results show the superiority of the proposed approach outperforms all existing methods by significant margin on zero-shot generalization. achieves relative gain of up to 6.03% over the most recent baseline. Evaluating on all datasets, we find an average improvement of 2.69% over MaPLe on novel categories. As is evident from the tables none of the existing methods beat pre-trained CLIP (except MaPL.e) underlining the challenges of achieving satisfactory shot performance while learning new task in few-shot setting.
4.3.2 Cross-Datase Evaluation
Table presents the results on cross-dataset eval- uation where the model is trained solely on Ima- geNet and evaluated on other 9 datasets under the zero-shot setting. We find APOLLO surpasses all other methods and observe the best zero-shot im- provement of 1.69% over MaPLe (SOTA) on the UCF dataset.
Our method also enjoys an average improvement of 2.80% over CLIP on novel classes across all the datasets. This can be attributed to unified multi- modal adapter and prompting technique that ables the model to leverage the mutual association between visual and language modalities.
4.3.3 Domain Generalization
Table 3 illustrates domain generalization results of APOLLO.. We use the ImageNet dataset the base and test our model on ImageNetV2, Im- ageNetS, ImageNetA, and ImageNetR where all of them from vastly different distributions. We notice strong generalizability across different do- mains as APOLLO outperforms all prior baselines by achieving the best zero-shot improvement 1.35% over MaPLe on the ImageNetR dataset.
Generalization and Performance on Base Classes: CoCoOp extends CoOp by introducing image-conditioned prompts thus securing consid- erable generalizability and obtaining a significant

---

Accuracy (%)
Base
Bate
Base
Novel
0.5
Prompt Layer
Adaster Layer
Loss Balancing Factor (a)
Figure 3: Impact of Prompt Layer, Adapter Layer, and Loss Balancing Factor (a) on the performance of APOLO.
Intra-modal Adap.
Cross-atin.
Avg.(Base)
Avg.(Novel)
Impact of Prompt Layers: We evaluate the av- erage performance of APOILO on 10 datasets for
x
X
81.49
75.62
x
x 82.36
76.45
the base-to-noves generalization with respect to the prompt depth, i.e., the number of prompt layers (see Figure 3). Unlike MaPLe (Khattak et al., 2023), while conducting ablation, we consider the same depth (num layers) in both vision and language encoders. The general trend suggests that with increased prompt depth, the overall performance improves. Further, unlike MaPLe, obtain a slight improvement in accuracy values for the gen- eralization task when prompt depth is increased from 9 to 12. The reason for such an observation can be attributed to the presence of adapter lay- ers and intra-modal contrastive consistency losses which negatively impact performance related to in- creasing prompt depth beyond 9. Therefore, we maintain the value of prompt depth to 12 in all our experiments.
X 83.05
77.22
82.30
76.21
V
82.93
77.07
V 83.77
78.03
Table 4: Impact of Intra-modal contrastive consis- tency, Adapter tuning, and Cross-attention strategies on the performance of APOILO. Our method gives the best accuracy values (averaged 10 datasets for base- to-novel generalization task) when all three components are considered.
4.4 Ablations
Impact of Intra-modal Contrastive Consis- tency, Adapter, and Cross-attention: We assess the importance of the individual components in APOLLO and report the average accuracy scores (on 10 datasets) for the Base-to-Novel Class Gener- alization task (see Table 4). We show 3 main com- ponents in APOLLO i.e., Intra-modal Contrastive Consistency, Adapter Layers, and the utility of the Cross-attention strategy. Note that Prompt Learn- ing is present in these experiments. Further- more, input augmentation is present in cases where intra-modal consistency is taken into account. We observe the lowest performance with only prompt learning, shown in (1*f row) in Table 4. Upon addi- tion of intra-modal contrastive consistency, we no- tice significant boost (+0.87% in base and +0.83 % in novel) performance. Therefore, enforcing the mutual synergy between the same modalities re- sults in a much better generalization across several datasets. Intra-modal consistency when paired with an adapter gives a further improvement (+0.69% in base and +0.77 % in novel) in performance sug- gesting that they mutually aid each other in the generalization task. Finally, when cross attention is employed in the adapter layers we obtain +0.63 % and +0.86 % relative improvements in the base and novel classes respectively.
Impact of Adapter Layers: To evaluate the im- portance of adapters employ dual setup strat- egy. First, we study the utility adapters in in- dividual modalities (see Appendix Table 8). We notice that adding a text adapter more beneficial than adding only an image adapter. This is consis- tent with the findings outlined in Gao et al. c). However, unlike them, we notice increase in accuracy values when adapters used with both modalities. A key difference in this respect (as compared Gao et al. (2021c)) the presence of cross-attention which plays an important role in alignment. This demonstrates the importance of yet another component of APOIJO i.e., the intra-modal contrastive consistency. This under- lines that dual-modality adapters are not beneficial in the case of naive few-shot tuning, whereas approach (having both cross-attention and intra- modal consistency) leads to better generalization with more tunable parameters on both modalities. Second, we study the role the number of adapter layers using 4 different configurations as shown

---

Prompt: photo goldfish.
Prompt: photo ambulance.
Prompt: of Indigo bunting.
04
OV
Prompt: A photo of broccoll.
Prompt: photo of pineapple.
Prompt: photo of coffee mug.
Figure 4: Cross-attention visualizations as heatmaps superimposed on the respective original images showing how objects (in red) in text prompts attend relevant regions in the images. These maps depict the cross-modal alignment which improves generalization in downstream tasks.
Figure 3. Note that increasing adapter depth from I to leads to an improvement in performance in- dicating its role in understanding more complex relationships between the modalities via cross- modal alignment. However, the performance drops beyond the depth of 2 suggesting that overparame- terization may lead to an overfitting issue under a few-shot setting.
intra-modal loss leads to poor cross-modal align- ment affecting the overall performance.
Different LLMs as Text Augmentors Table 5 re-
ports the performances of ApoLLo under different LLM- based text augmenters.
LLM Text Augmentor Avg.eBase) Avg.(Novel)
VICUNA
83.16
77.29
Impact of Input Augmentations: Augmentation plays pivotal role in enforcing consistency across different branches (Chen et al., 2020; Zbontar et al., 2021). For text augmentation, we use two strategies Easy Data Augmentation (Wei and Zou, 2019) and LLM-generaale augmentation. We show that us- ing more descriptive text of the object employing LLM is more beneficial for learning intra-modal consistency than using the EDA-based approach (see Table 9). For image augmentation, we consider two schemes standard image augmentations such as cropping, flipping, solarization, and color inver- sion, and text-conditioned diffusion- based augmen- tation. We observe that the latter performs better in our case (Table 9).
GPT
83,77
78.03
Table 5: Performance comparison between different LLM-based text augmenters.
Adapters on Different Image and Text Branches Our method contains two image and text branches (where Image Branch I, Image Branch 2, Text Branch I, and Text Branch are in order from left to right in Figure 2 and represent the augmented im- age, original image, original text, and augmented text respectively). Here, we add the following table which demonstrates the effects of adapter layers on different image and text branch combinations. Please note that all these combinations involve both self- and cross-attentions. We obtain similar accu- racy values in two cases: (a) Image Branch Text Branch combination, and (b) when all four branches are taken into account. However, we se- lect case (a) with Image Branch 2 and Text Branch combination because the number of trainable pa- rameters in this scenario is half that in the latter. Moreover, Image Branch 2 Text Branch combi- nation adapters gives the highest (mean) accu- racy values on base and novel classes as shown Table 6.
Impact Loss Balancing Factor: To study the impact of the Loss Balancing (Weighting) Factor (a), we report the average accuracy values (on 10 datasets) of Base and Novel classes in the Base- to-Novel Generalization task. As shown in Figure 3, increase in the value of leads to in- creased weightage of intra-modal loss compared the inter-modal similarity loss this leads to an improvement performance. We achieve op- timal results for a = 2 which we use in all the experiments. However, an over-emphasis on the

---

Img. Br.
Img. Br. 2 Txt. Br.
Txt. Br.
(rgse
(Nvell)
trained on ImageNet using our method.
83,28
77.32
77.24
83.16
77.23
83.77
78.03
Table Impact adding adapters on different im- age and text branches.
Prompt: A photo of cake.
Prompt: photo of cup.
Incorporating Attentions in Adapter Layers We add the following ablations in Table w.r. the Adapter layers: (a) Adapter layers having no attention (b) Adapter layers with only self-attention (c) Adapter layers with self cross attention. Ex- perimental results in Table 7) demonstrate optimal performance is achieved (c) when cross attention is employed between the adapters leading to better multi-modal alignment.
Prompt: photo a girl.
Prompt: A photo of a pizza.
No-attn. Self-attn. Cross-attn. (Base
Avov(Novel)
Prompt: photo of horse.
Prompt: A photo of woman.
82,54
76.98
83.05
77.22
Figure 5: Cross-attention visualizations as heatmaps superimposed the respective original images in multi-object setting.
83.77
78.03
Table 7: Impact incorporating attention adapter layers.
Conclusions and Future Work
4.5 Qualitative Results
We present APOLLO, unified multi-modal adapter and prompt tuning method facilitate few- shot learning without compromising zero-shot gen- eralizability. To this end, we leverage contrastive- consistency loss and ensure cross-modal synergy which results in an improved performance on threc carefully chosen downstream tasks: base-to-novel generalization, cross-datased evaluation, and do- main generalization on 10 benchmark datasets observed from the experimental results. Morcover, the ablation analysis outlines the salient contribu- tions of each of the sub-modules. Future work can include assessing the performance of such frame- works for fine- ane-ggrined tasks which remains an open problem. Moreover, the adapter layers may lead to parmmetcccctiinno which can be further opti- mized in subsequent works.
Through Figure and Appendix Figure 6 visu- alize cross-attention maps guided by the respective text prompts. The cross-attention scores between the image and the associated text token referring to the object (highlighted in red each prompt) are extracted and bilinearly interpolated to match the image dimension and superimposed on the original image. These maps depict the alignment between images and corresponding texts for variety of classes, including fish, food, and vehicles. Such cross-modal awareness leads to better downstream tasks.
Qualitative Analysis on Multi-object Setting Examples of the cross-attention visualization on images with multiple objects are shown Figure 5. Here, each of the images contains more than one object category. We generate textual prompts using the same template "A photo of <CLASS>> for individual objects as shown in the figure. The attention maps are activated in the corresponding re- gions showing fine-grained alignment of APoLLo under challenging scenarios with objects from mul- tiple categories. Please note that here we did a zero-shot transfer on these images with the model
Acknowledgement: The research has been sup- ported by ARO grants W911NF2310352 and W91 10022

---

References
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
Shekoofeh Azizi, Simon Korbblith, Chitwan Saharia, Mobammad Norouzi, and David J Fleet. 2023. Syn- thetic data from diffusion models improves imagenet classification. arXiv preprint Xiv:2334.008-66
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan and Phillip Isola. 2022. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 17274, 1(3):..
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang. Lu Yuan, Nanyun Peng, et al. 2022. An empirical study of training end-to-end lision-and-languaa transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18166-18176
Hanoona Bangalath, Muhammad Maaz, Muham- mad Uzair Khattak, Salman H Khan, and Fahad Shah- baz Khan. 2022. Bridging the gap between object and image-level representations for open-vocabulary detection. Advances in Neural Information Prucess- ing Systems, 35:33781-33794
Li Fei-Fei, Rob Fergus, and Pictro Perona. 2004, Learn- ing generative visual models from few training ex- amples: An incremental bayesian approach tested on 101 object categories. In 2004 conference com- puter vision and pattern recognition workshop, pages 178-178. IEEE
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Atlah Khan, Shivanshu Purchit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXii:230400137.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang. Yongfeng Zhang. Hongsheng Li. and Yu Qiao. 2021a. Clip-adapter: Better vision- language models with feature adapters. arXiv preprint arXiv:2110.0454.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014 4. with random forests. In Computer Vision-ECCV ECCV 2014: 13th European Conference. Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer.
discriminative components
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2021b. Clip-adapter: Better vision- language models with feature adapters. arXiv preprint arxiv:2110.04444.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-19990.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2021c. Clip-adapter: Better vision- language models with feature adapters. arXiv preprint
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607. PMLR
Tianxiang Hao, Hui Chen, Yuchen Guo, and Guiguang Ding. 2023. Consolidator: Mergeable adapter with grouped connections for visual adaptation. arXiv preprint 205.02300..0003..
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. 2022. Vision trans- former adapter for dense predictions. arXiv preprint arXiv:2205.08534
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. 2021a. The many faces of robustness: A critical analysis of out-of outoof-diitribution generalization. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pages 8340-8349.
Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. 202 lb. Natural adversarial examples. Proceedings of the IEFF//VF Confer- ence on Computer Vision and Pattern Recognition. pages 15262-15271.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-effiieent transfer learning for nip. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXXvv1880..44

---

Xuming Hu, Aiwei Liu, Zeqi Tan, Xin Zhang, Chenwei Zhang. Irwin King, and Philip S Yu. 2023. Gda: Generative data augmentation techniques for relation extraction tasks. arXiv preprint arXiv:2305. 16663.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021a. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprimt arXiv:21
Tony Huang, Jack Chu, and Fangyun Wei. 2022. Unsu- pervised prompt learning for vision-language models. arXiv preprint
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692..
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916, PMLR.
7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021b. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 10012-10022.
Muhammad Uzair Khattak, Hanoona Rasheed, Muham- mad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Maple: Multi-modal prompt learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang. Yajing Liu, and Xinmei Tian. 2022. Prompt distribution learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition. pages 5206-5215.
Konwoo Kim, Michael Laskin, Igor Mordatch, and Deepak Pathak. 2021. How to adapt your large-scale vision-and-lane vision-and-languaae model.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 2013. Fine-grained visual classification of aircraft. arXiv preprint arXiv:130..515I
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei- Fei. 2013. 3d categorization. In Proceedings of the IEEE inter- national conference on computer vision workshops, pages 554-561.
object representations for fine-grit grained
Maria-Elent Nilsback and Andrew Zisserman. 2008. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Com- puter Vision, Graphics & Image Processing, pages 722-729. IEEE
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuning. arXiv preprint
Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hong- sheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xi- aogang Wang. Wenhai Wang. et al. 2023a. Uni- perceiver v2: A generalist model for large-scale vi- sion and vision-langue tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2691-2700.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman. and CV Jawahar. 2012. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3508 IEFF.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a. Blip: Bootstrapping language-image pre-training for unified languael understand- ing and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-551.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning:
Optimizing continuous prompts for generation. arXiv preprint 2xxv:2111100111
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. 2023b. Scaling language-iman pre-training via masking. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. Advances in newral informa- tion processing systems, 30.
Yanghao Li. Hanzi Mao, Ross Girshick, and Kaim- ing He. 2022b. Exploring plain vision transformer backbones for object detection. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part IX, pages 280-296. Springer.
Sylvestre-AIvee Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2018 Efficient parametrization of multi- domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pateer Recognition, pages 81 19-81177

---

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakbutdinov. 2023. Effective data aug- mentation with diffusion models. arXiv preprint
arXiv:2302.07944.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. 2022. High- resolution image symthesis with latent diffusion mod- els. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10695.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. 2019. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32.
Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. 2023a. Position-guided text prompt for vision-language pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 33242-23255
Amir Rosenfeld and John K Tsotsos, 2018. Incremen- tal learning through deep adaptation. IEEE transac- tions on pattern analysis and machine intelligence, 42(3):651-665 163..
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2023b. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 19175- 19186.
Teven Le Scao, Angela Fan. Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph. William Fedus, Xinyun Chen, al. 2023. Flan-moe: Scaling cnrruction-finetunee language models with sparse mixture of experts. arXiv preprint 14705
Chen Wei, Huiyu Wang. Wei Shen, and Alan Yuille, 2020. Co2: Consistent contrast for unsuper- vised visual representation learning. arXiv preprint arXiv:2010.02217.
Jason Wei and Kai Zou. 2019. Eda: Easy data augmenta- tion techniques for boosting performance on text clas- sification tasks. arXiv preprint arXiv: 111990111111. 11196.
Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang. and Clinton Fookes. 2023. Diversity definitely needed: Improving model-agnostic zero- shot classification via stable diffusion. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 769-778.
Chenxi Whitehouse. Monojit Choudhury. and Al- ham Fikri Aji. 2023. Lim-powered data augmen- tation for enhanced crosslingual performance. arXiv preprint arXiv:2305. 14288.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. 2022. Test-time prompt tuning for zero-shot gener- alization in vision- language models. arXiv preprint arXiv:2209.05511.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: Large-scale scene recognition from abbey to 20o. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3492. IEEE
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. Ucf101: A dataset of 101 human ac- tions classes from vidcos in the wild, arXiv preprint arXiv:1212.0402.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Li- juan Wang. 2022. Unitab: Unifying text and box outputs for grounded vision-language modeling. Computer Vision-ECC' 2022: 17th European Con- ference, Tel Aviy, Israel, October 23-27, 2022, Pro- ceedings, Part XXXVI, pages 521-539. Springer.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. VI-adapter: marameter-efficien transfer learning for vision-and- vision-and-languaag tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 5227-5237.
Mingxing Tan and Quoe Le. 2019. Efficientnet: Re- thinking model scaling for convolutional neural net- works. In International conference on machine learn- ing, pages 6105-6114, 14. PMLR.
Hantao Yao, Rui Zhang, and Changsheng Xu. 2023. Visual-language prompt tuning with knowledge- guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6757-6767.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet. Marie-Aune Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.15 139771.
Yuwei Yin, Jean Kaddour, Xiang Zhang. Yixin Nie. Zhenguang Liu, Lingpeng Kong, and Qi Liu. 2023. Ttida: Controllable generative data augmentation via text-to-text and text-to-image models. arXiv preprimt arXiv:2304.0882.

---

Lu Yuan, Dongdong Chen. Yi-Ling Chen, Noel Codella. Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. 2021. Florence: new foundation model for computer vision. arXiv preprint arXi::2111.11432.
Yuhang Zang. Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. 2022. Open-vocabulay, detr with conditional matching. In Computer Visi ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 106-122. Springer.
Jure Zbontar, Li Jing. Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310- 12320. PMLR.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li. Jifeng Dai, Yu Qiao, and Hongsheng Li. 2021. Tip-adapter: Training-free free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pat- tern Recognition, pages 16816-16825.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Learning to prompt for vision- language models. International Journal of Computer Vision, (330)22337722388.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühh, and Ishan Misra. 2022c. Detecting twenty-thousand classes using image-level supervi- sion. In Computer Vision-ECCV 2022: 17th Eu- ropean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 350-368. Springer.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. 2022. Prompt-aligned gradient for prompt tuning, arXiv preprint ar0iv:2205. 14865.

---

A Appendix
The similarity between the positive p and the neg- ative keys n; {1, 21....... .)) in the form of probability P(i) is written as:
})
A.1 Details for Radar Chart Figure! 1
In this section, explain the details of the radar chart shown in Figure 1. Each axis denotes accu- racy value, either on base or novel class (specified accordingly) for the Base-to-Novel Generalization task. total, there are 10 datasets for this task and therefore we plot 10 vertices where each denotes a ratio relative to our performance. This is calcu- lated on the basis of normalizing the performance of either APOLLO previous SOTA (i.e., MaPLe) by that of APOLL,, and is therefore kept in the range of (0, 1]. Here we set the radar chart's origin be 90% and the outermost frame be 100%. This essentially separates the adjacent frames for better readability. The accuracy values annotated against the vertices are the absolute values (without normalization) obtained using both methods.
P()) exp(p n;/ ni/Teon) exp((p Dk/Tean) n
(10)
Consistency between the probability distributions P and Q is imposed in the form of Symmetric KL Divergence Loss.
Lconsistency = 2 (KL(P,2) KI(2,))))) (I))
The predicted similarity distribution of the positive key to each crop of the other data, P, acts as a soft pseudo label to that of the query. Q. The total loss is weighted combination of contrastive and consistency losses given
Lco: LNCE BRomsistency
(12)
A.2 Intra-modal Contrastive Consistency Loss
where /3 is the balancing coefficient. We follow the recommended values of Feon and 3 as mentioned in Wei al. (2020) since they have shown to perform the best in our experiments.
Our intra-modal contrastive consistency loss differs from a conventional contrastive loss in the aspect that it contains consistency terms that prevent class- collision problem (Wei et al., 2020). Therefore, has two terms contrastive loss and a consistency loss.
A.3 Inter-modal Similarity Maximization
During training we consider a mini-batch (N) con- taining image-captions pairs, { TT), Dy where I and T, represent jth image and text pair, respec- tively. After passing these image-text pairs through respective encoders (and adapter layers) we obtain the normalized image embedding as zf C Rd and
A.2.1 Contrastive Loss
A typical contrastive loss function in the form of an InfoNCE loss is given as:
text embedding as zf R.d. These representa- tions are also aware of each other (modality- They are subsequently used to compute this inter- modality Image-Texx Text Contrastive (ITC) loss given as IT
exp(q p/T) exp(q p/T) + Σ exp(q nz/T) n
LNCE log
(8)
where p represents the positive key, represents the query, and na represents the negative key in minibatch.
sim
A.2.2 Consistency Loss
C
1
N
log
exp
(zf,22))/T)
2N
ΣN exp
(F)T
Taking inspiration from semi-supervised learning, consistency loss is proposed by Wei al. (2020) to strengthen the consistency between the similarities the query data and the positive data.
j=1
Contrasting images with the texts
exp (z//,
Σ
log
The similarity between the query q and the neg- ative keys ng can be represented in the form of probability ((i) which is denoted as:
2NV
V C exp
Contrasting texts with the images
(13)
exp(q nz/Teon)
2(1) Σ explq D2/ Tcon) ni
where ( denotes inner product, and r is the temperature parameter.

---

Prompt: photo of tiger shark.
Prompt: photo passenger car.
Prompt: photo of umbrella.
Prompt: photo persian cat.
Prompt: A photo strawberry.
Prompt: photo electric guitar.
Figure 6: Cross-attention visualizations as heatmaps superimposed on the respective original images showing how objects (in red) in text prompts attend relevant regions the images (Extension of Figure 4).
A.4 Generative Data Augmentation
SGD optimizer with a base learning rate of 0.004. We follow MaPLe for setting the values of prompt depths and lengths in our experiments.
Generative data augmentation an emerging field where generative models are used to generate aug- mented (perturbed) samples of the data (Hu et al.. 2023; Trabucco et al., 2023; Azizi et al., 2023; Shipard et al., 2023; Yin et al., 2023; Whitehouse et al., 2023). It is prominent in the field of Natural Language Processing where Large Language Mod- els (LLMs) are used data augmenters (White- house al., 2023; Yin al., 2023). Given prompt, LLMs provide several descriptive versions of which can potentially be used to increase the size of the dataset, especially a low-dataset regime. Further, text-to-image generation and dif- fusion models (Yin et al., 2023; Trabucco et al., 2023; Azizi et al., 2023; Shipard et al., 2023) are also used for effective data augmentation strategies. However, these types of augmentation schemes have not been explored together previously in multi-modal (vision-language) context. To the best our knowledge, ours the first to leverage the capabilities of LLMs and text-conditioned image generation (diffusion) models together in a unified framework for effective augmentation in a multi- modal learning context.
A.6 Ablation Tables
A.6.1 Impact of Adapter Layers
The impact of adapter layers is shown in Table 8.
Average
Image
Text
Base
Novel
x
83.16
77.31
83.41
77.48
83.77
78.03
Table Impact of using different adapters on the performance of APOLL.. Our method gives the best accuracy values when both image and language adapters are used.
A.6.2 Impact of Input Augmentations
Impact of input augmentation is shown Table 9.
Average
Modality
Augmentation
Base
Novel
Standard
83.28
77.45
Image
Diffusion
83.77
78.03
A.5 Implementation Details
EDA L.M
82.81
77.04
Text
Following MaPLe (Khattak et al., 2023), we uti- lize the pre-trained ViT-B/16 CLIP model vision-language backbone. In all experiments, fine-tune the model employing few-shot train- ing strategy with 16 samples randomly sampled from each class for all known classes. We train our model on a single GPU for 10 epochs using an
83.77
78.03
Table 9; Impact of different input augmentation strategies on the performance of APOLL.. Our method gives the best accuracy values when LLM is used for text augmentation and the text-condition image generation model is used for image augmentation.


Reviews:
[{"review_id": "B0CNln42S2", "reviewer": "Reviewer_BqGK", "summary": "", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive comments. Please find our responses below. The tables here represent mean accuracies reported over all the 10 datasets.\n\n$\\textbf{1. Ablations on adapter architecture}$\n\nPlease refer to Figure 3 (main paper) where we add ablations with varying number of adapter layers and also Table 4 (main paper) which reports the impact of intra-modal contrastive consistency, adapter tuning and cross attention strategies.\n\nFurthermore, our method contains two image and text branches (where Image Branch 1, Image Branch 2, Text Branch 1, Text Branch 2 are in order from left to right in Figure 2 of main paper and represent the augmented image, original image, original text and augmented text respectively). Here, we add the following table which demonstrates effects of adapter layers on different image and text branch combinations. Please note that all these combinations involve both self- and cross-attentions. We obtain similar accuracies in two cases: (a) Image Branch 2 - Text Branch 1 combination, and (b) when all four branches are taken into account. However, we select the former scenario (a) with Image Branch 2 and Text Branch 1 combination because the number of trainable parameters in this scenario is half than that in the latter. Moreover, Image Branch 2 - Text Branch 1 combination of adapters gives the highest (mean) accuracy values on base and novel classes.\n\n$\\textbf{Table 1. Impact of adding adapters on different image and text branches}$\n\n| Image Branch 1 | Image Branch 2 | Text Branch 1 | Text Branch 2 | Avg. (Base) | Avg. (Novel) |\n| :----: | :----: | :-----: | :--------: | :-----------------: | :-------------: |\n| &check; | - | &check; | - | 83.28 | 77.32 |\n| - | &check; | - | &check; | 83.19 | 77.24 |\n| &check; | - | - | &check; | 83.16 | 77.23 |\n| &check; | &check; | &check; | &check; | 83.77 | 78.01 |\n| **-** | **&check;** | **&check;** | **-** | **83.77** | **78.03** |\n\nWe add the following ablations w.r.t to the Adapter layers:\n(a) Adapter layers having no attention \n(b) Adapter layers with only self-attention \n(c) Adapter layers with self + cross attention\n\nExperimental results show the optimal performance is achieved in (c) when cross attention is employed between the adapters leading to better multi-modal alignment.\n\n\n$\\textbf{Table 2. Impact of incorporating attentions in the adapter layers}$\n\n| No-Attn | Self | Cross | Avg. (Base) | Avg. (Novel) |\n| :----: | :----: | :-----: | :--------: | :-----------------: |\n| &check; | - | - | 82.54 | 76.98 |\n| - | &check; | - |  83.05 | 77.22 |\n| **-** | **&check;** | **&check;** | **83.77** | **78.03** |\n\nWe will add these tables in the supplementary of the camera ready version and make the necessary changes. Also, as mentioned in the paper, we will make the codebase public upon acceptance to facilitate reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a framework that unifies adapter and prompt learning to improve Vision Language Models' performance.\n2. Data augmentation approach with LLM and diffusion models are used to enhance contrastive learning performance. \n3. Good empirical results on multiple benchmarks, and detailed ablation studies support the claims well. \n4. The paper is well-written and easy to follow.", "reasons_to_reject": "1. Both adapters and prompt tunning have been explored in VLP for some time. More ablations on adapter architecture choice might be discussed to bring more novelty to the approach. ", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper proposes a unified adapter and prompt learning to improve Vision Language Models' zero-shot and few-shot capabilities. Specifically, they introduce trainable cross-attention based adapter layers in both modalities. To better train the model to learn cross-modal alignment, this paper first proposes multi-modal augmentation strategy to create positive and negative pairs for intra-modal and cross-modal contrastive learning. Besides, both intra-modal contrastive consistency and inter-modal similarity maximization are proposed for training the model. Empirical results on multiple benchmarks demonstrate the effectiveness of the proposed approach in generalization in zero-shot and few-shot settings. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "Suv7I1KGRI", "reviewer": "Reviewer_bnbq", "summary": "", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive comments. Please find our responses below. The tables here represent mean accuracies reported over all the 10 datasets.\n\n$\\textbf{1. Choice of LLM text augmentor}$\n\nGPT-3 has been widely used and validated as a text-augmentor method (Whitehouse et al., 2023, Yin et. al. 2023, Peng et al. 2023,). In particular, Whitehouse et al. (2023) demonstrated the superiority of GPT variants over models like VICUNA (Wei-Lin et al. 2023) as used in InstructBLIP and MiniGPT4. Taking inspiration from these previous works, we chose GPT-3 as the LLM-based text augmentor.\n\nHowever, as suggested by the reviewer, we also experimented with VICUNA (LLM for InstructBLIP and MiniGPT4) as a text augmentor. Experimental results show that our GPT based (Brown et al. 2020) text augmentation strategy is more superior possibly due to the diversification of the samples generated and the robustness of GPT (as found in Whitehouse et. al. 2023). Table 3 demonstrates the performance comparison when APoLLo is equipped with these two text augmentors:\n\n$\\textbf{Table 3. Performance comparison between different LLM based text augmentors}$\n\n| LLM Text Augmentor | Avg. (Base) | Avg. (Novel) |\n| :----: | :----: | :-----: |\n| VICUNA  | 83.16 | 77.29 |\n| **GPT** | **83.77** | **78.03** |\n\n\n$\\textbf{2. Ablation on adapter choices}$\n\nAdapter layers were initially added in both augmented image and as well as text branches. However, that increased the number of trainable parameters of the pipeline by nearly two times without substantial performance gains. In fact, our proposed architecture with two adapters (as shown in Fig. 2 in main paper) has performed the best. Please refer to the following table (Table 1 from previous response Reviewer $\\textcolor{Blue}{\\text{BqGK}}$) for more details:\n\n$\\textbf{Table 1. Impact of adding adapters on different image and text branches}$\n\n| Image Branch 1 | Image Branch 2 | Text Branch 1 | Text Branch 2 | Avg. (Base) | Avg. (Novel) |\n| :----: | :----: | :-----: | :--------: | :-----------------: | :-------------: |\n| &check; | - | &check; | - | 83.28 | 77.32 |\n| - | &check; | - | &check; | 83.19 | 77.24 |\n| &check; | - | - | &check; | 83.16 | 77.23 |\n| &check; | &check; | &check; | &check; | 83.77 | 78.01 |\n| **-** | **&check;** | **&check;** | **-** | **83.77** | **78.03** |\n\n$\\textbf{3. Cross-attention visualization on images with multiple objects}$\n\nThank you for bringing this to our notice. As suggested by the reviewers, we provide some examples of the cross-attention visualization on images (from COCO validation set) with multiple objects embedded in the anonymous link (as approved by the program chairs): https://ibb.co/dczvjcN\n\nHere, each of the images contains more than one object category. We generate textual prompts using the same template “A photo of a <CLASS>” for individual objects as shown in the figure. The attention maps are activated in the corresponding regions showing fine grained alignment of APoLLo under challenging scenarios with objects from multiple categories. Please note that here we did a zero-shot transfer on these images with the model trained on ImageNet using our method.\n\n$\\textbf{References}$\n\nLLM-powered Data Augmentation for Enhanced Crosslingual Performance (Whitehouse et al. 2023)\n\nTTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models (Yin et al. 2023)\n\nCheck Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback (Peng et. al. 2023)\n\nVicuna: An Open-Source Chatbot Impressing GPT-4 with 90\\%* ChatGPT Quality (Wei-Lin et al. 2023)\n\nLanguage models are few-shot learners (Brown et al. 2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The APoLLo achieves state-of-the-art performance on multiple-benchmarks\n2. The authors first combine prompt tuning and adaptor on foundational models and nicely design the contrastive loss for intra and inter-modalities with sufficient ablation study to prove its effectiveness.\n3. The augmentation method is simple and effective, and more interesting work could base on the authors' work as generative models evolve fast recently.", "reasons_to_reject": "1. I am wondering why the author chose GPT-2 as their LLM text augmentor while instructBLIP and MiniGPT4 are already available, but this concern is minor\n2. Could you explain why not apply the adaptor on the augmented images and texts\n3. I am wondering how the cross-attention visualization works on images with multiple objects", "questions_to_authors": "See weakness", "paper_topic_and_main_contributions": "The authors propose APoLLo, which combines adapter and prompt learning to improve the generalization of these models when they are applied to new tasks with few examples. APoLLo introduces several novel aspects, such as a multi-modal augmentation strategy that generates descriptive texts and image augmentations using large language models (LLMs) and text-conditioned diffusion models. It also employs a contrastive-consistency loss that enforces consistency between the respective encoder branches and prevents overfitting in downstream tasks. APoLLo outperforms existing methods by a significant margin and sets a new state-of-the-art on 10 challenging datasets. The page concludes with some future directions for improving the performance and robustness of vision-language models", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aF6b6wwR90", "reviewer": "Reviewer_4n34", "summary": "", "questions": "- Which LLM and diffusion models were used in the paper? \n- Is it one generated image per sample? \n- Have evaluations been conducted on the generated images and text?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive comments. Please find our responses below. \n\n$\\textbf{1. Quality of augmentations}$\n\nQuality of generated data in an augmentation pipeline has been previously examined in (Whitehouse et al., 2023, Yin et. al. 2023, Peng et al. 2023,). Whitehouse et al. (2023) demonstrated the superiority of GPT variants over models such as VICUNA (Wei-Lin et al. 2023) as LLM-based text-augmentor using a Generation Success Rate metric. On the other hand, the quality of generated augmented samples have also been explored by Azizi et al. (2023) using FID scores. Taking inspiration from these works we considered pre-trained Stable Diffusion (Rombach et al. 2022) as the image augmentor and GPT as the text augmentor generating diverse yet meaningful image-text pairs. Furthermore, it is worth noting that the standard (non-generative) techniques of image data augmentation restrict themselves to simple transformations like rotations, flipping etc. However, these augmented images are not semantically diverse enough. In contrast, our method edits images to change their appearances using an off-the-shelf diffusion model, and generalizes to novel visual concepts from only a handful of samples.\n\nTo the best of our knowledge, this is the first work to leverage the potential of these pretrained generative frameworks for data augmentation in a few-shot unified multimodal prompt and adapter learning setting. Please note quality assessment of the generated augmentations is beyond the scope of this work (can be a possible future work) and we conform to these prior works (mentioned before) which concludes that the augmentation strategies are indeed trust-worthy.  \n \n$\\textbf{2.  LLM and diffusion models}$\n\nAs addressed above, the image augmentation branch consists of a pre-trained text-to-image stable diffusion model (Rombach et al. 2022) which is applied on input images to generate diverse yet meaningful image samples. The text branch comprises a pre-trained LLM, GPT (Brown et al. 2020), to obtain a descriptive sentence from an input prompt text of a generic format (a photo of a ‘class’). \n\n$\\textbf{3. Number of generated images per sample}$\n\nYes, for each input image-text pair we obtain a corresponding augmented image-text counterpart. \n\n$\\textbf{4. Evaluations protocol}$\n\nPlease note that the evaluations are not conducted on generated images and text due to the lack of a baseline performance of any existing model. For a fair comparison against all prior works and to be consistent with the datasets used in our work (see Section 4.2), we strictly use the original text and images for all our evaluations. \n\n$\\textbf{4. Typos}$\n\nThank you, we will update this in the camera ready version.\n\nAlso, as mentioned in the paper, we will make all our code public upon acceptance to facilitate reproducibility.\n\n$\\textbf{References}$\n\nLLM-powered Data Augmentation for Enhanced Crosslingual Performance (Whitehouse et al. 2023)\n\nTTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models (Yin et al. 2023)\n\nCheck Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback (Peng et. al. 2023)\n\nVicuna: An Open-Source Chatbot Impressing GPT-4 with 90\\%* ChatGPT Quality (Wei-Lin et al. 2023)\n\nLanguage models are few-shot learners (Brown et al. 2020)\n\nHigh resolution image synthesis with latent diffusion models. (Rombach et al. 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is the first to unify adapter tuning and prompt tuning, facilitating the learning of new tasks in a few-shot setting without compromising zero-shot generalization. By clarifying the differences from recent related research, the position of this study is clearly defined. Through comparison with the latest models on ten datasets, the utility of the method is demonstrated by its superior results. This contribution greatly contributes to the advancement of the related fields, yielding highly useful results.\n- With the introduction of multi-modal augmentation via LLMs and text-conditioned diffusion models, the adoption of multi-modal cross-attention adapter layers, and the introduction of Intra-modal Contrastive Consistency, the impact of these components is clearly revealed through appropriate ablation studies. Furthermore, by visualizing cross-attention, the understanding of the proposed method is promoted, reflecting the exceptional quality of this paper.", "reasons_to_reject": "- One issue that seems to remain is the lack of clarity regarding the impact of the quality and quantity of augmentations generated by the LLM and the text-conditioned diffusion model on the proposed method. The paper has demonstrated through experiments that input augmentation was effective, but there is a lack of detailed descriptions about the LLM and diffusion models used in the paper. It seems necessary to address the matter of their quality and quantity.", "questions_to_authors": "- Which LLM and diffusion models were used in the paper? \n- Is it one generated image per sample? \n- Have evaluations been conducted on the generated images and text?", "paper_topic_and_main_contributions": "This paper proposes APoLLo, a fine-tuning method for visual-language pretrained (VLP) models for multi-modal downstream tasks. The proposed method is a novel integrated adapter and prompt learning approach for VLP models, addressing the issue of generalization in few-shot scenarios. The main contributions of this paper are as follows:\n\n- The proposed method is the first to unify adapter tuning and prompt tuning for VLP models, facilitating the learning of new tasks in a few-shot setting without compromising zero-shot generalization.\n- By leveraging LLMs to generate explanatory text for augmented samples and using text-conditioned diffusion models for image augmentation, a new multi-modal augmentation strategy is proposed.\n- A novel application of multi-modal cross-attention adapter layers is introduced, bridging the gap between the two modalities and fostering a synergistic effect between them.\n- Through evaluation on ten datasets, the effectiveness of APoLLo is demonstrated, setting new state-of-the-art benchmarks on various downstream tasks, including generalization from base to novel, cross-dataset recognition, and domain generalization.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- p.5 l366: A period is missing."}, "extra_text": {}}, {"review_id": "M6kbcX8L4t", "reviewer": "Reviewer_SvZA", "summary": "", "questions": "- Could you please provide some statistics for the training cost? I wonder if the augmented images are generated offline to accelerate training. If both the LLM and diffusion models augment data on the fly, it seems very expensive in terms of a single GPU. \n- The augmentation greatly depends on the diffusion and LLM models. It may be helpful to report performance impact with different size of diffusion/LLM models. This may point out the hard requirements for those models.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive comments. Please find our responses below. The tables here represent mean accuracies reported over all the 10 datasets.\n\n$\\textbf{1. Novelties, contributions and performance gains}$\n\nWe would like to mention that one of our primary contributions is the unification of adapter and prompt learning within the same framework for VLP models (Section 3.2). To the best of our knowledge, this has not been explored by the community before. Our method facilitates learning new tasks in a few-shot setting without compromising on their zero-shot generalizability.\n\nPlease refer to Table 1 of the main paper which shows the superior performance of ApoLLo over prior SOTA MaPLe (Khattak et. al. 2023). Note that the highest performance gain of $\\textbf{6.03%}$ is obtained in Novel classes of DTD dataset. To show the overall efficacy of our method we report in Table A.6.1 and A.6.2 the average (over 10 diverse datasets) accuracy improvements. It is to be noted that, as outlined in CLIP (Radford et al. 2021), there is less room for improvements under zero-shot setting in some of the datasets. Similar trends in the results and corresponding improvements have also been found in studies such as MaPLe, KgCoOp (Yao et. al 2023). Therefore, reporting the mean of all the datasets (as shown in Table A.6.1 and A.6.2), does not fully capture those cases where we obtain significant improvements on the individual datasets (refer Table 1). However, for ablation studies we report the mean accuracy values following previous works for fairness (MaPLe).\n \nIn addition to this as outlined in the paper (under main results in page 2), we make the following novel contributions:\n\n(a) We propose a novel multi-modal augmentation strategy by leveraging LLMs to generate descriptive texts as augmented samples in the text branch, and text-conditioned diffusion models to generate image augmentations for the image branch. Although LLM based augmentations have been explored before, to the best of our knowledge, multi-modal augmentation is relatively unexplored. \n\n(b) Our novel application of multi-modal cross attention adapter layers bridges the gap between the two modalities by generating text-guided visual  features and vice-versa. This promotes the synergy  between the two modalities.  \n\n(c) Finally, extensive evaluation on 10 challenging datasets demonstrates the effectiveness of APoLLo as it outperforms existing methods by a significant margin and set a new SOTA (Figure 1) for a range of downstream tasks including base-to-novel generalization, cross-dataset recognition, and domain generalization.\n\n$\\textbf{2. Applicability of diffusion and LLMs as data augmentors}$\n\nGenerative data augmentation is an emerging field where generative models are used to generate augmented (perturbed) samples of the data (Hu et al. 2023; Trabucco et al., 2023; Azizi et al., 2023; Shipard et al. 2023; Yin et al., 2023; Whitehouse et al. 2023,). It is prominent in the field of Natural Language Processing where Large Language Models (LLMs) are used as data augmenters (White house et al., 2023; Yin et al. 2023,). Given a prompt, LLMs provide several descriptive versions of it which can potentially be used to increase the size of the dataset, especially in a low-dataset regime. Further, text-to-image generation and diffusion models (Yin et al.,2023; Trabucco et al.,2023; Azizi et al.,2023; Shipard et al.2023,) are also used for effective data augmentation strategies.\n\nHence it is a common practice to use LLMs as augmentors by leveraging the capabilities of these models which are trained on varied datasets. Therefore we believe using them effectively as a tool for data augmentation in few-shot settings can boost the performance (as mentioned in Table A.6.2).\n\n$\\textbf{3. Cross-attention visualization on images with multiple objects}$\n\nThank you for bringing this to our notice. As suggested by the reviewers, we provide some examples of the cross-attention visualization on images (from COCO validation set) with multiple objects embedded in the anonymous link (as approved by the program chairs): https://ibb.co/dczvjcN\n\nHere, each of the images contains more than one object category. We generate textual prompts using the same template “A photo of a <CLASS>” for individual objects as shown in the figure. The attention maps are activated in the corresponding regions showing fine grained alignment of APoLLo under challenging scenarios with objects from multiple categories. Please note that here we did a zero-shot transfer on these images with the model trained on ImageNet using our method.\n\n$\\textbf{4. Implementation details}$\n\nIn our current set up, we generate the images using a text-to-image stable diffusion model a priori. On the other hand, the text augmentor was leveraged to generate on-the-fly text augmentations to ensure diversification in the generated samples.\n\nExperimental results show that this strategy yields superior performance in terms of computational resources as compared to when both the image and text augmentations are done offline. We also observe better model performances in the former case. We attribute this to the fact that when we use the LLM on-the-fly to obtain the textual augmentations they are able to generate diverse captions instead of a fixed caption (when done offline). To ensure the generated images are regulated within a certain variability (using FID scores), we make the image augmentation offline. Furthermore, for all our experiments, we have used a single A100 GPU and adopted memory efficient techniques such as gradient checkpointing as commonly used in the deep learning community.\n\n$\\textbf{5. Impact of different text augmentors}$\n\nAs also suggested by Reviewer $\\textcolor{Green}{\\text{bnbq}}$, we add the ablation studies with different diffusion based image augmentors and LLM based text augmentors. Following table reports the performances of ApoLLo under different LLM-based text augmentors. We will add these results in the Supplementary section of our paper.\n\n$\\textbf{Table 3. Performance comparison between different LLM based text augmentors}$\n\n| LLM Text Augmentor | Avg. (Base) | Avg. (Novel) |\n| :----: | :----: | :-----: |\n| VICUNA  | 83.16 | 77.29 |\n| **GPT** | **83.77** | **78.03** |\n\n\nAlso, as mentioned in the paper, we will make the codebase public upon acceptance to facilitate further research in this direction.\n\n$\\textbf{References}$\n\nMaPLe: Multi-modal prompt learning (Khattak et. al 2023)\n\nLearning Transferable Visual Models From Natural Language Supervision (Radford et. al. 2021)\n\nVisual-language prompt tuning with knowledge-guided context optimization (Yao et. al, 2023)\n\nGda: Generative data augmentation techniques for relation extraction tasks (Hu et. al. 2023)\n\nEffective data augmentation with diffusion models (Trabucco et. al. 2023)\n\nSynthetic data from diffusion models improves imagenet classification. (Azizi et. al. 2023)\n\nLLM-powered Data Augmentation for Enhanced Crosslingual Performance (Whitehouse et al. 2023)\n\nTTIDA: Controllable Generative Data Augmentation via Text-to-Text and Text-to-Image Models (Yin et al. 2023)\n\nDiversity is definitely needed: Improving model-agnostic zero shot classification via stable diffusion (Shipard et. al. 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The designed network architecture is interesting. \n- The proposed method achieves strong improvements on three categories of downstream tasks.\n- The authors provide a large number of empirical results and interesting discussions.", "reasons_to_reject": "- The submission is more like an empirical/engineering study. While the developed model and training recipe are greatly appreciated, the performance improvements are straightforward and expected, mainly attributed to the training data augmentation. In other words, there is limited novelty in the presented method. Both the loss functions are explored in prior works. LLM-based and text-conditioned augmentation are also well-explored. The only new thing in the paper is to combine the two. But this improves little performance as shown in Table A.6.1 and A.6.2.\n- As the diffusion model and LLM need to be well pre-trained with a variety of different datasets, I am not sure if the reported experiments are of fair comparison.\n- Visualization in Fig 4 is not convincing. For each demonstrated example, it only has a single salient object in the image and the text is of a template \"A photo of a [OBJ]\". Under the well-restricted setting, it is expected to see attention around the salient object. It is not conclusive to validate the effectiveness of cross-modal alignment.", "questions_to_authors": "- Could you please provide some statistics for the training cost? I wonder if the augmented images are generated offline to accelerate training. If both the LLM and diffusion models augment data on the fly, it seems very expensive in terms of a single GPU. \n- The augmentation greatly depends on the diffusion and LLM models. It may be helpful to report performance impact with different size of diffusion/LLM models. This may point out the hard requirements for those models.", "paper_topic_and_main_contributions": "This paper presents an interesting framework called APoLLo. The authors propose a method that includes both prompt tuning and adapter tuning in the scenario of multimodalities.  \nFollowing the previous works, the authors use the classic ITC and a consistency-based contrastive loss for training.\nTHe authors demonstrate good results on three selected downstream tasks.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The overall clarity and significance of the paper are strong as indicated by all reviewers. The proposed method of combining adaptors and prompt tuning is novel, and experimental results are significant.
Decision: Accept-Main