Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
Authors: Chen Huang
Skyler Seto
Samira Abnar
David Grangier
Navdeep Jaitly
Joshua M. Susskind
Venue: NeurIPS 2024 poster (2024)
Keywords: CLIP
Downstream Generalization
Prompt Learning
Natural Language Prompts
Primary area: machine_vision
TL;DR: Prompt learning that distills the textual knowledge from natural language prompts to improve the downstream generalization of CLIP.

Abstract: Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models) where the visual concepts are unseen or under-represented during pretraining. Prompt learning offers a parameter-efficient finetuning framework that can adapt CLIP to downstream tasks even when limited annotation data are available. In this paper, we improve prompt learning by distilling the textual knowledge from natural language prompts (either human- or LLM-generated) to provide rich priors for those under-represented concepts. We first obtain a prompt ``summary'' aligned to each input image via a learned prompt aggregator. Then we jointly train a prompt generator, optimized to produce a prompt embedding that stays close to the aggregated summary while minimizing task loss at the same time. We dub such prompt embedding as Aggregate-and-Adapted Prompt Embedding (AAPE). AAPE is shown to be able to generalize to different downstream data distributions and tasks, including vision-language understanding tasks (e.g., few-shot classification, VQA) and generation tasks (image captioning) where AAPE achieves competitive performance. We also show AAPE is particularly helpful to handle non-canonical and OOD examples. Furthermore, AAPE learning eliminates LLM-based inference cost as required by baselines, and scales better with data and LLM model size.

A Aggregate-and-Adapl Natural Language Prompts for Downstream Generalization of CLIP
Chen Huang, Skyler Seto, Samira Abnar, David Grangier, Navdeep Jaitly & Josh Susskind
Apple
{chen-huang, sseto, abnar, grangier njaitly, jaaaaayyyyjsssttnnoppppel..oo
Abstract
Large pretrained vision-language models like CLIP have shown promising general- ization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models) where the visual concepts are un- seen or under-tepresentte during pretraining. Prompt learning offers parameter- efficient finetuning framework that can adapt CLIP to downstream tasks even when limited annotation data are available. In this paper, we improve prompt learning by distilling the textual knowledge from natural language prompts (either human- or LLM- (eMe-eeeeeted) to provide rich priors for those under-represented concepts. We first obtain a prompt "summary" aligned to each input image via learned prompt aggregator. Then we jointly train a prompt generator, optimized produce prompt embedding that stays close to the aggregated summary while minimizing task loss the same time. We dub such prompt embedding as Aggregate-and and Adapted Prompt Embedding (AAPE). AAPE is shown to be able to generalize to different downstream data distributions and tasks, including vision-language un- derstanding tasks (e.g. few-shot classification, VQA) and generation tasks (image captioning) where AAPE achieves competitive performance. We also show AAPE is particularly helpful to handle non-canonical and OOD examples. Furthermore, AAPE learning eliminates LLM- inference cost as required by baselines, and scales better with data and LLM model size.
Introduction
Most existing vision-language tasks rely on large pretrained models like CLIP [0]. which are often adapted to downstream tasks using small amount of labeled data (as compared to the web-scale pretraining data). This is shown by many studies (667 68) to be likely to obtain poor generalization performance in special domains, such satellite imagery and fine-grained classification of models or flower species. Such overfitting behavior is a result of limited data for those tail class concepts in both pretraining and downstream tasks. The domain gap between pretraining and downstream data further compounds the generalization problem. For instance, CLIP may not see enough image-text pairs to identify different car models during pretraining. This makes downstream generalization to fine-grained car models difficult, especially in low-data scenarios.
In this paper, investigate using pure text-based knowledge to boost the downstream generalization of CLIP over different data distributions and tasks, with special focus on few-shot and OOD tasks.
Note similar ideas have been explored in recent works that leverage the implicit textual knowledge in Large Language Models (LLMs) to aid vision-language tasks. For example in [48] [56). GPT-2/3 [5] used generate image descriptions for the tasks of Visual Question Answering (VQA) and image captioning. While for HLIP-based image classification, [34] 39 (55) use GPT-3 to generate natural language attributes or captions for each class, and then classify images based on such information. Despite the success of these methods, they all suffer from large inference cost due to the use of LLM
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Prompt learning stage
Natural language prompts
(b) Inference stage
"The exterior of Compass SUV 2012 very sleek and sporty looking."
How can identily air) (Jeep Compass 2012)?
"Th extarior of a Jeep Compase BUV 2012 very sleek and modern."
Jeep Compase SUV 2012 boxy shape with sloping roofine."
Or Human-generater image captions
The Jeep Compass SUV 2012 is car that seats five people."
This mage is of red Jeep Compass SUV from 2012."
Image embeddding
Prompt aggregator
Prompt generator
Distilation (12 locs)
Adaptation
Task loss
AAPE
CLIP reward
Image classificttion Image-lo-teed retrieval
Prompt generator
VOA
Image embedding
Image captioning
Figure I: Aggregate-and-adee the textual knowledge in natural language prompts for downstream tasks. (a) For classification of object-centric images, we query GPT-3 to obtain list of prompts for each class, e.g., the car model of "Jeep Compass SUV 2012". Note how redundant the reference prompts can be (e.g., s. the first two), and how they can be irrelevant to the image (e.g.. the last prompt). Alternatvely, for complex tasks like VQA, we use human-generated image captions to depict multi- object images. For all tasks, we first learn to aggregate the reference prompts into image-aligned "summary" (prompt embedding) based on CLIP reward. Then a prompt generator is jointly trained to generate Aggregate-and-AAoe Adddted Prompt Embedding (AAPE), such that the distance between AAPE and the aggregated summary is minimized and the task loss is minimized too for adaptation purpose. (b) At test time, we only keep the prompt generator with the prompt aggregator discarded. Our AAPE is applicable to different lisoun-lagguage tasks with strong generalization performance.
at test time. More critically, the LLM--generater texts are not necessarily beneficial, since they might be noisy and irrelevant to the considered task (see one example in Fig. [Ta).
To address the two issues, we propose a new prompt learning method that is boosted by task-relevant language priors but does not incur any LLM cost test time. The high-level idea is to learn prompts via distillation of input-adapted textual knowledge, which is especially useful to recognize under- represented visual concepts. Specifically, for classification of object-centric images, we follow [39] to first query GPT-3 for set of natural language prompts that describe each class. While for more complex tasks like VQA, we use -eeerrete image captions that can depict multi-object images with object interactions in cluttered background. More importantly, for both cases, we learn to aggregate the collected reference prompts into single prompt embedding, which is optimized by the CLIP reward to have high similarity with input image. This allows us to obtain a condensed prompt embedding that is image-aligned, ruling out redundant and irrelevant information, e.g., ignoring the prompt elements about headlights for an image of rear facing car. Finally, we jointly train prompt generator conditioned on input image to generate prompt embedding with two objectives: 1) staying close to the aggregated embedding (iee, distillation from the condensed textual knowledge), 2) minimizing the task loss (i.e., downstream adaptation).
Fig. I Tillustrates our "aggregate-and-aa addapt method for prompt learning. Note prompt aggregation and distillation is only required for the learning stage. At test time, we will discard the aggregator and use the learned prompt generator as a standalone module. This leads to compute- -ffiiiii-fff
when compared to prior works [34 39 48 55 56) since we entirely eliminate the LLM- --iduuuueed inference cost. Our generated Aggregate-ande and- and-adapteddd Prompt Embedding (or AAPE) prove highly effective on various downstream rision-language tasks. We show AAPE is a new state-of-the-art for few-shot image classification on 11 datasets, under different OOD generalization settings. AAPE can also generalize zero-shot to tasks like image-to-text retrieval, image captioning and VQA. When finetuned on these tasks, AAPE achieves even better performance than SOTA vision- languulangguae models (e.g., MAGMA [II) whose entire image and text networks are fine-tuned at large cost.
To summarize, our main contributions are:
A new prompt learning method that distills the textual knowledge from human- or L.LM- generated natural language prompts to improve the downstream generalization of CLIP.
Our learned AAPE achieves compelling performance on various downstream vision- langual
tasks, including image-to-text retrieval, few-shot classification, image captioning and VQA.
We offer insightful findings that AAPE is especially helpful when there are under-represented concepts in few-shot and OOD settings or ambiguous visual cues in non-canonical image views. AAPE learning is also data-efficient and scales better than baselines with LLM model size.

---

2 Related Work
Vision-language models. Large-scale languae models achieve remarkable performance on variety of downstream tasks. One learning paradigm is based on generative encoder-decoder models, which allows a sequence-to-sequue -eeencee----euuecce learning format that can connect visual data to free- form language prompts 2 [1 33 Recent works like LiMBeR (35] show it is also possible for an LLM to operate on simple linear mappings of visual features. Another learning paradigm is based on contrastive learning with image-text pairs, which is popularized by CLIP [40] and numerous follow-ups [8, 24 29 41 [46 59 63 [6]. However, both categories of lainnnlanguage models (e.g., CLIP and generative PaLI (8) are found to struggle with special visual concepts or domains. In this paper, we focus on CLIP and improve its downstream generalization, especially under few-shot and OOD settings, via distillation of language priors Nevertheless, approach can be applied to other vision-language models, which we leave as future work.
Prompt learning is parameter-efficient yet effective framework finetune CLIP even in low-data settings. Most prompt learning methods learn text prompt vectors [67 68 in place of haad-written sentence prompts. Other methods show the possibility of learning prompts in the image space [99] or in both image and text spaces [20) [64]. To reduce overfitting to seen classes during prompt learning,
recent works focus on new class feature synthesis [52 651 improved optimization 225 45 and
regularization [1 strategies. More related to our approach are 7 58 69 that align the learned prompts with hand-written prompts, with the goal of not forgetting the text knowledge from human input. These methods can be interpreted as way of knowledge distillation from only short prompt templates. We will show the distillation from such prompt templates suboptimal when compared to distillation from natural language prompts using LLMs.
Leveraging language in vision tasks. There is a long line of works on leveraging language to aid vision or multimodal tasks. One family of methods rely on external natural language datasets to retrieve text knowledge of image categories. For example, 16 44] show improvements on ImageNet classification using the class descriptions retrieved from WordNet [36] and ImageNet- Wiki (6] More
recent works use LLMs to generate text for downstream tasks. GPT-3 is used in 48 56 to help
with the VQA and image captioning tasks. GPT-3 is also used in 34 39
55
generate class-wise
attributes or captions for Cl.IP-based classification. Unfortunately, all these prior works suffer from the noisy text that may be task-irreeevant. In this paper, we learn to adapt L.L.-geeerated text the target task, but without incurring any LLM-induced inference cost.
3 Method
Our Tiggregate-and-adde method for prompt learning consists of three key components: 1) gen- erating natural language prompts per image class, 2) learning an aggregated prompt embedding that aligns with input image and 3) learning to generate Aggregate-and-- and-Ad Prompt Embedding (AAPE) for downstream tasks. In the following, we provide the details for each component. Note method is based on CLIP [40, but it can be easily applied to other CLIP-like vision-lane language models.
3.1 Generating Natural Language Prompts
Prompt engineering. For CLIP-based image classification, the standard approach requires of hand-written prompts, e.g., "a { in video game" and "a dark photo of a ( which are completed with the class name. However, this is costly because one needs hand-construct different set of prompt templates for each dataset (e.g., 80 for ImageNet in [40)), and that requires excessive prior knowledge about the target domain. Moreover, such prompt templates lack the descriptive details for discriminating fine- siee-graindd classes.
LLM L-g-enerrteed prompts. For image classification, we make use of the rich knowledge in LLMs to generate natural language prompts for a given class. One benefit of using LLMs is the ability to generate an arbitrary number of prompts, without relying on any domain knowledge. In particular, we follow the CuPL. method 39) and query GPT-3 [5] for prompt set in a scalable way. Specifically, GPT-3 queried with a few ..MM-ppoppss such as "Describe what a(n) () looks like" and "How can you identify a(n) ? Then for each LLM-prompt, GPT-3 generates 10 reference prompts using
3

---

"A goldtish long, gold body with baok
"Tiddton a chogay medium ength that etthrr black, while. boown gray, or cream."
ImageNet
"A goldiian has shing goldan body with fine that hwip swim.
"The Titwtan Terrer medium awd, theireed stowggy Tenter "This image is of Tioetan terver standing in foldd of tall grass."
belly" "A goldiion is small, orange fss with long tall" 'A goldifish awemmning in faltboos"
golatich long body
identiited long, shopgy ooot"
"This Tibwlen Tierier, breed of that ratiow Tibel:
"a bottroom with bath tub near windows"
"A groels and mebos cleekking eech otter out.."
COCO
"A bathroom scene is shown with a tub and courter." bathroom with buttruub blue oceen."
tooknn
grotte zebra through African plain.
feeld water
aratto and zebro
"A grails looks ciown all valina on the fiek"" the wild
"A bathrocm with white tue with logo."
bathroom complete
alyle
rebra
attractive
Flickr30k
coins for strect porformen ers.
people ride mourtain through junglo enirooment?
"Twe woreen enjoyy the entertiimment of afreet performers."
"Men, aumouneee by nelure, are riding mourtein tikee..
"There are mountai
in forest terrain
orctond be statunos womeen on." "Twe yourng womee stop and watch streel performmna. artiste."
People riding bikes on "A group of paoople is bike rading the woods.
forest"
"Pedeetrians merrec
Figure 2: LLM-generated image prompts for ImageNet categories, and the hand-constructed image captions on COCO and Flickr30k datasets. Note ImageNet mainly contains object-centric images with relatively clean background, and the LLM-generated image prompts can describe distinct characteristies of the given classes. While COCO and Flickr30k contain multi-object images with cluttered background, and the hand- construcedd captions can represent varying object relations.
high temperature of 0.99 for diversity. Fig. aa) shows some example prompts for particular model "Jeep Compass SUV 2012". Note how the reference prompts specify the car's discriminating characteristics in its sleek exterior. For the 11 classification datasets considered in our work, we follow the full generation setting in [39]: for each dataset, we use a different set of LLM-prompts (between 2 9), resulting in 20-90 reference prompts generated for each class.
Human-generated image captions. illustrated Fig. 2 the LLM--gnnrrted class-wise prompts are mainly suited for the classification task, where there are often object-centric images with clean background. For more complex vision-language tasks like VQA, deeper understanding required for multi-object scenes with varying object interactions and cluttered background. To capture the textual knowledge for describing multi-object images, we use their image captions available from image-text datasets as a source of natural language prompts. Here we use COCO dataset [31] that consists of 5 humannaamann annotated captions per image. will be shown that our prompt embedding learned on COCO suffices to generalize three difficult vision- language tasks (image-to-ter) retrieval, image captioning and VQA) with varying data distributions.
3.2 Input-Adapted Prompt Aggregator
The LLM-ggnerated prompts have one notable issue: they are not necessarily good representation of input image. For example in Fig. M [a), it inaccurate describe the input image of silver Jeep SUV as red one in the last prompt, whereas other prompts are more relevant. Obviously, it would be detrimental to directly use the noisy prompts to supervise the following learning stage. Another issue with both the LLM- and guman--ennrated prompts is that they are highly redundant with repeated information. To find a better supervisory signal, we propose to first aggregate the reference prompts into an image-aligned, condensed "summary". Such prompt summary is expected to have filtered noise as well as reduced redundancy.
Given n generated prompts, we first use the text encoder of CLIP to obtain the prompt embeddings P [pi. P2, ... P... Then we learn an adaptive prompt aggregator to condense P into m (& n) embeddings of the same size. Ideally, the aggregation should be invariant to permutations of P, and scales as On + m). Here we set m 1 for efficiency concerns. The aggregated result, a single prompt embedding, is denoted as p One simple aggregator that has the properties of permutation invariance and high efficiency is based on just averaging P into P. Simple averaging is widely used in However, the mean p would still be compromised by the irrelevant information P. Here we introduce an attention-basee aggregator which allow us to align the aggregated pd with input image. At the same time, our prompt aggregator remains efficient and permutation invariant.
prior works 34 39

---

(b)
Input-adapted prompt aggregator
"a photo of (close))
Text Encoder
W,
Aggregated embedding
Updated p
AAPE
Prompt aggregator
Leour Eq. (2)
Projection
P
Reference prompts
CLIP reward
Layer (
Prompt
generator
Image
P
Ps
Encoder
Image
Luzk
embedding
Meen
Reference prompt embeddiings
Eq. (4)
Figure 3: (a) Input-adapted prompt aggregator which aggregates the embeddings of reference prompts P into an image-aligned, condensed prompt embedding pa based on CLIP reward. (b) Instantiation of our prompt learning approach for image classification. The CLIP model is kept frozen.
Fig. 3(a) shows the architecture of our input-adapted prompt aggregator based on just one attention layer. The attention layer takes as input the reference prompts P and a learnable prompt embedding (initialized as p). Then all the embeddings are updated as follows:
[p". Pd] AttentionLayer ((p,
(1)
where pa is the desired prompt aggregation. The attention layer consists of standard multi-head cross-attention and feed-forward networks together with LayerNorm [3].
To make pd semantically related the input image (with embedding æ), we optimize our prompt aggregator using the CLIP reward 177 in form of CLIP-S(x, S(x, p*) max (x.s((, p²), 0). The CLIP reward allows pd to selectively blend image-related prompts through the attention mechanism. Fig. 7 in Appendix B.1 confirms that redundant or irrelevant reference prompts tend to have low attention scores, hence they are suppressed during prompt aggregation.
3.3 Learning AAPE
The per-image prompt aggregation pd offers useful textual knowledge to supervise the following prompt learning stage. In this section, we elaborate how to improve prompt learning by distilling the aggregated text knowledge from p" Note CLIP is kept frozen during prompt learning.
As key innovation of this paper, we propose to train prompt generator h that directly generates the prompt embedding h hxx) conditioned on image features æ. We parameterize h as a lightweight network with two fully connected layers and ReLU nonlinearity. h is trained using distillation loss Laisin for knowledge distillation from p", as well as a task loss Lask for downstream adaptation. We call such learned (ar) Aggregate-and-Adapte Prompt Embedding (AAPE). AAPE can also be viewed as an image captioning embedding in the latent space, since useful text knowledge is distilled in AAPE. In the following, detail the training losses.
Distillation loss is simply defined as the Euclidean distance between h(x)) and p":
Caistil
[[((x)) -1
Task loss image classification. Besides distilling the textual knowledge from pa, h((=)) should adapt the downstream task too. Here we start with the instantiation of adapting h(x) to the most studied task of image classification.
Note existing prompt learners for classification (e.g., 67 68
learn individual word tokens {vr)-
prompt, and then combine the learned tokens with class name embeddings Cg[[I....C to obtain the full prompt. By contrast, directly generate full prompt embedding h(x))) without token-wise prediction. For classification, we simply combine h(z) and the embedding of a prompt template "a photo of (class)" act as the classifier weights (to be matched image features σ))
Fig. 3 b) shows the overall prompt learning framework. For the prompt template filled with the i-th class name, we use the text encoder of CLIP to obtain the template embedding w; E IR Next, concatenate w; and our h(x)= E Rd followed by projection g. giving wx = g ([w? h(x) [")) Note wo (2x) does not involve any prompt engineering effort. We rely on un to mainly encode the class name, while l(x) enriches that with input-adapted class descriptions in the latent space.
5

---

We parameterize g: R² + R by one fully connected layer with ReLU nonlinearity. The nonlinearity important since it ensures w; (x) is not trivially equivalent to the linear combination of w, and h(x). which will ignore h(x) if we match the linear combination to z for classification. The classification probability given as:
exp (cos (x. w. w,(x))) /T)
( = c Ei=1 exp (cos (z. w. (x)))) /T)
where C' is the total number of classes, T and cos(.,)) denote the temperature and cosine similarity. Appendix B.2 (Table compares with an h(xz)- baseline for classification, without combining w; or using projection g. Results show our default classification framework achieves solid gains over the onnl baseline.
Finally, we arrive at the overall loss function to train the prompt generator / together with projection g for image classification:
L = ddsiuun + Lusk, where
Lask
log p (y x),
(4)
and À = 5 is weighting parameter. Table in Appendix C provides the sensitivity analysis for X, which shows performance is quite robust to the À value a wide range.
Note during testing we only use the prompt generator h, without querying LLM or using the prompt aggregator anymore. This removes the LLM-induced inference cost as required in [34]]]]]]]]]]] 39 48 55 56)
shifting such cost into our learning stage.
Task loss - beyond classification. Will the textual knowledge in h(x) or AAPE benefit other vision- vision-language tasks? We consider three tasks beyond classification: image-to-text retrieval, image captioning & VQA. Note these tasks involve multi-objec images as mentioned in Section 3.] Hence we use COCO image captions that contain textual descriptions of object relations. For all the three tasks, we train both the prompt aggregator and prompt generator h on 5 COCO captions per image. The same Laisin in Eq. (2) is used, while Lusk the CLIP loss.
For image-text retrieval, we simply use AAPE as the query and evaluate its zero-shot text retrieval performance on Flickr30k dataset [60]. We also evaluate the finetuning performance on Flickr30k when the prompt generator h is finetuned using Edistlll and the corresponding retrieval loss Lrask-
For image captioning and VQA tasks, conjecture that the textual knowledge encoded in AAPE will be especially useful when the visual cues are confusing or missing. To test this hypothesis, use the POCO-trainer AAPE in straightforward manner. Concretely, we follow the LiMBeR baseline in [SS] which linearly transforms the CLIP image representation into a sequence of prompt "prefix"
embeddings that LLM can process. Then AAPE is appended to the prompt sequence to offer rich language priors. We similarly evaluate performance of the zero-shot or finetuned prompt generator i on downstream datasets. For finetuning, h is optimized using dissslll and the corresponding task loss Lusk (captioning or VQA).
4 Experimental Setup and Datasets
4.1 Few-shot Image Classification
Datasets. We use Il datasets: ImageNet [0), Caltech101 [12, OxfordPets [38], StanfordCars 23 Flowers 1 02 [37], Food101 4. FGVC-Aircraft (32). SUN397 [54] UCF101 [7). DTD 99 and EuroSAT 14. These datasets cover wide range of generic objects and scenes, fine-grained object classes, well special domains with textural and satellite images. The various visual concepts in these datasets are perfect to test whether and when the textual knowledge in LLM will help. We further evaluate domain generalization on ImageNetV2 42), ImageNet-Sketee (5I), ImageNet-A [16] [5]. which have different types of domain shift from ImageNet.
and IetgeNet-R
Implementation. We follow the prompt learning details in 677 including the CLIP vision backbone (ViT-B/16), learning rate schedule and the number of epochs for each dataset. Appendix (Table provides a detailed analysis of the compute cost measured on Nvidia V100 GPU, where prompt leamners are evaluated for fair efficiency comparisons.

---

Table 1: Image-to-Text Retrieval: zero-shot and finetuned results (Recall K) Flickr30k dataset. Note our AAPE is learned on the COCO dataset.
COCO
Flickr30k
R
@@5
R@ 10
R@1
R@5
R@ 10
Unicoder-V VL
26
62.3
87.1
92.8
86.2
96.3
99.0
Train or Finelune
Oscar 28
73.5
92.2
96.0
ERNIE-ViL
61
88.7
98.0
99.2
AAPE
10.1
20.5
97.4
94.2
99.3
99.7
Zero-shot
CLIP 40
58.4
81.5
88.1
88.0
98.7
99.4
SigLIP 66
65.4
85.1
91.5
98.1
99.4
Llip 124
68.1
87.6
92.5
93.2
99.0
99.4
BLIP-2 27
96.9
100.0
100.0
AAPE
90.8+0.2
98.6.0.1
99.4.101
Evaluations. We follow the few-shot evaluation protocol in [4). using 1, 2, 4, 8 and 16 shots per class for training (default 16), and the full testset for evaluation. Two OOD generalization settings are considered in [67]. 1) Generalization from base the base class split but testing on both base and new class splits. This helps evaluate the ID and OOD performance under class-incts maas-inncemental domain shift. We follow [53] also measure the harmonic mean of base and new class accuracies to quantify the ID and OOD performance trade-off. 2) Domain generalization where trains ImageNet (with 16 shots) and evaluates four ImageNet variants. For all experiments, we report results as an average over three random seeds.
classes within
dataset, i.e., training
4.2 Vision-Language Understanding and Generation Tasks
we perform prompt learning on COCO dataset [3] before evaluation on three vision-language lasks. For the task of image-to-text retrieval, we use the same CLIP vision backbone VIT-L/14 (Recall @)) K) on Flickr30k
As mentioned in Section 3.3
24 40
for fair comparisons. We show both zero-shot and finetuned results
660
For captioning and VQA tasks, follow LiMBeR B5 to the same language model and CLIP
vision backbone (RN50x 16). We evaluate on image captioning datasets COCO and NoCaps [ Zero- shot and finetuned results reported in terms of CIDEr-D [50], CLIPScore, and Ref-CLIPSccre (17). For VQA, we prompt the model with the "[image] Q: [q] A:" format. The generation is truncated the length of the longest ground truth answer. For evaluation, we use the VQA2 dataset [13 and follow the few-shot setting in [[]] to report accuracy metric for every K-shots.
Appendix C (Table shows the high efficiency with the straightforward
of AAPE for tasks
beyond classification. When compared to the SOTA fully fine-tuned model MAGMA 11 AAPE is about 2.8/1.2 times faster for training/inference Nvidia A A100 GPU.
Results
5.1
Image-to- Text Retrieval
We use this task to verify if AAPE can act as meaningful image captioning embedding, which is learned distill image-aligned text knowledge from available image captions. We do not aim push for tate--of-the-and performance for the retrieval task. Table that we can indeed achieve strong training and finetuning performance on COCO and Flickr30k datasets, respectively. This indicates our prompt learning method is competent with producing high-quality text or captioning embedding that can successfully fulfill the task at hand. It is also worth noting that our AAPE learned on COCO can perform zero-shot retrieval on Flickr30k, obtaining competitive results with SOTA zero-shot models (e.g., CLIP and SigLIP). This further demonstrates the good generalization capability of AAPE over different data distributions.

---

Table 2: Few-shot classification in the base-to-new class generalization setting. OGEN denotes the OGEN+PromptSRC variant. Our AAPE follows CuPL to query an LLM to obtain natural language prompts, but further learns from those prompts. H: Harmonic mean of base and new class accuracies.
Prompt learning without language priors
Human-generated prompts
L.M--based
CoOp
CoCoOp
MaPL.e
CLiPoood
PromptSRC
OGEN
ProGrad KgCoOp
LASP-V
CuPL.
AAPE
Avg across
Base
82.69
80.47
82.28
83.90
84.26
84.17
82.48
80.73
83.18
74.31
84.72
New 63.22
71.69
75.14
74.50
76.10
76,86
70.75
73.60
76.11
75.25
17,54
11 datasets
H
71.66
75.83
78.55
78.90
79.97
80.34
76.16
77.00
79.48
74.78
80.97
Base 76.47
75.98
76.66
77.50
77.60
77.50
77.02
75.83
76.25
75.05
78.10aa11
ImageNet
New 67.88
70.43
70.54
70.30
70.73
70.97
66.66
69.96
71.17
68.43
41.98
H
71.92
73.10
73.47
73.70
74.01
74.09
71.46
72.78
73.62
71.59
74.92:012
Base 98.00 08.00
97.96
97.74
98.70
98.10
98.32
98.02
97.72
98.17
98.24
1am
Caltech101
New 89.81
93.81
94,36
94.60
94.03
94.76
93.89
94,39
94,33
94.34
94.79
93.73
95.84
96.02
96.60
96.02
96.50
95.91
96.03
96.21
96.25
96.53
Base 93.67
95.20
95.43
95.70
95.33
95.96
95.07
94.65
95.73
95.30
96.89
OxfordPets
New 95.29
97.69
97.76
96.40
97.30
97.48
97.63
97.76
97.87
97.74
98.02
H
94.47
96.43
96.58
96.00
96.30
96.71
96.33
96.18
96.79
96.50
97.45
Stanford Cars
Base 78.12
70.49
72.94
78.60
78.27
77.59
77.68
71.76
75.23
68.88
77.5 7
New
60.40
73.59
74.00
73.50
74.97
75. 17
68.63
75.04
71.77
75.09
77.37 20.56
68.13
72.01
73.47
75.90
76.58
76.38
72.88
73.36
73.46
71.85
77.444042
Base 97
94.87
95.92
93.50
98.07
34
95.54
95.00
97.17
77.79
Flowers 102
New 59.67
71.75
72.46
74.50
76.50
77.67
71.87
74.73
73.53
78.10
78.75
H
74.06
81.71
82,56
82,90
85.95
86.39
82.03
83.65
83.71
77.94
87.25
Base 88.33
90.70
90.71
90.70
90.67
90.69
90.37
90.50
91.20
90.56
91.82
Food101
New 82.26
91.29
92.05
91.70
91.53
91.68
89.59
91.70
91.90
91.86
92.66+a11
85.19
90.99
91.38
91.20
91.10
91.19
89.98
91.09
91.54
91.21
92.244:000
Base 40.44
33.41
37.44
43.30
42.73
41.26
40.54
36.21
38.05
33,29
41,46:012
FGVC Aireraft
New 22.30
23.71
35.61
37.20
37.87
40.26
27.57
33,55
33.20
37.60
40.37
H
28.75
27.74
36.50
40.00
40.15
40.75
32.82
34.83
35.46
35.31
40.91
Base 80.60
79.74
80.82
81.00
82.67
82.57
81.26
80.29
80.70
73.39
82193
SUN397
New 65.89
76.86
78.70
79.30
78.47
78.83
74.17
76.53
79.30
75.69
79287
H
72.51
78.27
79.75
80.20
80.52
80.65
77.55
78.36
80.00
74.52
81.37.018
Base 79.44
77.01
80.36
80.80
83.37
83.75
77.35
77.55
81.10
62.45
83,97
DTD
New 41.18
56.00
59.18
58.60
62.97
62.54
52.35
54,99
62.57
60.31
63.64
54.24
64.85
68.16
67.90
71.75
71.60
62.45
64.35
70.64
61.36
72.41 :039
Base 92.19
87.49
94.07
97.50
92.90
93.40
90.11
85.64
95 00
66.38
95.40ga33
EuroSAT
New 54.74
60.04
73.23
64.10
73.90
76.74
60.89
64.34
83.37
69.97
76.30
H
68.69
71.21
82,35
77.30
82,32
84,25
72.67
73.48
88.86
67.60
84,79
Base 84.69
82,33
83.00
85.70
87.10
87.44
84.33
82.89
85.53
77.13
87.69
UCFI01
New 56.05
73.45
78.66
79.30
78.
79.28
74.94
76.67
78.22
78.58
79.21
67.46
77.64
80.77
82.40
82.74
83.16
79.35
79.65
81.70
77.85
83.23
5.2 Few-shot Image Classification
Base-to-new class generalization. Table 2 compares AAPE with two categories of prompt learning methods: 1) CoOp [67) CoCoOp 68,, MaPLe 20 CLIPood 45), PromptSRC [21] and OGEN 65 These methods learn prompt vectors without using any text-based knowledge, but heavily rely on advanced optimization and regularization strategies to improve generalization. Our AAPE, on the other hand, outperforms by distilling the textual knowledge from LLMs. Notably, on average (across 11 datasets), AAPE achieves better classification accuracies than the previous SÔTA approach OGEN for both base and new classes, setting a new SOTA mean accuracy 80.97% (vs. 80.34%). 2) ProGrad [69], KgCoOp (58) and LASP-V [7]. These methods choose to align the learned prompts with hand-written ones like "a dark photo of {class)", hence distilling knowledge from only basic, non-descriptive templates. This proves less effective than our -dd-iviieed natural language priors.
Recall our prompt learning method is built on top of the CuPL approach 39 to obtain the knowledge of GPT-3. Here we show both the LLM knowledge and our learning algorithm (that adaptively distills the knowledge) are indispensable. We first compare with the CuPL baseline that leverages LLM knowledge, but without any learning. Specifically, CuPL averages the L.M--enerated prompts to perform zero-shot classification. This is contrasted with our method that learns to aggregate the noisy prompts into AAPE which is then adapted for classification. Table[]shows such "aggregate-and-adda"" learning method leads to significant gains over the learning-free ffree CuPL, especially for base classes.
8

---

AAPE AAPE w/o
on base classes
AAPE vs. AAPE w/o
new classes
19
19
9.45
gat
7.47
7.92
2.75
5.48
2011
0:97
2.75
Iragelee
Conaect
Fimeara to
Feetfi
roVCARcaR
SUNDR
0r8
Euroter
UCP107
Imageeiet
Cantech1
OetorPaee
Fower ne
Foodfot
ravsAwash
oro
Resbbrr
uon
Figure 4: Quantifying the role of LLM knowledge (distilled with Ldisti)) in prompt learning. distilll consistently improves the base and new class accuracies 11 classification datasets.
ImageNet (easy)
ImageNet (hard)
Distractor label: church GT label: alter
GT label: golf
Prompt template
W
Score: 0.94
Prompt template
W
Score: 0.27
"a photo of (galf bally"
photo (atar))
AAPE
Score: 0.97
AAPE
Score: 0.87
Reference prompts
Reference prompts
"A golf ball amall, white with dimplee it.' "A golf ball a small, round, white sphere."
"An altar a raised platform table used for religious ceremonies." "The altar the church raligious rituats."
DTD (OOD)
EuroSAT (OOD)
Distractor lined GT label: grooved
Prompt template
w,
Score: 0.41
Distractor labelt residential
Prompt template
Score: 0.48
"a photo of (grooved) texture"
industrial
photo [industrial area]"
AAPE
A0X
0.93
ААPE
Score: 0.89
Reference prompts
Reference prompts
*A grooved surface is surface that has ridges furrows cut into a.' "Grooves are usually long narrow, trench."
"An industrial aroa is typicaly a large, flat arna few large bulldings." "industrial
typically identified large buildinge and parking lots."
Figure 5: AAPE helps disambiguate the classification task. To highlight the textual knowledge encoded in AAPE, we show some reference prompts generated by GPT-3. For both the prompt template and AAPE (before concatenation and projection), with the image. Note the similarity score can be small when using basic prompt template to match the "altar" class instance on ImageNet. Indeed, in this non-canonical image view, the altar small and the whole scene can be classified as the easily confused class of "church".. Whereas AAPE is able eliminate confusion by providing additional cues like altar "is raised table" often the location of "church". This results in increased image-text similarity. Similarly, the textual cues from AAPE helpful for the OOD examples in special domains of DTD and EuroSAT.
measure their Cosine similarity score
Next we compare with variant of our approach, with only task loss Lusk but no Laisin to distill LLM knowledge. This variant allows decoupling the contribution of LLM knowledge for prompt learning under a fair setting. Fig. 4 shows that using Ldisin leads to consistent gains for both seen and unseen classes from all the considered datasets. The distilled textual knowledge makes an especially large impact for those fine-graines actions (UCF101) and visual classes (StanfordCars, Flowers 102 and FGVCAireraft) which can be under-represented during both CLIP pretraining and prompt learning. Large gains are also observed for the special domains of textures (DTD) and satellite images (EuroSAT) with large distribution shift.
We further show how our distilled AAPE can disambiguate the classification task. Fig. shows our AAPE augments the basic prompt template with descriptive details for each image, exemplified by the reference prompts. Such input-specific details are often helpful for non-canonical views (e.g., hard cases on ImageNet) and OOD examples (e.g., on DTD and FuroSAT), where the visual cues are either ambiguous barely visible (hence low similarity between the image and basic template). Eventually, we use a projection network to blend textual information from the template and AAPE, resulting in increased image-text similarity.
More comparisons. Table Appendix D includes domain generalization results. We see that AAPE is robust to different types of domain shift, outperforming prior works on 4 ImageNet variants. Table and Appendix F further show the advantage of AAPE over two recent prompt learning methods ProText [22] and ArGue-N 49

---

AAPE
AAPE who Love
CuOcOp
OUPL
CLIP (aro-hoo-
(8)
82
(
H lavg disessett @
Hang over 11 delaseet)
H Hang over 11 danceets)
74
78
76
78
75
73
73
70
0.0004
(deln Bf)
(pwcn
$ shots per class
# prompts per promppt
LLM size (OpenA) API cost, log scald)
Figure 6: AAPE scales better with data (a-b) and LLM size (C) than alternatives. Experiments conducted under the base-to-new generalization setting for few-shot classification. We measure the Harmonic mean (H) of base and new class accuracies. To adjust the total number of reference prompts per class supervise AAPE learning, vary the number of prompts generated by each LLM-prompp template. Four models of GPT-3 are considered: Ada, Babbage, Curie and Davinci.
Table 3: Image captioning and VQA performance. Note our AAPE is learned on COCO dataset, and we show both its zero-shot and finetuned results on the two tasks with different testing datasets.
Captioning
VQA2 K-shots
COCO
NoCaps (CIDEr-D
NoCaps (All)
2
CIDFrDD
CLJP-S
Ref-S
Out
Near
All
CLIPS
Ref-S
MAGMA 1 LiMBeR [51
47.5
75.3
79.6
30.4
43.4
36.7
38.7
74.3
78.7
24.6
39.3
40.6
41.5
54,9
76.2
80.4
34.3
48.4
41.6
43,9
74.7
79.4
33.3
39.9
40.8
40.3
LiMBeR+AAPE (train/linetune)
$7.8
80.8
83.6
42.1
49.8
44.2
47.3
77.6
81.7
36.5
42.7
44.2
45.9
LAPE (sero-shat)
36.1
48.8
42.9
45.1
76.3
80.3
34.9
41.0
42.3
43.1
Ablation studies. 6(.. shows that AAPE learning is data-efficient. We see AAPE consistently outperforms two prompt learners that do not benefit from LLM's 's text knowledge, i.e. AAPE w/o Laisiil and similar baseline CoCoOp 68 Encouragingly. using shot for AAPE is already far better than using 16 shots for the compared baselines. AAPE with varying shots is also consistently better than the LLM-based but learning-free approach CuPL 39). Fig. (o() further shows that AAPE scales better with the number of used prompts than CuPL. Note when we use only 1 prompt generated by each LLM- -LM-pptppt, CuPL is even worse than the CLIP baseline. Whereas AAPE performs much better by distilling task-related information from the limited number of prompts. Finally. Fig. (c) shows the benefits of AAPE over CuPL. in terms of the scaling performance with LLM model size.
5.3 Image Captioning & VQA
3 shows that AAPE can adapt to other vision-language tasks. Note AAPE is trained on COCO captions that deseribe complex scenes other than object-centric images. We once again find the benefits of distilling the text knowledge from image captions into AAPE. We observe consistent gains over the LiMBeR baseline, using either trained or finetuned prompt generator (finetuned on NoCaps captioning and VQA2 tasks). More importantly, AAPE shows great generalization capability on both tasks. Its zero-shot performance consistently better than that of LiMBeR and MAGMA, the latter of which finetunes both image and text networks. Fig. 9 in Appendix [ help on the captioning task, especially when the visual cues are ambiguous.
eexemplifiss how AAPE can
6 Conclusion
In this paper, we show the distillation of text-based knowledge into CLIP improves its downstream generalization. We propose new prompt learning method where a prompt embedding AAPE is distilled from human- or I.LM-generated natural language prompts. A prompt generator is trained to predict AAPE, which is shown to generalize to various vision-language tasks. We further demonstrate the benefits of AAPE for handling non-canonical examples as well as few-shot and OOD settings.
Limitations and future work. Given sufficiently large set of image prompts, it is preferable to aggregate them into more than one prompt embeddings to model text diversity. However, learning to predict such an embedding set hard on data-deficient tasks, since will not only increase the computation cost but also incur performance degradation. For future work we hope to address this limitation by scaling up data to learn a diversified universal prompt generator. Another plan is to go beyond CLIP and apply AAPE learning more vision-language models (contrastive generative).
10

---

References
[1] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. NoCaps: novel object captioning at scale. In ICCV, 2019.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. In NewrIPS, 2022.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint 1607006000 2016.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 1ond-10 discriminative components with random forests. In ECCV, 2014.
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[6] Sebastian Bujwid and Josephine Sullivan. Large-scale zero-shot image classification from rich and diverse textual descriptions. In LANTERN, 2021,
[7] Adrian Bulat and Georgios Tzimiropoulos, LASP: Text-to-text optimization for language-aware soft prompting of vision & language models. In CVPR, 2023.
[8] Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sehas- tian Alexander Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scaled multilingual language-image model. In ICLR, 2023.
[9] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[11] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA multimodal augmentation of generative models through adapter-based finetuming, arXiv preprint arXiv:21/2.05253, 2022.
[12] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. CVPR workshop, 2004.
[13] Yash Goyal, Tejas Khot, Douglas Summers-Ste Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In CVPR, 2017.
[14] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 2019.
[15] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021.
[16) Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardr, and Dawn Song. Natural adversarial examples. In CVPR, 2021.

---

[17] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In EMNLP, 2021.
[18] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[19] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[20] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz. Khan. MaPLe: Multi-modal prompt learning. In CVPR, 2023.
[21] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, 2023.
[22] Muhammad Uzair khattak, Muhammad Ferjad, Naseer Muzzamal, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision-language models. arXiv preprint arXivv2401.024118 2024.
[23] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
[24] Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mahmoud Assran, Andrew Gordon Wildon, Aaron Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pretraining. arXiv preprint 2xXi:22405.007400 2024.
[25] Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J. Kim. Read-only prompt optimization for vision-language few-shot learning. In ICCV, 2023.
[26] Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-VL. VL: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint (00..vv9900000666 2019.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[28] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre- pre-training for lasion-language tasks. In ECCV, 2020.
[29] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm. In ICLR, 2022.
[30] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022.
[31] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
[32] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
[33] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier Biard, Sam Dodge, Philipp Dufter, Bowen Zhang, Dhruti Shah, Xianzhi Du, Futang Peng, Haotian Zhang, Floris Weers, Anton Belyi, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu He, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. MM1: Methods, analysis & insights from Multimodal L.LM pre-training. arXiv preprint arXiv:2403.096.1 2024.
12

---

[34] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
[35] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In ICLR, 2023.
[36] George A. Miller. Wordnet: lexical database for english. Commun. ACM, 1995.
[37] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008
[38] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
[39] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, 2023.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[41] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang. Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. DenseCLIP: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
[42] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
[43] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[44] Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang. Pengchuan Zhang, Anna Rohrbach, Zhe Gan, Lijuan Wang, Lu Yuan, et al. K-lite: Learning transferable visual models with external knowledge. In NeurlPS, 2022.
[45] Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. CLIPood: Generalizing clip dustif-diitributiions. In ICML, 2023.
[46] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational languag and vision alignment model. In CVPR, 2022.
[47] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXivv1212.04000 2012.
[48] Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang. Lingpeng Kong, and Nigel Collier. Language models can see: Plugging visual controls in text generation. arXiv preprint 2rXiv:2205.026555 2022.
[49] Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. ArGue: Auided Prompt Tuning for Vision-Language Models. In CVPR, 2024.
[50] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In CVPR, 2015.
[51] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In NeurIPS, 2019.
[52] Zhengbo Wang, Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. In ICCV, 2023.
13

---

[53] Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In CVPR, 2017.
[54] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[55] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Gallison-Burche and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, 2023.
[56]
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang, An empirical study of GPT-3 for few-shot knowledge-basee VQA. In AAAI, 2022.
[57] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung- Lin, Zicheng Liu, and Lijuan Wang. The dawn of LMMs: Preliminary explorations with GPT--V(ision). arXiv preprint arXiv:2309. /7421, 2023.
[58] Hantao Yao. Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In CVPR, 2023.
[59] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. In ICLR, 2022.
[60] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014.
[61] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE- ViL.: Knowledge enhanced vision-language representations through scene graph. In AAAI, 2021.
[62] Tao Yu, Zhihe Lu, Xin Jin, Zhibo Chen, and Xinchao Wang. Task residual for tuning vision- language models. In CVPR, 2023.
[63] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.1144 2021.
[64] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022.
[65] Yuhang Zang, Hanlin Goh, Joshua M. Susskind, and Chen Huang. Overcoming the pitfalls of vision-language model finetuning for OOD generalization. In ICLR, 2024.
[66] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
[67] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[68] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-languuae models. In CVPR, 2022.
[69] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. ICCV, 2023.
14

---

A Alternative Methods of Generating Natural Language Prompts
Text retrieval for object-centrie image classification. is possible to the class name to retrieve class descriptions from WordNet [36] or image-text datasets, just like in previous works [6 44). Unfortunately, single dataset, including the large-scale LAION-5B 43 contains arbitrary classes specified in downstream tasks (e.g., fine-grained flower species). In other words, the retrieval-basee approach can limit the possible classes that can be recognized. Furthermore, the image captions retrieved from image-text datasets can be noisy and irrelevant to the target class. One typical example that the retrieved image captions instances. Instead, the captions can be depicting their relations with other objects in a multi-object image. Therefore, retrieved image captions may be suited for object-centric classification. This is evidenced by Pratt et al. [39 who explored similar retrieval ideas for ImageNet classification, where each class is guaranteed have class descriptions from WordNet [36] Wikipedia articles (ImageNet- Wikii [6] ). The retrieved captions prove less effective than LLM generated customized prompts in terms ImageNet Top-1 accuracy.
really focused the descriptive details of the target class
Large Multimodal Models (LMMs) for object-centric classification and other vision-language tasks, One alternative way of acquiring natural language prompts is to query LMMs, e.g., GPT- 4V (57 For the task of object-centric classification, it is easy to imagine that LMMs can be more competent than pure text-based LLMs in generating helpful prompts, since LMMs have both vision and language supervision in their pretraining. For the same reason, vision-language tasks like VQA are likely to benefit more from LMM-generatee image prompts than aaman-annotated image captions. However, our goal here is to study the role of text-only knowledge for downstream generalization. Hence LMMs with additional vision supervision fall outside of the scope of our study.
B
More Ablations and Analyses
B.1
Input-Adapted Prompt Aggregator
We rely on the input-adapted prompt aggregator to provide good textual supervision for prompt learning. The prompt aggregator learned to aggregate all the reference prompts P into an image- aligned prompt embedding pd with reduced noise and redundancy.
Fig. 7 confirms that redundant and noisy (irrelevant) reference prompts are often suppressed with low attention scores during prompt aggregation.
Next, we compare with three alternative methods for prompt aggregation:
Two baselines are learning-fin free, obtaining p by either random sampling from P or simple averaging (i.e., p).
We further compare with learning method using a different architecture other than the default attention network. Specifically, we start with the mean p and learn to transform it via simple MLP layers such that the transformed embedding is aligned with input image. Hence we have pd o:((p) + (1 a) where f is two-layer bottleneck MIP with ReI.U nonlinearity, while is learnable parameter weight the residual connection. Note such MIP-haaed architecture differs from attention mechanism in that the ML.P-aggregated pd is not able to attend to individual prompts in P for dynamic information fusion.
Table 4: Ablation study on our prompt aggregator: We experiment under the base-to-new class generalization setting for few-shot classification. CLIP performance (zero-shot) is listed baseline. H: Harmonic mean of base and new class accuracies.
Avg across 11 datasets
Base
New
H
Zero-shot CLIP 40
69.34
74.22
71.70
Random sampling
70.1344.93
74.61
72.30+1.15
Mean p
82.05-+0.1
75.64+00.1
78.71
MI.P-based aggregation
84.39
76.33
80.16
Attention-bass aggregation (default)
84.72-H0.18
77.54
80.97 +0.19
Table 4 summarizes the comparison results on few-shot classification under the base-to-new class generalization setting, It is clear that random sampling is not good choice, whose performance has
15

---

Label:
Attention score
LLM-generated reference prompts
Jeep Compass SUV 2012
Stanford Cars
0.30
"The exterior of Jeep Compass SUV 2012 is very sleek and sporty looking."
0.11
"The exterior of Jeep Compass SUV 2012 is very sleek and modern."
"A Jeep Compass SUV 2012 has boxy shape
sloping roafline."
0.14
"The Jeep Compass SUV 2012 car that seats five people."
0.08
"This image is of red Jeep Compass SUV from 2012."
Label: toucan
ImageNet
"Toucans brightly colored birds large beaks."
0.60
"A toucan has a long, thin beak that curved at the end and black and white."
0.19
"A toucan is a large, brightly colored bird with a long, curved bill."
0.07
"A toucan with its large beak perches on branch."
"A Keel-billed Toucan
Label: abbey
0.09
"A Abbey looks large stone building with many stained glass windows.
SUN397
0.24
"Abbeys are typically large stone buildings with high cailings and tall windows."
0.35
"Each Abbey is different, but typically they are large stone buildings with tall spires."
0.02 "There is certain way identify abbey."
0.29 "A abbey typically looks like large church cathedrall."
Figure 7: Visualizing the attention score of each reference prompt during prompt aggregation. Note the attention score is re-normalized among the illustrated prompt samples. We observe low attention scores for prompts that are redundant or noisy (irrelevant to input image), e.g. the 2nd and 5th prompt in the car image.
Table 5:
Comparison with the onnyy baseline for few-shot classification. We experiment under base-to-new class generalization setting. H: Harmonic mean of base and new class accuracies.
Avg across datasets
Base
New
H
AAPE (default)
84.7240.18
77.54 20.29
80.97
LLM- bused
(AAPE) onnly
84.01 10.12
75.93
79.77
CuPL
44.3
75.25
74.78
w/o priors
AAPE w/o Casill
79.47
73.25
4..34
76.23
CoCoOp 68
80.47
71.69 20.37
75.83
large variance and only marginally better than that of zero-shot CLIP. This because the reference prompts often noisy and not related to input image. Hence randomly sampled prompt is likely poor source of textual knowledge to distill from. When we use the mean prompt p as supervision, significant gains observed due reduced noise well enriched information. MLP-based aggregation leads to larger gains, since learning is introduced now to find a better and image-aligned supervisory signal. With the attention mechanism, achieve the best results with dynamic prompt aggregation, which is our default approach.
B.2 Simple Classification Framework Based on AAPE Only
The classifier weights are built from the combination AAPE (=x) and a template embedding w, using a projection g. Here compare with simpler f(x (oxyy baseline for classification, without combining w, or using projection g. This baseline uses the standard text classifier weights wi, while h(x) acts as proxy image query to be matched to w, Such setup is extremely similar to the image-to-text retrieval task. Evaluating the h(x))-nny baseline a more direct quantification of how well the text knowledge in AAPE, the image captioning embedding,
In the main paper, we introduce our default classification framework in Section 3.3
distinguish different classes.
Table shows the (x))-nnly baseline achieves reasonable performance for both base and new classes, which indicates AAPE's good generalization on the classification task. When compared to CuPL. that captures text knowledge by simply ensembling --neerree image prompts, the hi(x)-only baseline is much more performant by learning adaptive prompt aggregation. While the AAPE
16

---

Table 6: Sensitivity analysis of the distillation loss weight X. We report the few-shot classification
results -base-to-new class generalization setting) averaged across 11 datasets terms of H, the Harmonic mean of base and new class accuracies.
H 09.14 80.65 2.22 80.84 80.97 10.19 80.90
80.93 00.1 80.76
Table 7: Inference cost for few-shot classification and other tasks beyond classification. For few-shot classification, we report accuracy averaged over 11 datasets (base-to-new setting). We implement CoCoOpt have a bigger prompt prediction network than CoCoOp, such that CoCoOpt has matching parameter count with AAPE.
Compute cost
Few-shot accuracy
Method
Comment
params
GFLOP
FPS
Base
New
CoOp
67
Non-adaptive text prompts
2k
162.5
1344
82.69
63.22
71.66
OGEN 65
2k
162.5
1351
84.17
76,86
80.34
MaPI.e
Text+image
3.55M
162.7
1365
82.28
75.14
78.55
PromptSRC
prompts
46k
162.8
1380
84.26
76.10
79.97
CoCoOp 68
35k
162.5
15.08
80.47
71.69
75.83
CoCoOpf
84k
162.6
14.67
81.05
70.12
75.19
AAPE h(()+t
Input-adaptive text prompts
82k
162.6
14.92
84.72
77.54
80.97
for classification AAPE k(x)
33k
162.5
15.06
beyond classification
baseline w/o Laisilll and CoCoOp both learn input-adapted prompts but without language supervision. We can observe evident benefits of our h(x))-on) baseline over the language-free methods.
Lastly, our default AAPE-based classification framework consistently outperforms the h-zr)-only baseline, only at small overhead incurred by projection g. In the meantime, since the default classi- fication framework simply combines and projects AAPE h(x) and w;, it enables easy interpretation of the roles of the two components for classification, see Fig.
C Hyperparameter Sensitivity and Compute Cost
Sensitivity analysis of the distillation loss weight À in Eq. (4. Table reports the results for the few-shot classification task. is shown that AAPE performs robustly with overlapping confidence intervals when À E [3, 9). AAPE even outperforms the strong baseline OGEN (average H: 80.34) in by default.
this wide range of X. We set
Compute cost. Table 7 compares the inference cost for different tasks in terms of # parameters, GFLOP and FPS. For the few-shot classification task, we compare AAPE with three types of prompt learning methods. CoOp and OGEN are the first type of methods that learn fixed text prompts. The efficiency benefits of these methods are evident: the number of learned parameters is small (2k), and high inference speed (FPS) can be achieved without requiring forward pass to predict adaptive prompts for every input image. MaPLe and PromptSRC belong to the multimodal prompting methods that learn prompts for both text and image. These methods have comparable GFLOP and FPS with fixed prompt Iearners but have much more parameters to learn, thus risk generalization with sub-optimal accuracy for new classes
CoCoOp and our AAPE both learn input-adaptive text prompts, with reasonable parameter count and GFLOP. However, they suffer from low FPS because of the input-conditional prompt prediction. This low speed also translates the learning stage. The training time (min) for AAPE and CoCoOp are 41.92 and 39.53 respectively, in comparison to 10.08 of CoOp. Despite the equally low time efficiency, AAPE outperforms CoCoOp drastically accuracy, and scales much better with model size than CoCoOp-style methods. To show this, we implement a CoCoOpi baseline that has similar parameter count with AAPE. As expected, CoCoOpř has lower speed than CoCoOp. CoCoOpt also found to have lower class accuracy, i.e., worse generalization.
17

---

Table 8: Few-shot classification in the domain generalization setting. Note AAPE follows
CuPL to query an LLM to obtain natural language prompts, but further learns from those prompts.
Source
Target
ImageNet
ImageNerV2
ImageNet Skeeth
ImageNet-A
ImageNet- R
Zero-shot CLIP 40
66.73
60.83
46.15
47.77
73.96
Prompt learning w/o
MaPLe 20
70.72
64.07
49.15
50.90
76,98
language priors
CoCoOp 68
71.02
64.07
48.75
50.63
76.18
PrompiSRC 21
71.27
64.35
49.55
50.90
77.80
CoOp 67
71.51
64.20
47.99
49.71
75.21
CLIPood 45
71.60
64.90
49.30
50.40
77.20
RPO 25
71.67
65.13
49.27
50.13
76.57
UPT 66
72.63
64.35
48.66
$0.66
76.24
TaskRes
73.07
65.30
49.13
50.37
77.70
I.M-- Basic bused prompts
LASP 7
71.10
63.96
49,01
50.70
77.07
KgCoOp
58
71.20
64.10
48.97
50.69
76.70
ProGrad
69
72.24
64.73
47.61
49.39
74.58
CuPL
68.86
63.14
47.85
48.63
75.11
AAPE
73.56
65.97
50.124023
51.62
77.52+0.4
When we turn our attention to complex vision-language tasks beyond classification, AAPE shines in both efficiency and performance. Note in the tasks of text retrieval, image captioning and VQA, AAPE is used as a standalone image captioning embedding (without projection g) to provide rich language priors. As shown in Table 7 (last row), this setup involves fewer parameters than the classification setting (33k vs. 82k), and leads to slightly higher inference speed. But on the vision- language tasks, AAPE not only achieves SOTA performance (Table 3) but also has higher efficiency than prior works, e.g., about 2.8/1. times faster than MAGMA I1 for training/inference.
In summary, AAPE is designed to be universal text embedding directly applicable to various vision- languge tasks. This is not possible with most prompting methods designed for classification, only that AAPE's generality sacrifices the efficiency in the classification task. We leave future work to speed up AAPE inference the classification setting, e.g., via pruning or distillation techniques to simplify the forward pass of prompt prediction.
D Results of Few-Shot Classification under Domain Generalization
Table 8 shows our approach is robust to the different types of domain shifts on 4 ImageNet variants. Overall, our AAPE outperforms prior works all but the ImageNet-R dataset (where AAPE is still competitive with the state-of-the-art PromptSRC method). AAPE outperforms most prompt learners that do not leverage any language priors, sometimes by a large margin. When compared the prompt learners using basic prompt templates. AAPE shows notable gains thanks to the rich knowledge contained LLMs. The zero-shot CuPL method based on LLM too, but lags far behind due the lack of learning components for downstream adaptation.
E
Comparison with Recent Prompt Learners ProText and ArGue-N
Table compares AAPE with two recent prompt learning methods on the few-shot classification task. The compared methods ProText and ArGuc-N are closely related to AAPE since the former similarly learn text prompts from LLM-generated image prompts or visual attributes. However, both ProText and ArGue-N learn fixed text prompts, and are not adapted to input image. By contrast, AAPE anput-adaptive during both prompt aggregation and prompt prediction recall that our prompt aggregation aligned with the input image, and AAPE prediction is mage-conditional.
We conjecture that our input-adaptive framework makes better use LLM's textual knowledge. The resulting image-aligned AAPE also promotes image-text alignment, giving rise to improved optimization during prompt learning. Table[confirms this with better performance of AAPE on seen class data from the base split source dataset. Input-adaptive AAPE improves generalization too,
18

---

Table 9: Comparison with ProTeex and ArGue-N for few-shot classification in both settings of base-to- nese-to-new class generalization and domain generalization.
Base-to-New Generalization
Domain Generalization
Avg across 11 datasets
Source
Target
Base
New
H
ImageNet
-V2
-Sketch
-A
-R
ProText 22
72.95
76.98
74.91
70.22
63.54
49.45
51.47
77.35
ArGue-N 49
83.77
78.74
81.18
71.84
65.02
49.25
51.47
76.96
AAPE (ours)
84.72
77.54
80.97
73.56
65.97
50.12
51.62
77.52
ImageNet
EuroSAT
SUN397
75
86
82
76
78
73
Gap distance
Gap distance
Gap distance
Optimal gap distance after promot learning
-.. Defoult dislance of pretrained CLP
Figure 8: Optimal gap distance obtained by AAPE learning on the few-shot classification task (base-to-new class generalization setting). Y axis indicates the Harmonic mean (H) of base and new class accuracies on each dataset. axis indicates the gap distance varied by shifting the image and text embeddings following [30]. We see the optimal gap distance can be increased (on SUN397) or decreased (on ImageNet and EuroSAT) over the default gap obtained from pretraining.
achieving comparable stronger generalization performance in two generalization settings than the fixed prompt learners. More importantly, AAPE can be used as a standalone captioning vector in various vision vision-language tasks beyond classification. This is not possible with ProTeex ArGue-N.
F
Additional Image Captioning Results
Fig. showws example captioning results on the NoCaps dataset. We compare LiMBeR 35 with and without AAPE learned on COCO and transferred zero-shot to NoCaps. Thanks the textual knowledge encoded in AAPE, often augments the image features to generate more descriptive captions even when the visual cues are ambiguous (e.g., in the first image, AAPE identifies the turtles which can be easily confused with rocks).
G Discussion on the Image- Text Modality Gap for AAPE Learning
In this section, examine the image- modality gap, concept introduced in [30]. We aim to gain insights behind the strong downstream performance and generalization of AAPE-based prompt learning. Is that because AAPE learning can mitigate modality gap?
To answer this question, we first note that AAPE is predicted from input image features æ via a conditional prompt generator h. Such h can be viewed an image-to-tex mapping function, which should at least maintain (if not improve) feature alignment between the image and text modalities. For empirical evidence, we measure the average cosine feature similarity for the image-prompl pairs on ImageNet. We find AAPE scores 0,91, slightly higher than that of pretrained CLIP (0.89). This fact makes it seem like AAPE should be able to reduce modality gap. Here we perform systematic analysis following [30], which defines modality gap as the Euclidean distance between the centers of image and text features.
Fig. X shows the analysis results for few-shot classification on three example datasets. As expected, after AAPE-based prompt learning, the optimal gap distance that attains maximum accuracy deviates
19

---

from the default gap distance of pretraind CLIP. We see the optimal gap is increased on SUN397 dataset, while decreased on ImageNet and EuroSAT datasets. In fact, the optimal gap is reduced out of total of 11 datasets and moderately increased on the remaining 4. On the other hand, AAPE consistently improves classification accuracy on each dataset, see Table and Fig. [ This suggests that modality gap is not highly correlated with downstream generalization, which is in line with one of the main arguments in [30] that good generalization does not necessarily need a reduced modality gap.
Then why does AAPE generalize when the modality gap not reduced? We hypothesize that our multi-task learning reshapes the loss landscape in way that encourages generalizable solutions. Specifically, we optimize two loss functions for AAPE: the task loss that (over)fits the seen class data, and the distillation loss that moves AAPE closer to some aggregation of external text knowledge. We find the distillation loss often promotes generalization (Fig- 4 and avoids overfitting caused by the task loss. Hence the two losses could move AAPE in opposite directions, modifying the modality gap differently on the loss landscape. As shown in Fig. [8] the gap change (increase or decrease) is highly dependent on the image-text distribution on the considered dataset.
Note the above optimization perspective is not limited to classification. We use our hypothesis similarly explain AAPE's good generalization in complex vision-language tasks like VQA where multi-task loss is used (distillation + task loss). We leave future work study 1) how multi- task learning affects modality gap dynamically and 2) the relationship between modality gap and downstream generalization.
H Broader Impact
The main contribution of this work is the use of text-based knowledge to improve the downstream generalization of CLIP. The textual knowledge is distilled from either human-annotated image captions or LLM-generated natural language prompts. Such knowledge significantly improves CLIP's downstream performance but carries potential socictal impacts. Specifically, when the image captions/prompts reflect (unintentional) biases, our distillation and learning methods could inherit or amplify these biases in the learned feature embeddings. This would potentially lead to perpetual unfair discriminative outcomes in a variety of vision- vision-language tasks and more critical applications such as Al-driven planning and decision-making.
20

---

Ground Truth:
"Sea turtles lie on the beach while a wave pushes in the background." "
LIMBeR:
"Hard rocks by beach."
LiMBER+AAPE:
"Sea turties
sand of the beach."
Ground Truth:
"A laboratory contains lots of scientific equipment, rolling chairs, and long black countertops."
LiMBeR:
"An office with no people in it."
LIMBeR+AAPE:
"Laboratory with black countertops and medical equipment."
Ground Truth:
"Crowd of people in bright wigs and clothes."
LiMBeR:
"A group of people gathering together."
LiMBeR+AAPE:
"A group of people dressed up colorful clothes."
Ground Truth:
"A lone camel standing on a hill in the desert sand."
LiMBeR:
camel is standing on top of hill."
LiMBeR+AAPE:
"A camel is walking sand hill ahead of plant."
Ground Truth:
"A neatly made bed with swans made out of linens it."
LiMBeR:
"Two swans sitting on a bed."
LIMBeR+AAPE:
"A bed with swans made of towels."
Ground Truth:
"A white jack lantern is sitting the hay with other orange pumpkins around."
LiMBeR:
"A group of pumpkins are smiling."
LiMBeR+AAPE:
"A and smiling pumpkin sitting stacks hay."
Ground Truth:
"A group people paddle boat with lifejacket river."
LiMBeR:
"A group of people in the river."
LiMBeR+AAPE:
"A group of people paddling against the waves on river."
Figure Captioning results on NoCaps dataset: LiMBeR vs. LiMBeR+ (zero-shot).
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are stated in both Abstract and Introduction sections, and each claim is supported by experimental results. We further enumerate our contributions in the Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations and future work the Conclusion section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification:
No theoretical results are included in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Experimental setup and dataset details are included in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Section describes the experimental setup, dataset and implementation details to run and reproduce experiments.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section describes all the experimental setup, dataset and training/testing details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the average result and the standard deviation of three runs with random seeds for our experiments/ablations image classification and text retrieval. Please refer to Table
6 and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is included in Section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htattps:/neeuriii http/peu.p...e. cccppuulic/tthiiccciiiieell.iii.
Answer: [Yes]
Justification: We confirm that the work presented in this paper is performed in a manner consistent with NeurIPS Ethics Guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact of our work in Appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work presents a finetuning algorithm of CLIP models so this question is not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We include this information in experimental details in Section
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "7VvUCTA98N", "reviewer": "Reviewer_UfdC", "summary": "The paper proposes a prompt learning method that distills the knowledge from pre-trained LLM while conditioning them on the image embedding. A Prompt Aggregator module combines the LLM generated prompts per image and the Prompt Generator module, generates a prompt from the image embedding. The modules are trained using a regularization loss between the aggregated and generated prompts, and the downstream task loss. Experimental results on 11 image classification datasets, four imagenet domain variants, two image captioning datasets and VQA dataset show the effectiveness of the proposed prompt learning method.", "questions": "- What are the number of parameters learnt in comparison to other prompt learning methods? Is the final projection layer necessary? \n- The zero shot performance on the captioning task seems promising. Is there a projection layer $g$ for the VQA and captioning tasks as well? Perhaps the projection layer being trained can be giving a significant boost in the classification task. What are the authors thoughts on this?", "limitations": "none", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The experimental results cover a broader spectrum that includes image classification, vision-language understanding, such as captioning and VQA. The results show the proposed method is effective across the range of tasks.", "weaknesses": "- The method is shares a lot if similarity to [1] which distills the LLM knowledge through prompts to the CLIP text encoder. It is important to compare and distinguish how this work differs from [1]. \n- See questions\n\n[1] Khattak, Muhammad Uzair, et al. \"Learning to Prompt with Text Only Supervision for Vision-Language Models.\" arXiv preprint arXiv:2401.02418 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback on our work. Regarding the efficiency concern, please refer to our **Response to Common Concern**. For other comments, our point-by-point response is as follows.\n\n**Compare and discuss how this work differs from ProText (arXiv:2401.02418)**\n\nThanks for bringing this paper to our attention! ProText and our AAPE both learn text prompts that distill natural language priors from LLM, with the same goal of improving generalization to novel classes or data distributions. However, the prompt learning mechanisms are different: ProText learns individual prompt token vectors. They are then combined with a text template to generate the embedding of a full prompt sentence, which stays close to those LLM-generated sentences. On the other hand, we learn to directly generate the prompt sentence embedding which we call AAPE, supervised by both LLM-generated prompts and task loss. Note our AAPE prediction is conditioned on the image feature $x$ via a prompt generator $h(x)$.\n\nWe share the following benefits:\n* 1. Our input-conditioning mechanism focuses on extracting prompt features from the input image, while ProText learns unconditional text prompts for each class. Without considering input information, ProText has a higher risk of overfitting to training classes (thus worse generalization to new classes). 2. Our image-conditional prompt generator $h(x)$ can be viewed as an image-to-text mapping function, which helps to bridge the modality gap between image and text feature spaces. This further gives rise to improved generalization during prompt tuning.\n* Here we provide empirical support for AAPE's better generalization on the few-shot classification task. In the base-to-new class generalization setting, we have the average Base/New/H class accuracy on 11 datasets: ProText (72.95/76.98/74.91) vs. AAPE (84.72/77.54/80.97). While in the domain generalization setting, we have the accuracy for ImageNet/-V2/-Sketch/-A/-R: ProText (70.22/63.54/49.45/51.47/77.35) vs. AAPE (73.56/65.97/50.12/51.62/77.52).\n* 3. More importantly, our AAPE is designed to be a universal embedding directly applicable to various vision-language tasks. For example, in our text retrieval, image captioning and VQA experiments, AAPE $h(x)$ is used as a standalone ``image captioning latent'' that achieves SOTA performance (Table 2 in main paper). This is not possible with ProText.\n\nWill add the above discussions and comparisons in revised paper.\n\n**Is the projection $g$ necessary for classification? Is $g$ used for VQA/captioning tasks?**\n\nIn response, we apply AAPE $h(x)$ alone to the tasks of image-to-text retrieval, image captioning and VQA, which proves the efficacy of AAPE thanks to the distilled language priors in it. But for classification, it is not possible to only use $h(x)$ to fulfill the task. We argue that it's necessary to include both $w_i$ and $g$ to build a well-functioning classifier.\n\nSpecifically, the image-conditional $h(x)$ can be viewed as an ``image captioning latent vector''. It does not necessarily encode the explicit class name, which prevents it from acting as a good text classifier for each class. Therefore, we start with a basic template ($w_i$) that allows us to manually encode the $i$-th class name. Then we combine $w_i$ with $h(x)$ so that $h(x)$ can provide extra class descriptions that are adapted to input image. For combination strategy, we found it not working when linearly combining $w_i$ and $h(x)$ based on element-wise addition or using a linear projection $g$, since that tends to ignore $h(x)$ if we match the linear combination to $x$ for classification (see L187-188). Hence we choose to use a nonlinear $g$ on top of the concatenation of $w_i$ and $h(x)$ for their non-trivial fusion.\n\nWe acknowledge that $g$ introduces more parameters for prompt learning. One might wonder whether the increased model size contributes significantly to our classification accuracy gains. To this end, we compare with a variant of the CoCoOp method that similarly conditions prompt generation on input image. Our implemented CoCoOp variant has a $g$ attached after the text encoder, such that its total parameter count is similar to ours. We show such CoCoOp variant achieves an average H of 75.32 on 11 datasets, even worse than that of projection-free CoCoOp (75.83) mainly because of the worse generalization on new classes. This indicates that adding projection $g$ with increased capacity does not necessarily mean higher performance for few-shot classification. On the other hand, our similar-sized AAPE with $g$ achieves H 80.97, which benefits a lot from the learned language priors. Figs. 3-5 all confirm the positive contributions of language priors in AAPE. Will add the above clarifications in paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sjTLM7SRom", "reviewer": "Reviewer_7B5Q", "summary": "This framework first aggregates textual knowledge from human or large language model (LLM) generated prompts into a summary aligned with each input image. This is achieved using a prompt aggregator. A prompt generator is then jointly trained to create prompt embeddings that are close to this aggregated summary while also minimizing task-specific loss. The method demonstrates improvements in performance on various downstream vision-language tasks, including few-shot classification, visual question answering (VQA), and image captioning without incurring LLM inference costs during testing.", "questions": "In Figure 1, what input is provided to GPT-3? Is it only text? Why does the last sentence of the description contain \"the image is...\"?\n\nWhere are human-generated image captions obtained from?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation is clear and the method is reasonable to me. The proposed method extends the application of downstream tasks from the common image classification to image-text retrieval, visual question answering (VQA), and image captioning. The performance on Flickr30k dataset is remarkable.", "weaknesses": "This paper misses one relevant research in prompt learning “ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models.” ArGue introduces an attribute-guided prompt tuning approach that outperforms traditional prompt learning methods in specific tasks and datasets. The failure to reference and compare these latest methods may limit the comprehensiveness and advancement of the proposed method.\n\nThe comparison methods used for the image-to-text retrieval task in this paper are somewhat outdated and do not incorporate the latest research advancements. For example, BLIP-2, as a new image-to-text retrieval method, has demonstrated superior performance across various tasks and datasets. To more accurately assess the effectiveness of the proposed method, the latest comparison methods such as BLIP-2 should be included in the experiments.\n\nThe method in this paper shows overfitting on the base classes, with significantly lower performance on the novel classes compared to the base classes. This indicates a lack of generalization ability when dealing with new categories, potentially leading to poor performance on unseen data in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Compare with ArGue (CVPR 2024)**\n\nArGue and our AAPE both learn text prompts that distill language priors from LLM. However, the prompt learning mechanisms are different: ArGue learns individual prompt token vectors. They are combined with the embeddings of the class name and class-wise visual attributes generated by LLM. On the other hand, we learn to directly generate the embedding of a full prompt sentence which we call AAPE, supervised by both LLM-generated natural language prompts and task loss. Note our AAPE prediction is conditioned on the image feature $x$ via a prompt generator $h(x)$.\n\nWe share the following benefits:\n\n* 1. Our input-conditioning mechanism focuses on extracting prompt features from the input image, while ArGue learns unconditional text prompts for each class. Without considering input information, ArGue has a higher risk of overfitting to training class distributions. 2. Our image-conditional prompt generator $h(x)$ can be viewed as an image-to-text mapping function, which helps to bridge the modality gap between image and text feature spaces. This further gives rise to improved generalization during prompt tuning.\n* Here we provide empirical support for AAPE's good generalization on the few-shot classification task. In the base-to-new class generalization setting, we have comparable Base/New/H class accuracy averaged on 11 datasets: ArGue-N (83.77/78.74/81.18) vs. AAPE (84.72/77.54/80.97). While in the domain generalization setting, we have better accuracy for ImageNet/-V2/-Sketch/-A/-R: ArGue-N (71.84/65.02/49.25/51.47/76.96) vs. AAPE (73.56/65.97/50.12/51.62/77.52).\n* 3. More importantly, our AAPE is designed to be a universal embedding directly applicable to various vision-language tasks. For example, in our text retrieval, image captioning and VQA experiments, AAPE $h(x)$ is used as a standalone ``image captioning latent'' that achieves SOTA performance (Table 2 in main paper). This is not possible with ArGue.\n\n**Compare with latest methods like BLIP-2 for the image-to-text retrieval task**\n\nAs mentioned in Section 5.1 and Appendix B, we treat the image-to-text retrieval task as a small-scale proof of concept of our prompt learning method. Specifically, we learn AAPE on COCO dataset only, and test its zero-shot/finetuning performance on Flickr30k. The goal is not to push for SOTA performance, but to verify if the learned AAPE can successfully distill text knowledge from COCO captions which can be useful for downstream text retrieval.\n\nTable 3 provides supportive results as AAPE obtains strong performance with both zero-shot and finetuned models on Flickr30k. What's most interesting is that our zero-shot Flickr30k performance is nearly on par with that of SOTA zero-shot models, including CLIP/SigLIP and the latest work Llip. This proves good generalization of AAPE. We will add results of more recent/competitive methods like BLIP-2 as suggested. But note this won't be a fair comparison since we only train on COCO while other methods like BLIP-2 are trained on billions of image-text pairs.\n\nIn the future, we plan to scale up training data to learn universal AAPE. This not only enables fair comparisons with SOTA multimodal models for image-to-text retrieval, but also on more complex vision-language tasks e.g. using AAPE+LiMBeR which proves promising already.\n\n**AAPE overfits base classes, with lower performance on novel classes; poor generalization on unseen data**\n\nWe humbly argue that overall, AAPE improves generalization over unseen classes and data distributions. We acknowledge there exists a gap between the base and new class accuracies (7.18 points averaged across 11 datasets) for AAPE's few-shot classification experiments. One reason is that our prompt generator $h(\\cdot)$ won't be trained sufficiently well on some datasets with limited class data, while we rely on a good $h(\\cdot)$ to provide useful language priors for new class generalization. For example, DTD and EuroSAT datasets have only 47 and 10 training classes respectively (16 shots per class), and their base-new accuracy difference is high (20.33 and 19.10 points). While the SUN397 dataset has 397 classes (hence better learned $h(\\cdot)$), and the base-new accuracy gap is reduced to 3.06 points. This motivates us to scale up training data in real-world tasks to learn universal prompt generator that generalizes over unseen categories.\n\nNevertheless, under the few-shot classification setting, our AAPE learning does not significantly widen the average base-new accuracy gap (7.18) in comparison to other methods, e.g. the competitive OGEN (gap 7.31) that doesn't leverage language priors. With the language priors, AAPE can actually improve both the base and new class accuracies for each of the 11 datasets (see Fig. 3), sometimes by a large margin. When well trained on ImageNet, AAPE also achieves SOTA results for domain generalization (Table 4). Furthermore, we have successfully trained a competent $h(\\cdot)$ on COCO to describe complex scenes. It shows impressive zero-shot generalization to multiple tasks with varying data distributions, including image-to-text retrieval (on Flickr30k), image captioning (on NoCaps) and VQA (on VQA2).\n\n**Fig 1: what is the input to GPT-3? Why does the last sentence of the description contain \"the image is...\"?**\n\nAs mentioned in L120-127, we follow the CuPL method to query GPT-3 with more than one LLM-prompt templates (i.e., GPT-3 inputs). Examples are \"How can you identify a(n) {}?\" and \"Describe an image from the internet of a(n) {}\". The latter example leads to the GPT-generated image prompts that start with \"this image is...\". Will clarify in the paper.\n\n**Where are human-generated image captions obtained from?**\n\nAs mentioned in L128-136, we use human-annotated captions (5 per image) from COCO dataset to represent complex scenes. We train on COCO captions for 3 vision-language tasks, i.e., image-to-text retrieval, image captioning and VQA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "owgOR4MzaS", "reviewer": "Reviewer_khEq", "summary": "The paper proposes a new prompt embedding named Aggregate-and-Adapted Prompt Embedding (AAPE), which improves prompt learning by distilling knowledge about more detailed descriptions of classes into prompt embeddings. Concretely, the “summary” prompt is obtained by aggregating diverse reference prompts. Then, the prompt generator is trained to produce a prompt embedding that stays close to the aggregated summary while minimizing task loss at the same time. From the experiments, the proposed AAPE shows good performance on diverse tasks such as few-shot classification, VQA, and image-to-text retrieval.", "questions": "Please refer to the weaknesses.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- How to well train prompt embeddings is one of the important topics in fine-tuning Vision-Language Models.\n- The paper is well written.\n- The proposed approach shows good performance in multiple tasks.", "weaknesses": "- More details on using the CLIP reward are required. From my understanding, there is another loss to enhance the CLIP reward $\\texttt{CLIP-s}\\left(\\boldsymbol{x}, \\boldsymbol{p}^\\alpha \\right)$.\n- It would be better to include the performance of the model with only Aggregate-and-Adapted Prompt Embedding (AAPE) $h\\left(\\mathbf{x}\\right)$ in Table 1 without using text embedding $\\mathbf{w}_i$ and projection $g$. This result clearly shows the performance gain induced by the Aggregate-and-Adapted Prompt Embedding.\n- It would be better if the efficiency comparison was included in the paper compared to other prompt learning methods such as CoOp, MaPLE, and PromptSRC, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive feedback and constructive suggestions. For the efficiency concern, please refer to our **Response to Common Concern**. Answers to your other questions below.\n\n**More details on the CLIP reward**\n\nAs mentioned in L160, we use the same CLIP reward as in [17], formulated as $\\texttt{CLIP-S}(x,p^a) = w \\cdot \\max(\\cos(x,p^a),0)$ - will add in paper. This reward works well in our experiments. Studies of more advanced loss/reward functions are left for future work.\n\n**To show the benefits AAPE, use $h(x)$ only but no $w_i$ and $g$ for classification**\n\nIn response, we have successfully applied AAPE $h(x)$ alone to the tasks of image-to-text retrieval, image captioning and VQA, which proves the efficacy of AAPE thanks to the distilled language priors in it. But for classification, it is not possible to only use $h(x)$ to fulfill the task. We argue that it's necessary to include $w_i$ and $g$ to build a well-functioning classifier.\n\nSpecifically, the image-conditional $h(x)$ can be viewed as an ``image captioning latent vector''. It does not necessarily encode the explicit class name, which prevents it from acting as a good text classifier for each class. Also, it's easy to imagine that directly matching the image features $x$ to $x$-conditional $h(x)$ would always lead to a high similarity score, which is not discriminative. Therefore, we start with a basic template ($w_i$) that allows us to manually encode the $i$-th class name. Then we combine $w_i$ with $h(x)$ so that $h(x)$ can provide extra class descriptions that are adapted to input image. For combination strategy, we found it not working when linearly combining $w_i$ and $h(x)$ based on element-wise addition or using a linear projection $g$, since that tends to ignore $h(x)$ if we match the linear combination to $x$ for classification (see L187-188). Hence we choose to use a nonlinear $g$ on top of the concatenation of $w_i$ and $h(x)$ for their non-trivial fusion. Because of such design choices, to isolate the impact of natural language priors under the classification setting, we always use $w_i$ and $g$ and compare $h(x)$ learned with and without the language prior distillation loss. Figs. 3-5 all confirm the positive contributions of language priors. Will add the above clarifications in paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pZiEfVqwAH", "reviewer": "Reviewer_yCM3", "summary": "The proposed method leverages language priors for better downstream adaptation and generalization of CLIP [37], which is similar to CuPL [36] that utilizes prompts generated by a LLM (e.g., GPT-3) for zero-shot image classification using CLIP. Unlike CuPL, the proposed method aggregates multiple LLM-generated or human-generated prompts to construct an aggregated text feature. Using the aggregated text feature as a guidance, the authors adaptively transform CLIP image features to tackle few-shot image classification, image-to-text retrieval, image captioning, or VQA. A notable difference from CuPL is that the proposed method does not run an inference of LLM at test time.", "questions": "(i) Despite the image-text modality gap in CLIP vision-language space [A], the proposed method simply transforms CLIP image features using an aggregated text feature. Does “h(x)” overcome the image-text modality gap? How do the authors deal with the modality gap? If the proposed method fails to overcome the modality gap, the authors should justify why the proposed method leads to better downstream adaptation and generalization of CLIP. \n\n[A] Liang et al., “Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning”, NeurIPS, 2022.\n\n(ii)  According to L176-180, the authors claim that learning h(x) is much more parameter-efficient than learning a sequence of token embeddings. Is it true? For example, suppose that we have 16 learnable word tokens, where the dimension size of each word embedding is 512. In this case, there are 16 * 512 learnable parameters. In contrast, as described in L169-170, “h($\\cdot$)” is a network which consists of two fully connected layers. Given that the dimension size of CLIP features is 768 or 1024, the network seems to have 2 * 768 * 768 or 2 * 1024 * 1024 learnable parameters. What does it mean that learning h(x) is much more parameter efficient than learning a sequence of token embeddings? \n\n(iii) To validate the statement in L191-196 (“Performance is found to be not very sensitive to the $\\lambda$ value in a wide range.”), the authors need to provide experimental results obtained with different $\\lambda$ values.", "limitations": "Yes, in page 9 and 16.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper has the following strengths:\n\n(i) The proposed method is timely, and seems to be effective and efficient. By aggregating text features produced from LLM-generated or human-generated prompts and then using the aggregated feature as a guidance, the proposed method could effectively leverage language priors for better downstream adaptation and generalization of CLIP. After the distillation of rich language knowledge during the training time, the proposed method does not need to run an inference of LLM at test time. \n\n(ii) The authors validate the effectiveness of the proposed distillation method for downstream tasks (few-shot image classification, image-to-text retrieval, image captioning, VQA). It demonstrates that the learned adaptive transformation guided by language priors actually improves the adaptation and generalization of CLIP for downstream tasks.", "weaknesses": "This paper has the following weaknesses:\n\n(i) This paper lacks the justification of how the adaptive transformation of CLIP image features, guided by an aggregated text feature, overcomes the image-text modality gap which exists in CLIP vision-language space [A]. Despite the modality gap, the proposed method simply encourages transformed CLIP image features to be located nearby an aggregated text feature (using L2 distillation loss).\n\n[A] Liang et al., “Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning”, NeurIPS, 2022.\n\n(ii) The proposed method might not be able to improve the model robustness to distribution shifts which are difficult to be explained by natural language. For example, the Terra Incognita dataset [B] contains distribution shifts caused by different camera locations in wild environments. The widely used benchmark of in-the-wild distribution shifts, WILDS [C], also contains such distribution shifts. Since the proposed method leverages only language priors, it might not be good at handling those distribution shifts.\n\n[B] Beery et al., “Recognition in Terra Incognita”, ECCV, 2018.\n\n[C] Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts”, ICML, 2021.\n\n(iii) This paper lacks experimental results obtained with different distillation loss coefficient values, although the distillation loss is the main contribution of this paper. According to L191-196, it simply states that the proposed model is not sensitive to different coefficient values. Without relevant experimental results, it is difficult to evaluate this statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition of our work and the detailed feedback! We respond to specific comments below.\n\n**Does AAPE $h(x)$ mitigate the image-text modality gap?**\n\nGreat question! Short answer is yes. Essentially, AAPE belongs to those text prompt tuning methods that only operate at the text branch, and one tuning signal is from some aggregation of external text knowledge. This sounds like we may risk breaking the image-text feature alignment after finetuning. However, AAPE is directly predicted from the image feature $x$ via our prompt generator $h(\\cdot)$ (or an image-to-text mapping function) — AAPE does not involve separate token vectors as learned in CoOp and CoCoOp. Then $h(\\cdot)$ can be viewed as a bridge between the image and text modalities, encouraging their alignment via an explicit feature mapping function. This way, our finetuning process in Fig. 2(b) can be re-interpreted from the perspective of modality gap mitigation: first we compute two text embeddings that are already image-aligned — $w_{i}$ (from frozen CLIP text encoder) and $h(x)$; they are then combined using a projection $g$ to compute an image-text alignment loss to further reduce modality gap.\n\nNote our image-text mapping function $h(\\cdot)$ is similar to that in MaPLe, only that MaPLe learns both image and text prompts (vs. our unimodal prompts). The key difference between the two methods is that our $h(\\cdot)$ maps to some natural language equivalent while MaPLe does not introduce any language priors. This makes the text distribution for our finetuning not deviate too much from the free-form texts used for CLIP pre-training. As a result, our learned image-to-text mapping is expected to preserve CLIP's feature alignment level after finetuing. For empirical evidence, we compare the average cosine similarity score for the ground-truth image-prompt pairs on ImageNet. The higher the similarity score, the better the image-text alignment. We show AAPE scores 0.91, slightly higher than CLIP (0.89) which confirms our hypothesis. AAPE is also comparable to multimodal prompting method MaPLe (score 0.92), yet with language priors, AAPE achieves better classification accuracy and generalizes to more vision-language tasks.\n\n**Natural language may not be able to handle particular distribution shifts e.g. on Terra Incognita and WILDS datasets**\n\nThanks for this insightful comment. We do rely on the language priors learned in AAPE to specify descriptive class details that are often invariant to image-based distribution shifts. Table 4 confirms our robustness across ImageNet variants. Intuitively, the description of \"boxy shape and sloping roofline\" for a \"Jeep car\" is useful to distinguish it from other similar classes, regardless of the style of input image, e.g., sketch images in ImageNet-Sketch or paintings in ImageNet-R. That said, we acknowledge that our model robustness (to domain shifts) depends on the descriptiveness of training image prompts and their coverage of discriminating image information in the considered domain. The suggested Terra Incognita and WILDS datasets are both good examples where one might need customized image prompts (preferably generated by LLM at scale) with domain knowledge, in order to generalize across e.g. different structural scaffolds of molecules in WILDS dataset. We leave such investigations of LLM prompting to future work, and will add the above discussions in main paper.\n\n**Ablation on distillation loss coefficient $\\lambda$**\n\nThe attached **Rebuttal.pdf** includes the ablation results in Table 2. It is shown that our AAPE performs robustly (with overlapping confidence intervals) when $\\lambda \\in [3,9]$, and still outperforms the strong baseline OGEN (H: 80.34) in this wide range of $\\lambda$. By default, $\\lambda=5$ is used for classification task.\n\n**L176-180: is it true that learning h(x) is more parameter-efficient than learning a sequence of token embeddings?**\n\nSorry about the confusion. We meant for the purpose of augmenting $w_i$, how to efficiently learn an image-conditional embedding that is equivalent to a full prompt sentence. So the comparison is only under the context of learning sentence embedding conditioned on image $x$. We could simply generate a single embedding for the sentence of length $L$ based on $x$ using $h(x)$. Or we could generate $L$ embeddings for individual word tokens based on $x$. Obviously the token-wise prediction is much less efficient, since we need to train either $L$ conditional generator networks {$h_l$}$_{l=1}^L$ or a large-scale autoregressive network $h$. Will clarify in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "swlLtpMx8v", "reviewer": "Reviewer_ZqyB", "summary": "This paper proposes a new method to enhance the downstream generalization of CLIP by distilling knowledge from LLM- or human-generated text prompts. The proposed method involves training a prompt generator to predicts prompt embeddings (AAPE) based on images.", "questions": "Can Input-Adapted Prompt Aggregator correctly distinguish between noisy and accurate text prompts?  For example, in fig1, the last text prompt in the natural language prompts (“This image is of a red Jeep Compass SUV from 2012.”) inaccurately describes the training image. What’s the attention score of this prompt during aggregation?\n\nFLOPs of the proposed method.\n\nHow long does it take to evaluate the proposed method and CLIP on ImageNet1k dataset?", "limitations": "The analysis on the aggregated text prompt is missing.\nThe proposed method should be applied on more vision-language models, such as SigLIP[1]. This limitation has been mentioned in the paper.\n\n[1] Sigmoid Loss for Language Image Pre-Training.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe method aggregates and distills task-related knowledge from LLM- or human-generated text prompts, leading to better downstream generalization of CLIP.\n2.\tThe approach demonstrates good performance on various downstream vision-language tasks.", "weaknesses": "1.\tThe analysis on the aggregated text prompt is missing. There is no evidence to prove that Input-Adapted Prompt Aggregator can alleviate the influence from noisy text.\n2.\tThe evaluation efficiency may be significantly impacted by the need to connect predicted prompts base text features.\n3.\tA minor error is found in line 181, where “a photo of a {class} + category type” should be “a photo of a {class}”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful suggestions to improve our work. Below is our point-by-point response.\n\n**Whether the prompt aggregator can suppress noisy text prompts.**\n\nThe attached **Rebuttal.pdf** (Fig. 1) visualizes the attention score for some prompt samples. We do observe low scores for those noisy or redundant prompts (see the example of car image), which will be suppressed during aggregation. We plan to add a larger-scale visualization and a correlation analysis between the attention score and image-prompt similarity.\n\n**Inference efficiency, FLOPs, evaluation time on ImageNet1k.**\n\nPlease refer to our **Response to Common Concern** for comparisons and discussions of inference cost in terms of # params, GFLOP and FPS. AAPE evaluation on ImageNet1k takes a reasonable amount of time (12.6 min).\n\n**Application to more vision-language models, such as SigLIP.**\n\nWe have shown the benefits of the language priors learned in our AAPE for two categories of vision-language models: 1) contrastive CLIP and 2) generative LiMBeR that connects the vision encoder with an LLM. For future work, we do plan to study more multimodal models to test the generality of AAPE, as mentioned in L318-320. We can go the contrastive route and apply AAPE to e.g. SigLIP as suggested. What's more interesting is the application to larger-scale generative models like BLIP-2 and LLaVA (work in progress) for general purpose visual and language understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Response to Common Concern**

Thanks to all reviewers for the thoughtful comments. Before responding to the questions raised by each reviewer, we first address the common concerns around efficiency.

The attached **Rebuttal.pdf** (Table 1) compares the inference compute cost between our AAPE and 3 types of prompt learning methods. CoOp and OGEN are the first type of methods that learn fixed prompts with no adaptation to input. The efficiency benefits in these methods are evident: the number of learned parameters is small (2k), and the fixed prompts only need a single forward pass for all batch data, leading to high inference speed (FPS). MaPLe and PromptSRC belong to another type of methods that learn prompts for both text and image modalities. These methods have comparable GFLOP and FPS with CoOp but much more parameters to learn, thus risks generalization with often sub-optimal new class accuracy.

CoCoOp and our AAPE both learn input-adaptive text prompts, with reasonable parameter count and comparable GFLOP. However, they suffer from low speed (FPS) since they require forwarding input-conditional prompts to the text encoder each time. Despite the low FPS, we argue that AAPE still shines:

* Not only because of its highest classification accuracy, but also because AAPE scales much better with model size than CoCoOp-style methods that similarly condition prompt generation on input. Table 1 in **Rebuttal.pdf** provides empirical evidence by comparing AAPE with a CoCoOp$\dagger$ baseline that has similar parameter count (with bigger prompt generation network). Note the degraded FPS and new class generalization with CoCoOp$\dagger$, which makes it underperform AAPE in both efficiency and accuracy.
* More importantly,  AAPE is designed to be a universal embedding directly applicable to various vision-language tasks. We have successfully applied AAPE to few-shot classification (with the help of a projection $g$), as well as to text retrieval, image captioning and VQA where AAPE is used as a standalone ``image captioning latent'' (without $g$) to provide rich language priors. This is not possible with CoCoOp, OGEN or many other prompting methods for few-shot classification, only that AAPE's generality sacrifices the efficiency in the classification task. While in complex vision-language tasks, AAPE can achieve SOTA performance (Table 2 in main paper) with higher efficiency than prior works, e.g., about 2.8/1.2 times faster than MAGMA for training/inference.
* We leave as future work to further speed up AAPE inference particularly for classification, via pruning or distillation techniques to simplify the forward pass of prompts.

Meta-review: The paper introduces Aggregate-and-Adapt Natural Language Prompts for improving the downstream generalization of CLIP. Reviewers acknowledged the novelty of distilling textual knowledge from natural language prompts, either human-generated or from LLMs, to enhance performance across a range of vision-language tasks. The experimental results demonstrate the effectiveness of the method on tasks like few-shot classification, image captioning, and VQA.

However, concerns were raised about missing comparisons with relevant baseline methods, such as ProText, and some reviewers questioned the efficiency of the proposed approach compared to existing prompt-learning methods. Additionally, there was a call for more detailed analysis of the aggregated text prompt’s ability to handle noisy data. The authors provided a detailed rebuttal, addressing these concerns, as well as other, with additional comparisons and clarifications, which were generally well-received by the reviewers. Consequently, all reviewers vote for accepting the paper post-rebuttal. 

Considering the novelty, the strong experimental results, and the authors’ comprehensive response which resolve majority of the concerns, the AC recommends accepting the paper. However, the authors are encouraged to further strengthen the final version by including the suggested comparisons and additional analysis as promised in the rebuttal.
Decision: Accept (poster)