AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation
Authors: Yuhan Zhu
Yuyang Ji
Zhiyu Zhao
Gangshan Wu
Limin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-Language Model
Zero-shot/Few-shot Transfer
Image Classification
Video Action Recognition
Optimal Transport
Primary area: machine_vision
TL;DR: Transferring vision-language models to visual classification via AWT framework (Augmentation, Weighting, and Transportation).

Abstract: Pre-trained vision-language models (VLMs) have shown impressive results in various visual classification tasks.
However, we often fail to fully unleash their potential when adapting them for new concept understanding due to limited information on new classes.
To address this limitation, we introduce a novel adaptation framework, AWT (Augment, Weight, then Transport). AWT comprises three key components: augmenting inputs with diverse visual perspectives and enriched class descriptions through image transformations and language models; dynamically weighting inputs based on the prediction entropy; and employing optimal transport to mine semantic correlations in the vision-language space.
AWT can be seamlessly integrated into various VLMs, enhancing their zero-shot capabilities without additional training and facilitating few-shot learning through an integrated multimodal adapter module.
We verify AWT in multiple challenging scenarios, including zero-shot and few-shot image classification, zero-shot video action recognition, and out-of-distribution generalization. AWT consistently outperforms the state-of-the-art methods in each setting. In addition, our extensive studies further demonstrate AWT's effectiveness and adaptability across different VLMs, architectures, and scales.

AWT: Transferring Vision-Language Models via T Transportation
A Agmeetaatio,, Weighting, and
Yuhan Zhu Yuyang Ji
Zhiyu Zhao Gangshan Wu' Limin Wang" 1.2+ ...
State Key Laboratory for Novel Software Technology, Nanjing University
SShanghai AI Laboratory
https://githinn httpps///gthuub..cc////////////
Abstract
Pre-trained vision-language models (VLMs) have shown impressive results in vari- ous visual classification tasks. However, we often fail to fully unleash their potential when adapting them for new concept understanding due to limited information on new classes. To address this limitation, we introduce a novel adaptation framework, AWT (Augment, Weight, then Transport). AWT comprises three key components: augmenting inputs with diverse visual perspectives and enriched class descriptions through image transformations and language models; dynamically weighting inputs based on the prediction entropy: and employing optimal transport to mine semantic correlations in the Languag space. AWT can be seamlessly integrated into various VLMs, enhancing their zero-shot capabilities without additional training and facilitating few-shot learning through an integrated multimodal adapter module. We verify AWT in multiple challenging scenarios, including zero-shot and few-shot image classification, zero-shot video action recognition, and --d-rii-iiiitttoioon generalization. AWT consistently outperforms the state-of-the-ar methods in each setting. addition, our extensive studics further demonstrate AWT's effectiveness and adaptability across different VLMs, architectures, and scales.
Introduction
Recent advances in vision-language models (VLMs) [1-8], which undergo extensive pre-training on web-scale image-text pairs, have exhibited remarkable success in various classification tasks. VLMs are trained to associate images with relevant textual descriptions. In the standard protocol (Fig. (a)), raw images and class names are projected into a joint vision- embedding space, where the class with the shortest distance to the image representation is selected as the prediction result.
However, directly using raw images and class names in testing has limitations [1, 9J. Visually, the broad scope of pre-training compels VLMs analyze all image elements, lacking capability of focusing on specific interested regions. For instance, model might miss critical facial features of cat while unnecessarily focusing on irrelevant elements like "bench" and "grass" (Fig. (a)). Textually, since VLM pre- pre-training associates visual elements with diverse and rich textual descriptions (e.g.. colors and textures), merely using class names during test falls short of capturing the full spectrum of visual content. To enhance input effectiveness, the literature focuses on post-training prompts [9-33] (Fig. (bb) that provide contextual cues, thereby helping the model in prioritizing relevant features, such as cat's ss attributes. However, this approach often depends on the availability of training resources, which may not be always practical.
In this study, we are interested in enhancing inputs for better adaptation of VLMs without training prompts. We advocate for data augmentation simple yet effective strategy, depicted in Fig. (cc).
Corresponding author: Imwang@nju. edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Add visual prompts
Aug, views
75.0
Aceaua
TAL
10.0
colzulate distance
calculate distance
colculate distance
Ancrogg
63.5
Aug. views
80.0
Regdoll
Ragdoll
[fiendly exprression
blue eyes
$7.5
Inelaxed and docle
longhaired
Add text prompts
ligt-colloed body dark
Image
(8) Standard protocol
(b) Prompt-besed
(u)mmentttebbeed (ours)
(d) Performance Comparison
Figure I: (a) Standard protocol directly calculates distances between raw images and class names in the joint V-L space. (b) Prompt-based methods enhance inputs with post-trained visual textual prompts to provide the task-specific context. (c) Augment- -based method enriches raw inputs with image transformations and class descriptions, requiring additional training. Upon this, we propose AWT. which considers both intra- intra-modal importance variations and cross-modal semantic correlations. (d) AWT is evaluated against SOTA methods across four tasks: zero-shot and few-shot image classification, out-of cut-of-distribution generalization, and zero-shot video action recognition.
Techniques like random resized cropping and image flipping enrich the input with varied and multiscale perspectives, while detailed textual descriptions for each class provide richer visual narratives. Although manually crafting diverse descriptions for each class is expensive, employing Large Language Models (LLMs) [34-6]] presents an efficient alternative.
Nonetheless, several challenges remain. First, the intra-modal importance of each augmented image and description needs assessment, not all views contribute equally to class recognition- may be irrelevant background elements or non-visual descriptors such as the cat's personality. Second, the inter-modal interaction requires consideration, as descriptions such as "dark face" or "light-colored body" might have direct semantic correlations with some image crops (Fig. I(c)).
To tackle these challenges, we propose AWT, novel framework that augments raw inputs into diverse views, weights view importance in each modality dynamically, and transports semantic correlations across modalities. Initially, AWT augments LLMs. Subsequently, it weights the importance of each view on the fly based on its prediction entropy, as more confident predictions typically indicate higher accuracy [37]. This method allows AWT to identify and prioritize significant views, and adjust the importance distribution dynamically according the task-specific context (e.g., candidate class names). AWT then formulates the image-text distance calculation as an optimal transport problem [38, 39], considering each augmented view as quantity of sand. The importance assessed for each view determines the mass of its corresponding sand pile, and distances are calculated using cosine similarity. This formulation can effectively discover cross-modal correlations by solving the optimal transport mroblem-wiich which minimizes the effort required to transport sand from one modality to another. Additionally, generating class descriptions from LLMs using a simple prompt like "Describe a {class)." often results in overly generic descriptions. Inspired by chain-of-thought approach [40], we introduce two-step, dataset-aware prompting method. This approach encourages LLMs produce class descriptions that are both diverse and dataset-relevant.
inputs via image transformations and
We implement AWT using the CLIP model [1] and evaluated its performance across 21 datasets covering four challenging tasks: zero-shot and few-shot image classification, out-of-distribution generalization, and zero-shot video action recognition. As shown in Fig. (d), AWT consistently surpasses the existing state-of-the-ar art methods in each setting. Our extensive analysis further examines AWT's flexibility with diverse architectures, its scalability with different model sizes, and its potential applicability to other VLMs.
Related Work
Vision-Languagee Models. Leveraging the extensive pre-training on web-scale text-image pairs, languuel models (VLMs) such as CLIP [1] and ALIGN [6] excel in acquiring versatile

---

representations that span multiple modalities. These models adeptly embed texts and images into a shared vision-language feature space, enabling the proximity of inputs with analogous semantics, The inherent flexibility of natural language allows VI.Ms to be effectively utilized across wide range of open-set tasks including image classification [1, 9, 13], object detection [41-43], image generation [44, 45], video action recognition [46-48]. However, such general-purge purposs models often fail to focus on task-specitte details, which can result in sub-optimal performance. This study aims to overcome this limitation by proposing novel adaptation framework, namely AWT, for VLMs.
Adapt VLMs to downstream tasks. Direct adaptation of pre-trained VLMs downstream tasks often results in suboptimal and unstable performance [9]. To overcome this. the existing literature has primarily focused on the use of post-training enrich task context. This includes strategies such as few-shot prompt learning [9, 11, 12, 29, 32], cross-dataset prompt generalization [13-22, 25- 27, 30, 31, 33], unsupervised prompt tuning [23, 24, 28], test-time prompt tuning [10, 49-54], and adapter tuning [55-59]. Conversely, other approaches aim to augment inputs using various resources such as the WordNet relationship hierarchy [60], Large Language Models [61-64], or Stable Diffusion models [50, 65, 66]. Nonetheless, these methods mainly enhance only one modality. In contrast, our study innovatively applies augmentation to both visual and textual modalities and addresses significant challenges in dual-modality augmentation scenarios.
Optimal Transport (OT). Optimal transport (OT), originating from the Monge problem [38] the eighteenth century, serves as a metric for quantifying the distance between mathematical entities [67] while considering their intricate geometric structures [39]. Historically rediscovered in various forms, OT first gained fame in computer vision under the name of earth mover's is distances [68].
The development of efficient approximate solvers [69] has recently propelled a resurgence in OT's popularity, broadening its utility across multiple domains, including object detection [70, 71], domain adaptation [72-74], generative modeling [75-78], semantic correspondence [79], point clouds [80- 82). prompt learning [83-85] and video understanding [86, 87J. Of particular relevance to our study are PI.OT [83] and Wang et al. [85], which leverage OT for fine-grained prompt learning to enhance VLMs. Distinct from these two studies, our research diverges by eschewing the need for additional
training resources, opting instead for an augmentation- based direction.
3 Methodology
3.1 Preliminaries
Contrastive Language-Iman Pre-training (CLIP). CLIP integrates dual encoders-an an image encoder f()) and a text encoder g()) map images and textual descriptions into a shared vision- minimize the cosine distance between embeddings of semantically related image-text pairs. Thanks to the flexibility of natural language, CLIP enables direct application classification tasks without the need for task-specific training. For instance, given an image X E |x W and a set of candidate class names { where C
language (V-L.) embedding space. CLIP is designed
denotes the class count. CLIP computes the embeddings I E Idd for the image and {e; E Rxdd for all class names, where d is the feature dimension. Subsequently, the classification probability for image X being of class 4 can be formulated as:
cxp (cos (e,, I) /T) Eg exp (cos (e,, /T)
p (44 X)
where 7 is a temperature parameter.
Optimal Transport (OT). Optimal transport (OT) theory, originating from the Monge problem [38], provides framework for structural distance measurements. This theory conceptualizes scenarios such as relocating sand at a construction site with the goal of minimizing effort. Mathematically. the initial and target distributions of sands modeled discrete measures:
a and B
Σ by byouyy
(2)
where dz, denotes the Dirac with a concentrated mass a; centered at v4, and similarly for (9. Here, N and M represent the number of source and target locations, respectively. The cost of transporting
3

---

Aug. views
desc
dest
Importance decreases
Input image
random resized grop Flip
Importance Score
Data Trans.
Augment
Candidate class
Weight
Transport
pho.
Minarets
Bell Tower
photo of Bell Tower
has a bell hanging from it
has dome- shaped top
hos
windows
Lighthouse
Encrstep datase t-awore
Aug. views
openings
prompt
Importance Score
Figure 2: Pipeline of AWT: Augment, Weight, then Transport. Given an image and candidate class names, we first augment each input into diverse views. These views are then fed into the CLIP model to obtain coarse predictions. To assess the importance of each view, we use prediction confidence proxy and introduce an entropy-based weighting mechanism. Next, we measure the distance between image-text view sets by solving optimal transport (OT) problem. Finally, the resulting OT distance used represent the distance between the input image and each class name.
sands from any source location z; any target location yj is given by the cost function c(z;, Wj) To extend the application to broader and more intricate scenarios, egg., cross-modal correlation, the Kantorovich relaxation [88] is employed. This relaxation introduces flexibility in the transport plan and ensures symmetric transport solutions. The transport plan P R. x where element P... indicates the mass transported from I; to Yj. must satisfy the constraints:
U(a,b) b) def. (P E
NxM PlM = and P'Iy=bb
(3)
Kantorovich's formulation seeks minimize the total transportation cost:
Co B)
min (C,P) PEU(a.b)
CigPiis i
(4)
where C.2 = c( Bj) defines the cost matrix.
3.2 AWT: Augment, Weight, then Transport
Pre-trained VLMMs often underperform when adapted to new concepts due to insufficient information about new classes. Moreover, their extensive pre-training scope leads them to analyze all elements of an image, causing them to miss contextually important cues crucial for specific downstream applications. To overcome these limitations, introduce a novel framework, termed AWT (Augment, Weight, then Transport), to enhance the adaptability of VLMs without additional training. The AWT framework, as depicted in Fig. 2, consists of three critical components: augmenting raw inputs to generate diverse and content-rich views, weighting the significance of these views within each modality, and transporting semantically correlated elements across modalities.
3.2.1
A Augment Raw Inputs
The augmentation process begins with an image X E R3xHxW and the class name set [t aiming transform these inputs into various views that offer different perspectives and details.
For visual augmentation, we apply standard data augmentation including random resized cropping and random flipping to produce a set varied views [+ This set includes augmented images alongside the original (denoted as the 0 index), enriching the input with diverse and multiscale perspectives. An illustrative example is shown in Fig. 2.
To enrich the textual modality, we utilize Large Language Models (LLMs) to generate class descrip- tions. Typical prompts like "Describe {class)." often result in descriptions that are either vague-lacking in specific visual details-or contextually misaligned. For instance, in contexts

---

such as classifying sketches, generic deseriptions of a category may not correspond well with the sketch images. To address this, we adopt two-step, dataset-aware prompt strategy. Initially, we prompt LLMs to generate multiple questions that probe different aspects of the category, which is crucial for eliciting detailed and varied descriptions. To ensure the queries are aligned with the visual content, we incorporate dataset-level description into the initial prompts. Specifically, we start by asking LLMs to "Generate questions to classify images from a dataset, which {dataset descirption). ". Using the datasct-related questions generated from the first step, we proceed to the second step where these questions are combined with the specific class name to obtain tailored descriptions. The set of augmented views for each class L, is denoted as {( M4+ m==1 including an additional view formed by the raw class name. This method ensures both diversity in the descriptions and their relevance to the visual content. More details can be found in Appendix C.2.
3.2.2 Weight Augmented Views
Following augmentation, it is essential to assess the significance of each augmented view, as not all views contribute equally classification. Some views may be critical while others might be less informative or even noisy. To address this variation, we introduce an entropy based weighting mechanism to quantify each view's importance. Our key insight that the impact of a view on classification confidence metric often correlated with accuracy [37]- can serve as a proxy for its importance. A view that significantly enhances classification confidence is considered more vital.
To assess the importance of n-th image view X", we maintain a constant text context and compute the averaged embedding for each class as {z;- = M4T EM+1 e, J where e,"h is the CLIP embedding of 4TT The classification probability p(t | X²) is then calculated using the image embedding I" and text embeddings {2, following Eq. (1). Predictive confidence is then quantified using the entropy formula (t) = - E. p(t X") log logp(t x²"). Lower entropy indicates higher confidence,
allowing us to evaluate view importance through the negative entropy as follows:
exp (- (1)/vo)
ag
Σ exp (-H,(1)/7.)
n 1,...... N +1,
(5)
where 7o is a temperature parameter adjusting the distribution's sharpness.
Similarly, to determine the importance of m-th description for i-th class, iee, IT", we calculate the classification probability p'n (t x"), with the image embedding I and text embeddings {e,") U (e, (e,) The classification entropy is given by Hi, (l) = - E. pm (1) X*) log (4 x²).
We then calculate the importance scores for all descriptions within the i-th class as follows:
exp H (t)/74)
b m
m = M +1,
(6)
exp (+H)()))7))
where % is the temperature parameter. This entropy- weighting mechanism ensures the prioritization of the contextually significant views. By dynamically adjusting the importance based on the direct impact on classification confidence, the augmented view sets can be well-prepared for the optimal transport process.
3.2.3 Transport Across Modalities
Our primary goal is precisely measure the distance between image and its candidate names. Through the augmentation, we have transformed each original image or class name into a set of augmented views. Typically, the distance between these sets measured by simply averaging the embeddings within each set. However, such practice often fails to capture the dynamic correlation across modalities. Consider the scenario depicted in Fig. 2, where specific textual descriptions such as "has a dome-shaped top" might correlate directly with certain image crops. The conventional averaging strategy typically overlooks these intuitive and meaningful correlations.
To address this issue, we propose a novel approach by formulating distance measurement as an optimal transport (OT) problem, which facilitates richer interactions between modalities. We model each view within the V-L. space as a mass located at its embedding position:
N+1
M-1 by
a m and
g
8eg
i=1
5

---

Table Zero-shot image classification. We report top-1 accuracy (%) for each dataset. The "Train"
column indicates whether the methods necessitate additional training (including test-time training).
Numbers in grey indicate that the method was trained on ImageNet and is therefore not zero-shot.
IN-14000
Campany
ECP
SUNP9
SAYYON
Birda 100g
Caz2001011
CURIIC
Arg((a)
Truio
CLIP
66.74
67.44
44.27
88.25
65.48
65.13
93.35
83.65
62.59
23.67
12.01
42.80
82,50
54,90
63.06
CoOp [9]
71.51
68.71
41.92
99.14
64.51
66.55
95.70
85.
64.15
18.47
46.39
41.43
82.91
53.18
63.42
CoCoOp [133]
68.21
62.36
22.94
[15L
70.72
72:23
46.29
90.49
65
68.69
95.55
86 20
67.01
24. 74
48.06
44.06
85.58
57.18
65.75
PLOT [83]
72.48
38.42
90.49
61
68.94
91.32
86.07
61.59
24.84
49.90
36.37
84.30
48.58
63.11
POMP [22]
70.16
72.72
89.05
66.70
68
94.65
86.28
67.27
25 47
52.65
43.94
86.76
56.92
ProVP-Rof [32)
71.14
71.62
45,97
91,58
65
67.72
93.79
86.17
66.29
24.51
51.95
06
TPT [19]
84.67
65.50
44.56
85.71
DIITPT [50]
70.30
70.10
47.00
89.22
67
68.22
92.49
N7.23
65.74
25. 641
43.13
63.91
PromptAlign [52]
71:44
72.39
47.24
90.76
68.
69.47
94,01
86.65
67.54
24.
-1..86
45.28
86.05
57,90
66.42
Self-TPT-x [10]
72.96
71.79
49.35
91.26
68.81
69.50
94.71
85.41
68.18
27.57
51.91
CuPL [63]
69.62
71.30
64.56
89.13
65 29
66.83
92.95
96.11
62.59
24.90
47.84
41.24
86.30
56.28
64.64
[61]
67.12
WaffleLIP [62]
x
68.81
72.35
45.21
89.95
63.57
67.19
94,02
86.08
67.23
25. 39
55.07
43.92
87.04
57.17
63.97
SUS-X-SD|P [65]
69.88
73.81
51,55
90.57
66.13
66.59
93.96
86.08
67.73
28.68
57.49
45.53
87.45
57.11
67.51
71.32
75.07
55.56
72.51
95.54
85.54
71.58
29.22
58.61
48.75
Here, the importance weight of each view, derived from Eqs. (5) and (6), determines the mass allocation. The transportation cost between any two points (e.g., an image and a textual description) quantified using the cosine distance between their embeddings, C = 1 cos( e). which serves as an intuitive measure of semantic similarity. The goal of optimal transport is to minimize the total cost of transporting mass from visual modality into textual modality. Specifically, the distance between the image view set [X*)N++ and i-th class description set [ is redefined as OT problem between a and B, as formulated in Eq. (4). We employ Sinkhorn's Algorithm [69] to efficiently approximate the solution, denoted as P. Consequently, the classification probability can be expressed as:
N+1
n=1
por (4 IX) exp (S,/T)
(8)
j=1 exp (s
By employing the optimal transport framework, ensure that semantically related views receive more attention, enhancing the accuracy and relevance of the classification process.
4 Experiments
4.1 Zero-shot Image Tasks
Datasets. For zero-shot image tasks, we consider image classification and out-of-distribution (OOD) generalization. Our study encompasses 18 datasets that span a wide array of recognition tasks: ImageNet [89], Caltech101 [95] and Caltech256 [101] for generic object recognition, Oxford- Pets [92]. StanfordCars [93], OxfordFlowers [90]. Food101 [96], FGVCAireraft [98], Birdsnap [100] and CUB [102] for fine-grained classification, SUN397 [97] for scene recognition, DTTD [91] for texture classification, EuroSAT [99] for satellite recognition, and UCF101 [94] for action recognition. Besides, four ImageNet variant datasets involved to assess the model's 's capability for OOD gener- alization: ImageNet-A [103]. ImageNetV2 [104]. ImageNet-R [105] and ImageNet-Sketch [106].
Competitive methods. We mainly compare three distinct categories of approaches: 1) Prompt learning methods: These involve additional data to post-train visual or textual prompts, including CoOp [9], CoCoOp [13], MaPLe [15], PLOT+ [83], POMP [22]. 2) Test-time prompt tuning methods: These optimize prompts during inference, such as TPT [49], DiffTPT [50], and Promp- tAlign [52]. 3) Augment-based method: These use LLMSs or diffusion models to augment inputs,
including CuPL [63], VisDesc [61], WaffleCLIP [62], and SuS-X-SD [65].

---

Implementation details. We implemented the AWT framework using the CLIP-B/16 model [1]. Image augmentations include random resized cropping and flipping, and class descriptions are generated via GPT-3.5 [35]. We set the number of augmented images N and descriptions M to 50 each. Dataset-level descriptions are provided in Appendix C. For both visual and textual modalities, we configured the importance distribution temperatures at % 1/2 and Yt- = 1/2. The optimal transport problem is approximated using Sinkhorn's 's Algorithm with an of 0.1 [69]. All experiments are conducted on one NVIDIA 1100-XXM-800GB GPU.
Results. In Tab. 1, we compare AWT with three categories of CLIP adaptation methods: prompt learning, test-time prompt tuning, and existing augmentation-basse methods. Remarkably, with- out additional training, AWT outperforms all ex- isting methods by a significant margin, achiev- ing state-of wef-the-an art performance datasets and surpassing the previous best results by an average accuracy of 2.05%. Further, the out-of distribuuiiiinoo (OOD) generalization capabilities of AWT are detailed in Table 2. Leveraging dataset-aware prompting and dynamic weighting approach that adjusts in real-time during testing, AWT effectively manages complex scenarios encountered in OOD. Consequently, AWT stands out by delivering the highest performance across all four OOD datasets, surpassing the previous arts by an average accuracy improvement of 3.62%.
Table 2: Out of distribution generalization.
Method
IN-A [103]
IN-V2 (104) IN-R [105] IN-K [[0]]001)
CLIP [U]
47.74
60.75
73.98
46.13
57.15
TPT [491
54.77
47.94
60.81
DISTUT [50]
55.68
65.10
75.00
46.80
60.65
CaPL [63]
63.27
49.02
60.02
VisDese [61]
49.07
61.80
75.13
47.997
58.49
13 out of 14
WaMeCLIP [6]]
AWT
60.33
65.15
80.64
$1.60
64.43
4.2 Zero-shot Video Tasks
Setup. Here, we focus on the zero-shot video action recognition task, using three representa- tive datasets: UCF101 194], HMDB51 [107], and
Table 3: Zero-shot video action recognition.
UCF [S]
HMDEST [10]]
Method
K600 [108]
EPI
EP2
EPI
Kinetics-600 [108]. For UCF101 and HMDB51,
ActionCTIP [d]]
714
173
450
48.2F133
623-12
X-CLIP [109]
72.012.3
41.6.1.22
65.21044
we adopt two evaluation protocols: 1) EP1: test model on all 101 UCF classes and 51 HMDB classes [47, 117], and report the top-1 accuracy. 2) FP2: evaluate the model using three official splits and averaging the results each split. The average top-1 accuracy and standard deviation are reported. For Kinetics-600, the top-1 accuracy and standard deviation are reported on three validation sets split by Chen and Huang [118].
al. [110]
69.31:4.2
44.31:2.2
$5.8:0.7
Texs4Vis [111]
79.6
49.8
689+10
AIM AIM (112)
79.0
79.4+110
49.5
50.3+0.8
66.740.5
ST-Adageer [E13]
77.9
77.6
50.3
51.1
6022:118
Visa-CIPP [114]
75.0:10.6
48.610.6
67.4
VIFi-LLP [115]
76.610.7
51.31066
7121110
AdaptFommer [116]
80.5
80.31:1.0
50.5
$1.01008
67.0:04
Open-VCLIP [47]
83.5
83.++1.2
53.2
53.9++122
730+008
FROSTER [48]
85.0
84.8+1.1
54.5
54,8+1.
748+009
AWT
85.4
85.2+0.7
56.1
57,240.9
76.1-06
Implementation details. To model temporal dynamics, we follow Open- [47] to use neighbor-free frame attention and fine-tune CLIP on Kinetics-400 [119]. Note that three test subsets of Kinetics-600 have a disjoint class set compared to Kinetics-400. All AWT configurations are the same for zero-shot image tasks except for the visual augmentation, in which we directly use the different sampled temporal and cropped video frames.
Results. In Tab. 3, we present comparison of our AWT with existing CLIP-based zero-shot video action recognition methods. Although AWT was not originally tailored for video tasks, it sets new records in this domain, outperforming the recent state-of-the- method, FROSTER, by 1.6% and 2.4% on HMDB51, and by 1.3% on Kinetics-600. These results suggest that our AWT framework could be effectively extended to video understanding tasks.
4.3 Fhw-shot Image Tasks
Setup. We assessed the few-shot transfer capabilities of our method across 11 datasets: Ima- geNet [89], Caltech101 [95], OxfordPets [92], StanfordCars [93], OxfordFlowers [90]. Food101 [96], FGVCAircraft [98], SUN397 [97], DTD [91], EuroSAT [99], and UCF101 [94]. We trained our model using 1, 2, 4, 8, and 16 shots. Results are averaged over three runs.
Implementation details. All AWT configurations are the same as zero-shot image tasks. this task, we introduced multi-modal Adapter for efficient learning, inserted after each Multi-Head
7

---

Average datasets
ImageNat
Caltech101
OxfordPats
85
97-
Top-1 Artuweyy 1ho
74
Top3 Accuresty (26)
topa Accurecy (R0
8D
Accunes
75
92-
DeOa
70-
Coop
Promuttic
94
Premussic
30-
APET
AFE
ARST
AFCT
66
88
012
per
per
StanfordCars
Food101
FGVCAircraft
UCF101
1o Accurscy 198)
85
Toge1 Accuracy (SQ
500
Topa Accorney 199)
80
Top1 Accurscc
80-
75
ILT
FOOT
70
84
AFCS
PrompiSCC
30
A757
70
PrampiSiic
65
20
MNT
65
12
AMT
16
16
012
shoss plrclass
shots per class
shots per class
4 shotts per class
Figure 3: Few-shot image classification. We present the average accuracy across 11 datasets and specific accuracy for three datasets. Numerical values can be found at Tab. 11.
Table 4: Ablation experiments across 18 image datasets. The default configuration is colored grey
(a) Main component analysis: Aug- (b) Number of augmented image (c) Number of generated class
Me)t(A), Weight(W) and Transport(T).
views N.
descriptions M.
Step
OOD
Avg.(14)
N
OOD
Avg.
M OOD
Av4.
Raw inputs
57.15
63.06
2
61.40
68.57
2
61.58
66.67
A(Img.)
56.77
63.40
5
62.59
68.98
5
62.87
67.55
A(Name)
59.76
67.36
10
63.31
69.16
10
63.42
68.53
A(Img. -Nmme)
59.92
67.13
25
64.13
69.33
25
64.07
68.98
A+T
60.48
67.80
50
64.43
69.59
50
64,43
69,59
A+W+T
64.43
69.59
100
64.54
69.52
100
64.46
69.73
(d) Different prompt methods for LLMs.
(c) Weighting temperature "/.
(f) Weighting temperature "t.
Method
OOD
Av4.(14)
OOD
Avg-(14)
%
OOD
Avg.(14)
VisDese [61]
63.55
67.78
100
61.07
68.71
100
63.97
68.39
CuPL (base) [63]
63.57
68.87
63.54
69.43
1
64.47
69.27
CuPL (full) [63]
63.88
69.33
1/2
64.43
69.59
1/2
64.43
69.59
1/3
64.42
69.37
1/3
64.08
69.50
Ours
64.43
69.59
1/4
64.38
69.19
1/4
63.80
69.38
Self-Attention and MLP block in every layer. We adopt a distillation technique similar to [21] to prevent overfitting. Further details about the Adapter available in Appendix
Results. In Fig. we compare the performance AWT with existing methods in the few-shot transfer learning task. Impressively, AWT surpasses the previous otate-of-the-ar by average accuracies of 2.76%, 2.16%. 1.62%, 1.57%, and 1.75% for 2, 4. 8, and 16 shots respectively. Notably, on the ImageNet dataset, AWT significantly outperforms all prior methods. While PLOT+ also utilizes optimal transport, is limited local image features and class names, neglecting the multiscale image perspectives and the rich textual semantics, leading to suboptimal transfer capabilities. In contrast, AWT leverages diverse augmented views, effectively maintaining intra-modal importance trade-offs while establishing dynamical cross-modal correlations, achieving superior few-shot performance.
4.4 Ablation Study
Main component analysis. In Tab. 4a, we analyze the key components of AWT. Initially, we augment raw inputs and apply basic ensembling. The results (rows two and four) show that directly augmenting images is ineffective, likely due to background image crops. Conversely, textual enhance- ments significantly boost performance, thanks to our carefully designed prompt strategies for LLMs. We then shift from basic ensembling to optimal transport (OT), leading to consistent improvements across two tasks. However, the full potential of OT is not realized due to ineffective mass (i.e.,

---

59
78
Avgrage Accuracy (%)
75
Zero Shat-Shat
CLIP w/ AWT
58
ALIGN
SigUP
w/ AWT
EVA-CUP
70
57
Average Accuracy (96
74
65
Average Accuracy
56
Averaae Accuracy
74
72
72.7
55
60
70
55
-3
52
70-
68
A8.1
51
60
RN50
RN1101
VIT-8/32 VI-60/16 4 U14g333
CLIP Backbone
50
ADIGN
SigLiP
EVA-CLIP
(a) AWT across different backbones.
(b) AWT generalizes different VLMs.
Figure 4: Versatility analysis of AWT. Average top-1 accuracy (%) on 18 image datasets is reported.
importance) weighting. By incorporating our entropy-based weighting method, which accurately assesses the importance of each view, we again achieve substantial performance gains.
Number of augmentation views. We present the study on the augmented view quantities for both visual and textual sides in Tabs. 4b and 4c, respectively. The results clearly demonstrate that performance tends to increase with the number of views. Our findings suggest that about 50 views per modality are sufficient to achieve decent performance. The number of augmentation views is crucial for AWT's effectiveness. Given that AWT is augmentation-driven, this correlation is intuitive. However, increasing the augmented view quantities can also lead to higher computational costs during inference, we also include an
trade-off analysis in Fig. 9.
LLM prompting strategy. We evaluated the effectiveness of our LLM prompting strategy, detailed Tab, 4d. Our method compared with two established approaches: VisDese [61] and CuPL [63]. VisDesc uses uniform prompt template across different datasets, while CuPL employs tailored, dataset-specific manual prompting strategy, enriching context for LLMs. We developed a refined two- step process that enhances context comprehension through dataset-level descriptions and increases diversity by utilizing chain-of thouuhtt queries. Our strategy consistently outperforms the existing methods in both evaluated tasks.
Temperature in weighting, We evaluated our entropy-based weighting method by conducting an ablation study on the sof tmax function's 's temperature parameter (see Eqs. (5) and (6)). A higher temperature creates a more uniform importance distribution. The findings for both modalities are presented in Tabs. 4e and 4f, respectively. Our results reveal that a very high temperature (e.g.. 100) leads to suboptimal performance, likely due to insufficient emphasis on contextually significant views. Conversely, lowering the temperature enhances focus on these important views, improving performance. Empirically, temperature of 1/2 has been identified as optimal for both modalities. For clearer understanding of our weighting strategy, visualizations are provided in Appendix A.
4.5 Versatility Study
Our AWT is applicable any VLM using dual encoders map images and text into joint space with appropriate distance metrics (e.g.. cosine similarity). Therefore, it is crucial to assess AWT's effectiveness across various scenarios. We conduct evaluations with both ResNet [120] and ViT [121] architectures, explore AWT's scalability from ViT-B/32 to ViT-L/14@ 336, and assess its generalizability across three VLMs: ALIGN [6]. SigLIP [122], and EVA-CLIP [123]. We conduct experiments across 18 image datasets and present the results in Fig. 4. Our findings reveal that AWT
consistently achieves performance gains in all tested scenarios, highlighting its broad applicability.
4.6 Failure Case Analysis
Although AWT has shown success across various datasets and tasks, we identified certain limitations when applying to the CIFAR datasets. As detailed in Tab. 5, AWT resulted in performance declines of 0.91% % and 0. 17% on two CIFAR datasets, respectively. To investigate this issue, we analyzed the images produced by the transformations used our study. We discovered that with low-resolution images, such 32 x 32 pixels, the random resized crop operation tends to overly blur the images,

---

Original Image
Random Resized Cropping & Flipping
Diffusion Model Generated Images
CIFAR-100 CIFAR-10
Figure 5: Comparison of image augmentation techniques on low-resolution images. We present images from the CIFAR-10/100 datasets, where each image is 32 x 32 pixels. The comparison includes images generated by traditional image transformations and DALL-E
Table 5: Failure case analysis. We focus on low-resolution datasets CIFAR-10 and CIFAR-100 and evaluate the effectiveness of AWT when equipped with different image augmentation techniques.
Method
Image augmentation
CIFAR10 [124]
CIFARI00 [124]
CLIP [1]
90. 16 (baseline)
67.78 (baseline)
AWT
Random resized crop and flip
89.25 (0.91)
67.61 (0.17) 4
AWT
DALL-E 2 [44]
92.30 (2.14)
68.69 (0.91)
obscuring the objects within them, as illustrated in Fig. 5. To address this, we integrated a diffusion model, specifically DALL-E 2 [44], substitute for traditional data augmentations. Fig. 5 shows examples enhanced image views generated by DALL-E 2, which produce sharper images and offer variety of perspectives. By incorporating this advanced technique into the AWT framework, have significantly improved its performance. The updated benchmark results, presented in Tab. 5, demonstrate that AWT now consistently achieves performance gains over baselines.
Conclusion
In this paper, have introduced the AWT (Augment, Weight, then Transport) framework. designed to enhance the transferability of pre-trained vision-language models (VLMs). Rather than using raw images and class names directly, approach enriches the inputs by augmenting them with diverse visual perspectives and detailed class descriptions. We further develop an entropy-based weighting strategy dynamically prioritize these augmented views and employ optimal transport measure the cross-modal distance in the structured visual-language space. The AWT framework not only boosts the zero-shot performance of VL.Ms without the need for additional training but also facilitates few-shot transfer learning via an integrated multimodal adapter module. Our evaluations across four challenging tasks demonstrate that AWT significantly outperforms existing wftc--of-hhe-ad methods.
This work is supported by the National Key R&D Program of China (No. 2022ZD01609000) National Natural Science Foundation of China (No. 620761 19), the Funda- mental Research Funds for the Central Universities (No. 0202143801 19), the Nanjing University- China Mobile Communications Group Co., Ltd. Joint Institute, and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
Acknowledgments.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-876 8763. PMLR,
2021. 1, 3, 10, 19,23 23
[2] Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang. Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative
learning. arXiv preprint arXiv:2212.03113
2022.
[3] Yi Wang. Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan
Xu, 7un Wang. al. Intervviee22: Scaling video foundation models for multimodal video understanding.
arXiv preprint arXiv:2403. /5377,, 2024.
[4] KunChang Li, Yinan He, Yi Wang. Yizhuo Li. Wenhai Wang. Ping Luo, Yali Wang, Limin Wang, and
Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0660 06355, 2023.
[5] Kunchang Li. Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zan Wang. Jilan Xu, Guo Chen, Ping Mybench: A comprehensive
Luo,
video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-22206 2024,
[6] Chao Jia, Yinfci Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021. 2,9
[7] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learming, pages 12888- 12900. PMLR. 2022.
[8] lean-Baptist Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson. Karel Lenc, Arthur Mensch, Katherine Millican. Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 15:23716-2377
2022. 1
[9] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language
models. International Journal of Computer Vision (IJCV), 2022. 1, 3, 6, 20, 21, 23
[10] Yuhan Zhu, Guozhen Zhang, Chen Xu, Haocheng Shen, Xiaoxin Chen. Gangshan Wu, and Limin Wang- Efficient test-time prompt tuning for vision- laiiun- models. arXiv preprint arXiv:2408.05775 2024. 3.
6
[11] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5205552001551225 2022.
[12] Feng Wang. Manling Li, Xudong Lin, Hairong Lv, Alex Schwing, and Heng Ji. Learning to decompose visual features with latent textual prompts. In The Eleventh International Conference on Learning
Representations, 2022. 3
[13] Kaiyang Zhou. Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for Lingon-Language models. In JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR).
2022. 3, 6, 20,21 21
[14] Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge-guide context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6757-6767, 2023. 20
[15] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 191 1311122, 2023. 6, 20
[16] Adrian Bulat and Georgios Tzimiropoulos. Lasp: Text-to-text optimization for language-awaa soft prompting of vision & language models. Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 2 2023.
[17] Dongjun Lee, Scokwon Song, Jihee Suh, Joonmycong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1401-1411, 2023. 202.
[18] Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-awar prompt tuning for generalizable vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 15670-15680. 2023.
[19] Hantao Yao, Rui Zhang, and Changsheng Xu. Tep: Textual-based class-aware prompt tuning for vasual-language model. arXiv preprint arXiv:2311.188 2023.
[20] Ji Zhang, Shihan Wu, Lianli Gao, Hengtao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning.
arXiv preprint wtXiv:2309.07453 2023.
[21] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz. Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15190- 15200, 2023.
8, 21
[22] Shuhuai Ren, Aston Zhang, Yi Zhu, Shuai Zhang. Shuai Zheng, Mu Li, Alexander Smola, and Xu Sun. Prompt pre-training with twenty- thousand classes for voca-vocy visual recognition. Advances in
11

---

Neural Information Processing Systems, 36, 2024, 3, 6. 20
[23] Zixian Guo, Bowen Dong, Zhilong Ji. Jinfeng Bai, Yiwen Guo, and Wangmeng Zuo. Texts as images in prompt tuning for multi-label image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2808-28 2817, 2023, 3
[24] Muhammad Uzair Khattak, Muhammad Ferjad Nacem, Muzammal Nascer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision-language models. arXiv preprint arXiv:2401.000 00 2024. 3
[25] Cheng Shi and Sibei Yang. Logoprompt: Synthetic text images can be good visual prompts for vision- language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2932-2941, 2023. 3
[26] Zhengbo Wang, Jian Liang. Ran He, Nan Xu, Zilei Wang. and Tieniu Tan. Improving zero-shot general- ization for clip with synthesized prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3032-3042, 2023.
[27] Beier Zhu, Yulci Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15659-15669, 2023. 3,20
[28] Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.0366
2022. 3
[29] Yuhang Zang, Wei Li. Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language
prompt learning. arXiv preprint arXiv:2210007225, 2022. 3
[30] Sifan Long, Zhen Zhao, Junkun Yuan. Zichang Tan, Jiangjiang Liu, Luping Zhou, Shengsheng Wang, and Jingdong Wang. Task-oriented multi-modal mutual leaning for laiionl models. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 21959-21999 1969, 2023. 3
[31] Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang. Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, meta-prompt learning for generalizable vision- language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
Tat-Seng Chua, and Yueting Zhuang.
2551-2562, 2023. 3
[32] Chen Xu, Yuhan Zhu, Haocheng Shen, Boheng Chen, Yixuan Liao, Xiaoxin Chen, and Limin Wang. Progressive visual prompt learning with contrastive feature re-formation. International Journal of Computer Vision, pages 1-16, 2024. 3, 6,
[33] Chen Xu, Yuhan Zhu, Guozhen Zhang. Haocheng Shen, Yixuan Liao, Xiaoxin Chen, Gangshan Wu, and Limin Wang. Dpl: Decoupled prompt learning for vision-language models. arXiv preprint
arXiv:2308. 10061, 2023. 3,
[34] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.
Advances in neural information processing systems, 33:1877-1900 2020.
[35] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022. 7, 20
[36] OpenAL. Gpt-4 technical report. ArXiv, abs/2303.08774 2023. URL https: //ap: semanti iescaana..ccco.......
org/ sng/CorpusIDD ID: 257532815. 2, 20
[37] Dequan Wang. Evan Shelhamer. Shaoteng Liu. Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time
adaptation by entropy minimization. In International Conference on Learning Representations, 2020. 2, 5
[38] Gaspard Monge. Mémoire sur la théoric des déblais et des remblais. Mem. Math. Phys. Acad. Royale
Sei., pages 666-704, 1781. 2,
[39] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning. 1 (556))33556607 2019. 2, 3
[40] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
[41] Chengjian Feng. Yujie Zhong. Zequn Jie. Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open- vocabulary detection using uncurated images. In European Conference on Computer Vision, pages 701-717. Springer, 2022. 3
[42] Xiuye Gu, Tsung-Yi Yi Lin, Weieheng Kuo, and Yin Cui. Zero-shot detection via vision and language knowledge distillation. arXiv preprint arXiv: 20te:2200.//300 1, 2(3):4, 2021.
[43] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14084-14093, 2022. 3
[44] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hicrarchical text cexticonditional image generation with clip latents. arXiv preprint arXiv:2204.0611
1(2):3, 2022. 3, 10
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolutien image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022. 3
[46] Mengmeng Wang, Jiazheng Xing, and Yong Liu, Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.084/ 2021. 3,7
12

---

[47] Zejia Weng, Xitong Yang, Ang Li. Zuxuan Wu, and Yu-Gang Jiang. Open-velip: Transforming clip to an open-vocabulary video model via interpolated weight optimization. In ICML, 2023. 7
[48]
Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. Froster: Frozen clip is a strong teacher for open- vocabulary action recognition. arXiv preprint arXiv:2402.0322
2024. 3,
[49]
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei
Xiao. Test-time prompt tuning for shot- generalization in vision- Saiionlan models. Advances in
Neural Information Processing Systems, 15:14474-14289, 2022. 6, 7, 20, 23
[50]
Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation
with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International
Conference on Computer Vision. pages 2704-2714, 2023. 6.
[51] Xiaosong Ma. Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. Advances in Neural Information Processing Systems. 36, 2024.
[52] Jameel Abdul Samadh, Mohammad Hanan Gani, Noor Hussein, Muhammad Uzair Khattak, Muham- mad Muzammal Nascer, Fahad Shahbaz Khan, and Salman H Khan. Align your prompts: Test-time
prompting with distribution alignment for zero-shot generalization. Advances in Neural Information
Processing Systems, 36, 2024. 20
[53] Yabin Zhang, Wenjie Zhu, Hui Tang. Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang, Dual memory networks:
A versatile adaptation approach for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 287 18 11181822 2024.
[54] Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb EI Saddik, and Eric Xing. Efficient test-time adaptation of languaee models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14162-14171, 2024. 3
[55] Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, pages 1-15, 2-23. 3
[56] Renrui Zhang. Wei Zhang. Rongyao Fang. Peng Gao, Kunchang Li. Jifeng Dai, Yu Qiao, and Hongsheng
Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European Conference on Computer Vision, pages 493-510. Springer, 2022.
[57] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang. Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2605-2615, 2023.
[58] Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, and Manabu Okumura. Ad- vancing cross-domain discriminability in continual learning of vison-language models. arXiv preprint arXiv:2406. 18868, 2024.
[59] Lingxiao Yang. Ru- Yuan Zhang, Yanchen Wang, and Xiaohua Xie. Mma: Multi-modal adapter for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23826-23837, 2024. 3
[60] Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayaman, and Jiaping Zhao. Improving zero-shot generalization and robustness of multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11093-11101, 2023. 3
[61] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In
The Eleventh International Conference Learning Representations, 2022. 3, 6, 7, 8, 9,
[62] Karsten Roth, Jac Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15746-15751 2023, 6,7
[63] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pages 15691-15701, 2023, 6, 7, 8, 9,
[64] Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In International Conference on Machine Learning, pages (6342-263622 PMLR. 2023. 3
[65] Vishaal Udandarao, Ankush Gupta, and Samuel Albanic. Sus-x: Training-free name-only transfer of vision-language models. Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 2725-2736, 2023. 3, 6
[66] Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng
Li. Prompt. generate, then cache: Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15211- 1- 15222, 2023. 3
[67] Abdelwahed Khamis, Russell Tsuchida, Mohamed Tarek, Vivien Rolland, and Lars Petersson. Scalable optimal transport methods in machine learning: A contemporary survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024. 3
[68] Yossi Rubner, Carlo Tomasi, and Leonidas Guihas. The earth mover's 's distance as metric for image retrieval. International journal of computer vision, 40:99-121, 2000. 3
13

---

[69] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advunces in neural
information processing systems, 26, 2013. 3.6,7 7
[70] Zheng Ge, Songtao Liu, Zeming Li. Osamu Yoshie, and Jian Sun. Ota: Optimal transport assignment for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pages 903-312, 2021.
[71] Henri De Placn, Picrre-François De Plaen, Johan AK Suykens, Mare Proesmans, Tinne Tuytelaars, and Luc Van Gool. Unbalanced optimal transport: A unified framework for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3198-3207, 2023.
[72] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamoniy. Optimal transport for domain adaptation. IEEE trunsactions on pattern analysis and machine intelligence, 39(9):1833-188 2016. 3 [73] Yuguang Yan, Wen Li, Hanrui Wu, Huaqing Min, Mingkui Tan, and Qingyao Wu. Semi-supervised optimal transport for heterogeneous domain adaptation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2969-2975, 2018.
[74] Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty, Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings
the European conference on computer vision (ECCV), pages 447-463, 2018. 3
[75] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on muchine learning, pages 214-223. PMLR, 2017.
[76] Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport.
In International Conference on Learning Representations, 2018
[77] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017. [78] Antoine Liutkus, Umut Simsekli, Szymon Majewski, Alain Durmus, and Fabian-Robert Stöter. Sliced- wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions. In Interna-
tional Conference on Machine Learning, pages 4104-4113. PMLR, 2019. 3
[79] Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence as an optimal transport problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4463-4472, 2020. 3
[80] Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow on point clouds guided by optimal transport. In European conference on computer vision, pages 527-544. Springer, 2020. 3
[81] Zhengyang Shen, Jean Feydy, Peirong Liu, Ariel H Curiale, Ruben San Jose Estepar, Raul San Jose Es- tepar, and Mare Niethammer. Accurate point cloud registration with robust optimal transport. Advances in Neural Information Processing Systems, 34:5373-5389, 2021.
[82] Ruibo Li. Guosheng Lin, and Lihua Xie. Self seef-point-foor ffow: Sele-supervised scene flow estimation from point clouds with optimal transport and random walk. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15586, 2021.
[83] Guangyi Chen, Weiran Yao, Xiangehen Song, Xinyue I., Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language Learning Representations, 2022. 3, 6, 20, 21, 23
models. In The Eleventh International Conference on
[84] Kwanyoung Kim, Yujin Oh, and Jong Chul Ye. Zegot: Zero-shot segmentation through optimal transport of text prompts. arXiv preprini
221.1
[85] Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen, and Hanwang Zhang. Tuning multi-mod token-level prompt alignment across modalities, Advances in Neural Information Processing Systems, 36, 2024.
[86] Tongjia Chen, Hongshan Yu, Zhengeng Yang. Zechuan Li, Wei Sun, and Chen Chen. Ost: Refining text knowledge with optimal spatio-temporal descriptor for general video recognition. arXiv preprint arXiv:2312.00099 2023. 3
[87] Kaihua Zhang, Zicheng Zhao, Dong Liu, Qingshan Liu, and Bo Liu. Deep transport network for unsupervised video object segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9781-8790, 2021. 3
[88] L. Kantorovich. On the translocation of masses. Journal of Mathematical Sciences, 133, 03 2006. doi: 1011007/11958-00000000
4
[89] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pottern recognition, pages
248-255. Icce, 2009. 6, 7, 20, 21
[90] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image procesxing, pages
722-729. IEEE, 2008 6, 7.21 21
[91] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606 3613, 2014. 6, 7, 20, 21
[92] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 20/2 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEFF, 2012. 6, 7, 21
[93] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference computer vision workshops, pages
14

---

554-561, 2013, 6, 7, 20, 21
[94]
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions
classes from videos in the wild. arXiv preprint urXiv:1212.0402 2012. 6, 7, 20, 21
[95] Li Fei-Fei. Learning generative visual models from few training examples: An incremental bayesian
appreach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition
workshop, pages 178- 178. IEEE, 2004, 6, 7, 20, 21, 23
[96] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014. 6, 7, 20, 21
[97] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to Z0o. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010. 7.
[98]
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual 1306.5/51, 2013. 6, 7,
classification of aircraft. arXiv preprint
[99] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 12(7)22217-2233 2019. 6, 7, 21, 23
[100] Thomas Berg. Jiongxin Liu, Seung Woo Lee, Michelle L. Alexander, David W. Jacobs, and Peter N. Belhumeur. Birdsnap: Large-scale fine-graines visual categorization of birds. In Proc. Conf. Computer
Vision and Pattern Recognition (CVPR), June 2014. 6
[101] Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007. 6, 23
[102] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd
birds-200-2011 dataset. 2011. 6
[103] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271. 2021. 6. 20,23
[104] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR,
2019. 6, 20, 23
[105] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribation generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021. 6, 7, 20, 23
[106] Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems. 32, 2019. 6. 7.
20, 23
[107] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdh: a large video database for human motion recognition. In 201 International conference computer vision, pages 2556-2563. TEEF, 2011. 23
[108] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman, A short note about kinetics-600. arXiv preprint 18Xiv 130.:08800.0140, 2018. 7.23
[109] Bolin Ni. Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng. Jianlong Fu, Shiming Xiang. and Haibin Ling. Expanding language-image pretrained models for general video recognition. In European Conference on Computer Vision, pages 1-18. Springer, 2022. 7
[110) Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xic. Prompting visual-language models for efficient video understanding. In European Conference on Computer Vision, pages 105- 124. Springer,
2022. 7
[111] Wenhao Wa, Zhun Sun, and Wanli Ouyang. Revisiting classifier: Transferring vision-language models for video recognition. Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 2847-2855, 2023. 7
[112] Taojiannan Yang. Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video action recognition. In The Eleventh International Conference on Learning Representations, 2022.
[113] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-effiiinnt image-to-vidco transfer learning. Advances in Neural Information Processing Systems, 35:26462-26477,
2022." 7
[114] Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23034-23044, 2023. 7
[115] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6545-6554, 2023. 7
[116] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664- 16678, 2022. 7
15

---

[117] Biagio Brattoli. Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking zero- shot video classification: End-to-end training for realistic applications. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 4613-4623, 2020. 7
[118] Shizhe Chen and Dong Huang. Elaborative rehearsal for zero-shot action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13638- 13647, 2021. 7
[119] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vjjayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv preprint 1rXiv:1705.0690
2017. 23
[120] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[121] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020. 9
[122] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
11975-11986, 2023. 9
[123] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. 2333:22003.1389, 2023. 9
[124] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009, 10 Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 09685,
[125]
2021. 19
[126] Shuhuai Ren, Lei Li. Xuancheng Ren, Guangxiang Zhao, and Xu Sun. Delving into the openness of clip. Findings of the Association for Computational Linguistics: ACL 2023, pages 9587-9606, 2023. 20 [127] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023. 20
[128] Blya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2016. 22
[129] Hya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139-1147.
PMLR, 2013. 22
[130] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.0840 08415,
2016. 22
[131] Guozhen Zhang, Jingyu Liu, Shengming Cao. Xiaotong Zhao, Kevin Zhao, Kai Ma, and Limin
Wang. Dynamic and compressive adaptation of transformers from images to videos. arXiv preprint arXiv:2408.0668
2024. 25
[132] Xinhao Li and Limin Wang. Zeroi2v: Zero-cost adaptation of pre-trained transformers from image to video. arXiv preprint arXiv:2310001324 2023. 25
[133] Guozhen Zhang, Yuhan Zhu, Haonan Wang. Youxin Chen, Gangshan Wu, and Limin Wang. Extracting motion and appearance via fnter-frame attention for efficient video frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5682-5666 (692,5692, 2023. 25
[134] Guozhen Zhang, Chunxu Liu, Yutao Cui, Xiaotong Zhao, Kai Ma, and Limin Wang. Vfimamba: Vidco frame interpolation with state space models. arXiv preprint arXiv:2407.023:
2024.
[135] Chunxu Liu, Guozhen Zhang, Rui Zhao, and Limin Wang. Sparse global matching for video frame interpolation with large motion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19125-19134, 2024. 25
[136] Mengeheng Lan, Xinjiang Wang. Yiping Ke, Jiaxing Xu, Litong Feng, and Wayne Zhang. Smooseg: Smoothness prior for unsupervised semantic segmentation. Advances in Neural Information Processing Systems, 13:31333-1113 11373, 2023. 25
[137] Mengcheng Lan. Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang, Proxyclip: Proxy attention improves clip for open-vocahulary segmentation. arXiv preprint a4Xiv:2408. 04883, 2024.
[138] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense rision-languee inference. arXiv preprint ar 240: 12442,
2024. 25
[139] Yuhan Zhu, Guozhen Zhang. Jing Tan, Gangshan Wu, and Limin Wang. Dual detrs for multi-label temporal action detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18559-18569, 2024. 25
[140] Chen Xu, Tianhui Song, Weixin Feng, Xubin Li, Tiezheng Ge, Bo Zheng, and Limin Wang. Accelerating image generation with sub- -ath linear approximation model. arXiv preprint arXiv:2404.1 13903, 2024. 25 [141] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847,
2023. 25
16

---

Appendix
The appendix provides supplementary material including additional visualization results, comparative studies, ablation experiments, detailed methodologies, examples of failure cases, and discussion on the limitations of our research. The contents are structured as follows:
Visualizations of weighting and transportation processes (Appendix A).
Extended experiments and ablation studies (Appendix B).
Method details and experimental procedures (Appendix C).
Discussion on societal impacts (Appendix D).
Discussion on limitations and future research directions (Appendix E).
A
Visualization
Candidate
Classes (animals)
Augmented Views
Alpaca
Horse
Dog Lion
Importance
0.303
0.031
0.258
0.310
0.098
Candidate
Classes
(nature)
Augmented Views
Mountain
Deser!
Ocean Jungle
Importance
0.097
0.324
0.178
0.060
0.341
Ineland
(a)
Candidate
Classes (animals)
Augmented Views
Aloaca
Dog Lion
Importance
0.304
0.022
0.022
0.326
0.326
Candidate
Classes (nature)
Augmented Views
Mountain
Desert
Ocean
Junglo
Importance
0.228
0.262
0.266
0.149
0.095
Iceland
(b)
Figure 6: Visualization of weighting image views. We show the weights assigned to the same image view set under varying candidate class names. Our dynamic weighting strategy effectively allocates importance to contextually relevant image views.
A.1 Ebarop Weight Strategy
Pre-trained VLMs often struggle to adequately focus on contextually significant features due to their open-set design, which requires processing every element within an image. To overcome this limitation, we introduce an entropy-based weighting strategy that dynamically evaluates the importance of each image view based on the prediction entropy. This method allows us prioritize relevant views while diminishing the impact of less pertinent ones.
Visualization of weighting image views. We have included visualizations to demonstrate the effectiveness of our entropy-baa based weighting strategy in managing image views, as shown in Fig. 6.
17

---

Through these visualizations, we observe two primary benefits: 1) Priority to Significant Views: The strategy effectively prioritizes image views that are intuitively significant, while efficiently excluding irrelevant background crops. 2) Dynamic weight assignment: In different contexts, such as varying candidate class names, the strategy dynamically assigns weights, emphasizing views that are contextually important.
Importance
Chihuahua
0.1241
They
body.
0.3889
They displey excitement by jumping around.
0.1146
Their fur color include tan, brown, white, black.
0.3725
They may have a wide-eyed look with their eare perked
Bengal
0.2234
They distinctive
spotted pattem.
0.2453
They might
0.2130
They have large, rounded eyes that can be gold or green.
0.3183
Their legs are long and well-proportioned to its body.
Samoyed (GT)
0.0546
tend have sturdy, straight legs They thick, fluffy ears are erect and triangular, covered in fur. The nose black, contrasting with its white fur.
0.3794
0.1452
0.4207
Figure 7: Visualization of weighting class descriptions.
Visualization of weighting class descriptions. The strategy not only enhances image view man- agement but also optimizes the handling of textual descriptions. In Fig. 7, observe the following: 1) For descriptions related to the ground-truth class, our strategy prioritizes key textual elements while effectively filtering out irrelevant content. 2) For non-matching classes, the importance levels are relatively uniform, indicating a general irrelevance to the specific image content.
A.2 Optimal Transport
Bengal's spotted fur.
0.0450
0.0718
0.0458
0.0032
0.0843
Bengal's triangular ears.
0.0389
0.0461
0.0530
0.0906
0.0214
Bengal's oval-shaped eyes.
0.0255
0.0327
0.0592
0.1033
0.0293
Bengal's energetic legs.
0.0907
0.0494
0.0420
0.0029
0.0650
Figure 8: Visualization of cross-modal correlations captured by optimal transport.
After augmenting the input image and class names create diverse views, there is potential for these views to exhibit direct and meaningful correlations across modalities. For instance, image crops that focus on the eyes of a could closely correlate with textual descriptions concerning the eyes. In our main paper, we introduce the use of optimal transport (OT) to effectively handle these correlations. illustrate this, we provide a visualization of the meaningful correlations captured by the OT approach. As depicted in Fig. 8, the visualization of the transport plan clearly shows that image and text pairs with direct semantic relationships are given priority. Conversely, pairs that are semantically irrelevant typically disregarded by the transport plan.
18

---

B Additional Experiments
B.1 Error bar analysis
Table 6: Error bar analysis on 18 image datasets.
Method
OxfordFlowers
DTD
OxfordPets
StanfordCars
UCF101
Caltech101
CLIP
67.44
44.27
88.25
65.48
65.13
93.35
74.42-0.09
55.30.10.03
92.30110.34
69.83110.10
72.4510.26
94.94_0.11
Method
Food101
SUN397
FGVCAireraft
EuroSAT
Birdsnap
Caltech256
CUB
83.65
23.67
42.01
42.80
82.50
54.90
AWT
85.51+0.03
70.26+00.23
28.34=0.26
61.00 +.03
48.41+0.13
88.79:0.19
60.36=00.20
Method
ImageNet-A
ImageNet
Imtge
Skatehht-Sketch
ImageNet
00D
Avg.(14)
CLIP
47.74
60.75
73.98
46.13
66.74
57.15
63.06
AWT
60,42:10.2
64.86.10.21
80.27:10.09
51,6411.03
71.33:10.02
64.30.10.03
69.52:10.08
We conducted an analysis of error bars across 18 image datasets, performing three runs each to ensure robust statistical evaluation. The results, which include both the mean and the standard deviation of the top-1 accuracy, are presented Tab. 6. Overall, AWT demonstrates robust performance, exhibiting low variability across most datasets.
B.2
Performancc--fficiiney trade-off
70
Average Accuracy (%)
69
Average Accuracy (%)
82
AWT(Ours)
AWT(Ours)
CLIP
CoOp
66
CoOp
CoCoOp
65
TPT
POMP
78
MaPLe
PLOT+
DifFTPT
PromptSRC
PromptAlign
APE-F
7153
222
74
FPS (logarithmically scaled) 10
100
FPS (logarithmicaall scaled)
comparison. AWT is assessed using 2, 5, 10, 25, and 50 image views to present its trade-off between performance and computational efficiency. We show the results on both zero-shot (left) and few-shot (right) tasks.
Figure 9: Performance-efficiene
Our AWT introduces additional computational costs during inference by utilizing multiple views. Although text embeddings can be pre-computed as suggested in [1], the primary expense arises from processing additional image views. In our study, explore the trade-off between performance and
efficiency by employing 2, 5, 10, 25, and 50 augmented views of the input image. We assess this trade-off by reporting both the frames per second (FPS) and the average accuracy in zero-shot and few-shot settings, as shown in Fig. 9.
The results indicate that even with a limited number of views, AWT can substantially outperform existing methods. As the number of views increases, there a consistent enhancement in performance, albeit at the expense of reduced inference speed. This analysis enables practitioners tailor AWT application to real- world scenarios by selecting an optimal balance between accuracy and processing speed, based on specific requirements.
In the few-shot settings, we incorporate two additional adapter modules at each transformer layer to facilitate transfer learning. This configuration initially makes our method slower than other prompt learning approaches, likely due to adapters requiring sequential processing, whereas prompts are processed in parallel. Considering the computational costs, propose that AWT could be more efficiently transferred using test-time zero-cost methods like LoRA [125], which allow for structural reparameterizatioo after training.
19

---

B.3 Study on the multi-modal adapter
Table 7: Fine-tuned modalities in few-shot learning. The model's performance evaluated with the adapter module applied to visual, textual, or both modalities. Performance metrics are reported for the ImageNet test set.
Adapter
1shot
4shot
16shot
Visual
71.66
72.52
74.34
Textual
71.24
72.99
74.54
Multi-modas 72.14
73.41
75.17
In this study, explore the impact of applying multi-modal adapters to both visual and textual modalities. We conducted experiments using an unimodal adapter either on the visual or textual side and analyzed their performance on the ImageNet dataset. The results detailed in Tab. 7. Our findings indicate that when training samples are extremely limited, such as in a 1-shot scenario, fine-tuning the visual adapter yields more benefits. However, the number of training samples increases, fine-tuning the textual adapter becomes more advantageous. This shift is likely due to the need to reduce the similarity between class embeddings suggested in [126]. Despite these variations, our results ultimately show that fine-tuning both modalities simultaneously delivers the best overall performance.
B.4 Domain generalization
Table 8: Domain generalization. All methods are trained on 16-shot ImageNet and directly evaluated on four datasets with domain shifts. We report the top-1 accuracy (%) each method.
Method
IN-IK [89]
IN-A [103]
IN-V2 [104]
IN-R [105]
IN-K [106]
00D
CoOp [9]
71.51
49.71
64,20
75.21
47.99
59.28
CoCoOp [13]
71.02
50.63
64.07
76.18
48.75
59.91
ProGrad [27]
72.24
49.39
64.73
74.58
47.61
59.07
KgCoOp [14]
71.20
64.10
76.70
48.97
PL.OT++ [83]
72.48
47.05
65.07
74.27
47.13
58.38
MaPLe [15]
70.72
50.90
64.07
76.98
49.15
60.28
PromptSRC [1:5]
71.27
50.90
64.35
77.80
49.55
60.65
POMP [22]
70.16
50.83
63,32
77.37
49.74
60.32
TPT [49] CoOp [9]
73.61
57.95
66.83
77.27
49.29
62.83
TPT [49] + CoCoOp [13]
71.07
58.47
64.85
78.65
48.47
62.61
DifTTPT [50] + CoOp [9]
75.00
58.09
66.80
73.90
49.50
62.07
DiffIPT [50] CoCoOp [13]
69.30
52.56
63.20
75.30
47.50
PromptAlign [52]
71.44
59.37
65.29
79.33
50.23
63.56
AWT
75.17
63.01
68.22
81.20
51.59
66.01
Previous studies [9, 13] suggested assessing method robustness by directly applying a model trained on ImageNet to four variant datasets derived from ImageNet. We adhere to this protocol and present our findings in Tab. 8. The results demonstrate that our approach, AWT, not only excels in few-shot performance on ImageNet but also outperforms domain generalization robustness across the four ImageNet variant datasets, effectively balancing the popplaationngnnealiizttioo trade-off.
B.5
Study on different LLMs
Table 9: Study on different LLMs in the description generation process.
AWT
IN-1K (89]
IN-A [103]
IN-V2 [104]
DTD [91]
Cars [93]
[94]
Cal101 [95]
Food [96]
GPT-3.5
71.32
60.33
65.15
55.56
69,93
72.51
95.54
85.54
w/ GPT-4o [36]
71.36
60.13
64.63
$6.03
69.46
72.14
95.21
85.39
Qwen [127]
65.21
95.17
85.42
In this study, we utilize GPT-4o [36] and Owen-Plus [127] for our description generation process, integrating them into AWT. The benchmark accuracy is presented in Tab. 9. Overall, AWT maintains robust performance across different LLMs. Interestingly, more advanced LLMs not necessarily lead to significant performance improvements. This may be because AWT primarily relies on the simple instruction-follore followiwn and knowledge-recall capabilities of L.LMs, for which models like GPT-3.5 are sufficient.
20

---

B.6 Description generation without dataset description
Table 10: Compare AWT-base with other description generation methods.
Method
Manual Design Component
OOD
Avg.(14)
AWT w/ VisDesc [6]]
glohal question
63.55
67.78
AWT w/ CuPL-buse [63]
global questions
63.57
68.87
AWT w/ CuPL-full [63]
4.56 questions per dataset
63.88
69.33
AWT- base
1 global question
64,03
69.27
AWT
(1 global) + (1 description dataset)
64.43
69.59
Here, we examine the performance of AWT when no dataset-level description available, a situation often encountered in real-world applications. We introduce a base version of AWT, which replaces our original instruction with "Generate questions to classify images. while leaving other components unchanged. The performance of AWT-base is detailed in Tab. 10. Despite its simplicity, AWT-base demonstrates favorable performance compared to previous methods.
B.7 Numerical values in few-shot learning
For the benefit of future research, we present numerical values of our experimentation in the few-shot transfer learning setting in Tab. 11.
Table 11: Few-shot image classification results of different methods on 11 datasets. All results are averaged over three runs.
Flowers/ 901
Airrratosy
18-1k(89)
Pets92]
DTD
Food/oo)
Avg
Shots
Handcrafted
88.25
67.44
23.67
44.27
42.01
65.48
83.65
62.59
93.35
65.13
66.73
63.87
90.37
77.53
21.37
50.23
$4.93
67.43
84.33
66.77
92.60
71.23
66.33
67.56
87.33
26.20
53.60
65.17
70.50
84.40
66.53
93.07
73.43
67.07
70.65
CoOp [9]
92.57
92.17
30.83
58,70
70.80
74.47
84.47
69.97
94.40
77.10
68.73
74.02
91 27
94.97
39.00
64.77
78.07
79, 30
82.67
71.53
94.37
80.20
70.63
76.98
91.87
97.07
43.40
69.87
84.93
83.07
84.20
74.67
82.23
71.87
79.89
91.27 27
72.08
12.68
54
55.33
67.22
85.65
68.33
93.83
70.30
69.43
66.79
92.64
75.79
15.06
$2.17
46.74
68.37
86.22
69.03
94.82
73.51
69.78
67.65
CoCoOp [13]
78.40
24.79
65
70.21
94.98
74.82
70.39
71.21
93.45
84.30
26.61
58.89
68.21
70.44
86.97
70.84
95.04
77.14
70.63
72.96
93.34
87.84
31.21
63.04
73.32
71.57
87.25
72.15
95.16
78.14
70.83
74.90
91.89
80.48
28.60
54.57
65.41
68.81
86.16
66.77
94.34
74.31
66.45
70.71
92.29
89.81
31.14
56.72
76.80
73.17
86.33
68.06
94.69
76.76
68.28
74.00
PLOT+ [83]
92.55
92.93
35.29
62.43
83.21
76.25
86.46
71.73
9.5.13
79.76
70.40
76.92
93.02
95.44
66.49
88.37
81.26
86.58
73.93
95.51
82.
71.31
79.65
93.59
97.56
46.74
71.43
92.00
84.55
87.11
76.03
96.04
85.34
72.60
82.09
92.00
85,93
27.67
56.23
73.13
69.
84.87
69.67
93.67
74,
68.1 63
72.32
92.50
91.17
31.70
59.97
79.37
73.40
85.70
71.60
94.53
78.50
69.77
75.29
PromptSRC [21]
93.43
93.87
37.47
65.53
86.30
77.13
86.17
74.00
95.27
81.57
71.07
78.35
93.50
96.27
43.27
69.87
88.80
80.97
86.90
75.73
95.66
84.30
72.33
80.69
97.60
72.73
92.43
87.50
77.23
86.47
73.17
84.76
66.09
72.
67.29
71.37
92.31
86.
30.55
57.23
86.57
67.99
85.54
68.42
95.00
76.07
688 S54
74.03
Ref [32)
93.02
91.70
34.40
63.33
88.97
71.57
86.13
70.27
95.28
13
69.25
76.73
93.27
95.89
43.47
68.42
91.78
77.33
86.41
72.48
95.86
83.57
70.23
79.88
16
93.89
98.16
53.18
73.62
941.19
84.85
87.00
74.72
96.63
86.43
71.14
83.07
91.17
90.58
31.50
60.58
72.
69
85. 94
71.20
93.88
75.26
70.88
73.94
35.85
76.73
72.19
79.17
71.43
76.27
APE-T [57]
92.34
95.70
39.00
67.55
83.57
76.61
86.61
74.47
94.73
80.99
71.99
78.51
93.46
96.91
42.87
70.63
87.02
78.86
86.85
76.13
95.70
83,72
73.37
80.50
93.16
97.81
48.33
74.29
90.17
83.93
87.22
77.29
84.85
74.33
82.49
92.89
85.61
33.92
59.44
76.33
73.43
85.92
72.65
95.13
78.38
72.14
75.08
92.98
89.89
37.77
62.63
83.17
76.96
86.05
73.72
95.67
80.32
72.86
77.46
AWT
93.50
42.14
88.51
86.46
74.72
82.91
73.41
93.83
96.47
50,21
70.51
89.72
84,50
87.44
75.79
96.58
85.42
74.38
82.26
94.63
97.69
57.08
74.65
93.68
87.59
88.11
77.57
97.12
87.53
75.17
84.62
21

---

Adapter
Scale
MLP
A
Up
LayerNorm
GeLU
Adapter
Self-Attention
LayerNorm
Trainable
Firozen
Figure 10: Architecture of the adapter module and its integration with the Transformer.
Table 12: Few-shot transfer learning settings. We select the model from the final epoch for testing.
Configuration
1-shot
2-shot
4-shot
8-shot
16-shot
adapter layers
all transformer layers
downsample rate À
384
384
384
256
256
learning rate
2c-3
2e-3
2c-3
8c-3
8c-3
learning rate schedule
cosine decay [128]
warmup epochs warmup type
5
linear
weight decay
5e-4
training epochs
20
20
20
25
25
batch size optimizer optimizer momentum image views N (train/test)
16
SGD [129]
0.9
8/50
class descriptions M (train/test)
8/50
GPU numbers augmentation
1
random resized crop and random flip
C
Additional Details
C.1 Few-shot learning with the multi-modal adapter module
For few-shot learning, we employ a multi-modal adapter to achieve parameter-efficient transfer learning. The architecture of the multi-modal adapter is depicted Fig. 10. We integrate the adapter module subsequent to the Multi- Self-Attention and the MLP module within each transformer layer. The adapter module applied both the visual and textual side of the CLIP model. The core of the adapter module is a bottleneck structure designed to efficiently manage model parameters while maintaining performance. This structure comprises down doon-projection layer W down R d that reduces the dimensionality of the input features, and up-projection layer W up R & that restores the original dimensions. Between these layers, a GeLU activation layer [130] introduces non-linearity. learnable scale parameter s used to modulate the output of the adapter. The feature dimension is denoted by d. For an input feature vector x E g.xxd where L is the sequence length, the adapter's forward process is expressed as:
GeLU(x W down) W
(9)
up
Detailed settings and hyperparameters for our few-shot learning experiments outlined in Tab. 12.
22

---

Table 13: Dataset-level descriptions employed in the first step of our prompt strategy are provided, along with corresponding source weblinks for each description. ImageNet-A and ImageNetV2 are omitted as they share the same dataset-level descriptions as ImageNet.
Dataset
Description
Desc. Source
ImageNet
"an image database contains millions of images across thou- sands of categories"
[Website]
ImageNet-R
"contains images in forms art, cartoons, deviantart, graf- fiti, embroidery, graphics, origami, paintings, patterns, plastic objects, plush objects, sculptures, sketches, tattoos, toys, and video game renditions"
[Wehsite]
ImageNet-Sketch
"consists of black and white sketches of ImageNet categories"
[Wehsite]
Caltech101
"contains images from 101 object categories"
[Website]
OxfordPets
"a pet dataset whose images have large variation in scale, pose, and lighting"
[Website]
StanfordCars
"contains images of cars whose classes rypically at the level of Make, Model, Year, ex"
[ebsite)]
OxfordFlowers
"the flowers chosen to be flowers commonly occurring in the United Kingdom with large scale, pose, and light variations" " "consists of 101 food categories with some amount of noise" " "contains images of different aircraft model variants, most of
[Website]
Food101
[Wehsite]
FGVCAireraft
[Website]
which are airplanes"
SUN397
"a Scene UNderstanding dataset with 397 categories"
[Website]
DTD
"has collection of textural images in the wild"
[Website]
EuroSAT
"based on Sentinel-2 satellite images for land use and land cover classification"
[Website]
UCF101
"an action recognition data set of realistic action videos"
[Website]
Caltech256
an object recognition dataset containing real-world images" "a challenging dataset of 200 bird species"
[Website]
CUB
[Website]
Birdsnap
"a large bird dataset with 500 bird species"
[Website]
C.2
Two-step dataset-aware prompting for LLMs
Here, we provide additional details of our proposed two-step dataset-aware prompting strategy for LLMs. Initially, the first step, we leverage dataset-level description for each dataset inspire the generation of diverse and contextually relevant questions. The descriptions for each dataset are sourced from their official websites or from the paperswithcode website, and we include the dataset description and the reference to the source's URL in Tab. 13.
To better illustrate the effectiveness of our strategy, we present sample of questions generated in the first step. For each dataset, we list three examples in Tab. 14. From these examples, we can observe the following: 1) the questions generated are not only more diverse compared to traditional prompts such as "Describe a {class}", but they also delve into more detailed inquiries, and 2) the questions incorporate dataset-specifie information, thereby aiding LLMs in producing more accurate and relevant descriptions.
C.3 License information of the assets used in this work
Datasets. Below are the datasets used in this paper that have known license information:
MIT License: ImageNet-A [103), ImageNetV2 [104], ImageNet-R [105), ImageNNttSSketch [106),
EuroSAT [99].
CC BY 4.0 License: 101 [95], Caltech256 [101], HMDB51 [107], K400 [119], K600 [108].
CC BY-SA 4.0 License: OxfordPets [92].
Source code. Source code used in this paper are under the MIT License: CLIP [1], CoOp [9].
TPT [49], PLOT [83].
23

---

Table 14: A selection of generated questions for each dataset.
Dataset
Generated questions (a selection)
How would you describe the overall appearance of (1 in the image? Describe the color scheme and patterns present image of ().
ImageNet
What shapes and structures are noticeable in the image of a ()2
How is the style of the (} depicted in this art piece different from a realistic portrayal?
ImageNet-R
What aspects of the sculpture rendition of () stand compared to other forms?
What elements the origami version of {) capture its essence a creative way?
What basic shapes can you identify the sketch of [
ImageNet-Sketch
How would you describe the linework in the sketch of {}?
Are there any particular lines strokes that define the sketch of {)?
How would you describe the overall appearance of {
Caltech101
Are there any recognizable patterns or markings on {}? What colors dominate the image of {)?
What fur patterns or textures are visible on [}?
OxfordPets
Are there any distinetive features like ears, tails, paws that help identify (I?
What colors are commonly seen on () in dataset?
How would you describe body shape {)?
StanfordCars
Are there any distinctive logos or emblems on []?
What kind of wheels or tires are visible on [}?
What are the prominent colors the petals of )
OxfordFlowers
Can you describe the shape the petals on {)?
Do you notice any specific features like stamens or pistils on (}?
What are the key ingredients or toppings present on (}?
Food101
Are there any garnishes accompaniments that come with ()?
How would you describe the presentation plating of ()?
How would you describe engine or propulsion system of aircraft in {)?
FGVCAircraft
What details can you observe in the cackpit area of the aircraft in (}2
Can you describe the wingspan of the aircraft in ()?
What kind of objects structures typically found in {)?
SUN397
Are there any specific landmarks or features that are commonly associated with ()?
Can you identify any natural elements in ))?
How would you describe the surface of ()?
DTD
Are there any patterns irregularities in the texture of ()?
Are there any specific features that indicate the material of ()?
What natural features can you identify in satellite view of []?
EuroSAT
What land patterns or formations are visible in a satellite view of {)?
Are there any specific landmarks that can help in identifying satellite view of ()?
What body movements are being performed in the action of (}?
UCF101
Can you describe the posture and gestures of the person in the action of ()?
What objects or tools are involved in the action of ()?
Can you describe the overall shape of ()?
Caltech256
Are there any specific patterns or markings {}?
How would you describe the color scheme of {]?
What specific colors
seen the feathers of [)?
CUB
Can you describe the beak shape {)?
What are the distinguishing features of (F's wings?
How would you describe the shape of the hody of ()?
Birdsnap
What the coloration of the feathers {)?
Do you notice any distinct features on the head of ()?
24

---

D Societal Impacts
This paper introduces AWT enhance the transferability of vision-language foundation models. We have evaluated the effectiveness of AWT across a variety of classification tasks, employing different types, scales, and architectures of VLMs. Our findings suggest that AWT holds significant potential for integration into diverse downstream applications involving VLMs. Beyond classification tasks, we anticipate that the insights gained from this study will stimulate further research into other areas of computer vision, such as object detection and semantic segmentation. Currently, we are not aware of any ethical issues associated with the real-world applications of this technology. Nonetheless, continuous monitoring and evaluation are recommended to ensure responsible deployment.
E Limitations and Future Work
While the AWT framework achieves commendable performance using a limited number of augmented image views, attaining higher performance necessitates increase in the number of views. This expansion, however, inevitably leads to additional computational costs. Notably, the visual elements in different augmented views often contain duplicates, particularly when they are all derived from the same original image (e.g., random cropping). An interesting avenue for future research would be to explore strategies for reducing duplicates among batches to enhance inference speed. Furthermore, the application of test-time augmentation extends beyond our current scope and holds potential for various other domains, such as video understanding [131, 132], frame interpolation [133-135], semantic segmentation [136-138], and action detection [139]. Another intriguing area for future research involves the integration of efficient and controlled diffusion models [140, 141] enhance the quality of visual augmentations.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See our results and discussion in Section 4,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See implementation details in Section and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [No]
Justification:
We will release our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct /ttp:///ggiieeeeeeeuuuubiiiiiiii /CodeSubmic ssisnnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See experimental setting/details in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Appendix B.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerrp/.ee/ee cc htcp/ppbic///hhccuuuiiiiieeeeeee
Answer: [Yes]
Justification: This paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix D.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
[Yes]
Justification: We have cited the creators and mentioned the license of the assets (See Ap- pendix C.3).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
30

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve such experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve such risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "fXpl0ImcIL", "reviewer": "Reviewer_CYNL", "summary": "This paper proposes AWT, an adaptation framework that can boost pre-trained vision-language models (VLMs) for understanding new concepts on new classes. To summarize, AWT first do augmentations on both visual images and textual class names. Then an entropy metric is employed for weighting multiple augmented data. Finally, it measures the distance by OT algorithm associated with the confidence weight computed from the W step. AWT performances best in multiple scenarios including ZS, FS image classification and action recognition and OOD generalization.", "questions": "1.What about applying more complex visual augmentations more than random cropping and flipping? How about introducing additional visual training data like Support Set in SuS-X?\n2.Why only 50 class descriptions M are generated? The performance seems not saturated at M=50 according to Tab 4(c).\n3.Any inference speed limitation? Since 50 times more images and class descriptions are generated by augmentation.\n4.Are the evaluation results stable or constant? If AWT generate different image crops or different class descriptions (affected by GPT version) in multiple experimental trials, then the weighting and OT processes should be affected, which may give different final distance measure. If so, then the mean and variance of several rounds of experiments should be reported.", "limitations": "No limitation discussion in the original paper.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.This paper is well-written, and well-organized with clear statements, standard structure, intuitive motivation, etc.\n2.AWT shows superior performance and strong generalization ability across multiple evaluation benchmarks.\n3.Utilizing LLM to augment existing class names seems interesting and effective, also distinctive from previous works. May bring new insight for the community.", "weaknesses": "1.Augmenting images by simple random resized cropping and random flipping seems to be too naïve compared with nowadays prompting methods. As in previous work SuS-X [1], a much more diversified Support Set can be constructed by stable diffusion and retrieval. Meanwhile, according to Tab 4(a), we can not see any improvement by introducing visual augmentations (Raw inputs vs. A(Img.)). \n[1] SuS-X: Training-Free Name-Only Transfer of Vision-Language Models. Vishaal Udandarao et al.\n2.The final inference step, measuring distance across two modalities by OT algorithm is not new as it was proposed in [2] and [3]. However, the difference with these two studies is not clearly claimed in the AWT paper. The original statement “Distinct from these two studies, our research diverges by eschewing the need for additional training resources, opting instead for an augmentation-based direction”  looks confused and fuzzy.\n[2] Tuning Multi-mode Token-level Prompt Alignment across Modalities. Dongsheng Wang et al.\n[3] PLOT: Prompt Learning with Optimal Transport for Vision-Language Models. Guangyi Chen et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and insightful comments! Please allow us to address your concerns below.\n\n#### **W1 and Q1: 1) Random resized cropping and flipping is naïve. 2) Cannot see any improvement by visual augmentations. 3) What about applying more complex visual augmentations?**\n\nThank you for raising these points.\n\n- We opted for random resized cropping and flipping primarily due to their **efficiency** when compared to more advanced augmentation methods. For instance, employing diffusion models can significantly slow down inference speed, as discussed in the last part of Section 4.3 in DiffTPT [1], where inferring 10 test images takes 36 minutes. Meanwhile, methods involving data retrieval often require substantial storage space, such as LAION-5B used in SuS-X [2].\n\n- The incremental improvements with visual augmentations are also noted by us, as mentioned in Lines 277-278 of our paper. Directly applying such augmentations can generate information-sparse image views. **This motivates us to propose the weighting strategy**. We provided an additional ablation study below, which demonstrates that, with the weighting module, even simple visual augmentations can yield significant improvements.\n\n| Step             |  OOD  | Avg.(14) |\n| ---------------- | :---: | :------: |\n| Raw inputs       | 57.15 |  63.06   |\n| A(Img.)          | 56.77 |  63.40   |\n| A(Img.) + Weight | 62.76 |  66.21   |\n\n- We have explored more complex augmentations like diffusion models in Appendix Section D and found them superior to naive methods. Additionally, we included results for the EuroSAT dataset this time, summarized in the table below. However, given the efficiency concerns, we choose to use simple augmentation by default.\n\n| Augmentation                           | EuroSAT | CIFAR10 | CIFAR100 |\n| -------------------------------------- | :-----: | :-----: | :------: |\n| Random resized crop and flip (default) |  58.61  |  89.25  |  67.61   |\n| DALL·E 2                               |  60.41  |  92.30  |  68.69   |\n\n#### **W2: Difference between prior OT methods and AWT is not clearly claimed.**\n\nWe apologize for any confusion caused. For a detailed response, please refer to our responses to common reviewer concerns. We will ensure to clarify any ambiguities in the final version of our paper.\n\n#### **Q2: Why only 50 class descriptions are generated?**\n\nThank you for your query. As detailed in Table 4(c), increasing the number of descriptions from 50 to 100 provides only marginal gains (0.03% and 0.14% improvements, respectively), while doubling the query costs for GPT. We believe that maintaining 50 descriptions strikes a good cost-effectiveness balance.\n\n#### **Q3: Any inference speed limitation?**\n\nThank you for your question. We have discussed the inference speed limitations in Sections B.2 and F of our Appendix.\n\n#### **Q4: Are the evaluation results stable or constant?**\n\nThe results are stable and almost constant. The stability of AWT is supported by error bar analysis (mean and variance) in Section B.1 of the Appendix. For each run, images are augmented using different seeds, and descriptions are regenerated. Your suggestion to explore the impact of different GPT versions is insightful. Following your advice, we conducted further experiments with an alternative GPT version and another LLM, and presented the results below. Overall, AWT maintains robust performance across these variations. An interesting observation is that more advanced LLMs do not necessarily yield significant performance improvements. This may be attributed to the fact that, in this scenario, AWT primarily relies on the simple instruction-following and knowledge-recall capabilities of LLMs, making models like GPT-3.5 sufficient.\n\n| Method                   | ImageNet | ImageNet-A | ImageNet-V2 |  DTD  | StanfordCars | UCF101 | Caltech101 | Food101 |\n| ------------------------ | :------: | :--------: | :---------: | :---: | :----------: | :----: | :--------: | :-----: |\n| AWT w/ GPT-3.5 (default) |  71.32   |   60.33    |    65.15    | 55.56 |    69.93     | 72.51  |   95.54    |  85.54  |\n| AWT w/ GPT-4o            |  71.36   |   60.13    |    64.63    | 56.03 |    69.46     | 72.14  |   95.21    |  85.39  |\n| AWT w/ Qwen-Plus         |  71.48   |   60.21    |    65.21    | 56.15 |    70.25     | 72.01  |   95.17    |  85.42  |\n\n#### **L1: No limitation discussion in the original paper.**\n\nThank you for your comment. The limitations have been discussed in Section F of the Appendix.\n\n***We hope that these responses adequately address your queries and concerns. We look forward to your feedback and further guidance.***\n\n[1] Feng, Chun-Mei, et al. \"Diverse data augmentation with diffusions for effective test-time prompt tuning.\" ICCV 2023.\n\n[2] Udandarao, Vishaal, Ankush Gupta, and Samuel Albanie. \"Sus-x: Training-free name-only transfer of vision-language models.\" ICCV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFUkfcBoWi", "reviewer": "Reviewer_Vbjj", "summary": "This paper proposes a novel adaptation framework for vision-language models, which replaces point-to-point alignment of text and images with set-to-set alignment. Specifically, the authors use image transformation and LLM generation to create augmentation sets. After weighting the augmentations based on entropy, optimal transport is used to compute the distance between vision and textual sets to measure their semantic correlation. Experiments are conducted on standard benchmarks.", "questions": "See weaknesses, especially the difference and connection with [71,73]", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Replacing point-to-point distance measurement with set-to-set distance measurement is a significant improvement in measuring semantic correlation.\n2. The proposed method is well-motivated, with its three modules—augmentation, weight, and transport—being closely related, and their effects are ablated separately.\n3. Experiments are conducted on a wide range of tasks.", "weaknesses": "1. PLOT [71] and Wang [73] also introduce optimal transport (OT) into vision-language models and are closely related to this work. The relationship and differences between these works need to be clearly clarified.\n2. Some recent works should be acknowledged. For example, [a, b, c] have also been proposed for the adaptation of vision-language models. These works were accepted by CVPR 2024 and typically appeared on ArXiv in March, about two months before the NeurIPS submission deadline. Therefore, it is recommended to discuss these works as well.\n3. The computational complexity should be analyzed and compared against prior methods. \n\n[a] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models, CVPR2024\n\n[b] MMA: Multi-Modal Adapter for Vision-Language Models, CVPR2024\n\n[c] Efficient Test-Time Adaptation of Vision-Language Models, CVPR2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you have dedicated to reviewing our paper. Please allow us to address your concerns below.\n\n#### **W1: Relationship and differences between prior OT methods and AWT.**\n\nThanks for your comment. For a detailed response, please refer to our responses to common reviewer concerns. We will include this discussion in the final version of our paper.\n\n#### **W2: Some recent works should be acknowledged and discussed.**\n\nThank you for bringing these excellent studies to our attention. We have carefully considered their relevance with AWT and plan to include the following discussions in our revised paper:\n\n- **DMN [1]** introduces a versatile adaptation framework suitable for various settings. However, DMN's applicability is constrained as it necessitates historical test data and is limited to scenarios where test and candidate classes are identical. AWT, while effective, also has its limitations: it requires additional LLMs and incurs higher computational costs. Noted that AWT and DMN could be potentially combined: DMN could utilize its capability to store AWT's augmented image features in memory alongside LLM augmented textual features, then replace the final dot product with our proposed entropy-based weighted OT distance.\n- **MMA [2]** introduces advanced adapter-based methods for VLMs. In zero-shot settings, MMA falls short compared to AWT due to its dependency on additional training data. In contrast, for few-shot adaptation/generalization settings, while AWT uses a standard adapter module, MMA introduces an advanced adapter that balances discrimination and generalization. Integrating MMA's adapter into AWT could potentially enhance performance.\n- **TDA [3]** proposes the use of a memory cache to mitigate the extensive computational demands of test-time adaptation methods. TDA is akin to the zero-shot adaptation version of DMN, which necessitates historical test data from identical candidate classes. Based on our analysis alongside DMN, our AWT method is also orthogonal to TDA, and integrating these two approaches could potentially enhance performance.\n\n#### **W3: The computational complexity should be analyzed and compared against prior methods.**\n\nThank you for your suggestions. We have discussed the computational limitations in Sections B.2 and F of our Appendix, and compared our method with previous methods in Figure 8 of our paper. Below, we provide additional details to more comprehensively address your concern:\n\n- **Augment and CLIP forward:** Given that the text features can be pre-computed (as suggested in the CLIP paper), AWT theoretically involves computing complexity of $N+1$ times that of the original CLIP model, where $N$ is the number of augmented image views.\n- **Weight Calculation:** This process is relatively swift and primarily involves several matrix operations such as Softmax function.\n- **Optimal Transport:** Thanks to the efficient Sinkhorn’s Algorithm, we can solve the optimal transport problem with high efficiency.\n\nThe bulk of the computational effort is concentrated in the CLIP forward process. Thanks to the parallel processing capabilities of modern GPUs and efficient deep learning libraries like PyTorch, the computational cost of AWT remains manageable. Notably, AWT is significantly more efficient than test-time adaptation methods, which require model training during inference, such as TPT [4] (refer to Figure 8 of our paper). Moreover, the cost-performance trade-off can be easily modulated by adjusting $N$. We believe that these limitations can be further addressed through future research, as discussed in Lines 776-779 of our paper.\n\n***We hope that these responses adequately address your queries and concerns. We look forward to your feedback and further guidance.***\n\n[1] Zhang, Yabin, et al. \"Dual memory networks: A versatile adaptation approach for vision-language models.\" CVPR 2024.\n\n[2] Yang, Lingxiao, et al. \"MMA: Multi-Modal Adapter for Vision-Language Models.\" CVPR 2024.\n\n[3] Karmanov, Adilbek, et al. \"Efficient Test-Time Adaptation of Vision-Language Models.\" CVPR 2024.\n\n[4] Shu, Manli, et al. \"Test-time prompt tuning for zero-shot generalization in vision-language models.\" NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0kgeXjYRGI", "reviewer": "Reviewer_kG9e", "summary": "The paper considers a task of low-shot (zero-shot and few-shot) adaptation of vision-language models (VLMs). The proposed approach is based on diversifying the inputs of VLMs using augmentations for images and LLMs for class names. Each input is then weighted based on its prediction entropy, while optimal transport is employed to measure the distance between the test image and classes. The evaluation is performed across a wide range of datasets for different tasks, such as zero-shot classification and video action recognition, few-shot classification, and out-of-distribution generalization.", "questions": "- What is the performance of the proposed method when using LLM prompts that do not include a description of the dataset?\n- What is the performance of the method without optimal transport but using weighted averaging of features?", "limitations": "The authors have adequately addressed the limitations of the work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- One set of hyper-parameters is used across all datasets, which differs from most other methods that tune hyper-parameters per dataset, sometimes even per shot. This makes the proposed method really a low-shot one, as others tune hyper-parameters on a large annotated validation set.\n- The paper tackles an important problem of adapting VLMs in low-shot settings. This is a very active area of research, and the proposed method is achieving state-of-the-art results across various benchmarks.\n- The method is presented in a simple way that is easy to understand.", "weaknesses": "- Prompts for LLM to generate class descriptions contain a description of the dataset, which is a slight limitation. It would be beneficial to show the result for prompting without knowing the dataset description. CuPL and VisDesc comparisons in Table 4d are going in this direction; however, considering that the LLM version could be different, they are not a direct comparison.\n- The paper shows that simple averaging of features from multiple image and text inputs does not work. However, a comparison with weighted averaging (weights based on prediction entropy) is missing. This would give a proper indication of how important optimal transport is in the method.\n- The related work section could be significantly extended, e.g., methods based on adapters (CLIP-Adapter [1]) and based on memory (TIP-Adapter [2], APE [3]) are not mentioned at all. Additionally, a comparison with APE for few-shot classification would be beneficial.\n\n*Minor comments:*\n\n- Are $\\gamma_v = 2$ and $\\gamma_t = 2$ as in L217 or are the equal to $1/2$ as in tables 4e and 4f? \n\n*References:*\n\n[1] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao, \"CLIP-Adapter: Better Vision-Language Models with Feature Adapters\", IJCV, 2024\n\n[2] Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li, \"Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification\", ECCV, 2022\n\n[3] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, Peng Gao, \"Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement\", ICCV, 2023\n\n**After rebuttal**\nThe rebuttal has adequately addressed all of my concerns from the initial review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful feedback. Please allow us to address your concerns below.\n\n#### **W1: 1) The dataset description is a slight limitation. 2) Results for prompting without dataset description. 3) The LLM version for AWT, VisDesc [1], and CuPL [2] might differ.**\n\nThank you for raising these issues.\n\n- **Dataset Description Limitation**: We acknowledge this limitation. However, to enable the LLM to perceive visual domains (e.g., sketches), manually inputting this prior information is necessary. CuPL (in its `full` version) addresses this by **manually designing multiple questions for each dataset**. We have **streamlined this process** by directly providing LLMs with dataset-level descriptions, enabling them to **automatically generate dataset-tailored questions**.\n- **Prompting without Dataset Description**: Your suggestion to explore scenarios without dataset descriptions is valuable, considering real-world applications where such information might not always be available. We have designed a \"`base`\" version for AWT where we replaced the instruction in Lines 146-147 with \"Generate questions to classify images.\" while keeping other parts unchanged.\n- **LLM Version Consistency**: For VisDesc, we regenerated the descriptions using GPT-3.5 (the same version as used for AWT) using their official code. For CuPL, the descriptions were initially from SuS-X [3] and generated by GPT-3. We apologize for this oversight. To correct this, we have regenerated CuPL descriptions with GPT-3.5 and implemented both `base` and `full` versions of CuPL.\n\nIn summary, we now provide additional results for AWT (`base`) and GPT-3.5-based CuPL (`base` and `full`) in the table below.\n\n| Method             | GPT version | Manual Design Component                  | OOD   | Avg.(14) |\n| ------------------ | ----------- | ---------------------------------------- | ----- | -------- |\n| AWT w/ VisDesc     | GPT-3.5     | 1 global question                        | 63.55 | 67.78    |\n| AWT w/ CuPL-`base` | GPT-3.5     | 3 global questions                       | 63.57 | 68.87    |\n| AWT w/ CuPL-`full` | GPT-3.5     | ~4.56 questions per dataset              | 63.88 | 69.33    |\n| AWT-`base`         | GPT-3.5     | 1 global question                        | 64.03 | 69.27    |\n| AWT                | GPT-3.5     | (1 global) + (1 descirption per dataset) | 64.43 | 69.59    |\n\n#### **W2: Results for weighted averaging are missing.**\n\nThank you for pointing this out. We aimed to demonstrate the significance of OT by contrasting A(Img.+Name) and A+T in Table 4(a). We have also included the results for weighted averaging below for your reference.\n\n| Step  | OOD   | Avg.(14) |\n| ----- | ----- | -------- |\n| A+W   | 63.67 | 68.71    |\n| A+W+T | 64.43 | 69.59    |\n\n#### **W3: The related work section could be significantly extended.**\n\nWe apologize for the oversight in our related work section. CLIP-Adapter [4] and Tip-Adapter [5] are pioneering studies proposing an adapter module and a training-free cache module for few-shot adaptation, respectively. APE [6] further leverages a CLIP prior for initialization in a more efficient manner. We will incorporate these studies into our related work section. Moreover, to enable a comparison between APE and AWT, we implemented APE-T (its highest-performing variant) using the CLIP-base/16 architecture across 11 few-shot learning datasets, utilizing the official code. We detail this comparison in Figure 1 of our attached PDF. APE-T will also be included in Figure 3 of our paper.\n\n#### **Minor: Are $\\gamma_v=2$ and $\\gamma_t=2$ or are the equal to 1/2 as in tables 4e and 4f?**\n\nWe apologize for the confusion; both $\\gamma_v$ and $\\gamma_t$ are indeed set to 1/2.\n\n***We hope that these responses adequately address your queries and concerns. We look forward to your feedback and further guidance.***\n\n[1] Menon, Sachit, and Carl Vondrick. \"Visual Classification via Description from Large Language Models.\" ICLR 2023.\n\n[2] Pratt, Sarah, et al. \"What does a platypus look like? generating customized prompts for zero-shot image classification.\" ICCV 2023.\n\n[3] Udandarao, Vishaal, Ankush Gupta, and Samuel Albanie. \"Sus-x: Training-free name-only transfer of vision-language models.\" ICCV 2023.\n\n[4] Gao, Peng, et al. \"Clip-adapter: Better vision-language models with feature adapters.\" IJCV 2024.\n\n[5] Zhang, Renrui, et al. \"Tip-adapter: Training-free clip-adapter for better vision-language modeling.\" ECCV 2022.\n\n[6] Zhu, Xiangyang, et al. \"Not all features matter: Enhancing few-shot clip with adaptive prior refinement.\" ICCV 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pU2emS3iqp", "reviewer": "Reviewer_qvMQ", "summary": "This paper introduces an adaptation framework AWT to enhance vision-language models. The authors propose to augment inputs from visual and textual perspectives. The augmented inputs are dynamically reweighted based on the prediction entropy. Finally, the authors propose to use optimal transport to find the best prediction from VLM. The experimental results demonstrate strong performance under various settings.", "questions": "Please see the weakness above.", "limitations": "The limitation of this paper is sufficiently discussed by the authors.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well illustrated and easy to follow.\n- The proposed AWT framework is novel and effective, which improves performance of VLM without additional training costs.\n- The experiments demonstrate the strong performance of AWT under image/video tasks.", "weaknesses": "- Performance-efficiency trade-off under few-shot settings. In supplementary material Figure 8, the performance increases with more augmented images under a zero-shot setting. I'd like to see a similar analysis under the few-shot image classification setting, which can further improve the significance of this work.\n- How do the authors decide the hyper-parameters? The authors provide the average results of 14 datasets and the OOD under different choices of hyper-parameters in Table 4. However, it seems to lack details about how the hyper-parameters were chosen. Could authors provide more information on this?\n- Why choose flipping as one of the image augmentation strategies? If I understand correctly, image augmentation aims to enable the VLM to capture the fine-grained matching between image regions and different text descriptions, which is already achieved by random crop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback. Please allow us to address your concerns below.\n\n#### **W1: I'd like to see performance-efficiency trade-off under few-shot settings.**\n\nThank you for your suggestion. We presented this experiment in Figure 2 of our attached PDF. In comparison to the zero-shot scenario, the few-shot settings incorporate two additional adapter modules at each transformer layer to facilitate transfer learning. This configuration results in our method being initially slower than other prompt learning approaches. This slowdown is likely due to the adapters needing sequential processing, whereas the prompts are processed in parallel. Taking into account the computational costs, we propose that AWT could be more efficiently transferred using test-time zero-cost methods like LoRA [1], which allows for structural reparameterization after training.\n\n#### **W2: How do the authors decide the hyper-parameters?**\n\nThank you for your question. As detailed in Table 4 of our paper, AWT involves four main hyper-parameters: number of augmented image views ($N$), number of generated class descriptions ($M$), and the weighting temperatures $\\gamma_v$ and $\\gamma_t$.  We maintain consistent hyper-parameter values across all scenarios—whether zero-shot or few-shot, and for both image and video tasks.  This uniformity is due to our primary focus on the zero-shot capabilities, where typically no validation set is available for further tuning of hyper-parameters. Here is how we determined these values:\n\n- We used a subset of the ImageNet training set as the initial and only validation set for selecting hyper-parameters. We chose the hyper-parameter values based on the accuracy:\n  - For $\\gamma_v$ and $\\gamma_t$, we conducted a broad search and observed optimal performance around the value of 1/2, which we adopted as the default value.\n  - For $N$ and $M$, our selection process was similar, but we opted for the most cost-effective values rather than the absolute best. This decision was based on the observation that while increasing $N$ and $M$ can marginally improve results, it also significantly raises the computational costs associated with querying GPT.\n\n#### **W3: Why choose flipping as one of the image augmentation strategies?**\n\nYou are correct in noting that cropping alone can capture fine-grained information. However, we believe that flipping introduces an additional layer of diversity in visual perspectives, which can potentially enhance the cross-modal matching process. To support our choice, we've conducted an ablation study detailed below. Our findings suggest a modest improvement, but given the minimal additional cost, we have decided to retain this method.\n\n| Augmentation                         |  OOD  | Avg.(14) |\n| ------------------------------------ | :---: | :------: |\n| Random resized cropping              | 64.22 |  69.45   |\n| Random resized cropping and flipping | 64.43 |  69.59   |\n\n***We hope that these responses adequately address your queries and concerns. We look forward to your feedback and further guidance.***\n\n[1] Hu, Edward J., et al. \"LoRA: Low-Rank Adaptation of Large Language Models.\" ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate the time and effort you have invested in reviewing our submission. This paper received 4 review comments, and 3 reviewers gave positive scores. All reviewers acknowledged the strong writing, the excellent benchmark performance, and the extensive experiments conducted. Additionally, Reviewer CYNL noted that our study may bring new insights to the community, Reviewer Vbjj highlighted our significant improvements in measuring semantic correlation, Reviewer kG9e recognized that our paper addresses an important issue in low-shot learning, and Reviewer qvMQ commended the novelty and clarity of our method. We sincerely thank all reviewers for their recognition and valuable insights!

Below, we first address a common concern raised by the reviewers, followed by detailed responses to each reviewer's specific comments. We hope that our clarifications help enhance the understanding of our work.

#### **Common concern: Relationship and differences between prior works (PLOT [1], Wang et al. [2]) and AWT.**

**Relationship:** Both [1] and [2] lay the groundwork for AWT by demonstrating that the cross-modality distance can be effectively measured using Optimal Transport (OT).

**Differences:** AWT distinguishes itself from [1] and [2] in several key aspects:

- **Purpose:** While [1] and [2] utilize OT to learn comprehensive and semantically diverse prompts **during training**, AWT leverages OT to **directly enhance VLMs during inference**.
- **Elements for OT:** Both [1] and [2] apply OT at the **token level**, requiring **fine-grained alignment capabilities** that are often lacking in models like CLIP. This necessitates additional training data to close the gap. In contrast, AWT implements OT at the **holistic level**, treating entire images or sentences as single elements, thus eliminating the need for extra data.
- **Element Distributions:** Both [1] and [2] consider elements as **uniform distributions**, which may not be appropriate. AWT, on the other hand, uses real-time prediction entropy to **dynamically adjust distribution probabilities**. Our ablation studies support that relying on uniform distributions often leads to sub-optimal results (refer to Tables 4e and 4f, Lines 299-304 in our paper).

Consequently, **AWT significantly outperforms both methods** in zero-shot image classification tasks **without any training**. Below, we present a comparative analysis:

| Method          | Train |  Flowers  |    DTD    | OxfordPets | StanfordCars |  UCF101   | Caltech101 |  Food101  |  SUN397   | FGVCAircract |  EuroSAT  |
| --------------- | :---: | :-------: | :-------: | :--------: | :----------: | :-------: | :--------: | :-------: | :-------: | :----------: | :-------: |
| PLOT++  [1]     |   √   |   69.10   |   38.42   |   90.49    |    61.20     |   68.94   |   91.32    |   86.07   |   61.59   |    24.84     |   49.90   |
| Wang et al. [2] |   √   |   73.75   |   46.75   |   90.55    |    65.84     |   69.60   |   93.91    | **86.40** |   67.59   |    24.95     |   47.25   |
| AWT             |   ×   | **75.07** | **55.56** | **92.53**  |  **69.93**   | **72.51** | **95.54**  |   85.54   | **70.58** |  **29.22**   | **58.61** |

[1] Chen, Guangyi, et al. "PLOT: Prompt Learning with Optimal Transport for Vision-Language Models." ICLR 2023.

[2] Wang et al., Dongsheng, et al. "Tuning multi-mode token-level prompt alignment across modalities." NeurIPS 2023.

Meta-review: This work involves the combination of three distinct, already understood, steps to improve the performance of vision-language models on classification centric tasks. The reviewers were broadly supportive of the paper, and agreed that while the methodological novelty was not very high the method performs very well when compared to existing baselines. The evaluation is thorough and the description of the method and analysis of the results is clear. 

Given the above, this AC agrees with the reviewers and recommends that the paper is accepted. The authors are strongly encouraged to take the reviewer recommendations and suggestions into account when revising the paper for the camera ready version. Specifically, they should focus on the following:  
* Add the comparisons flagged by the reviewers, e.g. PLOT, Wang et al, CLIP-Adapter, TIP-Adapter, etc.
* Make reference to Section C2 in the main paper so that it is clearer to the reader why dataset specific information is used by the LLM. Also add the new results from the rebuttal related to dataset descriptions to the appendix. 
* Make the additional inference cost clearer in the results text e.g. point to Fig 8 in this section and add more of the methods from the main results table to the plot. It should be made very clear in the main text how many forward passes you need, and how that is more than some existing methods.  
* Add the results from Fig 2 from the rebuttal PDF to the appendix.
Decision: Accept (poster)