AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation

Authors: yuhan  zhu
yuyang  ji
zhiyu  zhao
gangshan  wu
limin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 06:31:09

Abstract: Pre-trained vision-language models (VLMs) have shown impressive results in various visual classification tasks.
However, we often fail to fully unleash their potential when adapting them for new concept understanding due to limited information on new classes.
To address this limitation, we introduce a novel adaptation framework, AWT (Augment, Weight, then Transport). AWT comprises three key components: augmenting inputs with diverse visual perspectives and enriched class descriptions through image transformations and language models; dynamically weighting inputs based on the prediction entropy; and employing optimal transport to mine semantic correlations in the vision-language space.
AWT can be seamlessly integrated into various VLMs, enhancing their zero-shot capabilities without additional training and facilitating few-shot learning through an integrated multimodal adapter module.
We verify AWT in multiple challenging scenarios, including zero-shot and few-shot image classification, zero-shot video action recognition, and out-of-distribution generalization. AWT consistently outperforms the state-of-the-art methods in each setting. In addition, our extensive studies further demonstrate AWT's effectiveness and adaptability across different VLMs, architectures, and scales.

Main paper: 
 introduction: 
 Recent advances in vision-language models (VLMs) [1] [2] [3] [4] [5] [6] [7] [8] , which undergo extensive pre-training on web-scale image-text pairs, have exhibited remarkable success in various classification tasks. VLMs are trained to associate images with relevant textual descriptions. In the standard protocol (Fig. 1(a) ), raw images and class names are projected into a joint vision-language embedding space, where the class with the shortest distance to the image representation is selected as the prediction result. However, directly using raw images and class names in testing has limitations [1, 9] . Visually, the broad scope of pre-training compels VLMs to analyze all image elements, lacking capability of focusing on specific interested regions. For instance, a model might miss critical facial features of a cat while unnecessarily focusing on irrelevant elements like "bench" and "grass" (Fig. 1(a) ). Textually, since VLM pre-training associates visual elements with diverse and rich textual descriptions (e.g., colors and textures), merely using class names during test falls short of capturing the full spectrum of visual content. To enhance input effectiveness, the literature focuses on post-training prompts (Fig. 1(b) ) that provide contextual cues, thereby helping the model in prioritizing relevant features, such as cat's attributes. However, this approach often depends on the availability of training resources, which may not be always practical.In this study, we are interested in enhancing inputs for better adaptation of VLMs without training prompts. We advocate for data augmentation as a simple yet effective strategy, as depicted in Fig. 1(c) . Techniques like random resized cropping and image flipping enrich the input with varied and multiscale perspectives, while detailed textual descriptions for each class provide richer visual narratives. Although manually crafting diverse descriptions for each class is expensive, employing Large Language Models (LLMs) [34] [35] [36] presents an efficient alternative.Nonetheless, several challenges remain. First, the intra-modal importance of each augmented image and description needs assessment, as not all views contribute equally to class recognition-some may be irrelevant background elements or non-visual descriptors such as the cat's personality. Second, the inter-modal interaction requires consideration, as descriptions such as "dark face" or "light-colored body" might have direct semantic correlations with some image crops (Fig. 1(c )).To tackle these challenges, we propose AWT, a novel framework that augments raw inputs into diverse views, weights view importance in each modality dynamically, and transports semantic correlations across modalities. Initially, AWT augments raw inputs via image transformations and LLMs. Subsequently, it weights the importance of each view on the fly based on its prediction entropy, as more confident predictions typically indicate higher accuracy [37] . This method allows AWT to identify and prioritize significant views, and adjust the importance distribution dynamically according to the task-specific context (e.g., candidate class names). AWT then formulates the image-text distance calculation as an optimal transport problem [38, 39] , considering each augmented view as a quantity of sand. The importance assessed for each view determines the mass of its corresponding sand pile, and distances are calculated using cosine similarity. This formulation can effectively discover cross-modal correlations by solving the optimal transport problem-which minimizes the effort required to transport sand from one modality to another. Additionally, generating class descriptions from LLMs using a simple prompt like "Describe a {class}." often results in overly generic descriptions. Inspired by chain-of-thought approach [40] , we introduce a two-step, dataset-aware prompting method. This approach encourages LLMs to produce class descriptions that are both diverse and dataset-relevant.We implement AWT using the CLIP model [1] and evaluated its performance across 21 datasets covering four challenging tasks: zero-shot and few-shot image classification, out-of-distribution generalization, and zero-shot video action recognition. As shown in Fig. 1(d) , AWT consistently surpasses the existing state-of-the-art methods in each setting. Our extensive analysis further examines AWT's flexibility with diverse architectures, its scalability with different model sizes, and its potential applicability to other VLMs.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: See our results and discussion in Section 4.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Vision-Language Models. Leveraging the extensive pre-training on web-scale text-image pairs, vision-language models (VLMs) such as CLIP [1] and ALIGN [6] excel in acquiring versatile representations that span multiple modalities. These models adeptly embed texts and images into a shared vision-language feature space, enabling the proximity of inputs with analogous semantics. The inherent flexibility of natural language allows VLMs to be effectively utilized across a wide range of open-set tasks including image classification [1, 9, 13] , object detection [41] [42] [43] , image generation [44, 45] , video action recognition [46] [47] [48] . However, such general-purpose models often fail to focus on task-specific details, which can result in sub-optimal performance. This study aims to overcome this limitation by proposing a novel adaptation framework, namely AWT, for VLMs.Adapt VLMs to downstream tasks. Direct adaptation of pre-trained VLMs to downstream tasks often results in suboptimal and unstable performance [9] . To overcome this, the existing literature has primarily focused on the use of post-training to enrich task context. This includes strategies such as few-shot prompt learning [9, 11, 12, 29, 32] , cross-dataset prompt generalization [13-22, 25-27, 30, 31, 33] , unsupervised prompt tuning [23, 24, 28] , test-time prompt tuning [10, [49] [50] [51] [52] [53] [54] , and adapter tuning [55] [56] [57] [58] [59] . Conversely, other approaches aim to augment inputs using various resources such as the WordNet relationship hierarchy [60] , Large Language Models [61] [62] [63] [64] , or Stable Diffusion models [50, 65, 66] . Nonetheless, these methods mainly enhance only one modality. In contrast, our study innovatively applies augmentation to both visual and textual modalities and addresses significant challenges in dual-modality augmentation scenarios.Optimal Transport (OT). Optimal transport (OT), originating from the Monge problem [38] in the eighteenth century, serves as a metric for quantifying the distance between mathematical entities [67] while considering their intricate geometric structures [39] . Historically rediscovered in various forms, OT first gained fame in computer vision under the name of earth mover's distances [68] . The development of efficient approximate solvers [69] has recently propelled a resurgence in OT's popularity, broadening its utility across multiple domains, including object detection [70, 71] , domain adaptation [72] [73] [74] , generative modeling [75] [76] [77] [78] , semantic correspondence [79] , point clouds [80] [81] [82] , prompt learning [83] [84] [85] and video understanding [86, 87] . Of particular relevance to our study are PLOT [83] and Wang et al. [85] , which leverage OT for fine-grained prompt learning to enhance VLMs. Distinct from these two studies, our research diverges by eschewing the need for additional training resources, opting instead for an augmentation-based direction.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Appendix E.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Contrastive Language-Image Pre-training (CLIP). CLIP [1] integrates dual encoders-an image encoder f (•) and a text encoder g(•)-to map images and textual descriptions into a shared visionlanguage (V-L) embedding space. CLIP is designed to minimize the cosine distance between embeddings of semantically related image-text pairs. Thanks to the flexibility of natural language, CLIP enables direct application to classification tasks without the need for task-specific training. For instance, given an image X ∈ R 3×H×W and a set of candidate class names {t i } C i=1 , where C denotes the class count. CLIP computes the embeddings I ∈ R d for the image and {e i } C i=1 ∈ R C×d for all class names, where d is the feature dimension. Subsequently, the classification probability for image X being of class t i can be formulated as:EQUATION)where τ is a temperature parameter.Optimal Transport (OT). Optimal transport (OT) theory, originating from the Monge problem [38] , provides a framework for structural distance measurements. This theory conceptualizes scenarios such as relocating sand at a construction site with the goal of minimizing effort. Mathematically, the initial and target distributions of sands are modeled as discrete measures:EQUATIONwhere δ xi denotes the Dirac with a concentrated mass a i centered at x i , and similarly for β. Here, N and M represent the number of source and target locations, respectively. The cost of transporting Importance decreases Importance ScoreImportance Score sands from any source location x i to any target location y j is given by the cost function c(x i , y j ).To extend the application to broader and more intricate scenarios, e.g., cross-modal correlation, the Kantorovich relaxation [88] is employed. This relaxation introduces flexibility in the transport plan and ensures symmetric transport solutions. The transport plan P ∈ R N ×M, where element P i,j indicates the mass transported from x i to y j , must satisfy the constraints:EQUATIONKantorovich's formulation seeks to minimize the total transportation cost:EQUATIONwhere C i,j = c(x i , y j ) defines the cost matrix.Pre-trained VLMs often underperform when adapted to new concepts due to insufficient information about new classes. Moreover, their extensive pre-training scope leads them to analyze all elements of an image, causing them to miss contextually important cues crucial for specific downstream applications. To overcome these limitations, we introduce a novel framework, termed AWT (Augment, Weight, then Transport), to enhance the adaptability of VLMs without additional training. The AWT framework, as depicted in Fig. 2 , consists of three critical components: augmenting raw inputs to generate diverse and content-rich views, weighting the significance of these views within each modality, and transporting semantically correlated elements across modalities.The augmentation process begins with an image X ∈ R 3×H×W and the class name set {t i } C i=1 , aiming to transform these inputs into various views that offer different perspectives and details.For visual augmentation, we apply standard data augmentation including random resized cropping and random flipping to produce a set of varied views {X n } N +1 n=1 . This set includes N augmented images alongside the original (denoted as the 0 index), enriching the input with diverse and multiscale perspectives. An illustrative example is shown in Fig. 2 .To enrich the textual modality, we utilize Large Language Models (LLMs) to generate class descriptions. Typical prompts like "Describe a {class}." often result in descriptions that are either vague-lacking in specific visual details-or contextually misaligned. For instance, in contexts such as classifying sketches, generic descriptions of a category may not correspond well with the sketch images. To address this, we adopt a two-step, dataset-aware prompt strategy. Initially, we prompt LLMs to generate multiple questions that probe different aspects of the category, which is crucial for eliciting detailed and varied descriptions. To ensure the queries are aligned with the visual content, we incorporate a dataset-level description into the initial prompts. Specifically, we start by asking LLMs to "Generate questions to classify images from a dataset, which {dataset descirption}.". Using the dataset-related questions generated from the first step, we proceed to the second step where these questions are combined with the specific class name to obtain tailored descriptions. The set of augmented views for each class t i is denoted as {t m i } M +1 m=1 , including an additional view formed by the raw class name. This method ensures both diversity in the descriptions and their relevance to the visual content. More details can be found in Appendix C.2.Following augmentation, it is essential to assess the significance of each augmented view, as not all views contribute equally to classification. Some views may be critical while others might be less informative or even noisy. To address this variation, we introduce an entropy-based weighting mechanism to quantify each view's importance. Our key insight is that the impact of a view on classification confidence-a metric often correlated with accuracy [37] -can serve as a proxy for its importance. A view that significantly enhances classification confidence is considered more vital.To assess the importance of n-th image view X n , we maintain a constant text context and compute the averaged embedding for each class as{ē i = 1 M +1 M +1 m=1 e m i } C i=1, where e m i is the CLIP embedding of t m i . The classification probability p(t | X n ) is then calculated using the image embedding I n and text embeddings {ē i } C i=1 , following Eq. ( 1). Predictive confidence is then quantified using the entropy formulaH n (t) = -t p(t | X n ) log p(t | X n ).Lower entropy indicates higher confidence, allowing us to evaluate view importance through the negative entropy as follows:EQUATIONwhere γ v is a temperature parameter adjusting the distribution's sharpness.Similarly, to determine the importance of m-th description for i-th class, i.e., t m i , we calculate the classification probability p i m (t | X 0 ), with the image embedding I 0 and text embeddings {e m i } ∪ {ē j } C j=1,j̸ =i . The classification entropy is given by H i m (t) =t p i m t | X 0 log p i m t | X 0 . We then calculate the importance scores for all descriptions within the i-th class as follows:EQUATIONwhere γ t is the temperature parameter. This entropy-based weighting mechanism ensures the prioritization of the contextually significant views. By dynamically adjusting the importance based on the direct impact on classification confidence, the augmented view sets can be well-prepared for the optimal transport process.Our primary goal is to precisely measure the distance between an image and its candidate names. Through the augmentation, we have transformed each original image or class name into a set of augmented views. Typically, the distance between these sets is measured by simply averaging the embeddings within each set. However, such practice often fails to capture the dynamic correlation across modalities. Consider the scenario depicted in Fig. 2 , where specific textual descriptions such as "has a dome-shaped top" might correlate directly with certain image crops. The conventional averaging strategy typically overlooks these intuitive and meaningful correlations.To address this issue, we propose a novel approach by formulating distance measurement as an optimal transport (OT) problem, which facilitates richer interactions between modalities. We model each view within the V-L space as a mass located at its embedding position:EQUATIONTable 1 : Zero-shot image classification. We report top-1 accuracy (%) for each dataset. The "Train" column indicates whether the methods necessitate additional training (including test-time training). Numbers in grey indicate that the method was trained on ImageNet and is therefore not zero-shot. Here, the importance weight of each view, derived from Eqs. ( 5) and ( 6), determines the mass allocation. The transportation cost between any two points (e.g., an image and a textual description) is quantified using the cosine distance between their embeddings, C = 1 -cos(I, e), which serves as an intuitive measure of semantic similarity. The goal of optimal transport is to minimize the total cost of transporting mass from visual modality into textual modality. Specifically, the distance between the image view set {X n } N +1 n=1 and i-th class description set {t m i } M +1 m=1 is redefined as an OT problem between α and β i , as formulated in Eq. ( 4). We employ Sinkhorn's Algorithm [69] to efficiently approximate the solution, denoted as P. Consequently, the classification probability can be expressed as:EQUATIONEQUATIONwhere s = i j Pij (1 -C) ij . By employing the optimal transport framework, we ensure that semantically related views receive more attention, enhancing the accuracy and relevance of the classification process.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
zero-shot image tasks: 
 Datasets. For zero-shot image tasks, we consider image classification and out-of-distribution (OOD) generalization. Our study encompasses 18 datasets that span a wide array of recognition tasks: ImageNet [89] , Caltech101 [95] and Caltech256 [101] for generic object recognition, Oxford-Pets [92] , StanfordCars [93] , OxfordFlowers [90] , Food101 [96] , FGVCAircraft [98] , Birdsnap [100] and CUB [102] for fine-grained classification, SUN397 [97] for scene recognition, DTD [91] for texture classification, EuroSAT [99] for satellite recognition, and UCF101 [94] for action recognition. Besides, four ImageNet variant datasets are involved to assess the model's capability for OOD generalization: ImageNet-A [103] , ImageNetV2 [104] , ImageNet-R [105] and ImageNet-Sketch [106] .Competitive methods. We mainly compare three distinct categories of approaches: 1) Prompt learning methods: These involve additional data to post-train visual or textual prompts, including CoOp [9] , CoCoOp [13] , MaPLe [15] , PLOT++ [83] , POMP [22] . 2) Test-time prompt tuning methods: These optimize prompts during inference, such as TPT [49] , DiffTPT [50] , and Promp-tAlign [52] . 3) Augment-based method: These use LLMs or diffusion models to augment inputs, including CuPL [63] , VisDesc [61] , WaffleCLIP [62] , and SuS-X-SD [65] .Implementation details. We implemented the AWT framework using the CLIP-B/16 model [1] . Image augmentations include random resized cropping and flipping, and class descriptions are generated via GPT-3.5 [35] . We set the number of augmented images N and descriptions M to 50 each. Dataset-level descriptions are provided in Appendix C. For both visual and textual modalities, we configured the importance distribution temperatures at γ v = 1/2 and γ t = 1/2. The optimal transport problem is approximated using Sinkhorn's Algorithm with an ϵ of 0.1 [69] . All experiments are conducted on one NVIDIA A100-SXM4-80GB GPU. Setup. Here, we focus on the zero-shot video action recognition task, using three representative datasets: UCF101 [94] , HMDB51 [107] , and Kinetics-600 [108] . For UCF101 and HMDB51, we adopt two evaluation protocols: 1) EP1: test model on all 101 UCF classes and 51 HMDB classes [47, 117] , and report the top-1 accuracy.2) EP2: evaluate the model using three official splits and averaging the results of each split. The average top-1 accuracy and standard deviation are reported. For Kinetics-600, the top-1 accuracy and standard deviation are reported on three validation sets split by Chen and Huang [118] .Implementation details. To model temporal dynamics, we follow Open-VCLIP [47] to use neighbor-frame attention and fine-tune CLIP on Kinetics-400 [119] . Note that three test subsets of Kinetics-600 have a disjoint class set compared to Kinetics-400. All AWT configurations are the same for zero-shot image tasks except for the visual augmentation, in which we directly use the different sampled temporal and cropped video frames.Results. In Tab. 3, we present a comparison of our AWT with existing CLIP-based zero-shot video action recognition methods. Although AWT was not originally tailored for video tasks, it sets new records in this domain, outperforming the recent state-of-the-art method, FROSTER, by 1.6% and 2.4% on HMDB51, and by 1.3% on Kinetics-600. These results suggest that our AWT framework could be effectively extended to video understanding tasks.Setup. We assessed the few-shot transfer capabilities of our method across 11 datasets: Ima-geNet [89] , Caltech101 [95] , OxfordPets [92] , StanfordCars [93] , OxfordFlowers [90] , Food101 [96] , FGVCAircraft [98] , SUN397 [97] , DTD [91] , EuroSAT [99] , and UCF101 [94] . We trained our model using 1, 2, 4, 8, and 16 shots. Results are averaged over three runs.Implementation details. All AWT configurations are the same as zero-shot image tasks. In this task, we introduced a multi-modal Adapter for efficient learning, inserted after each Multi-Head Self-Attention and MLP block in every layer. We adopt a distillation technique similar to [21] to prevent overfitting. Further details about the Adapter are available in Appendix C.Results. In Fig. 3 , we compare the performance of AWT with existing methods in the few-shot transfer learning task. Impressively, AWT surpasses the previous state-of-the-art by average accuracies of 2.76%, 2.16%, 1.62%, 1.57%, and 1.75% for 1, 2, 4, 8, and 16 shots respectively. Notably, on the ImageNet dataset, AWT significantly outperforms all prior methods. While PLOT++ also utilizes optimal transport, it is limited to local image features and class names, neglecting the multiscale image perspectives and the rich textual semantics, leading to suboptimal transfer capabilities. In contrast, AWT leverages diverse augmented views, effectively maintaining intra-modal importance trade-offs while establishing dynamical cross-modal correlations, achieving superior few-shot performance.Main component analysis. In Tab. 4a, we analyze the key components of AWT. Initially, we augment raw inputs and apply basic ensembling. The results (rows two and four) show that directly augmenting images is ineffective, likely due to background image crops. Conversely, textual enhancements significantly boost performance, thanks to our carefully designed prompt strategies for LLMs.We then shift from basic ensembling to optimal transport (OT), leading to consistent improvements across two tasks. However, the full potential of OT is not realized due to ineffective mass (i.e., importance) weighting. By incorporating our entropy-based weighting method, which accurately assesses the importance of each view, we again achieve substantial performance gains.Number of augmentation views. We present the study on the augmented view quantities for both visual and textual sides in Tabs. 4b and 4c, respectively. The results clearly demonstrate that performance tends to increase with the number of views. Our findings suggest that about 50 views per modality are sufficient to achieve decent performance. The number of augmentation views is crucial for AWT's effectiveness. Given that AWT is augmentation-driven, this correlation is intuitive. However, increasing the augmented view quantities can also lead to higher computational costs during inference, we also include an efficiency-performance trade-off analysis in Fig. 9 .LLM prompting strategy. We evaluated the effectiveness of our LLM prompting strategy, detailed in Tab. 4d. Our method is compared with two established approaches: VisDesc [61] and CuPL [63] .VisDesc uses a uniform prompt template across different datasets, while CuPL employs a tailored, dataset-specific manual prompting strategy, enriching context for LLMs. We developed a refined twostep process that enhances context comprehension through dataset-level descriptions and increases diversity by utilizing chain-of-thought queries. Our strategy consistently outperforms the existing methods in both evaluated tasks.Temperature in weighting. We evaluated our entropy-based weighting method by conducting an ablation study on the softmax function's temperature parameter (see Eqs. ( 5) and ( 6)). A higher temperature creates a more uniform importance distribution. The findings for both modalities are presented in Tabs. 4e and 4f, respectively. Our results reveal that a very high temperature (e.g., 100) leads to suboptimal performance, likely due to insufficient emphasis on contextually significant views. Conversely, lowering the temperature enhances focus on these important views, improving performance. Empirically, a temperature of 1/2 has been identified as optimal for both modalities.For a clearer understanding of our weighting strategy, visualizations are provided in Appendix A.Our AWT is applicable to any VLM using dual encoders to map images and text into a joint space with appropriate distance metrics (e.g.. cosine similarity). Therefore, it is crucial to assess AWT's effectiveness across various scenarios. We conduct evaluations with both ResNet [120] and ViT [121] architectures, explore AWT's scalability from ViT-B/32 to ViT-L/14@336, and assess its generalizability across three VLMs: ALIGN [6] , SigLIP [122] , and EVA-CLIP [123] . We conduct experiments across 18 image datasets and present the results in Fig. 4 . Our findings reveal that AWT consistently achieves performance gains in all tested scenarios, highlighting its broad applicability.Although AWT has shown success across various datasets and tasks, we identified certain limitations when applying it to the CIFAR datasets. As detailed in Tab. 5, AWT resulted in performance declines of 0.91% and 0.17% on two CIFAR datasets, respectively. To investigate this issue, we analyzed the images produced by the transformations used in our study. We discovered that with low-resolution images, such as 32 × 32 pixels, the random resized crop operation tends to overly blur the images, obscuring the objects within them, as illustrated in Fig. 5 . To address this, we integrated a diffusion model, specifically DALL•E 2 [44] , as a substitute for traditional data augmentations. Fig. 5 shows examples of enhanced image views generated by DALL•E 2, which produce sharper images and offer a variety of perspectives. By incorporating this advanced technique into the AWT framework, we have significantly improved its performance. The updated benchmark results, presented in Tab. 5, demonstrate that AWT now consistently achieves performance gains over baselines. 
conclusion: 
 In this paper, we have introduced the AWT (Augment, Weight, then Transport) framework, designed to enhance the transferability of pre-trained vision-language models (VLMs). Rather than using raw images and class names directly, our approach enriches the inputs by augmenting them with diverse visual perspectives and detailed class descriptions. We further develop an entropy-based weighting strategy to dynamically prioritize these augmented views and employ optimal transport to measure the cross-modal distance in the structured visual-language space. The AWT framework not only boosts the zero-shot performance of VLMs without the need for additional training but also facilitates few-shot transfer learning via an integrated multimodal adapter module. Our evaluations across four challenging tasks demonstrate that AWT significantly outperforms existing state-of-the-art methods.Through these visualizations, we observe two primary benefits: 1) Priority to Significant Views:The strategy effectively prioritizes image views that are intuitively significant, while efficiently excluding irrelevant background crops. 2) Dynamic weight assignment: In different contexts, such as varying candidate class names, the strategy dynamically assigns weights, emphasizing views that are contextually important.They have a distinctive marbled or spotted fur pattern.They might have their fur raised and ears flattened when scared.They have large, rounded eyes that can be gold or green.Their legs are long and well-proportioned to its body.They tend to have sturdy, straight legs and a curled tail.They have a thick, fluffy white coat.Its ears are erect and triangular, covered in fur.The nose is black, contrasting with its white fur.They have short, smooth fur that lies close to body.They display excitement by jumping around.Their fur color include tan, brown, white, black.They may have a wide-eyed look with their ears perked up. Visualization of weighting class descriptions. The strategy not only enhances image view management but also optimizes the handling of textual descriptions. In Fig. 7 , we observe the following: 1) For descriptions related to the ground-truth class, our strategy prioritizes key textual elements while effectively filtering out irrelevant content. 2) For non-matching classes, the importance levels are relatively uniform, indicating a general irrelevance to the specific image content. 
 
 Appendix: Bengal's spotted fur.Bengal's triangular ears.Bengal's oval-shaped eyes.Bengal's energetic legs. After augmenting the input image and class names to create diverse views, there is a potential for these views to exhibit direct and meaningful correlations across modalities. For instance, image crops that focus on the eyes of a cat could closely correlate with textual descriptions concerning the eyes. In our main paper, we introduce the use of optimal transport (OT) to effectively handle these correlations. To illustrate this, we provide a visualization of the meaningful correlations captured by the OT approach. As depicted in Fig. 8 , the visualization of the transport plan clearly shows that image and text pairs with direct semantic relationships are given priority. Conversely, pairs that are semantically irrelevant are typically disregarded by the transport plan. AWT is assessed using 2, 5, 10, 25, and 50 image views to present its trade-off between performance and computational efficiency. We show the results on both zero-shot (left) and few-shot (right) tasks.Our AWT introduces additional computational costs during inference by utilizing multiple views.Although text embeddings can be pre-computed as suggested in [1] , the primary expense arises from processing additional image views. In our study, we explore the trade-off between performance and efficiency by employing 2, 5, 10, 25, and 50 augmented views of the input image. We assess this trade-off by reporting both the frames per second (FPS) and the average accuracy in zero-shot and few-shot settings, as shown in Fig. 9 .The results indicate that even with a limited number of views, AWT can substantially outperform existing methods. As the number of views increases, there is a consistent enhancement in performance, albeit at the expense of reduced inference speed. This analysis enables practitioners to tailor AWT application to real-world scenarios by selecting an optimal balance between accuracy and processing speed, based on specific requirements.In the few-shot settings, we incorporate two additional adapter modules at each transformer layer to facilitate transfer learning. This configuration initially makes our method slower than other prompt learning approaches, likely due to the adapters requiring sequential processing, whereas prompts are processed in parallel. Considering the computational costs, we propose that AWT could be more efficiently transferred using test-time zero-cost methods like LoRA [125] , which allow for structural reparameterization after training. In this study, we explore the impact of applying multi-modal adapters to both visual and textual modalities. We conducted experiments using an unimodal adapter either on the visual or textual side and analyzed their performance on the ImageNet dataset. The results are detailed in Tab. 7.Our findings indicate that when training samples are extremely limited, such as in a 1-shot scenario, fine-tuning the visual adapter yields more benefits. However, as the number of training samples increases, fine-tuning the textual adapter becomes more advantageous. This shift is likely due to the need to reduce the similarity between class embeddings as suggested in [126] . Despite these variations, our results ultimately show that fine-tuning both modalities simultaneously delivers the best overall performance. Previous studies [9, 13] suggested assessing method robustness by directly applying a model trained on ImageNet to four variant datasets derived from ImageNet. We adhere to this protocol and present our findings in Tab. 8. The results demonstrate that our approach, AWT, not only excels in few-shot performance on ImageNet but also outperforms in domain generalization robustness across the four ImageNet variant datasets, effectively balancing the adaptation-generalization trade-off. In this study, we utilize GPT-4o [36] and Qwen-Plus [127] for our description generation process, integrating them into AWT. The benchmark accuracy is presented in Tab. 9. Overall, AWT maintains robust performance across different LLMs. Interestingly, more advanced LLMs do not necessarily lead to significant performance improvements. This may be because AWT primarily relies on the simple instruction-following and knowledge-recall capabilities of LLMs, for which models like GPT-3.5 are sufficient. Here, we examine the performance of AWT when no dataset-level description is available, a situation often encountered in real-world applications. We introduce a base version of AWT, which replaces our original instruction with "Generate questions to classify images." while leaving other components unchanged. The performance of AWT-base is detailed in Tab. 10. Despite its simplicity, AWT-base demonstrates favorable performance compared to previous methods.For the benefit of future research, we present numerical values of our experimentation in the few-shot transfer learning setting in Tab. 11. For few-shot learning, we employ a multi-modal adapter to achieve parameter-efficient transfer learning. The architecture of the multi-modal adapter is depicted in Fig. 10 . We integrate the adapter module subsequent to the Multi-Head Self-Attention and the MLP module within each transformer layer. The adapter module is applied to both the visual and textual side of the CLIP model. The core of the adapter module is a bottleneck structure designed to efficiently manage model parameters while maintaining performance. This structure comprises a down-projection layer W down ∈ R d× d λ that reduces the dimensionality of the input features, and an up-projection layer W up ∈ R d λ ×d that restores the original dimensions. Between these layers, a GeLU activation layer [130] introduces non-linearity. A learnable scale parameter s is used to modulate the output of the adapter. The feature dimension is denoted by d. For an input feature vector x ∈ R L×d , where L is the sequence length, the adapter's forward process is expressed as:EQUATIONEQUATIONDetailed settings and hyperparameters for our few-shot learning experiments are outlined in Tab. 12. C.2 Two-step dataset-aware prompting for LLMsHere, we provide additional details of our proposed two-step dataset-aware prompting strategy for LLMs. Initially, in the first step, we leverage a dataset-level description for each dataset to inspire the generation of diverse and contextually relevant questions. The descriptions for each dataset are sourced from their official websites or from the paperswithcode website, and we include the dataset description and the reference to the source's URL in Tab. 13.To better illustrate the effectiveness of our strategy, we present a sample of questions generated in the first step. For each dataset, we list three examples in Tab. 14. From these examples, we can observe the following: 1) the questions generated are not only more diverse compared to traditional prompts such as "Describe a {class}", but they also delve into more detailed inquiries, and 2) the questions incorporate dataset-specific information, thereby aiding LLMs in producing more accurate and relevant descriptions.Datasets. Below are the datasets used in this paper that have known license information: MIT License: ImageNet-A [103] , ImageNetV2 [104] , ImageNet-R [105] , ImageNet-Sketch [106] ,EuroSAT [99] . CC BY 4.0 License: Caltech101 [95] , Caltech256 [101] , HMDB51 [107] , K400 [119] , K600 [108] . CC BY-SA 4.0 License: OxfordPets [92] .Source code. Source code used in this paper are under the MIT License: CLIP [1] , CoOp [9] , TPT [49] , PLOT [83] . This paper introduces AWT to enhance the transferability of vision-language foundation models. We have evaluated the effectiveness of AWT across a variety of classification tasks, employing different types, scales, and architectures of VLMs. Our findings suggest that AWT holds significant potential for integration into diverse downstream applications involving VLMs. Beyond classification tasks, we anticipate that the insights gained from this study will stimulate further research into other areas of computer vision, such as object detection and semantic segmentation. Currently, we are not aware of any ethical issues associated with the real-world applications of this technology. Nonetheless, continuous monitoring and evaluation are recommended to ensure responsible deployment.While the AWT framework achieves commendable performance using a limited number of augmented image views, attaining higher performance necessitates an increase in the number of views. This expansion, however, inevitably leads to additional computational costs. Notably, the visual elements in different augmented views often contain duplicates, particularly when they are all derived from the same original image (e.g., random cropping). An interesting avenue for future research would be to explore strategies for reducing duplicates among batches to enhance inference speed. Furthermore, the application of test-time augmentation extends beyond our current scope and holds potential for various other domains, such as video understanding [131, 132] , frame interpolation [133] [134] [135] , semantic segmentation [136] [137] [138] , and action detection [139] . Another intriguing area for future research involves the integration of efficient and controlled diffusion models [140, 141] to enhance the quality of visual augmentations.NeurIPS Paper Checklist 


Decision: This work involves the combination of three distinct, already understood, steps to improve the performance of vision-language models on classification centric tasks. The reviewers were broadly supportive of the paper, and agreed that while the methodological novelty was not very high the method performs very well when compared to existing baselines. The evaluation is thorough and the description of the method and analysis of the results is clear. 

Given the above, this AC agrees with the reviewers and recommends that the paper is accepted. The authors are strongly encouraged to take the reviewer recommendations and suggestions into account when revising the paper for the camera ready version. Specifically, they should focus on the following:  
* Add the comparisons flagged by the reviewers, e.g. PLOT, Wang et al, CLIP-Adapter, TIP-Adapter, etc.
* Make reference to Section C2 in the main paper so that it is clearer to the reader why dataset specific information is used by the LLM. Also add the new results from the rebuttal related to dataset descriptions to the appendix. 
* Make the additional inference cost clearer in the results text e.g. point to Fig 8 in this section and add more of the methods from the main results table to the plot. It should be made very clear in the main text how many forward passes you need, and how that is more than some existing methods.  
* Add the results from Fig 2 from the rebuttal PDF to the appendix.

Reviews: review: paper_summary: paper_summary: This paper proposes AWT, an adaptation framework that can boost pre-trained vision-language models (VLMs) for understanding new concepts on new classes. To summarize, AWT first do augmentations on both visual images and textual class names. Then an entropy metric is employed for weighting multiple augmented data. Finally, it measures the distance by OT algorithm associated with the confidence weight computed from the W step. AWT performances best in multiple scenarios including ZS, FS image classification and action recognition and OOD generalization.
strength_weakness: strength_weakness: 1.Augmenting images by simple random resized cropping and random flipping seems to be too naïve compared with nowadays prompting methods. As in previous work SuS-X [1], a much more diversified Support Set can be constructed by stable diffusion and retrieval. Meanwhile, according to Tab 4(a), we can not see any improvement by introducing visual augmentations (Raw inputs vs. A(Img.)). 
[1] SuS-X: Training-Free Name-Only Transfer of Vision-Language Models. Vishaal Udandarao et al.
2.The final inference step, measuring distance across two modalities by OT algorithm is not new as it was proposed in [2] and [3]. However, the difference with these two studies is not clearly claimed in the AWT paper. The original statement “Distinct from these two studies, our research diverges by eschewing the need for additional training resources, opting instead for an augmentation-based direction”  looks confused and fuzzy.
[2] Tuning Multi-mode Token-level Prompt Alignment across Modalities. Dongsheng Wang et al.
[3] PLOT: Prompt Learning with Optimal Transport for Vision-Language Models. Guangyi Chen et al.
questions: questions: 1.What about applying more complex visual augmentations more than random cropping and flipping? How about introducing additional visual training data like Support Set in SuS-X?
2.Why only 50 class descriptions M are generated? The performance seems not saturated at M=50 according to Tab 4(c).
3.Any inference speed limitation? Since 50 times more images and class descriptions are generated by augmentation.
4.Are the evaluation results stable or constant? If AWT generate different image crops or different class descriptions (affected by GPT version) in multiple experimental trials, then the weighting and OT processes should be affected, which may give different final distance measure. If so, then the mean and variance of several rounds of experiments should be reported.
limitations: limitations: No limitation discussion in the original paper.
paper_summary: paper_summary: This paper proposes a novel adaptation framework for vision-language models, which replaces point-to-point alignment of text and images with set-to-set alignment. Specifically, the authors use image transformation and LLM generation to create augmentation sets. After weighting the augmentations based on entropy, optimal transport is used to compute the distance between vision and textual sets to measure their semantic correlation. Experiments are conducted on standard benchmarks.
strength_weakness: strength_weakness: 1. PLOT [71] and Wang [73] also introduce optimal transport (OT) into vision-language models and are closely related to this work. The relationship and differences between these works need to be clearly clarified.
2. Some recent works should be acknowledged. For example, [a, b, c] have also been proposed for the adaptation of vision-language models. These works were accepted by CVPR 2024 and typically appeared on ArXiv in March, about two months before the NeurIPS submission deadline. Therefore, it is recommended to discuss these works as well.
3. The computational complexity should be analyzed and compared against prior methods. 

[a] Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models, CVPR2024

[b] MMA: Multi-Modal Adapter for Vision-Language Models, CVPR2024

[c] Efficient Test-Time Adaptation of Vision-Language Models, CVPR2024
questions: questions: See weaknesses, especially the difference and connection with [71,73]
limitations: limitations: NA
paper_summary: paper_summary: The paper considers a task of low-shot (zero-shot and few-shot) adaptation of vision-language models (VLMs). The proposed approach is based on diversifying the inputs of VLMs using augmentations for images and LLMs for class names. Each input is then weighted based on its prediction entropy, while optimal transport is employed to measure the distance between the test image and classes. The evaluation is performed across a wide range of datasets for different tasks, such as zero-shot classification and video action recognition, few-shot classification, and out-of-distribution generalization.
strength_weakness: strength_weakness: - Prompts for LLM to generate class descriptions contain a description of the dataset, which is a slight limitation. It would be beneficial to show the result for prompting without knowing the dataset description. CuPL and VisDesc comparisons in Table 4d are going in this direction; however, considering that the LLM version could be different, they are not a direct comparison.
- The paper shows that simple averaging of features from multiple image and text inputs does not work. However, a comparison with weighted averaging (weights based on prediction entropy) is missing. This would give a proper indication of how important optimal transport is in the method.
- The related work section could be significantly extended, e.g., methods based on adapters (CLIP-Adapter [1]) and based on memory (TIP-Adapter [2], APE [3]) are not mentioned at all. Additionally, a comparison with APE for few-shot classification would be beneficial.

*Minor comments:*

- Are $\gamma_v = 2$ and $\gamma_t = 2$ as in L217 or are the equal to $1/2$ as in tables 4e and 4f? 

*References:*

[1] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao, "CLIP-Adapter: Better Vision-Language Models with Feature Adapters", IJCV, 2024

[2] Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li, "Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification", ECCV, 2022

[3] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, Peng Gao, "Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement", ICCV, 2023

**After rebuttal**
The rebuttal has adequately addressed all of my concerns from the initial review.
questions: questions: - What is the performance of the proposed method when using LLM prompts that do not include a description of the dataset?
- What is the performance of the method without optimal transport but using weighted averaging of features?
limitations: limitations: The authors have adequately addressed the limitations of the work.
paper_summary: paper_summary: This paper introduces an adaptation framework AWT to enhance vision-language models. The authors propose to augment inputs from visual and textual perspectives. The augmented inputs are dynamically reweighted based on the prediction entropy. Finally, the authors propose to use optimal transport to find the best prediction from VLM. The experimental results demonstrate strong performance under various settings.
strength_weakness: strength_weakness: - Performance-efficiency trade-off under few-shot settings. In supplementary material Figure 8, the performance increases with more augmented images under a zero-shot setting. I'd like to see a similar analysis under the few-shot image classification setting, which can further improve the significance of this work.
- How do the authors decide the hyper-parameters? The authors provide the average results of 14 datasets and the OOD under different choices of hyper-parameters in Table 4. However, it seems to lack details about how the hyper-parameters were chosen. Could authors provide more information on this?
- Why choose flipping as one of the image augmentation strategies? If I understand correctly, image augmentation aims to enable the VLM to capture the fine-grained matching between image regions and different text descriptions, which is already achieved by random crop.
questions: questions: Please see the weakness above.
limitations: limitations: The limitation of this paper is sufficiently discussed by the authors.

Comments: title: Rebuttal reply
comment: Dear Reviewer CYNL,

We would like to remind you that there are only 4 hours remaining in the discussion phase. Please let us know if all your concerns and questions have been addressed satisfactorily.

We look forward to your feedback.

Best regards,

Authors of Paper #8459
Dear Reviewer CYNL,

We greatly appreciate the time and effort you have dedicated to reviewing our submission. As the discussion phase is set to conclude in 48 hours, we wish to ensure that all your queries and concerns are thoroughly addressed. Please do not hesitate to reach out if there are additional aspects of our submission that you would like us to clarify or elaborate on.

Thank you once again for your insightful feedback.

Best regards,

Authors of Paper #8459
Thanks for your responses. My concerns have been addressed and I will increase my score to 6.
The rebuttal has adequately addressed all of my concerns from the initial review. As a result, I will raise my score to accept.
Thank you for your positive recognition of our work!
The authors' response addresses all my concerns. I will raise my score.