Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model
Authors: Yushu Li
Yongyi Su
Adam Goodge
Kui Jia
Xun Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-Language Model
Label Propagation
Training-Free
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A graph-based method for VLM label-efficient adaptation, enabling dynamic graph construction with computationally efficient label propagation and context-aware feature re-weighting without task-specific tuning.

Abstract: Vision-language models (VLMs) have revolutionized machine learning by leveraging large pre-trained models to tackle various downstream tasks. Although label, training, and data efficiency have improved, many state-of-the-art VLMs still require task-specific hyperparameter tuning and fail to fully exploit test samples. To overcome these challenges, we propose a graph-based approach for label-efficient adaptation and inference. Our method dynamically constructs a graph over text prompts, few-shot examples, and test samples, using label propagation for inference without task-specific tuning. Unlike existing zero-shot label propagation techniques, our approach requires no additional unlabeled support set and effectively leverages the test sample manifold through dynamic graph expansion. We further introduce a context-aware feature re-weighting mechanism to improve task adaptation accuracy. Additionally, our method supports efficient graph expansion, enabling real-time inductive inference. Extensive evaluations on downstream tasks, such as fine-grained categorization and out-of-distribution generalization, demonstrate the effectiveness of our approach. The source code is available at https://github.com/Yushu-Li/ECALP.

Published as a conference paper at ICLR 2025
EFFICIENT AND CONTEXT-AWARE LABEL PROPAGA TION FOR ZERO-/FEW-SHOP TRAINING-FREE ADAP- TATION OF VISION-LANGUAGE MODEL
Yushu Li Yongyi Su Adam Goodge", Kui Jia", Xun Xu
South China University of Technology
2 Institute for Infocomm Research (I²R), A*STAR
3 School of Data Science, The Chinese University of Hong Kong, Shenzhen
Shanghai AI Laboratory
eeyushul i @mail........mmam scut. edu. en, eesuyongyi@mail scut. edu. cn, goodge..dam.dav wwt.aaaiiddddadiee...... a-star. edu. edu.sq, kuijiaßcuhk, edu. en, xu xun0i22 a-star, edu.
ABSTRACT
Vision-language models (VLMs) have revolutionized machine learning by lever- aging large pre-trained models to tackle various downstream tasks. Although la- bel, training, and data efficiency have improved, many state-of-the-art VLMs still require task-specific hyperparameter tuning and fail to fully exploit test samples. To overcome these challenges, propose a graph-based approach for label- efficient adaptation and inference. Our method dynamically constructs a graph over text prompts, few-shot examples, and test samples, using label propagation for inference without task-specifi tuning. Unlike existing zero-shot label propa- gation techniques, our approach requires no additional unlabeled support set and effectively leverages the test sample manifold through dynamic graph expansion. We further introduce a context-aware feature re-weighting mechanism to improve task adaptation accuracy. Additionally, our method supports efficient graph expan- sion, enabling real-time inductive inference. Extensive evaluations on downstream tasks, such fine-grained categorization and cittr--distribution generalization, demonstrate the effectiveness of our approach. The source code available at htttp:://githu: com www..occooo/o//oeeeco...
INTRODUCTION
Emerging foundation models have transformed the traditional paradigm of machine learning model development (Jia et al., 202 Kim al. 202T Li al. 2022). By pre-training vision-language models (VLMs) on image-laageeeee language pairs at massive scale, these models have demonstrated strong inference capabilities across wide range of downstream tasks, all while reducing the need for
extensive data collection and labeling (Zhou et al. 2022b; Zhang et al. 2022 Shu et al. 2022).
Recent research on adapting pre-trained VLMs to downstream tasks has primarily focused on im- proving label efficiency, training efficiency, and data efficiency. DMN (Zhang et al. (024b) repre- sents a comprehensive approach that addresses all three dimensions. Specifically, DMN leverages text prompts, test samples, and few-shot samples construct three-branch classifier, with final predictions being the fusion of the individual branches. While DMN achieves state-of-the-art per- formance fine-graine grained categorization tasks, it requires tuning task-specific coefficient to fuse predictions, often using the test set for hyperparameter tuning in the absence of a dedicated valida- tion set. Additionally, DMN introduces memory bank of test samples to synthesize classifier for each sample. However, we hypothesize that these test samples can be used more effec- tively-not just for classifier synthesis but also better capture the data manifold for transductive inference (Joachims) 2003), particularly when labeled samples are limited.
adaptive
'Equal contribution. *Coresponndence Xun Xu: xu_xun x wtw.aiiraataarrdd....... This work was done during Yushu Li and Yongyi Su's visit to I'R.

---

Published as a conference paper at ICLR 2025
To reduce the need for task-specific hyperparameter tuning and better utilize test samples, pro- pose a graph- based adaptation and inference method for downstream tasks. Our approach dynami- cally constructs graph based on available few-shot samples (if applicable), test samples, and text prompts linked to semantic labels. This graph captures the intrinsic data manifold, and we use label propagation (Zhu & Ghahramani 2002) for inference. Unlike DMN, this method eliminates the need for task-specific hyperparameter tuning and enhances the use of information embedded in the unlabeled test samples.
ZLaP (Kalantidis et al. 2024) recently introduced a zero-shot VLM adaptation method based on label propagation, employing an external dataset to build the manifold and using a closed-form solution to propagate labels from text prompts to test samples. However, we argue that ZLaP is suboptimal for label-efficient VLM adaptation for three key reasons. First, a closed-form solution becomes computationally expensive for larger datasets like ImageNet, which includes 50,000 test samples, due to the costly inversion of the Laplacian matrix, limiting the graph's connectivity. Sec- ond, using static graph based solely on the training set without incorporating test samples fails to leverage the test data manifold, which can lead to performance degradation when there is a dis- tribution shift between the training and test sets. Third, relying cosine similarity to measure affinities between test samples and prompts can be problematic. Since VLMs are pre-trained on diverse image-text pairs, their vision-encoded features may capture irrelevant semantic information, such as background objects or image style, making cosine similarity biased for downstream tasks.
Dynamic Graph Expansion
Context-aware Edge Re-weighting
Incremental Label Propagation
* - n. & - W] u, (Nrrm(dtugte))))
2W 12 Y. [P. g1
Test Samples
Calculate Smilrrity S +
Imoige Fnosder
Previous Graph
Wilf - Norm (dlng (u)
uiyy' (1 agye
Sample
Update Pseudo Label Y
Rabbii
A photo of a <cllsss>
Swan
Text Enender
Dog
K-NN Graph
Image
(Optomab Dew-dbbt Sazipl
Encoder
Sample
Sample
Pronnage
Sompls
Memory Updar
Figure I:
Illustration of the overall framework ECALP. We use the text prompts, few-shot sam- ples, and test samples to build the graph. The graph is dynamically expanded upon seeing new sample with context-aware re-weighted edge weights. An iterative solution is adopted to predict labels for all test samples.
In this work, we introduce holistic label propagation approach label-efficient adaptation of VLMs with an overview presented in Fig. T To address the challenge of computational efficiency, we employ an iterative solution to label propagation instead of the closed-form solution, which could benefit from incremental label propagation and label reset. Furthermore, we employ context- aware feature dimension re- weiiggttiing to better adapt to downstream tasks. We require only the text prompts and/or few-shot samples to provide contextual information for feature re- re-weighting. We also present an efficient graph expansion mechanism to allow inductive inference on a stream of test samples, without requiring all test data for transductive labeling. Our proposed method is referred to as Efficient and Context- range of downstream tasks, including fine fine-grained categorization, distribution shift and few-shot categorization. We summarize the contributions of this work as follows:
Label Propagation (ECALP). Experiments are conducted on a wide
We propose unified label-efficient adaptation of vision-language models from a label propagation perspective. Compared with state-of-the-arr methods, the proposed method achieves higher inference speed and fixed hyperparameters.
We account for the diversity of information captured by the vision encoder and propose to re-weight the vision feature dimensions using the statisties of the text embeddings and/or few-shot samples' features from the downstream task.
2

---

Published as a conference paper at ICLR 2025
We demonstrate the effectiveness of approach through experiments on wide range of downstream tasks, including fine- geeenga categorization and out-of-distribute generaliza- tion, achieving state-of-the aftt results.
RELATED WORK
2.1
ZERO/FEW-SHOW ADAPTATION OF VISION-LANGUAGE MODEL
Vision-language models (VLMs) Zhang et al. 2024a Jia et al. 2021 Li ct al. 2022 2023)
have achieved remarkable success open- open-vocabulary classification tasks. However, their per- formance significantly declines in fine-grained scenarios (Zhao et al.) (017) or under distribution shifts (Hendrycks & Dietterich] 2019), highlighting the limitations of large-scale pre-training. To address these challenges, two primary strategies have been developed to improve VLM adaptability in zero-shot and few-shot settings: prompt tuning and adapter tuning.
Prompt tuning Yoon et al., 2024 Feng al., 2023), inspired by techniques in language mod- els (Brown et al.J 2020). improves performance by optimizing learnable input prompts. For in- stance, CoOp (Zhou al. 2022b) and CoCoOp (Zhou et al., 2022a) replace static text prompts with learnable word vectors, fine-tuned via few-shot classification loss. TPT (Shu et al. (022)) and Swap- Prompt (Ma al. 2024) extend this concept zero-shot settings, employing contrastive learning on augmented image inputs to generate pseudo-labels. Although effective, prompt tuning requires full backpropagation through the computational graph, which results in high computational costs.
Adapter tuning (Zhu et al. 2023 Wang et al. 2024), exemplified by Clip-Adapter (Gao et al. 2024), adapts image or text features using lightweight adapters, thereby avoiding full backpropagation through complex encoders. Recently, training-free adapter approaches, such as Tip-Adapter (Zhang al. 2022), have gained traction by leveraging few-shot image features as prototypes, removing the need for learned adapter weights. DMN (Zhang et al.) (024b) and TDA (Karmanov et al.) 2024) extend these methods to zero-shot scenarios by building memory banks from test data and pseudo- labels. However, these approaches often introduce additional costs due to image augmentation and repeated sample processing. Moreover, their reliance on complex architectures and extensive hyper- parameter tuning limits their practicality in real-world applications.
In this work, we propose a unified, training-fret free adaptation framework for VLMs based on label propagation. Our method exploits the manifold structure of test data, eliminating the need for aug- mentation and exhaustive hyperparameter searches, offering a more efficient and practical solution compared to previous approaches.
2.2 LABEL PROPAGATION
Label propagation (LP) is a well-established method widely used for label-efficient learning ZZuu cet al., 20003 Iscen et al. 2019; Xu & Lee| 2020; Zhu & Koniusz 2023). LP operates under the assumption that labels vary smoothly across a graph, with adjacent nodes likely sharing the same label. transductive learning, where all test samples are visible during inference, LP propagates labels from labeled nodes to unlabeled ones. It has been particularly effective in retrieval tasks, where a new query is appended to the graph, and its affinity is propagated to prototype nodes (Iscen et al. (017). Recent studies have demonstrated the benefits of L.P in adapting VL.Ms to downstream tasks (Hu et al. 2024 Kalantidis et al.) 2024). where a graph is constructed from downstream test samples, and labels are propagated from text prompt prototypes test samples using closed-form LP solution. Despite its strengths, this approach faces several challenges, as previously discussed. In response, we propose a holistic solution for label-efficient adaptation of VLMs, built upon label propagation framework.
3 METHODOLOGY
3.1 PROBLEM FORMULATION
We begin by formally defining the task zero-/few-shot adaptation of vision-language mod- els. We denote the image encoder and text encoder of VLM, such CLIP, as f(x) and g(=)
3

---

Published as a conference paper at ICLR 2025
respectively. The downstream task consists of the encoded features of unlabeled testing data Du = {44 = f and optionally the features and labels of few-shot labeled data D, == -1,-
Known class names combined with prompt templates to form textual prompts. Fach class consists of multiple textual prompts (g(=mm) and we take the av- erage the textual prototype Pe 32 Em O ( for the e-th class and denote all prototypes as P = fce.c.....C. The task is to infer the labels of the unlabeled data D,, using the textual prototypes
P and optionally the few-shot labeled data Dj.
3.2 REVISITING TRANSDUCTIVE LABEL PROPAGATION
We revisit transductive label propagation and introduce the graph construction process for a holistic, label-efficient approach VLM adaptation. Instead of directly comparing the similarity between unlabeled samples and class prototypes, we aim to exploit the manifold of all available downstream data. We denote a graph built upon all downstream data as G (V. E) where each node downstream data sample prototype, i.e. PUD: D.. The adjacency matrix for the graph is denoted as W +.N. A normalized adjacency matrix is obtained as W D-0.5 W D-0.5 with being the degree matrix. Labels are propagated and refined according the rule Eq. in iterative manner, where Y are the initial labels, is the iteration count and is weighting hyperparameter. The propagation will converge upon infinite steps of propagation, i.. t + oo with closed- form solution (Zhu al. 2003).
y aWY' + (1 a)Y"
y = (I (I-
(1)
Under VLM adaptation protocol, the initial label matrix is follows, where Yp, Y, Y, refer to the label of textual prototypes, few-shot labeled samples and unlabeled samples.
Yd = [Y, YP, Y.l. s.t. Y = diag(1 ))) Y E 10.1) THe 1 = =c, Y = N.CC
Efficient Label Propagation via Iterative Solution: The closed-form solution to label propagation was adopted by Kalantidis et al. (2024). However, argue that this closed-form solution is sub- optimal in practice for two reasons. First, the closed-form solution requires solving linear system using the conjugate gradient method ( 2006). This is expensive step that prohibits efficient inference. Furthermore, the closed-form solution is the converged solution to the iterative method. Each iteration step involves propagating labels between all connected nodes, including the inferred label information from test samples back to the text prototypes and few-shot labeled samples, which undesirable. For this reason, we employ the iterative solution and we reset the labels for textual prototypes and labeled samples after each iteration, i.c. y Y2 Y YP
3.3 DYNAMIC GRAPH EXPANSION FOR INDUCTIVE INFERENCE
In this section, we introduce the graph construction process. In particular, Static Graph Construc- tion elaborates how graph is constructed from all observed data samples and text prompts, and Dynamic Graph Expansion introduces how to efficiently expand the graph upon observing new testing data. Finally, introduce stream-based Incremental Label Propagation method for inductive inference.
Static Graph Construction: We first introduce the way construct static graph when the whole set of unlabeled test samples D.,, textual prototypes P, and few-shot samples (optional) Dj are avail- able. Nodes are denoted as V {ui, ... UN, P1, ..., PN,, 4, .... (N.). J. We write the adjacency matrix W blockwise in Eq. where W., E IR. NNN.N,, Wp IR N,,,, Wi RNX; Wap R.N x N,, Wai XN, Wpt IR N,NN, We do connect any nodes between textual prototypes and few-shot samples because we are only interested in inferring the labels of unlabeled test samples. This results in W, 0, Wi Wμ
W
W
W at
W
W
W
up
up
W
W
W
W pl
W
W up
T
(3)
up
W
W
W
W
pl
ul
4

---

Published as a conference paper at ICLR 2025
For label propagation, sparsify W to improve computation cost and robustness. Observations made in /hu et al. (2023); Kalantidis et al. (2024) suggest that cosine similarities between inputs from different modalities can vary significantly. For example, the cosine similarity between the features of two images is generally significantly lower than between an image feature and a text embedding. Therefore, we search for the K-nearest neighbors within each modality separately as follows, with D referring to the k nearest neighbor search within nodes from set D.
u Ujo u, Pj. u 1j,
if uy NNKD, (u:) pf E NNkpp
W6
(4)
if NNKD, (u:) and Dr + otherwise
Dynamic Graph Expansion: The static graph construction assumes unlabeled test samples are all available once. This assumption prohibits inductive inference where test samples arrive in stream fashion. Previous work (Kalantidis et al.) (2024) proposed an inductive inference approach by exploiting an external training dataset to build the manifold. With access to such an external dataset, naive way to support inductive inference is to build new graph from scratch upon seeing new testing data, which results in high computation complexity. For example, with a total number of N. Np + Ni N2 nodes, each with d dimensions, the computational complexity is O(d N + N² log N.). which prohibits realistic inference on large scale downstream task. To enable inductive inference while exploiting the testing data manifold, we reduce the complexity to O Nº) by a dynamic graph expansion mechanism. Specifically, with an existing adjacency matrix W, we use the new test sample uN to query all existing nodes and replace the weakest connections with connections to the test sample following the update rule in Eq. 5 The update rule will insert new test sample into the existing graph without re-calculating the K nearest neighbor the existing nodes.
Vi,j E 1........ Nu. S. UN Uer
(5)
WiN.11 = S, 1(, min Wy), Wy= = W.y 1(S, min W.,) 1(j # arg min Woy)
Incremental Label Propagation: With the dynamically expanded graph, we can perform label propagation incrementally. Specifically, we re-use the pseudo labels from the previous iteration to speed up the convergence the label propagation process. After inference for each test sample, the pseudo labels are attenuated for label propagation for the next test sample. We follow the rules in Eq. to produce the incremental labels Y for the observed test samples for inference on future test samples, where Y? represents the label predictions obtained after 7 propagation steps.
BYT uies
if arg max, YT uic
Yus
Y = -i,,01X0
(6)
10
otherwise.
3.4 CONTEXT-AWARE EDGE RE- NE-WEIGHTING
The graph construction process depends on properly chosen distance metric. Measuring the dis- tance between samples based on the features from the vision encoder is subject the biases learnt by the vision encoder during large-scale pre-training. For example, it is known that VLMs encode features that capture all visual instances in the image, e.g. objects, background, style, etc. For this reason, directly measuring the cosine similarity between the raw features encoded by the VLM subject to irrelevant information for the downstream task. For example, adapting the VL.M to downstream task of car model classification requires less attention to certain aspects like background or color information. Motivated by this, we propose to re-weight the importance of VLM vision- encoded features for calculating similarity. Specifically, we first calculate the statistics of textual prompts feature dimensions as follows, where e is the feature dimension index.
1 =
1 Σ Pies IP]
of = |P]
> (Pic me)²
(7)
5

---

Published as a conference paper at ICLR 2025
A higher variance in e-th dimension suggests the higher discriminative capability of the dimension, ie. the more capable it is to discriminate between text prompts. Therefore, we increase the im- portance of these dimensions with higher variance. Additionally, on the few-shot samples, resulting in 14g and o- In contrast to the context of textual prompts, we argue that high-variance feature dimensions in the context of few-shot samples highlight the intra- class variance and that this should be suppressed when calibrating the similarity. Therefore, we use the reciprocal of the variance for re-weighting graph edge similarity. The new edge weights presented in Eq. 8
calculate the same statistics
u4 Norm(diag(o")u)) W = 1. N Norm(diag(1/o)))
(8)
Graph Sparsification:
The constructed graph via KNN search and similarity re- re-weighting via fea- ture importance cater the requirements of VLM adaptation. To further sparsify the graph to prune out erroneous connections and highlight the difference between different semantic classes in the downstream task, we further apply power operation on the afîinity matrix as Wi, W
3.5 OVERALL ALGORITIMM
We present the overall algorithm for unified iterative label propagation for vision-language models in Algorithm
Algorithm Iterative Label Propagation for Adapting Vision-Language Model
1: Input: Test data stream Da, textual prototypes P, few-shot features D,, and label propagation
iterations T
2: Output: Predicted labels {v:}
3: Initial graph W = O, initial label yp, y, & Y by Eq. Initialization
4: for I; E DD, do
5: Calculate Wy. and Wal according to Eq. 8 # Testing Node Edge Update
6: Update W.. W., and W.t according Eq. Dynamic Graph Expansion
7:
W + WT W = W W D
w D # Symmetrize, sparsify normalize graph
8:
for f = 0 to (T 1--1] 1) do
9:
y = WY² # One step label propagation
10:
y!+1 y y YP # Reset prototype and few-shot labels
end for
12: y: arg max, y Test sample label
13: Update y! by eq.
14: end for
return {y:}
4 EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Datasets: We selected diverse range of datasets for thorough evaluation, grouped into three main categories: fine-grained datasets, style-transfer datasets, and out-of-distribution datasets, which cover total of 30 specific test scenarios. Fine-Grained Datasets: We evaluated 11 widely rec- ognized benchmarks, including ImageNet (Deng et al., 2009). Flowers102 (Nilsback & Zisserman 2008 DTD (Cimpoi et al.) 2014), OxfordPets (Parkhi et al. 2012). StanfordCars (Krause et al,
2013 UCF101 (Soomro) 2012), Caltech101 (Fei-Fei et al. 2004). Food101 (Fei-Fei et al. 2004).
SUN397 (Xiao et al. 2010), FGVCAircraft (Maji et al., 2013). and EuroSAT (Helber et al. 2019
Style-Transfer Datasets: We used five benchmarks: ImageNet- V2 (Recht et al. 2019), ImageNet-
A (Hendrycks et al. 2021b), ImageNet-R (Hendrycks et al.) 2021a). and ImageNet--Sktth (Wang
al. 2019). Out-of Distri--tittiittinnn Datasets: We extensively assessed 15 unique different domains

---

Published as a conference paper at ICLR 2025
from ImageNet-C (Hendrycks & Dietterich, 2019) with the highest severity 5, including Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Frosted Glass Blur, Motion Blur. Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic Transform, Pixelate, and JPEG corruptions.
For all experiments, we use the pretrained CLIP model (Radford et al. 202T] with both ResNet-based and ViT-based architectures, specifically ResNet50 (He et al. 2016) and ViT-B/16 (Dosoviskkiy (020). Unlike previous works, which tuned hyperparameters separately for each experiment, we simplity the process by using a consistent set of hyperparameters across all experiments. In constructing the KNN graph, each test image is connected to 3 text embeddings, 8 embeddings from the test memory bank, and 8 from few-shot images (if applicable). The exponential factor y for graph sparsification set to 10, the update factor /3 for Yuu is set to 0.2, and the factor for label propagation is set to 1.0. For efficient inference, we limit the label propagation iterations to 3. We also adopt the dataset splits and textual prompts from Pratt et al. (0023); ; ZZhang et al. 2022).
Implementation Details:
Competing Methods: We selected diverse set of competing methods for both zero-shot and few- shot adaptation scenarios to create a comprehensive benchmark. These methods can be broadly categorized into two groups: prompt-tuning methods and adapter-based methods. For prompt tun- ing, include TPT (Shu et al. 2022), DifITPT (Feng al., 2023), CoOp (Zhou et al. 2022b), CoCoOp (Zhou et al. 2022a), among others. For adapter-based methods, we consider CLIP- Adapter (Gao et al. 2024). ITP-Adapte (Zhang et. 2022), and the most recent approaches such as
DMN Zhang et al. 2024b), TDA (Karmanov et al. [0224) and ZLaP (Kalantidis et al. (024), along with several other methods. Finally, benchmark our proposed method, ECALP, on all tasks.
4.2 EVALUATION OF ZERO-SHOT ADAPTATION
Fine-Graide Grained Categorization Tasks: We present the results of zero-shot adaptation of CLIP model to fine-grained categorization downstream tasks in Tab. [ We make the following observations from the results. i) Our proposed method consistently outperforms other methods across both chitectures, showcasing robust generalization and strong performance, especially in the ViT-B/16 architecture. The ViT-based models generally perform better than their ResNet-based counterparts, reflecting the advantages of transformer-basee architectures in handling diverse benchmarks. ii) DMN performs strongly across both architectures, particularly excelling in fine-grained classifica- tion tasks like Pets and Aircraft. However, the original DMN, denoted as DMN* in the table, im- plements exhaustive searching for the fusion hyperparameters based on testing set for the individual downstream tasks and the performance drops by more than 1% when hyperparameter searching is removed. Nevertheless, both *MN* and its variant, DMN with fixed hyperparameters, are worse in general compared with us. Also, ZLaP requires additional unlabelled training set data to initialize the graph. fii) Some methods show strengths in specific benchmarks. For example, DiffPT excels in handling corruption datasets. The state-of-the offthe-artt label propagation based method, ZLap, achieves much lower performance compared with us. This is attributed to the more effective context-aware reweighting and iterative propagation strategies.
We further evaluate the performance on style-transfer downstream tasks with results presented in Tab. 2 Our analysis reveals several key insights. i) Our proposed method consis- tently surpasses the baseline CLIP models, such as CLIP-RN50 and CLIP-ViT/B-16, with significant improvements across all datasets, especially in the averaged scores, with an average gain of approx- imately 7%. This highlights the effectiveness of our approach in improving generalization without the need for additional training. ii) In comparison state-of-the-an adaptation methods such as DMN and TDA, our method achieves higher accuracy averaged over all datasets, demonstrating that our method robust across different architectures. iii) Remarkably, even when compared to adaptation methods that require training, such as CoOp and CoCoOp, our method maintains clear advantage, showcasing the superior performance of our method, even in scenarios where competing methods undergo task-specific fine-tuning.
Style-Transfer Tasks:
Finally, we evaluate the model's adaptation to downstream tasks. As seen from Tab. 3 proposed method consistently outperforms state-of- the-art competing methods on almost all types of corruptions with both CNN and transformer back- bones.
cat-of-Distribution Tasks:
7

---

Published as a conference paper at ICLR 2025
Table 1: Zero-shot adaptation of CLIP on fine-gras fine-grained categorization downstream tasks. DMN*
refers to the original method that exhaustively searches the fusion hyperparameter with testing data ground-truth on each individual downstream task (not for direct comparison with other methods). ZLaPf requires additional unlabelled training set data.
Method
ImageNet
Flower
DTD
Pets
Cars
UCF
Caltech
Food
SUN
Aircraft
EuroSAT Mean
CLIP-RN50 (Radford et al.
2021
58.16
61.75
40.37
83,57
55.70
58.84
85.88
73.97
58.80
15.66
23.69
56.04
DN
Zhou et 2025
60.16
63.32
41.21
81.92
56.55
55.60
87.25
74.64
59.11
17.43
28.31
56.88
TPT
60.74
62.69
40.84
84,49
58.46
60.82
87.02
74.88
61.46
17.58
28.33
57.94
DifTIPT
Feng CC m1 2023
60.80
63.53
40.72
83.40
60.71
62.67
86.89
79.21
62.72
17.60
41.04
59.94
VisDese
Menoon
2023
59.68
65.37
41.96
54.76
58.47
88.11
76.80
59.84
16.26
37.60
58.29
Ensemble Zhann el m 21122
60.32
66.10
40.07
85.83
55.71
61.33
83.94
71.32
58.53
17.10
37.54
58.53
CALIP (Gbo al. A724
60.57
66.38
42.39
86.21
56.27
61.72
87.71
77.42
58.59
17.76
59.45
CuPL (Pratt a. 2015
61.45
65.44
48.64
84.84
57.28
58.97
89.29
76.94
62.55
19.59
38.38
60.31
SuS-X Lodaanarrrto dt
2023
61.84
67.72
50.59
85. 34
57.27
61.54
89.53
77.58
62.95
19:47
45.57
61.76
DMN (hhang el 2024b
62.02
68.33
50.53
86 29
58.36
64.02
89.09
74.69
63.70
20.22
44.94
62.02
DMN (Zhaing ef al.
21244h
63.87
67.93
50.41
86 78
60,02
65.34
90.14
76.70
64.39
22.77
48.72
63.37
TDA (Karmanov
61.35
68.74
43.74
86,
57.78
64.18
89,70
77.75
62.53
17.61
42.11
61.06
aI (Kalamidds al. 2024
62.20
69.27
42.79
80.32
56.42
62.81
86,90
77.87
61.83
17:37
31.85
59.06
ECAL (Ours))
62.64
69.39
54.49
88.20
60.56
66.67
89.94
76.97
64.97
21.12
49.09
64.00
CLIP-VITB/16 VTT/116 (Radford et al.
[021]
66.73
64.44
44.27
88.25
65
65.13
93.35
83
62,59
23.67
42,01
63.87
Ensemble (Zhang et el.
2022
68.34
66.99
45.04
86,92
66.11
66.16
93.55
82.86
65.63
23.22
50.42
64.93
TPT (Shu al 2014
68.98
68.98
47.75
87.79
66.87
68.04
94.16
84.67
65.50
24.78
42,44
63:45
DiffiT (Feng el al 2023
70.30
70.10
47.00
88 20
67.01
68.22
92.49
87.23
65.74
25.60
43.13
65.90
DMN
(hang
2124h
70.51
75.32
54.85
91.22
67.01
71.95
93.63
84.05
69.14
28.29
56.22
69.29
DMN Zhang el al. 2024b
72.25
74.49
55.85
92.04
67.96
72.51
95.38
85.08
70.18
30.03
59.43
70:47
TDA (Karmanoy
69.51
71.42
47.40
88.63
67.28
70.66
94.24
86. 14
67.62
23.91
58.00
67.71
ZLaP (Kalantidi al.
2024
70.17
73.49
48.58
87 14
65.63
71.45
93.06
86.92
67 67.44
25.44
55.62
67.72
ECAL))OOS))
71.26
75.96
56.32
92.31
68.20
75.44
94.40
85.72
70.35
29.49
56.53
70.54
Table 2: Zero-shot adaptation of CLIP on style-transfer downstream tasks. Methods marked with are pre-finetuned using additional 16-shot training samples for each category in ImageNet.
Method
ImageNet-A ImageNet-V2 V2 ImageNet-R ImageNet-S
Mean
CLIP-RN50 Radford
20211
21.83
51.41
56.15
33.37
40.69
CoOpl
Zhou al.
20226
23.06
55.40
56.60
34.67
42.43
CoCoOp Zhou et al.
2022a
23.32
$5.72
57.74
34.48
42,82
TPT Shu et al. 2022
26.67
$4.70
59.11
35.09
43.89
Diff Feng et
2023
31.06
55.80
58.80
37.10
45.69
CALIP
Guo ot al,
23.96
53.70
60.81
35.61
43.52
TDA (Karmanoy
2h24
30.29
55.54
62.58
12
46.63
DMN Zhhnn et
28.57
56.12
61.44
39.84
46.49
ECALP (Ours)
28.80
56.92
63.68
41.51
47.73
CLIP- V/1- Radford et
2021
47.87
60.86
73.98
46.09
$7.20
CoOp
Zhoo et all 20226
49.71
64.20
75.21
47.99
59.14
CoCoOp Zhou et 2022a
50.63
64.07
48.75
MaPLe Khattak ct...
50.90
64.07
49.15
TPT Shu al. 2022
54,77
63.45
77.06
47.94
60.81
Dif'
[T PP et ml
2023
55.66
65,10
75.00
46.80
60.65
TDA
Karmanov et al 2024
60.11
64.67
80.24
50.54
63.89
DMN Zhang et al
20246
58. 28
65.17
78.55
53,20
63.80
ECALP (Ouns)
58.52
65.72
80.77
54.66
64.92
Table 3: Zero-shot adaptation of CLIP on out-of-distribution downstream tasks. ZLaPt requires additional unlabelled training set data.
Method
Gauss.
Shot
Impu.
Defo.
Glass.
Moti.
Zoom
Snow
Pros.
Brig.
Cont.
Elas.
JPBG
Avemge
CLIP-RN50 (Rafford et al. 2021
1.63
2.18
L61
10.06
3.12
7.85
12.83
12.58
15.67
21.95
10.27
6.28
4.75
11.12
13.03
11.02
TDA
Karmanov el al. 2024
2.26
3.10
2.31
11.30
5.12
9.26
15:43
15:47
19.11
26.45
45.5
8.34
7.30
13.01
1583
13.31
Kalanidis
1.77
3.54
7.99
13.47
13,66
17.15
23,20
44.67
6.55
5.15
11.82
DMN
Zhang ct 20246
2.14
2.78
2.30
10.91
4.48
8.59
14.31
14.14
17.92
24.16
44.57
7:88
6.11
12:40
14.93
12.51
BCALP (Curs)
2.71
3.30
2.83
12.29
5.49
10.56
16.82
16,66
20.60
27.83
47.02
9.08
7.72
14.46
16.88
14.28
CLIP- VTB/16 (Radford et al. 2021
11.34
12:31
11.85
23.78
15.12
24.05
22.73
32.70
30.43
36.69
54,57
16.84
12.77
31.22
33.00
24.63
TDA
Karmanov et al. 2024
15:42
16.46
16.03
26.53
17.91
27.35
25.90
36.50
34.84
10.53
58.57
20.16
16.62
35.65
36.69
28.34
Kalantdis 2015
12.85
16.13
25.77
24.36
34,43
17.53
26.42
DMN
Zhang al. 20246
14.33
15.33
14.69
26.06
17.19
26.61
25.23
34.81
33.48
38.93
58.
70.70
19.38
15:40
35.32
36.49
27.46
ECALP (Ours)
15.92
16.84
16.32
27.85
18.78
28.59
27.62
37.82
36.01
41.65
60.57
21.26
17.77
37.39
38.11
29.50
8

---

Published as a conference paper at ICLR 2025
Average Accuracy of 11 Datasets
Accuracy on ImageNet
Accurecy on Flowers102
63.0
Accurac (51
45
DMN
ECALP (Ours]
62.5
DNN APE
ECALP (Ours)
APE
62.0-
ECALP (Ours]
Accuracy
50
TIP-Adepter
615-
TIP-Ndapter
Zoro-shot
6100
TTD-kdepter
TIP-
15
605
Zoro shot CLPP
20
0
16
16
Shats Number
Shots Number
Shots Number
Accuracy on DTD
Accuracy on StanfordCars
Accuracy on UCF101
Accurary (h1
Accuracy 1NN
65
70-
DNN
ECALP (Ours)
15
ECALP (Durs)
DMN
60
DMN
ECALP (Ours)
Accuracy
65
ARE
TIP-Adapter
TIKAdapper
TiP-sdapter
TIPX
TIP-
Zoro shot
Zero shoe CUP
0
16
012
16
0
0.2
16
Shots Number
Shots Number
Shots Number
Accuracy on SUN397
Accuracy on FGVCAircraft
Accuracy on EuroSAT
68
ECALP (Ouns)
30.0
DMN
70-
(Ours)
Accunccy (N)
64
APE
DMK
27.5
25.0
TiP-bdepier
Accurccy
60
APE
DMN
12
TIPx
TiNAclapie
22.5
Zuro shot CLP
50
TiP-Adapter
Zero
20.0
40
17.5
30
0
16
16
2
16
Shats Number
Shots Number
Shots Namber
Figure 2: Few-shot adaptation of VLM for fine- -haigrr categorization downstream tasks with CLIP- RN50 model.
4.3 EVALUATION OF FEW-SHOT ADAPTATION
For few-shot adaptation tasks, we evaluate the adaptation performance on fine -raiird categoriza- tion benchmarks, with [1. 2, 4, 8, 16] shots of each class, as presented in Fig. 2 In line with (Silva-Rodriguez et al. 2024), we establish a fair benchmark by comparing existing methods using their default hyperparameter values, as specified in the corresponding papers, without conducting a hyperparameter search. We compare our method with several existing training-free free approaches, including DMN et al. 2024b), APE (Zhu et al. 2023). TIP-Adapter (Zhang et al. 2022), and TIP-X (Udandarao et al. 2023). Our method outperforms all competing methods on average across 11 datasets and consistently achieves superior results across most datasets and shot counts. Furthermore, it maintains high performance even in low-shot scenarios, underscoring its efficiency and robustness.
4.4 ABLATION & ADDITIONAL STUDY
Unveiling the Impact of Individual Components: We first conduct comprehensive ablation anal- ysis investigating the effectiveness of individual components within ECALP. As shown in Tab we first observe that label propagation (Label Prop.) improves the results with significant margin, on DTD and UCF datasets, compared to naive nearest neighbor classification. However, the result drop a little bit on ImageNet (58. 16% - 55.34%), probably owing to the diversity of contents within ImageNet. This is quickly remedied when edge weights are reweighted by text embeddings (Tex. Reweight) (55.34% - 62.64%). suggesting the context information is essential to identify the rel- evant information from the feature encoded by VLM. When 16 shots of each class are available, we make similar observations that edge weights re- weighted by few-shot sample feature distribu- tion (F.S. Reweight) are essential to maintain good performance while text reweighting is less im- portant. We also noticed that ImageNet benefits less from few-shot than DTD and UCF because of the high intra-class variation, which renders few-shot labeled samples less effective.

---

Published as a conference paper at ICLR 2025
Table Ablation study on the components of ECALP. (ZS: zero shot; FS: few-shot)
Table 5: Ablation study on label propagation.
Siratey
Lab, Prop. Few-shot Tex, Reweight ES Reweight ImtageNet
DTD
UCF
Stralogy
Wop W. Wis JInagoNet
DTD UCF
55.34
49.76 61.17
ECALP-ZS
62.64
54.49
66.67
63.97
ECALP-ZS
62.64
34,49 66.67
V
47.25
59.34
66.98
63.00
64.48
72.01
63.47
65.13 74.18
ECALP-FS
/
63.20
65.37 74.44
ECALP
63.20
65.37
74.44
Investigating Graph Construction: We further investigate the necessity of each subgraph of the overall graph. Specifically, we consider the three subgraphs including test samples to prototypes Wap: within test samples W. and test samples to few-shot samples Wi.- As seen from Tab. [5] all three datasets benefit substantially from having the subgraph connecting to textual prototypes. When the subgraph within test samples Wa is included, the performance further improves, though more significantly on DTD and UCF, suggesting the manifold is helpful to the propagation of labels. Finally, including the test sample subgraph Wa is also helpful to the few-shot cases.
Computational Efficiency. To assess the computational efficiency of our proposed ECALP method, we measure the wall-clock time during the zero-shot adaptation task on fine- gree-graned datasets, as shown in Tab. [6] All experiments are conducted using single RTX 3090 GPU and AMI) EPYC 7302 CPU. ECALP, along with other training-free adaptation methods, does not significantly in- crease computational overhead and demonstrates least 30x speedup compared approaches requiring training. Notably, ECALP achieves the best performance with low computational cost.
Table 6: Comparison of wall-clock time with CLIP-RN50.
ImageNet
DTD Accurccy
UCF
Method
Testing Time
Accumey
Gain
Testing Time
Gain
Testing Time
Accuracy
Gain
CLIP-RN5O Radford
Thnining-rqquine
TPT
Shu
2022
898. 8m8.7ms
60.74
+2.58
881.9 2359, Ams
40.84
+0.47
871.6um
60.82
+1,98
DiIFIT (Feg et al
2023
2472
60,80
+2.61
40.72
+0.35
2115.7ms
62.67
+3.83
Training-froe
TDA
Karmanov
2024
61.35
DMN
Zhane al.
2624
1ms
62.02
+3.86
18.8ms
50.53
+10.16
15.5ms
64.02
+5.18
aP
Kalantidis et al.
2024
29. 3ms
62.20
+4.04
15.1ms
42.79
+2.42
13.4ms
62.81
+3.34
ECALP (Ours)
28. 28.0ms
62.64
+4.47
16.1ms
$4.49
+14.12
14.5ms
66.67
+7.83
5 CONCLUSION
Adapting pre-trained vision language model for downstream tasks introduces new paradigm for crafting computer vision models. We aim to better exploit the observed unlabeled data, i.e. the data manifold and propose a unified method based on label propagation. In particular, we addressed the challenges of improving the computation efficiency of label propagation via iterative and in- cremental solution. We also proposed a simple dynamic graph expansion strategy to accommodate inductive inference. Furthermore, ods overlooked the diverse information captured by VLM and thus propose a context-aware edge reweighting based on the downstream task information. We carried out experiments on fine-grained categorization and out-distribution downstream tasks and achieved the state-of-the-art performance on all datasets.
observe that the similarity metric adopted by existing meth-
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 62106078; by the Agency for Science, Technology and Research, Singapore (A *STAR) under Grant
M23L760021; by the Guangdong Provincial Key Laboratory of Human Digital Twin under Grant
2022B 121 121000004: by the National Natural Science Foundation of China (NSFC) under Grant 6170118 181; by the Guangdong R&D key project of China under Grant 20198010155001.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 2009.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, 2004.
Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 2023.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 2024.
Leo Grady, Random walks for image segmentation. IEEE transactions pattern analysis and machine intelligence, 2006.
Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEE conference on computer vision and pattern recognition, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference Learning Represen- tations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A criti- cal analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2021b.
Xuefeng Hu, Ke Zhang, Lu Xia, Albert Chen, Jiajia Luo, Yuyin Sun, Ken Wang, Nan Qiao, Xiao Zeng. Min Sun, et al. Reclip: Refine contrastive language image pre-training with source free domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, 2024.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Teddy Furon, and Ondrej Chum. Efficient diffusion on region manifolds: Recovering small objects with compact cnn representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
11

---

Published as a conference paper at ICLR 2025
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and laiguala representation learning with noisy text supervision. In International conference on machine learning, 2021.
Thorsten Joachims. Transductive learning via spectral graph partitioning. In Proceedings of the 20th international conference machine learning (ICML-03), 2003.
Yannis Kalantidis, Giorgos Tolias, et al. Label propagation for zero-shot classification with vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Adilbek Karmanov, Dayan Guan, Shijian Lu, Abdulmotaleb Saddik, and Eric Xing. Efficient test-time adaptation of vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-a and-language transformer without convo- lution or region supervision. In International conference on machine learning, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- aee-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, 2013.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman image pre- training for unified vision- lsiion-aagguage understanding and generation. In International conference on machine learning, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping inguage-imagee pre-training with frozen image encoders and large language models. In International conference on machine learning, 2023.
Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Swapprompt: Test-time prompt adaptation for vision-language models. In Advances in Neural Information Processing Systems, 2024.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: (30x:v:306..1..1 2013.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference on Learning Representations, 2023.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, 2012.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2023.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, 2019.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances in Neural Information Processing Systems, 2022.
Julio SilvieRo, Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. A closer look at the few- shot adaptation of large vision- saiiunla models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
K Soomro. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:12/2.0402, 2012.
Vishaal Udandarao, Ankush Gupta, and Samuel Albanie. Sus-x: Training-free free name-only transfer of vision-language models. In Proceedings of the IEEE/CVE International Conference on Computer Vision, 2023.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 2019.
Zhengbo Wang. Jian Liang Lijun Sheng, Ran He, Zilei Wang, and Tieniu Tan. A hard-to-beat baseline for training-free clip-based adaptation. arXiv preprint arXiv:2402.0408/ 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, 2010.
Xun Xu and Gim Hee Lee. Weakly supervised semantic point cloud segmentation: Towards 10x fewer labels. In Proceedings of the IEEF/CVF conference computer vision and pattern recog- nition, 2020.
Hee Suk Yoon, Eunscop Yoon, Joshua Tian Jin Tee, Mark A Hasegawa-Johnson. Yingzhen Li, and Chang D Yoo. C-tpt: Calibrated test-time prompt tuning for vision-language models via text feature dispersion. In The Twelfth International Conference on Learning Representations, 2024.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-fres free adaption of clip for few-shot classification. In European conference on computer vision, 2022.
Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024b.
Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. A survey on deep learning- based fine-grained object classification and semantic segmentation. International Journal of Automation and Com- puting, 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- -aniunl models. In Proceedings of the IEEE/CVF conference on computer vision and pastern recognition, 2022a.
13

---

Published as a conference paper at ICLR 2025
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 2022b.
Yifei Zhou, Juntao Ren, Fengyu Li, Ramin Zabih, and Ser-Nam Lim. Distribution normalization:
An" effortless" test-time augmentation for contrastively learned visual-language models. arXiv preprint arXiv:2302. 1084, 2023.
Hao Zhu and Piotr Koniusz. Transductive few-shot learning with prototype- based label propagation by iterative graph refinement. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang. Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. Proceedings of the IEEE/CVF International Conference Computer Vision, 2023.
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propa- gation. ProQuest number: information to all users, 2002.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), 2003.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 FURTHER ANALYSIS OF THE COMPUTATIONAL COMPLEXITY IN GRAPH
CONSTRUCTION
In this section, we analyze the computational complexity of constructing graph in two scenarios: first, the static graph construction, and then, how our dynamic graph expansion method significantly reduces this complexity. The analysis focuses two key components: distance matrix compu- tation and nearest neighbor selection. For simplicity, we assume graph consists of a total of No N N₁ No nodes, where each node is represented by feature vector of dimension d. Since this analysis is conducted in the context of inductive inference, we consider the process as a sequential addition of nodes from to N.-
A.1.1 STATIC GRAPH CONSTRUCTION COMPLEXITY
Distance Computation Complexity: In K-Nearest Neighbors (KNN) graph, the first step is to compute the pairwise distances between nodes. For n nodes, each represented by a d-dimensional feature vector, the time complexity compute the distance between two nodes is O(d). Since distances need to be calculated for all pairs nodes, the number of comparisons is O(n)). Hence, the computational complexity for distance computation step is:
CD. Onn²... d)
Accumulating the complexity across all steps, the total computational complexity for distance com- putation is:
N₂ (d.NN)
Cp Σ CD.
Σ On d)= d
n²
(10)
Neighbor Selection Complexity: After computing the distances, the next step is to select the K- nearest neighbors for each node. Sorting the list of distances for single node requires O(n log n) time. The total complexity for neighbor selection can be approximated by treating this continu- ous function and integrating:
N Σ
o (f,
x1log log log x dx (
Os = O
n log
O (N3 log N.)
(II)
The overall time complexity for static graph construction the sum of the complexities of distance computation and neighbor selection. Thus, the total time complexity is:
Total Complexity:
C= C= = ONN + N3 log N.)
(12)
A.1.2 DYNAMIC GRAPH EXPANSION COMPLEXITY
Distance Computation Complexity: In our dynamic graph expansion method, the distance com- putation for each new node limited the current node and the existing nodes in the graph. For nodes, the number of distance computations needed is O(n d). Accumulating this across all steps, the total complexity for distance computation becomes:
N, On d) -0 dl
CD
CD
Σ n
0d (d.dd3
(13)
Neighbor Selection Complexity: In this dynamic method, maintain sparse affinity matrix, and only need to find the nearest neighbor for each node, which requires a constant time operation. Thus, the total complexity for neighbor selection is:
15

---

Published as a conference paper at ICLR 2025
Cs Σ -
(14)
Total Complexity: The overall time complexity for dynamic graph expansion is the sum of the distance computation and neighbor selection complexities. Therefore, the total time complexity is:
C CD + Cs= - 0 Nē + N.) = O(d N?)
(15)
A.2 FURTHER ANALYSIS ON ECALP
A.2.1 KNN CONNECTION STUDY
We conducted an analysis of the number of connections K-Nearest Neighbors (KNN) models using CLIP-RN50 on the DTD dataset in Tab. 7 Specifically, experimented with varying the number connections for both NNKp and NN D., selecting values from the range [1, 3, 5, 8, 10]. It presents the performance across these different configurations, highlighting the highest accuracy achieved when kp = and p.. 8, where the model reached an accuracy 54.49%. This suggests that increasing the number of neighbors enhances performance up certain point, beyond which the impact diminishes.
Table 7: Studies on KNN connections number with CLIP-RN50 for zero-shot adaptation.
Bp\KD.
8
10
1
48.64
48.88
49.05
49,00
49.00
3
52.36
53,90
54.20
54.49
54.20
5
51.60
53.19
53.90
54.26
54.14
8
51.18
52.96
53.72
53.01
52.66
10
51.00
52.12
52.13
52.25
52.30
A.2.2 DYNAMIC GRAPH CONSTRUCTION Vs. STATIC
our dynamic graph expansion enables ECALP to achieve an approximate 6x speedup compared the static graph construction method the ImageNet dataset with CLIP- RN50.
As illustrated in Fig. 3
6xFaster
Time (nS Sample)
Construction Method
Figure Wall-clock time comparison between statically-coosstuutedd graphs and dynamically- expansive graphs.
A.2.3 GENERALIZATION TO DIFFERENT VLMS
The ability of ECALP to improve zero-shot adaptation extends beyond CLIP, showcasing its ef- fectiveness across various Vision-Language Models (VLMs). As demonstrated in Tab. applying
16

---

Published as a conference paper at ICLR 2025
ECALP to multiple VLMs results in noticeable improvement in ImageNet accuracy. For instance, when applied to BLIP (Li et al. 2022), the base ImageNet accuracy of 53.40% is significantly enhanced to 58.16% Similarly. in the of BLIP v2 (Li et al. 2023), the accuracy increases from 41.22 41.22% to 43.72% with ECALP. A notable improvement is also observed with ALBEF (Li et al. (02T), where ECALP raises the accuracy from 36. 15% 40.31 %. These results highlight the generalization ability of our approach, making it applicable across different state-of-the-art VLMs, consistently improving their performance.
Table 8: sero adaptation accuracy with our method applied to different VLM.
VLM
ImageNet Accuracy
BLIP Li al.
2022)
53.40
+ ECALP
58.16
BLIP v2 Li et
2023
41.22
+ECALP (Ours)
43.72
ALBEF (Li et al. +ECALP (Ours)
2021
36.15
40.31
A.2.4 COMBINATION WITH COOP
We integrate ECALP with the traditional prompt learning method CoOp (Zhou al. 2022b), utiliz- ing text embeddings with their trained prompts on 16-shot samples from the DTD dataset. As illus- trated in Fig. 4 CoOp combined with ECALP significantly outperforms the original CoOp across different shot numbers. Notably, with 8 and 16 shots, CoOp with ECALP surpasses standalone ECALP, indicating that ECALP can be seamlessly integrated with traditional prompt learning meth- ods to enhance their performance. However, in low-shot scenarios like 1-shot, CoOp with ECALP performs worse than ECALP alone. This is likely due to overfitting and bias in CoOp's trained prompts when training data is very limited, which is detrimental for label propagation using the complete test data manifold.
Accuracy on DTD
60
Accuracy (%6
ECALP (Ours)
45
CoOp
CoOp ECALP (Ours)
Zero-shot CLIP
16
Shots Number
Figure 4: Combining proposed ECALP with CoOp (Zhou et al. 2022b) the DTD dataset.
A.2.5 IMPACT OF LABEL PROPAGATION ITERATION T
We explored the effect of varying the label propagation iteration T from 1 to 6 on the DTD dataset, as illustrated in Fig. 5 The results demonstrate that performing multiple iterations of label propagation enhances accuracy compared to single iteration. This suggests that ECALP effectively utilizes the manifold structure of the test data. However, increasing iterations also raises computational costs, yt++1 becomes denser with larger 1, which may affect efficiency. Thus, opt for T across
17

---

Published as a conference paper at ICLR 2025
all experiments to maintain a balance between computational efficiency and accuracy. As indicated Tab. 66 ECALP achieves an optimal trade-off between these factors.
$5.00
Accuracy
34.15
-4,00
53.25
Iteration
Figure Effect of label propagation iterations on the DTD dataset.
A.2.6 PERFORMANCE ON DIFFERENT CORRUPTION SEVERITIES
We further examine the performance of our proposed ECALP on corrupted downstream tasks using the ImageNet-C dataset, across severities ranging from 1 to 5. 'he average accuracy of ECALP compared to Res.--Re over types of corruption (as detailed in Tab 3) is presented in Fig. ECALP consistently surpasses CLIP by margin of approximately 3%-5% across all severity levels, demonstrating its robustness.
46.70
CLIP-AN50
45
ECALP
40
37.68
Accuracy (%)
20:05
32.51
22
25.25
21.55
20
17:33
14.28
11.02
Severity
Figure 6: Average accuracy across 15 different corruption subsets under various severity levels on the ImageNet-C dataset.
A.2.7 ROBUSTNESS TO POOR INITIALIZATION
To assess the robustness of our dynamically constructed graph for label propagation, conducted experiments under various dtta-stream conditions. As presented in Tab. 9 we evaluated ECALP's performance with three different test set configurations: standard Random Sampling: random sample of the test stream. Hard Samples First: The initial 5% of the test stream consists of only hard samples (those misclassified by CLIP),, followed by a random sampling of the remaining 95%. 10% % Hard Samples First: The initial 10% of the test stream consists of only hard samples, with the rest being randomly sampled.
typical
The results demonstrate minimal performance drop, with less than 0.4% decrease in accuracy even with a 10% hard sample initialization. This resilience can be attributed to our graph's 's dynamic nature, where edge connections between data points are continuously updated as the test sequence progresses. This adaptability ensures that our method remains robust against poor initializations and maintains high accuracy despite challenging conditions.
18

---

Published as a conference paper at ICLR 2025
Table 9: Zero-shot adaptation accuracy with our method under different data-stream initializations on the DTD dataset.
Random Sampling
5% Hard Samples First
10% Hard Samples First
Accuracy
54.49
54.31
54.14
A.2.8 VISUALIZATION SAMPLES FOR CORRUPTION TASK
Fig. 7 iillustrates an image from the ImageNet-C (Hendryeks & Dietterich 2019) dataset subjected
to varous types of corruption at severity level 5. The examples include Gaussian Noise, Motion Blur, Saturation, and Snow. These visualizations highlight the substantial domain shifts present in this challenging task, demonstrating the robustness required to handle such severe corruptions effectively.
(u) Raw Image
(b) Gaussian Noise
(c) Motion Blur
(d) Saturate
(c) Snow
Figure 7: Samples from different corruption types at severity level 5 in the ImageNet-C dataset.
19

---

Published as a conference paper at ICLR 2025
A.3 FULL RESULTS FOR FEW-SHOT ADAPTATION
Average Accuracy of Dalasels
Accuracy Imageliet
Accuracy on Flowers102
70
DMN
APE
ECALP (Ours)
TIP-Adapter
62.5
02.5
DNX
ECALP rOurs)
AccU
TR- TR-Adepter
APE
DMN
ECALP (Durs)
80
TIP-
61.5
APE
TIXX
Zerpatot
TIP-Acapter
Zeno-shol CuP
25
16
0
16
Shots Number
Shots Number
Shots Number
Accuracy on
Accuracy on OxtardPets
Accuracy StanfordCars
67-
60-
Aroureyy (ha)
Accumicy 194
53
10
53
Til-aadapiier
TIX-X
TIP-X
Zero shot CUP
Zoro shot CUP
Zero shot CUP
40-
Accuracy on UCF101
Accuraty on Coltech101
Accurety on Food101
75
90
70-
(Durs)
Acturntyy Pho
TIP-Sdapter
50-
DHN
ECALP IOurs!
Accurccy
APE
TiPx
ARE
TIXX
Zaro-hott CUP
TIP Adapter
TIF-X
Zaro-shot CUP
Zero-shot
1.2
16
Shors Number
Shots Number
Shors Number
SUN397
Accuracy FGVCAirzraft
Accuracy EUroSAT
32.5
(Dursi
30.0
ECALP (Cura)
DVN
APE
Accuracyy 1982
15
DMN
ECALP (Cura)
APE
IP-Adaaper
TIX
Zero shot CUP
APE
TP-Aulapter
TIPX
CUP
shot
17.5
Shatks Nurn but
Shatis Number
Shats Nam ber
Figure 8: Full results for few-shot adaptation of VLM for fine-grained categorization downstream tasks.
A.4 FAILURE CASE ANALYSIS FOR FEW-SHOT ADAPTATION
In this section, we delve into the reasons behind the suboptimal performance of our ECALP method on the OxfordPets and Food101 datasets under few-shot adaptation. We bring in other two datasets, Flowers 102 and EuroSAT, for comparative studies. The test sample features space via t-SNE (yan der Maaten & Hinton 2008), as illustrated in Fig. 9 We make the following observations. The test samples more cluttered and mixed up in OxfordPets and Food101 than in Flowers 102 and EuroSAT. There are more separated sub-classes, i.c. isolated clusters within each semantic class, for OxfordPets and Foo This suggests having few-shot labeled samples is less effective for OxfordPets and Food101 datasets. In certain cases, an inappropriate selection of few-shot samples may even bias the adaptation.
projected into 2D
20

---

Published as a conference paper at ICLR 2025
(a) Flowers 102
(b) EuroSAT
Ouford Pers Image Festures
Pood1:1 Image Feet
(c) OxfordPets
(d) Food101
Figure 9: t-SNE visualization of image features from the CLIP ResNet50 model across four datasets: Flowers EuroSAT, Oxford Pets, and Food101. Circles represent test image samples, while triangles indicate few-shot samples. The visualization focuses on the first 20 categories and 4-shot scenario for clarity.
21


Reviews:
[{"review_id": "YbIo47Eg9Y", "reviewer": "Reviewer_9D8M", "summary": "The paper proposes ECALP, a novel graph-based framework for training-free adaptation of vision-language models. The proposed method constructs a graph over text prototypes, optional training samples and testing samples in a simple, dynamic fashion, and adopts label propagation for inference. Furthermore, edge re-weighting is carefully designed to take into account the semantic context of different classes. Throughout training and inference, the framework introduces little augmentation and hyper-parameter search, showing its efficiency and practicality. Finally, comprehensive experimental results over a wide range of datasets on zero-shot/few-shot fine-grained classification, style-transfer and out-of-distribution detection tasks show significant improvement in performance of the proposed approach over previous methods.", "questions": "1. As is mentioned above, I wonder if the authors could offer some explanation on ECALP’s behavior on certain datasets in low-shot settings.\n\n2. In the ablation study, in Table 4, it’s interesting to see that without Text Reweight, the model achieves slightly better performance under the 16-shot setting on ImageNet. It would be nice if the authors could shed some light on this phenomenon.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Comprehensive experiments and great results:\n\nThe paper provides a comprehensive comparison of ECALP against a number of highly competitive prior works, and carried out experiments on a variety of datasets as well as visual tasks, which is sufficient to validate the approach. The empirical results clearly demonstrate that ECALP achieves significantly better performance than previous methods. The experimental design, including the main setting, comparison baselines as well as ablation studies, are well justified. Moreover, the method proves to be computationally efficient.\n\n2. Nice presentation and clear structure\n\nThe figures in this paper are well-designed and effectively highlight the pipeline as well as advantages of the proposed framework. The formulation, presentation of the results as well as the visuals are aesthetically pleasing and easy to follow.\n\n3. Clear motivation and good writing\n\nThe introduction and abstract are well written and lays a good foundation for the paper. It is easy to see the motivation behind the work.", "weaknesses": "1. In Figure 1, the inputs to the second image encoder and the text encoder seem to be mixed up. \n\n2. As is discussed in the paper, ECALP builds upon similar ideas with ZLaP (i.e. label propagation), and although I acknowledge that the proposed method achieves improvement in performance and eliminates the need for additional unlabelled datasets, the contribution seems somewhat limited.\n\n3. In the detailed few-shot results provided in the appendix, it seems that ECALP does not always achieve the best performance. Especially on datasets like OxfordPets and Food101, the accuracy of ECALP even goes down as the number of samples per class increases. This weakens the claim of ECALP’s robustness in low-shot scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7eAkaZnf6J", "reviewer": "Reviewer_qcpB", "summary": "This paper proposes a graph-based approach for label-efficient adaptation and inference. The method dynamically constructs a graph based on available samples to enable real-time inductive inference.", "questions": "1. It appears that ECALP requires additional storage. How does its CUDA memory compare to that of previous methods?\n2. If the initial testing samples are used to create an incorrect adjacency graph, will this lead to a cumulative error?\n3. In corrupted downstream tasks, is ECALP still effective when the severity is low (e.g., at level 1)?\n4. Is the method sensitive to the label propagation iterations T?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method requires no additional unlabeled support set and effectively leverages the test sample manifold through dynamic graph expansion.\n2. This paper is clear and easy to follow.\n3. This paper conducts extensive evaluations on diverse downstream tasks.", "weaknesses": "1. Compared to the SOTA methods, the performance improvement seems limited. Additionally, the second-highest performance can be highlighted with an underline for clarity.\n2. In the computational efficiency section, does the testing time for ECALP include the dynamic graph construction process, or does it only account for the label propagation time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yi4HU5MwA6", "reviewer": "Reviewer_uWXi", "summary": "The paper proposes a approach for adapting vision-language models (VLMs) using efficient, context-aware label propagation. It introduces dynamic graph construction to improve inference accuracy without task-specific tuning, and employs context-aware feature re-weighting for better task adaptation. The method demonstrates superior performance and efficiency across various downstream tasks, highlighting its potential in zero-/few-shot scenarios.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of dynamic graph expansion is innovative, allowing for real-time adaptation and efficient use of test samples. \n2. The method enhances feature relevance by re-weighting based on context, which is a novel approach to improving model adaptability to downstream tasks. This can potentially lead to better performance in diverse scenarios.\n3. Employing an iterative solution rather than a closed-form one reduces computational costs and enhances scalability. This is particularly beneficial for handling large datasets.", "weaknesses": "1. More detailed motivation behind the model design is preferred. It is important to explain why the authors propose the method in this work.\n2. The method may be complex to implement, particularly for large-scale or varied tasks. Detailed guidelines for implementation would be beneficial.\n3. The approach assumes that the context-aware feature re-weighting will generalize well across different tasks, but this might not hold true for all types of data or in cases with significant domain shifts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yWUPbLSh3n", "reviewer": "Reviewer_9AGk", "summary": "This paper proposes graph-based efficient adaption and inference for vision-language models. Specifically, the paper maximizes the use of test samples through iterative label propagation without the need for task-specific tuning. Additionally, the paper introduces a novel method to mitigate biases in VLMs through context-aware re-weighting and a interesting approach for identifying KNNs within each modality. Extensive experiments on various downstream tasks demonstrate the effectiveness of the proposed graph-based method.", "questions": "1. In Figure 1, it seems that the second image encoder and text encoder should be switched.\n2. In Line 161, how can we obtain multiple textual prompts $z_{cm}$ for the $c$-th class? It seems that the value of $M$ can vary from class to class.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed use of K-nearest neighbors within each modality and context-aware edge re-weighting are novel approaches that align well with the property of pre-trained VLMs.\n2. The paper is well-written and easy to understand.\n3. Extensive experiments were fairly conducted with recent baselines across a range of datasets and architectures. \n4. The paper improves the feasibility of the proposed method by using dynamic graph expansion, supported by a practical analysis of wall-clock time in Table 6 and time complexity in Appendix A.1.", "weaknesses": "I think the paper has no significant weaknesses.\n1. Evaluating the performance degradation when using one-step label propagation instead of iterative label propagation could provide valuable insights for practitioners. This analysis would shed light on the trade-off between reduced complexity and potential accuracy decline, assisting in making informed decisions for applications where computational efficiency is crucial.\n2. When utilizing an additional 16-shot training samples in Table 2, it appears that ECALP uses the training data as a component of the graph rather than for training. Wouldn’t it be possible to apply ECALP in addition to traditional prompt learning methods such as CoOp and CoCoOp?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)