Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin
Authors: Yuchen Wang
Xuefeng Bai
Xiucheng Li
Weili Guan
Liqiang Nie
Xinyang Chen
Venue: ICML 2025 poster (2025)
Keywords: semi-supervised learning
unsupervised learning
vision-language models
prompt tuning
Primary area: general_machine_learning->unsupervised_and_semisupervised_learning
TL;DR: This paper addresses imbalance in pseudolabels generated by VLMs, identifying concept mismatch and concept confusion as key causes. It proposes a framework with concept alignment and confusion-aware margin to improve pseudolabel balance and accuracy.

Abstract: Adapting vision-language models (VLMs) to downstream tasks with pseudolabels has gained increasing attention. 
A major obstacle is that the pseudolabels generated by VLMs tend to be imbalanced, leading to inferior performance.
While existing methods have explored various strategies to address this, the underlying causes of imbalance remain insufficiently investigated.
To fill this gap, we delve into imbalanced pseudolabels and identify two primary contributing factors: concept mismatch and concept confusion. 
To mitigate these two issues, we propose a novel framework incorporating concept alignment and confusion-aware calibrated margin mechanisms. 
The core of our approach lies in enhancing underperforming classes and promoting balanced predictions across categories, thus mitigating imbalance. 
Extensive experiments on six benchmark datasets with three learning paradigms demonstrate that the proposed method effectively enhances the accuracy and balance of pseudolabels, achieving a relative improvement of 6.29\% over the SoTA method. Our code is avaliable at https://github.com/Noahwangyuchen/CAP

Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margin
Yuchen Wang' Xuefeng Bai Xiucheng Li"
Weili Guan
Liqiang Nie
Xinyang Chen'
Abstract
Accuracies
Cluster Distributions
Adapting vision-language models (VLMs) to downstream tasks with pseudolabels has gained increasing attention. A major obstacle is that the pseudolabels generated by VLMs tend to be im- balanced, leading to inferior performance. While existing methods have explored various strategies to address this, the underlying causes of imbal- ance remain insufficiently investigated. To fill this gap, delve into imbalanced pseudolabels and identify two primary contributing factors: concept mismatch and concept confusion. To mitigate these two issues, we propose novel framework incorporating concept alignment and confusion-awee -warre calibrated margin mechanisms. The core of our approach lies in enhancing un- derperforming classes and promoting balanced predictions across categories, thus mitigating im- balance. Extensive experiments on six benchmark datasets with three learning paradigms demon- strate that the proposed method effectively en- hances the accuracy and balance of pseudola- bels, achieving a relative improvement of 6.29% over the SoTA method. Our code is avaliable
DO1
0.00
Figure I. left: The lowest 5 per-class accuracies in RESISC45, right: The distribution of samples of them in clusters. The pink bar represents samples in the cluster in which they appear most frequently, the gray bar represents samples appear in other clusters.
previous research (Zhou et al., 2021; Gao et al., 2021; Zhang et al., 2022) shows that they still require adaptation using annotated data from downstream datasets achieve optimal performance, which incurs substantial annotation costs.
Building upon the observation that VLMs, such as CLIP, inherently possess zero-shot image classification capabili- ties, previous studies (Huang et al., 2022; Menghini et al., 2023; Zhang et al., 2024b) explore adapting VLMs for down- stream tasks by leveraging pseudolabels generated by the VLMS themselves. A critical challenge is that VLMM have biased preferences for different classes, which results in imbalanced pseudolabels, thus suffering from confirmation bias (Huang et al., 2022; Zhang et al., 2024b). While exist- ing literature has explored strategies such as enforcing equal number of pseudolabels assigned to all classes (Huang et al., 2022; Menghini et al., 2023) and assigning candidate set of pseudolables each sample (Zhang et al., 2024b), limited research investigates the rationale behind the issue.
1. Introduction
Large vision- language models (VLMs; Radford et al., 2021; Li et al., 2022: 2021; Alayrac et al., 2022; Zhang et al., 2024a; Gao et al., 2024) pre-trained extensive image- text pairs achieve remarkable performance across a wide range foundamental vision tasks, such image classifica- tion (Zhou et al., 2021). semantic segmentation (Xu et al., 2022), and object detection (Gu et al., 2022). Nonetheless,
To fill this gap, we begin by studying the underlying causes of the imbalanced pseudolabels. We identify that the im- balance in pseudolabeling originates from the semantic gap (Xing et al., 2023) inherent in VLMs, where certain class names do not sufficiently correspond to visual con- cepts. To illustrate this, we identify the five classes with the lowest accuracies given by CLIP and visualize their per- class accuracies along with the cluster distribution of cor- responding image features'. As shown in Figure 1, despite
'School of Computer Science and Technology. Harbin Insti- tute of Technology (Shenzhen). Correspondence to: Xinyang Chen <chenxinyang956 wewiinnnnnnggggnlll.ooooommmm gmail.com²,, Xuefeng Bai <baixue- feng@hit.edu.cn>>.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We apply K-Means clustering to image features of samples in RESISC45 extracted by the image encoder of CLIP, forming 45 clusters since there are 45 classes RESISC45.

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
shiftad text feature
shifted text feature
mitigating the confirmation bias introduced by incorrect pseudolabels arising in the training process.
We conduct experiments on six image classification bench- marks across three learning paradigms, comparing our ap- proach with previous methods. The results show that our framework consistently improves performance, achieving new state-of-ther results. Further analysis confirms that the proposed method effectively mitigates the issues of con- cept mismatch and concept confusion, resulting in more balanced pseudolabels. The contributions of this work can be summarized as follows:
mage festumes * ik test feotures
image fustures l text features
O Imnommrtt
O correct
O incormett O correct
Figure 2. left: concept mismatch. right: concept confusion. Please see Appendix A for realistic examples of them.
the extremely low prediction accuracies for these classes, their image features exhibit good clustering performance, as they mostly concentrated in a single cluster. This indicates that CLIP fails to relate the name of certain classes to the corresponding visual concepts, thus resulting in imbalanced classification accuracies in zero-shot predictions.
We identify and analyze the causes of imbalance in VLMs' zero-shot predictions, attributing to concept mismatch and concept confusion.
We propose concept alignment and confusion-awee calibrated margin to address these issues, enhancing pseudolabel balance and accuracy.
To further study the issue of semantic gap, we delve into the erroneous classification results and identify that it leads to two key consequences, as depicted in Figure 2: concept mismatch and concept confusion'. Concept mismatch arises from a severe form of the semantic gap, where the text feature of a class is significantly misaligned with its corre- sponding image features. This leads to incorrect samples being assigned to the class during pseudolabeling based prediction confidence, resulting in persistently low accuracy after fine-tuning. Concept confusion, on the other hand, is more commonly observed between similar classes, when the text features of the class names fail to capture the most distinguishable visual concepts between them. This leads to bias seen in zero-shot prediction among these classes, causing imbalance in prediction and pseudolabels.
We conduct extensive experiments across unsupervised learning, semi-supervised learning, and transductive zero-shot learning, achieving a relative improvement of 6.29% over the SoTA method.
2. Related Works
Vision-Language Models. Recently, VLMs have demon- strated impressive performance on various downstream vi- sion tasks, such as image classification (Zhou et al., 2021; Addepalli et al., 2024), semantic segmentation (Xu et al., 2022; Shi et al., 2024), and object detection (Gu et al., 2022; Kim al., 2024; Wang et al., 2024). The key idea of VL.Ss is to learn representations that bridge the gap between visual and textual modalities, which facilitates general-purpor understanding and reasoning between modalities (Van den Oord et al., 2018), for example, CL.IP (Radford et al., 2021), ALIGN (Li et al., 2021), Florence (Yuan et al., 2021). De- spite great success, recent research (Zhou et al., 2021: Zhang et al., 2024a) indicates that a significant amount of labeled data remains crucial for adaptating VLMs across various downstream tasks, which incurs substantial labeling costs. In this paper, we focus on fine-tuning CLIP widely adopted VLM, in downstream tasks with abundant unla- beled data, thus eliminating dependence on labeled data.
To address these, we propose a concept-adaptive pseudola- beling (CAP) framework to fine-tune mene VLMs with balanced pseudolabels of unlabeled data on downstream tasks. To crase the concept mismatch, we propose a concept align- ment strategy in which we employ iterative clustering to de- tect concept-mismatcher classes and utilize large language models to generate enhanced textual deseriptions. This ap- proach aligns the text feature with the corresponding image features, ensuring that more correct samples are ussigned to the class in the initialization stage. To tackle concept con- fusion, we introduce a confusion-awand -warre calibrated margin that encourages the model to make more distinguishable predictions between similar classes and more balanced pre- dictions over all classes, thereby improving the accuracy of pseudolabels. Moreover, we employ independent adapters on the visual branch to separately learn from highly reliable pseudolabels obtained through concept alignment and dy- namically generated pseudolabels during training, thereby
Learning from Unlabeled Data. In real-world downstream applications, obtaining a considerable amount of labeled data expensive. With the zero-shot classification capa- bility inherent in VLMMs, recent work explores the use of pseudolabeled data for task adaptation. For instance, UPL (Huang et al., 2022), or FPL (Menghini et al., 2023) select top-k confident samples for each class to form a balanced distribution among classes. Building upon this idea, GRIP (Menghini et al., 2023) exploits an iterative strategy, grad- ually increasing the value of with each iteration until all
Take CLIP's zero-shot prediction for RESISC45 as an exam- ple, we discovered approximately 5% classes existing concept mismatch, and about 30% classes suffer from concept confusion, thus harming the accuracy.
2

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
unlabeled data are incorporated in the final iteration. Mean- while, CPL (Zhang et al., 2024b) takes similar iterative strategy to GRIP, assigning each sample a set of candidate pseudolabels each iteration, expecting the true label to be among them. Different from previous work which employ post-hoc methods to optimize pseudolabels, we identify the rationale behind the imbalanced pseudolabels and pro- pose two approaches to address classes with inherently low accuracy, encouraging the model to make more balanced predictions among classes.
Mismatch Detecting
mismathh classes remain
most similar pair
most pair
2 Text Augmentation
3 Mismatch Erased
Please describe the most distingusshng visual froturns of sweel kelliamm type of flowers one sentrnce.
This strategy was initially applied to large language models as a replacement for manually designed prompts and served as a fine-tuning method to provide task- specific information. With CoOp (Zhou et al., 2021) pi- oneering the application of prompt tuning for fine-tuning CLIP, this technique has been increasingly adopted in VLMs (Zhou et al., 2022; Bahng et al., 2022; 7ang et al., 2022). Prompt tuning applied to different modalities has been ex- plored. CoOp concatenate group of continuous vectors to the input at the textual branch, optimizing these vectors to complete fine-tuning. VPT (Jia et al., 2022) introduces a small budget of additional parameters the image encoder, which are prepended into the input sequence of each layer. MaPLe (Khattak et al., 2023) proposes a joint prompting approach by learning context prompts in the textual branch, and projecting them to the visual branch through a linear projection. In this paper, we adopt MaPLe as the prompt tuning method, as it allows learning from both modalities.
Prompt Tuning.
Sweet Williams are characterized by their dereo, flat topped clusters of small, frilly-edged bluoms...
image fexture
*
x bect feature
AAA A. clustor cemers
samplos selected diusee
peredo-laballed semples
Figure 3. The process of concept alignment. We first take an itera- tive clustering strategy to detect the concept-mismatched classes. We then utilize LLMs to generate enhanced descriptions for them, and obtain images with top-k similar image features to the en- haneed text feature each class as pscudolabeled samples.
guides VLMs to generate more distinguishable predictions.
Overview. The overall workflow of our method can be divided into three steps: 1) we employ concept align- ment ($3.1) to refine pseudolabels for cancept-mismatcher classes by detecting them and enhancing their text descrip- tions via large language model, aligning the descriptions with visual concepts; 2) we then alleviate concept confusion with a confusion aware calibrated margin ($3.2). This mar- gin, derived from similarity between classes and model pre- diction tendencies, encourages the model to produce more discriminative and balanced predictions; 3) we propose a fine-tuning framework ($3.3) that utilizes both the pseu- dolabeled data generated in the concept alignment step and the remaining unlabeled data. We deploy main and pseudo adapters on the visual branch to learn from pseudolabeled and unlabeled data separately, and use the confusion-await calibrated margin to compute the loss.
3. Methodology
Problem Definition. This paper studies how to adapt CLIP to downstream tasks without human- annottaee data. For- mally, given a collection of downstream unlabeled data denoted as DUL = {( comprising N instances, we explore how to assign accurate pseudolabels to each instance within a classification label space y = {c)_- The obtained datasets are then utilized to fine-tune CLIP, enhancing its applicability to the downstream tasks. We consider three learning paradigms: unsupervised learning (UL), semi-supervised learning (SSL) (Zhang et al., 2021; Chen et al., 2023; Wang et al., 2023), and transductive zero- shot learning (TRZSL), since all the paradigms explore the exploitation of unlabeled data.
3.1. Concept Alignment
As illustrated in Figure 1 and Figure 2. concept mismatch leads to markedly low accuracy for certain classes. Conse- quently, very few correct pseudolabels are assigned to these classes and the accuracy for these classes remains exception- ally low after fine-tuning. To address this issue, we propose concept alignment process designed to assign more pre- cise pseudolabels to the concept-mismatched classes. As depicted in Figure 3, the process initiates with a mismatch detection algorithm that iteratively excludes well-matched classes, thereby isolating the concept-mismatee instances and their corresponding labels. Subsequently, it utilize a
Motivation. While VLMs have exhibited inherent zero- shot capabilities for pseudolabel generation, the semantic gap inherent in VLMs significantly restricts the accuracy of generated pseudolabel. We conduct an in-depth study on the semantic gap and observe that it manifests two phenomena: concept mismatch and concept confusion. In this paper, we first address concept mismalch through a detect-then- enhance framework; Subsequently, we alleviate concept confasion with a confusion-aware calibrated margin which
3

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
Algorithm 1 Mismatch Detection
Courect Incurcett
Coreect Incoerect
Input: Image feature set I {o; j text feature set
Density
Density
T {un, ) class labels y = {c) threshold
Output: Remaining image feature set Tinal, remaining
class labels Vinal
while y 2 t do
0.2
Confidunce Score
0.4
0.5
0.8
1.0
0.2
Confidence Score
0.4
0.5
1.0
C {c, ) = (Meann(I, (TT)
Figure 4. Density curve of confidence score for samples in concept- confused groups by left: zero-shot CLJP and right: CL.IP fine-tuned with confusion-awee aware calibrated margin.
STC = sim, cy),
Yw; E ET, EC
P softmax(S!!. (S
(i* argmax PTC ij
using large language model (LLM) to generate enhanced text descriptions. Specifically, for each class e in JimM, we query the LLM times to generate corresponding descrip- tions. Akin to a step in Alg. 1, we identify the optimal description as the one that exhibits the highest similarity to one of the centroids derived from clustering on Tinal-
I.- (or Uh II, k E Clustery-)
T - TI (ws-) }. I I II,,, y t 21
end while
Ifmal- = I, Vinal = y
Finally, for each class VMm, we assign corresponding im- ages of the image features with top-k cosine similarity to the text feature of enhanced description as pseudolabeled sam- ples of this class. For each class in V'M'M.. we follow previ- ous work (Menghini et al., 2023) by assigning pesudolabels based on the top-k confidence scores obtained through zero- shot CLIP. We denote pseudolabeled samples generated this stage as DpL = {(r, 5)} with size M k x C. Please refer to Appendix for more details of concept alignment.
large language model to enhance text descriptions, ensuring more accurate alignment with their respective visual con- cepts. In this way, reduces the occurrence of mismatches between visual data and their associated pseudolabels.
Given the text encoder &é and the image encoder & of CLIP, we obtain the image feature a; of each image in DuL and text feature w. of each class label with the template "a photo of [CI.S]", forming:
3.2. Confusion-Aware Calibrated Margin
I= ==(())))= 0. = <(;)),
Confusion frequently arises among similar classes, imped- ing the model's s ability to distinguish these classes. This further leads to biased predictions that lean towards one class, disrupting the balance of the pseudolabels. To ad- dress this, we propose confusion-aware calibrated margin inspired by logit adjustment (Menon et al., 2021), which gradually reduces concept confusion by improving local calibration among confused groups
T= = ((w=)) we ((lemplate(c))
We started by detecting the concept-mismatchen classes by an iterative clustering algorithm based on I, T, and Y. The mismatch detection algorithm is presented in Algorithm In this algorithm, we gradually remove the image features classes, thereby retaining
and text features of only the concept-mismatched classes. For each iteration, we begin by applying K-means clustering to I, forming [T] clusters and obtain the centroids C. We then compute the similarity matrix TTC and probability matrix TCC of T and C. Finally, we figure out the pair of text feature and centroid with the highest confidence score, W; and Cj., and we remove the image features in the cluster corresponding to cy. from I, the text feature wi- from T, the corresponding class label i" from y, since we assume that i" relates to the best-matched class in this iteration.
As shown in Figure 4, zero-shot CLIP often makes incor- rect predictions with high confidence due to concept confu- sion. A 0.6 threshold for many errors and amplify confirmation bias. In contrast, our confusion-aware calibrated margin provides local calibra- tion, reducing confidence for incorrect predictions, making pseudo-labels selected at the same threshold more accurate and improving learning stability.
would introduce
Specifically, we compute the confusion-aware calibrated margin based on the similarity of classes and the model's s prediction tendency. Given an instance æ, we obtain the logit output by CLIP as z. With the instance's label the confusion-ane aware calibrated margin can be defined as a variant of cross-entropy loss:
The algorithm terminates when the size of y falls below predefined threshold t. We denote the remaining image features and class labels as Zrinal and Vinai. We further obtain the classes with the fewest-t samples predicted to be as -io and finally identify the concept-mismatched classes as VAM = Vanal O Vlow-t-
E
Ly 2) = -log c=0 + Ecfy ezo
(1)
For the classes in VAM, we then perform text augmentation

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
where M is the margin matrix, which is constructed from the similarity matrix S and the class-wise margin scales m.
it encourages the model to make more confident predictions between classes with high similarity and in classes with low tendency, thus enhancing the model's discriminative ability for these classes. It should be noted that the margin matrix M is updated at each epoch, thereby facilitating the progressive alleviation of concpet confusion in the generated pseudolabels and improving their accuracy.
We start by computing the similarity matrix S of all classes. Given CLIP with learnable parameters 6, the image features of samples with pseudolabel c are calculated as
0j = (0)))) (ætj, I = (oy 12
E DrL).
We then compute the prototypes of all classes to further obtain their similarity. The visual prototypes are computed as the average of all image features corresponding to certain class, and the text prototypes T are text features extracted by the model:
3.3. Fine- Tine-Tunig with Pseudolabels
Figure 5 illustrates the overall fine-tuning framework of our method. We use MaPLe (Khattak et al., 2023) as our prompt tuning strategy, which learns context prompts separately each layer from layer 1 to layer L,, from both textual and visual branches of CLIP. Different from GRIP (Menghini et al., 2023) and CPL (Zhang et al., 2024b) which only train on pseudolabeled data generated in initialization, we also utilize the remaining unlabeled data by dynamically generating pseudolabels for them during training. Thus, we separate the training data into two parts, the unlabeled samples DuL and the pseudolabeled samples DpL, DpL. is obtained in Section 3.1.
I = ((wa))==
V. = avg(
T= ((w)) w. = vo(())
Then, we obtain the similarity matrix S by computing the maximum similarity between visual prototypes and textual prototypes for each pair of classes
Sij = max(sim (T,, Uy), sim(w,, w,)).
To better utilize the high label accuracy of Dri. and the abun- dance of DUL. we deploy the main adapter and the pseudo adapter on the visual branch. The main adapter learns only from Dpr. and is used to generate pseudolabels for DUL. while the pseudo adapter learns from only DUL under the supervision of these pseudolabels. Since the main adapter only learns from DPL with highly accurate pseudolabels, this strategy avoids the accumulation of errors introduced by the pseudo adapter which learns from less accurate pseu- dolabels. We also deploy an adapter on the textual branch. It is worth noting that we disable all adapters in inference.
To determine the class- wise margin scales m, first com- pute ofc), the number of samples in Dpr. that are classified as class e with confidence exceeding threshold r:
M
(c)) (max((p,) 2 T) 1(arg max(p,) = c). (3)
Based on (c), the model's 's class-wise tendency &, and the overall imbalanced degree A can be calculated as
(c)
8e- 1- max(o(f))
c,j E
(4) Formally, at each iteration, we have {(x)", ))) and ((=;)] where b is the batch size, as subsets of DrL. and DUL, respectively. We use 6 to denote CLIP's text encoder with adapter, and 6Th and 6² CLIP's image encoder with main adapter and pseudo adapter, respectively. For each sample in {(x,' y ((())'' ) we compute the logit 2 as
A max
(5)
For all classes, we define the class-wise margin scale as
m = =mm,, m2, mm)...mmo) m. = m x A x de
(6)
zPl = (21, 22, ...,. =c), 2e = sim( (x""), (c)). (8)
where m is a predefined margin scale.
We then compute the loss on pseudolabeled samples using L. defined in Eq. 1:
Finally, we compute margin matrix M using similarity ma- trix S obtained in Eq. and class-wise margin scale m obtained in Eq. 6 by
CPL
Lm (z) pl .
(9)
b i=1
M S m,
For samples in {(x )i we first follow FixMatch (Sohn et al., 2020) to generate their corresponding pseudolabels, denoted as (or))² using confidence threshold 7.
where @ represents the hadamard product. The margin matrix Mi is combination of inter-class similarity and the class-wise prediction tendency of the model, thus providing a margin that adaptively adjusts between classes based on their similarity and the model's prediction behavior. By incorporating M to cross-entropy loss, as defined in Eq. 1,
Next, we compute the logit as
= (21,22, 22,.... ;0)), &e = sim(o"(9(zz)) 60 (c)),
(10)
5

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
Text Encoder
(C(c)
2Pl
Cr
Layer 1-11
Adapter
Layer
Layer
(/(())
(xe*))
photo
(1((c)
a
Concept
DrL
gu
Alignment
Main Adlepter
4 (200)
(**)
Margin Matrix
Intich emboekange
Layer 1
Layer 1-11
Layer I.
gi (xe2)
Pieudo
DUL + DuL \ DeL
(*(*")
z
CUL
Image Encoder
(x"")
Figure 5. Overview of our framework. In the initialization stage, we use concept alignment to obtain DEL. the fine-tuning stage, deploy the main adapter and pseudo adapter to the visual branch, allowing for separate learning from pseudolabeled and unlabeled samples, and we utilize the confusion-awee calibrated margin matrix M to compute the loss.
where Ω represents image augmentation operation, and com- pute the loss on unlabeled data by
et al., 2017), DTD (Cimpoi et al., 2014), EuroSAT (Helher et al., 2019), FGVC-Aireraft (Maji et al., 2013), CUB (Wah et al., 201 Flowers 102 (Nilsback & Zisserman, 2008).
8' Cm
CUL
(11) Learning Paradigms. To ensure thorough evaluation, investigated three distinct learning paradigms: 1) un- supervised learning (UL) which provides with unlabeled data without any prior labels; 2) semi-supervised learning (SSL) which combines 2 labeled data per class with a larger pool of unlabeled data; 3) transductive zero-shot learning (TRSSL) which classes are devided into seen classes with fully labeled data and unseen classes with fully unlabeled data. We set the ratio of seen to unseen classes at 62-38.
6 1=1
where D' is the number of samples with pseudolabels. Finally, the overall loss formulated as
L CPL + LUL-
(12)
In the manner of semi-supervised learning and transductive zero-shot learning, also have access to labeled samples {aa), v/) } After we obtain z' similarly in Eq. 8, we compute L1 as
Model Configuration. In concept alignment, we set / TO
to determine the concept-mismatched classes. We use ChatGPT 4o-mini and set the query times n to obtain the enhanced descriptions. We set k = 16 to generate DPL- We set m as the predefined margin scale to compute the confusion-aware calibrated margin. Please refer to Appendix C.1 for more details.
h
C.
Lm (2/. w!),
(13)
and add it to the overall loss:
== LPL + LUL + LL.
(14)
Baselines. We compare our method with three existing methods, namely, Few Pseudolabels (FPL; Menghini et al., 2023), Grow and Refine Iteratively Pseudolabels (GRIP; Menghini et al., 2023), and Candidate "xeudo-Labeling (CPL; Zhang al., 2024b). Since these methods can be ap- plied with different prompting modalities, report the re- sults for the modality with the highest overall performance.
4. Experiments
To examine the effectiveness of the proposed method, we conduct extensive experiments under three different learning paradigms across six benchmarks and compare our method with SoTA methods (64.2). Moreover, we conduct ablation study ($4.3) and analysis ($4.4) to explore how the proposed method improve the performance on downstream tasks.
Evaluation Metric. Following CPL (Zhang et al., 2024b), we employ accuracy as the metrie for evaluating model per- formance on test sets. We report the harmonic mean of the accuracies of seen and unseen classes TRZSL. Specif- ically, we report the performance by calculating the test accuracy averaged over three seeds with standard deviation.
4.1. Experimental Settings
Datasets. We consider six image classification datasets covering diverse domains, including RESISC45 (Cheng
6

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
Table 1. Comparison results of test accuracy (%) on six henchmarks. The highest accuracies are hold.
Flowers102
RESISC45
DTD
Methods
SSL
UL
TRZSL
SSL
UL
TRZSL
SSL
UL
TRZSL
zero-shot CLIP
63.67a.00
63.40p.00
54.488.00
54.466.00
43.24p
43.45
FPI. (Menghini et al., 2023)
75.960.74
65.67
80.97c.00
68.130.55
63.07o.as
72. 1o.00
37. 37.10s.45
44.96g.ss
46.30g.os
GRIP (Menghini et al., 2023)
83. 53.60p.48
69.841
86 36.266.00
74.1lo.08
70.55p.a5
81.076.00
56.07a.8s
46.091.6
65.30g.01
CPI. (Zhang et al., 2024b)
89.66g.30
72.90..
87 37.350.76
80.98o.11
77.390.44
85.850.49
61.21a.36
51.91
68.00g.34
CAP (Ours)
89.96..46
76.80
89,53
83.32..39 59
81.48).45
88.82o.18
62.33
55.29
69.55
FA
CUB
FGVCAircrafi
Methods
SSL
UL
TRZSL
SSL
UL
TRZSL
SSL
UL
TRZSL
zero-shot CLIP
32.88o.00
30.54g.00
51.82a.00
51.577.00
17.58p 00
17.86g.00
FPL (Menghini et al., 2023)
62.05,.a .s4
48.96 .99
53. 53.70bs.st s7
55 65.29%..5 50
53.04o.sa 53
65.440.20
20.022..7
16.62
17.55g.37
GRTP (Menghini et al., 2023)
68.662.44
57.21,
92.33p.0
56.635 06.655.39
51.42..a1
59.48p.38
16.980.82
15.22
26.08g.25
CPL (Zhang et al., 2024b)
77.51
67.2 67.26p.44
93.78o.12
58.530.24 24
53.47o.a6
66.20o.50
22.48%.ca
18.35p.27
30.86u.70
CAP (Ours)
92.78h.31
75.01 94
96.649.27
08.044.25
55.76).u 20
61.35g.05
21.799.16
18.42
29.039.49
4.2. Main Results
Table 2. Ablation results of Concept Alignment (CA) and
Irnfusion-Awart Calibrated Margin (CACM).
Table I compares the performance of different methods un- der vairous learning settings. We compare the proposed method with zero-shot CLIPP FPI., GRIP, and CPI. across six datasets. It can be observed that the proposed method consistently surpasses existing methods under UL setting. Notably, our approach achieves significant improvements Flowers 102, RESISC45, and EuroSAT datasets, surpassing the CPL baseline by 3.90%, 4.09%, and 7.75%, respec- tively. These results underscore the efficacy of our method in leveraging unlabeled data to achieve superior classifi- cation outcomes. Moreover, our approach demonstrates competitive performance under the SSL setting across all datasets. This indicates that our method is capable of effec- tively integrating scarce labeled data with unlabeled data to enhance model performance. Under the TRZSL setting, our method surpasses CPL on Flowers102. RESISC45, DTD and ETroSAT by 2.18% 2.97%, 1.55% and 2.86%, respec- tively. This suggests the capability of our method to leverage abundant labeled data with unlabeled data, further validating its versatility regarding different paradigms.
CA
CACM
RESISC45
DTD
Flowers
X
68.41
49.57
70.85
x
72.48
53.13
72.58
X
V
78.77
52.76
74.49
V
82.03
55.26
76.77
Table 3. Ablation results of Independent Adapters (IA).
Dataset
CPL
w/ IA
w/o IA
DTD RESISC45
51.9
55.3
54.6
77.4
81.5
80.6
EuroSAT
72.9
76.2
78.3
baseline across all datasets. This underscores that each com- ponent independently facilitates better performance than the baseline. Compared with CA, CACM exhibits greater improvements on the RESISC45 and Flowers datasets, while achieving slightly lower results on the DTD dataset. A possible reason is that CLIP performs inferior extracting image features in DTD, leading to a substantial amount of mutual confusion among image features which is hard to address with prompt tuning. Moreover, the combination of both methods consistently yields the highest accuracy, highlighting their complementary effects in improving the accuracy of pseudolabels.
In addition, as illustrated in Figure 14, our method forms remarkably more balanced predictions compared to the base- line, which is inline with our motivation. Furthermore, our method is considerably less time-consuming than CPL and GRIP, achieving about 3.5 times the speedup over GRIP. Please refer Appendix D.I and D.2 for more details.
4.3. Ablation Study
A Ablation of Independent Adapters. In CAP, we deploy in- dependent adapters on the visual branch to separately learn from DpL. and DUL, thus avoiding the accumulation of er- rors introduced by incorrect pseudolabels. To explore the effect of independent adapters, we report the test accuracy under UL setting across three dataset with and without in- dependent adapters in Table 3. It can be observed that both model gives better results than CPL, and shared adapters gives generally comparable results to independent adapters.
Ablation of Concept Alignment and Confusion-Aware Calibrated Margin. To evaluate the effectiveness of each component of our method, we conducted ablation study by independently removing each module and assessing the model's performance across three datasets under the UL setting. As shown in Table 2, both components individu- ally result in noticeable performance improvements over the

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
0.86
With Concept Coerection
Widhout Concept Correction
0.64
Pecudolabe Accuracy
1.0
1.0
Acourcy
0.54
Accuracy
0.62
0.8
Tost A.caraaa
0.8
0.6
0.6
0.82
0.4
0.4
02.
0.2
0.80
12
14
16
18
20
Value of
Value of
Figure 6. Ablation of different values of under SSL setting. We
report the test accuracies on RESISC45 and DTD.
Figure 8. Evaluation of concept alignment. left: The accuracy of pscudolabels generated for concept-mismatchie classes in Flow-
ers102. right: The test accuracy after fine-tuning under UL setting.
RESISOOS
LTTD
Accuracy
Accuracy
0.4
1.0
Lovaal BCB
0.5
0.
Value of 1
Figure 7. Ablation of different values of 7 under UL setting. We
report the test accuracies RESISC45 and DTD.
0.
Confusion Groups
Confusion Groups
Figure 9, Evaluation of confusion aware calibrated margin on RE- SISC45, left: Local ECE of concept-contee confused groups. right: Test accuracy of concept-confused groups after fine-tuning.
Ablation of Hyperparameters. In CAP, two hyperparam- eters are directly related to the generation of pseudolabels. Specifically, determines the number of pscudolabels gener- ated for each class in the initialization stage, while serves as the confidence threshold for dynamically assigning pseu- dolabels during training. We conduct ablation studies on different values of k and 7 using the RESISC45 and DTD datasets, and present the results in Figure 6 and Figure 7, re- spectively. The performance remains relatively stable across a wide range of values for both & and r, indicating that CAP generally robust to moderate fluctuations in these settings.
produces superior results compared to the top-k method, notably within the classes "globe flower", "love in the mist". and "great masterwort". This confirms the effectiveness of concept alignment in improving pseudolabel quality and further enhancing overall performance.
Does Confusion-Aware Calibrated Margin Effectively Reduce Concept Confusion? We first evaluate its impact on local Expected Calibration Error (ECE) for different concept-confused groups in RESISC45. As illustrated in the left sub-figure of Figure 9, it evident that directly fine-tuning the model using cross-entropy loss leads to substantial increase in local ECE for most concept-confusee groups. This indicates that naive fine-tuning approach re- sults in poorer calibration and reduces reliability of the pseudolabels. contrast, our proposed confusion-aware calibrated margin addresses this issue by explicitly promot- ing more distinguishable logits among concept-confused classes By promoting greater separation among logits of similar classes, this mechanism helps the model avoid over- confident yet incorrect predictions. This results in notably lower local ECE values, reflecting improved calibration within the concept-confused groups, thus enhancing the accuracy and balance of pseudolabels. Furthermore, the right sub-figure of Figure compares the downstream per- formance of different methods. Our approach consistently outperforms both zero-shot CLIP and fine-tuning without the calibrated margin across all concept-confused groups. This not only highlights the effectiveness of our method in improving calibration but also demonstrates its ability to enhance the model's discriminative ability.
4.4. Analysis
Does Concept Alignment Effectively Reduce Concept Mismatch? The core of concept alignment lies in accu- classes,
rately detecting and correcting moncept-mismatchht thereby improving the accuracy of pseudolabels. To vali- date this, we examined the accuracy of the pseudolabels cor- rected by our approach and compared it to the top-k strategy employed by UPL (Huang et al., 2022) and GRIP (Menghini al., 2023)². The left sub-figure of Figure 8 visualizes the accuracy of generated pseudolabels for concept-mismatcher classes in the Flowers 102 dataset. As observed, the concept alignment approach results in consistently higher accura- cies compared to the top-k strategy, with significant boost over 60% seen in four classes. This indicates that our pro- posed concept alignment mechanism effectively mitigates the concept mismatch issue. To further examine the impact of concept alignment, we evaluate the test accuracy after fine-tuning downstream datasets under the UL setting. As shown in the right figure of Figure 8, concept alignment
We also present the mismatch detection results, please refer to Figure 16 for detailed information.
8

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
0.34
Table 4. Results of test accuracy (%) using ViT-L/14 as the visual backbone. The highest accuracies are bold.
RESISC4S
Accurasy
00.0
Accurcy
Methods
SSI.
UI.
TRZSI.
0.50
0.56
Zero-shot CLIP
52.45
51.61
GRIP
60.91
54.40
64.92
0.78
10
12 Margin Scnle
12
13
0.52
9
10
11
12
13
14
CPL
69.82
57.20
71.97
Margin Scale
CAP (Ours)
70.26
65.85
73.61
Figure 10. Evaluation of margin scale m. We report test accuracy of RESISC45 and DTD under UL setting.
RESISC4S
Zero-shot CLIP
62.67
62.13
GRIP
81.53
76.86
86.88
CPL
87.75
80.88
89.73
1.0
1.0
CAP (Ours)
88.09
84.46
90.21
Accuracy
Accuracy
Flowers 102
Zero-shot CLIP
73.98
73.05
0.6
0.6
GRIP
94.21
82.33
96.18
CPL
96.80
83.94
97.34
0.2
SSL on DTD
0.2
SSL RESISSAAS
CAP (Ours)
98.09
84.19
95.40
DTD
UL on RESISC45
0..0
0.2
0.1
0.6
0.8
1.0
0.000
0.6 Datla Scale
0.1
10 0.8
Data Scale
Figure 11. Performance DTD and RESISC45 under SSL and UL. settings with different proportion of unlaheled data.
mance experiences rapid improvement more unlabeled data becomes available, but eventually reaches point where further increases in unlabeled data provide diminishing re- turns. The observed plateaus in performance indicate that the proposed method is highly efficient in utilizing unla- beled data, achieving substantial gains with relatively small amounts of unlabeled samples.
Analysis of Margin Scale. In the confusion-aware cali- brated margin, we use a predefined margin scale m in com- puting the margin matrix, as shown in Eq. 6. Figure 10 presents the test accuracy under the UL setting for various values of m on RESISC45 and DTD. Generally, the results show that the test accuracy remains relatively stable across different margin scales, suggesting that the proposed method is robust to variations in margin scale. Closer examination reveals a trend with drops at both ends. When m is too small, the margin introduced to concept-confused groups becomes insufficient, making difficult for the model to effectively resolve confusions between them. This results in limited improvement in the distinguishability of the model between concepts- confuuee classes. Conversely, setting m too large introduces excessively strong separation between class logits, which can lead to unstable training dynamics, harming the improvement of performance. Therefore, se- lecting moderate value such as m 12 offers good trade-off between enhancing inter-class distinguishability and maintaining stable optimization dynamics.
Different Image Encoders. To evaluate the generalizability of our method on different visual backbones, we conduct additional experiments on DTD, RESISC45 and Flowers 102 utilizing a larger image encoder ViT-14/L as the visual back- bone. As shown in Table 4, our method consistently out- performs other approaches across all three datasets. The performance demonstrates the robustness and adaptability of our method when integrated with different visual back- bones, and that our method effectively leverage the representational capacity of different vision models.
5. Conclusion
In this paper, we delved into the issue of imbalance in pseu- dolabels generated by VLMs, identifying two core under- lying causes: concept mismatch and concept confusion. Building on the analysis, proposed novel framework incorporating concept aligmment and confusion-aware cal- ibrated margin to address two challenges. Our approach is capable of focusing on the underperforming classes and promoting balanced predictions across categories, thus im- proving the accuracy and balance of pseudolabels, leading to optimized performance. Extensive experiments on six benchmark datasets with three learning paradigms show that the proposed method effectively mitigates the issues of concept mismatch and concept confusion, resulting in more balanced and reliable pseudolabels, achieving a relative im- provement of 6 29% over the SoTA method.
Impact of Data Scale. To investigate whether our approach can efficiently leverage limited data, we evaluate our meth- ods with different proportion of available unlabeled data. Figure 11 illustrates the impact of the scale of unlabeled data on the model performance under SSL and UL settings on DTD and RESISC45. The curves display a characteristic pattern of initial rapid growth followed by a gradual plateau, with around 0.6 and 0.4 on DTD and RESISC45, respec- tively. This indicates that while increasing the amount of unlabeled data leads to clear performance gains initially, the marginal benefit diminishes as the scale grows beyond a certain point. This pattern suggests that the model's perfor-

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
Impact Statement
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision- language models with feature adapters, ArXiv preprint, abs/2110.00544, 2021.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gao, P., Geng, S., Zhang, R., Ma. T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision-language models with feature adapters. Int. J. Comput. Vis., 132 (2):581- 2024. doi: 10. 10.1007/S1 1007/S11263-022--188--1
Acknowledgement
This work was supported by the National Natural Science Foundation of China (62306085, 62406091, 62206074, 62476071, 62236003), Shenzhen College Stability Support Plan (GXWD20231 1301502
Gu, X., Lin, T., Kuo, W., and Cui, Y. Open-vocabulary object detection via vision and language knowledge dis- tillation. In Proc. of ICLR. OpenReview.net, 2022.
SXWD20220811172233888 Guangdong Basic and Ap- plied Basic Research Foundation (2025A1515012952)
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote 2019.
References
Sensing, 12
Addepalli, S., Asokan, A. R., Sharma, L.,, and Babu, R. V. Leveraging vision-language models for improving do-
Huang, T., Chu, J., and Wei, Unsupervised prompt
learning for vision-language models. ArXiv preprint, abs/2204.03649, 2022.
main generalization in image classification. In JEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. pp- 23922-23932. IEEE, 2024. doi: 10.1 19/CVVP52233 2024.02258.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In Proc. of ECCV, pp. 709-727. Springer, 2022.
Alayrac, J., Donahue, J., Luc, P., Miech, A., Barr, I., Has- son, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick. J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: visual language model for few-shot learning. In Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Proc. of NeurlPS, 2022.
Khattak, M. U., Rasheed, H. A., Maaz, M., Khan, S. H., and Khan, F. S. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 19113-19123 IEEE, 2023. doi: 10. 1109/CVPR52729.000 202.32222.222218...
Kim, J., Ku, Y., Kim, J., Cha, J., and Baek, S. Vim-pl: Advanced pseudo labeling approach for class incremen- tal object detection via vision-language model. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4170-4181 2024.
Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, Exploring visual prompts for adapting large-scale models. 2022.
Li, J., Selvaraju, R. R., Gotmare, A., Joty, S. R., Xiong. C., and Hoi, S. C. Align before fuse: Vision and language representation learning with momentum distillation. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Proc. of NeurIPS, pp. 9694- 9705, 2021.
Chen, H., Tao, R., Fan, Y. Wang. Y., Wang, J., Schiele, B., Xie, X., Raj, B., and Savvides, M. Softmatch: Addressing the coantity-quality tradeoff in semi- superviee learning. In Proc. of ICLR. OpenReview.net, 2023.
Li, J., Li, D.,. Xiong, C., and Hoi, S. C. H. BLIP: boot- strapping language- image pre-training for unified vision- language understanding and generation. In Chaudhuri, K., Jegelka, S., Song, I.., Szepesvári, C., Niu, G., and Sabato, S. (eds.), Proc. of ICML, volume 162 of Proceedings of Machine Learning Research, pp. 12888 -19800. PMLR, 2022.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105(10))1855-1883, 2017.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp. 3606-3613, IEEE Computer Society, 2014. doi: 10. 10.1109/CVPR.2014.44
Maji, S., Rahtu, E.,, Kannala, J., Blaschko, M., and Vedaldi, A. Fine- line-grained visual classification of aircraft, arXiv preprint arXiv: TrXiiv/3366.55 306.5111, 2013.
10

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
Menghini, C., Delworth, A., and Bach, S. H. Enhancing CLIP with CLIP: exploring pseudolaheling for limited- label prompt tuning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Proc. of NeurlPS, 2023.
Xing, Y., Kang, J., Xiao, A., Nie, J., Shao, L.,, and Lu, S. Rewrite caption semanties: Bridging semantic gaps for innguage-superisse semantic segmentation. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Proc. of NeurlPS, 2023.
Menon, A. K., Jayasumana, S., Rawat, A. S., Jain, H., Veit, A., and Kumar, S. Long-tail learning via logit adjustment. In Proc. of ICLR. OpenReview.net, 2021.
Xu, M., Zhang, Z.., Wei, F., Lin, Y., Cao, Y., Hu, H., and Bai, X. A simple bascline for open-vocabulary semantic seg- mentation with pre-trained vision- laiionl model. In Avi- dan, S., Brostow, G. J., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Proc. of ECCV, volume 13689 of Lec- ture Notes in Computer Science, pp. 736-753. Springer, 2022.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In ICVGIP, pp. 722-729, 2008.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T. (eds.), Proc. of ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Yuan, L.,, Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang. X., Li, B., Li, C., et al. Florence: A new foundation model for computer vision. ArXiv preprint, 0ts/2111.111433 2021.
Zang, Y., Li, W., Zhou, K., Huang, C., and Loy, C. C. Uni- fied vision and language prompt learning, ArXiv preprint, abs/2210.07225, 2022.
Shi, H., Dao, S. D., and Cai, J. LInformer: Large lan- guage model for open open-vocabulary semantic segmentation. International Journal of Computer Vision, 2024.
Zhang, B., Wang, Y., Hou, W., Wu, H., Wang. J., Oku- mura, M., and Shinozaki, T. Flexmatch: Boosting semi- supervised learning with curriculum pseudo labeling. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, W. (eds.), Proc. of NeurIPS, pp. 18408- 18419, 2021.
Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C., Cubuk, E. D., Kurakin, A., and Li, C. Fix- match: Simplifying semi-supereieed learning with con- sistency and confidence. Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Proc. of NeurlPS, 2020.
Zhang, J., Huang, J., Jin, S., and Lu, S. Vision-language
models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Van den Oord, A., Li, Y., and Vinyals, O. Representa- tion learning with contrastive predictive coding. arXiv e-prints, pp. ar0iv-1807, 2018
Zhang, J., Wei, Q., Liu, F,, and Feng, L. Candidate pseu- dolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data. In Proc. of ICML. OpenReview.net, 2024b.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset. Technical Report ANS-TR--011-001 California Institute of Tech- nology, 2011.
Zhang, R., Zhang, W., Fang. R., Gao, P., Li, K., Dai, J.. Qiao, Y., and Li, H. Tip-adapter: Training- free adaption of CLIP for few-shot classification.. In Avidan, S., Brostow, G. J., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Proc. of ECCV, volume 13695 of Lecture Notes in Computer Science, pp. 493-510. Springer, 2022.
Wang, K., Cheng, L., Chen, W., Zhang, P., Lin, L., Zhou, F., and Li, G. Marvelovd: Marrying object recognition and lasionnlanguage models for robust voan-voca object detection. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vision ECCV 2024 18th European Confer- ence, Milan, Italy, September 29- October 4, 2024, Pro- ceedings. Part XVII, volume 15075 of Lecture Notes in Computer Science, pp. 106-122. Springer, 2024. doi: 10.10077978-3-031--226333
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. ArXiv preprint, abs/2109.01134, 2021.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Condi- tional prompt learning for vision-languee models. In IEEE/CVF Conference on Computer Vision and Par- tern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 16795-16804 IEEE, 2022. doi: 10.1109/CVPR52668 2022. 111.0000..22882022166611....
Wang, Y., Chen, H., Heng. Q. Hou, W., Fan, Y., Wu, Z Wang, J., Savvides. M.. Shinozaki, T., Raj. B., Schiele, B., and Xie, X. Freematch: Self-adaptive thresholding for semi-supervised learning. In Proc. of ICLR. OpenRe- view.net, 2023.

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
Appendix for "Handling Imbalanced Pseudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin"
A. Examples of Mismatch and Confusion
In this section, we provide realistic examples of concept mismatch and concept confusion. We show the t-SNE plots of image features extracted by CLIP from certain classes in RESISC45 dataset with their corresponding true labels and predicted labels generated by zero-shot CLIP.
Figure 12 shows examples of concept mismatch. It indicates that although the image features of these classes are relatively distinguishable, the samples represented as pink are completely misclassified. This illustrates that the text features of certain classes fail to capture the corresponding visual concepts, leading to significant semantic misalignment.
Figure 13 shows examples of concept confusion. The interwoven distribution of image features between these classes suggests a high degree of similarity. In the zero-shot prediction, most of samples are predicted to be certain class, leaving relatively scarce samples predicted as the other. This indicates the text feature of the minority class fails capture the most distinguishable visual concepts to align with corresponding image features, resulting imbalanced prediction.
True Distribution
Predicted Distribution
True Diseibuiio
Predicaa Dottibution
Figure 12. Two examples concept mismatch in RESICS45. Pink represents the classes exisiting concept mismatch.
The Distribution
Prodicted d Distribution
True Distribationn
Predictal Distibution
Figure 13. Two examples of concept confusion RESICS45. Yellow represents the minority class.
B. Details of Concept Alignment
In this section, we introduce more details of Alg. 1, and present examples of the prompt we use and the enhanced descriptions generated from LLMs.
B. Details of Removing Samples
In Alg. 1, we remove the image features from the cluster corresponding to the class with best cross-modal alignment. However, the quantity of samples in this cluster might be too large, potentially containing samples for other classes. Thus,
12

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
only remove top-s samples predicted to be the best-matched class sorted by confidence, where is the average size of image features per class
Z
In each iteration, after we obtain i", we obtain the image features predicted to be class i" as
I.- {z E I, Uk- = =
where Dh represents the predicted label of the sample corresponding to Uk- Then, we obtain the intersection of Ię- and I,., sort it by confidence score, and finally, select the top-s samples as the samples to remove:
Ij- + sIr [I
B.2. Details of Text Augmentation
For text augmentation, we use the prompt template "Please describe the most distinguishing visual features of [CLS], in sentence.". We present the enhanced descriptions of concept- concept-mismatched classes detected in RESISC45 below an example.
Q: "Please describe the most distinguishing visual features of a satellite photo of chaparral, in one sentence."
A: "A satellite photo of chaparral features dense, patchy, and irregular mosaic of shrubland vegetation, with a dominant brown or beige tone, and speckled or mottled texture created by the mixture of shrubs, bare soil, and rocky outcrops."
Q: "Please describe the most distinguishing visual features of a satellite photo of terrace, in one sentence."
A: "A satellite photo of terrace is distinguished by series of flat or gently sloping, rectangular or stepped areas, usually with distinct boundaries and varying tones or textures that distinguish the terraced fields."
B.3. Details of Identifying the Optimal Description
Given remaining image features Iinal and the corresponding text features of n candidate descriptions of class denoted as Te, we first execute K-Means clustering to Tanal forming centroids Crnal by
Cinal = {c;),== Tel = KMeans(Zanat Tel))
We then compute the similarity matrix and probability matrix of Te and Crnnl and identify the pair with highest confidence score similar to Alg. 1, as
STC = sim (w,, cj), Yw; E Te, E Canal
TTC = softmax (
(f' argmax PTC
Finally, we select the description corresponding to wi- as the optimal enhanced description for class
B.4. Details of the usage of DPL
We add unlabeled data with high confidence into DPL to enhance the abundance of DPL during training. Specifically, we increase the size of DPL by [Puul every 5 epochs. Denote as epochs to complete in one training, we compute as
t = Once every 5 epochs, for each class label c, we obtain the unlabeled samples with predicted label c with top- txC confidence, and add them into DrL as pseudolabeled samples.
[Puu]
13

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
Table Detailed settings for experiments.
Flowers102
RESISC45
DTD
CUB
EuroSAT
FGVCAírcraft
Statistic data
Class number
102
45
47
200
10
100
Training set size
2040
6300
3760
5594
27000
6667
Testing set size
6149
25200
1880
5794
5000
3333
Training Setting
Prompt Layers L
Prompl per Layer Image Augmentation
random resized crop
Confidence Threshold & in top-k strategy
0.85
0.5
16
Network
ViT-B /2
Batch size
32
Epoch
where the epoch warmup
Optimizer
SGD
Momentum
0,9
Learning rate (LR)
0.01
Weight decay
0.1
LR scheduler
CosineAnneulingLR
C. Experimental Details
C.1. Training Settings
We present the detailed training setting in Table 5.
C.2. Comparison Methods
We briefly introduce the baselines in this section.
Few-pseudolabels (FPL) (Menghini et al., 2023): FPL the same as UPL (Huang et al., 2022), which generates offline pseudolabels by selecting the top-k confident samples per class in zero-shot predictions of CLIP.
Grow and Refine Iteratively Pseudolabels (GRIP) (Menghini al., 2023): GRIP is built upon FPL, taking iterative training strategy. For each Iteration, GRIP select top-k confident samples per class in predictions by CLIP with soft prompts trained in last iteration. All soft prompts re-initialized then, and GRIP start new training iteration. Notably, the value of l progressively increases after each iteration, and all unlabeled data will be included in the last iteration.
Candidate Pseudolabel Learning (CPL) (Zhang et al., 2024b): CPL takes similar iterative strategy of GRIP, with a different strategy to select pscudolabels each iteration. CPL draw inspiration from the concept the multiple annotations in crowdsourcing, constructing set of potential true labels for model learning. CPL utilize inter-instance and intra-instance label selection assign set of candidate pseudolabels a sample, and employ loss fuction designed for partial-label learning update the soft prompts.
D. More Experimental Results
D.1. Effect of our method CAP
To illustrate the overview of the effect of our method CAP, we present the results after fine-tuning with and without CAP in Figure 14.
D.2. Training Time
We present the time consumed training on EuroSAT with CAP (out method), CPL and GRIP in Figure 15. Our method takes significantly less time to complete fine-tuning, since the other two methods take an iterative strategy which executes the training process several times while our method only trains to converge once.
14

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
Wath Concegt Ara Pieodolabeling
Without Coucept Adapure Preodolabeling
1.0
0.4
0.6
Figure 14. Test accuracy after fine-tuning with CAP and without CAP RESISC45 under UL setting. Note that still reserve the fine-tuning framework CAP in the control group. It is clear that CAP forms a significantly more balanced prediction.
Training Time on EuroSAT
42min
29mmm
CAPiOurs)
CPL
GRIP
Figure 15. Time consumed training EuroSAT with CAP, CPL and GRIP.
D.3. Number of Mismatch Classes Detected
We present the number of concept-mismatched classes detected in Figure 16. Our approach achieves significant performance improvement for previously underperforming classes while maintaining the exceptionally high accuracy of well-performing classes, demonstrating remarkable balance in model predictions.
Mismatch Detected Over Different Datasets
0
EuroSAT (10 classe)
RESICS45 (45 classes)
DID (47 classes)
Flowers102 (102 classes)
ROVCÁrraaa (100 classes)
CUB (200 classes)
Figure 16. Visualization of the number of classes detected with concept mismatch six datasets.
D.4. More Evaluation of Confusion-Awart Calibrated Margin
The purpose of cinfusion-award calibrated margin is encourge CLIP to generate more distinguishable logits, thus gradually improve the balanced degree of CLIP's prediction among classes. In Figure 17, we illustrate the class-wise test accuracy on RESISC45 after menituning under UL setting, highlighting the impact of incorporating the confusion aware calibrated margin. From the left part of Figure 17, we observe that incorporating the confusion-awart calibrated margin significantly improves the accuracy of the powest-performing classes. This indicates that the margin effectively mitigates the imbalance in predictions and reduces concept confusion among similar classes. On the right side of Figure 17, we see that the accuracy remains stable for well- peelicredd classes when applying confusion-awee aware calibrated margin. These results highlight the effectiveness of confusion-awee aware calibrated margin by addressing low-performing classes while maintaining high accuracy
15

---

Handling Imbalanced Pscudolabels for VLMs with Concept Alignment and Confusion-Aware Calibrated Margin
for others.
With Confusion-Aware Margin
Without Confusion-Aware Margin
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
3
10
12
13
Class ID
Class ID
Figure 17. Class-wise test accuracy on RESISC45 under UL setting as an evaluation of confusion-awer aware calibrated margin. We disable concept alignment here. left: Visualization of lowest- 10 class-wise accuracies. right: Visualization highest- class wise accuracies.
16


Reviews:
[{"review_id": "SeavrF2YqO", "reviewer": "Reviewer_d39N", "summary": "The paper addresses concept mismatch and confusion when adapting VLMs to downstream tasks using pseudo-labeled data. The authors argue that this issue primarily arises from an imbalance in the pseudo-labels generated by VLMs. To mitigate this, they propose a concept alignment mechanism and a confusion-aware calibrated margin strategy. Additionally, they introduce independent adapters that separately process the original labeled data and pseudo-labeled data. Extensive and diverse experiments demonstrate the effectiveness of the proposed methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and helpful comments! We address each of your concerns below.\n\n> **Q1:** The ablation studies could be more comprehensive\n\nFollowing your feedback, we have conducted additional ablation studies on the components you mentioned.\n\n**Ablation of gamma Selection**\n\nWe conducted experiments on different values of $\\tau$ (we assume that the reference to gamma was intended to refer to $\\tau$, the confidence threshold) under UL setting. The results are as follows:\n\n*Table 1: Performance regarding $\\tau$ under UL setting.*\n|   | $\\tau$ = 0.80 | $\\tau$ = 0.82 | $\\tau$ = 0.85 (Default) | $\\tau$ = 0.87 | $\\tau$ = 0.90 |\n|----------|---------|---------|------------------|---------|---------|\n| DTD | 55.9 | 56.3 | 57.1 | 56.4 | 56.7 |\n| RESISC45 | 80.9 | 81.2 | 81.4 | 80.5 | 81.6 |\n\nIt can be observed that our method is generally robust to changes in $\\tau$.\n\n**Ablation of Independent Adapters**\n\nWe evaluate our method with and without the independent adapters under UL setting. The results are as follows:\n\n*Table 2: Performance of different adapter configurations under UL setting.*\n|    | CPL (baseline) | w/ Independent Adapters | w/o Independent Adapters |\n|----------|---------|---------|---------|\n| DTD | 51.9 | 55.3 | 54.6 |\n| RESISC45 | 77.4 | 81.5 | 80.6 |\n| EuroSAT | 72.9 | 76.2 | 78.3 |\n\n\nIt can be observed that both model gives better results than CPL, and shared adapters gives generally comparable results to independent adapters.\n\n**Impact of Prompt Tuning Methods**\n\nWe replace MaPLe with two prompt tuning methods (i.e., UPT [1] and VPT [2]) and compare the performance under UL setting.\n\n*Table 3: Impact of different prompt tuning methods.*\n|   | DTD | EuroSAT |\n|----------|---------|---------|\n| **CAP w. UPT** | 54.9 | 77.5 |\n| **CAP w. VPT**  | 56.2 | 77.3 |\n| **CAP w. MaPLe** |  55.3 | 75.0 |\n\nThe results demonstrate that CAP can be effectively integrated with various prompt tuning methods.\n\n[1] Unified Vision and Language Prompt Learning. ArXiv, abs/2210.07225, 2022.  \n[2] Visual Prompt Tuning. In Proc. of ECCV, 2022.\n\n> **Q2:** Abbreviations (e.g., CA, CACM) are not properly explained\n\nThank you for pointing this out. We apologize for the oversight. **CA** and **CACM** are abbreviations for **C**oncept **A**lignment and **C**onfusion-**A**ware **C**alibrated **M**argin. We will ensure that these abbreviations are properly explained in the next version.\n\n> **Q3:** Analysis of underperformance on CUB TRZSL\n\nThanks for pointing this issue out. By analyzing the confusion matrices, we found that our method has more classes with zero accuracy than CPL. To further study this, we conduct a in-depth analysis of the results on CUB.\n\nCUB consists of 200 fine-grained bird species and exhibits **frequent concept mismatches**. For instance, CUB includes **four species of orioles**, but CLIP extracts **highly similar text features** for their category names. As a result, all four species are aligned with the visual features of a single dominant oriole species, leading to incorrect pseudo-labeling. Due to the **simplicity of our mismatch detection algorithm**, many of these mismatches go undetected, preventing affected classes from receiving correct pseudo-labels and limiting fine-tuning improvements.\n\nIn contrast, CPL assigns multiple pseudo-labels to each sample. Since the misclassified categories are **semantically related**, there is **a higher chance that the correct label appears within CPL’s candidate label set**. This enables CPL to improve accuracy on these classes more effectively, which explains its superior performance under this particular setting. However, such extreme concept mismatches are relatively rare across datasets.\n\n> **Q4:** Rationale behind (2)\n\n**The choice of similarity:** In Equation (2), we use cosine similarity to measure the similarity between the text and visual prototypes of two classes, as cosine similarity is the most commonly used similarity metric in CLIP and relevant literature [3,4].\n\nWe take the maximum similarity value across the text and visual features because if two classes have high text feature similarity but low visual similarity, misclassification can still occur, and vice versa. By selecting the maximum value, we ensure that our method accounts for class pairs where concept confusion is most likely to happen.\n\n[3] Learning Transferable Visual Models From Natural Language Supervision. In Proc. of ICML, 2021.  \n[4] Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?. In Proc. of ICLR, 2024.\n\n---\n\nWe will incorporate all of the above experimental results and analyses in the next version.", "claims_and_evidence": "I believe the imbalance of pseudo-labels is a crucial challenge in adapting VLMs to downstream tasks using pseudo-labeled data. The proposed methods offer a reasonable solution to this issue, and their effectiveness is well-supported by appropriate experiments. The papers are well written to follow the core concept and motivation\n\n- However, the ablation studies could be more comprehensive. Given the complexity of the method, additional components should be ablated, such as the choice of gamma, the use of independent adapters, and, if feasible, the selection of prompt tuning. Additionally, some abbreviations (e.g., CA, CACM) are not properly explained, which could enhance clarity.\n\n- While the proposed method generally outperforms baselines, certain metrics, particularly on CUB TRZSL, show significant underperformance. Analyzing the root cause of this discrepancy would be needed.\n\n- I wonder about the rationale behind (2). There can be various similarity calculation strategies.", "methods_and_evaluation_criteria": "Methods and evaluation criteria seem reasonable. As far as I know, the evaluation criteria follow the previous works (common benchmarks).", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The effectiveness of the proposed methods is well validated through various experiments (Figures 4, 6, 7, 8, and 9). Additionally, experiments conducted with different data scales and backbone sizes demonstrate the consistency of the methods across varying settings.", "relation_to_broader_scientific_literature": "Considering the frequent use of pseudo labels in this literature, the proposed idea is worth to share", "essential_references_not_discussed": "To the best of my knowledge, there are no missing essential references, though I am not deeply familiar with this specific literature.", "other_strengths_and_weaknesses": "Described above", "other_comments_or_suggestions": "Described above", "questions_for_authors": "Described above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IfAWwnIApu", "reviewer": "Reviewer_1DgE", "summary": "This paper proposes a novel framework, CAP (Concept-Adaptive Pseudolabeling), to address the problem of imbalanced pseudolabels when fine-tuning Vision-Language Models (VLMs) like CLIP for downstream tasks using unlabeled data. The authors identify two key causes of imbalance: concept mismatch (where text features of a class are misaligned with image features) and concept confusion (where similar classes are hard to distinguish). CAP tackles these issues with two main components: 1) a concept alignment strategy that iteratively detects and corrects concept-mismatched classes using LLMs to generate enhanced textual descriptions, and 2) a confusion-aware calibrated margin that encourages the model to make more distinguishable predictions between similar classes. The framework also employs independent adapters on the visual branch to learn from both highly reliable (concept-aligned) and dynamically generated pseudolabels. Extensive experiments across six image classification benchmarks and three learning paradigms (unsupervised, semi-supervised, and transductive zero-shot) demonstrate that CAP consistently improves performance, achieving a relative improvement of 6.29% over the state-of-the-art.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and helpful reviews! We address each of your concerns below.\n\n> **Q1:** A more formal or quantitative definition of concept mismatch and concept confusion\n\nGenerally, concept mismatch arises from a severe form of the semantic gap while concept confusion is more commonly observed between similar classes. We provide more formal definitions below:\n\n**Concept Mismatch:** A class $y$ is considered to exhibit **concept mismatch** if the predicted label $\\hat{y}$ for its visual prototype $\\boldsymbol{v}_y$ (the average of visual features for class $y$) is **incorrect**, i.e.,  \n\n$$\n\\hat{y} = \\arg\\max p(\\boldsymbol{v}_y) \\neq y\n$$\n\nwhere $p(\\boldsymbol{v}_y)$ is the predicted probability distribution over all classes for prototype $\\boldsymbol{v}_y$.  \n\n**Concept Confusion:** Given the confusion matrix **$\\mathbf{M}$** predicted by CLIP, two classes $i$ and $j$ are considered to exhibit **concept confusion** if the misclassification between them is significant, specifically:\n\n$$\n\\mathbf{M} _ {ij} + \\mathbf{M} _ {ji} > 0.25 \\times (N_i + N_j)\n$$\n\nwhere $N_i$ and $N_j$ are the total number of samples belonging to class $i$ and $j$, respectively.  \n\nBased on these definitions, we report the number of classes exhibiting concept mismatch and concept confusion in RESISC45 and DTD. \n\n*Table 1: Number of classes exhibiting concept mismatch and concept confusion.*\n| Dataset | Concept Mismatch | Concept Confusion |\n| --------|------------------|-------------------|\n| **RESISC45** | 6 | 17 |\n| **DTD** | 9 | 24 |\n\nWe will incorportate these definitions into the next version of our paper to improve clarity and precision.\n\n> **Q2:** Computational cost of CAP\n\nCAP, CPL, GRIP share two stages:  \n\n1. **Pseudolabeling stage** – Responsible for generating pseudo-labels, only needs to be run once per dataset.\n2. **Fine-tuning stage** – Where the model is trained using the generated pseudo-labels. \n\nAdditionally, CAP employ MismatchDetection algorithm to fix mismatched concepts.\n\nBelow, we compare the computational cost of these two stages of training on EuroSAT for CPL, GRIP and CAP, using an NVIDIA RTX 4090 GPU and a Intel(R) Xeon(R) Silver 4314 @2.40GHz CPU:\n\n*Table 2: Computation time for each stage of CPL, GRIP and CAP.*\n| Method | MismatchDetection Time | Pseudolabeling Time | Fine-tuning Time |\n|--------|-------------------|------------------|------------------|\n| **CPL** | - | 3min 32s | 74min |\n| **GRIP** | - | 3min 32s | 102min |\n| **CAP** | 47s | 3min 32s | 29min |\n\nIn can be observed that the mismatch detection algorithm of CAP takes about 47s. In fine-tuning, CPL requires more time as it takes an **iterative strategy requiring training to convergence multiple times** (10 iterations in the original implementation). In each iteration, CPL expand the pseudo-labeled dataset. GRIP shares a similar trend with CPL.\n\n*Table 3: Computation time for each iteration of CPL.*\n| Iteration of CPL | Training Time |\n|-----------|-------------------------|\n| **1st**  | 1 min |\n| **5th**  | 7 min |\n| **10th** | 13 min |\n| **Total** | 74 min |\n\n> **Q3:**  Discussion of the limitations\n\nOne limitation of our method (CAP) is that it relies on a mismatch detection algorithm when handling **concept mismatch**. Currently, our detection method is a **simple clustering-based algorithm**, which, despite its computational efficiency, has room for improvement in detection accuracy and robustness.\n\nIn future work, we plan to explore more **precise mismatch detection algorithms** to better identify  mismatched concepts, thereby further improving the robustness and accuracy of our method.\n\n---\n\nWe will incorporate all of the above experimental results and analyses in the next version.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "The experimental designs are rational, and the results are convincing.", "relation_to_broader_scientific_literature": "The paper builds upon a growing body of work on adapting VLMs to downstream tasks using pseudo-labels. It directly addresses the limitations of existing methods that suffer from imbalanced pseudo-labels due to confirmation bias.", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "Strength:\n\n1. The identification of concept mismatch and concept confusion as distinct sources of imbalance is insightful and well-supported by the analysis and visualizations .\n\n2. The confusion-aware calibrated margin provides a mechanism for improving local calibration and mitigating bias among confused classes.\n\n3. The paper is well-written, clearly explains the proposed approach, and provides sufficient details for implementation\n\nWeakness:\n\n1. While the paper provides a good qualitative explanation of 'concept mismatch' and 'concept confusion,' a more formal or quantitative definition of these concepts would be beneficial.\n\n2. While the paper demonstrates improved training time compared to CPL and GRIP, a more detailed breakdown of the computational cost of each component of CAP (e.g., concept alignment vs. fine-tuning) would be useful.\n\n3. A more thorough discussion of the limitations would benefit the paper", "other_comments_or_suggestions": "No", "questions_for_authors": "See strength and weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "odHIQ4FUnf", "reviewer": "Reviewer_YUBC", "summary": "This paper proposes a concept-adaptive pseudo-labeling framework to generate balanced pseudo-labels for fine-tuning vision-language models (VLMs) on downstream tasks.\n\nIn the first stage, the paper introduces concept alignment to address the issue of concept mismatch by assigning precise pseudo-labels to misclassified instances. In the fine-tuning stage, the paper proposes a confusion-aware calibrated margin, built upon logit adjustment, to mitigate concept confusion.\n\nExperiments on six benchmarks demonstrate the effectiveness of the proposed approach, with ablation studies highlighting the contribution of each component.\n\n## update after rebuttal\n\nThe authors have addressed my concerns, and I keep my initial positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and helpful comments! We address each of your concerns below.\n\n> **Q1:** The number of baselines is limited\n\nIn our paper, we primarily compare against state-of-the-art methods **specifically designed for leveraging CLIP’s zero-shot capabilities** through pseudolabel generation. The number of such methods is relatively small—for instance, CPL only compares with three baselines.\n\nTo further enrich our comparisons, we have incorporated **general semi-supervised learning methods** as additional baselines, and the results under SSL setting are presented in the table below. \n\n*Table 1. Comparison of different methods under SSL setting.*\n|   | RESISC45 | DTD | Flowers102 | EuroSAT |\n|----------|---------|---------|------------------|---------|\n| **FixMatch [1]** | 66.1 | 51.2 | 85.0 | 79.1 |\n| **FreeMatch [2]** | 76.6 | 53.5 | 87.7 | 90.0 |\n| **SoftMatch [3]** | 69.5 | 47.8 | 86.9 | 83.8 |\n| **CAP (Ours)** | **83.3** | **62.3** | **90.0** | **92.8** |\n\nThe results show that CAP consistently outperforms these baselines on all datasets, with a significant improvement of 6.7% on RESISC45 and 9.8% on DTD.\n\n[1] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. In Proc. of NeurIPS, 2020.  \n[2] FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning. In Proc. of ICLR, 2023.  \n[3] SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning. In Proc. of ICLR, 2023.\n\n\n> **Q2:** The scope of evaluation is somewhat narrow\n\nWe appreciate your suggestion to evaluate our method on more challenging tasks such as segmentation. However, our method is primarily designed to address the issue of pseudolabel imbalance caused by the image-level classification bias of VLMs, making it less directly applicable to segmentation tasks.\n\nA key requirement of our approach is the ability to extract image features for each class and use them for clustering or similarity computation at different stages of the algorithm. In segmentation, however, a single image **typically contains multiple classes**, prevents the extraction of clean, class-specific visual prototypes. This intrinsic difference in problem formulation currently limits the applicability of our method.\n\nIn fact, existing works in top machine learning conferences that address this problem, such as CPL [4], LaFTer [5], and FineSSL [6], have similarly concentrated on classification tasks. This is because classification serves as a fundamental benchmark for evaluating improvements in pseudolabeling and adaptation strategies for VLMs.\n\nWhile extending these techniques to segmentation or other structured prediction tasks is an interesting direction, it would require task-specific modifications beyond pseudo-labeling (e.g. in segmentation, one often needs to design a more advanced segmentation network). That being said, we fully acknowledge the importance of extending these techniques to structured prediction tasks like segmentation, and we will include this promising avenue in our future work discussion.\n\n[4] Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data. In Proc. of ICML, 2024.  \n[5] LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections. In Proc. of NeurIPS, 2023.  \n[6] Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning. In Proc. of ICML, 2024.\n\n---\n\nWe will incorporate all of the above experimental results and analyses in the next version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Overall, the technical contributions and evaluation are sound.\n\nThis paper focuses on fine-tuning VLMs for downstream tasks, specifically image classification, and conducts experiments on six benchmarks. The authors identify two key issues in generated pseudo-labels: concept mismatch and concept confusion. To address these issues, the paper proposes Concept Alignment (CA) and Confusion-Aware Calibrated Margin (CACM). The authors further support their approach with ablation studies under different datasets.", "theoretical_claims": "N/A—this paper does not include theoretical proofs.", "experimental_designs_or_analyses": "The proposed method is evaluated on six image classification benchmarks under SSL, UL, and TRZSL settings. Additional ablation studies on three datasets further demonstrate the effectiveness of the proposed components. Overall, the experimental setup and analysis are comprehensive and convincing.\n\nOne concern is the limited number (i.e., 4) of baselines for comparison. It would strengthen the paper if more baselines were included in the main results (i.e., Tab. 1).", "relation_to_broader_scientific_literature": "This paper addresses the issues of mismatch and bias in VLM-generated pseudolabels, contributing to broader research areas, particularly in VLM-based training and pseudo-labeling techniques. The proposed approach has the potential to improve the reliability of pseudo-labeling for various downstream tasks, extending its impact beyond image classification.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Pros:\n- Well-organized paper with clear motivation and strong writing.\n- The proposed methods are effective and demonstrate SOTA performance.\n\nCons:\n- As mentioned earlier, the number of compared baselines is limited—only four baselines are considered.\n- The scope of evaluation is somewhat narrow. It would be more compelling if the method were tested on more challenging tasks, such as segmentation, rather than just image classification.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7et6s5Nk6V", "reviewer": "Reviewer_xucA", "summary": "This paper proposes a novel approach to improving pseudolabels generated by vision-language models (VLMS). The authors identify two key errors which contribute to the degradation of pseudolabel quality: concept mismatch and concept confusion. Concept mismatch occurs when the class text name provides a description that is misaligned with the visual features. Concept confusion occurs when two (or more) classes contain significant overlap, and the text description fails to capture the most striking visual differences. To address these, the authors propose concept alignment, in which concept-mismatched classes are identified and their labels enhanced with an LLM, and the confusion-aware calibrated margin, in which a margin matrix is calculated per-class based on the inter-class similarity and class-wise detection confidence margin and integrated into the cross-entropy loss. The authors perform experiments in the unsupervised, semi-supervised, and transductive zero shot learning settings, and demonstrate improvement in fine-tuning across a wide variety of settings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and encouraging comments! We address each of your concerns below.\n\n> **Q1:** Ablations of hyperparameters are not included\n\nFollowing your suggestion, we have conducted ablations of several hyperparameters in CAP.\n\n**Ablation of $t$**\n\nIn CAP, we use $t$ to identify concept mismatched classes. Below, we report the results under UL setting with different values of $t$.\n\n*Table 1. Impact of threshold $t$.*\n|   | $t$ = $\\frac{C}{14}$ | $t$ = $\\frac{C}{12}$ | $t$ = $\\frac{C}{10}$ (Default) | $t$ = $\\frac{C}{8}$ | $t$ = $\\frac{C}{7}$ |\n|----------|---------|---------|------------------|---------|---------|\n| **RESISC45** | 81.5 | 81.5 | 81.5 | 81.5 | 81.7 |\n| **DTD** | 54.9 | 54.9 | 55.4 | 55.4 | 55.4 |\n\n$C$ is the number of classes in the dataset. The results show that CAP maintains consistent performance across different threshold values.\n\n**Ablation of $k$**\n\nIn CAP, we use $k$ to determine the number of pseudo labels generated for each class. Results of ablation of different values of $k$ under SSL setting are presented below.\n\n*Table 2. Impact of pseudolabel number $k$.*\n|   | $k$ = 12 | $k$ = 14 | $k$ = 16 (Default) | $k$ = 18 | $k$ = 20 |\n|----------|---------|---------|------------------|---------|---------|\n| **RESISC45** | 82.9 | 83.0 | 83.3 | 83.5 | 83.1 |\n| **DTD** | 61.5 | 61.1 | 61.3 | 61.1 | 60.6 |\n\nThe results demonstrate that CAP is generally robust to different choices of $k$.\n\n**Ablation of $\\tau$**\n\nIn CAP, we use $\\tau$ as the confidence threshold to dynamically generate pseudolabels. We conducted experiments on different values of $\\tau$ under UL setting. The results are as follows:\n\n*Table 3. Impact of confidence threshold $\\tau$.*\n|   | $\\tau$ = 0.80 | $\\tau$ = 0.82 | $\\tau$ = 0.85 (Default) | $\\tau$ = 0.87 | $\\tau$ = 0.90 |\n|----------|---------|---------|------------------|---------|---------|\n| **RESISC45** | 80.9 | 81.2 | 81.4 | 80.5 | 81.6 |\n| **DTD** | 55.9 | 56.3 | 57.1 | 56.4 | 56.7 |\n\nThe results confirm that our method is robust to changes in $\\tau$.\n\n> **Q2:** Confusing Figure 5\n\nFollowing your valuable suggestion, we have revised Figure 5 to provide clearer visualization of CAP's key components while simplifying its overall presentation. The new figure is available at:\n\nhttps://anonymous.4open.science/r/CAP-C642/framework_updated.pdf\n\nWe appreciate your insights and hope this revision improves clarity. We will incorporate all of the above experimental results and figures in the next version, and we welcome any additional feedback you may have!", "claims_and_evidence": "Yes, the authors claims are well supported empirically.", "methods_and_evaluation_criteria": "Yes, the benchmark datasets and various fine-tuning settings make sense for the task.", "theoretical_claims": "N/a", "experimental_designs_or_analyses": "Yes, the experimental design and analysis is valid.", "relation_to_broader_scientific_literature": "Previous works, such as CPL, have investigated prompt-tuning to improve pseudolabeling by VLMs [A]. This work expands on this in a principled way through investigating what causes the misalignment between text and visual features, and directly addressing those with improvements.\n\n[A] Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data. In Proc. of ICML.\nOpenReview.net, 2024b", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow\n- CAP achieves good performance in a wide variety of experimental settings, consistently improving the SOTA.\n- The method is well-motivated based on empirical observations, and detailed analysis (e.g. density of confidence scores in Fig 4) support the authors principle claims.\n\nWeaknesses:\n- CAP is largely heuristic and involves several differing hyperparameters, however detailed ablations of these are not included.", "other_comments_or_suggestions": "I find Figure 5 to be a little confusing; although the figure displays the entirety of the pseudolabel finetuning pipeline, the key components of CAP are not well highlighted. I think the figure should be simplified, and more emphasis should be placed on the components that are the authors contributions.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to investigate the underlying causes of the class imbalance issues when generating pseudolabels by VLMs. This paper identifies two primary contributing factors, including concept mismatch and concept confusion. To address the two issues, this paper proposes a novel framework incorporating concept alignment and confusion-aware calibrated margin mechanisms, which would mitigate class imbalance by enhancing underperforming classes and promoting balanced predictions across categories. Experimental results demonstrate that the proposed method effectively enhances the accuracy and balance of pseudolabels, outperforming SOTA methods.

This paper finally receives 1 accept and 3 weak accept, which means all the reviews are positive. Addressing class imbalance when generating pseudolabels by VLMs is a crucial challenge, and this paper identifies two key factors and proposes an effective method. The contribution of this paper meets the ICML standard.

I commend the authors adopt the reviewers' constructive suggestions and add the provided new experimental results into the final version.
Decision: Accept (poster)