Revisiting Semi-Supervised Learning in the Era of Foundation Models
Authors: Ping Zhang
Zheda Mai
Quang-Huy Nguyen
Wei-Lun Chao
Venue: NeurIPS 2025 poster ()
Keywords: semi-supervised learning
self-training
vision foundation models
foundation models
benchmark
parameter-efficient fine-tuning
Primary area: deep_learning

Abstract: Semi-supervised learning (SSL) enhances model performance by leveraging abundant unlabeled data alongside limited labeled data. As vision foundation models (VFMs) become central to modern vision applications, this paper revisits SSL in the context of these powerful pre-trained models. We conduct a systematic study on tasks where frozen VFMs underperform and reveal several key insights when fine-tuning them. First, parameter-efficient fine-tuning (PEFT) using only labeled data often surpasses traditional SSL methods---even without access to unlabeled data.  Second, pseudo-labels generated by PEFT models offer valuable supervisory signals for unlabeled data, and different PEFT techniques yield complementary pseudo-labels. These findings motivate a simple yet effective SSL baseline for the VFM era: \emph{ensemble pseudo-labeling across diverse PEFT methods and VFM backbones}.  Extensive experiments validate the effectiveness of this approach, offering actionable insights into SSL with VFMs and paving the way for more scalable and robust semi-supervised learning in the foundation model era.

Revisiting Semi-Supervised Learning in the Era of Foundation Models
Ping Zhang* Zheda Mai* Quang-Huy Nguyen Wei-Lun Chao
The Ohio State University
(zhang. 14217, mai 145, nguyen. 2959, chao. chao.209) Cosu edu
Abstract
Semi-supervised learning (SSL) enhances model performance by leveraging abun- dant unlabeled data alongside limited labeled data. As vision foundation models (VFMs) become central to modern vision applications, this paper revisits SSL in the context these powerful pre-trained models. We conduct a systematic study on tasks where frozen VFMs underperform and reveal several key insights when fine-tuning them. First, parameter-efficint fine-tuning (PEFT) using only labeled data often surpasses traditional SSL methods without access to unlabeled data. Second, pscudo-labels generated by PEFT models offer valuable supervisory signals for unlabeled data, and different PEFT techniques yield complementary pseudo-labels. These findings motivate simple yet effective SSL baseline for the VFM era: ensemble pseudo-labeling across diverse PEFT methods and VFM back- bones. Extensive experiments validate the effectiveness of this approach, offering actionable insights into SSL with VFMs and paving the way for more scalable and robust semi-supervised learning in the foundation model era.
Introduction
The quality of machine learning (ML) models often closely tied the amount of labeled data available, but annotation can be costly or labor-intensive. Semi-supervised learning (SSL), which leverages abundant unlabeled data alongside limited labeled data, has thus emerged
promising paradigm for developing ML models without the need for extensive labeling [82 I0 67 74 Over the past few decades, numerous SSL algorithms have been developed advance this field. Exemplar methods from the deep learning era include Mean Teacher [61], MixMatch [6], and FixMatch (59),
which dynamically impose objective functions on unlabeled data based on distillation and consistency, thereby enhancing learning performance. It is worth noting that many of these methods were originally designed to train neural networks "from scratch."
Recently, pre-training on external labeled or unlabeled data has become the de facto standard across many machine learning application domains [29] 46). For example, computer vision, many recent algorithms are built upon vision foundation models (VFMs) such as CLIP (55) and DINOv2 [533] These models were pre-trained on massive datasets-millionss
not billions, of data points. As result, they have demonstrated remarkable generalizability across a wide range of tasks, often requiring only slight fine-tuning or, some cases, functioning effectively as frozen backbone 779
Given the promising advancements in both fields, we explore their interaction this paper. Specifi- cally, we seek address the following questions: Are existing SSL algorithms still effective when using VFMs as the backbone? What adjustments, any, are needed to improve their performance? Finally, can we leverage the power of VFMs to develop more effective, yet simpler, SSL algorithms?
'Equal contribution.
"Corresponding author: zhang, 142170osu edu. Our code is available at pattps:///gt //tthub. cota//90l.MM//SSt-oouaaaaiiioiot
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CLIP
DINOv2
79
96
41
46
60
58
103
95
Accuracy
188
219
21
36
39
66
34
PET LoRA AdaptFormer
VAPET
W..ET
CLIP (LoRA)
DINOv2
Labeled-On Only
CLIP (AdaptFormer)
DINOv2 (AdaptFormer)
Figure 1: Left: The Venn diagram of top 20% highest-confidence predictions various VFMs and PEFT under DTD 3-shot, illustrating their intrinsic property of producing a diverse range of high-confidence predictions. Right: Ensembling these diverse pseudo-label predictions progressively boosts downstream performance for increasingly ensembles (Self-Training PET V-PET), highlighting the quality improvements from diversity. Results are averaged across 12 settings benchmark.
Study design. To this end, we introduce new SSL benchmark datasets based on the Visual Task Adaptation Benchmark (VTAB) (76), diverse suite of classification tasks designed to evaluate visual representations. Our focus is on tasks where frozen VFMs underperform, indicating the need for further fine-tuning, and where SSL could offer beneficial solution. We then systematically evaluate three representative SSL Hyperparameters carefully selected using techniques proposed for unsupervised domain adaptation 23 ensuring that the data leakage issue discussed in [52] is avoided.
Seailodd--ixMMtaaa [59], Flex (77] and SoftMatch [TT.
Key insights. Our empirical results highlight two main findings. First, fine-tuning VFMs with representative SSL algorithms offers limited advantages over using only labeled data for fine-tuning Second, parameter-efficient fine-tuning (PEFT) [47 72 60)-hhich which updates small subset of pa- rameters or adds lightweight Icarnable modules while keeping the VFM largely frozen bozzen-consistentty yields substantial performance gains, regardless of the learning paradigm.
These observations imply two important directions. First, there a need for SSL methods tailored specifically for VFMs. Second, since PEFT models trained only on labeled data already match the performance of standard SSL approaches, effectively leveraging their predictions pseudo- a promising path to further improve performance in semi-supervised settings.
An SSL baseline in the VFM Building on these insights, we introduce simple yet effective SSL approach that leverages VFMs as backbones. Our method is grounded in the principle of self- training-2 straightforward SSL strategy in which the model generates pseudo-labels for unlabeled data to guide further training 71I 13 73]. While traditional self- self-training often struggles with low-quality pseudo-labels we address this by exploiting two key properties VFMs and PEFT methods: their strong initial performance and complementary behaviors.
Specifically, as observed in 63) different VFM backbones and PEFT methods frequently produce diverse predictions-evet when their overall accuracies are similar. As shown in Figure 1 this diversity motivates ensembling predictions from multiple VFM-PEFT pairs [8]. By explicitly compensating for their varied confidence distributions, we obtain significantly more robust pseudo- labels-withen requiring explicit filtering [59 51). The result simpler yet more reliable self-
training pipeline (Figure 2) that effectively leverages both labeled and unlabeled data, achieving substantial gains over existing SSL methods.
We extensively validate our approach, VFM-PEFT Ensemble Training (V-PET), on newly proposed benchmark datasets. V-PET consistently outperforms existing SSL methods across most tasks, including the recently proposed FineSSL [25), which also builds on VFMs. These results establish PET as simple, effective, and competitive SSL baseline for the foundation model era.
I pseudo-labeling 41 and ensemble methods (00 have been extensively studied in the ML literature, spanning both the pre-deep learning and deep learning eras. Our goal is
Remark. Self-training

---

Unlabeled
One-hot Pseudo-Labels
(c) Pseudo-Lbbel Eusemble
Paeudo-Labeled Data
CLIP
CLIP
CLIF
CLIP
LoRA
Adagat Poormer
LoRA
Aduul
DINO
DINOv2
DINO-2
DUNOWE
Adlags Pormar
Adayi
TORA
Supervised Parameter Efficient Fine-Tuning
(b) Pscudo-Label
(d) Self-Training
Gemeration
Figure 2: Illustration V-PET. To effectively leverage abundant unlabeled data alongside scarce labeled data in the era of VFMs, our approach follows four phases: (a) Supervised Parumeter- Fine- Tne-Tuning, where we hamess labeled data by fine-tuning pre-trained VFMs using various PEFT algorithms; (b) Pseudo-Labes Generation, where exploit fine-tuned VFMs' generalization ability to generate pscudo-labels for unlabeled data; (c) Pseudo-Late Label Ensemble, where we enhance robustness by aggregating pscudo-labeks labuls from multiple fine-tuned VFMs; and (d) Self-Training where we consolidate knowledge into one model.
compete with existing methods but to establish a simple yet effective semi-supervised learning baseline that harnesses their strengths while incorporating the unique properties of foundation models. We note that key step ensembling obtaining multiple base learners that are diverse and equally performant. Our contribution lies in leveraging the complementary behaviors of multiple foundation models and PEFT methods- an approach tailored specifically for the foundation model era.
Consistent with representative SSL studies 6 59 77], our primary focus is on classification. Nonethe- less, we believe that our insights are transferable to various tasks and hope they will inspire future research on other downstream SSL tasks, such as segmentation and detection.
2
Related Works
Semi-Supervised Learning (SSL), In recent years, many SSL methods have centered on generating and selecting reliable pseudo-labels [59 5] FixMatch [59] uses fixed confidence threshold, while FlexMatch [77) adopts class-specific thresholds and SoftMatch [ applies soft threshold to balance label quality and quantity. However, these approaches were designed for training from scratch, leaving open questions about their compatibility with VFMs. Another key paradigm self-training [1], which (1) trains teacher on labeled data, (2) generates pseudo-labels for unlabeled samples, and (3) trains a student on both. Ensuring pseudo-label reliability typically involves unsupervised pre- training [13,, confidence thresholds, consistency constraints [59][]] [1]. We instead leverage VFMs to produce more robust pscudo-labels, questioning whether scratch-oriented SSL algorithms remain effective in the ionddatiio-moda era.
Transfer Learning & Self-Supervised Learning. Transfer learning 83 65
and specifically PEFT (or PETL) for efficiently adapting foundation models (31 -has long leveraged pretrained models to boost downstream tasks. In our approach, we fine-tune VFMs via PEFT on labeled data, then use the adapted models to generate pscudo-labels for unlabeled examples, thus bridging transfer learning with semi-supervised methods. A related paradigm, self-supervised learning, also exploits unlabeled data and often serves as a baseline in SSL comparisons [13] 2 but assumes abundant unlabeled
data assumption that may fail in low-resource settings.
Vision Foundation Models. Vision Transformers (ViT) 21 pre-trained massive amounts of data,
have become indispensable to modern development vision foundation models (VFMs), have demonstrated superior performance wide range of
28 These models, often referred to

---

tasks. For example, CLIP-ViT 55 trained with millions of image-text pairs shows an unprecedented zero-shot capability, robustness distribution shifts, and serves the encoders for various powerful generative models [57 42 Meanwhile, DINOv2 5
pre-trained with self-supervised objectives on extensive sets of well-curated images effectively captures fine-grained localization features. Moreover, it is increasingly becoming common sense that strategically fusing multiple VFMs can unlock synergistic gains [47 63], leading to state-of-the-ar performance across tasks from visual question answering object detection.
Leveraging VFMs for Semi-Supervised Learning, With the growing prominence of VFMs, few recent studies have explored their usage SSL.. The most relevant to ours is FineSSI. 25 which specifically investigates the use of CLIP vision backbone in an SSL. setting, accompanied by pseudo- labels refinement using a balanced margin softmax. The evaluation was limited to simple, small-scale datasets like CIFAR-16 10 [40]. where frozen VFMs can already achieve high accuracy. A few other works constrained the study only to VFMs with zero-shot capabilities, such as the full CLIP model with both vision and text encoders [50 78 Our work, in contrast, extends the scope in both the diversity VFMs and the evaluation benchmarks, establishing the first comprehensive study of SSL in the era of foundation models.
Table I: Comparison supervised linear probing performance (%) between popular SSL datasets (CIFAR, Food101) and our benchmark, where denotes the number of labeled samples per class. While frozen VFMs already excel in standard SSL datasets, they struggle in our benchmark comprising diverse tasks, domains, and sizes unde--nndrsc the potential of SSL. for unleashing the full potential of VFMs.
CIFAR-10
CIFAR-100
FOOD-101
DTD
SUN397
RESISC45
Retinoputhy 80
CLEVR-C
KITTI
N
100
10
2
20
10
CLIP
85.0
78.3
61.8
63.7
69.3
33.1
51.1
DINOV2
91.7
S8.1
83.1
66.7
65.3
$2.4
41.2
30.5
51.3
On Evaluation of SSL in the Era of VFMs
3.1 Problem Definition
We consider a C-class classification problem in SSL with a large unlabeled dataset 24 == {x²") 1= and much smaller labeled dataset L : (a/ g/) where a! and æx training samples and of is the ground-truth class label. [24] and | denote the unlabeled and labeled dataset size with [4]] >> [C]. SSL aims to learn a model fo parameterized by using U/ and L. Unlike conventional SSL that initializes randomly, our framework starts with VFM (e.g.. CLII or DINOv2) and
fine-tunes it for downstream tasks.
3.2 A Comprehensive SSL Image Classification Benchmark
continue to evaluate on classic datasets such as 1IFAR-10/100 40). STL-I0 [17], and Food101 g However, these benchmarks exhibit two key limitations: Diminishing difficulty under VFMs. Following prior PEFT-on- VFM work 37 66) 47 70 80). we use linear probing as our primary performance indicator. As shown in Table linear probing on frozen VFM backbones already delivers remarkable accuracy- even with limited labeled data. We also report results for various classification-heat sizes in appendix [.4 and Narrow domain coverage. Because these benchmarks focus mainly natural images, they offer only a narrow view of real-world VFM applications. To effectively evaluate SSL methods in the VFM era- focusinn exclusively semi-superviied image classification- propose benchmark designed to capture the complexities of real-world applications across different domains and dataset sizes.
Despite recent advancements in SSL, most studies 225
Dataset & Regime Following the VTAB protocol 76 we select six classification datasets covering the three VTAB categories Natural, Specialized and Structured Specifically, we choose two datasets from each category: DTD and SUN397 from Natural RESISC45 and Retinopathy from Specialized and CLEVR-C and KITTI from Structured These datasets span diverse domains, including texture recognition, scene understanding, remote sensing, medical imaging, synthetic reasoning, and autonomous driving. To evaluate the robustness of SSL

---

methods, we vary the number of labeled samples per class and adopt linear probing the evaluation protocol, with shot counts chosen to keep each task sufficiently challenging for frozen VFMs. This configuration imposes substantial difficulty on frozen representations (Table 1}, underscoring the necessity of SSL for unlocking their full potential. A summary of the datasets appears in (Table and detailed descriptions are provided in Appenddxx A] Together, this benchmark offers a more diverse and comprehensive boundatonnmoddll-ee evaluation of semi-supervised image classification.
3.3 Fair Hyperparameter Tuning for SSL.
Hyperparameter tuning has long been persistent challenge in SSL [52 remaining ambiguous and unstandardized throughout much of the existing literature. Due to the labeled data scarcity, a standard train-validation split is often infeasible. Tuning a held-out labeled validation even test set can lead to significant data leakage, resulting in over-optimistic results and unfair comparisons. To tackle this issue and complement our benchmark, we establish more rigorous protocol for hyprraraaete tuning in SSL.
Our core insight is to harness the defin- ing characteristic of SSI-ann abun- dance of unlabeled training data to tune parameters in an unsupervised manner and avoid the pitfalls of data leakage. One recent study has ex- plored such an idea for domain adapta- tion, using unsupervised criteria such as RankMe [26] and AMI [68] to es- timate the effectiveness of each hyper- parameter configuration [23
Table 2: Summary of benchmark. Rem. Sen.: Remote sensing; Recog.: Recognition; [|: # labeled training data; M] unlabeled training data.
Dataset
Task
Domain
+ [4]
Classes
Recog
SUN397
Recoy.
Natural
RESISC45
Recog.
Rem, Sen.
20,160
45
Retinopathy
Recog.
Medical
36,825
4,8
CLEVR-C
Count
Synthetic
56,000
8
1.2
KITTI
Depth
Auto Drive
5,416
4
5,
also revealed that no single criterion could reliably select suitable hyperparameters across all scenarios. Motivated by this finding, we propose integrating seven unsupervised criteria-five derived from features, AMI 168]
However, 23
ARI
30 V-Measure (58),
FMI (24) and BNM [18), and two from logits, RankMe (26 and CHI 9 for more robust tuning. Concretely, for each hyperparameter configuration and its corresponding model, we compute all seven {=))]]]
criteria using held-out unlabeled validation set == (x:] 1 Next, we rank every hyperparameter configuration based on each criterion and choose the one achieving the lowest average rank across all. We provide formal definitions and detailed procedures in appendix C.T and the effectiveness of the proposed method in section 6.3 By eliminating reliance on held-out labeled validation sets, our method mitigates data leakage and promotes a more practical and fair tuning protocol for SSL.
4 Systematic Evaluation of SSL with VFMs
Given the struggling performance of frozen VFMs in our benchmark (Table we consider two straightforward strategies enhance their effectiveness: (1) exploit the inherent generalizability of VFMs by tne-ttning only on labeled data; (2) employ SSL to leverage both labeled and unlabeled data. With the diverse benchmark and hyperparameter tuning protocol introduced in section 3 we aim to investigate whether existing SSL algorithms remain effective when adopting VFMs as their backbone.
Evaluation Setup. We focus on two representative VFMS-ViT-BIA VIT-B/16 CLIP [55] and ViT-B/14 DINOv2 53 covering language-imann contrastive pre-training and self- supervied pre-training strategies, respectively. We examine four representative SSL methods, including FixMatch [59], FlexMatch 777). SoftMatch [II], and FineSSL [25] and use labeled-only fine-tuning as the baseline. We employ the Adam W optimizer 44 with a batch size of 32 and weight decay x 10-4 to fine-tune the model for 35 epochs. Learning rates and other hyperparameters are tuned with our proposed tuning protocol. The classification performance evaluated using the Top-1 accuracy on the test dataset.
5

---

Surprising Effectivenesss of Labeled-only Fine-tuning. Surprisingly, under fair comparison, fill fine-tuning with even few labeled images per class can match or surpass SSL methods, as illustrated in Figure 3) In other words, even with large amount of additional unlabeled data, SSL provides little advantage over fine-tuning VFMs with limited labeled data. This finding is reminiscent of observations made in comprehensive investigation conducted seven years ago [52].
Due to the inherently noisy supervised signals associated with unlabeled data in SSL methods, we hypothesize that allowing SSL to update all parameters in VFMs may inadvertently reduce their built-in generalizability.
Can PEFT Come to the Rescue? When labeled downstream data searce, recent research has shown that parameter-efficien fine-tuning (PEFT)-whiee updates only a small subset of parameters introduces a lightweight learnable module to frozen
outperforms full fine-tuning 666. When considering SSL scenarios with limited labeled data, PEFT compatible with SSL? Although a recent study has made initial attempt to apply PEFT in SSL 25 a comprehensive analysis of its compatibility with different VFM backbones, PEFT methods, and SSL approaches still lacking. We, therefore, assess two commonly used PEFT strategies: LoRA [32), which trains additive low-rank matrices for Transformer layers to approximate weight updates, and AdaptFormer [12]. an adapter-bad based approach proven effective in computer vision. We follow the same protocol outlined in section to tune the hyperparameters of these PEFT methods.
on VFMs 47
72 32
35
natural question arises:
Our results in Figure show that PEFT indeed improves SSL on VFMs. However, it also enhances labeled-only fine-tuning, with results remaining comparable to SSL. This highlights the limited effectiveness of using unlabeled data in existing SSL methods when paired with VFMs.
Discussion. Three key takeaways emerge from these findings. First, well-tuned labeled-only PEFT serves competitive baseline for SSL with VFMs. Second, the ineffectual use of unlabeled data in existing SSL methods underscores the need for SSL approaches specifically designed for VFMs. Finally, given that labeled-only PEFT can achieve accuracy comparable SSL, its predictions unlabeled data generate pseudo-labels of sufficient quality. A promising direction is to leverage these pseudo-labels to further enhance model performance.
5 A Simple, Effective SSL Baseline for VFMs
To wrap up the discussion in sectioo 4 we propose a new semi-supervised learning method tailored to vision foundation models. Specifically, build on self-training [64 41
49 48 a conceptually
simple SSL baseline that uses pseudo-labels from unlabeled data as additional supervision to improve the model. In the following section, we first briefly review self-training.
5.1 Self-training
Let us denote the predicted posterior probability of class given x as p(c|æ; f(0)). The core idea of self-training is gradually assign pseudo-labels to unlabeled data L {xx] based on high-confidence predictions
[2]]
1 = arg max p (cl=";;
if max : So) 2 T,
and add these pseudo-labeled data to the labeled set L = (( 1/ for supervised learning. Here, 0 represents the weights of the current classifier, and the confidence threshold. If (a) high-confidence predictions are correct and (b) the updated model gradually increases its confidence in the remaining data, self-training can be as effective as fully supervised learning on the entire dataset with true labels.
In context, a VFM fine-tuned with PEFT using labeled dataset L serves as the current model to generate labeds for 44, which are then used to further fine-tune the VFM.
Challenges. Although self-training conceptually simple, choosing the confidence threshold difficult. A high- excludes wrongly labeled samples but leaves too few for learning; low 7 admits

---

errors that reinforce bias. Moreover, the optimal T shifts across iterations. Despite extensive study of pseudo-label selection 744,, most approaches remain heuristic or overly complex, undermining self-training's 's practicality and reliability.
Our objective. To leverage the high-quality pseudo-labels from label-only PEFT while circumventing the aforementioned challenges, we aim develop self-training- simple and effective in practice. Specifically, we seek to eliminate the need for complex pseudo-label selection by utilizing all available pseudo-labels (i.e.. setting r = 0). This also allows self- training to be completed in single round, thereby removing an additional hyperparameter.
SSL algorithm that is both
5.2 Ensembling of multiple PEFT and VFMs
A necessary step toward achieving our objective is to ensure pseudo-labels of even higher quality to prevent er-
Algorithm 1: V-PET (Figure 2
ror propagation. To this end, we em- ploy ensembling techniques [9], which combine the predictions from multiple models to enhance the over- all quality and reliability of pseudo- labels.
Input Labbledd dataset and unlabeled datasct
PEFT methods indexed by v E [1.
VFMs indexed by m E [1.
Initialized parameters of VFM on PEFT On.m
Output: Srpput:Opii Optimal parameter 0* // (a) Supervised Parameter Efficient Fine-Tuning
What to Ensemble? Unlike con- ventional methods that train multi- ple equally performant yet diverse base learners through bootstrapping, random initialization, or cyclic learn- ing rate schedules (33, our ap- proach exploits the unique proper- ties of PEFT [47] and VFMs [63] 62). Specifically, it has been ob- served that different PEFT methods, while achieving similar overall accu- racy on downstream tasks, generate diverse predictions for individual sam- ples. Similarly, VFMs trained on dif- ferent datasets with varying objective functions exhibit diverse capabilities, with no single model demonstrating clear overall dominance.
for n E [1, N1, m E M1] do Fine-tune On.m on C to obtain On.m
// (b) Pseudo-Label Generation for E N], E [1, MM] do
Compute one-hot pscudo-label set Pn.m
Pa,m one hot(arg max (u)) Yu Eu
// (c) Pseudo-Label Ensemble
Ensemble pseudo-labe label P = ( 1 where:
P. = NXM
Σ
Pa
// (a) Self Self-Training
Choose n E [I, m² E [1, M] and fine-tune On. on P to get 0*
How to Ensemble? Common strate- gies for ensembling different predictions are (a) Mean averaaing class logits, and (b) Mean Probabilities avvbaagn class probabilities, both assuming similar output scales (typical with identi- cal architectures and bootstrapped training). However, VFMs trained differently and fine-tuned via varied PEFT methods produce inconsistent scales (Figure 4), causing some models to dominate and weakening the ensemble.
To address this, we introduce a simple solution called Mean Labels. We first obtain predictions from each model, convert them into one-hot encoding to ensure uniformity, and average them to obtain soft pscudo-label. Finally, we generate the augmented dataset P using averaged soft pseudo-label and fine-tune PEFT-based VFM models. We present proposed pipeline in algorithm I
Time Efficiency of Ensembling, Among steps (a)-(d) in algorithm step (d) dominates the wall- clock time since the labele-labeled set is orders of magnitude larger than the labeled one. Although our method ensembles N M models, the overall time overhead remains marginal: step (a) is fast owing to the small labeled set, and steps (b) and (c) are computationally light. For instance, for V-PET, the estimated runtime is only about 1.16x that of other SSL baselines.
7

---

Experiment
conduct experiments to address four questions- (1) How does our method compare to existing SSL? (2) Does it scale with more PEFT' and VFMs? (3) How effective is the ensemble strategy? Following the setups in Sectiiiinn and section 4] we evaluate two variants: V-PET (ensembles over PEFT methods and VFMs) and PET (ensembles over PEFT methods on single VFM). We use ST (self-training without ensembling) as baseline and, in all cases, reinitialize from the original pretrained VFM before fine-tuning with pseudo-labels.
CLIP
DINOV2
SUN397
Clevr-C
60
1.4
Accuraty (98)
50
Average Entropy
1.2
40
0.8
Adapt-
LoRA
Adápt-
Labeled-Onlly
FisMatch
0.4
SoftMatch
CLIP LoRA
DINOv2 LoRA DINOv2 AdaptFormer
Figure 3: Average SSL accuracy with full fine-tuming PEFT across 12 settings shows that, with fair hyper- parameter tuning. fine-tuning limited labels out- perform SSL; PEFT boosts SSL yet matches labeled- only performance, indicating minimal unlabeled-datt benefit in current #FM-based SSL (see appendix B.3).
CLIP AdaptFormer
Figure 4: The average entropy of predicted proba- bility distributions from different iine-tuned VFMs, highlighting the entropy gap among pseudo-labels, in- dicating poor calibration.
6.1 Performance Comparison
We summarize the performance comparison between ST, PET, V-PET and existing SSL methods across all the 12 settings in Table 3 Overall, V-PET achieves superior performance compared with others, as shown in the Average column. To further analyze the results, FFiure 5) visualize the ranking frequency, where each matrix entry (i,j) indicates how often method i ranked j-th across 12 settings. We then compute the mean rank (number in the bracket) of each method and sort them accordingly. Although V-PET does not claim first place in every single setting, it secures the top rank most frequently, showcasing the effectiveness of pseudo-label ensemble and establishing it as simple yet powerful baseline the VFM era.
Digging deeper into the three ST-based methods, we observe a consistent performance boost as more diverse pseudo-labels are introduced for the ensemble, i.e., ST PET - V-PET, as shown in Figure This underscores the importance of diversity among pseudo-labele
6.2 Scalability of Ensembling
While most of our experiments explore ensembles of LoRA and AdaptFormer, we also examine how well our method scales when integrating additional pseudo-label sources. particular, we expand our evaluation across two dimensions Cimnensios--MMM... and PEFT methods to assess the broader applicability of our ensembling strategy.
VFMs Beyond ViT-B (CLIP, DINOv2), incorporate four more VFMs, for total of six, including four ViT-B (CLIP, DINOV2, OpenCLIP, ImangeNer21kk [1 56)) and two VIT-L (CLIP, DINOv2).
PEFT Following the same setup, we start with LoRA and AdaptFormer, then expand our selection with four additional PEFT: ConvPass 33], BitFit 775. VPT-Deep (36), and Fact-TT 38 These methods span a broad range PEFT approaches, including selective-based based, adapter-based, and prompt-based techniques.
train CLIP with LoRA on DTD N3, generate pseudo-labels, and ensemble groups of 1-6 labels. We compare (1) PEFT: ensemble of pseudo-labels from models using different PEFT methods on the same CLIPP backbone, and (2) VFMs: ensemble pseudo-labels from different VFMs all fine-tuned with LoRA. Results averaged over five runs with distinct random pseudo-label sets (see Figure 6).

---

DTD
SUN397
RESISC45
Retinopathy
CLEVR-C
KITTI
Average
40
80
20
5
10
CLIP
LoRA
Labeled Only
64.1
60.7
66.8
35.4
35.2
60.1
Fixmatch
56.8
68.6
62.2
72.8
47.6
77.4
32.5
39.1
32.2
42.6
59.8
52.6
33.7
Maxx
66.4
32.8
38.7
36.9
75.4
44.6
SoftMatch
59.8
68.0
66.3
70.9
78.7
83.7
42.1
42.9
29.8
71.2
42.9
60.5
59.7
FineSSL
56.3
35.0
30.4
38.1
53.7
ST
38.9
63.8
49.2
52.7
64.1
71.4
35.7
42.8
36.5
51.6
59.8
64.3
34.7
PET
63.4
68.2
65.6
70.8
67.5
78.6
36.4
42.4
38.3
54.6
60.3
65.7
59.3
V-PET
71.7
67.2
72.8
77.2
38.9
51.1
36.7
58.5
58.0
62.3
60.5
AdaptFormer
Labeled Only
61.5
65.0
61.1
68.2
61.0
72.8
34.4
39.0
37.8
48.6
57.8
59.1
55.6
Fixmatch
67.3
64.6
71.2
63.1
80.5
29.0
42.4
33.1
44.8
30.0
57.2
FlexMatch
60.3
68.8
64.3
70.7
58.1
78.1
31.0
35.9
38.5
52.2
34,2
54.2
53.9
SoftMatch
69.3
65.3
70,2
65.8
82.7
38.2
42.7
34.5
56.1
33.9
55.1
FineSSt.
61.9
68.1
58.4
64.9
73.9
84,2
25.3
29.8
26.0
36.9
42.2
47.1
51.6
ST
62.0
67.2
58.1
61.3
68.8
79.3
34.4
39.6
39.0
50.3
58.2
61.9
56.7
66.8
71.7
37.1
41.5
55.8
60.5
61.7
V-PET
65.7
71.8
67.8
73.2
67.9
78.4
38.9
51.2
37.0
59.0
58.2
63.3
61.0
DINOv2
LoRA
Labeled Only
61.0
68.4
59.7
66.8
48.6
63.1
40.7
48.0
35.9
61.7
46.7
71.7
56.0
Fixmatch
57.4
68.4
56.6
71.4
36.6
66.7
38.1
41.2
35.7
41.0
14.8
42.9
47.6
Flex.Match
52.8
65.4
55.7
67.7
33.3
72.6
36.6
35.8
37.1
66.9
26.2
33.6
48.6
SoftMatch
53.6
64.0
60.4
67.3
46.6
79.5
32.1
44.5
33.6
65.1
27.4
27.9
50.2
FineSLL
66.9
72.7
54.6
62.1
69.8
87.4
31.0
24.6
24.8
31.7
31.9
39.8
52.3
36.9
"45.4"
55.5
54.7"
72.4
42.6
49.6"
36.3
62..
33.2
777.8
552.5
PET
66.7
72.2
58.6
65.3
55.8
70.6
41.0
54.3
33,1
56.9
42.6
63,3
56.7
V-PET
74.1
67.1
73.2
66.0
77.0
39.2
52.5
36.8
59.3
58.2
63.7
61.2
AdaptFormer
Labeled Only
63.0
67.6
58.9
67.0
57.4
34.4
53.1
27.3
41.8
52.3
49.8
51.6
Fixmatch
59.5
70.4
57.9
69.7
26.4
62.3
38.7
45.6
32.7
38.9
19.8
50.4
47.7
FlexMatch
57.1
65.9
56.0
67.5
64.0
33.4
40.2
33.4
42.2
34.3
SoftMatch
58.7
66.2
60.0
66.7
53.1
78.1
33.5
32.2
32.9
45.0
24.5
30.9
48.5
FineSSL
71.2
56.0
63,3
69.7
17.7
24.3
25.8
30.0
31,8
62.3
67.0
59.0
67.1
54.1
66.2
34.9
56.0
27.4
43.1
43.9
48.7
52.5
PET
67.2
73.0
63.5
70,9
56.2
72.0
40.2
55.5
33.1
58.0
45.2
60.9
V-PET
68.2
74,3
67.7
73.4
66.3
78.0
39.4
52.5
36.8
60.0
59.2
61.7
61.5
Table 3: Performance (%) comparison of baselines, existing SSL approaches, and proposed methods on six diverse datasets (12 settings). The best result within each PEFT and VFM is highlighted in bold. We report the full results appendix B.2
Mean Label Consistently Outperforms. As also shown in Figure 6 we investigate three ensemble strategies: (1) Mean Label (our proposed); (2) Mean Logits; and (3) Mean Probabilities. We find that the Mean Label strategy is the most effective among the three. It consistently outperforms the other two strategies across different ensemble sources and ensemble sizes. This suggests that the Mean Label approach better captures the diversity of pseudo-labels and maximizes the information utilization from the ensemble sources.
Both PEFT and VFMs Help. Whether we ensemble the pseudo-labels generated across different PEFT or VFMs, the performance is consistently improved, indicating that both PEFT and VFMs contribute the diversity of pseudo-labels and improve the ensemble robustness.
Diminishing Return. The performance improvement diminishes as we increase the number of ensemble sources, suggesting the performance gain from ensemble is not linearly scalable. In order achieve balance between performance and computational cost, we recommend using around 2 to 4 in this case.
Impact of Pseudo-Label Quality, When applying V-PET, one might wonder: the performances of different VFMs vary significantly, can their pseudo-labels still be ensembled to boost downstream results? The answer is yes, we demonstrate the result in appendix B3.6]
6.3 The Effectiveness of Hyperparameter Tuning
We also investigated whether the proposed hyperparameter tuning strategy accurately gauges SSL methods' performance. To illustrate its advantages, we compare it with two baselines, both measured via the absolute difference from the oracle test performance. Specifically, the oracle test performance is defined the highest accuracy across the entire hypeparrameter search space; each method selects

---

(ear)
12.20
[4-0]]
(9.6
13.01
(837)
D.06
(5.77
[1.0]
(9.43]
Pank
Park
Rani
Bark
Figure 5: Ranking frequency across SSL methods by proposed benchmark. The number (4.j) indicates the frequency of method is ranked j-th across 12 settings. The number in brackets indicates the average rank, where the higher rank is better.
Hyperparameter tuning comparison: our method versus baselines (random or chosen by one criterion). The absolute errors (%) from oracle test accuracy (smaller the better) are reported averaged across 12 datasets, 5 SSL baselines, and VFMs trained on 2 PEFT. The reported values in the format: mean + standard deviation.
Table 4:
RankMe
AMI
ART
V-Measure
FMI
CHI
BNM
Random
Ours
7.918.6
2.8.5.0
2.814.9
2.8149
2.914.9
3.717.0
12.0112.1
8.9
2.6.14.3
hyperparameter configuration, and we record how far its resulting accuracy is from this optimum. By taking the absolute value of this difference, quantify how closely each tuning method approximates the best possible result.
Our baselines are: (1) Random expectation, which represents the average performance ob- tained by randomly choosing hyperparameters from the search space, and (2) Single criterion, which selects the best hyperparameter configu- ration using only one metric. Comparing with (1) offers insight into how well method leverages the search space relative to a purely random choice, while comparing with (2) illus- trates the advantage of integrating multiple eval- uation criteria. As shown in Table we mea- sure the average absolute error between each method's chosen accuracy and the oracle's 's best accuracy (i.e., the maximum accuracy achiev- able in the search space) across datasets, SSL baselines, 2 VFMs (CLIP and DINOv2), and PEFT methods (LoRA and AdaptFormer). Our method outperforms both baselines in most cases, suggesting that combining multiple met- rics produces more reliable and robust assess- ment of SSL methods. We hope future research adopts this approach to strengthen SSL evalua-
CLIP (PEFT)
CLIP (VFMs)
6254
Accuracc 1962
60%
58%
DINOV2
Accuracy 1%
67%
56%
Num
+ Mean Label
Mean Logits
+ Moan
Figure 6: Scaling analysis for ensembling more PEFT methods and VFMs. Starting from LoRA finc-tuning, performance improves with diverse pseudo-labels, but the gain diminishes with more ensemble sourees. Mean Label consistently outperforms Mean Logits and Mean Probabilities.
tions.
7 Conclusion
We conduct a comprehensive and systematic empirical study for SSL the era of VFMs, revealing that VFMs can still benefit from SSL but require specifically tailored approach. In particular, we propose simple yet highly effective self-training algorithm that ensembles pseudo-labels from multiple VFMs and parameter-efficient fine-tuning (PEFT) techniques. Overall, our study offers actionable insights into SSL with VFMs and lays the groundwork for more scalable and practical semi-supervised learning in the of foundation models.
10

---

Acknowledgment
This research is supported by grants from the National Science Foundation (ICICLE: OAC-2112606), We appreciate the generous support of computational resources from the Ohio Supercomputer Center (OSC).
References
[1] Massih-Reza Amini, Vasilii Feofanov, Loic Pauletto, Lies Hadjadj, Emilie Devijver, and Yury Maximov. Sclf-training: A survey, Neurocomputing, 616:128904 2025.
[2] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning, 2022.
[3] Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundation models defining new era in vision: a survey and outlook. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2025.
[4] Maria-Florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: Towards bridging theory and practice. In NIPS, 2005.
[5] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring, 2020.
[6] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning, 2019.
[7] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In COLT, 1998.
[8] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. components with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pages 446-461. Springer, 2014.
discriminative
[9] Tadeusz Caliński and Jerzy Harabasz. A dendrite method for cluster analysis. Communications in Statistics-theory and Methods, 3(1):1-27, 1974.
[10] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised learning. adaptive computation and machine learning series, 2006.
[11] Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-qualle trade-off in semi- supervised learning, 2023.
[12] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition, 2022.
[13] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi- sumi-supervised learners, 2020.
[14] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1855-188,, 2017.
[15) Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
[16] M. Cimpoi, S. Maji, Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.

---

[17] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsu- pervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, Fl., USA, 11 1--13 Apr 2011. PMLR.
[18] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear norm maximization under label insufficient situa- tions. In Proceedings of the IEEF/CVF conference on computer vision and puttern recognition, pages 3941-3950, 2020.
[19] T. G. Dietterich. Ensemble methods in machine learning. Lecture Notes in Computer Science, pages 1-15, 2000.
[20] Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. A survey on ensemble learning. Frontiers of Computer Science, 14:241-258, 2020.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[22] Emma Dugas, Jared, Jorge, and Will Cukierski. Diabetic retinopathy detection. https //kaggle.com
2015. Kaggle.
[23] Linus Ericsson, Da Li, and Timothy M. Hospedales. Better practices for domain adaptation, 2023.
[24] Edward B Fowlkes and Colin L Mallows. A method for comparing two hierarchical clustcrings. Journal of the American statistical association, 78(383):553-500 1983.
[25] Kai Gan and Tong Wei. Erasing the bias: Fine-tuning foundation models for semi-supervised learning, 2024.
[26] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme Assessing the downstream performance of pretrained self-supprvised representations by their rank, 2023.
[27] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[28] Jianyang Gu, Samuel Stevens, Elizabeth G Campolongo, Matthew J Thompson, Net Zhang, Jiaman Wu, Andrei Kopanev, Zheda Mai, Alexander E White, James Balhoff, et al. Bio- clip 2: Emergent properties from scaling hierarchical contrastive learning. arXiv preprint wrXiv:2505.238882 2025.
[29] Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[30] Maria Halkidi, Yannis Batistakis, and Michalis Vazirgiannis. Cluster validity methods: part i. ACM Sigmod Record, 31(2):40-45, 2002.
[31] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024.
[32] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[33] Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hoperoft, and Kilian Q. Wein- berger. Snapshot ensembles: Train 1, get m for free. In International Conference on Learning Representations, 2017.
[34] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth, 2016.
12

---

[35] Menglin Jia, Luming Tang, Bor- Ch-C Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[36] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning, 2022.
[37] Shibo Jie and Zhi-Hong Deng. 2022.
Convolutional bypasses are better vision transformer adapters,
[38] Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer, 2023.
[39] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016.
[40] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
[41] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896. Atlanta, 2013.
[42] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems, volume 36, 2024.
[43] Bly Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, 2017.
[44] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019.
[45] Zheda Mai, Arpita Chowdhury. Zihe Wang. Sooyoung Jeon, Lemeng Wang, Jiacheng Hou, and Wei-Lun Chao. Ava-bench: Atomic visual ability benchmark for vision foundation models. arXiv preprint (rXXv:25006000882. 2025.
[46] Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hao Tu, Hong- You Chen, Vardaan Pahuja, Tanya Berger- Wolf, Song Gao, Charles Stewart, Yu Su, et al. Fine-tuning is fine, if calibrated. Advances Neural Information Processing Systems, 37: 17:133088-136119 2024.
[47] Zheda Mai, Ping Zhang, Cheng-Hao Tu, Hong- You Chen, Quang-Huy Nguyen, Li Zhang, and Wei-Lun Chao. Lessons and insights from a unifying study of cerraeteerefffcieee fine-tuning (peft) in visual recognition. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14845-14857, 2025.
[48] David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In ACL, 2006.
[49] David McClosky, Eugene Charniak, and Mark Johnson. Reranking and self-training for parser adaptation. In ACL, 2006.
[50] Cristina Menghini, Andrew Delworth, and Stephen H. Bach. Enhancing clip with clip: Exploring pseudolabeling for limited-label prompt tuning, 2024.
[51] Khanh-Binh Nguyen and Joon-Sung Yang. Boosting semi-supervised learning by bridging high and low-confidence predictions, 2023.
[52] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Real- istic evaluation of deep semi-supervised learning algorithms. Advances in neural information processing systems, 31, 2018
[53] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Ro-Yao Huang, Shang- Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision, 2024.
13

---

[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[56] Tal Ridnik, Emanuel nen-Baruch,, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the masses, 2021.
[57] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[58] Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), pages 410-420, 2007.
[59] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence, 2020.
[60] Mengyang Sun, Yihao Wang, Tao Feng, Dan Zhang, Yifan Zhu, and Jie Tang. A stronger mixture of low-rank experts for fine-tuning foundation models, 2025.
[61] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, 2018
[62] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, Ziteng Wang. Rob Fergus, Yann LeCun, and Saining Xie. Cambrian- 1: A fully open, vision-centric exploration of multimodal 1lms, 2024.
[63] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9568-9578 2024.
[64] Isaac Triguero, Salvador Gareía, and Francisco Herrera. Self-labeled techniques for semi- supervised learning: taxonomy, software and empirical study. Knowledge and Information systems, 42(2):245-285 2015.
[65] Cheng-Hao Tu, Hong- You Chen, Zheda Mai, Jike Zhong, Vardaan Pahuja, Tanya Berger-Wolf, Wolf, Song Gao, Charles Stewart, Yu Su, and Wei-Lun Harry Chao. Holistic transfer: towards non-disruptive fine-tuning with partial target data, Advances in Neural Information Processing Systems, 36:29149-291111 2023.
[66] Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao. Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7725-7735, 2023.
[67] Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine learning, 109(2)373-440) 2020.
[68] Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for clus- terings comparison: a correction for chance necessary? In Proceedings of the 26th annual international conference machine learning, pages 1073-1080, 2009.
14

---

[69] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pages 3492, 2010.
[70] Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang, Yuxiang Wei, Jihun Hamm, and Min Xu. Visual instance-award prompt tuning. arXiv preprint arXiv:2507.07796. 2025.
[71] Qizhe Xie, Minh-Thnaaa Luong, Eduard Hovy, and Quoe V. Le. Self-training with noisy student improves imagenet classification, 2020.
[72] Yi Xin, Siqi Luo, Xuyang Liu, Yuntao Du, Haodi Zhou, Xinyu Cheng, Christina Luoluo Lee, Junlong Du, Haozhe Wang, MingCai Chen, et al. V-petl bench: A unified visual parameter- efficient transfer learning benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[73] I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification, 2019.
[74] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-supervised learning. IEEE Transactions Knowledge and Data Engineering, 35(9))8934-88)) 8954, 2022.
[75] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-basee masked language-models, 2022.
[76] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. large-scale study of representation learning with the visual task adaptation benchmark, 2020.
[77] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling, 2022.
[78] Jiahan Zhang, Qi Wei, Feng Liu, and Lei Feng. Candidate pseudolabel learning: Enhancing vasion-l models by prompt tuning with unlabeled data, 2024.
[79] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision models for vision tasks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[80] Yunbei Zhang, Akshay Mehra, and Jihun Hamm. Ot-vp: Optimal transport-guin guuded visual prompting for test-time adaptation. In 2025 IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1122-1132. IEEE, 2025.
[81] Zhi-Hua Zhou. Ensemble Methods: Foundations and Algorithms. Chapman & Hal/CCRC 1st edition, 2012.
[82] Xiaojin Zhu. Semi-supervised learning literature survey. Comput Sci, University of Wisconsin- Madison, 2, 07 2008
[83] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning, 2020.
15

---

The appendix is organized as follows:
appendix A
provides details about the datasets used in experiments.
appendix presents details about our experiments and additional experiments conducted.
appendix C
provides detailed description of the hyperparameter search algorithms we proposed and metrics considered for the experiments in the main paper.
A Dataset Details
We select 6 datasets with diverse ranges of applications for our experiments, including:
The Describable Texture Dataset (DTD) is designed for texture pattern recogni- tion. It includes a diverse collection of images from 47 distinct texture categories, offering rich and varied examples to evaluate texture feature extraction and classification methods under natural conditions.
DTD 16]
SUN397 (69] SUN397 is large-scale scene recognition benchmark that covers 397 categories of both indoor and outdoor scenes. It provides a comprehensive testbed for scene classification algorithms.
Resisc45 is a remote sensing image dataset specifically created for scene classification tasks. It comprises thousands of images across 45 categories, representing diverse landscapes such as urban, rural, and industrial areas. This dataset instrumen- tal in developing and benchmarking algorithms for geographic information systems and environmental monitoring.
Resisc45 14]
Diabetic-Retinopatty [22]: This dataset focuses on high-resolution retinal images used for detecting diabetic retinopathy. It includes images that span various stages of the disease, providing a robust resource for training and evaluating deep learning models aimed at early diagnosis and automated medical analysis in ophthalmology.
Derived from the CLEVR family of datasets, Clevr-Count is centered on visual reasoning, particularly object counting tasks. Using synthetically generated images, it challenges models to accurately count objects within complex scenes, thereby assessing their ability to understand spatial relationships and compositional structures in visual data.
Clevr-Count 33
KITTI-Dist 27 KITTI-Dist is based on the renowned KITTI benchmark and is tailored for depth estimation in vehicular contexts. It provides high-quality stereo images along with ground-truth depth maps, supporting research in autonomous driving, 3D reconstruction, and stereoscopic vision, and serves as a key resource for evaluating depth estimation algorithms.
The detailed statistics of the datasets, specifically the validation and test set sizes, are presented Table
Dataset
N=
Test Size
DTD
752
1,880
SUN397
17.401
21.750
Resiso4S
5,040
6,300
Retinopathy
9,207
42670
Clevr-C
14,000
15,000
KITTI
1,354
Table 5: Dataset statisties.
B Experiment Details
In this section, we provide additional details on the hyperparameters and computation settings used in our experiments.
16

---

B.1
Computation Details
Our experiments were conducted on a workstation equipped with eight NVIDIA RTX 6000 Ada GPUs, two AMD EPYC 9554 64-Core Processors, and 800GB of RAM. Additionally, utilized NVIDIA Tesla V100, NVIDIA Tesla A100, and NVIDIA RTX H100 GPUs for certain experiments. All experiments were implemented using PyTorch 54
B.2 SSL and Labeled-Only Baseline Details
optimizer with a cosine annealing learning rate scheduler [3] with warm-up period with 2.5% of total iterations. We use batch size of 32 + 32 for all experiments, where the first 32 corresponds to labeled data and the second 32 corresponds to unlabeled data. We list the hyperparameter search spaces for the SSL., Labeled-Only, and ST settings in Table 6) including the drop path rate (dpr) (34). training augmentation (train-aug), LoRA dimension (lora_ (lora_dim), adapter bottleneck size (adapter_ bottleneck), weight decay, momentum, learning rate (Ir), and number of epochs. For other unmentioned SSL algorithm related hyperparameters, we use the default values provided in the original papers.
For all the experiments, we employ Adam 44
Hyperparameter
SSL
Labeled-Only
ST
Upr aug
0
0.2)
0
[weak, strong]
loca, dian
4
[4, 16]
4
adapter bottleneck
[4,
weight_ decay
5e-4
5c-4
5c-4
momentum
0.9
Full PT: [le-4, le-5, [c-6]
Full PT: [Ie-4, le-5, [e-6]
Fall FT: [1e-4, le-5, [c-6] PEFT: (1c-3, le-5] Fall 35
PEFT: [1e:3, 1o-4, le-5]
PEFT: [1e=3, 1o-4, le-5]
Full FT: 35
Full FT: 60
epochs
PEFT: 30
PEFT: 50
PEFT: 30
Table 6: Hyperparameter search spaces for SSL, Labeled-Only, and ST settings.
As a completion of Table 3 we present the detailed results of the SSL bascline experiment with
different SSL methods, including full fine-tuning, in Table 7
B.3 PEFT Experiment Details
As completion Figure 3 we present the detailed results of the PEFT experiment with different SSL methods in Table We present the detailed results of the PEFT experiment with different SSL methods in Table
B.4
Linear Probing with Different Classification Heads
We provide results of frozen backbone with different classification heads on our benchmark in Table 9
B.5 Impact of Labeled-Only Initialization on SSL
Per discussion in section natural question is whether the performance of SSL methods can be improved by initializing from a model trained on labeled data using PEFT. To explore this, we conduct experiments on three datasets: DTD (N6), Diabetic Retinopathy (N80), and KITTI (N10). We first train DINOv2 models on labeled data using LoRA and then fine-tune them with SoftMatch. These datasets were chosen because their labeled-only performance exceeds that of SoftMatch large margin, as shown in Table
The results, summarized in Figure 10) suggest that even when starting from the labeled-only trained model, SSL. performance still falls short. some cases, it performs even worse than training from scratch.
17

---

DTDD
Sun397
Risised
Reiinguath
CleveC
KETTI
AVERAGR
40
30
10
20
10
CLIP
Full
Labeled Ouly
53.46%
62.52%
52.79%
59.499
52. 03.88%
65.51%
31.78%
32.69%
25.03%
29.62%
37.55%
65.54%
47.44%
Hocmarch
46.97%
69) 00.17%
65. 49%
60.97%
91. 79595
74 74.43%
24. 648
45.955 1.999
21 2075
31 1550
45.4%%
45. 1.15
43.00%
RloMaachh
$0,495
61 06.0055
64.41%
61.35%
41. 6758
70.05%
26. 8555
18. 1758
25.318
30.75%
44.87%
47.41%
45.18%
SoftMandh
$2.29%
62.55%
50,6915
60.70%
59. 835855
84.67%
32.91%
25.15%
19.55%
20005%
31.25%
57.52%
46, 3..34%5
FineSSL.
7,02%6
17.18%
1.4015
4.0015
7.75%
8,37%
39.64%
16.9%%
19.55%
26.05%
41.49%
44.595
19.51%
ST
54. 1..0%
61 A15
51.88%
59, M15
55 9915
69.002 097
34. 8999
92.9%
24 6.%
79 7696
14.4
40F6
67 579
PET
63.19%
68.57%
63.28%
65.335
68. 68.3216
79.735
39 19.46%
43.35%
37.35%
49.87%
60.34%
63.55%
58,556
VPET
66.33%S
70.64%
66.4216
72.23%
67. 00.49%
79.11
39 49.45%
55.01%
35.61%
54.85%
$7.24%
65.12%
50,3015
LoRA
Oady
57,87%
64.10%
59 .8.
72,0556
42.166
35.11%
50,90%
60.06%5
56.815
68.56%
62.17%6
72.789
17-566
77.45%
22 45%
39.99
32 20%
4261
19.77
52.00%
53,67%
MecMatch
160 1.0%
SuliMards
47
79
FineSLL
56.2515
74 14
25
38075
53
53.56
621635
77.
51.5%
65.59%
71.65%
72.83%
02
66.50%
38,85%
51.05%
36
62.31%
60.53%
AdaptFormer
Labeled Only
61.5 11.50%
64.99%
61.14%
23.
60,9015
72.87%
34 3913
99%
37
48.6%
57.1%
59.7%
59.55%
Fiomatch
61.22%
67.34%
64.63%
71.19
63.00 13.0658
80./59
28,
42.41%
33. 13.139
44.76%
29.96/%
$7.24%
S3.70%
ResMauch
60.27%
68.78%%
64.32%
73735
58,1398
75.085
31.023
35935
38.50%
52.11%
34.15%
54.15%
53.86%
SuflMarch
61.39%
69.11 31
65.39%
70.17 13.17%
65 905
87.80%
96 266
42465
14 487
56 08/
11407S
$5,159
S6. 6..5%
FinSSL
61.91%
68. 14%
54.1 176
64 869
71 13.05%
841 84175
25 2213
25.978
36877
42.19'
47.12
$1.55 555
ST
61.97%
67.236
38. 58.10%
61.39
35336
31 220
30 30.01%
50.3556
58336
61.58%
56 66.96
PETT
63.78%
66.795
71.71%
69.445
53. 14.148
41.51%
36 454
55.18%
50. 50.08%
61.585
55.77
V-PECT
71. 7656
67.81%
73.29%
67 9018
51.2%%
36 90/
59.15
98.39%
63.39%
51 14.0%
DINO<2
Full
Labeled Ouly
46.44%
61.76%
47.58%
53.535
5226
55.59
34.26%
41.54%
26.77%
26.95%
53.77%
62.31%
43.5155
Hicmarch
40.57%
69. 195
33.98%
64:715
$51 1100
20 26.552
20 205555
In 15-55
1682%
11 11.08
50-055
SoftMandh
FineSSI.
RexMaach
44.00%
27.025
23.78%
60.50 505
57.55%
49.40%
44.97%
48.7316
2.704
S5239
393000
-1.
23.
22.255
26
2518
-
250004
21177
15.55%
17 1414
-
25538-
15800
12718-
19.95
16-172
30:40
400005 100005
239336
ST
VEETT
PET
64.41%
66.97%
49.47%
09.54%
00.45%
73.78%
59.9415
65.5316
66.311
65575
71.48%
07.3345
79.25%
61. 20
34.398
36.55%
56550
52.95
065556
21 110
31 11.35%
35.95%
55055
16811
56.55%
55.95%
46.88%
$4.85
64.70%
56,115
50.5915
LoRA
Laby
61.01%
68.40%
99.71%
48 5..9%
63095
40 69.9
4R025
35 87%
61,73%
46.9%
71.736
68.40%
56.626
71,38
36 11.
41111%
35 65.
4099%
14.7%
42.9%
47.59%
MexMatch
SuliMadd
48%
6507%
PineSSL
72.06%
61.0%
66
30 19.99
24. 24.77%
31 11.93%
39
57.50 57.5
50.565
50477
45
36 SVR
40
14.10%
73.24%
77.00%
63.71%
AdaptFormer
Labelad-Oal Ouly
62.98%5
67.61%
58.87%
67.33%
47.00%
57.419
34.428
53.07%
27.35%
41.79%
$2.32%
49.79%
$1.64%
Foczatch
99.47%
70.37%
57.85%
69,689
26.41.5
62.50%
36.739
45.35%
32.65%
28.89%
19.35%
50.35%
47.67%
ResMarc
57.07%
63.20%
56.01%
07.33%
$1.29%5
64.02%
33.439
40.20%
33.35%
42.16%
34.32%
27.85%
47.766
SallMarch
S8.77%
66. 17%
90.97%
65.85%
51 (65%
78.064
11.47%
32.17%
12.918
45
24.7%%
30.969
48.46%
FisSSL
64.00%
71.17%
$3.965
63.349
63. 306
69.7
17 696
24.908
25.79'
30030
11.79%
45.01%
46.76%
ST
62.269%
67.024
56.00%
67.111
540994
67.
38 8776
55366
27.08
0..55
66.66%
55.366
PETT
67.18%
72. 13.9
67.47%
70.91%
26.21
724044
40 1998
150
5799
45.150
57.500 97.555
VIPET
68.24%
74.15%
67.65%
73.43%
66.326
77.98%
39.39%
$2.538
36.77%
59.97%
59.118
61.74%
61.45%
Table 7: Complete results of the SSL baseline experiment with different SSL methods.
Table 10: Performance of starting from labeled-only model then fine-tuning with SoftMatch. Results on DTD (N6), Retinopathy (N80), and KITTI (N10). LoRA dim=4 fixed.
Labeled-Only SoftMatch
Dataset
Labeled-Only
SoftMatch
DTD N6
68.4%
64.0%
63.5%
Retinopathy N80
44.9%
44.5%
43.9%
KITTI N10
54.0%
27.9%
50.1%
B.6 The Effectiveness of VFMs Ensemble
To assess the benefit of pseudo-label ensembling across VFMs, even when their individual accuracies differ significantly, we performed an ablation study on the CLIP-B model. We trained three variants on different pseudo-label sets: (1) labels generated by CLIP itself, (2) labels from ViT-B-IN211 a comparatively weaker VFM, and (3) the ensemble of both sources. Results in Table II show that, despite ViT-B-IN2IK's lower standalone accuracy, combining its labels with CLIP's improves overall performance.
C Hyperparameter Tuning Details
We provide the detailed hyperparameter tuning procedure in this section.
18

---

DTD
Sun399
Revisck5
Retinopally
ClevsC
KFTTI
N3
N6
N5
N6
N1
N2
N40
N99
N10
N20
N5
N10
Tull
53,86%6
02. 825
52. 7916
59. 0956
52. 9899
65.51%
11.78%
32,09%5
25 15.02%
29,6215
57.55%
65.59%
Labeled-Onll
AdkepFFomeer
LoRA
61.548
57.87%
64. 10%
54.90 95%
50.054
61. 14%
66. 79%
68.2 68.214
50. 00.9855
59 05.85%
72.87%
22.016
35 5358
535
42. 1015
58. 58,905
53 89824
55.21%
48.6315
50.89%
50.05%
57.815
64 64.28%
59.07%
A
6.256
1.70%
8.11%
5.02%
7.25%
6.9%
1.10%
7.885
11.47%
20. 1999
21.355
-1.87%
Full
40.5%
60.375
25,405
51.76%
74,455
24.64%
45.95%
21 21.700
21,05%
45.43%
45.15%
FisMatch
Adpptfommer
LoRA
56.81%
61.22%
07.54%
55.50%
02.125
616655
72 72:298
71.18%
11111
03 03/0555
43.50%%
$0,555 50.995
77.45%
23 300
32,455
42.455
39.09%
23.135
32.20%
44,70%
42.65%
59.775
25,95%
52. 22.60%S
37 37.28%
CLIP
A
12.05%
7.58%
17 17.92%
23.55
0.07%
5:20
10.991
22,6346
0150%
9.77%
Full
50.699
46 600
25 25858
35-3618
20755
44
47 7014
Flexhatch
AdapiFumcc
LoRA
60.2778
56.49%
66.
7 22-
58.828
75 75-059
52:155
54.185
54.196
9999
7.695
20344
2.58%
12:39
12:50%
43.031
-5.495
7.17%
Full
52.2946
62.55%
50,9%
19.521
20/005
51, 11.225
52%
SoftMatch
06.
712200
62,90%
AdaptFommer
2.17%
15.1 15.19
7.19%
Full
4015
19.531
41.49%
44. 4955
FineSSL.
MapptFommr
LRRA
63.14%
61.91%
55.51%
58.37%
55.85%
56.259
59,51%
64 19%
$4.179
15:22
10.03
11.590
53.73%
48 11.99
47.12%
Full
46 44%%
767
47.55 5856
56 8%%
5 S2756
56 999
1L 1335
41 S45
56444
76
51.87%
62 62.31%
Laby
AdaytFormer
LaRA
A
15.56/3
61.01%
02.98%
68. 409
6.25%
619
99.7149 71.7
58,879
11.719
66 79%
67 119
8.78
45 17.00
43. 43.55%
28 599
ST7115
61,009
1299
54
5.35%
48000
23.356
58.5543
$139
617755
41.705
-1366
45.00%
$2.535
49.79%
71 11.75%
11.55%
Full
69.57%
63. 19%
64.73%
30,99%
55.164
25,05
20,55%
16,751
168313
51.055
50.49%
FieMatch
LoRA
59.47%
57.39%
68. 60%
70.37%
57.85%
56.25
71 19.
62. 3M4
58735
53 111S
45555
11.10%
15,3645
17:00
40999
15.85
14.75%
50.35 50.35%
42 2..00%
DINO
A
6.8m
6.20%
14.25%
10: 10.54
11.45%
21.11
-13 1889
-3.87%
Full
44.14%5
60.90%
11975
24.72
63.758
23388
26.02%
17:148
15.99
40,373
30.24%
Flexbaach
MapptFmmer
LoRA
57.01%%
52.82%
65.377%
63.99%
50.01%
55,695%
671999
67 03.33%
23.32
51.225
726335
305-05
371455
40220%
23:3558
13.1
66135
42,165
26.166
54.32%
27.55
33.61%
A
10.90%
4.84%
10,89%
9. 9.36%
11.59%
4.58%
11.215
11:92%
18.10%
39.71%
10.15%
0.49%5
Full
27.1256
57.55%
48.73%
59-16/5
51.52%
77.41%
26,288
36.17%
16.54%
17.715
16.17%
46
SoftMatch
MapptFommr
LoRA
58.075
53.6258
64.045
59.97%
60.41%
66.05%
67.34%
26, 600%
53,00 33,00%
78,00 13.00%
79,54%
33,4%%
52-17%
14,158
32.91%
33-57%
45.00%
65.07%
27.43%
24,47%
20.1256
7.55%
11.47%
7.947
6.69
1,395
6.48%
2.166
16. 16.70%
$7.525
9.77%
-17.165
Full
23.76/6
20.25%
20.7598
45.44%
FineSSI.
AllapFFumne
LaRA
64.144%
54.62%
67.38%
30.99%
24.309
24.55%
25. 25.799
45.01%
80:0
43.73%
52.53%
46.13%
55,519
2,08%
3.25%
6.96%
Table 8: Performance comparison of various SSL methods on our proposed benchmark datasets. For each dataset, model, and SSL baseline, we report the arithmetic mean of test accuracy across different PEFT, subtracted by the test accuracy of full fine-tuning, A positive value indicates that PEFT outperform full fine-tuning in the given scenario. Overall, the results show that PEFT consistently outperform full fine-tuning, with very few exceptions. This demonstrates the effectiveness of PEFT in enhancing the performance of VFMs within SSL frameworks.
DTD
SUN397
Resise45
Retinopathy
Clevr-C 20
KITTI
N
6
6
2
80
10
CLIP-LINEAR
61.80%
63.70%
69.30%
35.90%
33.10%
51.10%
CLIP-MLP-90
63.09%
60.29%
67.89%
36.91%
38.11%
50.77%
CLIP-MLP-766
62.29%
62.52%
69.62%
39.67%
39.51%
48.10%
CLIP LoRA
64.10%
66.79%
72.05%
42.16%
50.80%
64.28%
DINOV2-LINEAR
66.70%
65.30%
52.40%
41.20%
30.50%
51.30%
DINOv2-MLP-99
66.97%
62.41%
50.30%
44.33%
36.37%
50.91%
DINOv2-MLP-768 67.98%
61.16%
52.06%
35.66%
35.21%
50.77%
DINOV2 LORA
68.40%
66.78%
63.08%
48.02%
61.72%
71.73%
Table 9: Linear probing results compared to LoRA under various scenarios, where LINEAR denote a single linear layer and MLP-x denotes an MLP with single hidden layer of size x. The results demonstrate that LoRA is a more effective approach for fine-tuning vision foundation models.
C.1 Hyperparameter Tuning Procedure
and models trained with them held-out unlabled validation set D = {(x=))) is use to calculate the 7 unsupervised criteria. Then we calculate the rank of each criterion for each hyperparameter set. We will choose the hyperparameter set with the lowest average rank (R) across all criteria.
Concretely, given set of hyperparameters (61,02,... .2, J.
f(0,0,) ((0.0)):((800)) J )..
19

---

Table 11: We use pseudo-labels from different source train CLIP-B model. Demonstrating when performing ensembling training, even weaker pseudo-labels will boost performance.
Model
Test Accuracy
CLIP
56.17%
ViT-B-IN2IK
25.27%
CLIP + ViT-B-IN2IK
58.62% %
Problem Formulation Formally, the objective to find a suitable function,
= g gog E ER' Rd E R + R
(model),
(feature extractor),
R + R
(linear projector).
where h serves feature extractor, and is linear projector. The aim is to use L and 24 to learn h and g that minimize the generalization error on the test set T.
For any dataset = æi; - regardless whether is labeled, we denote the output of h as features, the output of logits, and the argmax of the logits predictions.
ΦD (h(x)))
(features);
E = (o(h(z;)))
(logits).
= {arg max ((h(())))
(predictions).
Tuning without Labels The primary challenge in tuning hyperparameters within an unsupervised setting lies in effectively adjusting them without access to labels. Inspired by the approach for standardizing hyperparameter tuning in Unsupervised Domain Adaptation (UDA) (23]. we propose tuning the hyperparameters using oD.,, LD.,, and TD. Specifically, we define function v as validator:
= LD. TD. RR
who receives the features, logits, and predictions of the validation set D, and outputs a scalar and models trained with them as
value. When given set of hyperparameters {01, 00.,0..... ......,,... 0n). {Ja, "aa....,,,,hl We first extract the features, logits, and predictions of the validation set D,, using each model,
Φ0. = (M(a;))
la (o(h(z:)))
ne, = farg max g(h(r:))) e (1,2....,
Then, given set of validators {ul, ....... ), we evaluate the performance of each model on the validation D. using the validators. Specifically, given extracted features, logits, and predictions of the validation set D, for each model as:
{(do, fo, No, ). (0o,, loz; To₂)) ... .. (0o, lo,, To }), calculate the performance of each model using each validator as,
v1 vi lo,, To,): vz(de, loz, ToI):
Un (do,, La,, )
20

---

Suppose we have validators and à models. We denote the performance of each model on the validation set D using each validator as a matrix M:
U1 11000 10000,.00 to,, TT, U1 (09,, Lez, T82)
Un (0000 lo, To, )
Un (002 Cz T02
M
2n (00 ly.: TB,))
Un (co,, f Te,))
Where each row represents the performance of model under different validators. And each column represents the performance of different models under the same validator. Then across every column, we calculate the rank of each model based on the performance under the corresponding validator. We then get rank matrix R:
T11
F12
71h
121
122
72h
R
Lnl
Tn2
Tych
Here each row represents the rank of model under different validators. Then we can approximate the performance of each model by calculating the average rank across all validators. We denote the average rank of each model as a vector A:
h
j= r2j
A
Σ
j=1
Thj
To get the best performance model, we select the model with the lowest value in A. We put the validators and more detailed examples in appendix C.2
C.2
Validators
We deploy semi-supervised learning validators, including
RankMe Score 26 We adapt RankMe Score to compute the feature matrix rank of the pre-trained data over both source and target data domains.
Adjusted Mutual Information (AMI) 68
We adapt AMI compute the adjusted mutual informa- tion between predicted and cluster labels of the validation set.
Adjusted Rand Index (ARI) 30 ART is a widely used metric in cluster analysis by measuring the similarity between two clustering solutions. Given RT is the Rand Index between the true and predicted clustering, ARI is defined by:
RI Expected RI
ARI
max RR) Expected RI
V-Measure 58 Beside AMI, we also adapt V-Measure to measure the harmonic mean between homogeneity and completeness over the clustering labels and prediction.
Fowlkes-Mallows Index (FMI)
24 Measures the similarity between clusterings. FMI is defined
by:
TP
TP
FMI = TP+FF FP+FN
21

---

Harabbss Index (CHI) 19 CHI is an internal clustering measurement. Given n sets of data point {=] grouped into clusters {C} with centroids {c} and c the overall centroid, CHI is defined by:
Σ nille - /kk-1) -1)
CHI
Ei Li=1 Enso, || - cal2] (n k)
Batch Nuclear-norm Minimization (BNM)
An unsupervised domain adaptation method aims to generate predictions that favor both diversion and confidence; BNM maximizes the nuclear norm of the prediction matrix in batch and repurposes it a validation criterion.
18
C.3 Hyperparameter Tuning Results
We present complete hyperparameter tuning results in Table 12
D Limitations
Our study primarily focuses on the image classification setting, following representative SSL frame- works such FixMatch. While this scope allows for controlled and fair evaluation of SSL principles, may limit the direct applicability of our findings to other tasks (e.g., detection or segmentation) that often demand task task-specific architectures and training paradigms. Nevertheless, we believe the core insights from our analysis such as the behaviors of pseudo-labeling, ensembling, and label-efficiency dynamics- are conceptually transferable. We leave systematic exploration of these directions to future work.
22

---

DTD
Sand99
Rmbeet
Retinepalhy
Clarst
KITTI
CUP LaKA
Crade
Labelakceely
1355
53.99%
19.85%
17%
41
59.15%
36.77%
53
673115
HaaMach
09.00%
72.28%
57105%
37.39%
41.22%
20.99%
73
53.995
55.13%
So0Mac
59.99
66.16
70.976
78.70%
83 03.00
42 1.00%
43.506
56.30%
71 103
65.126
49
RonMMe
Labelsl-Odly
185
68.29%
57,046
55:33
59.759
50.77%
55.21%
4101%
54.216
Ramash
$1.513
205
51 51.
63.03%
15.35
02 5.575
33.01%
14.05%
57.30%
43.6%%
50 1T7
$3.60
PanMatch
207
12.00%
31.24%
31825
303411
27-08%
58.21%
64 13355
57.00%
16.9%
11.64%
21.05%
38
48.226
Rasden
Cabe
10005
35
41:225
1.%
20 0.000
20 67675
17.46G
23.75%
50 5381%
51.78G
FroMach
57.00%
5455 555
13.965
40
00.995
08
Sotthbacs
33300
28.123 12:22
N0.17
54.93
81 3,322
36 5L.25
41.12%
50.1%%
43 10.0
12.25
12-12%
20
38-12%
29.27%
32
Famaahh
55.21%
5655
11.744
41.5
7744%
52 42.5%
19099%
$2.31%
42 62619
59.776
FasMarc
975
71 155
71.25
1.24%
78 100
16.75
75 355
41
63 11.790
188
55 15.00%
51.555
58
AlaztPome:
Orade
Labelok-iily
61.45
67:50 17.508
61140
58,255
00.85%
7285
53.955
46.34%
31.01%
45 4552
57.11
Fameadh
61
03006%
45.37%
34.27%
73.0%
Sodihaaca
01.35%
31%
65.35%
90.17%
65.31%
56.10%
43.65%
56 56.099
47.12%
57.34%
01
7195
31 17%
76
11.2%
14
Pramach
16
11.35
41
Fashlach
59054
02.155 155
18. 15.34%
45.35%
50.00%
73.56
57.57%
35.30%
31.0%%
43450
14.55
54.55
So0Mbacs
47.35%
38
35.00%
55
15.15
61.11
13.05%
Ronden
11:41
17%
17 13:015
11.0%
13.73%
515
111155
36A15
76 555
16 1.19
15.05
Paamasch
57.05
11.7%
21023
41:05
11A15
Sonbiauc
57.51%
19.995
54.34%
13.1%%
11611
88 1817
12.0%5
FieSSLL
44.35%
54.56
55.156
66.8%
2500
31,0%
24.75
33 3358
68.57%
40416
Ozz
64
36
50,07
61.22%
345
64.89
63.68
91.04
33.56
41766
Paasktcch
00.77
TW5
54 5.305
10371
15.17%
11 11.005
15475
5331/
34 111
Sontitucs
70.25
19.95%
82675
X
41.8%
31,85
19 19.85
55.175
7195%
17%
Don01 LoRA
Oraee
Labebsl-Olly
99.55
080
50
53,406
61
71,71G
40%
11.38G
35.05%
1..15
556550
FesMasch
36.01
68 55%
55 65.
67.56
4506%
12.05%
56.565
41.06%
51.140
05 5555
26.37%
S.0Mac
53,22
65.50
00.16
67:345
$4.73
51 86%
36 64665
14.15
36.5%%
65 1100%
27.656
KunkMM
Labelad-Okly
46.08%
S00th
41.21%
41354
50,55
41.35%
11.55%
35
46
Pareasch
42.1%%
64 841
36.01
15/011
16.9%%
30 100
5A 1184
10.17%
25 535
Se0March
13.385
$3.306
45.0%
53
16,00
51.5%
51.05
RleSSt
47 188
25.55
51 186
15:55
57
24.55
33355
31 11.05
Randon
Cabeld-Odly
(0.50 595
156
63 01-155
4611%
23 1000
16
11.37
37.14%
FlaMatch
31.43%
01:11
50055
55.154
11.5
15.34%
525615
48 25634
24.10
Sedibaaca
30335%
01500
54794
4511 1511%
41.3%%
32144
47
25.25
Oux
66.360
in
51.55
61 1355
Ramash
17 W.
mm
km
11 11:383
64 6.00
10 115
11:16
15
00 0300
FieMacch
27%
55
11:24
36
35.51%
37.15%
09
11./1
Fin:SL
05.77%
54525
87 55
30.90
34.5%
24.7%
51.50
AlasPaom:: Orade
Labciok-hily
0399
51/2256
4505%
03 2079
55.15%
34:33%
45 215
52 22400
58.23%
FvaMech
17.0%
4..0
67.11
6100
11.75
51.44
47
SonMburs
12%
59 0m
11.0%
14 p.
11.7%
46 505
Fin:SL
71.17%
56.17
02.38%
19%
25
31.5%
25.79%
30
RaAMi
54.25
Famach
$1.65%
16.289
45.55%
55,77
23.56%
5 5
45.55%
26.99%
29 160
50.35%
FeaMtach
52.15%
62 1434
41944
51,546
45.9%%
03 100
27 27.50%
33.55%
25,20%
27 5555
21.55%
41.87%
122
53.15
56.10
15%
68 104
17.005
23.045
15.55
Ronden
Labvad-ciyyy
31
14.2% 2575
16.175
48.7%%
30.0%
24 12
28 1355
74 1/5
(2
Faneads
54.11%
65386
23.7%
56 60%
35.1%
23.4%
35 3597%
33.5%
T09
11.60
SunMaaca
52.54%
18.
52.556
51.2%
11655
37.17%
50754
40 100
34666
11816
Fin:SLL
57.77%
62.415
55.15%
$1.146
4001%
07 47
24 141
16326
25.2%
29 63
31.25
46656
Lanved--iely
59.87%
71.37%
5T.55
86.6%
26.1%
16
45.1%%
34
1940
Pasktach
17473
975
1. K.005
51.775
11.20%
6100
41.35
11.4%
42 165
34.105
31.5%
5.0Naacs
06
35
45
02.32%
24.3%%
25,79%
4500
Table 12: Complete hyperparameter tuning results for all datasets, models, and SSL baselines. For single validators we only include RankMe.
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We carefully write the abstract to accurately reflect our contribution and scope.
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretial results.

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the environmental details reproduce the experiments are disclosed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
25

---

Justification:
We will include code in supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ccipss.cc/pullcc/guiic gwwp/// Cowoeeobmisisiii..c. wwwbbbmmsssooPPoiooy.. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present the detailed experiments setting in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide error bars in main figure.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We present environment required for experiments in appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We discussed the positive societal impacts in the "Conclusion" section. To our knowledge, the current stage, our paper does not introduce any additional negative societal impacts compared to existing papers.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We will include the original license files in our released code if any references.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will have it well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
30


Reviews:
[{"review_id": "OL2ltT5zQ6", "reviewer": "Reviewer_DLdY", "summary": "The authors propose a new pipeline for semi-supervised learning in the regime of pretrained (vision) foundation models (VFMs). They demonstrate several existing SSL methods (which utilize both labelled and unlabelled data for the fine-tuning task) perform worse than a baseline of only using the labelled subset of the data to fine-tune the model on new tasks. The authors then propose their own method, which is more involved but does benefit from unlabelled data, that involves using multiple low-rank fine-tuning methods to fine-tune multiple VFMs to pseudo-label the unlabelled samples and create an ensembled training target.", "questions": "See above.", "limitations": "Not sufficiently addressed. The proposed method requires more compute than the competitors. The authors do recommend limiting the number of fine-tuned models in the ensemble to 2-4 (L329-330) due to diminishing returns, but do not point out that their approach requires more compute than the competitors since it requires fine-tuning N models on the labelled partition before performing the SSL step.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe work is well presented and addresses a timely problem. I believe the ideas and implementation are novel. For the most part, I am generally happy with the paper as submitted.\n\nI appreciate that the authors practice a robust experiment protocol with the partition used for hyperparameter optimization distinct from the partition for final model evaluation. I couldn't immediately see details about the partitioning availability so ask the authors to please ensure these splits are made publicly available for reproducibility.\n\n\n## Weaknesses\n\n**Datasets**: I understand the authors wanting to focus on transfer learning datasets where there is a challenge rather than ones which are trivial to solve given a pretrained VFM out-of-the-box. However, some caution should be taken as the datasets which the authors have selected are relatively far out-of-distribution from the training distribution of these VFMs. Previous work has investigated domain shift and categorized datasets as near- or far-OOD from ImageNet-1k, and the datasets which the authors select are generally on the far side of OOD for image datasets ([Guo, 2019](https://arxiv.org/abs/1912.07200); [Yang, 2022](https://arxiv.org/abs/2210.07242); [Lowe, 2024](https://arxiv.org/abs/2406.02465)). You can tell by eye looking at samples that describable textures (DTD), retinopathy eye scans, Clevr, and satellite imagery (remote sensing) are very different from the natural images that comprise the training datasets usually used for VFMs. For completeness, I think the work would benefit from adding a dataset which is near-OOD. However, being near-OOD does not necessarily mean the dataset isn't challenging - I recommend including a fine-grained classification dataset, which will still be very challenging for the model to learn accurately. There are many datasets used in the fine-grained image classification literature, examples include biodiversity datasets such as [iNaturalist-2021](https://github.com/visipedia/inat_comp/tree/master/2021) and [TreeOfLife-200M](https://huggingface.co/datasets/imageomics/TreeOfLife-200M).\n\n**Experiments**: Unless I missed it, the authors do not seem to explore the interaction between number of VFM models used and number of PEFT methods used in their label-ensembling experiments. It would be helpful to expand the experiments Figure 5 to show the cross between these two to make it clearer when to add more models and when to add more PEFT methods. If I already have used (VFM, PEFT) pair $(f, a)$, would I be better off adding $(g, a)$ a new VFM with an existing PEFT, or $(f, b)$ a new PEFT on an existing VFM, or $(g, b)$ a new PEFT on a new VFM, to the ensemble? Or does it not matter as long as at least one is new?\n\n**Take-away message**: The paper would benefit from giving a take-away message on which VFMs and PEFT methods to prioritize when building the ensemble for pseudo labelling, or a conclusion that it doesn't matter which models/methods are used as long as there is diversity in one/both of these.\n\n**Formatting**: Table 3 is very hard to parse as it is a very large array of numbers. I think this would be easier to read if there were cell background highlighting indicating the magnitude of the values for each column or showing the top K values in each column. This makes the table more like a heatmap so readers can determine trends in which values are large (good) and small (bad). The table would also be easier to read if the redundant % symbol were not shown in every cell. Good practice for table design recommends the units are shown in the first cell and not repeated if the same unit is used in the rest of the row/column. If the same unit is used throughout the table it can instead be described in the caption. Moving the % symbol from the cells to the caption will make the table less busy and hence more legible. Additionally, the font size can be increased with this change.", "was_revised": true, "final_justification": "The authors have addressed my concerns:\n- All datasets being far-OOD: I later realized that SUN397 is near-OOD. The authors also showed similar results on CUB-200, which is a fine-grained near-OOD dataset.\n- Combinations of increasing the number of PEFT methods and VFMs have now been explored sufficiently.\n- Computational requirements have been detailed more thoroughly. It transpires that the proposed method is actually less computationally demanding than (most? all? unclear) of the SSL baselines compared against, since the alternative methods rely additional multi-view forward passes of the unlabelled data throughout training. In comparison, the proposed PET method only needs a single additional pass of the unlabelled data through each of the PEFT foundation models before ensembled-target fine-tuning. This difference more than makes up for the additional overhead of the multi-VFM PEFT stage.\n- I anticipate formatting issues will be address in revision.", "rebuttal": "Thank you so much for your interest in our work and your valuable feedback. We would like to address all your concerns, point by point, as follows:\n\n**Concern 1: Dataset Selection**\n\nThank you for your valuable insight regarding our dataset selection, particularly the observation that our selections are relatively \"out-of-distribution (OOD)\". We would like to respectively clarify our rationale. We first emphasize that we do not deliberately choose OOD datasets. As our focus is on semi-supervised learning (SSL) in the downstream tasks, we focus on tasks where frozen VFMs underperform, indicating where SSL could offer a beneficial solution [lines 37 - 38]. In retrospect, these datasets turn out to be those that exhibit a larger distribution shift between the pertaining and fine-tuning. That said, with PEFL techniques, the performance gap due to domain shifts can be effectively reduced, suggesting that the pre-training is still beneficial for downstream tasks even under distribution shifts. \n\nNonetheless, we fully agree with your point of view that incorporating relatively \"in-distribution\" datasets would create a more representative benchmark. Due to the large scale of [INaturalist-2021](https://github.com/visipedia/inat_comp/tree/master/2021) and [TreeOfLife-200M](https://huggingface.co/datasets/imageomics/TreeOfLife-200M), and limited time during the rebuttal period, we opted to evaluate on [CUB-200](https://huggingface.co/datasets/cassiekang/cub200_dataset) — a smaller-scale, but good quality, and also widely adopted dataset for fine-grained natural image classification. We conducted experiments on it following the same setting as *Table 3* on 4-SHOT task. The results are attached in the table below, with the best performance in each column bolded:\n\n|              | CLIP LoRA  | CLIP AdaptFormer | DINOv2 LoRA | DINOv2 AdaptFormer |\n|--------------|------------|------------------|-------------|--------------------|\n| Labeled Only | 63.95     | 64.65           | 77.30      | 81.03             |\n| FixMatch     | 62.82    | 64.43           | 78.84      | 80.62             |\n| FlexMatch    | 69.62     | 69.21           | 78.05      | 79.06             |\n| SoftMatch    | 71.94     | 71.32           | 81.45      | **82.33**         |\n| FineSSL      | 73.47     | 69.66           | 82.07      | 81.48             |\n| **PET**      | 58.42     | 58.51           | 64.50      | 74.65             |\n| **V-PET**    | **74.61** | **73.68**       | **82.26**  | 82.26             |\n\nPreliminary results on CUB-200 demonstrate that our proposed method also performs well in the fine-grained classification setting. And we’ll leave further exploration to future works.\n\n**Concern 2: Experiment**\n\nWe appreciate your thoughtful suggestion regarding the interaction between VFMs and PEFT methods for ensemble methods. We agree that evaluating the interplay between these components is highly valuable and would strengthen our paper. \n\nIn response, we conducted the proposed ablation study by varying the number of VFMs (horizontally) and PEFT methods (vertically) from 1 to 6, following the same experimental setup as in Figure 5 of the main paper using DINOv2 and LoRA. The results are presented in an orthogonal grid of configurations, where for each setting, we report the average performance and standard deviation over three randomly sampled combinations.\n\n| N_PEFT \\ N_VFM | 1                 | 2                 | 3                 | 4                 | 5                 | 6                 |\n|----------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|\n| 1              | 37.96 ± 9.97    | 61.65 ± 0.79    | 63.99 ± 1.02    | 64.17 ± 0.59    | 63.40 ± 1.02    | 63.40 ± 0.19    |\n| 2              | 61.72 ± 1.30    | 62.68 ± 0.76    | 63.00 ± 0.58    | 62.80 ± 0.67    | 64.45 ± 0.25    | 63.92 ± 0.37    |\n| 3              | 63.23 ± 0.47    | 63.10 ± 0.70    | 63.28 ± 0.76    | 63.81 ± 0.81    | 65.27 ± 0.11    | 64.27 ± 0.46    |\n| 4              | 59.68 ± 3.55    | 63.05 ± 0.84    | 63.86 ± 1.01    | 63.94 ± 0.94    | 65.00 ± 0.16    | 65.12 ± 0.21    |\n| 5              | 61.33 ± 0.04    | 63.40 ± 1.02    | 63.44 ± 0.25    | 63.35 ± 0.87    | 64.11 ± 0.41    | 65.20 ± 0.45    |\n| 6              | 62.48 ± 0.52    | 61.84 ± 0.30    | 63.96 ± 1.24    | 64.22 ± 1.14    | 65.16 ± 0.26    | 65.25 ± 0.16    |\n\nThroughout the table, we can compute the Pearson correlation coefficient between the performance and the number of PEFT (or VFMs), as shown below:\n\n| n_peft | n_vfm |\n|---------------|----------------|\n| 0.272        | 0.446           |\n\nThis indicates that the number of VFMs is more related to the overall ensemble performance gain. However, the ensembling over PEFTs method is also useful and can serve as a supplement to VFMs, which is in accordance with the observations made by previous work [A].\n\n**Concern 3: Take-away message for VFMs vs. PEFT**\n\nThank you for the valuable suggestion. As shown in Figure 5 and our reply to **Concern 2**, the marginal gains from ensembling VFMs consistently exceed those from applying PEFT. Based on this observation, we recommend prioritizing VFMs over PEFT when resources are limited. However, when only a few (or one) VFMs are accessible, PEFT offers an effective alternative to improve performance. We will include this key takeaway message in the revised version of the paper.\n\n\n**Concern 4: Computation Time**\n\nThank you for your concern. We believe that the computational cost should not be a concern in our work. Like other ensembling-based methods, our approach introduces additional computational overhead. However, thanks to the efficiency of few-shot fine-tuning and pseudo-label generation, the additional cost is actually not severe. Thus, it keeps the overall cost relatively low compared to other ensemble methods.\n\nTo quantify this, we report the end-to-end runtime of PET and V-PET compared to standard baselines for DTD 3-shot setting. Based on our average measurements on experiments on an A100 GPU, we estimate the per-epoch time to be approximately 93 seconds.\n\n|           | ~Sec/Epoch | Epochs | ~Time (Mins) |\n|-----------|------------|--------|--------------|\n| FixMatch  | 93         | 30     | 46.5         |\n| FlexMatch | 93         | 30     | 46.5         |\n| SoftMatch | 93         | 30     | 46.5         |\n| FineSSL   | 93         | 30     | 46.5         |\n\n|       | ~Sec/Epoch (SFT) | Epochs (SFT) | #SFT Models | ~Sec/Epoch (Ensemble) | Epochs (Ensemble) | ~Total Time (Mins) |\n|-------|------------------|--------------|-------------|------------------------|-------------------|---------------------|\n| PET   | 2                | 50           | 2           | 93                     | 30                | 49.8                |\n| V-PET | 2                | 50           | 4           | 93                     | 30                | 53.1                |\n\nAs shown, PET incurs only ~3.3 additional minutes compared to baselines (1.07×), and V-PET ~6.6 minutes (1.14×). We believe this is a reasonable overhead given the performance gains achieved.\n\n**Concern 5: Formatting issues**\n\nThank you for pointing out the formatting issues, both about the typos and the table 3 formatting. We will carefully revise the manuscript to correct these problems and ensure a clean, professional presentation in the final version.\n\n**References**\n\n[A] Lessons and insights from a unifying study of parameter-efficient fine-tuning (PEFT) in visual recognition. CVPR’25."}, {"review_id": "ISmlOVAgSz", "reviewer": "Reviewer_DJbk", "summary": "This paper investigates the applicability of semi-supervised learning (SSL) to vision foundation models (VFMs), specifically CLIP and DINOv2. The paper systematically evaluates whether existing SSL methods remain effective when applied to VFMs. Based on findings, the paper proposes V-PET, a simple yet effective self-training SSL method that ensembles pseudo-labels generated across diverse parameter-efficient fine-tuning (PEFT) methods and VFM backbones.", "questions": "1. Beyond empirical observations, could you explain more clearly why ensemble pseudo-labeling works better than using individual PEFT pseudo-labels in VFM settings? While this is discussed in Figure 1 and lines 255–263, a clearer reason would strengthen the claims.\n\n2. In Table 3, what backbones and PEFT methods were used for the existing methods? Please clarify. If the same backbones and PEFT methods were used, your method’s training cost would increase with the number of PEFT methods and VFMs used. In that case, are there any additional benefits beyond accuracy, such as improved training stability? If different backbones or PEFT methods were used, reporting training time or FLOPs would help highlight the advantages of your approach.\n\n3. In Table 3, the proposed methods seem to show slightly lower performance on only some datasets (e.g., Resisc45 and Retinopathy). Could you explain why this happens?", "limitations": "It would be helpful to explicitly discuss the limitations of the proposed method, particularly regarding its applicability beyond classification tasks.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "This paper provides a thorough empirical study with extensive evaluations. The structure of the paper is clear, and it offers insights into SSL in the era of foundation models. While the proposed method is effective despite its conceptual simplicity, the paper offers limited theoretical insights into why ensemble pseudo-labeling is particularly effective in VFM settings.", "was_revised": true, "final_justification": "The authors have provided a detailed and thoughtful rebuttal that successfully addresses the main concerns raised in the initial review. In particular, their explanation of why ensemble pseudo-labeling is effective is well-reasoned and supported by additional empirical evidence.\n\nI maintain my original score, but I am now more confident in my assessment. I hope the authors incorporate the discussed points to further improve the clarity and completeness of the final version.", "rebuttal": "Thank you for your interest in our work, and we greatly appreciate your valuable feedback. We would like to address all your concerns, point by point, as follows:\n\n**Concern 1: Why Ensembling Works Better**\n\nThank you for raising this insightful question. We would like to reiterate and further elaborate on why ensemble pseudo-labeling is particularly beneficial in the VFM+PEFT setting, building on the evidence already discussed in Section 5.2.\n\nUnlike traditional SSL methods that ensemble models trained via random initialization, our method leverages the inherent diversity across different PEFT strategies and VFM pretraining objectives. As shown in Section 5.2, even when different PEFT methods achieve similar overall accuracy, they tend to produce divergent predictions on individual samples (Figure 1) – this is in accordance with the observations in previous work [A]. Similarly, VFMs pretrained with varying data and objectives demonstrate complementary strengths, with no single model dominating across all examples.\n\nThis natural diversity leads to a situation where individual PEFT-VFM pseudo-labels may be biased or inconsistent. By ensembling these diverse predictions, we reduce model-specific noise and extract consensus signals, thereby producing pseudo-labels that are more robust and informative.\n\nFurthermore, we address the practical challenge of inconsistent output scales across models (as discussed in *How to Ensemble?* in section 5.2 and shown in Figure 4) by proposing the *Mean Labels* strategy, which converts predictions into one-hot vectors before averaging. This avoids scale mismatch and ensures a more balanced ensemble. \n\nTogether, these designs allow our method to effectively leverage the diversity inherent in VFMs and PEFTs to construct high-quality pseudo-labels. We will take your advice and revise the corresponding section in the paper to make this motivation clearer.\n\n\n**Concern 2: Clarification on Table 3**\n\nWe appreciate your feedback and apologize for any confusion. As indicated in the header and the first column of *Table 3*, all baselines and our proposed methods are trained using the same backbone and PEFT method within each part of the table. We will make this clearer in the final revision of the paper.\n\nRegarding your concern about training time, we acknowledge that our ensemble-based methods may introduce additional computational overhead compared to other baselines. However, due to the efficiency of few-shot fine-tuning and pseudo-label extraction, we find the overhead to be _relatively modest considering the performance improvements achieved_. We provide detailed runtime comparisons in the following tables for DTD 3-shot setting, where we estimate the per-epoch wall-clock time as 93 seconds based on our average observation of experiments on one single A100 GPU:\n\n|           | ~Sec/Epoch | Epochs | ~Time (Mins) |\n|-----------|------------|--------|--------------|\n| FixMatch  | 93         | 30     | 46.5         |\n| FlexMatch | 93         | 30     | 46.5         |\n| SoftMatch | 93         | 30     | 46.5         |\n| FineSSL   | 93         | 30     | 46.5         |\n\n|       | ~Sec/Epoch (SFT) | Epochs (SFT) | #SFT Models | ~Sec/Epoch (Ensemble) | Epochs (Ensemble) | ~Total Time (Mins) |\n|-------|------------------|--------------|-------------|------------------------|-------------------|---------------------|\n| PET   | 2                | 50           | 2           | 93                     | 30                | 49.8                |\n| V-PET | 2                | 50           | 4           | 93                     | 30                | 53.1                |\n\nAs shown above, PET incurs only ~3.3 additional minutes compared to baselines (1.07×), and V-PET ~6.6 minutes (1.14×). We believe this is a reasonable and practical trade-off considering the performance gains obtained. We will also take your advice and incorporate this table in the revised version of the paper.\n\n\n**Concern 3: Lower Performance on Some Datasets**\n\nThank you for your careful observation. We acknowledge that although our method consistently outperforms other approaches across the benchmark, its performance gain is less pronounced on certain datasets, namely Resisc45. Diving deep into it, we believe this is due to the nature of the dataset itself. Compared to the rest of the benchmark, Resisc45 is relatively less challenging — as evidenced by the higher performance across other methods in Table 3. In such cases, the outputs of different PEFT models and VFMs tend to be more aligned, leaving limited room for ensembling to further reduce bias or extract complementary information. As a result, the benefit of our ensemble approach is naturally diminished.\n\nHowever, we would like to respectfully clarify that on **Retinopathy**, our method actually performs competitively — in fact, we outperform most baselines across almost all of the settings, as shown in Table 3. While there may be marginal variations in specific configurations, the overall trend supports the effectiveness of our approach, even on this challenging medical dataset.\n\n\n**Concern 4: Limitations on Scope**\n\nThank you for your valuable feedback. We acknowledge that extending our work to tasks beyond classification is valuable, but these tasks often require task-specific architecture and training paradigms. To remain focused on core machine learning problems, we align with representative SSL works (e.g., FixMatch) by prioritizing image classification. We humbly believe the fundamental SSL principles and insights from our work are transferable across tasks. For example, UniMatch [B], a popular SSL segmentation method, was inspired by FixMatch. Many SSL segmentation methods [C, D] use self-training & pseudo-labelling. We’ll also take the suggestions and discuss the limitations of our work in the revised version of the paper.\n\n**References**\n\n[A] Lessons and insights from a unifying study of parameter-efficient fine-tuning (PEFT) in visual recognition. CVPR’25.\n\n[B] Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. CVPR’23.\n\n[C] St++: Make self-training work better for semi-supervised semantic segmentation. CVPR’22.\n\n[D] Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation. ICCV21."}, {"review_id": "3NC1bPvPrJ", "reviewer": "Reviewer_gEWg", "summary": "This paper studies the problem Semi-supervised Learning (SSL) in the context of foundation models, and found that standard SSL technique only incrementally benefits foundation models.To make unlabelled data pay off, the authors introduce V-PET—a one-shot self-training routine that simply averages the one-hot predictions of several differently fine-tuned backbones and then retrains on that soft target—showing consistent gains over prior SSL baselines on a new, harder VTAB-derived benchmark.", "questions": "1. I think another informative point authors could try to add is how foundation models' performances sclaes as we increase the number/proportion of the labeled data, this is expected to behave somewhat differently to standard SSL classifiers as they perhaps need more labeled data since they are trained from the scratch.\n\n2. Are foundation models more vulnerable to the problem of imbalance comparing to classifical SSL classifiers?", "limitations": "N.A.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The problem being studied in this paper is very crucial and of high interest to the SSL community - what role does SSL play in the era of foundation models, and what sets it apart from classical SSL? \n\n2. The take-away message is informative and perhaps stirs further thinking in this domain - naively utilizing unlabeled data does not seems to contribute much for performance gain.\n\n3. The tested cases covered in the experiment are rather comprehensive and informative. \n\n4. This paper is overall well-written and easy to follow. \n\n**Weaknesses:**\n\n1. One major weakness of this paper appears to be the computational complexity of the proposed method; this method requires an ensemble of multiple foundation models, and considering the complexity of the foundation models, the computational cost of the proposed method seems significant. Moreover, efficiency studies are missing in this paper.\n\n2. More ablation studies are needed to further highlight the importance of self-training; authors mainly proposed two techniques, self-training and ensemble, and combining them together witness performance gain. However, such performance gain could solely stem from ensemble rather than self-training. \n\n3. The main insights from this paper are not significant enough - the resulting methodology essentially stems from ensemble and self-training, and all of these are not very relevant to the \"unique properties of foundation models.\" \n\n4. The coverage of techniques and tricks is not comprehensive enough; some other very important aspects, such as threshold of pseudo-label (FlexMatch, AdaMatch), data-augmentation and consistency training (FixMatch), exponential moving average, label alignment, and other techniques to counter training imbalance, are not discussed or mentioned. However, researchers in SSL might be interested in learning why or why those tricks still work for foundation models.", "was_revised": true, "final_justification": "After a through discussion with the authors, while most of my concerns are somewhat addressed, it seems that when authors reporting the computational efficiency, their original experiments lacks rigor and soundness, making the computational overhead seemingly smaller (see authors' last two responses for details), therefore my concern remain for this aspect. \n\nIn addition, the technical novelty of this paper remains limited in my view. Therefore, I tend to maintain my original evaluation as leaning towards rejection.", "rebuttal": "Thank you for your interest in our work and for your constructive feedback. Meanwhile, we respectfully think some of the concerns may result from misunderstandings, and we apologize if these were caused by the organization of our paper. Before addressing your specific concerns, we would like to provide a brief overview of our paper and algorithm. \n\nIn the conventional setting of **few-shot semi-supervised learning**, our method follows a **4-stage pipeline**:\n\n1. Supervised Parameter-Efficient Fine-Tuning: We select a set of either *(a) Different PEFT methods (PET)* or *(b) Different VFMs and PEFT methods (V-PET)*, and fine-tune them using **only a few labeled samples**.\n2. Pseudo-Label Generation: The fine-tuned model is then used to generate pseudo-labels for the unlabeled data through **single forward pass**.\n3. Pseudo-Label Ensemble: Multiple sets of pseudo-labels are ensembled through our *mean-label* strategy.\n4. Ensemble Training: The final VFM is trained on the originally unlabeled data, using the ensemble pseudo-labels as supervision.\n\nBased on the workflow. We’d like to address your concerns one by one as follows:\n\n**Concern 1: Computational Complexity**\n\nWe acknowledge that, like other ensemble approaches, our method introduces some additional computational cost overhead. However, thanks to **the efficiency of stage 1 and stage 2** listed above, this extra cost remains negligible. To better illustrate this trade-off, we summarize the end-to-end computation time in the table below:\n\n|           | ~Sec/Epoch | Epochs | ~Time (Mins) |\n|-----------|------------|--------|--------------|\n| FixMatch  | 93         | 30     | 46.5         |\n| FlexMatch | 93         | 30     | 46.5         |\n| SoftMatch | 93         | 30     | 46.5         |\n| FineSSL   | 93         | 30     | 46.5         |\n\n|       | ~Sec/Epoch (SFT) | Epochs (SFT) | #SFT Models | ~Sec/Epoch (Ensemble) | Epochs (Ensemble) | ~Total Time (Mins) |\n|-------|------------------|--------------|-------------|------------------------|-------------------|---------------------|\n| PET   | 2                | 50           | 2           | 93                     | 30                | 49.8                |\n| V-PET | 2                | 50           | 4           | 93                     | 30                | 53.1                |\n\nAs shown, PET incurs only ~3.3 additional minutes compared to baselines (1.07×), and V-PET ~6.6 minutes (1.14×). We believe this is a reasonable overhead given the performance gains achieved.\n\n**Concern 2: Self-Training vs. Ensemble**\n\nWe appreciate your question. Before we discuss ensemble and self-training, we want to first clarify that our contributions extend beyond simply employing these techniques, but also lie in the insight that PEFT and multiple VFMs can generate high-quality, complementary pseudo-labels that make ensemble learning effective. \n\nRegarding the performance gain, both ensemble and self-training are essential components. Self-training, as a SSL method, provides a principled way to leverage unlabeled data by generating pseudo-labels and fine-tuning the model in a supervised manner. Ensemble, on the other hand, serves to enhance the quality of these pseudo-labels. It is important to note that self-training is indispensable in our framework — without it, ensemble alone cannot fine-tune the pre-trained VFM.\nIn our main results (Table 3), ST, PET, and V-PET all incorporate self-training. The distinction lies in how pseudo-labels are generated. We observe that ST alone does not consistently outperform the labeled-only baseline, indicating that the unlabeled data have not been effectively leveraged—likely due to the low quality of the pseudo-labels. In contrast, our approaches—PET and V-PET—use pseudo-label ensembles from multiple VFMs and PEFTs, enabling self-training to yield significant improvements over the labeled-only setting. This highlights the critical role of pseudo-label quality and underscores the effectiveness of our approach.\n\n**Concern 3: Limited Novelty and Relevance to Foundation Models**\n\nWe respectfully disagree with the view that our methodology does not leverage the unique properties of VFMs. As mentioned in the paper [lines 51 - 56], our approach is built upon two core properties of VFMs. First, their ability to adapt quickly with minimal supervision — i.e., few-shot generalization. Second, the difference in their pretraining distribution or objectives. Our method explicitly exploits these capabilities by using VFMs to generate high-quality and complementary pseudo-labels from a few labeled examples, which enables us to further learn from unlabeled data. \n\nWhile the use of ensemble and self-training may appear conventional, our key contribution lies in showing how these techniques can be adapted to the VFM regime — not as arbitrary choices, but as mechanisms that align with VFMs’ unique strengths (e.g., stable zero-shot predictions, diverse representations). Moreover, by establishing a controlled benchmark and introducing a simple yet effective baseline, we aim to lay the groundwork for a principled exploration of SSL in the context of VFMs. We believe this is both timely and impactful, given the growing interest in adapting large-scale models under limited supervision.\n\n\n**Concern 4: Incorporating Other SSL Techniques**\n\nThank you for your question. We acknowledge that there are many existing SSL techniques. While thoroughly revisiting these tricks would be valuable to the community, we humbly believe this is orthogonal and beyond our focus. Indeed, as highlighted in paper [lines 34, 67], one of our aims is to establish a simple SSL pipeline—ideally with as few tricks as possible — by appropriately leveraging the unique properties of VFMs and PEFT.\n\nThat said, we do think that the existing SSL tricks and techniques (e.g., data augmentation, self-consistency) have the potential to further improve our method. \n\nTo explore this direction, we conducted additional experiments to evaluate the compatibility of our method with both weak and strong augmentations and self-consistency techniques. In the experiments, we fixed the hyperparameter settings under the **DTD-3SHOT, DINOv2** configuration, using **LoRA** for ensemble training. We evaluated the performance of our method in combination with existing SSL techniques under two main settings:\n\n1) **Self-Consistency**: Pseudo-labels are generated from weakly augmented (random horizontal flip) unlabeled data and averaged over **10** stochastic forward passes.\n\n2) **Strong Data Augmentation**: During ensemble training, we add a cross-entropy loss term computed on RandAugment(m=3, n=5)-augmented samples.\n\n|                     | **w/o Self-Consistency** | **+ Self-Consistency** |\n|---------------------|------------------------|----------------------|\n| **w/o Strong Aug**  | 63.62                | 63.35              |\n| **w/ Strong Aug**           | 64.26                | 64.26              |\n\nWe observe that strong augmentation yields modest performance gains, whereas self-consistency offers no additional benefit. This suggests that the effectiveness of such techniques may be less pronounced in our method or requires further tuning. We consider a more comprehensive exploration of such combinations as a promising direction for future work.\n\n\n**Question 1: Foundation Models’ Scaling Law for Labeled Data**\n\nThank you for your constructive suggestion. Compared to training standard SSL classifiers from scratch, foundation models indeed require less labeled data – that is precisely why existing works leveraging foundation models for SSL [B][C], including us, primarily focus on low-shot settings. That said, we agree that systematically studying this question is an interesting direction. In response, we have extended our supervised fine-tuning part of **Table 3** on the *DTD* dataset from **3-SHOT** to **48-SHOT** setting. The updated results are shown below, \n\n\n|                    | DTD N3 | DTD N6 | DTD N12 | DTD N24 | DTD N48 |\n|--------------------|--------|--------|---------|---------|---------|\n| DINOv2 LoRA        | 61.01 | 68.40 |  74.26 |  78.19 |  82.45 |\n| DINOv2 AdaptFormer | 62.98 | 67.61 |  74.52 |  80.48 |  83.03 |\n| CLIP LoRA          | 57.87 | 64.10 |  69.84 |  74.57 |  78.46 |\n| CLIP AdaptFormer   | 61.54 | 64.95 |  71.76 |  76.17 |  79.36 |\n\nAs we can see, performance on low-shot settings (**N3, N6**) is already strong, thanks to the rich priors encoded in foundation models. As the number of labeled examples increases, performance continues to improve but with diminishing returns – indicating a flatter scaling curve compared to standard SSL classifiers trained from scratch. We appreciate the reviewer for highlighting this important difference, and we’ll revise the final version to have this study.\n\n**Question 2: Foundation Models’ Vulnerability to Imbalanceness**\n\nThank you for the question. While we do not explicitly explore this question in the main paper, we respectfully believe that using foundation models as the backbone would make the overall SSL training more robust to class imbalance, compared to training from scratch (in conventional SSL methods). This is because foundation models often enable few-shot adaptation, which prevents minor categories from being simply dominated by major categories. Notably, our V-PET approach is a simple pseudo-labeling baseline without any rebalancing techniques, yet it still performs competitively. This suggests a certain degree of inherent robustness to imbalance in foundation models. We consider this a promising direction for future exploration.\n\n**References**\n\n[A] Lessons and insights from a unifying study of parameter-efficient fine-tuning (PEFT) in visual recognition. CVPR’25.\n\n[B] Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning. ICML’24.\n\n[C] USB: A Unified Semi-supervised Learning Benchmark for Classification. NeurIPS’22."}, {"review_id": "kNcfJq6X6k", "reviewer": "Reviewer_BzLX", "summary": "Foundation models are often used as backbones for downstream tasks, like classification. When applied to bespoke classification datasets, baseline performance may not be as high as users require. In these settings, it is common that users have a small set of labelled data and much more unlabelled data, making it suitable for semi-supervised learning.\n\nThe authors construct an evaluation dataset with classification tasks in which existing foundation models (DINOv2 and CLIP) perform poorly (30-70% accuracy).\n\nThe authors explore if existing semi-supervised learning methods (FixMatch, FlexMatch, SoftMatch) can be used to fine-tune foundation models to improve their performance on these bespoke datasets. When fine-tuning the models, the authors test both full parameter fine-tuning and parameter-efficient fine-tuning (LORA and Adaptformer). \n\nThe authors find that:\n- the simple approach, full parameter fine-tuning on just a few labelled data samples is very effective and existing SSL methods do **not** confer an advantage.\n- parameter-efficient fine-tuning improves performance of all training methods, again providing no advantage to SSL methods which make use of extra unlabelled data.\n\nThe authors propose a new method for SSL on foundation models that uses parameter efficient self-training and ensembling to improve model performance. They use a small amount of labelled data to fine-tune the model and use the resulting model to generate pseudo-labels for self-training. They ensemble several base models and parameter-efficient fine-tuning methods. They try three different supervisory signals leveraging the ensembling: (Mean Logits, Mean Probs, and Mean Labels). They empirically demonstrate that their approach with Mean Labels performs the best on their evaluation dataset.", "questions": "None", "limitations": "The authors test on classification and on 6 datasets, this is good evaluation benchmark but not comprehensive. It's addressed to some degree in the text, but not in a specific section independently.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The problem choice is important. Many users try to apply foundation models to specific problems and find mixed results. Establishing a benchmark and methods for this specific setting is worthwhile.\n- The paper is written clearly. It is easy to follow the investigation of prior work and the intuition behind their proposed approach.\n- The authors ablate various components of their algorithm to precisely understand the contributions of each component. They also explore several options for each step. They choose recent methods for their baseliens and conduct hyperparameter sweeps to try and achieve a fair comparison.\n-  The method they are proposing is a novel combination of established methodologies. \n\nWeaknesses:\n- While the evaluation benchmark is reasonable, it is still relatively limited in comparison to the diversity of tasks users may be interested in. \n- The methodology is fairly similar to the one proposed in FineSSL. The authors work does introduce two additional contributions: (1) more breadth in the evaluation tasks, backbones and baselines explored and (2) the idea to ensemble different methods to achieve better performance. \n\nMinor: \n- L113 - \"first\" comprehensive study of SSL in the era of foundation models is a broad claim. A more specific claim would be more appropriate. \n- L187 - \"few\" is ambiguous, refer to Table 2.\n- L240 - would be better to keep section header with the text it refers to.\n- L867-869 - potentially capitalize appendix?\n\nJustification: I recommend a 5 because the problem is important, the paper is technically sound, and presented well. I do not believe the results or method is ground-breaking, so I cannot recommend a higher score.", "was_revised": true, "final_justification": "I have read the other reviews and their responses and I have no new concerns. I intend to maintain my rating (5) as the significance, clarity, and soundness of the work are strong, but the methodological novelty remains limited.", "rebuttal": "Thank you for your support of our work and your constructive feedback. \n\n**Concern 1: Selection of Evaluation Benchmark**\n\nWe appreciate this observation and thank you for pointing out the limitation regarding task diversity in our evaluation benchmark. \n(1) Regarding the dataset in our benchmark: ​​While our study centers on classification, we deliberately ensured diversity by following the VTAB sub-categories (*NATURAL*, *SPECIALIZED*, and *STRUCTURED*), selecting at least two representative datasets from each. This design balances task diversity, domain coverage, and practical feasibility. To our knowledge, such a structured evaluation within the context of SSL remains underexplored. We hope our benchmark helps fill this gap and serves as a solid foundation to inspire future research on semi-supervised learning with VFMs across a broader range of tasks beyond classification.\n\n(2) Regarding the task type of our benchmark: We agree that extending to tasks beyond classification is valuable. However, such tasks often involve task-specific architectures and training paradigms. To maintain focus on core machine learning challenges, we align with prior SSL works (e.g., FixMatch) by prioritizing image classification. We believe the core SSL principles and insights from our study are transferable. For instance, UniMatch [A], a popular SSL segmentation method, was directly inspired by FixMatch, and many SSL segmentation approaches [B, C] adopt self-training and pseudo-labeling. We appreciate the suggestion and will discuss this limitation in our revised manuscript.\n\n**Concern 2: Comparing to FineSSL**\n\nThank you for your thoughtful observation. We agree that our work shares a similar high-level methodology with FineSSL [D] – both adopt pseudo-labeling in conjunction with PEFT to improve sample efficiency in semi-supervised learning with VFMs. FineSSL presents a simple and well-structured approach, and our work also provides valuable extension along two key axes: (1) building a more systematic and diverse benchmark across datasets, backbones, and PEFT methods, and (2) exploring model diversity through ensembling to further enhance pseudo-label quality.\n\nWe see FineSSL as a timely and valuable contribution to this emerging research direction. Our aim is to build upon such efforts to establish a strong and extensible baseline for future work. We will revise the related work and discussion sections to better clarify the connection and distinction between our method and FineSSL.\n\n**Concern 3: Scope of the Work**\n\nThank you for your positive evaluation regarding the importance of the problem, technical soundness, and clarity of presentation. We also fully respect your assessment regarding the level of novelty. While the current results may not appear ground-breaking in isolation, we believe the proposed framework and benchmark introduce a new perspective in the SSL community that would likely lead to broader implications. We see this as a step toward a new paradigm that could catalyze future research and lead to impactful applications within the community.\n\n**Concern 4: Section for Limitations**\n\nThank you for the constructive suggestion. We agree that, while we briefly acknowledge certain limitations in the main text (e.g., the focus on classification tasks [line 75] and the selection of six datasets [line 135]), these points would be clearer and more transparent if presented in a dedicated section. In the revised version, we will add an explicit \"Limitations\" section to clearly outline the scope of our current study and discuss directions for broader task coverage in future work.\n\n**Concern 5: Formatting issues**\n\nThank you for pointing out our flaws in the formatting. We’ll revise them in the future manuscript and ensure a clean and professional final version.\n\n**References**\n\n[A] Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. CVPR’23.\n\n[B] St++: Make self-training work better for semi-supervised semantic segmentation. CVPR’22.\n\n[C] Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation. ICCV21.\n\n[D] Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning. ICML’24."}]

Author rebuttal:

Meta-review: This paper revisits semi-supervised learning (SSL) in the context of vision foundation models (VFMs). Through a systematic study, the authors show: (1) parameter-efficient fine-tuning (PEFT) on labeled data alone can outperform classic SSL baselines; (2) PEFT models provide high-quality pseudo-labels; and (3) complementary signals emerge when ensembling pseudo-labels across diverse PEFT methods and VFM backbones. They propose a simple baseline—ensemble pseudo-labeling across heterogeneous PEFT techniques and VFMs—and validate it extensively.

All reviewers recommend acceptance, highlighting strong empirical results, a clear and timely problem framing, and actionable takeaways for practitioners adapting SSL to the “foundation model era.” The paper offers clear, empirically grounded insights that reshape how we think about SSL in the presence of strong VFMs and PEFT. The proposed ensemble pseudo-labeling baseline is simple and likely to influence practice. Please address the notes made by the reviewers to strengthen the camera-ready version.
Decision: Accept (poster)