Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Models

## Abstract

Abstract Semi-supervised learning (SSL) alleviates the cost of data labeling
process by exploiting unlabeled data and has achieved promising results. Meanwhile, with the development of large foundation models, exploiting pre-trained models becomes a promising way to address the label scarcity in the downstream tasks, such as various parameter-efficient fine-tuning techniques. This raises a natural yet critical question: When labeled data is limited, should we rely on unlabeled data or pre-trained models? To investigate this issue, we conduct a fair comparison between SSL methods and pre-trained models (e.g., CLIP) on representative image classification tasks under a controlled supervision budget.
Experiments reveal that SSL has met its “Waterloo” in the era of large models, as pre-trained models show both high efficiency and strong performance on widely adopted SSL benchmarks. This underscores the urgent need for SSL researchers to explore new avenues, such as deeper integration between the SSL and pre-trained models. Furthermore, we investigate the potential of Multi-Modal Large Language Models (MLLMs) in image classification tasks. Results show that, despite their massive parameter scales, MLLMs still face significant performance limitations, highlighting that even a seemingly well-studied task remains highly challenging.

## 1 Introduction

Deep neural networks have demonstrated performance comparable to that of humans in certain supervised learning tasks, such as image classification (Geirhos et al., 2018; He et al., 2016; Lee et al., 2013b). However, these impressive results rely heavily on large amounts of labeled training data. In many real-world applications, obtaining such labeled data often demands substantial human and financial resources (Yang et al., 2022). To reduce the dependence on labeled data and thereby cut costs, researchers have proposed Semi-Supervised Learning (SSL), leveraging unlabeled data to enhance model performance for image classification tasks.

In recent years, Pre-trained Models have attracted considerable attention. Among them, vision-language models (VLMs) such as CLIP (Radford et al., 2021), which achieves semantic alignment during pre-training, has become a widely adopted backbone for image classification tasks. Building upon CLIP, various fine-tuning strategies (e.g., prompt tuning (Zhou et al., 2022b)) have been proposed to enhance task-specific performance using minimal labeled data by leveraging the rich knowledge acquired during pre-training (Zhou et al., 2022b; Khattak et al., 2023a; b). More recently, multi-modal large language models (MLLMs) such as Qwen-VL (Bai et al., 2023) and GPT-4o-min (Hurst et al., 2024) have emerged, featuring significantly larger parameter scales and stronger general capabilities in visual tasks.

As shown in Figure [1](https://arxiv.org/html/2505.13317v4#S1.F1), both SSL and pre-trained models are well suited to image classification tasks with limited labeled data, by leveraging unlabeled data and pre-trained knowledge, respectively. This makes us come up with three questions naturally:

- 1.
When labels are scarce, should we prioritize utilizing unlabeled data or pre-trained models?
- 2.
What are the strengths and limitations of leveraging unlabeled data versus exploiting pre-trained models?
- 3.
How can we design methods that mitigate the shortcomings of both approaches while maximizing their complementary benefits?

Figure: Figure 1: Pipelines of semi-supervised learning and pretrain-finetuning.
Refer to caption: x1.png

To answer these questions, we adopt widely used CLIP-based VLMs as representative pre-trained models, owing to their strong generalization capabilities and proven effectiveness in zero- and few-shot learning across diverse domains (Radford et al., 2021; Zhou et al., 2022b; Khattak et al., 2023b). We extracted and designed a fair evaluation framework based on papers from top-tier conferences and journals to compare various SSL methods and pre-trained models under consistent experimental conditions. Specifically, we standardize the amount of labeled data used across different experimental setups, including semi-supervised, open-set semi-supervised, open-world semi-supervised, and long-tailed semi-supervised settings, ensuring equal supervision signals for all approaches.

The experimental results show that the SSL method faces its Waterloo, with problems such as low training efficiency and poor performance. Moreover, when the test data contains unseen classes, SSL struggles to learn and adapt effectively. In contrast, pre-trained models, endowed with rich prior knowledge, can quickly adapt to downstream tasks with limited data, demonstrating strong generalization to open-world scenarios and long-tailed distributions. Although pre-trained models may suffer from adaptation bias due to distribution discrepancies between pre-training data and downstream tasks, these issues can be alleviated through additional training or data distillation techniques. Such approaches also yield performance gains and higher efficiency compared to SSL methods.

To bridge the gap between the two paradigms, we further explore hybrid algorithms. By leveraging their generalization ability, these methods can generate more reliable pseudo-labels, thereby substantially enhancing the efficiency and stability of semi-supervised training. We explore the performance of larger-scale MLLMs on image classification tasks, and the results show that they are likewise constrained by adaptation bias and other issues arising from the limitations of their visual encoders. To further advance research in this area, we propose the following directions:

- 1.
It is no longer sufficient for SSL researchers to compare only with previous SSL methods in image classification; systematic comparisons with pre-trained models are necessary to ensure fair and meaningful evaluations.
- 2.
Exploring effective strategies to integrate SSL techniques with the adaptation of pre-trained models represents another promising approach to enhance both the efficiency of SSL and the adaptability of pre-trained models.
- 3.
Exploring the capabilities of MLLMs on image classification tasks, particularly across varying distributions and resolutions, as experiments highlight that even a seemingly well-studied task like image classification remains challenging for MLLMs.

## 2 Related Work

Semi-Supervised Learning. The goal of Semi-Supervised Learning is to reduce the cost of data annotation by leveraging unlabeled data, under the assumption that both labeled and unlabeled samples belong to a predefined set of known classes. The primary objective is to assign accurate ground-truth labels to unlabeled instances. Existing SSL methods can be broadly categorized into three main approaches: Hybrid and Holistic Methods (Berthelot et al., 2019a; b). Pseudo-Labeling assigns pseudo-labels to unlabeled data based on the model’s own predictions throughout training (Lee et al., 2013a). FixMatch (Sohn et al., 2020) combines regularization and confidence-based filtering by enforcing consistency between weakly and strongly augmented samples, while only using high-confidence pseudo-labels for training, resulting in significant performance gains with limited labeled data.

Pre-trained Models. Recent advances in pre-trained models have attracted considerable attention, particularly with the emergence of VLMs such as CLIP (Radford et al., 2021) and MLLMs such as Qwen-VL (Bai et al., 2023). While prior work (Zoph et al., 2020) has compared pre-trained models with self-training algorithms in object detection and segmentation tasks, it did not consider VLMs, MLLMs, or even SSL settings. To address this gap, we focus on the widely studied image classification task and systematically investigate the performance of VLMs and their fine-tuning strategies.
As the pioneering work for fine-tuning VLMs, CoOp (Zhou et al., 2022b) for the first time introduces learnable prompts to transfer the task-specific knowledge to VLMs. Building on this idea, MaPLe (Khattak et al., 2023a) and PromptSRC (Khattak et al., 2023b) extend prompt tuning to both visual and textual encoders, enabling joint vision-language adaptation. Multi-modal large language models, on the other hand, are primarily geared toward reasoning tasks, and their capabilities in image classification remain underexplored. Therefore, this paper selects three MLLMs for a comparative study.

## 3 Evaluation Settings

We summarize four common SSL settings from 50 recent top-tier publications, which represent the primary and most challenging focuses of current research. Then, we reformulate these settings for pre-trained VLMs to ensure fair comparisons. The dataset $\mathcal{D}$ used in this setting consists of a small labeled subset $\mathcal{D}_{l}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N_{l}}$ and a larger unlabeled subset $\mathcal{D}_{u}=\{\mathbf{x}_{j}\}_{j=1}^{N_{u}}$, where the number of labeled samples is significantly smaller than the unlabeled samples $(N_{l}\ll N_{u})$. The following provides an overview of each setting, and details are shown in Appendix [B](https://arxiv.org/html/2505.13317v4#A2).

- •
Standard Semi-Supervised Learning (Sohn et al., 2020). In this setting, the labels for both subsets belong to the same known class set $\mathcal{Y}^{l}$. For pre-trained models, which do not rely on unlabeled data for training, our setting reduces this setting to a standard few-shot scenario, provided the number of labeled samples remains the same.
- •
Open-Set Semi-Supervised Learning (Oliver et al., 2018). In this setting, the unlabeled dataset $\mathcal{D}_{u}$ may include instances from unseen classes, i.e., $\mathcal{Y}^{l}\subset\mathcal{Y}^{u}$ with $\mathcal{Y}^{\text{new}}=\mathcal{Y}^{u}\setminus\mathcal{Y}^{l}$. Since pre-trained models do not utilize unlabeled data, our open-set setting reduces to the semi-supervised setting.
- •
Open-World Semi-Supervised Learning (Niu et al., 2024). This setting extends the open-set scenario by aiming to classify both known and novel classes. Pre-trained models naturally support this via zero-shot classification, making the open-world semi-supervised setting analogous to the widely used base-to-novel setting (Zhou et al., 2022a).
- •
Long-Tailed Semi-Supervised Learning (Kukleva et al., 2023). This setting features a highly imbalanced class distribution, where major (head) classes have many samples and minor (tail) classes have few. The imbalance ratio (IR) is defined as the sample count ratio between the largest and smallest classes. Since VLMs use a fixed number of samples per class in few-shot settings, aligning the shot count with tail classes reduces the problem to a standard few-shot scenario, enabling fair comparison.

## 4 Experiments

### 4.1 Baselines Selection

Given that SSL is primarily applied to image classification tasks, we selected FixMatch (Sohn et al., 2020) as a representative method for standard semi-supervised learning, DWD (Ban et al., 2024) for open-set semi-supervised learning, OwMatch (Niu et al., 2024) for open-world semi-supervised learning, and CCL (Zhou et al., 2024) for long-tailed semi-supervised learning. For pre-trained models, we consider CLIP (Radford et al., 2021) along with its prompt tuning variants: CoOp, which was the first to introduce prompt tuning for CLIP (Zhou et al., 2022b), and PromptSRC (Khattak et al., 2023b), a state-of-the-art method in this line of work.

### 4.2 Comparisons in Standard SSL Setting

**Table 1: Accuracy on CIFAR-10, CIFAR-100, and STL-10. The best performance for each dataset is in bold, while the top scores in semi-supervised setting are underlined.**
| Method | CIFAR-10 | CIFAR-100 | STL-10 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 40 labels | 250 labels | 4000 labels | 400 labels | 2500 labels | 10000 labels | 1000 labels |  |
| MixMatch | 52.46±11.50 | 88.95±0.86 | 93.58±0.10 | 32.39±1.32 | 60.06±0.37 | 71.69±0.33 | 89.59±0.61 |
| UDA | 70.95±5.93 | 91.18±1.08 | 95.12±0.18 | 40.72±0.88 | 66.87±0.22 | 75.50±0.25 | 92.34±0.56 |
| ReMixMatch | 80.90±9.64 | 94.56±0.05 | 95.28±0.13 | 55.72±2.06 | 72.57±0.31 | 76.97±0.56 | 94.77±0.45 |
| FixMatch (RA) | 86.19±3.37 | 94.93±0.65 | 95.74±0.05 | 51.15±1.75 | 71.71±0.11 | 77.40±0.12 | 92.02±1.50 |
| FixMatch (CTA) | 88.61±3.35 | 94.93±0.33 | 95.69±0.15 | 50.05±3.01 | 71.36±0.24 | 76.82±0.11 | 94.83±0.63 |
| ZSCLIP | 79.30±0.00 | 46.00±0.00 | 97.40±0.00 |  |  |  |  |
| CoOp | 80.60±1.79 | 85.10±0.29 | 89.13±0.12 | 49.00±0.83 | 57.43±0.25 | 61.40±0.14 | 98.23±0.17 |
| PromptSRC | 86.03±0.48 | 90.10±0.08 | 93.90±0.14 | 60.33±0.17 | 66.33±0.29 | 70.10±0.22 | 98.57±0.12 |

**Table 2: Performance of SSL and pre-trained VLMs on the ImageNet dataset.**
| Method | UDA | FixMatch | ZSCLIP | CoOp | PromptSRC |
| --- | --- | --- | --- | --- | --- |
| Label | 10% labels | 10% labels | 0 labels | 2% labels | 2% labels |
| Accuracy | 68.78±0.43 | 71.46±0.52 | 66.70±0.00 | 73.43±0.12 | 74.03±0.09 |

Datasets and Experimental Setup.
We evaluate both SSL and pre-trained VLMs on CIFAR-10/100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Deng et al., 2009), under varying annotation levels and augmentation strategies. For VLMs, we simulate few-shot learning by training on the same labeled data only, without accessing unlabeled samples. Full implementation details are provided in Appendix [B.2](https://arxiv.org/html/2505.13317v4#A2.SS2).

Experiment Results.
As shown in Table [1](https://arxiv.org/html/2505.13317v4#S4.T1), SSL methods generally perform well on low-resolution datasets such as CIFAR-10 and CIFAR-100, which have an image resolution of 32 $\times$ 32, surpassing zero-shot CLIP and its fine-tuning variants. This advantage stems from the challenge of extracting meaningful features from low-resolution inputs due to the “adaptation bias” between the pre-training data of VLMs and the target downstream tasks.
In contrast, pre-trained VLMs excel on higher-resolution datasets like STL-10 (96 $\times$ 96), where even zero-shot CLIP surpasses the best-performing SSL baselines.
Moreover, as shown in Table [2](https://arxiv.org/html/2505.13317v4#S4.T2), on the ImageNet dataset, FixMatch requires a significantly larger proportion of labeled data (around 10%) but still lags behind in accuracy. By comparison, VLMs achieve over 73% accuracy using only 2% of labeled samples, with performance improving as more labeled data becomes available.
In addition to accuracy, training SSL models on CIFAR datasets takes GPU hours (Berthelot et al., 2019a; Sohn et al., 2020), whereas prompt tuning approaches for VLMs (Khattak et al., 2023a; b; Zhou et al., 2022b) can be completed in just a few GPU minutes. In summary, the above observations highlight the complementary strengths of SSL and pre-trained models, with the latter generally offering a higher performance ceiling and faster training speed.

Figure: Figure 2: Performance of different backbones.
Refer to caption: Backbone.png

Analysis of “Adaptation Bias”.
We argue that “adaptation bias” is typically caused by two main factors: the capacity of the visual encoder and discrepancies between training and testing styles (e.g., differences in resolution and image style). To evaluate the impact of resolution, we test CLIP models with different backbones on the CIFAR-10/100 datasets. As shown in Table [3](https://arxiv.org/html/2505.13317v4#S4.F3), performance improves steadily with the number of visual encoder parameters, indicating that the suboptimal results on low-resolution images are not due to the encoder itself.
Since the original training data of CLIP is unavailable, we approximate this bias through input resolution (CLIP is fixed at an input resolution of 224 $\times$ 224). We then evaluated various methods under different input resolutions on the STL-10 dataset (Figure [3](https://arxiv.org/html/2505.13317v4#S4.F3)). As the resolution increases, model performance consistently improves, and adaptation bias is gradually reduced. This suggests that the distribution shift between training and testing is the fundamental cause of “adaptation bias”.
Fine-tuning can partially mitigate the adaptation bias of models toward low-resolution images (e.g., PromptSRC outperforms CLIP at 16 $\times$ 16 resolution). Additionally, model distillation techniques have been shown to alleviate this bias (Lee et al., 2025; Zhou et al., 2023). For instance, RADIO (Heinrich et al., 2024) enhances the stability and performance of student models across varying input resolutions by distilling knowledge from teacher models and incorporating strategies such as multi-resolution training and balanced teacher losses.

### 4.3 Comparisons in Open-Set SSL Setting

**Table 3: Performance comparison over three datasets. We report the mean accuracy averaged over three seeds, along with standard error.**
| Dataset | FixMatch | MPL | IOMatch | DWD-SL | ZSCLIP | CoOp | PromptSRC |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR-10/100 | 78.91±0.15 | 70.95±0.34 | 77.66±0.22 | 80.05±0.14 | 79.30±0.00 | 87.87±0.26 | 92.27±0.12 |
| ImageNet-30 | 70.07±0.26 | 72.65±0.70 | 79.23±0.29 | 82.20±0.38 | 98.80±0.00 | 99.33±0.12 | 99.30±0.00 |
| ImageNet-100 | 65.11±0.32 | 68.43±0.33 | 66.85±0.19 | 82.81±0.31 | 87.70±0.00 | 91.37±0.12 | 91.70±0.08 |

Datasets and Experimental Setup. We evaluate both paradigms on standard open-set benchmarks, including CIFAR-10/100 (Krizhevsky et al., 2009), ImageNet-30 (Hendrycks et al., 2019), and ImageNet-100 (Cao et al., 2021). For CIFAR-10/100, we use 100 labeled images per class from CIFAR-10 as the supervised data and the entire CIFAR-100 dataset as the unsupervised data. For ImageNet-30 and ImageNet-100, we adopt a 50-shot setting, using 50 labeled samples per class from the selected in-distribution (ID) classes.
It is important to note that the test set in this setting contains only the classes seen during training. For pre-trained models, the experimental setups are consistent with those in the semi-supervised setting. Further details can be found in Appendix [B.3](https://arxiv.org/html/2505.13317v4#A2.SS3).

Experiment Results. As shown in Table [3](https://arxiv.org/html/2505.13317v4#S4.T3), zero-shot CLIP and its fine-tuned variants consistently outperform other methods across all datasets, with particularly strong results on ImageNet-30 and ImageNet-100. This advantage stems from their data efficiency—without reliance on noisy unlabeled data, VLMs are less impacted by out-of-distribution (OOD) samples in low-quality datasets.
In contrast, SSL methods are more sensitive to such data quality issues, as they require specialized mechanisms to identify and filter out OOD samples. This reliance on additional OOD handling procedures inevitably affects both training efficiency and final performance, often leading to suboptimal results.
Moreover, pre-trained models derive distinct advantages from two key stages: first, they undergo extensive large-scale pre-training on diverse datasets, and second, they are refined through targeted fine-tuning on task-specific data. This two-step process enables them to achieve consistently lower performance variance across different data distributions and a higher performance ceiling. Based on the analysis, we conclude that pretrained models are better suited for scenarios where unlabeled data contain previously unseen classes, and that the impact of noisy data is far more severe than that of “adaptation bias”.

### 4.4 Comparisons in Open-World SSL Setting

**Table 4: Average accuracy on CIFAR-10/100 and ImageNet-100 datasets with both new class ratio and label ratio of 50%.**
| Method | CIFAR-10 | CIFAR-100 | ImageNet-100 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Base | New | All | Base | New | All | Base | New | All |  |  |
| FixMatch | 71.5 | 50.4 | 49.5 | 39.6 | 23.5 | 20.3 | 65.8 | 36.7 | 34.9 |  |
| DS^3L | 77.6 | 45.3 | 40.2 | 55.1 | 23.7 | 24.0 | 71.2 | 32.5 | 30.8 |  |
| CGDL | 72.3 | 44.6 | 39.7 | 49.3 | 22.5 | 23.6 | 67.3 | 33.8 | 31.9 |  |
| DTC | 53.9 | 39.5 | 38.3 | 31.3 | 22.9 | 18.3 | 25.6 | 20.8 | 21.3 |  |
| RankStats | 86.6 | 81.0 | 82.9 | 36.4 | 28.4 | 23.1 | 47.3 | 28.7 | 40.3 |  |
| SimCLR | 58.3 | 63.4 | 51.7 | 28.6 | 21.1 | 22.3 | 39.5 | 35.7 | 36.9 |  |
| ORCA | 88.2 | 90.4 | 89.7 | 66.9 | 43.0 | 48.1 | 89.1 | 72.1 | 77.8 |  |
| NACH | 89.5 | 92.2 | 91.3 | 68.7 | 47.0 | 52.1 | 91.0 | 75.5 | 79.6 |  |
| OpenLDN | 95.7 | 95.1 | 95.4 | 73.5 | 46.8 | 60.1 | 89.6 | 68.6 | 79.1 |  |
| OpenCon | 89.3 | 91.1 | 90.4 | 69.1 | 47.8 | 52.7 | 90.6 | 80.8 | 83.8 |  |
| OwMatch | 93.0 | 95.9 | 94.4 | 74.5 | 55.9 | 65.1 | 91.7 | 72.0 | 81.8 |  |
| OwMatch+ | 96.5 | 97.1 | 96.8 | 80.1 | 63.9 | 71.9 | 91.5 | 79.6 | 85.5 |  |
| ZSCLIP | 88.0 | 88.1 | 79.3 | 57.4 | 53.1 | 46.0 | 89.5 | 87.9 | 87.7 |  |
| CoOp | 91.1 | 90.4 | 81.3 | 65.4 | 51.0 | 48.4 | 89.7 | 83.2 | 85.5 |  |
| PromptSRC | 92.9 | 90.0 | 83.8 | 72.3 | 62.8 | 58.9 | 91.2 | 89.7 | 89.4 |  |

Datasets and Experimental Setup. We conduct evaluations on CIFAR-10/100 (Krizhevsky et al., 2009), and ImageNet-100 (Cao et al., 2021). In each dataset, the classes are divided into known (base) and new classes, with the proportion of new classes set to 50% by default. A portion of the samples from the known classes are labeled according to a given label ratio, while the remaining part, together with all the new class data, formed the unlabeled set. This setup requires the model to distinguish and classify both base and new classes simultaneously. For more detailed settings, please refer to Appendix [B.4](https://arxiv.org/html/2505.13317v4#A2.SS4).

Experiment Results. In the open-world setting, SSL methods aim to identify unseen classes from unlabeled data during training, typically using clustering-based pseudo-labeling, while evaluating both base and novel classes at test time. As shown in Table [4](https://arxiv.org/html/2505.13317v4#S4.T4), SSL methods achieve the best performance on CIFAR-10 and CIFAR-100 datasets, indicating that specially tailored losses outperform pretrained models affected by “adaptation bias”, although requiring substantial GPU training time.
In contrast, on the ImageNet-100 dataset, zero-shot CLIP already approaches the performance of OwMatch on base classes and significantly outperforms it on novel classes. This can be attributed to the richer feature representations captured by pre-trained models as image resolution increases, thereby reducing “adaptation bias”.
Notably, fine-tuned models exhibit a smaller performance gap between base and novel classes on ImageNet-100 compared to SSL methods, highlighting their strong generalization capabilities. In summary, pre-trained models that overcome adaptation bias offer advantages such as rapid adaptability and strong generalization, whereas SSL methods require careful design to effectively handle both novel and unseen classes.

### 4.5 Comparisons in Long-Tailed SSL Setting

**Table 5: Performance of SSL and VLMs under consistent imbalance ratio setting.**
| Method | CIFAR10-LT | CIFAR100-LT |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\gamma=\gamma_{l}=\gamma_{u}=100$ | $\gamma=\gamma_{l}=\gamma_{u}150$ | $\gamma=\gamma_{l}=\gamma_{u}=10$ | $\gamma=\gamma_{l}=\gamma_{u}=20$ |  |  |  |  |  |
|  | $N_{1}=500$<br>$M_{1}=4000$ | $N_{1}=1500$<br>$M_{1}=3000$ | $N_{1}=500$<br>$M_{1}=4000$ | $N_{1}=1500$<br>$M_{1}=3000$ | $N_{1}=50$<br>$M_{1}=400$ | $N_{1}=150$<br>$M_{1}=300$ | $N_{1}=50$<br>$M_{1}=400$ | $N_{1}=150$<br>$M_{1}=300$ |
| Supervised | 47.3±0.95 | 61.9±0.41 | 44.2±0.33 | 58.2±0.29 | 29.6±0.57 | 46.9±0.22 | 25.1±1.14 | 41.2±0.15 |
| w/ LA | 53.3±0.44 | 70.6±0.21 | 49.5±0.40 | 67.1±0.78 | 30.2±0.44 | 48.7±0.89 | 26.5±1.31 | 44.1±0.42 |
| FixMatch + LA | 75.3±2.45 | 82.0±0.36 | 67.0±2.49 | 78.0±0.91 | 47.3±0.58 | 58.6±0.36 | 41.4±0.93 | 53.4±0.32 |
| w/ DARP | 76.6±0.92 | 80.8±0.62 | 68.2±0.24 | 76.7±1.35 | 50.4±0.65 | 59.0±0.53 | 43.6±1.05 | 55.3±0.84 |
| w/ DASO | 77.±0.88 | 82.5±0.60 | 71.0±1.68 | 79.0±2.23 | 50.1±0.75 | 61.0±0.72 | 43.5±0.81 | 56.1±0.72 |
| FixMatch + ACR | 81.6±0.19 | 84.1±0.20 | 77.0±1.19 | 82.0±0.41 | 51.1±0.80 | 61.3±0.26 | 46.1±0.34 | 56.5±0.33 |
| FixMatch + CPE | 80.7±0.96 | 84.4±0.29 | 76.8±0.53 | 82.3±0.36 | 50.6±0.45 | 59.8±0.13 | 43.6±0.88 | 55.8±0.65 |
| FixMatch + CCL | 84.5±0.38 | 86.2±0.35 | 81.5±0.99 | 84.0±0.21 | 53.5±0.49 | 63.5±0.39 | 46.8±0.45 | 57.5±0.55 |
| ZSCLIP | 79.30±0.00 | 46.00±0.00 |  |  |  |  |  |  |
| CoOp | 80.73±1.90 | 84.20±0.50 | 80.60±1.79 | 82.23±0.75 | 49.83±1.04 | 55.23±0.50 | 44.83±0.45 | 52.57±0.12 |
| PromptSRC | 86.50±0.37 | 88.70±0.08 | 86.03±0.48 | 87.67±0.42 | 60.33±0.17 | 64.37±0.34 | 58.03±0.46 | 62.83±0.48 |

Datasets and Experimental Setup.
We compare different methods on four long-tailed datasets: CIFAR-10-LT (Krizhevsky et al., 2009), CIFAR-100-LT (Krizhevsky et al., 2009), STL-10-LT (Coates et al., 2011), and ImageNet-127 (Fan et al., 2022). For synthetic long-tailed datasets (CIFAR, STL-10), the class-wise labeled/unlabeled sample counts follow an exponential decay controlled by imbalance ratios $\gamma_{l}$ and $\gamma_{u}$ (Wei & Gan, 2023). For ImageNet-127, a naturally long-tailed dataset, we follow (Zhou et al., 2024) and use 10% of the training data as labeled. To ensure fair comparison, we adopt the same splits and protocols as SSL methods (Niu et al., 2024), and report performance of pre-trained models in a few-shot setting where the number of shots is determined by the smallest labeled class (e.g., 5-shot for CIFAR-10 when $\gamma_{l}=100$ and $N_{1}=500$). Full setup details and shot derivations are available in Appendix [B.5](https://arxiv.org/html/2505.13317v4#A2.SS5).

Experiment Results.
As shown in Table [5](https://arxiv.org/html/2505.13317v4#S4.T5), in the long-tailed semi-supervised setting, SSL methods are designed to ensure that the learning process effectively captures all classes, even when the distribution is dominated by a few head classes. However, as shown in Table [1](https://arxiv.org/html/2505.13317v4#S4.T1), their performance often declines in the same datasets compared to standard SSL, highlighting their ongoing vulnerability to the challenges of long-tailed distributions. In contrast, the evaluation results of pre-trained models underscore their efficiency, which can be fine-tuned with only a small amount of labeled data and still perform well, even on low-resolution datasets, a common challenge for these models.

We also conducted experiments about STL-10 and Imagenet-127 datasets (detailed results are shown in Appendix [C](https://arxiv.org/html/2505.13317v4#A3)), and concluded that pre-trained models perform well on imbalanced datasets but struggle with semantically coarse tasks due to ambiguous class definitions. Although downstream fine-tuning can partially mitigate this issue, the underlying semantic constraints remain. We provide a more in-depth explanation and visualization of this phenomenon in Appendix [C](https://arxiv.org/html/2505.13317v4#A3). Ultimately, we concluded that the highly abstract nature of downstream task categories inherently leads to “adaptation bias”, a fundamental issue that negatively impacts the performance of VLMs.

### 4.6 Comparisons between VLMs and MLLMs

Datasets and Experimental Setup. We compare the performance of CLIP with existing MLLM approaches such as Qwen and GPT-4o-mini on the CIFAR-10, STL-10, ImageNet-30, and ImageNet-100 datasets. The detailed prompts for MLLMs are provided in Appendix [D](https://arxiv.org/html/2505.13317v4#A4).

Experiment Results. Intuitively, increasing the number of model parametercan improve classification accuracy; however, this phenomenon is primarily observed in CLIP models. For MLLMs, which are generally assumed to handle image classification tasks with ease, often underperform compared to CLIP, and simply increasing the model size can even lead to performance degradation (see Table [6](https://arxiv.org/html/2505.13317v4#S4.T6)). In contrast, GPT-4o-min demonstrates strong performance on these datasets, suggesting that its developers have explicitly enhanced the visual capability of its encoder. Regarding the observation that larger-parameter MLLMs do not consistently outperform CLIP-based models in image classification tasks, our case studies in Appendix [D](https://arxiv.org/html/2505.13317v4#A4) reveal that MLLMs often fail to identify which subjects in an image to focus on. This can be attributed to their autoregressive training paradigm, which is less conducive to effective vision-language alignment. Taken together, our analysis indicates that while scaling up parameters enhances a model’s training capacity, it does not necessarily improve classification ability—a factor that has been overlooked in prior Qwen research.

**Table 6: Classification performance comparison between VLMs and MLLMs.**
| Method | CIFAR-10 | ImageNet-30 | STL-10 | ImageNet-100 |
| --- | --- | --- | --- | --- |
| CLIP-ViT-B/16 | 79.30 | 98.40 | 97.40 | 87.70 |
| CLIP-ViT-L/14 | 86.80 | 99.31 | 98.90 | 89.70 |
| Qwen2.5-VL-7B-Instruct | 74.51 | 97.63 | 92.27 | 65.92 |
| Qwen2.5-VL-32B-Instruct | 67.06 | 99.00 | 87.52 | 71.60 |
| GPT-4o-mini | 89.82 | 99.43 | 98.31 | 85.36 |

## 5 Conclusions and Discussions

Conclusions We conduct a comprehensive and fair comparison of the strengths and weaknesses of SSL methods and pre-trained models. The key insights are as follows:

- 1.
SSL methods suffer from high training cost overhead and usually suffer lower performance (Takeaway 1). Moreover, they suffer from severe performance degradation in open environments, such as class space and distribution changes between labeled and unlabeled data, whereas pre-trained models generalize well owing to their pre-trained knowledge (Takeaway 2, 3).
- 2.
Though fine-tuning pre-trained models typically enable rapid performance gains, they may suffer from “adaptation bias” when downstream task data differs from pre-trained data distribution (Takeaway 1, 4). For example, the CLIP model, which is pre-trained on data with $224\times 224$ resolutions, achieves poor performance for low-resolution data, such as CIFAR-10.

**Table 7: Performance of representative methods in the fusion paradigm on the Flowers102 dataset.**
| Method | Domain Data | SSL | Open-World |
| --- | --- | --- | --- |
| CLIP Radford et al. (2021) | Zero-shot | 72.08 | 77.80 |
| FPL (Menghini et al., 2023) | Few-shot<br>+<br>Unlabeled | 75.96 | 80.97 |
| IFPL (Menghini et al., 2023) | 78.68 | 82.08 |  |
| GRIP (Menghini et al., 2023) | 83.60 | 86.26 |  |
| PromptKD (Li et al., 2024) | 98.23 | 86.27 |  |
| CPL (Zhang et al., 2024) | 89.66 | 87.35 |  |

Discussion 1: Potential for combining SSL and pre-trained models.
Integrating SSL with pre-trained models presents a promising direction for enhancing classification performance, combining the ability to exploit unlabeled data with strong generalization and zero-shot capabilities. This synergistic effect will help tackle challenges such as few-shot learning, class imbalance, and other domain-specific scenarios.
For example, GRIP (Menghini et al., 2023) and CPL (Zhang et al., 2024) bridge SSL methods and VLMs through a pseudo-label-based fine-tuning strategy. Futhermore, PromptKD (Li et al., 2024) introduces a two-stage framework that distills knowledge from teacher to student via learnable prompts and unlabeled data.
As shown in Table [7](https://arxiv.org/html/2505.13317v4#S5.T7), incorporating unlabeled data and using labels generated by pre-trained models not only leverages the rich prior knowledge embedded in pre-trained models, but also enables semi-supervised learning algorithms to effectively mine domain-specific knowledge from unlabeled data, thereby achieving substantial improvements in model performance and generalization.

Discussion 2: Effects of training with long-tailed data on pre-trained models.
When training data follows a long-tailed distribution, excessive fine-tuning of pre-trained models can be harmful, particularly degrading performance on tail classes. Aggressive fine-tuning can disrupt the pre-trained class-conditional structure and weaken rare category representations (Shi et al., 2024). To mitigate this, LIFT fine-tunes only a small part of the model, enabling adaptation to long-tailed distributions while avoiding catastrophic forgetting.
However, most existing approaches focus on reducing fine-tuning intensity, often at the cost of head class performance. This highlights the urgent need for new strategies that can adapt to imbalanced data while minimizing learning bias.

## 6 Recommendations for Future Research

Facing label scarcity, choosing between SSL methods and pre-trained models is a critical decision. Based on our practical considerations, here are our recommendations:

One should first attempt to address the problem using pre-trained models along with fine-tuning strategies. If “adaptation bias” arises, it is advisable to incorporate additional pseudo-labeling to facilitate adaptation. If these approaches prove ineffective and the discrepancy between training and testing classes is relatively small, then it may be beneficial to design a task-specific SSL method—either a traditional SSL approach or one that leverages pre-trained models for pseudo-label generation—to train a specialized model for the downstream task. Our study also highlights five promising directions for future research:

- •
Datasets: Incorporate higher-resolution and more challenging benchmarks to better assess SSL methods and pre-trained models, aligning with the steady rise in image resolution.
- •
SSL methods: Future work on image-classification SSL should include head-to-head comparisons with strong pre-trained baselines under matched conditions (e.g., identical data, resolution, augmentation, and compute).
- •
Pre-trained models: Current pre-trained models remain sensitive to distribution shift (e.g., resolution mismatches between pre-training and testing data). More research is needed on robust adaptation strategies and evaluation protocols to mitigate these effects.
- •
MLLMs Although MLLMs demonstrate strong semantic understanding capabilities, their limitations in visual encoders lead to challenges in recognizing low-resolution images, which remains an important direction for future research.
- •
General directions: Explore principled integrations of SSL with adaption of pre-trained models to enhance performance in label-scarce settings.

One limitation of our study is that, since most SSL methods are primarily designed for image classification tasks, the range of experimental settings we compared is relatively narrow. In future work, we plan to explore the integration of these two approaches (e.g., leveraging the zero-shot classification capability of VLMs for pseudo-labeling and iterative training) and extend our research to broader multimodal scenarios.

#### Ethics Statement

This work aims to investigate the performance of existing SSL methods and pretrained models. It does not involve human subjects, sensitive personal data. All datasets used are publicly available, and we adhere to the ICLR Code of Ethics.

#### Reproducibility Statement

We have taken extensive measures to ensure the reproducibility of our results. An anonymous code repository is available at [https://anonymous.4open.science/r/Rethinking-SSL-and-Pretrain-Finetuning-5566/](https://anonymous.4open.science/r/Rethinking-SSL-and-Pretrain-Finetuning-5566/), containing implementations of code and running scripts. Details of experimental settings are described in Section [4](https://arxiv.org/html/2505.13317v4#S4) and Appendix [B](https://arxiv.org/html/2505.13317v4#A2). All datasets used are publicly available.

## References

- Bai et al. (2023)
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.
*arXiv preprint arXiv:2308.12966*, 2023.
- Ban et al. (2024)
Seonghyun Ban, Heesan Kong, and Kee-Eung Kim.
Data augmentation with diffusion for open-set semi-supervised learning.
*Advances in Neural Information Processing Systems*, 37:64970–64995, 2024.
- Berthelot et al. (2019a)
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.
Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.
*arXiv preprint arXiv:1911.09785*, 2019a.
- Berthelot et al. (2019b)
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.
Mixmatch: A holistic approach to semi-supervised learning.
*Advances in neural information processing systems*, 32, 2019b.
- Cao et al. (2021)
Kaidi Cao, Maria Brbic, and Jure Leskovec.
Open-world semi-supervised learning.
*arXiv preprint arXiv:2102.03526*, 2021.
- Coates et al. (2011)
Adam Coates, Andrew Ng, and Honglak Lee.
An analysis of single-layer networks in unsupervised feature learning.
In *Proceedings of the 40th international conference on artificial intelligence and statistics*, pp. 215–223, 2011.
- Deng et al. (2009)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
ImageNet: a large-scale hierarchical image database.
In *Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition*, pp. 248–255, 2009.
- Fan et al. (2022)
Yue Fan, Dengxin Dai, Anna Kukleva, and Bernt Schiele.
Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 14574–14584, 2022.
- Geirhos et al. (2018)
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.
Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.
In *International conference on learning representations*, 2018.
- He et al. (2016)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 770–778, 2016.
- Heinrich et al. (2024)
Greg Heinrich, Mike Ranzinger, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, Pavlo Molchanov, et al.
Radio amplified: Improved baselines for agglomerative vision foundation models.
*arXiv e-prints*, pp. arXiv–2412, 2024.
- Hendrycks et al. (2019)
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song.
Using self-supervised learning can improve model robustness and uncertainty.
*Advances in neural information processing systems*, 32, 2019.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Khattak et al. (2023a)
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.
MaPLe: Multi-modal prompt learning.
In *Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 19113–19122, 2023a.
- Khattak et al. (2023b)
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan.
Self-regulating prompts: Foundational model adaptation without forgetting.
In *The 2023 IEEE/CVF International Conference on Computer Vision*, pp. 15144–15154, 2023b.
- Kim et al. (2020)
Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin.
Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning.
*Advances in neural information processing systems*, 33:14567–14579, 2020.
- Krizhevsky et al. (2009)
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.
2009.
- Kukleva et al. (2023)
Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht.
Temperature schedules for self-supervised contrastive methods on long-tail data.
2023.
- Le & Yang (2015)
Yann Le and Xuan Yang.
Tiny imagenet visual recognition challenge.
*CS 231N*, 2015.
- Lee et al. (2013a)
Dong-Hyun Lee et al.
Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.
In *Workshop on challenges in representation learning, ICML*, pp. 896, 2013a.
- Lee et al. (2013b)
Dong-Hyun Lee et al.
Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.
In *Workshop on challenges in representation learning, ICML*, pp. 896, 2013b.
- Lee et al. (2025)
Jungsoo Lee, Debasmit Das, Munawar Hayat, Sungha Choi, Kyuwoong Hwang, and Fatih Porikli.
Customkd: Customizing large vision foundation for edge model improvement via knowledge distillation.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 25176–25186, 2025.
- Li et al. (2024)
Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang.
Promptkd: Unsupervised prompt distillation for vision-language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 26617–26626, 2024.
- Ma et al. (2024)
Chengcheng Ma, Ismail Elezi, Jiankang Deng, Weiming Dong, and Changsheng Xu.
Three heads are better than one: Complementary experts for long-tailed semi-supervised learning.
In *Proceedings of the AAAI Conference on Artificial Intelligence 38*, pp. 14229–14237, 2024.
- Menghini et al. (2023)
Cristina Menghini, Andrew Delworth, and Stephen Bach.
Enhancing clip with clip: Exploring pseudolabeling for limited-label prompt tuning.
*Advances in Neural Information Processing Systems*, 36:60984–61007, 2023.
- Niu et al. (2024)
Shengjie Niu, Lifan Lin, Jian Huang, and Chao Wang.
Owmatch: Conditional self-labeling with consistency for open-world semi-supervised learning.
*Advances in Neural Information Processing Systems 37*, pp. 99836–99866, 2024.
- Oh et al. (2022)
Youngtaek Oh, Dong-Jin Kim, and In So Kweon.
Daso: Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 9786–9796, 2022.
- Oliver et al. (2018)
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow.
Realistic evaluation of deep semi-supervised learning algorithms.
*Advances in neural information processing systems*, 31, 2018.
- Pham et al. (2021)
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le.
Meta pseudo labels.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11557–11568, 2021.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In *Proceedings of the 38th International Conference on Machine Learning*, pp. 8748–8763, 2021.
- Rizve et al. (2022)
Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah.
Towards realistic semi-supervised learning.
In *European conference on computer vision*, pp. 437–455, 2022.
- Shi et al. (2024)
Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin-Yan Han, and Yufeng Li.
Long-tail learning with foundation model: Heavy fine-tuning hurts.
In *Proceedings of the 41th International Conference on Machine Learning*, 2024.
- Sohn et al. (2020)
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.
Fixmatch: Simplifying semi-supervised learning with consistency and confidence.
*Advances in neural information processing systems*, pp. 596–608, 2020.
- Wei et al. (2021)
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang.
Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 10857–10866, 2021.
- Wei & Gan (2023)
Tong Wei and Kai Gan.
Towards realistic long-tailed semi-supervised learning: Consistency is all you need.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 3469–3478, 2023.
- Yang et al. (2022)
Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu.
A survey on deep semi-supervised learning.
*IEEE transactions on knowledge and data engineering*, 35(9):8934–8954, 2022.
- Zhang et al. (2024)
Jiahan Zhang, Qi Wei, Feng Liu, and Lei Feng.
Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data.
In *Forty-First International Conference on Machine Learning*, 2024.
- Zhou et al. (2023)
Andy Zhou, Jindong Wang, Yu-Xiong Wang, and Haohan Wang.
Distilling out-of-distribution robustness from vision-language foundation models.
*Advances in Neural Information Processing Systems*, 36:32938–32957, 2023.
- Zhou et al. (2022a)
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.
Conditional prompt learning for vision-language models.
In *Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 16816–16825, 2022a.
- Zhou et al. (2022b)
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.
Learning to prompt for vision-language models.
*International Journal of Computer Vision*, pp. 2337–2348, 2022b.
- Zhou et al. (2024)
Zi-Hao Zhou, Siyuan Fang, Zi-Jing Zhou, Tong Wei, Yuanyu Wan, and Min-Ling Zhang.
Continuous contrastive learning for long-tailed semi-supervised recognition.
*arXiv preprint arXiv:2410.06109*, 2024.
- Zoph et al. (2020)
Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le.
Rethinking pre-training and self-training.
*Advances in neural information processing systems*, 33:3833–3845, 2020.

## Appendix A Use of Large Language Models (LLMs)

In this work, LLMs were used solely for language polishing and translation; they did not contribute to research ideation, experiment design, or result interpretation. Multimodal LLMs (e.g., Qwen, GPT-4o-min) were employed only as baselines in our experiments. The authors take full responsibility for all content and results.

## Appendix B Detailed Experimental Setup

### B.1 Overal Experimental Setup

The overall VLM setup experiments are all conducted on a single NVIDIA A800 GPU, and the SSL method directly reported the results of the most recognized articles.

### B.2 Detailed Semi-supervised Datasets and Experimental Setup

Datasets Setup.
We evaluate the performance of both classical SSL methods and VLMs on standard image classification benchmarks: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Deng et al., 2009). Each dataset is tested under multiple supervision levels, typically using 40, 250, or 4000 labeled samples depending on the benchmark. We also consider different augmentation strategies (e.g., weak/strong) following the standard SSL protocol.

Experimental Setup.
For classical SSL baselines (FixMatch (Sohn et al., 2020), MixMatch (Berthelot et al., 2019b), ReMixMatch (Berthelot et al., 2019a)), models are pre-trained on the labeled subset and optimized using a combination of supervised and consistency-based unsupervised losses. We directly use results reported in the FixMatch paper to maintain consistency. The unlabeled data is drawn from the remaining training set without labels.
For Vision-Language Models (VLMs), we adopt the ViT-B/16 architecture across all experiments. Since VLMs are pre-trained on large-scale image-text pairs, we do not require any additional unlabeled samples. Instead, we fine-tune them using only the labeled portion of the data, treating the task as a pure few-shot classification problem. We use prompt tuning methods such as CoOp and PromptSRC (depending on the experiment), and all models are trained for 50 epochs using the labeled data only. Hyperparameters are kept consistent with those used in the Base-to-New setting. For CoOp, we follow the settings reported in the original paper: 4-shot training for 100 epochs and 25/100/400-shot training for 200 epochs. PromptSRC strictly follows the configurations reported in its paper, with key parameters as follows: both textual and visual prompts consist of 4 tokens, and the first nine transformer layers are modified with learnable prompt tokens. All PromptSRC experiments are run for 20 epochs. Experiments are conducted on a single NVIDIA A800 GPU.

### B.3 Detailed Open-set Semi-supervised Datasets and Experimental Setup

Dataset Setup. We follow commonly used OSSL benchmarks, including CIFAR-10/100, ImageNet-30, and ImageNet-100. A detailed description of these datasets is provided below:

- •
CIFAR-10/100. This task uses CIFAR-10 as the labeled dataset and CIFAR-100 as the unlabeled dataset. While the entire CIFAR-100 dataset serves as the unlabeled portion, we sample 100 images per class from CIFAR-10 to simulate a scarce-label scenario. For our VLMs, since training is still conducted solely on the labeled CIFAR-10 data in a few-shot manner, the results reflect a 100-shot training setup for CIFAR-10 under the SSL setting.
- •
ImageNet-30. This task uses the ImageNet-30 dataset (Hendrycks et al., 2019), a 30-class subset of ImageNet. Following Pham et al. (2021), we select 5% of the data (approximately 50 images per class) from the first 20 alphabetically sorted classes as labeled data, and treat the rest as unlabeled. For VLMs, the minimum number of samples among these 50 classes is 754; 10% of that equals roughly 70. To maintain consistency and simplify comparisons, we fix the labeled set to 50-shot per class for fine-tuning.
- •
ImageNet-100. This task is based on the ImageNet-100 dataset, a 100-class subset of ImageNet as defined in Cao et al. (2021). The classes are split evenly into 50% in-distribution (ID) and 50% out-of-distribution (OOD) based on alphabetical order. For the labeled data, we select 50 images per class from the first 20 ID classes, while the remaining data from all 100 classes is used as unlabeled data. VLMs are fine-tuned using only the 50-shot labeled data from the first 20 ID classes, and evaluated on the corresponding test set following the same setup.

Experimental Setup. Since VLMs do not utilize unlabeled data, the Open-set semi-supervised setting on above datasets essentially becomes a few-shot learning problem with a reduced number of classes. The results of OSSL methods refer to DWD-SL (Ban et al., 2024).

### B.4 Detailed Open-World Semi-Supervised Datasets and Experimental Setup

Dataset Setup. We evaluate our approach on CIFAR-10/100 (Krizhevsky et al., 2009), ImageNet-100 (Cao et al., 2021), and Tiny ImageNet (Le & Yang, 2015). Specifically, the ImageNet-100 dataset contains 100 classes sub-sampled from ImageNet-1k following the selection in Deng et al. (2009). For all datasets, we split classes into seen and new categories with a predefined new class ratio (defaulting to 50% unless otherwise specified). From the seen classes, a fixed ratio of data is randomly selected as labeled data, while the remaining seen samples, together with all samples from the new classes, are assigned to the unlabeled set.

Experimental Setup.
Under the Base-to-New setting commonly used for evaluating VLMs (Khattak et al., 2023b; Zhou et al., 2022b), each dataset’s classes are split evenly into base and new groups. VLMs are trained exclusively on the base classes and then evaluated separately on both base and new classes to assess their generalizability. OWSSL setting closely aligns with the Base-to-New setting, but differs in how unseen (new) classes are treated during training: SSL methods assume the unlabeled pool contains OOD (new) samples and employ clustering or OOD-detection mechanisms to identify and leverage them. While VLMs use only labeled base-class data, without any exposure to OOD examples. At evaluation time, VLMs simply rely on their pre-trained image–text alignment to generalize to new categories, whereas OWSSL methods benefit from the self-label strategy during training (Niu et al., 2024; Rizve et al., 2022).

We report OWSSL results using OwMatch (Niu et al., 2024). For VLMs, we adopt ViT-B/16 backbones across all variants. CoOp and PromptSRC are fine-tuned using only the labeled data under varying label ratios (e.g., 1%, 5%, 10%), and are trained for 50 epochs following Khattak et al. (2023b); Zhou et al. (2022b). All other hyperparameters remain consistent across datasets and methods unless otherwise stated.

### B.5 Detailed Long-Tailed Datasets and Experimental Setup

Dataset Setup.
We evaluate methods on widely-used long-tailed SSL benchmarks: CIFAR-10-LT, CIFAR-100-LT (Krizhevsky et al., 2009), STL-10-LT (Coates et al., 2011), and ImageNet-127 (Fan et al., 2022). For synthetic datasets, the number of samples for each class $c$ is defined as $N_{c}=N_{1}\cdot\gamma_{l}^{-(c-1)/(C-1)}$, where $N_{1}$ is the number of samples in the most frequent class and $\gamma_{l}$ is the imbalance ratio. Unlabeled data follows a similar distribution with $\gamma_{u}$.

- •
CIFAR-10-LT:
Following DASO (Wei & Gan, 2023), we adopt $(N_{1},M_{1})\in\{(500,4000),(1500,3000)\}$ and $\gamma_{l}=\gamma_{u}\in\{100,150\}$. We also consider uniform/reversed unlabeled distributions with $\gamma_{u}\in\{1,1/100\}$ and $\gamma_{l}=100$ fixed. Based on the minimum labeled class size, we define the corresponding few-shot settings for VLMs:
–
5 shots for $(\gamma=100,\ N_{1}=500)$
–
15 shots for $(\gamma=100,\ N_{1}=1500)$
–
4 shots for $(\gamma=150,\ N_{1}=500)$
–
20 shots for $(\gamma=150,\ N_{1}=3000)$
- •
CIFAR-100-LT:
We adopt $(N_{1},M_{1})\in\{(50,400),(150,300)\}$ with $\gamma_{l}=\gamma_{u}\in\{10,20\}$.
The corresponding shots for VLMs are:
–
5 shots for $(\gamma=10,\ N_{1}=50)$
–
15 shots for $(\gamma=10,\ N_{1}=150)$
–
3 shots for $(\gamma=20,\ N_{1}=50)$
–
8 shots for $(\gamma=20,\ N_{1}=150)$
- •
STL-10-LT:
We only control labeled imbalance ratio with $\gamma_{l}\in\{10,20\}$, as unlabeled data lacks class labels.
- •
ImageNet-127:
This is a real-world long-tailed dataset with naturally imbalanced label distribution. Following CCL (Zhou et al., 2024), we sample 10% of training data for the labeled set. Due to this sampling strategy, there is no fixed $\gamma$ value.

Experimental Setup.
Since the LTSSL setting only affects the training set distribution, we evaluate all models on the same balanced test sets as prior work. For example, in CIFAR-10-LT with $\gamma_{l}=\gamma_{u}=100$, $N_{1}=500$ implies that the rarest class has only 5 labeled samples. For VLMs, we treat this as a 5-shot setup and do not use any unlabeled data. All VLMs follow the same architecture (ViT-B/16), hyperparameters, and optimization schedules as in the Section about SSL.

## Appendix C SSL vs. VLM Finetuning in Long-Tailed Semi-Supervised Setup

**Table 8: Test accuracy under inconsistent imbalance ratio setting ($\gamma_{l}\neq\gamma_{u}$) on STL10-LT and ImageNet-127 datasets. $\gamma=10/20$ for STL10-LT dataset.**
| Method | STL10-LT | STL10-LT | ImageNet-127 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\gamma=10$ | $\gamma=20$ | $\gamma=\text{None}$ |  |  |  |  |
| $N_{1}=150$<br>$M=100k$ | $N_{1}=450$<br>$M=100k$ | $N_{1}=150$<br>$M=100k$ | $N_{1}=450$<br>$M=100k$ | 32 $\times$ 32 | 64 $\times$ 64 |  |
| FixMatch | 56.1 $\pm$ 2.32 | 72.4 $\pm$ 0.71 | 47.6 $\pm$ 4.87 | 64.0 $\pm$ 0.27 | 29.7 | 42.3 |
| w/ DARP (Kim et al., 2020) | 66.9 $\pm$ 1.66 | 75.4 $\pm$ 1.05 | 59.9 $\pm$ 2.17 | 72.3 $\pm$ 0.60 | 30.5 | 42.5 |
| w/ CReST+ (Wei et al., 2021) | 56.0 $\pm$ 3.13 | 71.5 $\pm$ 0.96 | 55.4 $\pm$ 3.03 | 68.5 $\pm$ 1.15 | 32.5 | 44.7 |
| w/ DASO (Oh et al., 2022) | 61.3 $\pm$ 1.84 | 78.4 $\pm$ 0.96 | 68.5 $\pm$ 1.78 | 75.3 $\pm$ 0.44 | 40.9 | 55.9 |
| w/ ACR (Wei & Gan, 2023) | 70.2 $\pm$ 3.04 | 83.0 $\pm$ 1.40 | 75.1 $\pm$ 1.81 | 80.5 $\pm$ 0.25 | 57.2 | 63.6 |
| w/ CPE (Ma et al., 2024) | 69.6 $\pm$ 0.46 | 83.0 $\pm$ 1.14 | 69.6 $\pm$ 0.46 | 81.7 $\pm$ 0.34 | - | - |
| w/ CCL (Zhou et al., 2024) | 79.1 $\pm$ 0.43 | 84.8 $\pm$ 1.05 | 77.1 $\pm$ 0.33 | 83.1 $\pm$ 0.18 | 61.5 | 67.8 |
| ZSCLIP (Radford et al., 2021) | 97.4 $\pm$ 0.00 | 46.00 $\pm$ 0.00 |  |  |  |  |
| CoOp (Zhou et al., 2022b) | 97.77 $\pm$ 0.19 | 98.13 $\pm$ 0.17 | 97.73 $\pm$ 0.25 | 97.83 $\pm$ 0.09 | 49.83 $\pm$ 1.04 | 55.23 $\pm$ 0.50 |
| PromptSRC (Khattak et al., 2023b) | 98.27 $\pm$ 0.05 | 98.47 $\pm$ 0.05 | 98.13 $\pm$ 0.12 | 98.30 $\pm$ 0.08 | 60.33 $\pm$ 0.17 | 64.37 $\pm$ 0.34 |

Experiment Results.
As shown in Table [8](https://arxiv.org/html/2505.13317v4#A3.T8), pre-trained models perform well on highly imbalanced datasets like STL-10 but struggle on semantically coarse datasets such as ImageNet-127 (Fan et al., 2022).
ImageNet-127 is obtained by collapsing the 1,000 fine-grained ImageNet-1K classes into 127 coarse-grained categories according to their positions in the WordNet semantic hierarchy, yielding a long-tailed distribution (e.g., grouping “abacus,” “acoustic guitar,” and “analog clock” under the broader category “device”). Zero-shot CLIP evaluations reveal that this hierarchical coarsening injects semantic ambiguity, making it challenging for pretrained VLMs to confidently associate a fine-grained concept with a specific image.
Although downstream fine-tuning can partially mitigate this issue, the underlying semantic constraints remain. By contrast, SSL learns directly from neural representations rather than predefined semantics, making it less vulnerable to such ambiguity and better suited to tasks with limited or ambiguous class structure.

Detailed Analysis and Visualization of ImageNet-127.
ImageNet-127 compresses the original 1,000 fine-grained ImageNet classes into 127 higher-level categories, inevitably producing a long-tailed distribution. For example, 124 distinct classes—including *abacus*, *acoustic guitar*, and *analog clock*—are merged into the super-category device, whereas a single class such as *bubble* constitutes the entire globule category. Thus, categories that absorb many fine-grained classes become *head* classes, while those represented by only one class form the *tail*.

This hierarchical regrouping introduces substantial semantic ambiguity: the pre-training stage of VLMs fails to adequately encode the latent hierarchy, resulting in frequent misalignments, as shown in Figure [4(a)](https://arxiv.org/html/2505.13317v4#A3.F4.sf1). Although fine-tuning enables pre-trained VLMs to better bridge these granularity gaps and exploit limited downstream labels, the fundamental semantic imbalance remains, as further illustrated in Figure [4(b)](https://arxiv.org/html/2505.13317v4#A3.F4.sf2).

Figure: (a) Zero-shot alignment of fine-grained and coarse-grained category embeddings.
Refer to caption: x2.png

## Appendix D Detailed Analysis and Visualization of MLLMs

We present several misclassification cases of GPT-4o-mini on ImageNet-100 along with the corresponding prompts. For example, the model mislabels a “poke bonnet” as a “sarong.” In these four cases, GPT-4o-mini is clearly distracted by secondary scene cues (e.g., the ship in Figure [5](https://arxiv.org/html/2505.13317v4#A4.F5) (b)) and performs poorly when recognizing blurred targets. Furthermore, as the number of candidate classes increases, distractors consistent with alternative categories more easily divert the model’s attention away from the true object regions, amplifying errors. These observations suggest that, although GPT-4o-min exhibits strong recognition capabilities, its autoregressive training paradigm often hinders accurate identification of the primary target in complex multi-class scenarios, leading to misclassifications. More specifically, since autoregressive training represents both images and categories as token sequences, the model struggles to align image tokens with text tokens, making it prone to assigning labels corresponding to secondary regions of the image. As shown in Figure [6](https://arxiv.org/html/2505.13317v4#A4.F6), the model disregards the main object, the pedestal, and instead focuses on the obelisk. Decoupling perception from cognition thus appears to be a promising avenue for enhancing classification robustness and, more broadly, the reasoning capabilities of multimodal large language models. Additionally, we prompted the GPT-4o-mini to provide the classification for Figure [5](https://arxiv.org/html/2505.13317v4#A4.F5) (b) using long-chain reasoning. As the results indicate, the model sometimes relies solely on learned knowledge to infer the image category, making it prone to errors. The generated reply is as follows:

Figure: (a) True: Foxhound Pred: American coot
Refer to caption: Figures/English_foxhound_American_coot.jpeg

Figure: Figure 6: Average attention visualization of “pedestal”. Darker colors indicate higher attention.
Refer to caption: Figures/7.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 2
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: The paper presents comprehensive experiments.
summary: This paper investigates traditional self-supervised learning (SSL) methods and CLIP-based prompt-tuning approaches. The authors conduct experiments across various image classification settings and present several empirical findings.
weaknesses: 1. This paper focuses solely on experimental comparisons without proposing any new method. Therefore, the experimental setup is critical and should be described in detail in the main text.

2. The comparison settings are entirely unfair. For example, traditional SSL methods use ResNet backbones (e.g., ResNet-28-2), whereas prompt-tuning methods use ViT/B-16. Moreover, comparing SSL and few-shot learning is conceptually unsound. SSL with CLIP has already been explored, and under such unfair experimental conditions, the conclusions drawn from the experiments are not meaningful.

3. Since CLIP is trained on diverse web-scale data, its robustness to OOD (out-of-distribution) data is expected and not surprising.

4. Recent studies [1] have shown that MLLMs perform better in image classification tasks using no-thinking or direct-answering prompts. It would be more informative to report the performance of MLLMs under such settings, as this would better reflect their true classification capabilities.

5. CLIP models can also be used for SSL settings. For example, [2]. It is questionable why comparing few-shot CLIP-based prompting methods with ResNet-based SSL methods. 

6. Overall, the contribution of this paper appears limited. The authors merely conduct inference-based experiments and report results that are largely unsurprising. There are no technical or methodological innovations.

Reference:
[1] To Think or Not To Think: A Study of Thinking in Rule-Based Visual Reinforcement Fine-Tuning. NeurIPS 2025.

[2] Candidate pseudolabel learning: enhancing visionlanguage models by prompt tuning with unlabeled data. ICML 2024.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: NA
soundness: 3
strengths: This is a timely study in the era of large models and has the potential to deliver useful insight for the SSL community. The experiments are relatively comprehensive, covering main settings in SSL research. Takeaways are clearly stated.
summary: The paper presents a controlled comparison of classic semi-supervised learning (SSL) methods versus pre-trained vision–language models (VLMs) across four regimes: standard SSL, open-set, open-world, and long-tailed classification. Under matched supervision budgets, the study finds that VLMs (e.g., CLIP with prompt/adapter tuning) often match or surpass SSL in accuracy while being more compute-efficient.
weaknesses: The benchmark datasets—CIFAR-10/100, STL-10, and ImageNet—may overlap semantically with VLM pretraining data, potentially advantaging the VLMs in this comparison. By contrast, in domains where the pre-trained model has seen less data (e.g., medical imaging), it is unclear whether the conclusions would still hold. Experiments probing this low-overlap setting are currently missing.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - For open-set and open-world SSL learning, is the method using pre-trained models the same as standard semi-supervised learning since it does not use unlabeled data?
soundness: 4
strengths: - The problem studied, i.e., comparing unlabeled data and pre-trained models, is interesting and important to the literature.

- The experiments are comprehensive and cover various semi-supervised learning settings, including standard, open-set, open-world, and long-tailed semi-supervised learning. Therefore, the conclusions are convincing.

- The writing is excellent, making the paper easy to understand.

- This paper is an important position paper for the field of weakly supervised learning.
summary: This paper compares unlabeled data and pre-trained models with a limited amount of labeled data. Various semi-supervised learning scenarios were considered, including semi-, open-set, open-world, and long-tailed semi-supervised learning. Based on extensive experimental results, valuable conclusions have been drawn, showing that pre-trained models can often perform better, though they still face many limitations and require further research.
weaknesses: - There is a lack of related work discussing the use of zero-shot CLIP for traditional weakly supervised learning, a more general form of semi-supervised learning. Additionally, some work discusses semi-supervised learning with CLIP.

- The title would be better with "semi-supervised learning" instead of "SSL" since "SSL" has multiple meanings, such as self-supervised learning.

- In experiments on standard semi-supervised learning, SOTA SSL algorithms such as FixMatch seem to achieve better performance on CIFAR-10 and CIFAR-100. Therefore, concluding that pre-trained models are better for standard semi-supervised learning is incomplete.

[1] Delving into Weakly Supervised Learning with Pre-Trained Models, https://openreview.net/forum?id=RgWATMmWmz


Decision: Reject