Title: Revisiting Semi-Supervised Learning in the Era of Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: semi-supervised learning

self-training

vision foundation models

foundation models

benchmark

parameter-efficient fine-tuning
Abstract: Semi-supervised learning (SSL) enhances model performance by leveraging abundant unlabeled data alongside limited labeled data. As vision foundation models (VFMs) become central to modern vision applications, this paper revisits SSL in the context of these powerful pre-trained models. We conduct a systematic study on tasks where frozen VFMs underperform and reveal several key insights when fine-tuning them. First, parameter-efficient fine-tuning (PEFT) using only labeled data often surpasses traditional SSL methods---even without access to unlabeled data.  Second, pseudo-labels generated by PEFT models offer valuable supervisory signals for unlabeled data, and different PEFT techniques yield complementary pseudo-labels. These findings motivate a simple yet effective SSL baseline for the VFM era: \emph{ensemble pseudo-labeling across diverse PEFT methods and VFM backbones}.  Extensive experiments validate the effectiveness of this approach, offering actionable insights into SSL with VFMs and paving the way for more scalable and robust semi-supervised learning in the foundation model era.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFoundation models are often used as backbones for downstream tasks, like classification. When applied to bespoke classification datasets, baseline performance may not be as high as users require. In these settings, it is common that users have a small set of labelled data and much more unlabelled data, making it suitable for semi-supervised learning.\n\nThe authors construct an evaluation dataset with classification tasks in which existing foundation models (DINOv2 and CLIP) perform poorly (30-70% accuracy).\n\nThe authors explore if existing semi-supervised learning methods (FixMatch, FlexMatch, SoftMatch) can be used to fine-tune foundation models to improve their performance on these bespoke datasets. When fine-tuning the models, the authors test both full parameter fine-tuning and parameter-efficient fine-tuning (LORA and Adaptformer). \n\nThe authors find that:\n- the simple approach, full parameter fine-tuning on just a few labelled data samples is very effective and existing SSL methods do **not** confer an advantage.\n- parameter-efficient fine-tuning improves performance of all training methods, again providing no advantage to SSL methods which make use of extra unlabelled data.\n\nThe authors propose a new method for SSL on foundation models that uses parameter efficient self-training and ensembling to improve model performance. They use a small amount of labelled data to fine-tune the model and use the resulting model to generate pseudo-labels for self-training. They ensemble several base models and parameter-efficient fine-tuning methods. They try three different supervisory signals leveraging the ensembling: (Mean Logits, Mean Probs, and Mean Labels). They empirically demonstrate that their approach with Mean Labels performs the best on their evaluation dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The problem choice is important. Many users try to apply foundation models to specific problems and find mixed results. Establishing a benchmark and methods for this specific setting is worthwhile.\n- The paper is written clearly. It is easy to follow the investigation of prior work and the intuition behind their proposed approach.\n- The authors ablate various components of their algorithm to precisely understand the contributions of each component. They also explore several options for each step. They choose recent methods for their baseliens and conduct hyperparameter sweeps to try and achieve a fair comparison.\n-  The method they are proposing is a novel combination of established methodologies. \n\nWeaknesses:\n- While the evaluation benchmark is reasonable, it is still relatively limited in comparison to the diversity of tasks users may be interested in. \n- The methodology is fairly similar to the one proposed in FineSSL. The authors work does introduce two additional contributions: (1) more breadth in the evaluation tasks, backbones and baselines explored and (2) the idea to ensemble different methods to achieve better performance. \n\nMinor: \n- L113 - \"first\" comprehensive study of SSL in the era of foundation models is a broad claim. A more specific claim would be more appropriate. \n- L187 - \"few\" is ambiguous, refer to Table 2.\n- L240 - would be better to keep section header with the text it refers to.\n- L867-869 - potentially capitalize appendix?\n\nJustification: I recommend a 5 because the problem is important, the paper is technically sound, and presented well. I do not believe the results or method is ground-breaking, so I cannot recommend a higher score.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nThe authors test on classification and on 6 datasets, this is good evaluation benchmark but not comprehensive. It's addressed to some degree in the text, but not in a specific section independently.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem Semi-supervised Learning (SSL) in the context of foundation models, and found that standard SSL technique only incrementally benefits foundation models.To make unlabelled data pay off, the authors introduce V-PET—a one-shot self-training routine that simply averages the one-hot predictions of several differently fine-tuned backbones and then retrains on that soft target—showing consistent gains over prior SSL baselines on a new, harder VTAB-derived benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The problem being studied in this paper is very crucial and of high interest to the SSL community - what role does SSL play in the era of foundation models, and what sets it apart from classical SSL? \n\n2. The take-away message is informative and perhaps stirs further thinking in this domain - naively utilizing unlabeled data does not seems to contribute much for performance gain.\n\n3. The tested cases covered in the experiment are rather comprehensive and informative. \n\n4. This paper is overall well-written and easy to follow. \n\n**Weaknesses:**\n\n1. One major weakness of this paper appears to be the computational complexity of the proposed method; this method requires an ensemble of multiple foundation models, and considering the complexity of the foundation models, the computational cost of the proposed method seems significant. Moreover, efficiency studies are missing in this paper.\n\n2. More ablation studies are needed to further highlight the importance of self-training; authors mainly proposed two techniques, self-training and ensemble, and combining them together witness performance gain. However, such performance gain could solely stem from ensemble rather than self-training. \n\n3. The main insights from this paper are not significant enough - the resulting methodology essentially stems from ensemble and self-training, and all of these are not very relevant to the \"unique properties of foundation models.\" \n\n4. The coverage of techniques and tricks is not comprehensive enough; some other very important aspects, such as threshold of pseudo-label (FlexMatch, AdaMatch), data-augmentation and consistency training (FixMatch), exponential moving average, label alignment, and other techniques to counter training imbalance, are not discussed or mentioned. However, researchers in SSL might be interested in learning why or why those tricks still work for foundation models.\n\nQUESTIONS:\n1. I think another informative point authors could try to add is how foundation models' performances sclaes as we increase the number/proportion of the labeled data, this is expected to behave somewhat differently to standard SSL classifiers as they perhaps need more labeled data since they are trained from the scratch.\n\n2. Are foundation models more vulnerable to the problem of imbalance comparing to classifical SSL classifiers?\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the applicability of semi-supervised learning (SSL) to vision foundation models (VFMs), specifically CLIP and DINOv2. The paper systematically evaluates whether existing SSL methods remain effective when applied to VFMs. Based on findings, the paper proposes V-PET, a simple yet effective self-training SSL method that ensembles pseudo-labels generated across diverse parameter-efficient fine-tuning (PEFT) methods and VFM backbones.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper provides a thorough empirical study with extensive evaluations. The structure of the paper is clear, and it offers insights into SSL in the era of foundation models. While the proposed method is effective despite its conceptual simplicity, the paper offers limited theoretical insights into why ensemble pseudo-labeling is particularly effective in VFM settings.\n\nQUESTIONS:\n1. Beyond empirical observations, could you explain more clearly why ensemble pseudo-labeling works better than using individual PEFT pseudo-labels in VFM settings? While this is discussed in Figure 1 and lines 255–263, a clearer reason would strengthen the claims.\n\n2. In Table 3, what backbones and PEFT methods were used for the existing methods? Please clarify. If the same backbones and PEFT methods were used, your method’s training cost would increase with the number of PEFT methods and VFMs used. In that case, are there any additional benefits beyond accuracy, such as improved training stability? If different backbones or PEFT methods were used, reporting training time or FLOPs would help highlight the advantages of your approach.\n\n3. In Table 3, the proposed methods seem to show slightly lower performance on only some datasets (e.g., Resisc45 and Retinopathy). Could you explain why this happens?\n\nLIMITATIONS:\nIt would be helpful to explicitly discuss the limitations of the proposed method, particularly regarding its applicability beyond classification tasks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new pipeline for semi-supervised learning in the regime of pretrained (vision) foundation models (VFMs). They demonstrate several existing SSL methods (which utilize both labelled and unlabelled data for the fine-tuning task) perform worse than a baseline of only using the labelled subset of the data to fine-tune the model on new tasks. The authors then propose their own method, which is more involved but does benefit from unlabelled data, that involves using multiple low-rank fine-tuning methods to fine-tune multiple VFMs to pseudo-label the unlabelled samples and create an ensembled training target.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe work is well presented and addresses a timely problem. I believe the ideas and implementation are novel. For the most part, I am generally happy with the paper as submitted.\n\nI appreciate that the authors practice a robust experiment protocol with the partition used for hyperparameter optimization distinct from the partition for final model evaluation. I couldn't immediately see details about the partitioning availability so ask the authors to please ensure these splits are made publicly available for reproducibility.\n\n\n## Weaknesses\n\n**Datasets**: I understand the authors wanting to focus on transfer learning datasets where there is a challenge rather than ones which are trivial to solve given a pretrained VFM out-of-the-box. However, some caution should be taken as the datasets which the authors have selected are relatively far out-of-distribution from the training distribution of these VFMs. Previous work has investigated domain shift and categorized datasets as near- or far-OOD from ImageNet-1k, and the datasets which the authors select are generally on the far side of OOD for image datasets ([Guo, 2019](https://arxiv.org/abs/1912.07200); [Yang, 2022](https://arxiv.org/abs/2210.07242); [Lowe, 2024](https://arxiv.org/abs/2406.02465)). You can tell by eye looking at samples that describable textures (DTD), retinopathy eye scans, Clevr, and satellite imagery (remote sensing) are very different from the natural images that comprise the training datasets usually used for VFMs. For completeness, I think the work would benefit from adding a dataset which is near-OOD. However, being near-OOD does not necessarily mean the dataset isn't challenging - I recommend including a fine-grained classification dataset, which will still be very challenging for the model to learn accurately. There are many datasets used in the fine-grained image classification literature, examples include biodiversity datasets such as [iNaturalist-2021](https://github.com/visipedia/inat_comp/tree/master/2021) and [TreeOfLife-200M](https://huggingface.co/datasets/imageomics/TreeOfLife-200M).\n\n**Experiments**: Unless I missed it, the authors do not seem to explore the interaction between number of VFM models used and number of PEFT methods used in their label-ensembling experiments. It would be helpful to expand the experiments Figure 5 to show the cross between these two to make it clearer when to add more models and when to add more PEFT methods. If I already have used (VFM, PEFT) pair $(f, a)$, would I be better off adding $(g, a)$ a new VFM with an existing PEFT, or $(f, b)$ a new PEFT on an existing VFM, or $(g, b)$ a new PEFT on a new VFM, to the ensemble? Or does it not matter as long as at least one is new?\n\n**Take-away message**: The paper would benefit from giving a take-away message on which VFMs and PEFT methods to prioritize when building the ensemble for pseudo labelling, or a conclusion that it doesn't matter which models/methods are used as long as there is diversity in one/both of these.\n\n**Formatting**: Table 3 is very hard to parse as it is a very large array of numbers. I think this would be easier to read if there were cell background highlighting indicating the magnitude of the values for each column or showing the top K values in each column. This makes the table more like a heatmap so readers can determine trends in which values are large (good) and small (bad). The table would also be easier to read if the redundant % symbol were not shown in every cell. Good practice for table design recommends the units are shown in the first cell and not repeated if the same unit is used in the rest of the row/column. If the same unit is used throughout the table it can instead be described in the caption. Moving the % symbol from the cells to the caption will make the table less busy and hence more legible. Additionally, the font size can be increased with this change.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nNot sufficiently addressed. The proposed method requires more compute than the competitors. The authors do recommend limiting the number of fine-tuned models in the ensemble to 2-4 (L329-330) due to diminishing returns, but do not point out that their approach requires more compute than the competitors since it requires fine-tuning N models on the labelled partition before performing the SSL step.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}