Enhancing Semi-supervised Learning with Zero-shot Pseudolabels
Authors: Jichan Chung
Irene Y. Chen
Venue: Accepted by TMLR ()

Abstract: The high cost of data labeling presents a major barrier to deploying machine learning systems at scale. Semi-supervised learning (SSL) mitigates this challenge by utilizing unlabeled data alongside limited labeled examples, while the emergence of foundation models (FMs) offers powerful zero-shot capabilities that can further reduce labeling cost. However, directly fine-tuning large FMs is often impractical in resource-constrained settings, and naïvely using their pseudo-labels for unlabeled data can degrade performance due to its unreliablity or domain mismatch with target task. In this work, we introduce ZeroMatch, a novel SSL framework that integrates knowledge distillation with consistency-based learning to jointly leverage labeled data, unlabeled data, and pseudo-labels from FMs. ZeroMatch trains a compact student model and access FMs only through inference services, making it suitable for low-resource environments such as personal devices with limited compute. Experiments on six vision and language classification benchmarks show that ZeroMatch consistently outperforms standard SSL and zero-shot augmented methods, demonstrating its effectiveness and robustness across a range of foundation model qualities.

Published in Transactions on Machine Learning Research (12/2025)
Enhancing Semi-supervised Learning with Zero-shot Pseudolabels
Jichan Chung
Deportment of EECS
University of California, Berkeley
Irene Y. Chen
kychen&@berkeley. edu
Department of EECS
University of Colifornia, Berkeley
Reviewed on OpenReview: https: // openreview net/ forum? id= WB05Do129V
net/forum?
Abstract
The high cost of data labeling presents a major barrier to deploying machine learning systems at scale. Semi-superviser learning (SST.) mitigates this challenge by utilizing unlabeled data alongside limited labeled examples, while the emergence of foundation models (FMs) offers powerful zero-shot capabilities that can further reduce labeling cost. However, directly fine-tuning large FMs is often impractical in sesource-constrainet settings, and naively using their pscudo-labels for unlabeled data can degrade performance due to its unreliablity or domain mismatch with target task. In this work, we introduce ZeroMatch, a novel SSL framework that integrates knowledge distillation with consistency-basst learning to jointly leverage labeled data, unlabeled data, and pseudo-labels from FMs. ZeroMatch trains compact student model and access FMs only through inference services, making it suitable for low-resource environments such as personal devices with limited compute. Experiments on six vision and language classification benchmarks show that ZeroMatch consistently outperforms standard SSL and zero-shot augmented methods, demonstrating its effectiveness and robustness across a range of foundation model qualities
1 Introduction
The growing scale of machine learning applications has made data labeling costs a critical bottleneck in deploying ML systems (Northeutt et al. 2021 Sun et al. 2017 Shen et al. 2024). Semi-supervised learning (SSL) addresses this challenge by leveraging unlabeled data alongside limited labeled examples (Tarvainen) & Valpola 2017). Traditional SSL approaches like pseudo-labeling and consistency regularization have demonstrated strong performance across domains, particularly in computer vision and natural language processing (Sohn et al. 2020 Laine & Aila 2022 Tarvainen & Valpola 2017].
In parallel, the emergence of foundation models (FMs) has opened new opportunities for reducing reliance labeled data. These large-scale pre-trained models exhibit strong zero-shot capabilities, enabling them to generalize to novel tasks without requiring task-specific finc-tuning (Brown et al., 2020 Liang et al. 2022
To this end, recent elforts have explored integrating foundation models into the SSL pipeline. Proposed strategies include fine-tuning foundation models with labeled and unlabeled data (hi et al.) 2023 Zhang
ct al. et al.
2024 Gan & Wei. 2024), using zero-shot predictions as pscudo-labels (Hegsclmann et al. 2023 Nam
2023 and distilling knowledge from foundation models into smaller student models (Yang
2025
Shi et al. Vemulapalli al. 2023: Zhao et al. 2023 Jiang et al. 2023
We motivate problem setting with practical on-device training scenario: user with personal modest- sized dataset and limited computational resources (e.g., single GPU), with access to foundation model
'Experiment codes available in https://gåthhb. http:////ttaaabma/cccca///////a/

---

Published in Transactions on Machine Learning Research (12/2025)
Teacher Inference (via API)
Pseudo-labels
p
"goldrinch"
Foundation Model (e.g. LLM, VLM)
"geldiinca" "solamander"
Student Stage 1 (KD)
Model Pradictions PGJA)
x²
Prediction Head
H(}*)66(k>)) + H(9", #(0".p(rlz"))
Encoder
v
KD Loss
Student
H(y",pClk
Stage 2 (SSL + KD)
Prediction Head
Unsupervised Learning (with x
Encoder
U
KD Head
KD Loss
Figure 1: Illustration of the proposed ZeroMatch algorithm. Pseudo-labels from the teacher foundation model obtained for both labeled and unlabeled input data through APT inference service. In stage of student model training, knowledge distillation is performed with the obtained pseudo-labels. In stage 2, the student model is trained with supervised and unsupervised objective of an SSL algorithm, with weights and initial confident, prediction learned from the previous stage. We add an auxiliary classifier head (green box) that runs the same knowledge distillation task from the previous stage, to mitigate potential catastrophic forgetting that may occur during running SSL.
inference services. In such settings. direct finetuning of large foundation models (including parameter- efficient fine-tuning (Houlsby et al. 2019: Hu et al. (022) and test-time adaptation (Sun al.) 2020 ) is infeasible due to their large parameter size and high computational costs for backpropagation. Furthermore, naively using pseudo-labels from foundation models to supervise unlabeled data can degrade performance particularly when pseudo-labels are inaccurate [Zhu et al. 2024) - which can be common when the foundation model has not encountered data similar to the user's personal data. Among existing approaches, knowledge distillation-basel methods (Zhao et al., 2023: Vemulapalli et al. 2023 Jiang et al. 2023 offer promising solution for such resource-constrained settings. However, current methods typically leverage either the tcacher's predictions the labeled data. independently, missing opportunitics to combine these complementary supervision sources. Thus, fully leveraging labels, unlabeled data, and pseudo-labels from foundation models in unified framework remains an underexplored direction.
In this paper, we introduce ZeroMatch, a method that integrates knowledge distillation and semi-supervised learning to jointly supervise a student model using both pseudo-labels from a foundation model and a confidence-basod SSL objective. Our approach based on the insight that the student model can iteratively refine its predictions on unlabeled data by drawing from the strengths of both the teacher (via distillation) and labeled data (via SSL). We conduct extensive experiments across six datasets in vision and language classification, evaluating ZeroMatch using pseudo-labels generated by both high- and low-quality foundation models. Our results show that ZeroMatch consistently outperforms standard SSL basclines, zcro-shot classification, and other zero-shot augmented approaches, highlighting its robustnest and practical applicability in low-resource scenarios.
2

---

Published in Transactions on Machine Learning Research (12/2025)
In summary, our key contributions are follows:
1. We propose ZeroMatch: novel knowledge distillation-basst semi-supervised learning approach that effectively leverages labels, unlabeled data, and teacher pseudo-labels from foundation models.
2. Our framework trains small models while directly leveraging zero-shot predictions from foundation models as pseudo-labels, enabling training on modest. hardware, and eliminating the need for the teacher model to be present. This addresses practical setting where users have limited computational resources but access to foundation model inference services.
3. We provide extensive experiments on six datasets across vision and language domains, which demonstrate that ZeroMatch outperforms both standard SSL methods and zero-shot augnented approaches.
2 Related Work
2.1 Semi-supervised Learning
Semi-supervised learning (SSL) has evolved from foundational consistency regularization methods to so- phisticated deep learning approaches. The II-Model (Laine & Aila 2022 and Mcan Teacher (Tarvainen & Valpola 2017) established core principles by enforcing consistent predictions across different model states. FixMatch (Sohn al., 2020) later unified these ideas by combining weak and strong augmentations with pseudo-labeling. Subsequent work focused on reliability: UPS (Rizve al. 2021) introduced confidence-based filtering while FlexMatch (Zhang et al. 2021) developed adaptive thresholding for pseudo-label selection. Recent approaches like SimMatch (Zheng et al. 2022) have further advanced the field by incorporating contrastive learning principles.
2.2
Foundation Models in SSL
The integration of foundation models into SSL frameworks is an emerging direction. One of the main emerging
approaches is to the-tune foundation models with SSL or unsupervised algorithms (Shi et al. 2023 You et al.
2024
Hegselmann et al. 2023 Menghini et al. 2023 Zhang et al. 2024 Gan & Weil 2024j. These methods
provide a strong and robust learner in general, but are not often applied to state-of-the-art foundation models due to reliance on model-spocific features like cosine similarity of emboddings, and the significant compute for the fine-tuning step involved in tuning large foundation models. In contrast, our setup leverages foundation models as pscudo-label generators. Rescarchers have also proposed thcoretical guarantees to accommodate potentially noisy pseudo-labels (Zhu et al. 2024). Our work builds this goal to develop a practical method for incorporating unreliable pseudo-labels.
2.3 Knowledge Distillation in SSL
Knowledge distillation (KD) has been widely used to transfer knowledge from large teacher models to smaller student models (Buciluă et al.) 2006 Hinton et al.) 2015 West et al. 2021 Beyer et al. 2022 Hsieh et al.) 2023
Vöge et al. 2024h, allowing the benefits of large models on low-compute devices. Many KD approaches
applied in the semi-supervised learning (SSL) setting pre-train the teacher model, either via supervised or
self-supervised objectives (Chen et al. 2020 He et al.] 2021 Yang et al.] 2025 Xie et al. 2022 Shi et al,
Vemulapalli et al. 2023). Among them, SRD (Yang et al. (2025) deploys an auxiliary objective match teacher and student outputs, similar to ours. Methods that rely teacher model pre-training often make it impractical to apply them to recent large foundation models like Llama-70B due to their large computational requirements for tuning. Recent alternative KD attempts avoid explicitly training a teacher by optimizing the student. models on teacher's zero-shol outpuls, but trains the student model on either one of (eacher model's outputs (Zhao et al. 2023 Jiang et al. 2023) or labeled data (Vemulapalli et al., 2023), missing the synergic opportunity of using both. To the best of our knowledge, there is no existing KD approach that combines SSL-specific features (such as strong augmentation and confidence- based sampling) with label-based KD, which satisfies our goal of fully leveraging labels, unlabeled data, and teacher outputs simultaneously.
3

---

Published in Transactions on Machine Learning Research (12/2025)
3 Preliminaries
3.1 Problem Setup
We consider an on-device training setup with a compute-limited device (e.g. single GPU, edge device with memory less than 8GB), semi-supervised training data, and read-only access to a foundation model f through an inference service. The training data consists of a labeled Dz = (a/² y [Nt] and an unlabeled sot Dv -x4f) E [Nu], where inputs of both sets come from the same distribution. D Dr. 'Du represents the full dataset where [M] denotes integers 1, 2... N. The foundation model f generates pscudo-label distributions 0.- = f(x!") for labeled data and d.¹ = f(x(") ) for unlabeled data. The goal is to train a classifier f that outputs class distributions p(y2=) using DL. Du, and pscudo-labels 0.. g." We assume of was not trained on D, or Dr. The pseudo-label quality depends on the model's capabilities and prompt design, with potential failure modes including hallucinations and out-of-domain responses.
Our main objective to develop an SSL method that maximally leverages labels, unlabeled data, and pseudo-labels from the foundation model. This implies that the algorithm should fully utilize pseudo-labels when they are high quality, and gracefully degrade to standard SSL performance when pseudo-labels are unreliable.
3.2 Semi-supervised learning methods
Recent advancements in SSL have led to the development of methods that jointly train classifier from both supervision on the labeled set, and unsupervised feature learning on unlabeled data. The training objective is typically expressed as L = L. + Lu where L, the supervised loss, often the cross-entropy loss computed on the labeled data:
B1 Hly! p(yl=!)).
C.
BL
where B, the size of the data batch sampled from the labeled set Dz.
On the other hand, fu represents the unsupervised loss, which leverages different strategics incorporate unlabeled data into the training process. Some notable strategies include the following:
Self-training: This strategy involves using the model trained on labeled data to generate temporary predictions for the unlabeled samples. These predictions are then incorporated into the model's supervision objective Lee 2013).
Confidence-based sample selection: Unlabeled samples with less confidence potentially incorrect pseudo-labels are filtered out based predefined thresholds. This ensures that only samples with reliable predictions contribute to the training process (Xie et al. 2020
Strong augmentation: To further enhance feature learning, the model is optimized that its predictions weakly augmented and strongly augmented versions of the same sample agree, encouraging the model to learn robust features (Xie et al. 2020).
Distribution alignment: This technique adjusts the output probability distribution of each class based on the input data distribution [Berthelot et al. 2020
With strong augmentation A. and weak augmentation Au (x of unlabeled samples x!' we denote prediction logits from classifier f of these augmented samples by pf P(ylA. and p." = Am )), respectively.
2Our usage of "pseudo-labek differs from traditional SSL literature, where they typically predictions on Dy from model partially trained Dr.- Here, they represent foundation model predictions without any tpek-specific training.
4

---

Published in Transactions on Machine Learning Research (12/2025)
When data batch of size By is sampled from Du, the unsupervised training objective in typical SSL methods takes the following form:
By 1 (max(p?")
Lu
T)H( plyla!))
BU
where 03" DA(p)) the label prediction for input Au (x and DA represents the distribution aligument. process.
4 The ZeroMatch Framework
In order to develop an SSL method that effectively leverages labeled data, unlabeled data, and pseudo-labels generated by a foundation model, we propose a hybrid approach that combines the strengths of knowledge distillation and SSL tochniques.
The two key intuitions behind our approach are:
1. Knowledge distillation methods utilize the teacher model's predictions to improve the student modcl's prediction output on unlabeled data.
Semi-supervised learning algorithms utilize the labeled data refine the model's predictions on unlabeled data.
These imply that both methods can be used to improve predictions unlabeled data from different supervision sources in a complementary manner. To this end, we propose the following two-stage algorithm that integrates both paradigms to maximize the utility of all available supervision signals.
Stage 1: Knowledge Distillation (KD) We start by following standard knowledge distillation procedure with the teacher's output. A student model is trained using available unlabeled data (including the unlabeled input in labeled data) with the pseudo-labels , V generated from the teacher foundation model f with the following objective:
N₂ H(T! x())))) + Σ HO? p(yl=,))))
Nu
1 N
CKD
Σ
where N = Ny + No. The resulting student model obtains high-confidence predictions for unlabeled data that match the teacher's pseudo-labels.
Stage 2: Semi-supervised learning with auxiliary KD loss The objective of the second stage is to train the student model with an SSL objective. The key component is that, unlike running SSL from scratch, predictions obtained from the previous stage can be utilized as initial high-confidence samples in the SSL algorithm, allowing more unlabeled samples to utilized from the beginning stage of SSL learning, achieving laster convergence as a result.
One possible failure mode of running an SSL algorithm in stage catastrophic forgetting (Goodfellow et al. 2013), where learned knowledge from the teacher's pseudo-label can be forgotten during training for the downstream task (SSL in this case). This can happen frequently in a low-label setting, where the SSL algorithm can develop inaccurate predictions unlabeled data due to inherently limited information in labels, leading to inaccurate pseudo-label that overwrites the knowledge learned. To address this issue, we include knowledge distillation as an auxiliary objective alongside the main SSL objective, motivated by Kar ct al. (2022). When the student classifier f consists of a non-lincar projector head h(-) and a feature encoder g()), such that f g, we introduce an additional linear projector head h.). which learns knowledge distillation task, while sharing the encoder g with the main classification task. Given batch of labeled and
5

---

Published in Transactions on Machine Learning Research (12/2025)
unlabeled data of size By and Bu, stage 2 optimizes hp o g with the following loss:
By
H(t ((vlat)) + Σ HOl alola!))
LKD= B
where B = By + By and ((ylz) indicates the predicted class distribution of hp(g(z)). Together with the SSL objective, the overall loss function of stage 2 is:
LKD-SSL. L. + Lu + Oe. XpCKDz
where ag is an anncaling paramcter that lincarly increases its value from 0 to 1 during training based on training step I, and Ap fixed scalar hyperparameter indicating the relative weight of pseudo-label prediction task. We denote the inclusion of annealing with a binary variable Op-
The auxiliary KD head acts an indirect regularizer that mitigates the catastrophic forgetting issue, by providing continuous supervision of the teacher's pseudo-label to student's 's model during the SSL training. Because hp and h share the encoder 9, the KD loss constrains g's representations to remain aligned with teacher pseudo-labels even as h adapts to the SSL objective. This prevents the encoder from drifting toward erroneous SSL predictions in the early stage of training, which may easily override the teacher knowledge learned from stage 1. Ao and a can be set to control the balance between the SSL objective and KD loss to ensure that the KD loss does not overwhelm the SSL objective but is strong enough to keep the stage knowledge. Note that when Ap is set too large, the KD lask may dominate the training objective and interfere with the SSL's objective, reducing the flexibility of the SSL head h to adapt to the downstream task. Our algorithm is illustrated in Figure
Our algorithm can robustly improve using both labels and pseudo-labels. When the pseudo-labels are accurate, the model benefits from the knowledge distillation task throughout both stages of training. Conversely, when the pscudo-labels are noisy, the model can still benefit from the SSL's objective, with less impact from the inaccurate pseudo-label due the KD task being learned at the separate classification head hy. We integrate our method with AdaMatch (Berthelot et al., 2021), which is a highly performant SSL baseline across various datasets and domains, as demonstrated by benchmark results in Wang et al. (0222)). We denote this method by ZeroMatch.
While our method is designed to train an accurate student classifier for a given task, note that its encoder may have representation drift due to forcing supervision with teacher pseudo-labels, which may be different from the student's 's prediction. For applications that requires embeddings from the encoder, we suggest training a separate student model with an additional distillation stage with the current student's output.
5 Aren't existing LLM adaptation solutions good enough?
The problem of transferring the knowledge of large language models (LLMs) to domain-specific tasks has been widely studied, and many apporaches are applicable to semi-supervised learning as well. Two prominent strategies among them are: fine-tuning (including its parameter-effeiennt variants) and prompt engineering. While these approaches have proven effective in many fully supervised settings, they present limitations when applied to scmi-supervised learning tasks under low-compute environments. In this scction, we highlight these limilations and discuss how ZeroMatch provides a more practical alternative.
(Paraneter-efficient) fine-tuning cannot avoid large memory footprint. In terms of statistical performance, fine-tuning wein-w
parfieter-efficient fine-tuning methods (uu et al.
Lester et al. 2021 and test-time adaptation techniques (Sun et al. 2020 Hong et al. 2023 Song ct al. often achieves the strong results, as these methods allow models to fully exploit their rich paramelerization. However, such approaches are resource-intensive: both model loading and gradient backpropagation scale with the full model size. We illustrate this limitation in Fig. 2 which compares LoRA fine-tuning and ZeroMatch using the L.AMA-70B model as an example. LoRA requires roughly 140 GB of GPU memory just to load model parameters, and more than 160 GB when including intermediate activations during backpropagation, making it infeasible for users without access to high-end GPU clusters. In contrast,
2022
6

---

Published in Transactions on Machine Learning Research (12/2025)
Cloud:
y1
Llama-70B
Llama-70B
y2
LoRA
One-time inference
User:
(a) User's VRAM usage: 160GB
User:
Bert- base
Model training
Model inference (no activation)
(b) User's VRAM usage: 5.4GB
Model inference with activation
Figure 2: Comparing training setup and memory usage (a) LoRA fine-tuning and (b) ZeroMatch using the Llama-70B foundation model (with 16 bit quantization). (a) While LoRA offers parameter- efficieet adaptation, still requires loading the entire Llama-70B model and storing intermediate activations for backpropagation, leading to a large memory footprint (approximately 160 GB), requiring users to have access to data center-level GPUs. (b) In contrast, ZeroMatch trains a lightweight student model (e.g., BERT-base) locally while delegating heavy LLM inference to cloud servers via one-time API calls, where cloud servers effectively hide the computation cost by batch processing requests from multiple users. This design offloads both computation and memory from the user, enabling training on low-menory devices (e.g., laptops or smartphones) with only 5.4 GB VRAM usage.
ZeroMatch enables the use of LLMs an SSL fine-tuning task with a smaller training backbono on modest hardware, such as a single GPU or even edge devices. This decouples the use high quality information from LLMs from the computational demands of training, allowing effective model training locally without incurring high computational costs.
Prompt engineering cannot utilize semi-supervised datasets effectively. Prompt engineering approaches, such as in-context learning, are often used to leverage few labeled examples by placing them directly within the model's input context. While this strategy enables models to adapt to tasks without computation-heavy parameter updates, it commonly yields underwhelming performance compared to fine- tuning-basse methods (Lehman et al. 2023), particularly when the target task involves complex fcature-labe relationships that require extensive supervised training. Moreover, prompt-based methods are inherently limited in utilizing large unlabeled datasets due to the restricted context window of foundation models. Even if becomes technically possible with increased context window size, most foundation models are not trained to interpret or exploit unlabeled data, making effective semi-supervised learning infeasible with this solution.
While in-context learning could, in principle, be employed to generate pseudo-labels for unlabeled data-potentially complementing ZeroMatch's 's zero-shot responses- we exclude this option in our experimental analysis to isolate and evaluate the effects of labeled supervision within our framework.
In addition to the computational and met hodolggiaa ad advantages discussed above, ZeroMatch offers further practical benefits that extend its usability across a wider range of real-world constraints, as described below.
ZeroMatch is compatible with closed-source LLMs Many high-performing LLMs (e.g., GPT-4o) are closed-weight and accessible only through limited APIs. These APIs often restrict training capabilities for instance, OpenAT's fino-tuning APT supports only supervised lcarning with fixed input-output pairs, making training with an SSL algorithm infeasible. ZeroMatch remains compatible in such scenarios. Extracting pseudo- labels from the LLM's text output is straightforward and does not require access to internal representations
7

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Summary of data used the experiments. For each dataset, we run experiments with at most 3 psoudo-label sets from different foundation models. The foundation model f used to generate cach pscudo- label are shown below.
Domain
Dataset
# Train
# Val
#Test
# Class
Foundation model f
Amazon Review
250,000
25,000
650,000
5
GPT-4o, Tlama333-7083
NLP
AG News
100,000
10,000
7,600
4
FLAN-15
Yahoo! Answer
500,000
50,000
60,000
10
CIFAR100 50,000
10,000
100
CV
Flowers102 1,020
6,149
102
GPT-4.1, CLIP
Resisc45
3,150
25,200
45
or gradients. This makes our approach practical even when constrained to inference-only interactions with proprietary LLMs.
ZeroMatch reduces data-leakage risks. Using LLMs often requires uploading data to remote servers where the model is hosted, raising concerns of data leakage especially when working with privacy-sensitive domains such as healthcare or finance. Assuming that the model training happens on a private device, our setup significantly reduces this risk by limiting data exposure to remote machines to a one-time inference call. Once the data is processod, it can be immediately discarded, climinating the need for persistent storage or repeated transmission. This is notably safer than fine-tuning LLMs, which requires storing the data remote machine for continuous access thronghout the training process.
6 Experiments
6.1 Experiment Setup
Datasets We use 6 publicly-vvailale datasets across vision and natural language processing (Table Dataset sizes range from 100s to 100,000s of data points. To evaluate the effect of label size, for each dataset, we create up to three tasks with different sizes of labeled data while the total number of unlabeled data remains the same. When sampling the labeled datapoints, same number of labeled samples are drawn for each class. For example, in the CIFAR100 dataset, we create sets with 100, 200, and 400 labeled data points, each having 1, 2, 4 labels per class, respectively. Detailed explanations of cach dataset are provided in Appendix A.1
Teacher pseudo-labels from foundation models For pseudo-labels of each dataset, we consider multiple sets of zero-shot responses with varying quality. For NLP datasets, generate pseudo-labels with GPT- 30B (AT 2024), and FLAN-T55 (Chung et al. 2024) with standard wero-shot prompt of task description (e.g. "Select the topic that the given article is about. or "What is the sentiment of this review?") ") with the list of the task's label options, and the response is parsed to find the predicted label. Among FLAN-T5 models with different model sizes, we choose two models: one with the best quality, and one having the lowest quality, based on their test accuracy. We name each FLAN-T5 pseudo-label set 'A' and 'B' in order of decreasing quality. See Appendix A.2.2 for exact models used. For image datasets, use GPT-4.1 (OpenAI, 2025) and CLIP (Radford et al. 2021 GPT-4.1 is prompted via chat-completion API with the image select the correct label with standard prompt (e.g. 'Classify the input image into eractly one of the following types." followed by candidate labels, while CLIP uses its standard zero-shot classification pipeline by comparing image embeddings with texl-prompted class embeddings with cosine similarity score. All pseudo-labels are computed once through offline inference without any additional calibration or consistency adjustments. We provide benchmark results of zero-shot psoudo-labels on the test set in Table 12 in Appendix A.2.2 Further details on pseudo-label generation procedures including the exact prompt used are discussed in Appendix A.2
4o (OpenAT 2024b. Llamma.3-70B
8

---

Published in Transactions on Machine Learning Research (12/2025)
Training setup Data-specific training configurations are explained in detail in Appendix A.3 For image classification, we train ViT-Small (Dosovitskiy) 2020) with AdamW optimizer for T =044 steps for CIFAR100, and T' =002,400 steps for Flowers102 and Resisc45. For NLP classification tasks, we use pre- trained BERT-Base (Devlin 2018) with the AdamW optimizer for T =102,400 steps. To ensure a fair comparison with AdaMatch (an SSL baseline); we use the exact same hyperparameters used for AdaMatch. For hyperparmeters specific to our method, we use Op = (indicating that the annealing is applied) and A for all experiments. Discussions on the default choice of ap and Aps and their sensitivity analysis is provided in Appendix 6.5.3
Implementation details We implement our method using the Unified SSL Benchmark (USB) frame- work (Wang et al. 2022a). To minimize the complexity of integrating our method with existing SST. baselines, we simply add an additional multi-layer perceptron (MLP) classification head with the same architecture as the bascline model's MLP head. All experiments are implemented in PyTTrch and run with a single NVIDIA A5000 24GB GPU, but we note that our experiments can run on a GPU with much smaller VRAM due to the training backbone being small (see Section 6.5.4 for computational requirements of our method.).
6.2 Baselines
6.2.1
supuri-supervision
When given semi-supervised dataset that includes pseudo-labels, straightforward approach is to fill in the prediction targets for the unlabeled samples using the pscudo-labels, resulting in a fully pscudo-labeled unlabeled set Dú = {(>{ ). (x) )....... (~x NNG }- The model can then be trained using supervised learning algorithu on this fully-labeled dataset D = Dr, DD We refer to this method as pseudo-supervision. Pseudo-supervision has been shown to improve performance in semi-supervised learning scenarios, particularly when zero-shot method provides accurate pseudo-labels for the task at hand (Hegselmann et al., 2023 Nam et al. 2023 However, when the zero-shot predictions inaccurate, it remains unclear whether pseudo-superviice will still be beneficial. We refer to Appendix A.4 for implementation details.
6.2.2 Pseudo-label as feature input
Another straightforward approach use pscudo-labels improve learning to include them as one-hot features, which is often employed in tabular learning settings include categorical features (Borisov al. SSL algorithm (AdaMatch) with one-hot converted pseudo-labels as
2022 For this baseline, we additional feature input to the model. We name this approach PL feature input. This approach can learn to accommodate or ignore the pseudo-label features in SSL depending on how helpful they are in learning the task. See Appendix A.4 for implementation details.
6.2.3 Additional comparisons with previous works
We discuss additional comparisons of ZeroMatch with other previous works in Appendix B While our work mainly focuses on involving FM to an SSL setting in pseudo-label format, there other previous works outside this category that also try to enhance SSL with FMs, such as fino-tuning FMs for SSL (e.g.
GRIP (Menghini et al. 2023 CPL (Zhang et al. 2024h and FineSSL (Gan & Wei 2024)) ). We provide an overview of how these methods compares with ZeroMatch in practice in Appendix B.T We also discuss comparisons with the doubly-robust seff-training method (Zhu et al. 2024] which addresses a problem similar
ours in Appendix B.2
6.3 ZeroMatch Enhances SSL with High Quality Pseudo-labels from Large Foundation Models
Benchmark results for the NLP datasets with GPT-4o and Mlama3.3-70B are presented in Table 2 and image datasets with GPT-4.1 are presented Table 3 When combined with pseudo-labeks from high-quality large foundation models, ZeroMatch achieves the highest scores most of the settings (22 of 23 total sottings, except 2000-label case of Yahoo Answers with Llama3.3-7055 30B).. Our results show that ZeroMatch consistently improves upon SSL baseline and zero-shot, implying effective usage of both labels and zero-shot pseudo-labels
9

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Accuracy (%) in Yahoo Answers, AG News, Amazon Review with pseudo-labels from GPT-4o and Llama3.3- 70B. Median and standard deviation of different random seeds are reported. Best score among experiments with same pseudo-labed set are in bold. 'FM' refers the foundation model used to generate the lebedo-label set.
Datsoes
Anewirs
News
Amnoro Reciow
FM
Labal size
250
500
2000
20
40
200
125
250
1000
Nome
Adamateh
61.8::122
67.320.58
09.4220,42
58.1242.56
85.2121.44
05.33=1
45,00::4005
$2.29:=1665
56:18411:45
Zaroohot
68.61=0.0
68.81+0.00
68.81+0.00
86.25+0.00
96,25+00.00
86.25-000
50.14=000
08.14=0.00
69.16+0.00
GPT-4o
Ponulo-superviee
lapu:
67 0.0000000
70.56:0227
67.3011.200
71.0010.57
71.4710.87
67.5610.06
86.25400.40
66.1310.115
86.33:0.00
88.0010.90
86.33-0.14
25.39_110
36.94:0.27
30,38-0.67
56.65:0.21
58,30:1.23
56 58.831:0.25
57.0810.299
Zenoaath (ours)
70.90=1.07
71.11=0.00
72.00+0.32
88.60+00.22
88.70++0.22
88.76=1.20
60.12++0.8
59.82=0.89
60.19+00.48
Zero-slict
09:15=000
08 18.1520.00
19.161
58.0110000
58.414000
00000000
55,79:000
50.79.0000
66.79410.000
3.ma333
07.85=0:21
07.67+0.39
67.17+0.14
58.47+00088
58.53+0.38
88.43 05.4550008
33.74=0.33
64.34-0.11
64.1440.53
-70B
PL featurr input
70.70=0022
71.37:10221
12.18:10.25
66. 60.70:17.00 7.08
88,41:00002
88 0.52
51 01.55:00:95
64.30:00,79
57.01
ZevoMasch (ours)
71.2810..29
71.4610995
71.6810.19
88.7910.05
88.96100007
88.86110.17
57.3611.881
59.46-00.13
60.0610.22
Table 3: Accuracy (%) in CIFAR-100, Flowers102, Resisc45 with pscudo-labels from GPT-4.1. Modian
and standard deviation of 3 different random seeds are reported. Best score among experiments with same pseudo-label set are in bold. 'FM' refers to the foundation model used to generate the pseudo-label set.
Dataset
CIFARI00
Flowers102
Resisc45
FM
Label size
100
200
400
204
90
None
Adamatch
71.43.12.48
78.32100.6
84.02.10.74
86.7110.69
78.8710.80
Zero-shot
83.2510.00
83.25::0.00
83.2510.00
88.37:00.00
79.2810.00
Pseudo-supervise
84.8420.03
84.931:..7
85.1210.13
85.4011.25
79.5911.18
GPT-4.1
PL. feature input ZeroMatch (ours)
1..8
82.78
85.78+00.41
89.53++1.44
82.01+1.009
88.01=00.05
87.97
88.12:10.14
95.17=0.88
87.83:10.72
as supervision sources. For example, in Amazon Review dataset with 40 labels and pseudo-label set from Llama3.3-70B, ZeroMatch achieves 71.28% compared to zero-shot accuracy of 69. 69.15% and Adamatch 64.81%, yielding an improvement of 2.13% and 6.47% respectively. Our method also outperforms psoudo-supervision and PL feature imput basclines, with highest improvement coming from 100-label setting of CIFAR100 with GPT-4.1 where ZeroMatch produces 15.2% improvement compared to PL. feature input, and 8.23% improvement in the 90-label setting of Resisc45 with GPT-4.1 for pseuuo-superviiiinn This indicates that ZeroMatch is the most efficient in utilizing all supervision sources among basclines that uses the same data.
6.4 ZeroMatch Improves Performance on SSL Baselines Despite Low-Quality Teacher Pseudo-labels
We also present benchmark results with pseudo-labels from FLAN-T5 and CLIP. While FLAN-T5 and CLIP models are generally less powerful than GPT-4 and Llama models, this configuration demonstrates the following practical settings:
Unseen task data: Users may train with personal data that is not seen by the foundation models, Based on training data sources listed in technical reports of FLAN-T5 (Chung al.] 2024) and CLIP (Radford et al. 2021 models, we ensure that these models are not trained with our benchmark dataset. On the other hand, recent models like GPT and Llama do not reveal the data source, therefore we cannot ensure that our benchmark datasets (which are public) are not in the training source of those models.
Restrictions in model access: Users may not be able to access statc-of-the-art large foundation models and may be limited to using certain types of less powerful foundation models, due to company restrictions, contracts, or high implementation cost. involved in accommodating the new model.
NLP benchmark results with FLAN-T5 are presented in Table 4 Image classification results with CLIP models are presented in Table 5 Our results show that ZeroMatch consistently outperforms or maintains the performance of basclines within one standard deviation all other basclines most cases (26 of 28 total settings, except CIFAR100 with 400-labels and Flowers102), demonstrating its robustness against low-quality teacher pseudo-labels. For example, for Amazon Review with FLAN-T5 pseudo-label set A, ZeroMatch
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: Accuracy (%) in Yahoo Answers, AG News, Amazon Review with pseudo-label from FLAN-T5 models. 'A' and 'B' in FM column indicates using highest and lowest quality pseudo-label sets (docreasing in order) from FLAN-T5 models. Median and standard deviation of 3 different random seeds are reported. Best score among experiments with same pseudo-label set are in bold.
Dataset
Yalso Auswers
AG News
Amnzzo Review
FM Nome
Label
250
500
2000
20
40
260
125
250
1000
Adamattc
64.81-1.29
67.3++053
60.42+0.42
13+255
85.21++1444
88.33+1.00 199
45.03++4.06
32.39+1.65
56.18+0.45
Zeco-siut
66.63-0.00
66 66.6510.00
66.6310.00
91:43:0.00
91.4510.00
91,4310:00
52.3710000
52.57:0.000
52,3710.00
A
Parudo-muperiie
PL feature iaput
67. 07.993.0.800
05.79=0.18
00.31100:0
69.6110.99
70.82.10:25
63.9220:33
91 4315,51
913774007
91.43:10.01
91 21.22400.08
90,70.10.85
91.4240088
50.0011.01
51.11:0:277
51.5810.67
50.86110.27
56.53100.11
50.4110:41
ZaroMatch (ours)
69.78440.94
70,51:00119
71.8129.07
91.61:0.122
91.4240.21
01.62=0.15
98.88-10.61
09.6910.60
60,95=00113
Zero-intt
36.29=0.00
35 36.299+0.00
85,29+0.00
87.06+0.00
87 0760000000
87.06+0.00
35.7400.00
$5.7+0.00
35.7+0.00
B Pueeei-neeerviie
36.38=0.51
36 29:30.33
36.37:0.51
68.710016
88 51:10.200
88.41:0.188
36, 06.02:00.01
35.9 35.97:10.05
36.0 36.0410.05
festare Topet
65.52-0.48
65 89.88900.89
60-29+0.53
84.61+2.08
86.45++1655
96.66+1.00
48.03+1.89
53.76+1.62
56606+0-30
ZeroMatch (ours)
67.05:1:0.76
07:13::0
69.01:00.2
90.42:10.19
90.21110.08
90.25:20.16
10.23:1:.35
53.78:11.74
56.12:00664
Table 5: Accuracy (%) in CIFAR-100, Flowers102, Resisc45 with pseudo-labels from CLIP models. Median and standard deviation of 3 different random seeds are reported. Best score among experiments with same pseudo-label set are in bold. 'FM' refers to the foundation model used to generate the pseudo-label set.
Dataset
CIFARI00
Flowers102
Resisc45
FM
Label size
100
200
400
204
90
None
Adamatch
71.43=2.48
78.32=0.46
84.02=#0.74
86.71
78.87=0.80
Zero-shot
60.65=0.000
60.65=0.00
60.65=00.00
72.13+0.00
60.32=0.00
Psendo-supervise
70.97+0.41
71.86=0.63
74.05=+0.42
74.19+0.57
63.98++0.21
CLIP-Large
PL feature input
71.21+2.56
80.11=0.49
84.42:10.22
89.5110.42
10.85=1143
ZeroMatch (ours)
84.30=t0.71
85.77:10.05
86.1310.011
88.9411.66
83.91=1.54
Zero-shot
48.07=0.00
48.07=0.00
48, 0.00070000
60.17:0.00
49.63=0.00
Pseudo-supervise
62.19-10.26
65.96-0.37
71.03:0.23
61.09
55.82-10.84
CLIP-Base
PL feature input
71.3.1..26
7.45
83.88:00773
87.56.10.99
80.85
ZeroMatch (ours)
79.78+t0.99
80.87+0.62
82.69+0.23
89.61+2.39
02.58+00.85
achieves accuracy of 58.83%, outperforming PL feature input's accuracy of 50.00%. Notably, when pseudo-label quality is low, ZeroMatch consistently improves (or at least maintains the performance within one standard deviation) when the number of labels increases. For example, the scores of Yahoo Answers with FLAN-T5 set increase from 67.05% to 69.61% as the number of labels increases. This implics that our method utilizes labeled samples lo correct and improve when pseudo-label may not be helpful. On the other hand, PT. feature input baseline fails to improve AG News when increasing labels from 40 to 200 with FLAN-T5 A set, where the score decreases from 91.43% to 90.70%. Our method remains largely effective against pscudo-supervision under low-quality pseudo-label settings. For example, in the 250-label setting of Yahoo Answers, pseudo-supervisg reaches 36.38% accuracy with pseudo-label set B, which is worse than 67.05% of ZeroMatch by a large margin.
6.5 A Deeper Look into ZeroMatch
6.5.1
Accomodating embeddings can further improve ZeroMatch
While our problem setup focuses on leveraging pseudo-labels from foundation models, embeddings (from foundation models trained with embedding tasks) may also provide information that is helpful for learning the task. We also provide way to accommodate these embeddings in our framework to further improve performance on target tasks. Pleasc see Appendix A.4 for implementation details. We provide benchmarks of our method leveraging pseudo-labels of GPT-4.1 and embeddings from the CLIP-large model on Flowers102 and Resisc45 in Table 6 Our result shows that adding embedding to our framework can outperform cases where only pscudo-labels from the same foundation models are included, achieving highest score in both datasets. This indicates that our method can be easily extended to mix and match pseudo-labels with embeddings to further improve the performance.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: Accuracy (%) in Flowers102, Resisc45 with pseudo-labels and embeddings from CLIP models and GPT-4.1. Median and standard deviation of 3 different random sceds are reportod.
Method
FM inference info used
Flowers
Resisc45
Zero-shot
GPT-4.1 pseudo-label
88.3710.00
79.28=0.00
Zero-shot
CLlle-laree pseudo-label
72.13:10.00
60.32=0.00
ZeroMatch (ours)
GPT-4.1 pseudo-label
95.17:10.88
87.65=0.733
GPT-4.1 pseudo-labed +
ZeroMatch (ours)
97.20=+0.41
88.52:10.77
CLIP-large embedding
Table 7: Comparison with ZeroMatch without auxliary loss in Yahoo Answers and Amazon Review with pscudo-labels from GPT-4o and LLama3.3 models. Median and standard deviation of accuracy (%) of 3 different random seeds are reported. Best score among experiments with samne pseudo-label set are in bold. 'FM' refers the foundation model used to generate the pseudo-label set.
Dataset
Yahoo Answers
Amazon Review
FM
Label size
2000
125
250
ZM W.o. aux.
08.2240.32
69.23:10.33
71.45=0.23
67.8010.47
59.60=0.33
60.16::0.31
GPT-4o
ZeroMatch (ours)
70.90+1.07
71.11=0.09
72.09:1..32
60.12:10.33
59.82++0.39
00.19:10.48
ZM w.o. aux. ZeroMatch (ours)
69.30+20.33
70.54:=0.54
71.54=0.31
56.59=1.78
59.04=0.21
60.13::0.08
LLama3.3-70B
71.28=0029
71.46:+0.95
71.68+0.19
57.36+t0.81
59.46+00.3
60.06+0.228
Table 8: Comparison with ZeroMatch Stage 2 only in Yahoo Answers and Amazon Review with pseudo-labels from FLAN-T5 models. Median and standard deviation of accuracy (%) of 3 different random seeds are reported. Best score among experiments with same pseudo-label set are in bold. 'FM' refers to the foundation model used to generate the pseudo-label set.
Dataset
Yahoo Answers
Amazon Review
FM
Label size
250
500
2000
125
250
1000
ZM w.o stage
70.01
60.76:0.22
71.22::0.27
55.16:10.30
55. 65.75
58. 08.5310.36
ZeroMatch (ours)
69.78++0.94
10.51
71.81
58.83:10.51
59.69-0.60
60.95:10.13
ZM w.o stage
66.64:013
67.05::0.41
69.22:10.33
45.534:4.91
53.19411.68
56.9+0.63
B
ZeroMatch (ours)
67.05:10.76
0..1
69.61
50.28:11.35
53.78+1.74
56.12::0.4
6.5.2
Ablation study
Two main components that constructsour method (other than SSL training) are separate knowledge distillation (KD) stage, and learning KD as auxiliary loss for SSL's objective. We validate the effectiveness of each component through the following ablation study.
Y Is auziliary KD loss necessary? To validate this, we compare method with the case without auxiliary KD loss, which essentially runs plain AdaMatch algorithm after running KD in stage 1. We name this approach ZeroMatch without auxiliary loss ('ZM w.o. aux.' in tables) and provide benchmark results in Table 7 Our results show that adding auxiliary KD loss helps, especially in the low-label setting, For example, in the 125-label setting of Amazon Review with GPT-4o, our original method improves plain AdaMatch by 2.22%. Since pseudo-labell inferred from labeled data can become inaccurate low-label settings, adding additional supervision with teacher pscudo-labels can be particularly helpful in this case.
2 Does having separate KD stage necessary? Since stage 2 of our algorithm also includes auxiliary KD loss, may wonder if rumning stage 2 without stage 1 in our algorithm is enough to achieve the goal of merging benefits of KD and SSL. To clarify this, we compare our method with the case without running stage knowledge distillation. We name this approach ZeroMatch without stage 1 ('ZM w.o stage 1' in tables) and provide benchmark results in Table Our results show that while ZeroMatch without stage 1 can sometimes achieve similar performance original ZeroMatch (ex. in Yahoo Answers) it may also produce a large gap performance depending on the dataset. For example, in the 250-label setting of Amazon Review with 'A' pseudo-label set, 7M stage 2 only reveals 3.94% accuracy drop compared to original.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Table 9: Hyperparameter sensitivity results in Yahoo Answers (250 labels) and Amazon Review (125 labels) with pseudo-labels from FLAN-T5 models.
a, A
Yahoo Auswers
Amazon Review
(0, 0.5)
69.1510.46
59.55= 1.69
(0, 1.0)
69.90:1.87
60.20=1.022
(1, 0.5)
68.26:10.91
59.31=11.43
(1, 1.0)
70.71=0.36
59.17-11.20
Table 10: Computational cost on training with CIFAR100.
Method
VRAM(GB)
Training time (sec)
Adamatch
4.2
41100
ZeroMatch
5.4
61662
Table 11: Matchss performance combined with different SSL basolines (osselincs(Fixbllle (FixMatch, FreeMatch, SoftMatch) in Yahoo Answers with different FM pseudo-label sets.
SSL algo.
FicMatch
FoweMrcc
Malth
PM
sise
250
2000
300
Nome
SST. only
54.01-0.55
63.56422.03
66.50-00-48
60.84+2.91
66,27+0.45
67:29-0.77
62.30-0091
6484-0554
67.45+0.87
Zero-olict
G8.87 00.81=0000
08.0100000
66:810000
00.1:0000
0880=0000
08.81600000
ZarrMatch (ours)
70.43+0.18
70.06+0.33
72.11+0.22
0..37
70.72+00.1
72.52+0.28
T1.46+0.900
71.20+0.20
72.22+0.25
LLamaa33
Zertosick
00:15-00.00
00.16+0.00
6011H1000
80.152000
60:15=0.00
30:1500000
60:16=0.00
69.16+0.00
-70h
(eurs)
70.45++.559
71.96+0.55
71.1740.07
71.30++0,25
A
ZeroMatch (ours)
06:63-0.00
09.64:0.72
71.19:0.20
66.63460 00
71.78+0..0
66.63+00.00
70.36:0040
06.65400.00
70.56++0.83
066634000
71.47=0.27
06:63-000
69.92:t0.75
06:630000
7..68:0.64
06.63-0.00
0..05
66 66.63+00000
B
ZexoMatch (ours)
66.66=11.49
66.82=0040
20.20110000
30.20100000
60.600:0022
67.79=00.82
36.20100000
68.24440088
33.2910000
70.16=00.10
00:20-0000
67.64=0.577
20.000
07.47=0.62
69.98=20.2
20100000000
6.5.3
Hyperparameter sensitivity analysis
For simplicity of settings, we mainly benchmark our method with hyperparameters ap = 1 and Ap = 1 without finding the optimal setting. The reasoning behind this choice is to make the KD loss the same as fully-supervised training when pseudo-labels are accurate, and to enable annealing to allow more freedom to accommodate labeled data in the early stage of AdaMatch, which develops initially confident predictions for unlabeled data.
In this section, run a sensitivity analysis on combinations of these parameters. We conduct experiments with four different sets: (ap, Ap) = (0, 0.5), (0, 1.0), (1, 0.5), (1, 1.0). The results with NLP datasets with FLAN-T5 models are shown in Table g We find that while (ap. Ap) = 1.0) can be a good candidate on some datasots (ex. Yahoo Answers), there may exist other optimal parameters that can further improve performance.
6.5.4 Computational cost analysis
We provide memory usage and computation cost associated with our method and the SSL baseline AdaMatch) for CIFAR100 dataset in Table I0 The usages are measured on our machine with 1 NVIDIA A10G GPU. We note that our method consumes around 50% more training time compared with AdaMatch. This is due to the separate knoweledge distillation at stage 1, which utilizes half of the batch size of AdaMatch. Additionally, our method uses slightly more VRAM due to propaattoggg through weal-anggmented samples in KD auxiliary loss in stage 2, while in AdaMatch weak-augmented samples are only used with a forward pass to get predictions and back-propagation happens only in strong augmented samples. Possible strategics to reduce training runtime and memory usage are discussed in Appendix C.1 and Appendix
C.2
6.5.5 ZeroMatch with other SSL algorithms
While we mainly use AdaMatch as the SSL bascline to use with ZeroMatch we also provide benchmarks of our method with 3 additional SSL algorithms: FixMatch (Sohn et al.) (020). FreeMatch (ang et al. 2022c), and SoftMatch (Chen et al. 2023). Results the Yahoo Answers dataset with 4 different FMs (GPT-40,
13

---

Published in Transactions on Machine Learning Research (12/2025)
Llama3.3-70b, FLAN-T5 (A), FLAN-T5 (B)) are shown in Table 11 The results show that ZeroMatch improves both SSL and zero-shot accuracy in all cases, implying ZeroMatch's applicability to wide range of SSL methods. While we expect that ZeroMatch can work with many types of SSL algorithms with confidence-baset sampling, we note that counterexamples may exist. One such example is UDA (Xie et al., 2020), which uses training signal annealing (TSA) to utilize only the low-confidence labeled samples in the initial training stage. This procedure conflicts with our solution since we have high-confidence samples at the start of SSL training, resulting in labeled samples not being utilized.
7
Conclusion
This work introduces ZeroMatch, framework for robustly integrating foundation model predictions into semi-supervised learning. Through extensive experiments across multiple datasets from different domains, we demonstrate that Zero.atth achieves Saate-of-the-art performance while maintaining robustness to varying pscudo-label quality. Our learning- baaning-based mechanism effectively balances between limited labels and plentiful pseudo-labels, enabling practitioners to leverage foundation models without the computational overhead of direct fine-tuning. Future work could explore extending Marhh to more complex tasks and investigating theoretical guarantees for the weighting mechanism.
Broader Impact
Our work aims to make semi-supervised learning more accessible by reducing the computational resources needed to leverage foundation models. This has positive implications for democratizing Al development, particularly benefiting researchers and organizations with limited computing infrastructure.
However, improved semi-supervised learning techniques could also lower the barrier for developing Al systems with potential misuse cases. While method focuses improving efficiency rather than expanding capabilitics, we acknowledge the need for careful consideration of deployment contexts.
The adaptive weighting mechanism we introduce could polentially mitigale some biases in foundation models by automatically reducing reliance on unreliable predictions. However, when the number of labeled data are low, this may not hold, and our method may propagate or amplify model biases into downstream student models. Further investigation is needed regarding the method's behavior across different demographic groups and task domains.
8 Acknowledgements
This work used the Savio computational cluster resource provided by the Berkeley Research Computing program at the University of California, Berkeley (supported by the UC Berkeley Chancellor, Vice Chancellor for Research, and Chief Information Officer). This work also used Delta GPU at NCSA through allocation C1S250269 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
14

---

Published in Transactions on Machine Learning Research (12/2025)
References
Meta AI. Llama 3.3-70b instruct model card, 2024. URL hatpps//uuuuuugeeee.ee/eeeeeaaar
1-70B-Innstrue
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In 8th International Conference on Learning Represenlations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL http:://openrreiii..eeeeeeeee/ee
David Berthelot, Rebecca. Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alexey Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. In International Conference on Learning Representations, 2021.
Lucas Beyer, Xiaohua Zhai, Amélic Royer, Larisa Markoeva, Rohan Anil, and Alexander Kolesnikov. Knowl- edge distillation: A good leacher palient and consistent. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10925-10934 2022.
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE transactions on neural networks and learning systems, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. 2020.
Advances in neurol information processing systems,
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compre ssio. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535 555,,,1,- 2006.
Ming-Wor Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. Importance of semantic representation: dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI008, pp. 830 835. AAAI Press, 2008. ISBN 9781577353683.
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning. arxiv preprint arXic:2301.10921 2023.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33: 22243-22255, 2020.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and stale of the art. Proceedings of the IEEE, 105(10):1865 1883, 2017.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghann, Siddhartha Brahma, et al. Scaling inst wuutionn-ineeuune language models. Journal of Mochine Learning Research, 25(70):1-5) 53, 2024.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition workshops, pp. 702-703, 2020.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprinl Xr:::11810.044 04805, 2018.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprinl ar Xiv:20000. 11929, 2020.
Kai Gan and Tong Wei. Erasing the bias: Fine-tuning foundation models for semi-supervised learning. In forly-forst International Conference Machine Learning, 2024.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 1:xi::1122 6211, 2013.
Lingxiao He, Wu Liu, Jian Liang, Kecheng Zheng, Xingyu Liao, Peng Cheng, and Tao Mei. Semi-supervised domain generalizable person re-identification. ar preprint wrXin:2108.056 2021.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tablim: Few-shot classification of tabular data with large language models. In International Conference Artificial Intelligence and Statistics, pp. 5549-5581. PMLR, 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint orXiv:1503.02555 2015.
Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. Mecta: Memory-cconomic continual test-time model adaptation. In 2023 International Conference Learning Representations, 2023.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiient transfer learning for nlp. In International conference machine learning, pp. 1790-2799, PMLR, 2019.
Cheng-Yu Yu Hsieh, Chun-Liang Li, Chin-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint ar arXiv:2305.0280 023011 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Menglin Jia, Luming Tang, for-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, pp. 109-727.22 Springer, 2022.
Weifeng Jiang, Qianren Mao, Chenghua Lin, Jianxin Li, Ting Deng, Weiyi Yang, and Zheng Wang. Disco: distilled student models co-training for semi-supervised text mining. arXiv preprint 2rX1v:2305.12074,
2023.
Sudipta Kar, Giuseppe Castellucci, Simone Filice, Shervin Malmasi, and Oleg Rokhlenko. Preventing catastrophic forgetting in continual learning of new natural language tasks. In Proceedings of the 28th
ACM SIGKDD Conference on Knowledge Discovery and Date Mining, pp. 3137-3145, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations, 2022.
Hong-HYun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. 2013. URL https:://ppl: semaaticcccctnncccacrr.. http:e///pp..ssmmtticcllaa...r.a http:aaaw.rrg/ooppsei::111::101. 18507866
Eric Lehman, Evan Hernandez, Diwakar Mahajan, Jonas Wulff, Micah J Smith, Zachary Ziegler, Daniel Nadler, Peter Szolovits, Alistair Johnson, and Emily Alsentzer. Do we still need clinical language models? In Conference on health, inference, and learning, pp. 578-597. PMLR, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-eflcient prompt tuning. Xrxiv preprint Xiv:21100..0 08699 2021.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2022.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, pp. 165-172, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450324090. doi: 10.1145/2507157 57.25071633
URL hTR.attttt// http::///lel..rrr//...1////////0 2507163
16

---

Published in Transactions on Machine Learning Research (12/2025)
Cristina Menghini, Andrew Delworth, and Stephen Bach. Enhancing clip with clip: Exploring pseudolabeling for limited-label prompt tuning. Advances in Newral Information Processing Systems, 36:60984-61000 2023.
Jachyun Nam, Woomin Song, Scong Hyeon Park, Jihoon Tack, Sukmin Yun, Jachyung Kim, and Jinwoo Shin. Semi-supervised tabular classification via in-context learning of large language models. In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sirth Indian conference computer vision, graphies & image processing, pp. 722-729. IEEE, 2008.
Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
OpenAl. Gpt-40 technical report, 2024. URL https///ppenai..cmmmnee////// 2025.
OpenAL. Introducing gpt-4.1 in the api. https://opente wwt.:/nddw//ggt---1111 http::://ppessi..c////aaea/////a
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations, 2021.
Judy Hanwen Shen, Inioluwa Deborah Raji, and Irene Y Chen. The data addition dilemma. In Machine Learning for Healthcare Conference. PMLR, 2024.
Liangliang Shi, Zhengyan Shi, and Junchi Yan. Selkd: Selective knowledge distillation via optimal transport perspective. In The Thirteenth International Conference on Learning Representations.
Zhengxiang Shi, Francesco Tonolini, Nikolaos Aletras, Emine Yilmaz, Gabriella Kazai, and Yunlong Jiao. Rethinking semi-supervised learning with language models. In Findings of the Association for Computational Linguistios: ACL 2023, pp. 5614-5634, 2023.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexcy Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised lcarning with consistency and confidence. Advances in neural information processing syslems, 33:596 608, 2020.
Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-effccient continual test-time adaptation via self-distilled regularization. Tn Proceedings of the IEFE/CVF Conference on Computer Vision and Pottern Recognition, pp. 11920 11929, 2023.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference machine learning, pp. 9229 9248. PMLR, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve supervise deep learning results. Advances in neural information processing systems, 30, 2017.
Raviteja Vemulapalli, Hadi Pouransari, Fartash Faghri, Sachin Mchta, Mchrdad Farajtabar, Mohammad Rastegari, and Oncel Tuzel. Knowledge transfer from vision foundation models for efficient training of small task-specific models. Xr preprint arXi::2311.1888 2023.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Lukas Vöge, Vincent Gurgul, and Stefan Lessmann. Leveraging zero-shot prompting for efficient language model distillation. arxiv preprint Xrx:224003. 15886, 2024.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and S Yu Philip. Generalizing to unseen domains: A survey domain generalization. IEEE transactions on knowledge and data engineering, (5(8))8052-8077 2022a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Kegin Chen, Xuejing Läu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. Xie prepriul or X0X: 12191, 2024.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, et al. Usb: A unified superviee learning benchmark for classification. Advonces in Neural Information Processing Systems, 35:3938 100119961111 2022b.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, et al. Freematch: Self-adaptive thresholding for semi-supervised learning. Xiviv preprint orx::22005.022 2022c.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. Xiv preprint arXin:2110.07178. 2021.
Bangquan Xie, Zongming Yang, Liang Yang, Ruifa Luo, Jun Lu, Ailin Wei, Xiaoxiong Weng, and Bing Li. Spd: Sumi-super learning and progressive distillation for 3-d detection. IEEE Transactions on Neural Networks and Learning Systems, 35(3):3506 (333333533-3513 2022.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmen- tation for consistency training. In Н. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6256-6268. Cur- ran Associates, Inc 2020. URL https://proceedings www.gg...urip..... cccppps.ccppppprrrr filles//aper:2200//ie// 044800901a3333112277788888300
.pf
Jing Yang, Xiatian Zhu, Adrian Bulat, Brais Martinez, and Georgios Teimiropoulos. Knowledge distillation meets open-set semi-supervised learning. International Journal of Computer Vision, 133(1):315 334, 2025.
Zebin You, Yong Zhong, Fan Bao, Jiacheng Sun, Chongxuan Li, and Jun Zhu. Diffusion models and semi-supervised learners benefit mutually with few labels. Advances in Neural Information Processing Systems, 36, 2024.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 1118408--8419) 2021.
Jiahan Zhang, Qi Wei, Feng Liu, and Lei Feng. Candidate pseudolabel learning: Enhancing vision-languee models by prompt tuning with unlabeled data. arXiv preprint arXiv:2406. 0000000.15002 10502, 2024.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, Benjamin Rozonoyer, Md Arafat Sultan, Yoon Lee, Mohit Tyyer, and Andrew McCallum. Multistage collaborative knowledge distillation from large language model for semi-supervised sequence generation. arXiv preprint ar trXii:2311.08666 2023.
Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi-supervised learning with similarity matching. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14471-1448) 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lision-language models. International Journal of Computer Vision, 130(9):2337-29)) 2348, 2022.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Banghua Zhu, Mingyu Ding, Philip Jacobson, Ming Wu, Wei Zhan, Michael Jordan, and Jiantao Jiao. Doubly-robust self-training. Advances in Neural Information Processing Systems, 36, 2024.
Jinhua Zhu, Yingee Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. Incor- porating bert into neural machine translation. In International Conference on Learning Representations, 2020.
19

---

Published in Transactions on Machine Learning Research (12/2025)
A Experiment Details
A.1 Datasets
We use six datasets from the NLP and image classification domains. We provide description of each of them.
CIFAR-100 CIFAR-100 (Krichessky 2009) is 32x32 pixel natural image recognition dataset consisting of 100 classes. There are 500 training samples and 100 test samples per class.
Flowers102 Flowers102 (Nissback & Zisserman 2008 is an image dataset containing 102 flower categories commonly found in the United Kingdom. The dataset contains a minimum of 40 and a maximum of 258 images per class.
Resisc45 RESISC45 (Cheng et al. 2017) is a publicly avaílable benchmark for Remote Sensing Image Scene Classification. Images from 45 kinds of scenes are provided.
Yahoo Answer Yahoo Answers (Chang et al. 2008 is a question text topic classification dataset with 10 topic categories. Each class contains 140,000 training samples and 6,000 test samples. Training and validation are created by sampling 50,000 and 5,000 samples per class from the training set, and the test set is unchanged, following the data settings in USB |Wang et al. 2022a
AG News The AG News (Zhang et al. 2015) dataset is a news topic classification dataset with 4 classes. The original dataset contains 30,000 training samples and 1,900 test samples per class. In our experiments. 25,000 samples and 2,500 samples per class are sampled from the training set, following USB (Wang et. al. 2022a). The test dataset is unchanged.
Amazon Review The Amazon Review (McAuley & Leskoved 2013) dataset is sentiment classification
dataset with product review text input. There are 5 classes indicating the review score. Each class contains 600,000 training samples and 130,000 test samples. Following USB (Wang et al., 2022a 50,000 samples and 5,000 samples per class from the training set are used as the training and validation datasets respectively. The test dataset is unchanged.
We the data splits provided by USB (Wang el al. 2022b) for the NLP datasets, and the splits provided by CPL Zhang et al. 2024 for Flowers102 and Resisc45.
A.2
Obtaining pseudo-labels from foundation models
In this section, we discuss details of obtaining the zero-shot pseudo-labels used to train with Zarchatch.
A.2.1 Details of pseudo-labe label generation process
Image classification datasets For the image classification datasets, mainly GPT-4.1 and CLIP models to generate pseudo-labels.
For GPT-4.1 (OpenAl 2025 1, we use the chat completion feature to prompt for image classification. We use the following system prompt to describe the task:
You are image classification system, asked to classify (topic). Classify the input image into exactly one of the following types: {candidate labels) Respond with only the type that best fits the image.
The topic variable set as 'flowers' for Flowers102, 'scene' for Resisc45, and 'tiny image' for CIFAR100. Candidate labels are set as the possible labels in text format. The same procedure applied when using other chat-completion--aseed models (GPT-4o (OpenA 2024), Qwen2-VI. VT. (Wang et al.] 2024 IJama3. 2- 2-Vision AA]
20

---

Published in Transactions on Machine Learning Research (12/2025)
2024
After obtaining the output, parse the output text to check if text labels in the candidate label set exist in the output. In cases where no label is detected, we use a default pseudo-label: rose for Flowers commercial area for Resisc45, and road for CIFAR100.
When using the CLIP model, we follow the procedures zero-shot classification in Radford et al. (2021) J. We compute the feature embedding of a given image, and for each label the candidate labels we prompt the model with the following template: This is a photo of {label} and compute its embedding. We then obtain the pscudo-label by finding the label that gives the closest embedding to the image embedding based cosine similarity.
NLP classification datasets - topic classification: AG News, Yahoo Answers To obtain pscudo- labels for topic classification tasks, we use the following prompt template with GPT-4o, Llama3.3-70B, and FLAN-T5 modols for a given input text:
Select the topic that the given article is about. The topics are: Landidate_labelss
Article: fimput_ text),
Answer:
For AG News, the candidate labels are world, sports, business, technology. For Yahoo An- swers, the candidate labels are society, science, health, education, computer, sports, business, entertainment, relationship politics. If the output does not contain labels from the candidate label set or is not parsable, we set the pscudo-label to be world for AG News and society for Yahoo Answers the default label. For GPT-4o and Llama models, we use the chat completion template with the following helpful assistant.
system prompt: You
NLP classification datasets sentiment classification: Amazon Review We use the following prompt template with GPT-4o, Llama3.3-70B, and FLAN-15 models for given input text to obtain the sentiment of the review:
fimput_ text)
What is the sentiment of this review?
OPTIONS:
very negative
negative
neutral
positive
very positive
If the model output is not recognized as one of the candidate labels, we select neutral as the default pscudo-label. For GPT-4o and LLaMA models, the chat completion template with the following system prompt: You are helpful assistant.
A.2.2 Zero-shot benchmark results and foundation model selection
To select foundation models to be used with ZeroMatch, we consider (1) the best-performing large foundation models that can provide maximum statistical performance with our method and (2) the lowest-quality foundation models to test the robustness of our method against inaccurate pseudo-labels.
We provide zero-shot benchmark results on the test set in Table 12 (NLP) and Table 13 (Image).
We found that GPT-4o and Llama3.3 were the highest performing overall, scoring top-2 on all datasets. Among FLAN-T5 models, we find that FLAN-T5-XXI. performs the best on Yahoo Answers and AG News, while the FLAN-T5-XI. model performs better on Amazon Review, To benchmark our method with highest and lowest accuracy, FLAN-T5-XXT, and FLAN-T5-small were selected A and B sets of pseudo-labels for
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table 12: Zero-shot pseudo-label accuracies (%) for NLP datasets (test set).
FM
Yahoo Answers
AG News
Amazon Review
GPT-40
68.81
86.25
59.14
Llama3. 3-70B
69.15
88.41
55.79
FLAN-T5-XXT.
66.62
91.43
45.61
FLAN-T5-XI.
63.97
91.39
52.37
PLAN-15-large
61.33
87.66
51.96
FLAN-T5-bes
55. 17
88.68
42.23
FLAN-T5-sman
29.44
87.07
35.7
AG News and Yahoo Answers, and for Amazon Review, FLAN-T5-XL and FLAN-T5-small as A and B sels.
Table 13: Zero-shot pseudo-label accuracies(%) for image datasets (test set).
FM
CIFARI00
Flowers102
Resisc45
GPT-4.1
83.25
88.37
79.28
GPT-4o
81.5
86.0
73.46
Qwen2- VL
62.58
78.11
68.25
Llama3.2 2-90B- Vision
36.05
59.02
44.58
CIJP--larg-patcc11
62.27
72.13
60.32
HLIP-base-patchie
49.49
60.17
49.63
For the image classification task, we benchmarked GPT-4.1, GPT-4o, Qwen2-VT, and T.I.MA3.2-900 2-..-------iiiiio
with our target datasets, and found that GPT--411 was best in all tasks (see Table 13 As a result, use GPT-4.1, CLIP-large, and CLIP-base models for our benchmark.
A.3 Training setup
Sctup for image classification We use VIT (Dosovitskiy 2020) as the backbone of our model and employ pre-trained ViT-Small models provided by the USB framework (Wang et al., 2022a For CIFAR100, ViT-Small with patch size of 2 and image size of 32 is used, and for Flowers and Resisc45 datasets, ViT-Small with a patch size of 16 and image size of 224 is used. Optimization is performed using the AdamW optimizer with a cosine learning rate schedule given by 7) 7o cos [66 ), where the initial learning rate is set 7md
to No = 5 10-1 for CIFAR100 and to = 1 x 10-4 for Flowers102 and Resisc45. For CIFAR100, we train the model for a total of = 204,800 steps, with a warm-up phase of 5,120 steps, and for Flowers102 and Resisc45, we run for half the number of training steps, = 102,400, with 2,560 warm-up steps to reflect the smaller training set size. The batch sizes for both labeled and unlabeled data are set to 16. For AdaMatch-specife parameters, the threshold for the utilization cutoff mask is set to 0.95. For data augmentation, we apply random cropping and random horizontal flipping for weak augmentation, while RandAugment (Cubuk et al, 2020) is utilized for strong augmentation. For the knowledge distillation stage (stage 1 of ZeroMatch), we run supervised training with teacher pseudo-labels for the same number of training steps. Regarding the hyperparameters specific to our method, we report results with Xp = and with annealing (ap = 1). Detailod hyperparameter settings are provided in Table 14
Setup for NLP classification For NLP tasks, we fine-tuee the pre-trained BERT-Base (Devlin 2018 using the AdamW optimizer with a cosine learning rate schedule. The total number of training steps is set to 102,400, with a warm-up phase of 5,120 steps. Both the labcled and unlabeled batch sizes are set to 4. All input lext is iruncated lo ensure its length remains within the context length of BERT-Base. For data augmentation, we employ back- translatnnntoon (Zhu et al. 2020 using German -nnglihh and Russian Russian-English translation as strong augmentations. No weak augmentation is applied, and the original text input used instead. We run AdaMatch with cutoff threshold of 0.95. Dataset-specffic hyperparameters are detailed in Table 15 For the knowledge distillation stage (stage 1 of ZeroMatch),
supervised training with
22

---

Published in Transactions on Machine Learning Research (12/2025)
teacher pseudo-labels for the same number of training steps. We report the performance of ZeroMatch with Ap 1 and with annealing (ap- = 1).
Table 14: Hyperparameters of Image classification datasets.
Dataset
CIFAR100
Flowers102
Resisc45
Image Size
32
224
224
Model
ViT-S-P4-32
VIT-S-PP6-224
ViT-S-P16-224
Weight Decay
5e-4
Labeled Batch size
16
Unlabeled Batch size
16
Learning Rate
5e-4
le-4
1e-4
Layer Decay Rate
0.5
0.65
0.65
Scheduler
77
17u COS 16K
7mk
Model EMA Momentum
0.0
Prediction EMA Momentum
0.999
Random Crop, Random Horizontal Flip Random Horizontal Flip
Random Resized Crop and Interpolation,
Weak Augmentation
Strong Augmentation
RandAugment (Cubuk et al. 2020
ap
Ap
1.0
Table 15: Hyperparameters of NLP tasks.
Dataset
AG News
Yahoo! Answer
Amazon Review
Max Length
512
Model
Bert-Base Basc
Weight, Decay
le-4
Labeled Batch size
4
Unlabeled Batch size
4
Learning Rate
5e-5
1e-4
5e-5
Layer Decay Rate
0.65
0.65
0.75
Scheduler
7/o cos
7ark I6K
Model EMA Momentum
0.0
Prediction EMA Momentum
0.999
Weak Augmentation
None
Strong Augmentation
Back- Translaaannn
Xie et al. 2020
ap X
1
1.0
A.4 Additional implementation details
Pseudo-supervision For the pseudo-supervision baseline, a supervised learning algorithm is run with fully labeled data (filled with FM pseudo-labels). To ensure a fair comparison with SSL baselines, the same number of labeled and unlabeled batches participating during optimization. In each optimization step, when labeled batch of size Br, and an unlabeled batch of size By are given, pseudo-supervision optimizes the following loss:
B1 Hly! p(yla()) + > Hil! p(yl=!1))
Bu
LPs B
Σ
where B Br, By-
23

---

Published in Transactions on Machine Learning Research (12/2025)
Pseudo-label as feature input To implement the sarudo--abells--fatu------- baseline, we change the model architecture to accommodate one-hot pseudo-labels. When a training sample with its corresponding one-hot converted pseudo-label Vi passes through the model with encoder and classifier head h, the model concatenates encoder output g(z;) and the one-hot pscudo-label Di as additional features, producing h(g(x,) (6)) as output. Note that the input dimension of head h is adjusted to accommodate the additional feature dimension (number of classes). With each sample paired with pscudo-label, this modification allows running the SSL baseline (AdaMatch) without further modification, and the model can learn to accommodate ignore the pseudo-label feature depending how helpful it is for learning the task.
Accommodating embeddings to ZeroMatch To accommodate feature embeddings from foundation models in ZeroMatch, we concatenate embeddings with our encoder output before passing them to the classifier head. When the model has an encoder and classifier head h, we modify the input dimension the MLP head so that encoded features and embeddings are concatenated and fed into h. As a result, when training sample z₁ with its embedding e; is given, the model outputs hgg(zz), ez). The same procedure is applied to the auxiliary KD classifier head for stage 2 in our algorithm, producing hp(g(2;), e;) to be supervised with teacher lseudo-labels.
B Additional comparsion with previous works
B.1 Compare with fine-tuning foundations models for semi-superrised learning
As discussed in Sec. 5 direct fine-tuning large foundation models (or using their parameter-efficient variants) may offer higher statistical performance than ZeroMatch by leveraging their vast number of parameters. However, such approaches incur significantly higher memory and computational costs. In contrast, our set up trains a compact backbone model that requires only modest resources (less than 8GB of GPU memory), making it suitable for resource-constrained environments. For fair comparison, we compare our method against recent approaches that fine-tune FMs for SSL and have computational requirements comparable to ours.
We compare our approach with GRIP Menggini et al. 2023 CPL Zhang el al. 2024 and FineSSL GGan & Wei 2024), which adopt prompt-tuning strategies for vision language models such as CLIP (Radford et al, 2021 and iteratively refine pseudo- label predictions based on confidence and features.
While direct comparison of these works with ours not possible due to settings that differ largely in model and training algorithms, present scenarios where method can outperform these approaches, and provide comparison details on training and information used. We benchmark our method the exact same data settings that these works have conducted their experiments on.
Visual prompt tuning methods Visual Prompt Tuning (VPT) (Jia et al. 2022 is a prompt-tuning method for langune models that tunes the prefix part of the input layer while keeping the textual encoder fixed. FineSSL, CPL and GRIP provide VPT benchmarks with CLIP models. We provide comparisons of ZeroMatch FineSSL's CIFAR-100 results in Table 16 and results of GRIP and CPL with Flowers102 and Resisc45 in Table 17
Table 16:
Comparison with FineSSL's VPT (visual prompt tuning) results on CIFARI00 with 400 labels (4 labels per class). indicates numbers published in FineSSL.
Method
FM Inference info used
Training backbone
Fine-tuning method
Accuracy
F
CLIP VIT-B-16
VPT-deep
80.44=0.24
Zero-shot
CLIP ViT-L-14
pseudo-label
60.65=0.00
ZeroMatch (ours)
CLIP ViT-I-14 pseudo-label
ViT-S-2
Full fine-tuming
86.73+1..42
Our results show ZeroMatch outperforming all the baseline methods. This largely due to leveraging high-quality outputs of larger foundation models: for CIFAR-100, we use pseudo-labels from CLIP VIT-I-14,
24

---

Published in Transactions on Machine Learning Research (12/2025)
Table 17: Comparison with GRIP and CPL's VPT results on Flowers102 and Resisc45 datasets with labels
per class. * indicates numbers published in the paper.
Method
FM Inference info used
Training backbone
Finetuning method
Flowers102
RESISC
Zero-shut*
CLIP ViT-B-32 pscudo-label
63.67
54.48
GRIPs
CLIP text encoder embedding
CLIP VIT-B-32
VPT
67.95+1.2
71.22+0.77
CPL*
CLIP toxt encoder embedding
CLIP VIT-B-32
VPT
73.52-0.37
78, 78.4610.74
Zero-shot
GPT-4411 pscudo-labcl
88.37
79.28
ZeroMatch (ours)
GPT--111 pseudo-label
VIT-S-16
Full fine-tuning
95.17:10.88
87.65=10.73
and for Flowers102, leverage GPT-4.1. We also note that the training backbone use is smaller than the baselines. We use ViT-Small, which uses fewer attention heads compared to CLIP-ViT-B models.
This result implies that our method can outperform many baselines that fine-tune foundation models by simply leveraging the highest-quality foundation model, while training a smaller model for the task on a low-compute device.
Textual prompt tuning methods Textual Prompt Tuning (Zhou et al.) 2022) is another common prompt-tuning method which tuncs the input prefix of the textual encoder of vision-language models while keeping the visual encoder fixed. The textual encoder is trained with embeddings obtained from large visual encoder, and it can be implemented using the visual encoder in inference mode only. This setting resembles our problem setup in that the inference outputs of larger models are used to train smaller model. We provide benchmarks comparing our method to GRIP and CPL's textual prompt tuning results in Table 18
Table 18: Comparison with GRIP and CPL's TPT (Textual Prompt Tuning) results Flowers102 and Resisc45 datasets with labels per class. indicates numbers published in the paper.
Method
FM Inference info used
Training backbone
Finetuning method
Plowers102
RESISC
Zero-shot*
CLIP VIT-B-32
peeudo- label
63.67
54.48
GRIP-
CLIP VrT-B-32
embedding embedding
CT.IP text encoder
Textual prompt tuming
83.6++0.68
74.11-0.66
CPL,
CLIP ViT-B-32
CLIP text encoder
Textual prompt tuming
0.366
8.198=0.11
Zero-shot*
CLIP VIT-L-14
pseudo-label
73.98
62.67
CPL*
CLIP VIT-L-14
embedding
CLIP text encoder
Textual prompt tuming
96.80=0.63
87.75=0.29
Zero-shot
GPT-4.1 pseudo-label
88.37
79.28
ZeroMatch (ours)
GPT-4.1 pseudo-label
VIT-S-16
Full fine-tuning
95.17:10.88
87.6 87.65-0.73
GPT-4.1 pseudo-label +
ZeroMatch (ours
VIT-S-16
Full fine-tuning
97.20=10.41
88.5211077
CLIP VIT-L-14 embedding
While our method paired with pseudo-labels from GPT-4.1 can outperform CPT. and GRIP with CLIP-VIT: B, CPL's results with CLIP ViT-L outperform our score. We additionally benchmark our method using embeddings from the CLIP ViT-L model, which CPL uses as the supervision source. The results show that our method improves with added embedding information, outperforming other baselines, implying that embeddings are a helpful information source for learning the benchmark task.
B.2 Comparison with doubly-robust self-training
Doubly-robust self-training (Zhu et al. 2024 addresses problem setting similar to ours, proposing method that robustly improves performance despite varying quality of pseudo-labels. The method employs the following loss function:
N1 Σ
N2 Σ N2 HiG p()l=())
Nr
No HCo p()la) + Σ Hil p(ol?))
CDR
1
H(y! p(ulo())
1
Σ
N1
When the pseudo-labels are accurate, the loss function approximates training with the full dataset D2 and Du, both with ground truth labels. Conversely, when the pseudo-labels are inaccurate, the loss function
25

---

Published in Transactions on Machine Learning Research (12/2025)
asymptotically approaches training with only the labeled samples Dr. The work provides convergence improvement guarantees over training with labeled data alone, under varying pseudo-label quality, under certain regularity conditions. While the study presents experimental results using single pseudo-label set inferred from a pre-trained model, its practical effectiveness with inaccurate pscudo-labels in low-label settings is unexplored. To implement this method and ensure a fair comparison with ours, we use the same number of labeled and unlabeled batches participating during the optimization process. In each optimization step, when labeled batch of size Bt and an unlabeled batch of size By are given, the method optimizes the following:
Br.
Br Hot p(yl+t)) +
Br.
By
1 B
Hot p(ul=()) + Σ nol p(wle!))
LDa BL
Hls p(ul)t))
Σ
Σ
where B BL + By and 04 represents the anncaling parameter, following the original implementation in Zhu
et al.
2024
We provide comparison of the doubly-robust method with ZeroMatch on Yahoo Answers and Amazon Reviews in Table 19] Our results show that our method outperforms the doubly-robust approach in all settings. We hypothesize that this is due to the asymptolic guarantees of the doubly-robust. self-training method failing to hold in the low-label regime.
Table 19: Comaparison with doubly-robust self-training method in Yahoo Answers and Amazon Review with pscudo-labels from FLAN-T5 models. Median and standard deviation of accuracy (%) of 3 different random seeds are reported. Best score among experiments with same pseudo-label set are in bold. 'FM refers to the foundation model used to generate the pseudo-laber set.
Dataset
Yahoo Answers
Amazon Review
FM
Label size
250
500
2000
125
250
1000
Doubly-robust
$1.39+12.13
14.32:10.91
47.915.68
46.94:10.22
47.25:10.3
47.7610.87
A
Zero-shot
66.63:10.00
66.63+40.00
66.63:10.00
52.37:10.00
52.37=10.00
52.37:+0.00
ZeroMatch (ours)
69.78
70.51=0.19
71.81=0.07
58.83+00.1
59.69+0.60
60.95:0.13
Doubly-robust
24.5314.88
22.52+5.54
33.88:+2.07
35.8+0.13
35.9+0.07
35.961:008
B
Zero-shot
35.29:10.00
35.29
35.29:10.00
35.7+0.00
35.7+0.00
35.7+0.00
ZeroMatch (ours)
67.05:10.76
67.1310.73
69.61
50.23+1.335
53.7811.77
56.12:0.64
C Additional experiments
C.1 Strategy to reduce training time overhead
ZeroMatch's KD stage (stage 1) causes around a 50% increase in training time compared with SSL (shown in Table 10p. This is due to using the exact same number of training steps from the original SSL algorithm to avoid introducing new hyperparameters in our method. To mitigate this issue, we can use smaller number of training steps with larger learning rate to reduce the runtime. In Table 20 we provide experimental results on the Amazon Review dataset with new setting that reduces the mumber of KD training steps to 10% and increases the learning rate to 5e-4, and compare with the original ZeroMatch (denoted by KD step ratio 100%) and ZeroMatch without stage 1 (denoted by KD step ratio 0%).
We observe that the new setting (10%) performs close to the original (100%), and still produces a large improvement compared to the case without stage (0%). Note that reducing the KD steps to 10% reduces the overall training runtime overhead of our method from 50% 5%.
C.2 Strategy to reduce memory overhead
In AdaMatch, weak augmentations are used only to obtain predictions and do not require storing activations for backpropagation. In contrast, ZeroMatch's auxiliary KD loss in stage involves labeled samples and weakly augmented unlabeled samples, requiring activation storage until backpropagation happens. When
26

---

Published in Transactions on Machine Learning Research (12/2025)
Table 20: Experimental results of reducing KD steps on Amazon Reviews dataset.
Label size
FM
KD step ratio
KD learning rate
125
250
1000
0%
N/A
55.16+0.30
55.75+0.65
58.53=0030
FLAN-T5-XL
10%
57.1810.22
59. 59.12:1..2
60.31=0.37
100%
le-4
58.83-10.51
59.69
60.95:10.13
0%
N/A
45.53+4491
53.1921.68
56,920.63
FLAN-T3-smal)
10%
5e-4
50.01#2.20
54.28
56.49-10.40
100%
le-4
50.23+1.35
53.78=1.74
56.12=0.64
combined with an SST. algorithm, ZeroMatch needs activations for added strongly augmented unlabeled batches for the SSL algorithm, increasing VRAM usage compared to running SSL only (shown in Table I0]. To reduce memory usage in ZeroMatch, the auxiliary KD loss in stage 2 can be modified by incorporating strongly augmented samples into the KD loss:
Bi Σ H(g! a(vla)))
1
Σ Hig ((vlA.(a))))
CKD2
where A, is the same strong augmentations applied to SSL loss discussed in Sec. ensures that activations of weakly augmented sample is not stored in method.
3.2
This modification
Table 21: Experimental results of ZeroMatch-RM on Amazon Reviews dataset.
FM
Label size
125
250
1000
ZeroMatch
58.83:10.51
59.6911.60
60.95-10.13
FLAN-T5-XL
ZeroMatch-RM
58.93-10.66
59.50+0.42
60.53++0.27
ZeroMatch
50.23:11.33
53. 53.7841.77
56.12:0..6
ZeroMatch-RM
19.23:12.40
53.03.0.599
56.80=10.86
We provide a performance comparison of the new method (denoted by ZeroMatch-RMi on the Amazon Reviews dataset in Table 21 We observe that ZeroMatch-RM performs close to original ZeroMatch while consuming 4.3 GB of memory, which is close to AdaMatch's usage (4.2 GB) in Table 10
Potential inductive bias mismatch
In our work, we implicitly assumed that pseudo-labels are generated by a reasonably reliable foundation model. These models are typically trained on well-curated datasets to ensure that their outputs follow a meaningful distribution. We assume that pseudo-labels and ground-truth labels have high correlation, and the method may not work well this not the case. One possible inductive bias mismatch case using FM pscudo-labels with added noise (c.g., label flipping with fixed probability). In such cases, the KD task may shift from learning meaningful representations to memorizing inpul noise pairs, which demands significantly more parameter space. This can be an issue in our setup where the backbone model is small, and may lead to significant underperformance compared with SSL.
E Limitations and Future Directions
While our work focuses on incorporating pseudo-labels from a single foundation model, future extensions could explore using temperatuue-ccontollldd logits or using multiple FMs and ensembling their contributions to provide richer supervision.
Because ZeroMatch already includes a knowledge distillation component that mitigates catastrophic forgetting and integrates smoothly with semi-supervised learning, future directions explore applying our method
27

---

Published in Transactions on Machine Learning Research (12/2025)
to continual semi-supervised learning setup. For example, when learning task, ZeroMatch could combine (1) pseudo-labels generated by the student model trained on previous tasks to reduce forgetting, and (2) new pseudo-labels from foundation models relevant to the new task.
ZeroMatch also naturally extends to semi-supervised domain adaptation settings, since its underlying SSL algorithm (AdaMatch) was designed for domain-shift. scenarios. For such settings, it may be beneficial include mixed batches of both source-domain and target-domain samples during the stage 1 KD phase help stabilize distillation and improve alignment across domains. Below are limitations of our work.
We benchmark our method only on the case where a fixed number of labeled data are sampled per class. This may not demonstrate practical scenarios where labels and classes have imbalanced distributions. Future directions can include benchmarking method in such settings.
While we assume that input data come from a single distribution, practical scenarios may include distribution shifts in the data, especially if collected from sensors on moving edge devices (e.g., cameras in different environments). Future work can explore solutions to adapt to such shifts.
While most of our experimental results show that our method outperforms the zero-shot method, there is no methodological guarantee that this always holds. Future directions can explore ensuring robust improvement over zero-shot results.
We also note that our method's performance inherently depends on the quality of the pseudo-labels generated from foundation models. Robustness against labels that are not from foundation models (e.g., weak supervision signals, patterned noise) not tested and may not work with our method. Future dircctions can further explore application different types of label noisc.
We do not include analysis on FM inference API cost. This cost can become a practical issue for large-scale unlabeled data when used with models with high API price or requiring multiple inferences for reasoning. Future directions can explore reducing API cost by using cheaper models for easier samples and requesting high-quality FM predictions only when needed.
28


Reviews:
[{"review_id": "M3MALTODhm", "reviewer": "Reviewer_Pb9P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary of the Paper**\n\nThis paper introduces ZeroMatch, a semi-supervised learning framework that integrates pseudo-labels from foundation models (FMs) into a two-stage training procedure. First, the student model undergoes knowledge distillation using FM-generated pseudo-labels on both labeled and unlabeled data; second, the model is refined via a standard SSL objective augmented with an auxiliary KD head to mitigate catastrophic forgetting. The method is motivated by a realistic on-device training scenario with strict compute and privacy constraints, where users can access FMs only through inference APIs. Extensive experiments across six datasets (vision and NLP), different label budgets, and multiple pseudo-label sources of varying quality show that ZeroMatch consistently improves over SSL baselines, zero-shot performance, and alternative pseudo-label integration strategies. The paper includes thorough ablations and demonstrates robustness even when pseudo-label quality is low.\n\n**Strengths**\n* _Well-motivated problem and clear narrative._ The authors present a compelling, realistic on-device training scenario and articulate why existing approaches (PEFT, prompt engineering, and simple pseudo-supervision) fail under these constraints. The storyline is coherent and easy to follow throughout the paper.\n* _Extensive and convincing empirical evaluation._ Experiments span six datasets across NLP and vision, multiple label regimes, and pseudo-label sources ranging from high-quality (GPT-4o/4.1, Llama 3.3-70B) to low-quality (FLAN-T5 B, CLIP-base). The method consistently improves performance, even when the quality of the pseudo-labels is poor. Ablations are comprehensive and include removal of key components, hyperparameter sensitivity, runtime analysis, and compatibility tests with other SSL baselines.\n* _Practical, easy-to-integrate method._ The approach only requires adding an auxiliary head and a KD loss; no FM finetuning or gradient access is needed. This makes it practical for settings with inference-only FM access and limited compute.\n* _Robustness to pseudo-label noise._ The method shows strong resilience across pseudo-label qualities: even low-quality pseudo-labels rarely degrade performance, and improvements typically grow with the number of labeled samples.\n\n**Weaknesses**\n* _Method description lacks important details._ The paper does not clearly specify how pseudo-labels are obtained and mapped into the task label space, especially for vision tasks using GPT-4.1 or CLIP. The mechanics of the auxiliary KD head (e.g., temperature, loss normalization, gradient interactions with the main head) remain vague, limiting reproducibility.\n* _Figure 1 is unclear and hard to interpret._ The figure does not visually distinguish teacher and student components, nor does it clearly illustrate the flow of pseudo-labels through the two stages. The role of the auxiliary KD head is particularly hard to infer.\n* _Limited discussion of limitations and failure modes._ The method increases training time and requires large-scale API inference over all samples, which may be costly in practice.\n* _Presentation issues and minor errors._ The paper contains multiple typos (“a SSL method”, “the both metrics”, “accomodate”, etc.) and occasional grammar issues.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s empirical claims are supported by extensive and carefully executed experiments across six datasets, multiple label budgets, and a diverse set of pseudo-label sources. The authors compare against strong SSL baselines, zero-shot performance, and several reasonable pseudo-label integration strategies, and the results consistently support the claimed improvements and robustness. The ablation studies further validate the contribution of each component, and the computational analysis aligns with the stated on-device training motivation. Overall, the evidence provided is thorough, convincing, and consistent with the paper’s claims.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a timely and practically relevant problem at the intersection of semi-supervised learning and foundation model usage under realistic compute and privacy constraints. Many researchers and practitioners in the TMLR community work with limited labeled data and increasingly rely on large foundation models; understanding how to effectively incorporate FM-generated pseudo-labels into SSL pipelines is of clear interest. The findings are likely to be valuable to a broad subset of the TMLR audience.", "requested_changes": "**Major**\n1) Clarify the pseudo-label generation process. Provide a detailed description of how pseudo-labels are produced and mapped to task labels across all modalities (e.g., prompt formats for NLP, label mapping or similarity scoring for vision models like CLIP or GPT-4.1).\n2) Improve the explanation of the auxiliary KD head and training dynamics. Elaborate on how the auxiliary KD head interacts with the main classifier (e.g., temperature settings, gradient flow, loss weighting). This part of the method is currently under-specified and needs clearer exposition.\n3) Revise Figure 1 for clarity. Clearly distinguish teacher vs. student components, show the flow of pseudo-labels, and better illustrate the two-stage process. The current figure is difficult to interpret.\n\n**Minor**\n4) Correct minor typos and improve grammar throughout the text.\n5) Discuss limitations and potential failure cases. Briefly describe scenarios where low-quality pseudo-labels might harm performance, and comment on the cost of large one-time FM inference. This will present a more balanced view.\n6) Add a short discussion on inductive bias mismatch. Explain how pseudo-labels from large FMs interact with smaller student models and whether label distributions require calibration or filtering.", "broader_impact_concerns": "No major ethical risks are apparent beyond standard concerns with using foundation models. The existing Broader Impact statement is adequate, but it could more explicitly acknowledge that FM-generated pseudo-labels may propagate or amplify model biases into downstream student models. A brief discussion of this risk would strengthen the section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZPGkJPvpJ5", "reviewer": "Reviewer_GdrS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces ZeroMatch, a framework that combines knowledge distillation from foundation models with semi-supervised learning to simultaneously leverage labeled data, unlabeled data, and zero-shot pseudo-labels. The method trains a compact student model while relying only on inference access to large foundation models, making it suitable for resource-constrained on-device training.\n\nStrengths:\n- The paper addresses a timely and interesting problem where users cannot fine-tune large foundation models but can query them for labels.\n- The two-stage design is intuitive and experimentally validated across many datasets, with different foundation model types and multiple baselines.\n- The method works with closed-source FM APIs, needs minimal VRAM, and reduces data-leakage risks.\n\nWeaknesses:\n- While effective, the framework would benefit from a clearer motivation and justification for how the auxiliary head stabilizes training and mitigates forgetting.\n- The pseudo-label generation pipeline is detailed in the appendix, but the main text could more clearly explain prompting, calibration, and consistency considerations.\n- Stage 1 KD cost is non-trivial: training time increases by ~50%, mitigation strategies could be better discussed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides comprehensive experimental evidence across domains and baselines. Improvements over both semi-supervised learning and foundation models-based zero-shot baselines are consistent and substantial.", "audience_yn": "Yes", "audience_explanation": "Leveraging foundation models in data-scarce settings is a major current topic and the results show clear performance gains without requiring fine tuning, which is attractive.", "requested_changes": "- Expand the explanation of why the auxiliary head mitigates catastrophic forgetting. Provide more analysis of the potential representation drift between the main head and the auxiliary KD head. Clarifying when and how auxiliary supervision helps, or might interfere, would strengthen the paper.\n- Discuss computational trade-offs in more detail, including possible strategies to reduce training time or memory overhead.\n- Briefly comment on whether ZeroMatch could synergize with related approaches in continual learning or domain adaptation, given its emphasis on preserving knowledge from multiple supervision sources.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0NbLFA67bq", "reviewer": "Reviewer_vqY2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "ZeroMatch is a two-stage Semi-Supervised Learning (SSL) framework that integrates Knowledge Distillation (KD) with consistency-based SSL. The goal is to train a compact student model effectively on limited labeled data and large amounts of unlabeled data, by leveraging powerful zero-shot pseudo-labels from Foundation Models (FMs). The entire process is designed for low-resource environments, accessing the FM only via a single inference API call.\n\nPros:\n1. The paper is well-formatted and easy to follow.\n2. The motivation for using SSL in a low-compute device scenario is interesting.\n3. Extensive results show the effectiveness of the proposed method.\n\nCons:\n1. Lack of novelty. The method combines existing common methods (i.e., KD and SSL) and does not propose any novel components or algorithms. It looks more like an engineering solution to me. And it is quite similar to the one[1] mentioned in the related works, but moving to a different setting and making minor modifications.\n2. Lack of comparison. There are many SSL methods introduced in the related works but none of them are compared with the proposed method. It casts doubt on the true effectiveness of the proposed method.\n\n[1] Knowledge distillation meets open-set semi-supervised learning, IJCV 2025\n\nGiven the above two major concerns, I think the paper does not meet the TMLR standard.", "claims_and_evidence_yn": "Yes", "claims_explanation": "see above", "audience_yn": "Yes", "audience_explanation": "see above", "requested_changes": "see above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)