Re-Evaluating the Impact of Unseen-Class Unlabeled Data on Semi-Supervised Learning Model
Authors: Rundong He
Yicong Dong
Lan-Zhe Guo
Yilong Yin
Tailin Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Safe Semi-Supervised Learning
Unseen-Class Unlabeled Data
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Re-Evaluating the Impact of Unseen-Class Unlabeled Data on Semi-Supervised Learning Model

Abstract: Semi-supervised learning (SSL) effectively leverages unlabeled data and has been proven successful across various fields. Current safe SSL methods believe that unseen classes in unlabeled data harm the performance of SSL models. However, previous methods for assessing the impact of unseen classes on SSL model performance are flawed. They fix the size of the unlabeled dataset and adjust the proportion of unseen classes within the unlabeled data to assess the impact. This process contravenes the principle of controlling variables. Adjusting the proportion of unseen classes in unlabeled data alters the proportion of seen classes, meaning the decreased classification performance of seen classes may not be due to an increase in unseen class samples in the unlabeled data, but rather a decrease in seen class samples. Thus, the prior flawed assessment standard that "unseen classes in unlabeled data can damage SSL model performance" may not always hold true. This paper strictly adheres to the principle of controlling variables, maintaining the proportion of seen classes in unlabeled data while only changing the unseen classes across five critical dimensions, to investigate their impact on SSL models from global robustness and local robustness. Experiments demonstrate that unseen classes in unlabeled data do not necessarily impair the performance of SSL models; in fact, under certain conditions, unseen classes may even enhance them.

Published as a conference paper at ICLR 2025
RE-EVALUATING THE IMPACT OF UNSEEN-CLASS UN- LABELED DATA ON SEMI-SUPERVISED LEARNING
MODEL
Rundong He"* Yicong Dong" Lanzhe Guo
Yilong Yin Tailin Wu
Department of Artificial Intelligence, School of Engineering, Westlake University
2 School of Software, Shandong University
3 School of Intelligence Science and Technology, Nanjing University
rundong tee he@ma sdu. edu. en, ylyinesdu edu. cn, wutallineestlake.edd
ABSTRACT
Semi-supervised learning (SSL) effectively leverages unlabeled data and has been proven successful across various fields. Current safe SSL methods believe that unseen classes in unlabeled data harm the performance of SSL models. However, previous methods for assessing the impact of unseen classes on SSL model per- formance are flawed. They fix the size of the unlabeled dataset and adjust the proportion of unseen classes within the unlabeled data to assess the impact. This process contravenes the principle of controlling variables. Adjusting the proportion of unseen classes in unlabeled data alters the proportion of seen classes, meaning the decreased classification performance of seen classes may not be due to an increase in unseen class samples in the unlabeled data, but rather decrease in seen class samples. Thus, the prior flawed assessment standard that "unseen classes unlabeled data can damage SSL model performance" may not always hold true. This paper strictly adheres to the principle of controlling variables, maintaining the proportion of seen classes in unlabeled data while only changing the unseen classes across five critical dimensions, to investigate their impact on SSL models from global robustness and local robustness. Experiments demonstrate that unseen classes in unlabeled data do not necessarily impair the performance of SSI. models; in fact, under certain conditions, unseen classes may even enhance them.
INTRODUCTION
Semi-supervised learning (SSL) has achieved tremendous success across various fields. For instance, in the realm of natural language processing, SSL has been instrumental in improving language models. An example the development of advanced text classifiers that can understand context and sentiment with minimal labeled data. Another field where it has made significant impact is image recognition. Here, SSL techniques have enabled the creation of more accurate and efficient image classification models, which can recognize and categorize images with limited human-annotater examples. These successes demonstrate the effectiveness of SSL. handling complex tasks in different domains.
SSL effectively leverages unlabeled data and has been proven successful across various fields. Current safe SSL methods generally believe that unseen classes unlabeled data will damage the performance of SSL models. Therefore, their common practice is to identify and either remove these unseen classes from the unlabeled data or assign them lower weights. Guo et al. Guo et al. (2020) assigned lower weights to possible unseen-class unlabeled data by bi-level optimization. Opth Saito et al. (202I) combines FixMatch SShnn et al. (2020) and a one-vs-all classifier, enhancing the model's ' anomaly detection capabilities by introducing open soft consistency regularization loss. Fix A_ A_Stepp Huang et al. (2023a) adjusts the relationship between labeled and unlabeled data from gradient perspective by integrating MixUp Zhang al. (2017) technology with an update direction correction strategy. However, previous safe SSL methods for assessing the impact of unseen
Work done as an intern at Westlake University
Corresponding authors

---

Published as a conference paper at ICLR 2025
DL
Ace
DL
Acc
Lo
Di
DL
D
D
(1) old
(2) ours
The structural causal model of unseen-class evaluation process. DL. Dÿ Dj, and Acc denote labeled data, seen-class unlabeled data, unseen-class unlabeled data, and SSL model's per- formance of seen-class classification, respectively. The dashed line represents a confounding factor, which is the fundamental reason for the failure of previous evaluations.
Figure 1:
classes on SSL model performance are flawed. Especially, previous safe SSL methods fix the size of the unlabeled dataset and adjust the proportion of unseen classes within the unlabeled data to assess the impact. This process contravenes the principle of controlling variables. Inspired by Peters et al. (2017), we utilize structural causal model to model the process in Fig. [ As shown in Fig. 2(a) adjusting the proportion of unseen classes in unlabeled data alters the proportion of seen classes, meaning decreased classification performance of seen classes may not be due to an increase in unseen class samples the unlabeled data, but rather decrease seen class samples. Thus, the prior flawed assessment standard that "unseen classes in unlabeled data can damage SSL model performance" may not always hold true.
To more reliably assess how unseen-class unlabeled data affects SSL model performance, removing the spurious correlation between seen-class unlabeled data D and unseen-class unlabeled data Dÿ eritical. Based on this, we propose the RE-SSL evaluation framework. Fig. 2(b) illustrates the dataset construction for our RE-SSL evaluation framework. Ts denotes the ratio of selected [ seen-class data in D to all seen-class data in D that is r. = [D where D is the size of the
sampled seen-class data, and D is the size of the total set of seen-class data. Ta denotes the ratio D where DY
of selected unseen-class data in Dy to all unseen-class data in D4 that is r. DgT
the size of the sampled unseen-class data, and D is the size of the total set of unscen-class data. In assessment process, the size of the unlabeled dataset is no longer fixed; rather, the quantity of seen-class unlabeled data DG remains constant (r, is fixed) while only the quantity of unseen-class unlabeled data D is varied (rw, variable).
P
Q
D
(5x SUAO
DI 50001
D 15 1000
D (3 5000)
Dr
Dr.
Tv
0.8
0.2
Fu 0.2
D/
5 20
D
D
5 1000
4000
x 1000
1000
(a) Previous evaluation
(b) RE-SSL evaluation
Figure 2: An example of dataset construction the previous evaluation and our RE-SSL evaluation framework. т. denotes the ratio of selected seen-class data to all seen-class data. тu denotes the ratio of selected unseen-clate data to all unseen-class data. P denotes the distribution of seen classes, and Q denotes the distribution of unseen classes. D and D denote the initial set of seen classes and unseen classes, respectively. Dr denotes the labeled set, Dv denotes the unlabeled set. DE and DY together form Dr. where Då and D are sampled from D and DV according to r. and r.. respectively. The red bidirectional arrow represents the confounding factor mentioned in Figure
2

---

Published as a conference paper at ICLR 2025
In addition to studying the impact of sample-numbe factor Tu SSL models, our RE-SSL evaluation framework also explored four additional settings with four factors, including category-number factor Cn, category-index factor C₁ nearness factor, and label distribution factor Ca. Moreover, RE-SSL introduces five evaluation metrics, including the Slope of Regression function (Rsoope). Global Magnitude (GM), Worst-case Adjacent Discrepancy (WAD), Best-case Adjacent Discrepaney (BAD), and Probability that AD 2 0 (PAD20), and defined global and local robustness to assess the robustness of these algorithms against unseen classes. RE-SSL has conducted extensive experimental demonstrations and provided detailed analysis and insights.
Our Contributions. Our contributions contain five aspects: ① We first point out that the previous evaluations of the impact of unseen classes on SSL models are flawed, and construct structural causal model analyze the fundamental reasons leading to the flawed evaluations. 2 We propose new evaluation framework RE-SSL, which includes setting factors, constructing data, selecting comparative methods, formulating robustness evaluation metrics, and analyzing experimental results. ! For the first time, we study the impact of unseen classes on SSL models from five perspectives, including unseen classes' sample-number, category-number, category-index, nearness, label distri- bution. 4 To effectively and intuitively analyze the impact of unseen classes on SSL models under different factors, we start from global and local robustness, proposing five metrics. S We conduct fair and reasonable experiments and provide detailed analysis based the experimental results, offering strong guidance for SSL. learning in scenarios where unseen classes appear.
2 FORMALIZATION OF SSL WITH UNSEEN CLASSES
Semi-supervsied learning hamnesses the large number of unlabeled data alleviate reliance on limited labeled data. In a static environment, labeled data and unlabeled data are sampled from the identical distribution P(x, y), X, Y' on consistent data space X' y C Rd x (0, 1.. K- 1}. where and denote the dimension of the input space and the number of classes. During training stage, we are given labeled set Dr. {( y: ) (3r,, yi) ~ P(æ, y) 1 and unlabeled set Du {<x:) a ~ P(x)) } where ny and n. denote the number of samples in Dr. and De. P(æ) the marginal distribution of P(x, y). Then, the goal of SSL to learn a model with the smallest generalization error on P(æ, y).
In a dynamic environment, labeled data and unlabeled data may not originate from the identical distribution. For example, unlabeled data may contain unseen classes which are not in labeled data. We assume that labeled is sampled from the distribution P(x, y). and unlabeled set contains seen-class unlabeled set Dÿ and unseen-class unlabeled set Dj where Dÿ are sampled from the distribution (()), but Dÿ samples are from an additional distribution Q(x), the label spaces of P(x:) and (xx) have no intersection.
3 RE-SSL EVALUATION FRAMEWORK
Previous most safe SSL works, such as DS3L Guo et al. (2020) and Safe-Student He et al.
2022).
come the conclusion: unseen-class unlabeled data impair the performance of SSL model. However, such a conclusion is based on an unfair evaluation process. Especially, as shown in Fig. 2(a) they fix the size of the unlabeled dataset Dy and adjust the proportion of unseen classes within the unlabeled data to assess the impact. Adjusting the proportion Tu of unseen classes in unlabeled data alters the proportion rs of seen classes, which results in the confounding factor between DG and Dj- The decreased classification performance of seen classes may not be due to an increase in unseen class samples in the unlabeled data, but rather a decrease seen class samples. Thus, the impact of unseen-class unlabeled data on SSL models remains to be further verified. To evaluate the impact of unseen-class unlabeled data in a fair way, we propose the RE-SSL evaluation framework.
3.1 DATASET CONSTRUCTION
RE-SSL reconstructs the dataset used for evaluation, as shown in Fig 2(b) We only fix r. and no longer fix the size of Du. By changing тu (abbreviated r), verify the impact of different numbers of unseen classes on the performance of SSL models. Furthermore, we also explore four additional settings: (1) varying the number of unseen class categories; (2) changing the indices of
3

---

Published as a conference paper at ICLR 2025
unseen classes; (3) differing degrees of nearness in unseen classes; (4) diverse label distributions in unseen classes.
3.2 INTRODUCTION OF EVALUATION METRICS
Traditional SSL algorithms only consider the state where the unlabeled data distribution is consistent with the labeled data. Even most of the safe SSL methods (such DS3L [uoo et al. (2020) and Safe-Student He et al. (2022)) only consider one of all inconsistent states of unlabeled data and cannot directly and effectively reflect the impact of unseen classes the performance of SSL models. Furthermore, previous safe SSL. methods' analysis of unseen classes contradicts the standard controlled variable criterion. In fact, study the impact of unseen classes SSL models, it is necessary to strictly adhere to the controlled variable criterion. Currently, there is a lack of tools and evaluation metrics directly and straightforwardly analyze the impact of unseen classes on the performance of SSL models from a dynamic, global perspective. Although some previous works in safe SSL (like DS3L and safe-Student) have drawn curves describing the change in accuracy with the inconsistency of label space, they are based on an incorrect controlled variable criterion and have studied a limited range of inconsistency changes (such as 0-0.6). addition, they lack global metric to analyze the curves of different methods and cannot provide a direct comparison and analysis. To address the issues mentioned above, we propose Ace Regression, which considers each state and its corresponding Acc as points and performs linear regression to obtain the Regression function. This Regression function can globally measure the impact of unseen classes on different SSL methods.
To more comprchensively evaluate the impact of unseen classes SSL models, we have defined multiple evaluation metrics to assess the robustness of these algorithms against changes in the label space distribution within unlabeled data. Unlike traditional SSL evaluations that solely focus assessing Acc(O) and secure SSL evaluations that only assess Acc(r) for a specific r, our evaluation framework, meticulously based on the Regression function, allows for more comprehensive and direct assessment of the impact of unlabeled data on SSL models.
Slope of Regression function (Rsiope): The slope of Regression function reflects the over- all trend of changes in SSL models as unseen classes are continuously added. We define Re- gression function as Ace(r) = Ralope r + where is just a constant. Then, based on (r. Acc(r)), r = 0, 0.2, 0.4, 0.5, 0.6, 0.8, 1.0, where AĆC represents the empirical accuracy corre- sponding use linear regression obtain the regression function:
Rtope (Ace) = arg min [ ('(Acc(r)- - Acc(r))² ) dr
(I)
Ratope
Global Magnitude (GM): Global magnitude reflects the influence of unseen classes on SSL models from global perspective, which measures the integral of the magnitude of Acc for each to the expected accuracy Acc, where ACC represents the mean of accuracy values of the model across all values of Unlike Rolope. this metric does not require imposing increasing decreasing constraints on the r:
GM(Acc) = =/o" [Ace(r) Acc(r)) dr
Worst-case Adjacent Discrepancy (WAD):The maximum of adjacent discrepancy reflects the influence of unseen classes on SSL models from a local perspective, where adjacent discrepancy represents the local slope, calculating the change in Ace between two adjacent r. WAD displays the worst-case impact of unseen classes have on SSL models:
WAD(Acc) min { (cc(F++1) Acc(rr) n-1
ri+1 Tú
Best-case Adjacent Discrepancy (BAD): The minimum of adjacent discrepancy reflects the influence of unseen classes SSL models from a local perspective. BAD displays the best-case impact of unseen classes have on SSL models:
Acc((s+1) - Acc(ri) n-1
BAD(Ace) = max
(4)
Ti+1
4

---

Published as a conference paper at ICLR 2025
Table I: Evaluation on CIFAR1O with 100 labels under inconsistent label spaces.
METHOD
r=0
r=0.2
>=0.4
r=0.5
r=0.6
r=0.8
r=1.0
Raieze
GM
BAD
WAD
PaPzo
SUPERVISED
0.617
0.617
0.617
0.617
0.617
0.617
0.617
0.000
0.000
0.000
0.000
1.000
PSEUDOLABEL
0.333
PIMODEL
0.667
0.651
0.644
0.637
0.636
0.636
0.630
-0.034
0.066
0.000
-0.080
0.167
FIXMATCH
0.612
0.547
0.515
0,514
0.505
0.442
0.365
-0.223
0.386
-0.010
-0.585
0.000
FLEXMATCH
0,770
0.717
0.702
0.704
0.704
0.686
0.636
-0.107
0.166
0.020
-0.265
0.333
UDA
0.645
0.621
0.604
0.601
0.591
0.590
0.552
-0.082
0.137
-0.005
-0.190
0.000
SOFTMATCH
0.761
0.710
0.689
0.697
0.692
0.682
0.607
-0.112
0.193
0.050
-0.375
0.167
VAT
0.710
0.677
0,663
0.663
0.657
0.651
0.639
-0.063
0.111
0,000
-0.165
0.167
FREEMATCH
ICT
0.618
0.619
0.621
0.620
0.621
0.619
0.621
0.002
0.007
0.010
-0.010
0.667
UASD
0.618
0.619
0.617
0.616
0.617
0.618
0.618
0.000
0.005
0.010
-0,010
0.667
CAFA
0.640
0.653
8.641
0.642
0.640
-0.013
0.040
0.130
-0. 40.120
D.500
OPENMATECE
0.713
0.606
0.586
0.595
0.579
0.517
0.473
-0.211
0.350
0. 0.090
-0.535
0.167
STXPA_STEP
0.662
0.634
0.615
0.582
0.555
0.585
0.509
-0.138
0.272
0.150
-0.580
0.167
Probability that AD 2 0 (PAD>0): The probability that AD 2 0 reflects the influence of unseen classes on SSL models from a global perspective. PAD'O measures the probability that the addition of unseen classes does not decrease the performance of SSL models:
n-1 PAD2o(Acc) = Σ D(Ac((r(4)) - Acc(r:) 0)
(5)
Overall, we propose several targeted metries from both global and local perspectives assess the impact of unseen-class unlabeled data on SSL models. From global perspective, we propose R Ratopes GM, and PADZO- From local perspective, we introduce WAD and BAD. The integrals in Eq. and Eq. 2 are computed as accumulation operations based on empirical values.
4 DEFINITION OF ROBUSTNESS TO UNSEEN CLASSES
Due to the existing definitions of robustness in previous work, avoid confusion in the definition, we define global robustness (including Rsiope robustness) and local robustness (WAD robustness) based on Ratope and WAD metrics.
In our safe SSL setting, if SSL model does not exhibit a significant decline as unseen-class samples continuously added, consider the model to possess global robustness (Rstope robustness) towards unseen classes. If, at any point during the ongoing addition of unseen-class samples, the performance of the SSL model does not decrease or even improves, we consider the model have local robustness (WAD or BAD robustness) towards unseen classes.
Definition 1 (Global robustness.) SSL algorithm A returns model fr E F using Dr. and Du with unseen-class ratio r where F is the hypothesis space of A. Let Acc(r) denote the accuracy of fr on test set with only seen classes. If there exists 8g such that Reiope 2 8g. we suy that SSL algorithm A exhibits 8g slope algorithmic robustness.
Definition (Local robustness.) If there exists 6ep such that WAD S 6w. we say that SSL algorithm
A exhibits dn wwosstct-a algorithmiic robustness. If there exists 6, such that BAD 85, we say that SSL algorithm A exhibits ò, by-best-case algorithmic robustness.
5
EXPERIMENTS
5.1 EVALUATED ALGORITHMS
Based on diversity and performance, we comprehensively consider 10 classical algorithms: Pseu- doLabel Lee et al. (013)), PiModel Laine & Aila (2017), ICT Verma et al. (2022), VAT Miyato jet al. 20018), MixMatch Berthelot et al. (2019), UDA Xie et al. (2020). FixMatch Sohn et al. (2020),
Flex Match
Zhang Zhang et al. 2021), FreeMatch Zhang al. 2021) and Mothhatch Chen al. (2023). We
5

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation on CIFAR 100 with 1000 labels under inconsistent label spaces.
METHOD
r=0
r=0.2
y=0.4
r=0.5
r=0.6
r=0.8
r=1.0
Raiese
GM
BAD
WAD
Papao
SUPERVISED
0.422
0.422
0.422
0.422
0.422
0.422
0.422
0.000
0.000
0.000
0.000
1,000
PSEUDOLABEL
0.413
0.411
0.060
0.167
PIMODEL
0.421
0.422
0.421
0.423
0.421
0.412
0.120
-0.007
0.018
0.040
-0.045
0.333
FIXMATCH
0.441
0.332
0.223
0,260
0.219
0.205
0.176
-0.244
0.485
0.370
-0.545
0.167
FLEXMAICH
0.143
0.112
0,390
0.377
0.382
0.344
0.317
-0.120
0.208
0.050
-0.190
0.167
0.388
0.029
0.060
SOFTMATCH
0.455
0.449
0.440
0.432
0.434
0.432
0.408
-0.042
0.074
0.020
-0.120
0.167
VAT
0.146
0.435
0.426
0.422
0.424
0.417
0.403
-0.039
0.065
0.020
-0.070
0.167
FREEMATCH
0.455
0.435
0.416
0.410
0.408
0.401
0.354
-0.088
0,144
-0.020
-0.235
0.000
0.414
0.412
0.413
0.414
0.413
0.411
-0.002
0.010
0.500
UASD
0.415
0.411
0.411
0.411
0.415
0.412
0.411
-0.002
0.011
0 0441
-0.000
D.500
MTCP
0.248
0.226
0.220
0.227
0.221
0.227
0.222
-0.018
0.041
0.070
-0.110
D.333
CAFA
0.418
MPENN
0.367
0.329
0.325
0.309
0.
30707
0.292
-0.063
0.115
0.150
190
D.167
PIX_A_STEP
0.448
0.393
0.359
0.342
0. 326
0.311
0.241
-0.188
0.326
-0.075
40.350
0.000
also select 5 robust algorithms: UASD Chen et al. (2020) CAFA Huang et al.
2021). MTCF Yu et al.
(2020), OpenMatch Saito et al. (2021) and Fix A Step Huang et al. (023a).
5.2 EXPERIMANTAL SETTING
For all the experiments, we use supervised learning algorithm as baselines, which trains model based on DL with empirical risk minimization. The evaluation of the algorithm was performed based on the LAMDA-SSL toolkit Jia et al. (2022). We use ResNet50 He et al. (2016)as the backbone. All experiments of deep SSL algorithms are conducted with NVIDIA RTX A6000 GPUs. To ensure reliability, we conduct three experiments for each sampling point with seed '0, 1, 2' to obtain average Acc. Different seeds are used to sample from the source dataset to create different training sets. Code is available at httptts://///m /tt:://gittttt cww. ccom//undonggg//E.....
5.3
MAIN RESULTS AND ANALYSIS
Table 1 and presents the evaluation of various SSL methods on CIFARIO and CIFARI00, with assessments conducted using 100 labels within inconsistent label spaces and 1000 labels within inconsistent label spaces, respectively. The r value indicates the degree of inconsistency within the label space, with r = signifying complete consistency (i.e., no unseen classes) and r representing the highest degree of unseen class introduction. Performance metrics include accuracy (with values listed from r = 0 to r = 1), as well as Ralope, GM, WAD and BAD, and PAD20- Based we can obtain the following conclusions:
on Table I
From the perspective of global robustness: as unseen-class data continuously integrated, the Ace of the vast majority of SSL methods and robust SSL methods shows declining trend. In CIFAR10. assume that og equals -0.020, according to Definition several SSL methods PseudoLabel and ICT, as well robust SSL methods UASD) and CAFA, which are relatively robust unseen classes, the addition of unseen classes essentially has no impact on their algorithmic performance. On the contrary, some SSL methods such FixMatch and FreeMatch, OpenMatch, are very sensitive to unseen classes, with their Ratope values being -0.223, -0.287, and -0.211 respectively. The introduction of unseen classes can severely deteriorate their performance. In CIFAR100, assume that og also equals -0.020, according to Definition I several SSL methods PseudoLabel. PiModel, UDA and ICT. as well robust SSL methods UASD, MTCF, and CAFA, which are relatively robust to unseen classes. Then, examining both CIFAR 10 and CIFAR 100, we find that the SSL. algorithms ExeudoLaabe and ICT, as well as the robust SSL algorithms UASD and CAFA, are robust to unseen classes in both CIFAR 10 and CIFAR100. This also reflects the stability of these algorithms' robustness to unseen classes, unaffected by the choice of dataset.
well as robust SSL methods like
From the perspective of local robustness: In the context of local robustness, analysis of SSL algorithms on CIFAR 10 and CIFAR100 datasets centers on the performance metrics of Worst- case Adjaceet Discrepancy (WAD) and Best-case Adjacent Discrepancy (BAD). For the CIFAR10 dataset, the analysis of WAD reveals the extent performance degradation under locally worst-case scenarios. For instance, Mitch exhibits significant drop (WAD at -0.385), indicating weaker

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation on CIFAR1O with 100 labels under the different number of unseen classes.
METHOD
base
Ca=1
C,m2
C,=3
C,m4
C,m5
Raiaps
GM
BAD
WAD
PADSO
SUPERVISED
0.617
0.617
0.617
0.617
0.617
0.617
0.000
0,000
0.000
0.000
1.000
PSEUDOLABEL
0.677
0.648
0.652
0.663
0.664
0.668
0.005
0,036
0.011
0.001
1.000
PIMODEL
0.667
0.622
0.628
0.640
0.642
0.651
0.007
0.046
0.012
0.002
1.000
FIXMATCH
0.612
0.710
0.674
0.574
0.441
0.547
-0.056
0,411
0.106
-0.133
0.250
FLEXMATCH
0.770
0.658
0.644
0.689
0.699
0.715
0,017
0.120
0.045
-0.014
0.750
UDA
0.645
0.620
0.599
0.594
0.621
0.621
0.002
0.058
0.027
-0.021
0.500
SOFTMATCH
0.671
0.662
0.715
0.710
0.013
0.119
0.062
-0.009
0.250
VAT
0.710
0.606
0.627
0.671
0.676
0.677
0.019
0.139
0.044
0.001
1.000
FREEMATCH
0.760
0.668
0.639
0.724
0.688
0.723
0.016
0.140
0.085
-0.036
0.500
ICT
0.618
0.611
0.613
0.615
0.619
0.619
0.002
0.014
0.004
0.000
1.000
UASD
0.618
0.618
0.620
0.617
0.619
0.619
0.000
0.004
0.002
-0.003
0.750
MTCP
0.772
0.723
0.704
0.737
0.736
0.743
0.007
0.060
0.033
-0.019
0.500
CAFA
0.652
0.642
0.645
0.642
0.643
0,648
0.001
0.010
0.005
-0.003
0.750
OPENMATCH
0.713
0.642
0.634
0.561
0.606
-0.015
0.127
0.045
-0.072
0.250
STEP
0.662
0.674
0.647
0.631
0.625
0.632
-0.011
0.075
0.007
-0.027
0.250
Table 4: Evaluation on CIFARIO with 100 labels under the different index of unseen classes
МETHOD
near
for
bose
Clm5
C2m6
Cim?
C.=8
C2m9
GM
Cim5
CLm6
Cim?
Ci=8
Clm9
GM
SUPERVISED
0.617
0.617
0.617
0,617
0.617
0.617
0.000
0.617
0.617
0,617
0.617
0.617
0.000
PSEUDOLABEL
0.677
0.648
0.649
0.650
0.650
0.648
0,004
0.614
0.618
0.626
0.621
0.623
0.018
PIMODEL
0.667
0,638
0.642
0.639
0.637
0.622
0.027
0.621
0.624
0.625
0.621
0.624
0.008
FIXMATCH
0,612
0.635
0.504
0,533
0.527
0.710
0.363
0.704
0.694
0,689
0.695
0.702
0.025
FLEXMATCH
0.770
0,656
0.640
0,682
0,662
0.658
0.050
0.610
0.586
0.619
0.606
0.616
0.046
UDA
0.645
0.627
0.646
0.627
0.608
0.620
0.046
0.644
0.640
0.640
0.643
0.633
0.014
SOFTMATCH
0.764
0.654
0,725
0.687
0.671
0.646
0.644
VAT
0.710
0.654
0.632
0.657
0.623
0.606
0.084
0.564
0.565
0.577
0.568
0.581
0.032
FREEMATCH
0.760
0.655
0.610
0.709
0.628
0.668
0.140
0.642
0.633
0.641
0.639
0.637
0.014
ICT
0.618
0.623
0.619
0.622
0.608
0.611
0.028
0.597
0.596
0.598
0.597
0.597
0.002
UASD
0.618
0.617
0.616
0.618
0.619
0.618
0.004
0.617
0.618
0.618
0.618
0.617
0.002
MTCF
0.772
0.703
0.685
0.697
0.689
0.723
0.054
0.632
0.573
0.619
0.596
0.596
0.089
CAFA
0.652
0.639
0.639
0.638
0.634
0.630
0.016
0.606
0.597
0,592
0.612
0.613
0.038
OPENMATCH
0.713
0.636
0.546
0.648
0.566
0.642
0.206
0.607
0.559
0.569
0.629
0.622
0.133
STEP
0.662
0.630
0.630
0.638
0.634
0.630
0.014
0.693
0.684
0.693
0.691
0.693
0.014
local robustness against unseen classes. In contrast, the ICT algorithm's WAD value is nearly zero (0.010), demonstrating stronger local worst-case robustness. Moreover, BAD values suggest that FlexMatch (0. 166) and VAT (0.111) possess higher local best-case robustness, implying that these algorithms can capitalize on favorable conditions to maximize performance enhancements. Moving to the CIFAR 100 dataset, (-0.545), which shows high sensitivity to unseen classes and insufficient local worst-case robustness. Concurrently, Flex shows significant advantage in local best-case robustness with a BAD value of 0.208, while Tix_A_STEPP BAD value (-0.350) indicates a lower capacity for performance improvement under optimal conditions. In summary, regarding local robustness, FlexMatch and ICT display relatively better performance stability, both under worst-case scenarios and best-case scenarios. On the contrary, Fix.a shows deficiencies in both aspects. These insights are crucial for the design and deployment of SSL algorithms that are robust in dynamic environments.
observe universally lower WAD values, particularly for FixMatch
5.4 EXTENDED EXPERIMENTS
Before this, our research had only examined the impact of a single element (the number of unseen class samples) on SSL models. However, in more complex scenarios, unseen class samples may have varying numbers of unseen class categories, unseen class samples may have different unseen class category indices, unseen class samples may have varying degrees of nearness, and unseen class samples may have diffeerntt labeling distributions. Studying these elements is also of significant instructive value for SSL models.
Category-number factor C,,: C, refers to the factor that controls the number of unscen-class categories from which the unseen class samples originate. We validated different approaches on
7

---

Published as a conference paper at ICLR 2025
Table 5: Evaluation on CIFARIO with 100 labels under different imbalance factors of unseen classes.
METnOO
bese
Cg=0.01
Ca-0.02
Ca-0.05
Cu
Ca-D.20
Ratops
GM
BAD
WAD
PAD22
SUPERVISED
0.617
0.617
0.617
0.617
0.617
0.000
0.000
0.000
PAEUDOLABEL
0.677
0.660
0.660
0.663
0.658
0.662
0.005
0.008
0.100
-0.100
0.750
PIMODEL
0.667
0 6464
0.649
0 0.642
0.646
0.642
-0 022
0.013
0.200
-0.233
0.500
FIXMarcn
0.612
0.529
0.534
0.541
0.603
0.560
0.208
0.112
1.240
-0.430
0.750
FLEXMATCH
0.706
0.717
0.022
-0.700
UDA
0.645
0.620
0.647
0.634
0.615
0.619
-0. -0.088
0.054
2.700
-0.433
0.500
SOPIMATCH
0.764
0.730
0.731
0.725
0.221
0.698
-0.170
0.046
0.100
-0.230
0.250
VAT
0.710
0.681
0.682
0.682
0.678
0.675
-0.037
0.012
0.100
-0.080
0.500
RTEEMATCH
0.760
0.730
0.723
0.674
0.670
0.677
256
0.127
0:070
-1.633
0,250
ICT
0.618
0.622
0.621
0.621
0.622
0.622
0.003
0.002
0.020
-0.100
0.750
UASD
0.618
0.618
0.618
0.617
0.618
0.617
-0.004
0,002
0.020
-9.033
0,500
MTCF
0.722
0.721
0.722
0.004
0.067
-0.200
CAFA
0.652
0.6417
0.650
0.613
0.648
0.616
-0.007
0.009
0.300
-0.233
0.500
OPENMATCH
0.713
0.613
0.613
0.620
0.650
0.639
0.163
0.070
0.600
-0.130
0.750
FIX.A.STEP
0.622
0.685
0.638
0.632
0.633
0.628
-0.179
0.084
0.020
-4,700
0,250
CIFARIO, with the results shown in Table Here, base refers to the scenario where r=0, meaning no unseen class data is added. In other cases, r is set to 0.2. On CIFARIO, C, is set to 1, 2, 3, 4, 5.
On the CIFARIO dataset, most methods show high accuracy under the base scenario, and the accuracy generally rises as the number of unseen classes Cn increases. The Ratope column indicates that the impact the SSL model is continuously mitigated for most methods with the increase in the number of unseen class categories, except for FixMatch, OpenMatch, and Fix_A.St A.Step. Specifically, FixMatch shows a sharp decline in accuracy C, 4, suggesting that it may need additional improvements when dealing with higher category diversity.
Category-index factor Cj: C2 refers to the factor that controls the index of unseen-class category from which the unseen class samples originate. We validated different approaches on CIFAR 10 and the results can be seen in Table 4 with near part. Unlike the previous and C., the different values of C do not have a progressive relationship, so we can only use GM to measure the impact of O2 on the SSL model. Because we select the first five classes as seen classes in CIFARIO, the range of C2 is 5, 6, 6, 8, 9. base still refers to the scenario where r=0, meaning no unseen class data is added.
Almost all methods perform better under the base scenario than when unseen classes with different indices are added. As C; varies from 5 to 9, the accuracy trends are inconsistent, reflecting the varied impact of unseen class indices on different methods. For instance, Fixch shows a significantly low accuracy at C. = 6 (0.504), but its performance recovers and even exceeds the base level when C; is at or other values. This indicates that unseen class data with different indices can have a large and variable impact SSL models, potentially improving or degrading their performance. PseudoLabel and UASD have very low GM values, suggesting they maintain better robustness across different unseen class indices. On the other hand, FixMatch and OpenMatch have high GM values, indicating that these two methods are quite sensitive to C,,
Nearness factor: Prior to this, the unseen classes in our experiments originated from near out-of- distribution (OOD). For instance, in our experiments on CIFARIO, 10, we default to treating the first five classes as seen and the latter five as unseen. However, since they all come from the same dataset, there is a considerable degree of relatedness in terms of data categories and distribution. But what if the nearness of the unseen classes is different, such when the unseen classes come from far OOD? How would these unseen classes affect the SSL models? To investigate the impact of nearness on SSL models, we have also conducted experiments where the unseen classes come from MNIST, based on the previous CIFAR10 10 experiments, to make horizontal and vertical comparisons. The experimental results can be seen in Table 4 with far part.
When comparing the "near" and "far" results, it's apparent that almost all SSL models have notably lower GM in the far scenario than in the near one, indicating that the SSL models are more robust to the Category-index factor C; in the far scenario. Moreover, we find that except for FixMatch and Fix_A.Step. the performance of the majority of SSL models under the same settings is better in the near scenario than in the far one. This suggests that the smaller the semantic shift and distribution shift from the seen classes to the unseen classes, the less damage to the SSL models. This conclusion is clearly valid. For example, when mixup is applied directly to the seen classes as unseen classes,
8

---

Published as a conference paper at ICLR 2025
GM metrics on CIFAR10
0.5
0.4
0.3
0.1
0.0
PSRUDOLASEE
PRiOpEL
FIAMATCH
PLEXMATCH
UDA
SOFMMATCH
VAT
PROEMATCH
KT
UASO
HYZF
CAPA
OPEMMATEH
PM,A A.SEP STEP
Method
Figure 3: GM metries on CIFARIO with 100 labels under different factors.
this time these unseen classes have the smallest semantic shift and distribution shift from the classes. The classes obtained after mixup are clearly helpful for the classification of seen classes.
Label distributions factor Cis= C. refers the factor that controls the labeling distribution of the unseen class data. Specifically, we achieve varying degrees of class imbalance by adjusting the sampling strategy of the samples. We the imbalance factor (Cis) 0.01, 0.02, 0.05, 0.10, and 0.20. The smaller the Ca., higher degree of class imbalance.
Through Table
observe that different SSL methods react differently changes in Cib- Most methods are quite robust to C.b, with R slsoope and GM values close to O. However, FixMatch and OpenMatch show a generally increasing trend Ct. increases. Similarly, FreeMatch and Fix_A_Step also tend to exhibit an increasing trend with the increase in Cibb-
ANALYSIS OF EXPERIMENTAL RESULTS
6.1 ANALYSIS OF FACTORS
Table and Fig. provides the GM ods the CIFARTO dataset with 100
values
for various SSL meth- under
labels
different
factors.
These factors include the number of unseen class samples (r), the number of unseen class categories (C,),, the index of unseen class categories (C;), and the label distribution of unseen classes (Ct)). Faeg represents the mean of each row, while Aang repre- sents the mean of each column. As for the degree of nearness of unseen classes, since we only verified two scenarios of nearness for unseen classes, namely near unable obtain GM
Table 6: GM under different factors.
Method
C.
C;
Ca
Fosy
PscudoLabel
0.038
0.036
0.004
0.008
0.021
PiModel
0.066
0.046
0.027
0.013
0.038
FixMarch
0.386
0.411
0.363
0.112
0.318
FlexMatch
0.166
0.120
0.050
0.022
0.089
UDA
0.137
0.058
0.046
0.054
0.073
SofiMatch
0.193
0.119
0.102
0.046
0.1
OOD and far OOD, metrics for the degree of nearness unseen classes. In this table, the GM metric is used to evaluate the robustness of various SSL methods under the influ- ence of different factors. The lower the GM metrics, the less sensitive the model is to that specific factor, indicating greater robustness. According to Table the average GM value for r is the highest at 0.170. indicating that SSL. models are generally sensitive to changes in the number of unseen classes in the unlabeled sample set. Methods like FreeMatch. with GM value 0.496, are particularly sensitive to this factor. The average GM value for C, is 0.097, which, although lower than the impact of r. still indicates a certain level of sensitivity. This suggests that altering the number of unseen classes can also affect the robustness of the model. The average GM value for C is 0.081, showing that models also have certain sensitivity to changes in the index of unseen classes, but this sensitivity is less than the impact of changes in the number of unseen class samples and the number of unseen class categories. The average GM value for Ci. is
VAT
0.111
0.139
0.084
0.012
0.086
FreeMatch
0.496
0.140
0.140
0.127
0.225
ICT
0.007
0.014
0.028
0.002
0.012
UASD
0.005
0.004
0.004
0.002
0.003
MTCF
0.119
0.060
0.054
0.004
0.059
CAFA
0.040
0.010
0.016
0.009
0.018
OpenMatch
0.350
0.127
0.206
0.070
0.188
A
0.272
0.075
0.014
0.084
0.111
Auee
0.170
0.097
0.081
0.040

---

Published as a conference paper at ICLR 2025
the lowest 0.040, meaning that the models are relatively robust to changes in the label distribution of unseen classes. This indicates that changes in the label distribution of unseen classes have a less significant impact model performance compared to other factors. Overall, these results reveal that SSL. learning models are most sensitive to the quantity of unseen class samples when dealing with unseen categories, while being relatively robust to changes in the label distribution of unseen classes. This insight helps us understand how models perform under different types of data uncertainty.
6.2 ANALYSIS OF ALGORITHMS
Based on Table 6 we can analyze the overall differences between various SSL methods. The smaller the Fang. the more robust the algorithm is to unseen classes. Pseudol abel and PiModel have very small Faug. indicating that they are quite robust to unseen classes. PseudoLabel relies solely on the inconsistency between soft and hard labels, while PiModel utilizes the inconsistency caused by weak data augmentation. The unsupervised loss derived from their inconsistencies is minimal, hence the algorithms exhibit strong robustness. ICT's Favg is very small, even smaller than that of PseudoLabel and PiModel. This is because ICT employs Mixup operations between labeled and unlabeled data, which can effectively mitigate the impact of unseen classes in the unlabeled data on the classification of seen classes, thus ensuring the robustness of ICT to unseen classes. In classic SSL algorithms, FixMatch has the largest Forg of 0.318. This demonstrates that FixMatch the most sensitive to unseen classes and is not robust enough. We analyze this as being due to FixMatch using a fixed threshold, which not suitable for complex scenarios. Methods that are improvements based on FixMatch, such as FlexMatch, FreeMatch, and SoftMatch, have abandoned the use of fixed thresholds and instead employ adaptive thresholds or appropriate weighting strategies. Therefore, they have achieved certain degree of enhancement in robustness.
As for safe SSL. models, UASD exhibits the strongest robustness. We analyze that this is because it leverages the inconsistency between predictions from the current and previous rounds update the model, thus exhibiting greater robustness to unseen classes. CAFA, which is designed based on PiModel and Mixup both inherently more robust to unseen classes alss demonstrates enhanced robustness to unseen classes. MTCF employs curriculum learning and multi-task learning to filter out unseen classes, therefore also showing more robust nature to unseen classes. OpenMatch, while relatively less robust, is based on FixMatch design, but is more robust than FixMatch because it introduces one-vs-all (OVA) classifiers, which can help filter unseen classes. Similarly, Fix-A-Step also shows relatively less robustness as it is based on the FixMatch design. However, it is more robust than FixMatch because it selectively discards the unlabeled loss by calculating the cosine similarity of the gradients of the labeled and unlabeled losses, which is very interesting and promising approach.
7 LIMITATION
This manuscript primarily explores the impact of unseen classes within unlabeled data on semi- supervised models. However, in practical applications, the appearance of unseen classes in unlabeled data is not the only change; seen classes in unlabeled data may also experience shifts in feature and label distributions. Investigating the influence of unseen classes on comi-supervised models in more complex scenarios is also crucial. Additionally, another weakness of our manuscript is the lack of corresponding theoretical analysis.
CONCLUSION
This article illuminates the deficiencies in prior assessments concerning the influence of unseen classes on SSL models, and pionecrs a structural causal model to dissect the underlying causes of these evaluation inaceuracies. We introduce the RE-SSL framework, a holistic approach encompassing factor specification, dataset structuring, comparative methodology, development of robustness metrics, and comprehensive experimental analysis. This study marks the inaugural investigation into the effects of unseen classes on SSL models across five critical dimensions. By leveraging both global and local robustness metrics, we offer a nuanced and intuitive examination of the unseen classes' impact under varied conditions. Our rigorous and methodologically sound experiments, coupled with an de-tepth analysis, pave the way for significant advancements in SSL, particularly in environments featuring unseen classes, thereby setting new standards for model robustness.
10

---

Published as a conference paper at ICLR 2025
9 ACKNOWLEDGMENT
This research was supported by Westlake University Research Center for Industries of the Future, Westlake University Center for High-performance Computing, the National Natural Science Founda- tion of China (Grant No. U23A20389, 62176139, 62306133). The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities. We would like to thank the reviewers for their constructive suggestions.
REFERENCES
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurlPS, 2019.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bertt Schiele, Xing Xie, Bhiksha Raj. and Marios Savvides. Softmatch: Addressing the quantity-uuality trade-off in semi-supervised learning. arXiv preprini 2cXiv:2301.1092) 2023.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In AAAI, 2020.
Lan-Zhe Guo, Zhenyu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hur Zhou. Safe deep semi-supervised learning for inseen-clase unlabeled data. In ICML, 2020.
Lan-Zhe Guo, Lin-Han Jia, Jie-Jing Shao, and Yu-Feng Li. Robust semi-supervised learning in open environments. Frontiers of Computer Science, 1988),,93445, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Rundong He, Zhongyi Han, Xiankai Lu, and Yilong Yin. Safe-student for safe deep semi-supervised learning with unseen-class unlabeled data. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14565-14574, 2022. doi: 10. WWWD9CCVRR2268.20222.00
Zhe Huang, Mary-Joy Sidhom, Benjamin Wessler, and Michael Hughes. Fix-a-step: Semi- supervised learning from uncurated unlabeled data. In International Conference on Artificial Intelligence and Statistics, pp. 8373-8338 8394. PMLR, 2023a.
Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. Advances in Neural Information Processing Systems, 34:26714-2677 2021.
Zhuo Huang, Li Shen, Jun Yu, Bo Han, and Tongliang Liu. Flatmatch: Bridging labeled data and unlabeled data with choss-sharpness for supervuped learning. In Advances in Neural Information Processing Systems, volume 36, pp. 18474-18494 2023b.
Lin-Han Jia, Lan-Zhe Guo, Zhi Zhou, and Yu-Feng Li. Lamda-ssl: Semi-supervised learning in python. arXiv preprint arXiv:2208, 046100 2022.
Maria Brbic Kai-Di Cao and Jure Leskovec. Open-world supe-vupe learning, In Proceedings of the 10th International Conference on Learning Representations, 2022.
Alex Kurakin, Colin Raffel, David Berthelot, Ekin Dogus Cubuk, Han Zhang, Kihyuk Sohn, and Nicholas Carlini. Remixmatch: Semi-supervised learning with distribution matching and augmen- tation anchoring. In ICLR, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Mong-Hyun Lee et al. Pseudo-label: The simple and efficient semi sumi-supervised learning method for deep neural networks. In ICML., volume pp. 896. Atlanta, 20133
11

---

Published as a conference paper at ICLR 2025
Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9475-9484, 2021.
Zekun Li, Lei Qi, Yinghuan Shi, and Yang Gao. lomatch: Simplifying open-set semi-supervised learning with joint inliers and outliers utilization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15870-15879, 2023.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 11(8):1979-199) 2018
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: and learning algorithmss The MIT Press, 2017.
fogferance foundations
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: O semi-supervised learning with open-set consistency regularization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 25956-25967. Curran Associates, Inc., 2021.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. In NeurIPS, 2016.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurlPS, 2020.
Yiyou Sun, Zhenmei Shi, and Yixuan Li. A graph-theoretic framework for understanding open-world semi-supervised learning. Advances in Neural Information Processing Systems, 36, 2024.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
Jesper E Van Engelen and Holger H Hoos. survey on semi-supervised learning. Machine learning, 109(2):373 440, 2020.
Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Arno Solin, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. Neural Networks, 145: 90-106, 2022.
Renzhen Wang, Xixi Jia, Quanziang Wang, Yichen Wu, and Deyu Meng. Imbalanced semi-supervised learning with bias adaptive classifier. The Eleventh International Conference on Learning Representations, 2022a.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, et al. Usb: A unified semi-supervised learning benchmark for classification. Advances in Neural Information Processing Systems, 35:3938-3961, 2022b.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, et al. Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint 2rXhh:2205.07244 2022c.
Yu Wang, Pengchong Qiao, Chang Liu, Guoli Song, Xiawu Zheng, and Jie Chen. Out-of-distributed semantic pruning for robust cemi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22889, 2023.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33:6256-6266 2020.
Lihe Yang. Zhen Zhao, Lei Qi, Yu Qiao, Yinghuan Shi, and Hengshuang Zhao. Shrinking class space for enhanced certainty in semi-superrised learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 16187-16196. 2023.
12

---

Published as a conference paper at ICLR 2025
Xiangli Yang. Zixing Song, Irwin King, and Zenglin Xu. A survey deep semi-superviied learning. IEEE Transactions on Knowledge and Data Engineering, 2022.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In ECCV, pp. 438-454. Springer, 2020.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:18408-1841 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. 2017.
Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi- supervised learning with similarity matching. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14471-14481, 2022.
A RELATED WORK
A.1 DEEP SEMI-SUPERVISED LEARNING
Semi-supervised learning (SSL) holds immense potential within the realm of machine learning Chapelle et al. (2009); Van Engelen & Hoos (2020); Wang et al. (2022b); Yang et al. (2023] (022); Zheng al. (2022). Compared supervised learning, escapes the shackles of being entirely reliant on labeled data, shifting its focus towards the rich sample characteristics and structural information concealed within unlabeled data Chen et al. (2023). In contrast with unsupervised learning, successfully propagates the semantic information from labeled data to unlabeled counterparts, thereby streamlining its application in downstream tasks.
Self-training and consistency regularization stand out cornerstone techniques of contemporary deep SSL Laine & Aila (2017); Li et al. (202T: 2023); Sajjadi et al. (2016); Tarvainen & Valpola (2017): Wang al. (2023). In particular, the pseudo-labeling technique within self self-training plays a pivotal role. It iteratively enlarges the "labeled data" by speculatively assigning pseudo-labels to unlabeled samples, enabling unlabeled data to generate supervisory signals for model training Lee et al. (2013); Wann et al. (2022a). This strategy has become almost inescapable in current SSL paradigms. Consistency regularization, on the other hand, operates under the presumption that training samples share identical labels with their proximate synthetic counterparts (or similar views). By disseminating the labels from seen-class samples and confident predictions of unseen- class samples, it harnesses unlabeled data even more effectively Berthelot et al. (211)); Miyato et al. (2088): (Satoo et al.) (02T)); (Sohn et al. [2020); Huang al. (2023b); Guo et al. (2025). For example, FixMatch Sohn et al. (2020) an efficient approach that capitalizes on both techniques. It refines consistency regularization through the distinct deployment of weak and strong augmentations. And there are many other works that have made important technical contributions to the research of SSL. Re.MixMatch Kurakin et al. (2020) introduces distribution alignment and augmentation anchoring. Flex Zhang et al. (2021 and FreeMatch Wang et al. (2022c) propose introduce currency pseudo labeling and self-adaptive threshold to deal with the learning difficulty of different classes.
Despite the promising strides in SSL, there inherent limitations that need to be addressed. The most salient among these is the prevalent assumption that labeled and unlabeled data samples share identical category spaces. When this assumption breached, there may be tangible dip in the performance of seen-class classification.
A.2 SAFE SEMI-SUPERVISED LEARNING
To address the issues mentioned above, mainstream safe SSL strategies tend to detect and exclude unseen-class samples to avoid misunderstandings in the training process. For example, UASD Chen et al. (2020) relies on the confidence threshold of model predictions to identify outliers. DS3L Guo al. 2020) assigns soft weights to each unlabeled instance by a weighting function based level optimization. Safe-Student He He et al. (2022) utilizes the logit output of the model to define an energy difference score. Furthermore, methods like MTCF Yu et al. (020), CAFA Huang et al. (021))
13

---

Published as a conference paper at ICLR 2025
OpenMatch Saito al. 2202]), and Fix_A Stee Huang al. (2023a) have introduced specific modules dedicated to the detection of unseen-classes.
However, previous methods for assessing the impact of unseen classes on SSL model performance are flawed. They fix the size of the unlabeled dataset and adjust the proportion of unseen classes within the unlabeled data to assess the impact. This process contravenes the principle of controlling variables. Adjusting the proportion of unseen classes in unlabeled data alters the proportion of seen classes, meaning the decreased classification performance of seen classes may not be due to an increase in unseen class samples in the unlabeled data, but rather a decrease in seen class samples. Thus, the prior flawed assessment standard that "unseen classes in unlabeled data can damage SSL model performance" may not always hold true.
B FURTHER INTRODUCTION
B. INTRODUCTION OF COMPARED SSL METHODS
We conduct experiments 10 popular SSL methods, 5 safe SSL methods. The 10 SSL methods we employed can be divided into three categories. Pseudo-Labeling Methods: This category involves generating pseudo-labels for unlabeled data and using these labels to guide model training.
PseudoLabel| [ee et al. 2013)) assigns the highest confidence class prediction as the pseudo-label.
FixMatch [Sonn et al. (2220) further refines this approach by training only on pseudo-labels exceeding a certain confidence threshold, combined with data augmentation to enhance label quality. FlexMatch Zhang et al. (202T) and FreeMatch Zhang et al. (202T) dynamically adjust thresholds based on the learning difficulty of different classes to balance learning progress. SoftMatch [Chnn et al. (2023) optimizes the quality and quantity of pseudo-labels by weighting them with truncated Gaussian function based on sample confidence. Consistency Regularization Methods: These methods emphasize the importance of the model maintaining consistent outputs response to minor perturbations of input data. PiModel Laine & Aila (217)) applies data and model perturbations to the same unlabeled sample, defining consistency loss through the mean squared error (MSE)
between two outputs. MixMatch [Berthelot et al. (2019) employs the MixUp Zhang al. (2017)
technique on both labeled and unlabeled data, creating new training data by mixing samples and their labels to calculate consistency loss for improved model generalization. UDA Xie et al. (2020) introduces perturbations with advanced data augmentation techniques and uses Kullback-Leible (KL) divergence as consistency loss to ensure output consistency. ICT Verma et al. (2022) ensures prediction consistency for interpolated samples with the original samples' prediction interpolations. Adversarial Training Methods: Represented by VAT Miyato al. (2018), , this approach defines consistency loss by identifying the virtual adversarial direction that maximizes the perturbation in model output without relying on label information. This method aims to improve the model's robustness to minor variations in input data, thereby enhancing its generalization capability.
In addressing the challenges of SSL., researchers have proposed various safe SSL methods designed to effectively utilize unlabeled data while identifying and filtering out Out-Of-Distribution (OOD) samples. UASD Chen et al. (2020) derives ensemble predictions by averaging all previous network predictions, conducting uncertainty- aware deff-distillation based on these ensemble predictions. MTCF Yu et al. (2020) initializes the OOD scores for both unlabeled and labeled samples and alternates between updating the network parameters and OOD scores, cleansing the noisy OOD scores of unlabeled samples in a joint optimization process. CAFA Huang et al. (202T) introduces a scoring mechanism to identify both labeled and unlabeled data within shared categories and employs domain adaptation to increase the efficiency of unlabeled data utilization. OpenMatch Saito et al. (202T) combines FixMatch with novel detection techniques based on one-vs-all classifiers. introducing an open-set soft-consistency regularization loss to enhance outlier detection capabilities. Fix_A Huang et al. (2023a) integrates MixUp with step direction modification strategies, optimizing the model's 's ability to utilize unlabeled data while maintaining the accuracy of the labeled set by adjusting the relationship between labeled and unlabeled data from gradient perspective.
B.2 FURTHER STATEMENT ON EVALUATION FRAMEWORK
In real-world applications, the primary focus is often on model performance with a fixed size of unlabeled data, rather than on how performance changes the proportion of unseen classes (r) varies.
14

---

Published as a conference paper at ICLR 2025
Fixing the size of the unlabeled set is practical approach, the exact proportion of unseen classes typically unknown during data collection. This framework provides valuable insights for practical cases.
However, evaluating sumi-supervised learning models under a single fixed r value may not fully capture their robustness generalizability. Evaluating performance across multiple values offers comprehensive understanding of model behavior under diverse conditions. This approach enables a systematic assessment of how the proportion of unseen classes impacts performance, highlighting robustness and adaptability.
Both evaluation approaches address distinct aspects of semi-supervised learning. Fixing the size of the unlabeled set emphasizes practical relevance for real-world scenarios, while evaluating varying proportions of unseen classes provides insights into model robustness and adaptability. Together, these perspectives contribute to more holistic understanding of semi-supervised learning models.
C EXPERIMENTS
C.I EVALUATION METRIC
Table 7 shows evaluation metric with range, interpretation of positive/negative values, and how relates to model performance.
C.2 EXPERIMENTAL DETAIL
C.2.1 REGARDING THE NUMBER OF UNSEEN-CLASS EXAMPLES
For CIFAR-10, we select the first classes seen classes and the last 5 classes as unseen classes. From the training set of known classes (which contains 25,000 samples), we randomly select 100 samples form the labeled set DL. We then randomly select rs x 25,000 samples from the remaining samples of the training set of known classes and put them into D From the training set of unknown classes (which contains 25,000 samples), we randomly select ru x 25,000 samples and put them into Dy. D and Dú: together form the unlabeled set Dt.
For CIFAR-100 100, we select the first classes as seen classes and the last classes as unseen classes. Other settings are the same CIFAR-10.
C.2.2 REGARDING THE NUMBER OF UNSEEN-CLASE CATEGORIES
We fix the number of unseen-class examples by setting 0.2. Unlike the first setup, instead of randomly selecting 25,000 samples from the entire training set of unknown classes, we randomly select 25,000 samples from the last C, classes of the unknown classes and put them into D
C.2.3 REGARDING THE INDICES OF UNSEEN CLASSES
We fix the number of unseen-class examples by setting r 0.2. Unlike the first and second setups, we randomly select x 25,000 samples from the C,-th class of the unknown classes and put them into D
C.2.4 REGARDING THE DEGREES OF NEARNESS OF UNSEEN CLASSES
Similar to the third setup, instead of randomly selecting x 25,000 samples from the C.-th class of the unknown classes, we randomly select x 25,000 samples from the C--th class of the external MNIST dataset and put them into D..
C.2.5 REGARDING THE LABEL DISTRIBUTION UNSEEN CLASSES
We determine the number of samples to be adopted from each unknown class to obtain D based on Ca.
15

---

Published as a conference paper at ICLR 2025
Table 7: A detailed description for each metric.
Metric
Range
Description
Interpretation
Ratgee
(-00, 00)
Indicates the overall trend of accuracy (Ace(r))
A larger Ratope (closer to 0 or positive) in- dicates better global robustness to unseen classes.
classes added. Posi- tive values imply incrensing accuracy, while negative values suggest decline.
GM
(0,00)
Measures the cumulative magnitude of ac- curacy deviations from the average mey (ACC), Higher values imply larger fluctuations.
A smaller GM indicates higher stability and robustness the additionn unseen classes.
WAD
(-00, 00)
Captures the largost negative accuracy drop between adjacent values.
A larger WAD indicates better local robust- to unseen-elass variations.
BAD
(-20,
Reflects the maximum positive or minimal negative accuracy change between adjacent r values. Positive values indicate improve- ments.
A larger BAD (positive or close to 0) in- dicales better performance in the best-cuse scenario.
PAn2o
10, I]
Represents the probubility that addding un- seen classes does not degrade performance (AD 2 O).
A higher Pango (close to 1) indicates bet- ter global robustness to unseen class addi. Bions.
Table 8: Evaluation on CIFAR100 with 1000 labels under the different number of unseen classes.
METHOO
boac
C,-10
CL-20
Cu=30
Cg=40
C,=50
Ralope 10
GM
BADx10
WADx10
PAD22
PSEUDOLAREL
0.421
0.415
0.416
0.415
0.417
0.417
0.000
0.003
0.001
0.000
1,000
PIMODEL
0.424
0.428
0.426
0.009
0.002
-0.001
FIXMATCH
0.441
0.384
0.338
0.398
-0.005
0.137
0.060
FLEXMATCH
0:423
0.012
UDA
0.380
0.385
0.382
0.379
0.386
0.583
0.000
0.030
0.007
0.250
SOFTMATCH
0.455
0.447
0.440
0.441
0.452
0.002
0.021
0.011
-0.00 02007
0.500
VAT
0.486
0.426
0.430
0.425
0.435
0.435
0.002
0.018
0.009
0.904
0.750
FRIEMATCH
0.455
0.451
0.421
B.426
0.439
0.435
0.003
0.028
0.013
-0.310
0.500
ICT
0.413
0.415
0:413
0.411
0.413
0.414
0.000
0.005
0.002
-0.002
0.500
LASD
0.415
0.409
0.415
0.414
0.413
0.411
0.000
0.010
0.006
-0.302
0.250
MTCF
0.248
0.258
0.250
0.238
0.227
0.226
-0.009
0.057
-0.001
-0.012
0.000
CAFA
0.424
0.421
0.413
0.014
0.004
-0.008
ORINMATCH
0.367
0.352
0.327
0.020
PIXEA.STEP
0.448
0.400
-0.033
0.013
C.3 CATEGORY-NUMBER FACTOR IN CIFAR 100
We validated different approaches on CIFAR100, with the results shown Table Here, base refers to the scenario where r=0, meaning no unseen class data is added. In other cases, is set to 0.2. To ensure a sufficient sample size, C, is set to 10, 20, 30, 40, and 50, respectively. On the CIFARI00 dataset, the accuracy of all methods is generally lower, reflecting the complexity of the dataset. As On increases from 10 to 50, the performance of many methods declines, but the rate of decline is more gradual compared to CIFAR1O. The performance of MTCF decreases significantly, while PiModel and PseudoLabel change little, indicating that these methods are more robust to different Category-number factors. Overall, looking at the results of the two datasets, some methods show strong robustness and adaptability when dealing with different unseen class categories (such as ICT, CAFA).
C.4 INCORPORATING NEARNESS FACTOR INTO GATEGORY-NUMERS FACTOR
We incorporate nearness factor into category number factor and the experimental results can be seen in Table 9 By comparing Table 9 and Table 3 in main body, we can conclude that in the far OOD scenario, SSL models are more robust to changes in the number of unseen classes. This indicates that the farther the OOD, the smaller the impact on SSL models (whether positive or negative).
C.5 THE IMPACT OF SEEN-CLASS RATIO r. FOR SSL MODEL
Previous works, such as DS3L Guo et al. (2020) when studying the impact of unseen classes on SSL models, employed a flawed approach that violated the principle of controlling variables. In their setup, they fixed the total number of unlabeled samples and merely adjusted the proportion of unseen class samples within this total. The decrease in SSL model performance, caused by the increase in the proportion of unseen classes, is not solely due to the increase in unseen class samples but
16

---

Published as a conference paper at ICLR 2025
Table 9: Evaluation on CIFAR1O with 100 labels under the different number of unseen classes from far OOD scenario.
METHOD
base
C,.1
C,
C,,3
C m4
C
Rylupe
GM
BAD
WAD
PADe0
SUPERVISED
0.617
0.617
0.617
0.617
0.617
0.617
0.000
0.000
0.000
0.000
1.000
PSEUDOLABEL
0.677
0,623
0.624
0.622
0.623
0.621
0.000
0.004
0.001
-0.002
0.500
PIMODEL
0.667
0.624
0.624
0.625
0.624
0.622
0.000
0.004
0.001
-0.002
0,500
FIXMATCH
0.612
0.702
0.703
0.678
0.689
0.691
-0.004
0.040
0.011
-0.025
0.750
FLEXMATCH
0.770
0.616
0.617
0.621
0.611
0.614
-0.001
0.013
0.004
-0.010
0.750
0.645
0.633
0.643
0.631
0.621
0.636
-0.002
0.027
0.015
-0.012
0.500
SOFTMATCH
0.764
0,642
0,656
0,639
0.642
0,644
-0.001
0.023
0.014
-0.017
0.750
VAT
0.710
0.581
0.578
0.588
0.585
0.582
0.001
0.015
0.010
-0.003
0.250
FREEMATCH
0.760
0,637
0.644
0.631
0.628
0.635
-0.002
0.022
0.007
-0.013
0.500
ICT
0.618
0.597
0.598
0.599
0.598
0.599
0.000
0.003
0.001
-0.001
0.750
UASD
0.618
0.617
0.619
0.618
0.619
0.618
0.000
0.003
0.002
-0.001
0.500
MTCF
0.772
0.596
0.557
0.605
0.567
0.603
0.002
0.094
0.048
-0.039
0.500
CAFA
0.652
0.621
0.620
0.621
0.617
0.608
-0.003
0.020
0.001
-0.009
0.250
OPENMATCH
0.713
0.617
0.614
0.605
0.598
-0.006
0.039
-0.009
0.000
FIX.A.STEP
0.622
0.693
0.682
0.689
0.677
0.690
-0.001
0.027
0.013
-0.012
0,500
Table 10: Evaluation on CIFARIO with 100 labels when we fix fu and change Ts-
METROD
rD-D
P.-0.2
---0.4
F2-0.5
7.-0.6
7.-0.8
r.-1.0
Ratage
GM
RAD
WAD
Paago
SUPERVISED
0.617
0.617
0.617
0.617
0.617
0.617
0.617
0.000
0.000
0.000
0,000
1.000
PSBUDOLABEL
0.653
0.669
0.672
0.673
0.675
0.674
0.675
0.018
0.037
0.080
-0.005
0.833
PIMODEL
0.631
0.651
0.655
0,659
0.655
0.660
0.668
0.030
0.053
0,100
-0.040
0.833
FIXMATCH
0.419
0.548
0.579
0.581
0,588
0.597
0,603
0.154
0.303
0,645
0.020
1.000
FLEXMATCH
0.626
0.749
0.747
0.757
0.756
0.109
0.229
0.460
40.025
0.667
MIXMATCH
0.678
0.708
0.722
0,723
0.728
0.731
0.729
0.047
0.096
0,150
-0.010
0.833
UDA
0.543
0.621
0.640
0.648
0.638
0.641
0.654
0.088
0,390
100
0.833
SOFTMATCH
0.596
0.710
0.737
0.743
0.735
0.752
0.755
0.131
0.261
0,570
-0.080
0.833
VAT
0.640
0.677
0.691
0.696
0.695
0.699
0,701
0.054
0.108
0,185
-0.010
0.833
PRREMATCH
0.488
0.723
0.689
0.751
0.707
0.751
0.725
0.184
0.408
1.175
-0.440
0.500
ICT
0.621
0.620
0.621
0.618
0.619
0.620
0.621
0.000
0.006
0,010
-0.030
0.667
UASD
0.618
0.619
0.620
0.617
0.619
0.618
0.618
-0 -0.001
0.005
0.020
-0.030
0.667
MTCF
0.689
0.744
0.759
0.762
0.762
0.766
0.772
0.069
0.136
0.275
0.000
1.000
CAFA
0.643
0.641
0.015
0.160
-0.030
0.500
OPENMATCH
0.555
0.605
0.629
0.674
0.632
0.670
0.648
0.095
0.205
0.450
-0.420
0.667
FIX_A_STEP
0.587
0.632
0.678
0.644
0.655
0.650
0.653
0.052
0.133
0.230
-0.340
0.667
could also result from the decrease in seen class samples. To test this hypothesis, we conducted the following experiment: We fixed the number of unseen class samples in the unlabeled data and varied the number of seen class samples in the unlabeled data to verify the previous conjecture. The results are shown in Table TO The table shows that when the ratio r, decreases from 1.0 to O, almost all methods experience& significant performance drop, indicating that the quantity of seen class samples in the unlabeled data also has a significant impact on SSL models. This also proves that the previous evaluation strategy was flawed. That is, the previous evaluation strategy could accurately describe the relationship between unseen classes and SSL models.
C.6 EVALUATION ON A LARGER-SCALE DATASET
Based on the results in Table we can see that under Imagenet- 100, the influence of the addition of unseen classes on the SSL methods smaller compared to CIFAR-10 and CIFAR- 100. Most SSL. methods exhibit similar properties across CIFAR-10, CI0, and Imagenet- 100. For example, ICT and UASD demonstrate robustness to the addition of unseen classes across all three datasets. SoftMatch and FixMatch relatively sensitive to the addition of unseen classes across all three datasets.
Additionally, similar to Tables I [ndd we also calculated five metrics measure the degree to which semi-st mani--sperviseed learning models are affected by the addition of unseen classes on Imagenet- 100, from both global robustness and local robustness perspective.
17

---

Published as a conference paper at ICLR 2025
Table 11: Evaluation on ImageNet- 100 with 5000 labels under inconsistent label spaces.
METHOD
r=0
r=0.2
r=0.4
r=0.5
r=0.5
r=0.8
r=1.0
Raiose
GM
BAD
WAD
PAZ2o
P-VALUE
PSRUBOLARDI
0.258
0.253
0.262
0.257
0.261
0.263
0.258
0.004
0,018
0.045
-0.050
0.500
0.541
PIMODEL
0.272
0.265
0.255
0.250
0.260
0.260
0.249
-0.017
0.044
0.100
-0.055
0.333
0.001
FIXMATCH
0.282
0.268
0.268
-0.025
0.053
-0.110
FLEXMATCH
0.291
0.291
0.291
0.287
0.284
0. 281
D.285
-0.009
0.023
0.019
-0.040
0:500
0.039
UDA
0.240
0.246
0.248
0.256
0.255
0.252
0.256
0.014
0.034
0.080
-0.015
0.666
0.001
SOFTMATCH
0.295
0.040
-0.069
0.269
0.265
0,260
0.269
0.269
-0.004
0.022
0.090
-0.035
0.500
0.035
PREEMATCH
0.296
0.288
0.289
0.284
0.287
0. 292
0.291
-0.002
0.020
0.030
-0.050
0.500
0.001
ICT
0.259
0.261
0,262
0,262
0.258
0. 0.260
0.258
-0.001
0.010
0,010
-0.040
0.666
0,180
0.261
0.260
0.255
MICF
0.091
0.091
0.096
0.110
0.099
0.102
0.098
0.005
0.027
0.136
-0.106
0.666
0.028
CAFA
0.257
0.254
0.261
0.264
0.258
0.255
0.263
0.004
0.022
0.040
-0.060
0.500
0.258
OPENMATCH
0.052
0.052
0.056
0.059
0.054
0.056
0.057
0.005
0.014
0.029
-0.049
0.833
0.013
FIX.A. STEP
0
0.261
0.260
0.264
0.257
0.261
0.255
-0.003
0.017
0,010
-0.070
0.600
C.7 EVALUATION ON A FINE-GRAINER DATASET
We conducted experiments for r=0, 0=5.5 and r=1. The results are shown in the Table 12 Experimen- tal observations are similar to CIFAR-10 and CIFAR-100, where the performance of most methods decreases as unseen classes are added (e.g.. FreeMatch and FlexMatch), while some methods exhibit robustness to the addition of unseen classes (c.g., UASD and ICT).
Table 12: Evaluation on CUB with 5000 labels under inconsistent label spaces.
METHOD
r=1.0
Raiaps
P-VALUE
PI MoDEL.
0.487
0.487
0.507
0.001
0.422
FIXMATCH
0.494
FLEXMATCH
0,529
0.447
0.413
-0.115
0.028
SOFTMATCH
0.510
0.517
0.477
-0.033
0.582
VAT
0.510
0.490
0,490
-0.019
0.000
FREEMATCH
0.509
0.448
0.410
-0.098
0,052
ICT
0.473
0,000
UASD
0.668
0.668
0.668
0.000
PIX-A-STEP
0.393
0,000
0.183
C.8 EVALUATION ON TABULAR DATASETS
Based the results from Table 3 and Table 14 it can be observed that compared to image modality data, tabular data is less affected by unseen classes, and semi-supervised models demonstrate greater robustness tabular data. We attribute this primarily to the simplicity of tabular data structure and the more effective feature representation. Moreover, similar to CIFAR, ImageNet, and CUB, ICT and UASD are relatively less affected by unseen classes compared other semi-supervised methods, demonstrating stronger robustness against unseen classes.
Table 13: Evaluation on Forest under inconsistent label spaces.
METHOD
0
0.2
0.4
r= 0.6
r=
1.0
Raiops
GM
BAD
WAD
PAD2O
P-VALUE
PSEUDOLABEL
0.610
0.603
0.606
0.607
0.604
0.503
-0.004
0.011
0.010
-0.032
0.400
0.000
PIMODEL
0.593
0.595
0.543
0.599
0.597
0.601
0.007
0.015
0.029
-0.009
0.600
0.022
FIXMATCH
0.577
0.580
0.580
0.579
0.586
0.008
0.016
0.027
-0.006
0.800
0.007
FLEXMATCH
0.584
0.591
0.591
0.590
0.586
0.593
0.004
0.016
0.036
-0.018
0.600
0.001
UDA
0.609
0.608
-0.003
0.200
0.024
SorT MATCH
0.592
0.587
0.591
0.592
0.593
0.600
0.008
0.015
0.035
-0.022
0.800
0.783
VAT
0.609
0.613
0.614
0.611
0.613
0.611
0.000
0.008
0.019
-0.014
0.600
0.000
FREEMATCH
0,591
0,603
0. 0.602
0.593
0.595
0.607
0.007
0.033
0.062
-0.041
0. 0.600
0.008
ICT
0.610
0.613
0.607
0.611
0.611
0.612
0.001
0.008
0.019
-0.029
0.600
0.455
UASD
0.605
0.604
0.606
0.598
0.598
0.603
-0.004
0.016
0.023
-0.038
0.600
0.084
MICF
0,590
0.592
0.594
0.596
0.596
0.598
0.007
0.013
0.010
0.001
1,000
0.001
0.604
0.602
0.604
0.602
0.611
0.603
0.012
-0.038
0.818
OPENMATCH
0.615
0.617
0.610
0.612
0.613
0.615
-0.001
0.011
0.011
-0.031
0.800
0.222
FIX_A_STRP
0.605
0.604
0.610
0.607
0.612
0.616
0.010
0.022
0.032
-0.018
0.600
0.048
18

---

Published as a conference paper at ICLR 2025
Table 14: Evaluation Letter under inconsistent label spaces.
MRTHOD
0
0:2
0.4
-=
Rylape
BAD
PADze
P-VALUE
PARUDOLABEI.
0.719
0.717
0.718
0.718
0.215
0.714
-0.004
0.009
0.005
-0.015
0.400
0.032
PIMODEL
0.726
0.719
0.716
0.715
0.717
0.710
-0.012
0.021
0.008
-0.037
0.200
0. 0.002
FIXMATCH
0.740
0.718
0.708
0.689
0.679
0.688
-0.056
0.110
0.044
-0.110
0.200
0.003
0.749
0.735
0.724
UDA
0.719
0.718
0.714
0.715
0.719
0.718
0.000
0.010
0.020
-0.016
0.400
0.085
SOFTMATCH
0.726
0.712
0.707
0.711
0.717
0.715
-0.005
0.027
0.026
-0,072
0.400
0.001
MEANTEACHER
0.739
0.738
0.738
0.737
0.738
0.740
0.000
0.004
0.011
-0.002
0.400
0.177
VAT
0.735
0.729
0.726
0.725
0.721
0.721
-0.013
0.023
0.000
-0.029
0.000
0.002
FREEMATCH
0.724
0.614
0.615
0.620
0.614
0.611
-0.079
0.181
0.025
-0.549
0.400
0. 000
ICT
0.728
0.729
0.729
0.730
0.727
0.730
0.000
0.005
0.012
-0.015
0.800
0.141
UASD
0.723
0.726
0.725
0.723
0.725
0.731
0.005
0.012
0.029
-0.011
0.600
0.089
MICF
0.706
0.607
0.544
0.542
0.529
0.512
-0.172
0.332
-0.011
-0.495
0.000
0.001
OPENMATCH
0.738
0.723
0.718
0.719
0.718
0.716
-0.017
0.033
0.007
-0.075
0.200
0.000
FIX STRP
0.760
0.747
0.740
0.738
0.736
0.733
-0.024
0.045
-0.006
-0.062
0.000
0.001
C.9 EVALUATION ON TEXT DATASET
According to Table 15 compared tabular data, text modality data is relatively more affected by unseen classes. We attribute this to the greater difficulty in feature representation and the higher complexity of text-based tasks. Consistent with previous experimental findings, UASD relatively less affected by unseen classes compared to other semi-suppr methods, demonstrating stronger robustness against unseen classes
Table 15: Evaluation on AGNews under inconsistent label spaces.
METHOD
r=0
-= 0.2
0.4
= 0.5
0.6
r= 018
1.0
R.iqpe
GM
BAD
WAD
PADZO
P-VALUE
PSSUDOLARRI
0.960
0.865
0.917
0.850
0.893
0.916
0.868
-0.047
0.212
0.430
-0.670
0.500
0.001
PIMonIS.
0.971
0.944
0.465
0.665
0.004
0.500
FLENMATCH
0.734
0.823
0.957
0.
0.874
0.867
0.108
0.881
-1.543
0.500
UDA
0.858
0.781
0.845
0. 889
0.922
8.902
8. 0.890
0.086
0.253
0.442
-0.384
0.500
0.529
SOFTMATCH
0.757
0.919
0.915
0.925
0.857
0.798
0.843
0.000
0.362
0.509
-0.678
0.500
0.002
MEANTEACHER
0.964
0.942
0.967
0.941
955
0.941
0.941
-0.018
0.071
0.142
-0.260
0.500
0.015
LASD
0.960
0.970
0.919
0.955
0.959
0.500
0.395
C.10 EXPANDED EXPERIMENTAL RESULTS WITH STANDARD DEVIATIONS
Standard deviations is important to provide a more comprehensive evaluation of the experimental results, We have added the standard deviations and the updated results can be seen in Table [6 Table 17
Table 18 and Table 19
C.11 EVALUATION ON OPEN- WORLD SEMI-SUPERVISED LEARNING METHODS
To expand the application scenarios of the RE-SSL framework, we also conducted validation in an open-wond world semi-supervised learning setting, exploring the impact of unseen classes in unlabeled data on the classification of seen classes and unseen classes by the cuni-supervieed model during the testing phase. Unlike safe semi-supervised settings, open-world semi-supervised learning also requires classification unseen-class data during testing. We evaluated two representative open- world SSL methods, ORCA Kai-Di Cao & Leskoved (2022) and SORL. et al. (2024). The experimental results are shown in Table 20 According to the above results, we know the addition of unseen classes is beneficial for both seen-class classification and cassen-class classification in both the ORCA and SORL.
19

---

Published as a conference paper at ICLR 2025
Table 16: Evaluation on CI with 100 labels under inconsistent label spaces.
METHOD
r=0.2
r-0.4
r-0.5
r-0.6
r=0.8
r=1.0
PerinoLAnm
0.67740.010
0.668-0.010
0.664-0.011
0.660+0.011
0.660+3.010
0.660=0.0) 811
0.65400013
PIMODEL
0.66710.011
0.65110.011
0.64410.000
0.63610.006
0.636.0.00
0.63010.007
PLEXMATCH
0.770=11.026
0.717=0.017
0.702=0.017
0.706=0.006
0.704=0.009
0.616=0.015
0.63#=0.0U5
MIXMATCH
0:731=0.013
0.708-00.00
0.708=0.017
706=0.015
0.701+0.017
0.697=0.00
0 0676+00.01
UDA
0.64510.025
0.621 022
0.60410.030
0.501 10.029
0.39110.023
0.39010.022 024
0.55210.025
MATTH
0.764++11016
0.710=0.022 026
0.69
0 0.607=0.032
0.692+00.018
0.682=0.014
D 0.607=000000
VAT
0.710:00.055
0.677:00.055
0.663:0 008
0 663:20.112
0.657::0.014
0.65120.017
O 0.931100000 007
PREEMATCH
0.76020.019
0.723=:0012
0.66520.058
0.608.:0.012 0.63520.015
0.605
D 0.44020.0441
ICT
0.618+0.080
0.619=0.011
0.621=0.012
O 620+00.011
0.621+0.012
0.619=0.014
O 0.62+00.013
UASD
0.61810.010
0.61910.00 009
0.61710.008
0.616.000007
0.61710.008
0.618 008
0.61810.008
MTCF
0.772+0.012
0.743=2..0
0.731=0.017
0.725+0.010
0.725=00.015
0.716
0.692+0.015
CAFA
0.65210.014
0.60010.014
0.65310.010
0.04110.015
0.642.10.0 012
0.601000066
OPENMATCH
0.713:81.010
0.006=0.075
0.586=0.052
0.595=0.033
0.579:00027
0.517=0.030
0.473#0.044
FIX,A_STRP
0.662:40.081
0.634+0.06 061
0.615=0.062
0. 0.582:0.095
0.555+00.30
0.585+0.013
0.509+0.013
Table 17: Evaluation on CIFARIO with 100 labels under the different number of unseen classes.
METHOD
C,51
C242
C2=3
C,
G,=5
PSHUBOLARRL
0.648
0.652-0:012
66333-0
0.664+0.011
0.668+0.010
PIMODEL
0.622:10.00 808
0.628 005
0.610::0.01 014
0.612:0.014
0.651.10011
FLEXMATCH
0.658:0.002 034
0.644=0.015
0 089922 008
0.699:1.009
0.7155:.0019
UDA
0.620++0.01
0 0.59++0.05 0.590-0.010
0.621#0.014
0.621++0.022
SOFTMATCH
0.671:10.013
0.72410.011 0.662=0.047
0.71510:021
0.71010.026
VAT
0.606:20.000 028
0.627=0.034
0 0.671::0000 02)
0.676::0001
0.67710.015
FREEMATCH
0.66810.033
0.639_0.054
0.7241:0.00 001
0.58820.0027
0.723:10012
ICT
0.611:0.015
0.613=0.008
0.6152:0.010
0.61920.011
0.61910.011
0.620=0.009 009
0 0.617=-0000
0.619=0.007
0.619+0.009
MTCF
0.723:50.007
0. 0.7371:00009 0.704
0.7361:0.022
0.743-000099
CAFA
0.642+0.015
0.645-0.016
0 0.642+0.010
0.643+00.009
0.648+00013
OPENMATCH
0.642:00.08
0.634=0.049
0.633=00031
0.561=0.035
0.606+0.075
FIX.A A STOP
0.674:20.0 U26
0.647=0.037
0.631100.300
0.625=000055
0.63240.061
Table 18: Evaluation on CIFARIO with 100 labels under the different index of unseen classes.
near
/sr
4lob
Cick
CLas
41,7
B 66130000000 637
6.519-610 806
D 60 m0
01675 11 678
0.631
4 1000
Dr E7606 ror
PLEMMATH
0.62220.015
0.882-8.815
3.S1005.009
0.606:9.249
857
0.411-6 511
20
8004224100 100
0 0.633233955
0:723-8.806
0.68720000 013
2.66425-221
B.00220122
046320 033
0.844:3.222
D.64220 00.3
066558.00 m6
0 0661663007 031
6.700-6:00 004
D 7665669000 075
0:058-0.00 035
0665560061
0 0.60110000 053
0.616-6:000 065
D1 D163740 061
ICT
R4225240099
0.61220403
0.822.9.509
0.11220.013
0.391120004
0.5062100077
0.99322.007
0.897:8.006
0.29720
LAAD
B81758.010
0.615533081
0af0500007 0,518=9.007
0.61859.003 022
3.611:5::06
0:61855-003
0.618=8.209
ATTCFF
0.701
688 47.
0.547 6.014
061410003
0.546 1.566-6.566
0186240125
0 2354222415
0,549-5.315
0.366200 027
0.442300134
3.80125:041
8.309241005
0 0.567221445
0.825=39246
D.82220 (00
Table 19; Evaluation on CIFARIO with 100 labels under different imbalance factors of unseen classes.
Caz0.01
Cuz0.02
Coz0.05
Ca=D.
Cu=0.30
PAEUDOLABEL
0.660=0.014
0.60000:117
0.66540.004
0.662=000112 0.658:0.015
PIMons..
0.647=0001 011
0.64900.00
0.62200.000
0.646:0.000 008
0.642:00.0
PLEXMATCH
0.71010.019
0.70310.021
0.71610.010
0.70610.017
0.71710.014
UDA
8.620=0.035
0164700.099
0.654=00.29
0.615=00043
0.619=0.0522
SOFTMATCH
0.007
0.731:00014
0. 7.511000009
0.721100011
0.698:0.033
VAT
0.6811200116
0.68220.013
0.68220.012
0.6782:0.012
0.67520.019
FRIEMATCH
0,730=0.013
0.723-0.015 016
O. 0.674+00067
0.670++0.06 06063
0.677=0.002
ICT
0.62210.010
0.62110.011
0.62110..09
0.622
0.62210.008
UASD
0.618
0.618=0.00 0n8
0.617#0.008
0.618=0.008 008
0.617
0.722.100144
0.72010.011
0.72210.016
0.72110.011
0.72210.00 0.3
CAFA
0.647=0.0005
0.650:00000
0.64320.001
0.64820.013
0.646:20.01
OPENMATCH
0.61 3-:0--0000 068
0.613=0.074
6.00-0.077
0.650+0.006
0 867
FIX STEP
0.685110033
0.63810.066
0.632.10.00
0.60311.004
0.62810.066
Table 20: Evaluation of open-world semi supervised learning methods on CIFAR-10 with 100 labels under inconsistent label spaces.
0=
==0.4
r=
r=0.8
1.0
METHOD
SEEN/UNSEEN
SEEN/UNSEEN
SEEN/UNSEEN
SBEN/UNSEEN
SEEN/UNSEEN
ORCA
0.512/0.631
0.607/0.6 633
0.616/0.622 0.651/0.881
0.669/0.774 0.663/0.893
0.682/0.750 0.729/0.888
SORL
0,502/0.709
0.615/0.888
20


Reviews:
[{"review_id": "U1azeg98Tv", "reviewer": "Reviewer_Mtt1", "summary": "The paper investigates the impact of unseen classes in unlabeled data on the performance of semi-supervised learning (SSL) models. It challenges the prevailing assumption that unseen classes are detrimental to SSL performance, highlighting flaws in previous assessment methods that altered the proportion of unseen and seen classes simultaneously. This paper adheres to the principle of controlling variables by keeping the proportion of seen classes constant while varying unseen classes across five dimensions. Through rigorous experimentation, the authors demonstrate that unseen classes do not inherently degrade SSL model performance; in some scenarios, they may even enhance it. The findings suggest a reevaluation of the role of unseen classes in SSL, emphasizing that their effects are more nuanced than previously thought.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper effectively identifies a flaw in previous methods for assessing the impact of unseen classes on SSL model performance. By fixing the size of the unlabeled dataset and adjusting the proportion of unseen classes, earlier approaches fail to control for variables appropriately. This adjustment alters the proportion of seen classes, meaning that any decrease in classification performance for seen classes may not be solely due to an increase in unseen class samples, but rather to a decrease in seen class samples. This clarification helps refine our understanding of how unseen classes affect SSL model performance.\n2. The experiments presented in this paper are thorough, as the authors control various variables to validate their claims from different perspectives. This comprehensive approach strengthens the credibility of their findings.\n3. The writing in this paper is clear and easy to understand, making it accessible to readers.", "weaknesses": "1. In the proposed five metrics, some formulas are not clearly defined. For instance, the meaning of \\(\\hat{ACC}\\) in Eq. 1 and \\(\\bar{ACC}\\) in Eq. 2 is not explicitly explained. Providing clearer definitions for these terms would enhance the understanding of the metrics and their implications in the context of the research.\n2. In the tables, it would be helpful to use bold text or dashed lines to highlight specific data points, emphasizing the advantages of certain algorithms under particular settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SN57D7XM4o", "reviewer": "Reviewer_244R", "summary": "This paper studies the problem of safe semi-supervised learning. It found that the previous works are based on flawed evaluations, which did not adhere to the principle of controlling variables. They fixed the size of unlabeled data but change the proportion of unseen classes, which would also affect the proportion of seen classes. To deal with this problem, the paper proposes a re-evaluation framework to keep the proportion of seen classes unchanged and adjust the proportion of unseen classes. Furthermore, the paper also propose five metrics to comprehensively evaluate the impact of unseen classes. Based on the proposed framework, extensive experiments are performed to evaluate the impact of unseen classes on various SSL models.", "questions": "There are five evaluation metrics. Which one is the most important?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper does not provide any new method. It gives an insightful perspective on the previous studies and finds that these methods have issues accurately assessing the impact of unseen classes. The motivation is very clear and seems to be reasonable. \n\n2. Based on the above motivation, the paper proposes to re-evaluate previous safe SSL models by fixing the proportion of seen classes. Furthermore, the paper proposes five evaluation metrics to access the impact of unseen classes on SSL models.\n\n3. Extensive experiments based on the proposed framework are performed with various safe SSL methods. These results provide a comprehensive and fair comparison among existing methods.", "weaknesses": "The main contribution of this paper lies in proposing a more reasonable experimental setup; however, its major issue is the credibility of the experiments. The experiments are conducted solely on CIFAR-10 and CIFAR-100, which makes the results unconvincing. While I understand that past studies has primarily been conducted on these two datasets, given that this paper aims to make innovations from an experimental perspective, it should propose more practical experimental setups and conduct experiments on more realistic datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kx2oRkTow6", "reviewer": "Reviewer_CS3Q", "summary": "The paper proposes an evaluation framework and then evaluates whether unlabeled samples from unseen classes deteriorate the performance of semi-supervised learning methods on seen classes.", "questions": "I hope the authors can clarify the following question.\n\n**Q1.**  As mentioned in line 269-299, there are three experiments (generated by three random seeds) for each kind of dataset construction. So, the experiment results given in the paper are still with high randomness. So, it is difficult to accept these conclusions derived from these experiment results. In order to dispel this doubt, the authors should strengthen the experiment from the following aspects.\n\n    a. The number of datasets should be increased, e.g., 10 or more.\n\n    b. The diversity of datasets should also be increased, rather than being limited to image datasets.\n\n    c. For each dataset construction (labeled and unlabeled samples setting),  more experiments should be carried out, e.g., 10 times or more.\n\n    d. Some statistical significance tests should be introduced.\n\n**NOTE: I am willing to increase my rating to a 6 or 8, if the authors can dispel my doubts through more adequate experiments.**", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**S1.** The motivation of the paper is valid and interesting.\n\n**S2.** The proposed evaluation framework follows the principle of controlling variables by fixing the confounding factor, i.e. the proportion of unlabeled samples from visible classes. So, the proposed evaluation framework overcomes the shortcomings of current evaluation schemes.\n\n**S3.** The influence of unlabeled samples from unseen classes is evaluated from five perspectives and five evaluation indices.\n\n**S4.** Some representative methods are chosen for experiment and the experiment results can provide some guidance for understanding the working mechanism of existing methods and designing new ones.", "weaknesses": "In order to more clearly present the findings of the paper, the presentation needs to be further improved, specifically.\n\n**W1.** Since $r$ (or $r_u$) is recorded in the tables in the experiment section, there is a precise definition for $r$. To reduce ambiguity, the formal definition of $r$ should be given based on the symbol system in Section 2. \n\n**W2.** It is necessary to provide a table or brief description for each indicator, showing its range, interpretation of positive/negative values, and how it relates to model performance.\n\n**W3.** To facilitate readers to intuitively assess the degree of impact and facilitate cross dataset comparisons, the evaluation indicators in formulas (1)-(5) should to be normalized if possible.\n\n**W4.** More experiment details should be recorded, such as the names of seen and unseen classes, the number of labeled samples for each class, and the number of unlabeled samples for each class. If there is not enough space, put it in the appendix.  Furthermore, the authors should provide a link to their code or dataset splits, if possible, to further enhance reproducibility.\n\n**W5.** In addition, some clerical errors need to be corrected, specifically.\n\n    a. In line 072, ”unseen classes in ---> “unseen classes in.\n\n    b. In line 141, Fig.2(b) ---> FIg.2(a).\n\n    c. In line 480-481, the table’s title should be above the table.\n\n    d. In the experiment, how are the integrals in formulas (1) and (2) calculated? It seems that the integral operation should be the accumulation operation.\n\n    e. In line 190, how can we obtain the “the expected accuracy”? In my opinion, it should be “the mean of ....”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dreVNACySA", "reviewer": "Reviewer_wdK8", "summary": "This paper proposes an evaluation framework for safe semi-supervised learning (SSL) with unseen class data. The paper observes that previous works fix the total number of seen and unseen class data, which may not reflect the true influence of unseen class data. Therefore, they fix the number of seen class data and change a number of quantities of unseen class data to investigate their influence. Extensive experiments are performed with much analysis.", "questions": "- Are the experimental results in tables the average accuracies with multiple seeds?\n- Is $C_{ib}$ the ratio between unseen and seen data?\n- Is each entry in Table 6 the mean of several $C$, i.e. the mean of each row?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The shortcomings of previous evaluation approaches are new to the SSL field.\n- Many experiments are performed, providing a solid and comprehensive evaluation for robust SSL.\n- The experimental analysis is also very comprehensive.\n- The paper is well written and easy to understand.", "weaknesses": "- My main concern is the practicality of the framework. The main purpose of the paper is to test the influence of *increasing numbers* of unseen class data on the classification performance on seen class data. The proposed metrics are mainly *one-order statistics* of the classification performance, which show the change in performance with more unseen class data. However, the main interest of SSL with unseen class data in real-world applications may not be focused on one-order statistics. In real-world applications, we only care about the performance of a certain number of unseen class data instead of the performance change with more data, although many papers provide the figures with different numbers of unseen class data. Therefore, although the framework is valid for different sizes of unseen class data, the practicability for real applications may be limited.\n- Only CIFAR datasets were used for experiments. Since previous SSL benchmarks provide more datasets from different modalities, such as text or tabular data, it is beneficial to consider more text and tabular datasets.\n- The standard deviations of the experimental results should also be included.\n- Some typos should be checked. For example, in Eq. (1), it should be argmin. In Line 431, it should be Table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)