Realistic Evaluation of Semi-supervised Learning Algorithms in Open Environments
Authors: Lin-Han Jia
Lan-Zhe Guo
Zhi Zhou
Yu-Feng Li
Venue: ICLR 2024 spotlight (2024)
Keywords: Semi-Supervised Learning; Robustness; Open Environments
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Semi-supervised learning (SSL) is a powerful paradigm for leveraging unlabeled data and has been proven to be successful across various tasks. Conventional SSL studies typically assume close environment scenarios where labeled and unlabeled examples are independently sampled from the same distribution. However, real-world tasks often involve open environment scenarios where the data distribution, label space, and feature space could differ between labeled and unlabeled data. This inconsistency introduces robustness challenges for SSL algorithms. In this paper, we first propose several robustness metrics for SSL based on the Robustness Analysis Curve (RAC), secondly, we establish a theoretical framework for studying the generalization performance and robustness of SSL algorithms in open environments, thirdly, we re-implement widely adopted SSL algorithms within a unified SSL toolkit and evaluate their performance on proposed open environment SSL benchmarks, including both image, text, and tabular datasets. By investigating the empirical and theoretical results, insightful discussions on enhancing the robustness of SSL algorithms in open environments are presented. The re-implementation and benchmark datasets are all publicly available. More details can be found at https://ygzwqzd.github.io/Robust-SSL-Benchmark.

Published as a conference paper at ICLR 2024
REALISTIC EVALUATION OF SEMI-SUPERVISED LEARN- ING ALGORITHMS IN OPEN ENVIRONMENTS
Lin-Han Jia', Lan-Zhe Guo . Zhi Zhou', Yu-Feng Li
National Key Laboratory for Novel Software Technology
School of Artificial Intelligence? School of Intelligence Science and Technology
Nanjing University, Nanjing 210023, China
{jialh, guolz, zhouz, liyff eyffee. @ lamda nju. edu.cn
ABSTRACT
Semi-supervised learning (SSL) a powerful paradigm for leveraging unlabeled data and has been proven to be successful across various tasks. Conventional SSL studies typically assume close environment scenarios where labeled and unlabeled examples are independently sampled from the same distribution. However, real- world tasks often involve open environment scenarios where the data distribution, label space, and feature space could differ between labeled and unlabeled data. This inconsistency introduces robustness challenges for SSL algorithms. In this paper, we first propose several robustness metries for SSL based the Robustness Analysis Curve (RAC), secondly, we establish theoretical framework for studying the generalization performance and robustness of SSL algorithms in open environ- ments, thirdly, we re- imimleemen widely adopted SSL algorithms within a unified SSL toolkit and evaluate their performance on proposed open environment SSL benchmarks, including both image, text, and tabular datasets. By investigating the empirical and theoretical results, insightful discussions on enhancing the robustness of SSL algorithms in open environments and benchmark datasets are all publicly available. More details can be found at https:/lygzwqzd.. wwtpg//d..iii..iiiiie LooRRobsttSSL--eeee
presented. The ce-implemennation
INTRODUCTION
Semi-supervised learning (SSL) aims leverage unlabeled data improve learning performance when labels are limited or expensive to obtain (Chapelle et al.] 2006). SSL. algorithms have been repeatedly reported to achieve highly competitive performance purely supervised learning and save a lot of labeling costs, by exploring the structure of unlabeled data.
All of the positive results, however, are based on the close environment assumption where labeled and unlabeled data are sampled from the same distribution independently. However, many practical applications involve open environments [hhou] 2022) where the data distribution, feature space, and label space could be inconsistent between labeled and unlabeled data. SSL methods suffer severe robustness problems in open environments and could be even worse than simple supervised learning
model without exploiting more unlabeled data (Guo & Li 2018) Oliver et al. [2018] Guo et al.) 2020a Li et al., (02T). Such phenomena undoubtedly go against the expectations of SSL. and limit its effectiveness in more practical tasks.
The robustness of SSL in open environments has attracted great attention recent years and various robust SSL algorithms have been proposed from different perspectives, such as inconsistent label
space (Guo et al., 2020a Chen et al. 2020 Yu et al.] 2020; Saito et al., 2021 Guo & Li 2022
Wei et al. 2022). inconsistent data distribution (Guo et al. 2020b Zhou et al. 2021 Huang et al. 2021 et al. 2023a). However, these algorithms primarily focus on robustness from a singular perspective and overlook the utilization of practical metrics for robustness analysis. Consequently, it remains challenging to ascertain the suitability of SSL. algorithms in real-world open environments.
Cooresponding Authors: Lan-Zhe Guo and Yu-Feng L.i.

---

Published as a conference paper at ICLR 2024
In this paper, we first propose several metrics considering different aspects of performance in open environments to achieve fair and comprehensive evaluation of SSL algorithms. Then. we establish a theoretical framework for studying the generalization performance and robustness of SSL algorithms, and the results show that the generalization error in SSL, consists of five components: bias caused by the learner, variance caused by data sampling, and three types of inconsistencies caused by open environments. Finally, we re-implement widely adopted SSL algorithms within a unified SSL toolkit and evaluate their performance on proposed open environment SSL benchmarks, including both image, text, and tabular datasets. Some interesting findings include:
Inconsistency between the feature and label space has a more detrimental impact compared cases where there is inconsistency in data distribution.
Traditional statistical SSL algorithms can often outperform deep SSL algorithms in terms of both performance and robustness when applied to tabular datasets. Thus, more advanced SSL algorithms tabular datasets should be studied.
Certain robust SSL algorithms currently proposed do not consistently exhibit enhanced robustness and may not surpass ordinary deep SSL. algorithms in most scenarios. We argue that the robustness of SSL algorithms should be evaluated under more reasonable metries.
Inconsistency between labeled and unlabeled data does not invariably result in negative effects. On the contrary, leveraging inconsistent unlabeled examples may improve perfor- mance in some cases. Thus, it is important to study how to exploit helpful information from inconsistent unlabeled data.
2 ROBUST SSL IN OPEN ENVIRONMENTS
2.1 NOTATIONS
SSL algorithms leverage both labeled and unlabeled data for the learning process. In the close environments, all data generated from consistent distribution P(z, y), X, on consistent data space X' x y C Rdd x {1,... k) where and respectively represent the number of features and classes. In SSL,, and ni unlabeled examples Dy = { [ ~ P(x)) where P(r) is the marginal distribution of P(x, y). The purpose is to learn predictor with the smallest generalization error on P(x, g).
given ny labeled examples Dr = {(I,,, (x),, y:) P(x, y) ==
In the open environments, we assume that all examples originate from a global data space X x C
Rd x (1,...,,k* } where d' and k" respectively represent the number of features and classes that appear throughout the entire learning process. There exists invariant data distribution P(y lx") for x² E X" and y" E y* We denote the degree of inconsistency of data distribution, feature space, or label space between unlabeled and labeled data E [o, 1]. A higher indicates a greater inconsistency. For any f, there is an inconsistent distribution denoted as Px y² ). However, we can only obtain projected distribution Pt (x,,, ye) in a subspace X; x V., where X, C X' and V C C
We denote (tt as the function describing the ratio of inconsistent examples in the unlabeled dataset to t. For robust SSL with any t, we are given ny labeled examples from Po y). (1 (())) ny consistent unlabeled examples from Po(x)) and 0(1) n,, inconsistent unlabeled examples from (z)).
2.2 PERFORMANCE METRICS
To achieve a fair and comprehensive evaluation, we introduce a set of performance metrics tailored for robust SSL in open environments. These metrics begin by defining a function Ace(t), which quantifies the change in model accuracy as a function of the inconsistency level t. This function is used to construct the Robustness Analysis Curve (RAC) that maps the inconsistency level to the corresponding accuracy Acc(t). Unlike traditional SSL evaluations that focus solely on Acc(O)or robust SSL evaluations that consider only a specific Acc(t), our proposed metrics are derived from the RAC and provide a more comprehensive evaluation of SSL algorithms. These metrics include Area Under the RAC Curve (AUC) which captures the overall robustness of the SSL algorithm; Expected Accuracy (EA) which describes the average performance across all inconsistency levels; Worst-Case Accuracy (WA) which identifies the lowest accuracy level, representing the worst- case scenario; Expected Variation Magnitude (EVM) which captures the average magnitude of
2

---

Published as a conference paper at ICLR 2024
Table Performance Metrics for Robust Sunieisuee Learning in Open Environments. Ace(t) describe the change in model accuracy with the inconsistency extent t, Pr(() is the distribution for t, A-c(()) indicate the first derivative.
Area Under the Curve (AUC)
AUC(Acc) So Acc(t)dt
Expected Accuracy (EA)
EA(Pr, Acc) (Pr, Acc) = So (rc()) WA(Ace) min Acc(t)
Worst-Case Accuracy (WA)
Expected Variation Magnitude (EVM)
EVM(Acc) = -6 AAce (t)ldt VS(Acc) -01 [Ace' (t) ( Ace' Acd (t)dt)))dtt
Variation Stability (VS)
1 Aco(t).zdt Io Aoc(t)dt
Robust Correlation Coefficient (RCC)
RCC(Acc) VSa Pdt VV Acc (S Aoc(t)at)
performance variation; Variation Stability (VS) which quantifies the stability of the performance variation; Robust Correlation Coefficient (RCC) which captures the overall trend of performance variation. The detailed formulation of these metrics is presented in Table By considering these diverse metrics, we can provide comprehensive evaluation of the robustmess of SSL algorithms, capturing different aspects of their performance. Moreover, these metrics are not limited to accuracy and can be extended to other performance measures by replacing the funetion Acc(t).
2.3 DEFINITION OF ROBUSTNESS SSL IN OPEN-ENVIROMMENTS
Based on the proposed metrics, we propose a formal definition of the robust SSL in open environments, including the expected robustness and worst-case robustness.
Definition 1. An SSL algorithm A returns model F using labeled data DL and unlabeled data for any inconsistency level t where F is the hypothesis space of A. Let Acc(t) denote the accuracy of ft on the test data. When the inconsistency follows a distribution Pt(t), if there exists o such EA(Pr, Acc) S. OE, we say that A exhibits a og-expected algorithmic robustness. If there exists ow such that (Acc(O)- WA(Ace) S ow, we say that A exhibits ow hoy-wors--cad algorithmic robustness.
that [Acc(0)-
In open environments, the SSL algorithm is employed to generate models with different inconsistency ratios t. If the algorithm consistently deliver satisfactory performance across range of we deem it to exhibit expected algorithmic robustness. the algorithm can maintain acceptable performance levels in the worst case, consider it to demonstrate worst-case algorithmic robustness.
THEORETICAL STUDIES ON ROBUST SSL
To analyze how to improve the robustness of algorithms in open environments, establish theoretical framework for studying the generalization performance and robustness SSL algorithms in open environments.
Specifically, we first define the projection operations Tx and hl to project data distributions originating from different features and label spaces the same spaces with labeled data. Secondly, we formally define two types of inconsistencies: feature space inconsistency Disce and label space inconsistency DiscI, both of which represent additional generalization errors caused during the space projection process. Combined with the distribution inconsistency within the same data space Discp defined in (Jia et al., 2023a), these constitute three types of inconsistencies in total. Finally, we analyze the SSL process in an open environment and ultimately conclude that the generalization error in SSL consists of five components: bias caused by the learner, variance caused by data sampling, and three types of inconsistencies caused by open environments.
Theorem 3.1. For any target predictor f E F, pseudo-label predictor h E H, 0 S 8, S 1. 0 82 and dg 1, with the probability of least (1 51)(1 62) 6z):
FJJ, Po(x, 1), w. mapx, - Dí. Du.))
ÍC Dr)
Th
B(J. D..) ) var(F n + n ko, 61)
+ n
3

---

Published as a conference paper at ICLR 2024
Net (f' (ttss) (P. (3))
ni n
0" (t) (Ty P (z)], [c]], y (xx)) maxx
8t (t) (T1x Thy P( (x" y)l, Po(2,
(É(h, DL) + (H, nu k 62) + var(H, nu Ko: 03) net
Tu m
O" (H)Discc(PE (x" ), 0t 00 Discp (1y, [Pt Hy (PT (xx mapx:-xo-
-0" (()ss ! v)ll po(
(1)
where E(f, Di.) is the weighted disagreement rate between the noisy pseudo-labets and the predic- tion results of f on the weighted unlabeled dataset Dū.-
The conclusions drawn from theoretical analysis are as follows: inconsistencies in data distribution, feature space, and lahel space can all harm the generalization performance of the model. To alleviate the issue of data distribution inconsistency, it is primarily dependent on aligning the distributions based on the existing predictor, thereby optimizing the term DiscD. To alleviate the issue of feature space inconsistency, it is primarily dependent the feature mapping function, which requires the learning algorithm to accurately infer unobserved features based on the observed features, thereby optimizing the term Discp. To alleviate the issue of label space inconsistency, it primarily relies on sample selection and weighting functions, which require robust SSL algorithms to accurately detect and mitigate the negative impact of unfavorable examples, thereby optimizing the term Disc.. The detailed theoretical analysis and proof are shown in appendix A.2
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
In our experiments, we evaluate both statistical SSL and deep SSL algorithms. For statistical SSL, we select 6 classical algorithms including the Semi-supervised Gaussian Mixture Model (SSGMM) (Shahshahani & Landgrebe| 1994) from the generative SSL algorithms, TSVM (Joachims) al.) 1999) from the semi-supervised support vector machine algorithms. Label Propagation (Zhu &
Ghahramani 2003) and Label Spreading (Zhou et al. 2003) from the graph-based SSL algorithms,
Tri-nnaining (Zhou Li 2005) from the disagreement- based SSI. algorithms and Assemble (Benneti
ce al. 2002) from the ensemble-based SSL algorithms. For deep SSL, we select 10 representative
algorithms: Pseudo Label (Lee) 2013). Pi-Model (Laine & Aila 2017). Mcan Teacher (Tarvainen & Valpola 2017), ICT (Verma al. 2022). VAT (Miyato et al., 2018), UDA (Xie et al. [2020). FixMatch (Sohn et al. (0200). FlexMatch (Zhang et 2022). Match (Wang et al., 2022b) and
SoftMatch (Chen al.) [202]]. We also considered robust SSL algorithms: UASD (Chen al.)
2020). CAFA (Huang et al. (021), MTCF (Yu et al. 2020), and Fix-A-Step (Huang et al.) 2023).
We conduct experiments various types of datasets, including 3 tabular datasets: iris, wine, letter;
3 image datasets: Image-CLEI CLEF (Caputo et al., 2014)). CIFAR-10, 10, and CIFAR- 100; 3 text datasets: Amazon reviews (MeAuley Leskovee 2013). IMDB movie reviews (Maas al. 2011). I), and
agnews (Zhang et al. 2015).
For all the experiments. we use mainstream supervised learning algorithms as baselines. For tabular data, we use XGBoost (Chen & Guestrin 2016) as the benchmark for statistical learning algorithms and adopt FT-Transformer (Wang et al. 2022a) as the baseline and backbone for deep learning algorithms. For visual data, we use ResNer50 (He et al.) (016) as the baseline and backbone. For text data, we use the RoBERTa (Liu et al.) 2019) model as the benchmark and backbone. All SSL. algorithms are re- re-implemented based on the LAMDA-SSL toolkit (Jia et al. 2023b).
We plotted the RAC and performed statistical analysis on various evaluation metries for different methods. For the plotting of the RAC curve, we sampled six t values [0, 0.2, 0.4, 0.6, 0.8, 1] for all open environments. To ensure reliability, we conducted three experiments for each sampling point with seed values of 0 2. The average of these experiments used to plot the curve. Linear interpolation was performed between adjacent sampling points. More detailed settings of the experiments are presented in appendix A.5
4

---

Published as a conference paper at ICLR 2024
Table Evaluation of SSL algorithms using letter dataset under inconsistent data distributions
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.643
0.643
0.643
TSVM
0.624
0.650
0.607
0.012
0.012
-0.733
SSGMM
0.276
0.334
0.245
0.022
0,029
-0.740
Label Propagation
0.524
0.629
0.486
0.029
0.037
-0.833
Label Spreading
0.588
0.653
0.563
0.020
0.025
-0.780
Tri-Training
0.625
0.689
0.600
0.018
0.024
-0.851
Assemble
0.644
0.646
0.641
0.003
0.003
0.037
Transosffrrm
0.660
0.660
0.660
Pseudo Label
0.658
0.667
0.652
0.003
0.002
-0.977
Pi-Model
0.683
0.698
0.673
0.005
0.003
-0.661
Mean Teacher
0.687
0.687
0.687
0.000
0.000
VAT
0.702
0.725
0.686
0.008
0.005
-0.669
ICT
0.677
0.677
0.677
0.000
0.000
UDA
0.693
0.693
0.693
0.000
0.000
FixMatch
0.644
0.739
0.588
0.032
0.022
-0.947
Flex
0.649
0.687
0.618
0.014
0.010
-0.973
FreeMatch
0.474
0.629
0.406
0.050
0.053
-0.113
SoftMatch
0.584
0.654
0.564
0.020
0.032
-0.664
UASD
0,701
0.702
0.700
0.001
0.002
0.008
CAFA
0.658
0.659
0.657
0.001
0.001
-0.666
MTCF
0.365
0.612
0.270
0.081
0.102
-0.702
Fix-A-Step
0.682
0.739
0.642
0.019
0.013
-0.976
Table 3: Evaluation of deep SSL methods using ImageNee/Caltech dataset.
Dataset
Model
AUC
Ace(O)
WA
EVM
VS
RCC
Supervised
0.909
0.909
0.909
Pseudo Label
0.907
0.908
0.907
0.001
0.001
-0.621
Pi-Model
0.909
0.907
0.907
0.001
0.001
0,655
Mean Teacher
0.903
0.904
0.900
0.003
0.003
0.169
VAT
0.888
0.881
0.881
0.002
0.002
0.928
ICT
0.907
0.909
0.903
0.003
0.004
-0.359
UDA
0.896
0.904
0.891
0.006
0.007
-0.512
ImageNet/Caltech
FixMatch
0.902
0.905
0.887
0.005
0.007
-0.726
FlexMatch
0.906
0.921
0.893
0.008
0.010
-0.861
FreeMatch
0.864
0.916
0.832
0.031
0.028
-0.786
SoftMatch
0.904
0.908
0.891
0.007
0.007
-0.805
UASD
0.897
0.897
0.897
0.000
0.000
CAFA
0.893
0.892
0.889
0.002
0.002
0.820
MTCF
0.880
0.904
0.855
0.016
0.015
-0.841
Fix-A-Step
0.869
0.876
0.856
0.007
0,011
-0.347
Table Evaluation on IMDB/Amazon dataset with 100 labels under inconsistent data distributions
Dataset
Model
AUC
Acc(O)
WA
EVM
VS
RCC
Supervised
0.571
0.571
0.571
Pseudo Label
0.634
0.545
0.545
0.084
0.092
0.296
Pi-Model
0.597
0.615
0.535
0.051
0.056
0.504
Mean Teacher
0.601
0.570
0.538
0.096
0.101
-0.012
UDA
0.599
0.523
0.523
0.071
0.080
0.484
IMDB/Amazon
FixMatch
0.530
0.540
0.500
0.027
0.031
-0.604
Match
0.545
0.502
0.502
0.064
0.071
0.169
FreeMatch
0.537
0.591
0.502
0.036
0.035
-0.615
SoftMatch
0.532
0.553
0.513
0.019
0.020
-0.347
UASD
0.593
0.541
0.541
0.046
0.055
0.580
5

---

Published as a conference paper at ICLR 2024
Table 5: Evaluation of SSL algorithms using letter dataset under inconsistent feature space
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.694
0.694
0.694
TSVM
0.683
0.721
0.635
0.017
0.005
-0.991
SSGMM
0.412
0.503
0.309
0.039
0.008
-0.996
Label Propagation
0.589
0.642
0.542
0.020
0.012
-0.978
Label Spreading
0.668
0.695
0.598
0.019
0.022
-0.857
Tri-Training
0.696
0.716
0.664
0.010
0.008
-0.945
Assemble
0.675
0.675
0.671
0.004
0.005
0.341
FT
T--nrossfrrerr
0.490
0.490
0.490
Pseudo Label
0.534
0.538
0.532
0.002
0.002
-0.401
Pi-Model
0.541
0.552
0.537
0.003
0.004
-0.761
Mean Teacher
0.517
0.517
0.517
0.000
0.000
VAT
0.541
0.561
0.535
0.007
0.008
0.720
ICT
0.540
0.540
0.540
0.000
0.000
UDA
0.537
0.537
0.537
0.000
0.000
FixMatch
0.499
0.548
0.470
0.022
0.035
-0.237
FlexMatch
0.435
0.470
0.406
0.020
0.029
-0.015
FreeMatch
0.447
0.409
0.409
0.014
0.008
0.977
SoftMatch
0.501
0.536
0.475
0.020
0.022
-0,231
UASD
0.552
0.553
0.549
0.001
0.002
-0.530
CAFA
0.511
0.511
0.510
0.001
0.001
-0.358
MTCF
0.415
0.278
0.278
0.042
0.034
0.932
Fix-A-Step
0.511
0.561
0.490
0.020
0.032
-0.311
4.2
SSL UNDER INCONSISTENT DATA DISTRIBUTIONS
We report the performance SSL algorithms on letter, Image-CIEE, and IMDA/Amazon datasets in T Table 3 and T respectively. Results on more datasets are reported in appendix A.5) For tabular datasets, we calculate the centroids of each class and use the distance between examples and class centroids to filter examples, thus constructing environment with inconsistent data distribution. For image and text datasets, we adopt the natural distribution shift to simulate the inconsistent distribution between labeled and unlabeled datasets.
4.3 SSL UNDER INCONSISTENT FEATURE SPACES
To simulate the inconsistent feature space, we randomly mask features for tabular data and each masked portion is filled with the mean value of the labeled data. For image datasets, we adopt the CIFAR-IO and CIFAR 100 datasets and convert the images to grayscale, resulting the loss two color channels. For text data, we adopt the agnews (Zhang et al.) 2015) dataset and employ text truncation. Truncated portions are filled with "< pad > simulate inconsistent feature spaces. The experimental results are reported in Table Table and Table 7
4.4 SSL UNDER INCONSISTENT LABEL SPACES
The inconsistent label space between labeled and unlabeled data the most widely studied problem in robust SSL. Following previous studies (Guo et al. 2020a Oliver et al. 2018), we construct inconsistent labeled space by randomly selecting some classes and discarding the labeled data belonging to these classes for both tabular, image, and text datasets. The experimental results are reported in T Table 9 and Table
4.5 EXPERIMENTAL RESULTS ANALYSIS
Based on the experimental results, we further conduct comprehensive analysis from the perspectives of environments, algorithms, and performance metrics.

---

Published as a conference paper at ICLR 2024
Table 6: Evaluation CIFARIO 10 dataset under inconsistent feature spaces
Dataset
Method
AUC
Acc(O)
WA
EVM
VS
RCC
Supervised
0,473
0.473
0.473
Pseudo Label
0.519
0.524
0.515
0.002
0.003
-0,874
Pi-Model
0.500
0.511
0.485
0.007
0.007
-08822
Mean Teacher
0.470
0.486
0.457
0.006
0.005
-0.962
VAT
0.501
0.550
0.466
0.020
0.018
-08880
ICT
0.468
0.476
0.456
0.005
0.005
-0.929
UDA
0.498
0,505
0.438
0.019
0.025
-0.707
CIFARIO
FixMathh
0,517
0.551
0,430
0.037
0.042
-0.661
FlexMatch
0.552
0.607
0.431
0.041
0.039
-0921
FreeMatch
0.555
0.645
0.423
0.045
0.029
-0.962
SoftMatch
0.559
0.661
0.453
0.042
0.009
-0.998
UASD
0.481
0.486
0.479
0.003
0.003
-0.625
CAFA
0,484
0.502
0,469
0.007
0.003
-0,888
MTCF
0.496
0.625
0.316
0.107
0.130
00.604
Fix-A-Step
0.516
0.551
0,424
0.025
0.032
-0.832
Table 7: Evaluation Agnews under inconsistent feature spaces
Dataset
Method
AUC
Acc(O)
WA
EVM
VS
RCC
Supervised 0.844
0.844
0.844
Pseudo Label
0.849
0.847
0.844
0.005
0.006
0.480
Pi-Model
0.865
0.870
0.859
0.003
0.003
-0.874
Mean Teacher
0,851
0,856
0.841
0.004
0.004
-0.890
UDA
0.844
0,862
0.802
0.022
0.029
-0.686
Agnews
FixMatch
0.870
0.880
0.858
0.005
0.005
-0.944
FlexMatch
0.848
0,877
0,810
0.021
0.021
-0,829
FreeMatch
0.876
0.872
0.868
0.008
0.009
-0.131
SoftMatch
0.875
0,880
0,865
0.005
0.005
-0.815
UASD
0.849
0.854
0.837
0.010
0.012
-0.007
Table 8: Evaluation of SSL algorithms using letter dataset under inconsistent label space
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.694
0,694
0.694
TSVM
0.683
0.721
0.635
0.017
0.005
-0.991
SSGMM
0.412
0.503
0.309
0.039
0.008
-0.996
Label Propagation
0.589
0.642
0.542
0.02
0.012
-0.978
Label Spreading
0.668
0.695
0.598
0.019
0.022
-0.857
Tri-Training
0.696
0.716
0.664
0.010
0.008
-0.945
Assemble
0.675
0.675
0.671
0.004
0.005
0.341
FT-
Trannssfformr
0.628
0.628
0.628
Pseudo Label
0.628
0.634
0.620
0.003
0.002
-0.970
Pi-Model
0.649
0.653
0.639
0.005
0.007
-0.673
Mean Teacher
0.635
0.635
0.635
0.000
0.000
VAT
0.640
0.656
0,622
0.007
0.004
-0.884
ICT
0.607
0.607
0.607
0.000
0.000
UDA
0.606
0.605
0.605
0.001
0.001
0.657
FixMatch
0.602
0.663
0.556
0.021
0.012
-0.983
Fatch
0.644
0.662
0.621
0.008
0.006
-0.975
FreeMatch
0.528
0.634
0.447
0.042
0.041
-0.937
SoftMatch
0,638
0.657
0.613
0.009
0.008
-0.946
UASD
0.638
0.640
0.628
0.005
0.007
0.138
CAFA
0.620
0.622
0.615
0.002
0.002
-0.918
MTCF
0.547
0.668
0.417
0.050
0.027
-0.984
Fix-A-Step
0.648
0.668
0.614
0.013
0.008
-0.966

---

Published as a conference paper at ICLR 2024
Table 9: Evaluation CIFARIO under inconsistent label spaces
Dataset
Method
AUC
Acc(O)
WA
EVM
VS
RCC
Supervised
0.643
0.643
0.643
Pseudo Label
0.692
0.708
0.676
0.006
0,004
-0.933
Pi-Model
0.672
0.703
0.654
0.01
0.009
-0.937
Mean Teacher
0.639
0.647
0.634
0.003
0.005
-0.333
VAT
0.697
0.734
0.661
0.015
0.011
-0.974
ICT
0.643
0.647
0.642
0.002
0.002
-0.819
UDA
0.676
0.73
0.594
0.027
0.015
-0,963
CIFARIO
FixMatch
0.608
0.705
0.479
0.047
0.036
-0.933
FlexMatch
0.731
0.806
0.614
0.038
0.02
-0.965
FreeMatch
0.733
0.815
0.640
0.035
0.012
-0.994
SoftMatch
0.723
0.806
0.601
0.041
0.021
-0.968
UASD
0.644
0.641
0.641
0.002
0.002
0.404
CAFA
0.675
0.674
0.672
0.005
0.006
0.093
MTCF
0.747
0.798
0.681
0.024
0.008
-0.989
Fix-A-Step
0.681
0.757
0.517
0.048
0.048
-0.908
Table 10: Evaluation on Agnews under inconsistent label spaces
Dataset
Method
AUC
Acc(()
WA
EVM
VS
RCC
Supervised
0.961
0.961
0.961
Pseudo Label
0.960
0.956
0.956
0.007
0.006
0.307
Pi-Model
0.962
0.968
0.950
0.006
0.006
-0.785
Mcan Teacher
0.965
0.964
0.961
0.004
0.004
0.261
UDA 0.956
0,965
0.938
0.010
0.009
-0,816
Agnews
FixMatch
0.957
0.974
0.927
0.012
0.009
-0.902
FlexMatch
0.937
0.973
0.889
0.011
0.017
-0.975
FreeMatch
0.936
0.972
0.811
0.036
0.056
-0.752
SoftMatch
0.961
0.974
0.939
0.012
0.012
-0.862
UASD
0.954
0.948
0.944
0.013
0.014
0.112
Environments. We calculate the average expected robustness (under the uniform distribution of Pr) and the average worst-case robustness of SSL algorithms under different inconsistency settings. The results are reported in TT According to the definition, lower values of de and dw imply stronger robustness. The results show that the robustness of SSL algorithms is much lower in cases where there is inconsistency between the feature space the label space, compared to cases when there is inconsistency in data distribution. Actually, inconsistencies between the feature and label spaces can both be considered as a greater degree of inconsistency in data distribution. The former be viewed while the latter can be seen as a distribution shift where the probability of all missing classes for examples O. These tell us that more attention needs be paid to feature and label inconsistency between labeled and unlabeled data.
distribution shift where all missing features assumed take default values,
Algorithms. We compare the robustness of different algorithms in various environments and report the results in Table 12 We found that SSGMM (Shahshahani & Landgrebe 1994) shows the poorest robustness among all algorithms, the main reason relies on the assumption of data distribution. For other statistical SSL algorithms, Assemble (Bennett et al. 2002) demonstrates the best performance and remarkable robustness, showcasing the advantage of using ensemble learning. For deep SSL algorithms, we find that the reported SOTA methods such as FixMatch, FlexMatch, SoftMatch, and FreeMatch, suffer severe robustness problems. One possible reason is that these methods adopt threshold to select pscudo-labels for unlabeled data, which might overly centralize the distribution of unlabeled data. In comparison, UDA (Xie et al.) 2020) sets thresholds for both labeled and unlabeled data, mitigating the inconsistency induced by sample selection
large extent and significantly improving the robustness over FixMatch. For robust deep SSL algorithms, we find that UASD and CAFA achieve good robustness, but for other methods, they achieve lower robustness compared with
8

---

Published as a conference paper at ICLR 2024
Table 11: Average Robustness of SSL Algorithms in Different Environments
Environments
Excepted Robustness (ōg)
Worst-case Robustness (ōw)
Inconsistent Data Distributions
0.015
0.028
Inconsistent Feature Spaces
0.019
0,039
Inconsistent Label Spaces
0.020
0.044
Table 12: Average Robustness of SSL Algorithms
Algorithms SSGMM
Excepted Robustness (ØE)
Worst-case Robustness (ow)
0.062
0.120
TSVM
0.017
0.040
Label Propagation
0.030
0.053
Label Spreading
0.021
0.045
Tri- Training
0,024
0.041
Assemble
0.009
0.017
Pseudo Label
0,008
0.014
Pi-Model
0,012
0.021
Mean Teacher
0,014
0.027
VAT
0.034
0.065
ICT
0.009
0.022
UDA
0.020
0.066
FixMatch
0.065
164
FlexMatch
0,055
0.143
FreeMatch
0.066
0.157
SoftMatch
0.067
0.154
UASD
0.003
0.002
CAFA
0,010
0.022
MTCF
0.077
0.118
Fix-A-Step
0.062
0.197
ordinary SSL algorithms. Therefore, when designing robust SSL algorithm, need consider more comprehensive environments and evaluation metrics.
Performance Metrics. We also analyze the performance of different SSL algorithms under different metrics. First, we find that Acc(O) is not consistent with other metrics, SSL algorithms that have high Acc(O) may perform even worse than supervised learning under the proposed robustness metrics. Second, we find that the EVM and VS metrics exhibit a high level of consistency, despite their different definitions. This indicates that for robust SSL algorithm, its performance less sensitive to changes in inconsistency level, and the direction of performance change is more stable and predictable. On the other hand, for a non-robust SSL algorithm, not only does it exhibit larger variations in performance, but the performance changes are also more unstable, showing greater randomness. Using such an algorithm in an open environment extremely unsafe, as we cannot estimate its worst-case performance. Moreover, we find that the non-identically distributed unlabeled data not always harmful, in some cases, exploiting more unlabeled data from inconsistent distributions may improve the performance. This inspires to study SSL algorithms that fully exploit helpful information from inconsistent unlabeled data.
5 CONCLUSION
The research robust SSL is an essential step toward the practical application of SSL. This paper provides a reshaped perspective on problem definition, performance metrics, theoretical frameworks, and evaluation of benchmark datasets. Our results provide evidence that SSL is still not robust in open environments, especially when the feature and labeled space are inconsistent between labeled and unlabeled data. These problems often overlooked in previous studies and more efforts need to be devoted. The subsequent details about this work will be continuously supplemented and improved. We hope that our work can help push the successes of SSL towards the real world.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This research was supported by he National Science Foundation of China (62176118, 18, 62306133).
REFERENCES
Kristin Bennett, Ayhan Demiriz, and Richard Maclin. Exploiting unlabeled data in ensemble methods. In Proceedings of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 289-296, 2002.
Barbara Caputo, Henning Müller, Jesus Martinez-Gomez, Mauricio Villegas, Burak Acar, Novi Patricia, Neda Marvasti, Suzan Üsküdarh, Roberto Paredes, and Miguel Cazorla. Imageclef 2014: Overview and analysis of the results. In Proceedings of 5th International Conference of the Cross-Language Evaluation Forum for European Languages, pp. 192-211, 2014.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-Supervised Learning. The MIT Press, 2006.
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-qullity trade-off in semi-supervised learning. In Proceedings of the 12th International Conference on Learning Representations, 2023.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 785-794, 2016.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong, Semi-supervised learning under class distribution mismatch. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, pp. 3569-3576, 2020.
Lan-Zhe Guo and Yu-Feng Li. A general formulation for safely exploiting weakly supervised data. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp. 3126-3133, 2018.
Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. 2022. In Proceedings of the 39th International Conference on Machine Learning, pp.
Lan-Zhe Guo, Zhen- Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In Proceedings of the 37th International Conference on Machine Learning, pp. 3897-3906, 2020a.
Lan-Zhe Guo, Zhi Zhou, and Yu-Feng Li. Record: Resource constrained semi-supervised learning under distribution shift. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1636-1644, 2020b.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Zhe Huang, Mary-Joy Sidhom, Benjamin Wessler, and Michael C Hughes. Fix-a-step: Semi- supervised learning from uncurated unlabeled data. International Conference on Artificial Intelligence and Statistiics, pp. 8373 0394, 2023.
Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal sumi-supervised learning. In Advances in Neural Information Processing Systems, pp. 267 26714-26725, 2021.
Lin-Han Jia, Lae Guo, Zhi Zhou, Shao Jiejing, Yu Ke-Xiang, and Yu-Feng Li. Bidirectional adaptation for robust cemi-supervised learning with inconsistent data distributions. In Proceedings of the 40th International Conference on Machine Learning, 2023a.
Lin-Han Jia, Lan-Zhe Guo, Zhi Zhou, and Yu-Feng Li. Lamda-ssl: Semi-supervised learning in python. Science China Information Sciences, 2023b.
Thorsten Joachims et al. Transductive inference for text classification using support vector machines. In Proceedings of the 16th International Conference Machine Learning, pp. 200-209, 1999.
10

---

Published as a conference paper at ICLR 2024
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Proceedings of the 5th International Conference on Learning Representations, 2017.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of the 30th International Conference on Machine Learning Workshop on Challenges in Representation Learning, pp. 896, 2013.
Yu-Feng Li, Lan-Zhe Guo, and Zhi-Hua Zhou. Towards safe weakly supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):334-346, 2021.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: / 1907. 1692, 2019.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pp. 142-150, 2011.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference Recommender systems, pp. 165-172, 2013.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi cemi-supervised learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 11(8):1979-1993, 2018.
Balas Natarajan. On learning sets and functions. Machine Learning, 4(1):67-97, 1989.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. Advances in Neural Information Processing Systems, pp. 3239-3250, 2018.
Kuniaki Saito, Donghyun Kim. and Kate Saenko. Openmatch: Open-set supe-vipe learning with open-set consistency regularization. Advances in Neural Information Processing Systems, pp. 25956-25967, 2021.
Behzad M Shahshahani and David A Landgrebe. The effect of unlabeled samples in reducing the small sample size problem and mitigating the hughes phenomenon. IEEE Transactions on Geoscience and Remote Sensing, 32(5):1087-1095. 1994.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi suni-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, pp. 596-608, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, pp. 1195-1204, 2017.
VN Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability Its Applications, 16(2):264-280, 1971.
Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Amno Solin, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. Neural Networks, 145: 90-106, 2022.
Feiyu Wang, Qin Wang, Wen Li, Dong Xu, and Lue Van Gool. Revisiting deep semi-supervised learning: An empirical distribution alignment framework and its generalization bound. arXiv preprint arXiv:2203.0660 06639, 2022a.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides. Takahiro Shinozaki, Bhiksha Raj, et al. Freematch: Self-adaptive thresholding for semi-supervised learning. Proceedings of the 11th International Conference on Learning Representations, 2022b.
11

---

Published as a conference paper at ICLR 2024
Tong Wei, Qian- Yu Liu, Jiang- Xin Shi, Wei-We Tu, and Lan-Zhe Guo. Transfer and share: semi- supervised learning from long-taills data. Machine Learning, pp. 1-18, 2022.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, pp. 6256-6268, 2020.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set supi-super learning. Proceedings of the 16th European Conference on Computer Vision, pp. 438-454, 2020.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems, pp. 14408-18419, 2021.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 2015.
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances Neural Information Processing Systems, pp. 321-328, 2003.
Zhi Zhou, Lan-Zhe Guo, Zhanzhan Cheng. Yu-Feng Li, and Shiliang Pu. Step: Out-of-distributioo detection the presence of limited in-distribution labeled data. Advances in Neural Information Processing Systems, pp. 29168-29180, 2021.
Zhi-Hua Zhou, Open- environeit machine learning. National Science Review, 08))nwac1223 2022.
Zhi-Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions Knowledge and Data Engineering, 17(11):159-1141, 2005.
Xiaojin Zhu and Zoubin Ghahramani. Semi-supervised learning using gaussian fields and harmonic functions. Proceedings of the 20th International Conference on Machine Learning, pp. 912-919, 2003.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 LIMITATIONS
Although this paper has modeled and constructed complex open environments, it should be noted that the complexity of the real world may exceed the dataset we have constructed, and it may be difficult evaluate, analyze, and explain with limited evaluation metrics and theoretical frameworks. For example, when multiple inconsistencies coexist and their degrees vary simultaneously. high- dimensional vector i rather than a one-dimensional variable & is required represent the combination of multiple inconsistencies. While the evaluation methods and metrics we employed can be readily extended to sigh-dimensional cases, this leads to an exponential increase in computational resource consumption, specifically the order of O (di((t)), ), where s represents the number of examples taken for each dimension. Finding ways to reduce the evaluation complexity in scenarios involving high-dimensional inconsistencies is an urgent and unresolved issue.
A.2 THEORETICAL FRAMEWORK ON ROBUST SSL
Natarajan dimension Natarajan 1989) is an extension of Vapnik-Chervone dimension Vapnik & problems. We denote Ndimm(H) the Natarajan dimension of hypothesis space H. To simplify the expression, we denote the variance term associated with the hypothesis space complexity in the generalization error with the number of examples n, the number of classes k. and the probability 5:
Chervonenkis (1971) in multi-
16. In 22k + 8ln 3
var par(H, n, n,k,n
(2)
n
In an open environment, we assume that all data originates from a global space X x ) S Rdd x fO....,.* k - 13. There exists an invariant data distribution P(y"|z*) for x- E X and y" y². For any t, there is an inconsistent distribution denoted P( y.). According to the total probability theorem, Px(x") g. (p. (a for all x²' E X
However, we can only obtain a projected distribution Pa(x,, Me) in subspace X₁ x 34, from the global distribution Pr(x", ,y") ) in the global space X x y"* where X; C X and V C y" We denote X₂ = x²/ /22 as the unobserved features and Dr. = /D/ as the unobserved classes when the inconsistency rate is t. In this case, the observed inconsistent data follows the distribution Pz( Eg (PP.
where x Xx S Rdd and E & S Rd according to Vy, E Vr. P.(ue) 0 and Px(x*)= Pt( Pt (.)l).)
lx))
In SSL,, labeled data are used to train a pseudo-label predictor h E H X, - Vo where H is the hypothesis space of pseudo label predictor to obtain unlabeled dataset with pseudo-labels, denoted as Du = {(X,, of ), (x vi')
).!),,.. (X", Dh }}. There is also function w No - R used for sample weighting or selection. We denote the weighted unlabeled dataset without pseudo-labels as Dy w( w(De) Dy) and the weighted unlabeled dataset with noisy pseudo-labels as Dy w (Dy). We denote the sum of weights of all unlabeled examples n. = ((.w)e w(z). We additionally denote the proportion of inconsistent examples in the unlabeled dataset after sample weighting as Σn (1 (0-0)))) +1 m0(2).)
p.,
8t (t)
We define weighted distribution as the inner product of distribution function and a weighting function, such as (*(x)) = w(x) and p"(x,,) = w(x) y).
th w(x.)
SSL algorithms use both Dr and DY for training target predictor f E F where F is the hypothesis space of the target predictor. Due to different feature spaces, feature mapping functions mapx, X, + Xy is also required to map the input into the domain of definition of the model.
For the distribution (x(",,, ), we define its projection onto any target label space )" as Hy [Px")) EnEN,, I(N E P. (((x)) The joint distribution after projecting onto the label space )" is 1y [P(x" .y) = (E I(y E (M) (v:)) P x))
For the distribution P)) define its projection onto any feature space X' as Ix (P(()) P. (mmapx--px -xx' (x=xxxxx))))) (x*)), ~ Px(x)). The joint distribution after projecting onto the feature space X² is xx (P(""", (v)",)) .w) = P.maaxx- x (x"), (x* y) P.(x", ,y).
13

---

Published as a conference paper at ICLR 2024
We define the discrepancy between the distribution (z(x)) and the label space y" as:
Disc)( =11 (m(p)((r)) 1ly [Ptx [dd""
For the data distribution Pt(x=) and the observed data (c(),, we define their discrepancy on an arbitrary feature space X' with respect to the feature mapping function mapx,-x" and the model function defined on the domain of X' as
Discy (Pe(z), mapx = |P( w)) (f +
-Pie.)eEP(t..)) (1 (mapxxxxx vy)l
For two data distributions (z, y) and (a y) defined on the same feature space and label space, their distributional difference to the model function f can be defined by the following discrepancy:
Disco (z. v), P" (z. v). f) (Pa.,)--"aa. (f(x) # ((..))-"(a. (f(x) v)1 (5)
As a result, we can obtain the error rate of pseudo-labeling in the weighted or filtered unlabeled dataset obtained through robust SSL.
Theorem A.1. For any pseudo-label predictor h E H. 0 s 81 S 1 and 0 s 82 S 1, with the probability of at least (1 ō1)( -8):
Fhh, w. map,, -oo Du.)
< <h, DL) + mar(H, nı, ko, 81) + var (H, nw ho, 52) 0" (P") +0" Discr (Ily, ! (z)], Hy [P (x*) )} mapx.
(t) DiscD(II, [IT,, [P. (x", .V)l. Po(z. h)
(6)
where Eh,, DL) is the empirical error of h Dr, and B(,, w. mapx Dv.) is the empirical error of h on Du with ground truth labels.
Based on the above label noise rate bound of the unlabeled dataset, we can estimate the generalization error bound of the robust SSL algorithm trained with the labeled dataset and this unlabeled dataset.
Theorem A.2. For any target predictor f E F, pseudo-label predictor h E H. 0 < 6, S 1.
0 82 and < da 1, with the probability of least (1 61)(1 62) 63):
E(f, Po(x, h,, w. mapx.-X6 DL. Du.)
n DL)
+
B(J Di.)) ) + var(F, (ar(F,mm my n ko: 81)
n4 n.. (isc ), Tu + -
Th + n
n"
nt (I)DiscF(lls [PT (z)), Hy, mapp
mapx:
n (iDe (TTx 1y P. (x" y)]l Po(x,
n (F(h, Dr.) var(H n4 k 52) + var(H, n ko, 83)
n4 -
8d (t)Dissz(PP" (x") (a"), Vo) 8d (t) Discr (Ty; (P)())), Tyy (x"))))), mapx,-xxo h) - Dtsssis) [1, (x" vill. P)(
(7)
where ECS, Dy. ) is the weighted disagreement rate between the noisy pseudo-labels and the predic- tion results of f on the weighted unlabeled dataset Dy.
A.3 FORMULA DERIVATION AND THEORETICAL PROOF
A.3.1 DERIVATION EVALUATION METRICS
Since the formulas for the metrics AUC, EA, WA, and EVM can be directly obtained through definitions, primarily focus deriving the formulas for the metrics VS and RCC.
14

---

Published as a conference paper at ICLR 2024
1. VS: According to the definition, the metric VS is used to measure the stability of model per- formance changes, that is whether the derivative of Acc(t), Acd' (t), fluctuates significantly. We describe the magnitude of fluctuations using the variance, and thus, VS is defined the variance of Acd' (1). We denote the expectation of the variable x as ((x)) and the standard deviation of the variable (X))
VS(Acc)
0² (Ace')
/ (1o E(Ace)]
[Ace((]) - -1/0 Acd (
2. RCC: According to the definition, the metric RCC used to measure the correlation between model performance and the inconsistency factor t. The Pearson correlation coefficient effectively quantifies the correlation between two variables. We denote the covariance between the variables and Y as COV(x) (x:) and the Pearson correlation coefficient between the variables x and Y as p(X,
P(X,Y)
COV(X,Y)
(X))((
E(XY)- E(X)E(Y)
E E
We can directly apply the formula for the Pearson correlation coefficient.
RCC(Acc) =p(Acc, 1)
E(Ace- 1) E(A
E((co E²(Acc) / E(12) (t(
So' Acc(t) tdtt - So Acc(t) So tdt
Jo (tcd(t)dt Uo Acc(t)dt)2 vib tt - (So tdt)2
Jo Acc(t) tdt So Acc(t)dt
(10)
So (cc (To Acc(t)dt)2 v tddt
A.3.2 PROOF OF THEORETICAL RESULTS
Proof of Theorem
In the case of using only ny labeled examples for supervised learning, for any h H and 0 S 51 with the probability of at least 81:
Fihh, Po(x. B(, D1.) var(H, n4 k. 51)
(11)
where E(h, Dr) is the empirical error of h the dataset Dr and E(f, Po(æ, a)) is the generalization error of f on the distribution labeled data Po(æ, y).
In SSL, when all examples are from the same distribution, for any i, dataset Du, with n. examples are from the same distribution P((x, y) = Po(x, y). For any E H and 0 S 82 S 1, with the probability of at least dz:
E(h, Db) < E(h, P((x, y)) + var (H, nuu ks 52) E(h, Po(x. v)) var(H, Nou k 52)
(12)
15

---

Published as a conference paper at ICLR 2024
According to eqs. 11 and (12), for any pseudo-label predictor h E H, - 61 - and 0 S d2 S 1,
with the probability of at least (1 51)(1 62):
Bh,, Dv.) ) S Eh, DL) + var(H, ny, k. 61) + var(H, tue k, 82)
(13)
When labeled data and unlabeled data are from different distributions, for any h E H:
Ehh, (((
<E(h, Po(z, v)) Po (b(x) + Pa + y)l -b,b, Po( v)) Disc(h, Po(x, v), Pr v))
(14)
According eqs. I1
12) and (4), for any pseudo-label predictor h H. S 81 S and
0 s 62 $ 1, with the probability of at least (1 61) - -2)):
E(h, Pz(x, v))
Po(z,, v)) DiscD(Po(x:) P((z.) v).
<bth, DL) + var(H. ny, k, 61) + var var(H, nus k 82) + Discp(Po(z, v), P((z, y)
(15)
Taking into account that SSL,, a weighting function w is often used to either weigh or filter unlabeled examples, it's the weighted unlabeled data that truly plays role in the learning process.
[h,, Du.; w) <E(h, Po(x, v)) DiscD(Pu(as) v). w(P v)), h))
<h(h, DL) + var(H, m,, k, 51) + (H, n k 82) + Discp (Polz, v), Pt" (z, v)
(16)
Now considering that labeled and unlabeled data are not only from inconsistent data distributions but also inconsistent data spaces, we need an extra feature mapping function to complete the features and an extra weighting function to filter out examples from new classes. Both the mapping function and the weighting function aim to project unlabeled data to the same space as labeled data.
According eqs. 3 and (16), for any pseudo-label predictor h H. 0 S 8, 1 S 82 S 1, with the probability of at least (1 81) (1)):
(h, w. mapx Du.)
< Ebt,, Dr) + var nı, ko, 61) + var var(H, n ko 52) + 0 (P" +0 (t) (1y,, ( (z)], 11,, [P (x")), mapxx
Discp v), IIx [ (x" .2)],,
(17)
where E(h, DL) is the empirical eπror of h on Dr and E(h, w. mapx Du,) is the empirical error of h on Dv, with ground truth labels.
Proof of Theorem 2
We denote the mixture of two distributions DD and D2 with proportion a as:
Mix.(Di D2) aD (1 a)
(18)
In SSL with inconsistent distributions, the target predictor is trained with both labeled dataset Dr. and weighted unlabeled dataset with noisy pseudo-labels Du,- D2 and Dy, be considered mixed dataset with ny + Tu. examples from the mixed distribution M (Po(I,)) 3). (x y))
whose noisy rate is mitt É(h, Due)
So, for any target predictor f E F, pseudo-label predictor h E H. 0 S 83 S 1, with the probability of at least dz:
E(f, Mir
(Po(I, y), Pe(x, 4)) DL, Du)
mitwung
Th
B(J.Dr) +
Ra. BC Dv) + var F, + nuyes hi 63)
Bue E(b, Dv
T11 Thus
T1 nu.
(19)
16

---

Published as a conference paper at ICLR 2024
(Po v). Pi(z, v)) Dr, Du) is the generalization enror of f on the dis- where Eff, Mix my
ortrug (Po u), Pt(x, v) corresponding pseudo-label predictor h.
tribution Mix "t my-M
When labeled data and unlabeled data are from different distributions, for any f E F:
E(S, Po(a, w) Dr, DU.)
<F(S Mir m (Po(o.)). (r U))l,, Dr, Dv.) mitnue
-P (h(x) # y) Pin me (Pola,)) + v)l
=E(f,Mis (Po v). P v) DL,
+DiseD(f, Pola, v), Mia my (Po v) (r
(20)
According to eqs. (6). 199 and 20 for any target predictor f E F, pseudo-label predictor E H. 0 . 61 1,0 82 and 0g 1, with the probability of at least (1 51)(1 52)( 63):
E(S, Po(x, u)l,, DL, Du.)
<F(S, Mix mi-
(Po(ar,)) v), Pa(a, Dr, Du;)
Discc Po(x, v). Mir
(Po(x, v), Pc(x.
m
"rtrut
< n11 + Thus
11 DL). D1. +
Thu
ECs, Du.) + (Far(F ni + Tuys ki 65)
n1 + Tar
Thet
(Po v), (,(2,,))) B(h, Du.) + (Jscc([, Po(x, u), Mix Tn metmue
n1 + Duss
T11
Ë(f, D1)+ + ni + nur
The
Do.) + var(F, mi + nu. 8))
T4 + nuu
+DiscD(f, Po(r. (z. v), Mir P1 wrtnue
(k(h, Dr.) (Hr ny, 82) (Hr,H, nue k. 6g) + (((x,) v))) The
14 + nuu
(21)
where E(S, Dv.) is the weighted empirical inconsistency rate between the noisy pseudo-labels and the prediction results of f on the unlabeled dataset Dv.-
Taking into account inconsistent label spaces and weighting function w:
E(S, Po(x, h)(h, DL, Du,, w) n.
E(J. D1) + nh
B(f, DV,) + var(F, m + n 81)
ni + n
14 + n
+Diccp Po v), Míx
(Po(æ, u), P (x (x, v)))
nu
(É(h, DL) - var ny, k 82) + var(H, nic k. 8z)
+ nw.
Discn(h, Po(a, v). (x,,)))
(22)
Taking into account inconsistent feature spaces and mapping function mapxx-x6, the final error bound can be obtained.
According to eqs. H. < 81 52) 8z): F[S, Pa(a, 1h, w. mapxx+26 Dr. Dl)
21
and
22
for any target predictor f e F, pseudo-label predictor
02
and S 8a < with the probability of at least (1 - 61)
17

---

Published as a conference paper at ICLR 2024
nt D(). + T4 + nu.
B(f. Di,) + var(F, , nw: ko. 61)
ny + nu
(0 (I) (P (x"(((())))))))))))""))) (,))),)))))))) 0r (i)crrr P mapxx
ny + nu,
S0 Disen (11x, [1y [ (a", Po f).
(R(h, DL) var(H. n. k 62) var(H, ni Ko, 83)
T4 + nu,
+0"h Diser (x') (x"), Vo) + (1) ()) (1Dis (x)(,,) Hy (P"((x)) [],"], mapx,
+0" Disep x [1y [P"
(23)
where E(f. Di, is the weighted disagreement rate between the noisy pseudo-labels and the predic- tion results of f on the weighted unlabeled dataset D
A.4 SSL ALGORITHMM EVALUATED THE BENCHMARK
A.4.1 STATISTICAL SSL ALGORITHMS
1. SSGMM (Shahshahani & Landgrebe 1994) assumes that data is generated by a Gaussian mixture model, that is, the marginal distribution of examples can be expressed as the result mixing several Gaussian distributions, and each distribution given a weight.
2. TSVM (Joachims et al. 1999) infers labels of unlabeled examples and finds dividing hyperplanes that maximize the distance from support vectors. It continuously finds pairs of unlabeled heterogeneous examples and exchanges their labels until more pairs can be found.
3. Label Propagation (Zhu Ghahramani 2003) uses examples as nodes, and the relationship between the examples as edges. The purpose of the Label Propagation algorithm is to propagate labels from labeled data to unlabeled data through the graph.
4. Label Spreading (Zhou al. 2003) penalizes misclassified labeled examples rather than banning misclassification completely which is different from Label Propagation fixing labels of labeled examples during the spreading process.
5. Tri- Training (Zhou & Li 2005) is a representative three learners with divergence and makes divergence by data sampling. The disagreement between learners is utilized for optimizing interactively.
nanggrrement-basee SSL algorithm. It uses
6. Assemble [Bennett et al. 2002) extents AdaBoost to the field SSL by giving pseudo- labels unlabeled data. It pays more attention to the examples with poor learning effects of the current ensemble learner in each round and continuously improves the robustness using new base learners.
A.4.2 DEEP SSL ALGORITHMS
1. Pseudo Label Leee 2013) takes the label with the highest confidence the pseudo-label and uses cross-entropy obtained from the pseudo-label as the unsupervised loss.
2. Pi-Model (Laine & Aila, 2017) augments the data twice randomly and uses the results of the two augmentations as inputs of the neural network respectively. The inconsistency of the prediction results is used as the unsupervised loss.
3. Mean Teacher (Tarvainen & Valpola 2017) relies the idea of knowledge distillation, where the prediction results of the teacher model are used as pseudo-labels to train the student model to ensure the consistency of the prediction results. It uses EMA for the student model's parameters as the teacher model.
4. VAT (Miyato al. 2018) adds adversarial noise rather than random noise the data so that the worst performance of the model can be better when the data is affected by noise within a certain range, which corresponds to the zero-sum game game theory and Min-Max problem in optimization.
5. ICT (Verma et al. 2022) linearly interpolates data and prediction results by Mixup. The unsupervised loss is obtained by the interpolation consistency.
18

---

Published as a conference paper at ICLR 2024
6. UDA (Xie et al. 2020) performs data augmentation the unlabeled examples and then compares the prediction results before and after the augmentation. The thresholds are used for sample selection for both labeled and unlabeled data respectively.
7. FixMatch (Sohn et al. 2020) uses both strong and weak data augmentation and the inconsis- tency of prediction results between them used as the unsupervised loss. fixed threshold is used for sample selection.
8. FlexMatch (Zhang et al. 2021) uses a dynamic threshold based on FixMatch. It sets a lower confidence threshold for the classes that are more difficult to learn.
9. FreeMatch (Wang et al. 2022b) employs a more precise dynamic threshold, where the threshold setting takes into account both the model's training phase and the disparities be- tween categories. It also incorporates a regularization term to facilitate equitable predictions between categories.
10. SoftMatch (Chen et al., 2023) no longer adheres to the paradigm of filtering examples through confidence threshold, and instead replaces sample selection with sample weighting. The sample weights are utilized to achieve better balance between the quantity and quality of pscudo-labeled data.
A.4.3 ROBUST DEEP SSL ALGORITIMMS
2020) ensembles model predictions to produce probability predictions for unlabeled examples, es, and uses threshold based on confidence to filter out OOD examples.
UASD (Chen et al.
2. CAFA (Huang et al.) 202I) takes into account both the inconsistency in labeling spaces and data distributions. It employs scoring mechanism filter out examples from new classes and then utilizes unsupervised domain adaptation to alleviate distribution inconsistency, thus obtaining higher-quality pseudo-labels.
3. MTCF Yu et al.) (020) leverages the concept of curriculum learning. It uses a joint optimiza- tion framework, which updates the network parameters and the OOD score alternately to detect the OOD examples and achieve high performance on the classification simultaneously.
4. Fix-A-Step (Huang et al. 2023) views all OOD unlabeled examples as potentially helpful. modifics gradient descent updates to prevent optimizing a multi-task SSL loss from hurting labeled-set accuracy.
A.5 EXPERIMENTS
A.5.1 DATASETS PREPARATION
Inconsistent Data Distribution
1. Wine, Iris, Letter: To construct datasets with inconsistent distributions, in each class, we calculate the center of all examples and sort these examples according to the distance between them and the center in ascending order. The first n. * 0.5 examples are used as labeled data which can be which can be regarded as being obtained by sampling from Po(x, v). and the rest of ni * 0.5 examples are used as inconsistent unlabeled data. For each which can be regarded as being obtained by sampling from (z).. (t) = 1 for every t. 5 examples per class from source domain data used as labeled data and the rest are used test data.
the n. 0.5 * (t (t- !)) to n. 0.5 + examples
used as labeled data which
2. Image-CLEF: The Image-CLEF dataset consists of 3 domains, which can be combined into 6 source-domain target-domain pairs. All source domain data can be regarded as being obtained by sampling from P Psource (2. y) and all target domain data can be regarded as being obtained by sampling from Ptarget (x., y/- We set Pa y) = Psourcee(:; y) and Pr v) = P Ptargee (xr, y) for all t F O. From the source-domain data, 100 examples are taken as labeled data. Half of the remaining source-domain examples are used as test data, while the other half is combined with the target-domain data to form an unlabeled dataset. The total number of unlabeled data n. is min (n. 100). n2) where n, is the number of examples in the source domain and n, is the number of examples in the target domain. 0(t) = for every t. For inconsistency rate t, the unlabeled dataset is combined with nu (1 t) examples for the source domain and ny examples from the target domain.
19

---

Published as a conference paper at ICLR 2024
The IMDB and Amazon datasets can be considered as source and target 3. domains respectively. All source domain data can be regarded as being obtained by sampling from (a, y) and all target domain data can be regarded as being obtained by sampling from Prorget(2; y). We set Po(x, y) = Psource (2. y) and (r. y) Ptarget (1. y) for all t o O. From the source-domain data, 100 examples are taken labeled data. Half of the remaining source-domain examples are used as test data, while the other half is combined with the target-domain data form an unlabeled dataset. The total number of unlabeled data Th is mén(0.5 * (n, 100). n²) where n. is the number of examples in the source domain and ny the number of examples in the target domain. {(t) = for every t. For inconsistency rate I, the unlabeled dataset is combined with nh (1 1) examples for the source domain and ny * examples from the target domain.
Inconsistent Feature Space
1. Wine, Iris, Letter: 50% of all examples can be used as source domain data, and the rest used as target domain data. 5 examples per class of source domain data are used as labeled data which can be regarded as being obtained by sampling from (Z, y), and the rest are used as test data. For every t, All target domain data randomly dropping ( x d features are used as unlabeled data which can be regarded as being obtained by sampling from Pz y). 0(t) = 1 for every t.
2. CIFARIO, CIFAR100: 50% of all examples can be used as source domain data which can be regarded as being obtained by sampling from Psourea(C (I, y), and the rest are used as target domain data. 20 examples per class of source domain data are used as labeled data. All target domain data are transformed to grey images by dropping 2 channels which can be regarded being obtained by sampling from P P'trger (z, y). We set Po y) = Psource (Ir, y) and P,(x, y) = (targee(x,, y) for all + O. O(t) = t for every t. For inconsistency rate t, the unlabeled dataset is combined with n. (1 t) examples for the source domain and n. examples from the target domain.
3. Agnews:
50% of all examples can be used as source domain data which can be regarded as being obtained by sampling from Psource (3r, y). and the rest are used target domain data. 100 examples of source domain are used as labeled data and the rest are used as test data. 50% target domain sentences are used as IID examples and the other 50% target domain sentences that drop 50% tokens are used as OOD examples which can be regarded as being obtained by sampling from Pearget (a, y)- We set Po(x. y) = Psource (z. y) and Py (2x y) = Ptarget (I, y) for all A O. The number of unlabeled data n. is set to min (nz /
t). no /t where and nD are the numbers of IID and OOD examples respectively. The unlabeled dataset is
combined with nu, (1 1) IID and n,, / OOD examples. ((t) = / for every
Inconsistent Label Space
1. Wine, Iris, Letter: 50% of all examples can be used as source domain data, and the rest are used as target domain data. (k+1)///2 classes of source data are saved and the rest examples are dropped which can be regarded being obtained by sampling from Psource (1:, y). 5 examples per class of saved source domain data are used as labeled data and the rest are used as test data. The target domain examples with saved classes are used as OOD examples which can be regarded as being obtained by sampling from Prarget (I. y). and the target examples with dropped classes are used as IID examples. The number of unlabeled data nu is set to min(ni/ (11 - t), nt/ft) where n. and nD are the numbers of IID and OOD examples respectively. The unlabeled dataset combined with n, - t) IID and n. t OOD examples. We set Po v) = (source(I, y) and (z Pearges (z. y) for all t + O. 0(t) = t for every t.
2. CIFARIO, CIFARI00: (A: + 1)/2 classes of all examples are used as source domain data, and the rest are used target domain data. 20 examples per class of the source domain are used as labeled data. For inconsistency rate f, the unlabeled dataset is combined with n, (1 - t) examples for the source domain and n, examples from the target domain where n, is the number of target domain examples. We set Po y) = P Psource (2. y) and (s y) Pearget (3. y) for all I. + O. 0(t) for every t.
20

---

Published as a conference paper at ICLR 2024
3. Agnews: (A- - 1)/2 classes of all examples are used as source domain data and the rest used as target domain data. 100 examples the source domain are used as labeled data. For inconsistency rate t, the unlabeled dataset is combined with n, (1 t) examples for the source domain and ny * I examples from the target domain where n4 is the number of target domain examples. We set (2 y) P Psource v) and (1 (1,, y) Paarget (I, y) for all * O. ((t) = t for every t.
A.5.2 HYPER- PARAMETERS OF COMPARED ALGORITHMS
Baselines
1. XGBoost: the parameter of _as__abel_encodee is set to False, the parameter metric is set "logloss".
2. PT-Traasformmrs: the number of layers is set to 8, the dimension of tokens is set to 192, and the number of heads is set to
3. ResNet50: the Resnet50 pre-trained ImageNet from torchvision.modess is directly used.
4. Roberta: the pre-trained model "roberta-base" from transformers package is directly used.
Statistical SSL Algorithms
1. SSGMM: the number of iterations is set to 300.
2. TSVM: the parameter C is set to 15, the parameter C, is set 0.0001, and the method to deal with multi-classification tasks is set to "one rest".
3. Label Propagation: the hyperparameters provided by scikit-learn in default are used.
4. Label Spreading: the hyperparameters provided by scikit-learn in default are used.
5. Tri-Training: all the base learners set to XGBoost classifier consistent with the baseline.
6. Assemble: the number of iterations T is set to 30, and all the base learners are set to XGBoost classifier consistent with the baseline.
Deep SSL Algorithms
1. Pseudo Label: the ratio of unsupervised loss A is set 1.0, and the threshold is set 0.95.
2. Pi-Model: the ratio of unsupervised loss Au is set to 1.0, the warmup rate of unsupervised
loss wP set to 0.4, and the ratio unsupervised loss X, is set to max : 1.0) where is current iteration and T is the number of iterations.
3. Mean Teacher: the EMA decay is set to 0.999, the warmup rate of unsupervised loss w. is set to 0.4, and the ratio of unsupervised loss X is set to max( t. 1.0) where f. is current iteration and 7' is the number of iterations.
4. VAT: the ratio of unsupervised loss A, is set 0.3, the ratio of entropy minimization loss Aentmin is set to 0.06, the number of iterations for adversarial training it is set to the degree of adversarial noise set to 6.
5. ICT: the ratio of unsupervised loss A set to 100, and the parameter of Beta distribution in Mixup is set to 0.5.
6. UDA: the ratio of unsupervised loss A is set to 1.0, the threshold is set to 0.8, and the temperature of softmax is set to 0.4.
7. FixMatch:
the ratio of unsupervised loss A is set to 1.0, the threshold is set to 0.95, and the temperature of softmax is set 0.5.
8. FlexMatch:
the ratio of unsupervised loss X, is set to 1.0, the basic threshold is set 0.95, the temperature of softmax is set to 0.5, and the threshold warmup mechanism is used.
9. FreeMatch:
the ratio of unsupervised loss X, is set 1.0, the EMA decay is set 0.999. the threshold is set to 0.95, and the temperature of softmax is set to 0.5.
10. SoftMatch: the ratio of unsupervised loss X, is set to 1.0, the basic threshold is set to 0.95, the temperature of softmax is set to 0.5, and the distribution alignment mechanism is used.
21

---

Published as a conference paper at ICLR 2024
Robust Deep SSL Algorithms
1. UASD: the ratio of unsupervised loss X is set to 1.0, and the threshold is set to 0.95,
2. CAFA: the base SSL algorithm used Pi Model, the warmup rate of unsupervised loss
Wh. is set I5. The perturbation magnitude is set 0.014 and the Beta distribution 0.75, the warmup rate adversarial loss wa is set to 15 the ratio of unsupervised loss X, is exp(- 5- (1 min( T 1.0))²) and the ratio of adversarial loss X is exp(-5- (1 - min( F. 1.0))²) in the t-th iteration where I" is the number of iterations.
parameter a is
3. MTCF: the ratio of unsupervised loss Au is set to 75, the temperature T is set to 0.5, and the parameter of Beta distribution in Mixup is set to 0.75.
4. Fix-A-Step: the parameter of Beta distribution in Mixup is set to 0.75, FixMatch set to the base SSL method, and all the hyperparameters are the same FixMatch.
Data Augmentation
1. Agnews and IMDB/Amazor the weak and strong augmentations synonyms replace- ments with and 5 words respectively.
2. wine, irís, letter: weak and strong augmentations are Gaussian noise with and 0.2 rates respectively.
3. CIFARIO, CIFAR100, Image-CLEF- the weak augmentation is RandomHorizontallliip and the strong augmentation is RandAugment.
Others Hyper-Parameteer
batch size: the batch size for the IMDB/Amazon dataset 8. the batch size for the Agnews dataset is 16, the batch size for the Image-CLEi CLEF dataset is 32, the batch size for CIFARIO and CIFAR100 64, the batch size for tabular datasets is 64.
2. iteration: the iteration for Image-CIEFF dataset is 2000, the iteration for the tabular dataset is set to 1000, the iteration for ag ig_news and IMDB/Amaze is set to 5000, the iteration for CIFARIO and CIFAR100 100000.
3. optimizer: the optimizer for all datasets SGD with a learning rate 5e-4 and momentum
of 0.9.
4. scheduler: the scheduler for all datasets is Cosine with num_cyyles 7/16.
A.6 RESULTS UNDER INCONSISTENT DISTRIBUTION
The experimental results can be referenced tables 13
A.7
RESULTS UNDER INCONSISTENT FEATURE SPACE
The experimental results can be referenced in tables 18 20
A.8
RESULTS UNDER INCONSISTENT LABEL SPACE
The experimental results can be referenced in tables 21 to 23
22

---

Published as a conference paper at ICLR 2024
Table 13: Evaluation of SSL algorithms using iris dataset under inconsistent data distributions
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.940
0.940
0.940
TSVM
0.973
0.983
0.960
0.015
0.018
-0.377
SSGMM
0.835
0.867
0.803
0,033
0.042
0,042
Label Propagation
0.978
0.973
0.970
0.011
0.012
0.412
Label Spreading
0.977
0.973
0.967
0.012
0.013
0.438
Tri- Tri-Training
0.959
0.977
0.950
0.009
0.012
-0.310
Assemble
0.973
0.947
0.947
0.011
0.016
0.787
FT
TT-rransformer
0.980
0.980
0.980
Pseudo Label
0.988
0.987
0.987
0.001
0.002
-0.414
Pi-Model
0.981
0.987
0.973
0.009
0.010
-0.029
Mean Teacher
0.987
0.987
0.987
0.000
0.000
VAT
0,986
0.987
0.983
0.001
0.002
-0.993
ICT
0.987
0.987
0.987
0.000
0.000
UDA
0.986
0.987
0.983
0.002
0.002
-0.414
FixMatch
0.977
0.983
0.967
0.011
0.012
-0.380
Flexhatch
0.978
0.980
0.967
0.016
0.017
0.109
FreeMatch
0.978
0.970
0.967
0.010
0.011
0.603
SoftMatch
0.981
0.987
0.967
0.009
0.012
-0.441
UASD
0.984
0.980
0.977
0.008
0.009
0,217
CAFA
0.989
0.990
0.987
0.002
0.002
-0.414
MTCF
0.984
0.993
0.973
0.007
0.009
-0.605
Fix-A-Step
0.987
0.987
0.983
0.003
0.003
-0.338
Table 14: Evaluation of SSL algorithms using wine dataset under inconsistent data distributions
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.872
0.872
0.872
TSVM
0.944
0.955
0.925
0.012
0.018
-0.68
SSGMM
0.446
0.429
0.379
0.049
0.070
0.006
Label Propagation
0.928
0.944
0,909
0.017
0.016
-0.596
Label Spreading
0.927
0.939
0.899
0.020
0.024
0.037
Tri-Training
0.952
0.965
0.936
0.012
0.012
-0.765
Assemble
0.941
0.939
0.912
0.014
0.017
-0.604
FT- Transformer
0.875
0.875
0,875
Pseudo Label
0.931
0.931
0.925
0.005
0.006
-0.575
Pi-Model
0.961
0.973
0.936
0.017
0.018
-0.641
Mean Teacher
0.920
0.920
0.920
0.000
0.000
VAT
0.972
0.984
0.955
0.010
0.011
-08224
ICT
0.957
0.957
0.957
0.000
0.000
UDA
0,940
0.952
0.904
0.027
0.032
-0.138
FixMatch
0.958
0.971
0.928
0.027
0.031
-0.362
Fatch
0.948
0.955
0.915
0.027
0.029
-0.326
Free
0.903
0.907
0.853
0.031
0.039
0.204
SoftMatch
0.960
0.976
0.944
0.015
0.019
-0.134
UASD
0.912
0.912
0.912
0.000
0.000
CAFA
0.876
0.864
0.848
0.028
0.036
0.249
MTCF
0.962
0.981
0.947
0.017
0.017
-0.493
Fix-A-Step
0.963
0.979
0.939
0.013
0.013
-0.893
23

---

Published as a conference paper at ICLR 2024
Table 15: Evaluation of deep SSL methods using ImageNet/Catch and ImageNet/Pascal datasets.
Dataset
Model
AUC
Acc(0)
WA
EVM
VS
RCC
Supervised
0.909
0.909
0.909
Pseudo Label
0.907
0.908
0.907
0.001
0.001
-0.621
Pi-Model
0.909
0.907
0.907
0.001
0.001
0.655
Mean Teacher
0.903
0.904
0.900
0.003
0.003
0.169
VAT
0.888
0.881
0.881
0.002
0.002
0.928
ICT
0.907
0.909
0.903
0.003
0.004
-0.359
UDA
0.896
0.904
0.891
0.006
0.007
-0.512
ImageNet/Caltech
Fix FitMatch
0.902
0.905
0.887
0.005
0.007
-0.726
FlexMatch
0.906
0.921
0.893
0.008
0.010
-0.861
FreeMatch
0.864
0.916
0.832
0.031
0.028
-0.786
SoftMatch
0.904
0.908
0.891
0.007
0.007
-0.805
UASD
0.897
0.897
0.897
0.000
0.000
CAFA
0.893
0.892
0.889
0.002
0.002
0.820
MTCF
0.880
0.904
0.855
0.016
0.015
-0.841
Fix-A-Step
0.869
0.876
0.856
0.007
0.011
-0.347
Supervised
0.909
0.909
0.909
Pseudo Label
0.906
0.908
0.901
0.004
0.004
-0.375
Pi-Model
0.907
0.907
0.901
0.005
0.006
-0.085
Mean Teacher
0.904
0.909
0.900
0.003
0,003
-0.813
VAT
0.884
0.889
0.875
0.006
0.007
-0.781
ICT
0.906
0.909
0.903
0.004
0.004
0.000
UDA
0.894
0.904
0.883
0.013
0.017
0.173
ImageNet/Pascal
FixMatch
0.901
0.905
0.889
0.013
0.017
-0.528
FlexMatch
0.911
0.921
0.897
0.006
0.007
-0.915
FreeMatch
0.855
0.876
0.824
0.022
0.025
-0.665
SoftMatch
0.888
0.908
0.839
0.014
0.018
-0.846
UASD
0.897
0.896
0.896
0.000
0.001
0.655
CAFA
0,890
0.897
0.887
0.003
0.003
-0.768
MTCF
0.875
0.909
0.840
0.014
0.012
-0.928
Fix-A-Step
0.877
0.885
0.867
0.009
0.010
-0.224
24

---

Published as a conference paper at ICLR 2024
Table 16: Evaluation of deep SSL methods using Caltech/ImageNet and Caltech/Pascal datasets.
Dataset
Model
AUC
Acc(0)
WA
EVM
VS
RCC
Supervised
0.945
0.945
0.945
Pseudo Label
0.953
0.949
0,949
0.001
0.002
0.655
Pi-Model
0.950
0.945
0.945
0.001
0.002
0.655
Mean Teacher
0.951
0.955
0,948
0.003
0.003
-0.714
VAT
0.953
0.959
0.945
0.005
0.005
-0.895
ICT
0.950
0.945
0.945
0.003
0.003
0.247
UDA
0.949
0.943
0,943
0.003
0.003
0.769
Caltech/lmageNet
FixMatch
0.952
0.957
0.943
0.003
0.003
-0.928
FlexMatch
0.960
0.965
0.953
0.007
0.007
-0.498
FreeMatch
0.896
0.947
0.800
0.029
0.021
-0.947
SoftMatch
0.941
0.960
0.912
0.011
0.011
-0.91
UASD
0.962
0.963
0.961
0.000
0.001
-0.655
CAFA
0.962
0.965
0,960
0.002
0.002
-0.690
MTCF
0.946
0.947
0.939
0.007
0.007
-0.300
Fix-A-Step
0.940
0.948
0.931
0.007
0.008
-0.474
Supervised
0.945
0.945
0.945
Pseudo Label
0.951
0.949
0.949
0.001
0.001
0.781
Pi-Model
0.950
0.945
0.945
0.001
0.002
0.655
Mean Teacher
0.951
0.953
0.948
0.002
0.003
-0,825
VAT
0.944
0.956
0.936
0.006
0.006
-0.678
ICT
0.949
0.945
0.945
0.002
0.003
-0.222
UDA
0.946
0.943
0.939
0.006
0.007
0.030
Caltech/Pascal
FixMatch
0.933
0.957
0.852
0.022
0.036
-0.729
FlexMatch
0.952
0.965
0.937
0.007
0.007
-0.920
FreeMatch
0.876
0.956
0.777
0.036
0.024
-0.980
SoftMatch
0.933
0.960
0.892
0.017
0.013
-0.920
UASD
0.962
0.963
0,961
0.000
0.001
-0.655
CAFA
0.961
0.960
0.960
0.001
0.001
0.169
MTCF
0.946
0,943
0.943
0.004
0.005
0.146
Fix-A-Step
0.942
0.928
0,928
0.010
0.011
0.590
25

---

Published as a conference paper at ICLR 2024
Table 17: Evaluation of deep SSL methods using Pascal/Caltech and Pascal/ImageNet datasets.
Dataset
Model
AUC
Acc(()
WA
EVM
VS
RCC
Supervised
0.732
0.732
0.732
Pseudo Label
0.726
0.724
0.724
0.001
0.002
0.804
Pi-Model
0.723
0.724
0.723
0.000
0.001
-0.655
Mean Teacher
0.727
0.728
0.721
0.007
0.008
-0.398
VAT
0.721
0.731
0.712
0.007
0.009
-0.216
ICT
0.727
0.727
0.721
0.006
0.006
-0.241
UDA
0.731
0.728
0.725
0.003
0.003
0.916
Pascal/Caltech
Fix Match
0.698
0.713
0.611
0.027
0.043
-0.628
FlexMatch
0.719
0.719
0.701
0.013
0.018
-0.529
Frtc
0.634
0.704
0.545
0.035
0.028
-0.964
SoftMatch
0.694
0.691
0.676
0.011
0.013
-0.486
UASD
0.724
0.724
0.724
0.000
0.000
CAFA
0.724
0.732
0.719
0.005
0.007
-0,516
MTCF
0.69
0.708
0.661
0.016
0.016
-0.869
Fix-A-Step
0.694
0.697
0.689
0.003
0.002
-0.854
Supervised
0.732
0.732
0.732
Pseudo Label
0.724
0.724
0.721
0.002
0.002
-0.414
Pi-Model
0.724
0.724
0.723
0.001
0.001
0.497
Mean Teacher
0.728
0.727
0.724
0.003
0.004
-0,549
VAT
0.715
0.721
0.707
0.007
0.007
-0.665
ICT
0.727
0.727
0.724
0.002
0.003
-0.177
UDA
0.722
0.728
0.715
0.005
0.007
0.272
Pascal/ImageNet
FixMatch
0.717
0.713
0.713
0.004
0.004
0.425
Flex
0.716
0.719
0.708
0.007
0.009
-0.128
FreeMatch
0.659
0.708
0.633
0.021
0.025
-0.629
SoftMatch
0.694
0.691
0.688
0.003
0.004
0.763
UASD
0.724
0.724
0.724
0.000
0.000
CAFA
0.716
0.717
0.715
0.004
0.004
0.000
MTCF
0.703
0.707
0.684
0.016
0.018
-0.617
Fix-A-Step
0.696
0.700
0.680
0.008
0.008
-0.751
26

---

Published as a conference paper at ICLR 2024
Table 18: Evaluation of SSL algorithms using iris dataset under inconsistent feature space
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.920
0.920
0.920
TSVM 0.887
0.937
0.850
0.023
0.021
-0,929
SSGMM
0.756
0.92
0.677
0.049
0.054
-0.906
Label Propagation
0.797
0.820
0.780
0.011
0.011
-0.705
Label Spreading
0.832
0.843
0.817
0.007
0.008
-0.912
Tri-Training
0.861
0.893
0.843
0.020
0.025
-0.638
Assemble
0.917
0.907
0.907
0.015
0.018
-0.266
F
TT-Trrnsformer
0.937
0.937
0.937
Pseudo Label
0.922
0.920
0.920
0.001
0.002
-0.293
Pi-Model
0.913
0.910
0.910
0.003
0.004
-0.293
Mean Teacher
0.917
0.917
0.917
0.000
0.000
VAT
0.918
0.923
0.907
0.007
0.010
-0.345
ICT
0.923
0.923
0.923
0.000
0.000
UDA
0.926
0.927
0.92
0.005
0.006
-0.744
FixMatch
0.933
0.940
0.927
0.007
0.008
-0.319
Flex
0.909
0.893
0.857
0.024
0.028
0.825
FreeMatch
0.844
0.913
0.790
0.025
0.016
-0.976
SoftMatch
0.934
0.923
0.923
0.003
0.004
0.925
UASD
0.913
0.913
0.900
0.009
0.012
-0.026
CAFA
0.931
0.933
0.927
0.003
0.003
-0.071
MTCF
0.900
0,860
0,860
0.014
0.017
0.845
Fix-A-Step
0.898
0.877
0.877
0.014
0.017
0.339
Table 19: Evaluation of SSL algorithms using wine dataset under inconsistent feature space
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.846
0,846
0,846
TSVM
0.890
0.938
0.832
0.039
0.038
-0.865
SSGMM
0.500
0.722
0.359
0.096
0.110
-0.606
Label Propagation
0.800
0.930
0.703
0.075
0.087
-0.487
Label Spreading
0.887
0.924
0.822
0.022
0.015
-0.958
Tri-Training
0.924
0.962
0.895
0.018
0.018
-0.912
Assemble
0.897
0.938
0.841
0.021
0.012
-0.977
FT Transformer
0.938
0.938
0.938
Pseudo Label
0.888
0.911
0.868
0.012
0.010
-0.937
Pi-Model
0.932
0.943
0.916
0.005
0.006
-0.934
Mean Teacher
0.914
0.914
0.914
0.000
0.000
VAT
0.928
0.943
0.905
0.015
0.016
-0.789
ICT
0.916
0.916
0.916
0.000
0.000
UDA
0.924
0.935
0.881
0.03
0.035
0.326
Mithat
0.912
0.959
0.881
0.022
0.025
-0.564
Fltxhatch
0.910
0.932
0.851
0.048
0.056
-0.153
FreeMatch
0,888
0.922
0.835
0.031
0.036
-0.068
SoftMatch
0.887
0.957
0.816
0.050
0.069
0,000
UASD
0.962
0.962
0,962
0.000
0.000
CAFA
0,928
0.951
0.873
0.035
0.046
-0.228
MTCF
0.905
0.951
0.881
0.015
0.012
-0.913
Fix-A-Step
0.917
0.946
0.873
0.023
0.021
-0.923
27

---

Published as a conference paper at ICLR 2024
Table 20: Evaluation on CIFARIO and CIFARI00 under inconsistent feature spaces
Dataset
Method
AUC
Acc(O)
WA
EVM
VS
RCC
Supervised
0.473
0.473
0.473
Pseudo Label
0.519
0.524
0.515
0.002
0.003
-0,874
Pi-Model
0.500
0.511
0.485
0.007
0.007
-0.882
Mean Teacher
0.470
0.486
0.457
0.006
0.005
-0,962
VAT
0.501
0.550
0.466
0.020
0.018
-0.880
ICT
0.468
0.476
0.456
0.005
0.005
-0,929
UDA
0.498
0.505
0.438
0.019
0.025
-0.707
CIFARTO
FixMatch
0.517
0.551
0.430
0.037
0.042
-0.661
FlexMatch
0.552
0.607
0.431
0.041
0.039
-0.221
FreeMatch
0.555
0.645
0.423
0.045
0.029
-0.962
SoftMatch
0.559
0.661
0.453
0.042
0.009
-0.998
UASD
0,481
0.486
0.479
0.003
0.003
-0,625
CAFA
0.484
0.502
0.469
0.007
0.003
-0.988
MTCF
0.496
0.625
0.316
0.107
0.130
-0.604
Fix-A-Step
0.516
0.551
0.424
0.025
0.032
-0.832
Supervised
0.368
0.368
0.368
Pseudo Label
0.371
0.374
0.369
0.003
0.002
-0.126
Pi-Model
0.367
0.369
0.365
0.003
0.002
-0.147
Mean Teacher
0.342
0.368
0.301
0,013
0.007
-0,972
VAT
0.364
0.384
0.342
0.008
0.005
-0.985
ICT
0.343
0.367
0.306
0.006
0.012
-0,970
UDA
0.349
0.350
0.343
0.004
0.003
-0.555
CIFAR100
FixMatch
0.378
0.401
0.336
0.011
0.013
-0.93
FlexMatch
0.376
0.397
0.331
0.019
0.016
-0.799
FreeMatch
0.382
0.420
0.327
0.019
0.012
-0.958
SoftMatch
0.373
0.406
0.316
0.014
0.018
-0,934
UASD
0.368
0.364
0.364
0.002
0.003
0.256
CAFA
0.344
0.365
0.316
0.010
0.007
-0,966
MTCF
0.273
0.170
0.170
0.052
0.066
0.629
Fix-A-Step
0.328
0.394
0.161
0.047
0.047
-08822
28

---

Published as a conference paper at ICLR 2024
Table 21: Evaluation of SSL algorithms using iris dataset under inconsistent label space
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.930
0.930
0.930
TSVM
0.986
1.000
0.955
0.013
0.015
-0.721
SSGMM
0.944
1.000
0.83
0.040
0.065
-0.831
Label Propagation
0.993
1.000
0.99
0.003
0.004
-0.524
Label Spreading
0.996
1.000
0.99
0.006
0.007
-0.272
Tri-Training
0.997
1.000
0.985
0.005
0.007
-0.618
Assemble
1.000
1.000
1.000
0.000
0.000
FT-
TT-Transformer
1.000
1.000
1.000
Pseudo Label
0.994
1.000
0.990
0.003
0.004
0.355
Pi-Model
0.995
1.000
0.985
0.003
0.004
-0878
Mean Teacher
1.000
1.000
1.000
0.000
0.000
VAT
0.999
1.000
0.996
0.001
0.002
-0.655
ICT
1.000
1.000
1.000
0.000
0.000
UDA
0.996
1.000
0.990
0.002
0.002
-0,924
FixMatch
0.995
1.000
0.985
0.003
0.002
-0.930
FlexMatch
0.996
1.000
0.990
0.003
0.005
-0.497
FreeMatch
0.956
0.991
0.871
0.028
0.023
-0,905
SoftMatch
0.996
1.000
0.989
0.006
0.007
-0.393
UASD
0.987
0.987
0.987
0.000
0.000
CAFA
1.000
1.000
1.000
0.000
0.000
MTCF
0.995
1.000
0.985
0.007
0.007
-0,776
Fix-A-Step
0.998
1.000
0.985
0.003
0.006
-0.655
Table 22: Evaluation of SSL algorithms using wine dataset under inconsistent label space
Model
AUC
Acc(O)
WA
EVM
VS
RCC
XGBoost
0.846
0.846
0.846
TSVM
0.924
0.938
0.905
0.012
0.014
-0.721
SSGMM
0.643
0.722
0.560
0.035
0.025
-0,972
Label Propagation
0.889
0.93
0,833
0.023
0.027
-0,903
Label Spreading
0.883
0.924
0.833
0.028
0.033
-0.831
Tri- Tri-Training
0.879
0.962
0.829
0.038
0.043
-0.834
Assemble
0.904
0.938
0.862
0.021
0.018
-0.929
FT Trannssffrrerr
0.964
0.964
0.964
Pseudo Label
0.885
0.898
0.862
0.015
0.016
-0.815
Pi-Model
0.862
0.880
0.840
0.015
0.016
-0.611
Mean Teacher
0.920
0.920
0.920
0.000
0.000
VAT
0.882
0.920
0.837
0.031
0.029
-0.891
ICT
0,920
0.920
0.920
0.000
0.000
UDA
0.881
0.873
0.858
0.024
0.028
0.138
FixMatch
0.879
0.913
0.833
0.016
0.009
-0.978
Match
0.871
0.927
0.833
0.031
0.036
-0.648
FreeMatch
0.831
0.877
0.790
0.026
0.038
-0,601
SoftMatch
0.943
0.976
0.920
0.013
0.011
-0.911
UASD
0.923
0.923
0.923
0.000
0.000
CAFA
0.880
0.815
0.815
0.042
0.043
0.779
MTCF
0.908
0.971
0.825
0.041
0.045
-0.853
Fix-A-Step
0.947
0.960
0.927
0.007
0.004
-0.976
29

---

Published as a conference paper at ICLR 2024
Table 23: Evaluation on CIFARIO and CIFARI00 under inconsistent label spaces
Dataset
Method
AUC
Acc(O)
WA
EVM
VS
RCC
Supervised
0.643
0.643
0.643
Pseudo Label
0.692
0.708
0.676
0.006
0.004
-0,973
Pi-Model
0.672
0.703
0.654
0.01
0.009
-0.937
Mean Teacher
0.639
0.647
0.634
0.003
0.005
-0.333
VAT
0.697
0.734
0.661
0.015
0.011
-0,974
ICT
0.643
0.647
0.642
0.002
0.002
-0.819
UDA
0.676
0.73
0.594
0.027
0.015
-0,963
CIFARTO
FixMatch
0.608
0.705
0.479
0.047
0.036
-0.933
FlexMatch
0.731
0.806
0.614
0.038
0.02
-0.965
FreeMatch
0.733
0.815
0.640
0.035
0.012
-0.994
SoftMatch
0.723
0.806
0.601
0.041
0.021
-0.968
UASD
0.644
0.641
0.641
0.002
0.002
0.404
CAFA
0.675
0.674
0.672
0.005
0.006
0.093
MTCF
0.747
0.798
0.681
0.024
0.008
-0,989
Fix-A-Step
0.681
0.757
0.517
0.048
0.048
-0.908
Supervised
0.444
0.444
0.444
Pseudo Label
0.453
0.459
0.448
0.002
0.002
-0.933
Pi-Model
0.438
0.441
0.437
0.002
0.002
-0.614
Mean Teacher
0.442
0.437
0.437
0.001
0.002
0.817
VAT
0.444
0.473
0.411
0.012
0.008
-0.975
ICT
0.441
0.441
0.44
0.001
0.001
0.54
UDA
0.416
0.434
0.381
0.011
0.01
-09006
CIFAR100
FixMatch
0.392
0.496
0.25
0.049
0.024
-0.975
FlexMatch
0.416
0.483
0.345
0.028
0.01
-0.992
FreeMatch
0.465
0.518
0.386
0.026
0.011
-0.978
SoftMatch
0.473
0.521
0.407
0.023
0.008
-0,981
UASD
0.437
0.438
0.436
0.002
0.002
-0.142
CAFA
0.441
0.442
0.44
0.001
0.001
-0.055
MTCF
0.254
0.28
0.234
0.009
0.007
-0.953
Fix-A-Step
0.414
0.486
0.299
0.037
0.024
-09554
30


Reviews:
[{"review_id": "ilTO4sg4Mm", "reviewer": "Reviewer_2gQd", "summary": "This paper establishes a benchmark for robust semi-supervised learning in open environments, providing reliable evaluation and analysis methods. The proposed evaluation method is used to assess the robustness of current mainstream semi-supervised learning algorithms. The robustness of semi-supervised learning algorithms in open environments is analyzed based on both theoretical considerations and experimental results.", "questions": "In practical applications, how can the metrics for measuring real-world inconsistency be correlated with the inconsistency in the evaluation framework?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1. Scientifically sound tools and metrics, such as RAC curves and evaluation metrics like AUC, EA, and WA are employed. This has made a positive contribution to the evaluation of existing semi-supervised learning algorithms and the standards for designing future new algorithms.\n\nS2. This research is comprehensive, considering various open environments, data modalities, and types of semi-supervised learning algorithms.\n\nS3. The analysis section is based on a solid theoretical foundation and experimental results, ensuring its reliability.", "weaknesses": "The main weakness, as discussed in the limitations section by the authors, lies in the fact that the proposed evaluation framework cannot assess highly complex real-world scenarios with low calculation complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EExcscTulS", "reviewer": "Reviewer_dSe9", "summary": "This paper studies an important aspect of the research field of semi-supervised learning, specifically focusing on semi-supervised learning in open environments. It considers issues such as inconsistent data distributions, label spaces, and feature spaces. The authors build a comprehensive benchmark, including various data types such as tabular data, image data, and text data, and adequate evaluation metrics for this problem. Additionally, the paper provides a solid theoretical analysis to guide future research in this direction.", "questions": "1.\tThis paper proposes various metrics to evaluate the robustness of existing semi-supervised learning methods in different aspects. Is it possible to propose a unified method for ranking existing methods, as there are many experiment results but we cannot know which algorithm is better?\n2.\tPlease refer to some questions raised in the weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThis paper studies an important problem of semi-supervised learning, called robust semi-supervised learning in open environments. This line of research attracts a lot of attention and is becoming more and more important recently. Therefore, building a benchmark for this problem is meaningful and reasonable.\n2.\tThis paper has built a comprehensive benchmark for robust semi-supervised learning in open environments. This benchmark includes various data types, 20 semi-supervised methods, and extensive experimental results and analyses. Therefore, this paper makes a significant contribution to advancing the development of the semi-supervised learning community.\n3.\tThe authors provide both empirical results and theoretical analysis, making this paper technically sound. The provided theoretical framework discusses three different challenges in a unified manner, which are non-trivial and insightful.", "weaknesses": "1.\tAlthough the content of this paper is sufficient and informative (up to 46 pages), the main paper contains excessive text and lacks tables and figures to present the experiment results and conclusions. \n2.\tSection 7 discusses the theoretical results of this paper. It would be more appealing for authors to present some theorems and conclusions in the main paper, rather than presenting only the proof sketch of the theorems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SUwbyijqRS", "reviewer": "Reviewer_ynTZ", "summary": "This paper introduces a new robustness evaluation framework for semi-supervised learning, comprising multiple evaluation metrics. It assesses a wide range of semi-supervised learning algorithms including statistical semi-supervised learning algorithms, classical deep semi-supervised learning algorithms, and robust deep semi-supervised learning algorithms in three open environments. This paper also establishes a solid theoretical framework and provides a valuable analysis of the robustness of current semi-supervised learning based on experimental results and the theoretical foundation.", "questions": "Some algorithms have been evaluated on image and tabular datasets but not on text datasets. What could be the reason for this difference?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe evaluation method employed in this benchmark is reliable, and the adopted metrics are novel and diverse. These reflect the robustness of SSL algorithm performance from different perspectives.\n2.\tThis paper conducted numerous experiments, evaluating a wide set of algorithms which includes commonly used statistical semi-supervised learning algorithms, classical deep semi-supervised learning algorithms, and robust semi-supervised learning algorithms. The experimental results are comprehensive and convincing.\n3.\tIt is good to see that the paper is well-supported with abundant materials, providing extensive theoretical results and detailed descriptions of experiments, rendering it highly reliable.", "weaknesses": "It appears that some algorithms were only evaluated on tabular and image datasets, with no evaluation on text datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vi4mOSJs26", "reviewer": "Reviewer_Et8T", "summary": "This paper addresses the challenge of non-robust performance in semi-supervised learning (SSL) due to inconsistencies between labeled and unlabeled data, especially in open environments with different data sources. The authors introduce the Robustness Analysis Curve (RAC) and related metrics, reshaping the framework for robust SSL to achieve global robustness rather than just local robustness. They evaluate a variety of statistical and deep SSL algorithms across diverse datasets (tabular, image, and text) in three open environment scenarios: inconsistent data distributions, label spaces, and feature spaces, providing a comprehensive benchmark and detailed analysis to enhance the robustness of SSL algorithms in open environments.", "questions": "The current theoretical results appear to be applicable only when the degrees of inconsistency across the three types of variations are the same. It raises the question of whether the theory would still hold if the degrees of inconsistency were different.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Clarity and Precision in Motivation: The manuscript excels in presenting a clear and well-articulated motivation for the research conducted. The authors emphasize the challenges posed by inconsistencies between labeled and unlabeled data in open environments. They have set a strong foundation for the study, making it easier for readers to understand the significance of the work and the potential impact it could have in advancing the field of robust machine learning. \n2. Comprehensive Experimental Evaluation: The manuscript stands out in its thorough and well-structured experimental evaluation. The authors have gone to great lengths to ensure that the performance of various statistical and deep SSL algorithms is rigorously assessed across a wide range of datasets, including tabular, image, and text data. The inclusion of three distinct open environment scenarios (inconsistent data distributions, label spaces, and feature spaces) adds depth to the evaluation, ensuring that the results are comprehensive and reliable. The meticulous design of the experiments and the extensive coverage of different scenarios demonstrate a commitment to empirical rigor, which significantly enhances the credibility and value of the research findings.\n3. Solid Theoretical Foundation: The strength of the manuscript is further augmented by its robust theoretical framework. The introduction of the Robustness Analysis Curve (RAC) and the associated metrics offers a novel perspective to approach the problem of robustness in SSL. The detailed theoretical analysis provided in the paper not only supports the empirical findings but also offers deeper insights into the underlying mechanisms that contribute to the robustness of SSL algorithms in open environments. This solid theoretical grounding ensures that the contributions of the manuscript are not just empirical but also provide a conceptual advancement in the understanding of robust SSL.", "weaknesses": "1. The manuscript lacks clarity in some of its detail descriptions, such as the explanation of the variable 't', which is not very comprehensible. The author could provide a detailed example to help readers better understand the meaning and function of 't'.\n2. The robust semi-supervised learning referred to in the manuscript actually mainly pertains to safe semi-supervised learning. Perhaps utilizing the term \"safe semi-supervised learning\" would be more suitable and precise in this context.\n3. In Figure 2, there are too many methods presented, making it difficult to read. It would be sufficient to showcase a comparison of some representative methods instead.\n4. The manuscript introduces numerous evaluation metrics; however, the inherent relationships among these metrics are not very clear. The individual motivations for proposing each of these metrics could be further elucidated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)