SemiReward: A General Reward Model for Semi-supervised Learning
Authors: Siyuan Li
Weiyang Jin
Zedong Wang
Fang Wu
Zicheng Liu
Cheng Tan
Stan Z. Li
Venue: ICLR 2024 poster (2024)
Keywords: Semi-supervised Learning
Reward Model
Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper proposes a Semi-supervised Reward model (SemiReward) that predicts reward scores to evaluate and filter out high-quality pseudo labels, which is pluggable to mainstream SSL methods in wide task types and scenarios.

Abstract: Semi-supervised learning (SSL) has witnessed great progress with various improvements in the self-training framework with pseudo labeling. The main challenge is how to distinguish high-quality pseudo labels against the confirmation bias. However, existing pseudo-label selection strategies are limited to pre-defined schemes or complex hand-crafted policies specially designed for classification, failing to achieve high-quality labels, fast convergence, and task versatility simultaneously. To these ends, we propose a Semi-supervised Reward framework (SemiReward) that predicts reward scores to evaluate and filter out high-quality pseudo labels, which is pluggable to mainstream SSL methods in wide task types and scenarios. To mitigate confirmation bias, SemiReward is trained online in two stages with a generator model and subsampling strategy. With classification and regression tasks on 13 standard SSL benchmarks across three modalities, extensive experiments verify that SemiReward achieves significant performance gains and faster convergence speeds upon Pseudo Label, FlexMatch, and Free/SoftMatch. Code and models are available at https://github.com/Westlake-AI/SemiReward.

Published as a conference paper at ICLR 2024
SEMIREWARD: A GENERAL REWARD MODEL FOR SEMI-SUPERVISED LEARNING
Siyuan Li Weiyang Jin Zedong Wang" Fang Wu Zicheng Liu Cheng Tan Stan Z. Li Al Lab, Research Center for Industries the Future, Hangzhou, China;
"Zhejiang University, College of Computer Science and Technology; "Westlake University 1is weiyuan@@weste edu.. en; wayney j in gma [wangzedong; wufang; siuuiiiiiii cheng; tancheng; stan. zq. 11] @westlake. edu. cn
gma il. com;
ABSTRACT
Semi-supervised learning (SSL) has witnessed great progress with various im- provements in the self-training framework with pseudo labeling. The main chal- lenge is how to distinguish high-quality pseudo labels against the confirmation bias. However, existing pseudo-label selection strategies limited pre-defined schemes or complex hand-crafted policies specially designed for classification, failing to achieve high-quality labels, fast convergence, and task versatility si- multaneously. To these ends, we propose a Semi- seee-supe Reward framework (SemiReward) that predicts reward scores evaluate and filter out high-quality pseudo labels, which is pluggable to mainstream SSL methods in wide task types and scenarios. To mitigate confirmation bias, SemiReward is trained online stages with a generator model and subsampling strategy. With classifica- tion and regression tasks on 13 standard SSL benchmarks across three modalities, extensive experiments verify that SemiReward achieves significant performance gains and faster convergence speeds upon Pseudo Label, FlexMatch, and Free/- SoftMatch. Code and models are available https: / thub. con/ West a ake- AAT/Sem Reward
UitrsSoand-Rk 400
Yelp Review
DSoisyy 18k 1775
AG News 200
UturaSound-ae 8k 100
Yelp Review 1000
ESC-500
Yahoo A Answor 2000.
FSDDuissy-18 18
1773
21
CIFAR-100 400
20.1
20.9
ESC-50 250
Yelp 2550
2..
1x2
3.54
2i.7
STI-101
#
Ag:DB 122
Yolp Revicw 1900
BunSAT 40
EuroSAT 20
IMDD Wiki
CIFAR 100 200
EuroSAT 20
1915
PunsSAT 40
STL-10 40
Pseudo Label
Pseudo Label+SR
FlexMutch
FlexMatch+SR
Previous SOTA
Previous SOTA
Frew/SofiMathh-S
(a) SSL classificatio and regression henchmarks (b) classiffiiin benchmarks with SOTA methods SemiReward (abbreviated as SR) enables existing SSL methods to select high-quality pseudo labels on both classification and regression tasks with fast convergence speeds (Figure 2). Error rates of SSL algorithms are plotted on CV. NLP, and Audio datasets. Note that previous SOTA marks the best performance among a set of methods, which denotes general SSL methods used for classification and regression tasks in (a) and 17 SSL methods in USB (Wang et al., 2022a) in (b). SemiReward noticeably improves performance when plugged into existing SSL methods.
Figure 1:
First authors contribute equally. CCorsponding author.

---

Published as a conference paper at ICLR 2024
+3.7
+1.9 and
12.5 aoc 1.7x
06.0
Sextkaore
15.
bandiin 2041
Tumnii (2440)
(a) CV: Euro-SAT (20)
(b) NLP: Yahoo! Answer (2000)
(c) Audio: ESC-50 (250)
Figure 2: Top-1 accuracy v.S. training iterations (x2048) on SSL datasets (the number of used
labels) of three modalities. Employing SemiReward with SOTA SSL methods produces +1. +1.9~3.7
performance gains with at least 1.7 times fewer training iterations compared to the baseline. We apply early-stop when the validation performance reaches the peak.
INTRODUCTION
In the past decades, deep learning (DL) has made great progress in various applications with differ- ent modalities (He et al., 2016; Devlin et al., 2018; Dong et al., 2018; Li et al., 2024). However, most tasks are in a supervised learning (SL) manner that requires manually labeling data, which is lim- ited in quantity and labor-exhaustive. To extend SL with massive unlabeled data, semi-supervised learning (SSL) exploits the information of unlabeled data with limited labeled data (Tarvainen & Valpola, 2017; Sohn al., 2020) in the self-training paradigm of pseudo-labeling (Lee al., 2013), i.e. training models with unlabeled data and pscudo labels assigned by models' predictions.
As a widely used technique, the main problem of SSL is how to generate accurate pseudo labels without or with tolerable effects of confirmation bias (Arazo et al., 2020), i... overfitting to incorrect pseudo labels from teacher models. There were three main strands of research, aiming at obtaining high-quality pseudo labels and a high sampling rate while being capable of various tasks and scenar- ios. Firstly, mainstream methods utilize threshold-based pseudo labeling (Sohn et al., 2020; Zhang et al., 2021; Kim et al., 2022; Wang et al., 2022b) with ad-hoc or complex hand-crafted strategies to select high-quality pseudo labels. However, these algorithms are predefined and task-specific, i.e., they are designed for classification tasks but cannot handle more challenging regression tasks. The second strand introduces pre-trained teacher models (Zhou & Li, 2010; Xie et al., 2020b) to generate high-quality pseudo labels, which require extra computational cost (e.g., double training times (Pham et al., 2021)) or suffer from confirmation bias (Yalniz et al., 2019). The third line explores consistency regulaizations (Xie et al., 2020a; Sohn et al., 2020; Li et al., 2021) to prevent confirmation bias of inaccurate pseudo labels, e.g., optimizing the consistency loss with weak-strong augmentation, which only work for specific modalities with prior augmentations. Therefore, none of the previous SSL methods achieved three goals simultancously.
This work answers core question in SSL training: how to efficiently evaluate a pseudo label com- prehensively? ? We introduce a reward score based on cosine similarity between pseudo and ground- truth labels as the quality standard, which is a smooth and well-calibrated metric for classification and regression tasks. Then, we propose Semi-supervised Reward framework (SemiReward) that predicts reward scores based on pseudo labels and corresponding unlabeled data for pseudo-label selection and can be used as add-on module for mainstream SSL methods. Specifically, a warder network predicts credible reward scores to filter pseudo labels for the student training and is learned to fit ground-trute reward scores online. To disentangle its training from the student, a two- stage training pipeline is designed with the assistance of a generator network, which generates "fake labels" that only train the rewarder. The rewarder and generator are first pre-trained alternatively on the labeled dataset in stage 1 to alleviate confirmation bias, then trained on randomly subsampled set of labeled data and selected unlabeled data in stage 2. Empirical studies show that SemiReward predicts calibrated reward scores to select high-quality pseudo labels with a high sampling rate to boost SSL training. We conduct comparison experiments on SSL benchmarks with three modalities and two task types, verifying that SemiReward improves both general and modern SSL algorithms in performance and convergence speeds. Our main contributions are three folds:
From fresh perspective, we introduce the reward score to evaluate pseudo-label qualities and design the rewarder to predict it by modeling unlabeled data and pseudo-label, together.
We propose a general and pluggable SemiReward framework that selects high-quality pseudo labels with reward scores. A two-stage training pipeline and a generator network are designed to train the rewarder online with negligible extra cost.
2

---

Published as a conference paper at ICLR 2024
SSL Datasel
Selection Strategy
SSL Dalusel
Teacher Model fr
Pseudo lubels
Selected lakell data
Rewarder R
Pseado
label
Selecoed labels unlsiciaa
Duta Augientttion
Studentt Model fa
Predicted targets
Model fs
Prodicted targets
I Lshel
Labels labeled data
Label
Labcls oflabcled date
(a) Consistency regularization framework
(b) SSL training with pre trained rewarder
Figure 3: Illustration of SSL training paradigm, where blue lines denote yseudo-labeling pipeline and red lines denote gradient propagation. (a) Confidenced-band based label selection strategy and strong- weak augmentations for consistency are task-specific and modality-speciin (requiring ad-hoc aug- mentations). (b) Rewarder is plug-ana aagg-and-pay label selection module for general SSL scenarios.
Extensive experiments on 13 datasets validate that SemiReward markedly increases per- formance and convergence speeds of popular SSL methods in classification and regression tasks. We also empirically verify the reliability of reward scores and designed modules.
PRELIMINARY
Semi-supervised training pipeline. SSL is an extended scenario of SL, where given a labeled dataset D, [/) v/ and unlabeled dataset Du {a NL < Ng. Considering any classification or regression task, y E R denotes the encoded ground- truth label, where C is the label dimension, and the model fs(.) learns to predict [s(x) = y E IRC As for C-class classification, one-hot encoding is adopted for y while converting the model output to arg max pyy). To utilize all training data, the general SSL training pipeline with pscudo-labeling contains three steps: (a) Pseudo-label generation. Given a teacher model fr(-) that is well-trained on Dz. it can generate pseudo-labels y" = fI (x") for Du. (b) Pseudo-label selection. High- quality pscudo labels Dy { {I(p"" T)y?' T are filtered by label selection mechanism -(. where T E [0, 1] is the threshold. (c) Supervised and unsupervised losses computation, denoted as L = Ls Lu. Given mini-batch of Bz data, Ls is written
N Ji=1 with the sample numbers
1 Cs Br.
H (1. (s(((x))))
where w(.) denotes stochastic data augmentations and H(:.)) is the loss function used for the SL task, such as cross-entropy and E loss for classification and regression tasks. Similarly, given a mini-batch of By unlabeled data, taking popular consistency regularization frameworks (Sohn al., 2020) as an example, the unsupervised loss is
By
Cu Bu
1
EIOT T)H T (F
where ((x;)) represents the strong augmented unlabeled data. As shown in Figure 3(a), the con- sistency regularization framework usually has three design aspects: (i) Jr and Is share the same network architecture and parameters of fs are updated to fr by copying or exponential moving erage (EMA). (ii) For most consistency-bases SSL methods, a hand-crafted I(.,)) requires predicted classification confidence to distinguish reliable labels. (ii) Since the teacher fr is more reliable than the student fs, the consistency that between fr and fs is introduced by constructing sample pairs (w(x²) R(xx))) with strong-weak augmentations proposed by UDA (Xie al., 2020a) and optimizing consistency through Lü.
Breaking Through Limitations of Co Label Selection. Existing label selection strategies in step (ii) only use y" or the confidence pd to evaluate pscudo labels in hand-crafted policies, which cannot guarantee the quality and stability of Dy. Meanwhile, the designed steps (ii) and (iii) limit the task and modality generalities of the pseudo-labeling pipeline. To tackle these problems, we parameterize II(,-) as a lightweight rewarder model R(x" yy) = r, where reward score r E [0. 1] represents the label quality and is defined in Sec. 3.1. In Figure 3(b), the pre-trained R can evaluate the label quality comprehensively based on both x" and y", rather than solely depended on y". And we define L in a simple and general form:
By
C = BL
H 1/ (s((aa)) +
Σ
T)HL T) ( (s((a)))) +Luxs (3)
Bu
Ce: where Laux denote training losses of the rewarder R with generator discussed in Sec. 3.2.
3

---

Published as a conference paper at ICLR 2024
3 SEMIREWARD
Here, we introduce SemiReward for high-quality pseudo-label selection in general SSL tasks. In Sec. 3.1, first define reward score pseudo-label evaluation metric and approximate it by rewarder model. Then, Sec. 3.2 describes how to learn the rewarder through a two-stage pipeline.
3.1 MEASUREMENT OF LABEL QUALITY
Unlike popular ranking loss (Ouyang al., 2022) in reinforcement learning (RL) (Schulman al., 2017), we define a continuous metric of pseudo-label quality based on label similarity.
Definition 3.1 (Reward Score). The reliability of a pseudo label y" of data is measured by label similarity -) with its ground truth label y', which can also be approximated by a rewarder ((::))
r(y", y S(y", y = R(x, v²) E [].
The ideal reward score should satisfy monotonicity and smoothness (not increasing dramatically) and strive meet the trend of calibration curve (Clark, 1975), where lower reward confidence indicates poorer label quality. Therefore, we define the label similarity based on cosine similarity.
Definition 3.2 (Label Similarity). Given vectorized label y E RC, the label similarity between y and Bj is defined as scaled cosine similarity:
S(y:. VU) 2 11 ||,
y1 My
0.5 [0.
(5)
Figure 4(a) verifies the properties r(y" ,') by changing the label similarity metrics to negative Le distance and andJSS-a JS-divergence, and it shows that Eq. (5) can be the better choice.
Calibration Curve
Calibeation Curve
Calibrooon Curve
1. Diszave
Crilas Seniliiiy
Rosa dar R/CC
25 Stutlacity
Reaacde si Amirics
Rova dar M.
Unitne Sialertz
Rewinder Aliction
80%
Howarder KiLh
Assinog
08%-
65%
67%
40%
677-
40%
05-
0.
O4
1.5
36
1,0
04
Reword Scree Throdhold
Reward Sunm: Threchold
Resond Score Throhold
(a) Various reward similarities
(h) Attention module in Rewarder (c) MLP module in Rewarder
Figure 4: How rewarder works illustrated by reward scores v.s. top-1 accuracy on CIFAR- 100 (400 labels). (a) Analysis of alternative reward similarities; (b) Ablation of cross-attention module in R, which is the vital component to learn calibrated reward scores; (c) Ablation of MLP layers.
To support both classification and regres- sion tasks, we determine the encoding strategies ensure that used labels are in vector format. This paper mainly dis- cusses the cases of one-hot classification or single attribute regression. Given raw scalar label, it can be encoded in "one-hot" format for classification. As for raw re- gression label y E [0, Cj, we propose a soft one-hot encoding that equally divides the scalar into bins and sets the k-th po- sition in the vector to 1 + (y - k), where k < y < + 1, while other positions are set to O. Afterward, we verify Eq. (4) with regression tasks in Figure 5 and find that it can serve as reliable metric and reduce the confirmation bias of raw pseudo labels. As for multi-label scenarios (Lin et al., 2017), we first encode raw labels for each task separately and then concatenate them the final labels.
Rowadd Store Threaaki
(a) Reward score E.S. MAE (b) MAE V.S. training steps Figure 5: Credible reward scores ensure the stable op- timization of the student model, while pseudo la- bels in general SSL methods gravely misled the student for regression task on RCF-MNIST (1% labels).
Rewarder.
As defined in Eq. (4), R(: :) tries to solve a regression problem: the model should extract semantic information of y' from " and tell the similarity between x" and y" according to their semantic correlation. As shown in Figure 6, R is designed as:
R(x". y² Sigmoid [MPP (CA(Emb)) (f)))) Emb(s")))
(6)
4

---

Published as a conference paper at ICLR 2024
where the input data and label are first linear embedded to the same dimension by Emb(.), and their correlations modeled by cross-attention module CA(-,-) and a (L.(()) module, then predict the reward score through Sigmoid function. Notice that x is converted to last- layer features by a pre-trained backbone model f(), e.g. an image in H x W resolutions will be encoded dim feature 2" E RRD, which is easy for the network to capture high-level information directly related to y'. As shown in Figure 4(b), we ablate modules in R. and find that C:(::)) is the most essential component to learn credible reward scores. Meanwhile, the backbone f(-) is also important to provide highly embedded features, or it will be hard and costly to learn such information by the lightweight R. On the contrary, the number of layers verified
Label
Data
Pretrained
Pretrained Backbone
Backbone
+
FC
FC
FC
D-
Norm
Norm
Norm
Cross-attention
MLP
MLP
Sigmoid
Rewarder score
Fake Label
in MLP(-) has less impact performance, in Figure 4(c). As for implementation, R uses a 2-layer MLP(.) with D = 128 and we simply apply the inherent teacher fr as f(-) in Eq. (6).
(a) Rewarder
(b) Generator
Figure 6: Network structures of effi- cient R and 9 anaayzed in Table
3.2 EFFICIENT TWO-STAGE TRAINING OF SEMIREWARD
Synchronizing with self teafnngaining paradigms, we train the rewarder R in a supervised manner with a Na
reward training set DR = (w(x]) ) 1- where y" is considered as the ground-trun srunthhhuthhhh label here. As discussed in Sec. 2, we expect reliable R to filter pseudo labels to ensure high label quality train fs. Hence, we design a two-stage training paradigm for R in Figure 7, and Dn will be dynamically constructed by D,, and Du. View Appendix B for detailed analysis of training processes.
Generator.
To train R. first design generator G(x*)) y' E RCC generate pseudo labels but not participate in the training process of fs. Thus, we denote them as "fake labels". Similar to Chen al. (2022a), 9 decouples the training of fs and R. avoid confirmation bias. Meanwhile, the fake labels generated by G gradually change from random to accurate, which helps R steadily fit reward scores on high-quality pseudo-label distributions. Its network is also as lightweight R, containing the pre-trained f followed by sample embedding Emb(-) and MLP(-) module in Figure 6.
Pre-training Rewarder. R and G will be trained with fixed DR Dr before T training iterations. In the first stage, our main optimization goal is to approximate the ground truth reward scores with a wide range of fake labels without affecting the training of fs- Thus, R does not select pseudo labels for the student fs. and we introduce G(x") yf to generate fake labels that gradually get better. We compute losses for R and alternatively as the auxiliary loss Laux = LR + Lg:
1 Bg
CR
la
(R((,) ((F,))))))
BR
BR
Cg = BR Σ
l
((.))
(8)
where R and denote forward without requiring gradients, which prevents two losses from inter- fering with each other. In implementations, we adopt two independent optimizers for R. and G for
Cosine similarity
Label
Truth rewards
Label
Truth rewards
Data
Generntor g
MSE loss
labels
Generntor g
Fake Inbels
Rewarder R
Predicted rewards
MSE loss
Unlabeled Data
Student Model fo
Pacuudo
Pseudo rewards
Rewarder R
Predicted rewards
labels
Seloced by Rewander
(a) Stage 1: Pre- tr-iirainig with labeled data Figure 7: Two-stage training paradigm of SemiReward. (a) prevent from distorting fs, we pre-train R and G to convergence the carly stage of SSL training with Dz. (b) After T' iterations, R further learns from the Di sub-samples from Dr U Du with ignorable training cost.
(b) Stage 2: Semi- Semi-supervised training with DR
5

---

Published as a conference paper at ICLR 2024
Recrach
SeeaRoood
3.8
Sangderuu
Auda
(a) Pseudo-label quality after stage (b) Final pseudo-lahel quality (c) Sampling rate V.S. training steps
Figure 8: Evaluation of pseudo-label quality and quantity. (a) After stage in SemiReward training, it improves pseudo-label qualities by 1.7-2. 1 %% over FlexMatch on CIFAR-100. Ag News, and UrbanSound-8k datasets. (b) SemiReward improves pseudo-label qualities of final models by 3.1~3.5% over FlexMatch. (c) Class-average sampling rate on STL-10 (40). Despite earning high sampling rate at the very beginning of training in Flex/FreeMatch, it sustains below 90% during training. With SemiReward (SR), the sampling rate rapidly increases and remains 90-95%.
convenience, e.g., Adam (Kingma & Ba, 2014). Therefore, R and G only run forward and backward once for rewarder training in each iteration, which costs ignorable extra overheads in SSL training.
Semi-supervised training Rewarder. In the second stage, the core objective is to optimize fs using R to filter high-quality labels as in Figure 3(b). As fs is continuously optimized on Dr U'Du, R should also be efficiently optimized suppress the confirmation bias in Pseudo Labeling. i.e., fs is easily to overfit to incorrect pseudo-labels. We tackle this dilemma with a simple sub-sampling strategy: we further train R and 9 by Eq. (7) and Eq. (8) with randomly sub-sampled dataset DR C
Dr U Du. where NR = A(N& + NN) and Du is the reliable pscudo-label set selected by DR- We adopt À= = 0.1 by default. This strategy combines two merits: (i) training R can be as fast as the first stage; (ii) similar to 10-fold cross-validation, exploring different subsets to train R avoids overfitting by introducing more randomness. As shown in Figure 8(c), SemiRewarder achieves high sampling rates compared to two confidence-based baselines, which select high-quality pseudo labels after stage in Figure 8(a) and will maintain the high quantity in stage 2 as shown in Figure 8(b).
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Comparison Methods for Classification. In the context of classification tasks, we conducted ex- periments on 10 diverse datasets spanning three distinct modalities to assess the impact of integrating our SemiReward approach. All experiments are based on SSL benchmark USB (Wang et al., 2022a), which implement 14 SSL algorithms, including II model Rasmus et al. (2015), Pseudo Label Lee et al. (2013). Mean Teacher Tarvainen & Valpola (2017), VAT Miyato et al. (2018), MixMatch Berth- elot et al. (2019b), ReMixMatch Berthelot et al. (2019a). UDA Xie et al. (2020a), Fix.atthh Sohn al. (2020), Dash Xu al. (2021), CoMatch Li al. (2021), CRMatch Fan et al. (2021), Flex- Match Zhang et al. (2021), AdaMatch Berthelot et al. (2021), and SimMatch Zheng et al. (2022). We rigorously compare various SSL algorithms from them. Softmatch, Freematch, and Flexmatch, constituting the previous state-of-the-art, dubbed as Previous SOTA. Also, we choose the basic method Pseudo Label (Lee al., 2013; Arazo al., 2020) to illustrate the role of our approach in unlocking potential. Initially, we assess the performance of these algorithms based on classification error rates and training convergence speed, establishing a performance baseline. Subsequently, we can introduce SemiReward into the workflow and conduct a comparative analysis.
Task Settings for Classification. Here are tasks and specific settings on datasets of each modality. More information on datasets and experimental settings are detailed in Appendix A.
(a) For CV tasks, our investigations featured the deployment of renowned and challenging datasets, including CIFAR- 100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 201 1), EuroSAT (Helber et al., 2019), 9), and ImageNet (Deng et al., 2009), with the ImageNet pre-trained Vision Trans- formers (ViT) (Dosovitskiy et al., 2021) randomly initialized ResNet-50 (He et al., 2016) architectures serving as the backbone.
(b) In the domain of NLP, we leveraged 3 datasets, including AG News (Zhang al., 2015), Yahoo! Answers (Chang et al., 2008), and Yelp Review (yel, 2014), employing the self self-supervised pre- trained Bert (Devlin al., 2018) as the backbone.

---

Published as a conference paper at ICLR 2024
Table 1: Top-1 error rate (%), performance gain, and training speedup times on nine SSL classifica- tion datasets with CV, NLP, and Audio modalities in various label settings.
Pseudo Label
FlexMaich
SoflMalchFFreMMtcch
Average
Domain
Datasct (Seting)
Base
+SR
Base
+SR
Base
+SR
Gain
Speed.
ESC-50 (250)
38.42.1085
03.3
56.83 32.58 32.71
29.71=004
+4.11
x1773
ESC-50 (500)
28.92:2024
27.65
27.75
25.92.10.31
29.07
25.98:0.9
+2.06
x2.07
Audio
F DSDnoisy 18k (1273)
34.60
33.24
26.
19.3
26.10=
+1.77
x1.30
UrbanSound8k (100)
37.74:10.6
36.47:66.666
177.8810.1666..06100000
#..6
3.097:1102
+1.93
x1.70
UrhanSound8k (400)
27.45::0006
25.22
23.78
23.7
19.39
+2.30
x1.08
AG News (40)
1.5.19+3.07 13.90=
+1.69
10.67=0..0
+0.93
x2.77
AG News (200)
14.69
12.10
12.08
11.75
10.02:0.82
+1.78
X2.30
Yahoo! Answer (5m0)
14.887=n.:33355088882
33.02+00z
30.92:00.0
+0.99
x x1.80
NLP
Yahoo! Answer (2000)
33.1444..03325500.aa
10.34
29.11
+0,99
x3.53
Yelp Review (250)
13.9
42.68
+2.55
x1.40
Yelp Review (1000)
40.43
38.4
+2.26
X1.01
CIFAR-T00 (200)
$2.78:402233..44:0.077
25.7
23.74::..93
21.07
20.06=
+1.28
X1.04
CIFAR- 100 (100)
25.16::0.7
23.84
17.80
17.59:
15.97
15.62
+0.63
x1.57
STL-10 (40)
20.53++.12
17.37=00.77
11.82+051
10.20
+0.51
9.72+t0.2
+4,19
x1.07
CV
STL-10 (00)
11.25 10.88
7.13
1.59
8.10
7.10:1.9
+0.30
x1.11
Furo-SAT (20)
25.25
23.65
5.54
1.86
=..5
4.22+t0.4
+1.19
x1.03
Euro-SAT (40)
12.82408:
8.33
4.21
3.88
5.46
3.94
+2.21
X1.13
(c) For audio classification, we study the applications of SSL. on 3 datasets, including Urban- Sound8k (Salamon et al., 2014), ESC-50 (Piczak, 2015), and FSDNNisy1 (Fonseca et al., 2019), where Hubert (Hsu et al., 2021) played the role of the pre-trained backbone.
Comparison Methods and Task Settings for Regression. To demonstrate the versatility of our ap- proach, we extend our investigation to regression tasks alongside our primary focus. Specifically, we select Pseudo Label and its counterparts, namely the II model (Rasmus et al., 2015), CRMatch (Fan et al., 10 and Mean Teacher (Tarvainen & Valpola, 2017), as our baseline methods. We then eval- uate their performance in comparison to the integration of RemiReward on 3 regression datasets. The first two datasets, IMDB-WIL (Rothe et al., 2018) and AgeDB (Moschoglou et al., 2017) with only 1 1% labeled data, perform face age regression. Additionally, we conduct rotation angle estimation task using our custom RCF-MNIST dataset (Yao et al., 2022), featuring a more complex CIFAR- 10 (Krizhevsky al., 2009) background to align the samples closely natural images and make the task more difficult. Experimental results are assessed based on two standard regression metrics: Mean Absolute Error (MAE) and Root Mean Square Error (RMSE).
SemiReward Implementations. To train the rewarder R and generator G, we apply Adam (Kingma & Ba, 2014) optimizer with fixed learning rate of 0.0005 in two-stage training for all tasks. We set the scheduler's 's T to 10% of total SSL training iterations. During the inference process of R, we use the average reward score as the threshold 7 to filter pseudo labels dynamically. More specific hyperparameters are provided in Appendix A.2.
4.2 COMPARISON RESULTS ON SEMI-SUPERVISED BENCHMARKS
Results on Classification. Table 1 demonstrates the substantial performance improvements achieved by plugging SemiReward into representative SSL. algorithms across diverse modalities, with notable impacts in audio-related tasks. When augmenting Pseudo Label with SemiReward, outperforms SoftMatch on UrbanSound8k with 100 labeled instances and achieves an average per- formance gain of 4.11 % on ESC-50 with 250 labels. This enhancement effectively guides basic models, e.g., Pseudo Label, toward more favorable local minima. The inclusion of SemiReward consistently expedites model convergence, as evidenced by the "avg. speedup" column in Table 1, with acceleration factors ranging from x1.5 to x3.53 most cases. Total training times are shown in C. Meanwhile, the early stopping technique reduces training costs while maintaining desired performance, representing a valuable trade-off. Furthermore, using SemiReward can reduce train- ing times and achieve lower еπтоr rates on Imagenet, shown in Table 3. Notably, FlexMatch, in conjunction with SemiReward, surpasses previous SOTA methods, such as Freematch and Soft- match. The basic method with consistency regularization, FixMatch, also demonstrates substantial performance improvements when combined with SemiReward.
Results on Regression. We compare CRMatch, Mean Teacher, II model, Pseudo Label, and Pseudo Label added to SemiReward on RCF-MNIST, IMDB-WIKI, and AgeDB. The results are reported in Table 2. From the results of RMSE and MAE, SemiReward has great gain. Especially on RCF.
7

---

Published as a conference paper at ICLR 2024
Table 2: RMSE and MAE, performance gain, and training speedup Table 3: Top-1 error rate (%), times on three SSL regression datasets with 1 1% labels.
performance gain, and train- ing speedup times on ImageNet with 100 labels per class.
RCF-MNIST
IMDB-WIKI
AgeDB
Method
RMSE
MAE
RMSE MAE
RMSE
MAE
Supervised
62.02.1044
22.81
14.22..141..52..00
14.5|
Method
Top-1 Gain
Speedup
Pscudo Label
62.72=a11 23.07
14.90=o22
11.44=051
14.764n12
+055
FixMarch
43.66
+0.00
x1.00
TI-Model
63.24220
23.542063
14.80
11.35=a.1
14.762014
11. 11.92:00
FixMatch+RR
41.72 +1.94
x1.98
MeunTeacher
63,44.0.2
23.25
15.01
11.66.....
1499
12.07.0048
FlexMatch
41.85 +0.00
x0.00
CRMatch
=8.77
20.42.2......0
FreeMatch
40,57
+1.28
x1.50
PseudoLabel-SR
61.71=0.34 22.45
14.8000--00
10.91
14.01
10..2 10.77
SoftMatch
40.52
+1.33
x1.46
Gain
-0.90
-0.99
-0.10
-0.53
-0.75
-0.94
FlexMatch+SR
40.36
+1.49
x2.35
MNIST dataset, SemiReward can yield lower RMSE to 0.9 and MAE to 0.99, which is even better than the supervised baseline. On the contrary, CRMatcc performs poorly on various data sets, inferior to other SSL baselines, indicating the strong effect of confirmation bias.
4.3 ANALYSIS AND ABLATION
This section presents experimental analysis to demonstrate the functionality of SemiReward
Contribution of Each Component. We do extensive ablation experiments and place them in Appendix B and obtain the fol- lowing observations: (i) The number of MLP layers has little impact on the model's performance. The key lies in the design of the attention mechanism. (ii) Table 4 shows that replacing the used MSE (62) loss with BCE loss will make it difficult for the rewarder to converge and achieve poor scoring performance. Also, we find a scheduler that exceeds the reasonable setting range will cause the rewarder to be trapped in the wrong direc- tion. The empirical starting time T can be 10%. (iii) Compar- ing the training objectives of several models, we find that cosine similarity helps form the correspondence between pseudo labels
Ablation of rewarder training. We search the stage-2 start timing T in the two-stage scheduler and losses for Eq. (7)
Table 4:
and Eq. (8) on CIFAR- 100 (400).
Scheduler
Loss
Error
T
MSE BCE
(%)
0%
19.65
5%
17.89
10%
16.65
10%
17.66
15%
16.82
and scores. (iv) Using the mean of reward scores to dynamically adjust the threshold performs much better than a fixed value in Figure A1.
Simplicity of SemiReward. Table 5 shows SemiRe- ward is very streamlined regarding parameters and FLOPs based on ViT-S-P4-32 on the CIRFA-100 dataset. Compared with the student model, our model accounts for very low proportion of the training pro- cess, only requiring 1.28% and 0.056% extra parame- ters and FLOPs and computing two times forward and one times backward propagation in each iteration.
Table 5: Analysis of parameters and
computational overhead (MFlops) of the student model, Rewarder, and Generator.
Model
Params. (M)
FLOPs (M)
Student Model
21.7
607.9
Rewarder
0.140
0.198
Generator
0.137
0.139
Proportion 1.28%
0.056%
Regression Tasks with SemiReward. Existing consistency regularization methods are unsuitable for regression tasks, with CRMatch being the only open-source alternative. However, CRMatch consistently yields subpar results, primarily due to confirmation bias (Arazo et al., 2020). Simulta- neously, we note that in imbalanced regression datasets like IMDB-W and AgeDB, SemiReward encounters challenges in enhancing the selection of superior pseudo-labels, hampering improved model convergence. Conversely, in tasks with balanced data distributions, such as rotation angle estimation, SemiReward demonstrates notably superior performance. This phenomenon may be attributed to the inherent difficulty in accurately labeling data points located at the distribution's 's extremes in imbalanced datasets, leading to partial performance degradation in such scenarios.
RELATED WORK
Pseudo Label (Lee et al., 2013) pioneered the generation of artificial labels for unlabeled data with models trained on labeled data, followed by consistency regularization (Samuli & Timo, 2017) aim- ing to ensure consistent predictions for different views of the same data, which are two foundational techniques in SSL. However, confirmation bias (Arazo et al., 2020; Chen et al., 2022a) caused by inaccurate pseudo labels limits SSL performances. Subsequent works mainly address this problem from three aspects: (i) selecting high-quality pscudo labels, (ii) generating high-quality pseudo la- bels, (iii) enhancing the tolerance of inaccurate labels. View Appendix D) for detailed backgrounds.
8

---

Published as a conference paper at ICLR 2024
Improving Quality of Pseudo-labeling. Confidence-based thresholding techniques (Xie et al., 2020a; Xu et al., 2021) are designed to determine nigh-confidential pseudo labels. FixMatch (Sohn et al., 2020) relies fixed threshold but limits usage of more unlabeled data and leads to imbal- anced pseudo-lahels. Flex (Zhang et al., 2021) employs class-specific thresholds to alleviate class imbalance by reducing thresholds for challenging classes. SoftMatch (Chen et al., 2022c) ex- plores a trade-off between pseudo-label quantity and quality with a truncated Gaussian function to weigh sample confidence. FreeMatch (Wang et al., 2022b) introduces adaptive confidence thresh- olds based on the model's learning state. Moreover, contrastive learning is applied to thresholding methods, e.g., adaptive contraction of the class space in ShrinkMatch (Yang et al., 2023) and the se- mantic similarity for mutual calibration in SimMatch (Zheng et al., 2022). However, these methods broadly enhance classification tasks but are inapplicable in regression tasks. CR-Match (Fan et al., 2021) presents FeatDistl ooss, which also works for regression but does not yield satisfactory results.
Improving Tolerance of Inaccurate Labels. Early SSL models exhibit heightened sensitivity to low-quality pseudo-labels, necessitating the enhancement of the model's error tolerance and label quality. The II model (Rasmus et al., 2015) introduces dual perturbations to input samples, while Temporal Ensembling (Samuli & Timo, 2017) maintains an EMA of label predictions for each train- ing example. Mean Teacher (Tarvainen & Valpola, 2017) takes a step further by averaging model weights, reducing label dependency during training. Meanwhile, another line of research assumes the labeled datasets contain noisy labels and designs robust training strategies to discriminate in- accurate labels (Xu et al., 2021; Li et al., 2019a). Unlike them, SemiReward employs two-stage training approach to learn reward scores, separating rewarder and student model training.
Reward Modeling A reward funetion is crucial in conveying complex objectives to agents in reinforcement learning (RL) (Christiano et al., 2017). Most reward models (Leike et al., 2018) are supervised by classification losses, e-g., ranking loss (Bradley & Terry, 1952), on constructed preference datasets from users. SURF (Park et al., 2022) adopts confidence-based pseudo-labeling RL. Recently, InstructGPT (Ouyang et al.. 2022)
to learn a reward function for preference-basee provided a fine-tuning paradigm for aligning pre-trained large-scale language models (LLM) to human preference. However, reward modeling designed and used for RL optimizations (Schulman et al., 2017) but has not been introduced to SSL scenarios.
6 CONCLUSION AND LIMITATION
Contributions and Social Impacts This paper introduces SemiReward, a general and pluggable framework for SSL scenarios that evaluates and selects high-quality pseudo labels boost the per- formance and convergence speeds of self self-training techniques. The core idea is to select accurate pseudo labels by reward score reflecting pseudo-label quality based on unlabeled data and pseudo labels. To achieve this, a simple but efficient rewarder network is designed to model correlations and predict credible reward scores, which is trained online in a two-stage pipeline assisted by a generator network to avoid confirmation bias. Extensive experiments on diverse classification and regression datasets demonstrate consistent performance gains and convergence speedup when ap- plying SemiReward to popular SSL algorithms. We believe that SemiReward will be regarded as a new paradigm for measuring pseudo-label quality compared to previous confidence-basee strategies and will inspire the SSL community to design effective methods in many application scenarios.
Limitations and Future Works
We hope this work might be valuable and inspire the SSL com- munity and list some limitations and future directions: (1) The defined reward scores and rewarder only support sample-level labels, while fine-grained labels have been widely used in many scenarios requiring token-level rewarding, e.g., object detection (Liu et al., 2021). (2) Despite the rewarder predicting a reliable indicator for high-quality labels, it requires repeating the teacher model and the rewarder several times to get reliable pscudo labels (discussed in Appendix B. B.3). It costs extra com- putational costs and might lead to performance decreasing at the end of training in Figure 2. We may further design more efficient sampling and selection pipeline for SSL training.
(3) In real-world scenarios, it might be useful to pre-train a general rewarder with large-scale pre-trained backbones on open-source datasets (Yalniz et al., 2019). Then, transfer it to specific SSL downstream tasks. (4) In RL scenarios, SemiReward might be useful to popular RLHF (Christiano et al., 2017; Ouyang et al., 2022) and LLM instruction alignment tasks, combining SSL with reward modeling for RL training as Park et al. (2022). (5) Extending SemiReward with adaptive data augmentations, e.g., automatic mixup (Liu al., 2022; Qin et al., 2024), to further enhance SSL performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by National Key R&D Program of China (No. 1022000115100) National Natural Science Foundation of China Project (No. U21A20427), and Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University. This work was done when Weiyang Jin, Zedong Wang, and Fang Wu interned at Westlake University. We thank the Al Station of Westlake University for the support of GPUs and all anonymous reviewers for polishing the writing of the manuseript.
REFERENCES
Yelp dataset. attps://www, htttss//www.yeppoo.cmaa/ttttttt 2014. 6, 16
2014,
Eric Arazo, Diego Ortego, Paul Albert, Noel E OConnor, and Kevin McGuinness. Pscudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020. 6, 8,
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. arXiv preprint 19Xiv:19/100900 2019a. 6, 22, 23
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint
arXiv:1905.02249, 2019b. 6, 22
David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alexey Kurakin. Adamatch: A unified approach supervuper learning and domain adaptation. International Confer- ence on Learning Representations, 2021.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the eleventh annual conference Computational learning theory, pp. 92-100, 1998. 23
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(334))324-35, 1952.
Ming-Wei Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. Importance of semantic repre- sentation: Dataless classification. AAAI Conference on Artificial Intelligence (AAAT), volume 2, pp. 830- 805, 2008. 6, 16
Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self- self-training for cemi-supervised learning. Advances in Neural Information Processing Systems, 01322244-24332 2022a. 5,
Baixu Chen, Junguang Jiang, Ximei Wang, Jianmin Wang, and Mingsheng Long. Debiased pseudo labeling in self-training. arXiv preprint arXiv:2202.07136, 2022b. 23
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the crantity-quality tradeoff in semi- semi-supervised learning. In The Eleventh International Conference Learning Representations, 2022c. 9, 23
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020. 23
Paul Francis Christiano, Jan Leike, Tom B. Brown. Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Pro- cessing Systems (NeurIPS), 2017. 9
Malcom Clark. calibration curve for radiocarbon dates. Antiquity, 49(196):251-26,, 1975. 4,
18
10

---

Published as a conference paper at ICLR 2024
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 115-223. JMLR Workshop and Conference Proceedings, 2011. 6, 16
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.0900 2018. 17
Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Russ R Salakhutdinov. Good semi- supervised learning that requires bad gan. Advances in neural information processing systems, 30, 2017. 24
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (CVPR), pp. 248-255, 2009. 6, 16
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2.18. 2, 6
Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformerr a no-recurrence to-nuueee-- model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884-5888. IEEE, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. 6
Yue Fan, Anna Kukleva, and Bernt Schiele. Revisiting consistency regularization for semi- supervised learning. In DAGM German Conference on Pattern Recognition, pp. 63-78. Springer, 2021. 6, 23
Eduardo Fonseca, Manoj Plakal, Daniel PW Ellis, Frederic Font, Xavier Favory, and Xavier Serra. Learning sound event classifiers from web audio with noisy labels. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 21-25. IEEE, 2019. 7, 16
Yixiao Ge, Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for unsu- pervised domain adaptation on person re-identification. In International Conference on Learning Representations, 2019. 23
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004. 22
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pp. 770-778, 2016. 2,
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 9729-9738, 2020. 23
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12:2217-2226, 2019. 6, 16
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing,
29:3451-3460, 2021. 7
Jiwon Kim, Youngjo Min, Daehwan Kim, Gyuseong Lee, Junyoung Seo, Kwangrok Ryoo, and Seungryong Kim. Conmatch: supervsud learning with confidence -annded consistency reg- ularization. In European Conference Computer Vision (ECCV), 2022.

---

Published as a conference paper at ICLR 2024
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998,, 2014. 6, 7,
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6, 7, 16
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, pp. 896, 2013. 2, 6, 8, 22
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. ArXiv, abs/1811 18111178111, 2018. 9
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-
supervised learning. In International Conference on Learning Representations, 2019a. 9, 23
Junnan Li, Caiming Xiong, and Steven Hoi. Comatch: Semi-supervised learning with contrastive graph regularization. In International Conference on Computer Vision (ICCV), 2021. 2, 6, 22, 23
Siyuan Li, Di Wu, Fang Wu, Zelin Zang, Kai Wang, Lei Shang, Baigui Sun, Haoyang Li, and Stan.Z.Li. Architecture-agnoticc masked image modeling from vit back to cnn. ArXiv, abs/2205. 13943,, 2022. 23
Siyuan Li, Luyuan Zhang. Zedong Wang, Di Wu, Lirong Wu, Zicheng Liu, Jun Xia, Cheng Tan, Yang Liu, Baigui Sun, and Stan Z. Li. Masked modeling for sulf-supe representation learn- ing on vision and beyond. ArXiv, 01222401000997 2023a. 23
Siyuan Li, Zedong Wang. Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan Z. Li. Efficient multi-order gated aggregation network. In International Confer- ence on Learning Representations, 2024.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In International Conference on Learning Representations (ICLR), 2019b. 23
Xuhong Li. Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning (ICML), 2018. 23
Zicheng Li, Siyuan Liu, Zelin Zang, Di Wu, Zhiyuan Chen, and Stan Z. Li. Genurl: A general framework for unsupervised representation learning. IEEE Transactions on Neural Networks and Learning Systems, 2023b. 23
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In International Conference on Computer Vision (ICCV), 2017. 4
Yen-Cheng Liu, Chih-Yar Ma, Zijian He. Chia- Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, 7solt Kira, and Péter Vajda. Unbiased teacher for semi-supervised object detection. In Interna- tional Conference on Learning Representations, 2021.
Zicheng Liu, Siyuan Li, Di Wu, Zhiyuan Chen, Lirong Wu, Jianzhu Guo, and Stan Z. Li. Automix: Unveiling the power of mixup for stronger classifiers. In European Conference Computer Vision, pp. 441-458, 2022. 9
Zicheng Liu, Siyuan Li, Ge Wang, Cheng Tan, Lirong Wu, and Stan Z. Li. Harnessing hard mixed samples with decoupled regularizer. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 22
Iyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0399 2016. 17
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence (TPAMI), 1979 2018. 6, 24
12

---

Published as a conference paper at ICLR 2024
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In proceed- ings of the IEEE conference computer vision and pattern recognition workshops, pp. 51-59, 2017. 7, 16, 17
Augustus Odena. Semi-supervinn learning with generative adversarial networks. arXiv preprint 166.v:1606.01583, 2016. 24
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in Neural Information Processing Systems
(NeurlPS), 35:27730-27744, 2022. 4,
Jongjin Park. Younggyo Sco, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-effiiie preference-base, reinforcement learning. In International Conference Learning Representations (ICLR), 2022.
Sungrae Park, JunKeon Park, Su-Jin Shin, and II-Chul Moon. Adversarial dropout for supervised and supersupe learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 24
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11557-11568, 2021. 2, 23
Karol Piezak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015. 7, 16
Huafeng Qin, Xin Jin, Yun Jiang, Mounîm A. Yacoubi, and Xinbo Gao. Adversarial automixup. In International Conference on Learning Representations (ICLR), 2024.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. Advances in neural information processing systems, 28, 2015. 6, 7, 9, 23
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. In 2005 Seventh IEEE Workshops Applications of Computer Vision. Carnegie Mellon University, 2005. 22
Rasmus Rothe, Radu Timofte, and Lue Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2-4): 144-157, 2018. 7, 16, 17
Sebastian Ruder and Barbara Plank. Strong baselines for neural sumi-super learning under domain shift. In The 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2018. 23
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041- 1044, 2014. 7, 16
Laine Samuli and Aila Timo. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations (ICLR), volume pp. 6, 2017. 8, 9, 22,23
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv,
2017. 4,
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying supervised leam- ing with consistency and confidence. Advances in Neural Information Processing Systems (NeurIPS), 2020. 2, 6, 22
13

---

Published as a conference paper at ICLR 2024
Jong-Chyi Su and Subhransu Maji. The semi remi-supervised inaturalist-aves challenge at fgye? work- shop. In proceedings of the IEEE conference on computer vision and pattern recognition work- shops, 2020. 17
Teppei Suzuki and Ikuro Sato. Adversarial transformations for semi-supervised learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5916-5923, 2020. 24
Cheng Tan, Jun Xia, Lirong Wu, and Stan Li. Co-learning: Learning from noisy labels with self-supervision. In Proceedings of the 29th ACM International Conference on Multimedia, pp-
1405-1413, 2021. 23
Cheng Tan, Zhangyang Gao, Lirong Wu, Siyuan Li, and Stan Z Li. Hyperspherical consistency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7244-7255, 2022. 23
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. In 31st Conference on Neural Information Processing Systems (NeurIPS), 2017. 2, 93
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang. Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. Usb: A unified semi-supervised learning benchmark. In Neural Information Processing Systems (NeurIPS), 2022a. 1, 6, 17
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, el. Freematch: Self-adaptive thresholding for semi- supervised learning. In The Eleventh International Conference on Learning Representations, 2022b. 2, 23
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data augmenta- tion for consistency training. In Advances in Neural Information Processing Systems (NeurIPS), 2020a. 2, 3, 6, 9, 17,21 22
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10687-10698, 2020b. 2, 23
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 23
Wang Ximei, Gao Jinghan, Long Mingsheng, and Wang Jianmin. Self-tuning for data-efficient deep learning. In International Conference on Machine Learning (ICML), 2021. 23
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine
Learning (ICML), pp. 11525-11534 11536. PMLR, 2021. 6, 9, 23
Ismet Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Kumar Mahajan. Billion- scale semi-supervised learning for image classification. ArXiv, abs/ abs//995.0055. 900540, 2019. 2, 9, 23
Lihe Yang, Zhen Zhao, Lei Qi, Yu Qiao. Yinghuan Shi, and Hengshuang Zhao. Shrinking class space for enhanced certainty in semi-supervised learning. arXiv preprint 27Xiv22308.06777, 2023. 9,
23
Huaxiu Yao, Yiping Wang. Linjun Zhang, James Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. Advances in Neural Information Processing Systems, 35:3361-3376,
2022. 7, 16, 17
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd
annual meeting of the association for computational linguistics, pp. 189-196, 1995. 22
14

---

Published as a conference paper at ICLR 2024
Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 23
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo la- beling. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2, 6, 9, 17, 20, 22
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, volume 28, 2015, 6, 16
Mingkai Zheng, Shan You, Lang Huang. Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi- supervised learning with similarity matching. arXiv preprint arXiv:203.006115, 2022. 6, 9, 22, 23
Zhi-Hua Zhou and Ming Li. Semi-supervised learning by disagreement. Knowledge and Information Systems, 24:415-435 2010. 2, 23
15

---

Published as a conference paper at ICLR 2024
APPENDIX
The appendix is structured as follows:
(A) In Appendix A, we provide implementation details are provided including dataset settings, hyperparameter settings, and training schedule.
(B) In Appendix B, we describe extensive ablation studies presented analyzing the impact of differ- ent architectural choices, training techniques, and loss functions.
(C) In Appendix C, we provide additional experimental results, including detailed training time statistics across different datasets and settings.
(D) In Appendix D, we further provide extensive related work to highlight connections and differ- ences to the proposed approach.
(E) In Appendix E, we provide pseudocode for training pipelines SemiReward.
A
IMPLEMENTATION DETAILS
A.1 DATASET SETTING
For fair comparison, train and evaluate all methods with the same VIT backbones and hyper- parameters in Table A3. As for CV, we evaluate SemiReward on common benchmarks: CIFAR- 100 (Krizhevsky al., 2009), Euro-SAT (Helber al., 2019), STL-10 (Coates et al., 2011), I), and ImageNet (Deng et al., 2009) for image modality. Euro-SAT contains Sentinel-2 satellite images covering 13 spectral bands, which not a natural image dataset as the other three. As for NLP, AG News (Zhang et al., 2015) (news topic material), Yahoo! Answer (Chang et al., 2008) (topic classification), and Yelp Review (yel, 2014) (sentiment classification) to evaluate SSL algorithms on more fine- -ra-igr sentiment NLP classification tasks. For audio classification, we choose Urban- Sound8k (Salamon et al., 2014) with maximum length seconds, ESC-50 (Piczak, 2015) with a maximum length of 5 seconds, and FSDNoisy 18k (Fonseca et al., 2019) with the length between 3 seconds and 30 seconds.
Table Al: Settings and details classification datasets in various modalities.
Domain
Dataset
#Label per class
#Training data #Validation data
#Test data
#Class
CIFAR-100
2/4
50,000
10,000
100
CV
STL-10
4/ 10
5,000 100,000
8,000
10
EuroSat
2/4
16,200
5,400
10
ImageNet
100
1,28,
5,0000
1000
Yelp Review
50/200
250,000
25,000
50,000
NLP
AG News
10/50
100,000
10,000
7,600
Yahoo! Answer
/0/
500,000
50,000
60,000
ESC-50
5/ 10
1,200
400
400
Audio
UrbanSound8k
10/40
7.079
816
837
10
FSDnoisy188 88
52-171
1,772 / 15,813
947
20
Table A2: Settings and details of regression datasets CV.
Domain
Dataset
Task
#Label arrange
#Training data
#Validation data
RCF-MNIST
Rotation
360
50,000
10,000
CV
IMDB-WIKI
Face age
1, 101
167,562
23,938
AgeDB
Face age
[1, 101
106,750
15,250
We conducted age regression experiments on two datasets, IMDB-W WIKI (Rothe et al., 2018) and AgeDB (Moschoglou et al., 2017) with 1% labels. AgeDB contains images of various celebrities, such as actors, writers, scientists, and politicians, and each image is annotated with identity, age, and gender attributes. The minimum and maximum ages are 1 and 101, respectively. The IMDB-WIKS dataset contains around 167,562 face images. Each image has an age and gender label associated with it, and the age range is 101. The task here is to extract human features so that the model returns a continuous real value to predict age. Furthermore, we performed a rotation angle esti- mation task background CIFAR-10 10 (Krizhevsky et al., 2009), rather than the simple three-color backgrounds, to align the dataset's s images more closely with natural images and make it more difficult. This dataset
custom RCF-MNIST (Yao et al., 2022) dataset, which features
intricate
16

---

Published as a conference paper at ICLR 2024
can be solved with rotation features of objects except for the background image, allowing the model to regress a rotation angle of the foreground object.
A.2 HYPERPARAMETER AND TRAINING SETTINGS
Basic Settings. As for classification tasks, regarding hyperparameter settings of SSL classifica- tion benchmarks constructed USB (Wang et al., 2022a), we adopted the original settings with pre-trained Transformers as the backbone and made few adjustments to adapt SemiReward, as shown in Table A3. The total training iterations are set to 2 and an early stop technique used for calculating the convergence times. Meanwhile, we use the full experimental settings in Flex- Match (Zhang et al., 2021) for ImageNet, which uses 100 classes per class with ResNet-50 as the backbone. All methods are trained from scratch by SGD (Loshchilov & Hutter, 2016) optimizer with a momentum of 0.9, a basic learning rate of 0.03, and cosine learning rate decay as USB. Note that Semi-AVEs (Su & Maji, 2020) uses 224 x 224 input resolutions and ViT-S-P16- PP-6-22 with the labeled and unlabeled batch size of 32, and other settings are the same STL-10. We apply & loss as the basic regression loss. As for regression tasks, we follow CV settings in USB to construct similar experiment settings for IMDB- WIKI (Rothe al., 2018) (224 x 224 resolutions AVES in USB), AgeDB (Moschoglou et al., 2017) (as Semi-AVES), RCF-MNIST (Yao et al., 2022) (32 x 32 resolutions CIFAR-100), All experiments implemented with PyTorcc and run on NVIDIA A100 GPUs, using 4GPUs training by default.
Semi-
Table A3: Mlyper-parameee and training schemes of SSL classification tasks based on USB.
Domain
CV
NLP
Audio UibanSouxd8k FSDNoisy
Dataset
CIFAR-100
STL-I0
Baro-SAl
News Yahoo! Yolp-3
ESC-50
Image Size
32
96
32
Max Length
$12
4.0
5.0
5.0
Sampling Rate
16,000
Model
M4 32 V.T.B. P16-96 VT.5SP4.32
BERT Base
HuBERT-Ban Base
Weight Decay
lo-4
Se-4
Laheled Batch size
16
8
Unlabeled
Learting Rate
Sc-4
1c-4
50-5
5o-5
lc-4
5o-5
5c-3
5c-4
10-4
Layer Decay Rate
0.5
0.68
0.65
0.75
0.75
0.75
0.85
Scheduler
coel TEA
Model EMA Momentum
0.999
Eval EMA Momeutum
0.999
Augmeniation
Random Crop, Random Horizontal Flip
sample
Strong Augmentation
RandAngment(Cuuuu et 2018)
Back- Tramstaitn (Xic et al. Randomm Sub-sample, Gain, Pitch, Speed
Table A4: Hyper-parameters and training schemes of SemiReward for various tasks and modalities.
Hyperparameter
Classification
Regression
CV NLP Audio
RCFMNNIS IMDB-WIKI AgeDB
Threshold r
Average
Top-k
Average
Optimizer
Adam
Learning rate
0.0005
Loss Embedding dim.
MSE
128
MLP Layer-number
Schedule T
10% total iterations
Sun-sampling A
0.1
SemiReward Settings. In Table A4, provide detailed hyper-parameters and settings for SemiReward training. The two-stage online training of the rewarder R and generator G is trained by Adam (Kingma & Ba, 2014) optimizer with learning rate of 0.0005 for all tasks, independent of the student model's optimization. For each training step after T iterations, R. infers once and selects high-quality pseudo labels for the student with the average reward score as the threshold T, except for using top-k highest pseudo-labels for RCF-MNIST with k = 16. The generator G utilizes a 4-layer MI.P (only containing FC layers and ReL.U) with 256, 128, and 64 hidden dimensions.
B ABLATION STUDY DETAILS
B. CALIBRATION CURVE
To explore the properties of our proposed reward score Sec. 3.1, we visualize the correlation between ground truth or leamed reward scores and the quality of pseudo labels according to the
17

---

Published as a conference paper at ICLR 2024
concept of calibration curves (Clark, 1975) in Figure 4 and Figure 5. Our goal in Eq. 4 is to learn a mapping from pseudo-labels to scores, which can be approximately linear or positively correlated and will have good discrimination and reliability in evaluating pseudo-label qualities. The data will not be classified as particular points within a small range, leading to excessive random error interference.
Therefore, we plan to analyze the reward score from four aspects: the threshold of reward score screening, the pseudo-label accuracy, and the confidence of the reward score. Based the direct proportional relationship, we explore whether the model can achieve the required effect under differ- ent module designs and use this to illustrate through ablation experiments and theoretical analysis. Concretely, for each pseudo label that passes through the rewarder, we will return a correspond- ing score value and calculate the accuracy of the pseudo-labels after different thresholds by setting thresholds ds for different score values to draw graph. In Figure 4(a), when calculating similarity, there is sudden accuracy drop near threshold value close This is caused by adding epsilon in numerical calculations to prevent division by zero errors in PyTorch implementation.
B.2 NETWORK ARCHITECTURE OF REWARDER
From the model design perspective, our rewarder network mainly incorporates cross-attention mechanism extract the information interaction between labels and data. On the other hand, uses several layers of MLP to deepen feature processing further. Therefore, we conducted ablation experiments on CIRFA-100 with 400 labels to explore the impact of these mechanisms.
As presented in Table A5, we find that the incorpo- ration of the cross-attention mechanism within the architectural module exerts profound influence on both the pace of convergence and the ultimate ef- ficacy of the model. Subsequent to the integration of a more profound MLP, the performance of the rewarder in the context of SemiReward exhibits no statistically significant enhancements. Instead, is discernible that the augmentation has engendered a deceleration in the training process. Drawing upon our meticulous calibration curve analysis in Figure and Figure 5, it becomes readily apparent that the attention mechanism assumes a paramount role in evaluating the intrinsic performance of the rewarder model and the holistic training regimen. Its profound impact manifest in the capability to orchestrate seamless and continuous spectrum for score mapping of pseudo labels, opposed to engendering numerous isolated points that could precipitate a distortion the alignment between accuracies and reward scores.
Table A5: MLP number stands for the FC layers the rewarder.
attention MLP Accuracy(%)
iteration
83.35
100352 iters
83.26
129024 iters
83.32
145408 iters
81.99
194559 iters
2
82.20
194559 iters
3
82.25
204799 iters
B.3 SCHEDULER FOR SEMIREWRD
We conducted experiments on the CIFAR- 100 with 400 labels and ESC-50 with 250 labels datasets. The model training effects under different start timings were counted. Start timing represents the time node from pre-training (stage-1) to semi-supervised training (stage-2) of the rewarder, indicat- ing that SemiReward will utilize high-quality pseudo labels to ensure the further convergence the student model and itself. This is what define as SemiRewards scheduler.
Table A6: The starting time is the comparison of the round in which training starts to the total rounds. At the same time, we measured the convergence time and accuracy.
CV
Audio
Start Timing
Accuracy($5)
lteration
Accuracy(%5
iteration
0%
80.35
159743 iters
62.86
96255 iters
5%
82,11
169984 iters
65.59
65535 iters
10%
83.35
100352 iters
67.42
38911 iters
15%
83.18
174080 iters
67.10
69631 iters
It can be seen that when switching at 0%, the model achieves poor results the two data sets. However, there is experience value in the range of 5%-15%, and the robustness to nodes is main- tained. In fact, for the scheduler selection of the model, the intuitive understanding is that turning
18

---

Published as a conference paper at ICLR 2024
out of range is more likely to produce poor results. This is because premature means that the pre- training phase has not been completed, causing problems with the score mapping during the initial screening and causing subsequent online training to learn worse score targets. Too late will make the model converge slowly and easily fall into local optimality, making it difficult to achieve favorable performance in the early stage.
For the screening phase, we employed a multi-forward approach to generate multiple pseudo-labels for given dataset, facilitating iterative screening. The parameter de- cay denotes the frequency of forward passes. the sub- sequent stages, we introduced an annealing strategy. dy- namically adjusting decay throughout the training process. Specifically, we divided the total training steps by the cur- rent iteration, rounding up the result as the updated num- ber of forward passes. To underscore that the performance enhancement of our algorithm extends beyond the impact of decay alone, augmented the baseline algorithm with multiple forward passes and conducted comparative experiments A7. Our findings revealed that the algorithm achieves peak performance when decay and reward-based screening collaborate.
Table A7: Analysis of selecting pseudo labels on CIFAR-100 (400 la- bels) with or without decay. Top-1 ac- curacy (%) and the training speedup times are reported.
Method
FlexMatch
Baseline
82.12 (x1.0)
+Decay
79.42 (x1.4)
Semireward
82.90 (x2.7)
+Decay
(x222)
B.4 LOSS FOR SEMIREWRD
In the ablation experiment, we not only compared the results of replacing MSE (f2) loss with BCE loss. We also changed the algorithm of SemiReward total loss. Initially, two independent losses were used for gradient backpropagation, but we also considered the impact of weighting on the overall model training. We conducted ablation experiments on CIRFA-100 with 400 labels to compare their difference and find that the proposed MSE loss yields the best results.
As shown in Table A8, we can find that the weighted loss is more negative for model train- ing, which may cause the rewarder to not con- verge and introduce many low-quality labels into the training process. Therefore, the impor- tance of independent loss design can be seen here. On the other hand, BCE loss is also diffi- cult train the rewarder to convergence. This may be because our scoring model essentially follows the idea of regression tasks.
Table A8: Analysis of the loss types and loss
weight for the proposed reward loss.
MSE BCE Weighted Accuracy(%)
iteration
83.35
100352 iters
0.1
80.99
204799 iters
0.5
81.25
204799 iters
0.9
79.85
204799 iters
82.34
153600 iters
0.1
80.02
196608 iters
0.5
81.11
194559 iters
0.9 81.01
196608 iters
B.5 TARGET FOR SEMIREWRD
As for the reward score, i.e., the target of the rewarder model, its distance measurement is essential. We pursue that the scored pseudo-standards can be distributed evenly on the accuracy-score mapping with favorite properties mentioned in Sec. 3.1. Therefore, we constructed different score labels using different distance measures to train the rewarder and inferred why cosine similarity is an acceptable distance measure. We conducted ablation experiments on CIRFA-100 with 400 labels to compare the differences. As analyzed of Sec. 3, it can be seen that the divergence method represented by JS
Table A9: Analysis of the impact of training scoring targets calculated using different distance metric methods on the model, including using L2 distance and cosine similarity or not in SemiReward.
Cosine Similarity
L2 Distance
Accuracy($)
iteration
83.35
100352 iters
80.23
202751 iters
82.25
204799 iters
divergence has serious failures in the thinking of the calibration curve. This is because JS divergence may cause the scores of some tags to be too concentrated that bad labels with similar scores will be selected as reliable labels. In Table A9, we found that the target score derived from the negative La distance will cause the filtering ability of the rewarder to decline rapidly that many low-
19

---

Published as a conference paper at ICLR 2024
quality labels are selected, causing the training process of the student model trapped in relatively low accuracy.
B.6 THRESHOLD FOR SEMIREWRD
In Appendix AI, we ablate the thresholding strategy for SemiReward, which compares the average thresholding with several fixed threshold T settings, including 0.5, 0.7, and 0.9. The red dotted line denotes the result of the average strategy. In the context of reward score threshold-basee filtering, it becomes evident that the fixation of this threshold engenders a multitude of challenges. During the training of SSL., employing a static threshold for pseudo-label selection poses prominent challenges (Zhang et al., 2021). During the early epochs of training, a model is still in its nascent state of un- derfitting and unstable. Setting high threshold during these phases can inadvertently discard sub- stantial portion of potentially informative pseudo-labels. Such an action can curtail the model's 's abil- ity to learn from these early indicators, potentially decelerating the overall convergence trajectory. Conversely, as training progresses and the model refines its internal representations, static low threshold may fall short in filtering out subpar-quality pseudo-labels. This introduces the hazard of the model overfitting these less reliable markers, jeopardizing its generalization capabilities. We advocate for dynamic thresholding strategy grounded in averaging princi- ples to address these challenges. Instead of adhering to rigid threshold, our approach recalculates the threshold value within each mini-batch, considering the current quality distribution of the pseudo-labels. Such a mechanism ensures consistent retention of high- high-quality pseudo-labels throughout the train- ing lifespan while effectively sidelining low-quality ones. Our empirical evaluations underline the efficacy of this method, not only amplifying the model's rate of convergence but also bol- stering its performance on out-of-sample evaluations.
11.6
Fiod Tholda
Areage Thaaea
Figure Al: Thresholding for ward scores with adding SemiRe- ward to FlexMatch CIFAR-100 with 400 labels.
Method
Flex
Coupled Training
(x1.0)
+Gradient Ascent
82.23 (x1.2)
Decoupled Training
(x2.2)
Chanded
Lake
+Gradient Ascent
83.25(x1.7)
Lanch
Table A10: Analysis of two training
(a) Coupled V.S. Decoupled
(b) Pscudo labels v.s. Fake labels
processes and the gradient accent of Figure A2: Analysis of SR training on CIFAR- with Flex- pseudo labels on CIFAR-100 (400 Match. The mean and std of top-1 accuracy are plotted for (a) labels). Top-1 accuracy (%) and the pseudo labels for the coupled and decoupled training and (b) training speedup times are reported. pseudo and fake labels in the decoupled training.
B.7 DECOUPLING OF STUDENT AND REWARDER TRAINING
As discussed in Sec. 3.2, we decouple the training of the student model and the rewarder by intro- ducing the Generator and two-stage training pipeline to prevent confirmation bias. Here, we analyze the two training processes to verify whether the decoupled two-stage training with the Generator is an essential design. The first type of training process is to optimize the student and the Rewarder together without the Generator, where the teacher model generates candidate pseudo labels for the student and the Rewarder, which we call coupled training. Contrastively, the proposed two-stage training the decoupled training. There are two reasons for decoupling the training process of the student and the Rewarder. Firstly, the Rewarder requires diverse pseudo-labels as the training data to fit the ground truth reward scores rather than deterministic high-peeformance labels. Secondly, the student and the Rewarder might suffer from confirmation bias. To further enhance the gener- ated pseudo labels for the student training, we also designed a gradient ascent trick. Given selected reliable pseudo labels, we can modify them to generate more high-quality pseudo labels (or fake labels) by maximizing the reward scores with a step of gradient ascent in the inference process of the Rewarder.
20

---

Published as a conference paper at ICLR 2024
Table All: Training times and the average speedup times on nine SSL classification datasets with
CV, NLP, and Audio modalities in various label settings.
Pscudo Label
FlexMatch
SoftMatch//rreeMacchh
Modality
Dataset (Setting)
Avg. Speedup
Base +SR
Base
+SR
Base
+SR
ESC-50 (250)
5.700
7.125x088
10.053
3.142x32
9.100
7.583x
x1.73
ESC-50 (500)
6,750 3.214x2.1
10.806
4.912x2a
10.751
5.658
x2.07
Audio
FSDnoisy 8k (173))
7,467
8,297x0,9
12.133
8.089x15
11,467
7.645x15
x1.34
UrbanSound8k (100)
5.250
5,833x00
4.728
1.525x31
6.167
5.606x11
x1.70
UrbanSound8k (100)
4.217
6.024
2.833
2.361x12
3,033
x.1
x1.08
AG News (40)
2.400
x.
6.267
x.333x4.7
13.333
6.060x22
X2.77
AG News (200)
2.889
1.699
3.556
1.693
4.444
x,434x31
X2,30
Yahoo! Answer (500)
0.178
0.445
8.711
5.807x1.5
9.000
2.571
x1.80
NLP
Yahoo! Answer (200)
8.689
1.889
8.122
1.692x48
9.919
8.266x
x3.53
Yelp Review (250)
22,400
22.400xx..
20.066
20.066x1.0
22,400
10.667 x211
x1.39
Yelp Review (1000)
1.822
x.673
21.411
16.470x1.a
19.133
16.394
x1.00
CIFAR- 100 (200)
9.320
x1.314x0.5
54.280
49.345
54,889
49.899
x1.04
CIFAR- 100 (100)
14.920
13.564
100.240
45.564x2.22
94.044
x7.174x114
x1.57
STI,-10
0.528
x.320
T1.760
8.400
19.360
15.600
x1.07
CV
STI-10 (40)
0.268
0.693
9.556
7.351 x.3
20.267
13.889
x1.11
Euro-SAT (20)
1.196
5.980
14.320
17.900
10.755
5.121 x2.1
x1.03
Euro-SAT (40)
1.092
5.460x0.2
21.040
23.378
16,800
7.304 x23
x1.13
Table A12: Top-1 error rate (%), performance gain, and training speedup times additional SSL classification datasets with CV and NLP modalities in various label settings.
Pseudo Label
FlexMatch
SoftMatch/FreeMMttch
Average
Domain
Dataset (Setting)
Base
+SR
Base
+SK
Base
+SK
Gain
Speed.
Amazon Review (250)
53.45 49.13 45.73 45.733.075773..00
43,08 2.1 45.29
42.98
+3.09
x2.59
NLP
Amazon Review (1000)
47.00=
44.21 42.25 41.11 42.21 39.17=a0.2
+2.32
x2.92
Semi Aves 3959 (3959)
40.35
37.93
32.85
31.02
+1.82
x2.01
CV
Tissuemnist (80)
56.92:44.4
58.36238 54.27
58.24 23.00 53.52:1107
+4.22
x1.92
As shown in Table B.6, when using the coupled training of the student and the Rewarder, Flex- Match+SR yields worse performance than the baseline (82.12 vs. 82.20), and FlexMatch+SR with the gradient ascent can only obtain a limited performance gain and speedup over the baseline. As shown in Figure 2(a), selected pseudo labels the coupled training are unstable and affected by the student model, while the decoupled training produces high-quality pseudo labels steadily. Mean- while, the proposed two-stage training decouples the student and the Rewarder by the Generator (aiming to maximize the reward score). It achieves great trade-off between performance gains and speedup. Further applying the gradient ascent to decoupled training will yield a little per- formance gain with more extra computational costs and cause unstable training. As shown in Fig- ure 2(b), the quality of fake labels relatively diverse, and it is difficult to obtain high-quality labels steadily. Therefore, we intend to use the decoupled training process without the gradient ascent trick as the final design.
C EXTENSIVE EXPERIMENT RESULTS
C.I DETAILS IN SPEEDUP
In Sec. 4, we give the average speed gain but not the specific training time. Table A1l gives the different training times corresponding the nine of data in the three modes in the main text. We stipulate that the calculation is on single NVIDIA A100 GPU to carry out relevant statistics, and the reported unit is the total hours.
C.2 CAPACITY SEMIREWARD
From Table and Table AII. In few situations, SemiReward did not reach full convergence in shorter time frame for primitive SSL algorithms like Pseudo Label, especially when evaluated on certain datasets such as STL-10 and SATT-SAT. This may be attributed to the simplicity of those basic methods like pseudo-labeling and entropy regularization in SSL tasks, which do not guide the model effectively towards better local minimum. In contrast, SemiReward compensates
21

---

Published as a conference paper at ICLR 2024
for these shortcomings and unveils the potential of unlabeled data, allowing the model to progress toward better local minima, albeit requiring more time. This represents a trade-ofi off and specific decisions about early stopping times for the optimal balance between speed and quality.
Table A13: Top-1 error rate (%), performance gain, and training speedup times on SSL classification datasets with CV in more label settings.
Domain
CIFAR- (1000)
CIFAR-100 (200)
CIFAR-10K 100
(1000)
Average Speed.
Flexmatch
+SR
Flexmatch
+SR
Flexmatch
+SR
Gain
11.191079 9.24
10.82. 9.42..066
10.22
8.99104:
+1.29
x1.71
Table A14: Top-1 I accuracy rate (%) and performance gain on ImageNet with 1% and 10% labels.
Label Settings
FixMatch
CoMatch
SimMatch
Average
Dataset
Base
+SK
Base +SR
Base
Gain
1%
53.5
55.1
66.0
67.4
67.2
+1.5
Imagenet
10%
71.6
72.8
73.6
74.5
74.4
+1.1
C.3 RESULTS FOR ADDITIONAL DATASETS AND MORE LABEL SETTINGS
Due to the relatively antiquated nature and lower quality of the STL-I0 dataset, our approach did not achieve optimal mean gain while emphasizing speed and lightweight characteristics. This can be attributed to the fact that we selected different random seeds multiple times, resulting in varied averages. Consequently, we have supplemented our study with datasets from the CV and NLP domains that exhibit superior performance augmented the relevant tasks, as illustrated in A A13, with the ImageNet pre-trained Vision Trans- formers (ViT) architecture serves as the backbone. Additionally, we have supplemented the data results for 1% and 10% labeled datasets (i.e., 13 and 128 labels per class) in A14. We find that applying the proposed SemiReward (+SR) upon FixMatch (Sohn et al., 2020) and CoMatch (Li et al., 2021) can achieve around 1.3% performance gains, and SoMatch+ outperforms the current SOTA SimMatch (Zheng et al., 2022)).
A A12. In several settings of CIFAR-100, we have
D EXTENSIVE RELATED WORK
D.1 SEAFINNAINING
In semi-supervised learning (SSL.), self-training frameworks (Rosenberg al., 2005; Grandvalet & Bengio. 2004; Yarowsky, 1995) play a very important role in unlabeled data utilization. Then, pseudo-labeling (Lee al., 2013), as one of the classic self-training ways, pioneered the generation of artificial labels for unlabeled data. However, this embodiment faces the need for high-quality labels due to the problem of confirmation bias (Arazo et al., 2020). Subsequent work will mainly address this problem from two perspectives: one is to design a class or combine multiple methods to improve the quality of pseudo-label generation and application, and the other is to consider enhanc- ing the network's 's acceptance of pseudo-labels, that is, small number of low-quality pseudo-labels will not affect the overall prediction of the network.
Consistency Regularization. Samuli & Timo (2017) first proposed consistency regularization to ensure consistent predictions for similar data points, which has become basic method for gener- ating high-quality pseudo labels. Based on this, MixMatch (Berthelot et al., 2019b) and its vari- ants (Berthelot et al., 2019a; Liu et al., 2023) performs data augmentation on unlabeled data, in- puts multiple data into the same classifier, obtains different predicted classification probabilities, and uses a class method to make the average variance of multiple probability distributions smaller. UDA (Xie et al., 2020a) goes a step further and starts to use two branches of weak and strong aug- mented samples and regards the predictions of the weak augmentation branch as the target of the strong augmentation branch to improve the consistency of the pseudo-label and predictions. After that, ReMixMatch (Berthelot et al., 2019a) uses the distribution alignment method to encourage the marginal distribution of predictions for unlabeled data to be close to the marginal distribution of ground truth labels. Fixmatch (Sohn et al., 2020) designs fixed confidence threshold to filter pseudo labels so that the high-quality pseudo-labed can be used in the SSL training process. The following works, like Fltx (Zhang al., 2021), deeply explore the idea of confidence thresholds and
22

---

Published as a conference paper at ICLR 2024
propose curriculum learning to dynamically adjust the thresholds generated by pseudo labels based on the training process. Additionally, softmatch (Chen et al., 2022c) shows the trade-off between the quantity and quality of pseudo labels and also derives a truncated Gaussian function to weight sample confidence. Freematch (Wang et al., 2022h) proposes a free matching method that adaptively adjusts confidence thresholds based the model's learning state. The above methods essentially follow the strategy of training teacher-student distillation. Even the most advanced methods still rely on the manual design of confidence thresholds for screening. Although Meta Pscudo Labels (Pham et al., 2021) proposes to generate more accurate pseudo labels with meta learner through bi-level optimization, it doubles training times and requires large-scale teacher models. This why we proposed SemiReward as simple but efficient solution for pseudo-label selection.
Tolerance to Inaccurate Pseudo Labels. Early SSL models have certain sensitivity to low- quality pseudo labels. Then, another aspect of work starts by improving the model's tolerance to errors low-quality labels. II-Model (Rasmus et al., 2015) adds two different perturbations to an input sample, inputs the network twice to get the result, and then compares the consistency of the two results. This weakens the impact of low-quality labels but may be less efficient since two for- ward propagations are required to calculate the loss. Based on this. Temporal Ensembling (Samuli & Timo, 2017) maintains an EMA of label predictions on each training example and penalizes pre- dictions that are inconsistent with this goal. Mean Teacher (Tarvainen & Valpola, 2017) further averages model weights instead of label predictions. This allows the use of fewer labels than se- quential integration during training and also improves the accuracy of testing. Meanwhile, another branch of research assumes the labeled datasets are noisy and designs robust training or ad-hoc label selection policies to discriminate inaccurate labels (Xu et al., 2021; Li et al., 2019a; Tan et al., 2021).
D.2 DISAGREEMENT BASED MODELS
From the view of disagreement SSL., it is required to train two or three different networks simulta- neously and label unlabeled samples with each other (Zhou & Li, 2010) so that they are less affected by model assumptions and loss functions. Co-training (Blum & Mitchell, 1998) assumes that each data point has two different and complementary views, and each view is sufficient train good classifier. Noisy Student (Xie et al., 2020b) is assigned pseudo-labels by a fixed teacher from the previous round, while (Yalniz et al., 2019) scales up this training paradigm to billion-scale unlabeled datasets. MMT (Ge et al.. 2019), DivideMix (Li et al., 2019a) learn through multiple models or clas- sifiers through online mutual teaching. Multi-head Tri-training (Ruder & Plank, 2018) uses training to learn three classifiers from three different training sets obtained using bootstrap sampling. In these methods, each classifier head still trained using potentially incorrect pseudo-labels generated by other heads. Afterward, the classifier for pseudo-labels generated by DST (Chen et al., 2022b) is trained with unused pseudo-labed labels,, thus having better tolerance to inaccurate pseudo-labels.
D.3 SELF-SUPERVISED LEARNING FOR SSL
Self-supervised learning Xie et al. (2022); Li et al. (2023b; 2022; 2023a) techniques like contrastive learning (CL) approaches (Chen et al., 2020; He et al., 2020) are also widely applied to SSL, such as CoMatch (Li et al., 2021) that first introduced CL to the consistency regularization framework. ShrinkMatch (Yang et al., 2023) allows the model to search for contracted class space adaptively. In detail, for each uncertain sample, ShrinkMatch dynamically defines shrunk class space, includ- ing the original top-1 class and less likely classes. Similarly, SimMatch (Zheng et al., 2022) uses semantic and instance similarity for mutual calibration. It uses the labeled data to train seman- tic classifier and uses this classifier to generate pseudo labels for the unlabeled data. Meanwhile, ReMixMatch (Berthelot et al., 2019a) and CR-Match (Fan et al., 2021) utilize rotation prediction as the auxiliary task for SSL. Moreover, fine-tuning pre-trained model on labeled datasets is a widely adopted form of transfer learning (TL), and several recent works (Li et al., 2018; 2019b; You et al., 2020: Ximei et al., 2021) like Self-Tunine (Ximei et al., 2021) combining TL with SSL methods Self- Tuning (Ximei et al., 2021) 1) and HCR (Tan et al., 2022) introduce CL pre-tedined models as the regularization mitigate confirmation bias in TL.
23

---

Published as a conference paper at ICLR 2024
D.4 ADVERSARIAL TRAINING FOR SSL
In the realm of SSL, innovative approaches have emerged that utilize adversarial training. One ap- proach involves generating synthetic data (Odena, 2016; Dai al., 2017) using generator network and assigning it to a new "generated" class. The goal is to make the discriminator network pro- vide class labels for these synthetic samples. Another line of research creates adversarial examples through techniques like VAT (Miyato et al., 2018), which adds noise to imput data: VAdD (Park et al., 2018), introducing an adversarial exit layer into the model's architecture; and RAT (Suzuki & Sato, 2020), extending the concept of noise to input transformations. These methods aim to impose lo- cal smoothness constraints on the model's s learned representations without relying on pseudo-labels during training. These advancements enhance model robustness and generalization, particularly in data-scarce scenarios, by utilizing latent data distribution structures for more effective learning. This research contributes significantly to improving SSL algorithms, addressing challenges in leveraging unlabeled data to enhance the applicability and performance machine learning models in real- world applications. These innovative adversarial training approaches poised to advance SSL.,
24

---

Published as a conference paper at ICLR 2024
E ALGORITHM
SemiRewards algorithm flow, including two-stage training (SR_Train Stage and SR_Train Stage 2) and inference (SR.inference), is shown in Algorithm
Algorithm Pseudocode of SemiReward training and inference in a PyTorch-like style.
SR Train Szage
iteration
set Reward data for x_1y__ in loader:
x_1,y_1,B_i
load data 8_R 51ze batch, X_E labelod data yur is ground truth label for X_r,y_E,B_ in loader:
[x_E) f_s. (X_E) + feature
y_ G(feat (x_=) ges
(ffet (X_E) $ reward
cossimin (y_ (y_F,__E))
fget similarity as sargte
ca ou loulate ate loss LJR 1- MSE [2,
MSB
ALaux (L_R+L_)) /B__
adam update
update (G)
update (R)
SR Train Stage iteration >- T:
Roward data loader
for x_ in loader:
x _
get pseudolebel V-P
9_P Psoudolabel (x_r()_)))
R(y_B/X_)) calculate zeward each pseudolabel
f select top k reward in sorted np.ergaort [::-1]
J-P Y-p y=p_lorrte__indicee y_Pl-k:]
f get loader batch size B_R B_R (R_1+2_u) +k/N
load data B_R size batel, x unlabeled data
for x_E, in sr_dataleader::
y_ G(x_r) s(x_r,y_k) get reward
fake label
8 cossimin (0 (y_k,y_t)) #get label similarity as zargte
caloulate loss
R (1
L_G 1- MDE Iz, 1]
(1_R+L_))/BB /B__
adam update
I aux L.auxbackwar backward
update updsta(R)
SR Inforence iteration
for u, x x_1,y_1 lcader:
F get psoudolabel y_p Paeudolabell (x__) feat (x [_st++ .eatt.efetteee (X_U) get feature
R(test [x_ul,
ovaluate
T r.mean # get tareshold
mask__ where (rxT,
CrossFit (y (Y_P, fLa (x__) mmakk filter label (B (x
11
calculate loss
total loss
adamki update backkaaard
update
feat: feature of input; cosslmim n: normalized ensine similarity; cat: concatenation. Psoudolaber paeudolabel method
Label algurithm (htips//aassv..orgaaa/ httpa//////9999/999 908.02983)
25


Reviews:
[{"review_id": "elHL0mpBTu", "reviewer": "Reviewer_xKm4", "summary": "This work addressed the major challenge in semi-supervised learning, i.e. evaluating the quality of pseudo labels in anomaly detection. A rewarder network is introduced to predict good quality pseudo labels by training on real and generated samples. The proposed reward network is demonstrated to improve multiple semi-supervised learning baselines.", "questions": "A more clear analysis of why the rewarder can predict the label quality and why the rewarder is not used for inference are encouraged.\n\nMore discussions on hyper-parameters and the definitions of fake label are recommended.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strength:\n\n1. Evaluating the quality of pseudo label is a real and major challenge for semi-supervised learning.\n\n2. The experiment evaluation is extensive covering audio, nlp and computer vision tasks. The proposed rewarder also consistently improves the performance of multiple state-of-the-art semi-supervised learning baselines.", "weaknesses": "Weakness:\n\n1. It is not clear why this rewarder design could be effective. If such a rewarder function is effective can one infer the labels for unlabelled data by simply choosing the label that maximise the rewards score given an input sample? In addition, one may also infer the labels for testing data in the similar way. If this is true, does it mean one can use directly use the rewarder function for inference?\n\n2. According to Fig. 7, the semi-supervised training stage still requires a hard threshold to generate pseudo labels. The proposed method does not reduce the overall amount of hyperparameters to tune compared against existing SSL baselines.\n\n3. It is unclear the definition of fake label. Does it mean the fake label must deviate from the ground-truth label? But the objective of Eq (7) seems to encourage the fake label to be consistent with ground-truth label. There is not enough clarifications here.\n\n4. The current design seems to mix student model f_S with generator. Such a design may affect the training of student model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d9CREuLSJn", "reviewer": "Reviewer_uket", "summary": "This paper proposes Semi-supervised Reward (SemiReward), an add-on module that uses a reward network and a generator network in a two-stage training pipeline, to train a student model with labeled and unlabeled data. The rewarder network predicts scores to filter pseudo labels for the student training, and the generator network generates fake labels (to provide the rewarder with a variety of scenarios, including incorrect ones, so that it learns to distinguish good data from bad more effectively without affecting the training of the student network) that only train the rewarder. The paper proposes a two-stage training pipeline: 1) pre-training both the rewarder network and a generator network on a labeled dataset, 2)  the reward network is trained on a smaller, random subset of the labeled data along with selected unlabeled data. Empirical results on NLP, Vision and Audio datasets show the effectiveness of their framework.", "questions": "- The NLP experiments are performed using 1\\% labels, what is the labeled \\% used in the CV and Audio datasets?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper is well-written and easy to follow, the motivation and formulation of the paper are sound\n\n+ Figures and examples are informative and help understand the paper\n\n+ Numerous ablation studies and qualitative comparisons help in understanding the setting\n\n+ This paper shows strong empirical results that surpass current state-of-the-art methods on SSL\n\n+ This paper adds non-trivial contributions and builds on prior work to present a novel framework for SSL", "weaknesses": "- Missing experiments: recent works show the effectiveness of their proposed methods by varying the number of labeled samples, given the sensitivity of available data and their impact, it seems imperative to include such experiments to asses the proposed method.\n\n- Missing benchmarks: it is hard to asses the impact of the proposed framework without at least a large-scale experiment (e.g., ImageNet)\n\n- Some gains, especially in NLP and CV datasets seem marginal (e.g., TL +0.30)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XulcOMYW2P", "reviewer": "Reviewer_DH5L", "summary": "This paper studies the problem of semi-supervised learning, where the labeled data is limited and unlabeled data are massive during training. Previous pseudo labeling based methods are limited to pre-defined schemes or complex hand-crafted policies specially designed for classification. To address this, the authors present the Semi-supervised Reward framework (SemiReward), which assesses and chooses high-quality pseudo labels through predicted reward scores and is adaptable to mainstream SSL methods for both classification and regression tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand.\n2. The proposed integration of a reward model with an SSL model enhances pseudo label quality evaluation, with extensive experiments demonstrating its effectiveness across diverse tasks.\n3. The experimental protocol is detailed and conducive to reproduce the results.", "weaknesses": "1. The method involves multiple models and training processes, adding complexity.\n2. The manuscript would benefit from pseudo codes for improved clarity.\n2. The design of reward model and hyper-parameter selection influence the outcomes, affecting the method's feasibility in practical scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)