Reinforcement Learning Guided Semi-Supervised Learning
Authors: Marzi Heidari
Hanping Zhang
Yuhong Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Semi-Supervised Learning
Primary area: other

Abstract: In recent years, semi-supervised learning (SSL) has gained significant attention due to its ability to leverage both labeled and unlabeled data to improve model performance, especially when labeled data is scarce. However, most current SSL methods rely on heuristics or predefined rules for generating pseudo-labels and leveraging unlabeled data. They are limited to exploiting loss functions and regularization methods within the standard norm. In this paper, we propose a novel Reinforcement Learning (RL) Guided SSL method, RLGSSL, that formulates SSL as a one-armed bandit problem and deploys an innovative RL loss based on weighted reward to adaptively guide the learning process of the prediction model. RLGSSL incorporates a carefully designed reward function that balances the use of labeled and unlabeled data to enhance generalization performance. A semi-supervised teacher-student framework is further deployed to increase the learning stability. We demonstrate the effectiveness of RLGSSL through extensive experiments on several benchmark datasets and show that our approach achieves consistent superior performance compared to state-of-the-art SSL methods.

Reinforcement Learning Guided Semi-Supervised Learning
Marzi Heidari', Hanping Zhang', Yuhong Guo
School of Computer Science, Carleton University, Ottawa, Canada 2CIFAR Al Chair, Amii, Canada
[marziheiddari@caaill.. jagzhang@cmail, yuhong .rrniggguoooccccoe carleton.ca
Abstract
In recent years, semi-supervised learning (SSL) has gained significant attention due to its ability to leverage both labeled and unlabeled data to improve model performance, especially when labeled data is scarce. However, most current SSL methods rely on heuristics or predefined rules for generating pseudo-labels and leveraging unlabeled data. They are limited to exploiting loss functions regularization methods within the standard norm. In this paper, we propose novel Reinforcement Learning (RI.) Guided SSL method, RLGSSI, that formulates SSL as one-armed bandit problem and deploys on weighted reward to adaptively guide the learning process of the prediction model. RLGSSL incorporates carefully designed reward function that balances the use labeled and unlabeled data to enhance generalization performance. A semi-supervised teacher-student framework is further deployed increase the learning stability. We demonstrate the effectiveness of RLGSSL through extensive experiments several benchmark datasets and show that our approach achieves consistent superior performance compared to state-of-the-art SSL. methods.
innovative RL loss based
Introduction
Semi-supervised learning (SSL) is a significant research area in the field of machine learning. addressing the challenge of effectively utilizing limited labeled data alongside abundant unlabeled data. SSL techniques bridge the gap between supervised and unsupervised learning, offering practical solution when labeling large amounts of data is prohibitively expensive time-consuning, The primary goal SSL is to leverage the structure and patterns present within the unlabeled data to improve the learning process, generalization capabilities, and overall performance of the prediction model. Over the past few years, there has been considerable interest developing various SSL methods, and these approaches have found success in wide range of applications, from computer vision [ to natural language processing and beyond 3
Within the SSL domain, a range of strategies has been devised to effectively utilize the information available both labeled and unlabeled data. Broadly, SSL approaches can categorized into three key paradigms: regularization-base Regularization-base approaches form fundamental pillar SSL ( These methods revolve around the core idea of promoting model robustness against minor perturbations in the input data. A quintessential example this category Virtual Adversarial Training (VAT) (5). VAT capitalizes the introduction of adversarial perturbations the input space, thereby ensuring the model's predictions maintain consistency. The second category, Mcan-teacher based methods, encapsulates distinct class of SSL strategies that leverage the concept of temporal ensembling. This technique aids in the stabilization of the learning process by maintaining an exponential moving average of model parameters over training iterations. Mean Teacher g notably pioneered this paradigm with
mean-teacher-basee based, and pseudo-labeling methodologies.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Mean Teacher model, illustrating its efficacy across numerous benchmark tasks. Lastly, the category of aseudo-labeling approaches has attracted attention due to its simplicity and effectiveness. These methods employ the model's 's own predictions on unlabeled data as "pseudo-labeli to augment the training process. The Mix Match [ framework stands as one of the leading representatives of this category, demonstrating the potential of these methods in the low-data regime.
Despite these advancements, achieving high performance with limited labeled data continues to be a significant challenge in SSL, often requiring intricate design decisions and careful coordination of multiple loss functions. Recently RL has been increasingly used in fine-tuning complex models with non- differentiablle reward functions. This application establishes a reinforced alignment with the learning objective and enhances generalization to standard supervised learning scenarios [2] Enthusiastic about the potential of enhanced generalizability introduced by non-differentiable RL reward SSL,, we propose to approach SSL. outside the conventional design norms by developing Reinforcement Learning Guided Semi-Supervised Learning (RLGSSLL method, which brings fresh perspective to SSL. We employ Reinforcement Learning (RL. to optimize the generation of pscudo-labels in Semi-Supervise Learning (SSL). Conventional methods in SSL often face problems such as overfitting and difficulties in creating accurate pseudo-labels. RL introduces advantages in exploration capabilities and adeptly manages non-differentiable operations by considering the predictor as a policy function.
In RLGSSL, we formulate SSL as bandit problem, a special case of reinforcement learning, where the prediction model serves as the policy function, and soft pseudo-labeling acts as the actions. We define a simple reward function that balances the use of labeled and unlabeles data and improves generalization capacity by leveraging linear data interpolation, while the prediction model trained under policy gradient framework to maximize the policy-output weighted reward. Formulating the SSL problem as such an RL task allows our approach to dynamically adapt and respond to the data. Moreover, we further deploy teacher-student learning framework to enhance the stability of learning. Additionally, we integrate a supervised learning loss to improve and accelerate the learning process. This new SSL framework has the potential to pave the way for more robust, flexible, and adaptive SSL methods. We evaluate the proposed method through extensive experiments on benchmark datasets. The contribution of this work can be summarized as follows:
We propose RLGSSI, a novel Reinforcement Learning based approach that effectively tackles SSL by leveraging RL's power learn effective strategies for generating pseudo- labels and guiding the learning process.
We design a prediction assessment reward function that encourages the learning of accurate and reliable pseudo-labels while maintaining balance between the usage of labeled and unlabeled data.
We develop an innovative RL loss that allows reward from pseudo-labels be incorpo- rated into SSL as a non-differentiable signal a reinforced manner, promoting better generalization performance.
We conduct novel investigation on integration frameworks that combine the power of both RL loss and standard cuni-super loss, providing a brand new approach that has the potential to lead to more accurate and robust SSL models.
Extensive experiments demonstrate that our proposed method outperforms state-of-the-art SSL. approaches and validate the integration of RL strengths in SSL..
2 Related Work
2.1
Semi-Supervised Learning
Existing SSL approaches can be broadly classified into three primary categories: regularization-buse methods, teacher-student based methods, and pseudo-labeling techniques.
2.1.1
Regularization-Ban Methods
A prevalent research direction in SSL focuses on eggulaizattonnbasse methods, which introduce additional terms to the loss function to promote specific properties of the model. For instance, the I1-model [6] and Hamporal-Ensemblee (6) incorporate consistency regularization into the loss

---

function, with the latter employing the exponential moving average of model predictions. Virtual Adversarial Training (VAT) [5] is yet another regularization based technique that aims make deep neural networks robust to adversarial perturbations. In similar vein, Consistency Regularization for Generative Adversarial Networks (CR-GAN) 77) integrates a generative adversarial network (GAN) with consistency regularization term, facilitating the effective generation of pseudo-labels for unlabeled data.
2.1.2 Teacher-Student- Based Methods
Teacher-student-basee methods offer an alternative approach in SSL research. These techniques train a student network to align its predictions with those of a teacher network on unlabeled data. Mean Teacher (MT) 8), a prominent example in this category, leverages an exponential moving average (EMA) of successive weights from the student model to obtain the teacher model. To enhance performance, MT Fast SWA [T3 combines Mean Teacher with Fast Stochastic Weight Averaging. Smooth Neighbors on Teacher Graphs (SNTG) [14] takes a different approach, utilizing a graph for the teacher to regulate the distribution of features in unlabeled samples. Meanwhile, Interpolation Consistency Training (ICT) [15] aims to promote consistent predictions across interpolated data points by ensuring that model's predictions on an interpolated set of unlabeled data points remain consistent with the interpolation of the predictions those points.
2.1.3 Pseudo- Labeling Methods
Pseudo-labeling is an effective way to extend the labeled set when the number of labels is limited. Pseudo-Label [16] produces labels for unlabeled data using model predictions and filters out low- confidence predictions. MixMatch [ employs data augmentation to create multiple imput versions, obtaining predictions for each and averaging them generate pseudo-labels. In contrast, works such as ReMixMatch [17), UDA [18], and FixMatch [19] apply confidence thresholds to produce pseudo-labele labels for weakly augmented samples, which subsequently serve as annotations for strongly augmented samples. Label propagation methods, including TSSDL 20 and LPD [21, assign pseudo- labels based on local neighborhood density. DASO (22) combines confidence- based and density-based pseudo-labels in varying ways for each class. Approaches such as Dash 22 and FlexMatch [24] dynamically adjust confidence thresholds in a curriculum learning manner to generate pseudo-labels. Meta Pseudo-Label (25 uses bi-level optimization strategy, deriving the teacher update rule from student feedback, to learn from limited labeled data. Co-Training (26) is an early representative of pscudo-lableing which involves training two classifiers on distinct subsets of unlabeled data and using confident predictions to produce pseudo-labele for one another. Similarly, Tri- tri-Training [27] trains three classifiers on separate unlabeled data subsets and generates pseudo-labels based on the disagreements between their predictions.
2.2 Reinforcement Learning
Reinforcement Learning (RL) is a field of study that focuses on optimizing an agent's 's decision-maan maakin abilities by maximizing the cumulative reward obtained through interactions with its environment [28]. RL methodology has been widely applied to solve many other learning problems, including searching for optimized network architectures [29], training sequence models for text generation by receiving
reward signals [30 311, and solving online planning problems [32. Recently, RL has been applied to fine-tune complex models that typically fail to align with users' preferences. Moreover, based on RL from Human Feedback (RLHF; (1I1),, ChatGPT achieves great success in dialogue generation by fine-tuning Large Language Models (LLM) 333 It frames the training of LLM as a bandit problem, specifically a one-armed bandit problem (28) where the objective is to determine the optimal action (dialogue generation) for given state (user prompt) within single step, demonstrating the capacity of RL for prediction tasks.
The bandit problem was originally described as statistical decision model used by agents to optimize their
making process (34. this problem, an agent receives a reward upon taking an action and learns to make the best decision by maximizing the given reward. The bandit problem found its application in economies and has been widely used in market learning, specifically in finding the optimal marker demands or prices to maximize expected profits [35]. Bergemann et al. (36 and Lattimore et al. [37 have extensively discussed the literature and modern applications of the bandit problem. Additionally, Mortazavi et al. [38] introduced Single-Step Markov Decision Process
3

---

Unlabeled X
Teacher Palicy 0r
Action
Pseudo Label Y Pay
comu
Labeled x
Student Policy 0g
Mixup Data n (xmm Y
L
Labels y-
Lup
Predicted Label (x)(X*)
Reward Function R(x', x , x=
Figure 1: Overview of the RLGSSL Framework. The prediction networks (student 0s, teacher 0r) serve as the policy functions, and the soft pseudo-labeling (Poy. (X")) acts as the actions. The model has three loss terms in total: RL loss (Cn), supervised loss (fou)), and consistency loss (Leoms). The teacher policy function is used to execute the actions and compute the consistency loss, while the student policy function is used for all other aspects.
(SSMDP) to formulate the bandit problem manner that aligns with modern RI. techniques. This advancement enables the utilization of standard RL methods conventional bandit problems.
3 The Proposed Method
We consider the following semi- superviee learning setting: the training data consist of small number of labeled samples, D' (X', Y*) ((1), v!) and a large number of unlabeled samples, D" X" {x
1e'],N" Ji=1, with N > N where x (or x denotes the input instance and y denotes the one-hot label vector with length C. The goal is to train a C-class classifier fo + that generalizes well to unseen test data drawn from the same distribution as the training data.
In this section, we present the proposed RLGSSL method, which formulates SSL as a one-armed bandit problem with continuous action space, and deploys novel RL loss to guide the SSL process based on a reward function specifically designed for cumi-supervised data. Moreover, we further incorporate semi-supervised teacher-student framework to augment the RL loss with supervised loss and a prediction consistency regularization loss, aiming to enhance the learning stability and efficacy. Figure I illustrates the overall framework of the proposed RLGSSL, and the following subsections will elaborate on the approach.
3.1 Reinforcement Learning Formulation for SSL
We treat SSL special one-armed bandit problem with continuous action space. One-armed bandit problem can be considered a single-step Markov Decision Process (MDP) [38]. In this problem, the agent takes single action and receives reward based on that action. The state of the environment is not affected by the action. The one-armed bandit problem involves selecting an action to maximize immediate reward, which can be regarded as learning policy function under the RL framework. Formulating SSL as one-armed bandit problem within the RL framework and deploying RL techniques guide SSL requires defining the following key components: state space S, action space A, policy function S - A, and a reward function R. x A + R. The objective is to learn optimal policy n* that maximizes the expected one-time reward R(s, (-(ss) in the given environment (state s): n' = arg max (T( = E. ग (c))))((s,))
The state encapsulates the provided knowledge about the environment and is used input for the policy function. As the action does not affect the state of the environment under one-armed bandit problem, use the observed data from the SSL problem as the state, i.c., (x', Y X)..
State

---

Action and Policy Function As the goal of SSL is to learn an optimal classifier fo (i.e., prediction network parameterized with 8), we use the classifier for usually denoted by its parameters 8, as the policy function To to unify the goals of RI. and SSL.. In particular, we consider a probabilistic policy function/prediction network nT(-) Po(.). Since a policy function is used to project a mapping from the state s to the action space, = (-))s,,, by using probabilistic prediction network as the policy function, it naturally determines a continuous action space A. Specifically, given the fixed state taking an action is equivalent to making probabilistic predictions
the unlabeled data in y = (x(")) = [T<(-s) as the labeled data already has labels. For each unlabeled instance o; the action is probability vector produced y Po(a)) ) which can be regarded as soft pseudo-labels in the SSL setting. This links the action of RL to the pseudo lobulligg in SSL.
3.1.1 Reward Function
The reward function serves as feedback to evaluate the performance of the action (prediction) provided by the policy. needs to be thoughtfully crafted to maximize the model's ability to extract useful information from both labeled and unlabeled data, which is central to the SSL paradigm. The underlying motivation is to guide the learning process to induce a more generalizable and robust prediction model. To this end, we adopt data mixup [39] strategy to produce new data points from the given labeled data (X', Y') and pseudo-labeled data (x", Y"), which together form the state-action pair (s, a), through linear data interpolation, and assess the prediction model's s generalization ability on such data points as the reward signal. This decision is inspired by the proven effectiveness of mixup in enhancing model performance in various tasks. The idea of data mixup is to generate virtual training examples by creating convex combinations of pairs of input data and their corresponding labels. This technique encourages the model learn more fluid decision boundaries, leading to improved generalization capabilities.
Specifically, we propose generate new data points by performing inter-mixup between labeled and unlabeled data points, aiming to maintain a balanced utilization of both labeled and unlabeled data. In order to address the size discrepancy between the labeled dataset D and the unlabeled dataset D" with N >> N we replicate the labeled dataset D' by factor of r N times, resulting in an extended labeled dataset DJ_ After shuffling the data points in each set, we generate a mixup data point by mixing an unlabeled point x; E DD with labeled point x! D along with their corresponding pseudo-label y D and label y! E Di:
p p + (1 -0)= n))
+ m4
(1)
where the mixing parameter ja is sampled from Beta distribution, Beta(1,1). With this procedure, we can generate NT = N mixup samples by mixing all the unlabeled samples with the samples in the extended labeled set.
We then define the reward function to measure the negative mean squared error (MSE) between the model's prediction (a(x)) and the mixup label y for each instance in the mixup set. This results in a single, comprehensive metric that quantifies the overall negative disagreement between the model's predictions and the mixup labels over large set of interpolated data points:
R(s, az (0) R(x,,
Y sg((0)
C- Σ Po(x:)) 11
1
(2)
where C denotes the number of classes and sg[:] is the stop gradient operator which stops the flow of gradients during the backpropagation process. This ensures that the reward function solely employed for model assessment, rather than being directly utilized for model updating, enforcing the working mechanisms of RL. Mixup labels capture both the supervision information in the labeled data and the uncertainty in the pseudo-labels of unlabeled data. With the designed reward function, good reward value can only be returned when the prediction model not only exhibits strong alignment with the labeled data but also delivers accurate predictions on the unlabeled data. Consequently, through RL, this reward function will not only promote accurate predictions but also enhance the model's 's robustness and generalizability.
3.1.2 Reinforcement Learning Loss
By deploying the probabilistic predictions on the unlabeled data, Y = 7x(X") = PX"), as the action, we adopt a deterministic policy. Following the principle of one-armed bandit problem on
5

---

maximizing the expected one-time reward w.r.t. the policy output, we introduce a weighted negative reward based on the deterministic policy's 's output as the RL loss for the proposed RLGSSL. thereby exploiting non-differentiable reward signals while enabling policy gradient with a deterministic policy. Specifically, we treat the output of the policy network, Y", as uniform distribution over the set of N probability vectors, [y"," ... .YN",, J. predicted for the unlabeled instances. Let e = 1/C denote a discrete uniform distribution vector with length Ch-the number of classes. We design the following KL-divergence weighted negative reward as the RL loss:
(a = - -By (L, y (s,, az sg sg()) - ED KL(e, Po(x)))R(s, a; sg(())
where the KL-divergence term measures the distance of each label prediction probability vector y from a uniform distribution vector; R(s, a: sg. (0x)) is treated as a non-differentiabll reward function. Given that uniform probability distribution signifies the least informative prediction outcome, the expected Kl.-divergence captures the level of informativeness in the policy output and hence serves as meaningful weight for the reward, which inherently encourages the predictions to exhibit greater discrimination.
The minimization of this loss function over the prediction network parameterized by 0 is equivalent to learning an optimal policy function Te by maximizing the KL-divergence weighted reward, which aims at an optimal policy function (also the probabilistic classifier Po) that not only maximizes the reward signal but is also discriminative. From the perspective of SSL, the utilization of this novel RL loss introduces a fresh approach to designing prediction loss functions. Instead of directly optimizing the alignment between predictions and targets, it offers a gradual learning process guided by reward signals. This innovative approach presents more adaptive and flexible solution for complex data scenarios, where traditional optimization- based methods may fall short.
3.2 Teacher-Student Framework
Teacher-student models 8 have been popularly deployed to exploit unlabeled data for SSL, improving the learning stability. We extend this mechanism to provide a teacher-student framework for RL- guided SSL by maintaining a dual set of model parameters: the student policy/model parameters Os. and the teacher policy/model parameters 0r. The student model is directly updated through training, whereas the teacher model is updated via an exponential moving average (EMA) of the student model. The update is conducted as follows:
0 = Br + (1 0)0s
(4)
where denotes hyperparameter that modulates the EMA's decay rate. The utilization of the EMA update method ensures a stable and smooth transfer of knowledge from the student model to the teacher model. This leads to a teacher model with consistent and reliable parameter values that are not susceptible to random or erratic fluctuations during the training process. Leveraging this desirable characteristic, we propose to employ the teacher model for executing actions within the RL framework described in the section 3.1 above; that is, Y = Pe, (X"), while retaining the student model for other aspects. By doing so, we ensure that stable actions are taken, reducing the impact of random noise in the -seudo-labels and enhancing the accuracy of reward evaluation and reinforcement strength.
Within the seacher-student framework, we further propose to augment the RL. loss with a supervised loss Sup on the labeled data and consistency regularization loss Leons on the unlabeled data. We adopt a standard cross-entropy loss function leg to compute the supervised loss, promoting accurate predictions on D where the ground-truth labels are available:
Csup E(e'' (DED' [fCE (Pog (a)))')))'))
(5)
This loss can enhance effective exploitation of the ground- truth label information, providing a solid basis for exploring the parameter space via RL. The consistency loss Lcons is deployed to encourage prediction consistency between the student and teacher models on the unlabeled data D=:
(6) where (L(.,)) denotes the Kullback- Leibler divergence between two probability distributions. By enforcing consistency, this loss encourages the student model to make more confident and reliable predictions, reducing the impact of random or misleading information in the training set. It also acts as form of regularization, discouraging the student model from overfitting the labeled data.
Lcons Ex [KL (Pag (x"),, Po (=)))]

---

Algorithm Pseudo-Label Based Policy Gradient Descent
Input: D', D", and extended D'; initialized 0s, Or: for iteration to maxiters do
hyperparameters
for z. E "De do
Compute soft pseudo-label vector y Paz (xx²) to form y end for
Generate mixup data D =(X", Y on D and D' using Eq. with shuffling
for step maxsteps do
Draw a batch data B {x:"" yy"" from D
Calculate the reward function R( [::::ggl00)) s0[0s]) using the batch B
Compute the objective in Eq.(7)
Update the policy parameters 8g via gradient descent
Update teacher model B1 via EMA in Eq.44)
end for end for
Table I: Performance of RLGSSL and state-of-the-arr SSL algorithms with the CNN-13 network.
We report the average test emrors and the standard deviations of 5 trials.
Datasel
CIFAR-10
CIFAR- 100
Number of Labeled Samples
1000
2000
4000
4000
10000
Supervised
(9.95(0.75)
(7.67
20.42(o.21)
58.31(p.89)
44.56(0.30)
Supervised MixUp 39
(1.83
24.22 (0.15)
(0.37
54.87 (4.87
(0.97
11-model
(8.74
(7.47
(2.36(o.17)
(b.3)
(8.07
Temp-ensemble 6
25. (5.16
15.78 15.78
(0.44)
11.90
38.65 (8.65(0.51)
Mean Teacher
15.73 (0.53
12.31
(5.36(0.49)
35.96(0.77)
VAT S5
(8.12(0.82)
13.93 (0.33)
(1.10(0.24)
SNTG
14
18.41(0.52)
(3.64(0.52)
10.93
37.97 (0.29)
Learning to Reweight 40
11.74(0.12)
9. (..4(0.17)
46.62
37 37.31 (0.47)
MT Fast SWA 13
(5.58
11.02
(..5
(3.52
ICT [15
(2.44
(..9
(..8)
40.07 (0.38)
32.24 32.24(0.16)
RLGSSL (Ours)
9.15
(.90
6.11
(6.42(0.45)
29.12 (0.20)
3.3 Training Algorithm for RL-Guided SSL
The learning objective for the RLGSSL approach is formed by combining the reinforcement learning loss La with the two augmenting loss terms, the supervised loss Luu and the consistency loss Ceons- using hyperparameters λ₃ and A2:
[(0s) Las X1 Soup + X2hcoms
By deploying such joint loss, the RLGSSL framework can benefit from the strengths of both reinforcement exploration and semi-supervised learning. The RL component, in particular, introduces a dynamic aspect to the learning process, enabling the model to improve iteratively based on its own experiences. This innovative combination of losses allows the model to effectively learn from limited labeled data while still exploiting the abundance of unlabeled data.
We develop stochastic batch-wise gradient descent algorithm to minimize the joint objective in Eq. (7) for RL-guided semi-supervised training. The procedure of this algorithm summarized in Algorithm
4 Experiments
4.1 Experimental Setup
Datasets We conducted comprehensive experiments on four image classification benchmarks: CIFAR-10, CIFAR- 100 [43, SVHN (44,, and STL-10 (45). We adhere to the conventional dataset splits used in the literature. Consistent with previous works, each dataset we preserved the labels of randomly selected subset of training samples with an equal number of samples for each class, and left the remaining samples unlabeled. In order to compare with previous works in the same settings,
7

---

Table 2: Performance of RLGSSL and state-of-the-an SSL algorithms with the CNN-13 network.
We report the average test errors and the standard deviations of 5 trials.
VAT [5] II-model [6]
Temp-ensemble
MT
ICT [15]
SNTG [1] RLGSSL (Ours)
SVHIN/50
(..5)
(.12
4.18
(.23
3.99 (o.94)
3.12 (.12(0.07)
SVIIN/1000 5. o.)
(.82
(o.6
3. (.95
3.89
(..6
3. 3.05
Table 3: Comparison results in terms of mean test error and standard deviation using WRN-28-2 as the backbone on CIFAR-10 10 and SVHN and using WRN-28-8 8 as the backbone on CIFAR-100.
Dataset
CIFAR-I0
CIFAR- 100
SVHN
Number of Labeled Samples
250
1000
4000
2500
10000
1000
Mcan Teacher
(2.30
(7.32(4.00)
(0.36
(3.9l
35. (5.83
5.65
ICT
15
(.66
3.5310.07)
MixMatch
1
7. 7.75(0.32) (1.05
6,24 6.24ça.00
28.31 (9.94(0.37))
(.27
ReMixMatch
(.05)
(0.34)
6.24
27. (7.14
23.78 (0.12))
3.27 (.27
FixMatch 19
(..55
1. 1.266a.05)
28. (8.29(0.11)
22. (2.10(0.12)
(.2
Meta-Semi 41
7. (0.2))
6. 6.10
Meta Pseudo-Labels
25
3.89
(o.0))
MarginMatch 42
(..1)
3.98 02)
23.71 (0.13)
21.39 (0.12)
(..3
RLGSSL (Ours)
5.01
(..2(
23.18 (..2(
(0.48)
20.15(0.30)
(..00)
we performed experiments on CIFAR-10 10 with various numbers (N' E {250, 1, 000, 2, 000, 4. 000))
of labeled samples, on CIFAR- 100 with 2,500, 10,000 and 4,000 labeled samples, on SVHN with 1,000 and 500 labeled samples, and on STL-10 with 1,000 labeled images.
Implementation Details We conducted experiments using four different network architectures used in the literature: a 13-layer Convolutional Neural Network (CNN-13), a Wide-Residual Network with 28 layers and widening factor of 2 (WRN-28-2), RestNet-37-2 (WRN-37-2). For training the CNN-13 architecture, we employed the SGD optimizer with Nesterov momentum of 0.9. We used an L2 regularization coefficient of le-4 for CIFAR-10 10 and CIFAR-100, and 5e-5 for SVHN, The initial learning rate was set to 0.1, and the cosine learning rate annealing technique proposed in previous studies [6 [5] was utilized. For the WRN-28-2 architecture, we followed the suggestion from MixMatch I and used an L2 regularization coefficient of 4e-4. For WRN-37-2, the training configuration includes the SGD optimizer, an L2 regularization coefficient of 5e-4, and an initial learning rate of 0.01. Finally, the training configuration for the WRN-28-8 model includes using the SGD optimizer, L2 regularization coefficient of 0.001, and starting with a learning rate of 0.01. To compute the parameters of the teacher model, we employed the EMA method with decay rate B 0.999. We selected all hyperparameters and training techniques based on relevant studies to ensure a fair comparison between our approach and the existing methods. Specifically for RLGSSL, we set the batch size 128, and model for 50 epochs using the Mean-Teacher algorithm and then proceed to the training procedure of RLGSSL for 400 epochs. We ran each experiment five times and reported the mean test errors with their standard deviations.
Wide-RestNet-28-- (WRN-28-8) and Wide-
A2 = 0.1. We first pre-train the
4.2 Comparison Results
We compare RLGSSL with great number of SSL algorithms, including Supervised + MixUp 39
I-model (6. Temp-ensemble (6, Mean Teacher 88, VAT (5), SNTG [14,, Learning to Reweight
40 MT + Fast SWA [13), MixMatch 11 RaMcx [17]. FixMatch [19]. MarginMatch 42)
Meta-Semi [41, Meta Pscudo-Labels (25), and ICT [15, using CN,-13, WRN-28-2, WRN-28-8 or
WRN-37-2 as the backbone network.
Table D reports the comparison results on CIFAR-10 10 with 4,000, 2,000, and 1,000 labeled samples and on CIFAR-100 with 10,000 and 4,000 labeled samples when CNN-13 is used as the backbone network. On CIFAR-10, 10, RLGSSL outperforms all the other compared methods across all settings with different numbers of labeled samples. With 1,000 labeled samples, RLGSSL surpasses the second best method, Learning to Reweight, by a significant margin 2.59% on CIFAR-10, achieving an average test error of 9.15%. This pattern of outperformance continues with different numbers (2,000 and 4, 4,000) of labeled samples, where RLGSSIL yields lowest test error rates and outperforms ICT the second best method by margin of 1.79% and 1.07% respectively. The results on the
8

---

Table Comparison results in terms of mean test error and standard deviation using WRN-37-2 as the backbone on STL-10.
Model [6]
Mean Teacher
Mix
[I]
UDA [2
RLGSSI. (Ours)
STI.-10/ 1000
26. (0.88)
21 (3.39)
10.41 (0.61)
(.66
6.12 (0.52
Table 5: Ablation study results on CIFAR- 100 using 10000 and 4000 labels with CNN-13 as the backbone. The average test errors and standard deviations over 5 trials are reported.
RLGSSL
w/o Lat
w/o Suu
-w/o Leons
-w/o EMA
w/o mixup
CIFAR-100/4000
(6.92(0.45)
44.92
39.52
38.78/0.48)
(3.12(0.52)
40. (0.12
CIFAR-100/10000
29.12
33.12
32.67 (0.45)
31.48(0.322)
32.84 (0.45)
(1.48
REGSSI.
R
R.
R(MSE + KT.)
R(MSE JS)
R : w/o sg[@]
CIFAR-100/4000
36.92
39.52
39.54 (0.33)
38.02
39.32(0.15)
(0.62
CIFAR-100/10000
29.12
31.25
32.37 (0.57)
(1.12(0.52)
31.39(0.68)
32.12
CIFAR-100 dataset are similarly impressive. For 4,000 labeled samples, RLGSSL again outperforms ICT, the second best method, with a margin of 3. 15%. As the number of labeled samples escalates to 10,000, RLGSSL maintains its performance edge, outperforming the second best method ICT by margin of 3.12%.
Table the comparison results on the SVHN dataset with CNN-13 as the backbone network. Our method, RIGGSLL, surpasses all the other compared SSL techniques for both settings. Specifically, for 500 labeled samples, RLGSSL achieves the lowest test error of 3. 3.12%, which is 0.87% lower than the recond-best method, SNTG. For 1,000 labeled samples, RLGSSL also shows superior performance with test error of 3.05%, outperforming the second-best method, SNTG, by 0.81%.
Table 3 presents the comparative outcomes across three datasets, utilizing WRN-28-2 as the backbone network for CIFAR- 10 and SVHN, and WRN-28-4 for CIFAR-100. On CIFAR- 10, with the number of labeled samples increasing from 250 4,000, RLGSSL showed better performance compared to the other methods in most cases. For 1, 1,000 labeled samples, our method improved over the best competing method, ReMixMatch, by a margin of 1.35%. Additionally, for 4,000 labeled samples RLGSSL outperforms Meta Pseudo-labels which is the second best method. On CIFAR- 100, with 2,500 and 10,000 labeled samples, RLGSSL surpassed the state-of-the -hrrterrrt MarginMatch. Moreover, on SVHN with 1,000 labeled samples, RLGSSL also outperformed the MarginMatch. This confirms the robustness of RLGSSL across various settings, even when the labeled data is limited in quantity.
Table presents the comparison results of various SSL methods on the STL-10 dataset, utilizing WRN- 37-2 as the backbone network. With a fixed number of 1,000 labeled samples, RLGSSL achieves remarkable performance with a mean test error of 6. 12%, which outperforms previous state-of-the-ar methods, including MixMatch and UDA, showcasing the effectiveness of our RLGSSL.
4.3 Ablation Study
In order to evaluate the significance of various components of our RLGSSL approach, we conducted an ablation study on the CIFAR-100 dataset using the C3 network. In particular, we compared the full model RLGSSL with the following variants: (1) "-w/o Lu"" which drops the RL loss La (2) "-wlo Ssup", which excludes the supervised loss; (3) "-w/o Leons", ". which does not include the consistency loss; (4) "-w/o EMA", which drops the teacher model by disabling the EMA update; and (5) " "-w/o mixup", which only uses unlabelled data the reward function (μ = 1), and the mixup operation is excluded. The ablation results are reported in the top section of Table 5 The full model, RLGSSL, achieved the lowest test errors, confirming the overall effectiveness of our method. The most significant observation from study lies in the removal of the RL loss La. Upon removal of this component, we witness a substantial increase in test errors, which highlights the indispensable role played by the RL component in model. The ablation study further illustrates the importance of each component by analyzing the performance of the model when the component is removed. In each of these cases, we observe that the removal of any individual component consistently leads to an increase in test errors. This finding underpins the notion that each component of the RLGSSL model plays a significant role in the overall performance of the model.
In addition, we also conducted another set of ablation study centered on the proposed RL loss and the reward function. We compared the full model RLGSSL with the following variants: (1) "R 1".

---

33
32
Error
Error
31
30
29
29
0.0001 0.001
0.01
0.1
0.2
0.3
0.0001 0.001
0.01
0.1
0.2
0.3
A1
A2
(a)
(b) Az
Figure 2: Sensitivity analysis for hyper-parameters \ and X2 on CIFAR-100 using 10000 labeled samples.
which drops RL by setting the reward as a constant (2) "R p p== 0", which only uses labeled data compute the reward by setting μ 0: (3) "R(MSE KL)", which replaces the mean squared error of the reward function in Eq. ( with the KL-divergence loss; (4) "R(MSE - JS)", which replaces the mean squared eтror of the reward function Eq. ? with the JS-divergence loss; and (5)
"R w/o sg[0]", which removes the stop-graun -ggadind operator from the reward function and makes the reward function differentiable w.r.t The ablation results are reported in the bottom section of Table 5. We can see that all these variants with altered reward functions produced degraded performance comparing to the full model with the proposed reward function. In particular, the performance degradation of "R = 1" and "RR : w/o sg[0]" that drop RL in different manners further validates the contribution of the proposed framework of guiding SSL with RL.
4.4
Analysis
We conduct sensitivity analysis over the two hyper-parameters of the proposed RLGSSL: \ Ahe trade-off parameter for the supervised loss, and the trade-off parameter for the consistency loss. In the case of \1, lower values (e.g.. 1e-4 and 1e-3) result in less emphasis on the supervised loss term in the objective function. As result, the model might not learn effectively from the limited available labeled data, leading to increased test error rates. Conversely, higher values of λ (e.g., 0.2 and 0.3) may overemphasize the supervised loss term, potentially causing the model to overfit the labeled data and ignore useful information from the unlabeled data. The sweet spot lies in the middle (around 0.1), attaining a balance between learning from labeled data and leveraging information from unlabeled data. Regarding X2. very low values (c.g.. 1e-4 and 1e-3) may not enforce sufficient consistency in the model predictions on unlabeled data, resulting in model that fails to generalize well. However, if X2 is too high (e.g., 0.2 and 0.3), the model may overemphasize the consistency constraint, possibly leading the diverse patterns in the data. An optimal value of A2 (around 0.1 in our experiments) ensures good balance between encouraging prediction consistency and allowing the model to adapt to the diverse patterns in the data. The optimal value choice for hyperparameters λι and X2 (around 0. also validates that the RL loss the main leading term, while the supervised loss and consistency loss are augmenting terms.
The results are reported in Figure -
model that is too rigid capture
Conclusion
In this paper, we presented Reinforcement Learning-Guided Semi-Supervised Learning (RLGSSL), a unique approach that integrates the principles of RL tackle the challenges inherent in SSL. This initiative was largely driven by the limitations of conventional SSL techniques. RLGSSL employs distinctive strategy where an RL-optimized reward function is utilized. This function adaptively promotes better generalization performance through more effectively leveraging both labeled and unlabeled data. We also further incorporated ttuchrt-teacher framework to integrate the strengths of RL and SSL. Extensive evaluations were conducted on multiple benchmark datasets, comparing RLGSSL to existing state-of-the-an SSL techniques, and various ablation variants. RLGSSL consis- tently outperformed these other techniques across all the datasets, which attests to the effectiveness and generalizability of our approach. The results underline the potential of integrating RI principles into SSL, and the RLGSSL method introduced in this paper is significant stride in this direction.
10

---

Acknowledgement and Disclosure of Funding
This research was supported in part by an NSERC Discovery Grant, the Canada Research Chairs Program, and the Canada CIFAR AI Chairs Program.
References
[1] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, "Mixmatch: holistic approach to semi-supervised learning," Advances in neural information processing systems (NeurIPS), 2019.
[2] J. Howard and S. Ruder, "Universal language model fine-tuning for text classification," Pro- ceedings of Annual Meeting of the Association for Computational Linguistics (ACL), 2018.
[3] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, "Semi-superviiedd learning with ladder networks," Advances in neural information processing systems (NeurIPS), 2015.
[4] X. Zhu and A. B. Goldberg, "Introduction to cumi-supeeeied learning." " Synthesis lectures on artificial intelligence and machine learning, 2009.
[5] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, "Virtual adversarial training: regularization method for supervised and semi-supervised learning," " IEEE transactions on pattern analysis and machine intelligence, 2018.
[6] S. Laine and T. Aila, "Temporal ensembling for sumi-supervised learning." in International Conference on Learning Representations (ICLR), 2017.
[7] H. Zhang, Z. Zhang, A. Odena, and H. Lee, "Consistency regularization for generative adversar- ial networks," in International Conference on Learning Representations (ICLR), 2020.
[8] A. Tarvainen and H. Valpola, "Mean teachers are better role models: Weight- consis- tency targets improve semi-supervised deep learning results," Advances in neural information processing systems (NeurIPS), 2017.
[9] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforce- ment learning from human preferences," Advances in neural information processing systems (NeurIPS), 2017.
[10] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. "Fine-tuning language models from human preferences," arXiv preprint wxXiv:109.008993 2019.
[11] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano, "Learning to summarize with human feedback," Advances in Neural Information Processing Systems (NeurIPS), 2020.
[12] H. Le. Y. Wang. A. D. Gotmare, S. Savarese, and S. C. H. Hoi, "CodeRL: Mastering code generation through pretrained models and deep reinforcement learning," Advances in Neural Information Processing Systems (NeurIPS), 2022.
[13] B. Athiwaratkun, M. Finzi, P. Izmailov, and G. Wilson, "There are many consistent explana- tions of unlabeled data: Why you should average," in International Conference on Learning Representations (ICLR), 2019.
[14] Y. Luo, J. Zhu, M. Li, Y. Ren, and B. Zhang, "Smooth neighbors on teacher graphs for semi- supervised learning," in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
[15] V. Verma, K. Kawaguchi, A. Lamb, J. Kannala, A. Solin, Y. Bengio, and D. Lopez-Paz. "Interpolation consistency training for semi-supervised learning," Neural Networks, 2022.
[16] D.-H. Lee el al., "Pseudo-lahel: The simple and efficient semi-supervised learning method for deep neural networks," in Workshop challenges in representation learning, 2013.
11

---

[17] D. Berthelot, N. Carlini, E. D. Cubuk. A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, "Remix- match: Semi-supervised learning with distribution matching and augmentation anchoring." " in International Conference on Learning Representations (ICI.R), 2020.
[18] Q. Xie, Z. Dai, E. Hovy. T. Luong, and Q. Le, "Unsupervised data augmentation for consistency training." Advances in neural information processing systems (NeurIPS), 2020.
[19] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, "Fixmatch: Simplifying con-ssuper learning with consistency and confidence," Advances neural information processing systems (NeurIPS), 2020.
[20] W. Shi, Y. Gong, C. Ding, Z.. M. Tao, and N. Zheng, "Transductive semi-supervised deep learning using min-max features," in Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[21] A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, "Label propagation for deep semi-supervised learning." in Proceedings of the IEEE/CVF conference on computer vision and paltern recogni- tion (CVPR), 2019.
[22] Y. Oh, D.-. Kim, and I. S. Kweon, "Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning," in Proceedings of the IEEE/CVE conference computer vision and pattern recognition (CVPR), 2022.
[23] Y. Xu, L. Shang, J. Ye, Q. Qian, Y.-F. Li, B. Sun, H. Li, and Jin, "Dash: Semi-supervised learning with dynamic thresholding," in International Conference on Machine Learning (ICML), 2021.
[24] B. Zhang, Y. Wang. W. Hou, H. Wu, J. Wang, M. Okumura, and Shinozaki, "Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling." Advances in Neural Information Processing Systems (NeurIPS), 2021.
[25] H. Pham, Z. Dai, Q. Xie, and Q. V. Le, "Meta pseudo labels," in Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[26] A. Blum and T. Mitchell, "Combining labeled and unlabeled data with co-training," in Proceed- ings of the annual conference on Computational learning theory, 1998.
[27] Z.-H. Zhou and M. Li, "Tri-training: Exploiting unlabeled data using three classifiers," " IEEE Transactions on knowledge and Data Engineering, 2005.
[28) R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
[29] B. Zoph and Q V. Le, "Neural architecture search with reinforcement learning." International Conference on Leurning Representations (ICLR), 2017.
[30] D. Bahdanau, Brakel, K. Xu, A. Goyal, Lowe, J. Pineau, A. Courville, and Y. Bengio, "An actor-critic algorithm for sequence prediction," International Conference on Learning Representations (ICLR), 2017.
[31] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba, "Sequence level training with recurrent neural networks," International Conference on Learning Representations (ICLR), 2016.
[32] A. Fickinger, H. Hu, B. Amos, S. Russell, and N. Brown, "Scalable online planning via rein- forcement learning fine-tuning," Advances in Neural Information Processing Systems (NeurIPS), 2021.
[33] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems (NeurIPS), 2022.
[34] H. Robbins, "Some aspects of the sequential design of experiments." Bulletin of the American Mathematical Society, 1952.
[35] M. Rothschild, "A two-armed bandit theory of market pricing," Journal of Economic Theory, 1974.
12

---

[36] D. Bergemann and J. Välimäki, Bandit Problems. Cowles Foundation for Research in Eco- nomics, Yale University, 2006.
[37] Lattimore and C. Szepesvári, Bandit algorithms. Cambridge University Press, 2020.
[38] M. S. Mortazavi, T. Qin, and N. Yan, resenacceon. A single-step reinforcement learning method for design space exploration," arXiv preprint 2ttii::2211.020522. 2022.
[39] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," in International Conference on Learning Representations (ICLR), 2018.
[40] M. Ren, W. Zeng, B. Yang. and R. Urtasun, "Learning to reweight examples for robust deep learning," in International Conference on Machine Learning (ICML), 2018.
[41] Y. Wang, J. Guo, J. Wang, C. Wu, Song, and G. Huang, "Meta-semi: A meta-learning approach for semi-supervised learning," CAAI Artificial Intelligence Research, 2022.
[42] T. Sosea and C. Caragea, "Marginmatch: Improving semi-supervised learning with pseudo- margins," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2023.
[43] A. Krizhevsky, G. Hinton, al., "Learning multiple layers of features from tiny images." Technical report, 2009.
[44] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. "Reading digits in natural images with unsupervised feature learning," " Workshop on Deep Learning and Unsupervised Feature Learning in Advances in Neural Information Processing Systems (NeurlPS), 2011.
[45] A. Coates, A. Ng, and H. Lee, "An analysis of single-layer networks in unsupervised feature learning," " in Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011.
[46] I. Loshchilov and F. Hutter, "Sgdr: Stochastic gradient descent with warm restarts," in Interna- tional Conference on Learning Representations (ICLR), 2017.
13

---

A Computer Resources
Our experiments were performed setups featuring CPUs with 8 Intel Core processors and 64 GB of RAM. For graphics processing units, we utilized NVIDIA GeForce RTX 3060 cards, each offering 12 GB of VRAM.
B Limitation
While our RLGSSL method demonstrates significant improvements in semi-supervised learning tasks, we acknowledge certain limitations that can be addressed future work. Our current formulation assumes that the unlabeled data is drawn from the same distribution assumption might not hold certain real-world scenarios where labeled and unlabeled data come from different, albeit related, distributions. Future extensions could consider domain adaptation strategy to handle such scenarios.
the labeled data. This
Despite these limitations, the proposed work provides promising direction for integrating reinforce- ment learning with semi-supervised learning, paving the way for more adaptive and versatile machine learning algorithms.
C Broader Impacts
The proposed Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL) method has significant positive social impacts, particularly in access to advanced machine learning techniques and enabling more efficient use of data resources. By effectively leveraging both labeled and unlabeled data, RLGSSL reduce the dependency extensive and expensive labeled datasets, making high-performing machine learning models more accessible to organizations with limited labeling resources. This particularly benefit fields such as healtheare, education, and environmental monitoring, where acquiring labeled data can be challenging and costly.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Yes, the main claims in the abstract and introduction accurately reflect the paper's contributions and scope, highlighting the proposal of novel RL-Guided SSL method (RLGSSL), its innovative reward function, and the integration a teacher-student framework, validated by extensive experiments showing superior performance over state-of- the-art SSL methods.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Limitation Section in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper includes detailed descriptions of the experimental setup and parame- ters. Additionally, the paper includes the algorithm to aid reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification:
The code is currently not shared.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ussc//uuss////eeubbiii 7/nips cc/ wccppublic./. geuiic/ggiee//eeeeeeeeooeiieeeee Codeeubm bebbmmsss issiissiion aasionnoiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4.1 for details about training settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: All experiments were conducted five times, with the average results and standard deviations reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information about computer resources are provided in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Broader Impacts Section in Appendix C
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in our research are properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, thcoe com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "IvKXW6j1w5", "reviewer": "Reviewer_5CmG", "summary": "One of the bottlenecks for Semi-supervised learning (SSL) is achieving high performance with limited labeled data, as the model is often complex and needs multiple loss functions. Recently RL has been increasingly used in fine-tuning complex models with non-differentiable reward functions.  \n\nThus with these observations, the authors proposed the Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL) method, which uses RL to optimize the generation of pseudo-labels in SSL.  More specifically, the pseudo-label predictor serves as the policy function and soft pseudo-labeling acts as the actions.\n\nTechnically,  the authors formulate SSL as a one-armed bandit problem with a continuous action space and deploy a novel RL loss to guide the SSL process based on a reward function specifically designed for semi-supervised data. Moreover, they further incorporate a semi-supervised teacher-student framework to augment the RL loss with a supervised loss and a prediction consistency regularization loss, aiming to enhance learning stability and efficacy. In this way, RL could provide exploration and manage non-differentiable operations.", "questions": "1. The writing should be more concise,  and the logic here is quite messy. \n\nE.g. in \"We treat SSL as a special one-armed bandit problem with a continuous action space. One-armed  bandit problem can be considered a single-step Markov Decision Process (MDP) [39]. In this problem, the agent takes a single action and receives a reward based on that action. The state of the environment is not affected by the action....\". This content has been repeated a few times previously, but the reader still can not find the logic that connects these sentences.\n \n2. The research question in this paper is not quite emphasized, it should be focused on, with deeper analysis. \n\n\n3. Another thing is the authors may consider presenting the novelty more explicitly.  E.g. do more comparisons between related work and the proposed methods in the introduction and related work section.\n\n\n4. More related work. This is an extension of question 3, since the idea of RL to fine-tune complex models is prevailing, readers may expect the paper to show the related work of such an idea, e.g. in the paper, the authors said \"Recently, RL has been applied to fine-tune complex models that typically fail to align with users’ preferences.\", then the reader would expect more related work here. And if there exists a method that tries to combine SSL and RL in any sense, then it should appear in the baseline as well.", "limitations": "The authors did not mention the limitations of the proposed method. \n\n\nA possible limitation may fall on the training efficiency. E.g. according to Figure 1, there are multiple training loss and training objects.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of leveraging RL to optimize the pseudo-labels generator in Semi-Supervised Learning (SSL) is a good catch. \n\n2. The motivation is reasonable and in the experiment section, the author conducted broad experiments to showcase the effectiveness of the proposed method.", "weaknesses": "1. The comparison between the proposed method and other methods is not enough. E.g. the author could showcase the uniqueness of the proposed method in related work section, \n\n2. The presentation of this paper is below average. E.g. there should be more figures and the writing should be more concise and logical.\n\n3. The author did not mention the side effect of bringing RL into SSL. E.g. will the training process be more time-consuming?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort the reviewer has dedicated to reviewing our work.\n\n**About visualization**: We will add some figures to help visualize the results in future revisions of our paper. Nevertheless, the effectiveness of SSL is well captured in the test accuracy results reported, which is a primary and dominating evaluation norm in the literature. \n\n**About time complexity**: In training deep models, backpropagation is typically the most computationally intensive step. Our method, RLGSSL, features a non-differentiable reward function and a streamlined algorithm, ensuring minimal additional computational overhead. This contrasts with many state-of-the-art methods, which often significantly extend training times due to their complexity. The table below demonstrates that for a batch size of 32 on the CIFAR10 dataset using a WRN28 backbone, the Cuda time for RLGSSL is significantly lower than that for other methods in a single training iteration. This underscores its efficiency and reduced computational cost.\n\n| Algorithm      | CPU time | Cuda time   |\n|----------------|----------|-------------|\n| MixMatch       | 5.549s   | 51.039ms    |\n| FixMatch       | 5.167s   | 287.179ms   |\n| UDA            | 5.196s   | 287.088ms   |\n| RLGSSL (ours)  | 5.186s   | 23.472ms    |\n\n\n\n\n\n**About the research question**: In the 'Problem Setup' section, we formally and clearly define the research problem our study addresses, ensuring that readers clearly understand the framework and parameters within which our findings operate. \n\n**About more related works**: The 'Reinforcement Learning' subsection of  Related works in the manuscript provides a detailed overview of various studies that employ reinforcement learning (RL) techniques to tackle diverse challenges. We will expand the related work section to include additional studies that apply reinforcement learning (RL) to fine-tune complex models. It is important to note that to the best of our knowledge, our work is the first to use RL to guide semi-supervised learning (SSL), which represents the novelty of our approach.\n\n**About limitation**: The limitation section can be found in Appendix E of the manuscript. \n\n**About loss terms**: Our RLGSSL method adds just one extra loss term—the RL loss—to the standard framework of semi-supervised learning (SSL), which typically includes supervised and consistency losses.  The non-differentiable nature of the reward function within this reinforcement learning (RL) loss ensures that it does not significantly increase computational overhead during training (as shown above). Furthermore, the only trainable entity is the student model; the teacher model's parameters are updated using an Exponential Moving Average (EMA), which simplifies the training process. This streamlined approach addresses concerns about training efficiency,", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zdPuIYVbAc", "reviewer": "Reviewer_6MiT", "summary": "The authors proposes a novel Reinforcement Learning (RL) Guided semi-supervised learning (SSL) method, RLGSSL, that formulates SSL as a one-armed bandit problem and deploys an innovative RL loss based on weighted reward to guide the learning process of the prediction model adaptively. The core idea is to use RL to guide the selection of informative unlabeled samples, thereby improving the learning efficiency and effectiveness of SSL models.", "questions": "1.\tHow can the RLGSSL method be adapted to handle scenarios where the labeled and unlabeled data come from different distributions?\n2.\tHow can the hyperparameter tuning process be automated or simplified to make the RLGSSL method more user-friendly and less resource-intensive?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors have introduced RLGSSL, a new approach based on Reinforcement Learning that effectively handles Semi-Supervised Learning (SSL). This method uses RL to learn effective strategies for generating pseudo labels and guiding the learning process. \nThe authors have devised a reward function for assessing predictions that encourages the learning of accurate and reliable pseudo-labels while maintaining a balance between the usage of labeled and unlabeled data. They have also developed a new RL loss that allows the reward from pseudo-labels to be incorporated into SSL as a non-differentiable signal in a reinforced manner, promoting better generalization performance. \nFurthermore, the authors have investigated integration frameworks that combine the power of both RL loss and standard semi-supervised loss, providing a new approach that has the potential to lead to more accurate and robust SSL models. \nExtensive experiments have demonstrated that this proposed method outperforms state-of-the-art SSL approaches and validates the integration of RL strengths in SSL.", "weaknesses": "The integration of RL introduces additional computational overhead, which may require substantial computational resources, especially for large-scale datasets.\nThe current formulation assumes that the labeled and unlabeled data are drawn from the same distribution, which may not hold true in real-world scenarios. This limitation could affect the generalizability of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort the reviewer has dedicated to reviewing our work.\n\n**About computational overhead**: In training deep models, backpropagation is typically the most computationally intensive step. Our method, RLGSSL, features a non-differentiable reward function and a streamlined algorithm, ensuring minimal additional computational overhead. This contrasts with many state-of-the-art methods, which often significantly extend training times due to their complexity. The table below demonstrates that for a batch size of 32 on the CIFAR10 dataset using a WRN28 backbone, the Cuda time for RLGSSL is significantly lower than that for other methods in a single training iteration. This underscores its efficiency and reduced computational cost.\n\n| Algorithm      | CPU time | Cuda time   |\n|----------------|----------|-------------|\n| MixMatch       | 5.549s   | 51.039ms    |\n| FixMatch       | 5.167s   | 287.179ms   |\n| UDA            | 5.196s   | 287.088ms   |\n| RLGSSL (ours)  | 5.186s   | 23.472ms    |\n\n\n**About distribution mismatch scenarios**: The RL component's focus on decision-making and optimization is broadly applicable across different types of learning tasks. Therefore, while domain-specific adjustments might be necessary, the foundational idea of RLGSSL could be extended and applied to different domains and tasks. Techniques such as domain-adaptive pretraining, where models are initially trained on a source domain and then fine-tuned on a target domain, or incorporating domain adversarial training, which encourages the model to learn features that are invariant across different domains, could be particularly effective. Exploring the integration of these strategies with RLGSSL to robustly address distribution shifts could be a promising direction for future research.\n\n**About hyperparameter tuning automation**: To automate and simplify hyperparameter tuning for the RLGSSL method, leveraging tools such as Bayesian optimization[1], Hyperband[2], and AutoML frameworks can be effective. These methods efficiently explore the parameter space by balancing the exploration of new configurations with the exploitation of promising ones, thereby minimizing resource expenditure. Developing more advanced and integrated hyperparameter tuning strategies for RLGSSL could be a valuable direction for future research.\n\n\n\n\n[1] Santos, Maria. \"Bayesian Optimization for Hyperparameter Tuning.\" Journal of Bioinformatics and Artificial Intelligence 2.2 (2022).\n\n[2] Li, Lisha, et al. \"Hyperband: A novel bandit-based approach to hyperparameter optimization.\" Journal of Machine Learning Research 18.185 (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HrFPep8Ufz", "reviewer": "Reviewer_EQVV", "summary": "This paper presents a method called Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL), which frames SSL as a one-armed bandit problem. The method features a reward function that measures the discrepancy between the model's predictions on mixed data and pseudo-labels, guiding the learning process. Additionally, it employs a teacher-student model to enhance stability and reduce noise in pseudo-labels. The proposed joint loss function combines RL loss, supervised loss, and consistency regularization loss. Experiments validated the performance of the proposed method and the indispensability of each component.", "questions": "1. Why does Table 1 and Table 3 use different numbers of labeled samples for the same datasets, such as using 1000, 2000, 4000 for CIFAR-10 in Table 1, and 250, 1000, 4000 in Table 3? Additionally, why are the comparison methods different in these tables?", "limitations": "The authors have discussed limitations in their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper attempts to solve the problem of SSL from the perspective of the bandit problem, offering a fresh angle to the research.\n2. The ablation study convincingly demonstrates the indispensability of each component of the proposed method.\n3. The paper is written in a fluent and clear manner.", "weaknesses": "1. While the paper attempts to solve the SSL problem from a bandit perspective, it refers to the loss function as RL loss. The key distinction between RL and bandit problems is the presence or absence of state transitions, and the authors seem to have conflated these concepts.\n2. The paper employs bandit terminology to explain parts of the methodology where it might not be necessary. Forcing SSL into a bandit framework seems somewhat unnatural, despite the novel perspective.\n3. The paper appears to combine previously existing methods -- Regularization-Based Methods, Teacher-Student-Based Methods, and Pseudo-Labeling Methods. This raises questions about whether the innovation is sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort the reviewer has dedicated to reviewing our work.\n\n**About terminology**: The bandit problem can be viewed as a special case of reinforcement learning where there is only one transition in the trajectory. Given that the bandit problem is an older concept and less common in current literature, we formally define it as a Single-Step Markov Decision Process (SSMDP), as described in [1], to improve clarity. In this framework, we focus on maximizing the instant reward rather than the cumulative reward over multiple transitions.\n\n**About the necessity of bandit framework**: We use Reinforcement Learning (RL) and specifically one-armed bandit framework to enhance the exploration of pseudo-labels.We design the training procedure for the SSL predictor as the training of a policy function in RL. This approach enhances the predictor's performance by incorporating a novel, non-differentiable RL loss. We conducted an ablation study in Table 4 of our manuscript by dropping the RL which is the variant “$\\mathcal{R}:$ w/o sg[$\\theta$]\". This variant demonstrates poorer results compared to our RLGSSL method showcasing the necessity of the bandit framework in our work.\n\n**About novelty**: Our approach, RLGSSL, is not merely a combination of existing SSL strategies but a novel application of reinforcement learning principles specifically designed to enhance SSL. RLGSSL introduces a specialized RL framework that uses a unique prediction assessment reward function to generate accurate and reliable pseudo-labels. This method innovatively incorporates an RL-based loss to leverage the strengths of RL, promoting superior generalization performance. Our extensive experiments, particularly the ablation studies shown in Table 5, highlight the effectiveness and innovation of our approach. The significant drop in model accuracy when the RL loss is removed ( in variant $- \\text{w/o } \\mathcal{L}_\\text{rl}$ ) demonstrates the integral role of this component, confirming that RLGSSL is a fundamentally new method that transforms traditional SSL dynamics.\n\n\n**About the difference in tables**: The research field of standard semi-supervised learning (SSL) is extensive, with various papers employing different experimental setups. In this work, we have aimed to compare our method against a broad spectrum of standard SSL research. The primary difference between the setups in Table 1 and Table 3 lies in the backbone used. We rely on the results as reported in the related works in their respective papers. If a method is absent from any of the tables, it indicates that those authors did not provide results for that specific experimental setup.\n\n[1] M. S. Mortazavi, T. Qin, and N. Yan, “Theta-resonance: A single-step reinforcement learning method for design space exploration,” arXiv preprint arXiv:2211.02052, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TpZsfZqkuZ", "reviewer": "Reviewer_mW6a", "summary": "This paper introduces Reinforcement Learning-Guided Semi-Supervised Learning (RLGSSL), a novel method that combines reinforcement learning (RL) with semi-supervised learning (SSL). By formulating SSL as a one-armed bandit problem, the authors employ a RL-based loss function to guide the learning process. Additionally, the method incorporates a semi-supervised teacher-student framework to improve learning stability.", "questions": "1. In line 215, the authors mention the reward function that can enhance the model's robustness and generalization. How is this proven in the experiments?\n\n2. Could the authors further explain the description from lines 228 to 231?", "limitations": "In addition to the limitations discussed by the authors in the appendix, please refer to my comments in the Weaknesses section for further details.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a unique approach by framing semi-supervised learning as a one-armed bandit problem and integrating reinforcement learning to optimize the generation of pseudo-labels.", "weaknesses": "The rationality of the method design is not entirely convincing. Firstly, in Equation (2), why is the non-differentiable MSE used instead of the differentiable cross-entropy loss? Why use RL when supervised learning could be applied? Additionally, in the explanation of Equation (3), how is the \"meaningful weight\" defined? Would this design still be applicable when there is a significant imbalance in the number of samples for each class in the dataset?\n\nTraining with RL methods is challenging and often unstable, though RL has its own advantages. The authors are advised to add experiments demonstrating the benefits of the RL-based design compared to a supervised design.\n\nSome descriptions in the paper might be confusing for readers without background knowledge. For example, the terms \"convex combination\" in line 190 and \"fluid decision boundaries\" in line 191 might be unclear.\n\nAccording to the results in Table 3, the improvements brought by the proposed method are not significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort the reviewer has dedicated to reviewing our work.\n\n\n**About the rationality and advantage of using RL**: We use Reinforcement Learning (RL) to enhance the exploration of pseudo-labels. Traditional approaches in Semi-Supervised Learning (SSL) can encounter issues like overfitting and challenges in generating pseudo-labels. RL brings benefits of exploration and the ability to handle non-differentiable operations by treating the predictor as a policy function. Our proposed method establishes a pioneer and novel investigation framework for integrating the power of RL through non-differentiable losses with standard SSL loss. We conducted an ablation study in Table 5 of our manuscript by dropping the RL which is the variant “$\\mathcal{R}:$ w/o sg[$\\theta$]\". This variant demonstrates poorer results compared to our RLGSSL method showcasing the necessity of the bandit framework in our work.\n\n\n**About MSE loss**: We opted for the Mean Squared Error (MSE) when dealing with mixup data in the reward function, aligning with the established practices found in seminal works such as MixMatch [1], which utilizes L2 loss for this purpose.MSE loss is less sensitive to label noise compared to CE loss. Since pseudo-labels for the unlabeled data are not ground truth labels, they may contain some degree of uncertainty or noise, and hence using MSE loss can be a good option.  In addition, a non-differentiable reward function ensures that the reward function is solely employed for model assessment, rather than being directly utilized for model updating, enforcing the working mechanisms of RL. \n\n\n**About imbalance scenarios**:  In this work, we followed the established convention for research in classic semi-supervised learning (SSL). While our method has demonstrated promising results in this context, the specific scenario of class imbalance in the dataset was not directly investigated. The RL component's focus on decision-making and optimization is broadly applicable across different types of learning tasks. Therefore, while specific adjustments might be necessary, the foundational idea of RLGSSL could be extended and applied to different domains and tasks including but not limited to imbalance scenarios. Exploring the applicability and potential adaptations of our approach for imbalanced datasets presents a valuable direction for future work. \n\n**About experiments showing the advantage of RL**:  We conducted an ablation study in Table 6 of our manuscript with a variant labeled \"$\\mathcal{R}:$ w/o sg[$\\theta$]\", where we removed the stop-gradient operator from the reward function. This modification allowed the reward function to become differentiable with respect to $\\theta$, effectively transforming the setup into a supervised learning framework. This variant showed inferior performance compared to our original RLGSSL method which demonstrates the advantage of using RL.\n\n**About background knowledge**:  Given the constraints of space and the expected familiarity of NeurIPS audiences with standard machine learning terminology, we focused on detailing our novel method rather than explaining well-established concepts such as \"convex combination\" and \"fluid decision boundaries.\" We appreciate your understanding and encourage readers seeking definitions of these common terms to consult foundational ML literature or resources.\n\n**About improvement**: The improvement offered by our RLGSSL method is substantial as it significantly outperforms state-of-the-art works in semi-supervised learning (SSL). Specifically, RLGSSL surpasses the Interpolation Consistency Training (ICT) by a notable margin of 3.29% on CIFAR10 using just 1,000 labels, as detailed in Table 1 of our manuscript. Similarly, for CIFAR100 with 4,000 labeled instances, the improvement margin is 3.15%. it's noteworthy that our method in Table 3 has already achieved a very low error rate of 3.52%  for CIFAR10 using 4000 labels, where any additional improvement is inherently limited. Note that MarginMatch only achieves a 0.09% improvement over the previous Meta Pseudo-Labels method on the same dataset setting, which is much smaller than the performance gain our method achieved with an even lower error rate. \n\n\n**About effectiveness of the reward function** In the ablation studies presented in Table 5 of our manuscript, we examined the variation where \"$\\mathcal{R}=1$\", effectively removing the reward function and assigning equal rewards to all pseudo-labels. The results from this table clearly demonstrate that eliminating the reward function leads to an increased error rate in the model, thereby underscoring the critical role of the reward function in enhancing model performance.\n\n**About lines 228-231 and how the weights are defined**: We use a uniform probability distribution to represent the least informative prediction outcome in a reinforcement learning framework, where each class is considered equally likely, reflecting maximum uncertainty. The expected Kullback-Leibler (KL) divergence then measures how much the probabilistic outputs (policy outputs) of the model deviate from this non-informative, uniform distribution. By quantifying the divergence, the expected KL-divergence effectively gauges the level of informativeness or certainty in the model's predictions. This measurement is utilized as a weight for the reward function within the reinforcement learning setup. Consequently, this weighted approach incentivizes the model to produce predictions that are more distinct or discriminatory, moving away from the uniform distribution toward more informative and class-specific predictions, thereby enhancing the model's ability to discriminate effectively among different classes.\n\n\n[1] Berthelot, David, et al. \"Mixmatch: A holistic approach to semi-supervised learning.\" Advances in neural information processing systems (NeurIPS), 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel method to combine reinforcement learning (RL) with semi-supervised learning (SSL) to improve the generation of pseudo-labels. Particularly, the key insight is to first mix-up labeled and unlabled covariates, and also their label and pseudo-labels, respectively. Subsequently, the label predicted for the mixed-up covariate is made to match its mixed up label obtained from the previous step. This provides a new consistency loss for the label predictor. This loss is subsequently weighted by a KL term that encourages the labels to be distinctive, thereby preventing oversmoothing of the labels. This results in a complex objective that is treated as a non-differentiable reward to be optimized for. While reviewers raised some concerns regarding higher compute requirement when using  RL methods, authors provided in rebuttal reports of favorable compute time. Some more questions were raised regarding the choice of consistency loss (MSE vs cross entropy), and the authors provided additional experimental results in the rebuttal to justify their design choice.

Overall, the insights in the paper are interesting, and the gains in the performance are significant.

------
Some additional experiments to consider: I wonder if the method can provide even better result if the consistency loss is created for not only unlabeled-labeled pair, but also for labeled pairs of data?
Decision: Accept (poster)