Reinforcement Learning Guided Semi-Supervised Learning

Authors: marzi  heidari
hanping  zhang
yuhong  guo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 23:59:58

Abstract: In recent years, semi-supervised learning (SSL) has gained significant attention due to its ability to leverage both labeled and unlabeled data to improve model performance, especially when labeled data is scarce. However, most current SSL methods rely on heuristics or predefined rules for generating pseudo-labels and leveraging unlabeled data. They are limited to exploiting loss functions and regularization methods within the standard norm. In this paper, we propose a novel Reinforcement Learning (RL) Guided SSL method, RLGSSL, that formulates SSL as a one-armed bandit problem and deploys an innovative RL loss based on weighted reward to adaptively guide the learning process of the prediction model. RLGSSL incorporates a carefully designed reward function that balances the use of labeled and unlabeled data to enhance generalization performance. A semi-supervised teacher-student framework is further deployed to increase the learning stability. We demonstrate the effectiveness of RLGSSL through extensive experiments on several benchmark datasets and show that our approach achieves consistent superior performance compared to state-of-the-art SSL methods.

Main paper: 
 introduction: 
 Semi-supervised learning (SSL) is a significant research area in the field of machine learning, addressing the challenge of effectively utilizing limited labeled data alongside abundant unlabeled data. SSL techniques bridge the gap between supervised and unsupervised learning, offering a practical solution when labeling large amounts of data is prohibitively expensive or time-consuming. The primary goal of SSL is to leverage the structure and patterns present within the unlabeled data to improve the learning process, generalization capabilities, and overall performance of the prediction model. Over the past few years, there has been considerable interest in developing various SSL methods, and these approaches have found success in a wide range of applications, from computer vision [1] to natural language processing [2] and beyond [3, 4] .Within the SSL domain, a range of strategies has been devised to effectively utilize the information available in both labeled and unlabeled data. Broadly, SSL approaches can be categorized into three key paradigms: regularization-based, mean-teacher-based, and pseudo-labeling methodologies. Regularization-based approaches form a fundamental pillar of SSL [5] [6] [7] . These methods revolve around the core idea of promoting model robustness against minor perturbations in the input data. A quintessential example in this category is Virtual Adversarial Training (VAT) [5] . VAT capitalizes on the introduction of adversarial perturbations to the input space, thereby ensuring the model's predictions maintain consistency. The second category, Mean-teacher based methods, encapsulates a distinct class of SSL strategies that leverage the concept of temporal ensembling. This technique aids in the stabilization of the learning process by maintaining an exponential moving average of model parameters over training iterations. Mean Teacher [8] notably pioneered this paradigm with a Mean Teacher model, illustrating its efficacy across numerous benchmark tasks. Lastly, the category of pseudo-labeling approaches has attracted attention due to its simplicity and effectiveness. These methods employ the model's own predictions on unlabeled data as "pseudo-labels" to augment the training process. The MixMatch [1] framework stands as one of the leading representatives of this category, demonstrating the potential of these methods in the low-data regime. Despite these advancements, achieving high performance with limited labeled data continues to be a significant challenge in SSL, often requiring intricate design decisions and careful coordination of multiple loss functions. Recently RL has been increasingly used in fine-tuning complex models with non-differentiable reward functions. This application establishes a reinforced alignment with the learning objective and enhances generalization to standard supervised learning scenarios [9] [10] [11] [12] . Enthusiastic about the potential of enhanced generalizability introduced by a non-differentiable RL reward to SSL, we propose to approach SSL outside the conventional design norms by developing a Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL) method, which brings a fresh perspective to SSL. We employ Reinforcement Learning (RL) to optimize the generation of pseudo-labels in Semi-Supervised Learning (SSL). Conventional methods in SSL often face problems such as overfitting and difficulties in creating accurate pseudo-labels. RL introduces advantages in exploration capabilities and adeptly manages non-differentiable operations by considering the predictor as a policy function.In RLGSSL, we formulate SSL as a bandit problem, a special case of reinforcement learning, where the prediction model serves as the policy function, and soft pseudo-labeling acts as the actions. We define a simple reward function that balances the use of labeled and unlabeled data and improves generalization capacity by leveraging linear data interpolation, while the prediction model is trained under a policy gradient framework to maximize the policy-output weighted reward. Formulating the SSL problem as such an RL task allows our approach to dynamically adapt and respond to the data. Moreover, we further deploy a teacher-student learning framework to enhance the stability of learning. Additionally, we integrate a supervised learning loss to improve and accelerate the learning process. This new SSL framework has the potential to pave the way for more robust, flexible, and adaptive SSL methods. We evaluate the proposed method through extensive experiments on benchmark datasets. The contribution of this work can be summarized as follows:• We propose RLGSSL, a novel Reinforcement Learning based approach that effectively tackles SSL by leveraging RL's power to learn effective strategies for generating pseudolabels and guiding the learning process. • We design a prediction assessment reward function that encourages the learning of accurate and reliable pseudo-labels while maintaining a balance between the usage of labeled and unlabeled data. • We develop an innovative RL loss that allows reward from pseudo-labels to be incorporated into SSL as a non-differentiable signal in a reinforced manner, promoting better generalization performance. • We conduct a novel investigation on integration frameworks that combine the power of both RL loss and standard semi-supervised loss, providing a brand new approach that has the potential to lead to more accurate and robust SSL models. • Extensive experiments demonstrate that our proposed method outperforms state-of-the-art SSL approaches and validate the integration of RL strengths in SSL.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Yes, the main claims in the abstract and introduction accurately reflect the paper's contributions and scope, highlighting the proposal of a novel RL-Guided SSL method (RLGSSL), its innovative reward function, and the integration of a teacher-student framework, validated by extensive experiments showing superior performance over state-ofthe-art SSL methods.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
semi-supervised learning: 
 Existing SSL approaches can be broadly classified into three primary categories: regularization-based methods, teacher-student-based methods, and pseudo-labeling techniques.A prevalent research direction in SSL focuses on regularization-based methods, which introduce additional terms to the loss function to promote specific properties of the model. For instance, the Π-model [6] and Temporal-Ensemble [6] incorporate consistency regularization into the loss function, with the latter employing the exponential moving average of model predictions. Virtual Adversarial Training (VAT) [5] is yet another regularization-based technique that aims to make deep neural networks robust to adversarial perturbations. In a similar vein, Consistency Regularization for Generative Adversarial Networks (CR-GAN) [7] integrates a generative adversarial network (GAN) with a consistency regularization term, facilitating the effective generation of pseudo-labels for unlabeled data.Teacher-student-based methods offer an alternative approach in SSL research. These techniques train a student network to align its predictions with those of a teacher network on unlabeled data. Mean Teacher (MT) [8] , a prominent example in this category, leverages an exponential moving average (EMA) of successive weights from the student model to obtain the teacher model. To enhance performance, MT + Fast SWA [13] combines Mean Teacher with Fast Stochastic Weight Averaging. Smooth Neighbors on Teacher Graphs (SNTG) [14] takes a different approach, utilizing a graph for the teacher to regulate the distribution of features in unlabeled samples. Meanwhile, Interpolation Consistency Training (ICT) [15] aims to promote consistent predictions across interpolated data points by ensuring that a model's predictions on an interpolated set of unlabeled data points remain consistent with the interpolation of the predictions on those points.Pseudo-labeling is an effective way to extend the labeled set when the number of labels is limited. Pseudo-Label [16] produces labels for unlabeled data using model predictions and filters out lowconfidence predictions. MixMatch [1] employs data augmentation to create multiple input versions, obtaining predictions for each and averaging them to generate pseudo-labels. In contrast, works such as ReMixMatch [17] , UDA [18] , and FixMatch [19] apply confidence thresholds to produce pseudo-labels for weakly augmented samples, which subsequently serve as annotations for strongly augmented samples. Label propagation methods, including TSSDL [20] and LPD [21] , assign pseudolabels based on local neighborhood density. DASO [22] combines confidence-based and density-based pseudo-labels in varying ways for each class. Approaches such as Dash [23] and FlexMatch [24] dynamically adjust confidence thresholds in a curriculum learning manner to generate pseudo-labels. Meta Pseudo-Labels [25] uses a bi-level optimization strategy, deriving the teacher update rule from student feedback, to learn from limited labeled data. Co-Training [26] is an early representative of pseudo-lableing which involves training two classifiers on distinct subsets of unlabeled data and using confident predictions to produce pseudo-labels for one another. Similarly, Tri-Training [27] trains three classifiers on separate unlabeled data subsets and generates pseudo-labels based on the disagreements between their predictions.Reinforcement Learning (RL) is a field of study that focuses on optimizing an agent's decision-making abilities by maximizing the cumulative reward obtained through interactions with its environment [28] . RL methodology has been widely applied to solve many other learning problems, including searching for optimized network architectures [29] , training sequence models for text generation by receiving reward signals [30, 31] , and solving online planning problems [32] . Recently, RL has been applied to fine-tune complex models that typically fail to align with users' preferences. Moreover, based on RL from Human Feedback (RLHF; [9] [10] [11] ), ChatGPT achieves great success in dialogue generation by fine-tuning Large Language Models (LLM) [33] . It frames the training of LLM as a bandit problem, specifically a one-armed bandit problem [28] , where the objective is to determine the optimal action (dialogue generation) for a given state (user prompt) within a single step, demonstrating the capacity of RL for prediction tasks.The bandit problem was originally described as a statistical decision model used by agents to optimize their decision-making process [34] . In this problem, an agent receives a reward upon taking an action and learns to make the best decision by maximizing the given reward. The bandit problem found its application in economics and has been widely used in market learning, specifically in finding the optimal market demands or prices to maximize expected profits [35] . Bergemann et al. [36] and Lattimore et al. [37] have extensively discussed the literature and modern applications of the bandit problem. Additionally, Mortazavi et al. [38] serve as the policy functions, and the soft pseudo-labeling (P θ T (X u )) acts as the actions. The model has three loss terms in total: RL loss (L rl ), supervised loss (L sup ), and consistency loss (L cons ). The teacher policy function is used to execute the actions and compute the consistency loss, while the student policy function is used for all other aspects.(SSMDP) to formulate the bandit problem in a manner that aligns with modern RL techniques. This advancement enables the utilization of standard RL methods on conventional bandit problems.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer to Limitation Section in Appendix B Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: Information about computer resources are provided in Appendix A. 
the proposed method: 
 We consider the following semi-supervised learning setting: the training data consist of a small number of labeled samples, D l = (X l , Y l ) = {(x l i , y l i )} N l i=1 , and a large number of unlabeled samples, D u = X u = {x u i } N u i=1 , with N u ≫ N l , where x l i (or x u i ) denotes the input instance and y l i denotes the one-hot label vector with length C. The goal is to train a C-class classifier f θ : X → Y that generalizes well to unseen test data drawn from the same distribution as the training data.In this section, we present the proposed RLGSSL method, which formulates SSL as a one-armed bandit problem with a continuous action space, and deploys a novel RL loss to guide the SSL process based on a reward function specifically designed for semi-supervised data. Moreover, we further incorporate a semi-supervised teacher-student framework to augment the RL loss with a supervised loss and a prediction consistency regularization loss, aiming to enhance the learning stability and efficacy. Figure 1 illustrates the overall framework of the proposed RLGSSL, and the following subsections will elaborate on the approach.We treat SSL as a special one-armed bandit problem with a continuous action space. One-armed bandit problem can be considered a single-step Markov Decision Process (MDP) [38] . In this problem, the agent takes a single action and receives a reward based on that action. The state of the environment is not affected by the action. The one-armed bandit problem involves selecting an action to maximize an immediate reward, which can be regarded as learning a policy function under the RL framework. Formulating SSL as a one-armed bandit problem within the RL framework and deploying RL techniques to guide SSL requires defining the following key components: state space S, action space A, a policy function π : S → A, and a reward function R : S × A → R. The objective is to learn an optimal policy π ⋆ that maximizes the expected one-time reward R(s, π(•|s)) in the given environment (state s): π ⋆ = arg max π J r (π) = a π(a|s)R(s, a).The state encapsulates the provided knowledge about the environment and is used as input for the policy function. As the action does not affect the state of the environment under one-armed bandit problem, we use the observed data from the SSL problem as the state, i.e., s = (X l , Y l , X u ).As the goal of SSL is to learn an optimal classifier f θ (i.e., prediction network parameterized with θ), we use the classifier f θ , usually denoted by its parameters θ, as the policy function π θ to unify the goals of RL and SSL. In particular, we consider a probabilistic policy function/prediction network π θ (•) = P θ (•). Since a policy function is used to project a mapping from the state s to the action space, a = π θ (•|s), by using a probabilistic prediction network as the policy function, it naturally determines a continuous action space A. Specifically, given the fixed state s, taking an action is equivalent to making probabilistic predictions on the unlabeled data in s:Y u = P θ (X u ) = π θ (•|s), as the labeled data already has labels. For each unlabeled instance x u i , the action is a probability vector produced as y u i = P θ (x u i ), which can be regarded as soft pseudo-labels in the SSL setting. This links the action of RL to the pseudo-labeling in SSL.The reward function serves as feedback to evaluate the performance of the action (prediction) provided by the policy. It needs to be thoughtfully crafted to maximize the model's ability to extract useful information from both labeled and unlabeled data, which is central to the SSL paradigm. The underlying motivation is to guide the learning process to induce a more generalizable and robust prediction model. To this end, we adopt a data mixup [39] strategy to produce new data points from the given labeled data (X l , Y l ) and pseudo-labeled data (X u , Y u ), which together form the state-action pair (s, a), through linear data interpolation, and assess the prediction model's generalization ability on such data points as the reward signal. This decision is inspired by the proven effectiveness of mixup in enhancing model performance in various tasks. The idea of data mixup is to generate virtual training examples by creating convex combinations of pairs of input data and their corresponding labels. This technique encourages the model to learn more fluid decision boundaries, leading to improved generalization capabilities. Specifically, we propose to generate new data points by performing inter-mixup between labeled and unlabeled data points, aiming to maintain a balanced utilization of both labeled and unlabeled data. In order to address the size discrepancy between the labeled dataset D l and the unlabeled dataset D u with N u ≫ N l , we replicate the labeled dataset D l by a factor of r = ⌈ N u N l ⌉ times, resulting in an extended labeled dataset D l . After shuffling the data points in each set, we generate a mixup data point by mixing an unlabeled point x u i ∈ D u with a labeled point x l i ∈ D l along with their corresponding pseudo-label y u i ∈ D u and label y l i ∈ D l :EQUATION)where the mixing parameter µ is sampled from a Beta distribution, Beta (1, 1) . With this procedure, we can generate N m = N u mixup samples by mixing all the unlabeled samples with the samples in the extended labeled set.We then define the reward function to measure the negative mean squared error (MSE) between the model's prediction P θ (x m i ) and the mixup label y m i for each instance in the mixup set. This results in a single, comprehensive metric that quantifies the overall negative disagreement between the model's predictions and the mixup labels over a large set of interpolated data points:EQUATION)where C denotes the number of classes and sg[•] is the stop gradient operator which stops the flow of gradients during the backpropagation process. This ensures that the reward function is solely employed for model assessment, rather than being directly utilized for model updating, enforcing the working mechanisms of RL. Mixup labels capture both the supervision information in the labeled data and the uncertainty in the pseudo-labels of unlabeled data. With the designed reward function, a good reward value can only be returned when the prediction model not only exhibits strong alignment with the labeled data but also delivers accurate predictions on the unlabeled data. Consequently, through RL, this reward function will not only promote accurate predictions but also enhance the model's robustness and generalizability.By deploying the probabilistic predictions on the unlabeled data, Y u = π θ (X u ) = P θ (X u ), as the action, we adopt a deterministic policy. Following the principle of one-armed bandit problem on maximizing the expected one-time reward w.r.t. the policy output, we introduce a weighted negative reward based on the deterministic policy's output as the RL loss for the proposed RLGSSL, thereby exploiting non-differentiable reward signals while enabling policy gradient with a deterministic policy. Specifically, we treat the output of the policy network, Y u , as a uniform distribution over the set of N u probability vectors, {y u 1 , • • • , y u N u }, predicted for the unlabeled instances. Let e = 1/C denote a discrete uniform distribution vector with length C-the number of classes. We design the following KL-divergence weighted negative reward as the RL loss:EQUATIONwhere the KL-divergence term measures the distance of each label prediction probability vector y u i from a uniform distribution vector; R(s, a; sg[θ S ]) is treated as a non-differentiable reward function.Given that a uniform probability distribution signifies the least informative prediction outcome, the expected KL-divergence captures the level of informativeness in the policy output and hence serves as a meaningful weight for the reward, which inherently encourages the predictions to exhibit greater discrimination.The minimization of this loss function over the prediction network parameterized by θ is equivalent to learning an optimal policy function π θ by maximizing the KL-divergence weighted reward, which aims at an optimal policy function (also the probabilistic classifier P θ ) that not only maximizes the reward signal but is also discriminative. From the perspective of SSL, the utilization of this novel RL loss introduces a fresh approach to designing prediction loss functions. Instead of directly optimizing the alignment between predictions and targets, it offers a gradual learning process guided by reward signals. This innovative approach presents a more adaptive and flexible solution for complex data scenarios, where traditional optimization-based methods may fall short.Teacher-student models [8] have been popularly deployed to exploit unlabeled data for SSL, improving the learning stability. We extend this mechanism to provide a teacher-student framework for RLguided SSL by maintaining a dual set of model parameters: the student policy/model parameters θ S , and the teacher policy/model parameters θ T . The student model is directly updated through training, whereas the teacher model is updated via an exponential moving average (EMA) of the student model. The update is conducted as follows:EQUATIONwhere β denotes a hyperparameter that modulates the EMA's decay rate. The utilization of the EMA update method ensures a stable and smooth transfer of knowledge from the student model to the teacher model. This leads to a teacher model with consistent and reliable parameter values that are not susceptible to random or erratic fluctuations during the training process. Leveraging this desirable characteristic, we propose to employ the teacher model for executing actions within the RL framework described in the section 3.1 above; that is, Y u = P θ T (X u ), while retaining the student model for other aspects. By doing so, we ensure that stable actions are taken, reducing the impact of random noise in the pseudo-labels and enhancing the accuracy of reward evaluation and reinforcement strength.Within the teacher-student framework, we further propose to augment the RL loss with a supervised loss L sup on the labeled data and a consistency regularization loss L cons on the unlabeled data. We adopt a standard cross-entropy loss function ℓ CE to compute the supervised loss, promoting accurate predictions on D l where the ground-truth labels are available:EQUATIONThis loss can enhance effective exploitation of the ground-truth label information, providing a solid basis for exploring the parameter space via RL. The consistency loss L cons is deployed to encourage prediction consistency between the student and teacher models on the unlabeled data D u :EQUATIONwhere KL(•, •) denotes the Kullback-Leibler divergence between two probability distributions. By enforcing consistency, this loss encourages the student model to make more confident and reliable predictions, reducing the impact of random or misleading information in the training set. It also acts as a form of regularization, discouraging the student model from overfitting the labeled data.Input: D l , D u , and extended D l ; initialized θ S , θ T ; hyperparameters for iteration = 1 to maxiters do for x u i ∈ D u do Compute soft pseudo-label vector y u i = P θ T (x u i ) to form (x u i , y u i ) end for Generate mixup data D m =(X m , Y m ) on D u and D l using Eq.( 1) with shuffling for step = 1 to maxsteps do Draw a batch of data B = {(x m i , y m i )} from D m Calculate the reward function R(•; sg[θ S ]) using the batch B Compute the objective in Eq.( 7) Update the policy parameters θ S via gradient descent Update teacher model θ T via EMA in Eq.( 4) end for end for The learning objective for the RLGSSL approach is formed by combining the reinforcement learning loss L rl with the two augmenting loss terms, the supervised loss L sup and the consistency loss L cons , using hyperparameters λ 1 and λ 2 :EQUATIONBy deploying such a joint loss, the RLGSSL framework can benefit from the strengths of both reinforcement exploration and semi-supervised learning. The RL component, in particular, introduces a dynamic aspect to the learning process, enabling the model to improve iteratively based on its own experiences. This innovative combination of losses allows the model to effectively learn from limited labeled data while still exploiting the abundance of unlabeled data.We develop a stochastic batch-wise gradient descent algorithm to minimize the joint objective in Eq. ( 7) for RL-guided semi-supervised training. The procedure of this algorithm is summarized in Algorithm 1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
experimental setup: 
 Datasets We conducted comprehensive experiments on four image classification benchmarks: CIFAR-10, CIFAR-100 [43] , SVHN [44] , and STL-10 [45] . We adhere to the conventional dataset splits used in the literature. Consistent with previous works, on each dataset we preserved the labels of a randomly selected subset of training samples with an equal number of samples for each class, and left the remaining samples unlabeled. In order to compare with previous works in the same settings, we performed experiments on CIFAR-10 with various numbers (N l ∈ {250, 1, 000, 2, 000, 4, 000}) of labeled samples, on CIFAR-100 with 2,500, 10,000 and 4,000 labeled samples, on SVHN with 1,000 and 500 labeled samples, and on STL-10 with 1,000 labeled images.We conducted experiments using four different network architectures used in the literature: a 13-layer Convolutional Neural Network (CNN-13), a Wide-Residual Network with 28 layers and a widening factor of 2 (WRN-28-2), a Wide-RestNet-28-8 (WRN-28-8) and a Wide-RestNet-37-2 (WRN-37-2). For training the CNN-13 architecture, we employed the SGD optimizer with a Nesterov momentum of 0.9. We used an L2 regularization coefficient of 1e-4 for CIFAR-10 and CIFAR-100, and 5e-5 for SVHN. The initial learning rate was set to 0.1, and the cosine learning rate annealing technique proposed in previous studies [46, 15] was utilized. For the WRN-28-2 architecture, we followed the suggestion from MixMatch [1] and used an L2 regularization coefficient of 4e-4. For WRN-37-2, the training configuration includes the SGD optimizer, an L2 regularization coefficient of 5e-4, and an initial learning rate of 0.01. Finally, the training configuration for the WRN-28-8 model includes using the SGD optimizer, an L2 regularization coefficient of 0.001, and starting with a learning rate of 0.01. To compute the parameters of the teacher model, we employed the EMA method with a decay rate β = 0.999. We selected all hyperparameters and training techniques based on relevant studies to ensure a fair comparison between our approach and the existing methods. Specifically for RLGSSL, we set the batch size to 128, and set λ 1 = λ 2 = 0.1. We first pre-train the model for 50 epochs using the Mean-Teacher algorithm and then proceed to the training procedure of RLGSSL for 400 epochs. We ran each experiment five times and reported the mean test errors with their standard deviations.We compare RLGSSL with a great number of SSL algorithms, including Supervised + MixUp [39] , Π-model [6] , Temp-ensemble [6] , Mean Teacher [8] , VAT [5] , SNTG [14] , Learning to Reweight [40] , MT + Fast SWA [13] , MixMatch [1] , ReMixMatch [17] , FixMatch [19] , MarginMatch [42] , Meta-Semi [41] , Meta Pseudo-Labels [25] , and ICT [15] , using CNN-13, WRN-28-2, WRN-28-8 or WRN-37-2 as the backbone network.Table 1 reports the comparison results on CIFAR-10 with 4,000, 2,000, and 1,000 labeled samples and on CIFAR-100 with 10,000 and 4,000 labeled samples when CNN-13 is used as the backbone network. On CIFAR-10, RLGSSL outperforms all the other compared methods across all settings with different numbers of labeled samples. With 1,000 labeled samples, RLGSSL surpasses the second best method, Learning to Reweight, by a significant margin of 2.59% on CIFAR-10, achieving an average test error of 9.15%. This pattern of outperformance continues with different numbers (2,000 and 4,000) of labeled samples, where RLGSSL yields lowest test error rates and outperforms ICT-the second best method-by a margin of 1.79% and 1.07% respectively. The results on the CIFAR-100 dataset are similarly impressive. For 4,000 labeled samples, RLGSSL again outperforms ICT, the second best method, with a margin of 3.15%. As the number of labeled samples escalates to 10,000, RLGSSL maintains its performance edge, outperforming the second best method ICT by a margin of 3.12%.Table 2 reports the comparison results on the SVHN dataset with CNN-13 as the backbone network. Our method, RLGSSL, surpasses all the other compared SSL techniques for both settings. Specifically, for 500 labeled samples, RLGSSL achieves the lowest test error of 3.12%, which is 0.87% lower than the second-best method, SNTG. For 1,000 labeled samples, RLGSSL also shows superior performance with a test error of 3.05%, outperforming the second-best method, SNTG, by 0.81%.Table 3 presents the comparative outcomes across three datasets, utilizing WRN-28-2 as the backbone network for CIFAR-10 and SVHN, and WRN-28-8 for CIFAR-100. On CIFAR-10, with the number of labeled samples increasing from 250 to 4,000, RLGSSL showed better performance compared to the other methods in most cases. For 1,000 labeled samples, our method improved over the best competing method, ReMixMatch, by a margin of 1.35%. Additionally, for 4,000 labeled samples RLGSSL outperforms Meta Pseudo-labels which is the second best method. On CIFAR-100, with 2,500 and 10,000 labeled samples, RLGSSL surpassed the state-of-the-art MarginMatch. Moreover, on SVHN with 1,000 labeled samples, RLGSSL also outperformed the MarginMatch. This confirms the robustness of RLGSSL across various settings, even when the labeled data is limited in quantity.Table 4 presents the comparison results of various SSL methods on the STL-10 dataset, utilizing WRN-37-2 as the backbone network. With a fixed number of 1,000 labeled samples, RLGSSL achieves remarkable performance with a mean test error of 6.12%, which outperforms previous state-of-the-art methods, including MixMatch and UDA, showcasing the effectiveness of our RLGSSL.In order to evaluate the significance of various components of our RLGSSL approach, we conducted an ablation study on the CIFAR-100 dataset using the CNN-13 network. In particular, we compared the full model RLGSSL with the following variants: (1) "-w/o L rl ", which drops the RL loss L rl ;(2) "-w/o L sup ", which excludes the supervised loss; (3) "-w/o L cons ", which does not include the consistency loss; (4) "-w/o EMA", which drops the teacher model by disabling the EMA update; and ( 5) "-w/o mixup", which only uses unlabelled data in the reward function (µ = 1), and the mixup operation is excluded. The ablation results are reported in the top section of Table 5 . The full model, RLGSSL, achieved the lowest test errors, confirming the overall effectiveness of our method.The most significant observation from our study lies in the removal of the RL loss L rl . Upon removal of this component, we witness a substantial increase in test errors, which highlights the indispensable role played by the RL component in our model. The ablation study further illustrates the importance of each component by analyzing the performance of the model when the component is removed. In each of these cases, we observe that the removal of any individual component consistently leads to an increase in test errors. This finding underpins the notion that each component of the RLGSSL model plays a significant role in the overall performance of the model.In addition, we also conducted another set of ablation study centered on the proposed RL loss and the reward function. We compared the full model RLGSSL with the following variants: (1) "R = 1", 2) with the KL-divergence loss; (4) "R(MSE → JS)", which replaces the mean squared error of the reward function in Eq. ( 2) with the JS-divergence loss; and ( 5) "R : w/o sg[θ]", which removes the stop-gradient operator from the reward function and makes the reward function differentiable w.r.t θ. The ablation results are reported in the bottom section of Table 5 . We can see that all these variants with altered reward functions produced degraded performance comparing to the full model with the proposed reward function. In particular, the performance degradation of "R = 1" and "R : w/o sg[θ]" that drop RL in different manners further validates the contribution of the proposed framework of guiding SSL with RL.We conduct sensitivity analysis over the two hyper-parameters of the proposed RLGSSL: λ 1 -the trade-off parameter for the supervised loss, and λ 2 -the trade-off parameter for the consistency loss.The results are reported in Figure 2 . In the case of λ 1 , lower values (e.g., 1e-4 and 1e-3) result in less emphasis on the supervised loss term in the objective function. As a result, the model might not learn effectively from the limited available labeled data, leading to increased test error rates. Conversely, higher values of λ 1 (e.g., 0.2 and 0.3) may overemphasize the supervised loss term, potentially causing the model to overfit the labeled data and ignore useful information from the unlabeled data.The sweet spot lies in the middle (around 0.1), attaining a balance between learning from labeled data and leveraging information from unlabeled data. Regarding λ 2 , very low values (e.g., 1e-4 and 1e-3) may not enforce sufficient consistency in the model predictions on unlabeled data, resulting in a model that fails to generalize well. However, if λ 2 is too high (e.g., 0.2 and 0.3), the model may overemphasize the consistency constraint, possibly leading to a model that is too rigid to capture the diverse patterns in the data. An optimal value of λ 2 (around 0.1 in our experiments) ensures a good balance between encouraging prediction consistency and allowing the model to adapt to the diverse patterns in the data. The optimal value choice for hyperparameters λ 1 and λ 2 (around 0.1) also validates that the RL loss is the main leading term, while the supervised loss and consistency loss are augmenting terms. 
conclusion: 
 In this paper, we presented Reinforcement Learning-Guided Semi-Supervised Learning (RLGSSL), a unique approach that integrates the principles of RL to tackle the challenges inherent in SSL. This initiative was largely driven by the limitations of conventional SSL techniques. RLGSSL employs a distinctive strategy where an RL-optimized reward function is utilized. This function adaptively promotes better generalization performance through more effectively leveraging both labeled and unlabeled data. We also further incorporated a student-teacher framework to integrate the strengths of RL and SSL. Extensive evaluations were conducted on multiple benchmark datasets, comparing RLGSSL to existing state-of-the-art SSL techniques, and various ablation variants. RLGSSL consistently outperformed these other techniques across all the datasets, which attests to the effectiveness and generalizability of our approach. The results underline the potential of integrating RL principles into SSL, and the RLGSSL method introduced in this paper is a significant stride in this direction. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please refer to Broader Impacts Section in Appendix C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All datasets used in our research are properly credited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Our experiments were performed on setups featuring CPUs with 8 Intel Core processors and 64 GB of RAM. For graphics processing units, we utilized NVIDIA GeForce RTX 3060 cards, each offering 12 GB of VRAM.While our RLGSSL method demonstrates significant improvements in semi-supervised learning tasks, we acknowledge certain limitations that can be addressed in future work. Our current formulation assumes that the unlabeled data is drawn from the same distribution as the labeled data. This assumption might not hold in certain real-world scenarios where labeled and unlabeled data come from different, albeit related, distributions. Future extensions could consider a domain adaptation strategy to handle such scenarios.Despite these limitations, the proposed work provides a promising direction for integrating reinforcement learning with semi-supervised learning, paving the way for more adaptive and versatile machine learning algorithms.The proposed Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL) method has significant positive social impacts, particularly in access to advanced machine learning techniques and enabling more efficient use of data resources. By effectively leveraging both labeled and unlabeled data, RLGSSL can reduce the dependency on extensive and expensive labeled datasets, making high-performing machine learning models more accessible to organizations with limited labeling resources. This can particularly benefit fields such as healthcare, education, and environmental monitoring, where acquiring labeled data can be challenging and costly.NeurIPS Paper ChecklistJustification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper introduces a novel method to combine reinforcement learning (RL) with semi-supervised learning (SSL) to improve the generation of pseudo-labels. Particularly, the key insight is to first mix-up labeled and unlabled covariates, and also their label and pseudo-labels, respectively. Subsequently, the label predicted for the mixed-up covariate is made to match its mixed up label obtained from the previous step. This provides a new consistency loss for the label predictor. This loss is subsequently weighted by a KL term that encourages the labels to be distinctive, thereby preventing oversmoothing of the labels. This results in a complex objective that is treated as a non-differentiable reward to be optimized for. While reviewers raised some concerns regarding higher compute requirement when using  RL methods, authors provided in rebuttal reports of favorable compute time. Some more questions were raised regarding the choice of consistency loss (MSE vs cross entropy), and the authors provided additional experimental results in the rebuttal to justify their design choice.

Overall, the insights in the paper are interesting, and the gains in the performance are significant.

------
Some additional experiments to consider: I wonder if the method can provide even better result if the consistency loss is created for not only unlabeled-labeled pair, but also for labeled pairs of data?

Reviews: review: paper_summary: paper_summary: One of the bottlenecks for Semi-supervised learning (SSL) is achieving high performance with limited labeled data, as the model is often complex and needs multiple loss functions. Recently RL has been increasingly used in fine-tuning complex models with non-differentiable reward functions.  

Thus with these observations, the authors proposed the Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL) method, which uses RL to optimize the generation of pseudo-labels in SSL.  More specifically, the pseudo-label predictor serves as the policy function and soft pseudo-labeling acts as the actions.

Technically,  the authors formulate SSL as a one-armed bandit problem with a continuous action space and deploy a novel RL loss to guide the SSL process based on a reward function specifically designed for semi-supervised data. Moreover, they further incorporate a semi-supervised teacher-student framework to augment the RL loss with a supervised loss and a prediction consistency regularization loss, aiming to enhance learning stability and efficacy. In this way, RL could provide exploration and manage non-differentiable operations.
strength_weakness: strength_weakness: 1. The comparison between the proposed method and other methods is not enough. E.g. the author could showcase the uniqueness of the proposed method in related work section, 

2. The presentation of this paper is below average. E.g. there should be more figures and the writing should be more concise and logical.

3. The author did not mention the side effect of bringing RL into SSL. E.g. will the training process be more time-consuming?
questions: questions: 1. The writing should be more concise,  and the logic here is quite messy. 

E.g. in "We treat SSL as a special one-armed bandit problem with a continuous action space. One-armed  bandit problem can be considered a single-step Markov Decision Process (MDP) [39]. In this problem, the agent takes a single action and receives a reward based on that action. The state of the environment is not affected by the action....". This content has been repeated a few times previously, but the reader still can not find the logic that connects these sentences.
 
2. The research question in this paper is not quite emphasized, it should be focused on, with deeper analysis. 


3. Another thing is the authors may consider presenting the novelty more explicitly.  E.g. do more comparisons between related work and the proposed methods in the introduction and related work section.


4. More related work. This is an extension of question 3, since the idea of RL to fine-tune complex models is prevailing, readers may expect the paper to show the related work of such an idea, e.g. in the paper, the authors said "Recently, RL has been applied to fine-tune complex models that typically fail to align with users’ preferences.", then the reader would expect more related work here. And if there exists a method that tries to combine SSL and RL in any sense, then it should appear in the baseline as well.
limitations: limitations: The authors did not mention the limitations of the proposed method. 


A possible limitation may fall on the training efficiency. E.g. according to Figure 1, there are multiple training loss and training objects.
paper_summary: paper_summary: The authors proposes a novel Reinforcement Learning (RL) Guided semi-supervised learning (SSL) method, RLGSSL, that formulates SSL as a one-armed bandit problem and deploys an innovative RL loss based on weighted reward to guide the learning process of the prediction model adaptively. The core idea is to use RL to guide the selection of informative unlabeled samples, thereby improving the learning efficiency and effectiveness of SSL models.
strength_weakness: strength_weakness: The integration of RL introduces additional computational overhead, which may require substantial computational resources, especially for large-scale datasets.
The current formulation assumes that the labeled and unlabeled data are drawn from the same distribution, which may not hold true in real-world scenarios. This limitation could affect the generalizability of the model.
questions: questions: 1.	How can the RLGSSL method be adapted to handle scenarios where the labeled and unlabeled data come from different distributions?
2.	How can the hyperparameter tuning process be automated or simplified to make the RLGSSL method more user-friendly and less resource-intensive?
limitations: limitations: yes
paper_summary: paper_summary: This paper presents a method called Reinforcement Learning Guided Semi-Supervised Learning (RLGSSL), which frames SSL as a one-armed bandit problem. The method features a reward function that measures the discrepancy between the model's predictions on mixed data and pseudo-labels, guiding the learning process. Additionally, it employs a teacher-student model to enhance stability and reduce noise in pseudo-labels. The proposed joint loss function combines RL loss, supervised loss, and consistency regularization loss. Experiments validated the performance of the proposed method and the indispensability of each component.
strength_weakness: strength_weakness: 1. While the paper attempts to solve the SSL problem from a bandit perspective, it refers to the loss function as RL loss. The key distinction between RL and bandit problems is the presence or absence of state transitions, and the authors seem to have conflated these concepts.
2. The paper employs bandit terminology to explain parts of the methodology where it might not be necessary. Forcing SSL into a bandit framework seems somewhat unnatural, despite the novel perspective.
3. The paper appears to combine previously existing methods -- Regularization-Based Methods, Teacher-Student-Based Methods, and Pseudo-Labeling Methods. This raises questions about whether the innovation is sufficient.
questions: questions: 1. Why does Table 1 and Table 3 use different numbers of labeled samples for the same datasets, such as using 1000, 2000, 4000 for CIFAR-10 in Table 1, and 250, 1000, 4000 in Table 3? Additionally, why are the comparison methods different in these tables?
limitations: limitations: The authors have discussed limitations in their work.
paper_summary: paper_summary: This paper introduces Reinforcement Learning-Guided Semi-Supervised Learning (RLGSSL), a novel method that combines reinforcement learning (RL) with semi-supervised learning (SSL). By formulating SSL as a one-armed bandit problem, the authors employ a RL-based loss function to guide the learning process. Additionally, the method incorporates a semi-supervised teacher-student framework to improve learning stability.
strength_weakness: strength_weakness: The rationality of the method design is not entirely convincing. Firstly, in Equation (2), why is the non-differentiable MSE used instead of the differentiable cross-entropy loss? Why use RL when supervised learning could be applied? Additionally, in the explanation of Equation (3), how is the "meaningful weight" defined? Would this design still be applicable when there is a significant imbalance in the number of samples for each class in the dataset?

Training with RL methods is challenging and often unstable, though RL has its own advantages. The authors are advised to add experiments demonstrating the benefits of the RL-based design compared to a supervised design.

Some descriptions in the paper might be confusing for readers without background knowledge. For example, the terms "convex combination" in line 190 and "fluid decision boundaries" in line 191 might be unclear.

According to the results in Table 3, the improvements brought by the proposed method are not significant.
questions: questions: 1. In line 215, the authors mention the reward function that can enhance the model's robustness and generalization. How is this proven in the experiments?

2. Could the authors further explain the description from lines 228 to 231?
limitations: limitations: In addition to the limitations discussed by the authors in the appendix, please refer to my comments in the Weaknesses section for further details.

Comments: comment: Thank you for your reply.

Although I still have some concerns, I do find the idea of this paper genuinely interesting, so I’ve decided to increase my score by 1 point.

Good luck!
We thank the reviewer for the response.

 We would like to clarify that the bandit framework is not simply a narrative tool within our study. The entire training process is built upon the bandit framework, where the policy function is optimized through iterative interactions with pseudo labels based on feedback from the reward function, rather than using a simple policy gradient. Although there are differences between the solutions for bandit problems and traditional RL problems, the primary goal of the bandit problem remains to maximize the reward function. Our carefully designed KL-divergence weighted negative reward, as discussed in Section 3.1.2, is well-suited to the bandit setting and serves as an effective solution for maximizing the instant reward received by the agent through our policy.  

This formulation is not merely theoretical but operational, with the bandit's reward mechanism directly influencing the learning process through the RL loss. As shown in our experiments (Tables 1 and 2 in the paper), leveraging this bandit-inspired RL loss leads to measurable improvements over state-of-the-art SSL methods across multiple datasets, indicating a concrete, beneficial impact on performance rather than a superficial narrative alignment. Furthermore, the adaptation of RL loss in this context is well-founded, as our approach dynamically adjusts to both labeled and unlabeled data, akin to how RL algorithms optimize actions based on rewards—a principle core to both general RL and bandit problems.
Thanks for the response. The Rebuttal mitigates my concern about training efficiency. So I will keep my positive opinion of this paper.
We thank the reviewer for the response.


**About MSE loss**: Many outstanding  SSL methods [1,2,4] use MSE loss on unlabeled data, and with a similar idea MixMatch [3] applies MSE to the mixup of labeled and unlabeled data. MSE is chosen because it effectively enforces consistency, helping to stabilize training and prevent overconfidence in noisy pseudo-labels. Studies like [1] show that MSE consistently yields slightly better results than cross-entropy (CE) loss on unlabeled data. The table below highlights the effect of replacing MSE loss with CE loss in our RLGSSL in terms of test errors. These results are consistent with the studies in [1]:

| Dataset         |       CIFAR-100 (4000) | CIFAR-100 (10000) |
|-----------------|------------------------|-------------------|
| RLGSSL          | ${36.92}_{(0.45)}$     |${29.12}_{(0.20)}$ |
| R(MSE->CE)      | ${37.14}_{(0.53)}$     |${31.37}_{(0.52)}$ |                             



**About improvement on CIFAR-100**: The improvement on CIFAR-100 dataset is not marginal. As detailed in Table 1 of our manuscript, RLGSSL outperforms the second-best method, ICT, by 3.15% and 3.12% on CIFAR-100 with 4000 and 10000 labeled samples, respectively. Furthermore, in Table 3, using a WRN backbone, RLGSSL surpasses the state-of-the-art MarginMatch by 1.24% with only 2500 labeled samples on CIFAR-100. This consistent outperformance across different datasets and experimental setups clearly demonstrates the significant advantage of our proposed algorithm, RLGSSL.



**About robustness and generalization**: Our model demonstrates strong generalizability and robustness, evident in its consistent performance with the lowest average test error and low standard deviation across multiple datasets. RLGSSL not only achieves significant improvements on CIFAR-100 but also outperforms state-of-the-art methods on other datasets. This consistent success across various datasets highlights the robustness and adaptability of our model, confirming its effectiveness in varied scenarios.








[1] Laine, Samuli, and Timo Aila. "Temporal Ensembling for Semi-Supervised Learning." International Conference on Learning Representations. 2017.

[2] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." Advances in neural information processing systems 30 (2017).

[3] Berthelot, David, et al. "Mixmatch: A holistic approach to semi-supervised learning." Advances in neural information processing systems (NeurIPS), 2019.

[4] Verma, Vikas, et al. "Interpolation consistency training for semi-supervised learning." Neural Networks 145 (2022)
Thank you for your response. I find the idea behind this work to be quite innovative, however, I still lean towards maintaining my original score.

Regarding the use of the bandit framework, it seems more like a narrative tool rather than something strongly tied to the core method. Additionally, although bandit problems are a branch of RL, the primary focus and algorithms between them differ significantly. In this paper, the problem is framed using a bandit setting, yet the proposed solution is described as using an RL loss, which seems inappropriate.

I also agree with reviewer mW6a that this paper has significant room for improvement. I hope to see a more refined version in the future.
Thank you to the authors for their response.

However, I remain unconvinced by the rationale for using RL. The authors mentioned that "MSE loss is less sensitive to label noise compared to CE loss," which justifies using MSE as the reward function. I believe the opposite is true, as CE has better noise resistance. Therefore, building RL training based on this reward function is not convincing.

The explanation for the lack of significant improvement in the results in Table 3 is also unconvincing. While the test error on CIFAR-10 is low, and the differences may not be significant, there is also no noticeable improvement on CIFAR-100. This improvement might be due to model optimization and parameter selection, rather than demonstrating the significant advantage of the proposed algorithm.

The authors might have misunderstood my question 1. I asked how the proposed method enhances the model's robustness and generalization, as claimed in the original paper. The response was about the impact of the reward function on performance.

Therefore, while this work appears interesting, there is room for improvement. I hope the authors continue their efforts to further improve the paper. I cannot recommend accepting the current version and keep my initial scores.