RankUp: Boosting Semi-Supervised Regression with an Auxiliary Ranking Classifier

Authors: yen  huang
szu-wei  fu
yu  tsao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:05:55

Abstract: State-of-the-art (SOTA) semi-supervised learning techniques, such as FixMatch and it's variants, have demonstrated impressive performance in classification tasks. However, these methods are not directly applicable to regression tasks. In this paper, we present RankUp, a simple yet effective approach that adapts existing semi-supervised classification techniques to enhance the performance of regression tasks. RankUp achieves this by converting the original regression task into a ranking problem and training it concurrently with the original regression objective. This auxiliary ranking classifier outputs a classification result, thus enabling integration with existing semi-supervised classification methods.  Moreover, we introduce regression distribution alignment (RDA), a complementary technique that further enhances RankUp's performance by refining pseudo-labels through distribution alignment. Despite its simplicity, RankUp, with or without RDA, achieves SOTA results in across a range of regression benchmarks, including computer vision, audio, and natural language processing tasks. Our code and log data are open-sourced at [https://github.com/pm25/semi-supervised-regression](https://github.com/pm25/semi-supervised-regression).

Main paper: 
 introduction: 
 The effectiveness of deep learning models heavily depends on the availability of labeled data. However, obtaining labeled data can be challenging in various scenarios. For instance, tasks like quality assessment often require multiple human annotators to label a single data point [15, 8, 19] , resulting in a labor-intensive and time-consuming process. In domains where expert annotation is frequently required, such as medical data, the cost of acquiring labeled data can be extremely expensive [14, 21, 36] . To address these challenges, semi-supervised learning provides a powerful approach to reduce reliance on labeled data for training deep learning models [22, 34, 17, 24, 7] . By effectively leveraging the unlabeled data during model training, semi-supervised learning provides a means to enhance model performance while minimizing the need for extensive labeled data.Recent state-of-the-art (SOTA) semi-supervised learning methods, such as FixMatch and its variants, use a confidence threshold technique to obtain high-quality pseudo-labels [22, 34, 27, 4, 31] . This approach involves generating pseudo-labels from unlabeled data and then filtering out those with low confidence scores. The model is then trained to produce predictions consistent with these high-quality pseudo-labels. Despite its success across various classification tasks, directly applying this technique to regression tasks encounters several challenges. First, unlike classification models, regression models typically lack confidence measures for their predictions, making the confidence threshold technique unfeasible. Additionally, one of the motivations behind using pseudo-labels is to increase the model's confidence in its predictions for unlabeled data, based on the low-density assumption [6, 25] . However, as there are no confidence measures in the predictions of regression models, relying on the low-density assumption and increasing the confidence of unlabeled data becomes unfeasible.Figure 1 : Illustration of using FixMatch on the Auxiliary Ranking Classifier (ARC). This diagram uses the age estimation task as an example, where the goal is to predict the age of a person in an image. The auxiliary ranking classifier transforms this task into a ranking problem by comparing two images to determine which person is older. (Image sourced from the UTKFace dataset [37] ).In this paper, we introduce RankUp, a simple yet effective semi-supervised regression framework that leverages existing semi-supervised classification methods. RankUp achieves this by using an auxiliary ranking classifier, which concurrently solves a ranking task alongside the original regression task. The ranking task is derived from the original regression problem, where the objective is to compare the labels of pairs of samples to determine their relative rank (i.e., which one is larger or smaller). Since ranking problem is a type of classification problem, existing semi-supervised classification methods can be applied to assist in training the auxiliary ranking classifier (see Fig. 1 ).Our empirical results demonstrate that enhancing the performance of the auxiliary ranking classifier also improves the performance of the original regression task, as measured by metrics, such as mean absolute error (MAE) [29] , coefficient of determination (R 2 ) [23] , and Spearman rank correlation coefficient (SRCC) [33] . Furthermore, we show that applying existing semi-supervised classification methods to the auxiliary ranking classifier can effectively utilize unlabeled data, leading to further improvements in the classifier's performance. This improvement, in turn, translates to enhanced performance in the original regression task, showcasing the potential of applying semi-supervised classification techniques to enhance regression models.One of the key advantages of using the auxiliary ranking classifier is its ability to enhance the ranking relationship of pseudo-labels. Building upon this effect, we propose a novel Regression Distribution Alignment (RDA) method, designed to further improve RankUp's performance by refining the distribution of regression pseudo-labels. RDA adjusts the distribution of these pseudo-labels to better align with the true underlying distribution of the unlabeled data. This approach assumes that the distributions of labeled and unlabeled data are similar, allowing us to estimate the distribution of the unlabeled data based on that of the labeled data distribution. This assumption holds true in many cases, especially when labeled data are randomly sampled from the same pool as the unlabeled data. By aligning these distributions, RDA improves the quality of the pseudo-labels, ultimately leading to better model performance when training with these refined pseudo-labels.Our experimental results demonstrate that RankUp, even without RDA, achieves state-of-the-art (SOTA) results across a variety of regression datasets, including tasks in computer vision, audio, and natural language processing. Moreover, integrating RDA with RankUp provides an additional performance boost, leading to the highest performance observed in our experiments. For example, RankUp alone achieves at least a 13% improvement in MAE and a 28% improvement in R 2 compared to SOTA methods on the image age estimation dataset (UTKFace) with 50 labeled samples. The addition of RDA further boosts these results by an additional 6% and 7% in MAE and R 2 , respectively. The empirical results of our experiments demonstrate that existing semi-supervised classification methods can be effectively leveraged to improve the performance of semi-supervised regression tasks. These findings bridge the gap between future research in semi-supervised regression and classification, paving the way for further advancements in the field. 
related works: 
 In this section, we review related research in semi-supervised learning. We categorize the literature into two groups: methods applicable to regression tasks, which will be discussed in Section 2.1, and methods applicable only to classification tasks, detailed in Section 2.2.In semi-supervised regression, methods commonly rely on the smoothness assumption [6, 25] , which suggests that nearby data points in the feature space should share similar labels. Consistency regularization is a popular technique employed to achieve this assumption. It encourages models to generate consistent predictions for slightly perturbed data.For example, the Π-model [17] applies data augmentation to unlabeled data and minimizes the squared difference between the predictions of the augmented data and their original counterparts. Techniques like Mean Teacher [24] involve model-weight ensembling to align the predictions of the model with its ensemble counterpart. Similarly, UCVME [10] employs a bayesian neural network to ensure consistency in uncertainty predictions across co-trained models. Additionally, CLSS [11] utilizes contrastive learning to encourage features of similar labels to be closer together.In semi-supervised classification, in addition to the smoothness assumption, another commonly reliedupon assumption is the low-density assumption [6, 25] . This assumption suggests that a classifier's decision boundary should ideally pass through low-density regions in the feature space. Pseudolabeling [18] is a common approach used to achieve this assumption, where the highest probability class predictions on unlabeled data are utilized as pseudo-labels for training. By incorporating pseudo-labels, the model's confidence in predicting unlabeled data is increased, effectively pushing the decision boundary away from high-density regions towards low-density regions.Recent SOTA semi-supervised learning methods combine pseudo-labeling with consistency regularization to achieve both the low-density and smoothness assumptions, leading to improved performance. For example, MixMatch [3] utilizes a mixup [35] technique and averages predictions from multiple augmented instances to ensure consistency, while also using a sharpening technique to boost prediction confidence on unlabeled data. Similarly, FixMatch [22] builds on this concept by generating high-quality pseudo-labels from weakly augmented unlabeled data using a confidence threshold and enforcing consistency between weakly and strongly augmented versions of the same input.Despite the success of these methods on classification tasks. The low-density assumption doesn't directly translate to regression tasks, as regression models lack explicit confidence measures and decision boundaries like those in classification models. As a result, existing semi-supervised learning methods based on the low-density assumption cannot be directly applied in regression settings.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We have discussed the limitations of our proposed method in Section A.1. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 The proposed framework, RankUp, introduces two additional components: ARC and RDA. The design of ARC is inspired by RankNet [5] . To provide a clear understanding of the ARC's implementation, we first present background information on RankNet in Section 3.1. Subsequently, we will detail the implementation of ARC in Section 3.2 and introduce RDA in Section 3.3. Furthermore, we propose a warm-up scheme and techniques for reducing the computational time of RDA in Sections 3.4 and 3.5, respectively. Lastly, we outline the complete RankUp framework in Section 3.6.RankNet [5] is a deep learning model designed to predict the relevance scores of documents. The core idea behind RankNet is the use of a pairwise ranking loss. It compares two samples and predicts their relative ranking (i.e., which document is more relevant). This approach effectively transforms the relevance score prediction task into a pairwise classification problem. In the following, we will provide a detailed explanation of how the pairwise ranking prediction is performed and how the corresponding loss is calculated.Pairwise Ranking Prediction. The output of RankNet is a single scalar value indicating the ranking score of the sample, where a higher score indicates greater relevance. To obtain the pairwise ranking prediction, two samples are fed separately into the model to get their respective ranking scores. The difference between these scores is then passed through a sigmoid function, which generates a prediction in the range [0, 1]. This prediction indicates the likelihood that the first sample is more relevant than the second. If the output is greater than 0.5, the model predicts that the first sample has higher relevance; if the output is less than 0.5, the second sample is considered more relevant. Mathematically, for two samples, x i and x j , and the RankNet model g, the formula to obtain the pairwise ranking prediction p ij is as follows:p ij = sigmoid(g(x i ) -g(x j )) = 1 1 + e -(g(xi)-g(xj ))(1) Here, g(x i ) and g(x j ) represent the ranking scores for samples x i and x j , respectively. A higher value of p ij indicates a higher likelihood that the ranking of x i will be higher than that of x j .Pairwise Ranking Loss. The pairwise ranking loss is calculated by comparing the model's predicted pairwise ranking p ij with the ground truth label y ij . The label y ij indicates the true relative ranking between samples x i and x j . Specifically, y ij = 1 indicates that sample x i is ranked higher than sample x j , y ij = 0 indicates that sample x i is ranked lower than x j , and y ij = 0.5 suggests the two samples are equally ranked. Since this is fundamentally a binary classification task, the pairwise ranking loss is calculated using the cross-entropy loss function. Mathematically, the pairwise ranking loss for a batch of data is defined as follows:EQUATIONHere, N denotes the batch size, CE is the cross-entropy loss function, p ij is the predicted pairwise ranking between samples x i and x j , and y ij is the corresponding ground truth label. The loss iterates through all possible pairs of samples in the batch to calculate the average loss for the entire batch.The Auxiliary Ranking Classifier (ARC) is designed to solve a ranking task alongside the primary regression task. It can be easily integrated into existing regression model architectures like ResNet [13] , BERT [12] , or Whisper [20] . ARC is implemented as an additional output layer that shares the same hidden layers with the original regression model. This transforms the model into a multi-task architecture with two output tasks: the original output header continues to provide the regression output, while ARC generates a ranking score for the sample.The core idea behind ARC is to transform the original regression task into a multi-class classification problem, allowing existing semi-supervised classification methods to assist in its training. To achieve this, ARC's design is inspired by RankNet, which can effectively convert the regression task into a binary classification task. However, since a multi-class classification output is required, we introduce two key modifications to RankNet to adapt it for this purpose:1. The scalar output value of RankNet is changed to a two-class output, where each output class indicates which sample in a pair has a relatively greater ranking score.2. The sigmoid function is replaced with softmax, which converts the model's output into a multi-class classification probability distribution.Specifically, for the auxiliary ranking classifier r, which outputs a two-class output, the formula to obtain the pairwise ranking prediction pij of two data samples, x i and x j , is as follows:EQUATIONHere, pij represents a two-class prediction that indicates which sample in the pair has a relatively higher regression label. The loss calculation for ARC remains the same as described in Equation 2.This output format enables the integration of existing semi-supervised classification methods. We utilize FixMatch [22] as the semi-supervised classification technique for training ARC. An illustrative example of applying FixMatch to ARC can be found in Figure 1 , with further details in Algorithm 1.Algorithm 1 Auxiliary Ranking Classifier (with FixMatch)Input: Labeled batch X = { (x i , y i ) } N lb i=1 , unlabeled batch U = { u i } N ulb i=1, confidence threshold τ , unlabeled loss weight ω ulb , weak augmentation A w , strong augmentation A s for j = 1 to N ulb do 5: 1: ℓ lb = 1 (N lb ) 2 N lb i=1 N lb j=1 CE softmax r(A w (x i )) -r(A w (x j )) , 1{y i > y j } { Computepw ij = softmax(r(A w (u i )) -r(A w (u j ))) { Predictps ij = softmax(r(A s (u i )) -r(A s (u j ))) { Predict strong pairwise ranking } 7:ℓ ulb = ℓ ulb + 1{max(p w ij ) > τ } CE(arg max(p w ij ), ps ij ) { Accumulate unlabeled loss } 8:end for 9: end for 10:ℓ ulb = 1 (N ulb ) 2 ℓ ulb { Average unlabeled loss } 11: return ℓ arc = ℓ lb + ω ulb • ℓ ulb 3.3 Regression Distribution Alignment (RDA)Distribution alignment is a commonly used technique in semi-supervised classification [2, 16, 28, 4] , where pseudo-labels are refined by aligning their distribution with that of the labeled data. Training semi-supervised models with these refined pseudo-labels can lead to performance improvements. However, existing distribution alignment methods are designed for classification tasks involving discrete label distributions, making them unsuitable for regression settings. Moreover, applying distribution alignment to ARC is impractical, as the two output classes always have equal proportions. To address these challenges, we propose Regression Distribution Alignment (RDA), enabling the direct application of distribution alignment to regression tasks.The RDA process involves three key steps: (1) extracting the labeled data distribution, (2) generating the pseudo-label distribution, and (3) aligning the pseudo-label distribution with the labeled data distribution. These steps correspond to the orange, blue, and yellow parts of Figure 2 , respectively.Step 1: Extracting the Labeled Data Distribution. The labeled data is sorted according to its label values. To ensure a one-to-one correspondence with the pseudo-labels, the labeled data distribution must contain the same number of data points as the pseudo-label set. We use linear interpolation to extend the labeled distribution to match the size of the pseudo-labels.Step 2: Generating the Pseudo-Label Distribution. The model generates pseudo-labels for all the unlabeled data. These pseudo-labels are then sorted by their values, either in ascending or descending order, as long as the sorting direction is consistent with that of the labeled data distribution.Step 3: Aligning the Distributions. Once both distributions have been sorted and resized to the same size, the alignment is performed by replacing each pseudo-label value with its corresponding value from the labeled data distribution.In each training iteration, RDA is applied to refine the pseudo-labels. The loss is then computed between the model's predictions and the RDA-aligned pseudo-labels to minimize their difference. For an unlabeled data point u i with its corresponding regression prediction ŷi and RDA-aligned pseudo-label ỹi , the RDA loss ℓ rda for a batch of unlabeled data is defined as:EQUATIONHere, N ulb denotes the batch size of unlabeled data, L reg represents a regression loss function (e.g., MAE, MSE). The design of RDA is based on two key assumptions. First, it assumes that the distributions of labeled and unlabeled data are similar, which is often true since labeled data is typically randomly sampled from the unlabeled pool. Second, it assumes that the ranking of the pseudo-labels is reasonably accurate. Integrating RDA with ARC can reinforce this assumption, as ARC enhances the ranking relationships of pseudo-labels. Both assumptions are crucial for ensuring that RDA works properly.In the early stages of training, the pseudo-label rankings may be poorly predicted, which can degrade the quality of the pseudo-labels refined through RDA. To address this, we introduce a linear warm-up scheme to stabilize the RDA process. The adjusted RDA loss, ℓ ′ rda , is defined as follows:ℓ ′ rda = min iter α warm , 1.0 • ℓ rda (5)Here, iter denotes the current training iteration, and α warm is a hyperparameter that controls the duration of the warm-up phase. The min function ensures that the warm-up factor does not exceed 1.0, smoothly transitioning the model toward the full effect of ℓ rda .Applying RDA can be computationally expensive, as it requires inference all unlabeled data and sorting all pseudo-labels at every training iteration. This significantly increases the computational load compared to the original training process, especially when dealing with a large volume of unlabeled data, making the implementation of RDA impractical. To mitigate this challenge, we propose several techniques aimed at reducing the computational burden of RDA.Pseudo-label table. RDA creates a table of the same size as the unlabeled dataset. This table stores the model's predicted pseudo-labels for each instance of unlabeled data. For each training iteration, the model generates new pseudo-labels, which are stored and updated within this table. This approach eliminates the need to rerun inference on all unlabeled data when applying RDA, as it only requires a simple lookup from the pseudo-label table.Applying RDA only every T steps. To further reduce computational costs, RDA is applied only every T steps, where T is a hyperparameter. This is achieved by creating a second table of the same size as the unlabeled dataset, which stores the previously aligned results of the pseudo-labels generated by applying RDA. Between RDA updates, the model uses these stored aligned pseudo-labels, thereby avoiding the need to run RDA in every iteration. This strategy effectively reduces the computational cost associated with the RDA process to 1/T .We introduce the term RankUp to describe our proposed framework, which integrates two key components: ARC and RDA. The use of RDA is optional, depending on whether its underlying assumptions are satisfied. The final loss for RankUp is a combination of the regression loss and the ARC loss. The regression loss consists of the original labeled regression loss plus the unlabeled RDA loss. Specifically, the RankUp loss ℓ rankup is defined as follows:EQUATIONIn this equation, ℓ reg represents the loss from the original labeled regression task, while ℓ ′ rda denotes the RDA loss, as defined in Equation 5. The hyperparameter ω rda controls the weight of the RDA loss. If RDA is not employed, the term ω rda • ℓ ′ rda can be excluded from the equation. The term ℓ arc corresponds to the loss from the ARC module, as detailed in Algorithm 1. Additionally, the hyperparameter ω arc regulates the weight of the ARC loss.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this section, we evaluate RankUp's performance across various tasks. The experimental settings are described in Section 4.1. The main results for RankUp under different label configurations are presented in Section 4.2, while Section 4.3 provides additional results on audio and text datasets. Section 4.4 explores the use of alternative semi-supervised classification methods in place of FixMatch. Lastly, we discuss potential reasons why the smoothness and low-density assumptions are also effective for regression tasks in Section 4.5.Evaluation Metrics. We use three evaluation metrics: MAE, R 2 , and SRCC, to assess the performance of semi-supervised regression methods. MAE measures the average absolute difference between the model's predictions and the actual values. The R 2 score indicates the proportion of variance in the data explained by the model. SRCC evaluates the correlation between the predicted rankings and the actual rankings.Evaluation Robustness. To ensure the reliability of our evaluation results, each experiment is executed three times using fixed random seeds (0, 1, and 2). We report both the mean and standard deviation of each metric.Fair Comparison. To ensure a fair comparison between our proposed methods and related works, we implement and evaluate all methods within the same codebase. Specifically, we adapt the popular semisupervised classification framework USB [26] , modifying it for regression tasks to implement both our proposed methods and related works. Weak augmentation is applied consistently to the labeled data across all semi-supervised and supervised methods. For specific details on the modifications made to USB, please refer to Appendix A.5. The code and full training logs of the experiments presented in this paper are open-sourced at https://github.com/pm25/semi-supervised-regression.Hyperparameters. We use the hyperparameters of USB as the base for fine-tuning. We first fine-tune the hyperparameters in the supervised baseline setting and find the hyperparameters that lead to lowest MAE score. These same hyperparameters are then applied to all semi-supervised regression methods to ensure a fair comparison. Only the additional hyperparameters specific to each semi-supervised method are further tuned. For more details on the hyperparameters, please refer to Appendix A.13.Base Model. The base model used in our experiments varies depending on the data type. For image data, we use Wide ResNet-28-2 [32] , which is not pre-trained. For audio data, we use the pre-trained Whisper-base [20] , and for text data, we use the pre-trained Bert-Small [12] . Dataset. To simulate the semi-supervised setting, we randomly sample a portion of the dataset as labeled data, treating the remainder as unlabeled. To evaluate performance, we use three diverse datasets: UTKFace [37] , an image age estimation dataset; BVCC [8] , an audio quality assessment dataset; and Yelp Review [1], a text sentiment analysis (opinion mining) dataset. For more detailed information about these datasets, please refer to Appendix A.11. To evaluate the performance of RankUp under different labeled data settings, we conducted experiments using the UTKFace dataset with 50 and 250 labeled samples. We tested two configurations of RankUp: one incorporating the RDA (RankUp + RDA) and the other without it (RankUp). Their performance was compared against other semi-supervised regression methods, with MixMatch specifically representing the consistency regularization component of the approach. Additionally, we included a supervised setting that used only the labeled data without incorporating any unlabeled data during training, as well as a fully-supervised setting that used all available data (both labeled and unlabeled), assuming the unlabeled data had known true labels. We also conducted experiments with a 2000 labeled samples setting; however, due to space limitations, the results for this configuration can be found in Appendix A.2.The results are presented in Table 1 . We observed that RankUp (without RDA) consistently outperforms existing semi-supervised regression methods, especially when the amount of labeled data is scarce. Specifically, in the 50-label setting, RankUp achieves at least a 12.9% improvement in MAE, a 28.2% improvement in R 2 , and a 4.3% improvement in SRCC compared to other semi-supervised regression methods. In the 250-label setting, RankUp shows at least an 11.2% improvement in MAE, an 8.5% improvement in R 2 , and a 0.4% improvement in SRCC.Furthermore, the integration of RDA with RankUp further enhances the performance of RankUp. Specifically, in the 50-label setting, RankUp + RDA achieves an additional 6.3% improvement in MAE, a 7.4% improvement in R 2 , and a 9.5% improvement in SRCC compared to RankUp alone. Similarly, in the 250-label setting, RankUp + RDA achieves an additional 6.9% improvement in MAE, a 4.1% improvement in R 2 , and a 2.5% improvement in SRCC relative to RankUp.These empirical results demonstrate the effectiveness of RankUp and RDA across different labeled settings. Another notable observation is that RankUp + RDA in the 50-label setting outperforms the supervised model that utilizes five times the labeled data (in the 250-label setting) across all three metrics. Specifically, RankUp + RDA achieves a 1.0% improvement in MAE, a 2.2% improvement in R 2 , and an 8.1% improvement in SRCC while using only one-fifth of the labeled data, demonstrating its effectiveness in reducing labeling costs.To further assess the performance of RankUp across different data types and tasks, we evaluated it on the BVCC and Yelp Review datasets using 250 labeled samples. The results are presented in Table 2 . The table demonstrates that RankUp also consistently outperforms existing semi-supervised regression methods on both audio and text datasets. Specifically, on the BVCC dataset, RankUp (without RDA) achieves at least a 5.6% improvement in MAE, a 6.3% improvement in R 2 , and a 0.3% improvement in SRCC compared to other semi-supervised regression methods. On the Yelp To better understand the effect of RankUp's ARC component and its ability to utilize the unlabeled data, we analyzed ARC by training it with various semi-supervised classification methods. In this evaluation, we did not apply RDA to isolate its effect on ARC. Additionally, we compared these methods against a baseline with no ARC (denoted as "None") and a supervised setting where only labeled data was used to train ARC (denoted as "Supervised").The results are presented in Table 3 . The table indicates that using an ARC without any unlabeled data can already improve performance, with a 4.1% improvement in MAE, an 8.9% improvement in R 2 , and a 4.8% improvement in SRCC compared to the "None" setting. This suggests the beneficial impact of training ARC concurrently with the original regression head, even without leveraging unlabeled data. Furthermore, using different semi-supervised classification methods to utilize the unlabeled data further boost performance compared to using only labeled data. Specifically, we tested the Π-Model, Mean Teacher, and FixMatch, all of which demonstrated improvements over the "Supervised" setting across MAE, R 2 , and SRCC metrics. Among these methods, FixMatch achieved the best results, showing a 21.8% improvement in MAE, a 27.7% improvement in R 2 , and an 11.9% improvement in SRCC compared to the "Supervised" setting. This highlights the effectiveness of leveraging unlabeled data and semi-supervised classification techniques to improve ARC and RankUp's overall performance.The low-density assumption is crucial for understanding the effectiveness of semi-supervised learning methods. However, it does not directly apply to regression tasks due to the absence of decision boundaries and confidence measures. In this section, we explore why RankUp performs well in regression tasks by leveraging semi-supervised classification techniques that utilize the low-density assumption. This understanding can broaden our perspective on these assumption.We adopt a broader interpretation of the smoothness and low-density assumptions. Rather than viewing them solely in the context of classification, we interpret the smoothness assumption as an effort to group features with similar labels together, while the low-density assumption aims to separate features with dissimilar labels. These perspectives align with RankUp's approach. By training ARC with pseudo-labels, the model is encouraged to have greater confidence in the pairwise ranking predictions of the unlabeled data, thus pushing the features with dissimilar pseudo-labels further apart. Additionally, ensuring consistent predictions between weakly and strongly augmented data assists in grouping features with similar labels. The t-SNE visualization demonstrated in Figure 3 supports this claim, showing that within RankUp, similar labels are closer together, while dissimilar labels are pushed further apart in the feature space. 
conclusion: 
 Recent advancements in semi-supervised learning have achieved impressive results across various classification tasks; however, these methods are not directly applicable to regression tasks. In this work, we investigate the potential of leveraging existing semi-supervised classification techniques for regression tasks. We propose a novel framework, RankUp, which introduces two key components: the Auxiliary Ranking Classifier (ARC) and Regression Distribution Alignment (RDA). The empirical results of our experiments demonstrate the effectiveness of our methods across various labeled data settings (50, 250, and 2000 labeled samples) and different types of datasets (image, audio, and text). These findings show that semi-supervised classification techniques can be effectively adapted to regression tasks, bridging the gap between research in semi-supervised regression and classification, and paving the way for more advanced research in this area. 
acknowledgments: 
 I sincerely appreciate everyone who made this work possible. I am especially thankful to my coauthors, Szu-Wei Fu and Prof. Yu Tsao for their mentorship and our weekly discussions; their insights and comments have been invaluable in shaping this research. I am also very grateful for my experience at CLLab and the guidance of Hsuan-Tien Lin, which sparked my interest in weaklysupervised learning and greatly influenced my research mindset and approach. My heartfelt thanks go to my family for their unwavering support throughout my academic journey. I also thank my friend Chi-Chang, whose discussions helped me develop a deeper understanding of how to approach research. Lastly, I want to thank Chia-Ling for her continuous support and encouragement during the development of this research. This work would not have been possible without all these individuals. Thank you!A Appendix / supplemental material A.1 Limitations One of the limitations of RDA is that it relies on two key assumptions: the labeled and unlabeled data have similar label distributions, and the ranking of the pseudo-labels is accurate. If either of these assumptions is not met, the performance of RDA can be significantly impacted.Another limitation of RDA is that it may not perform well in tasks where distinct label values are few.As shown in Table 2 , in the Yelp Review dataset, SRCC decreases when RankUp is combined with RDA compared to using RankUp alone. This is likely because the limited number of distinct label values causes many pseudo-labels to align to the same value, disrupting the ranking relationships.Despite these limitations, RDA is still a powerful component that can work with RankUp to further boost its performance. However, users should carefully evaluate their dataset characteristics to determine whether applying RDA is an appropriate choice. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly cited all the code, data, and models used in this paper to ensure proper attribution and transparency.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The documentation, along with the code used in the paper, is open-sourced at https://github.com/pm25/semi-supervised-regression. It provides detailed instructions on how to use the code. The code is derived from the USB [26] codebase, and both our modifications and the original USB code are licensed under the MIT license, which allows for modification and redistribution. More details about the license can be found at https://github.com/pm25/semi-supervised-regression/blob/main/LICENSE. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.  
 
 Appendix: In We evaluated the impact of different semi-supervised regression methods on training RankUp's regression output. The ARC was trained using FixMatch, while the regression output was trained with various semi-supervised regression techniques. We compared these results with a supervised setting, where only labeled data was used for training the regression output (denoted as "Supervised").The results, shown in Table 5 , indicate that using different semi-supervised regression methods can further improve performance in MAE, R 2 , and SRCC metrics compared to the supervised method alone. Demonstrating the effectiveness of leveraging unlabeled data to directly train the regression head.Among all the semi-supervised regression methods we tested, our proposed RDA achieves the best performance in terms of MAE and R 2 , with at least a 7.7% improvement in MAE and a 1.7% improvement in R 2 compared to other methods. However, it shows a decrease in SRCC compared to MixMatch, with a 1.1% drop. In theory, RDA can also be used alongside the Π-Model, Mean Teacher, and MixMatch, as it focuses on refining pseudo-labels, which is distinct from the approaches of these methods. However, due to increased computational expense and the goal of maintaining a simpler framework, our proposed RankUp only uses a supervised method or RDA for training the regression output. To further understand the effect of ARC and RDA in RankUp, we conducted ablation studies comparing the performance of a supervised baseline, RDA only, ARC only, and ARC + RDA on the UTKFace dataset, using settings of 50 and 250 labeled samples.The results, as shown in Table 6 , demonstrate that RDA alone does not consistently outperform the supervised baseline, particularly in the 50 labeled samples setting (as reflected in the MAE and R 2 values). This may be due to one of the assumptions of RDA not being met, where it requires the pseudo-labels to have reasonably accurate ranking (the supervised baseline with 50 labeled samples only has an SRCC score of 0.371). In contrast, ARC alone ("RankUp" in the table) significantly improves performance over both the supervised baseline and RDA alone. The combination of ARC and RDA yields the best performance, highlighting their synergistic relationship, where it is beneficial to use RDA with ARC. The USB [26] codebase is originally designed for semi-supervised classification tasks and includes implementations of various existing semi-supervised classification methods. To adapt it for regression tasks, enabling the implementation of our proposed framework RankUp and other semi-supervised regression methods, we made the following key modifications:1. Replaced the cross entropy loss function with the MAE loss function.2. Adjusted the output layer to produce a single continuous output instead of multiple outputs used for multi-class classification.3. Removed one-hot encoding from the codebase.4. Normalized the regression labels to the 0-1 range.5. For methods like Mean Teacher [24] and Π-Model [17] , no changes were needed to the core algorithms, as they are inherently applicable to regression tasks.6. For MixMatch [3] , we excluded components specifically designed for classification, such as sharpening and one-hot label encoding, while retaining the input mixing and consistency regularization aspects, which are valuable for regression tasks.To further analyze the effect of weak augmentation on labeled data, we conducted experiments on the UTKFace dataset with a setting of 250 labeled samples, comparing the results with and without weak augmentation applied to the labeled data.Table 7 presents a comparison of the results, illustrating the significance of weak augmentation. The performance metrics (MAE, R 2 , SRCC) show a decline when weak augmentation is not applied (note that weak augmentation was applied to all the baselines in the paper). Despite the observed drop in performance when weak augmentation is not applied, the relative order of effectiveness among the methods remains consistent, regardless of the application of weak augmentation. This consistency further underscores the robustness of our proposed method. We followed the settings in the USB [26] codebase for augmentation operators, with an adjustment to the strong augmentation for audio data. This adjustment was necessary because our task involves quality assessment, and the original strong augmentation method would have affected the quality of the data. Specifically, we used the following augmentation techniques: A.8 Hardware SpecificationsAll experiments reported in this paper were conducted using an Nvidia Titan XP with 12 GB of VRAM and an Nvidia GeForce RTX 2080 Ti, also equipped with 12 GB of VRAM.A.9 More Feature Visualization ResultsAdditional feature visualization results beyond those presented in this paper are available. This includes t-SNE and UMAP visualizations, in both 2D and 3D, for different semi-supervised regression methods, all accessible at https://github.com/pm25/semi-supervised-regression.In our experiments, if the dataset provides a pre-defined train-eval-test split, we utilize the training split to train the model and evaluate its performance on the evaluation or test split. If the dataset does not provide such a split, we randomly sample 80% of the data as the training set and the remaining 20% as the test set. We open-source the train-test splits used for conducting the experiments in this paper at https://github.com/pm25/regression-datasets.Three datasets are utilized in the experiments conducted in this paper: UTKFace [37] , BVCC [8] , and the Yelp Review [1] dataset. Below, we provide a brief introduction to each dataset. BVCC. The VoiceMOS2022 (BVCC) dataset is an audio quality assessment dataset, where the objective is to predict the quality of an audio sample. The labels, ranging from 1 to 5, are obtained by averaging the scores provided by multiple listeners. The dataset is split into training (4,974 samples), evaluation (1,066 samples), and testing (1,066 samples) sets. In the experiments reported in this paper, we utilize only the training and evaluation splits to evaluate performance.Yelp Review. The Yelp Review dataset is a text opinion mining task, where the goal is to predict the rating of customers based on the comments they leave on the Yelp website. There are only five distinct ratings (0 to 4). We use the processed Yelp Review data provided by the USB [26] codebase. The dataset comprises training (250,000 samples), evaluation (25,000 samples), and testing (10,000 samples) sets. We only utilize the training split for model training and the evaluation set for evaluation.A.12 Hyperparameter Fine-TuningWe began by fine-tuning the hyperparameters of the supervised baseline for each dataset, initially setting them based on the configurations provided in the USB [26] codebase. We then fine-tuned the learning rate, weight decay, and layer decay hyperparameters for the supervised baseline model to identify the optimal set of hyperparameters that produced the lowest MAE score. The same hyperparameter settings were subsequently applied to all other methods evaluated in this study. Additionally, each method underwent further fine-tuning by adjusting its own specific additional hyperparameters, distinct from those used in the supervised setting.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Our main claim aligns with our contribution, which is that existing semisupervised classification methods can be successfully adapted for use in semi-supervised regression tasks. Our empirical results demonstrate the effectiveness of our proposed framework in achieving this adaptation. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper modifies existing semi-supervised classifications setups for regression. The reviews raised several issues, most significant among them was novelty. During rebuttal and discussions the authors satisfactorily addressed most of the concerns. This paper improves the performance of regression tasks by converting them into ranking problem.

Reviews: review: paper_summary: paper_summary: The work introduces a novel SSL-regression method called RankUp. A a pairwise ranking loss enables the SSL-method FixMatch to utilize also the unlabled split of the data to learn a regression task. The addition of the Regression Distribution Alignment (RDA) loss enables the method to also take the overall distribution of the labeled samples into account. RankUp achieves SOTA on multiple SSL regression benchmarks.
strength_weakness: strength_weakness: The scale of the experiments. Although this might be an issue of the SSL-regression domain in general.
questions: questions: Regarding the baseline methods: "Specifically, we adapt the popular semi-supervised learning codebase USB [20], modifying it for regression tasks." What modifications are necessary to turn e.g. the Mean Teacher approach into a regression method?
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The authors present two components to improve the problem of semi-supervised regression - 1) RankUp which considers the regression problem as a ranking problem and then adapts existing semi-supervised classification methods, and 2) Regression distribution alignment (RDA) which is to refine pseudo-labels. The experiments demonstrate the effectiveness on the problem.
strength_weakness: strength_weakness: I'm concerned about the technical novelty. 

  The proposed RankUp and RDA are combination from existing methods [1][2][3], and the core idea of adding ranking auxiliary objective to the regression object is not novel [4][5]. The author should include more discussion on the related work of using ranking auxiliary objective to improve regression problems.  

  The two proposed components seem to be orthogonal, especially for RankUp is directly designed to improving semi-supervised problems, but for more general regression tasks. 

[1] Burges et al., Learning to rank using gradient descent. ICML 2005

[2] Sohn et al., Fixmatch: Simplifying semi-supervised learning with consistency and confidence, NeurIPS 2020

[3] Kim et al., Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning, NeurIPS 2020 

[4] Rank-N-Contrast: Learning Continuous Representations for Regression, NeurIPS 2023

[5] Gong et al., RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression, ICML 2022
questions: questions: To better understand the effectiveness of each component on the semi-supervised regression, it can be better if the authors add more discussion and ablation studies on RankUp and RDA individually - e.g. only with RDA.
limitations: limitations: N/A
paper_summary: paper_summary: This paper presents a simple yet effective approach that adapts existing semi-supervised classification techniques to enhance the performance of regression tasks. The perspective is novel and can effectively use existing technologies to solve regression problems.
strength_weakness: strength_weakness: The method description needs to be improved.
questions: questions: 1. The quality of the figures in the article needs to be improved.
2. In the method, several variables are unclear. For example, ℓarc, ℓregression, etc.
3. The description of some processes in Section 3.3 is not very clear and needs to be improved.
limitations: limitations: as above
paper_summary: paper_summary: The submission presents a new semi-supervised learning algorithm for deep regression models: in addition to the standard supervised regression head, an auxiliary classification head is trained using a semi-supervised learning algorithm for classification that learns to classify pairs of examples. A pair is classified correctly if the example with the larger target value is ranked above the example with the smaller target value. The loss for supervised regression is used in a weighted sum with the loss for semi-supervised classification. A further improvement to the algorithm is obtained by introducing an additional component in the loss that encourages the distribution of predicted target values to be aligned with the distribution of ground-truth target values. Experiments on three regression problems, one involving images, one involving audio, and one involving text, indicate that the proposed method outperforms semi-supervised regression algorithms from the literature.
strength_weakness: strength_weakness: The only potential problem I see is the influence of augmentation on the results. The FixMatch method used to train the auxiliary classifier, shown in Algorithm 1, applies weak augmentation for the supervised component of the loss. For a fair comparison, it seems crucial to apply this weak augmentation also on the labeled data for all the other semi-supervised and purely supervised configurations evaluated in the empirical comparison. The submission does not state whether this was done. 

Another, smaller issue is that the submission does not state exactly which operators were used for augmentation in the three different domains.

My negative ratings for soundness, contribution, and the overall quality are based purely on this concern regarding the use of augmentation. *** I HAVE INCREASED BY RATINGS AFTER THE REBUTTAL ***
questions: questions: Was weak augmentation on the labeled data applied consistently across all the semi-supervised and supervised learning algorithm configurations evaluated in the comparison?

Which forms of augmentation were applied in the three different domains?
limitations: limitations: The submission discusses limitations of the proposed approach.

Comments: title: Response to rebuttal
comment: Dear Area Chair,

We would like to provide a summary following the authors' rebuttal. We sincerely thank all the reviewers for their thorough evaluations and valuable feedback. We are especially grateful to Reviewer Ngei for the opportunity to clarify potential issues and for being willing to update the rating **from 3 to 6**. We also appreciate Reviewers PMad and tpBB for recognizing our paper's technical soundness and providing **positive ratings (6)** prior to the rebuttal.

While we thank Reviewer 41bS for the thoughtful review, we note that they provided lower scores and raised several questions. We have carefully addressed the reviewer's comments in our rebuttal but have not received any feedback in response. As a result, we haven't had the opportunity to engage in further discussion with the reviewer. Nevertheless, we are confident that we have **thoroughly addressed their concerns**. 

Regarding the technical novelty concern raised by Reviewer 41bS, which mentions that some existing works have incorporated ranking loss in regression tasks, we would like to clarify the following:

* These methods are designed for **supervised** settings and cannot be applied to **semi-supervised** settings.  
* Additionally, these methods are based on contrastive learning approaches, which **do not align** with the objective of our paper. Our work aims to leverage existing semi-supervised classification methods for regression tasks.  
* Ranking loss is only one component of our proposed method; we have also introduced **RDA**, which is **highly beneficial** to semi-supervised regression when combined with our proposed RankUp.

More details are available in our rebuttal response to Reviewer 41bS.  

According to the input we provided, most reviewers indicated that their concerns were addressed, leading to positive ratings. We sincerely appreciate the reviewers for recognizing the innovation and potential impact of our work.

Thank you\!

Sincerely,   
The Authors
Thank you very much for your time and effort in reviewing our work. We have addressed your comments and welcome any further feedback or questions you may have. We would be happy to discuss them with you.
Thank you for your thoughtful review and positive feedback!
Thank you for your thoughtful review and positive feedback!
I thank the authors for their rebuttal. I keep my positive score.
Thank you for your rebuttal. I have no quetions.
Thank you for your thoughtful review and positive consideration!
Thank you for your rebuttal. My concerns have been addressed, and I will raise my score accordingly.