Semi-supervised regression with skewed data via adversarially forcing the distribution of predicted values
Authors: Dae-Woong Jeong
Kiyoung Kim
Changyoung Park
Sehui Han
Woohyung Lim
Venue:  (2021)
Keywords: Semi-supervised learning
Adversarial
regression

Abstract: Advances in scientific fields including drug discovery or material design are accompanied by numerous trials and errors. However, generally only representative experimental results are reported. Because of this reporting bias, the distribution of labeled result data can deviate from their true distribution. A regression model can be erroneous if it is built on these skewed data. In this work, we propose a new approach to improve the accuracy of regression models that are trained using a skewed dataset. The method forces the regression outputs to follow the true distribution; the forcing algorithm regularizes the regression results while keeping the information of the training data. We assume the existence of enough unlabeled data that follow the true distribution, and that the true distribution can be roughly estimated from domain knowledge or a few samples. During training neural networks to generate a regression model, an adversarial network is used to force the distribution of predicted values to follow the estimated ‘true’ distribution. We evaluated the proposed approach on four real-world datasets (pLogP, Diamond, House, Elevators). In all four datasets, the proposed approach reduced the root mean squared error of the regression by around 55 percent to 75 percent compared to regression models without adjustment of the distribution.

Under review conference paper at ICLR 2021
SEMI-SUPERVISED REGRESSION WITH SKEWED DATA VIA ADVERSARIALLY FORCING THE DISTRIBUTION OF PREDICTED VALUES
Anonymous authors Paper under double-blind review
ABSTRACT
Advances in scientific fields including drug discovery material design are accompanied by numerous trials and errors. However, generally only representative experimental re- sults reported. Because of this reporting bias, the distribution of labeled result data deviate from their true distribution. A regression model can be erroneous if it is built on these skewed data. In this work, we propose new approach to improve the accuracy of re- gression models that are trained using a skewed dataset. The method forces the regression outputs follow the true distribution; the forcing algorithm regularizes the regression results while keeping the information of the training data. We assume the existence of enough unlabeled data that follow the true distribution, and that the true distribution can be roughly estimated from domain knowledge or few samples. During training neural networks to generate regression model, an adversarial network used to force the dis- tribution of predicted values to follow the estimated 'true' distribution. We evaluated the proposed approach on four real-world datasets (pLogP, Diamond, House, Elevators). In all four datasets, the proposed approach reduced the root mean squared error of the regression by around 55 percent to 75 percent compared to regression models without adjustment of the distribution.
INTRODUCTION
Advances in scientific fields including drug discovery or material design are accompanied by numerous trials and errors. However, generally only representative experimental results are chosen be reported. As a consequence of this reporting bias, the distribution of the reported results can differ from the true distribution. For this reason, when data from the literature are used train a regression model, predictions from the regression model may differ from the true distribution because the model is derived using biased data (Lin et 2002. Galar et al. 2011)
In particular, pharmaceutical development is often affected by this problem. Quantitative structure-activing relationship (QSAR), including drug-target interaction (DTI), is consistently affected by the bias in the re- ported experimental data, because usually the targeted range of molecular property is clearly defined (Liu et al. 2015, Chen & Zhang 2013). When regression is performed using such skewed data, it often erro- neously predicts that the target properties are satisfied. As consequence, it is difficult to discover molecules that have the desired properties (Peng et al. 2017).
Active learning applications also have similar problem. Many active learning methods repeat the selection
of new data by applying certain criteria and retraining the surrogate model (Lookman et al.) 2019 Rouet- Ledue al. 2016 Yuan et al. 2018). During this process, the data be skewed according the criteria

---

Under review conference paper at ICLR 2021
(de Mello 2013; Prabhu et al. 2019). However, despite this problem, few studies have tried to improve the accuracy of regression models that have been trained on skewed data.
this work, we propose new approach improve the accuracy regression model that is trained using skewed data. We assume the presence of enough unlabeled data which follow the true distribution, and that the true distribution can be roughly estimated using domain knowledge or few examples. We use a semi-supervised learning framework with an adversarial network to force the distribution of the regression output to resemble the assumed true distribution (Figure [ At the same time, by sharing the front part of the regression model with the encoder of an adversarial autoencoder (AAE), the process of forcing the distribution of output values is regularized in a way that the information of the labeled data is represented stably. We created skewed datasets by selecting data that exceeded a certain threshold from four real-world datasets (pLogP, Diamond, House, Elevators), then evaluated the proposed approach using these skewed datasets. The proposed approach reduced the mean squared error (RMSE) of the regression model derived using each of the four datasets, compared to the regression model that had been trained using only the skewed datasets. We also verified that the proposed approach is feasible even when the estimate of the true distribution is not perfect.
Regressor
Rpoct
Regression Output y
Input
A ssumed true distribution
X
encoder
decoder
x"
Renc
Discriminator
+
Input
Adversarial network forcingotput distribution
AAE based regularization
Discriminator
Figure Architecture of regression model with proposed approach
2 RELATED WORK
Supi-Super Learning (SSL) is machine-learning strategy to learn using partially-labeled datasets (Chapelle et al., 2009p. the field of SSL, various methods have been developed, including those using generative models (Kingma al. 2014), graphs (Goldberg & Zhu (2006), self-training (Rosenberg al.) 2005) and consistency regularization (Sohn et al. 2020). SSL can improve classification and regression models by using information in large set of un unlabeled data to train relatively small set of labeled data (Xie jet al. 2019 Creswell et al.) 2018 Dimokranitou [007 |Reagholiradeh & Haidar (018).. These approaches generally assume that labeled and unlabeled datasets well distributed without distortion.
2

---

Under review conference paper at ICLR 2021
Several methods were tried to train model on data that have biased distribution. method proposed by (Zhu et al., 2003) incorporated class prior knowledge to adjust the class distribution. Another method proposed by (Kim et al. 2020) used model trained with skewed labeled data generate pseudo-labels of unlabeled data for retraining by considering the entropy of prediction. Positive Unlabeled (PU) learning uses data with only positive and unlabeled samples for binary classifications (Elkan & Noto 2008); PU learning has been studied intensively (Hsieh et al. [019 Youngs et al. 2013). However, to the best of our knowledge, few methods used skewed labeled datasets for regression.
Although other methods for learning with imbalanced data have been developed, such the Synthetic Minority Oversampling Technique (SMOTE) for regression (Torgo et al. 2013), these approaches can only be applied if even some of the data are in the sparse area.
3 METHOD
Our proposed approach is aimed to improve accuracy of regression models that are trained on dataset that is skewed according to labels. Dy denotes a skewed labeled dataset with n samples that satisfies
Y1 8, (x y4) E Dj { 9:
(1)
where x; E Rd is d-dimensional input, y: IR is ground truth target and @ is the bias threshold above which the labeled data are present. We assume that we have an estimate of the true distribution of targets y, p(V), and that we have unbiased unlabeled dataset TD,, (I #n Fm that has sufficiently large m >> nt and in which the distribution of ground truth targets follows p(y).
In this section, we describe our approach to make the regression outputs follow the true distribution closely as possible by using Dy and D. in semi-supervised fashion. The proposed approach consists of two parts: "Adversarial network for forcing output distribution" and "Regularization using AAE", In the first part, we present a method that uses an adversarial network to force the output distribution of the regression model to be similar to p(y). Then, we describe a method to properly regularize this forcing process by implementing an AAE.
3.1 ADVERSARIAL NETWORK FOR FORCING OUTPUT DISTRIBUTION
Let y be the output of regressor that aims to predict ground truth targets y, and ((ilar)) be the regression distribution. Then for given data distribution pa(z), distribution of predicted regressor output values ( can be defined as:
((0) = / 9 (p)
(2)
We use du to denote the predicted values for Di. and ÿ1 to denote the predicted values for B.. Then the distributions of predicted values are denoted as ((O.) and ((i)).
The proposed approach is to regularize the regressor to have g(ū.) close to p(y). To do so, an adversarial network is placed at the end of the regressor (Figure [T. The model is concurrently trained on Bj for re- gression, and on DD, to force the regression outputs to have similar distribution to p(y). The discriminator distinguishes outputs of the regressor from randomly-sampe values in p(y), and the regressor learns to deceive the discriminator.
Suppose we divide D,, into two subsets, Dabove that consists of x that have ground truth target values > 0, and Deetow which consist of æ that have ground truth target values 0. If the regressor is well trained
3

---

Under review conference paper at ICLR 2021
on Dj, then prediction on Doovve is expected to be relatively accurate because the distributions of of Wi and above are similar. In contrast, for Dielow, the regressor should predict values that have not ever been seen; therefore, we hypothesized that during adversarial training on ID,, the prediction on DD above should stay relatively still, whereas prediction on Dietow is flexible. If this hypothesis holds, then the portion of g g(ūw) that is > should be filled with prediction outputs on Dabones and prediction outputs on Detow should be < 0 to deceive the discriminator. These responses result in the regressor having ((Ua) close to p(y) while maintaining accurate prediction for outputs >> 0.
3.2 REGULARIZATION BASED ON AAE
Even if prediction on Dahove is intact and the predicted output distribution of D., fits the assumed true distri- bution, the prediction on Deelow will still have a large enror in most cases. To guide the prediction output on Dietow correctly, the information from above 9 must be conveyed below 9. We used the AAE to regularize the process of forcing the output distribution to achieve appropriate propagation of the information.
We set the encoder of an AAE to share the front part of the regression network (Figure T). We denote the shared part of the regression network as Rene and rest of the network as Rpost- Rene works as the encoder of the AAE and the front part of the regression network at the same time. Therefore, latent vectors of the useful feature for the Roost which functions to predict from the latent vectors. To
AAE should act fulfil its function, Rene must consider knowledge from the labeled dataset.
Also, during the process of forcing the output distribution, Rpost also transforms the given distribution of the latent vectors to ply). To be a useful feature for Rpost. latent vectors should be arranged in a similar way to ply), which possesses information about how the labeled dataset is skewed. This implies that latent vectors become more useful if Rene is aware of the skewed distribution of the labeled dataset. To facilitate this process, shaped the distribution of the latent vectors have similar characteristies to p(y) by adjusting the target distribution of AAE.
Consequently, hypothesized that the regularization process of AAE can be well guided by this informa- tion from labeled data and p(y). This good guidance results in well-controled propagation of information from the portion to the prediction on Dieloow
4 EXPERIMENTS
In this section, we present results of experiments on regression of artificially-skewee datasets. For a given dataset, we first randomly sampled a sufficient amount of data to construct an unlabeled train dataset. Then we divided the rest of the data arbitrarily into three datasets form a labeled train dataset, a validation dataset and a test dataset. To induce bias into the labeled dataset fairly, only data with labels that had values more than one standard deviation above the mean of all labels were selected randomly until the labeled dataset was sufficiently large.
For the experiments, we used four datasets: moses (QSAR on calculated pLogP), diamond, house and eleva- tors (Polykovskiy et al., 2018 Wickham [016]] Micala-Fdezz et 201T). Since QSAR on small molecules suffers from skewed datasets in many practical applications, our experiments are focused on the QSAR on calculated pLogP task. Other datasets were used to demonstrate the general applicability of the proposed approach.
We focused comparing regression accuracy of the models that were trained using our approach or without it. Further, to explore the effectiveness at which the adversarial network forced output distribution and AAE based regularization separately, we conducted ablation studies. As the proposed approach requires estimate of the true distribution, we tested the sensitivity of the model to the quality of the estimate. In each test, the true distribution of the data estimated to be Gaussian distribution that has mean and standard
4

---

Under review conference paper at ICLR 2021
Table 1: Comparisons of RMSE of different regression models. Unbiased data: Trained with fully labeled unbiased dataset, Skewed data: Trained with skewed dataset, Only AAE based regularization: Trained with skewed dataset and AAE based regularization, Only forcing output distribution: Trained with skewed dataset and forcing output distribution, Proposed approach (20 samples): Proposed approach is applied with the estimated true distribution from random 20 samples, Proposed approach (Full data): Proposed approach was applied with the estimated true distribution from full data.
Method
pLogP
Diamond
House
Hlevators 1.542:+0.226
Skewed data
1.689=0.087 1.654=0.035
0.629
1.694=0.027
Only AAE based regularization
Only foreing output distribution
0.982
Proposed approach (20 samples)
0.096
0.207+10.031
0.822+0.119
0.780:10.253
Proposed approach (Full data)
0.494=0.011
0.160
0.727=0.042
0.607=0.086 0.56510.026
Unbiased data
0.295=0.025
0.127
0.37410.017
deviation of 1) full data or 2) 20 randomly-sampled data. We set the assumed true distribution Gaussian distribution for experiments on all four datasets regardless of their real distributions.
These experiments should not consider the label of unlabeled dataset until the final accuracy test. Therefore, to tune hyperparameters. we only considered accuracy of the regression on labeled dataset and distribution mismatch between predicted values and the assumed true distribution.
4.1 EXPERIMENTS ON QSAR TASK
The agreement between ground truth and predicted pLogP were affected by the method used. The model trained using fully-labeled data accurately predicted pLogP (Figure 24). However, the model trained with skewed data without the proposed approach, showed decrease in accuracy as the ground truth value de- clined below the bias threshold (Figure 2); this result indicates that the model's 's extrapolation ability is not sufficient for this task. In contrast, the model trained with the proposed approach followed the data well (Fig- 2)) The RMSE [ also show that about 70% of error in the model without the proposed approach reduced using the proposed approach. This result suggests that information learned from skewed labeled data guide the outputs of the regression model to properly fit in the assumed true distribution during the training process; otherwise, the regression model with proposed approach would show predictions that have a distribution that matches the assumed true distribution, but would not be accurate.
4.2 ABLATION STUDY
To address the separate influence of the adversarial network to force output distribution and AAE for regu- larization, we conducted an ablation study that excluded each component.
When only the forcing output distribution part was excluded, the model showed almost uniform values outside of the labeled region (Figure(p)) The result indicates that regularization alone is not sufficient for extrapolation in this task. Comparison with (Figure[p) demonstrates that the regularization of the predicted values reduces the variation, but that the effect of skewed dataset holds the predicted values to near the lowest labeled data point. As result, the RMSE of the model is not meaningfully improved from that of the model without the proposed approach (Table
When only AAE for regularization was excluded, the prediction values were relatively accurate but the trend of prediction according to ground truth fluctuated slightly (Figure (b)) The fluctuation suggests a decrease in the ability to convey information from labeled data. Although the RMSE of the regression model was
5

---

Under review conference paper at ICLR 2021
(a)
(b)
(c)
2
2
Predictted valiu value
Prodictad valuue
0
0
-2
-2
-2
-4
4
-6
-6
-6
-6
-4
-
2
4
-6
Ground truth
Ground truth
Ground truth
(d)
(e)
(f)
Pooditad value
Frodcad vale
Prodcced sale
Cound trah
Grecad trda
Groan tut
Requeray
Estental
(A.U)
-6
-4
-
2
-6
-4
-2
0
2
4
-6
-4
-
0
4
pLogP value
pLogP value
pLogP value
Figure 2: Correlation plot of ground truth and predicted values of regression model (a) trained unbiased fully labeled data, (b) trained on skewed dataset and (c) trained on skewed dataset with proposed approach. Vertical dotted line: bias threshold of the dataset. (d-f) histogram of ground truth and predicted values of (a-c) respectively. Dark dashed line in (f): assumed true distribution for comparison.
400% lower than that of the model without the proposed approach (Table D. strong regularization seems wise, because in practice, we cannot evaluate the error of the model if fully-labeled data are not available.
4.3 GENERAL APPLICABILITY OF PROPOSED APPROACH
RMSE were also collected for the models for datasets other than QSAR (Table I For the other three datasets, RMSE was reduced by 55% to 75% compared to the RMSE of the regression model trained with the skewed dataset and without the proposed approach. The result was consistent even for the difficult task that shows high RMSE (e.g., house). This result demonstrates that even slight information from the labeled data may guide the process of forcing output distribution.
However, in practice, tasks may be performed on datasets for which the distribution cannot be estimated accurately even when the dataset is known to be skewed. The proposed approach relies strongly on the assumed true distribution, so we tested sensitivity of the proposed approach to the assumed true distribution. For this purpose, estimated the true distribution of the datasets by using only 20 random samples instead the full data. Interestingly, the degree of RMSE improvement over the model without the proposed approach reduced by only 10% to 20%. Considering that the true distributions of the datasets not exactly Gaussian, this result indicates that the proposed approach is not sensitive to error in the assumed true distribution, and is appropriate for practical applications.
6

---

Under review conference paper at ICLR 2021
(a)
(b)
2
Prodicteed Vallee value
Prodlictod value
0
-2
Biss Srrehald
Rias Blnebald
-4
-4
-6
-6
-6
Ground truth
Ground truth
(c)
(d)
Predicted value
Prediced value
Gruund truth
Ground snuth
Biss thrshald
Estimated
thrsholdd
Bioquency
Fraquency
(A.U.
(A.U)
-6
-4
-2
0
2
4
-6
-4
-2
2
4
pLogP value
pLogP value
Figure 3: Correlation plot of ground truth and predicted value of regression model (a) trained skewed dataset with only AAE based regularization and (b) trained on skewed dataset with only forcing output distribution. Vertical dotted line: bias threshold of the dataset. (c) and (d) represents histogram of ground truth and predicted values of (a) and (b) respectively. Dark dashed line: assumed true distribution for comparison.
5 CONCLUSION
We proposed new approach improve regression model that is trained skewed dataset. The method uses adversarial forcing to make the output distribution follow the assumed true distribution. The adversarial network to force the output distribution restrains regression model to have the same distribution the output distribution, and the AAE regularizes the model in the appropriate way. We evaluated the proposed approach four datasets. The proposed approach increased the accuracy of the regression model all datasets. Further assessment showed that the regression quality is maintained even when the proposed approach is applied with rough estimation of the true distribution.
7

---

Under review conference paper at ICLR 2021
REFERENCES
J. Alcala-Fdez, A. Fernandez, J. Luengo, J. Derrac, S. Garcia, L. Sanchez, and F. Herrera. KEEL Data- Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Frame- work. Journal of MMltiple--alllee Valued Logic and Soft Compuling, 17:255-287, 2011.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
Hailin Chen and Zuping 7hang. A superviee method for drug-target interaction prediction with con- sistency in networks. Plos one, 8:e62975, 2013.
Antonia Creswell, Alison Pouplin, and Anil A Bharath. Denoising adversarial autoencoders: classifying skin lesions using limited labelled training data. IET Computer Vision, 12(8): 105-1111, 2018.
Carlos Eduardo Ribeiro de Mello. Active Learning: an unbiased approach. PhD thesis, 2013.
Asimenia Dimokranitou. Adversarial auloencoders for anomalous event detection in images. PhD thesis, 2017.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 213-220, 2008.
Mikel Galar, Alberto Fernandez, Edurne Barrenechea, Humberto Bustince, and Francisco Herrera. A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(4))4433484, 2011.
Andrew B Goldberg and Xiaojin Zhu. Seeing stars when there aren't many stars: Graph-based semi- supervised learning for sentiment categorization. In Proceedings of TextGraphs: The first workshop on graph based methods for natural language processing, pp. 45-52, 2006.
Yu-Guan Hsieh, Gang Niu, and Masashi Sugiyama. Classification from positive, unlabeled and biased negative data. In International Conference on Machine Learning, pp. 2820-2829, 2019.
Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pseudo-label for imbalanced semi-supervised leamnin.. arXiv preprint arXiv:2007.08844 2020.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Yi Lin, Yoonkyung Lee, and Grace Wahba. Support vector machines for classification in nonstandard situa- tions. Machine learning, 46(1-3):111-22,, 2002.
Hui Liu, Jianjiang Sun, Jihong Guan, Jie Zheng, and Shuigeng Zhou. Improving compound- inter- action prediction by building up highly credible negative samples. Bioinformatics, 31:1221-i2223 2015.
Turab Lookman, Prasanna V. Balachandran, Dezhen Xue, and Ruihao Yuan. Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design. npj Computational Materials, 5, 2019.
Lihong Peng, Wen Zhu, Bo Liao, Yu Duan, Min Chen, Yi Chen, and Jialiang Yang. Screening drug-target interactions with positive-unlabeld
unlabeled learning. Scientific Reports, 7, 2017.
8

---

Under review conference paper at ICLR 2021
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alan Aspuru-Guzik, and Alex Zha- voronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. arXiv preprint arXiv:/811. 128223 2018.
Ameya Prabhu, Charles Dognin, and Maneesh Singh. Sampling bias in deep active classification: An empirical study. arXiv preprint arXiv:/909.0990 /909999399,, 2019.
Mehdi Rezagholiradeh and Md Akmal Haidar. Reg-gan: Semi-supervised learning based on generative adversarial networks for regression. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2806-2810. IEEE, 2018.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detec- tion models. 2005.
Bertrand Rouet-Ledue, Kipton Barros, Turab Lookman, and Colin J. Humphreys. Optimisation of GaN LEDs and the reduction of efficiency droop using active machine learning. Scientific Reports, 6, 2016.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Ku- rakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint TrXiv:2001.076855 2020.
Luís Torgo, Rita P Ribeiro, Bernhard Pfahringer, and Paula Branco. Smote for regression. In Portuguese conference on artificial intelligence, pp. 378-389. Springer, 2013.
Hadley Wickham, ggplo12: Elegant Graphics for Data Analysis. Springer- Verlag New York, 2016. ISBN 978-3-319-2427744 URL htall/tcpe////oooo2.....oeeeeeee
Yutong Xie. Jianpeng Zhang, and Yong Xia. Sepi-supervised adversarial model for benign- lung nodule classification on chest ct. Medical image analysis, 57:237-248, 2019.
Noah Youngs, Dennis Shasha, and Richard Bonneau. Positive-unlabeled learning in the face of labeling bias. In 2015 IEEE International Conference on Data Mining Workshop (ICDMW), pp. 639-645. IEEE, 2015.
Ruihao Yuan, Zhen Liu, Prasanna V. Balachandran, Deqing Xue, Yumei Zhou, Xiangdong Ding, Jun Sun, Dezhen Xue, and Turab Lookman. Accelerated Discovery of Large Electrostrains in BaTiO3- BBsse Piezo- electrics Using Active Learning. Advanced Materials, 30, 2018.
Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi- supervipe learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), pp. 912-919, 2003.

---

Under review conference paper at ICLR 2021
A
APPENDIX
A.1
EXPERIMENTS ON OTHER TYPES OF BIASES
(a)
(b)
4
2
Prodicter value
Prodictod
0
-2
-4
-4
RMSE 0.915
RMSE
-6
-6
-6
-2
4
-6
Ground truth
Ground truth
(c)
(d)
Predicted value
Prodicted value
Grund tuth
Ground tuth
Labeled samples
Labeled samples
Froquency
Frequency
(A.U.)
(A.U)
-6
-4
-2
0
2
4
-6
-4
-
4
pLogP value
pLogP value
Figure 4: Correlation plot of ground truth and predicted values of regression model (a) trained skewed dataset which have a gaussian distribution and (b) trained on the same skewed dataset with proposed ap- proach. (c, d) histogram of ground truth and predicted values of (a, b) respectively. Dark dashed line: histogram of labeled samples in the skewed dataset.
10

---

Under review conference paper at ICLR 2021
(a)
(b)
2
2
Predicted value
Predicted value
0
-2
-2
-4
-4
1013
=0.504
-6
-6
-6
-4
-2
2
4
-6
Ground truth
Ground
(c)
(c)
Predicted value
Predicted value
Grouni trah
Ground truth
Frequency
Frequercy
(A.U.)
(A.U)
-6
-4
-2 0
2
4
-6
-4
-2
pLogP value
pLogP value
Figure 5: Correlation plot of ground truth and predicted values of regression model (a) trained on skewed dataset and (b) trained on skewed dataset with proposed approach. Labeled samples only have the ground truth values in the grey region. (c, d) histogram of ground truth and predicted values of (a, b) respectively.
11

---

Under review conference paper at ICLR 2021
(a)
(b)
2
2
Predicted value
Predicted value
0
-2
-2
-4
-4
-6
-6
-6
-4
-2
0
2
4
Ground truth
Ground truth
(c)
(d)
Predicted value
Predicted value
Groun trat
Ground truh
Frequency
Frequercy
(A.U.)
(A.U)
-6
-4
-2
0
2
4
-6
-4
-2
pLogP value
pLogP value
Figure 6: Correlation plot of ground truth and predicted values of regression model (a) trained on skewed dataset and (b) trained on skewed dataset with proposed approach. Labeled samples only have the ground truth values in the grey region. (c, d) histogram of ground truth and predicted values of (a, b) respectively.
12


Reviews:
[{"review_id": "L7kdTa0N0bg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a semi-supervised learning approach to improve the regression model trained on output-skewed data. The key assumption is that, though the training outputs can be skewed, it is easy to estimate the true distribution of the output. The proposed model that combines an AAE that generates the output distribution, and an adversarial model that enforces the distribution of the predicted output to resemble the true distribution of the output. On several real datasets, the ablation study shows the proposed model can improve the regression accuracy.\n\nThe paper is in general well-written. However, the proposed method is straightforward, and a few important questions need to be addressed and/or clarified. \n\n(1)\t The paper assumes that the training data are often highly skewed (intentionally) but the true distribution of the output can be easily estimated or obtained. That seems to imply that ---  it is easy to fix the data skewness by simply collecting more labeled data. Then why should we use the proposed approach? Can you provide a concrete application where getting more labeled examples is very costly yet estimating the label distribution is easy and cheap? \n(2)\tThe paper did not explain where the target distribution in the adversarial part of AAE comes from. R_{enc} generates the latent features, which are aimed to fool the discriminator. But which target distribution do you believe the latent features should (approximately) follow?  How does it connect to using the information of the unlabeled data or the true output distribution?\n(3)\tThe evaluation is done by creating skewed training data from existing data. This is more like a simulation. Can you incorporate a real-world application, e.g., drug design (mentioned by the paper), to showcase the performance and usefulness of the proposed approach? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KOUMikXcbOC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Premise and Contributions:\nThis work presents a machine learning scenario where the training data is biased due to collection artifacts. The training data does not represent the true real world distribution. In order to overcome this obstacle, the work presents the following contributions:\n- A semi-supervised method that uses unlabeled data (that follows the real distribution) in order to improve predictions.\n- This method contains an adversarial network that forces the regression output distribution to be similar to the assumed true label distribution.\n- This method contains an adversarial autoencoder that propagates the information from the assumed true distribution to the latent vector of the regression model.\n\nStrengths:\n- The scenario seems plausible.\n- The proposed method seems sound. Although I have a concern with the AAE regularization.\n- The experiments are convincing in my opinion, but there are some issues that I talk about below.\n- The paper is pretty clear. Although, I believe the scenario is not explained in a way that is simple enough to understand instantly. But of course after reading the paper it becomes clear. Maybe there could be some improvements in the abstract and the introduction. Also I noticed that the introduction repeats the abstract, there could be some more precise explanations instead of this. For example (Kim et al. 2020)?\n\nIssues:\n- Section 3.2: \"To be a useful feature for Rpost, latent vectors should be arranged in a similar way to p(y), which possesses information about how the labeled dataset is skewed.\" - p(y) is a distribution of labels right? How can the latent vectors have same distribution as a distribution of labels. I do not understand this. \n- Is there generality with respect to different types of bias for this method? Not just biasing the dataset with labels above $\\theta$\n- How is the assumed true label distribution selected? What happens when it is very different from the true label distribution? Very interesting experiments could be undertaken here.\n- In Section 3.1: \"The model is concurrently trained on $D_l$ for regression, and on $D_u$ to force the regression outputs to have similar distribution to p(y)\" - How is the network \"trained\" on $D_u$ is there are no labels? Or is it just running a forward pass?\n- A real world application on a dataset that actually presents the issues of this specific scenario would make the value of the contributions much more convincing, instead of only testing on synthetic datasets. But I do not know about the availability of such datasets.\n- Experiments: For the ablation study, how many samples are used for the \"Only AAE regularization\" and \"Only forcing output distribution\"?\n\nMy biggest concern, and a question to other reviewers and the authors: Are there any other comparable methods to the proposed method, or other methods that solve this task in comparable scenarios? Should any other method be included in the experiments section for comparison purposes?\n\nCurrent decision: 5 - until I read other reviews and understand some of the moving parts better.\n\nUPDATE:\n\nAfter reading the author's updated paper and comments I have decided to improve my rating to a 6 since some of my concerns have been eased. The most important concern being eased was the type of bias was too constrained at first and now there are experiments with a more unconstrained version of bias that is more convincing.\n\nOverall, I would say that future versions of the paper could look into a task and dataset that are close to their domain of applicability and where they can contribute an increase in performance. That would strengthen the case of this paper. I think a rating of 6 is fair for this version of the paper and I thank the authors for their efforts in updating the work and addressing my concerns directly and efficiently.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1qxP012V_AZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to learn a regression model using \"skewed data\", which is defined as the subset of training samples with true target above certain threshold. The model consists of two components. First, the input x was mapped to its latent space through encoder R_enc. The latent representation was further mapped to the predicted output through regressor network R_post. The predictive distribution was forced to match the true target distribution p(y) through an adversarial network. Second, the latent space representations were also forced to match the true target distribution p(y). Experimental results on synthetic benchmark data showed the proposed approach performed better than naively applying regression model on the skewed data.\n\nHowever, defining the \"skewed data\" as the subset of training samples whose target values are above certain threshold is an overly strong assumption. In practice, it's more likely that values within certain ranges were over (or under) sampled, leading to mismatch between the empirical target distribution and the true target distribution. This strong assumption makes the proposed approach not applicable to most real-world problems.\n\nAssuming p(y) can be reliably estimated from the training data also seemed problematic. It's not clear how p(y) was estimated from the labeled dataset, where only samples with true target above certain threshold are available.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0LGRF2nSW7b", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper presents a novel approach to improve the accuracy of regression models that are learned from a skew dataset. The proposed approach consists of two parts, namely, (i) adversarial network for forcing output distributions and (ii) regularization based on an adversarial autoencoder. Experiments suggest that the proposed approach increases the accuracy of the regression model for all the four datasets considered in the paper.\n\n\n##########################################################################\n\nPros: \n\n(1) In practice, the distribution of reported data could be different from the true distribution. The proposed approach is potentially useful for analyzing such data.\n\n(2) This paper provides comprehensive experiments using four datasets. All the experiments agree that the proposed approach increases the accuracy of the regression model and outperforms some existing approaches in terms of RMSE.\n\n(3) The paper is clearly written and the motivation for the study is well explained in the introduction. Figure 1 is helpful to understand the architecture of the proposed approach.\n\n\n##########################################################################\n\nCons:\n\n(1) The proposed approach is based on the assumption that labeled data are skewed and unlabeled data follow the assumed true distribution. I am not sure how realistic this assumption is. For instance, the examples discussed in Introduction appear to be the ones in which both labeled and unlabeled data are skewed. Also, in the experiments given in Section 4, all the datasets are not originally skewed, but they are artificially changed to skewed datasets. It would be convincing to provide a real example in which the assumption of the proposed approach holds.\n\n(2) In the second paragraph of Section 4.3, it is claimed that the proposed approach relies strongly on the assumed true distribution. At the same time, it is also mentioned that the proposed approach is not sensitive to error in the assumed true distribution, and is appropriate for practical applications. These two claims seem contradictory to me. More explanations about these statements would be helpful.\n\n(3) It would be ideal if some theoretical results are presented to support the proposed approach. For example, is it possible to provide any theoretical background to support the claim that the proposed approach is not sensitive to error in the assumed true distribution?\n\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above.\n\n\n---\n\n### Updates:\n\nI thank the authors for their response. Some of my concerns are addressed. However, unfortunately, I still think the assumption of the proposed approach is too strong to have broad applications. I will keep my original score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the real-world problem of semi-supervised learning where the distribution from which the labeled examples are drawn is different from the distribution from which the unlabeled examples are drawn.  The task is motivated by structure-activity prediction for drug design (quantitative structure activity prediction, or QSAR).  Examples represent molecules, and we wish to predict a real-valued measure of binding affinity.  Exactly the general problem of data skew arose with exactly this task for example in one of the KDD Cup 2001 tasks.  While the authors here mention that labeled data may be focused more on active molecules (those with a high continuous-valued response), in the KDD Cup 200`1 data the reverse was true, and the unlabeled test data were skewed to higher activity level.  I say all this to agree with the authors about the real-world nature of the problem they address.  Also, some reviewers felt more empirical evaluation was needed, so that may be an additional data set for the authors to consider using.

Reviewer concerns including that the approach was simplistic, the empirical results were insufficient, and the claims were oversold.  The author replies and revisions, and the discussion, moved the reviews to be more favorable but still not strong enough to justify acceptance yet.  Nevertheless, the consensus is that the paper addresses an important problem and the revisions are headed in the right direction to make a strong future paper, and that the authors should be encouraged to continue this work.
Decision: Reject