Semi-supervised regression with skewed data via adversarially forcing the distribution of predicted values

1 INTRODUCTION . Advances in scientific fields including drug discovery or material design are accompanied by numerous trials and errors . However , generally only representative experimental results are chosen to be reported . As a consequence of this reporting bias , the distribution of the reported results can differ from the true distribution . For this reason , when data from the literature are used to train a regression model , predictions from the regression model may differ from the true distribution because the model is derived using biased data ( Lin et al. , 2002 ; Galar et al. , 2011 ) . In particular , pharmaceutical development is often affected by this problem . Quantitative structure-activity relationship ( QSAR ) , including drug-target interaction ( DTI ) , is consistently affected by the bias in the reported experimental data , because usually the targeted range of molecular property is clearly defined ( Liu et al. , 2015 ; Chen & Zhang , 2013 ) . When regression is performed using such skewed data , it often erroneously predicts that the target properties are satisfied . As a consequence , it is difficult to discover molecules that have the desired properties ( Peng et al. , 2017 ) . Active learning applications also have a similar problem . Many active learning methods repeat the selection of new data by applying certain criteria and retraining the surrogate model ( Lookman et al. , 2019 ; RouetLeduc et al. , 2016 ; Yuan et al. , 2018 ) . During this process , the data can be skewed according to the criteria ( de Mello , 2013 ; Prabhu et al. , 2019 ) . However , despite this problem , few studies have tried to improve the accuracy of regression models that have been trained on skewed data . In this work , we propose a new approach to improve the accuracy of a regression model that is trained using skewed data . We assume the presence of enough unlabeled data which follow the true distribution , and that the true distribution can be roughly estimated using domain knowledge or a few examples . We use a semi-supervised learning framework with an adversarial network to force the distribution of the regression output to resemble the assumed true distribution ( Figure 1 ) . At the same time , by sharing the front part of the regression model with the encoder of an adversarial autoencoder ( AAE ) , the process of forcing the distribution of output values is regularized in a way that the information of the labeled data is represented stably . We created skewed datasets by selecting data that exceeded a certain threshold from four real-world datasets ( pLogP , Diamond , House , Elevators ) , then evaluated the proposed approach using these skewed datasets . The proposed approach reduced the root mean squared error ( RMSE ) of the regression model derived using each of the four datasets , compared to the regression model that had been trained using only the skewed datasets . We also verified that the proposed approach is feasible even when the estimate of the true distribution is not perfect . 2 RELATED WORK . Semi-Supervised Learning ( SSL ) is a machine-learning strategy to learn using partially-labeled datasets ( Chapelle et al. , 2009 ) . In the field of SSL , various methods have been developed , including those using generative models ( Kingma et al. , 2014 ) , graphs ( Goldberg & Zhu , 2006 ) , self-training ( Rosenberg et al. , 2005 ) and consistency regularization ( Sohn et al. , 2020 ) . SSL can improve classification and regression models by using information in a large set of unlabeled data to train a relatively small set of labeled data ( Xie et al. , 2019 ; Creswell et al. , 2018 ; Dimokranitou , 2017 ; Rezagholiradeh & Haidar , 2018 ) . These approaches generally assume that labeled and unlabeled datasets are well distributed without distortion . Several methods were tried to train a model on data that have biased distribution . A method proposed by ( Zhu et al. , 2003 ) incorporated class prior knowledge to adjust the class distribution . Another method proposed by ( Kim et al. , 2020 ) used a model trained with skewed labeled data to generate pseudo-labels of unlabeled data for retraining by considering the entropy of prediction . Positive Unlabeled ( PU ) learning uses data with only positive and unlabeled samples for binary classifications ( Elkan & Noto , 2008 ) ; PU learning has been studied intensively ( Hsieh et al. , 2019 ; Youngs et al. , 2015 ) . However , to the best of our knowledge , few methods used skewed labeled datasets for regression . Although other methods for learning with imbalanced data have been developed , such as the Synthetic Minority Oversampling Technique ( SMOTE ) for regression ( Torgo et al. , 2013 ) , these approaches can only be applied if even some of the data are in the sparse area . 3 METHOD . Our proposed approach is aimed to improve accuracy of regression models that are trained on a dataset that is skewed according to labels . Dl denotes a skewed labeled dataset with n samples that satisfies yi > θ , ∀ ( xi , yi ) ∈ Dl = { ( xi , yi ) } ni=1 ( 1 ) where xi ∈ Rd is d-dimensional input , yi ∈ R is ground truth target and θ is the bias threshold above which the labeled data are present . We assume that we have an estimate of the true distribution of targets y , p ( y ) , and that we have an unbiased unlabeled dataset Du = { xn+1 , xn+2 , . . . , xm } that has sufficiently large m n and in which the distribution of ground truth targets follows p ( y ) . In this section , we describe our approach to make the regression outputs follow the true distribution as closely as possible by using Dl and Du in semi-supervised fashion . The proposed approach consists of two parts : “ Adversarial network for forcing output distribution ” and “ Regularization using AAE ” . In the first part , we present a method that uses an adversarial network to force the output distribution of the regression model to be similar to p ( y ) . Then , we describe a method to properly regularize this forcing process by implementing an AAE . 3.1 ADVERSARIAL NETWORK FOR FORCING OUTPUT DISTRIBUTION . Let ŷ be the output of a regressor that aims to predict ground truth targets y , and q ( ŷ|x ) be the regression distribution . Then for given data distribution pd ( x ) , a distribution of predicted regressor output values q ( ŷ ) can be defined as : q ( ŷ ) = ∫ x q ( ŷ|x ) pd ( x ) dx ( 2 ) We use ŷu to denote the predicted values for Du , and ŷl to denote the predicted values for Dl . Then the distributions of predicted values are denoted as q ( ŷu ) and q ( ŷl ) . The proposed approach is to regularize the regressor to have q ( ŷu ) close to p ( y ) . To do so , an adversarial network is placed at the end of the regressor ( Figure 1 ) . The model is concurrently trained on Dl for regression , and on Du to force the regression outputs to have similar distribution to p ( y ) . The discriminator distinguishes outputs of the regressor from randomly-sampled values in p ( y ) , and the regressor learns to deceive the discriminator . Suppose we divide Du into two subsets , Dabove that consists of x that have ground truth target values > θ , and Dbelow which consist of x that have ground truth target values < θ . If the regressor is well trained on Dl , then prediction on Dabove is expected to be relatively accurate because the distributions of x of Dl and Dabove are similar . In contrast , for Dbelow , the regressor should predict values that have not ever been seen ; therefore , we hypothesized that during adversarial training on Du , the prediction on Dabove should stay relatively still , whereas prediction on Dbelow is flexible . If this hypothesis holds , then the portion of q ( ŷu ) that is > θ should be filled with prediction outputs on Dabove , and prediction outputs on Dbelow should be < θ to deceive the discriminator . These responses result in the regressor having q ( ŷu ) close to p ( y ) while maintaining accurate prediction for outputs > θ . 3.2 REGULARIZATION BASED ON AAE . Even if prediction on Dabove is intact and the predicted output distribution of Du fits the assumed true distribution , the prediction on Dbelow will still have a large error in most cases . To guide the prediction output on Dbelow correctly , the information from above θ must be conveyed to below θ . We used the AAE to regularize the process of forcing the output distribution to achieve appropriate propagation of the information . We set the encoder of an AAE to share the front part of the regression network ( Figure 1 ) . We denote the shared part of the regression network as Renc and rest of the network as Rpost . Renc works as the encoder of the AAE and the front part of the regression network at the same time . Therefore , latent vectors of the AAE should act as a useful feature for the Rpost which functions to predict y from the latent vectors . To fulfil its function , Renc must consider knowledge from the labeled dataset . Also , during the process of forcing the output distribution , Rpost also transforms the given distribution of the latent vectors to p ( y ) . To be a useful feature for Rpost , latent vectors should be arranged in a similar way to p ( y ) , which possesses information about how the labeled dataset is skewed . This implies that latent vectors become more useful if Renc is aware of the skewed distribution of the labeled dataset . To facilitate this process , we shaped the distribution of the latent vectors to have similar characteristics to p ( y ) by adjusting the target distribution of AAE . Consequently , we hypothesized that the regularization process of AAE can be well guided by this information from labeled data and p ( y ) . This good guidance results in well-controlled propagation of information from the portion > θ to the prediction on Dbelow . 4 EXPERIMENTS . In this section , we present results of experiments on regression of artificially-skewed datasets . For a given dataset , we first randomly sampled a sufficient amount of data to construct an unlabeled train dataset . Then we divided the rest of the data arbitrarily into three datasets to form a labeled train dataset , a validation dataset and a test dataset . To induce bias into the labeled dataset fairly , only data with labels that had values more than one standard deviation above the mean of all labels were selected randomly until the labeled dataset was sufficiently large . For the experiments , we used four datasets : moses ( QSAR on calculated pLogP ) , diamond , house and elevators ( Polykovskiy et al. , 2018 ; Wickham , 2016 ; Alcala-Fdez et al. , 2011 ) . Since QSAR on small molecules suffers from skewed datasets in many practical applications , our experiments are focused on the QSAR on calculated pLogP task . Other datasets were used to demonstrate the general applicability of the proposed approach . We focused on comparing regression accuracy of the models that were trained using our approach or without it . Further , to explore the effectiveness at which the adversarial network forced output distribution and AAE based regularization separately , we conducted ablation studies . As the proposed approach requires an estimate of the true distribution , we tested the sensitivity of the model to the quality of the estimate . In each test , the true distribution of the data was estimated to be Gaussian distribution that has a mean and standard deviation of 1 ) full data or 2 ) 20 randomly-sampled data . We set the assumed true distribution as Gaussian distribution for experiments on all four datasets regardless of their real distributions . These experiments should not consider the label of unlabeled dataset until the final accuracy test . Therefore , to tune hyperparameters , we only considered accuracy of the regression on labeled dataset and distribution mismatch between predicted values and the assumed true distribution .

Summary: This paper proposed a semi-supervised learning approach to improve the regression model trained on output-skewed data. The key assumption is that, though the training outputs can be skewed, it is easy to estimate the true distribution of the output. The proposed model that combines an AAE that generates the output distribution, and an adversarial model that enforces the distribution of the predicted output to resemble the true distribution of the output. On several real datasets, the ablation study shows the proposed model can improve the regression accuracy.