Regression from Upper One-side Labeled Data

1 INTRODUCTION . This paper addresses a scenario in which a regression function is learned for label sensor values that are the results of sensing the magnitude of some phenomenon . A lower sensor value means not only a relatively lower magnitude than a higher value but also a missing or incomplete observation of a monitored phenomenon . Label sensor values for missing observations are lower than those for when observations are correct without missing observations and are also usually lower than an optimal regression line that is learned from the correct observations . A naive regression algorithm using such labels causes the results of prediction to be low and is thus biased and underfitted in comparison with the optimal regression line . In particular , when the data coverage of a label sensor is insufficient , the effect of missing observations causing there to be bias is critical . One practical example is that , for comfort in healthcare , we mimic and replace an intrusive wrist sensor ( label sensor ) with non-intrusive bed sensors ( explanatory sensors ) . We learn a regression function that predicts the values of the wrist sensor from values of the bed sensors . The wrist sensor is wrapped around a wrist . It accurately represents the motion intensity of a person and is used such as for sleep-wake discrimination Tryon ( 2013 ) ; Mullaney et al . ( 1980 ) ; Webster et al . ( 1982 ) ; Cole et al . ( 1992 ) . However , it can sense motion only on the forearm , which causes data coverage to be insufficient and observations of movements on other body parts to be missing frequently . The bed sensors are installed under a bed ; while their accuracy is limited because of their non-intrusiveness , they have much broader data coverage than that of the wrist sensor . In this case , the wrist sensor values for missing observations are improperly low and also inconsistent with the bed sensor values as shown in Fig . 1- ( 1 ) . This leads to severe bias and underfitting . The specific problem causing the bias stems from the fact that our data labeled with lower values than the estimations of the regression function are mixed with data that should be originally labeled above the regression line . Here , we call data labeled above the regression line upper-side data , depicted as circles in Fig . 1- ( 2 ) , and data labeled below the regression line lower-side data , depicted as squares in Fig . 1- ( 2 ) . When there are missing observations , that is , our scenario , it means that the original data with missing observations have been moved to the lower side , depicted as triangles in Fig . 1- ( 3 ) . We can not determine which data have been moved by just examining the label values . It follows that our lower-side data are mixed with the original upper- and lower-side data . We thus should assume our lower-side data to be unlabeled data , that is , a mix of original upperand lower-side data . We overcome the bias by handling this asymmetric label corruption , in which upper-side data are correctly labeled but lower-side data are always unlabeled . There is an established approach against such corrupted weak labels in regression , that is , robust regression that regards weak labels as containing outliers Huber et al . ( 1964 ) ; Narula & Wellington ( 1982 ) ; Draper & Smith ( 1998 ) ; Wilcox ( 1997 ) . However , since not asymmetric but rather symmetric label corruption is assumed there , it is still biased in our problem setting . In the classification problem setting , asymmetric label corruption is addressed with positive-unlabeled ( PU ) learning , where it is assumed that negative data can not be obtained but unlabeled data are available as well as positive data Denis ( 1998 ) ; De Comité et al . ( 1999 ) ; Letouzey et al . ( 2000 ) ; Shi et al . ( 2018 ) ; Kato et al . ( 2019 ) ; Sakai & Shimizu ( 2019 ) ; Charoenphakdee & Sugiyama ( 2019 ) ; Li et al . ( 2019 ) ; Zhang et al . ( 2019 ) ; Xu et al . ( 2019 ) ; Zhang et al . ( 2020 ) ; Guo et al . ( 2020 ) ; Chen et al . ( 2020 ) . The focus is on classification tasks , and an unbiased risk estimator has been proposed Du Plessis et al . ( 2014 ; 2015 ) . There is a gap between the classification problem setting and our regression problem setting , i.e. , we have to estimate specific continuous values , not positive/negative classes . We fill the gap with a novel approach for deriving an unbiased solution for our regression setting . In this paper , we formulate a regression problem from upper one-side labeled data , in which the upper-side data are correctly labeled , and we regard lower-side data as unlabeled data . We refer to this as one-side regression . Using these upper-side labeled and lower-side unlabeled data , we derive a learning algorithm in an unbiased and consistent manner to ordinary regression that uses data labeled correctly in both upper- and lower-side cases . This is achieved by deriving our gradient that requires only upper-side data and unlabeled data as an asymptotically equivalent expression of that for ordinary regression . This is a key difference from the derivation of unbiased PU classification where loss has been used . We additionally found that a specific class of losses enables us to make it so that an unbiased solution can be learned practically . For implementing the algorithm , we propose a stochastic optimization method . In numerical experiments using synthetic and real-world datasets , we empirically evaluated the effectiveness of the proposed algorithm . We found that it improves performance against regression algorithms that assume that both upper- and lower-side data are correctly labeled . 2 ONE-SIDE REGRESSION . Our goal is to derive a learning algorithm with upper one-side labeled data in an unbiased and consistent manner to ordinary regression that uses both upper- and lower-side labeled data . We first consider the ordinary regression problem ; after that , we formulate a one-side regression problem by transforming the objective function of the ordinary one . 2.1 ORDINARY REGRESSION PROBLEM . Let x ∈ RD ( D ∈ N ) be a D-dimensional explanatory variable and y ∈ R be a real-valued label . We learn a regression function f ( x ) that computes the value of an estimation of a label , ŷ , for a newly observed x as ŷ = f ( x ) . The optimal regression function f∗ is given by f∗ ≡ argmin f L ( f ) , ( 1 ) where L ( f ) is the expected loss when the regression function f ( x ) is applied to data , x and y , distributed in accordance with an underlying probability distribution p ( x , y ) : L ( f ) ≡ E [ L ( f ( x ) , y ) ] , ( 2 ) where E denotes the expectation over p ( x , y ) , and L ( f ( x ) , y ) is the loss function between f ( x ) and y , e.g. , the squared loss , L ( f ( x ) , y ) = ‖f ( x ) − y‖22 . L ( f ) can be written by using the decomposed expectations Eup when labels are higher than estimations of the regression function ( f ( x ) < y , upper-side case ) and Elo when labels are lower than the estimations of the regression function ( y < f ( x ) , lower-side case ) as L ( f ) = πupEup [ L ( f ( x ) , y ) ] + πloElo [ L ( f ( x ) , y ) ] , ( 3 ) where πup and πlo are the ratios for upper- and lower-side cases , respectively . Note that the decomposition in Eq . ( 3 ) holds for any f including f∗ , and we omitted the decomposed expectation when y = f ( x ) because it is always zero . 2.2 ONE-SIDE REGRESSION PROBLEM . We here consider a scenario in which we have training data , D ≡ { xn , yn } Nn=1 , that are correctly labeled only in the upper-side case because of the existence of missing label observations . The data in the lower-side case are a mix of original upper- and lower-side data and are considered to be unlabeled data . We can divide D by estimations of the regression function f into upper-side data { Xup , yup } ≡ { x , y ∈ D | f ( x ) < y } and unlabeled data Xun ≡ { x ∈ D | y < f ( x ) } . In the ordinary regression , where both upper- and lower-side data are correctly labeled for training , expectations Eup and Elo in Eq . ( 3 ) can be estimated by using the corresponding sample averages . In our setting , however , correctly labeled data from the lower-side case are unavailable , and , therefore , Elo can not be estimated directly . We can avoid this problem by expressing L ( f ) as L̃ ( f ) ≡πupEup [ L ( f ( x ) , y ) ] + E [ L ( f ( x ) , ỹlo ) ] − πupEup [ L ( f ( x ) , ỹlo ) ] , ( 4 ) where expectation E for x can be estimated by computing a sample average for our unlabeled data Xun , and ỹlo is a virtual label that is always lower than the estimations of the regression function f ( x ) , whose details will be given in the next paragraph . For this expression , the expected loss L̃ ( f ) is represented by only the expectations over the upper-side data and unlabeled data , Eup and E. Thus , we can design a gradient-based learning algorithm by using our training data . This transformation comes from Eqs . ( 2 ) and ( 3 ) with ỹlo as E [ L ( f ( x ) , ỹlo ) ] = πupEup [ L ( f ( x ) , ỹlo ) ] + πloElo [ L ( f ( x ) , ỹlo ) ] πloElo [ L ( f ( x ) , ỹlo ) ] = E [ L ( f ( x ) , ỹlo ) ] − πupEup [ L ( f ( x ) , ỹlo ) ] . ( 5 ) In practice , we can not properly set the value of ỹlo as being always lower than f ( x ) . However , for learning based on gradients , this is not needed when we set the loss function as losses whose gradients do not depend on the value of ỹlo but just on the sign of f ( x ) − ỹlo , which is always positive and sgn ( f ( x ) − ỹlo ) = 1 from the definition of ỹlo , i.e. , the loss functions satisfy ∂L ( f ( x ) , y ) ∂θ = g ( sgn ( f ( x ) − y ) , f ( x ) ) , ( 6 ) where θ is the parameter vector of f , g ( sgn ( f ( x ) − y ) , f ( x ) ) is a gradient function depending on sgn ( f ( x ) − y ) and f ( x ) , and sgn ( • ) is a sign function . Common such losses are absolute loss and quantile losses . For example , the gradient of absolute loss , |f ( x ) − y| , is ∂|f ( x ) − y| ∂θ = ∂f ( x ) ∂θ ( sgn ( f ( x ) − y ) = 1 ) −∂f ( x ) ∂θ ( sgn ( f ( x ) − y ) = −1 ) Undefined ( sgn ( f ( x ) − y ) = 0 ) , ( 7 ) which does not depend on the value of y but just on the sign of f ( x ) − y .

Summary: This paper considers a regression setting in which the missing values are observed with lower values than the true values. Authors provided appealing application for this problem setting. They rewrote the risk and provided an unbiased gradient estimator. However, there is a gap between the estimator and the actual implementation, thus making the overall paper less convincible.