Regression from Upper One-side Labeled Data
Authors: Takayuki Katsuki
Venue:  (2021)
Keywords: regression
weakly-supervised learning
healthcare

Abstract: We address a regression problem from weakly labeled data that are correctly labeled only above a regression line, i.e., upper one-side labeled data.
The label values of the data are the results of sensing the magnitude of some phenomenon.
In this case, the labels often contain missing or incomplete observations whose values are lower than those of correct observations and are also usually lower than the regression line. It follows that data labeled with lower values than the estimations of a regression function (lower-side data) are mixed with data that should originally be labeled above the regression line (upper-side data).
When such missing label observations are observed in a non-negligible amount, we thus should assume our lower-side data to be unlabeled data that are a mix of original upper- and lower-side data.
We formulate a regression problem from these upper-side labeled and lower-side unlabeled data. We then derive a learning algorithm in an unbiased and consistent manner to ordinary regression that is learned from data labeled correctly in both upper- and lower-side cases. Our key idea is that we can derive a gradient that requires only upper-side data and unlabeled data as the equivalent expression of that for ordinary regression. We additionally found that a specific class of losses enables us to learn unbiased solutions practically. In numerical experiments on synthetic and real-world datasets, we demonstrate the advantages of our algorithm.

Under review as a conference paper at ICLR 2021
REGRESSION FROM UPPER ONE-SIDE LABELED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
We address regression problem from weakly labeled data that are correctly labeled only above a regression line, i.e., upper one-side labeled data. The label values of the data are the results of sensing the magnitude of some phenomenon. In this case, the labels often contain missing or incomplete observations whose values are lower than those of correct observations and are also usually lower than the regression line. It follows that data labeled with lower values than the estimations a regression function (lower-side data) are mixed with data that should originally be labeled above the regression line (upper-side data). When such missing label observations are observed in a nog-negli amount, we thus should assume our lower-side data to be unlabeled data that are mix of original upper- and lower- side data. We formulate a regression problem from these upper-side labeled and lower-side unlabeled data. We then derive a learning algorithm in unbiased and consistent manner to ordinary regression that is learned from data labeled correctly in both upper- and lower-side cases. Our key idea is that we can derive gradient that requires only upper-side data and unlabeled data as the equivalent expression of that for ordinary regression. We additionally found that a specific class of losses enables us to learn unbiased solutions practically. In numerical experiments on synthetic and weal-world datasets, we demonstrate the advantages of our algorithm.
INTRODUCTION
This paper addresses scenario in which a regression function is learned for label sensor values that are the results of sensing the magnitude of some phenomenon. lower sensor value means not only relatively lower magnitude than higher value but also a missing or incomplete observation of monitored phenomenon. Label sensor values for missing observations are lower than those for when observations are correct without missing observations and are also usually lower than an optimal regression line that is learned from the correct observations. naive regression algorithm using such labels causes the results of prediction to be low and is thus biased and underfitted in comparison with the optimal regression line.
In particular, when the data coverage of a label sensor is insufficient, the effect of missing observations causing there to be bias is critical. One practical example that, for comfort in healtheare, we mimic and replace intrusive wrist sensor (label sensor) with non-intrusive bed sensors (explanatory sensors). We learn a regression function that predicts the values of the wrist sensor from values the bed sensors. The wrist sensor wrapped around wrist. It accurately represents the motion intensity of a person and is used such as for sleep-wake discrimination Tryon 2013);; Mullaney et al. 1980);; Webster et al. (1982) Cole al. (1992). However, it can sense motion only on the forearm, which causes data coverage to be insufficient and observations of movements on other body parts to be missing frequently. The bed sensors are installed under bed; while their accuracy is limited because of their non-intrusiveness, they have much broader data coverage than that of the wrist sensor. In this case, the wrist sensor values for missing observations are improperly low and also inconsistent with the bed sensor values as shown in Fig. (1). This leads to severe bias and underfitting.
The specific problem causing the bias stems from the fact that our data labeled with lower values than the estimations of the regression function are mixed with data that should be originally labeled above the regression line. Here, we call data labeled above the regression line upper-side data, depicted as circles in Fig. (2), and data labeled below the regression line lower-side data, depicted squares in Fig. (2). (2). When there are missing observations, that is, our scenario, it means that the original data with missing observations have been moved the lower side, depicted as triangles in Fig. (3). We

---

Under review as a conference paper at ICLR 2021
Upper-side data
Unlubeled data
Wrist sensor as label sensor
Lower-side data
Unbiased regression line
Sensor value
Bed sensors as explanatory semsor
Optimal regression line
Biused regression line
Label value
Label value
m
Observation point
Missing X
Wrist sensor's 's missing observations in which sensor valuss are lower than those of correct observations
Explanatory variable
Explanatory variable
(1) Example application: we mimic wrist sensor having insufficient data coverage with bed sensors having broader data coverage.
(2) Ordinary regression with upper- and lower-side data
(3) Our setting with upper one-side data
One-side regression problem, where, due to missing observations, data are correctly labeled only above regression line, i.e. upper one-side. Regression function must be learned in unbiased and consistent manner to ordinary regression, where data are labeled correctly in both upper- and lower-side.
Figure I:
cannot determine which data have been moved by just examining the label values. It follows that our lower-side data are mixed with the original upper- and lower-side data.
We thus should assume our lower-side data to be unlabeled data, that is, a mix of original upper- and lower-side data. We overcome the bias by handling this asymmetric label corruption, in which upper-side data are correctly labeled but lower-side data are always unlabeled. There is an established approach against such corrupted weak labels in regression, that is, robust regression that regards weak labels as containing outliers Huber et al. (1964); Narula & Wellington 1982); Draper & Smith (1998); Wilcox (199)). However, since not asymmetric but rather symmetric label corruption is assumed there, is still biased in our problem setting. In the classification problem setting, asymmetric label corrpption addressed with positive-unlabeler (PU) learning. where it assumed that negative data cannot be obtained but unlabeled data available well as positive data Denis (1998); De Comité jet al. 1999); Letouzey et al. (2000); : Shi et al. (2018): Kato et al. 2019); Sakai & Shimizu (2019);
Charoenphakdee & Sugiyama (2019 Li
(2019
; Zhang al. (2019 ; Xu et al. (2019); Zhang
et al..[.22200 (2020); Guo et al. 2020); Chen et al. (2020). The focus is on classification tasks, and an
unbiased risk estimator has been proposed Du Plessis al. (0144 2015). There a gap between the classification problem setting and our regression problem setting, I.e., we have to estimate specific continuous values, not positive/negative classes. We fill the gap with a novel approach for deriving an unbiased solution for our regression setting.
In this paper, we formulate a regression problem from upper one-side labeled data, in which the upper-side data are correctly labeled, and we regard lower-side data as unlabeled data. We refer to this as one-side regression. Using these upper-side labeled and lower-side unlabeled data, we derive a learning algorithm in an unbiased and consistent manner to ordinary regression that uses data labeled correctly in both upper- and lower-side cases. This is achieved by deriving our gradient that requires only uiper-side data and unlabeled data as asymptotically equivalent expression of that for ordinary regression. This is key difference from the derivation of unbiased PU classification where loss has been used. We additionally found that specific class of losses enables to make it so that an unbiased solution can be learned practically. For implementing the algorithm, we propose stochastic optimization method. In numerical experiments using synthetic and real-world datasets, we empirically evaluated the effecctvennes of the proposed algorithm. We found that it improves performance against regression algorithms that assume that both upper- and lower-side data are correctly labeled.
ONE-SIDE REGRESSION
Our goal is to derive a learning algorithm with upper one-side labeled data in an unbiased and consistent manner to ordinary regression that uses both upper- and lower-side labeled data. We first
2

---

Under review as a conference paper at ICLR 2021
consider the ordinary regression problem; after that, we formulate one-side regression problem by transforming the objective function of the ordinary one.
2.1 ORDINARY REGRESSION PROBLEM
Let æ (D E N) be a D-dimensional explanatory variable and y R be a real-valued label. We learn regression function f(x) that computes the value of an estimation of a label, D, for a newly observed æ as 9 = fx)) The optimal regression function f is given by
f* argmin L()),
(1)
where L(f) is the expected loss when the regression function f(x) is applied to data, and y. distributed in accordance with an underlying probability distribution p(a, y):
C(f) = ([)(f())
where E denotes the expectation over p(x, y). and ((())),)) y)) is the loss function between f(x)) and y, e-g., the squared loss, [([ y) = [ - v1
L()) can be written by using the decomposed expectations Eup when labels are higher than estima- tions of the regression function (f(x) < y. upper-side case) and E when labels are lower than the where Tup and Th are the ratios for upper- and lower-side cases, respectively.
estimations of the regression function (y < f(x), lower-side case) as
[ff) = upp (=x)) y)1 Eoo ((f(x). y)].
(3)
Note that the decomposition in Eq. (3) holds for any f including f", and we omitted the decomposed expectation when y f(z) because it is always zero.
2.2 ONE-SIDE REGRESSION PROBLEM
We here consider a scenario in which we have training data, D {Ix, yn that are correctly labeled only in the upper-side case because of the existence of missing label observations. The data in the lower-side case are a mix of original upper- and lower-side data and are considered to be unlabeled data. We can divide D by estimations of the regression function into upper-side data {X Jop = (æt y E DD f(x) < y} and unlabeled data X.. = {æ E D y < [())). In the ordinary regression, where both upper- and lower-side data are correctly labeled for training, expectations Eup and Es in Eq. (3) can be estimated by using the corresponding sample averages. In our setting, however, correctly labeled data from the lower-side case are unavailable, and, therefore. E, cannot be estimated directly.
We can avoid this problem by expressing L(f) as
Ž() (upl)()) v)] + ((T.( No)) Bup (T), Mo))
where expectation E for can be estimated by computing sample average for unlabeled data Xun. and Yo is a virtual label that always lower than the estimations of the regression function [()), whose details will be given the next paragraph. For this expression, the expected loss Z(J represented by only the expectations over the upper-side data and unlabeled data, Eup and E. Thus, we can design a gradient- gradient-based learning algorithm by using our training data. This transformation comes from Eqs. 2 and with Sr as
E [[L(f(x). Di = Tup EappEalL(f(()), 5n + Th Eloooo((((()) Mo)) mo Eio B (z). no)] [[I [m)) Eup [T(((())) (.))
(5)
In practice, we cannot properly set the value of Sıo as being always lower than f(x). However, for set the loss function losses whose gradients
learning based on gradients, this is not needed when do not depend on the value of iho but just on the sign of f(x) - I... which always positive and sgn(f( M)) = 1 from the definition of Soo. i... the loss functions satisfy
OL(((x) u) 00
g(sgn([(x)) w). s(x),
(6)
3

---

Under review as a conference paper at ICLR 2021
where 0 is the parameter vector of f. g (ssg((f(a)) - f(x)) is gradient function depending
(g)( y) and ((=), and sgn(*) is a sign function. Common such losses are absolute loss and quantile losses. For example, the gradient of absolute loss, (f(x) - y/. is
af(w) afle) 00 Undefined
(sgn (f y) 1)
i 0f(x) -
(sgn (f() ) y) = =-1)
d0
(sgn( (f() y) = =0)
which does not depend the value of but just on the sign of f(x) y.
LEARNING WITH GRADIENT USING UPPER ONE-SIDE LABELED DATA
In this section, derive the learning algorithm based on Eqs. 1 and 4 and show that it is unbiased to and consistent with ordinary regression. We consider the gradient of Eq. 4 by using losses that satisfy Eq. (6) for its second and third terms
aL(J) 00
(T([(x), [[gg(gn(f(())) nn.), f()))
Tup Eup
00
Tup Eup (gggng( - yno), (a=))]
(8)
Using upper-side and unlabeled sample sets, (Xup: Yup and Xuu. the gradient in Eq. can be estimated as
aā(f) 00
Tup
Σ
aL(f(x).8)
Σ e
g(sgn(f(x)) - (1o): f(x))
Nup
00
Run
( (X Mpp
Tup
Σ
g(sgn([(x) - (no) [(x))
Dup
=EXup
where (xr )) (Xup: Yup) represent coupled pairs of and y in the upper-side sample set, and Dup and Nun are the numbers of samples for the upper-side and unlabeled sets, respectively.
optimize Eq. (1) and learn the regression function. Its unbiasedness and consistency will be given in Section 3.1, and the specific implementation of the algorithm will be given in Section 3.2.
By using the gradient in Eq. (
3.1 UNBIASEDNESS AND CONSISTENCY OF GRADIENT
Our learning algorithm based on the gradient in Eq. (9) that uses only upper-side data and unlabeled data justified as follows.
Theorem 1.
Suppose that loss function L for the second term in Eq. (3) satisfies Eq. (6) Then, for any f, the gradient in Eq. (8) and its empirical approximation in Eq. (9 are unbiased to and consistent with the gradient of L(f) in Eq. (3)
In other words, learning based on the gradient of Eq. 19), which uses only upper-side data and unlabeled data (one-side regression), asymptotically produces the same result as learning based on the gradient of C(f) in Eq. which uses both upper- and lower-side data (ordinary regression).
Proof. First, by substituting Eq. (5) into the second and third terms in Eq. 8 aā(f)
(L(((x),))
ump Eup
mo fx)))
(10)
00
00
Then, from the definitions of Th. and E.,, both in which is always < f(x),
E1 (sogg(snn(((()) S).), f())) = Eio f(x))
(sgu((() v), f(x)))
(II)
4

---

Under review as a conference paper at ICLR 2021
and, thus, the gradient (T0) is essentially the same as the following gradient of the loss [(f) in Eq. for ordinary regression when we set the loss function for the second term in Eq. as losses that satisfy Eq. (
ac(f)
@L(f(x).8) 00
Tup Rup
+ To Eo y). [a)))
(12)
00
The gradient in Eq. 2 is also unbiased to and consistent with the gradient in Eq. (12). and its conver- gence rate of the order Op (///// + /Rm) in accordance with the central limit theorem Chung 19688), where Op denotes the order in probability.
3.2 IMPLEMENTATION OF LEARNING ALGORITHM BASED ON STOCHASTIC OPTIMIZATION
We scale our algorithm based on Eq. up by stochastic approximation with M M-mini-batcher and add regularization term, R(f):
ai(f) 00
M Σ
OL(f(x).))
Σ
Σ
(LL,
00
m=1
{
ZE (mm)
aR(S)
Σ s(1, [(x))
+
(13)
00
1 (m))
where (X -Ypp and X {m)
{mm)
are upper-side and unlabeled sets in the m-th mini-batch, respec- tively, À a regularization parameter, and the regularization term R(()) ()) is, for example, the Ll or L2 norm for the parameters Q. We also convert "ony (um) p ignoring constant coefficients and apply sgn(f(z) - M)) = 1. The hyperparameters and are optimized in training.
We can learn the regression function with the gradient in Eq. (13) by using any stochastic gradient method, such as Adam Kingma & Ba (015) and FOBOS Duchi & Singer (2009). The algorithm is deseribed in Algorithm 1 In the following experiments,
used Adam with the hyperparameters
recommended in Kingma & Ba 2015), and the number of samples the mini-batches was set to 32. By using the learned (xx) we can estimate ğ f(x) for new data.
From a practical perspective, the first term in Eqs. (9) and (33 requires that the estimations for the upper-side samples be higher than their label values as much possible because ( (X.::)) L(f(x). y) becomes zero when every estimation of the regression function f(x) is located in y S f(x). In contrast, the second term requires that the estimations for unlabeled samples be just as small as possible. The third term balances the effect of the second term.
Our algorithm and discussions applicable scenario that is opposite the one-side case, where the data are correctly labeled only on the lower side. Since the derivation is obvious from the analogy of the upper one-side case, just show its learning algorithm for the lower one-side case in the supplementary material. One example of the lower one-side case is when the label sensor has ideal coverage, but the cost of observation is high, and need to mimic sensor values with other cheaper sensors having smaller coverage.
4 EXPERIMENTAL RESULTS
We now empirically test the effectiveness of the proposed approach. Our goal to investigate the impact of our unbiased gradient, which is derived from the objective function based the assumption of upper one-side labeled data Eq. (4). We thus show how the proposed method improves performance against regression methods whose objective functions assume that both upper- and lower-side data are correctly labeled. We use the same model and optimization method for all of the methods, and the only difference is their objective functions.
4.1 EXPERIMENTAL SETUP AND DATASETS
We report the mean absolute error (MAE) and its standard error between the estimation results V. and the corresponding true labels ý across 5-fold cross-validation, each with different (
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1 One-side regression based on stochastic gradient method
Input: Training data D {In, V/ Jn and hyperparameters p, A 2 0
Output: Model parameters 0 for
I: Let A be an external stochastic gradient method and Gm be gradient for the m-th mani-bbtcch
2: while No stopping criterion has been met
3: Shuffle D into wi--mmi--bachhee and denote by {x (m)) ymm) } the m-th mini-batch whose
size is Nm
4: for m 1 to M
5: Gm +-
6: for n to Nm
7:
if f( (mm))
yn < 0 then
e >t(s(a!" (m)))) )m(m))))mm)))))))))))))))))))) [mmmm))
(1. (arh (m))) + (R())
8:
Gm t Gm
86
9:
else
Gm f- Gm + pg f (zh1m)) + AR())
10:
11: Update by A with Gm
randomly sampled training-testinn split. MAE is defined as MAE(Ú. = = I/n bml For each fold of the cross-validation, we used randomly sampled 20% of the training set as validation set to choose the best hyperparameters for each algorithm, where hyperparameters providing the highest MAE in the validation set were chosen. All of the experiments were carried out with a Python implementation workstations having 48-80 GB of memory and 2.3-4.0 GHz CPUs. With this environment, the computational time was a few hours for producing the results for each dataset.
Data 1: Synthetic dataset. Using a synthetic dataset, we investigated whether our algorithm could indeed learn from upper one-side labeled data. We randomly generated N training samples, X {I,n J.Y y==1, from the standard Gaussian distribution (æ,, 0, I), where the number of samples was N = I, 000, the number of features in x was D = 10, and is the identity matrix. Then, using X, we generated the corresponding N sets of true labels ý = {w.) J.Y from the distribution Ni, w 3), where w are coefficients that were also randomly generated from the standard Gaussian distribution Nww 0, I). / is the noise precision, and T denotes the transpose. For sim- ulating the situation in which a label sensor has missing observations, we created training labels y = (3mm [3,) ==1 by randomly selecting K percent of data in ÿ and replacing their values with the minimum value of ÿ. We finally added white Gaussian noise whose precision was the same as that of ý for the replaced K percent of data. We repeatedly evaluated the proposed method for each of the following settings. The noise precision was 3 = {100, 10-1), which corresponded to low- and high-noise settings, and the proportion of missing training samples was K {25, 50, 75) %. In the case of K = 75%, only 25 percent of the samples correctly corresponded to labels, and all of the other samples were attached with labels that were lower than the corresponding true values. In general, it is quite hard to learn regression functions using such data. In the experiment on Data 1, we used a linear model, x, for f(x and an implementation for Eq. (13) with squared loss for the first term, absolute loss, which satisfies Fq.. (6). for the second and third terms, and I. -reeullarizationn for the regularization literature, e.g., Huber loss Huber et al. (1964), Epsilon-insensitiv loss Vapnik (1995), and quantile losses Koenker & Bassett Jr (1978). We set the candidates of the hyperparameters, and X, to {10-8 10 10 J. We standardized the data by subtracting their mean and dividing by their standard deviation in the training split.
term. Loss functions having such a heterogeneous aspect are often used in the
Data 2: Kaggle dataset with synthetic corruption. We used a real-world sensor dataset collected from the Kaggle dataset [Sn (2016) that contains breathing signals. For this dataset, we used signals from chest belt as is the gold standard for measuring ventilation, as true labels ý (Un, } The dataset consisted of 1, 432 samples, and æ in each sample had D = number of features, i.e., the period and height of the expansion/coontaation of the chest. For our problem setting, we created training labels = (vn
(
== and signals obtained by the Douglas bag (DB) method, which
IN 1==1 by randomly selecting K percent of data in ÿ and replacing their value with the minimum value of ÿ. We finally added white Gaussian noise whose standard deviation was 0.1 x for the replaced K percent data, where s is the standard deviation of the original y. The setting for K was the same as that of Data K (25, 50, 75) %. In the experiment on Data 2, for its non-

---

Under review as a conference paper at ICLR 2021
Table 1:
Comparison of proposed method and methods based various objective functions in terms
of MAE (smaller is better). We show best methods in bold.
(1) Data 1: Synthetic dataset
Low-n setting (3 109)
High-noise setting (3 = 10-1)
25%
50%
75%
25%
K 50%
75%
MSE Proposed
0.77
1.53
2.30:10.02
1.03
1.62:10.03
2.36:1:0.03
0.58
0.60
0.60
0.78
0.79
0.79:00002
(2) Data 2: Kaggle dataset with synthetic corruption
K 25%
K 50%
K 75%
MSE Proposed
0.55+ 0.43
0.91+0.02 0.46
1.32+00.02
0.59
(3) Data Real- UCI dataset
Class A
Class B
Class C
Class D
Class E
Avg.
MSE
1.003
1.54
1.22
1.3710.01
1.2110.01
1.58
MAE
2.14
1.4610.01
1.4400.01
1.33
1.3110.01
1.54
Huber
2.04
1.66.10.01
1.45
1.50
1.32:10.01
1.59
Proposed-1
1.05
1.18
1.11
1.14#0.01
1.03
1.20
Proposed-2 -
1.32:1..01
0.99+:0011
0.94+0.01
0.86#0.01
0.97
1.02
linearity, used 0T (T for fax) where Φ is radial basis function, and is hyperparameter representing the kernel width that is also optimized in the training split. We set the candidates of the hyperparameters, p, X, and 0, to {10-3 10 2,100__ 1000. The other implementation was the same as that for Data 1.
Data 3: Real-world UCI dataset. We here applied the algorithm real sensor dataset, which was collected from the UCI Machine Learning Repository Velloso (2013): Velloso et al. (2013). contains sensor outputs from dumbbells and from wearable devices attached to the arm, forearm, and waist during exercise. We used all of the features from the dumbbell sensor that took "None" values less than ten times as X { /n where each sample had D 13 number of features. We used the magnitude of acceleration on the arm as training labels y frmm >n-1, which had insufficient data coverage and missing observations for the movements of other body parts. For testing, we used the magnitude of acceleration for the entire body as true labels ý [Ün W Because there were five classes for the exercise task with severe mode changes between classes, we divided the dataset into five datasets
the basis of class: A (N = 11, 159), B (N 7, 593), C (N 6, 844), (N = 6, 432), and E (N 214).. In the experiment on Data we used a 6-layer multilayer perceptron with ReLU Nair & Hinton 2010) (more specifically.
f(x) in order to demonstrate the usefulness of the proposed method in training deep neural networks. We also used a dropout Srivastava al. (2014) with rate of 50% after each fully connected layer. We used two implementations for the first term in Eq. (13) with absolute loss (Proposed-1) and squared loss (2roposed-2). For both implementations, we used the absolute loss, which satisfies Eq. (6). for the second and third terms and used Ll-regularization for the regularization term. We set the candidates of the hyperparameters, P and to {10 same as that for Data 1.
1-1000100-100-1111
100 1.0 The other implementation was the
4.2 PERFORMANCE COMPARISON
Table (1) and -(2) show the performance on Data and Data 2 for the proposed method and an ordinary regression method that uses mean squared error (MSE) assuming that both upper- and lower-side data are correctly labeled as its objective function. This comparison shows whether our method could learn from upper one-side labeled data, from which the ordinary regression method could not learn. From Table (1) and -(2), we can see that the overall performance of the proposed method was significantly better than that of MSE. We found that the performance of our method was not significantly affected by the increase in the proportion of missing training samples K even for K = 75%, unlike that of MSE. Table (3) shows a more extensive comparison using the real- world UCI dataset (Data 3) between methods, Proposed-1 and Proposed-2, and methods based
7

---

Under review as a conference paper at ICLR 2021
(MSE)
Estimation
True
(MSR)
1 1.
0 Sensor value
Sensor value
Estimation
True
200
400
600
800
1000
1200
1400
200
(Proposed)
(Pruposed-2) 2)
G Ro 4 o
200
400
600
800
1000
1200
1400
1000
Index (timestamp)
Index (timestamp)
(1) Data 2: Kaggle dataset with synthetic corruption
(2) Data 3: Real-world UCI dataset (class E)
Figure 2: Comparison of estimation results of proposed method and those of MSE. Orange line represents estimation results of each method, and blue line represents true label values.
various objective functions consisting of MSE, MAE, and Huber losses Huber et al. (1964); Narula & Wellington (1982); Wilcox (1997). The regression methods based MAE and Huber losses were robust regression methods that assume symmetric label corruption. From Table [T33), we can see that the performance of Proposed-1 1 and Proposed-2
totally better than that of the baselines. The robust regression methods did not improve in performance against MSE. In particular, Proposed-1 and Proposed-2 respectively reduced the error by more than 20% and 30% compared with the other methods on average.
Demonstration of unbiased learning and prediction. Figure 2 (1) compares the estimation results of the proposed method with true labels and those of MSE for the Kaggle dataset with synthetic corruption (Data 2). Since the ordinary regression method, MSE, regards both upper- and lower-side data as correctly labeled, we can see that it produced biased results due to the missing observations. The proposed method did not. Figure 2 (2) shows a comparison of the estimation results between the proposed method, Proposed-2, -2, and MSE for the real-world UCI dataset (Data 3). For ease of viewing, we show the results for the first 1, 000 samples for the class data, where the errors of most of the methods were the lowest. Although MSE showed the lowest error among the baselines for the class E data, we can see that the predictions by MSE were somewhat biased and underfitted for the real data having our assumed nature. This was not the case for the proposed method.
4.3 REAL HEALTHCARE CASE STUDY
We show the results of healthcare case study in the supplementary material, where we estimated the motion intensity of participant that measured accurately with an intrusive sensor wrapped around the wrist (ActiGraph) TTronn (013); Mullaney et al.) (1980); Webster et al. (1922); Cole al. (992) from non-intrusive bed sensors that installed under bed. The results showed that the intrusive sensor could be replaced with the non-intrusive ones, which would be quite useful for reducing the burden on users.
CONCLUSION
We formulated a one-side regression problem using upper-side labeled and lower-side unlabeled data and proposed learning algorithm for it. We showed that our learning algorithm is unbiased to and consistent with ordinary regression that uses data labeled correctly in both upper- and lower-side cases. We developed stochastic optimization method for implementing the algorithm. An experimental evaluation using synthetic and real-world datasets demonstrated that the proposed algorithm was significantly better than regression algorithms without the assumption of upper one-side labeled data.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Nontawat Charoenphakdee and Masashi Sugiyama. Positive-unlaheled classification under class prior shift and asymmetric error. In SDM, pp. 271-279, 2019.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-PU: Self boosted and calibrated positive-unlabeled training. In ICML, 2020.
Kai Lai Chung. A course in probability theory. Academic press, 1968.
Roger Cole. Daniel F Kripke, William Gruen, Daniel J Mullaney, and J Christian Gillin. Automatic sleep/wake identification from wrist activity. Sleep, 15(5)4461-49, 1992.
Francesco De Comité, François Denis, Rémi Gilleron, and Fabien Letouzey. Positive and unlabeled examples help learning. In ALT, pp. 219-230, 1999.
François Denis. Pac learning from positive statistical queries. In ALT, pp. 112-126, 1998.
Norman R Draper and Harry Smith. Applied regression analysis, volume 326. John Wiley & Sons, 1998.
Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation learning from positive and unlabeled data. In ICML, pp. 13..188666 1386-1394, 2015.
Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. NIPS, pp. 703-711, 2014.
John Duchi and Yoram Singer. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research, 10(Dec):2899-25)) 2934, 2009.
Tianyu Guo, Chang Xu, Jiajun Huang. Yunhe Wang, Boxin Shi. Chao Xu, and Dacheng Tao. On positive-unlabeled classification in GAN. In CVPR, pp. 8385-8393, 2020.
Peter Huber et al. Robust estimation a location parameter. The Annals of Mathematical Statistics, 35(1):73-101, 1964.
Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with selection bias. In ICLR, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Roger Koenker and Gilbert Bassett Jr. Regression quantiles. Econometrica: journal of the Economet- ric Society, pp. 33-50, 1978.
Fabien Letouzey, François Denis, and Rémi Gilleron. Learning from positive and unlabeled examples. In ALT, pp. 71-85, 2000.
Tianyu Li, Chien-Chih Wang, Yukun Ma, Patricia Ortal, Qifang Zhao, Bjorn Stenger, and Yu Hirate. Learning classifiers on positive and unlabeled data with policy gradient. ICDM, pp. 399 409.408. 2019.
DJ Mullaney, DF Kripke, and S Messin. Wrist-actigraphic estimation of sleep time. Sleep, 3(1): 83-92, 1980.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807-814, 2010.
Subhash C Narula and John F Wellington. The minimum sum of absolute emrors regression: A state of the art survey. International Statistical Review/Revue Internationale de Statistique, pp. 317-326. 1982.
Tomoya Sakai and Nobuyuki Shimizu. Covariate shift adaptation learning from positive and unlabeled data. In AAAI, pp. 4838-4845, 2019.
Sagar Sen. Kaggle dataset. http://www..eeggg..oo///sssssseu data-from------- 2016.
from- a- foom-a-ches--oe

---

Under review as a conference paper at ICLR 2021
Hong Shi, Shaojun Pan, Jian Yang. and Chen Gong. Positive and unlabeled learning via loss decomposition and centroid estimation. In IJCA,, pp. (689-2695, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-198, 2014.
Warren W Tryon. Activity measurement in psychology and medicine. Springer Science & Business Media, 2013.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1995.
Eduardo Velloso. UCI machine learning repository.
2013.
Eduardo Velloso, Andreas Bulling. Hans Gellersen, Wallace Ugulino, and Hugo Fuks. Qualitative activity recognition of weight lifting exercises. In AH, pp.
2013.
John B Webster, Daniel F Kripke, Sam Messin, Daniel J Mullaney, and Grant Wyborney, An activity-based sleep monitor system for ambulatory use. Sleep, 5(4):389-399 1982.
Rand R Wilcox. Introduction to robust estimation and hypothesis festing. Academic Press, 1997.
Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, XU Chunjing, Dacheng Tao, and Chang Xu. Fositive-unlabeled compression on the cloud. NeurIPS, pp. 2561-2570, 2019.
Chenguang Zhang, Yuexian Hou, and Yan Zhang. Learning from positive and unlabeled data without explicit estimation of class prior. In AAAI, pp. 6762-6769, 2020.
Chuang Zhang, Dexin Ren, Tongliang Liu, Jian Yang, and Chen Gong. Positive and unlabeled learning with label disambiguation. In IJCAI, pp. 1-7, 2019.
10


Reviews:
[{"review_id": "vpljbq0QdC5", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper considers a regression setting in which the missing values are observed with lower values than the true values. Authors provided appealing application for this problem setting. They rewrote the risk and provided an unbiased gradient estimator. However, there is a gap between the estimator and the actual implementation, thus making the overall paper less convincible.\n\nMain Concern:\n- A gap exists between Eq. (8) and Eq. (13). In Eq. (8), the expectation is taken over the distribution of \"up\". This distribution, as well as \\pi_{up}, is fixed throughout training. Unlike PU classification, this essential information is not given in this problem setting. However, according to Eq. (13) and Algorithm 1, this distribution and \\pi_{up} change ever minibatch with the current model. I admit this is a pratical algorithm, but it differs substaintialy from the first half of the paper. To fill the gap, investigation on how Eq. (13) approximate Eq. (8) should be conducted at least.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aOt6enQucgn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the problem of training a regression model when only for a subset of the datapoints (those for which their label lie above the current model prediction) the correct labels are available. A few comments,\n\n1) It is unclear if the labels y can be noisy. I assume they can't be, because all the derivations seem to be under the assumption they are not. \n2) The application of this setup is not clear to me. I think the authors would benefit from motivating it via other papers that study the same regression problem (if any), as opposed to citing other topics in motion sensor research. They would also benefit from writing down the existing work on classification more explicitly, and connecting it to the regression setup in the paper. It is unclear if the classification literature treats a similar (or exactly the same) problem in the classification setting and what is the hard part of translating these results into regression.\n\nThe solution to the problem proposed by the authors is quite simple. This would not be a downside if the motivation of the problem and the related work was established with more authority at the beginning of the problem. I am concerned that in the case of losses of the form g(sgn(f(x)-y), f(x)) the problem is not meaningful because in this case the learner only requires to know if f(x) < y or not. The initial problem the authors set to solve vanishes in this case. This means that Theorem 1 is not very informative. \n\nIn section 3.2, there is little explanation as to why using a \\rho multiplier in (13). This does not seem to be in accordance with Theorem 1. It is also unclear to me why do the last two terms on the RHS of (4) need to be written as a difference, when at the end gradients of an expectation of a loss of the form g(sgn(f(x)-y)), f(x)) over the unlabeled dataset can be computed directly. There doesn't seem to be any need of writing it as a difference. \n\nThe experimental evaluation is thorough. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "82cQtEXrp-Y", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors address a new weakly supervised regression problem. In this problem setting, upper-side data (labeled above the regression line) and unlabeled data are provided. To solve this problem, the authors derive a learning algorithm in an unbiased and\nconsistent manner to ordinary regression that is learned from data labeled correctly in both upper- and lower-side cases. Experiments demonstrate the advantages of the proposed algorithm.\n\nPros:\n1.\tTo the best of my knowledge, this paper is the first to solve the weakly supervised regression problem presented in the paper. I consider that it is the biggest advantage of this paper.\n2.\tThis paper proposes a consistent learning algorithm to solve the above problem.\n3.\tExperiments demonstrate the effectiveness of the proposed algorithm.\n\nCons:\n1.\tThe presentation of this paper needs to be improved. For example, I understand that in the introduction section, the authors try to justify that the weakly supervised regression problem (where upper-side and unlabeled data are available) is reasonable and could be encountered in real-world settings. However, I personally feel that the presentation is not very clear and I am not fully convinced. In addition, for the order of Eq. (4) and Eq. (5), I think it would be better to present Eq. (5) before Eq. (4), as Eq. (4) relies on Eq. (5).\n2.\tFor the proposed consistent algorithm, I would admit that it is novel to some degree, while the technical contribution of this algorithm is limited. It is worth noting that the proposed algorithm is adapted from the risk estimator of PU learning (Du Plessis et al., 2014; 2015). I think the only key contribution lies in Eq. (6), e.g., the authors show that instead of setting the value of ${\\tilde{y}}\\_{\\text{lo}}$, we can find the gradient only depends on the sign of $f(\\boldsymbol{x})-{\\tilde{y}}\\_{\\text{lo}}$.\n3.\tFor the experiments, it seems that the authors do not use a ground-truth regression line to separate the given data, and obtain upper-side and unlabeled data. Instead, they corrupt some selected data by setting their value to the minimum regression value. I feel that this practical operation does not accord with the proposed problem setting. Maybe we could use originally labeled data to obtain a well-trained regression line and then obtain the required upper-side and unlabeled data.\n\nIn summary, this paper proposed a novel problem setting and a novel learning algorithm, while the problem setting is not well justified and the technical contribution of the algorithm is limited. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses regression in a weakly supervised setting where the correct labels are only available for examples whose prediction lie above some threshold. The paper proposes a method using a gradient that is unbiased and consistent.

Pros:
- Problem setting is new and this paper is one of the first works exploring it.
- The procedure comes with some unbiasedness and consistency guarantees. 
- Experimental results on a wide variety of datasets and domains.

Cons:
- Novelty and technical contribution is limited.
- Motivation of the problem setting was found to be unclear.
- Some gaps in the experimental section (i.e. needing the use of synthetic data or synthetic modifications of the real data).

Overall, the reviewers felt that as presented, the paper did not convincingly motivate the proposed upper one-sided regression problem as important or relevant in practice, which was a key reason for rejection. The paper may contain some nice ideas and I recommend taking the reviewer feedback to improve the presentation. 
Decision: Reject