IBMresearch

Tokyo Research Laboratory

Regression with Interval Output Values
Hisashi Kashima
Kazutaka Yamasaki
Hiroto Saigo
Akihiro Inokuchi

IBM ResearchIBM Japan
Max Planck InstituteOsaka University

 We consider a regression problem, where
the target values in training data are given as “intervals”
 We propose an EM-based solution for this problem

1Problem definition: Regression with interval output values
In ordinary regression problems,
the output for a training instance is given as a point
input xIDx1

x2

x3

x4

output
y1

2

5

12

6

13

2

...

...

...

...

...

3539107An output isgiven as a point

In contrast, the output is given as an interval [ l, r ] in our problem
IDinput xoutput [l, r]

x1x2x3x4lr

12

5126

11

14

2

...

...

...

...

353

9

106∞

An output is given
as an interval

Examples of regression from interval output values
 Sales prediction from past sales data with occasional losses Prediction of number of troubles for quality assurance of projects
 Screening of chemical compounds for drug discovery (QSAR)

2

Existing methods cannot handle interval outputs appropriately
Existing regression methods can estimate p(y|x) from point outputs
input xx1x2x3x4

output
y12

5126

13

2...

...

...

...

...

353

9107IDmodel
estimationp(y|x)The existing regression methods can NOT handle interval outputs– Naive solution would be “Use only the point outputted instances(and neglect instances with interval outputs)”
input xID

output yx1x2

x3x4

lr

12

5

12

6

1114

2

...

...

...

...

353

910

model
estimationneglectedp(y|x)
6used

6

Our approach: Iterative estimation of the model and“representative values” of interval outputs
Iterate the following two steps:– Use the current model to give representative values to interval outputs– Use the representative values to estimate the new model
input xoutput yx1

x2

x3x4

lr

representativeoutput

1

2

5

12

6

11

14

13

2

...

...

...

...

...

...

3

5

3

910∞

7ID

6

The existing regression methods can be applied
if representative values are assigned

3

Representative output values are given as conditional meansby the current model Using the current model f (y | x; θ^ ) , the representative output for
the i -th instance is given as the conditional mean in [l(i), r(i)]– When the conditional mean is analytical intractable, use sampling Interpreted as an EM-algorithm when f is in the exponential family
Conditional mean in [l(i),r(i)]predictive distribution by thecurrent model f (y|x(i) ; θ^ )mean

Conditional mean in [l(i),r(i)]

l(i)

r(i)

y|xProcedure
Read a data set

where the i-th instance is given as (x(i), [l(i), r(i)])

Estimate
the initial modelby using only the instance where l(i) = r(i) （= y(i)）

Assignrepresentative outputs

by using the current model f

CallUpdate the modelNO

Call an existing
regression algorithm

YES

Converged ?

Output the model

4

Experiments with two benchmark data sets
 We used two data set– “Boston housing” data set: House price prediction– “EDKB” data set: Drug activity prediction of chemical compounds

 We compared the proposed method with two approaches– Method 1: Neglect instances with interval outputs– Method 2: Use (non-conditional) means as representative outputs

 We used a linear Gaussian model as the base regression method

Results 1: Price prediction (Boston housing) 506 instances (houses) with 12 input features (#rooms, crime rate, ...)– We used a Gaussian kernel as features
 Since this data set is originally point-outputted, half of the all instances
are converted to interval-outputted data by using uniformly randomvariables ∆l and ∆r ∈ 0.1 · y(i) for the i-th instance– Proposed method L uses [y(i) - ∆l , ∞] as the i-th interval– Proposed method LR uses [y - ∆l , y + ∆r ] as the i-th interval The proposed method outperformed the others
methodMethod1

Method Proposed Proposed
2
LLR
比較結果

M.S.E.

14.18

13.84

12.03

Proposed method
works well

11.37

5

Results 2: Drug activity prediction (EKDB) 59 chemical compounds with 13,600 features (found by a sub-graphpattern mining algorithm)
activity value(logRPP) Compounds with activities less than the 1/3-quantile of logRPP scoreare considered as “apparently inactive” by experts, and given theinterval outputs as [-∞, -0.8421 (=1/3-quantile)] The proposed method outperformed the others

methodMethod1Method
Proposed
2

M.S.E.

0.198

0.208

0.190

Proposed method
works well6