Utility-based Regression

Luis Torgo, Paula Branco and Rita Ribeiro
Dalhousie Univ., Canada
INESC Tec, Univ. Porto, Portugal

May, 2018(Torgo et. al.)Utility-based Regression

May, 2018

1 / 34

Predictive Analytics - 1
Standard Learning Tasks
Goal: obtain a good approximation of an unknown function
Y = f (X1 , X2 , · · · , Xp )Maps a set of p predictors into a target variable Y , which may benumeric (regression) or nominal (classification)
Use a training set D = {hxi , yi i}ni=1 to obtain the approximation, i.e.
the model, h(X1 , X2 , · · · , Xp )(Torgo et. al.)Utility-based Regression

May, 20182 / 34

Predictive Analytics - 1
Standard Learning Tasks
Goal: obtain a good approximation of an unknown function
Y = f (X1 , X2 , · · · , Xp )Maps a set of p predictors into a target variable Y , which may benumeric (regression) or nominal (classification)
Use a training set D = {hxi , yi i}ni=1 to obtain the approximation, i.e.
the model, h(X1 , X2 , · · · , Xp )Model is obtained using some preference criterion
On standard learning tasks this is usually some form of minimizationof the average prediction error(Torgo et. al.)Utility-based Regression

May, 20182 / 34

Predictive Analytics - 2

Utility-based Learning Tasks
End-user has a non-uniform preference over the target variable domain(Torgo et. al.)Utility-based Regression

May, 20183 / 34

Predictive Analytics - 2

Utility-based Learning Tasks
End-user has a non-uniform preference over the target variable domain
Some errors may be more costly than others
Some accurate predictions may bring more rewards than others(Torgo et. al.)Utility-based Regression

May, 20183 / 34

Challenges of Utility-based Learning

1

How to express user preferences?

2Standard evaluation metrics may be misleading3

How to bias learning algorithms to the cases the end user regards as
more relevant?

(Torgo et. al.)Utility-based Regression

May, 20184 / 34

UBL for Classification

This is a well studied problem for classification tasks (nominal target)
Adequate metrics have been proposed (e.g. precision/recall, etc.)
Learning methods or resampling strategies were developed to optimize
these metrics(Torgo et. al.)Utility-based Regression

May, 2018

5 / 34

UBL for Regression

Problem not so thoroughly studied for regression (numeric target)

Many relevant applications
Forecasting extreme values of some ecological parameters
Forecasting stock market returns
etc.

(Torgo et. al.)Utility-based Regression

May, 2018

6 / 34

An Example
Forecasting Stock Market Returns

Very high or low returns (% variations of prices) are interesting
Near-zero returns are very common but uninteresting for traders unable to cover transaction costs
Examples:Forecasting a future return of 3% and then it happens -5% is a very
bad error!
I Forecasting a return of 3% and then it happens 11% has the sameerror amplitude but it is not a serious error
I Forecasting 0.2% for a true value of 0.4% is reasonably accurate butirrelevant!
I Forecasting -7.5% for a true value of -8% is a good an useful prediction
I(Torgo et. al.)Utility-based Regression

May, 20187 / 34

UBL and Imbalanced Domains
A Side Note

Frequently, UBL problems are associated with Imbalanced Domains /Datasets
Actually, imbalanced domains are special cases of UBL
Properties of Imbalanced Domains:II

Non-uniform preferences (like UBL)
More important cases are rare (not a necessary condition in UBL)

More info at:
P. Branco, L. Torgo and R. Ribeiro. “A Survey of Predictive Modeling on
Imbalanced Domains”. In: ACM Comput. Surv. 49.2-31 (2016).

(Torgo et. al.)Utility-based Regression

May, 20188 / 34

Specifying User Preferences(Torgo et. al.)Utility-based Regression

May, 2018

9 / 34

UBL - basics
UBL is based on domain knowledge - what is relevant
How to express this information is a key stepII

Different types of information may be available (or missing!)
Quality and completeness of domain information may vary

For classification we typically require a cost/benefit matrix
What about regression? Infinite domain of the target...
Thus we need a Utility Surface!
A Utility Surface is a function that maps the prediction of value ŷ forthe true value y into a utility score, U(ŷ , y )I As in classification we assume positive values for correct predictions
and negative values for errors
I Not so obvious what is an error though...
I(Torgo et. al.)Utility-based Regression

May, 201810 / 34

Utility Surfaces - 1

Asking the user to fully specify a Utility Surface is not reasonable!
Two options remain:1

Use interpolation methods
FF

2

GeneralRequires the user intervention by providing some points of the surface

Use the automatic method proposed by Ribeiro (2011)
F
F

Only applicable to a special class of UBL problems
Does not require user intervention

R. Ribeiro (2011). “Utility-based Regression”. PhD on Computer Science, Univ. Porto.

(Torgo et. al.)Utility-based Regression

May, 201811 / 34Utility Surfaces through Interpolation
splinesUφ

Utility Surface
50Function Isometrics
1.01.00.8

40−0.50.5.6−010.0(Y^ , Y)30

splinesUφ0.4−0.8Y−0.5

0.0
0.6

−0.2

−1.050

0.4

2050

40

40

−0.50.2030

30020

1020Y10^Y

10−1.010−1.0−0.5

0.00.51.02030

4050^
Y

User specifies a few (important) points of the surface
Some interpolation method is used to obtain the surface
Method available in the R package UBL (Branco et al., 2016)
P. Branco, R. Ribeiro, L. Torgo (2016). “UBL: an R package for Utility-based
Learning”. arXiv:1604.08079 [cs.MS].
(Torgo et. al.)Utility-based Regression

May, 201812 / 34

Utility Surfaces through Automatic Derivation - 1
Ribeiro (2011) proposed a method to automatically defining Utility
Surfaces for a particular class of UBL regression applications:II

Users’ goals are rare extreme values of the continuous target
This is a special case of imbalanced regression domains

Although a special case, this matches many important applications ofutility-based regression
Branco et al. (2017) have recently shown that a similar approach canalso be applied to multi-class imbalance domains

R. Ribeiro (2011). “Utility-based Regression”. PhD on Computer Science, Univ. Porto.
P. Branco, L. Torgo and R. P. Ribeiro (2017). “Relevance-based Evaluation Metrics forMulti-class Imbalanced Domains”. In: 21th Pacific-Asia Conference, PAKDD’17.
Springer.

(Torgo et. al.)Utility-based Regression

May, 201813 / 34

Utility Surfaces through Automatic Derivation - 2
Ribeiro’s approach is based on the concept of relevance function, φ(),proposed in Torgo and Ribeiro (2007).
This function is used to specify user-preferences.
The function maps the continuous domain of the target into a [0, 1]
scale of importanceφ(Y ) : Y → [0, 1](1)This function can be automatically derived if assuming rare extremes
are the target

L. Torgo and R. Ribeiro (2007). “Utility-based Regression”. In: Proceedings ofthe 11th European Conference on Principles and Practice of Knowledge Discovery
in Databases (PKDD 2007). Springer.
(Torgo et. al.)Utility-based Regression

May, 201814 / 34Examples

Classification problem

1.0

Regression problem

0.0

0.2

0.4

φ(y)

0.6

0.8


0.2 if c = normal
φ(c) = 0.9 if c = rare1
1if c = rare2

23456y(Torgo et. al.)Utility-based Regression

May, 201815 / 34

Examples of Automatic Derivation of φ()Classification problem

0.36

80
0.8

0.06

1.0

1.0

Regression problem

1●

600.53
●

0.2

0.14

0.4

φ()

0.6

●

frequency
40

0.6
0.4

φ(Y)

0.04
0.02

0.69

0.31
●

0.2

0.08

20

0.0

0.00

0.06
0.13

0.04

0.12●0.0●0Density

0.80.32

10%(Torgo et. al.)46%
Y

Utility-based Regression

235Glass Classes

67May, 201816 / 34

From Relevance Functions to Utility Surfaces - 1

The Relevance Function tells us how important is a given value of thetarget variable to the end-user
How to translate that into the cost or benefit of forecasting ŷ
for a true value of y ?
In standard regression this is normally evaluated by |ŷ − y | or (ŷ − y )2
But these metrics do not take into account the end-user preferences!
Forecasting 10 for a true value of 11.5 will always have the same
“value” as forecasting -0.5 for a true value of 1
I But these may be very different situations for the end user!
I

How to use φ() to decide on the utility of predicting ŷ for y , U(ŷ , y ) ?

(Torgo et. al.)Utility-based Regression

May, 201817 / 34

From Relevance Functions to Utility Surfaces - 2
Torgo and Ribeiro (2007) have proposed to use φ() to define U(ŷ , y )as a net balance between the benefit and the costs of predicting ŷ foryThis definition was based on the proposal of the bi-variate relevancefunction, φp (ŷ , y ), that balances the relevance of both ŷ and y ,φp (ŷ , y ) = (1 − p) · φ(ŷ ) + p · φ(y )p defines the relative importance given to false alarms vs missed
important values (with p = 0.5 giving equal importance)

L. Torgo and R. Ribeiro (2007). “Utility-based Regression”. In: Proceedings of the 11th
European Conference on Principles and Practice of Knowledge Discovery in Databases(PKDD 2007). Springer.
(Torgo et. al.)Utility-based Regression

May, 201818 / 34

From Relevance Functions to Utility Surfaces - 3Ribeiro (2011) has proposed a refined definition of U(ŷ , y ),Uφp (ŷ , y ) = Bφ (ŷ , y ) − Cφp (ŷ , y )= φ(y ) · (1 − ΓB (ŷ , y )) − φp (ŷ , y ) · ΓC (ŷ , y )where ΓB (ŷ , y ) and ΓC (ŷ , y ) are two bounded-loss functions. Both“normalize” the standard regression loss (squared or absolute) that has
domain [0, ∞] to an interval [0, 1]
ΓB (ŷ , y ), ΓC (ŷ , y ) → 0 as ŷ gets near to yUφp (ŷ , y ) has domain [−1, 1]
The maximum of Uφp (ŷ , y ) is φ(y ) when ŷ = yThe minimum Uφp (ŷ , y ) is p · (1 − φ(y )) − 1
R. Ribeiro (2011). “Utility-based Regression”. PhD on Computer Science, Univ. Porto.
(Torgo et. al.)Utility-based Regression

May, 201819 / 34

From Relevance Functions to Utility Surfaces - 4

U0.9
Utility Surface
φU0.9
Utility Isometrics
φ6

1.0
1.0.6

−00.8

5

0.5

0.5−0.4

0.60.4−0.2
0.00.24p ^Uφ (Y , Y)

0Y

−0.5

00.063−1.0655−0.5
43Y2432^
Y2−1.02−1.0−0.5(Torgo et. al.)

0.0

0.5

1.0

Utility-based Regression

3456

^
Y

May, 201820 / 34

Performance Evaluation for UBR(Torgo et. al.)Utility-based Regression

May, 201821 / 34

Performance Evaluation in Utility-based Regression

Having defined a Utility Surface we can use it to evaluate a model,e.g. Mean Utility over a test setN1 XMU =U(ŷi , yi )Ni=1(Torgo et. al.)Utility-based Regression

May, 201822 / 34

Performance Evaluation for Imbalanced Regression
For forecasting rare extreme values, the concepts of Precision and Recall
were also adapted to regression (Torgo and Ribeiro, 2009; Branco, 2014),Pφ(ŷ )>t (1 + U(ŷi , yi ))prec φ = P i Rφ(ŷi )>tR (1 + φ(ŷi ))Pφ(y )>t (1 + U(ŷi , yi ))rec φ = P i Rφ(yi )>tR (1 + φ(yi ))L. Torgo and R. P. Ribeiro (2009). “Precision and Recall for Regression”. In: Discovery
Science’2009. Springer.
P. Branco (2014). “Re-sampling Approaches for Regression Tasks under ImbalancedDomains”. MSc on Computer Science, Univ. Porto.

(Torgo et. al.)Utility-based Regression

May, 201823 / 34

Learning Models that Maximize Utility(Torgo et. al.)Utility-based Regression

May, 201824 / 34

Learning by Optimizing Utility

Formalization
For a given test case q = hxk , yk i, with yk unknown, the optimal
prediction y ∗ can be determined as follows:Zy ∗ = argmax fY |X (y |X = xk ) · U(y , z) dy(2)z∈Y

This equation is an extension of the concept of miminization ofconditional risk to a utility-based regression setting
Having defined U() we only need an estimate of the conditional
density probability function, fY |X (y |X = xk )(Torgo et. al.)Utility-based Regression

May, 201825 / 34

Approximating the Conditional Density Probability
Function - 1
Branco et al. (2017) proposed an approach to approximate
fY |X (y |X = xk )The approach is based on using simpler class density probability
function estimators
Specifically, a method proposed by Frank and Bouckaert (2009) basedon ordinal classification
P. Branco, L. Torgo, R. Ribeiro, E. Frank, B. Pfahringer and M. Rau (2017). “LearningThrough Utility Optimization in Regression Tasks”. In: IEEE International Conferenceon Data Science and Advanced Analytics (DSAA’17).
E. Frank and R. Bouckaert (2009). “Conditional density estimation withclass probabilityestimators”. in Asian Conference on Machine Learning. Springer.

(Torgo et. al.)Utility-based Regression

May, 201826 / 34

Approximating the Conditional Density Probability
Function - 2
Assume we have a class probability estimator p̂ that given a test caseq produces an estimate of the class probability p̂(c|q).
These estimated probabilities are used as weights of each value of thetarget conditioned on q
wi (q) =p̂(cyi |q)
ncyi

(3)

Finally, a weighted kernel Gaussian estimator is used to approximate
fY |X (y |X = xk ),fY |X (y |X = xk ) =nX

wi (q)N (y ; yi , σ 2 )

(4)

i=1

(Torgo et. al.)Utility-based Regression

May, 201827 / 34

Approximating the Conditional Density Probability
Function - 3

The class probabilities p̂(c|q) are obtained by using a classification
methodThis method is trained on a discretized version of the training set
The continuous target variable is discretized into a set of equal width,non-overlapping bins

(Torgo et. al.)Utility-based Regression

May, 201828 / 34

0.15

−0.5

−0.05

0.00

0.0

0.05

fY|X

0.5

0.10

max fY|X
max utility
true value

Integral values (cf. Eq. 7)

1.0

An Example of the Adjustments

2345

LNO2(Torgo et. al.)Utility-based Regression

May, 201829 / 34

Results

Overall, the utility optimization strategy achieved interesting results,
particularly when using SVMs as learners

56

RFSVM

6455

42

1640

00Figure: Wins (left) and losses (right) of the utility optimization strategy against
the baseline for a utility surface with p=0.5.

(Torgo et. al.)Utility-based Regression

May, 201830 / 34

Summary and Challenges(Torgo et. al.)Utility-based Regression

May, 201831 / 34

SummaryUBR is not studied enough in ML and DM communities
Many relevant applications exist
Discretizing the target variable and handle the problem as autility-based classification task is not effective!
Several solutions have already been proposed for the main challenges
Specification of user preferences
Evaluation according to these preferences
I Obtaining models that optimize these preferences
II(Torgo et. al.)Utility-based Regression

May, 201832 / 34

Challenges

Making the ML and DM communities aware of these problems
Improve current solutions
Can we obtain utility surfaces in a easier way?
Can we provide new evaluation metrics?
I What about graphical tools for evaluation?
I New efforts are needed for learning regression models that optimize
utilityII

Better ways of approximating the conditional density probability
function
F Incorporating the utility function in the learning algorithm
FI

Provide new solutions and/or guidelines for the related problem ofimbalance regression(Torgo et. al.)Utility-based Regression

May, 201833 / 34

Utility-based Regression

Luis Torgo, Paula Branco and Rita Ribeiro
Dalhousie Univ., Canada
INESC Tec, Univ. Porto, Portugal

May, 2018(Torgo et. al.)Utility-based Regression

May, 201834 / 34