Intelligent

Data

Analysis

Elsevier

Science

Inc.

All

rights

reserved

Go

to

IDA

Home

Search

Table

of

Issues

Table

of

Contents

References

Decision

Combination

Based

on

the

Characterisation

of

Predictive

Accuracy

Kai

Ming

Ting

Manuscript

received

February

reviewed

April

accepted

April

Abstract

In

this

article

we

first

explore

an

intrinsic

problem

that

exists

in

the

models

induced

by

learning

algorithms

Regardless

of

the

selected

algorithm

search

methodology

and

hypothesis

representation

by

which

the

model

is

induced

one

would

expect

the

model

to

make

better

predictions

in

some

regions

of

the

description

space

than

others

We

present

the

fact

that

an

induced

model

will

have

some

regions

of

relatively

poor

performance

the

problem

of

locally

low

predictive

accuracy

Holte

Arker

Porter

-LSB-

-RSB-

addressed

this

intrinsic

problem

in

learning

systems

that

describe

the

induced

model

as

a

disjunction

of

conjunctions

of

conditions

In

this

article

we

investigate

the

characterisation

of

the

problem

in

instance-based

and

Naive

Bayesian

classifiers

Having

characterised

the

problem

of

locally

low

predictive

accuracy

we

propose

to

counter

the

problem

in

these

two

types

of

learning

algorithms

using

a

composite

learner

framework

The

strategy

is

to

select

an

estimated

better

performing

model

to

do

the

final

prediction

during

classification

Empirical

results

from

fifteen

real-world

domains

show

that

the

strategy

is

capable

of

partially

overcoming

the

problem

of

locally

low

predictive

accuracy

and

at

the

same

time

improving

the

overall

performance

of

its

constituent

algorithms

in

most

of

the

domains

studied

The

composite

learner

is

also

found

to

outperform

four

methods

of

stacked

generalisation

and

also

a

model

selection

method

based

on

cross-validation

in

most

of

the

experimental

domains

studied

-LRB-

Intelligent

Data

Analysis

Vol

No

-RRB-

Elsevier

Science

Inc.

INTRODUCTION

AND

MOTIVATION

One

important

measure

of

performance

used

in

the

machine

learning

community

for

learning

algorithms

is

estimated

accuracy

on

unseen

cases

This

measure

indicates

the

estimated

overall

performance

of

a

learned

model

over

the

whole

population

from

which

the

training

dataset

used

to

induce

the

model

is

generated

However

no

one

would

assume

that

every

prediction

made

by

a

model

would

have

the

same

probability

of

being

correct

-LRB-

i.e.

proportional

to

its

overall

estimated

accuracy

-RRB-

Regardless

of

the

selected

algorithm

search

methodology

and

hypothesis

representation

by

which

the

model

is

induced

one

would

expect

the

model

to

make

better

predictions

in

some

regions

of

the

description

space

than

others

This

is

a

characteristic

as

well

as

an

intrinsic

problem

in

learning

systems

Holte

Acker

and

Porter

-LSB-

-RSB-

were

the

first

to

address

this

problem

in

learning

systems

that

describe

the

induced

model

as

a

disjunction

of

conjunctions

of

conditions

The

coverage

of

each

rule

-LRB-

or

disjunct

-RRB-

to

characterise

this

intrinsic

problem

is

used

Holter

Acker

and

Porter

have

observed

that

rules

that

cover

a

small

number

of

instances

often

entail

high

error

rates

thus

they

name

the

phenomenon

the

problem

of

small

disjuncts

Several

other

refined

definitions

of

small

disjuncts

have

been

proposed

by

Ting

-LSB-

-RSB-

and

have

been

found

to

work

well

in

overcoming

relatively

poor

performance

regions

in

decision

trees

by

replacing

small

disjuncts

with

instance-based

methods

We

would

anticipate

the

existence

of

a

similar

problem

in

instance-based

learning

-LRB-

IBL

-RRB-

algorithms

-LSB-

-RSB-

and

Naive

Bayesian

classifiers

or

Naive

Bayes

-LSB-

-RSB-

Because

both

of

these

algorithms

employ

different

representations

from

decision

trees

and

rules

measures

other

than

disjuncts

are

needed

to

characterise

this

intrinsic

problem

in

these

representations

This

consideration

has

prompted

us

to

rename

the

problem

so

that

it

can

be

brought

to

a

more

general

perspective

We

term

the

fact

that

an

induced

model

would

have

some

regions

of

relatively

poor

performance

the

problem

of

locally

low

predictive

accuracy

Research

in

machine

learning

has

concentrated

on

the

hypothesis-space

bias

and

the

preference

bias

-LSB-

-RSB-

that

aims

to

improve

the

estimated

accuracy

of

learning

algorithms

Very

little

research

has

been

done

to

gain

insight

into

this

problem

that

is

intrinsic

to

all

learning

algorithms

The

difficulties

in

this

research

are

as

follows

-LRB-

i

-RRB-

how

to

characterise

the

various

regions

of

differing

predictive

accuracies

and

-LRB-

ii

-RRB-

how

to

estimate

predictive

accuracy

in

these

regions

The

general

strategy

of

our

research

is

to

probe

into

the

underlying

characteristics

of

the

learning

methods

to

look

for

an

answer

We

use

the

term

the

characterisation

of

predictive

accuracy

-LRB-

hereafter

shortened

to

the

characterisation

-RRB-

to

mean

the

use

of

a

measure

in

an

induced

model

as

an

indicator

for

its

predictive

accuracy

To

address

the

second

difficult

we

estimate

the

predictive

accuracy

of

the

characterisation

using

a

cross-validation

method

Once

we

can

characterise

predictive

accuracy

of

each

prediction

of

the

induced

models

we

can

overcome

the

problem

of

locally

low

predictive

accuracy

by

replacing

low

performance

regions

in

one

model

-LRB-

from

one

algorithm

-RRB-

with

those

of

another

model

-LRB-

from

another

algorithm

-RRB-

which

has

higher

predictive

accuracy

The

proposed

decision

combination

approach

dynamically

selects

a

learning

algorithm

in

each

prediction

during

classification

This

approach

is

different

from

most

of

the

current

work

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

in

that

explicit

characterisation

of

predictive

accuracy

is

used

to

guide

decision

combination

We

will

describe

how

predictive

accuracy

of

each

prediction

in

a

IBL

algorithm

and

a

Naive

Bayesian

classifier

can

be

characterised

in

Section

Then

in

Section

we

will

show

how

the

characterisation

can

be

used

to

overcome

the

problem

of

locally

low

predictive

accuracy

by

working

cooperatively

between

the

IBL

algorithm

and

the

Naive

Bayes

in

a

composite

learner

framework

and

conduct

a

comparative

study

with

four

methods

of

stack

generalisation

-LSB-

-RSB-

in

Section

Section

describes

some

related

work

and

Section

summarises

the

findings

CHARACTERISING

PREDICTIVE

ACCURACY

IN

IBL

AND

NAIVE

BAYES

Selecting

a

Characterisation

For

each

of

IBL

and

Naive

Bayes

we

choose

an

intuitive

method

of

characterising

the

accuracy

of

each

prediction

in

this

section

The

first

algorithm

a

IBL

algorithm

uses

typicality

This

characterisation

has

its

root

in

cognitive

psychology

-LSB-

-RSB-

In

our

setting

the

typicality

of

an

instance

-LRB-

Y

-RRB-

is

defined

as

-LSB-

-RSB-

where

the

inter-concept

distance

of

an

instance

-LRB-

Y

-RRB-

is

defined

as

its

average

-LRB-

Euclidean

-RRB-

distance

to

instances

of

different

classes

-LRB-

n

-RRB-

and

the

intra-concept

distance

of

an

instance

is

defined

as

its

average

distance

to

other

instances

of

the

same

class

-LRB-

p

-RRB-

Thus

a

typical

instance

would

have

a

high

value

of

the

typicality

measure

and

vice

versa

In

the

framework

of

IBL

we

would

expect

a

prediction

made

with

a

typical

nearest

neighbour

is

more

likely

to

be

correct

than

one

made

with

an

atypical

nearest

neighbour

The

second

algorithm

Naive

Bayes

uses

posterior

probability

predictions

made

with

high

posterior

probability

would

be

more

likely

to

be

correct

than

those

made

with

low

posterior

probability

This

characterisation

agrees

with

the

Bayesian

approach

in

decisionmaking

In

the

Naive

Bayes

approach

the

likelihoods

of

the

observed

attribute

values

-LRB-

X

xi

xI

-RRB-

are

assumed

to

be

mutually

independent

given

which

the

individual

independent

likelihoods

can

be

re-expressed

to

yield

As

only

the

relative

probabilities

of

the

different

classes

-LRB-

Cj

-RRB-

are

of

interest

the

terms

in

Equation

-LRB-

-RRB-

which

are

independent

of

the

class

can

be

ignored

We

have

just

selected

a

characterisation

for

each

of

IBL

and

Naive

Bayes

classifiers

The

next

step

is

to

estimate

the

predictive

accuracy

of

the

characterisation

Estimating

Predictive

Accuracy

of

the

Characterisation

Each

learning

algorithm

performs

a

w-fold

cross-validation

to

estimate

predictive

accuracy

from

the

characterisations

using

the

training

set

In

a

w-fold

cross-validation

a

dataset

is

partitioned

at

random

into

w

equal-size

subsets

At

ith

fold

the

learning

algorithm

is

trained

using

all

but

the

ith

subset

and

the

ith

subset

is

used

as

the

testing

set

and

this

process

is

repeated

w

times

Thus

each

instance

will

be

tested

only

once

and

a

training

set

of

size

n

will

have

the

cross-validated

testing

results

of

size

n

at

the

end

of

the

cross-validation

Figure

Transforming

individual

cross-validation

test

points

to

a

binned

graph

for

one

predicted

class

For

each

predicted

class

the

individual

test

results

from

the

w-fold

cross-validation

are

then

sorted

according

to

the

values

of

the

characterisation

-LRB-

i.e.

posterior

probability

or

typicality

on

the

x-axis

-RRB-

shown

in

the

left

plot

of

Figure

The

aim

is

to

produce

a

binned

graph

that

relates

the

average

value

of

the

characterisation

to

its

binned

predictive

accuracy

for

each

class

shown

in

the

right

plot

of

Figure

The

transformation

process

from

the

left

plot

to

the

right

plot

goes

as

follows

First

use

a

bin

which

contains

a

fixed

number

of

in

the

left

plot

This

bin

is

transformed

to

a

point

in

the

right

plot

by

averaging

the

values

of

the

characterisation

for

all

instances

in

the

bin

on

the

x-axis

and

converting

the

number

of

correct/incorrect

classifications

into

accuracy

on

the

y-axis

This

process

is

repeated

in

the

style

of

a

moving

window

-LRB-

i.e.

the

next

bin

is

obtained

by

dropping

the

leftmost

instance

and

adding

an

instance

adjacent

to

the

rightmost

instance

of

the

current

-RRB-

When

the

bin

size

is

bigger

than

fifteen

the

end

points

of

the

graph

are

extended

by

reducing

the

bin

size

one

instance

at

a

time

until

the

bin

size

reaches

fifteen

This

is

to

ensure

all

graphs

produced

have

the

same

bin

size

of

fifteen

at

the

end

points

Note

that

we

use

the

binned

graphs

for

classification

rather

than

function

approximation

problems

We

use

ten-fold

cross-validation

to

estimate

the

predictive

accuracy

of

the

characterisation

in

all

of

the

following

experiments

Experiments

The

two

algorithms

and

NB

-LSB-

-RSB-

are

used

to

test

the

above

characterisations

is

a

variant

of

-LSB-

-RSB-

that

incorporates

the

modified

value-difference

metric

-LSB-

-RSB-

and

NB

is

an

implementation

of

the

Naive

Bayes

-LSB-

-RSB-

algorithm

Both

algorithms

include

a

method

-LSB-

-RSB-

for

discretising

continuous-valued

attributes

in

the

preprocessing

This

preprocessing

improves

the

performance

of

the

two

algorithms

-LRB-

and

NB

-RRB-

in

most

of

the

continuous-valued

attribute

domains

studied

by

Ting

-LSB-

-RSB-

We

use

the

nearest

neighbour

for

making

prediction

in

and

the

default

settings

are

as

used

in

in

all

experiments

No

parameter

settings

are

required

for

NB

Specifically

we

conduct

experiments

to

test

a

hypothesis

for

each

of

and

NB

in

this

section

Hypothesis

The

more

typical

a

nearest

neighbour

used

in

a

prediction

the

higher

its

predictive

accuracy

in

Hypothesis

The

higher

the

posterior

probability

of

a

prediction

the

higher

its

predictive

accuracy

in

NB

We

conduct

the

experiments

in

fifteen

well-known

domains

obtained

from

the

UCI

repository

of

machine

learning

databases

-LSB-

-RSB-

They

are

the

breast

cancer

Wisconsin

-LRB-

bcw

-RRB-

pima

diabetes

Cleveland

heart

disease

-LRB-

heart

-RRB-

glass

hypothyroid

hepatitis

automobile

echocardiogram

-LRB-

echo

-RRB-

horse

colic

soybean

splice

junction

and

The

characteristics

of

the

experimental

domains

are

given

in

Table

The

example

binned

graphs

of

two

domains

are

shown

in

Figures

and

which

are

produced

from

one

trial

using

of

the

dataset

in

each

domain

Separate

graphs

are

produced

for

each

class

in

each

domain

for

the

two

methods

of

characterisation

-LRB-

i.e.

typicality

for

and

probability

for

NB

-RRB-

An

initial

examination

of

these

graphs

indicates

that

most

of

them

accord

with

our

hypotheses

Table

Details

of

experimental

domains

Domain

No

Instances

No

Classes

No

Attribute

Type

bcw

diabetes

heart

glass

hypothyroid

C

hepatitis

C

automobile

N

C

echo

C

horse

N

C

soybean

N

splice

B

Binary

N

Nominal

C

Continuous

Figure

Binned

graphs

in

the

diabetes

domain

Figure

Binned

graphs

in

the

hepatitis

domain

We

also

use

a

fairly

robust

-LRB-

sometimes

termed

distribution-free

-RRB-

test

of

significance

i.e.

a

Wilcoxon

ranksum

test

-LSB-

-RSB-

on

the

raw

data

-LRB-

e.g.

the

left

plot

depicted

in

Figure

-RRB-

to

see

whether

the

measure

of

characterisation

is

related

to

trends

of

correct

classification

For

each

individual

predicted

class

we

rank

the

individual

test

points

-LRB-

from

the

ten-fold

cross-validation

-RRB-

by

the

measure

of

characterisation

e.g.

the

lowest

value

of

typicality

-LRB-

or

probability

-RRB-

is

ranked

The

statistic

Ws

is

the

sum

of

ranks

of

all

test

points

that

have

correct

classifications

The

distribution

of

Ws

approximates

a

normal

distribution

We

can

then

calculate

z

from

the

mean

and

the

standard

error

of

a

normal

distribution

where

z

-LRB-

W

mean

-RRB-

-LRB-

standard

error

-RRB-

We

consider

a

significance

level

of

-LRB-

z

-RRB-

with

a

one-tailed

test

-LRB-

Ws-mean

-RRB-

-LRB-

standard

error

-RRB-

We

consider

a

significance

level

of

-LRB-

z

-RRB-

with

a

one-tailed

test

The

results

for

the

test

are

tabulated

in

Table

A

positive

value

for

z

indicates

that

the

classifications

are

more

likely

to

be

correct

with

higher

values

of

typicality/probability

A

negative

value

for

z

indicates

a

reverse

trend

Table

shows

the

results

of

the

test

The

insignificant

trends

are

marked

with

Table

The

results

of

a

Wilcoxon

ranksum

test

for

trends

of

correct

classification

with

respect

to

the

measure

of

characterisation

depicted

in

Figure

Predicted

Class

Typicality

Probability

Domain

z

z

bcw

diabetes

heart

glass

hypothyroid

hepatitis

automobile

echo

horse

soybean

splice

number

of

correct/incorrect

classifications

indicates

z

-LRB-

i.e.

an

insignificant

trend

-RRB-

Note

that

this

statistic

can

be

computed

even

though

a

class

has

a

number

of

instances

less

than

the

minimum

bin

size

-LRB-

i.e.

fifteen

-RRB-

The

numbers

of

correct

and

incorrect

classifications

for

each

predicted

class

are

shown

in

columns

labeled

and

respectively

These

figures

show

the

support

each

class

acquired

for

the

z

statistic

For

each

hypothesis

we

summarise

the

findings

as

follows

For

Hypothesis

all

classes

in

eight

domains

provide

consistent

evidence

to

the

hypothesis

that

the

prediction

made

with

a

typical

nearest

neighbour

-LRB-

i.e.

with

high

value

of

typicality

-RRB-

is

more

likely

to

be

correct

However

a

reverse

trend

is

observed

in

one

class

of

the

automobile

domain

Insignificant

trends

appear

in

one

class

in

each

of

the

and

echocardiogram

domains

and

in

three

classes

in

each

of

the

glass

automobile

soybean

and

domains

and

in

four

classes

in

the

domain

For

Hypothesis

all

classes

in

ten

domains

are

found

to

conform

to

the

hypothesis

that

the

prediction

made

with

high

posterior

probability

is

more

likely

to

be

correct

However

the

trend

appeared

to

be

the

opposite

in

one

class

in

the

domain

Insignificant

trends

appear

in

one

class

in

each

of

the

hepatitis

and

echocardiogram

domains

and

in

three

four

and

five

classes

in

the

automobile

and

domains

respectively

The

overall

result

is

that

most

classes

in

the

fifteen

domains

provide

consistent

evidence

to

the

hypotheses

The

typicality

and

the

posterior

probability

have

been

shown

to

be

a

satisfactory

characterisation

of

predictive

accuracy

in

and

NB

respectively

Discussion

There

are

a

number

of

reasons

for

these

methods

of

characterisation

to

show

insignificant

trends

in

some

classes

in

some

domains

First

when

the

underlying

concept

is

hard

to

learn

for

these

learning

algorithms

we

do

not

expect

any

methods

of

characterisation

to

provide

a

good

accuracy

predictor

To

describe

it

in

another

way

the

models

or

assumptions

of

these

learning

algorithms

could

be

all

wrong

in

these

cases

An

example

is

the

parity

concept

where

both

of

these

algorithms

are

known

to

have

difficulty

learning

it

The

second

reason

could

be

attributed

to

the

nature

of

the

characterisation

itself

Typicality

measures

the

global

property

of

instances

of

a

category

In

some

cases

local

properties

in

the

description

space

might

also

be

important

in

classification

exceptions

or

rare

cases

and

the

parity

problem

are

examples

where

typicality

would

fail

to

characterise

The

use

of

an

exponential

distance

function

rather

than

the

Euclidean

distance

function

in

Equation

-LRB-

-RRB-

may

improve

the

characterisation

Current

computation

of

typicality

does

not

take

into

account

the

relative

importance

of

attributes

Some

attribute

weighting

methods

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

may

be

employed

to

improve

this

characterisation

Another

important

reason

is

the

estimation

method

used

to

generate

the

graphs

Both

the

bin

size

and

the

moving

window

method

have

an

effect

on

the

shape

of

the

graphs

Sparse

data

in

some

classes

in

some

domains

further

compounds

the

problem

In

the

examples

shown

in

Table

a

number

of

classes

have

only

one

bin

because

they

are

less

than

the

minimum

bin

-LRB-

i.e.

fifteen

-RRB-

This

requires

an

extrapolation

in

order

to

use

the

bin

graphs

which

will

certainly

affect

the

accuracy

of

the

estimation

Note

that

using

these

methods

of

characterisation

one

does

not

explicitly

know

the

exact

regions

in

the

description

space

where

the

predictive

accuracy

is

high

or

low

as

in

the

case

of

small

disjuncts

in

decision

trees

or

rules

But

this

is

the

result

of

the

underlying

representation

rather

than

the

methods

of

characterisation

The

decision

boundaries

of

and

Naive

Bayes

are

intrinsically

implicit

in

their

representations

One

does

not

know

the

exact

decision

boundaries

unless

on

takes

extra

effort

to

draw

them

out

-LRB-

e.g.

into

a

Voronoi

diagram

-LSB-

-RSB-

for

-RRB-

The

characterisation

and

estimation

of

predictive

accuracy

described

here

can

be

viewed

as

a

means

of

calibrating

an

algorithm

's

estimate

-LRB-

i.e.

typicality

or

posterior

probability

-RRB-

with

its

-LRB-

estimate

-RRB-

predicted

accuracy

Cross-validation

is

used

here

as

a

tool

in

performing

calibration

In

statistics

the

idea

of

calibration

is

presented

as

a

way

of

realigning

ones

estimates

with

reality

or

some

standard

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

Statisticians

tend

to

use

a

linear

model

for

calibration

In

the

current

work

one

may

want

to

fit

a

model

for

the

characterisation

which

may

or

may

not

be

a

linear

model

For

instance-based

classifiers

there

are

other

choices

with

regard

to

the

method

of

characterisation

For

example

one

may

intend

to

use

some

method

of

kernel

density

estimation

-LSB-

-RSB-

which

may

solve

the

problem

faced

by

using

typicality

mentioned

above

We

have

attempted

another

method

of

characterisation

that

uses

the

nearest

neighbour

distance

The

intuition

is

that

one

prediction

made

with

a

closer

nearest

neighbour

would

be

expected

to

have

higher

predictive

accuracy

than

another

prediction

made

with

a

farther

nearest

neighbour

Mandler

and

Sch?rmann

-LSB-

-RSB-

employ

a

similar

approach

in

instance-based

classifiers

However

this

method

does

not

perform

well

for

a

number

of

reasons

In

domains

that

have

multi-modal

overlapping

distributions

-LRB-

i.e.

the

data

has

maximal

density

in

the

regions

of

decision

boundaries

and

sparse

density

elsewhere

-RRB-

the

nearest

neighbour

distance

would

produce

graphs

which

are

completely

opposite

to

the

hypothesis

The

test

instances

that

are

far

from

the

boundaries

are

more

likely

to

have

larger

nearest

neighbour

distances

than

those

near

to

the

boundaries

-LRB-

due

to

the

data

distribution

-RRB-

thus

they

are

more

likely

to

predict

correctly

Noise

is

another

factor

that

would

obscure

the

nearest

neighbour

distance

as

a

good

accuracy

predictor

in

Apart

from

this

we

have

also

experimented

with

several

other

coarse

methods

of

characterisation

-LRB-

e.g.

using

predicted

class

as

a

characterisation

and

they

have

failed

to

provide

satisfactory

results

-RRB-

While

the

methods

of

characterisation

and

estimation

are

not

perfect

for

one

reason

or

another

it

suffices

for

our

purpose

here

to

show

that

the

methods

can

be

used

to

characterise

and

estimate

predictive

accuracy

in

most

of

the

real-world

domains

studied

The

real

test

is

how

effective

are

these

methods

in

overcoming

the

problem

of

locally

low

predictive

accuracy

in

the

two

learning

algorithms

which

is

the

topic

of

the

next

section

THE

METHOD

OF

COMBINING

AND

NB

Having

characterised

and

estimated

predictive

accuracy

in

and

NB

we

can

now

make

use

of

their

individual

predictive

accuracy

estimation

for

each

classification

for

the

purpose

of

decision

combination

The

combining

strategy

that

we

use

is

to

select

an

estimated

better-performing

classifier

to

do

the

final

prediction

and

we

name

the

resultant

classifier

the

composite

learner

-LRB-

CL

-RRB-

Because

only

high

performance

predictions

are

selected

the

CL

's

performance

is

expected

to

be

better

than

the

individual

algorithms

Employing

the

characterisations

and

combining

the

two

algorithms

using

this

strategy

is

straight-forward

Both

and

NB

are

first

trained

independently

During

training

the

algorithms

perform

a

ten-fold

cross-validation

to

estimate

predictive

accuracy

of

each

prediction

from

the

characterisations

This

is

done

by

using

the

test

results

from

the

ten-fold

cross-validation

to

produce

a

binned

graph

that

relates

the

average

value

of

the

characterisation

to

its

binned

predictive

accuracy

-LRB-

as

described

in

the

last

section

-RRB-

At

the

end

of

the

training

process

in

each

learning

algorithm

a

model

induced

from

all

the

training

instances

and

the

binned

graphs

for

all

classes

are

stored

for

future

classification

Figure

Classificaton

process

of

the

composite

learner

To

classify

an

instance

X

it

is

input

into

both

models

and

the

model

which

has

the

higher

estimate

predictive

accuracy

for

its

output

is

chosen

to

make

the

final

prediction

This

process

is

shown

in

Figure

The

predictive

accuracy

PA

-LRB-

C

H

X

-RRB-

is

obtained

by

referring

to

the

predicted

class

-LRB-

C

-RRB-

binned

graph

with

the

corresponding

value

of

the

characterisation

-LRB-

H

-RRB-

Formally

the

selection

process

can

be

defined

as

follows

-LRB-

-RRB-

where

C

f

final

prediction

CM

HM

Model

M

's

prediction

characterisation

PA

-LRB-

C

H

X

-RRB-

predictive

accuracy

of

C

and

H

given

instance

X

Performance

of

the

Composite

Learner

We

conduct

experiments

in

the

same

fifteen

real-world

domains

as

in

Section

to

compare

the

performance

of

the

composite

learner

with

those

of

its

constituent

algorithms

and

a

model

selection

method

based

on

cross-validation

CL

combines

which

uses

typicality

and

NB

which

uses

posterior

probability

We

randomly

split

the

dataset

into

and

disjoint

subsets

Each

experimental

trial

uses

the

subset

as

a

training

set

and

the

subset

as

a

testing

set

and

it

is

repeated

over

trials

In

each

trial

a

ten-fold

cross-validation

is

performed

for

CL

using

the

training

data

in

each

learning

algorithm

as

described

in

Section

to

estimate

predictive

accuracy

from

the

characterisation

We

use

the

same

ten-fold

cross-validation

for

the

model

selection

method

This

experimental

method

is

used

throughout

the

rest

of

this

article

Table

Average

error

rates

of

CL

and

The

domains

are

ordered

by

the

absolute

error

r

Domain

NB

CL

horse

hypothyroid

heart

splice

bcw

soybean

glass

diabetes

hepatitis

echo

automobile

Table

Summary

of

Table

Comparison

to

the

better

of

and

NB

CL

No

wins

vs.

no

losses

No

signif

wins

vs.

no

signif

losses

Sign

test

Table

shows

the

average

classification

error

rates

and

their

standard

errors

of

NB

CL

which

uses

the

test

results

of

cross-validation

-LSB-

-RSB-

on

the

training

set

to

select

the

better

of

and

NB

and

the

best

algorithm

selected

among

NB

and

CL

based

on

the

test

results

of

cross-validation

on

training

data

The

best

result

in

each

domain

is

shown

in

boldface

The

-LRB-

or

-RRB-

symbol

in

front

of

a

figure

indicates

that

it

is

significantly

better

-LRB-

or

worse

-RRB-

than

the

better

of

and

NB

by

more

than

or

equal

to

two

standard

errors

-LRB-

confidence

-RRB-

The

domains

are

ordered

by

the

absolute

error

rate

difference

between

and

NB

This

ordering

divides

these

domains

into

two

groups

one

where

CL

performs

better

than

the

better

of

and

NB

and

one

where

CL

performs

worse

A

Wilcoxon

rank-sum

one-tailed

test

indicates

that

the

ordering

is

strongly

related

to

the

performance

of

CL

with

confidence

In

eleven

domains

where

the

performance

of

and

NB

have

small

differences

CL

achieves

better

results

than

the

better

of

its

constituent

algorithms

In

these

domains

it

is

better

to

use

CL

rather

than

In

the

other

four

domains

where

the

performance

of

and

NB

differ

substantially

CL

performs

in

between

the

performance

of

its

constituent

algorithms

but

never

performs

better

than

the

better

of

the

two

algorithms

performs

comparably

to

the

better

of

and

NB

in

all

domains

While

this

is

a

strong

point

of

the

method

it

also

demonstrates

that

this

method

can

not

produce

better

results

than

the

better

component

algorithm

behaves

more

conservative

than

CL

It

manages

to

overcome

the

weakness

of

CL

in

domains

where

CL

performs

badly

and

performs

slightly

worse

than

CL

in

domains

where

CL

performs

very

well

A

summary

of

these

results

is

given

in

Table

The

first

row

shows

the

number

of

domains

in

which

CL

or

achieves

higher

accuracy

than

the

better

of

and

NB

versus

the

number

in

which

the

reverse

occurred

The

second

row

considers

only

those

domains

in

which

the

difference

is

significant

with

at

least

confidence

The

third

row

shows

the

results

of

applying

a

sign

test

to

the

values

in

the

second

row

This

results

in

a

confidence

more

than

that

either

CL

or

is

a

more

accurate

learner

than

the

better

of

and

NB

has

no

significant

better

or

worse

results

in

comparison

to

the

better

of

and

NB

Ablation

Analysis

To

gain

insight

into

the

factors

that

affect

the

performance

of

the

composite

learner

we

first

examine

the

components

of

classifications

made

by

its

constituent

algorithms

The

classification

overlaps

consists

of

K

number

of

correct

classifications

and

number

of

incorrect

classifications

by

both

algorithms

Let

L

be

the

number

of

correct

classifications

made

by

only

one

of

them

and

LIB

and

LNB

the

portions

of

and

only

NB

making

correct

classifications

respectively

Then

the

total

number

of

classifications

T

consists

of

K

and

L

shown

in

Equation

-LRB-

-RRB-

and

the

components

of

L

is

shown

in

Equation

-LRB-

-RRB-

Now

let

us

consider

the

corresponding

components

of

classifications

made

by

the

composite

learner

-LRB-

i.e.

KCL

CL

and

LCL

-RRB-

The

composite

learner

improves

performance

if

it

makes

the

correct

selection

most

of

the

time

when

only

one

of

its

constituent

algorithms

makes

the

correct

classification

thus

LCL

L

The

composite

learner

can

do

nothing

when

both

algorithms

make

incorrect

or

correct

classifications

Since

the

composite

learner

always

makes

incorrect

classifications

in

component

whichever

algorithm

is

selected

CL

must

be

zero

The

opposite

applies

to

KCL

which

must

equal

to

K

Thus

the

total

number

of

correct

classifications

made

by

the

composite

learner

TCL

consists

of

LCL

and

KCL

shown

in

Equation

-LRB-

-RRB-

Table

Average

percentage

of

classification

overlap

and

correct

classification

Domain

Overlap

-LRB-

No

test

Inst

-RRB-

Only

Correct

NB

Only

Correct

CL

Correct

-LRB-

-RRB-

horse

-LRB-

-RRB-

hypothyroid

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

heart

-LRB-

-RRB-

splice

-LRB-

-RRB-

bcw

-LRB-

-RRB-

soybean

-LRB-

-RRB-

glass

-LRB-

-RRB-

diabetes

-LRB-

-RRB-

-LRB-

-RRB-

hepatitis

-LRB-

-RRB-

echo

-LRB-

-RRB-

automobile

-LRB-

-RRB-

Boldface

indicates

the

maximum

value

in

the

last

three

columns

in

each

domain

In

order

to

show

the

advantage

of

the

composite

learner

produced

out

from

L

the

percentage

of

LIB

LNB

and

LCL

are

computed

using

Equations

-LRB-

-RRB-

-LRB-

-RRB-

The

percentage

of

classification

overlap

is

computed

also

using

Equation

-LRB-

-RRB-

Table

summarises

the

results

in

the

fifteen

domains

All

figures

in

Table

are

calculated

using

Equations

-LRB-

-RRB-

-LRB-

-RRB-

which

correspond

to

the

last

four

columns

in

the

table

Here

we

provide

an

example

how

the

figures

in

Table

are

computed

In

the

diabetes

domain

the

various

figures

obtained

from

the

experiments

are

as

follows

The

overlap

component

consists

of

K

and

and

the

L

component

has

LIB

LNB

and

LCL

The

percentage

of

overlap

is

computed

using

Equation

-LRB-

-RRB-

The

percentage

of

only

correct

classification

-LRB-

LIB

-RRB-

is

computed

using

Equation

-LRB-

-RRB-

The

other

two

figures

in

Table

for

this

domain

can

be

similarly

computed

from

Equations

-LRB-

-RRB-

and

-LRB-

-RRB-

The

results

from

Table

reveal

that

the

degree

of

classification

overlap

does

not

affect

the

performance

of

CL

Summing

the

average

percentage

of

CL

correct

classification

over

fifteen

domains

in

Table

-LRB-

the

last

column

-RRB-

the

mean

is

-LRB-

an

oracle

would

achieve

and

a

random

choice

is

-RRB-

This

indicates

that

CL

can

obtain

better

results

than

either

of

its

constituent

algorithms

in

those

domains

in

which

LIB

or

LNB

is

less

than

Discussion

Because

the

composite

learner

employs

a

strategy

that

probes

into

the

intrinsic

problem

of

the

induced

models

of

its

constituent

algorithms

and

uses

only

those

predictions

that

are

more

likely

to

be

correct

it

is

able

to

improve

the

performance

of

the

individual

algorithms

Using

a

cross-validation

algorithm

such

as

does

not

get

better

performance

than

the

better

component

learning

algorithm

Cross-validation

as

employed

in

uses

global

information

-LRB-

i.e.

the

overall

estimated

error

rate

-RRB-

to

do

model

selection

before

classification

whereas

the

composite

learner

uses

local

information

-LRB-

i.e.

the

characterisation

of

predictive

accuracy

of

each

prediction

-RRB-

to

select

which

model

to

use

during

classification

The

fact

that

CL

achieves

only

correct

on

average

-LRB-

in

the

portion

where

only

one

of

and

NB

has

correct

classification

-RRB-

explains

why

the

current

implementation

of

the

composite

learner

can

only

achieve

better

results

in

domains

where

the

constituent

algorithms

have

comparable

performance

This

indicates

that

there

is

still

room

for

further

improvement

First

the

methods

for

characterisation

and

estimation

of

predictive

accuracy

can

be

improved

Some

of

these

problems

have

been

discussed

in

Section

Current

weaknesses

can

be

mitigated

by

employing

cross-validation

to

select

the

best

algorithm

among

the

composite

learner

and

its

constituent

algorithms

This

method

is

very

effective

in

domains

such

as

hepatitis

and

automobile

where

the

performance

between

and

NB

differs

substantially

Note

that

CL

does

not

perform

well

in

these

two

domains

because

the

characterisations

of

predictive

accuracy

are

poor

-LRB-

indicated

as

insignificant

trends

in

Table

-RRB-

Second

in

two-class

domains

an

improvement

can

be

made

by

simply

reversing

the

prediction

when

the

binned

graph

for

a

class

is

below

accuracy

-LRB-

see

the

bottom

left

portions

of

the

binned

graphs

for

typicality

in

Figures

and

-RRB-

There

are

two

situations

when

one

algorithm

performs

comparably

to

the

other

The

first

situation

is

when

the

decision

boundaries

almost

overlap

in

this

case

the

classification

overlap

would

be

very

high

and

we

say

that

the

algorithms

are

highly

correlated

In

the

second

situation

as

the

classification

overlap

is

relatively

low

the

proportion

classified

correctly

by

one

algorithm

while

the

other

algorithm

is

incorrect

is

about

the

same

for

both

algorithms

Nevertheless

the

composite

learner

is

capable

of

performing

well

under

both

situations

Classification

correlation

between

the

constituent

algorithms

in

some

domains

does

not

affect

the

performance

of

the

composite

learner

This

is

shown

in

the

bcw

and

hypothyroid

domains

where

the

percentage

of

classification

overlap

is

over

-LRB-

see

Table

-RRB-

COMPARISON

WITH

STACKED

GENERALISATION

Four

Methods

of

Stacked

Generalisation

Stacked

generalisation

-LSB-

-RSB-

is

proposed

as

a

more

sophisticated

version

of

cross-validation

-LSB-

-RSB-

that

uses

a

learning

algorithm

to

determine

how

the

outputs

of

-LRB-

primitive

-RRB-

learning

algorithms

or

generaliser

should

be

combined

The

inputs

to

the

primitive

generalisers

and

the

generalisers

themselves

are

referred

to

as

operating

in

the

zero-level

description

space

The

first-level

inputs

are

the

outputs

of

the

zero-level

generalisers

-LRB-

possibly

plus

the

zero-level

inputs

-RRB-

The

learning

algorithms

used

in

this

level

is

called

the

first-level

generalisers

Here

CL

is

compared

to

four

methods

of

stacked

generalisation

a

-RRB-

Use

only

the

predictions

of

and

NB

as

inputs

to

the

first-level

generaliser

SGa

b

-RRB-

Use

the

zero-level

inputs

and

the

predictions

of

and

NB

as

inputs

to

the

first-level

generaliser

SGb

c

-RRB-

Use

the

predictions

and

predictive

accuracies

of

and

NB

as

inputs

to

the

first-level

generaliser

SGc

d

-RRB-

Use

the

same

inputs

of

SGc

plus

a

binary

attribute

indicating

whether

the

predictive

accuracy

of

is

better

than

that

of

NB

as

inputs

to

the

first-level

generaliser

SGd

SGa

and

SGb

are

the

methods

originally

proposed

by

Wolpert

-LSB-

-RSB-

SGc

has

the

same

input

attributes

as

in

CL

which

allows

us

to

assess

the

advantage/disadvantage

of

having

the

additional

generaliser

The

additional

binary

attribute

presented

to

SGd

allows

a

simple

decision

tree

to

be

formed

to

represent

Equation

-LRB-

-RRB-

as

if

PA

-LRB-

CIB

HIB

X

-RRB-

PA

-LRB-

CNB

HNB

X

-RRB-

then

Cf

CIB

else

C

f

CNB

We

use

as

the

first-level

generaliser

for

SGa

and

SGb

and

use

-LRB-

i.e.

without

discretisation

in

the

preprocessing

-RRB-

with

k

nearest

neighbours

for

SGc

and

a

decision

tree

learning

-LSB-

-RSB-

for

SGd

The

computation

requirements

for

these

methods

of

stacked

generalisation

differ

Suppose

we

employ

a

v-fold

cross-validation

for

stacked

generalisation

and

a

w-fold

cross-validation

for

predictive

accuracy

estimation

SGa

and

SGb

require

v-fold

executions

for

each

zero-level

generaliser

to

setup

a

training

set

for

them

Predictive

accuracy

estimation

is

de-activated

as

it

is

not

used

for

both

SGa

and

SGb

However

both

SGc

and

SGd

need

vw-fold

executions

This

is

because

for

each

fold

required

for

stacked

generalisation

a

w-fold

cross-validation

is

needed

for

predictive

accuracy

estimation

in

a

zero-level

generaliser

internally

Both

v

and

w

are

set

to

ten

in

our

experiments

Thus

CL

SGa

and

SGb

have

the

same

computation

requirements

which

are

much

less

than

those

for

SGc

and

SGd

Experiments

Table

shows

the

average

classification

error

rates

of

these

four

methods

of

stacked

generalisation

with

comparison

to

the

composite

learner

The

results

of

are

restated

for

ease

of

comparison

The

-LRB-

-RRB-

symbol

in

front

of

a

figure

indicates

that

it

is

significantly

worse

-LRB-

better

-RRB-

than

CL

-LRB-

i.e.

the

difference

between

CL

and

a

stacked

generalisation

method

is

more

than

or

equal

to

two

standard

errors

-LRB-

confidence

-RRB-

-RRB-

The

best

result

in

each

domain

is

shown

in

boldface

The

domains

are

ordered

as

in

Tables

and

CL

performs

better

than

the

four

methods

of

stacked

generalisation

in

most

domains

and

a

considerable

number

of

the

differences

are

significant

For

example

CL

is

significantly

better

than

SGb

in

thirteen

out

of

fifteen

domains

Only

a

few

domains

do

some

version

of

stacked

generalisation

achieves

minor

improvements

over

CL

but

the

differences

are

not

significant

-LRB-

for

example

in

the

hepatitis

and

automobile

domains

-RRB-

Note

that

these

are

the

domains

which

the

algorithms

have

poor

characterisations

of

predictive

accuracy

-LRB-

i.e.

demonstrate

insignificant

trends

in

Table

-RRB-

Also

note

that

SGd

has

more

input

information

and

both

SGc

and

SGd

require

much

more

computation

than

CL

and

need

a

complex

learning

algorithm

for

decision

combination

yet

they

perform

significantly

worse

than

CL

in

most

of

the

domains

CL

achieves

significantly

better

results

than

in

eight

domains

and

significantly

worse

in

one

domain

Table

Average

error

rates

of

CL

stacked

generalisations

Domain

CL

SGa

SGb

SGc

SGd

horse

hypothyroid

heart

splice

bcw

soybean

glass

diabetes

hepatitis

echo

automobile

significantly

better

or

worse

than

CL

p

SGa

SGb

use

as

the

first

level

generaliser

SGc

used

with

k

and

SGd

uses

Table

Summary

of

Table

Comparison

to

CL

SGa

SGb

SGc

SGd

No

wins

vs.

no

losses

No

signif

wins

vs

no

signif

losses

Sign

test

Table

shows

a

summary

of

these

results

The

first

row

shows

the

number

of

domains

in

which

CL

achieves

higher

accuracy

than

a

stacked

generalisation

method

versus

the

number

in

which

the

reverse

happened

A

similar

comparison

for

only

those

domains

in

which

the

difference

is

significant

is

shown

in

the

second

row

The

third

row

shows

the

results

of

a

sign

test

on

the

values

of

the

second

row

This

reveals

that

CL

is

a

more

accurate

learner

than

the

four

types

of

stacked

generalisation

with

confidence

and

also

with

at

least

confidence

Discussion

There

are

a

number

of

reasons

why

the

composite

learner

performs

better

than

the

stacked

generalisation

Stacked

generalisation

merely

delays

the

decisions

to

be

made

when

one

is

confronted

with

a

learning

problem

The

decisions

regarding

a

learning

problem

such

as

the

number

of

inputs

to

be

used

-LRB-

the

feature

selection

problem

-RRB-

and

what

type

of

learning

algorithm

is

suitable

-LRB-

the

selective

superiority

problem

-LSB-

-RSB-

-RRB-

in

the

first-level

can

not

be

solved

in

stacked

generalisation

they

still

have

to

be

decided

-LRB-

manually

by

users

-RRB-

in

the

new

transformed

space

at

the

higher

level

The

same

problems

faced

in

the

new

transformed

space

do

not

become

simpler

in

fact

they

are

as

difficult

as

those

faced

in

the

original

space

The

poor

results

of

stacked

generalisations

are

due

to

the

fact

that

the

first-level

inputs

-LRB-

i.e.

the

zero-level

outputs

possibly

including

the

characterisations

-RRB-

are

highly

correlated

or

they

are

redundant

with

respect

to

the

primitive

inputs

-LSB-

-RSB-

The

composite

learner

addresses

the

selective

superiority

problem

given

that

instance-based

and

Naive

Bayesian

classifiers

are

chosen

to

solve

the

learning

problem

at

hand

by

using

the

characterisation

of

predictive

accuracy

in

each

classifier

Because

only

the

characterisations

are

used

the

feature

selection

problem

does

not

exist

in

the

composite

learner

framework

The

success

of

the

composite

learner

relies

on

the

accuracy

of

this

characterisation

and

its

predictive

accuracy

estimation

However

we

do

not

claim

that

our

implementation

of

stacked

generalisation

is

the

best

for

the

tasks

Its

successful

implementation

is

still

a

black

art

-LSB-

-RSB-

for

classification

tasks

to

the

best

of

our

knowledge

and

the

conditions

under

which

stacked

generalisation

works

are

still

unknown

except

for

regression

tasks

-LSB-

-RSB-

The

implementations

reported

in

Section

are

only

some

of

the

variations

we

have

tried

Nonetheless

it

shall

be

noted

that

some

versions

of

stacked

generalisation

in

our

experiments

do

outperform

the

better

of

two

zero-level

generalisers

in

some

domains

This

is

in

agreement

with

the

experimental

results

by

others

-LRB-

e.g.

Skalak

-LSB-

-RSB-

-RRB-

The

composite

learner

can

be

viewed

as

an

informed

simple

generaliser

that

makes

its

decision

to

select

a

classifier

based

on

the

weight

of

the

prediction

The

weight

of

the

prediction

can

be

regarded

as

the

classifier

's

confidence

concerning

the

prediction

made

This

simple

generaliser

is

demonstrated

to

be

better

than

more

complex

generalisers

-LRB-

i.e.

SGx

-RRB-

mainly

due

to

the

weight

truly

reflecting

the

classifier

's

confidence

about

the

prediction

made

Learning

algorithms

such

as

and

NB

-LRB-

and

also

decision-tree

or

rule-learning

algorithms

-RRB-

are

known

to

have

difficulty

learning

a

relation

such

as

Equation

if

no

special

treatment

is

made

The

composite

learner

makes

its

decision

solely

based

on

the

prediction

weights

and

it

does

not

generalise

in

a

new

transformed

space

What

it

does

is

a

superposition

of

the

models

learned

from

the

primitive

attributes

Figure

illustrates

how

one

model

is

superpositioned

on

another

model

in

a

one-dimensional

space

domain

The

vertical

axis

in

the

first

two

diagrams

-LRB-

for

and

NB

respectively

-RRB-

indicate

the

estimated

predictive

accuracies

in

each

region

of

the

description

space

-LRB-

plotted

in

the

same

scale

-RRB-

for

simplicity

they

are

assumed

to

be

uniform

in

each

region

P

shows

the

positive

class

regions

and

N

the

negative

class

regions

The

bottom

plot

shows

the

regions

cast

by

the

composite

learner

by

selectively

choosing

the

more

highly

predictive

regions

Figure

Model

superposition

based

on

predictive

accuracy

It

might

be

thought

that

the

task

of

estimating

a

relationship

between

the

characterisation

and

accuracy

-LRB-

i.e.

calibration

would

be

unnecessary

if

the

same

characterisation

could

be

applied

to

the

component

models

especially

these

models

are

derived

from

the

same

learning

algorithms

-RRB-

Ting

and

Low

-LSB-

-RSB-

have

applied

the

composite-learner

framework

in

the

multiple-data-batches

scenario

where

the

component

models

are

trained

using

disjoint

data

batches

from

a

single

learning

algorithm

Our

experiments

in

the

same

scenario

reveal

that

algorithms

with

calibration

give

better

results

than

those

without

calibration

in

most

cases

Sample

results

are

shown

in

the

Appendix

However

there

is

some

exception

where

calibration

can

be

omitted

without

loss

of

accuracy

This

indicates

that

the

cross-validation

used

for

calibration

-LRB-

as

well

as

the

calibration

itself

-RRB-

is

not

a

crucial

step

in

this

framework

as

in

the

case

of

stacked

generalisation

-LSB-

-RSB-

The

proposed

composite

learner

can

be

extended

to

include

other

learning

algorithms

provided

appropriate

characterisations

of

those

induced

models

can

be

found

For

example

the

probabilities

of

a

neural

networks

outputs

could

be

used

as

the

characterisation

In

such

cases

a

scheme

that

combines

the

weights/evidences

of

the

predictions

might

need

to

be

incorporated

into

the

system

-LRB-

such

as

those

in

-LSB-

-RSB-

-RRB-

Exactly

how

this

will

work

is

still

an

open

problem

RELATED

WORK

The

most

closely

related

work

that

exploits

local

information

in

combining

different

learning

algorithms

is

Tings

-LSB-

-RSB-

treatment

of

the

problem

of

small

disjuncts

in

decision

trees

The

key

idea

of

this

work

is

to

use

instance-based

methods

to

solve

the

problem

of

small

disjuncts

in

decision

trees

by

replacing

the

small

disjunct

regions

with

instance-based

methods

which

are

shown

to

have

better

performance

in

these

regions

in

most

domains

Although

a

similar

idea

is

used

here

the

roles

of

decision

trees

and

instance-based

methods

are

fixed

in

that

framework

decision

trees

work

on

large

disjuncts

regions

and

instance-based

methods

work

in

small

disjuncts

regions

This

arrangement

is

based

on

the

assumption

that

decision

trees

perform

better

than

instance-based

methods

in

the

regions

of

large

disjuncts

In

domains

where

this

assumption

does

not

hold

the

performance

of

this

composite

learner

will

be

worse

than

instance-based

methods

KBNGE

-LSB-

-RSB-

in

many

respects

resemble

Tings

work

without

using

an

explicit

characterisation

of

the

problem

of

small

disjuncts

It

uses

BNGE

-LRB-

i.e.

a

nested

generalised

exemplars

algorithm

-LSB-

-RSB-

-RRB-

in

regions

that

clearly

belong

to

one

class

and

a

k-nearest

neighbour

algorithm

otherwise

In

an

independent

work

Merz

-LSB-

-RSB-

describes

a

decision

combination

method

where

selection

of

algorithms

can

be

done

during

classification

The

DS

-LRB-

dynamic

selection

-RRB-

algorithm

builds

two

n

by

m

matrices

during

training

where

n

is

the

number

of

training

instances

and

m

is

the

number

of

learning

algorithms

The

first

matrix

records

the

predictions

of

the

induced

models

and

the

second

matrix

records

the

test

results

of

u

v-fold

cross-validations

for

each

training

instance

where

u

must

be

more

than

All

induced

models

are

used

in

classifying

a

new

instance

to

construct

a

row-vector

that

can

be

compared

in

the

first

matrix

to

identify

the

rows

closest

to

the

row-vector

of

the

new

instance

The

accuracies

for

each

induced

model

are

computed

from

the

corresponding

rows

-LRB-

by

averaging

-RRB-

in

the

second

matrix

The

induced

model

having

the

highest

accuracy

is

selected

for

final

prediction

There

are

several

differences

between

DS

and

CL

despite

the

facts

they

both

use

cross-validation

and

perform

algorithm

selection

during

classification

As

other

methods

of

stacked

generalisation

DS

uses

another

generaliser

-LRB-

i.e.

some

form

of

nearest-neighbour

classifier

-RRB-

in

the

high-level

transformed

space

whereas

CL

uses

cross-validation

for

predictive

accuracy

estimation

The

characterisation

of

various

regions

of

differing

predictive

accuracies

is

more

explicit

in

CL

and

it

is

much

harder

to

identifying

those

regions

in

DS

CL

is

using

a

binned

graph

for

each

class

whereas

DS

is

using

two

matrices

DS

requires

u

v-fold

cross-validations

and

CL

only

requires

one

v-fold

cross-validation

DS

relies

only

on

the

test

results

of

cross-validations

and

CL

requires

an

extra

measure

of

characterisation

for

each

induced

model

Merz

-LSB-

-RSB-

reports

that

DS

frequently

outperforms

a

cross-validation

algorithm

for

selecting

a

learning

algorithm

and

occasionally

outperforms

the

algorithm

with

the

best

test

accuracy

CL

performs

better

than

both

and

the

better

of

its

constituent

algorithms

in

most

of

the

domains

tested

LeBlanc

and

Tibshirani

-LSB-

-RSB-

describe

a

decision

combination

method

which

assumes

that

the

outputs

of

its

component

classifiers

are

probabilities

for

all

classes

They

show

in

only

one

artificial

domain

that

combining

these

probabilities

using

a

linear

regression

outperforms

the

two

component

classifiers

which

are

a

linear

discriminant

and

a

single

nearest

neighbour

method

Note

that

cross-validation

is

not

used

here

Their

method

also

uses

some

sort

of

characterisation

as

ours

but

does

not

perform

predictive

accuracy

estimation

as

described

in

Section

MCS

-LSB-

-RSB-

uses

a

hand-crafted

rule

to

guide

the

construction

of

three

different

models

at

the

nodes

of

a

decision

tree

and

each

model

is

trained

on

part

of

the

training

set

In

contrast

the

two

algorithms

in

the

composite

learner

are

trained

independently

on

the

entire

training

set

and

work

cooperatively

according

to

the

characterisation

of

predictive

accuracy

in

each

learned

model

Thus

no

hand-crafted

knowledge

is

used

in

the

composite

learner

CL

selects

a

model

during

classification

MCS

fixes

the

various

models

at

the

nodes

of

a

decision

tree

during

training

Methods

of

selecting

a

learning

algorithm

for

a

given

domain

-LRB-

e.g.

-LSB-

-RSB-

using

the

entire

set

of

training

data

only

choose

the

best

performing

algorithm

at

best

Some

other

methods

split

the

data

into

-LRB-

i

-RRB-

mutually

exclusive

subsets

-LSB-

-RSB-

or

-LRB-

ii

-RRB-

resampling

-LRB-

with

replacement

-RRB-

subsets

-LSB-

-RSB-

Different

classifiers

are

trained

using

these

subsets

Michie

et

al.

-LSB-

-RSB-

describe

meta-learning

methods

for

selecting

which

learning

algorithm

for

which

problem

Breiman

's

-LSB-

-RSB-

bagging

-LRB-

for

bootstrap

aggregating

-RRB-

predictors

combine

multiple

models

produced

from

a

single

algorithm

using

bootstrap

replicate

training

sets

Freund

and

Schapire

's

-LSB-

-RSB-

boosting

algorithm

-LRB-

see

also

in

-LSB-

-RSB-

-RRB-

modifies

the

data

distribution

in

multiple

runs

to

derive

multiple

models

In

both

algorithms

predictions

are

combined

either

using

a

majority

vote

or

averaging

-LRB-

weighted

-RRB-

the

estimated

class

probabilities

of

all

models

The

predictive

accuracy

of

the

characterisation

described

in

this

article

is

equivalent

to

boostings

class

probabilities

but

more

refined

due

to

the

calibration

process

One

can

apply

the

calibration

method

here

to

boosting

to

improve

its

class

probability

estimation

Kwok

and

Carter

-LSB-

-RSB-

also

use

voting

over

multiple

trees

generated

by

using

alternative

splits

In

another

approach

Kong

and

Dietterich

-LSB-

-RSB-

convert

a

d-class

problem

for

d

into

a

large

number

L

of

two-class

problems

A

decision-tree

learning

algorithm

is

used

to

derive

L

models

and

a

method

called

error-correcting

output

coding

is

employed

to

combine

these

models

Several

methods

of

reordering

ranks

when

combining

multiple

models

have

been

proposed

in

the

literature

Buntine

-LSB-

-RSB-

introduced

an

algorithm

design

strategy

based

on

the

approximating

Bayesian

decision

model

of

learning

class

probability

decision

trees

The

class

ranking

was

reordered

after

averaging

the

class

probabilities

from

the

multiple

trees

With

the

present

work

on

the

characterisation

of

predictive

accuracy

Buntine

's

strategy

can

be

readily

applied

to

include

different

models

Perrone

and

Coopers

-LSB-

-RSB-

ensemble

methods

work

in

a

similar

manner

by

merely

averaging

-LRB-

weighted

-RRB-

the

corresponding

outputs

of

the

multiple

neural

networks

Error

correlation

among

the

combined

algorithms

will

severely

affect

the

weighting

and

the

performance

of

the

ensemble

method

Smyth

Goodman

Higgins

-LSB-

-RSB-

and

Kononenko

Kovacic

-LSB-

-RSB-

use

Naive

Bayesian

combination

of

decisions

of

several

different

rules

in

re-ordering

ranks

Ho

Hull

and

Srihari

-LSB-

-RSB-

employ

logistic

regression

to

re-rank

classes

across

different

types

of

classifiers

and

only

the

ranks

of

the

classes

in

the

predictions

are

considered

CONCLUSION

The

main

contribution

of

this

article

is

two-fold

First

we

have

characterised

an

intrinsic

problem

-LRB-

i.e.

the

problem

of

locally

low-predictive

accuracy

in

IBL

and

Naive

Bayes

-RRB-

Two

intuitively

sound

methods

the

typicality

and

the

posterior

probability

have

been

found

to

be

a

satisfactory

characterisation

of

predictive

accuracy

in

and

NB

respectively

Second

knowing

the

weaknesses

in

each

algorithm

the

simple

strategy

of

selecting

as

the

final

prediction

the

one

that

has

the

higher

estimated

predictive

accuracy

has

been

demonstrated

to

partially

overcome

the

problem

of

locally

low

predictive

accuracy

When

and

NB

demonstrate

comparable

performance

we

strongly

recommend

that

the

proposed

composite

learner

is

used

to

improve

the

overall

performance

The

composite

learner

is

found

to

outperform

four

methods

of

stacked

generalisation

and

also

a

cross-validation

method

for

model

selection

in

most

of

the

experimental

domains

studied

Acknowledgment

Numerous

discussions

with

J.

R.

Quinlan

R.

M.

Cameron-Jones

Z.

Zheng

P.

Langley

and

D.

Wolpert

have

been

very

helpful

Thanks

to

D.

W.

Aha

for

providing

algorithm

The

anonymous

reviewers

have

provided

many

helpful

comments

Appendix

Table

A

-LRB-

i

-RRB-

average

error

rates

of

and

NB

using

single

data

batch

and

two

data

batches

for

the

composite

learner

This

result

shows

the

effect

of

using

calibration

versus

without

using

calibration

to

the

composite

learner

Train

Data

Size

NB

Single

Model

Composite

Learner

Single

Model

Composite

Learner

Typicality

NN

Distance

Probability

Calib

No

Calib

Calib

No

Calib

Calib

No

Calib

Calib

Calibration

Table

A

-LRB-

ii

-RRB-

average

error

rates

of

and

NB

using

a

single

data

batch

and

two

data

batches

for

the

composite

learner

Train

Data

Size

NB

Single

Model

Composite

Learner

Single

Model

Composite

Learner

Typicality

NN

Distance

Probability

Calib

No

Calib

Calib

No

Calib

Calib

No

Calib

This

appendix

shows

two

sample

resul

indicating

the

effect

of

using

calibration

without

using

calibration

to

the

composite

learner

in

a

scenario

where

two

batches

of

data

are

available

The

design

of

this

experiment

is

different

from

the

main

part

of

this

article

in

two

ways

First

the

component

models

of

the

composite

learner

are

induced

from

a

single

learning

algorithm

Second

each

of

its

component

models

are

trained

using

a

separate

batch

of

data

the

same

size

In

this

manner

we

examine

the

only

variable

-LRB-

i.e.

the

effect

of

calibration

to

the

composite

learner

-RRB-

For

each

batch

of

data

a

model

is

derived

from

using

-LRB-

or

NB

-RRB-

The

two

models

induced

from

the

same

learning

algorithm

are

then

combined

using

the

framework

described

in

this

paper

with

or

without

calibration

uses

either

typicality

or

nearest

neighbour

distance

for

its

characterisation

and

NB

uses

posterior

probability

Table

A

shows

the

results

of

this

experiment

in

two

datasets

-LRB-

i.e.

waveform

and

led

-RRB-

The

training

data

size

for

each

batch

of

data

is

varied

from

to

and

the

testing

data

size

is

fixed

at

For

each

training

data

size

the

experiment

is

repeated

over

trials

Table

A

reports

the

average

error

rates

The

columns

labelled

with

Single

Model

are

the

results

of

a

model

induced

from

a

single

batch

of

data

The

columns

labelled

with

Calib

and

No

Calib

are

the

results

of

the

composite

learner

using

calibration

and

without

using

calibration

respectively

The

results

show

that

in

all

cases

except

NB

in

the

dataset

the

composite

learner

that

employed

calibration

is

a

better

choice

However

the

exception

indicates

that

calibration

can

be

omitted

in

some

cases

without

loss

of

accuracy

our

experiment

it

is

half

of

the

total

number

for

each

class

of

fifteen

whichever

is

bigger

This

number

is

about

of

one

of

the

small

size

datasets

It

is

rather

ad

hoc

but

it

should

not

be

too

big

to

accommodate

small

datasets

or

too

small

to

avoid

graph

fluctuation

due

to

minor

changes

in

the

bin

there

are

ties

they

are

resolved

randomly

stores

all

training

instances

and

uses

maximum

differences

for

attributes

that

have

missing

values

and

computes

Euclidean

distance

between

any

two

instances

that

the

purpose

of

the

cross-validation

differs

algorithm

is

chosen

because

theoretically

Equation

-LRB-

-RRB-

can

be

expressed

as

a

tree

with

the

additional

binary

attribute

presented

to

SGd

However

the

test

selection

criterion

used

in

does

not

allow

the

binary

attribute

to

be

selected

as

the

root

node

This

will

be

indicated

in

the

results

shown

in

the

next

section

implies

that

a

cross-validation

algorithm

does

not

perform

better

than

thebest

performing

algorithm

We

have

the

same

finding

See

Section

Intelligent

Data

Analysis

Elsevier

Science

Inc.

All

rights

reserved

Go

to

IDA

Home

Search

Table

of

Issues

Table

of

Contents

References

decision combination
predictive accuracy

bayesian learning
cross-validation
instance-based learning