On-line

Prediction

and

Conversion

Strategies

N.

Cesa-Bianchi

Y.

Freundy

D.P.

Helmboldz

M.

Warmuthx

August

Board

of

Studies

in

Computer

and

Information

Sciences

University

of

California

Santa

Cruz

Santa

Cruz

CA

abstract

We

study

the

problem

of

deterministically

predicting

boolean

values

by

combining

the

boolean

predictions

of

several

experts

Previous

on-line

algorithms

for

this

problem

predict

with

the

weighted

majority

of

the

experts

predictions

These

algorithms

give

each

expert

an

exponential

weight

m

where

is

a

constant

in

-LSB-

-RRB-

and

m

is

the

number

of

mistakes

made

by

the

expert

in

the

past

We

show

that

it

is

better

to

use

sums

of

binomials

as

weights

In

particular

we

present

a

deterministic

algorithm

using

binomial

weights

that

has

a

better

worst

case

mistake

bound

than

the

best

deterministic

algorithm

using

exponential

weights

The

binomial

weights

naturally

arise

from

a

version

space

argument

We

also

show

how

both

exponential

and

binomial

weighting

schemes

can

be

used

to

make

prediction

algorithms

robust

against

noise

DSI

Universit?a

di

Milano

Via

Comelico

Milano

-LRB-

Italy

-RRB-

Email

cesabian@dsi.unimi.it

y

AT&T

Bell

Labs

Murray

Hill

New

Jersey

Email

yoav@research.att.com

z

University

of

California

Santa

Cruz

CA

Email

dph@cse.ucsc.edu

xUniversity

of

California

Santa

Cruz

CA

Email

manfred@cse.ucsc.edu

Introduction

This

paper

studies

a

simple

on-line

model

where

predictions

are

made

in

a

series

of

trials

At

each

trial

t

the

prediction

algorithm

receives

the

tth

observation

xt

and

produces

a

boolean

prediction

yt

It

then

receives

the

correct

outcome

yt

as

feedback

A

mistake

occurs

if

prediction

yt

and

outcome

yt

disagree

Following

Littlestone

-LSB-

-RSB-

we

seek

prediction

algorithms

that

minimize

the

number

of

mistakes

over

a

worst

case

sequence

of

xt

and

yt

Of

course

in

the

unconstrained

worst

case

a

mistake

can

occur

in

every

trial

In

order

to

make

good

predictions

the

predictor

needs

to

have

some

prior

knowledge

which

enables

it

to

makes

predictions

about

the

future

based

on

the

past

In

a

Bayesian

regression

framework

one

can

encode

this

knowledge

using

prior

distribution

over

the

set

of

sequences

or

over

a

set

of

sequence

models

In

this

work

we

are

interested

in

performance

bounds

that

make

no

probabilistic

assumptions

and

so

we

define

the

prior

knowledge

somewhat

differently

We

assume

that

there

are

N

experts

each

of

which

is

a

prediction

strategy

Our

goal

is

to

design

an

algorithm

which

we

shall

call

the

master

algorithm

that

combines

the

predictions

of

the

experts

in

the

following

way

At

the

beginning

of

trial

t

the

master

algorithm

feeds

the

given

observation

xt

to

all

experts

The

master

then

uses

some

function

of

the

N

predictions

produced

by

the

experts

to

form

its

own

prediction

yt

At

the

end

of

the

trial

the

feedback

yt

is

shared

with

all

experts

We

prove

worst-case

bounds

on

the

number

of

mistakes

made

by

the

master

when

the

number

of

mistakes

made

by

the

best

expert

is

bounded

Generalizations

of

the

above

model

where

the

predictions

of

the

experts

and/or

of

the

master

algorithm

may

be

in

the

continuous

range

-LSB-

-RSB-

have

been

studied

by

Vovk

-LSB-

-RSB-

Littlestone

and

Warmuth

-LSB-

-RSB-

Cesa-Bianchi

et

al.

-LSB-

-RSB-

and

Kivinen

and

Warmuth

-LSB-

-RSB-

In

this

paper

we

return

to

the

simplest

setting

where

all

predictions

and

outcomes

are

boolean

This

is

the

problem

solved

by

the

basic

Weighted

Majority

-LRB-

WM

-RRB-

algorithm

-LSB-

-RSB-

Here

we

study

the

boolean

case

in

more

depth

and

devise

a

better

algorithm

which

we

call

the

Binomial

Weighting

algorithm

or

BW

The

worst

case

number

of

mistakes

that

BW

makes

is

smaller

than

the

number

of

mistakes

made

by

previously

known

algorithms

In

fact

if

the

number

of

experts

is

large

enough

and

all

predictions

are

deterministic

and

boolean

then

we

show

that

BW

has

the

smallest

possible

worst

case

mistake

bound

among

all

master

algorithms

In

our

analysis

of

BW

we

explore

some

elegant

combinatorial

structures

that

might

be

applicable

elsewhere

The

Weighted

Majority

algorithms

cited

above

attempt

to

minimize

the

number

of

mistakes

made

as

a

function

of

the

number

of

mistakes

made

by

the

best

expert

They

assign

to

each

expert

E

a

weight

of

the

form

m

where

is

a

constant

in

-LSB-

-RRB-

and

m

is

the

total

number

of

mistakes

-LRB-

or

more

generally

the

total

loss

-RRB-

incurred

by

expert

E

so

The

essential

property

is

that

the

experts

making

many

mistakes

get

their

weights

rapidly

slashed

The

WM

algorithm

uses

the

weighted

average

of

the

experts

predictions

to

form

its

own

prediction

It

simply

predicts

if

the

weighted

average

is

greater

than

and

otherwise

The

new

master

algorithm

algorithm

BW

uses

its

weights

in

a

similar

way

to

WM

for

predicting

however

these

weights

are

not

in

exponential

form

Instead

they

are

tails

of

a

binomial

sum

A

further

difference

between

WM

and

BW

is

the

following

On

each

trial

similar

approach

can

be

taken

for

learning

the

best

combination

of

experts

although

different

forms

of

the

weights

are

used

when

the

loss

of

the

master

is

to

be

close

to

the

loss

of

the

best

convex

-LSB-

-RSB-

or

linear

-LSB-

-RSB-

combination

of

experts

Introduction

WM

predicts

if

and

only

if

the

total

current

weight

of

the

experts

predicting

is

bigger

than

the

total

current

weight

of

the

experts

predicting

BW

instead

predicts

if

and

only

if

the

total

updated

weight

resulting

from

the

outcome

being

is

bigger

than

the

total

updated

weight

resulting

from

the

outcome

being

This

binomial

weighting

scheme

is

motivated

by

a

version

space

argument

The

mistake

bound

of

the

Weighted

Majority

algorithm

approximates

the

mistake

bound

of

the

BW

algorithm

in

the

same

way

that

Chernoff

bounds

approximate

sums

of

binomial

tails

We

show

that

the

gap

between

the

mistake

bounds

of

the

Weighted

Majority

algorithm

and

our

new

algorithm

can

be

arbitrarily

large

Finally

a

perhaps

subtler

difference

between

exponential

weights

and

our

new

improved

scheme

is

that

each

expert

's

weight

in

the

latter

scheme

depends

not

only

on

the

current

mistake

count

of

the

expert

but

also

on

the

current

mistake

count

of

the

master

We

show

that

our

algorithm

has

the

best

possible

worst

case

mistake

bound

when

the

number

of

experts

is

very

large

compared

to

the

loss

of

the

best

expert

This

lower

bound

analysis

is

based

on

a

relation

between

our

prediction

problem

and

Ulam

's

searching

game

with

a

fixed

number

of

lies

-LSB-

-RSB-

We

also

present

a

second

lower

bound

argument

for

our

prediction

model

This

second

argument

use

a

probabilistic

construction

to

prove

that

both

the

BW

and

the

tuned

Weighted

Majority

algorithm

are

asymptotically

optimal

That

is

the

ratio

between

the

mistake

bound

of

either

algorithm

and

the

best

possible

worst

case

mistake

bound

goes

to

as

the

number

N

of

experts

or

the

loss

k

of

the

best

expert

go

to

infinity

An

equivalent

lower

bound

has

been

previously

obtained

by

Vovk

-LSB-

-RSB-

using

arguments

from

coding

theory

We

use

the

ideas

behind

the

BW

master

algorithm

to

devise

a

method

-LRB-

which

we

call

a

conversion

strategy

-RRB-

to

make

prediction

algorithms

robust

against

noise

The

conversion

strategy

feeds

different

feedbacks

to

several

copies

of

the

same

prediction

algorithm

If

the

noise

level

is

low

then

one

copy

will

get

noiseless

data

enabling

the

conversion

strategy

to

make

good

predictions

Our

upper

bound

has

slightly

better

constants

than

the

one

independently

obtained

by

Auer

and

Long

-LSB-

-RSB-

and

is

close

to

the

lower

bound

given

by

Littlestone

and

Warmuth

-LSB-

-RSB-

It

remains

open

whether

binomial

weights

also

lead

to

improved

master

prediction

algorithms

for

the

case

when

the

prediction

of

the

master

is

allowed

to

be

in

the

continuous

interval

-LSB-

-RSB-

In

this

more

general

setting

mistake

bounds

are

replaced

by

bounds

on

the

total

absolute

loss

There

are

master

prediction

algorithms

for

this

problem

-LSB-

-RSB-

using

exponential

weights

whose

mistake

bounds

are

exactly

half

of

the

corresponding

mistake

bounds

in

the

boolean

case

However

our

attempts

to

construct

a

continuous

prediction

algorithm

that

achieves

half

-LRB-

plus

possibly

a

constant

-RRB-

the

loss

of

the

BW

algorithm

have

so

far

been

unsuccessful

The

paper

is

organized

as

follows

In

Section

we

present

the

new

algorithm

BW

compare

it

against

WM

and

prove

general

lower

bounds

In

Section

we

introduce

two

conversion

strategies

one

based

on

binomial

weights

and

one

based

on

exponential

weights

Section

is

devoted

to

conclusions

Notation

The

set

X

represents

the

set

of

possible

observations

We

use

-LRB-

X

-RRB-

for

the

set

of

all

finite

sequences

over

-LRB-

X

-RRB-

of

nonzero

length

and

s

for

a

sequence

h

-LRB-

xt

yt

-RRB-

it

-LRB-

of

unspecified

length

-RRB-

in

-LRB-

X

-RRB-

of

observations

and

outcomes

Let

N

denote

the

natural

numbers

including

zero

The

notation

sn

for

any

n

N

represents

either

a

sequence

of

length

n

or

the

length

n

prefix

of

a

longer

sequence

s

The

correct

interpretation

will

be

clear

from

the

context

An

expert

is

any

function

mapping

-LRB-

X

-RRB-

X

to

In

this

paper

we

treat

experts

in

an

on-line

fashion

On

the

tth

trial

each

expert

E

makes

the

prediction

E

-LRB-

st

xt

-RRB-

where

xt

X

is

the

current

observation

and

st

is

the

sequence

of

observation/outcome

pairs

from

the

previous

t

trials

At

the

end

of

the

trial

the

expert

is

given

the

feedback

yt

for

the

current

trial

-LRB-

and

st

for

the

next

trial

is

created

by

appending

-LRB-

xt

yt

-RRB-

to

st

-RRB-

We

say

that

expert

E

either

is

wrong

makes

a

mistake

or

is

incorrect

when

its

prediction

at

trial

t

E

-LRB-

st

xt

-RRB-

is

different

from

yt

Also

we

use

dH

-LRB-

y

z

-RRB-

to

denote

the

Hamming

distance

between

any

two

boolean

sequences

y

and

z

of

equal

length

For

the

sum

of

binomials

we

use

the

notation

m

k

Pki

mi

for

all

integers

m

and

k

using

the

convention

m

k

when

m

or

k

negative

We

conventionally

set

mi

when

i

m

or

when

either

m

or

i

is

negative

We

will

often

make

use

of

the

well-known

combinatorial

identity

q

i

q

i

q

i

-LRB-

-RRB-

that

holds

for

all

non-zero

integers

q

and

all

integers

i

We

denote

the

binary

logarithm

by

log

and

the

natural

logarithm

by

ln

Master

Algorithms

for

Combining

the

Predictions

of

Experts

In

this

section

we

introduce

a

master

algorithm

that

sequentially

predicts

boolean

sequences

by

combining

the

predictions

of

a

set

of

experts

Throughout

the

section

we

assume

that

a

bound

k

on

the

number

of

mistakes

made

on

the

sequence

by

the

best

expert

in

the

set

is

available

and

known

to

the

master

algorithm

For

any

expert

E

and

for

any

sequence

s

-LRB-

X

-RRB-

of

instances

and

outcomes

we

denote

the

number

of

mistakes

-LRB-

i.e.

total

loss

-RRB-

of

expert

E

on

sequence

s

by

LE

-LRB-

s

-RRB-

Also

if

E

is

a

set

of

experts

we

use

LE

-LRB-

s

-RRB-

for

the

minimum

LE

-LRB-

s

-RRB-

over

the

experts

E

E

We

usually

make

the

assumption

that

LE

-LRB-

s

-RRB-

k

for

some

constant

k

known

to

the

master

algorithm

We

point

out

that

our

master

algorithms

are

domain

independent

using

the

information

provided

by

the

sequence

of

instances

hxtit

only

to

obtain

the

predictions

of

the

experts

Our

goal

is

to

solve

the

following

problem

Problem

Suppose

a

set

E

of

N

experts

is

available

and

the

task

is

to

predict

in

an

online

fashion

the

bits

y

of

some

sequence

s

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

x

y

-RRB-

in

a

set

of

sequences

-LRB-

X

-RRB-

Suppose

also

that

an

upper

bound

k

on

the

loss

of

the

best

expert

in

E

is

known

i.e.

for

each

s

LE

-LRB-

s

-RRB-

k

How

can

a

master

algorithm

combine

the

expert

's

predictions

so

that

its

worst

case

number

of

mistakes

is

minimized

If

the

master

algorithm

knew

which

expert

E

E

made

only

k

mistakes

then

it

could

simply

predict

the

same

way

that

expert

E

does

However

the

good

expert

-LRB-

or

experts

-RRB-

is

not

known

in

advance

In

the

fortunate

case

where

k

the

master

algorithm

knows

that

one

of

the

experts

predicts

perfectly

on

s

In

this

case

the

well-known

Halving

algorithm

-LSB-

-RSB-

can

be

used

On

each

trial

the

Halving

algorithm

predicts

the

same

way

as

majority

of

the

those

experts

that

have

never

made

a

mistake

-LRB-

the

consistent

experts

-RRB-

The

number

of

consistent

experts

Master

Algorithms

for

Combining

the

Predictions

of

Experts

is

reduced

by

at

least

a

factor

of

two

each

time

the

Halving

algorithm

makes

a

mistake

so

the

master

makes

at

most

log

N

mistakes

on

any

s

where

one

of

the

N

experts

always

predicts

correctly

We

now

present

a

simple

master

algorithm

called

the

Version

Space

algorithm

which

will

be

used

to

motivate

the

Binomial

Weighting

-LRB-

BW

-RRB-

algorithm

To

do

this

we

make

the

simplifying

assumption

that

the

length

of

the

sequence

of

instances

is

known

as

well

This

assumption

will

be

removed

shortly

Since

the

master

algorithm

knows

that

the

best

expert

makes

at

most

k

mistake

it

can

use

the

following

trick

The

master

algorithm

expands

each

expert

into

a

set

of

variants

so

that

some

variant

of

some

expert

predicts

perfectly

and

then

uses

the

Halving

algorithm

on

the

variants

If

expert

E

makes

exactly

j

mistakes

on

some

sequence

s

of

length

then

expert

E

can

be

expanded

into

a

collection

of

j

variants

containing

a

perfect

variant

Each

variant

in

the

collection

predicts

as

E

on

j

of

the

trials

and

predicts

with

the

opposite

of

E

's

predictions

on

the

other

j

trials

Thus

expert

E

is

expanded

into

a

collection

of

j

variants

including

one

which

changes

E

's

predictions

on

exactly

those

trials

where

E

predicts

incorrectly

For

Problem

the

master

algorithm

knows

that

at

least

one

of

the

N

experts

makes

at

most

k

incorrect

predictions

but

the

master

algorithm

knows

neither

which

expert

is

the

best

nor

the

exact

number

of

mistakes

made

by

the

best

expert

However

the

master

algorithm

can

expand

each

expert

into

a

collection

of

k

variants

The

union

of

these

collections

contains

at

most

N

k

variants

and

is

guaranteed

to

contain

at

least

one

variant

that

predicts

correctly

on

all

trials

Our

Version

Space

algorithm

runs

the

Halving

algorithm

on

the

union

of

these

collections

and

has

a

worst

case

mistake

bound

of

log

N

log

k

-LRB-

when

the

bounds

on

the

number

of

trials

and

k

on

the

number

of

mistakes

made

by

the

best

expert

are

known

in

advance

-RRB-

Intuitively

the

Version

Space

algorithm

uses

all

the

knowledge

it

has

about

the

experts

and

the

sequences

which

is

that

there

is

one

expert

which

makes

at

most

k

mistakes

on

the

sequence

It

does

not

know

which

expert

will

be

best

in

what

trials

the

best

expert

will

make

its

mistakes

or

even

how

many

mistakes

the

best

expert

will

make

-LRB-

other

than

the

upper

bound

k

-RRB-

Since

the

goal

of

the

algorithm

is

to

minimize

the

number

of

mistakes

that

it

makes

in

the

worst

case

it

has

to

treat

all

of

the

scenarios

that

are

possible

under

the

assumptions

equally

Observe

that

the

version

space

at

the

beginning

of

trial

t

can

be

represented

by

one

weight

per

expert

The

weight

of

an

expert

is

simply

the

number

of

its

k

variants

that

are

consistent

with

the

sequence

so

If

expert

E

makes

at

most

k

mistakes

on

the

trials

and

has

made

j

mistakes

in

trials

through

t

then

expert

E

can

make

at

most

k

j

more

mistakes

in

the

remaining

t

trials

Thus

the

weight

of

E

on

the

t

trial

should

be

t

k?j

which

is

exactly

the

number

of

variants

created

from

E

that

are

consistent

-LRB-

The

initial

weight

of

each

expert

is

k

-RRB-

Thus

the

Version

Space

algorithm

can

be

implemented

by

manipulating

binomials

representing

the

weights

-LRB-

number

of

consistent

variants

-RRB-

of

the

experts

If

expert

E

has

made

j

mistakes

in

the

first

t

trials

then

during

trial

t

expert

E

votes

with

weight

t

k?j

for

its

own

prediction

and

with

weight

t

k

-LRB-

j

-RRB-

for

the

opposite

prediction

Note

that

these

votes

correspond

to

the

number

of

E

's

variants

that

are

consistent

with

all

t

previous

trials

weighting

scheme

based

on

the

sum

of

binomial

coefficients

was

first

introduced

by

Berlekamp

-LSB-

-RSB-

and

agree

-LRB-

or

do

not

agree

respectively

-RRB-

with

the

prediction

of

E.

Also

expert

E

's

total

weight

is

split

between

the

two

choices

since

t

k?j

t

k?j

t

k?j

This

implementation

of

the

Version

Space

algorithm

totals

the

votes

for

outcome

and

outcome

and

predicts

with

the

majority

At

the

end

of

each

trial

t

the

Version

Space

algorithm

updates

the

weights

of

the

experts

to

reflect

the

outcome

on

that

trial

yt

In

addition

the

value

yt

is

given

to

all

the

experts

since

their

future

predictions

might

depend

on

the

past

sequence

The

Version

Space

algorithm

which

runs

the

Halving

algorithm

directly

on

the

N

k

variants

and

the

implementation

which

manipulates

binomial

weights

for

each

expert

clearly

make

the

same

predictions

The

Binomial

Weighting

-LRB-

BW

-RRB-

algorithm

is

similar

to

the

Version

Space

algorithm

using

weights

but

the

BW

algorithm

uses

another

trick

which

removes

the

requirement

that

the

algorithm

knows

the

length

of

the

sequence

This

trick

also

makes

the

upper

bound

on

the

number

of

mistakes

made

by

the

BW

algorithm

independent

of

There

are

two

versions

of

the

Halving

algorithm

one

that

discards

all

inconsistent

experts

in

each

trial

and

one

that

does

this

only

in

trials

when

the

Halving

algorithm

makes

a

mistake

-LRB-

such

algorithms

are

called

conservative

by

Littlestone

-LSB-

-RSB-

-RRB-

Both

versions

of

the

Halving

algorithm

have

the

same

worst

case

mistake

bound

-LRB-

log

N

-RRB-

so

nothing

is

lost

by

making

the

Version

Space

algorithm

conservative

The

Binomial

Weighting

algorithm

is

the

implementation

of

the

conservative

Version

Space

algorithm

with

binomial

weights

and

is

described

in

Figure

Because

the

BW

algorithm

is

conservative

we

do

not

need

a

variant

which

perfectly

predicts

the

outcome

It

suffices

to

have

only

those

variants

whose

mistakes

occur

when

the

BW

master

algorithm

predicts

incorrectly

Since

the

BW

algorithm

discards

variants

only

when

the

master

makes

a

mistake

such

a

variant

will

never

be

discarded

Thus

the

BW

algorithm

considers

only

m

k

of

each

expert

where

m

max

n

q

N

q

log

N

log

q

k

o

as

in

Figure

It

is

easy

to

show

that

BW

makes

at

most

m

mistakes

Assume

to

the

contrary

that

it

makes

m

mistakes

Since

at

least

one

of

the

N

experts

makes

at

most

k

mistakes

at

least

one

of

the

N?m

k

variants

is

consistent

with

the

m

outcomes

where

BW

made

mistakes

On

the

other

hand

the

number

of

consistent

variants

drops

by

a

factor

of

at

least

two

each

time

BW

makes

an

incorrect

prediction

Thus

the

number

of

consistent

variants

after

BW

makes

m

mistake

is

at

least

one

and

at

most

N?m

k

It

follows

that

N?m

k

and

equivalently

m

log

N

log

m

k

contradicting

the

definition

of

m

in

Figure

This

analysis

gives

us

the

following

theorem

Theorem

For

any

k

N

N

-LRB-

N

-RRB-

for

any

set

E

of

N

experts

and

for

any

sequence

s

-LRB-

X

-RRB-

if

LE

-LRB-

s

-RRB-

k

then

the

total

number

of

mistakes

of

BW

-LRB-

k

-RRB-

on

s

is

at

most

Expanding

each

expert

into

m

k

variants

instead

of

m

k

variants

-LRB-

where

m

is

defined

as

in

Figure

-RRB-

does

not

lead

to

the

mistake

bound

of

m

stated

in

Theorem

For

example

consider

the

case

where

there

is

N

expert

guaranteed

to

make

at

most

k

mistake

so

m

Assume

the

expert

is

expanded

into

just

m

k

variants

-LRB-

one

predicting

as

the

expert

and

one

predicting

the

other

way

-RRB-

and

the

expert

is

correct

on

the

first

trial

The

master

algorithm

would

see

a

tie

vote

and

could

predict

as

the

variant

and

make

a

mistake

Now

only

the

-LRB-

unmodified

-RRB-

expert

is

consistent

and

the

master

will

predict

as

the

expert

does

However

this

expert

still

has

a

mistake

to

make

and

thus

the

master

might

make

a

total

of

two

mistakes

Although

the

number

of

consistent

variants

has

been

reduced

to

one

-LRB-

the

original

expert

-RRB-

the

surviving

variant

may

still

have

mistakes

to

make

By

considering

m

k

variants

of

each

expert

we

guarantee

that

if

only

one

variant

is

consistent

then

the

expert

producing

that

variant

has

already

made

k

mistakes

-LRB-

and

thus

will

be

correct

on

all

future

trials

-RRB-

Master

Algorithms

for

Combining

the

Predictions

of

Experts

Master

Algorithm

BW

Input

A

set

of

N

experts

E

and

a

nonnegative

integer

k.

Let

m

max

-LRB-

q

N

q

log

N

log

q

k

-RRB-

Set

the

initial

weight

of

each

expert

to

m

k

and

set

the

number

of

mistakes

made

by

the

master

to

zero

For

each

trial

t

-LRB-

a

-RRB-

For

each

expert

E

E

Let

j

be

the

number

of

previous

trials

where

both

E

and

the

master

made

incorrect

predictions

Then

expert

E

has

current

weight

m

k?j

and

votes

for

its

own

prediction

with

weight

k?j

and

with

weight

k?j

for

the

opposite

prediction

-LRB-

b

-RRB-

Sum

the

votes

for

bit

zero

and

for

bit

one

and

predict

with

the

majority

-LRB-

arbitrary

in

case

of

a

tie

-RRB-

-LRB-

c

-RRB-

Get

the

correct

prediction

yt

-LRB-

d

-RRB-

If

a

mistake

occurred

then

increment

and

update

the

weight

of

each

expert

to

the

weight

with

which

it

voted

for

correct

bit

yt

Figure

The

Binomial

Weighting

algorithm

max

-LRB-

q

N

q

log

N

log

q

k

-RRB-

-LRB-

-RRB-

We

now

describe

a

variant

of

algorithm

BW

called

-LRB-

see

Figure

-RRB-

which

has

the

same

worst-case

mistake

bound

proven

in

Theorem

but

for

many

sequences

of

examples

the

new

algorithm

makes

fewer

mistakes

than

the

original

algorithm

The

current

weight

of

an

expert

E

is

now

m

k?j

where

j

is

the

number

of

mistakes

of

E

in

all

previous

trials

and

not

just

in

the

trials

in

which

the

master

made

mistakes

as

well

The

value

of

m

is

recomputed

at

the

beginning

of

each

trial

This

value

will

decrease

by

at

least

one

after

all

trials

in

which

the

master

made

a

mistake

because

the

total

weight

after

such

a

trial

is

at

most

half

of

what

it

was

before

the

trial

-LRB-

decreasing

m

by

at

least

one

corresponds

to

increasing

in

BW

-RRB-

The

value

of

m

can

never

increase

but

it

might

also

decrease

after

trials

in

which

the

master

made

no

mistakes

Again

it

can

be

shown

by

induction

that

the

number

of

mistakes

from

any

trial

onward

is

at

most

the

value

of

m

computed

at

the

beginning

of

that

trial

Comparison

with

Weighted

Majority

In

this

section

we

compare

the

performances

of

the

BW

and

Weighted

Majority

-LRB-

WM

-RRB-

algorithms

The

WM

algorithm

has

a

parameter

-LSB-

-RRB-

An

expert

E

votes

for

its

own

prediction

with

weight

kE

and

for

the

opposite

prediction

with

weight

kE

the

original

algorithm

expert

E

simply

votes

with

weight

kE

for

its

own

prediction

The

more

complicated

voting

scheme

given

in

the

text

is

more

similar

to

the

voting

scheme

of

the

BW

algorithm

Both

variants

of

the

WM

algorithm

generate

the

same

predictions

Master

Algorithm

Input

A

set

of

N

experts

E

and

a

nonnegative

integer

k.

For

each

expert

E

E

set

the

mistake

budget

kE

equal

to

k.

For

each

trial

t

-LRB-

a

-RRB-

Let

m

max

-LRB-

q

N

q

log

X

q

kE

-RRB-

-LRB-

b

-RRB-

For

each

expert

E

E

Expert

E

has

current

weight

m

kE

and

votes

for

its

own

prediction

with

weight

m

kE

and

with

weight

m

kE

for

the

opposite

prediction

-LRB-

c

-RRB-

Sum

the

votes

for

bit

zero

and

for

bit

one

and

predict

with

the

majority

-LRB-

arbitrary

in

case

of

a

tie

-RRB-

-LRB-

d

-RRB-

Get

the

correct

prediction

yt

-LRB-

e

-RRB-

Decrease

the

mistake

budget

kE

of

all

experts

that

predicted

incorrectly

in

this

trial

by

Figure

The

Modified

Binomial

Weighting

algorithm

Both

master

algorithms

predict

one

if

and

only

if

the

experts

predicting

one

the

experts

predicting

zero

The

weights

used

by

the

BW

algorithm

are

binomial

tails

whereas

the

WM

algorithm

uses

exponential

weights

of

the

form

j

We

often

refer

to

as

the

update

factor

of

the

WM

algorithm

because

an

expert

's

weight

gets

multiplied

by

when

the

expert

predicts

incorrectly

As

one

would

expect

the

choice

of

greatly

effects

how

the

WM

algorithm

performs

In

our

setting

the

master

algorithms

are

given

two

parameters

N

the

number

of

experts

and

a

bound

k

on

the

number

of

mistakes

made

by

the

best

expert

We

are

interested

in

worst

case

bounds

on

the

algorithm

's

performance

as

functions

of

N

and

k.

For

any

master

algorithm

A

define

the

worst

case

number

of

mistakes

WCA

-LRB-

N

k

-RRB-

as

WCA

-LRB-

N

k

-RRB-

def

max

E

of

N

experts

max

s

LE

-LRB-

s

-RRB-

k

-LSB-

number

of

mistakes

of

A

-LRB-

E

k

-RRB-

on

s

-RSB-

Furthermore

denote

the

performance

of

the

best

master

algorithm

by

WC

-LRB-

N

k

-RRB-

so

WC

-LRB-

N

k

-RRB-

def

min

algorithms

A

WCA

-LRB-

N

k

-RRB-

We

will

show

in

Subsection

that

if

the

number

of

experts

is

large

enough

then

the

BW

algorithm

is

-LRB-

essentially

-RRB-

optimal

That

is

for

any

k

there

exists

Nk

such

that

for

all

N

Nk

WCBW

-LRB-

N

k

-RRB-

WC

-LRB-

N

k

-RRB-

We

can

only

prove

the

above

for

Nk

-LRB-

-RRB-

However

we

show

in

Subsection

that

BW

is

asymptotically

optimal

i.e.

the

ratio

WCBW

-LRB-

N

k

-RRB-

WC

-LRB-

N

k

-RRB-

goes

to

when

N

or

k

goes

to

infinity

-LRB-

see

Theorem

-RRB-

algorithms

predict

arbitrarily

if

the

weights

are

tied

Master

Algorithms

for

Combining

the

Predictions

of

Experts

Comparing

the

BW

and

WM

algorithms

is

complicated

by

the

fact

that

WM

's

mistake

bound

depends

on

how

the

update

factor

is

chosen

-LRB-

as

a

function

of

N

and

k

-RRB-

For

-LSB-

-RRB-

let

WM

denote

the

WM

algorithm

which

chooses

the

update

factor

From

Littlestone

and

Warmuth

-LSB-

-RSB-

we

have

the

following

mistake

bound

for

the

WM

algorithm

WCWM

-LRB-

N

k

-RRB-

log

N

k

log

log

def

up

-LRB-

N

k

-RRB-

-LRB-

-RRB-

Let

be

the

value

of

-LRB-

as

a

function

of

N

and

k

-RRB-

that

minimizes

up

-LRB-

N

k

-RRB-

Vovk

-LSB-

-RSB-

gives

an

implicit

formula

for

An

explicit

approximation

to

is

given

in

Cesa-Bianchi

et

al.

-LSB-

-RSB-

With

set

to

this

approximation

they

show

that

up

-LRB-

N

k

-RRB-

pk

ln

N

log

N

We

show

that

up

-LRB-

N

k

-RRB-

WC

-LRB-

N

k

-RRB-

whenever

N

or

k

goes

to

infinity

-LRB-

see

Theorem

-RRB-

Although

both

up

-LRB-

N

k

-RRB-

and

WCBW

-LRB-

N

k

-RRB-

have

the

same

leading

term

when

N

and/or

k

is

large

there

can

be

significant

differences

between

them

We

show

below

that

our

bound

on

the

BW

algorithm

is

always

at

least

as

good

as

the

known

bounds

on

the

WM

algorithm

i.e.

that

WCBW

-LRB-

N

k

-RRB-

up

-LRB-

N

k

-RRB-

for

all

choices

of

N

and

k

-LRB-

see

Theorem

-RRB-

However

as

we

shall

discuss

below

at

least

for

small

values

of

N

the

upper

bound

on

the

WM

algorithm

up

-LRB-

N

k

-RRB-

is

weak

and

misleading

Let

WM

be

the

WM

algorithm

that

uses

update

factor

and

WM

be

the

WM

algorithm

that

chooses

as

a

function

of

N

and

k

so

that

WM

-LRB-

N

k

-RRB-

is

minimized

Unfortunately

we

do

n't

know

how

to

efficiently

compute

the

value

of

used

by

WM

The

value

of

WCWM

-LRB-

N

k

-RRB-

is

much

smaller

than

WCWM

-LRB-

N

k

-RRB-

for

some

choices

of

N

and

k

It

is

even

conceivable

that

WCWM

-LRB-

N

k

-RRB-

is

smaller

than

WCBW

-LRB-

N

k

-RRB-

for

some

N

k

pairs

although

this

disagrees

with

our

intuition

To

make

the

weakness

of

the

up

-LRB-

N

k

-RRB-

bound

concrete

consider

the

case

when

there

are

three

experts

-LRB-

N

-RRB-

It

is

easy

to

see

that

BW

-LRB-

k

-RRB-

which

is

the

best

possible

Also

WM

-LRB-

N

k

-RRB-

whenever

However

the

value

of

which

minimizes

up

-LRB-

k

-RRB-

approaches

one

when

N

and

k

becomes

large

In

fact

up

-LRB-

k

-RRB-

grows

as

-LRB-

pk

-RRB-

Thus

the

up

-LRB-

k

-RRB-

bound

overestimates

the

number

of

mistakes

made

by

WM

by

an

-LRB-

additive

-RRB-

-LRB-

pk

-RRB-

term

Intuitively

a

reason

for

this

is

that

when

is

large

then

two

poorly

performing

experts

can

outweigh

the

good

expert

and

cause

the

master

to

make

unnecessary

mistakes

The

main

difference

between

the

WM

and

BW

algorithms

is

how

the

weights

are

updated

The

WM

algorithm

uses

one

fixed

update

factor

throughout

the

entire

learning

process

The

update

factor

can

be

written

as

e

where

has

the

natural

interpretation

as

a

learning

rate

When

is

small

is

large

and

the

WM

algorithm

learns

slowly

When

is

large

is

small

and

the

WM

algorithm

rapidly

slashes

the

weights

of

poorly

performing

experts

The

disadvantage

of

a

high

learning

rate

is

that

the

algorithm

might

discount

experts

too

quickly

causing

its

predictions

to

be

dominated

by

only

a

few

experts

When

the

BW

algorithm

changes

an

expert

's

weight

from

k?j

to

k?j

then

this

can

be

seen

as

multiplying

the

expert

's

weight

by

an

update

factor

which

depends

on

the

number

of

mistakes

made

so

far

by

the

master

algorithm

-LRB-

as

well

as

j

the

number

of

mistakes

made

by

the

expert

N

and

k

-RRB-

These

update

factors

used

by

BW

become

less

drastic

as

the

number

of

mistakes

made

by

the

master

increases

-LRB-

and

the

upper

index

of

the

binomial

coefficients

decreases

-RRB-

This

represents

a

kind

of

annealing

schedule

performed

on

the

learning

rate

-LRB-

see

e.g.

-LSB-

-RSB-

for

examples

of

annealing

-RRB-

when

the

master

knows

nothing

the

learning

rate

is

relatively

high

and

as

the

master

learns

the

learning

rate

decreases

in

order

to

preserve

the

previously

acquired

knowledge

Although

one

could

use

any

of

a

number

of

ad

hoc

heuristics

for

cooling

down

the

learning

rate

we

have

seen

that

the

binomial

weights

are

theoretically

justified

by

the

version

space

argument

Our

belief

is

that

the

single

update

factor

used

by

WM

-LRB-

N

k

-RRB-

attempts

to

approximate

the

sequence

of

update

factors

used

by

BW

-LRB-

N

k

-RRB-

In

addition

to

the

update

relationships

between

the

two

algorithms

our

proof

techniques

provide

further

evidence

for

this

belief

Both

the

optimization

of

WM

's

update

factor

as

a

function

of

N

and

k

-LRB-

Lemma

-RRB-

and

the

proof

that

the

bound

for

WM

is

always

worse

than

the

BW

bound

-LRB-

Theorem

-RRB-

use

techniques

similar

to

those

used

to

prove

Chernoff

bounds

for

binomial

tails

-LSB-

-RSB-

We

now

proceed

to

compare

the

bounds

on

the

WM

and

BW

algorithms

beginning

with

an

examination

of

the

minimizing

up

-LRB-

N

k

-RRB-

Here

we

rederive

the

implicit

form

of

given

by

Vovk

-LSB-

-RSB-

Let

H

-LRB-

-RRB-

be

the

binary

entropy

function

H

-LRB-

x

-RRB-

x

log

x

-LRB-

x

-RRB-

log

-LRB-

x

-RRB-

defined

for

all

x

-LRB-

where

H

-LRB-

-RRB-

H

-LRB-

-RRB-

-RRB-

Lemma

-LRB-

See

also

-LSB-

-RSB-

-RRB-

Pick

any

N

k

and

-LSB-

-RRB-

If

m

k

-LRB-

-RRB-

fi

-LRB-

so

that

m

and

k

m?k

-RRB-

then

the

following

are

equivalent

a.

@up

-LRB-

N

k

-RRB-

b.

k

up

-LRB-

N

k

-RRB-

k

c.

m

up

-LRB-

N

k

k

m

k

-RRB-

and

d.

m

log

N

mH

-LRB-

k

m

-RRB-

Also

there

is

exactly

one

m

for

which

the

last

inequality

is

an

equality

and

the

corresponding

is

the

unique

minimum

of

up

-LRB-

N

k

-RRB-

Proof

Since

up

-LRB-

N

k

-RRB-

-LRB-

log

N

klog

-RRB-

log

we

have

@up

-LRB-

N

k

-RRB-

k

ln

up

-LRB-

N

k

-RRB-

-LRB-

-RRB-

ln

Note

that

ln

since

-LSB-

-RRB-

So

the

equivalence

between

-LRB-

a.

-RRB-

and

-LRB-

b.

-RRB-

is

easily

verified

by

setting

the

above

derivative

to

multiplying

by

-LRB-

-RRB-

ln

and

solving

for

The

equivalence

between

-LRB-

b.

-RRB-

and

-LRB-

c.

-RRB-

is

obtained

by

substituting

k

m?k

into

-LRB-

b.

-RRB-

and

solving

for

m.

To

show

equivalence

between

-LRB-

c.

-RRB-

and

-LRB-

d.

-RRB-

we

multiply

-LRB-

c.

-RRB-

by

the

denominator

of

up

-LRB-

N

k

k

m?k

-RRB-

Using

log

km?k

log

-LRB-

km

-RRB-

we

get

the

inequality

m

log

N

k

log

k

m

k

m

log

-LRB-

k

m

-RRB-

-LRB-

-RRB-

whose

RHS

equals

log

N

mH

-LRB-

km

-RRB-

Note

that

log

N

-LRB-

-RRB-

so

m

log

N

mH

-LRB-

km

-RRB-

for

m

close

to

Since

H

-LRB-

km

-RRB-

for

m

the

LHS

of

-LRB-

-RRB-

grows

faster

than

the

RHS

-LRB-

as

a

function

of

m

-RRB-

Thus

there

will

be

exactly

one

m

where

m

log

N

m?H

-LRB-

km

-RRB-

From

the

equivalences

it

follows

that

@up

evaluated

at

k

m

k

is

zero

and

this

is

the

unique

minimizer

of

up

-LRB-

N

k

-RRB-

Master

Algorithms

for

Combining

the

Predictions

of

Experts

Lemma

shows

that

when

N

and

k

are

fixed

the

solution

m

to

m

log

N

mH

-LRB-

km

-RRB-

is

the

minimum

value

of

up

-LRB-

N

k

-RRB-

Although

m

-LRB-

and

k

m

k

-RRB-

is

a

function

of

N

and

k

we

suppress

this

dependence

to

simplify

our

notation

Also

if

m

m

and

k

m?k

then

m

is

an

upper

bound

on

up

-LRB-

N

k

-RRB-

WCWM

-LRB-

N

k

-RRB-

Since

we

are

computing

integervalued

mistake

bounds

it

suffices

to

find

any

R

such

that

bm?c

Note

that

m

log

N

mH

-LRB-

km

-RRB-

when

m

m

and

m

log

N

mH

-LRB-

km

-RRB-

when

m

m

Therefore

we

can

find

an

appropriate

by

doing

binary

search

Since

WC

-LRB-

N

k

-RRB-

blog

Nc

-LRB-

as

proven

by

Littlestone

and

Warmuth

-LSB-

-RSB-

-RRB-

and

m

pk

ln

N

log

N

as

shown

by

CesaBianchi

et

al.

-LSB-

-RSB-

the

search

can

be

limited

to

the

range

-LSB-

blog

Nc

pk

ln

N

logN

-RSB-

Thus

the

binary

search

takes

at

most

O

-LRB-

log

k

log

log

N

-RRB-

time

Our

experience

indicates

that

m

tends

to

be

close

to

the

right

edge

of

this

range

For

N

m

is

within

one

of

ln

N

log

N

For

arbitrary

N

the

right

boundary

seems

to

be

at

most

log

N

greater

than

m

However

these

considerations

are

based

on

numerical

plots

and

have

not

been

verified

analytically

We

now

show

that

BW

beats

the

bound

obtained

by

minimizing

the

upper

bound

for

WM

We

need

a

preliminary

lemma

that

is

easily

derived

from

the

Binomial

Theorem

Lemma

For

all

m

k

N

such

that

k

m

and

for

all

m

k

-LRB-

-RRB-

m

k

-LRB-

-RRB-

Recall

that

m

up

-LRB-

N

k

-RRB-

for

k

m

k

is

the

minimum

of

up

-LRB-

N

k

-RRB-

over

all

-LSB-

-RRB-

Similarly

let

q

be

the

largest

integer

q

such

that

q

log

N

q

log

q

k

While

m

is

the

upper

bound

on

Weighted

Majority

derived

from

inequality

-LRB-

-RRB-

q

is

the

upper

bound

on

the

Binomial

Weighting

algorithm

in

Theorem

-LRB-

q

like

m

implicitly

depends

on

N

and

k

-RRB-

Theorem

Pick

any

integer

k

and

any

positive

integer

N

If

q

is

the

largest

integer

q

such

that

q

log

N

q

log

q

k

then

WCBW

-LRB-

N

k

-RRB-

q

and

q

up

-LRB-

N

k

-RRB-

for

all

-LSB-

-RRB-

Proof

The

fact

that

WCBW

-LRB-

N

k

-RRB-

q

follows

from

Theorem

Let

be

any

real

in

-LSB-

-RRB-

By

Lemma

the

solution

to

q

log

N

log

q

k

is

never

larger

than

the

solution

m

to

m

log

N

m

log

-LRB-

-RRB-

k

log

Since

solving

for

m

yields

m

log

N

k

log

log

up

-LRB-

N

k

-RRB-

proving

the

theorem

As

mentioned

above

when

N

the

worst

case

performance

of

WM

-LRB-

which

uses

the

best

choice

of

rather

than

the

minimizing

the

bound

-RRB-

equals

q

Furthermore

the

gap

between

these

two

and

m

grows

as

-LRB-

pk

-RRB-

If

N

is

large

compared

to

k

we

believe

that

the

upper

bound

m

is

much

closer

to

WCWM

-LRB-

N

k

-RRB-

However

even

when

N

is

large

q

can

be

significantly

less

than

m

Pick

any

k

If

N

k

N

k

values

are

chosen

to

make

the

algebra

tractable

rather

than

indicating

a

particular

region

of

interesting

behavior

then

q

With

a

bit

of

algebra

-LRB-

and

Stirling

's

approximation

-RRB-

it

can

be

shown

that

m

is

at

least

log

-LRB-

-RRB-

In

other

words

when

N

is

about

k

the

mistake

bound

on

BW

of

Theorem

is

at

least

log

-LRB-

-RRB-

better

than

the

best

known

bound

for

the

Weighted

Majority

algorithm

Although

our

bounds

on

the

BW

algorithm

are

better

than

the

up

-LRB-

N

k

-RRB-

bounds

on

the

WM

algorithm

asymptotically

the

two

bounds

have

the

same

leading

term

This

is

shown

in

the

following

section

Asymptotic

performance

of

the

algorithms

This

subsection

shows

that

both

BW

and

WM

are

asymptotically

optimal

in

the

worst

case

The

proof

uses

a

probabilistic

argument

to

show

the

existence

of

hard

sets

of

experts

Using

these

hard

sets

of

experts

an

adversary

can

force

any

prediction

algorithm

to

make

a

mistake

on

each

trial

proving

the

desired

lower

bound

We

use

the

notation

fi

gi

when

limi

fi

gi

We

define

the

following

functions

to

serve

as

a

lower

bounds

low

-LRB-

N

k

-RRB-

max

-LRB-

q

N

q

log

N

log

q

k

log

ln

q

k

-RRB-

Low

-LRB-

N

k

-RRB-

max

-LRB-

low

-LRB-

N

k

-RRB-

log

N

-RRB-

We

now

state

the

two

results

of

this

section

Theorem

For

any

integers

N

and

k

there

exists

a

set

E

of

N

experts

such

that

for

any

deterministic

master

algorithm

A

there

exists

a

sequence

s

of

trials

such

that

LE

-LRB-

s

-RRB-

k

and

A

makes

at

least

Low

-LRB-

N

k

-RRB-

mistakes

on

s

The

above

lower

bound

is

then

used

to

show

that

BW

and

WM

are

both

asymptotically

optimal

Theorem

For

any

sequence

f

-LRB-

Ni

ki

-RRB-

of

pairs

of

positive

integers

such

that

Ni

for

all

i

and

limi

Ni

or

limi

ki

Low

-LRB-

Ni

ki

-RRB-

WCBW

-LRB-

Ni

ki

-RRB-

WCWM

-LRB-

Ni

ki

-RRB-

up

-LRB-

Ni

ki

i

-RRB-

for

i

where

i

ki

up

-LRB-

Ni

ki

fi?i

-RRB-

ki

Before

proving

Theorem

we

need

some

definitions

and

lemmas

The

first

Lemma

is

from

Littlestone

and

Warmuth

Lemma

-LRB-

-LSB-

-RSB-

-RRB-

For

any

integers

N

and

k

there

exists

a

set

E

of

N

experts

such

that

for

any

deterministic

master

algorithm

A

there

exists

a

sequence

s

of

trials

such

that

LE

-LRB-

s

-RRB-

k

and

A

makes

at

least

log

N

mistakes

The

above

lemma

proves

the

first

lower

bound

used

in

the

definition

of

Low

The

second

lower

bound

is

proven

using

a

covering

argument

For

any

positive

integer

q

and

any

nonnegative

integer

k

q

a

k-covering

of

the

q-dimensional

boolean

hypercube

is

a

subset

B

of

such

that

for

any

v

there

is

a

p

B

such

that

dH

-LRB-

p

v

-RRB-

k

If

in

the

on-line

prediction

setting

the

expert

's

predictions

are

solely

a

function

of

the

trial

number

then

each

expert

can

be

viewed

as

a

sequence

of

bits

Furthermore

a

set

E

of

such

experts

is

a

k-covering

for

some

subset

tqg

of

trials

if

the

set

of

the

sequences

of

length

q

representing

the

predictions

of

the

experts

in

the

trials

tq

is

a

k-covering

of

Now

we

give

a

technical

lemma

showing

that

some

coverings

are

not

too

large

We

adapt

a

non-constructive

argument

of

Alon

and

Spencer

from

-LSB-

Theorem

p.

-RSB-

Master

Algorithms

for

Combining

the

Predictions

of

Experts

Lemma

Choose

N

and

k

and

let

m

low

-LRB-

N

k

-RRB-

Then

there

is

a

k-covering

of

of

size

at

most

N

Proof

We

prove

the

lemma

using

a

probabilistic

argument

Let

R

be

chosen

randomly

so

that

the

event

v

R

occurs

with

probability

p

-LRB-

to

be

specified

later

-RRB-

independently

for

any

v

Let

be

the

subset

of

containing

all

points

not

k-covered

by

R.

Clearly

R

-LSB-

is

a

k-covering

of

Observe

that

any

z

belongs

to

if

and

only

if

for

any

v

R

dH

-LRB-

z

v

-RRB-

k

This

implies

Pr

-LRB-

z

-RRB-

-LRB-

p

-RRB-

-LRB-

m

k

-RRB-

since

there

are

m

k

corners

of

the

m-dimensional

boolean

hypercube

with

Hamming

distance

at

most

k

from

z

-LRB-

z

itself

included

-RRB-

From

the

above

it

is

easy

to

compute

the

expectation

of

the

random

variable

jRj

E

-LSB-

jRj

-RSB-

-LRB-

p

-RRB-

-LRB-

m

k

-RRB-

Now

set

p

ln

-LRB-

m

k

-RRB-

-LRB-

m

k

-RRB-

Then

E

-LSB-

jRj

-RSB-

m

k

m

k

ln

m

k

m

k

-LRB-

m

k

-RRB-

ln

m

k

m

k

exp

ln

m

k

-LRB-

-RRB-

ln

m

k

m

k

where

inequality

-LRB-

-RRB-

holds

since

x

e?x

for

all

x

Thus

if

N

ln

-LRB-

m

k

-RRB-

-LRB-

m

k

-RRB-

then

the

m-dimensional

boolean

cube

is

k-covered

by

a

set

of

size

N

Solving

this

inequality

for

m

yields

that

m

lg

N

lg

m

k

lg

-LRB-

ln

m

k

-RRB-

or

equivalently

that

m

low

-LRB-

N

k

-RRB-

ensures

that

the

m-dimensional

boolean

cube

has

a

k-covering

of

size

N

Proof

of

Theorem

In

view

of

the

lower

bound

proven

in

Lemma

it

suffices

to

prove

a

second

lower

bound

of

low

-LRB-

N

k

-RRB-

mistakes

We

use

Lemma

to

do

this

Choose

a

sequence

of

distinct

observations

Choose

integers

N

and

k

Let

m

low

-LRB-

N

k

-RRB-

By

Lemma

there

exists

a

set

E

of

N

experts

whose

predictions

depend

only

on

the

trial

number

such

that

E

is

a

k-covering

for

the

first

m

prediction

trials

Now

notice

that

if

E

is

a

k-covering

for

the

first

m

trials

an

adversary

can

force

m

mistakes

on

any

deterministic

prediction

algorithm

The

adversary

simply

chooses

the

sequence

y

of

outcomes

of

length

m

such

that

yt

is

the

opposite

of

the

algorithm

's

prediction

on

the

tth

trial

Since

E

is

a

k-covering

of

for

any

such

sequence

y

of

outcomes

there

is

some

expert

in

E

which

makes

at

most

k

mistakes

on

-LRB-

-RRB-

-LRB-

xm

ym

-RRB-

Proof

of

Theorem

By

Theorem

we

know

that

Low

-LRB-

N

k

-RRB-

is

a

lower

bound

on

the

number

of

mistakes

for

any

deterministic

master

algorithm

Let

f

-LRB-

Ni

ki

-RRB-

be

a

sequence

as

in

the

statement

of

the

theorem

Since

by

Lemma

and

Theorem

Low

-LRB-

Ni

ki

-RRB-

WCBW

-LRB-

Ni

ki

-RRB-

up

-LRB-

Ni

ki

i

-RRB-

and

Low

-LRB-

Ni

ki

-RRB-

WCWM

-LRB-

Ni

ki

-RRB-

up

-LRB-

Ni

ki

i

-RRB-

it

is

sufficient

to

show

that

lim

i

Low

-LRB-

Ni

ki

-RRB-

up

-LRB-

Ni

ki

i

-RRB-

Suppose

for

contradiction

that

the

limit

does

not

hold

Since

Low

-LRB-

Ni

ki

-RRB-

up

-LRB-

Ni

ki

i

-RRB-

there

is

a

subsequence

f

-LRB-

N

i

-RRB-

of

such

that

limi

Low

-LRB-

N

i

-RRB-

up

-LRB-

N

i

fi?i

-RRB-

converges

to

some

constant

less

than

We

now

consider

two

cases

based

on

the

limiting

behavior

of

log

N

i

as

i

The

first

case

is

when

log

N

i

has

an

accumulation

point

at

zero

or

infinity

This

means

that

there

is

an

infinite

subsequence

f

-LRB-

N

-RRB-

of

such

that

limi

log

N

or

limi

log

N

In

either

case

we

use

the

upper

bound

on

the

function

up

proven

in

-LSB-

-RSB-

up

-LRB-

N

k

-RRB-

log

N

ln

N

-LRB-

-RRB-

to

get

lim

i

Low

-LRB-

N

-RRB-

up

-LRB-

N

i

-RRB-

lim

i

log

N

log

N

q

ln

N

lim

i

log

N

log

N

q

log

N

Since

is

a

subsequence

of

this

contradicts

the

assumption

that

fLow

-LRB-

N

i

-RRB-

up

-LRB-

N

i

i

-RRB-

g

converges

to

a

constant

strictly

less

than

For

the

other

case

we

assume

that

there

are

positive

constants

a

and

b

such

that

a

log

N

i

b

-LRB-

-RRB-

for

all

i.

Thus

both

N

i

and

go

to

infinity

For

the

remainder

of

the

proof

we

only

deal

with

the

sequence

f

-LRB-

N

i

-RRB-

and

thus

we

can

simplify

our

notation

by

dropping

the

primes

Let

m?i

denote

up

-LRB-

Ni

ki

i

-RRB-

Recall

from

Lemma

that

m?i

and

that

m?i

is

the

largest

real

solution

to

the

equation

x

log

Ni

xH

-LRB-

ki

x

-RRB-

Similarly

define

mi

as

the

largest

real

solution

of

the

equation

x

log

Ni

log

x

ki

log

ln

x

ki

We

will

now

show

that

mi

as

well

Observe

that

when

x

is

very

large

x

is

larger

than

log

Ni

log

x

ki

log

ln

x

ki

Also

as

log

Ni

ki

b

we

have

that

for

large

enough

i

log

Ni

log

ki

log

ln

ki

proving

that

mi

-LRB-

-RRB-

Finally

define

mi

as

the

maximum

of

log

Ni

and

mi

Note

that

mi

is

within

of

Low

-LRB-

Ni

ki

-RRB-

As

we

are

interested

in

asymptotics

we

use

mi

instead

of

Low

-LRB-

Ni

ki

-RRB-

In

addition

mi

mi

m?i

-LRB-

-RRB-

Master

Algorithms

for

Combining

the

Predictions

of

Experts

and

by

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

mi

log

Ni

pki

log

Ni

ki

a

a

-LRB-

-RRB-

Since

ki

for

i

it

follows

from

Inequality

-LRB-

-RRB-

that

mi

as

well

We

now

examine

the

asymptotic

behavior

of

mi

in

more

detail

mi

log

Ni

log

mi

ki

log

ln

mi

ki

log

Ni

log

mi

ki

log

ln

mi

ki

log

Ni

log

mi

ki

log

Ni

log

mi

ki

log

Ni

log

mi

ki

o

-LRB-

-RRB-

log

Ni

log

mi

ki

since

mi

-LRB-

o

-LRB-

-RRB-

-RRB-

log

Ni

log

mi

ki

-LRB-

-RRB-

-LRB-

o

-LRB-

-RRB-

-RRB-

log

Ni

miH

ki

mi

-LRB-

-RRB-

The

last

step

in

the

above

uses

the

equality

log

-LRB-

m

k

-RRB-

mH

-LRB-

k

m

-RRB-

log

m

O

-LRB-

-RRB-

-LRB-

see

-LSB-

-RSB-

exercise

-RRB-

and

the

fact

that

H

-LRB-

ki

mi

-RRB-

is

lower

bounded

by

a

constant

when

i

is

large

-LRB-

equations

-LRB-

-RRB-

and

-LRB-

-RRB-

-RRB-

Let

fi

-LRB-

x

-RRB-

log

Ni

xH

-LRB-

ki

x

-RRB-

From

the

definition

of

m?i

we

know

that

m?i

fi

-LRB-

m?i

-RRB-

Equation

-LRB-

-RRB-

means

that

for

any

ffl

there

exists

some

iffl

such

that

for

all

i

iffl

mi

-LRB-

ffl

-RRB-

fi

-LRB-

mi

-RRB-

Recall

that

mi

mi

m?i

We

need

to

show

that

mi

m?i

To

do

this

we

first

uniformly

bound

the

derivatives

of

the

functions

fi

-LRB-

x

-RRB-

in

some

ranges

Notice

that

f

i

-LRB-

x

-RRB-

log

-LRB-

x

-LRB-

x

ki

-RRB-

-RRB-

Thus

for

all

x

log

Ni

f

i

-LRB-

x

-RRB-

log

log

Ni

ki

log

Ni

log

-LRB-

ki

log

Ni

-RRB-

Since

ki

log

Ni

a

we

get

that

f

i

-LRB-

x

-RRB-

c

for

some

c

independent

of

i.

Using

the

mid-point

theorem

we

can

lower

bound

fi

-LRB-

mi

-RRB-

in

the

following

way

fi

-LRB-

mi

-RRB-

fi

-LRB-

m?i

-RRB-

f

i

-LRB-

-RRB-

-LRB-

m?i

mi

-RRB-

for

some

mi

m?i

Using

the

bound

on

the

derivative

we

get

that

fi

-LRB-

mi

-RRB-

fi

-LRB-

m?i

-RRB-

-LRB-

c

-RRB-

-LRB-

m?i

mi

-RRB-

c

-LRB-

m?i

mi

-RRB-

mi

-LRB-

-RRB-

On

the

other

hand

mi

-LRB-

ffl

-RRB-

fi

-LRB-

mi

-RRB-

and

f

i

-LRB-

x

-RRB-

for

all

x

As

mi

mi

-LRB-

Equation

-LRB-

-RRB-

-RRB-

we

get

that

fi

-LRB-

mi

-RRB-

-LRB-

ffl

-RRB-

mi

-LRB-

-RRB-

Combining

Equations

-LRB-

-RRB-

and

-LRB-

-RRB-

we

get

that

c

-LRB-

m?i

mi

-RRB-

mi

-LRB-

ffl

-RRB-

mi

which

implies

that

m?i

mi

-LRB-

c

ffl

-RRB-

c

As

we

can

choose

ffl

arbitrarily

small

we

get

that

mi

m?i

Lower

bounds

based

on

Ulam

's

game

In

this

section

we

give

lower

bounds

on

the

performance

of

prediction

strategies

We

show

that

for

any

fixed

number

of

mistakes

k

of

the

best

expert

and

for

any

prediction

algorithm

there

exists

a

set

E

of

experts

and

a

sequence

s

s.t.

k

LE

-LRB-

s

-RRB-

for

which

the

number

of

mistakes

made

by

the

prediction

algorithm

is

at

least

as

large

as

the

number

of

mistakes

made

by

BW

We

start

by

introducing

some

notation

that

lets

us

give

a

precise

statement

of

our

lower

bound

We

then

describe

Ulam

's

game

with

lies

and

its

relation

to

our

prediction

problem

Finally

we

show

how

Spencer

's

results

-LSB-

-RSB-

can

be

used

to

prove

our

lower

bound

In

all

of

the

following

discussion

we

shall

think

of

k

the

upper

bound

on

the

number

of

mistakes

made

by

the

best

expert

as

being

fixed

Let

J

-LRB-

k

q

-RRB-

be

the

following

sequence

of

numbers

indexed

by

q

J

-LRB-

k

q

-RRB-

q

k

It

is

easy

to

check

that

J

-LRB-

k

q

-RRB-

-LRB-

-RRB-

J

-LRB-

k

q

-RRB-

for

any

q

thus

the

sequence

J

-LRB-

k

q

-RRB-

increases

-LRB-

at

least

-RRB-

exponentially

Theorem

For

any

integer

k

there

exists

an

integer

Nk

such

that

for

any

N

Nk

the

following

holds

If

q

is

the

integer

such

that

J

-LRB-

k

q

-RRB-

N

J

-LRB-

k

q

-RRB-

then

WCBW

-LRB-

N

k

-RRB-

WC

-LRB-

N

k

-RRB-

If

J

-LRB-

k

q

-RRB-

N

WCBW

-LRB-

N

k

-RRB-

WC

-LRB-

N

k

-RRB-

Observe

that

the

upper

bound

on

algorithm

BW

is

always

guaranteed

to

be

within

one

mistake

of

the

optimal

algorithm

when

N

is

large

enough

Also

since

the

size

of

the

segment

J

-LRB-

k

q

-RRB-

N

J

-LRB-

k

q

-RRB-

increases

exponentially

with

q

as

q

increases

the

the

set

of

values

for

N

where

the

second

case

holds

-LRB-

i.e.

the

lower

bound

is

off

by

from

BW

's

upper

bound

-RRB-

becomes

an

insignificantly

small

fraction

of

the

possible

values

for

N

This

shows

that

BW

is

very

close

to

optimal

for

large

values

of

N

The

gap

of

when

N

J

-LRB-

k

q

-RRB-

arises

from

complicated

GCD

considerations

In

the

appendix

we

show

how

algorithm

BW

can

be

modified

so

that

it

is

completely

optimal

for

large

N

The

weakness

of

this

lower

bound

construction

is

that

the

threshold

Nk

above

which

the

lower

bound

holds

is

rather

large

on

the

order

of

This

double-exponential

dependence

on

k

arises

from

our

use

of

Spencer

's

results

-LSB-

-RSB-

Before

we

give

the

proof

of

Theorem

we

briefly

describe

Ulam

's

game

with

a

fixed

number

of

lies

and

show

how

this

game

relates

to

chip

games

and

to

the

problem

of

combining

the

predictions

of

experts

In

the

searching

game

introduced

by

Ulam

-LRB-

see

-LSB-

-RSB-

-RRB-

there

are

two

players

a

chooser

-LRB-

also

called

Carol

-RRB-

and

a

partitioner

-LRB-

also

called

Paul

-RRB-

A

game

is

defined

by

three

nonnegative

integers

N

k

and

q

that

are

known

to

both

players

Carol

is

assumed

to

select

a

secret

number

x

from

the

set

Ng

Paul

's

goal

is

to

find

out

what

this

number

is

by

asking

Carol

questions

of

the

form

Is

x

in

S

where

S

is

any

subset

of

Ng

Carol

is

required

to

answer

either

yes

or

no

However

she

is

allowed

to

lie

-LRB-

i.e.

give

the

incorrect

answer

to

Paul

's

question

-RRB-

up

to

k

times

We

say

that

Paul

wins

the

-LRB-

N

k

q

-RRB-

important

point

is

that

Carol

does

not

have

to

commit

to

a

specific

number

x

ahead

of

time

The

requirement

is

only

that

her

choice

of

answers

be

such

that

at

all

times

the

exists

x

Ng

which

is

consistent

with

all

but

at

most

k

of

her

answers

Master

Algorithms

for

Combining

the

Predictions

of

Experts

game

if

and

only

if

he

can

always

identify

Carol

's

secret

number

after

at

most

q

questions

regardless

of

Carol

's

strategy

The

interesting

fact

is

that

there

is

a

common

abstraction

of

Ulam

's

game

with

lies

and

of

our

problem

The

abstraction

can

be

seen

as

the

following

chip

game

-LRB-

for

more

work

on

chip

games

see

-LSB-

-RSB-

-RRB-

We

think

of

each

number

in

the

set

Ng

as

a

chip

and

consider

k

-LRB-

disjoint

-RRB-

subsets

of

these

chips

which

we

call

bins

and

denote

by

Bk

At

each

point

of

the

game

the

bin

Bj

contains

all

the

chips

that

correspond

to

a

number

x

Ng

with

the

property

that

if

x

is

the

number

chosen

by

Carol

then

j

of

the

answers

that

Carol

gave

so

far

have

been

lies

Thus

the

union

of

all

the

bins

contain

those

choices

of

x

which

are

consistent

with

the

bound

k

on

the

number

of

lies

that

Carol

is

allowed

to

make

Essentially

it

is

sufficient

to

describe

each

configuration

reached

during

the

game

by

the

number

of

chips

in

each

bin

We

denote

by

Ij

-LRB-

Ijk

-RRB-

the

configuration

of

the

chip

game

after

at

the

jth

trial

where

Iji

is

a

natural

number

which

denotes

the

number

of

chips

in

Bi

For

example

the

initial

configuration

is

always

-LRB-

N

-RRB-

When

Paul

asks

Is

x

in

S

his

question

partitions

the

chips

into

two

sets

those

in

S

versus

those

outside

S

If

Carol

answers

no

her

answer

constitutes

a

lie

with

respect

to

the

numbers

in

S

This

translates

to

advancing

each

chip

corresponding

to

a

number

in

S

from

its

current

bin

to

the

next

bin

-LRB-

e.g.

from

bin

Bj

to

Bj

-RRB-

If

a

chip

corresponding

to

a

number

in

S

is

already

in

the

last

bin

Bk

it

is

discarded

as

there

is

no

bin

Bk

If

Carol

answers

yes

then

those

chips

corresponding

to

numbers

not

in

S

are

advanced

Clearly

Paul

can

not

know

which

number

Carol

has

chosen

as

long

as

the

union

of

the

bins

contains

at

least

two

chips

Thus

Carol

's

goal

is

to

keep

two

chips

in

the

union

of

the

bins

for

as

long

as

possible

Paul

wins

the

-LRB-

N

k

q

-RRB-

iff

there

is

a

strategy

for

choosing

partitions

which

guarantees

that

after

q

steps

there

is

at

most

one

chip

remaining

in

the

union

of

the

bins

We

can

think

of

the

prediction

problem

as

a

prediction

game

where

the

predictor

is

playing

against

an

adversary

which

picks

both

the

predictions

generated

by

the

experts

and

the

outcomes

We

restrict

our

attention

to

those

adversary

strategies

which

force

the

prediction

algorithm

to

make

a

mistake

on

each

and

every

trial

for

as

long

as

possible

This

means

until

one

expert

has

made

k

mistakes

and

every

other

expert

has

made

more

than

k

mistakes

the

adversary

chooses

the

feedback

so

that

the

prediction

algorithm

makes

a

mistake

on

every

trial

From

this

point

on

the

predictions

of

the

single

best

expert

are

guaranteed

to

be

without

mistakes

and

by

copying

the

predictions

of

this

expert

the

master

algorithm

will

correctly

predict

the

remainder

of

the

sequence

This

restriction

is

helpful

to

map

to

the

prediction

game

into

a

chip

game

and

restricting

the

adversary

in

this

way

does

not

reduce

its

power

since

we

are

able

to

obtain

a

lower

bound

that

essentially

matches

the

upper

bound

of

the

BW

algorithm

We

can

easily

relate

this

prediction

game

to

a

chip

game

Each

chip

corresponds

to

an

expert

and

the

bin

Bj

for

j

k

contains

those

chips

corresponding

to

experts

which

have

made

exactly

j

mistakes

on

previous

trials

Each

iteration

of

the

game

starts

with

the

adversary

partitioning

the

chips

to

two

sets

according

to

the

predictions

given

by

the

corresponding

experts

The

prediction

algorithm

then

chooses

its

prediction

and

the

adversary

forces

a

mistake

by

generating

an

outcome

opposite

to

the

prediction

This

this

section

we

completely

ignore

the

instances

xt

that

are

given

as

inputs

to

the

experts

Because

we

are

dealing

with

worst

case

lower

bounds

we

can

assume

that

for

any

S

E

there

is

always

an

observation

xS

X

that

causes

the

experts

in

S

to

predict

and

the

experts

not

in

S

to

predict

Thus

the

adversary

can

control

the

predictions

of

the

experts

by

choosing

the

appropriate

observation

causes

those

chips

corresponding

to

experts

whose

predictions

were

mistaken

to

advance

one

bin

Thus

the

prediction

algorithm

-LRB-

indirectly

-RRB-

chooses

which

subset

of

the

chips

gets

advanced

so

the

prediction

algorithm

corresponds

to

Carol

and

the

adversary

corresponds

to

Paul

The

game

ends

when

the

configuration

-LRB-

-RRB-

is

reached

we

shall

refer

to

this

configuration

as

the

terminal

configuration

This

is

a

slight

difference

from

the

chip

game

that

corresponds

to

Ulam

's

game

with

k

lies

Another

much

more

significant

difference

is

that

the

goals

of

the

opponents

have

been

reversed

In

the

chip

game

corresponding

to

the

prediction

problem

Carol

-LRB-

the

prediction

algorithm

-RRB-

wants

to

shorten

the

game

as

much

as

possible

since

the

length

of

the

game

measures

the

number

of

mistakes

that

the

prediction

algorithm

is

forced

to

make

As

the

goals

of

Carol

and

Paul

have

been

reversed

it

would

seem

that

their

strategies

for

playing

the

two

games

would

be

very

different

Surprisingly

it

turns

out

that

the

optimal

strategy

for

Paul

is

the

same

in

the

two

games

when

the

different

ending

condition

is

ignored

If

N

Nk

then

this

optimal

strategy

Paul

can

force

both

games

to

have

the

same

length

regardless

of

the

actions

taken

by

Carol

In

other

words

if

Paul

uses

this

strategy

then

Carol

is

unable

to

make

the

game

either

longer

or

shorter

This

strategy

for

Paul

has

been

developed

by

Spencer

-LSB-

-RSB-

and

is

the

basis

of

the

proof

of

Theorem

We

shall

briefly

describe

the

strategy

give

a

result

of

Spencer

-LSB-

-RSB-

and

then

use

it

to

prove

Theorem

Spencer

identifies

the

same

binomial

weights

that

are

used

in

the

BW

algorithm

as

the

central

quantities

on

which

the

strategies

of

both

Carol

and

Paul

are

based

We

shall

denote

by

Wq

-LRB-

I

-RRB-

the

weight

associated

with

the

configuration

I

and

the

integer

q

i.e.

Wq

-LRB-

I

-RRB-

kX

i

Ii

q

k

i

Spencer

-LSB-

-RSB-

gives

a

strategy

for

Carol

Under

this

strategy

Carol

advances

those

chips

that

keep

the

future

configurations

as

heavy

as

possible

The

exact

opposite

choice

is

made

by

the

BW

algorithm

which

advances

the

heavier

chips

resulting

in

a

lighter

configuration

This

makes

intuitive

sense

because

Carol

has

the

opposite

goal

in

the

two

games

The

main

result

of

Spencer

's

paper

-LSB-

-RSB-

is

a

proof

that

when

N

is

large

enough

Paul

can

always

partition

the

chips

in

such

a

way

that

both

future

configurations

have

equal

weight

It

thus

completely

neutralizes

Carol

The

construction

of

the

strategy

is

based

on

the

observation

that

the

weight

associated

with

the

chips

in

bin

Bk

is

always

one

because

q

These

chips

are

appropriately

referred

to

as

pennies

It

is

clear

that

if

a

configuration

has

a

sufficient

amount

of

pennies

and

the

total

weight

is

even

then

by

moving

pennies

from

one

set

of

the

partition

to

the

other

one

can

equalize

the

weight

of

the

two

successor

configurations

Paul

's

strategy

is

to

choose

a

partition

whose

two

successor

configurations

are

almost

balanced

and

then

use

pennies

to

balance

them

completely

The

main

theorem

in

Spencer

's

paper

shows

that

given

appropriate

initial

conditions

Paul

can

use

this

technique

repeatedly

until

a

configuration

that

has

only

a

single

chip

in

the

union

of

all

the

bins

is

reached

We

now

give

the

main

result

from

Spencer

's

paper

-LSB-

-RSB-

in

a

form

that

fits

our

needs

here

Theorem

-LRB-

-LSB-

-RSB-

-RRB-

If

k

is

the

number

of

bins

then

there

exist

finite

integers

c

-LRB-

k

-RRB-

and

-LRB-

k

-RRB-

such

that

the

following

holds

for

any

q

-LRB-

k

-RRB-

If

-LRB-

-RRB-

is

an

initial

configuration

such

that

c

-LRB-

k

-RRB-

qk

and

Wq

-LRB-

-RRB-

then

there

exists

a

strategy

for

Paul

such

that

independent

of

the

choices

made

by

Carol

a

configuration

Im

is

reached

such

that

Pki

Imi

and

Wq?m

-LRB-

Im

-RRB-

m.

Master

Algorithms

for

Combining

the

Predictions

of

Experts

In

other

words

Paul

can

guarantee

that

the

total

weight

is

exactly

halved

at

each

step

until

only

a

single

chip

is

left

Proof

of

Theorem

The

proof

is

divided

into

two

parts

we

first

show

that

if

N

is

large

enough

then

from

the

initial

configuration

-LRB-

N

-RRB-

Paul

can

reach

in

k

steps

a

configuration

which

meets

the

conditions

of

Theorem

In

the

second

part

we

show

that

the

final

configuration

reached

in

Theorem

guarantees

the

bound

given

in

the

theorem

In

the

proof

we

make

use

of

the

idea

that

Paul

marks

chips

as

useless

If

a

chip

is

marked

on

some

particular

trial

then

this

chip

is

placed

arbitrarily

in

the

partitions

generated

by

Paul

on

subsequent

trials

We

shall

prove

that

Paul

can

delay

reaching

a

terminal

configuration

even

when

only

the

unmarked

chips

are

considered

It

is

clear

that

if

the

marked

chips

were

also

considered

then

reaching

the

terminal

configuration

would

be

delayed

for

at

least

as

long

which

proves

the

lower

bound

on

the

number

of

trials

Initially

all

N

chips

are

in

bin

It

takes

at

least

k

steps

to

get

chips

to

bin

Bk

and

thus

make

them

into

pennies

We

shall

devise

a

strategy

for

the

first

k

trials

that

is

guaranteed

to

give

rise

to

a

sufficient

number

of

pennies

at

the

kth

trial

First

Paul

marks

some

chips

so

as

to

make

the

number

of

unmarked

chips

divisible

by

Clearly

less

than

chips

need

to

be

marked

Ignoring

the

marked

chips

Paul

generates

the

following

partitions

The

-LRB-

unmarked

-RRB-

chips

in

each

bin

are

divided

into

two

equal

parts

one

part

from

each

bin

is

placed

in

the

first

set

of

the

partition

and

the

other

part

is

placed

in

the

second

It

is

easy

to

check

that

independently

of

Carol

's

actions

such

partitioning

of

the

unmarked

chips

is

possible

for

k

steps

It

is

also

simple

to

see

that

after

k

trials

exactly

a

fraction

of

k

of

the

unmarked

chips

reach

bin

Bk

and

become

pennies

Let

q

be

the

integer

such

that

J

-LRB-

k

q

-RRB-

N

J

-LRB-

k

q

-RRB-

FromEquation

-LRB-

-RRB-

it

is

clear

that

the

weight

that

is

associated

with

the

unmarked

chips

is

divided

by

two

at

each

step

Thus

independently

of

Carol

's

choices

the

weight

of

the

configuration

after

k

steps

satisfies

Wq?k

-LRB-

Ik

-RRB-

k

-LRB-

N

-RRB-

Wq

-LRB-

-RRB-

k

-LRB-

N

-RRB-

q

k

-LRB-

-RRB-

To

apply

Theorem

we

need

that

the

remaining

weight

-LRB-

after

k

steps

-RRB-

of

the

unmarked

chips

is

a

power

of

two

We

first

find

an

appropriate

q

such

that

W

q

-LRB-

Ik

-RRB-

q.

By

the

definition

of

q

J

-LRB-

k

q

-RRB-

N

J

-LRB-

k

q

-RRB-

If

N

is

large

enough

then

J

-LRB-

k

q

-RRB-

J

-LRB-

k

q

-RRB-

and

thus

N

J

-LRB-

k

q

-RRB-

This

implies

that

-LRB-

N

-RRB-

q

k

and

thus

by

inequality

-LRB-

-RRB-

Wq?k

-LRB-

Ik

-RRB-

k

It

follows

that

if

N

is

large

enough

then

we

can

always

choose

q

q

k

However

if

N

J

-LRB-

k

q

-RRB-

then

by

the

same

derivation

we

get

Wq?k

-LRB-

Ik

-RRB-

k

and

we

can

set

q

q

k

We

now

wish

to

apply

the

results

of

Theorem

to

the

configuration

Ik

whose

weight

satisfies

W

q

q.

However

in

order

to

obey

the

conditions

of

the

theorem

we

have

to

mark

some

more

chips

in

order

to

make

the

weight

of

the

configuration

satisfy

W

q

-LRB-

Ik

-RRB-

q

We

do

this

marking

carefully

so

that

afterwards

we

still

have

enough

unmarked

pennies

to

apply

the

theorem

We

mark

chips

using

the

following

simple

procedure

we

mark

nonpenny

chips

until

we

can

not

mark

a

non-penny

chip

without

reducing

W

q

-LRB-

I

-RRB-

below

q

We

then

mark

enough

pennies

to

reduce

the

weight

to

q

As

the

heaviest

chips

-LRB-

those

in

-RRB-

weigh

q

k

-LRB-

q

-RRB-

k

we

need

to

mark

at

most

-LRB-

q

-RRB-

k

pennies

Taking

into

account

both

the

initial

marking

of

less

than

chips

and

this

additional

marking

phase

we

get

that

the

number

of

unmarked

pennies

is

at

least

-LRB-

N

-RRB-

c

-LRB-

q

-RRB-

k

kN

-LRB-

q

-RRB-

k

On

the

other

hand

in

order

to

apply

Theorem

we

need

at

least

c

-LRB-

k

-RRB-

qk

unmarked

pennies

This

is

satisfied

if

kN

-LRB-

q

-RRB-

k

c

-LRB-

k

-RRB-

qk

As

for

any

fixed

value

of

k

q

and

thus

q

is

O

-LRB-

log

N

-RRB-

this

condition

is

satisfied

for

every

N

Nk

for

a

large

enough

Nk

We

can

thus

apply

Theorem

with

the

initial

configuration

being

the

unmarked

chips

in

the

kth

configuration

which

we

denote

by

Ik

The

weight

of

this

configuration

is

W

q

-LRB-

Ik

-RRB-

q

The

theorem

guarantees

that

Paul

can

find

partitions

so

that

after

some

m

steps

a

configuration

Ik

m

is

reached

such

that

Pki

Ik

m

i

and

W

q?m

-LRB-

Im

-RRB-

q?m

Thus

only

a

single

chip

will

be

left

It

is

easy

to

verify

that

as

the

weight

of

the

chip

is

q?m

it

must

be

in

bin

Bk

-LRB-

q?m

-RRB-

After

another

q

m

steps

the

single

chip

will

be

in

the

last

bin

and

the

game

is

over

Finally

we

sum

up

the

number

of

trials

or

mistakes

that

Paul

can

force

on

Carol

We

have

k

trials

before

getting

the

pennies

m

trials

using

the

Spencer

's

strategy

and

q

m

mistakes

at

the

end

Summing

these

terms

and

using

the

definition

of

q

we

get

that

Paul

can

always

force

at

least

q

mistakes

and

if

N

J

-LRB-

k

q

-RRB-

then

Paul

can

force

at

least

q

mistakes

Conversion

strategies

In

this

section

we

show

how

the

ideas

behind

the

BW

algorithm

can

be

used

to

modify

prediction

algorithms

so

that

they

can

tolerate

malicious

noise

Assume

we

are

given

a

prediction

algorithm

A

that

makes

at

most

k

mistakes

on

any

sequence

in

some

set

-LRB-

X

-RRB-

We

assume

that

algorithm

A

makes

at

most

k

mistakes

even

if

it

is

presented

with

a

subsequence

of

any

sequence

in

Formally

we

require

that

is

subsequence

closed

Any

deterministic

prediction

algorithm

can

be

converted

to

an

algorithm

that

changes

its

state

only

on

when

its

prediction

is

incorrect

This

is

achieved

by

resetting

the

state

of

A

after

each

trial

in

which

A

predicts

correctly

to

the

state

of

A

before

the

trial

This

conversion

does

not

increase

the

worst

case

number

of

mistakes

on

the

subsequence

closed

set

The

converted

algorithm

is

called

conservative

-LRB-

see

-LSB-

-RSB-

-RRB-

For

the

rest

of

this

section

we

shall

always

assume

that

the

set

of

sequences

is

subsequence

closed

and

that

the

prediction

algorithm

is

conservative

Algorithm

A

is

allowed

to

perform

arbitrarily

badly

if

given

an

instance/outcome

sequence

that

is

not

in

For

example

if

-LRB-

X

-RRB-

-LSB-

-LRB-

X

-RRB-

-LRB-

i.e.

all

sequences

where

the

outcome

is

held

constant

-RRB-

then

the

algorithm

A

which

always

predicts

with

the

first

outcome

seen

makes

at

most

one

mistake

when

given

a

sequence

in

However

if

the

first

label

is

corrupted

by

malicious

noise

then

all

subsequent

predictions

made

by

algorithm

A

will

be

incorrect

Here

we

show

how

to

convert

A

into

another

algorithm

which

performs

well

on

sequences

in

which

are

corrupted

by

noise

In

particular

for

any

r

we

can

build

an

algorithm

which

performs

well

on

those

sequences

which

can

be

created

from

a

sequence

in

by

arbitrarily

changing

up

to

r

examples

We

use

to

denote

this

set

of

noisy

sequences

As

the

above

example

indicates

algorithm

A

may

make

arbitrarily

many

mistakes

on

sequences

in

Furthermore

the

sequences

in

might

have

different

outcomes

for

the

same

instance

and

algorithm

A

might

not

even

be

defined

on

this

larger

set

of

sequences

We

use

the

methods

developed

in

Section

to

construct

master

algorithms

called

conversion

strategies

whose

mistake

bounds

increase

slowly

as

a

function

of

r

As

in

Section

we

use

a

version

space

argument

and

expand

A

into

a

set

of

variants

so

that

at

least

one

variant

will

be

correct

on

all

trials

where

the

conversion

strategy

makes

Conversion

strategies

a

mistake

However

here

the

elements

of

the

version

space

are

somewhat

dynamic

as

they

represent

computations

of

A

on

sequences

in

In

addition

to

discarding

irrelevant

computations

from

the

version

space

the

conversion

strategy

will

also

need

to

extend

certain

computations

by

simulating

A

on

the

current

trial

Since

the

members

of

the

version

space

managed

by

the

conversion

strategy

are

somewhat

dynamic

it

may

be

a

slight

misnomer

to

call

it

a

version

space

However

version

space

does

convey

the

proper

intuition

Since

our

conversion

strategies

are

conservative

we

can

concentrate

on

those

trials

where

the

conversion

strategy

itself

makes

mistakes

Here

we

use

m

for

a

bound

on

the

number

of

mistakes

made

by

the

conversion

strategy

k

to

denote

the

mistake

bound

of

algorithm

A

on

sequences

in

and

r

as

the

number

of

examples

corrupted

by

noise

We

first

outline

the

Cbin

conversion

strategy

which

is

based

on

binomial

weights

and

later

describe

a

second

conversion

strategy

Cexp

based

on

exponential

weights

These

strategies

are

described

in

more

detail

in

in

Sections

and

respectively

A

major

difference

between

the

conversion

problem

discussed

here

and

the

one

addressed

in

Section

is

that

with

experts

there

were

only

two

possibilities

for

each

trial

the

expert

was

either

correct

or

incorrect

Here

we

consider

three

different

cases

The

first

case

is

when

algorithm

A

correctly

predicts

the

outcome

In

the

other

two

cases

the

prediction

is

incorrect

In

the

second

case

the

wrong

prediction

is

due

to

the

fact

that

the

example

is

corrupted

by

noise

and

in

the

third

case

the

example

is

unchanged

but

the

algorithm

makes

a

mistake

in

predicting

the

label

Therefore

instead

of

associating

a

bit

string

to

each

member

of

the

version

space

the

Cbin

strategy

attaches

a

string

of

trits

from

the

set

noise

mstk

g.

Each

member

of

the

version

space

is

a

stored

state

of

algorithm

A

together

with

a

string

o

-LRB-

om

-RRB-

noise

mstk

gm

These

strings

have

an

interpretation

like

the

bit

strings

of

Section

If

a

-LRB-

state

o

-RRB-

pair

is

in

the

version

space

when

the

conversion

strategy

Cbin

makes

its

ith

mistake

then

the

value

of

oi

represent

the

following

possibilities

The

value

represents

the

possibility

that

A

predicted

the

label

of

the

example

correctly

The

values

noise

and

mstk

represent

the

possibility

that

A

predicted

incorrectly

where

the

cause

for

the

incorrect

prediction

is

attributed

to

noise

or

to

a

mistake

by

A

respectively

Since

algorithm

A

makes

at

most

k

mistakes

each

string

o

contains

mstk

at

most

k

times

Similarly

since

we

assume

that

at

most

r

of

the

trials

are

corrupted

by

noise

noise

appears

at

most

r

times

in

each

string

Therefore

only

some

of

the

strings

in

noise

mstk

gm

are

legitimate

In

particular

if

there

are

j

non-zero

elements

in

a

string

j

will

be

between

and

r

k.

Furthermore

at

most

r

and

at

least

j

k

of

the

elements

in

the

string

will

be

noise

This

gives

us

size

r

k

X

j

m

j

j

r

j

j

k

strings

that

must

be

considered

An

examination

of

the

term

in

brackets

shows

that

size

is

symmetric

in

r

and

k

as

expected

The

Cbin

conversion

strategy

starts

with

a

version

space

containing

size

elements

each

with

the

initial

state

of

algorithm

A

and

a

different

legitimate

string

o

The

conversion

strategy

manages

the

version

space

by

predicting

with

the

halving

algorithm

However

it

is

no

longer

quite

so

clear

what

this

means

Consider

the

situation

after

the

conversion

strategy

Cbin

has

made

i

mistakes

and

sees

instance

x

X

In

this

case

each

element

of

the

version

space

-LRB-

state

o

-RRB-

will

be

using

its

oi

to

see

if

its

variant

of

A

is

correct

has

a

noisy

trial

or

makes

a

mistake

Each

variant

will

see

how

A

-LRB-

in

state

state

-RRB-

predicts

If

its

oi

is

then

the

variant

predicts

the

same

way

otherwise

the

variant

predicts

with

the

opposite

value

Conversion

strategy

Cbin

may

update

the

version

space

after

getting

the

outcome

If

the

conversion

strategy

Cbin

predicted

correctly

then

all

variants

are

kept

unchanged

If

Cbin

predicted

incorrectly

then

those

variants

also

predicting

incorrectly

are

discarded

In

addition

when

Cbin

predicts

incorrectly

those

variants

predicting

correctly

may

be

updated

based

on

their

oi

values

There

are

three

cases

according

to

the

value

of

oi

Case

oi

This

means

that

the

variant

predicted

the

outcome

correctly

Since

A

is

conservative

Cbin

leaves

the

state

of

the

algorithm

A

for

this

variant

unchanged

Case

oi

noise

This

means

that

the

prediction

of

A

is

incorrect

but

would

have

been

correct

if

the

example

was

not

corrupted

by

noise

As

in

the

previous

case

Cbin

leaves

the

state

of

the

algorithm

A

unchanged

Case

oi

mstk

This

means

that

the

prediction

of

A

is

incorrect

because

A

has

made

one

of

its

k

allowed

mistakes

and

that

the

example

is

not

corrupted

by

noise

In

this

case

Cbin

updates

the

state

of

A

This

is

done

by

simulating

A

starting

from

the

old

state

on

the

example

received

in

the

current

trial

The

resulting

state

of

A

replaces

the

old

state

in

the

variant

We

show

in

Lemma

that

On

each

trial

where

Cbin

makes

a

mistake

the

size

of

the

version

space

drops

by

a

factor

of

at

least

For

any

sequence

in

at

least

one

variant

is

never

removed

from

the

version

space

during

the

run

of

the

master

algorithm

We

need

a

few

definitions

before

we

can

precisely

state

our

bounds

on

the

Cbin

conversion

strategy

For

all

n

N

and

for

all

pairs

s

-LRB-

-RRB-

-LRB-

yn

-RRB-

and

u

-LRB-

-RRB-

-LRB-

xn

zn

-RRB-

of

sequences

in

-LRB-

X

-RRB-

n

we

say

that

s

is

a

r-corrupted

version

of

u

if

and

only

if

-LRB-

xi

yi

-RRB-

-LRB-

-RRB-

for

at

most

r

indices

i

where

i

n

We

shall

also

use

the

notation

dC

-LRB-

s

u

-RRB-

r

to

indicate

that

s

is

an

r-corrupted

version

of

u

We

define

dC

-LRB-

s

u

-RRB-

if

the

sequences

differ

in

length

or

if

they

have

an

infinite

number

of

disagreements

We

will

show

in

Section

that

the

conversion

strategy

Cbin

achieves

the

following

bound

Theorem

Choose

a

subsequence-closed

set

-LRB-

X

-RRB-

of

sequences

Choose

a

conservative

deterministic

prediction

algorithm

A

such

that

for

some

k

N

LA

-LRB-

u

-RRB-

k

for

all

u

Choose

r

N

and

s

-LRB-

X

-RRB-

such

that

s

is

a

r-corrupted

version

of

some

sequence

u

in

Then

the

number

of

mistakes

made

by

Cbin

-LRB-

r

k

A

-RRB-

on

the

sequence

s

is

at

most

max

-LRB-

q

N

q

log

r

k

X

i

m

i

i

r

i

i

k

-RRB-

-LRB-

-RRB-

Note

that

the

Cbin

strategy

needs

to

know

the

upper

bounds

k

and

r

In

Section

we

describe

a

second

conversion

strategy

which

we

call

the

Cexp

strategy

The

Cexp

strategy

uses

exponential

weights

-LRB-

as

used

in

the

Weighted

Majority

algorithm

-RRB-

and

does

not

require

advance

knowledge

of

r

and

k

However

one

can

not

optimize

the

mistake

bounds

of

Cexp

without

knowing

these

parameters

The

following

theorem

gives

the

mistake

bound

we

prove

for

the

conversion

strategy

Cexp

Conversion

strategies

Theorem

Choose

a

subsequence-closed

set

-LRB-

X

-RRB-

and

a

conservative

deterministic

prediction

algorithm

A.

Choose

nonnegative

ff

such

that

ff

Then

the

number

of

mistakes

made

by

Cexp

-LRB-

ff

A

-RRB-

on

any

sequence

s

-LRB-

X

-RRB-

which

is

a

corrupted

version

of

some

sequence

in

is

at

most

min

max

dC

-LRB-

s

u

-RRB-

log

LA

-LRB-

-RRB-

log

log

ff

-LRB-

-RRB-

where

u

means

that

is

any

subsequence

of

u

It

is

easy

to

verify

numerically

that

by

choosing

ff

the

upper

bound

for

Cexp

displayed

in

-LRB-

-RRB-

is

at

most

min

max

-LRB-

dC

-LRB-

s

u

-RRB-

LA

-LRB-

-RRB-

-RRB-

Thus

we

get

a

reasonable

bound

that

holds

for

all

values

of

dC

-LRB-

s

u

-RRB-

and

LA

-LRB-

-RRB-

-RRB-

However

if

one

wants

to

set

ff

and

so

that

the

mistake

bound

of

Cexp

is

optimized

then

one

needs

to

know

upper

bounds

k

and

r

on

dC

-LRB-

s

u

-RRB-

and

LA

-LRB-

-RRB-

respectively

The

case

when

r

or

k

is

zero

is

degenerate

Thus

we

assume

that

min

-LRB-

r

k

-RRB-

The

following

inequality

was

numerically

checked

using

MAPLEtm

a

software

package

for

symbolic

computation

r

log

k

log

log

ff

-LRB-

r

k

-RRB-

q

rk

ln

-LRB-

e

max

-LRB-

r

k

-RRB-

min

-LRB-

r

k

-RRB-

-RRB-

prk

def

f

-LRB-

r

k

-RRB-

when

ff

r

f

-LRB-

r

k

-RRB-

r

k

and

k

f

-LRB-

r

k

-RRB-

r

k

If

r

k

then

by

dividing

the

inequality

by

k

we

are

left

with

an

inequality

in

r

k

where

r

k

-LSB-

-RRB-

We

plotted

the

difference

between

the

LHS

and

RHS

of

the

latter

inequality

as

a

function

of

r

k

and

checked

the

values

of

the

difference

and

its

derivatives

w.r.t.

r

k

at

the

end

points

one

and

One

can

also

show

that

there

is

no

constant

c

independent

of

r

and

k

such

that

the

mistake

bound

of

Cexp

-LRB-

with

ff

and

optimized

-RRB-

is

at

most

-LRB-

r

k

-RRB-

cprk

Notice

however

that

Cexp

has

a

worst-case

mistake

bound

larger

than

Cbin

In

much

the

same

way

we

proved

Theorem

in

Section

we

can

also

prove

the

following

-LRB-

see

Section

-RRB-

Theorem

r

N

and

-LSB-

-RRB-

such

that

-LRB-

ff

-RRB-

max

-LRB-

q

N

q

log

r

k

X

i

q

i

i

k

i

i

r

-RRB-

r

log

k

log

log

ff

-LRB-

-RRB-

To

show

an

immediate

application

of

Theorems

and

consider

the

special

case

when

the

set

-LRB-

X

-RRB-

of

uncorrupted

sequences

is

the

set

of

all

sequences

consistent

with

some

family

F

of

functions

f

on

X

That

is

F

h

-LRB-

xt

f

-LRB-

xt

-RRB-

-RRB-

it

f

F

hxtit

X

This

more

restricted

setting

was

studied

by

Littlestone

-LSB-

-RSB-

and

Littlestone

and

Warmuth

-LSB-

-RSB-

where

they

define

the

quantities

Opt

-LRB-

F

-RRB-

i.e.

the

optimal

worst-case

number

of

mistakes

over

all

sequences

from

F

and

Opt

-LRB-

F

r

-RRB-

i.e.

the

optimal

worst-case

number

of

mistakes

over

all

r-corrupted

sequences

from

F

Littlestone

and

Warmuth

-LSB-

-RSB-

show

that

Opt

-LRB-

F

r

-RRB-

Opt

-LRB-

F

-RRB-

but

the

problem

of

finding

an

equivalent

upper

bound

is

left

open

By

applying

Theorem

-LRB-

or

the

weaker

Theorem

-RRB-

when

F

and

the

subalgorithm

A

is

optimal

we

obtain

the

upper

bound

Opt

-LRB-

F

r

-RRB-

-LRB-

r

Opt

-LRB-

F

-RRB-

-RRB-

therefore

showing

Opt

-LRB-

F

r

-RRB-

-LRB-

r

Opt

-LRB-

F

-RRB-

-RRB-

Auer

and

Long

-LSB-

-RSB-

independently

developed

an

algorithm

essentially

equivalent

to

our

Cexp

strategy

All

of

our

conversion

schemes

use

deterministic

prediction

algorithms

This

means

that

the

algorithm

's

prediction

depends

only

on

its

current

state

and

the

observation

After

making

its

prediction

the

algorithm

enters

a

new

state

based

on

the

observation

and

the

outcome

We

denote

the

initial

state

of

the

prediction

algorithm

by

Sinit

and

use

AS

to

denote

prediction

algorithm

A

in

state

S

When

the

observation

is

fixed

the

next

state

entered

by

algorithm

A

depends

only

on

the

outcome

We

use

Sx

-LRB-

and

Sx

-RRB-

to

denote

the

-LRB-

possibly

identical

-RRB-

next

state

entered

by

AS

after

AS

receives

observation

x

and

outcome

-LRB-

or

outcome

respectively

-RRB-

In

the

rest

of

this

section

we

state

and

prove

the

mistake

bounds

for

Cbin

and

Cexp

The

conversion

strategy

Cbin

In

this

section

we

formally

describe

the

Cbin

strategy

and

prove

its

mistake

bound

The

Cbin

strategy

uses

a

concise

representation

of

the

version

space

in

much

the

same

way

that

the

BW

algorithm

keeps

a

single

binomial

weight

for

each

expert

In

order

to

avoid

confusion

with

the

states

of

the

algorithm

being

converted

we

call

the

states

of

the

Cbin

algorithm

configurations

Each

configuration

encodes

the

appropriate

version

space

as

well

as

a

value

-LRB-

which

we

usually

denote

-RRB-

indicating

an

upper

bound

on

the

number

of

mistakes

yet

to

be

made

by

the

conversion

strategy

The

Cbin

algorithm

changes

configurations

only

when

it

makes

a

mistake

The

version

space

is

encoded

in

a

configuration

as

a

-LRB-

multi

-RRB-

set

of

triples

representing

computations

of

algorithm

A

on

corrupted

versions

of

subsequences

of

the

past

trials

More

precisely

the

version

space

is

represented

by

a

collection

of

-LRB-

S

-RRB-

triples

where

S

is

a

possible

state

of

algorithm

A

and

the

other

two

components

are

integers

Intuitively

represents

the

maximum

number

of

future

examples

that

can

be

corrupted

by

noise

and

represents

the

maximum

number

of

mistakes

made

by

algorithm

A

in

the

remaining

trials

Thus

if

the

upper

bound

on

the

number

of

mistakes

yet

to

be

made

by

the

conversion

strategy

the

single

triple

-LRB-

S

-RRB-

represents

X

i

i

i

i

i

different

elements

in

the

version

space

-LRB-

or

-LRB-

S

o

-RRB-

pairs

for

o

noise

mstk

-RRB-

It

is

important

to

understand

that

the

and

values

all

start

at

the

r

k

and

m

upper

bounds

and

count

down

a

subsequent

paper

-LSB-

-RSB-

a

randomized

variant

of

their

conversion

strategy

is

introduced

The

worst-case

expected

number

of

mistake

of

their

randomized

strategy

is

significantly

lower

than

the

worst-case

mistake

bound

of

-LRB-

the

deterministic

strategy

-RRB-

Cbin

Conversion

strategies

The

initial

configuration

of

the

Cbin

conversion

strategy

contains

the

single

triple

-LRB-

Sinit

r

k

-RRB-

where

Sinit

is

the

initial

state

of

algorithm

A

r

is

the

bound

on

the

number

of

noisy

trials

and

k

is

the

mistake

bound

of

A

on

sequences

in

The

initial

configuration

of

Cbin

also

contains

the

mistake

m

one

greater

than

the

mistake

bound

of

Cbin

An

important

concept

is

the

successors

of

a

configuration

For

any

possible

state

S

of

algorithm

A

and

any

x

X

we

use

Sx

and

Sx

to

denote

the

states

entered

by

A

from

state

S

after

processing

the

single

observation-outcome

pair

-LRB-

x

-RRB-

or

-LRB-

x

-RRB-

respectively

Given

a

configuration

C

with

mistake

budget

we

define

the

successors

Cx

and

Cx

of

configuration

Ct

with

respect

to

observation

x

in

the

following

way

Both

successor

configurations

have

mistake

budget

For

each

triple

-LRB-

S

-RRB-

in

Ct

consider

the

prediction

of

AS

on

observation

x

If

AS

predicts

then

ffl

configuration

Cx

contains

the

single

triple

-LRB-

S

-RRB-

and

ffl

configuration

Cx

contains

the

triples

-LRB-

Sx

-RRB-

and

-LRB-

S

-RRB-

representing

the

possibilities

of

a

incorrect

prediction

by

A

and

a

noisy

trial

respectively

Similarly

if

AS

predicts

on

observation

x

then

ffl

configuration

Cx

contains

the

triple

-LRB-

S

-RRB-

and

ffl

configuration

Cx

contains

the

triples

-LRB-

Sx

-RRB-

and

-LRB-

S

-RRB-

We

define

the

weight

of

a

configuration

to

be

the

size

of

the

version

space

represented

by

that

configuration

In

particular

the

weight

-LRB-

S

-RRB-

of

the

triple

-LRB-

S

-RRB-

in

a

configuration

with

mistake

budget

is

X

i

i

i

i

i

and

the

weight

of

a

configuration

C

-LRB-

C

-RRB-

is

the

sum

of

the

weights

of

the

triples

in

C.

Triples

-LRB-

S

-RRB-

where

either

or

represent

sequences

disallowed

by

our

assumptions

and

these

disallowed

triples

are

given

weight

zero

Deleting

disallowed

triples

from

a

configuration

has

no

effect

on

the

strategy

's

predictions

On

each

trial

the

Cbin

conversion

strategy

in

configuration

C

receives

the

new

instance

x

and

computes

the

weights

of

the

two

successor

states

Cx

and

Cx

The

Cbin

conversion

strategy

predicts

if

the

weight

of

Cx

is

greater

than

the

weight

of

Cx

and

zero

otherwise

If

the

Cbin

strategy

predicted

correctly

it

keeps

the

configuration

C

If

the

Cbin

strategy

predicted

incorrectly

then

it

changes

its

configuration

from

C

to

Cx

b

where

b

is

the

outcome

of

the

current

trial

A

sketch

of

the

conversion

strategy

Cbin

is

given

in

Figure

The

algorithm

Cbin

can

be

further

improved

in

the

same

way

that

improved

BW

-LRB-

See

Section

-RRB-

However

these

changes

do

not

improve

the

worst-case

mistake

bounds

and

thus

we

chose

not

to

include

them

for

the

sake

of

the

simplicity

of

the

presentation

The

next

result

shows

some

useful

properties

of

sequences

of

configurations

Lemma

Choose

a

conservative

deterministic

prediction

algorithm

A

and

let

Sinit

be

A

's

initial

state

Choose

a

subsequence

closed

set

-LRB-

X

-RRB-

such

that

LA

-LRB-

u

-RRB-

k

for

some

k

N

and

all

u

Choose

r

N

and

a

sequence

s

h

-LRB-

xt

yt

-RRB-

i

in

-LRB-

X

-RRB-

from

footnote

that

using

m

can

lead

to

more

than

m

mistakes

Strategy

Cbin

Input

Two

positive

integers

r

k

and

a

prediction

algorithm

A

with

initial

state

Sinit

Let

g

m

where

m

max

-LRB-

q

N

q

log

r

k

X

i

q

i

i

r

i

i

k

-RRB-

-LRB-

-RRB-

Initialize

configuration

to

have

mistake

budget

g

and

contain

the

single

triple

-LRB-

Sinit

r

k

-RRB-

For

each

trial

t

-LRB-

a

-RRB-

Get

the

tth

observation

xt

-LRB-

b

-RRB-

Compute

the

successors

Cxt

t

and

Cxt

t

of

the

current

configuration

Ct

-LRB-

c

-RRB-

Predict

with

p

such

thata

Wct

-LRB-

Cpt

-RRB-

maxfWct

-LRB-

Cxt

t

-RRB-

Wct

-LRB-

Cxt

t

-RRB-

g

-LRB-

d

-RRB-

Get

the

outcome

yt

-LRB-

e

-RRB-

If

p

yt

then

decrease

the

mistake

budget

and

update

the

current

configuration

by

setting

Ct

Cxt

yt

t

if

p

yt

then

keep

the

current

configuration

by

setting

Ct

Ct

Figure

Pseudo-code

for

the

conversion

strategy

Cbin

aIf

Wct

-LRB-

Cxt

t

-RRB-

Wct

-LRB-

Cxt

t

-RRB-

then

arbitrarily

predict

which

is

an

r-corrupted

version

of

some

u

h

-LRB-

xt

zt

-RRB-

i

in

Let

be

the

configuration

with

mistake

budget

g

containing

the

single

triple

-LRB-

Sinit

-RRB-

and

let

Cg

be

the

sequence

of

distinct

configurations

generated

by

a

run

of

Cbin

applied

to

A

on

the

sequences

s.

Then

for

each

t

g

Wct

-LRB-

Ct

-RRB-

Wct

-LRB-

Cxt

t

-RRB-

Wct

-LRB-

Cxt

t

-RRB-

Wct

-LRB-

Ct

-RRB-

for

each

t

g

Wg

-LRB-

Ct

-RRB-

where

ct

is

the

mistake

budget

of

Ct.

Proof

To

prove

part

we

show

for

each

triple

-LRB-

S

-RRB-

that

the

sum

of

the

weights

of

the

successor

triples

equals

the

weight

of

the

original

That

is

if

the

example

is

xt

yt

then

Wct

-LRB-

S

-RRB-

Wct

-LRB-

Sxt

yt

-RRB-

Wct

-LRB-

S

-RRB-

X

j

ct

j

j

j

j

X

j

ct

j

j

j

j

X

j

ct

j

j

j

j

X

j

ct

j

j

j

j

X

j

ct

j

j

j

j

Conversion

strategies

X

j

ct

j

j

j

j

X

j

ct

j

j

j

j

X

j

ct

j

j

j

j

Wct

-LRB-

S

-RRB-

To

prove

part

choose

a

sequence

u

in

and

let

s

h

-LRB-

xt

yt

-RRB-

i

be

a

r-corrupted

version

of

u.

Let

v

be

the

subsequence

of

s

containing

all

the

pairs

-LRB-

xt

yt

-RRB-

where

Cbin

makes

a

mistake

by

predicting

yt

Let

w

be

the

subsequence

of

v

obtained

by

deleting

the

examples

corrupted

by

noise

Finally

for

each

t

let

p

-LRB-

t

-RRB-

t

be

the

number

of

uncorrupted

examples

in

vt

-LRB-

recall

that

vt

is

the

length

t

prefix

of

v

-RRB-

so

t

p

-LRB-

t

-RRB-

is

the

number

of

corrupted

examples

in

vt

and

wp

-LRB-

t

-RRB-

is

the

sequence

obtained

from

vt

by

deleting

the

corrupted

examples

Let

C

-LRB-

vt

-RRB-

be

the

set

of

-LRB-

S

-RRB-

triples

in

Cbin

's

configuration

immediately

after

Cbin

has

seen

the

sequence

vt.

Recall

that

C

-LRB-

-RRB-

f

-LRB-

Sinit

r

k

-RRB-

g

and

a

triple

-LRB-

S

-RRB-

is

discarded

from

the

configuration

if

either

or

To

prove

the

statement

in

part

of

the

lemma

it

suffices

to

prove

the

following

claim

Claim

For

each

t

jvj

there

is

a

triple

-LRB-

S

-RRB-

C

-LRB-

vt

-RRB-

such

that

S

is

the

state

of

A

-LRB-

wp

-LRB-

t

-RRB-

-RRB-

k

is

the

number

of

mistakes

made

by

A

on

sequence

wp

-LRB-

t

-RRB-

and

r

t

p

-LRB-

t

-RRB-

the

number

of

corrupted

trials

in

vt.

Proof

of

Claim

First

note

that

w

is

a

subsequence

of

u

so

A

makes

at

most

k

mistakes

on

w.

Furthermore

v

is

a

subsequence

of

s

and

s

contains

at

most

r

noisy

examples

so

v

contains

at

most

r

noisy

trials

Therefore

both

k

and

r

are

at

least

zero

We

now

prove

by

induction

on

t

that

an

appropriate

triple

is

in

the

configuration

C

-LRB-

vt

-RRB-

For

the

base

case

consider

t

and

recall

that

p

-LRB-

-RRB-

There

is

only

one

triple

-LRB-

Sinit

r

k

-RRB-

in

C

-LRB-

-RRB-

Since

is

the

empty

sequence

A

-LRB-

-RRB-

Sinit

and

A

makes

no

mistakes

on

sequence

Thus

all

three

conditions

are

satisfied

by

this

triple

For

the

inductive

step

assume

some

triple

-LRB-

S

-RRB-

C

-LRB-

vt

-RRB-

satisfies

the

three

conditions

of

the

claim

We

now

show

that

either

-LRB-

S

-RRB-

or

one

of

its

successors

in

C

-LRB-

vt

-RRB-

also

satisfies

the

claim

Case

the

t

trial

is

a

corrupted

trial

so

wp

-LRB-

t

-RRB-

wp

-LRB-

t

-RRB-

If

AS

agrees

with

the

corrupted

outcome

then

-LRB-

S

-RRB-

is

also

in

C

-LRB-

vt

-RRB-

and

the

three

parts

of

the

claim

continue

to

hold

If

AS

disagrees

with

the

corrupted

outcome

then

-LRB-

S

-RRB-

is

in

C

-LRB-

vt

-RRB-

and

since

vt

has

one

more

corrupted

trial

than

vt

the

three

parts

of

the

claim

also

holds

for

C

-LRB-

vt

-RRB-

Case

the

t

st

trial

is

not

a

corrupted

trial

so

vt

wp

-LRB-

t

-RRB-

wp

-LRB-

t

-RRB-

If

AS

predicts

correctly

on

wp

-LRB-

t

-RRB-

then

the

triple

-LRB-

S

-RRB-

remains

in

the

configuration

Also

since

A

is

conservative

S

A

-LRB-

wp

-LRB-

t

-RRB-

-RRB-

A

-LRB-

wp

-LRB-

t

-RRB-

-RRB-

and

the

claim

holds

for

C

-LRB-

vt

-RRB-

If

AS

predicts

incorrectly

then

so

does

A

-LRB-

wp

-LRB-

t

-RRB-

-RRB-

Thus

A

makes

k

mistakes

on

wp

-LRB-

t

-RRB-

Let

e

be

the

example

wp

-LRB-

t

-RRB-

and

thus

Se

is

the

state

A

-LRB-

wp

-LRB-

t

-RRB-

-RRB-

In

this

situation

the

triple

-LRB-

Se

-RRB-

is

in

C

-LRB-

vt

-RRB-

satisfying

the

claim

Proof

of

Theorem

Choose

n

k

N

and

a

sequence

sn

-LRB-

X

-RRB-

n

which

is

a

r-corrupted

version

of

some

u

Let

m

be

the

integer

defined

by

formula

-LRB-

-RRB-

and

assume

to

the

contrary

that

Cbin

-LRB-

r

k

A

-RRB-

makes

at

least

g

m

mistakes

on

s.

Let

be

the

trial

on

which

Cbin

-LRB-

r

k

A

-RRB-

makes

its

gth

mistake

and

the

mistake

budget

after

the

th

trial

We

will

show

that

-LRB-

C

-RRB-

Wg

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Let

tg

be

the

trials

at

which

algorithm

Cbin

makes

its

first

g

mistakes

and

be

the

associated

subsequence

of

u

Since

is

closed

under

subsequences

We

apply

Lemma

to

sequence

and

the

associated

sequence

Ctg

of

configurations

generated

by

the

algorithm

By

construction

the

algorithm

predicts

on

each

trial

t

-LRB-

t

n

-RRB-

according

to

the

heaviest

successor

of

the

current

configuration

Ct

The

current

configuration

is

unchanged

if

Cbin

predicts

correctly

If

the

algorithm

makes

a

mistake

on

trial

t

the

successor

Cxt

yt

t

corresponding

to

the

correct

prediction

yt

becomes

the

new

current

configuration

Because

algorithm

Cbin

predicts

on

each

trial

according

to

the

heaviest

successor

it

follows

from

part

of

Lemma

that

Wg

-LRB-

-RRB-

Wg

-LRB-

-RRB-

and

that

Wcj

-LRB-

Ctj

-RRB-

Wcj

-LRB-

Ctj

-RRB-

for

j

g

where

cj

-LRB-

for

j

g

-RRB-

is

the

mistake

budget

of

Ctj

This

implies

inequality

-LRB-

-RRB-

By

definition

of

m

in

-LRB-

-RRB-

and

the

fact

that

g

m

we

derive

inequality

-LRB-

-RRB-

Now

part

of

Lemma

shows

that

Wmt

-LRB-

Ct

-RRB-

contradicting

inequality

-LRB-

-RRB-

Thus

Cbin

makes

at

most

m

g

mistakes

on

s

concluding

the

proof

A

good

outcome

of

the

fact

that

Cbin

is

conservative

is

that

the

number

of

triplets

does

not

increase

on

trials

where

Cbin

predicts

correctly

However

it

seems

that

the

number

of

triples

kept

by

algorithm

Cbin

can

potentially

double

each

time

Cbin

makes

an

incorrect

prediction

We

now

show

that

this

apparent

worst

case

behavior

is

not

possible

and

that

the

maximum

number

of

triples

in

any

configuration

of

Cbin

-LRB-

r

k

A

-RRB-

is

bounded

by

m

minfr

kg

O

-LRB-

mminfr

kg

-RRB-

where

m

is

the

number

of

mistakes

made

by

Cbin

before

the

configuration

is

reached

Theorem

Choose

a

subsequence

closed

set

-LRB-

X

-RRB-

and

a

conservative

deterministic

prediction

algorithm

A

with

initial

state

Sinit

and

k

N

such

that

LA

-LRB-

u

-RRB-

k

for

all

u

Choose

m

r

N

and

any

sequence

s

h

-LRB-

xt

yt

-RRB-

i

in

-LRB-

X

-RRB-

which

is

a

r-corrupted

version

of

some

u

Let

be

configuration

with

mistake

budget

m

containing

the

single

triple

-LRB-

Sinit

-RRB-

and

let

Cm

be

the

sequence

of

distinct

configurations

generated

by

a

run

of

Cbin

applied

to

A

on

the

sequences

s

Then

for

all

t

m

configuration

Ct

contains

at

most

t

minfr

kg

triples

with

non-zero

weight

Proof

We

prove

the

theorem

when

r

minfr

kg

the

other

case

is

similar

For

all

t

m

and

i

r

let

Mt

-LRB-

i

-RRB-

be

the

number

of

triples

-LRB-

S

-RRB-

Ct

with

r

i.

Thus

-LRB-

-RRB-

-LRB-

for

the

initial

configuration

-RRB-

and

-LRB-

i

-RRB-

for

all

i

Note

that

some

triples

counted

in

Mt

-LRB-

r

-RRB-

might

have

zero

weight

if

their

From

the

definition

of

successors

Mt

-LRB-

i

-RRB-

Mt

-LRB-

i

-RRB-

Mt

-LRB-

i

-RRB-

The

unique

function

f

f

-LRB-

t

i

-RRB-

satisfying

f

-LRB-

-RRB-

f

-LRB-

i

-RRB-

for

i

r

f

-LRB-

t

i

-RRB-

f

-LRB-

t

i

-RRB-

f

-LRB-

t

i

-RRB-

for

t

and

i

r

Conversion

strategies

is

the

binomial

coefficient

ti

Therefore

Mt

-LRB-

i

-RRB-

ti

yielding

that

the

number

of

triples

-LRB-

S

-RRB-

in

Ct

with

r

is

at

most

rX

i

Mt

-LRB-

i

-RRB-

rX

i

t

i

t

r

as

desired

The

conversion

strategy

Cexp

We

now

move

on

to

the

description

of

the

conversion

strategy

Cexp

Where

Cbin

was

based

on

binomial

weights

Cexp

uses

exponential

weights

The

advantage

of

using

exponential

weights

is

that

the

conversion

strategy

does

not

need

to

know

the

bounds

r

and

k

which

Cbin

requires

as

inputs

However

if

one

wants

to

optimize

the

mistake

bound

of

Cexp

so

that

it

is

in

the

form

-LRB-

r

k

-RRB-

plus

a

square

root

term

then

knowledge

of

k

and

r

is

required

for

Cexp

as

well

Analogously

to

Cbin

the

bound

of

Cexp

does

not

depend

on

the

length

of

the

sequence

to

predict

The

weighting

scheme

used

by

Cexp

has

two

real

parameters

ff

and

such

that

ff

Here

we

define

a

configuration

by

a

set

of

triples

for

different

computations

of

algorithm

A.

Unlike

the

description

of

strategy

Cbin

given

before

here

a

configuration

does

not

have

a

mistake

count

or

mistake

budget

However

as

before

each

triple

is

of

the

form

-LRB-

S

i

j

-RRB-

where

S

is

a

possible

state

of

algorithm

A

and

i

j

are

both

nonnegative

integers

For

any

fixed

ff

the

weight

Wff

fi

-LRB-

S

-RRB-

of

the

triple

-LRB-

S

i

j

-RRB-

is

the

product

ffi

j

As

before

the

weight

of

a

configuration

Wff

fi

-LRB-

C

-RRB-

is

the

total

weight

of

the

triples

in

C

The

role

played

here

by

the

components

i

and

j

in

each

triple

is

analogous

to

the

role

respectively

played

by

the

components

and

in

the

triple

-LRB-

S

-RRB-

defining

algorithm

Cbin

We

use

essentially

the

same

definition

of

successors

as

the

one

introduced

in

Section

for

the

strategy

Cbin

with

only

two

differences

Namely

the

mistake

count

is

absent

and

a

triple

is

never

removed

since

its

weight

never

drops

to

zero

A

sketch

of

the

conversion

strategy

Cexp

using

the

above

weighting

scheme

is

given

in

Figure

The

next

lemma

establishes

some

properties

of

such

weighting

schema

Lemma

Fix

a

conservative

and

deterministic

prediction

algorithm

A

and

let

Sinit

be

its

state

after

the

initialization

Choose

a

subsequence

closed

set

-LRB-

X

-RRB-

and

a

corrupted

version

s

h

-LRB-

xt

yt

-RRB-

i

of

some

u

h

-LRB-

xt

zt

-RRB-

i

in

Let

Cn

be

the

sequence

of

distinct

configurations

generated

by

a

run

of

Cexp

applied

to

A

on

the

sequence

s.

Then

for

each

t

n

and

for

each

ff

Wff

fi

-LRB-

Ct

-RRB-

ff

Wff

fi

-LRB-

Ct

-RRB-

Wff

fi

-LRB-

Cn

-RRB-

ffdH

-LRB-

y

z

-RRB-

LA

-LRB-

u

-RRB-

Proof

Omitted

We

now

turn

to

the

proof

of

the

worst-case

mistake

bound

for

the

conversion

strategy

Cexp

Strategy

Cexpa

Input

Two

real

numbers

ff

such

that

ff

and

a

prediction

algorithm

A

with

initial

state

Sinit

Initialize

configuration

to

contain

the

single

triple

-LRB-

Sinit

r

k

-RRB-

On

each

step

t

-LRB-

a

-RRB-

Get

the

tth

observation

xt

-LRB-

b

-RRB-

Compute

the

successor

configurations

Cxt

t

and

Cxt

t

of

the

current

configuration

Ct

-LRB-

c

-RRB-

Predict

with

p

such

that

Wff

fi

-LRB-

Cxt

;p

t

-RRB-

maxfWff

fi

-LRB-

Cxt

t

-RRB-

Wff

fi

-LRB-

Cxt

t

-RRB-

g

-LRB-

d

-RRB-

Get

the

outcome

yt

-LRB-

e

-RRB-

If

p

yt

then

update

the

current

configuration

by

letting

Ct

Cxt

yt

t

or

if

p

yt

let

Ct

Ct

Figure

Pseudo-code

for

the

conversion

strategy

Cexp

aAn

alternative

way

of

arriving

at

the

same

prediction

is

the

following

Given

an

instance

x

each

triple

-LRB-

S

-RRB-

votes

with

weight

for

the

prediction

of

AS

on

the

instance

x

The

master

algorithm

then

predicts

with

the

vote

that

got

the

larger

total

weight

When

this

method

of

prediction

is

used

the

successor

configuration

has

to

be

computed

only

when

a

mistake

occurs

Proof

of

Theorem

Choose

any

sequence

s

h

-LRB-

xt

yt

-RRB-

i

and

choose

u

By

construction

Cexp

predicts

on

each

step

t

according

to

the

heaviest

successor

of

the

current

configuration

Ct.

If

a

mistake

occurs

then

the

successor

Cxt

yt

t

corresponding

to

the

correct

prediction

yt

becomes

the

new

current

configuration

Moreover

again

by

construction

of

Cexp

the

current

configuration

is

unchanged

if

the

algorithm

predicts

correctly

We

can

therefore

apply

Lemma

to

the

subsequence

s

determined

by

the

sequence

tm

of

the

indices

of

the

prediction

trials

where

Cexp

makes

a

mistake

Since

is

subsequence-closed

the

subsequence

of

u

that

corresponds

to

these

trials

lies

in

By

applying

part

of

the

same

lemma

and

given

that

ff

we

conclude

that

the

total

weight

of

the

current

configuration

decreases

by

a

factor

of

at

least

ff

each

time

Cexp

makes

a

mistake

Also

dC

-LRB-

-RRB-

dC

-LRB-

s

u

-RRB-

and

hence

if

Cfin

is

the

configuration

following

the

last

prediction

mistake

made

by

Cexp

on

s

part

of

Lemma

implies

that

Wff

fi

-LRB-

Cfin

-RRB-

ffdC

-LRB-

s

u

-RRB-

LA

-LRB-

-RRB-

Hence

assuming

Cexp

-LRB-

ff

-RRB-

makes

m

mistakes

on

s

and

recalling

that

Wff

fi

-LRB-

-RRB-

ff

m

Wff

fi

-LRB-

Ct

-RRB-

ffdC

-LRB-

s

u

-RRB-

LA

-LRB-

-RRB-

Solving

for

m

recalling

that

m

is

integer

yields

m

dC

-LRB-

s

u

-RRB-

log

LA

-LRB-

-RRB-

log

log

ff

Since

s

-LRB-

X

-RRB-

and

u

were

chosen

arbitrarily

the

proof

is

concluded

Conversion

strategies

We

conclude

this

section

by

proving

the

last

of

the

three

theorems

stated

in

Section

Proof

of

Theorem

We

shall

upper

bound

the

maximal

value

of

a

larger

set

max

-LRB-

q

N

q

log

r

k

X

i

q

i

i

k

-RRB-

r

log

k

log

log

ff

-LRB-

-RRB-

Inequality

-LRB-

-RRB-

is

proven

via

the

following

lemma

Lemma

r

m

N

such

that

m

r

k

and

-LSB-

-RRB-

such

that

-LRB-

ff

-RRB-

r

k

X

i

m

i

i

k

-LRB-

ff

-RRB-

m

ffr

k

Proof

of

Lemma

By

a

double

application

of

the

Binomial

Theorem

we

show

-LRB-

ff

-RRB-

m

m

X

i

m

i

-LRB-

ff

-RRB-

i

ffk

r

r

k

X

i

m

i

kX

j

i

j

and

this

concludes

the

proof

It

is

easy

to

see

that

r

k

is

a

lower

bound

on

the

number

of

mistakes

of

any

master

algorithm

The

LHS

of

Equation

-LRB-

-RRB-

is

an

upper

bound

on

the

number

of

mistakes

made

by

Cbin

therefore

it

is

larger

than

r

k.

Thus

we

can

apply

Lemma

to

-LRB-

-RRB-

obtaining

max

-LRB-

q

N

q

log

r

k

X

i

q

i

i

k

-RRB-

max

-LRB-

q

N

r

k

X

i

q

i

i

k

-RRB-

max

ae

q

N

-LRB-

ff

-RRB-

q

ffr

k

oe

max

q

N

q

r

log

k

log

log

-LRB-

ff

-RRB-

r

log

k

log

log

-LRB-

ff

-RRB-

concluding

the

proof

If

we

give

Cexp

an

additional

input

parameter

k

such

that

k

LA

-LRB-

-RRB-

the

strategy

can

exploit

this

information

in

order

to

minimize

the

number

of

states

in

each

configuration

In

particular

Cexp

can

discard

from

the

current

configuration

each

triple

-LRB-

S

i

j

-RRB-

such

that

j

k.

By

using

this

trick

we

can

show

analogously

to

what

we

did

for

Cbin

in

Theorem

that

the

maximum

number

of

triples

in

each

configuration

of

Cexp

-LRB-

ff

A

k

-RRB-

is

bounded

by

O

-LRB-

mk

-RRB-

where

m

is

the

number

of

mistakes

made

by

Cexp

up

to

the

current

configuration

Furthermore

as

we

mentioned

above

the

knowledge

of

bounds

r

or

k

can

be

used

to

optimize

the

parameters

ff

and

Note

that

both

the

conversion

strategy

Cbin

and

Cexp

are

conservative

in

the

sense

that

they

only

update

their

configuration

when

they

make

a

mistake

At

least

one

copy

of

algorithm

A

receives

only

the

subsequence

of

clean

examples

on

which

the

conversion

strategies

makes

a

mistake

Therefore

we

require

that

the

mistake

bound

of

algorithm

A

holds

on

all

subsequences

of

sequences

in

This

is

the

reason

we

assumed

that

the

set

of

sequences

in

theorems

and

is

subsequence-closed

We

would

like

conversion

strategies

which

do

not

require

this

assumption

It

seems

that

this

is

possible

only

for

a

mistake

bound

that

increases

with

the

length

of

the

sequence

If

we

somehow

could

give

A

the

correct

feedback

in

trials

in

which

the

conversion

strategy

makes

no

mistake

then

we

could

drop

the

assumption

and

update

the

configuration

in

all

trials

The

simple

method

of

using

the

prediction

of

the

conversion

strategy

as

feedback

does

not

work

This

is

illustrated

by

the

following

example

Assume

the

original

algorithm

A

predicts

in

the

first

trial

and

afterwards

it

simply

predicts

always

with

the

label

of

the

first

example

Now

let

the

sequence

of

examples

be

labeled

as

i

The

conversion

strategy

will

correctly

predict

in

the

first

trial

and

feeding

to

A

will

spoil

A

If

we

want

to

update

in

each

trial

then

we

need

to

simulate

noise

and

mistakes

on

all

trials

and

this

will

lead

to

increased

mistake

bounds

Conclusions

We

have

investigated

the

problem

of

on-line

boolean

prediction

from

two

different

viewpoints

We

first

improved

known

results

about

strategies

that

predict

deterministically

using

the

advice

from

a

set

of

experts

These

improvements

are

obtained

using

a

weighting

scheme

that

uses

Binomial

coefficients

rather

than

exponential

weights

of

the

form

m

These

binomial

coefficients

can

be

interpreted

as

counting

the

members

of

an

appropriate

version

space

In

the

expert

setting

the

mistake

bound

based

on

binomial

weights

is

never

larger

than

the

mistake

bound

based

on

exponential

weights

Furthermore

the

advantage

of

the

binomial

weights

can

be

made

arbitrarily

large

Nevertheless

both

bounds

can

be

shown

to

have

the

optimum

leading

term

using

probabilistic

techniques

We

also

prove

that

for

an

infinite

subset

of

the

possible

problem

parameters

the

bound

using

binomial

weights

is

best

possible

The

proof

of

this

fact

relies

on

a

new

translation

of

our

prediction

problem

to

Ulam

's

game

with

lies

Secondly

we

introduced

a

novel

approach

for

making

on-line

algorithms

robust

to

noise

We

show

how

to

convert

an

on-line

prediction

algorithm

that

is

guaranteed

to

make

at

most

k

mistakes

when

given

an

observation-outcome

sequence

from

its

domain

into

an

algorithm

that

works

well

when

up

to

r

of

the

outcomes

are

corrupted

by

noise

The

converted

algorithm

has

a

conjectured

mistake

bound

of

-LRB-

r

k

-RRB-

q

rk

ln

-LRB-

e

max

-LRB-

r

k

-RRB-

min

-LRB-

r

k

-RRB-

-RRB-

prk

on

any

of

the

corrupted

sequences

-LRB-

the

conjecture

is

supported

by

numerical

evidences

-RRB-

The

best

lower

bound

we

know

of

is

k

tightening

the

gap

between

these

bounds

remains

an

open

problem

Based

on

our

experience

binomial

weights

seem

to

lead

to

better

mistake

bounds

than

exponential

weights

They

have

the

advantage

of

being

motivated

by

a

version

space

argument

that

leads

to

a

deeper

understanding

of

the

on-line

learning

problem

The

exponential

weights

seem

to

approximate

the

binomial

weights

and

are

sometimes

easier

to

use

especially

when

the

number

of

mistakes

made

by

the

best

expert

is

unknown

-LRB-

although

optimizing

their

mistake

bounds

requires

knowledge

of

this

parameters

as

well

-RRB-

Also

exponential

weights

can

be

used

for

designing

randomized

prediction

algorithms

-LSB-

-RSB-

In

the

case

of

exponential

weights

the

worst-case

expected

number

of

mistakes

of

the

randomized

References

algorithm

is

exactly

half

of

the

worst-case

number

of

mistakes

of

the

deterministic

algorithm

-LSB-

-RSB-

We

were

unable

to

find

a

randomized

binomial

weighting

algorithm

that

had

an

expected

mistake

bound

significantly

smaller

than

the

deterministic

BW

algorithm

Acknowledgments

David

P.

Helmbold

was

supported

by

NSF

grant

Manfred

Warmuth

and

Yoav

Freund

were

supported

by

ONR

grant

Part

of

this

research

was

done

while

Nicol?o

Cesa-Bianchi

was

visiting

UC

Santa

Cruz

-LRB-

USA

-RRB-

partially

supported

by

the

Progetto

finalizzato

sistemi

informatici

e

calcolo

parallelo

of

CNR

under

grant

and

the

Institute

for

Theoretical

Computer

Science

at

the

Graz

University

of

Technology

-LRB-

Austria

-RRB-

References

-LSB-

-RSB-

E.

Aarts

and

J.

Korst

Simulated

Annealing

and

Boltzmann

Machines

John

Wiley

and

Sons

-LSB-

-RSB-

N.

Alon

J.H.

Spencer

and

P.

Erd

-RCB-

os

The

Probabilistic

Method

John

Wiley

and

Sons

-LSB-

-RSB-

D.

Angluin

Queries

and

concept

learning

Machine

Learning

-LCB-

-LSB-

-RSB-

J.A.

Aslam

and

A.

Dhagat

Searching

in

the

presence

of

linearly

bounded

errors

In

Proceedings

of

the

ACM

Symposium

on

the

Theory

of

Computation

pages

-LCB-

ACM

Press

-LSB-

-RSB-

P.

Auer

and

P.M

Long

Simulating

access

to

hidden

information

while

learning

In

Proceedings

of

the

ACM

Symposium

on

the

Theory

of

Computation

pages

-LCB-

ACM

Press

-LSB-

-RSB-

P.

Auer

and

P.M

Long

Structural

results

about

on-line

learning

models

with

and

without

queries

Machine

Learning

To

appear

-LSB-

-RSB-

J.M.

Bardzin

and

R.V.

Freivalds

On

the

prediction

of

general

recursive

functions

Soviet

Math

Dokl.

-LCB-

-LSB-

-RSB-

E.R.

Berlekamp

Error-Correcting

Codes

John

Wiley

and

Sons

-LSB-

-RSB-

N.

Cesa-Bianchi

Y.

Freund

D.P.

Helmbold

D.

Haussler

R.

Schapire

and

M.K.

Warmuth

How

to

use

expert

advice

In

Proceedings

of

the

ACM

Symposium

on

the

Theory

of

Computation

pages

-LCB-

ACM

Press

-LSB-

-RSB-

N.

Cesa-Bianchi

P.M.

Long

and

M.K.

Warmuth

Worst-case

quadratic

loss

bounds

for

a

generalization

of

the

Widrow-Hoff

rule

In

Proceedings

of

the

Annual

ACM

Workshop

on

Computational

Learning

Theory

pages

-LCB-

ACM

Press

-LSB-

-RSB-

H.

Chernoff

A

measure

of

asymptotic

efficiency

for

tests

of

a

hypothesis

based

on

the

sum

of

observations

Annals

of

Mathematical

Statistics

-LCB-

-LSB-

-RSB-

R.

Graham

D.

Knuth

and

O.

Patashnik

Concrete

Mathematics

Addison-Wesley

-LSB-

-RSB-

J.

Kivinen

and

M.K.

Warmuth

Using

experts

for

predicting

continuous

outcomes

In

Proceedings

of

the

First

Euro-COLT

Workshop

The

Institute

of

Mathematics

and

its

Applications

A

A

prediction

algorithm

that

is

strictly

optimal

for

a

large

number

of

experts

-LSB-

-RSB-

N.

Littlestone

Learning

quickly

when

irrelevant

attributes

abound

a

new

linearthreshold

algorithm

Machine

Learning

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

N.

Littlestone

Mistake

Bounds

and

Logarithmic

Linear-threshold

Learning

Algorithms

PhD

thesis

University

of

California

at

Santa

Cruz

-LSB-

-RSB-

N.

Littlestone

P.M.

Long

and

M.K.

Warmuth

On-line

learning

of

linear

functions

Technical

Report

University

of

California

at

Santa

Cruz

An

extended

abstract

appeared

in

Proceedings

of

the

ACM

Symposium

on

the

Theory

of

Computation

-LSB-

-RSB-

N.

Littlestone

and

M.K.

Warmuth

The

weighted

majority

algorithm

Technical

Report

University

of

California

at

Santa

Cruz

An

extended

abstract

appeared

in

Proceedings

of

the

Annual

Symposium

on

the

Foundations

of

Computer

Science

-LSB-

-RSB-

J.

Spencer

Ulam

's

searching

game

with

a

fixed

number

of

lies

Theoretical

Computer

Science

-LCB-

-LSB-

-RSB-

S.

Ulam

Adventures

of

a

Mathematician

Scribners

-LSB-

-RSB-

V.G.

Vovk

Aggregating

strategies

In

Proceedings

of

the

Annual

Workshop

on

Computational

Learning

Theory

pages

-LCB-

A

A

prediction

algorithm

that

is

strictly

optimal

for

a

large

number

of

experts

As

was

shown

in

Section

the

number

of

mistakes

that

the

BW

algorithm

makes

is

within

one

from

optimal

when

N

the

number

of

experts

is

large

enough

In

fact

we

have

shown

that

for

most

values

of

N

BW

obtains

strict

optimality

In

this

section

we

describe

a

variant

of

BW

which

we

call

EBW

-LRB-

Enhanced

Binomial

Weights

-RRB-

which

achieves

optimality

in

the

worst

case

for

all

sufficiently

large

values

of

N

This

modification

and

its

analysis

is

a

direct

adaptation

a

result

of

Spencer

's

-LRB-

-LSB-

-RSB-

Section

-RRB-

As

we

have

seen

in

the

proof

of

Theorem

the

only

slack

which

allows

for

the

gap

between

the

upper

and

the

lower

bounds

is

in

the

way

the

game

is

played

for

the

first

k

trials

In

these

trials

there

are

no

pennies

available

to

Paul

and

thus

in

some

cases

he

is

not

able

to

split

the

chips

into

two

sets

of

equal

weight

In

these

cases

Carol

can

force

a

reduction

of

the

weight

by

more

than

a

factor

of

two

If

Carol

plays

using

the

BW

strategy

then

this

possibility

is

ignored

When

using

EBW

Carol

takes

advantage

of

this

condition

whenever

possible

and

in

this

way

is

able

in

some

cases

to

reduce

the

number

of

mistakes

it

makes

by

one

This

improvement

is

the

best

possible

-LRB-

for

large

enough

N

-RRB-

as

there

also

exists

a

more

refined

strategy

for

Paul

that

can

force

the

exact

same

number

of

mistakes

for

every

N

We

now

describe

the

EBW

algorithm

Recall

step

in

BW

-LRB-

Figure

-RRB-

in

this

step

the

bound

on

the

number

of

mistakes

m

is

calculated

Algorithm

EBW

has

an

additional

step

between

steps

and

of

BW

In

this

step

EBW

checks

if

it

can

take

advantage

of

the

case

described

above

and

guarantee

that

at

most

m

mistakes

will

be

made

Specifically

it

computes

a

new

variable

m

which

is

equal

to

either

m

or

m

The

value

of

m

is

an

improved

upper

bound

on

the

worst

case

number

of

mistakes

The

rest

of

the

algorithm

stays

almost

the

same

the

only

difference

being

that

m

is

used

instead

of

m

in

steps

and

A

A

prediction

algorithm

that

is

strictly

optimal

for

a

large

number

of

experts

We

now

describe

the

computation

of

m

in

step

First

the

algorithm

checks

if

N

m

k

e

If

the

inequality

holds

then

it

is

known

from

Theorem

that

the

bound

can

not

be

improved

and

m

is

set

to

be

m.

Otherwise

EBW

computes

the

following

quantities

For

i

k

it

calculates

the

following

greatest

common

divisors

Ai

gcd

m

i

k

m

i

k

m

i

k

i

It

then

calculates

the

initial

weight

that

corresponds

to

m

N

m

k

and

then

calculates

inductively

for

i

k

the

following

numbers

Vi

max

ae

j

N

j

j

mod

Ai

and

j

Vi

oe

Now

the

algorithm

checks

if

Vk

k

If

this

condition

holds

then

the

algorithm

can

guarantee

at

most

m

mistakes

and

m

is

set

to

m

If

the

condition

does

not

hold

then

m

is

set

to

m

It

remains

to

be

shown

that

the

number

of

mistakes

made

by

EBW

is

at

most

m

and

that

no

other

algorithm

can

make

a

smaller

number

of

mistakes

for

large

enough

values

of

N

The

proof

of

both

of

these

claims

is

a

direct

translation

of

the

proof

of

the

theorem

in

section

of

Spencer

's

paper

-LSB-

-RSB-

and

we

omit

it

from

here

binomial weights
conversion strategies
expert advice
exponential weights
mistake bounds
online learning
weighted majority algorithm