proof of Chernoff-Cramer bound

proof of Chernoff-Cramer bound

Let

h

⁢

(

x

)

be the

step function

(

h

⁢

(

x

)

=

1

for

x

≥

0

,

h

⁢

(

x

)

=

0

for

x

<

0

); then, by generalized

Markov inequality

, for any

t

>

0

and
any

ε

≥

0

,

Pr

⁡

{

∑

i

=

1

n

(

X

i

-

E

⁢

[

X

i

]

)

>

ε

}

=

E

⁢

[

h

⁢

(

∑

i

=

1

n

(

X

i

-

E

⁢

[

X

i

]

)

-

ε

)

]

≤

≤

E

⁢

[

e

t

⁢

(

∑

i

=

1

n

(

X

i

-

E

⁢

[

X

i

]

)

-

ε

)

]

=

=

exp

⁡

(

-

ε

⁢

t

)

⁢

E

⁢

[

e

∑

i

=

1

n

t

⁢

(

X

i

-

E

⁢

[

X

i

]

)

]

=

=

exp

⁡

(

-

ε

⁢

t

)

⁢

E

⁢

[

∏

i

=

1

n

e

t

⁢

(

X

i

-

E

⁢

[

X

i

]

)

]

=

(by independence)

=

exp

⁡

(

-

ε

⁢

t

)

⁢

∏

i

=

1

n

E

⁢

[

e

t

⁢

(

X

i

-

E

⁢

[

X

i

]

)

]

=

=

exp

⁡

(

-

ε

⁢

t

+

∑

i

=

1

n

ln

⁡

E

⁢

[

e

t

⁢

(

X

i

-

E

⁢

[

X

i

]

)

]

)

=

=

exp

⁡

[

-

(

t

⁢

ε

-

ψ

⁢

(

t

)

)

]

.

Since this

expression

is valid for any

t

>

0

, the best bound is obtained taking the supremum:

Pr

⁡

{

∑

i

=

1

n

(

X

i

-

E

⁢

[

X

i

]

)

>

ε

}

≤

e

-

sup

t

>

0

⁡

(

t

⁢

ε

-

ψ

⁢

(

t

)

)

which proves part c).

To prove part a), let’s observe that

Ψ

⁢

(

0

)

=

sup

t

>

0

⁡

(

-

ψ

⁢

(

t

)

)

=

-

inf

t

>

0

⁡

(

ψ

⁢

(

t

)

)

and that

E

⁢

[

e

t

⁢

(

X

i

-

E

⁢

[

X

i

]

)

]

≥

E

⁢

[

1

+

t

⁢

(

X

i

-

E

⁢

[

X

i

]

)

]

=

=

E

⁢

[

1

]

+

t

⁢

E

⁢

[

X

i

]

-

t

⁢

E

⁢

[

E

⁢

[

X

i

]

]

=

=

1

=

E

⁢

[

e

t

⁢

(

X

i

-

E

⁢

[

X

i

]

)

]

t

=

0

that is,

t

=

0

is the infimum point for

E

⁢

[

e

t

⁢

(

X

i

-

E

⁢

[

X

i

]

)

]

∀

i

and consequently for

ψ

⁢

(

t

)

=

∑

i

=

1

n

ln

⁡

E

⁢

[

e

t

⁢

(

X

i

-

E

⁢

X

i

)

]

, so as a conclusion

Ψ

⁢

(

0

)

=

-

ψ

⁢

(

0

)

=

0

b) Let

x

>

0

be fixed and let

t

0

be the supremum point for

t

⁢

x

-

ψ

⁢

(

t

)

; we have to show that

t

0

⁢

x

-

ψ

⁢

(

t

0

)

>

0

.

By

differentiation

,

ψ

′

⁢

(

t

0

)

=

x

.

Let’s recall that the

moment generating function

is convex, so

ψ

′′

⁢

(

t

)

>

0

. Writing the

Taylor expansion

for

ψ

⁢

(

t

)

around

t

=

t

0

, we have, with a suitable

t

1

<

t

0

,

0

=

ψ

⁢

(

0

)

=

ψ

⁢

(

t

0

)

-

ψ

′

⁢

(

t

0

)

⁢

t

0

+

1

2

⁢

ψ

′′

⁢

(

t

1

)

⁢

t

0

2

that is

Ψ

⁢

(

x

)

=

t

0

⁢

x

-

ψ

⁢

(

t

0

)

=

t

0

⁢

ψ

′

⁢

(

t

0

)

-

ψ

⁢

(

t

0

)

=

1

2

⁢

ψ

′′

⁢

(

t

1

)

⁢

t

0

2

>

0

The convexity of

Ψ

⁢

(

x

)

follows from the fact that

Ψ

⁢

(

x

)

is the supremum of the linear (and hence convex)

functions

t

⁢

x

-

ψ

⁢

(

t

)

and so must be convex itself.

Eventually, in to prove that

Ψ

⁢

(

x

)

is an increasing function, let’s note that

Ψ

′

⁢

(

0

)

=

lim

x

→

0

⁡

Ψ

⁢

(

x

)

-

Ψ

⁢

(

0

)

x

=

lim

x

→

0

⁡

Ψ

⁢

(

x

)

x

>

0

and that, by Taylor formula with Lagrange form

remainder

, for a

ξ

=

ξ

⁢

(

x

)

Ψ

′

⁢

(

x

)

=

Ψ

′

⁢

(

0

)

+

Ψ

′′

⁢

(

ξ

)

⁢

x

≥

0

since

Ψ

′′

⁢

(

ξ

)

≥

0

by convexity and

x

≥

0

by
hypotheses.

Title

proof of

Chernoff-Cramer bound

Canonical

name

ProofOfChernoffCramerBound

Date of creation

2013-03-22 16:09:05

Last modified on

2013-03-22 16:09:05

Owner

Andrea Ambrosio (7332)

Last modified by

Andrea Ambrosio (7332)

Numerical id

26

Author

Andrea Ambrosio (7332)

Entry type

Proof

Classification

msc 60E15

Generated on Sat Feb 10 11:56:32 2018 by

LaTeXML