Probability

distributions

with

given

marginals

and

given

correlation

that

have

maximal

entropy

Report

Adrianus

M.H.

Meeuwissen

Tim

Bedford

Technische

Universiteit

Delft

Delft

University

of

Technology

Faculteit

der

Technische

Wiskunde

en

Informatica

Faculty

of

Technical

Mathematics

and

Informatics

ISSN

Copyright

c

by

the

Faculty

of

Technical

Mathematics

and

Informatics

Delft

The

Netherlands

No

part

of

this

Journal

may

be

reproduced

in

any

form

by

print

photoprint

microfilm

or

any

other

means

without

permission

from

the

Faculty

of

Technical

Mathematics

and

Informatics

Delft

University

of

Technology

The

Netherlands

Copies

of

these

reports

may

be

obtained

from

the

bureau

of

the

Faculty

of

Technical

Mathematics

and

Informatics

Julianalaan

BL

Delft

phone

A

selection

of

these

reports

is

available

in

PostScript

form

at

the

Faculty

's

anonymous

ftp-site

They

are

located

in

the

directory

pub/publications/tech-reports

at

ftp.twi.tudelft.nl

ABSTRACT

Probability

Distributions

with

Given

Marginals

and

Given

Correlation

that

have

Maximal

Entropy

For

two

random

variables

the

joint

distribution

with

maximal

entropy

given

the

marginal

distributions

and

the

-LRB-

rank

-RRB-

correlation

between

the

random

variables

is

studied

This

distribution

is

given

by

the

solution

of

a

non-linear

system

of

equations

A

recurrence

relation

for

the

coefficients

of

the

Taylor

series

expansion

of

this

solution

is

derived

An

algorithm

to

solve

the

non-linear

system

of

equations

numerically

for

the

computation

of

a

discretized

version

of

the

distribution

is

given

Adrianus

M.H.

Meeuwissen

Tim

Bedford

September

Mathematics

Subject

Classification

Introduction

In

this

paper

we

investigate

a

joint

probability

distribution

that

can

be

used

to

model

a

partially

unknown

distribution

In

modelling

practical

problems

concerning

random

variables

one

often

encounters

the

case

that

only

sparse

data

is

available

on

joint

distributions

The

problem

is

then

that

the

joint

distribution

is

not

entirely

characterized

by

this

data

i.e.

many

different

joint

distributions

comply

with

the

information

available

In

this

paper

we

consider

the

case

of

two

random

variables

X

and

Y

for

which

only

marginal

distributions

and

a

rank

correlation

coefficient

are

known

We

propose

to

model

the

joint

distribution

in

question

by

that

joint

distribution

which

has

maximal

entropy

among

all

candidate

distributions

Note

that

we

do

not

consider

the

problem

of

estimation

of

a

distribution

from

an

empirical

joint

probability

density

function

The

same

approach

has

been

followed

in

practice

see

for

example

the

technical

reports

-LSB-

-RSB-

There

one

reads

that

probability

distributions

with

maximal

entropy

have

been

used

in

the

analysis

of

the

propagation

of

uncertainty

of

model

parameters

in

models

trying

to

assess

the

risks

of

spacecraft

and

chemical

plants

Because

of

the

sparsity

of

data

available

to

accurately

quantify

the

uncertainties

involved

in

these

studies

the

most

uniform

most

random

distributions

complying

with

the

available

information

have

been

used

Another

classical

approach

to

construct

distributions

with

given

marginals

and

correlation

is

to

take

a

mixture

of

a

Fr?echet

bound

and

the

independent

distribution

This

approach

has

been

generalized

very

recently

-LSB-

-RSB-

for

the

construction

of

distributions

with

given

multivariate

marginals

and

a

given

correlation

matrix

This

classical

approach

seems

to

give

a

too

restrictive

distribution

for

an

uncertainty

analysis

The

approach

taken

in

this

paper

can

be

formulated

as

a

convex

optimization

problem

The

dual

formulation

of

this

problem

see

for

example

-LSB-

-RSB-

can

be

interpreted

as

the

maximum

likelihood

method

for

estimating

the

parameters

of

a

distribution

of

the

exponential

family

that

best

fits

a

given

sample

of

the

distribution

one

is

looking

for

It

is

also

this

dual

formulation

of

the

problem

which

is

an

uncontrained

optimization

problem

that

has

been

used

to

numerically

calculate

distributions

that

have

maximal

entropy

with

respect

to

a

specific

class

of

distributions

-LSB-

-RSB-

For

overviews

of

multivariate

probability

distributions

studied

in

the

past

see

for

example

-LSB-

-RSB-

In

contrast

to

these

lists

of

distributions

we

find

in

-LSB-

-RSB-

a

derivation

of

the

formulas

of

many

known

distributions

by

using

an

entropy

maximization

principle

applied

to

naturally

defined

classes

of

probability

distributions

Other

probability

distribution

related

research

found

in

the

literature

are

concepts

of

dependence

and

of

majorization

theory

-LSB-

-RSB-

A

recent

survey

on

bivariate

probability

distributions

is

-LSB-

-RSB-

It

collects

references

and

overviews

of

almost

all

non-entropic

items

in

bivariate

distribution

theory

Entropy

maximization

techniques

have

been

applied

fruitfully

in

many

fields

of

science

such

as

statistical

mechanics

-LSB-

-RSB-

queueing

theory

-LSB-

-RSB-

statistical

estimation

theory

-LSB-

-RSB-

and

many

more

General

literature

on

entropy

is

e.g.

-LSB-

-RSB-

In

this

paper

we

take

the

approach

of

first

stating

the

properties

we

want

the

distributions

to

have

and

then

trying

to

derive

the

formulas

for

them

We

use

an

entropy

maximization

approach

because

we

want

the

distribution

to

be

the

most

random

one

within

the

class

of

all

probability

distributions

with

the

same

desired

properties

The

paper

is

organized

as

follows

In

Section

we

discuss

in

short

some

relevant

topics

on

joint

distributions

dependence

and

entropy

In

Section

we

formulate

the

convex

optimization

problem

and

the

main

results

The

proof

of

the

main

theorem

is

deferred

until

we

have

examined

the

discretized

problem

in

Section

In

Section

we

calculate

the

limit

of

the

discretized

problem

and

show

that

this

limit

is

the

solution

of

the

original

continuous

problem

An

example

of

interesting

formulas

that

can

be

derived

once

one

has

this

result

is

calculated

in

Section

In

Section

some

numerical

results

are

shown

The

paper

ends

with

some

concluding

remarks

on

further

possible

generalizations

and

on

further

problems

which

may

be

approached

by

applying

the

same

solution

technique

developed

in

this

paper

Acknowledgment

We

would

like

to

thank

Roger

Cooke

Isaac

Meilijson

Michael

Keane

and

Jos

van

Kan

for

the

many

discussion

with

us

about

this

topic

Their

active

thinking

has

contributed

to

this

paper

To

be

mentioned

are

the

ideas

for

Theorem

of

Jos

van

Kan

the

ideas

for

Lemmas

of

both

Mike

and

Jos

and

Isaac

Meilijson

's

contribution

to

the

numerical

algorithm

in

Section

Joint

Distributions

Dependence

and

Entropy

Dependence

between

random

variables

is

often

focussed

on

dependence

between

pairs

of

random

variables

Let

us

restrict

ourselves

to

the

bivariate

case

Consider

the

pair

of

random

variables

X

Y

with

joint

distribution

FX

Y

-LRB-

x

y

-RRB-

and

marginal

distributions

FX

-LRB-

x

-RRB-

FY

-LRB-

y

-RRB-

respectively

In

the

past

various

measures

of

dependence

have

been

studied

in

various

contexts

see

-LSB-

-RSB-

Many

of

these

express

a

degree

of

congruence

between

the

two

random

variables

X

Y

i.e.

express

how

often

large

-LRB-

small

-RRB-

values

of

X

tend

to

come

together

with

large

-LRB-

small

-RRB-

values

of

Y

The

most

well

known

measure

of

dependence

is

the

correlation

Definition

The

correlation

ae

-LRB-

X

Y

-RRB-

of

X

Y

is

defined

as

ae

-LRB-

X

Y

-RRB-

covfX

Y

g

pvarfXgvarfY

g

The

correlation

value

of

two

random

variables

expresses

the

degree

of

linear

relationship

between

X

and

Y

If

X

Y

are

independent

then

ae

-LRB-

X

Y

-RRB-

On

the

other

hand

Y

is

a

linear

function

of

X

if

and

only

if

ae

-LRB-

X

Y

-RRB-

The

correlation

is

not

invariant

under

non-affine

transformations

of

X

and/or

Y

A

measure

of

dependence

which

is

invariant

under

general

strictly

increasing

-LRB-

decreasing

-RRB-

transformations

of

X

and/or

Y

is

the

so

called

rank

correlation

Definition

The

rank

-LRB-

or

grade

-RRB-

correlation

aer

-LRB-

X

Y

-RRB-

of

X

Y

is

defined

as

ae

-LRB-

FX

-LRB-

X

-RRB-

FY

-LRB-

Y

-RRB-

-RRB-

i.e.

aer

-LRB-

X

Y

-RRB-

-LRB-

X

-RRB-

FY

-LRB-

Y

-RRB-

g

The

constant

in

the

definition

stems

from

the

variance

of

uniform

-LRB-

-RRB-

distributed

variables

The

rank

correlation

measures

the

degree

of

monotone

relationship

between

X

and

Y

It

expresses

the

tendency

of

large

-LRB-

small

-RRB-

values

of

X

coming

together

with

large

-LRB-

small

-RRB-

values

of

Y

where

x

is

measured

with

respect

to

FX

X

x

is

called

large

-LRB-

small

-RRB-

with

respect

to

FX

if

and

only

if

FX

-LRB-

x

-RRB-

is

small

-LRB-

large

-RRB-

We

concentrate

in

this

paper

on

the

rank

correlation

as

a

measure

of

dependence

between

random

variables

for

several

reasons

First

of

all

it

resembles

the

well

known

correlation

and

second

it

is

a

distribution-free

measure

of

dependence

Many

characteristics

of

arbitrary

joint

distributions

are

distribution

free

i.e.

are

preserved

when

the

marginals

are

transformed

to

uniform

-LRB-

-RRB-

variables

There

exists

a

vast

literature

on

this

uniform

representation

or

copula

of

joint

distributions

-LSB-

-RSB-

This

paper

focusses

attention

mostly

on

a

pair

of

two

uniform

-LRB-

-RRB-

variables

with

a

given

correlation

We

prove

-LRB-

see

Theorem

-RRB-

that

after

application

of

a

non-linear

continuous

inverse-transform

on

the

marginals

a

distribution

is

obtained

with

arbitrary

marginals

and

a

given

rank

or

grade

correlation

and

that

this

distribution

has

maximal

entropy

as

well

Thus

basically

the

approach

we

follow

is

distribution

free

Let

X

be

a

discrete

random

variable

taking

values

in

xng

with

distribution

given

by

PX

-LRB-

X

xi

-RRB-

pi

pi

P

pi

-LRB-

i

n

-RRB-

xn

Definition

The

entropy

Hn

-LRB-

PX

-RRB-

of

the

discrete

distribution

PX

is

Hn

-LRB-

PX

-RRB-

n

X

i

pi

log

-LRB-

pi

-RRB-

The

entropy

Hn

-LRB-

-RRB-

is

positive

for

each

P

and

strictly

convex

In

the

definition

we

take

log

Changing

the

base

of

the

logarithm

changes

Hn

-LRB-

P

-RRB-

by

a

factor

We

take

base

e

It

is

well

known

and

easy

to

prove

that

Hn

-LRB-

P

-RRB-

is

maximal

for

the

uniform

distribution

i.e.

pi

It

is

minimal

if

there

exists

an

i

i

n

for

which

pi

Intuitively

one

could

say

that

more

entropy

corresponds

to

more

randomness

The

notion

of

entropy

can

not

be

generalized

to

continuous

densities

fX

-LRB-

x

-RRB-

since

the

summation

in

definition

-LRB-

-RRB-

would

become

Z

log

-LRB-

fX

-LRB-

x

-RRB-

dx

-RRB-

fX

-LRB-

x

-RRB-

dx

which

is

nonsense

Leaving

out

the

dx

in

the

argument

of

the

logarithm

makes

the

expression

scale

dependent

Therefore

the

notion

of

relative

information

has

been

introduced

-LSB-

-RSB-

Let

Y

be

another

discrete

random

variable

that

takes

values

in

xng

with

probabilities

given

by

the

distribution

PY

-LRB-

Y

xi

-RRB-

qi

qi

P

qi

-LRB-

i

n

-RRB-

xn

Definition

The

relative

information

In

-LRB-

PX

jPY

-RRB-

of

a

discrete

distribution

PX

with

respect

to

the

discrete

distribution

PY

is

In

-LRB-

PX

jPY

-RRB-

n

X

i

pi

log

-LRB-

pi

qi

-RRB-

Suppose

PX

PY

then

In

-LRB-

PX

jPY

-RRB-

if

PX

is

more

concentrated

than

PY

for

example

pi

qi

-LRB-

i

n

-RRB-

then

In

-LRB-

PX

jPY

-RRB-

is

positive

Conversely

the

relative

information

of

PX

with

respect

to

PY

is

negative

if

PX

is

less

concentrated

The

generalization

of

relative

information

to

continuous

distributions

with

densities

fX

-LRB-

x

-RRB-

fY

-LRB-

y

-RRB-

is

straightforward

I

-LRB-

fX

jfY

-RRB-

Z

Y

log

-LRB-

fX

-LRB-

-RRB-

fY

-LRB-

-RRB-

-RRB-

fX

-LRB-

-RRB-

d

-LRB-

-RRB-

-LRB-

we

assume

that

if

y

Y

then

fY

-LRB-

y

-RRB-

-RRB-

We

choose

to

model

the

-LRB-

joint

-RRB-

distribution

on

which

we

have

partial

information

by

that

distribution

that

has

minimal

relative

information

with

respect

to

the

uniform

density

on

the

unit

square

or

equivalently

by

the

distribution

with

maximal

entropy

among

all

distributions

which

satisfy

the

given

information

-LRB-

e.g.

on

marginals

and

correlations

-RRB-

The

principle

of

entropy

maximization

which

intuitively

corresponds

to

the

maximization

of

randomness

is

equivalent

to

the

minimization

of

the

relative

information

with

respect

to

the

uniform

distribution

-LRB-

density

-RRB-

Since

on

the

unit

interval

the

density

of

the

uniform

distribution

equals

Addition

of

extra

information

yields

in

this

context

a

distribution

which

has

less

entropy

So

conversely

a

distribution

that

is

not

the

distribution

with

maximal

entropy

among

all

distributions

with

the

given

properties

satisfies

some

extra

-LRB-

hidden

-RRB-

constraints

The

Optimization

Problem

and

its

Solution

Main

Result

For

notational

convenience

we

consider

a

continuous

bivariate

distribution

with

density

fX

Y

-LRB-

x

y

-RRB-

having

uniform

marginals

with

expectation

standard

deviation

oe

and

correlation

ae

The

support

of

fX

Y

is

restricted

to

the

shifted

unit

square

A

-LRB-

-RSB-

-LRB-

-RSB-

Let

F

be

the

class

of

all

probability

densities

on

A

that

have

uniform

marginals

and

let

Fae

be

the

subclass

of

all

distributions

with

uniform

marginals

and

a

correlation

ae

ae

-LSB-

-LSB-

-RSB-

Fae

F

Note

that

the

uniform

distribution

u

on

the

unit

square

is

a

member

of

and

that

and

F

each

contain

only

one

density

the

Fr?echet

bounds

with

correlation

and

respectively

Each

Fae

is

closed

with

respect

to

convex

combinations

The

distribution

with

density

fX

Y

Fae

that

has

minimal

relative

information

I

-LRB-

fX

Y

ju

-RRB-

among

all

distributions

in

Fae

with

respect

to

the

uniform

distribution

u

is

a

solution

of

the

following

optimization

problem

minimize

Z

A

fX

Y

-LRB-

x

y

-RRB-

log

-LRB-

fX

Y

-LRB-

x

y

-RRB-

-RRB-

dxdy

subject

to

Z

fX

Y

-LRB-

x

y

-RRB-

dx

-LRB-

-RSB-

Z

fX

Y

-LRB-

x

y

-RRB-

dy

-LRB-

-RSB-

-LRB-

-RRB-

Z

A

xyfX

Y

-LRB-

x

y

-RRB-

dxdy

fX

Y

-LRB-

x

y

-RRB-

Theorem

The

solution

fX

Y

-LRB-

x

y

-RRB-

of

-LRB-

-RRB-

equals

fX

Y

-LRB-

x

y

-RRB-

-LRB-

x

-RRB-

-LRB-

y

-RRB-

e?xy

-LRB-

-RRB-

where

the

coefficients

m

i

of

the

Taylor

series

expansion

-LRB-

x

-RRB-

X

m

m

X

i

m

-LRB-

-RRB-

are

given

by

the

recursive

relation

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

m

i

-LRB-

-RRB-

m?i

X

k

kX

j

k

j

-LRB-

-RRB-

j

A

for

i

m

and

-LRB-

-RRB-

m

m

X

k

kX

j

k

j

-LRB-

-RRB-

A

m

X

s

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

-LRB-

-LRB-

-RRB-

-RRB-

kX

j

k

j

-LRB-

-RRB-

j

A

s?k

X

j

s?k

j

-LRB-

-RRB-

j

A

-LRB-

-RRB-

The

relation

between

and

ae

-LRB-

X

Y

-RRB-

is

given

by

ae

-LRB-

-RRB-

X

n

bn

-LRB-

-RRB-

The

coefficients

bn

of

this

series

are

given

by

the

following

expression

in

the

k

i.

bn

n

X

s

sX

k

-LRB-

-LRB-

-RRB-

n?s

X

m

-LRB-

m

k

k

n?s?m

X

l

n?s?m

l

s

-RRB-

-RRB-

-LRB-

-RRB-

We

stress

the

fact

that

as

we

have

a

power

series

expansion

for

the

density

fX

Y

it

is

straightforward

to

derive

expressions

for

conditional

distributions

conditional

expectations

etcetera

We

illustrate

this

in

this

paper

by

the

derivation

of

the

relation

between

and

ae

-LRB-

Section

-RRB-

In

the

following

sections

we

prove

this

theorem

First

we

prove

in

Section

a

similar

theorem

for

step-densities

i.e.

densities

that

are

constant

on

little

squares

In

Section

the

theorem

is

proven

We

show

that

simply

taking

limits

suffices

to

obtain

the

solution

of

the

initial

problem

Further

Properties

of

the

Solution

The

following

theorem

states

that

it

is

sufficient

to

consider

joint

distributions

with

uniform

marginals

that

have

maximal

entropy

given

a

correlation

value

By

means

of

a

simple

transformation

of

the

random

variables

a

joint

distribution

with

-LRB-

arbitrary

-RRB-

given

marginal

distributions

and

a

given

rank

correlation

value

is

obtained

and

this

distribution

has

the

same

maximal

entropy

property

Theorem

Consider

the

random

variables

X

and

Y

with

continuous

distributions

GX

and

GY

respectively

and

with

compact

support

Let

the

densities

gX

gY

be

nonzero

on

the

essential

range

of

X

Y

The

joint

probability

density

gX

Y

-LRB-

x

y

-RRB-

with

given

marginal

densities

gX

-LRB-

x

-RRB-

gY

-LRB-

y

-RRB-

and

rank

correlation

aer

-LRB-

X

Y

-RRB-

ae

that

has

minimal

relative

information

with

respect

to

the

product

density

gX

gY

equals

gX

Y

-LRB-

x

y

-RRB-

fU

V

-LRB-

GX

-LRB-

x

-RRB-

GY

-LRB-

y

-RRB-

-RRB-

gX

-LRB-

x

-RRB-

gY

-LRB-

y

-RRB-

-LRB-

-RRB-

where

fU

V

-LRB-

u

v

-RRB-

is

the

density

with

uniform

-LRB-

-RRB-

marginals

and

correlation

coefficient

ae

-LRB-

U

V

-RRB-

ae

that

has

minimal

relative

information

with

respect

to

the

uniform

distribution

on

the

unit

square

Proof

The

joint

density

gX

Y

-LRB-

x

y

-RRB-

we

are

looking

for

is

the

solution

of

the

following

optimization

problem

minimize

Z

Z

gX

Y

-LRB-

x

y

-RRB-

log

gX

Y

-LRB-

x

y

-RRB-

gX

-LRB-

x

-RRB-

gY

-LRB-

y

-RRB-

dxdy

subject

to

Z

gX

Y

-LRB-

x

y

-RRB-

dy

gX

-LRB-

x

-RRB-

Z

gX

Y

-LRB-

x

y

-RRB-

dx

gY

-LRB-

y

-RRB-

Z

Z

GX

-LRB-

x

-RRB-

GY

-LRB-

y

-RRB-

gX

Y

-LRB-

x

y

-RRB-

dxdy

ae

This

optimization

problem

is

transformed

by

letting

u

GX

-LRB-

x

-RRB-

v

GY

-LRB-

y

-RRB-

and

consequently

dx

du

gX

-LRB-

x

-RRB-

dy

dv

gY

-LRB-

y

-RRB-

to

minimize

Z

Z

gX

Y

-LRB-

G

-LRB-

u

-RRB-

G

-LRB-

v

-RRB-

-RRB-

gX

-LRB-

G

-LRB-

u

-RRB-

-RRB-

gY

-LRB-

G

-LRB-

v

-RRB-

-RRB-

log

gX

Y

-LRB-

G

-LRB-

u

-RRB-

G

-LRB-

v

-RRB-

-RRB-

gX

-LRB-

G

-LRB-

u

-RRB-

-RRB-

gY

-LRB-

G

-LRB-

v

-RRB-

-RRB-

dudv

subject

to

Z

gX

Y

-LRB-

G

-LRB-

u

-RRB-

G

-LRB-

v

-RRB-

-RRB-

gX

-LRB-

G

-LRB-

u

-RRB-

-RRB-

gY

-LRB-

G

-LRB-

v

-RRB-

-RRB-

dv

Z

gX

Y

-LRB-

G

-LRB-

u

-RRB-

G

-LRB-

v

-RRB-

-RRB-

gX

-LRB-

G

-LRB-

u

-RRB-

-RRB-

gY

-LRB-

G

-LRB-

v

-RRB-

-RRB-

du

Z

Z

uv

gX

Y

-LRB-

G

-LRB-

u

-RRB-

G

-LRB-

v

-RRB-

-RRB-

gX

-LRB-

G

-LRB-

u

-RRB-

-RRB-

gY

-LRB-

G

-LRB-

v

-RRB-

-RRB-

dudv

ae

Now

write

fU

V

-LRB-

u

v

-RRB-

for

gX

Y

-LRB-

G

-LRB-

u

-RRB-

G

-LRB-

v

-RRB-

-RRB-

gX

-LRB-

G

-LRB-

u

-RRB-

-RRB-

gY

-LRB-

G

-LRB-

v

-RRB-

-RRB-

Definition

-LRB-

total

positivity

-RRB-

A

density

fX

Y

-LRB-

x

y

-RRB-

is

called

totally

positive

of

degree

n

-LRB-

TPn

-RRB-

if

and

only

if

for

all

xn

and

for

all

yn

matrix

M

with

elements

mi

j

fX

Y

-LRB-

xi

yj

-RRB-

obeys

det

-LRB-

M

-RRB-

If

the

same

equation

holds

with

the

sign

reversed

fX

Y

is

called

totally

negative

of

degree

n

-LRB-

TNn

-RRB-

-LSB-

-RSB-

Total

positivity

of

a

distribution

may

be

associated

with

smoothness

of

the

distribution

See

-LSB-

-RSB-

for

a

brief

overview

of

concepts

of

smoothness

Most

of

them

are

implied

by

Total

Positivity

For

example

the

joint

Gaussian

distribution

with

positive

correlation

coefficient

is

-LSB-

-RSB-

Theorem

Let

the

probability

densities

gX

gY

be

non-zero

on

the

essential

range

of

the

random

variables

X

Y

Let

the

pair

of

random

variables

X

Y

be

distributed

according

to

the

joint

probability

density

gX

Y

that

minimizes

I

-LRB-

gX

Y

jgX

gY

-RRB-

given

the

marginal

densities

gX

and

gY

and

given

rank

correlation

aer

-LRB-

X

Y

-RRB-

Then

for

all

n

IN

gX

Y

is

TPn

if

and

only

if

aer

-LRB-

X

Y

-RRB-

and

is

TNn

if

and

only

if

aer

-LRB-

X

Y

-RRB-

Proof

Let

fU

V

-LRB-

u

v

-RRB-

denote

the

joint

Uniform

-LRB-

-RRB-

distribution

with

correlation

ae

that

has

minimal

relative

information

with

respect

to

the

uniform

distribution

on

the

unit

square

As

we

have

that

gX

-LRB-

x

-RRB-

gY

-LRB-

y

-RRB-

and

-LRB-

x

-RRB-

-LRB-

y

-RRB-

are

strictly

positive

we

only

have

to

show

that

det

-LSB-

M

-RSB-

with

mi

j

e?xiyj

This

follows

because

the

matrix

G

with

elements

gi

j

g

-LRB-

X

Y

-RRB-

-LRB-

xi

yj

-RRB-

gX

-LRB-

xi

-RRB-

gY

-LRB-

yj

-RRB-

-LRB-

xi

-RRB-

-LRB-

yj

-RRB-

e?xiyj

can

be

written

as

a

product

of

matrices

G

AMH

where

A

is

the

diagonal

matrix

with

entries

gX

-LRB-

xi

-RRB-

-LRB-

xi

-RRB-

and

H

is

the

diagonal

matrix

with

entries

gY

-LRB-

yj

-RRB-

-LRB-

yj

-RRB-

That

the

n?n

matrix

M

has

a

positive

determinant

is

stated

in

-LSB-

-RSB-

pp.

actually

proven

on

pp.

The

total

positivity

-LRB-

negativity

-RRB-

of

degree

of

gX

Y

however

is

easy

to

show

We

know

that

-LRB-

ae

-RRB-

if

and

only

if

ae

so

we

obtain

det

-LRB-

M

-RRB-

which

equals

e

-LRB-

ae

-RRB-

-LRB-

ae

-RRB-

e

-LRB-

ae

-RRB-

e

-LRB-

ae

-RRB-

which

is

equivalent

to

-LRB-

ae

-RRB-

-LRB-

-RRB-

-LRB-

ae

-RRB-

-LRB-

-RRB-

which

gives

in

case

ae

-LRB-

-RRB-

-LRB-

-RRB-

which

is

true

by

assumption

for

ae

obviously

is

proven

in

the

same

way

Discretization

Introduction

In

order

to

avoid

technical

complexities

we

first

pay

attention

to

a

problem

similar

to

-LRB-

-RRB-

which

is

finite

dimensional

We

give

two

equivalent

formulations

of

it

In

one

we

consider

densities

fX

Y

that

are

constant

on

small

intervals

In

this

construction

all

integrals

in

-LRB-

-RRB-

transform

to

finite

sums

which

eases

the

analysis

In

the

other

one

we

consider

a

discrete

probability

distribution

on

points

It

is

easily

seen

by

a

simple

reformulation

that

this

discretized

problem

is

equivalent

to

the

classical

problem

considered

in

the

literature

where

one

is

maximizing

entropy

of

discrete

distributions

under

finitely

many

constraints

However

the

continuous

version

-LRB-

-RRB-

of

this

particular

problem

differs

from

the

continuous

version

of

the

classical

problem

as

in

this

case

the

number

of

constraints

does

not

stay

finite

-LSB-

-RSB-

In

the

literature

we

have

not

found

the

method

developed

in

this

paper

to

solve

the

classical

problem

It

uses

the

implicit

function

theorem

and

exploits

the

nice

structure

of

the

constraints

We

show

in

the

next

section

that

the

results

of

this

method

can

be

carried

over

to

the

continuous

case

Restriction

to

Step-densities

Let

n

IN

and

let

A

-LRB-

n

-RRB-

i

j

-LRB-

i

in

-RSB-

-LRB-

j

jn

-RSB-

i

j

ng

Further

let

-LRB-

x

-LRB-

n

-RRB-

i

y

-LRB-

n

-RRB-

j

-RRB-

-LRB-

n

n

-RRB-

denote

the

center

point

of

the

square

A

-LRB-

n

-RRB-

i

j

Definition

-LRB-

Step-density

of

order

n

-RRB-

A

step-density

fX

Y

of

order

n

is

a

den

sity

on

the

-LRB-

shifted

-RRB-

unit

square

that

is

constant

on

the

small

squares

A

-LRB-

n

-RRB-

i

j

i.e.

fX

Y

-LRB-

x

y

-RRB-

f

-LRB-

n

-RRB-

i

j

if

and

only

if

-LRB-

x

y

-RRB-

A

-LRB-

n

-RRB-

i

j

Now

we

restrict

in

the

optimization

problem

-LRB-

-RRB-

the

class

of

densities

F

to

the

subclass

of

step-densities

of

order

n

F

-LRB-

n

-RRB-

ae

F

Note

that

also

all

F

-LRB-

n

-RRB-

ae

-LRB-

and

F

-LRB-

n

-RRB-

-RRB-

are

closed

for

taking

convex

combinations

It

is

a

matter

of

simple

calculation

to

derive

-LRB-

-RRB-

from

-LRB-

-RRB-

minimize

n

X

i

n

X

j

f

-LRB-

n

-RRB-

i

j

log

-LRB-

f

-LRB-

n

-RRB-

i

j

-RRB-

subject

to

n

n

X

j

f

-LRB-

n

-RRB-

i

j

-LRB-

i

n

-RRB-

n

n

X

i

f

-LRB-

n

-RRB-

i

j

-LRB-

j

n

-RRB-

-LRB-

-RRB-

n

X

i

n

X

j

x

-LRB-

n

-RRB-

i

y

-LRB-

n

-RRB-

j

f

-LRB-

n

-RRB-

i

j

ae

and

f

-LRB-

n

-RRB-

i

j

This

is

a

convex

optimization

problem

with

linear

constraints

in

variables

f

-LRB-

n

-RRB-

i

j

It

is

easily

seen

that

it

has

at

least

one

solution

And

by

the

strict

convexity

of

the

objective

function

and

the

linearity

of

the

constraints

it

directly

follows

that

-LRB-

-RRB-

has

a

unique

solution

f

-LRB-

n

-RRB-

i

j

for

each

ae

-LSB-

-RSB-

We

remark

that

strictly

speaking

only

uniformity

constraints

are

a

set

of

independent

restrictions

the

one

might

as

well

be

dropped

For

reasons

of

symmetry

we

write

it

down

Later

by

virtue

of

the

symmetry

of

the

correlation

constraint

in

x

y

it

will

be

seen

easily

that

even

only

n

independent

uniformity

restrictions

remain

Theorem

The

solution

f

-LRB-

n

-RRB-

i

j

-LRB-

i

j

n

-RRB-

of

-LRB-

-RRB-

equals

f

-LRB-

n

-RRB-

i

j

-LRB-

n

-RRB-

i

-LRB-

-LRB-

n

-RRB-

-RRB-

-LRB-

n

-RRB-

j

-LRB-

-LRB-

n

-RRB-

-RRB-

e

-LRB-

n

-RRB-

x

-LRB-

n

-RRB-

i

y

-LRB-

n

-RRB-

j

-LRB-

-RRB-

where

the

coefficients

-LRB-

n

-RRB-

i

m

of

the

Taylor

series

expansion

-LRB-

n

-RRB-

i

-LRB-

-LRB-

n

-RRB-

-RRB-

X

m

-LRB-

n

-RRB-

i

m

-LRB-

-LRB-

n

-RRB-

-RRB-

are

given

by

the

recursive

relation

-LRB-

i

n

m

-RRB-

-LRB-

n

-RRB-

i

-LRB-

n

-RRB-

i

m

m

X

s

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

n

n

X

j

j

-LRB-

-RRB-

j

k

A

n

n

X

j

j

-LRB-

-RRB-

j

s?k

A

i

-LRB-

-RRB-

A

-LRB-

-RRB-

In

the

remainder

of

this

paper

we

do

not

write

the

suffix

-LRB-

n

-RRB-

except

when

confusion

might

arise

by

not

writing

it

Using

Lagrange

multipliers

and

the

implicit

function

theorem

a

fairly

straightforward

calculation

suffices

to

show

the

theorem

is

true

around

With

the

help

of

a

lemma

we

show

that

the

theorem

is

true

for

all

IR

The

proof

is

carried

out

in

the

next

subsection

Note

that

the

above

problem

can

be

rewritten

in

the

form

of

an

optimization

problem

for

a

discrete

probability

distribution

if

one

imagines

the

probability

mass

in

the

square

Ai

j

concentrated

in

its

center

point

-LRB-

xi

yj

-RRB-

i.e.

Pf

-LRB-

X

Y

-RRB-

-LRB-

xi

yj

-RRB-

g

pi

j

with

pi

j

Z

Ai

j

fi

j

dxdy

fi

j

-LRB-

-RRB-

Problem

-LRB-

-RRB-

now

becomes

maximize

n

X

i

n

X

j

pi

j

log

-LRB-

pi

j

-RRB-

log

-LRB-

n

-RRB-

subject

to

n

X

j

pi

j

n

-LRB-

i

n

-RRB-

n

X

i

pi

j

n

-LRB-

j

n

-RRB-

n

X

i

n

X

j

xiyjpi

j

ae

and

pi

j

Note

that

we

have

changed

the

sign

of

the

objective

function

This

formulation

reveales

that

for

all

n

IN

minimization

of

the

relative

information

with

respect

to

the

uniform

distribution

equals

maximization

of

the

entropy

of

the

discretized

distribution

Obviously

one

obtains

pi

j

fi

j

i

-LRB-

-RRB-

n

j

-LRB-

-RRB-

n

e?xiyj

-LRB-

-RRB-

where

the

coefficients

of

the

Taylor

series

expansion

of

<=-LRB-

-RRB-

are

given

by

the

same

recurrence

relation

-LRB-

-RRB-

In

the

next

subsection

we

solve

the

step-densities

formulation

of

the

problem

Solving

the

Problem

Restricted

to

Step-densities

Lagrange

multipliers

We

introduce

Lagrange

multipliers

i

j

connected

with

the

uniformity

restrictions

and

the

correlation

restriction

respectively

The

Lagrangian

L

of

-LRB-

-RRB-

becomes

L

n

X

i

n

X

j

fi

j

log

-LRB-

fi

j

-RRB-

n

X

i

i

-LRB-

n

n

X

j

fi

j

-RRB-

n

X

j

j

-LRB-

n

n

X

i

fi

j

-RRB-

-LRB-

n

X

i

n

X

j

xiyjfi

j

ae

-RRB-

-LRB-

-RRB-

Set

the

derivative

of

L

with

respect

to

fi

j

equal

zero

-LRB-

for

a

specific

-LRB-

i

j

-RRB-

-RRB-

and

solve

this

equation

This

gives

fi

j

e

n

i?n?j

xiyj

-LRB-

-RRB-

Symmetry

By

symmetry

one

knows

that

i

i

and

xi

yi

-LRB-

i

n

-RRB-

So

we

can

write

-LRB-

-RRB-

simpler

as

-LRB-

-RRB-

by

taking

i

e

n

i

e

n?i

further

we

take

fi

j

i

j

e?xixj

-LRB-

-RRB-

Substitution

of

these

fi

j

into

the

equations

requiring

uniform

marginals

leads

to

the

nonlinear

system

of

equations

-LRB-

-RRB-

g

-LRB-

-RRB-

-LRB-

-RRB-

where

gi

-LRB-

-RRB-

n

n

X

k

fi

k

n

n

X

k

i

ke?xixk

-LRB-

i

n

-RRB-

-LRB-

-RRB-

Existence

uniqueness

and

analyticity

of

f

around

By

-LRB-

-RRB-

it

is

clear

that

the

one-parameter

density

fae

exists

is

unique

and

analytic

if

and

only

if

exists

is

unique

and

analytic

fae

F

-LRB-

n

-RRB-

ae

is

the

density

that

has

minimal

relative

information

with

respect

to

the

uniform

distribution

u

on

A

The

system

of

equations

-LRB-

-RRB-

implicitely

gives

as

a

function

of

the

Lagrange

multiplier

connected

to

the

correlation

restriction

of

-LRB-

-RRB-

Now

we

prove

the

existence

uniqueness

and

analyticity

of

<=-LRB-

-RRB-

in

a

neigbourhood

of

by

applying

the

implicit

function

theorem

We

show

that

this

<=-LRB-

-RRB-

is

the

unique

and

global

solution

not

only

for

around

but

for

all

IR

by

applying

theory

from

-LSB-

-RSB-

Before

we

can

apply

the

implicit

function

theorem

we

have

to

check

that

-LRB-

i

-RRB-

the

pair

-LRB-

-RRB-

-LRB-

-RRB-

is

a

solution

to

-LRB-

-RRB-

and

that

-LRB-

ii

-RRB-

g

-LRB-

-RRB-

as

a

function

of

-LRB-

-RRB-

is

analytic

These

two

conditions

are

obviously

fullfilled

The

last

condition

to

be

checked

is

-LRB-

iii

-RRB-

the

non-singularity

of

the

Jacobian

G.

Calculation

of

the

derivatives

@gi

j

yields

G

-LSB-

Gi

j

-RSB-

Gi

j

fi

j

j

if

i

j

-LRB-

fi

i

-RRB-

i

if

i

j.

-LRB-

-RRB-

It

follows

from

-LRB-

-RRB-

and

-LRB-

-RRB-

that

both

fi

j

and

i

-LRB-

i

j

n

-RRB-

So

the

diagonal

matrix

D

diag

-LRB-<=-RRB-

is

non-singular

Now

we

can

apply

the

theorem

of

Gershgorin

to

the

product

matrix

GD

We

see

that

this

matrix

is

diagonally

dominant

so

all

its

eigenvalues

must

be

bounded

away

from

Because

D

is

non-singular

it

follows

that

G

is

non-singular

as

well

for

each

solution

fi

j

-LRB-

i

j

n

-RRB-

of

-LRB-

-RRB-

and

especially

also

for

-LRB-

-RRB-

-LRB-

-RRB-

For

the

special

solution

-LRB-

-RRB-

-LRB-

-RRB-

we

have

G

n

n

n

n

with

inverse

G

-LRB-

-RRB-

Theorem

-LRB-

application

of

the

implicit

function

theorem

-RRB-

Let

-LRB-

-RRB-

IRn

such

that

g

-LRB-

-RRB-

e.g.

-LRB-

-RRB-

-LRB-

-RRB-

Because

h

@gi

j

i

-LRB-

-RRB-

is

non

singular

and

g

is

analytic

it

follows

from

the

implicit

function

theorem

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

that

there

exists

an

open

neighbourhood

around

and

an

open

neighbourhood

around

such

that

for

each

there

exists

exactly

one

for

which

g

-LRB-

-RRB-

Write

OE

-LRB-

-RRB-

for

this

unique

then

OE

is

analytic

The

derivative

of

OE

with

respect

to

is

obtained

by

differentiating

-LRB-

-RRB-

using

the

chain

rule

One

finds

dOE

d

h

@gi

j

i

-LRB-

-RRB-

dg

d

-LRB-

-RRB-

Let

X

Y

be

two

uniform

variables

distributed

according

the

density

fX

Y

with

maximal

entropy

fX

Y

has

one

parameter

Corollary

The

correlation

ae

-LRB-

-RRB-

of

X

Y

is

an

analytic

function

ae

-LSB-

-RSB-

because

by

-LRB-

-RRB-

ae

-LRB-

-RRB-

n

X

i

n

X

j

xixj

OEi

-LRB-

-RRB-

OEj

-LRB-

-RRB-

e?xixj

Recursive

relation

for

the

Taylor

series

coefficients

Now

we

derive

the

recursive

relation

for

the

coefficients

of

the

Taylor

series

expansion

of

<=-LRB-

-RRB-

around

This

expansion

exists

because

is

analytic

Furthermore

by

symmetry

it

is

even

i

-LRB-

-RRB-

X

k

i

k

-LRB-

i

n

-RRB-

-LRB-

-RRB-

where

i

-LRB-

i

n

-RRB-

The

uniqueness

and

analyticity

of

implies

by

equation

-LRB-

-RRB-

the

existence

and

uniqueness

of

its

Taylor

series

expansion

coefficients

So

we

can

substitute

-LRB-

-RRB-

into

-LRB-

-RRB-

to

derive

the

recurrence

relation

for

the

coefficients

i

k.

Now

the

coefficients

i

k

can

be

derived

by

differentiating

-LRB-

-RRB-

To

ease

the

calculations

we

first

manipulate

the

sums

We

have

gi

-LRB-

-RRB-

n

n

X

j

X

k

i

k

X

k

j

k

e?xixj

-LRB-

-RRB-

which

using

the

series

expansion

for

the

exponential

function

is

equal

to

gi

-LRB-

-RRB-

n

n

X

j

-LRB-

X

t

-LRB-

tX

s

sX

k

i

s?k

j

k

-LRB-

xixj

-RRB-

-LRB-

-RRB-

-RRB-

X

t

-LRB-

tX

s

sX

k

i

s?k

j

k

-LRB-

xixj

-RRB-

-LRB-

-RRB-

-RRB-

-RRB-

-LRB-

-RRB-

-LRB-

i

n

-RRB-

Now

differentiate

-LRB-

m

-RRB-

times

with

respect

to

and

substitute

Then

the

following

is

obtained

-LRB-

m

-RRB-

gi

-LRB-

-RRB-

d

-LRB-

m

-RRB-

-LRB-

-RRB-

n

n

X

j

m

X

s

-LRB-

xixj

-RRB-

-LRB-

-RRB-

sX

k

i

s?k

j

k

-LRB-

-RRB-

-LRB-

i

n

-RRB-

This

can

be

rewritten

by

taking

the

j

m

apart

and

substituting

j

as

G

m

vm

-LRB-

-RRB-

with

m

B

m

n

m

CA

vm

B

m

vn

m

CA

and

vi

m

n

n

X

j

m

X

s

-LRB-

xixj

-RRB-

-LRB-

-RRB-

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

i

s?k

j

k

A

-LRB-

-RRB-

-LRB-

i

n

-RRB-

So

it

follows

for

m

that

it

can

be

written

as

a

function

of

m

by

virtue

of

the

non-singularity

of

G

i

m

m

X

s

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

n

n

X

j

j

-LRB-

-RRB-

j

k

A

n

n

X

j

j

-LRB-

-RRB-

j

s?k

A

i

-LRB-

-RRB-

A

-LRB-

-RRB-

-LRB-

i

n

-RRB-

This

completes

the

proof

of

Theorem

for

values

of

in

a

neighbourhood

around

In

the

next

subsection

we

use

a

relation

to

statistical

mechanics

to

prove

that

it

is

a

solution

for

all

IR

Relation

to

Statistical

Mechanics

Now

we

will

point

out

the

equivalence

of

the

discretized

problem

with

the

standard

problem

in

entropy

maximization

theory

of

distributions

under

constraints

The

optimization

problem

can

be

reformulated

so

as

to

become

a

special

case

of

a

problem

from

statistical

mechanics

We

follow

the

development

of

Lanford

-LSB-

-RSB-

Forgetting

the

geometric

structure

of

the

points

-LRB-

xi

yj

-RRB-

on

the

unit

square

we

can

consider

the

problem

as

one

where

we

have

N

points

-LRB-

N

-RRB-

let

fkjk

Ng

Now

let

the

k-th

point

bear

probability

mass

pk

-LRB-

k

N

-RRB-

and

define

a

number

of

functions

gi

IR

i

t

and

denote

g

-LRB-

gt

-RRB-

Let

the

expectations

xi

of

the

functions

gi

be

specified

-LRB-

i

t

-RRB-

i

e

Efgg

x

The

functions

g

that

can

be

identified

with

the

uniformity

constraints

in

the

previous

formulation

of

our

model

obviously

have

g

-LRB-

k

-RRB-

or

g

-LRB-

k

-RRB-

for

all

k

The

function

g

that

comes

from

the

correlation

constraint

is

such

that

its

value

in

k

is

equal

to

the

product

of

the

coordinates

of

the

point

in

the

-LRB-

shifted

-RRB-

unit

square

corresponding

with

point

k

Now

the

question

is

to

find

the

probability

vector

-LRB-

pN

-RRB-

with

maximal

entropy

giving

the

functions

g

the

required

expectations

Let

the

dot-product

be

defined

as

usual

by

-LRB-

g

-LRB-

k

-RRB-

IRt

-RRB-

g

-LRB-

k

-RRB-

tX

i

igi

-LRB-

k

-RRB-

For

each

IRt

we

define

Z

-LRB-

-RRB-

N

X

k

e

g

-LRB-

k

-RRB-

and

a

new

probability

vector

p

-LRB-

-RRB-

by

pk

-LRB-

-RRB-

e

g

-LRB-

k

-RRB-

Z

-LRB-

-RRB-

-LRB-

k

N

-RRB-

The

function

Z

-LRB-

-RRB-

plays

the

role

of

the

normalizing

constant

and

is

often

called

free

energy

The

probability

vector

p

-LRB-

-RRB-

is

the

measure

we

are

looking

for

The

only

thing

we

have

to

find

is

the

appropriate

to

give

g

the

required

expectation

x

The

following

two

lemmas

are

found

in

-LSB-

-RSB-

Lemma

-LRB-

lemma

in

-LSB-

-RSB-

-RRB-

If

x

is

in

the

interior

of

the

convex

hull

of

the

essential

range

of

g

then

there

is

a

unique

-LRB-

x

-RRB-

IRt

such

that

N

X

k

pk

-LRB-

-RRB-

g

-LRB-

k

-RRB-

x

Lemma

-LRB-

lemma

in

-LSB-

-RSB-

-RRB-

Let

g

be

a

bounded

measurable

function

on

with

values

in

IRt

and

such

that

the

components

of

g

are

linearly

independent

modulo

the

constants

For

x

in

the

interior

of

the

convex

hull

of

the

essential

range

of

g

let

-LRB-

x

-RRB-

be

the

unique

solution

of

PNk

pk

-LRB-

-RRB-

g

-LRB-

k

-RRB-

x

Then

if

Hg

x

is

the

maximum

entropy

possible

for

a

probability

vector

giving

g

expectation

x

we

have

Hg

x

log

Z

-LRB-

-LRB-

x

-RRB-

-RRB-

x

-LRB-

x

-RRB-

The

probability

vector

p

-LRB-

-LRB-

x

-RRB-

-RRB-

is

the

unique

probability

vector

with

entropy

Hg

x.

Furthermore

Hg

x

is

a

real-analytic

and

strictly

concave

function

of

x

i.e.

has

a

Taylor

series

converging

to

the

value

of

H.

Also

-LRB-

x

-RRB-

gradxHg

x

Applying

these

lemmas

to

our

problem

we

find

that

the

probabilities

pi

j

vary

as

analytic

functions

of

the

correlation

coefficient

ae

In

-LSB-

-RSB-

this

result

was

used

to

calculate

the

maximum

obtainable

entropy

H

We

use

this

result

to

prove

that

the

Taylor

series

of

the

probabilities

for

which

this

maximum

entropy

is

obtained

exists

and

is

unique

for

all

values

ae

-LRB-

-RRB-

We

have

calculated

the

Taylor

series

of

these

functions

by

use

of

the

implicit

function

theorem

The

two

conditions

to

be

fullfilled

in

these

lemmas

are

easily

seen

to

be

true

in

this

case

-LRB-

i

-RRB-

The

functions

g

are

linearly

independent

as

the

Jacobian

G

of

the

implicit

function

theorem

was

always

non-singular

-LRB-

ii

-RRB-

There

exists

always

a

density

satisfying

the

constraints

For

instance

one

can

take

an

appropriate

mixture

of

the

two

Fr?echet

bounds

the

density

having

all

mass

on

the

diagonal

x

y

and

the

density

having

all

mass

on

the

anti-diagonal

x

y.

Also

one

can

construct

a

candidate

solution

by

taking

an

appropriate

mixture

between

the

independent

distribution

and

one

of

the

Fr?echet

bounds

Hence

we

have

seen

that

the

maximal

entropy

measure

exists

is

unique

and

can

be

explicitly

calculated

for

each

-LRB-

ae

-RRB-

IR

Taking

Limits

In

Section

the

existence

uniqueness

and

analyticity

have

been

proven

for

the

step

densities

f

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

n

IN

Here

f

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

n

IN

denotes

the

density

that

has

uniform

-LRB-

-RRB-

marginals

that

has

correlation

ae

and

that

minimizes

I

-LRB-

f

-LRB-

n

-RRB-

ae

ju

-RRB-

the

minimal

relative

information

with

respect

to

the

uniform

density

u

-LRB-

x

y

-RRB-

First

we

will

show

in

this

section

that

in

the

class

of

continuous

densities

the

optimal

density

fae

-LRB-

x

y

-RRB-

if

it

exists

neccessarily

has

a

form

equivalent

to

the

form

of

the

optimal

den

sities

f

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

in

the

classes

of

step-densities

Then

it

will

be

shown

that

f

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

is

uniformely

bounded

and

that

one

can

take

for

all

-LRB-

x

y

-RRB-

A

the

limit

lim

n

f

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

fae

-LRB-

x

y

-RRB-

Further

we

have

not

only

the

existance

of

fae

-LRB-

x

y

-RRB-

but

also

an

expression

for

it

which

is

given

by

the

formulas

-LRB-

-RRB-

-LRB-

-RRB-

The

limit

fae

-LRB-

x

y

-RRB-

is

continuous

is

analytic

and

has

the

same

radius

of

convergence

-LRB-

in

x

-RRB-

as

f

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

Theorem

If

there

exists

a

solution

fae

-LRB-

x

y

-RRB-

of

the

continuous

optimization

problem

-LRB-

-RRB-

then

there

exists

a

IR

and

a

function

-LRB-

x

-RRB-

-LSB-

-RSB-

IR

such

that

fae

-LRB-

x

y

-RRB-

-LRB-

x

-RRB-

-LRB-

y

-RRB-

e?xy

Proof

Let

v

-LRB-

x

y

-RRB-

be

arbitrary

but

fixed

and

such

that

Z

-LRB-

y

-RRB-

v

-LRB-

x

y

-RRB-

dxdy

for

all

-LSB-

-RSB-

IR

-LRB-

-RRB-

Z

-LRB-

x

-RRB-

v

-LRB-

x

y

-RRB-

dxdy

for

all

-LSB-

-RSB-

IR

-LRB-

-RRB-

Z

Z

xy

v

-LRB-

x

y

-RRB-

dxdy

-LRB-

-RRB-

Suppose

fae

exists

i

e

fae

is

the

solution

to

the

continuous

optimization

problem

-LRB-

-RRB-

then

fae

is

unique

by

the

strict

convexity

of

x

log

x

We

now

derive

a

condition

that

fae

must

satisfy

Let

IR

and

consider

f

-LRB-

x

y

-RRB-

fae

-LRB-

x

y

-RRB-

v

-LRB-

x

y

-RRB-

It

is

immediately

verified

that

there

exists

a

constant

E

such

that

f

-LRB-

x

y

-RRB-

satisfies

the

constraints

of

-LRB-

-RRB-

for

all

j

j

E.

Differentiation

of

the

object

function

of

the

optimization

problem

with

respect

to

and

substitution

of

must

yield

as

fae

-LRB-

x

y

-RRB-

is

the

optimal

solution

We

get

by

differentiation

Z

Z

f

-LRB-

x

y

-RRB-

log

f

-LRB-

x

y

-RRB-

dxdy

Z

Z

ff

-LRB-

x

y

-RRB-

log

f

-LRB-

x

y

-RRB-

g

dxdy

Z

Z

v

-LRB-

x

y

-RRB-

logf

fae

-LRB-

x

y

-RRB-

v

-LRB-

x

y

-RRB-

g

v

-LRB-

x

y

-RRB-

dxdy

Now

substitute

and

remember

the

constraint

-LRB-

-RRB-

or

-LRB-

-RRB-

on

v

-LRB-

x

y

-RRB-

to

obtain

the

following

condition

on

fae

-LRB-

x

y

-RRB-

Z

Z

v

-LRB-

x

y

-RRB-

log

fae

-LRB-

x

y

-RRB-

dxdy

-LRB-

-RRB-

Condition

-LRB-

-RRB-

states

that

the

logarithm

of

the

optimal

solution

log

fae

-LRB-

x

y

-RRB-

must

be

perpendicular

to

v

-LRB-

x

y

-RRB-

This

condition

is

satisfied

if

and

only

if

log

fae

-LRB-

x

y

-RRB-

is

a

linear

combination

of

a

function

of

y

only

-LRB-

by

-LRB-

-RRB-

-RRB-

and

a

function

of

x

only

-LRB-

by

-LRB-

-RRB-

-RRB-

and

xy

-LRB-

by

-LRB-

-RRB-

-RRB-

This

means

that

condition

-LRB-

-RRB-

is

equivalent

with

fae

-LRB-

x

y

-RRB-

e

-LRB-

x

-RRB-

-LRB-

y

-RRB-

xy

By

the

symmetry

argument

we

know

that

-LRB-

x

-RRB-

-LRB-

x

-RRB-

and

hence

the

theorem

is

proven

if

we

write

e

-LRB-

x

-RRB-

-LRB-

x

-RRB-

Note

that

an

expression

for

-LRB-

x

-RRB-

may

be

obtained

by

solving

the

integral

equation

-LRB-

x

-RRB-

Z

-LRB-

y

-RRB-

e?xy

dy

This

expression

is

obtained

by

substitution

of

the

expression

for

fae

-LRB-

x

y

-RRB-

in

the

constraint

that

requires

uniform

marginals

One

could

solve

-LRB-

x

-RRB-

from

this

integral

equation

but

we

already

have

a

solution

for

the

problem

restricted

to

step-densities

This

stepdensities

solution

will

give

us

the

solution

for

-LRB-

x

-RRB-

immediately

Next

we

formulate

two

help

lemmas

Recall

the

system

of

equations

-LRB-

-RRB-

coming

from

the

fact

that

we

have

uniform

marginals

n

-LRB-

n

-RRB-

i

-LRB-

-RRB-

n

X

j

-LRB-

n

-RRB-

j

-LRB-

-RRB-

e?x

-LRB-

n

-RRB-

i

x

-LRB-

n

-RRB-

j

i

n

In

matrix-vector

notation

we

may

write

this

as

Mh

n

h

-LRB-

-RRB-

where

the

-LRB-

n

n

-RRB-

matrix

M

-LRB-

mi

j

-RRB-

has

elements

mi

j

e?x

-LRB-

n

-RRB-

i

x

-LRB-

n

-RRB-

j

the

vector

h

-LRB-

hi

-RRB-

has

elements

hi

-LRB-

n

-RRB-

i

-LRB-

-RRB-

and

where

with

we

denote

the

vector

with

elements

In

the

proofs

of

the

following

two

lemmas

we

use

that

hi

for

all

i

n

and

that

-LRB-

x

y

-RRB-

-LSB-

-RSB-

which

implies

that

mi

j

is

bounded

for

all

i

j

n

c

-LRB-

-RRB-

e?j?j

mi

j

ej?j

C

-LRB-

-RRB-

Lemma

There

exists

a

constant

d

-LRB-

n

-RRB-

such

that

Pni

hi

d.

Proof

Write

-LRB-

-RRB-

in

the

form

n

hi

n

X

j

mi

jhj

i

n

and

use

the

upperbound

C

-LRB-

-RRB-

on

mi

j

to

get

n

C

-LRB-

-RRB-

hi

n

X

j

hj

i

n

Sum

over

i

to

obtain

C

-LRB-

-RRB-

X

i

j

hihj

C

-LRB-

-RRB-

n

X

i

hi

then

it

is

clear

that

the

lemma

is

valid

with

d

-LRB-

n

-RRB-

n

pC

-LRB-

-RRB-

ne?j?j

Lemma

There

exists

a

constant

D

-LRB-

n

-RRB-

such

that

Pni

hi

D

-LRB-

n

-RRB-

Proof

As

in

the

proof

of

Lemma

we

get

from

the

other

side

n

c

-LRB-

-RRB-

hi

n

X

j

hj

i

n

and

c

-LRB-

-RRB-

X

i

j

hihj

c

-LRB-

-RRB-

n

X

i

hi

Hence

D

-LRB-

n

-RRB-

n

pc

-LRB-

-RRB-

nej?j

will

do

the

job

With

the

help

of

these

two

lemmas

the

boundedness

of

hi

i

n

is

easy

to

prove

Lemma

There

exist

constants

e

-LRB-

-RRB-

E

-LRB-

-RRB-

such

that

e

-LRB-

-RRB-

hi

E

-LRB-

-RRB-

for

all

i

n.

Proof

Using

-LRB-

-RRB-

again

we

see

that

hi

n

Pnj

mi

jhj

by

Lemmas

and

and

by

using

the

bounds

on

mi

j

we

have

c

-LRB-

-RRB-

d

-LRB-

n

-RRB-

c

-LRB-

-RRB-

n

X

j

hj

n

X

j

mi

jhj

C

-LRB-

-RRB-

n

X

j

hj

C

-LRB-

-RRB-

D

-LRB-

n

-RRB-

or

n

C

-LRB-

-RRB-

D

-LRB-

n

-RRB-

hi

n

c

-LRB-

-RRB-

d

-LRB-

n

-RRB-

Hence

we

can

take

e

-LRB-

-RRB-

n

C

-LRB-

-RRB-

D

-LRB-

n

-RRB-

e

j

and

E

-LRB-

-RRB-

n

c

-LRB-

-RRB-

d

-LRB-

n

-RRB-

Note

that

for

each

-LRB-

fixed

-RRB-

IR

the

constants

e

-LRB-

-RRB-

and

E

-LRB-

-RRB-

are

a

bound

for

-LRB-

n

-RRB-

-LRB-

-RRB-

and

thus

f

-LRB-

n

-RRB-

ae

-LRB-

-RRB-

-LRB-

x

y

-RRB-

is

bounded

independently

of

n.

Theorem

now

gives

the

limit

of

f

-LRB-

n

-RRB-

ae

-LRB-

-RRB-

-LRB-

x

y

-RRB-

for

n

Theorem

For

all

-LRB-

x

y

-RRB-

-LSB-

-RSB-

the

limit

lim

n

f

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

fae

-LRB-

x

y

-RRB-

exists

fae

-LRB-

x

y

-RRB-

is

given

by

fae

-LRB-

x

y

-RRB-

-LRB-

x

-RRB-

-LRB-

y

-RRB-

e?xy

with

-LRB-

x

-RRB-

X

k

m

-LRB-

x

-RRB-

For

the

coefficients

m

-LRB-

x

-RRB-

we

have

the

recurrence

relation

m

-LRB-

x

-RRB-

m

X

s

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

-LRB-

Z

-LRB-

-RRB-

k

-LRB-

-RRB-

d

Z

-LRB-

-RRB-

s?k

-LRB-

-RRB-

d

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

for

m

while

-LRB-

x

-RRB-

Proof

First

some

notation

Let

-LSB-

x

-RSB-

denote

the

entier

function

Let

x

-LRB-

-RSB-

and

let

-LSB-

x

n

-RSB-

-LSB-

-LRB-

x

-RRB-

-LRB-

n

-RRB-

-RSB-

i.e.

-LSB-

x

n

-RSB-

is

the

index

i

of

the

x-coordinate

of

the

center

point

of

the

little

interval

containing

x

at

the

n-th

order

discretization

We

must

show

that

lim

n

-LRB-

n

-RRB-

-LSB-

x

n

-RSB-

-LRB-

-RRB-

-LRB-

n

-RRB-

-LSB-

y

n

-RSB-

-LRB-

-RRB-

e?x

-LSB-

x

n

-RSB-

y

-LSB-

y

n

-RSB-

-LRB-

x

-RRB-

-LRB-

y

-RRB-

e?xy

It

is

clear

that

this

is

equivalent

with

showing

that

lim

n

X

m

-LSB-

x

n

-RSB-

m

X

m

m

-LRB-

x

-RRB-

For

this

it

suffices

to

show

that

-LRB-

i

-RRB-

for

all

m

IN

and

for

all

x

-LSB-

-RSB-

lim

n

-LSB-

x

n

-RSB-

m

m

-LRB-

x

-RRB-

-LRB-

-RRB-

-LRB-

pointwise

limit

-RRB-

and

that

-LRB-

ii

-RRB-

there

exists

a

constant

E

-LRB-

-RRB-

independent

of

n

such

that

for

all

n

IN

X

m

-LSB-

x

n

-RSB-

m

-LSB-

x

n

-RSB-

-LRB-

-RRB-

E

-LRB-

-RRB-

-LRB-

ii

-RRB-

follows

by

Lemma

now

we

prove

-LRB-

i

-RRB-

As

for

all

x

and

for

all

n

IN

we

have

that

-LSB-

x

n

-RSB-

-LRB-

x

-RRB-

we

can

prove

-LRB-

-RRB-

by

induction

So

if

we

assume

-LRB-

induction

assumption

-RRB-

that

-LRB-

-RRB-

is

true

up

till

m

then

we

have

to

show

for

m

that

lim

n

m

X

s

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

n

n

X

j

j

-LRB-

-RRB-

j

k

A

n

n

X

j

j

-LRB-

-RRB-

j

s?k

A

-LSB-

x

n

-RSB-

-LRB-

-RRB-

A

m

X

s

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

-LRB-

Z

-LRB-

-RRB-

k

-LRB-

-RRB-

d

Z

-LRB-

-RRB-

s?k

-LRB-

-RRB-

d

-LRB-

-RRB-

-RRB-

The

crucial

point

here

is

that

we

have

to

show

that

for

all

s

m

and

for

all

max

-LRB-

s

m

-RRB-

k

min

-LRB-

s

m

-RRB-

that

lim

n

n

n

X

j

j

-LRB-

-RRB-

j

k

Z

-LRB-

-RRB-

k

-LRB-

-RRB-

d

-LRB-

-RRB-

The

right

hand

side

integral

can

be

rewritten

as

Z

-LRB-

-RRB-

k

-LRB-

-RRB-

d

X

i

Z

-LRB-

xi

ffl

-RRB-

-LRB-

-RRB-

k

-LRB-

xi

ffl

-RRB-

dffl

-LRB-

-RRB-

Now

we

apply

the

induction

assumption

From

this

assumption

it

follows

that

for

all

ffi

there

must

exist

an

L

IN

such

that

for

all

L

X

i

Z

-LRB-

xi

ffl

-RRB-

-LRB-

-RRB-

k

-LRB-

xi

ffl

-RRB-

dffl

X

i

-LRB-

xi

-RRB-

-LRB-

-RRB-

k

-LRB-

xi

-RRB-

ffi

-LRB-

-RRB-

This

finishes

the

proof

of

-LRB-

i

-RRB-

as

by

the

induction

assumption

n

implies

j

k

-LRB-

xi

-RRB-

i

k

j

In

this

section

we

have

taken

so

far

IR

arbitrary

but

fixed

We

have

seen

that

f

-LRB-

n

-RRB-

ae

-LRB-

-RRB-

-LRB-

x

y

-RRB-

is

uniformly

bounded

-LRB-

in

n

-RRB-

and

have

calculated

its

limit

fae

-LRB-

-RRB-

-LRB-

x

y

-RRB-

One

must

be

very

carefull

here

f

-LRB-

n

-RRB-

ae

-LRB-

-RRB-

-LRB-

x

y

-RRB-

is

the

step-density

of

order

n

with

parameter

that

has

least

relative

information

with

respect

to

the

uniform

density

in

the

class

of

step-densities

of

order

n

with

uniform

marginals

and

correlation

ae

-LRB-

-RRB-

But

of

course

this

correlation

ae

depends

on

n

as

ae

-LRB-

n

-RRB-

-LRB-

-RRB-

Z

Z

A

xyf

-LRB-

n

-RRB-

-LRB-

x

y

-RRB-

dxdy

For

each

n

IN

f

-LRB-

n

-RRB-

-LRB-

x

y

-RRB-

is

optimal

but

is

this

still

true

if

we

take

the

limit

for

n

We

can

answer

this

question

by

yes

because

of

the

form

of

the

continuous

density

that

solves

the

optimization

problem

Theorem

gives

and

because

of

the

strict

convexity

of

x

log

x.

Theorem

The

continuous

density

fae

-LRB-

x

y

-RRB-

with

uniform

-LRB-

-RRB-

marginals

and

correlation

ae

which

attains

the

least

relative

information

I

-LRB-

faeju

-RRB-

with

respect

to

the

uniform

density

is

unique

Proof

We

prove

the

theorem

in

two

steps

First

suppose

there

exists

a

density

gae

that

has

strictly

less

relative

information

than

fae

with

respect

to

the

uniform

density

say

I

-LRB-

faeju

-RRB-

I

-LRB-

gaeju

-RRB-

ffi

with

ffi

We

can

construct

a

row

of

step-densities

g

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

g

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

F

-LRB-

n

-RRB-

ae

This

row

has

gae

-LRB-

x

y

-RRB-

as

limit

for

n

We

define

for

n

IN

g

-LRB-

n

-RRB-

ae

to

be

that

density

in

F

-LRB-

n

-RRB-

ae

that

minimizes

jjg

-LRB-

n

-RRB-

ae

-LRB-

x

y

-RRB-

gae

-LRB-

x

y

-RRB-

As

I

-LRB-

ju

-RRB-

is

continuous

we

have

that

for

all

ffi

there

exists

an

Ng

IN

such

that

for

all

n

Ng

jI

-LRB-

gaeju

-RRB-

I

-LRB-

g

-LRB-

n

-RRB-

ae

ju

-RRB-

j

ffi

-LRB-

-RRB-

Because

of

the

continuity

of

I

-LRB-

ju

-RRB-

and

Theorem

we

also

know

that

for

all

ffi

there

exists

an

Nf

IN

such

that

for

all

n

Nf

jI

-LRB-

faeju

-RRB-

I

-LRB-

f

-LRB-

n

-RRB-

ae

ju

-RRB-

j

ffi

-LRB-

-RRB-

Now

let

N

max

-LRB-

Nf

Ng

-RRB-

Then

it

follows

from

-LRB-

-RRB-

-LRB-

-RRB-

that

for

all

ffi

and

for

all

n

N

we

have

that

I

-LRB-

f

-LRB-

n

-RRB-

ae

ju

-RRB-

I

-LRB-

g

-LRB-

n

-RRB-

ae

ju

-RRB-

ffi

-LRB-

-RRB-

This

is

in

contradiction

with

the

fact

that

f

-LRB-

n

-RRB-

ae

is

the

step-density

in

F

-LRB-

n

-RRB-

ae

with

the

least

relative

information

with

respect

to

u

and

that

f

-LRB-

n

-RRB-

ae

is

a

unique

optimum

So

we

must

conclude

that

there

can

not

exist

a

continuous

density

gae

that

has

strictly

less

relative

information

with

respect

to

u

then

fae

has

Next

assume

that

there

exists

a

density

gae

fae

such

that

fae

and

gae

have

equal

relative

information

with

respect

to

u

i.e.

I

-LRB-

gaeju

-RRB-

I

-LRB-

faeju

-RRB-

Now

we

consider

the

mixture

ffgae

-LRB-

-RRB-

fae

This

mixture

is

in

Fae

as

well

Because

I

-LRB-

ju

-RRB-

is

strictly

convex

we

have

that

for

all

I

-LRB-

ffgae

-LRB-

-RRB-

faeju

-RRB-

ffI

-LRB-

gaeju

-RRB-

-LRB-

-RRB-

I

-LRB-

faeju

-RRB-

I

-LRB-

faeju

-RRB-

Thus

we

have

constructed

a

density

which

has

strictly

less

relative

information

with

respect

to

the

uniform

density

than

fae

This

is

in

contradiction

with

the

first

step

in

this

proof

that

there

can

not

exist

such

a

density

with

strictly

less

relative

information

with

respect

to

u

So

we

must

conclude

that

there

can

not

exist

a

density

gae

Fae

with

fae

gae

and

I

-LRB-

faeju

-RRB-

I

-LRB-

gaeju

-RRB-

Two

dimensional

Taylor

series

From

the

Taylor

series

expansion

-LRB-

x

-RRB-

X

m

m

-LRB-

x

-RRB-

where

the

m

-LRB-

x

-RRB-

are

given

by

the

recursive

relation

-LRB-

-RRB-

we

will

now

derive

the

formula

given

in

-LRB-

-RRB-

-LRB-

-RRB-

Careful

examination

of

relation

-LRB-

-RRB-

shows

how

to

write

m

-LRB-

x

-RRB-

explicitly

as

a

function

of

x

m

-LRB-

x

-RRB-

m

X

i

m

-LRB-

-RRB-

-LRB-

m

-RRB-

Obviously

we

have

-LRB-

x

-RRB-

Further

more

we

observe

that

for

all

r

IN

Z

m

-LRB-

-RRB-

d

Z

m

X

i

m

i

r

d

m

X

i

m

i

r

So

we

can

rewrite

-LRB-

-RRB-

getting

m

-LRB-

x

-RRB-

m

X

i

m

m

X

s

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

-LRB-

-LRB-

-RRB-

kX

i

k

i

m

-RRB-

s?k

X

i

s?k

i

m

sX

i

s

m

-RRB-

-LRB-

-RRB-

from

which

we

obtain

following

recursion

formula

for

the

coefficients

of

the

two

dimensional

Taylor

series

This

formula

is

stated

earlier

as

formulas

-LRB-

-RRB-

-LRB-

-RRB-

m

i

-LRB-

-RRB-

m?i

X

k

kX

j

k

j

-LRB-

-RRB-

j

A

for

i

m

and

m

m

X

k

kX

j

k

j

-LRB-

-RRB-

A

m

X

s

min

-LRB-

s

m

-RRB-

X

k

max

-LRB-

s?m

-RRB-

-LRB-

-LRB-

-RRB-

-RRB-

kX

j

k

j

-LRB-

-RRB-

j

A

s?k

X

j

s?k

j

-LRB-

-RRB-

j

A

Relation

between

and

ae

The

Taylor

series

expansion

of

the

density

fX

Y

-LRB-

x

y

-RRB-

allows

for

a

straightforward

though

maybe

cumbersome

calculation

of

for

instance

conditional

distributions

and

conditional

densities

As

an

example

of

those

calculations

we

will

derive

the

relation

between

and

ae

in

this

section

We

note

that

EfXg

EfY

g

and

that

varfXg

varfY

g

We

will

see

that

ae

-LRB-

-RRB-

is

an

odd

function

and

that

ae

-LRB-

-RRB-

The

Taylor

series

expansion

of

ae

-LRB-

-RRB-

that

we

will

derive

converges

as

the

analyticity

of

the

density

fX

Y

implies

the

analyticity

of

ae

-LRB-

-RRB-

-LRB-

see

Corollary

-RRB-

ae

-LRB-

-RRB-

Z

Z

xyfX

Y

-LRB-

x

y

-RRB-

dx

dy

Z

Z

X

m

m

X

k

m

X

n

n

X

l

n

e?xy

dx

dy

Z

X

m

m

X

k

m

X

n

n

X

l

n

l

Z

e?xy

dy

dx

Z

X

k

X

m

k

m

k

X

n

n

X

l

n

l

X

i

-LRB-

x

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

i

i

dx

If

we

calculate

further

we

obtain

Z

X

s

sX

k

X

m

k

m

k

X

n

n

X

l

n

l

-LRB-

-RRB-

-LRB-

-RRB-

s

dx

X

s

sX

k

-LRB-

-RRB-

X

m

m

k

k

X

n

n

X

l

n

l

s

X

s

X

t

-LRB-

t

sX

k

-LRB-

-RRB-

tX

m

m

k

k

t?m

X

l

t?m

l

s

-RRB-

X

n

n

X

s

sX

k

ae

-LRB-

-RRB-

n?s

X

m

-LRB-

m

k

k

n?s?m

X

l

n?s?m

l

s

-RRB-

-RRB-

So

we

have

found

the

Taylor

series

expansion

ae

-LRB-

-RRB-

X

n

bn

-LRB-

-RRB-

the

coefficients

bn

of

this

series

are

given

by

the

following

expression

in

the

k

l

bn

n

X

s

sX

k

ae

-LRB-

-RRB-

n?s

X

m

-LRB-

m

k

k

n?s?m

X

l

n?s?m

l

s

-RRB-

-RRB-

-LRB-

-RRB-

Numerical

Results

In

this

section

we

will

show

numerical

results

for

the

relation

between

and

ae

and

for

the

conditional

density

on

the

line

y

Basically

one

can

obtain

numerical

results

in

two

ways

First

one

can

calculate

the

coefficients

of

the

Taylor

series

expansion

of

-LRB-

x

-RRB-

using

the

derived

recurrence

relation

Second

one

can

solve

for

a

discretization

n

the

non-linear

system

of

equations

n

n

X

j

i

j

e?xixj

-LRB-

i

n

-RRB-

n

X

i

n

X

j

xixj

i

j

e?xixj

ae

numerically

This

system

of

non

linear

equations

has

been

derived

from

setting

the

derivatives

of

the

Lagrangian

L

equal

The

first

method

seems

to

be

the

most

elegant

one

but

the

convergence

of

the

series

expansion

is

rather

slow

The

second

methode

applies

to

the

discrete

case

only

but

is

much

more

efficient

One

can

proceed

in

two

ways

to

solve

this

system

of

equations

-LRB-

ae

-RRB-

ae

Figure

The

relationship

between

the

Lagrange

parameter

and

the

correlation

ae

The

first

one

is

to

apply

a

multi-dimensional

Newton-Raphson

method

i

e

to

apply

successive

linearization

Obviously

convergence

of

this

method

is

guaranteed

by

the

strict

convexity

of

the

e?functions

A

similar

approach

can

be

used

as

well

in

a

more

general

case

when

there

are

more

-LRB-

higher

order

-RRB-

cross

product

moment

constraints

than

only

a

correlation

constraint

Another

approach

applies

to

the

correlation

constraint

case

extensively

studied

in

this

paper

only

Let

us

concentrate

on

an

algorithm

for

the

correlation

constraint

case

Rewrite

the

system

of

equations

in

the

following

way

i

n

P

j

je?xiyj

i

n

P

i

j

xiyj

i

j

e?xiyj

ae

-LRB-

-RRB-

The

way

the

equations

-LRB-

-RRB-

are

written

already

suggests

how

to

calculate

the

solution

numerically

Let

-LRB-

k

-RRB-

denote

the

number

of

iterations

and

let

ffl

denotes

some

small

positive

real

number

The

main

steps

of

the

algorithm

are

Fix

a

value

for

ae

Let

k

Fix

values

for

-LRB-

k

-RRB-

i

-LRB-

k

-RRB-

i

-LRB-

-RRB-

-LRB-

i

n

-RRB-

Calculate

f

-LRB-

k

-RRB-

i

j

-LRB-

k

-RRB-

i

-LRB-

k

-RRB-

j

e?xiyj

-LRB-

i

j

n

-RRB-

Normalize

the

f

-LRB-

k

-RRB-

i

j

by

dividing

each

-LRB-

k

-RRB-

i

by

qP

i

j

f

-LRB-

k

-RRB-

i

j

Let

-LRB-

k

-RRB-

i

n

P

j

-LRB-

k

-RRB-

j

e?xiyj

-LRB-

i

n

-RRB-

-LRB-

x

-RRB-

-LRB-

-RRB-

x

ae

ae

ae

Figure

The

conditional

density

f

-LRB-

x

-RRB-

-LRB-

x

-RRB-

-LRB-

-RRB-

of

the

density

f

-LRB-

x

y

-RRB-

that

minimizes

I

-LRB-

f

ju

-RRB-

the

relative

information

with

respect

to

the

uniform

density

for

three

different

correlation

values

k

k

If

k

-LRB-

k

-RRB-

-LRB-

k

-RRB-

k

then

goto

step

Calculate

the

correlation

ae

of

the

maximal

entropy

distribution

P

-LRB-

-RRB-

that

has

been

calculated

If

this

is

not

equal

to

the

desired

ae

then

update

-LRB-

ae

ae

-RRB-

-LRB-

-RRB-

and

goto

step

The

following

notes

on

the

steps

of

the

algorithm

can

be

made

Steps

and

on

-LRB-

ae

-RRB-

To

find

good

starting

values

for

the

algorithm

one

can

of

course

use

the

first

couple

of

coefficients

of

the

Taylor

series

coefficients

derived

in

this

paper

It

appears

however

that

the

correspondence

between

the

bivariate

Gaussian

distribution

-LRB-

with

support

on

-RRB-

and

the

maximal

entropy

distribution

is

close

enough

to

use

well

known

formulas

to

obtain

good

starting

values

It

is

generally

known

-LSB-

-RSB-

-LSB-

-RSB-

that

the

standard

Gaussian

distribution

with

correlation

ae

is

the

distribution

with

maximal

entropy

among

all

distributions

on

with

correlation

ae

and

variance

For

smaller

correlations

the

Gaussian

density

conditional

for

y

is

about

uniform

on

-LSB-

-RSB-

and

for

larger

correlations

this

conditional

density

has

about

no

tails

stretching

outside

the

interval

-LSB-

-RSB-

In

Figure

one

sees

that

this

agrees

very

well

with

the

form

of

the

maximal

entropy

density

conditional

for

y

-LRB-

modulo

a

normalization

constant

-RRB-

By

some

manipulation

the

following

expression

can

be

derived

for

the

maximal

entropy

distribution

with

uniform

marginals

and

given

correlation

with

parameter

-LRB-

ae

-RRB-

fi

j

pfi

i

pfj

j

e

-LRB-

yj

-RRB-

-LRB-

-RRB-

Compare

this

with

the

density

OE

-LRB-

x

y

-RRB-

the

Gaussian

distribution

with

correlation

ae

OE

-LRB-

x

y

-RRB-

pOE

-LRB-

x

x

-RRB-

pOE

-LRB-

y

y

-RRB-

e

ae

-LRB-

-RRB-

-LRB-

-RRB-

The

equivalence

between

these

formulae

suggests

the

following

approximative

relationship

between

and

ae

ss

ae

-LRB-

-RRB-

Note

that

ae

ae

The

approximation

-LRB-

-RRB-

is

not

very

close

but

is

good

enough

to

start

an

efficient

algorithm

The

formula

used

in

update

step

no.

is

the

first

order

Taylor

series

expansion

of

it

i.e.

we

approximate

the

unknown

function

-LRB-

-RRB-

in

a

neighbourhood

of

the

point

ae

by

the

first

order

Taylor

series

expansion

of

-LRB-

-RRB-

In

most

cases

two

or

three

updates

of

are

enough

to

obtain

a

distribution

with

the

desired

correlation

ae

correct

up

more

than

two

or

three

digits

Step

on

-LRB-

-RRB-

i

Use

the

same

equivalence

between

the

relations

-LRB-

-RRB-

and

-LRB-

-RRB-

for

the

calculation

of

the

starting

values

of

i

this

yields

-LRB-

-RRB-

i

e

Steps

and

Normalization

Steps

and

are

during

the

iteration

process

together

taking

care

of

the

normalization

P

fi

j

should

equal

in

the

end

After

step

and

when

convergence

is

detected

in

step

sufficient

normalization

has

to

be

done

During

the

iteration

process

itself

however

the

normalization

steps

need

not

be

done

every

iteration

as

it

appears

that

the

vector

-LRB-

k

-RRB-

hardly

shrinks

or

grows

Step

Updating

the

i

The

formula

given

in

this

step

to

update

the

values

of

the

Lagrange

multipliers

i

is

directly

taken

from

-LRB-

-RRB-

Looking

more

closely

to

the

equation

reveals

that

it

is

quadratic

in

i

The

next

variant

for

the

update

step

uses

this

fact

It

makes

the

convergence

of

the

algorithm

faster

So

we

have

-LRB-

k

-RRB-

i

P

j

i

j

e?xiyj

q

-LRB-

P

j

i

j

e?xiyj

-RRB-

n

-LRB-

i

n

-RRB-

Note

that

only

the

positive

solutions

of

these

quadratic

equations

are

desired

Step

Convergence

Convergence

of

the

algorithm

is

guaranteed

if

in

the

update

step

of

the

algortihm

an

average

is

taken

of

the

old

value

and

the

newly

calculated

one

this

can

be

seen

directly

by

looking

at

the

case

with

n

and

follows

in

general

from

Lemma

which

shows

the

boundedness

of

the

i.

Results

Using

the

algorithm

sketched

above

with

a

discretization

of

n

the

Figures

and

show

some

numerical

results

The

relationship

between

the

Lagrange

parameter

and

the

correlation

ae

is

shown

in

Figure

The

conditional

density

for

y

is

depicted

in

Figure

for

three

different

correlation

values

Concluding

Remarks

A

recurrence

relation

has

been

derived

for

the

coefficients

of

the

Taylor

series

expansion

of

the

distribution

that

has

maximal

entropy

given

the

marginal

distributions

and

given

the

-LRB-

rank

-RRB-

correlation

It

has

been

proven

in

this

paper

that

this

distribution

generally

exists

is

unique

and

analytic

It

is

a

one-parameter

class

of

densities

with

parameter

-LRB-

ae

-RRB-

This

class

includes

the

Fr?echet

bounds

as

limiting

cases

i.e.

the

densities

with

all

mass

on

the

diagonal

y

x

or

with

all

mass

on

the

diagonal

y

x

-LRB-

for

and

respectively

-RRB-

and

it

includes

the

independent

density

xy

-LRB-

for

-RRB-

How

fast

this

Taylor

series

expansion

converges

as

a

function

of

-LRB-

ae

-RRB-

has

not

been

studied

in

detail

A

numerical

algorithm

has

been

given

and

some

results

have

been

shown

The

approach

that

has

been

followed

in

this

paper

to

solve

the

stated

problem

can

be

used

to

solve

any

entropy

maximization

problem

for

distributions

with

momenttype

constraints

like

constraints

on

mean

variance

and

quantiles

The

derivation

of

the

actual

recursion

formulas

however

may

be

more

difficult

if

the

structure

of

the

constraints

is

less

regular

As

a

formula

for

the

coefficients

of

the

Taylor

series

expansion

of

the

density

has

been

derived

it

is

a

matter

of

straightforward

calculation

to

derive

formulas

for

e.g.

conditional

distributions

and

conditional

expectations

As

an

example

the

relation

between

and

ae

has

been

derived

in

this

paper

Generalization

of

the

followed

approach

to

multivariate

distributions

with

n

variables

seems

straightforward

but

prohibitive

at

the

same

time

because

of

the

high

dimensional

Taylor

series

expansions

that

arise

References

-LSB-

-RSB-

N.

Agmon

Y.

Alhassid

R.D.

Levine

An

Algorithm

for

Finding

the

Distribution

of

Maximal

Entropy

J.

of

Comp

Physics

vol

pp

-LSB-

-RSB-

A.

Ben-Tal

A.

Charnes

A

Dual

Optimization

Framework

for

some

Problems

of

Information

Theory

and

Statistics

Problems

of

Control

and

Inf

Theory

vol

pp.

-LSB-

-RSB-

A.

Ben-Tal

M.

Teboulle

A.

Charnes

The

Role

of

Duality

in

Optimization

Problems

Involving

Entropy

Functionals

with

Applications

to

Information

Theory

J.

of

Optimization

Th

and

Appl

vol

pp

-LSB-

-RSB-

C.M.

Cuadras

Probability

Distributions

with

Given

Multivariate

Marginals

and

Given

Dependence

Structure

J.

of

Multiv

Analysis

vol

-LSB-

-RSB-

J.

Dieudonnee

Foundations

of

Modern

Analysis

Academic

Press

New

York

London

-LSB-

-RSB-

J.R.

van

Dorp

Dependence

Modeling

for

Uncertainty

Analysis

Delft

University

of

Technology

The

Netherlands

December

-LSB-

-RSB-

R.M.

Cooke

M.S.

Keane

A.M.H.

Meeuwissen

User

's

Manual

for

RIAN

Computerized

Risk

Assessment

Estec

Delft

University

of

Technology

The

Netherlands

February

-LSB-

-RSB-

S.

Guia?su

Information

Theory

with

Applications

McGraw-Hill

New

York

-LSB-

-RSB-

S.

Guia?su

Maximum

Entropy

Condition

in

Queueing

Theory

J.

Opl

Res

Soc

vol

pp.

-LSB-

-RSB-

T.P.

Hutchinson

C.D.

Lai

Continuous

Bivariate

Distributions

Emphasizing

Applications

Rumsby

Scientific

Publishing

Adelaide

Australia

-LSB-

-RSB-

J.N.

Kapur

Maximum

Entropy

Models

in

Science

and

Engineering

Wiley

New

York

-LSB-

-RSB-

S.

Karlin

Total

Positivity

Volume

Stanford

University

Press

Stanford

California

-LSB-

-RSB-

W.H.

Kruskal

Ordinal

Measures

of

Association

J.

Amer

Stat

Assoc.

vol

pp.

-LCB-

-LSB-

-RSB-

S.

Kullback

Information

Theory

and

Statistics

Wiley

-LSB-

-RSB-

Oscar

E.

Lanford

III

Entropy

and

Equilibrium

States

in

Classical

Statistical

Mechanics

Statistical

Mechanics

Mathematical

Problems

Ed

A.

Lenard

Springer

Lecture

Notes

in

Physics

no.

pp

-LSB-

-RSB-

E.L.

Lehmann

Some

concepts

of

dependence

Ann

Math

Stat

vol

pp.

-LCB-

-LSB-

-RSB-

K.V.

Mardia

Families

of

Bivariate

Distributions

Griffin

's

Statistical

Monographs

Courses

No.

Ed

A.

Stuart

Griffin

London

-LSB-

-RSB-

A.W.

Marshall

I.

Olkin

Inequalities

Theory

of

Majorization

and

Its

Applications

vol

in

Mathematics

in

Science

and

Engineering

ed

Richard

Bellman

Academic

Press

New

York

London

-LSB-

-RSB-

J.K.

Ord

Families

of

Frequency

Distributions

Griffin

's

Statistical

Monographs

Courses

Ed

A.

Stuart

Griffin

London

-LSB-

-RSB-

H.S.J.

Tsao

S.C.

Fang

D.N.

Lee

On

the

Optimal

Entropy

Analysis

European

J.

of

Opl

Res

vol

pp.

-LSB-

-RSB-

Y.L.

Tong

The

Gaussian

Distribution

-LSB-

-RSB-

J.B.M.

Uffink

Measures

of

Uncertainty

and

the

Uncertainty

Principle

PhD

Thesis

Rijksuniversiteit

Utrecht

The

Netherlands

-LSB-

-RSB-

J.-S

Wu

W.C.

Chan

Maximum

Entropy

Analysis

of

Multiple-server

Queueing

Systems

J.

Opl

Res

Soc

Vol

monte carlo simulation
multivariate probability distribution