conditional distribution of multi-variate normal variable

conditional distribution of multi-variate normal variable

Theorem

.

Let

X

be a

random variable

, taking values in

R

n

, normally distributed
with a

non-singular

covariance matrix

Σ

and a mean of zero.

Suppose

Y

is defined by

Y

=

B

*

⁢

X

for some

linear transformation

B

:

R

k

→

R

n

of maximum rank.
(

∗

to denotes the transpose operator.)

Then the

distribution

of

X

conditioned on

Y

is multi-variate normal,
with conditional means and

covariances

of:

𝔼

[

X

∣

Y

]

=

Σ

B

(

B

∗

Σ

B

)

-

1

Y

,

Var

[

X

∣

Y

]

=

Σ

-

Σ

B

(

B

∗

Σ

B

)

-

1

(

Σ

B

)

∗

.

If

k

=

1

, so that

B

is simply a vector in

R

n

,
these

formulas

reduce to:

𝔼

[

X

∣

Y

]

=

Σ

⁢

B

⁢

Y

Var

⁡

[

Y

]

,

Var

[

X

∣

Y

]

=

Σ

-

Σ

⁢

B

⁢

B

∗

⁢

Σ

Var

⁡

[

Y

]

.

If

X

does not have zero mean, then the
formula for

E

[

X

∣

Y

]

is modified by adding

E

⁢

[

X

]

and replacing

Y

by

Y

-

E

⁢

[

Y

]

,
and the formula for

Var

⁡

[

X

∣

Y

]

is unchanged.

Proof.

We split up

X

into two stochastically independent parts,
the first part containing exactly the

information

embodied in

Y

.
Then the conditional distribution of

X

given

Y

is simply
the unconditional distribution of the second part that is

independent

of

Y

.

To this end, we first
change variables to express everything in terms of a

standard

multi-variate normal

Z

. Let

A

:

ℝ

n

→

ℝ

n

be a “square root” factorization of
the covariance matrix

Σ

,
so that:

A

⁢

A

∗

=

Σ

,

Z

=

A

-

1

⁢

X

,

X

=

A

⁢

Z

,

Y

=

B

∗

⁢

A

⁢

Z

.

We let

H

:

ℝ

n

→

ℝ

n

be the

orthogonal projection

onto the range of

A

∗

⁢

B

:

ℝ

k

→

ℝ

n

, and decompose

Z

into

orthogonal

components

:

Z

=

H

⁢

Z

+

(

I

-

H

)

⁢

Z

.

It is intuitively obvious that orthogonality
of the two random

normal vectors

implies their stochastic independence.
To show this formally, observe that the

Gaussian

density function

for

Z

factors into a

product

:

(

2

⁢

π

)

-

n

/

2

⁢

exp

⁡

(

-

1

2

⁢

∥

z

∥

2

)

=

(

2

⁢

π

)

-

n

/

2

⁢

exp

⁡

(

-

1

2

⁢

∥

H

⁢

z

∥

2

)

⁢

exp

⁡

(

-

1

2

⁢

∥

(

I

-

H

)

⁢

z

∥

2

)

.

We can construct an orthonormal system of

coordinates

on

ℝ

n

under which the components for

H

⁢

z

are
completely

disjoint

from those components of

(

I

-

H

)

⁢

z

.
On the other hand, the

densities

for

Z

,

H

⁢

Z

, and

(

I

-

H

)

⁢

Z

remain

invariant

even after changing coordinates,
because they are radially

symmetric

.
Hence the variables

H

⁢

Z

and

(

I

-

H

)

⁢

Z

are

separable

in their joint density
and they are independent.

H

⁢

Z

embodies the information in the

linear combination

Y

=

B

∗

⁢

A

⁢

Z

.
For we have the

identity

:

Y

=

(

B

∗

⁢

A

)

⁢

Z

=

(

B

∗

⁢

A

)

⁢

(

H

⁢

Z

+

(

I

-

H

)

⁢

Z

)

=

(

B

∗

⁢

A

)

⁢

H

⁢

Z

+

0

.

The last term is null because

(

I

-

H

)

⁢

Z

is orthogonal to the range of

A

∗

⁢

B

by definition. (Equivalently,

(

I

-

H

)

⁢

Z

lies in the kernel of

(

A

∗

⁢

B

)

∗

=

B

∗

⁢

A

.)
Thus

Y

can always be recovered by a linear transformation on

H

⁢

Z

.

Conversely

,

Y

completely determines

H

⁢

Z

,
from the analytical expression for

H

that we now give.
In general, the orthogonal projection onto the range of an

injective

transformation

T

is

T

⁢

(

T

∗

⁢

T

)

-

1

⁢

T

∗

. Applying this to

T

=

A

∗

⁢

B

, we have

H

=

A

∗

⁢

B

⁢

(

B

∗

⁢

A

⁢

A

∗

⁢

B

)

-

1

⁢

B

∗

⁢

A

=

A

∗

⁢

B

⁢

(

B

∗

⁢

Σ

⁢

B

)

-

1

⁢

B

∗

⁢

A

.

We see that

H

⁢

Z

=

A

∗

⁢

B

⁢

(

B

∗

⁢

Σ

⁢

B

)

-

1

⁢

Y

.

We have proved that conditioning on

Y

and

H

⁢

Z

are

equivalent

, and so:

𝔼

[

Z

∣

Y

]

=

𝔼

[

Z

∣

H

Z

]

=

𝔼

[

H

Z

+

(

I

-

H

)

Z

∣

H

Z

]

=

H

Z

+

0

,

and

Var

⁡

[

Z

∣

Y

]

=

Var

⁡

[

Z

∣

H

⁢

Z

]

=

Var

⁡

[

H

⁢

Z

+

(

I

-

H

)

⁢

Z

∣

H

⁢

Z

]

=

0

+

Var

⁡

[

(

I

-

H

)

⁢

Z

]

=

𝔼

⁢

[

(

I

-

H

)

⁢

Z

⁢

Z

∗

⁢

(

I

-

H

)

∗

]

=

(

I

-

H

)

⁢

(

I

-

H

)

∗

=

I

-

H

-

H

∗

+

H

⁢

H

∗

=

I

-

H

,

using the defining property

H

2

=

H

=

H

∗

of orthogonal projections.

Now we express the result in terms of

X

, and remove the dependence on the transformation

A

(which is not uniquely defined from the covariance matrix):

𝔼

[

X

∣

Y

]

=

A

𝔼

[

Z

∣

Y

]

=

A

H

Z

=

Σ

B

(

B

∗

Σ

B

)

-

1

Y

and

Var

⁡

[

X

∣

Y

]

=

A

⁢

Var

⁡

[

Z

∣

Y

]

⁢

A

∗

=

A

⁢

A

∗

-

A

⁢

H

⁢

A

∗

=

Σ

-

Σ

⁢

B

⁢

(

B

∗

⁢

Σ

⁢

B

)

-

1

⁢

B

∗

⁢

Σ

.

Of course, the conditional distribution of

X

given

Y

is the same
as that of

(

I

-

H

)

⁢

Z

, which is multi-variate normal.

The formula in the statement of this

theorem

, for the single-dimensional case,
follows from substituting in

Var

⁡

[

Y

]

=

Var

⁡

[

B

∗

⁢

X

]

=

B

∗

⁢

Σ

⁢

B

.
The formula for when

X

does not have zero
mean follows from applying the

base case

to the shifted
variable

X

-

𝔼

⁢

[

X

]

.
∎

Title

conditional distribution of multi-variate normal variable

Canonical name

ConditionalDistributionOfMultivariateNormalVariable

Date of creation

2013-03-22 18:39:09

Last modified on

2013-03-22 18:39:09

Owner

stevecheng (10074)

Last modified by

stevecheng (10074)

Numerical id

5

Author

stevecheng (10074)

Entry type

Theorem

Classification

msc 62E15

Classification

msc 60E05

Generated on Sat Feb 10 12:03:35 2018 by

LaTeXML