Optimal Sensor Design and Zero-Delay Source Coding for
Continuous-Time Vector Gauss-Markov Processes

arXiv:1709.03026v1 [math.OC] 10 Sep 2017

Takashi Tanaka1

Mikael Skoglund2

Abstract— We consider the situation in which a continuoustime vector Gauss–Markov process is observed through a vector
Gaussian channel (sensor) and estimated by the Kalman–Bucy
filter. Unlike in standard filtering problems where a sensor
model is given a priori, we are concerned with the optimal
sensor design by which (i) the mutual information between
the source random process and the reproduction (estimation)
process is minimized, and (ii) the minimum mean-square
estimation error meets a given distortion constraint. We show
that such a sensor design problem is tractable by semidefinite
programming. The connection to zero-delay source-coding is
also discussed.

I. I NTRODUCTION
In this paper, we consider a situation in which a
continuous-time vector Gauss–Markov process (the source
random process) is estimated by the Kalman–Bucy filter
based on the output of a memoryless vector Gaussian channel
(the sensor). We study this estimation mechanism from the
perspectives of (i) the mean-square error (MSE) between
the source and the estimate, and (ii) the mutual information
rate between the source and the estimate. From standard
rate–distortion theory, it is intuitively clear that an accurate
sensing mechanism should lead to a small MSE and a
large mutual information, while a noisy sensing mechanism
implies a large MSE and a small mutual information. In this
paper, we make this intuition explicit by deriving a trade-off
curve between these two metrics by constructing trade-off
achieving sensor gain matrices. In particular, we show that
trade-off achieving sensor gain matrices are easily computed
by semidefinite programming, and consequently the trade-off
curve admits a convenient semidefinite representation.
There is a simple and explicit relationship (often called the
I-MMSE relationship in the literature) between the mutual
information (I) and the minimum mean-square error (MMSE)
when a random variable is observed through a Gaussian
channel. Guo et al. [1] showed that the derivative of the
mutual information with respect to the channel SNR (signalto-noise ratio) is equal to half the MMSE. They also considered causal estimation of random processes through Gaussian channels and provided a remarkably simple connection
between causal and non-causal MMSE. For continuoustime source processes observed through Gaussian channels,
Duncan [2] already derived a relevant result, stating that
1 Department of Aerospace Engineering & Engineering Mechanics, University of Texas at Austin, Austin, TX, USA. ttanaka@utexas.edu.
2 School of Electrical Engineering, KTH Royal Institute of Technology,
Stockholm, Sweden. skoglund@kth.se. 3 School of Engineering and
IT, The University of New South Wales at ADFA, Canberra, Australia.
v.ougrinovski@adfa.edu.au.

Valeri Ugrinovskii3

“twice the mutual information is merely the integration
of the trace of the optimal mean square filtering error.”
Kadota et al. [3] considered estimation of continuous-time
source over Gaussian channel with feedback (the source is
causally affected by channel output). Weissman et al. [4]
further studied the cases with feedback, where a fundamental
relationship between directed information and MMSE is
derived.
In parallel with the I-MMSE formulas for Gaussian observations, there exists a line of research for random processes
observed through Poisson channels. Guo et al. [5] studied a
relationship between mutual information and the estimation
error, measured by the mean value of the logarithm of the
ratio of the channel input plus dark current and its mean
estimate. Remarkably, the same formula as the I-MMSE
relationship for Gaussian case is recovered for Poisson cases
as well, provided that MMSE is replaced by a suitable loss
function for Poisson channels [6]. Estimation of continuoustime processes through Poisson channels with feedback is
studied by [4], [7]. Recently, an overarching theory unifying
the I-MMSE relationship for Gaussian channels and the
similar relationship for Poisson channels is proposed [8].
Applications of the I-MMSE formula can be found in
channel coding problems. Palomar and Verdú [9] extended
the result by [1] to vector Gaussian channels, where an
explicit formula relating gradients of mutual information
with respect to channel parameters and estimate covariance
matrices is obtained. Based on this result, they proposed a
gradient ascent algorithm for channel precoder design where
input-output mutual information is maximized subject to
input power constraints.
In this paper, we apply Duncan’s I-MMSE formula for the
aforementioned trade-off study. Our study is motivated by the
zero-delay source coding problem. Derpich and Østergaard
[10] showed that minimum the data-rate achievable by zerodelay source coding of a Gaussian source subject to a
quadratic distortion constraint is closely approximated by
the zero-delay rate-distortion function (also called sequentialor non-anticipative rate-distortion function in the literature).
For Gauss–Markov sources with mean-square distortion criteria, computation of zero-delay rate-distortion functions
and construction of optimal test channels are addressed by
recent literature [11], [12]. Stavrou et al. [11] showed that
the optimal test channel can be realized by a memoryless
Gaussian channel (sensor) with feedback and a Kalman filter.
Tanaka et al. [12] presented a different realization of the
test channel, using a memoryless Gaussian channel without
feedback and a Kalman filter. The latter observation implies

that the zero-delay rate-distortion function can be computed
by considering the I-MMSE trade-off with respect to the
Gaussian channel gain (sensor gain matrix) [12]. The IMMSE trade-off in the present paper can be viewed as a
continuous-time counterpart of a similar trade-off considered
in discrete-time [12]. From analogous discrete-time results,
it is conjectured that results in this paper provide fundamental performance limitations of zero-delay source coding
schemes for continuous-time sources, although zero-delay
source coding problems for continuous-time sources are not
fully explored in the literature.
This paper is organized as follows. Problem formulation
is presented in Section II. Section III summarizes the main
result. A connection to zero-delay source coding problem is
discussed in Section IV. Section V summarizes the paper
and discuss future work.
Notation: Let (Ω, F, P) be a probability space and let X
be a random variable in a measurable space (X , A). The
probability distribution µX of X is defined by
µX (A) = P{ω : X(ω) ∈ A} ∀A ∈ A.

A. Minimum mean-square error (MMSE) estimate
Let FtY ⊂ F be the σ-algebra generated by Ys , 0 ≤ s ≤ t.
Denote by X̂t , E(Xt |FtY ) the causal MMSE estimate of
the process (1) via the observation (2), calculated by the
Kalman–Bucy filter
dX̂t = AX̂t dt + Pt C > (dYt − C X̂t dt), t ∈ [0, ∞)

with X̂0 = 0. In (3), Pt is the unique solution to the matrix
Riccati differential equation
dPt
= APt + Pt A> − Pt C > CPt + BB > , t ∈ [0, ∞) (4)
dt
with P0 = 0.
For notational simplicity, we denote by X0T , Y0T , X̂0T the
random processes Xt , Yt , X̂t over the horizon 0 ≤ t ≤ T as
defined above. The MMSE performance over the considered
horizon is denoted by
ρ(X0T , X̂0T ) ,

Z

T

EkXt − X̂t k2 dt =

Z

0

If X and Y are random variables in the same measurable
space with distributions µX and µY , the relative entropy from
Y to X is defined by
Z
dµX
dµX
D(µX kµY ) = log
dµY
X
if the Radon-Nikodym derivative dµ
dµY exists. If random
variables X and Y have a joint probability distribution µXY ,
the mutual information between X and Y is defined by

I(X; Y ) = D(µXY kµX ⊗ µY )
where µX ⊗ µY is the product measure defined by the
marginal distributions. If µX is discrete, the entropy of X is
defined by
X
H(X) = −
µX (x) log µX (x).
x∈X

II. P ROBLEM F ORMULATION

dXt = AXt dt + BdWt , t ∈ [0, ∞)

(1)

with X0 = 0. The source process is observed through an
n-dimensional Gaussian channel (or sensor):
dYt = CXt dt + dVt , t ∈ [0, ∞)

T

Tr(Pt ) dt.
0

B. Mutual information
We are also interested in the mutual information
I(X0T ; X̂0T ) between X0T and X̂0T .
Theorem 1: Let the random processes X0T and X̂0T be
defined as above. Then
Z
1 T
T
T
I(X0 ; X̂0 ) =
EkC(Xt − X̂t )k2 dt.
2 0
Proof: Set Zt = CXt . The following identity is wellknown (e.g., Duncan [2]):
I(Y0T ; Z0T ) =

1
2

Z

T

EkC(Xt − X̂t )k2 dt.

For completeness, a proof of (5) is given in Appendix A. We
also have an identity
(6)

whose proof is provided in Appendix B.
Due to the property of the Kalman–Bucy filter, X̂0T is a
sufficient statistic of X0T for Y0T . Thus
I(X0T ; Y0T ) = I(X0T ; X̂0T ).

(7)

The claim follows from (5)–(7).
It is immediate from Theorem 1 that the mutual information
of interest can be written in terms of the solution Pt to the
Riccati equation (4) as

(2)

with Y0 = 0. We assume that (A, B) is a controllable pair.

(5)

0

I(Y0T ; Z0T ) = I(Y0T ; X0T ),

Let (Ω, F, P) be a complete probability space and Ft ⊂
F be a non-decreasing family of σ-algebras. Let (Wt , Ft )
and (Vt , Ft ) be n-dimensional independent standard Wiener
processes with respect to P. Assume that the source random
process is an n-dimensional Gauss–Markov process of the
form

(3)

I(X0T ; X̂0T ) =

1
2

Z
0

T

Tr(CPt C > )dt.

C. I-MMSE trade-off via observation channel design
In this paper, we construct the optimal observation gain
C ∈ Rn×n in the observation channel (2) that minimizes
the average mutual information while the average MMSE
is smaller than a given constant D. Formally, we seek an
optimal solution to the problem
R(D) , inf

C∈C

s.t.

1
I(X0T ; X̂0T )
T
1
lim sup ρ(X0T , X̂0T ) ≤ D.
T →+∞ T
lim sup

(8a)

T →+∞

(8b)

In (8), the underlying linear system model (1) is given. The
domain of optimization C ⊂ Rn×n is the set of matrices C
such that (A, C) is a detectable pair, i.e., A + LC is Hurwitz
stable for some matrix L. Below, we show that there exists
an optimal solution and thus “inf” can be replaced by “min.”
III. M AIN R ESULT
We first assume that a precoder matrix C ∈ C is given.
Since we assume (A, B) is controllable and (A, C) is detectable, the algebraic Riccati equation
AP + P A> − P C > CP + BB > = 0

(9)

admits a unique positive definite solution P [13, Theorem
13.7, Corollary 13.8]. Under the same assumption, the solution Pt to the Riccati differential equation (4) with P0 = 0
satisfies Pt → P as t → +∞ (e.g., [14, Theorem 10.10]),
where P is the unique positive definite solution to (9). Thus,
it follows from the convergence of Cesàro mean that
Z T
1
1
1
I(Y0T ; Z0T ) =
Tr(CPt C > )dt → Tr(CP C > )
T
2T 0
2
Z T
1
1
ρ(X0T , X̂0T ) =
Tr(Pt )dt → Tr(P ) as T → +∞.
T
T 0
Hence, the right hand side of (8) can be written as
inf

C∈C,P 0

s.t.

1
Tr(CP C > )
2
AP + P A> − P C > CP + BB > = 0

(10b)

Tr(P ) ≤ D.

(10c)

(10a)

Now we show that the optimization problem (10) is
reformulated as a semidefinite programming problem. First,
under the equality constraint (10b), the objective function
can be written as
1
1
Tr(CP C > ) = Tr(P C > CP P −1 )
(11a)
2
2
1
= Tr(A) + Tr(B > P −1 B)
(11b)
2
1
= min Tr(A) + Tr(Q)
(11c)
Q
2
s.t. B > P −1 B  Q
1
= min Tr(A) + Tr(Q)
(11d)
Q
2


Q B>
s.t.
 0.
B P

The equality constraint (10b) is used to obtain (11b) from
(11a). Equality (11c) holds since the unique solution to the
minimization problem in (11c) is Q = B > P −1 B. We have
applied the Schur complement formula in (11d).
The next lemma allows us to replace the nonlinear equality
constraint (10b) with a linear inequality constraint.
Lemma 1: If (A, B) is controllable, then the following
conditions are equivalent.
(i) ∃C ∈ C, P  0 s.t. AP + P A> − P C > CP + BB > = 0.
(ii) ∃P  0 s.t. AP + P A> + BB >  0.
Proof: The direction (i)⇒(ii) is trivial. To show
(ii)⇒(i), notice that if condition (ii) holds, then clearly there
exists a matrix C such that
AP + P A> − P C > CP + BB > = 0.

(12)

To complete the proof, we show that for every C satisfying
(12), (A, C) is a detectable pair. It is sufficient to show that
A − P C > C is stable. To this end, rewrite (12) as
(A−P C >C)P +P (A−P C >C)>+P C >CP +BB >= 0 (13)
and suppose that (A − P C >C)> is not stable. Let λ be an
unstable eigenvalue and x be the corresponding eigenvector:
(A> − C > CP )x = λx.

(14)

∗

Pre- and post-multiplying (13) by x and x, we have
(λ + λ̄)x∗ P x + x∗ (P C > CP + BB > )x = 0.
Since Re(λ) ≥ 0 and P  0, this implies CP x = 0
and B > x = 0. Thus, from (14), we obtain A> x = λx
and B > x = 0. This contradicts the Popov-Belevitch-Hautus
(PBH) test for controllability of (A, B).
Applying (11) and Lemma 1 to (10), we obtain the following
result,
1
R(D) = inf Tr(A) + Tr(Q)
(15a)
P 0,Q
2
s.t. AP + P A> + BB >  0
(15b)


Q B>
0
(15c)
B P
Tr(P ) ≤ D.

(15d)

The main result of this paper is given by the next theorem.
Theorem 2: Suppose (A, B) is controllable. The optimal
value R(D) of (8) admits a semidefinite representation
1
R(D) = min Tr(A) + Tr(Q)
P 0,Q
2
s.t. AP + P A> + BB >  0


Q B>
0
B P
Tr(P ) ≤ D.
In particular, there exists an optimal solution P  0, Q  0
attaining the optimal value. Moreover, any matrix C ∈ C
satisfying
AP + P A> − P C > CP + BB > = 0,

which always exists, is an optimal solution to (8).
Proof: Since we have (15), it is left to show that
the optimal value is attained. By continuity, inf P 0,Q in
(15) can be replaced with inf P 0,Q without changing the
optimal value. After this replacement, the existence of an
optimal solution is guaranteed by Weierstrass’ theorem [15,
Proposition A.8], since the feasible domain for (P, Q) is
closed and the objective function is coercive. Thus inf P 0,Q
can be written as minP 0,Q :
1
R(D) = min Tr(A) + Tr(Q)
P 0,Q
2
s.t. AP + P A> + BB >  0


Q B>
0
B P
Tr(P ) ≤ D.

(16a)

𝑡=0

𝑡=𝑇

Source
𝑋

Causal message generation
𝑘=𝐾

𝑘=0
Message
𝑀
𝑘 = 0,1,2, …

(16b)

Causal estimation

(16c)

𝑡=0

𝑡=𝑇

(16d)

Now, we show that if (P, Q) is an optimal solution to (16),
then P is nonsingular. To show this by contradiction, assume
P  0 is singular. Without loss of generality, assume


P1 0
P =
with P1  0.
0 0
Also, consider a corresponding partitioning of A and B:




A11 A12
B1
A=
,B =
.
A21 A22
B2
First, if (P, Q) is a feasible solution to (16), then it must be
that Im(B) ⊆ Im(P ). To see this by contradiction, suppose
there exists a matrix N such that N B 6= 0 and N P =
0. Then, pre- and post-multiplying (16c) by diag(I, N ) and
diag(I, N > ), we obtain


Q B>N >
 0.
NB
0

Reproduction
𝑋

Fig. 1.

Zero delay source coding of continuous-time signal.

1, 2, ..., K is generated. We assume m0 = 0. For each k =
1, 2, ..., K, we assume that ek is a Zn -valued map whose
domain is the space of sample paths Xt , 0 ≤ t ≤ kτ . A
simple example is n parallel scalar quantizers with uniform
quantizer step sizes ∆ = (∆1 , ..., ∆n ):
(mk )i = b∆i (Xkτ )i c, ∀i = 1, 2, ..., n.
Introduce a continuous-time process Mt = mb τt c as the zeroorder hold of mk , and its time integral
Z t
Yt = Y0 +
Ms ds.
(17)
0

However, this is a contradiction because a matrix of this
structure with non-zero off-diagonal entries must be indefinite. Thus, we conclude that Im(B) ⊆ Im(P ) and B2 = 0.
Next, it must be that A21 6= 0 since otherwise
dim(Im[ B AB A2 B ... An−1 B ]) < n
which contradicts the controllability of (A, B).
Finally, with the above observations, (16a) becomes


>
A11 P1 + P1 A>
P1 A >
11 + B1 B1
21
0
A21 P1
0
which is again a contradiction since off-diagonal matrices
are non-zero. Thus, we conclude that if (P, Q) is an optimal
solution to (16), then P is nonsingular.
IV. A PPLICATION
In this section, we consider an application of the optimization problem (8) to a zero-delay source coding scenario
depicted in Fig. 1.
Let Xt be a continuous-time source random process (e.g.,
video). The source process is encoded with sampling period
T
τ = K
, and a sequence of codewords mk = ek (X0kτ ), k =

At t = kτ , k = 1, 2, ..., K, the codeword mk is transmitted
to the destination. At the destination, the decoder estimates
Xt in continuous-time based on the received information:
X̂t = E(Xt |M0t ) = E(Xt |Y0t ).
RT
A function ρ(X, X̂) = 0 EkXt − X̂t k2 dt is introduced as
a distortion measure.
The above zero-delay source coding scheme is denoted
by ZDSC(τ, e). (Notice that we are free to choose sampling
period τ and encoding functions e1 , ..., eK .) Assuming that
the source process is given by (1), we arePinterested in
K
the fundamental trade-off between the rate
k=1 H(mk )
T
T
and the distortion ρ(X0 , X̂0 ) achievable by ZDSC(τ, e).
Here, we are interested in the entropy H(mk ) because it
is related to the minimum expected codeword length if mk
is represented by variable-length binary strings.
To analyze the fundamental performance limitation of
ZDSC(τ, e), we also consider a class of general causal
reproduction processes, denoted by GCR, described below.
Let (Ω, F, P) be a complete probability space and let Ft ⊂
F be a non-decreasing family of σ-algebras. Let (Wt , Ft )
and (Vt , Ft ) be mutually independent n-dimensional Wiener

processes. Let the source process Xt be defined by (1).
Consider a random process Yt that can be represented by
a stochastic integral
Z t
Z t
Yt = Y0 +
Ms (X)ds +
Ns (X)dVs
(18)
0

0

where for each 0 ≤ t ≤ T , functions Mt and Nt are FtX measurable. The source process is reproduced by
X̂t = E(Xt |Y0t ).
Notice that ZDSC(τ, e) is a special case of GCR where (18)
has a special form (17).
Notice that the following chain of inequalities holds.
T
X

min
ZDSC(τ,e):ρ(X0T ,X̂0T )≤D

≥

min
ZDSC(τ,e):ρ(X0T ,X̂0T )≤D

=

min
ZDSC(τ,e):ρ(X0T ,X̂0T )≤D

=

min
ZDSC(τ,e):ρ(X0T ,X̂0T )≤D

≥

min
ZDSC(τ,e):ρ(X0T ,X̂0T )≤D

≥

min
GCR:ρ(X0T ,X̂0T )≤D

H(mk )

(19a)

H(m1 , ..., mK )

(19b)

H(Y0T )

(19c)

I(X0T ; Y0T )

(19d)

I(X0T ; X̂0T )

(19e)

k=1

I(X0T ; X̂0T )

(19f)

Equality (19d) holds because H(Y0T ) = I(X0T ; Y0T ) +
H(Y0T |X0T ), and the second term is zero since under
ZDSC(τ, e) the map from X0T to Y0T is deterministic. (19e)
is the data-processing inequality. The last inequality (19f)
holds since ZDSC(τ, e) is a special case of GCR.
Therefore, the smallest data rate that the zero delay source
code can attain in average over the infinite horizon:
RZDSC (D) ,

min

lim sup

ZDSC(τ,e) K→+∞

s.t.

K
1 X
H(mk )
Kτ

an upper bound of a lower bound R∗ (D) of the smallest
achievable data rate RZDSC (D). Nevertheless, guided by the
analogy with the corresponding discrete-time results in [12],
[16], we conjecture that the inequality in (22) is actually the
exact equality:
Conjecture 1: R∗ (D) = R(D), ∀D > 0.
To establish Conjecture 1, one essentially needs to prove that
the optimal observation process (18) is linear in X, and has
the form (2).
V. C ONCLUSION
We considered a continuous-time vector Gauss–Markov
process being estimated by the Kalman–Bucy filter based on
the observation through a vector Gaussian channel (sensor).
The trade-off between the mutual information rate between
the source process and the estimation process and the
MMSE, as well as trade-off achieving sensor gain matrices, are studied by means of semidefinite programming. A
connection to the zero-delay rate-distortion problem is also
discussed. In this paper, we restricted ourselves to observation through Gaussian channels. However, in the future, it is
worth pursuing further whether or not the I-MMSE trade-off
can be improved by considering non-Gaussian and nonlinear
sensor mechanisms (Conjecture 1). Zero-delay source coding
schemes that (approximately) attain the obtained trade-off
function should also be considered in the future.
APPENDIX
A. Proof of equation (5).
Let C n = (C([0, T ], Rn ), B(C([0, T ], Rn ))) be the measurable space of continuous functions x = (xt , t ∈ [0, T ]),
x : [0, T ] → Rn with x0 = 0, equipped with the Borel
σ-algebra BC n = B(C([0, T ], Rn )). Consider two stochastic
processes Y = (Yt , t ∈ [0, T ]) and Z = (Zt , t ∈ [0, T ]) in a
probability space (Ω, F, P ) related by

k=1

1
lim sup
ρ(X0Kτ , X̂0Kτ ) ≤ D
K→+∞ Kτ

is lower bounded by
1
I(X0T ; X̂0T )
GCR T →+∞ T
1
s.t. lim sup ρ(X0T , X̂0T ) ≤ D.
T
T →+∞

R∗ (D) , min lim sup

(20a)
(20b)
∗

Thus, we are interested in computing the function R (D)
since it provides a fundamental performance limitation for
zero-delay source coding schemes.
Now, notice that the linear observation process (2) is
a special case of (18). Consequently, the functions R(D)
defined by (8) and R∗ (D) defined by (20) must satisfy
R∗ (D) ≤ R(D), ∀D > 0.

(21)

Since R(D) is semidefinite representable (Theorem 2), computing R(D) is straightforward. Unfortunately, the inequality
(21) is not of great use because it only shows that R(D) is

dYt = Zt dt + dVt ,

Y0 = 0

(22)

where V = (Vt , t ∈ [0, T ]) is the n-dimensional standard
Brownian motion independent of Z. Assume that Z satisfies
Z T
E
kZt k2 dt < ∞
(23)
0

and Z0 = 0. Let µY , µV and µZ be probability measures on
C n defined by
µY (BY ) = P {ω : Y (ω) ∈ BY }, BY ∈ BC n
µV (BV ) = P {ω : V (ω) ∈ BV }, BV ∈ BC n
µZ (BZ ) = P {ω : Z(ω) ∈ BZ }, BZ ∈ BC n .
In particular, µV is the Wiener measure. When µY  µV ,
denote by
dµY
: C[0, T ] → [0, ∞)
dµV
the Radon-Nikodym derivative.

Let µY Z and µV Z be joint measures on C n × C n defined
by the extensions of

dP
dP̃

= κ−1
T . Now,
Z

µY Z (BY × BZ ) =

dP (ω)
Z{ω:Y (ω)∈BY ,Z(ω)∈BZ }
=
κ−1
T dP̃ (ω) (24)

µY Z (BY × BZ ) = P {ω : Y (ω) ∈ BY , Z(ω) ∈ BZ },
µV Z (BV × BZ ) = P {ω : V (ω) ∈ BV , Z(ω) ∈ BZ },
for BY , BV , BZ ∈ BC n . Since V and Z are independent,
µV Z = µV ⊗ µZ where µV ⊗ µZ is the product measure.
Whenever µY Z  µV Z , denote by
µY Z
: C[0, T ] × C[0, T ] → [0, ∞)
µV Z
the Radon-Nikodym derivative.
YZ
First, we derive an explicit formula for dµ
µV Z .
Theorem 3 (Girsanov Theorem): [17, Theorem 6.3]:
Let κ = (κt , t ∈ [0, T ]) be a supermartingale of the form

 Z t
Z
1 t
kZs k2 ds
κt = exp −
Zs> dVs −
2 0
0
RT
where P ( 0 kZt k2 dt < ∞) = 1. If EκT = 1, then the
process Y defined by (22) is a Wiener process with respect
P̃
to a probability measure P̃ such that ddP
= κT .
Proof: See [17, Theorem 6.3].
RT
Note that condition (23) implies P ( 0 kZt k2 dt < ∞) =
1. To see this, consider
!
!
Z T
Z T
2
2
P
kZt k dt < ∞ ≥ sup P
kZt k dt ≤ R
0

R>0

≥ sup

0

E

1−

RT
0

R>0

kZt k2 dt
R2

!

=1
where the Chebyshev inequality is used in the second inequality. Moreover, since Z and V are independent, it follows
that EκT = 1 [17, Section 6.2, Example 4]. Thus, premises
of Theorem
3 are satisfied. Condition (23) also implies
RT
P (| 0 Zt> dVt | < ∞) = 1. This can be verified as


!
2
Z T
Z T
P
Zt> dVt < ∞ = P 
Zt> dVt < ∞
0

0


Z
≥ sup P 
R>0

≥ sup
R>0

= sup
R>0

2

T

Zt> dVt

On the other hand, since Y is a Wiener process under P̃ ,
the joint probability distribution of Y and Z under P̃ is the
same as the joint probability distribution of V and Z under
P . Therefore,
dµY Z
(Y (ω), Z(ω)) = κ−1
T (ω)
dµV Z
!
Z
Z T
1 T
2
>
kZt k dt
= exp
Zt dVt +
2 0
0
!
Z
Z T
1 T
>
2
Zt dYt −
= exp
kZt k dt .
2 0
0

≤ R

RT

!
Zt> dVt |2
1−
R2
!
RT
E 0 kZt k2 dt
1−
R2
0

=1
where the Itô isometry [18, Corollary 3.1.7] is used in the
fourth line. Hence P (κT = 0) = 0. Thus, by Theorem 3,
together with [17, Lemma 6.8], we also have P  P̃ and

(25)

dµY
Next, we derive an explicit formula for dµ
.
V
Theorem 4: [17, Theorem 7.13]: Let κ = (κt , t ∈ [0, T ])
be a supermartingale of the form
 Z t

Z
1 t
κt = exp −
Zs> dVs −
kZs k2 ds
2 0
0
RT
RT
where 0 EkZt kdt < ∞ and P ( 0 kZt k2 dt < ∞) = 1. If
EκT = 1, then µY  µV , µV  µY , and
!
Z
Z T
1 T
dµY
2
(Y (ω)) = exp
kẐt k dt
Ẑt dYt −
dµV
2 0
0

where Ẑt = E(Zt |FtY ), 0 ≤ t ≤ T .
Proof: See [17, Theorem 7.13].
Note that condition (23), together with Jensen’s inequality
!2
Z
Z
1 T
1 T
EkZt kdt
≤
EkZt k2 dt
T 0
T 0
implies
and

RT
0

EkZt kdt < ∞. Thus, Theorem 4 is applicable,



0

E|

{ω:Y (ω)∈BY ,Z(ω)∈BZ }

dµV
(Y (ω)) = exp −
dµY

Z
0

T

Ẑt> dYt

1
+
2

Z

!

T
2

kẐt k dt .
0

(26)
Finally, we derive an explicit formula for the mutual
information
Z
dµY Z
I(Y ; Z) ,
log
dµY Z .
d(µY ⊗ µZ )
C n ×C n
Using the definition of Radon-Nikodym derivative, one can
verify the chain rule
dµV
dµY Z
dµY Z
=
.
d(µV ⊗ µZ ) dµY
d(µY ⊗ µZ )

Thus
dµY Z
d(µY ⊗ µZ )
dµY Z
dµV
= log
+ log
dµV Z
dµY
Z
Z T
1 T
(kZt k2 − kẐt k2 )dt
=
(Zt − Ẑt )> dYt −
2 0
0
Z T
Z
1 T
=
(Zt − Ẑt )> dVt +
kZt − Ẑt k2 dt.
2 0
0

Proof: Notice that
Z
Z
f (z)µZ (dz) =
f (Cx)µX (dx)
Cn
c−1 (C n )
Z
=
f (Cx)µX (dx)

log

Cn

(27)
(28)

Equations (25) and (26) are used in (27). Taking the expectation, the first term in (28) vanishes [18, Theorem 3.2.1].
Thus,
Z
dµY Z
1 T
EkZt − Ẑt k2 dt.
I(Y ; Z) = E log
=
d(µY ⊗ µZ )
2 0

The first equality is a consequence of Claim 1. The second
equality holds since c−1 (C n ) = C n . To see this, notice
by definition c−1 (C n ) , {x ∈ C n : Cx ∈ C n } ⊆ C n .
Conversely, C n ⊆ c−1 (C n ) since Cx is continuous for any
continuous function x.
Now we prove equation (6).
Lemma 2: I(Y0T ; Z0T ) = I(Y0T ; X0T ).
Proof: In addition to µX , µY , µZ , consider the measures µY X and µY Z on the product space C n × C n defined
by the extensions of

B. Proof of equation (6)

µY X (BY × BX ) = P {ω : Y0T (ω) ∈ BY , X0T (ω) ∈ BX }

Let (C n , BC n ) be the measurable space of continuous
functions as defined in Appendix A. Consider stochastic
processes X = (Xt , t ∈ [0, T ]) and Y = (Yt , t ∈ [0, T ])
defined in (1), (2) and Zt = CXt . Since X is an Ito process,
its trajectory is a.s. continuous, and so are trajectories of Y
and Z. This allows us to define measures µX , µY , µZ on
C n by

µY Z (BY × BZ ) = P {ω : Y0T (ω) ∈ BY , Z0T (ω) ∈ BZ }.

µX (BX ) = P {ω : X0T (ω) ∈ BX }
µY (BY ) = P {ω : Y0T (ω) ∈ BY }
µZ (BZ ) = P {ω : Z0T (ω) ∈ BZ } = P {ω : CX0T (ω) ∈ BZ }
where BX , BY , BZ ∈ BC n . Consider a mapping c : C n →
C n defined by z = Cx. For each BZ ∈ BC n , define
c−1 (BZ ) ∈ BC n by c−1 (BZ ) , {x ∈ C n : Cx ∈ BZ }.
Claim 1: µZ (BZ ) = µX (c−1 BZ ) ∀BZ ∈ BC n .
Proof: Let ω be such that X0T (ω) ∈ c−1 (BZ ), then
we have that CX0T (ω) ∈ BZ . Therefore, {ω : X0T (ω) ∈
c−1 (BZ )} ⊆ {ω : CX0T (ω) ∈ BZ } and µX (c−1 (BZ )) ≤
µZ (BZ ). On the other hand, since X0T is an Ito process,
it is a.s. continuous [17], therefore P {ω : X0T (ω) 6∈
C([0, T ], Rn )} = 0. This allows us to conclude that
µZ (BZ )
= P {ω : CX0T (ω) ∈ BZ }
= P {ω : CX0T (ω) ∈ BZ , X0T (ω) ∈ C([0, T ], Rn )}
+ P {ω : CX0T (ω) ∈ BZ , X0T (ω) 6∈ C([0, T ], Rn )}
≤ P {ω : CX0T (ω) ∈ BZ , X0T (ω) ∈ C([0, T ], Rn )}
+ P {ω : X0T (ω) 6∈ C([0, T ], Rn )}
= P {ω : CX0T (ω) ∈ BZ , X0T (ω) ∈ C([0, T ], Rn )}
= µX (c−1 (BZ ))
Thus the claim holds.
Claim 2: For any measurable function f ,
Z
Z
f (z)µZ (dz) =
f (Cx)µX (dx).
Cn

Cn

where BX , BY , BZ ∈ BC n . Since C n is a Borel space [19,
Definition 7.7], by [20, Theorem 5.1.9 and Exercise 5.1.16]
there exists a Borel-measurable stochastic kernel µY |X on
C n given C n , such that µY |X (BY |X0T (ω)) is a version of
T
T
P ({ω : Y0T (ω) ∈ BY }|B X0 ); here B X0 denotes the σT
algebra of events generated by X0 . That is, the regular
T
conditional probability distribution given B X0 exists and
µY X (BY × BX ) = P {ω : Y0T (ω) ∈ BY , X0T (ω) ∈ BX }
Z
T
=
P (Y0T ∈ BY |B X0 )P (dω)
T
{ω:X0 (ω)∈BX }
!
Z
Z
(·)

=

Cxs ds + V ∈ BY

P
BX

P (ω : X0T ∈ dx)

0

Z
µY |X (BY |x)µX (dx).

=

(29)

BX

The identity in the second line follows from the existence of
T
the regular conditional probability distribution given B X0 ,
and the identity in the third line is due Rto the change of
(·)
variables. Here we have used the notation 0 Cxs ds + V to
stress that
R t we consider the entire path of the random process
Fx,t = 0 Cxs ds+Vt parameterized by x ∈ C n . The last line
in (29) holds due to the uniqueness of the Radon-Nikodym
derivative. This leads us to conclude, that for almost all x ∈
Cn,

T
µY |X (BY |x) = P ω : Fx,0
∈ BY
∀BY ∈ BC n .
In a similar fashion, it follows that there exists a Borelmeasurable stochastic kernel µY |Z on C n given C n , such
that µY |Z (BY |Z0T (ω)) is a version of P ({ω : Y0T (ω) ∈
T
BY }|B Z0 ) and for almost all z ∈ C n ,

µY |Z (BY |z) = P ω : GTz,0 ∈ BY
∀BY ∈ BC n ,
Rt
where Gz,t = 0 zs ds + Vt . It is also clear from these
expressions that
µY |X (BY |x) = µY |Z (BY |Cx).

(30)

We are now in a position to complete the proof. By
definition of the mutual information,
I(Y0T ; X0T ) = D(µY X kµY ⊗ µX ),
I(Y0T ; Z0T ) = D(µY Z kµY ⊗ µZ ).
Thus, the result follows from the chain of equalities:
D(µY Z kµY ⊗ µZ )
Z
=
D(µY |Z (·|z)kµY (·))µZ (dz)
ZZ
=
D(µY |Z (·|Cx)kµY (·))µX (dx)
ZX
=
D(µY |X (·|x)kµY (·))µX (dx)

(31a)
(31b)
(31c)

X

= D(µY X kµY ⊗ µX ).

(31d)

Equalities (31a) and (31d) follow from the chain rule of relative entropy [21, Lemma 1.4.3(f)]. Equation (31b) follows
from Claim 2, and (31c) follows from (30).
R EFERENCES
[1] D. Guo, S. Shamai, and S. Verdú, “Mutual information and minimum
mean-square error in Gaussian channels,” IEEE Transactions on
Information Theory, vol. 51, no. 4, pp. 1261–1282, 2005.
[2] T. E. Duncan, “On the calculation of mutual information,” SIAM
Journal on Applied Mathematics, vol. 19, no. 1, pp. 215–220, 1970.
[3] T. Kadota, M. Zakai, and J. Ziv, “Mutual information of the white
Gaussian channel with and without feedback,” IEEE Transactions on
Information theory, vol. 17, no. 4, pp. 368–371, 1971.
[4] T. Weissman, Y.-H. Kim, and H. H. Permuter, “Directed information,
causal estimation, and communication in continuous time,” IEEE
Transactions on Information Theory, vol. 59, no. 3, pp. 1271–1287,
2013.
[5] D. Guo, S. Shamai, and S. Verdú, “Mutual information and conditional mean estimation in Poisson channels,” IEEE Transactions on
Information Theory, vol. 54, no. 5, pp. 1837–1849, 2008.
[6] R. Atar and T. Weissman, “Mutual information, relative entropy, and
estimation in the Poisson channel,” IEEE Transactions on Information
theory, vol. 58, no. 3, pp. 1302–1318, 2012.
[7] Y. M. Kabanov, “The capacity of a channel of the Poisson type,”
Theory of Probability & Its Applications, vol. 23, no. 1, pp. 143–147,
1978.
[8] J. Jiao, K. Venkat, and T. Weissman, “Mutual information, relative
entropy and estimation error in semi-martingale channels,” Information
Theory (ISIT), 2016 IEEE International Symposium on, pp. 2794–
2798, 2016.
[9] D. P. Palomar and S. Verdú, “Gradient of mutual information in linear
vector Gaussian channels,” IEEE Transactions on Information Theory,
vol. 52, no. 1, pp. 141–154, 2006.
[10] M. S. Derpich and J. Østergaard, “Improved upper bounds to the causal
quadratic rate–distortion function for Gaussian stationary sources,”
IEEE Transactions on Information Theory, vol. 58, no. 5, pp. 3131–
3152, 2012.
[11] P. A. Stavrou, T. Charalambous, and C. D. Charalambous, “Filtering
with fidelity for time-varying Gauss–Markov processes,” The 55th
IEEE Conference on Decision and Control (CDC), pp. 5465–5470,
2016.
[12] T. Tanaka, K.-K. Kim, P. A. Parrilo, and S. K. Mitter, “Semidefinite
programming approach to Gaussian sequential rate–distortion tradeoffs,” IEEE Transactions on Automatic Control (To appear), 2014.
[13] K. Zhou, J. C. Doyle, and K. Glover, Robust and optimal control.
Prentice hall New Jersey, 1996.
[14] R. R. Bitmead and M. Gevers, “Riccati difference and differential
equations: Convergence, monotonicity and stability,” in The Riccati
Equation. Springer, 1991, pp. 263–291.
[15] D. Bertsekas, Nonlinear Programming. Athena Scientific, 1995.

[16] T. Tanaka, “Semidefinite representation of sequential rate–distortion
function for stationary Gauss–Markov processes,” The 2015 IEEE
Multi-Conference on Systems and Control (MSC), 2015.
[17] R. Liptser and A. Shiryaev, Statistics of Random Processes I: General
Theory. Springer, 2012.
[18] B. Øksendal, Stochastic differential equations. Springer, 2003.
[19] D. P. Bertsekas and S. Shreve, Stochastic optimal control: the discretetime case, 2004.
[20] R. Durrett, Probability: Theory and Examples, 4th ed. Cambridge
University press, 2010.
[21] P. Dupuis and R. S. Ellis, A weak convergence approach to the theory
of large deviations. John Wiley & Sons, 2011, vol. 902.

