IT ] 24 J an 2 00 9 On the rate distortion function of Bernoulli Gaussian sequences

In this paper, we study the rate distortion function of the i. i.d sequence of multiplications of a Bernoulli p random variable and a gaussian random variable ∼ N(0, 1). We use a new technique in the derivation of the lower bound in which we establish the duality between channel coding and lo ssy source coding in the strong sense. We improve the lower bound on the rate distortion function over the best kno wn lower bound byp log2 1 p if distortion D is small. This has some interesting implications on sparse signals wh ere p is small since the known gap between the lower and upper bound is H(p). This improvement in the lower bound shows that the lower and upper bounds are almost identical for sparse signals with small distortion because lim p→0 p log2 1 p H(p) = 1. I. BERNOULLI-GAUSSIAN MODEL AND SOME OBVIOUS BOUNDS ON ITS RATE DISTORTIONFUNCTIONS Notations: in this paper we usex , y , u for random variables andx, y, u for the realization of the random variables or constants. We denote by x Pr(A) the probability of eventA under measurex . We use bit andlog2 in this paper. Consider a sequence of signals x1, x2, . xn, wherexi’s are zero most of the time. When xi is non-zero, it is an arbitrary real number. In the signal processing literatu re, the signalsx is called sparse if most of them are zero. In their seminal work on compressive sensing [3] and [6 ], Candès, Tao and Donoho show that, to exactly reconstruct the sparse signals x, only a fraction ofn measurements are needed. Furthermore, the reconstruction can be done by a linear programming based efficient algorithm. In the compressed sensing literature, the non-zero part of the sparse signals are arbitrary real numbers without any statistical distribution assigned to them. Furthermore the compressed sensing system tries to recover the signals x losslessly without distortion of the reconstructed signals. These assumptions are not completely valid if the s ource statistics are known to the coding system, more importantly, if the goal of the sensing system is only to reco ver the data within a certain distortion. In the recent work by Fletcher etc. [8], [7], [9], the . What is lacking in th e previous study of this problem is a systematic study of the information theoretic bounds on the rate distortion f u ctions of the sources. In this paper, we give both lower and upper bounds on the rate distortion functions. A. Bernoulli-Gaussian random variable Ξ(p, σ) The information theoretic model of the “sparse gaussian” si gnals is captured in the following what we call a Bernoulli-Gaussian random variable. Definition 1: A random variablex is Bernoulli-Gaussian, denoted by Ξ(p, σ), if x = b × s, where s is a Gaussian random variable with mean 0 and varianceσ, s ∼ N(0, σ), andb is a Bernoullip random variable, Pr(b = 0) = 1 − p andPr(b = 1) = p, p ∈ [0, 1]. This random variable is a mixture of a continuous random vari able and a discrete random variable. This adds to the difficulties to study the rate distortion functions of th is random variable. The main result of this paper is a lower bound and an upper bound on the rate distortion functions of a sequence of independent random variables with distributionΞ(p, σ). It will be clear soon in Proposition 1 that we only need to stu dy the rate distortion functions for s ∼ N(0, 1), i.e. the rate distortion functions for Ξ(p, 1). First, we review the definition of rate distortion functions in both the average distortion and strong distort ion sense. B. Review of the rate distortion theory In the standard setup of rate distortion theory, the encoder mapsn i.i.d. random variablesx ∈ Xn, x ∼ px , into nR bits and then the decoder reconstruct the original signal wi thin a certain distortion. The encoder and decoder are denoted byfn andgn respectively: fn : X → {0, 1} and gn : {0, 1} → X̂, Cheng Chang is with Hewlett-Packard Labs, Palo Alto. Email: cchang@eecs.berkeley. edu and the distortion is defined as d(x, x̂) = 1 n n ∑ i=1 d(xi, x̂i). Definition 2: Rate distortion function ([4], pg. 341): the rate distortio n functionR(D) is the infinimum of rates R, such that(R, D) is in the rate distortion region of the source for a give disto rtion D. Where the rate distortion region is the closure of achievable rate distortion pairs (R, D) defined as follows. (R, D) is said to be achievable in the expected distortion sense if there exists a sequence o f (2, n) rate codes(fn, gn), such that lim n→∞ E (d(x, gn(fn(x )))) ≤ D (1) The strong sense of rate distortion function is defined simil arly with the following criteria for the codes: for all δ > 0 lim n→∞ Pr (d(x, gn(fn(x )) ≥ D + δ) = 0 (2) where, in this paper, the distortion function d(x, x̂) = 1 n n ∑ i=1 (xi − x̂i). It turns out that the rate distortions function for both the a v r ge distortion and the strong distortion are the same for discrete random variables Chapter 13.6 [4]. We can gener aliz this result easily to continuous random variables whose variance is finite and the probability density functio n satisfies the usual regularity conditions. The proof can be carried out by quantizing the probability density fun ction and then by using the proof for discrete random variables in [4]. A somewhat detailed sketch of how this work s is in Appendix A. A good lossy coding system in the strong sense is not necessarilygood in the expected distortion sense. Considering the following example, a good lossy coder can mi ss the distortion constraint for a subset Υn ⊆ Rn with asymptotically0 measure, lim n→∞ x Pr(Υn) = 0. However the good lossy coder can intentionallymake the distortion on Υn no smaller than 2D x Pr(Υn) , hence the expected distortion is at least 2D. However it is easy to see that given a good lossy coding system in the strong sense, we can easily make it also good in the expected sense if the mean and variance of x are finite. We sketch the proof in Appendix B. So from now on, when we say a lossy coding system is good in the strong s e se, that implies that the system is also good in the expected distortion sense. The following lemma characterizes the rate distortion func tion R(D). Lemma 1:Rate distortion theorem [10]: R(D) = min px̂|x : P x,x̂ px(x)px̂|x(x̂|x)d(x,x̂)≤D I(x ; x̂). (3) Corollary 1: Rate distortion theorem for Gaussian random variables [2]: for random variablex ∼ N(0, σ), the rate distortion function is: R(D, N(0, σ)) = { 1 2 log2 σ D , 0 ≤ D ≤ σ, 0, D > σ. (4) It is also shown that with the same variance and squared disto rtion measure, Gaussian random variables requires the most bits to be described. Both lower and upper bounds are given in Exercise 8 on Pg. 370 [4]. The proof can be found in [2]. Corollary 2: Rate distortion bounds for continuous random variables und er square distortion measure (Exercise 8 on pg. 372 [4]): the rate distortion function R(D) can be bounded as: h(x) − D 2 log(2πe) ≤ R(D) ≤ max{ 2 log σ D , 0} (5) The lower bound in Corollary 2 is known as the Shannon lower bo und in the literature [4].