# Sample Dependence in the Maximum Entropy Solution to the Generalized Moment Problem

Authors: Henryk Gzyl
Journal: Journal of Probability and Statistics
 (2016)

---

## Abstract

The method of maximum entropy is quite a powerful tool to solve the generalized moment problem, which consists in determining the probability density of a random variableX from the knowledge of the expected values of a few functions of the variable. In actual practice, such expected values are determined from empirical samples, leaving open the question of the dependence of the solution upon the sample. It is the purpose of this note to take a few steps towards the analysis of such dependence.


---

## Body

## 1. Introduction and Preliminaries

To state what the generalized moment problem is about, let(Ω,F,P) be a probability space and let (S,B,m) be a measure space, with m a finite or sigma-finite measure. Let X be an S-valued random variable, such that its distribution has a density with respect to the measure m. The generalized moment problem consists in determining a density f(x) such that (1)∫Shkxfxmdx=dkfork=0,1,…,M,where hk is a collection of measurable functions hk:S→R and dk are given real numbers, and we set h0≡1 and d0=1 to take care of the natural requirement on f. A typical example is the following. X stands for a positive random variable (a stopping time, o perhaps being total risk severity) and we can compute E[exp⁡(-αkX)]=dk by some Monte Carlo procedure at a finite number of points αk. The problem that we need to solve amounts to inverting the Laplace transform from such finite collection of values of the transform parameter α.Actually this last problem is of much interest in the banking and insurance industries, where the density is necessary to compute risk premia and regulatory capital of various types, samples may be small, and the estimation ofdk reflects that. We direct the reader to Gomez-Gonçalves et al. [1], where this issue was addressed in the context of risk modeling and Laplace transform inversion.Let us denote byf∗ the solution to problem (1) obtained by the maximum entropy method as explained in Section 3 below when the moments in the right hand side are known exactly. As in many situations the moments are to be estimated empirically as detailed in (2) below, it is to be expected that the maxentropic distribution obtained will depend on the sample X1,…,XN used to compute d^k. We shall denote this maxentropic density by f^N∗ to emphasize its dependence on the empirical moments d^k.The problem that we address in this note is the convergence on thefN∗ to f∗ as well as that of the oscillations of mean values computed with fN∗ about the oscillations of mean values computed with respect to f∗.When we have a sampleX1,…,XN, the empirical generalized moments (the sample averages) are given by (2)d^k=1N∑n=1NhkXnwhich fluctuate around the exact moments dk; we thus expect the output of the maxentropic procedure to somehow reflect this variability.For that, in the next section we recall in a (short) historical survey the notion of entropy of a density, and in the following section we present the basics of the maximum entropy method.In Section4 we take up the main theme of this work: the variability of f^N∗ that comes in through d^k. There we prove that f^N∗ converges pointwise and in L1 to the maxentropic density f∗ obtained from the exact data, and we shall examine how f^N∗ deviates from f∗ in terms of the difference between the true and the estimated (sample) moments. We examine as well the deviations of expected values like ∫g(x)fN∗(x)dm(x) from ∫g(x)f∗(x)dm(x). That the density reconstruction from empirical moments has to depend on the sample seems to be intuitive, but neither the behavior of the maxentropic density as the sample size increases nor the fluctuations of the expected values with the densities fN∗ seem not to have been studied before.


## 2. The Entropy of a Density

As there seem to be several notions of entropy, it is the aim of this section to point out that they are all variations on the theme of one single definition. Let us begin by spelling out what it is that we call the entropy of a density. LetP be a measure on (Ω,B). Suppose that P≪m and let f denote its density. The entropy Hm(P) (when we want to emphasize the density we write Hm(f)) is defined by (3)HmP=-∫Sfln⁡fdmwhenever ln⁡(f) is P-integrable, or -∞ if not. Hm(P) is called the entropy of P (with respect to m) and Hm(f) is called the entropy of f. Actually, we can also define Hm(f) for f∈{g∈L1(dm),g≥0} as follows. When P is not necessarily a probability measure having a density f with respect to m, (3) is to be modified as follows: (4)HmP=Hmf=-∫Sfln⁡fdm+∫Sfdm-1.When m is a probability measure, denote it by Q, and both P and Q are equivalent to a measure n, with densities given, respectively, by f=dP/dn and g=dQ/dn; then (3) can be written as (5)HnP,Q=-∫Sfgln⁡fggdn=-∫Sln⁡fgfdn=-∫Sln⁡fgdPand we call it the entropy of f with respect to g.Comment. For the applications that we shall be dealing with, Ω will stand for a closed, convex subset in some Rn, and m will be the usual Lebesgue measure. We also mention that when m is a discrete measure, then the integrals would become sums.The expression (3) seems to have made its first appearance in the work of Boltzmann in the last quarter of the XIXth century. There it was defined in R3×R3, where f(x,v)dxdv was to be interpreted as the number of particles with position within dx and velocity within v. The function happened to be a Lyapunov functional for the dynamics that Boltzmann proposed for the evolution of a gas, which grew as the gas evolved towards equilibrium. Not much later Gibbs used the same function, but now defined on R6N×R6N, whose points (q,p) denote the joint position and momenta of a system of N particles. This time dm=dqdp, and f(q,p)dqdp is the probability of finding the system within the specified “volume” element. Motivated by earlier work in thermodynamics, it was postulated that in equilibrium the density of the system yielded a maximum value to the entropy Hm(f). These remarks explain the name of the method.The expression (3) (with a reversed sign) made its appearance in field of information transmission under the name of information content in the density f; that is why it is sometimes called the Shannon-Boltzmann entropy. Also, expression (5) appeared in the statistical literature under the name of (Kullback-) divergence of the density f with respect to the density g, and it is denoted by K(f,g) and equal to -Hn(P,Q). See Cover and Thomas [2] or Kullback [3] for a detailed study of the properties of the entropy functions.Having made those historical remarks and having stated those equivalent definitions, we mention that we shall be working mostly with (3). In what comes below we make use of some interesting and well-known properties of (3) and (5), which we gather under the following.Theorem 1.
With the notation introduced above, one gets the following:(i)
The functionf→Hm(f) is strictly concave.(ii)
For any two densitiesf and g, Hn(f,g)≤0, and Hn(f,g)=0 if and only if f=g a.e. n.(iii)
For any two densitiesf, g such that Hn(f,g) is finite, one has (Kullback’s inequality) (6)12f-g12≤-Hnf,g.The reader is directed to either Cover and Thomas [2] or Kullback [3] for proofs.


## 3. The Standard Maximum Entropy Method

Here we recall some well-known results about the standard maximum entropy (SME) method along with some historical remarks. Even though the core idea seems to have been first made in the work of Esscher [4], where he introduced what nowadays is called the Esscher transform, it was not until the mid-1950s that it became part of the methods used in statistics, through the work of Kullback [3]. It seems to have been first formulated as a variational procedure by Jaynes [5] to solve the (inverse) problem consisting in finding a probability density f(y) (on the phase space of a mechanical system), satisfying the following integral constraints:(7)∫Shkxfxdmx=dkfork=0,1,…,M,where dk are observed (measured) expected values of some functions (“observables” in the physicist’s terminology), of the random variable X. That problem appears in many fields; see Kapur [6] and Jaynes [7], for example.Usually, we seth0≡1 and d0=1 to take care of the natural requirement on f(x). It actually takes a standard computation to see that when the problem has a solution, it is of the type(8)fM∗x=exp⁡-∑k=0Mλk∗hkxin which the number of moments M appears explicitly. It is usually customary to write e-λ0∗=Zλ∗-1, where λ∗=(λ1∗,…,λM∗) is an M-dimensional vector. Clearly, the generic form of the normalization factor is given by(9)Zλ=∫01e-∑k=1Mλkhkxmdx.With this notation the generic form of the solution can be rewritten as(10)f∗x=1Zλ∗e-λ∗,hx=e-∑k=0Mλk∗hky.Here a,b denotes the standard Euclidean scalar product in RM, and h(x) is the vector with components hk(x). At this point we mention that the simple minded proof appearing in many applied mathematics or physics textbooks is not really correct. That is because the set of densities is not open in L1(dm). There are many alternative proofs. Consider, for example, the work by Csiszar [8] and Cherny and Maslov [9].The heuristics behind (10) and what comes next are the following. If in statement (ii) of Theorem 1 we take g(x) to be any member of the exponential family λ→e-∑k=0M‍λkhkx, the inequality becomes(11)Hmf≤ln⁡Zλ+λ,dwhich suggests that if we find a minimizer λ∗ such that the inequality becomes an equality, by Theorem 1 we conclude that (10) is the desired solution. This dualization argument seems to have been first proposed in Mead and Papanicolaou [10] and is expounded in full rigor in Borwein and Lewis [11]. The vector λ∗ can be found minimizing the dual entropy:(12)Σλ,d=ln⁡Zλ+λ,d,where d is the M-vector with components dk, and obviously the dependence of f∗ on d is through the minimizer λ∗. We add that, technically speaking, the minimization of Σλ,d is over the domain of Z(λ) which is a convex set with a nonempty interior, and usually the minimum is achieved in its interior. In many applications the domain of Z(λ) is RM. And for the record, we state the result of the duality argument as follows.Lemma 2.
With the notations introduced above, if the minimizerλ∗ of (12) lies in the interior of the domain of Z(λ), then (13)Hmf∗=Σλ∗,d=ln⁡Zλ∗+λ∗,d.The proof goes as follows. Note that ifλ∗ is a minimizer of (12), the first order condition is -∇λln⁡Zλ∗=d, which written explicitly states that (10) satisfies the constraints (7). Since the entropy of this density is given by the right hand side of (13), it must be the density that maximizes the entropy.


## 4. Mathematical Complement

In this section we gather some results aboutZ(λ) that we need as follows.Proposition 3.
With the notations introduced above, suppose that the matrixC which we use to denote the covariance of h(x) computed with respect to the density f∗ is strictly positive definite. Let one suppose as well that the set {λ∈RM∣Zλ<∞} is an open set. Then one gets the following:(1)
The functionZ(λ) defined above is log-convex; that is, ln⁡Z(λ) is convex.(2)
Z(λ) is continuously differentiable as many times as one needs.(3)
If one setsϕ(λ)=-∇λln⁡Z(λ), then λ=ϕ-1(d) is continuously differentiable in d.(4)
The JacobianD of ϕ-1 at d equals the (negative) covariance matrix of h(x) computed with respect to f∗(x).The first two assertions are proved in Kullback’s book. Actually, the log-convexity ofZ(λ) is a consequence of Hölder’s inequality, and the analyticity of Z(λ) involves a systematic estimation procedure. The third drops out from the inverse function theorem in calculus. See Fleming [12], and the last one follows from the fact that the Jacobian of ϕ-1 equals the negative of the inverse of the Hessian matrix of ln⁡Z(λ), which is (minus) the covariance matrix C. As a simple consequence of item (4) in Proposition 3 we have the following result which is relevant for the arguments in the next section.Theorem 4.
With the notations introduced above, settingδd=d^N-d, the following assertions hold. The change in λ∗ as d→d+δd up to terms o(δd) is given by (14)δλ=Dδdand, more importantly, using (10) and, again, up to terms o(δd), (15)f^N∗x-f∗x=-f∗xhx-d,Dδd.To sketch a proof of (15) we proceed as follows. Let δλ=λN∗-λ∗; then(16)f^N∗x=e-λ∗,hxZλ∗=e-λ∗+δλ,hxZλ∗+δλ.Now, neglecting terms of second order in δλ, we approximate the numerator by (17)e-λ+δλ,hx=e-λ,hx1+δλ,hxand the denominator by(18)Zλ∗+δλ=Zλ∗1+∇λln⁡Zλ∗,δλ=Zλ∗1+d,δλ,where we used the fact that at the minimum(19)-∇λln⁡Zλ∗=d.Therefore (20)f^N∗x=e-λ,hx1+δλ,hxZλ∗1+d,δλfrom which the desired result readily drops out after neglecting terms of second order in δλ.


## 5. Sample Dependence

Throughout this section, we shall consider a sample{X1,…,XN} of size N of the random variable X. Here we shall relate the fluctuations of d^k=1/N∑n=1N‍hk(Xn) around its mean to the fluctuations of the density. The following is obtained from an application of the strong law of large numbers.Theorem 5.
Suppose thath is integrable, with mean d and covariance matrix Σ(h). Then, for each N, (2) is an unbiased estimator of d and(21)d^k=1N∑n=1NHkXn⟶da.s.P,whenN⟶∞.Consider now the following.Proposition 6.
Define the empirical moments as in (2). Denote by λ^N∗ the Lagrange multiplier determined as explained in Section 2. Then, as N→∞, d^N→d and therefore λ^N∗→λ∗ (a.s.) P.Iff^N∗ and f∗ are the maxentropic densities given by (10), corresponding, respectively, to d^ and d, then f^N∗→f∗ pointwise for every x and almost surely P.The proof hinges in the following arguments. From Theorems5 and 4 we obtain the first assertion. The rest follows from the continuous dependence of the densities on the parameter λ∗. Also, taking limits as N→∞ in (15), we obtain another proof of the convergence of f^N∗ to f∗.The next result concerns the convergence off^N∗ to f∗ in L1(dm).Theorem 7.
With the notations introduced above, one has(22)f^N∗→L1dmf∗asN⟶∞.Proof.
The proof is a consequence of the continuous dependence ofΣ(λ,d) on its arguments, of the identity (13) and item (iii) in Theorem 1 with f^N∗ playing the role of f and f∗ playing the role of g. In this case -Hm(f^M∗,f∗) happens to be(23)-Hmf^N∗,f∗=λ∗,d^N-λN∗,d^N+ln⁡ZλN∗-ln⁡Zλ∗,which, as mentioned, tends to 0 as N→∞.To continue, consider the following.Theorem 8.
With the notations introduced above, one has the following:(1)
f^N∗ is an unbiased estimator of f∗.(2)
For any bounded, Borel measurable functiong(x),(24)∫gxf^N∗xdmx-∫gxf∗xdmx2≤∫hxgxf∗x-d∫gxf∗x2δd,Dδd.The proof follows from (15). Multiply both sides of that identity by g(x) and integrate with respect to dm(x) and then invoke the Cauchy-Schwartz inequality in RK to obtain the inequality.What is interesting about (2) in Theorem8 is the possibility of combining it with Chebyshev’s inequality to obtain rates of convergence. It is not hard to verify that (25)PD1/2d^N-d>a≤trΣNDa2=trDΣNa2,where · is the Euclidean norm in RK and(26)ΣN=EPδdδdt=1NEPhX-dhX-dt=1NΣh.Corollary 9.
With the notations introduced in Theorem8 and the two lines above,(27)P∫gxf^N∗xdmx-∫gxf∗xdmx≤a≥1-∫hxgxf∗x-d∫gxf∗x2tr⁡DΣhNa2.Comment. If we take g=IA, we obtain a simple estimate of the speed of decay of ∫Af^N∗(x)dm(x)-∫Af∗(x)dm(x) to zero, or of the speed of convergence of ∫Af^N∗(x)dm(x) to ∫Af∗(x)dm(x) if you prefer. Regarding the fluctuations around the mean, consider the following two possibilities.Theorem 10.
With the notations introduced in Theorem4 and in the identity (15), one has (28)Nf^N∗x-f∗x=-f∗xhx-d,DNδd~N0,σ2xin law as N→∞. Also, for any bounded, Borel measurable g(x), (29)N∫gxf^N∗xdmx-∫gxf∗xdmx=∫hxgxf∗x-d∫gxf∗x,DNδd~N0,σ2gin law. Above, σ2(x)=v(x),Σ(h)v(x), σ2(g)=v(g),Σ(h)v(g), where v(x)=f∗(x)D1/2h(x)-d and v(g)=∫g(x)v(x)dm(x).The proof of the assertions involves applying the central limit theorem to the vector variableNh^N-d.Final Comments. Some numerical results illustrating the results presented here appear in the paper by Gomez-Gonçalves et al. [1]. There, they display graphically a “cloud of densities” corresponding to samples on various sizes and see how they shrink to the plot of the true (or exact) density as the sample size increases.



---

*Source: 8629049-2016-12-14.xml*