# Recovering Decay Rates from Noisy Measurements with Maximum Entropy in the Mean

Authors: Henryk Gzyl; Enrique Ter Horst
Journal: Journal of Probability and Statistics
 (2009)

---

## Abstract

We present a new method, based on the method of maximum entropy in the
mean, which builds upon the standard method of maximum entropy, to improve the parametric estimation of a decay rate when the measurements are corrupted by large level of
noise and, more importantly, when the number of measurements is small. The method is
developed in the context on a concrete example: that of estimation of the parameter in an
exponential distribution. We show how to obtain an estimator with the noise filtered out,
and using simulated data, we compare the performance of our method with the Bayesian
and maximum likelihood approaches.


---

## Body

## 1. Introduction

Suppose that you want to measure the half-life of a decaying nucleus or the life-time of some elementary particle, or some other random variable modeled by an exponential distribution describing, say a decay time or the life time of a process. Assume as well that the noise in the measurement process can be modeled by a centered Gaussian random variable whose variance may be of the same order of magnitude as that of the decay rate to be measured. We assume that the varianceδ of noise is known. To make things worse, assume that you can only collect very few measurements.We want to emphasize, that the method developed is tailored to this one important model in applications, and on the other hand, to the fact that the samples have to be small and contaminated by observational noise (on top of their inherent randomness). And what the method provides in general, is a technique for filtering noise out.Thus, ifxi denotes a realized value of the variable, one can only measure yi=xi+ei, for i=1,2,…,n, where n is a small number, say 2 or 3, and e1 denotes the additive measurement noise. When one knows that the distribution of X is exponential, the parameter should be estimated by (1/n)∑yi=(1/n)∑xi+(1/n)∑ei. In other words, assume that you know that the sample comes from a specific parametric distribution but is contaminated by additive noise, then your estimator of the relevant parameters is contaminated by the measurement noise. What to do? One possible approach is to apply to the small sample the standard statistical estimation procedures like maximum likelihood. But these work well when the sample size is larger than what concerns us here. In our particular example, the MLE is (1/n)∑yi in which the noise may be important (unless n is large.) Thus apart from the issues arising from the smallness of the sample, we have the issue of the presence of the observational noise. We should direct the reader to the work of Rousseeuw and Verboven [1], in which issues relating to estimation in small samples are discussed.Still another possibility, the one we want to explore here, is to apply a maxentropic filtering method, to estimate both the unknown variable and the noise level. For this we recast the problem as a typical inverse problem consisting of solving forx in
(1.1)y=Ax+e,x∈K,
where K is a convex set in ℝd,y∈ℝk and for some d and k, and A is an k×d-matrix which depends on how we rephrase the our problem. We could, for example, consider the following problem. Find x̂∈[0,∞) such that
(1.2)ŷ=x̂+ê.In our caseK=[0,∞), and we set ŷ=(1/n)Σjyj. Or we could consider a collection of n such problems, one for every measurement, and then proceed to carry on the estimation. Once we have solved the generic problem (1.1), the variations on the theme are easy to write down. What is important to keep in mind here, is that the output of the method is a filtered estimator x̂* of x̂, which itself is an estimator of the unknown parameter. The novelty then is to filter out the noise in (1.2).The method of maximum entropy in the mean (MEM) is rather well suited for solving problems like (1.1). See Navaza's [2] for an early development and Dacunha-Castelle and Gamboa [3] for full mathematical treatment. We shall briefly review what the method is about and then apply it to obtain an estimator x̂ from (1.2). In Section 3 we obtain the maxentropic estimator, and in Section 4 we examine some of its properties, in particular we examine what the results would be if either the noise level were small or the number of measurements were large. We devote Section 4 to some simulations in which the method is compared with a Bayesian and a maximum likelihood approaches.


## 2. The Basics of MEM

MEM is a technique for transforming a possibly ill-posed, linear problem with convex constraints into a simpler (usually unconstrained) but non-linear minimization problem. The number of variables in the auxiliary problem is equal to the number of equations in the original problem,k in the case of example 1. To carry out the transformation one thinks of the x there as the expected value of a random variable X with respect to some measure ℙ to be determined. The basic datum is a sample space (Ωs,ℱs) on which X is to be defined. In our setup the natural choice is to take Ωs=K,ℱs=ℬ(K), the Borel subsets of K, and X=idK the identity map. Similarly, we think of e as the expected value of a random variable V taking values in ℝk. The natural choice of sample space here is Ωn=ℝk and ℱn=ℬ(ℝk) the Borel subsets.To continue we need to select a prior measuresdQs(ξ) and dQn(v) on (Ωs,ℱs) and (Ωn,ℱn). The only restriction that we impose on them is that the closure of the convex hull of both supp(Qs) (resp., of supp(Qn)) is K (resp., ℝk). These prior measures embody knowledge that we may have about x and e but are not priors in the Bayesian sense. Actually, the model for the noise component describes the characteristics of the measurement device or process, and it is a datum. The two pieces are put together setting Ω=Ωs×Ωn;ℱ=ℱs⊗ℱn, and dQ(ξ,v)=dQs(ξ)dQn(v). And to get going we define the class
(2.1)ℙ={P∣P≪Q;AEP[X]+EP[V]=y}.Note that for anyP∈ℙ having a strictly positive density ρ=dP/dQ, then EP[X]∈int(K). For this standard result in analysis check in Rudin's book [4]. The procedure to explicitly produce such P's is known as the maximum entropy method. The first step of which is to assume that ℙ≠∅, which amounts to say that our inverse problem (1.1) has a solution, and we define
(2.2)SQ:ℙ→[-∞,∞)
by the rule
(2.3)SQ(P)=-∫Ωln(dPdQ)dP
whenever the function ln(dP/dQ) is P-integrable and SQ(P)=-∞ otherwise. This entropy functional is concave on the convex set ℙ. To guess the form of the density of the measure P* that maximizes SQ is to consider the class of exponential measures on Ω defined by
(2.4)dPλ=e-〈λ,Aξ〉-〈λ,v〉Z(λ)dQ,
where the normalization factor is
(2.5)Z(λ)=EQ[e-〈λ,Aξ〉-〈λ,v〉].
Here λ∈ℝk, if we define the dual entropy function
(2.6)Σ(λ):𝒟(Q)→(-∞,∞]
by the rule
(2.7)Σ(λ)=lnZ(λ)+〈λ,y〉
or Σ(λ)=∞ whenever λ∉𝒟(Q)≡{μ∈ℝk∣Z(μ)<∞}.It is easy to prove that,Σ(λ)≥SQ(P) for any λ∈𝒟(Q), and any P∈ℙ. Thus if we were able to find a λ*∈𝒟(Q) such that Pλ*∈ℙ, we are done. To find such a λ* it suffices to minimize (the convex function) Σ(λ) over (the convex set) 𝒟(Q). We leave for the reader to verify that if the minimum is reached in the interior of 𝒟(Q), then Pλ*∈ℙ. We direct the reader to [4, 5] for all about this, and much more. For a review of the use of maximum entropy on the mean for solving linear inverse problems, the reader might want to look at [6].


## 3. Entropic Estimators

Let us now turn our attention to (1.2). Since our estimator is a sample mean of an exponential (of unknown parameter), it is natural to assume, for the method described in Section 2, that the prior Qs for X is a Γ(n,α/n), where α>0 is our best (or prior) guess of the unknown parameter. Here in after we propose a criterion for the best choice of α. Similarly, we shall chose Qn to be the distribution of a N(0,δ2/n) random variable as prior for the noise component. Things are rather easy under these assumptions. To begin with, note that
(3.1)Z(λ)=eλ2δ2/2n(λ/nα+1)n,
and the typical member dPλ(ξ,v) of the exponential family is now
(3.2)dPλ(ξ,v)=(λ+nα)nξn-1Γ(n)e-(λ+nα)ξe-(v+δ2λ/n)2(n/2δ2)(2πδ2/n)1/2dξdv.
It is also easy to verify that the dual entropy function Σ(λ) is given by
(3.3)Σ(λ)=λ2δ22n-nln(λnα+1)+λŷ,
the minimum value of which is reached at λ* satisfying
(3.4)λ*δ2n-1/αλ*/nα+1+ŷ=0,
and, discarding one of the solutions (because it leads to a negative estimator of a positive quantity), we are left with
(3.5)λ*nα=12(-(1+ŷαδ2)+((1-ŷαδ2)2+4α2δ2)1/2),
from which we obtain that
(3.6)λ*nα+1=12((1-ŷαδ2)+((1-ŷαδ2)2+4α2δ2)1/2)
as well as
(3.7)x̂*=EP(λ*)[X]=n(λ*+nα)=[α2((1-ŷαδ)+(1-ŷαδ)2+4α2δ2)1/2]-1,ê*=EP(λ*)[V]=-δ2λ*n.Comment.
Clearly, from (3.4) it follows that ŷ=x̂*+ê*. Thus it makes sense to think of x̂* as the estimator with the noise filtered out, and to think of ê* as the residual noise.


## 4. Properties ofx̂*

Let us now spell out some of the notation underlying the probabilistic model behind (1.1). We shall assume that the xi and the ei in the first section are values of random variables Xi and εi defined on a sample space (𝕎,𝒲). For each θ>0, we assume to be given a probability law P(θ) on (𝕎,𝒲), with respect to which the sequences {Xk∣k=1,2,…} and {εk∣k=1,2,…} are both i.i.d. and independent of each other, and  with respect to P(θ),Xk~exp(θ) and εk~N(0,δ2). That is, we consider the underlying model for the noise as our prior model for it. Minimal consistency is all right. From (3.6) and (3.7), the following basic results are easy to obtain.Lemma 4.1.
If we takeα=1/ŷ, then λ*=0,  x̂*=ŷ, and ê*=0.Comment.
Actually it is easy to verify that the solution tox̂*(α)=1/α is α=1/ŷ.To examine the case in which large data sets were available, let us add a superscriptn and write ŷ(n) to emphasize the size of the sample. If x̂(n) denotes the arithmetic mean of an i.i.d. sequence of random variables having exp(θ) as common law, it will follow from the LLN the following.Lemma 4.2.
Asn→∞ then
(4.1)(x̂(n))*→x̃(α)≡[α2((1-θαδ2)+((1-θαδ2)2+4α2δ2)1/2)](-1).Proof.
Start from (3.7), invoke the LLN to conclude that ŷ(n) tends to θ and obtain (4.1).Corollary 4.3.
The true parameter is the solution ofx̃(α)-1/α=0.Proof.
Just look at the right hand-side of (4.1) to conclude that x̃(1/θ)=θ.Comment.
What this asserts is that when the number of measurements is large, to find the right value of the parameter it suffices to solvex̃(α)-1/α=0. And when the noise level goes to zero, we have the following.Lemma 4.4.
With the notations introduced above,x̂*→ŷ as δ→0.Proof.
Whenδ→0, the dQn(v)→ϵ0(dv), the Dirac point mass at 0. In this case, we just set δ=0 in (3.4) and the conclusion follows.When we chooseα=1/ŷ, the estimator x̂* happens to be unbiased.Lemma 4.5.
Letθ denote the true but unknown parameter of the exponential, and Pθ(dy) have density
(4.2)fθ(y)=∫-∞yθn(y-s)n-1e-θ(y-s)e-s2/2δ2Γ(n)2πδds
for y>0 and 0 otherwise. With the notations introduced above, one has EP(θ)[(x̂(n))*]=1/θ whenever the prior α for the maxent is the sample mean ŷ.Proof.
It drops out easily from Lemma4.1, from (1.2), and the fact that the joint density fθ of ŷ is a convolution.But the right choice of the parameterα is still a pending issue. To settle it we consider once more the identity |ŷ-x̂*|=|ê*|. In our particular case we shall see that α=0 minimizes the right-hand side of the previous identity. Thus, we propose to choose α to minimize the residual or reconstruction error.Lemma 4.6.
With the same notations as above,ê* happens to be a monotone function of α and ê*(α=0)=(1/2)(ŷ-ŷ2+4δ2) and ê*(α→∞)=ŷ. In the first case x̂*(α=0)=(1/2)(ŷ+ŷ2+4δ2), whereas in the second x̂*(α→∞)=0.Proof.
Recall from the first lemma that whenαŷ=1, then ê*=0. A simple algebraic manipulation shows that when αŷ>1 then ê*>0, and that when αŷ<1 then ê*<0. To compute the limit of ê* as α→∞, note that for large α we can neglect the term 4/δ2 under the square root sign, and then the result drops out. It is also easy to check the positivity of the derivative of ê* with respect to α. Also clearly |ê*(0)|<|ê*(∞)|.To sum up, with the choiceα=0, the entropic estimator and residual error are
(4.3)x̂*(0)=12(ŷ+ŷ2+4δ2),ê*(0)=12(ŷ-ŷ2+4δ2).


## 5. Simulation and Comparison with the Bayesian and Maximum  Likelihood Approaches

In this section we compare the proposed maximum entropy in the mean procedure with the Bayesian and maximum likelihood estimation procedures. We do that by simulating data and carrying out the three procedures and plotting the histograms of the corresponding estimators. First, we generate histograms that describe the statistical nature ofx̂* as a function of the parameter α. For that we generate a data set of 5000 samples of 1, 3, 5, and 10 measurements, and for each of them we obtain x̂* from (4.3). Also, for each data point we apply both a Bayesian estimation method, a simple-minded maximum likelihood estimation and a maximum likelihood method and plot the resulting histograms.


### 5.1. The Simple-Minded MLE

This consists of an application of the MLE method as if there was no measurement noise. We carried out this for the sake of comparison, to verify that when the sample size becomes larger, the effect of the measurement noise is washed away on the average. The plot of the results forn=1 is too scattered, and we do not display it. The result of the simulations is displayed in Figure 1.The simple MLE for differentn.
(a)(b)(c)


### 5.2. The Maxentropic Estimator

The simulated data process goes as follows. Forn=3 the data points y1,y2,y3 are obtained in the following way.(i)
Simulate a value forxi from an exponential distribution with parameter θ(=1).(ii)
Simulate a value forei from a normal distribution N(0,δ2=0.25).(iii)
Sumxi with ei to get yi, if yi<0, repeat first two steps until yi>0.(iv)
Do this fori=1,2,3.(v)
Compute the maximum entropy estimator given by (4.3).We then display the resulting histogram in Figure2.Histogram forE(x) with MEM for different n.
(a)(b)(c)


### 5.3. The Bayesian Estimator

In this section we derive the algorithm for a Bayesian inference of the model given byyi=x+ei, for i=1,2,…,n. The classical likelihood estimator of x is given by ŷ=(1/n)∑i=1nyi. As we know that the unknown mean x has an exponential probability distribution with parameter θ(x~𝔼(θ)), therefore the joint density of the yi and x is proportional to
(5.1)∏i=1n12πδ2exp{-(yi-x)22δ2}θexp(-θx)π(θ),
where θexp(-θx) is the density of the unknown mean x and where π(θ)∝θ-1 is the Jeffrey's noninformative prior distribution for the parameter θ [5].In order to derive the Bayesian estimator, we need to get the posterior probability distribution forθ, which we do with the following Gibbs sampling scheme [7].(i)
Drawx~N(ŷ-θδ2/n,δ2/n)1x>0.(ii)
Drawθ~𝔼(x).Repeat this algorithm many times in order to obtain a large sample from the posterior distribution ofθ in order to obtain the posterior distribution of E(x)=1/θ. For our application, we simulate data with θ=1, which gives an expected value for x equal to E(x)=1.We get the histogram displayed in Figure3 for the estimations of E(x) after 5000 iterations when simulating data for θ=1.Histogram forE(x) with Bayes Method for different n.
(a)(b)(c)


### 5.4. The Maximum Likelihood Estimator

The problem of obtaining a ML estimator is complicated in this setup because data points are distributed like(5.2)fθ(t)=∫-∞tθe-θ(t-s)e-s2/2δ2ds(2πδ2),fθ(t)=θe-θt+(θδ)2/2ℙ(S<t),
where S~N(θδ2,δ2). Therefore, after observing t1,t2, and t3, we get the following likelihood that we maximize numerically:
(5.3)θ3e-θ∑i=13ti+3(θδ)2/2∏i=13ℙ(S<ti).
If we attempted to obtain the ML estimator analytically, we would need to solve
(5.4)nθ-∑j=1n∫-∞tjθe-θ(tj-s)e-s2/2δ2ds/(2πδ2)∫-∞tjθe-θ(tj-s)e-s2/2δ2ds/(2πδ2)=0.Notice that asδ→0 this equation tends to (n/θ)-∑j=1ntj=0 as expected. We can move forward a bit, and integrate by parts each numerator, and after some calculations we arrive to
(5.5)nθ-∑j=1ntj+nδ2θ-∑j=1nδe-tj2/2δ2∫-∞tjθe-θ(tj-s)e-s2/2δ2ds/(2πδ2)=0.Trying to solve this equation inθ is rather hopeless. That is the reason why we carried on a numerical maximization procedure on (5.3). To understand what happens when the noise is small, we drop the last term in the last equation and we are left with
(5.6)nθ-∑j=1ntj+nδ2θ
the solution of which is
(5.7)1θ*=12(ŷ+ŷ2-4δ2)
or θ*=2(ŷ+ŷ2-4δ2)-1, and we see that the effect of noise is to increase the ML estimator. In Figure 4 we plot the histogram of (1/θ)* obtained by numerically maximizing (5.3) for each simulated data point.Histogram forE(x) with the MLE for different n.
(a)(b)(c)The results are summarized in Tables1, 2, and 3.Table 1
Means and standard deviations for different methods whenn=3.
MethodMeanStandard deviationMaximum entropy1.31120.5193Bayesian1.02710.5701Maximum likelihood1.80852.4630Sample average1.09280.5904Table 2
Means and standard deviations for different methods whenn=5.
MethodMeanStandard deviationMaximum entropy1.30900.4034Bayesian1.05320.4561Maximum likelihood0.58170.2016Sample average1.10090.4596Table 3
Means and standard deviations for different methods whenn=10.
MethodMeanStandard deviationMaximum entropy1.30930.2825Bayesian1.08460.3239Maximum likelihood0.26140.0627Sample average1.10970.3230When simulating data forθ=1, the MEM, Maximum likelihood and Bayesian histograms are all skewed to the right and yield a mean under the three simulated histograms close to 1. As shown in the table, compiled for n=3, the MEM method yields a sample mean of 1.3112 with a sample standard deviation of 0.5193, the Bayesian yields a sample mean equal to 1.0271 and sample standard deviation of 0.5701, and the Maximum Likelihood method yields a sample mean of 1.808 with a sample standard deviation of 2.463. All the three methods produce right skewed histograms for E(x). The MEM and Bayesian method provide better and similar results and more accurate than the Maximum Likelihood method, but keep in mind that we carried out an approximate calculation.Note as well that forln~10 the variability in the (approximate) MLE decreases, but it is far away from the true value. This could be due to the numerical approximation that we used, whereas for n~25 the estimator improves considerably. We owe this observation to one of our referee's, who pointed out that for very small samples, the MEM estimator outperforms the MLE estimator because it is biased, and that this advantage disappears as n becomes large.


## 5.1. The Simple-Minded MLE

This consists of an application of the MLE method as if there was no measurement noise. We carried out this for the sake of comparison, to verify that when the sample size becomes larger, the effect of the measurement noise is washed away on the average. The plot of the results forn=1 is too scattered, and we do not display it. The result of the simulations is displayed in Figure 1.The simple MLE for differentn.
(a)(b)(c)


## 5.2. The Maxentropic Estimator

The simulated data process goes as follows. Forn=3 the data points y1,y2,y3 are obtained in the following way.(i)
Simulate a value forxi from an exponential distribution with parameter θ(=1).(ii)
Simulate a value forei from a normal distribution N(0,δ2=0.25).(iii)
Sumxi with ei to get yi, if yi<0, repeat first two steps until yi>0.(iv)
Do this fori=1,2,3.(v)
Compute the maximum entropy estimator given by (4.3).We then display the resulting histogram in Figure2.Histogram forE(x) with MEM for different n.
(a)(b)(c)


## 5.3. The Bayesian Estimator

In this section we derive the algorithm for a Bayesian inference of the model given byyi=x+ei, for i=1,2,…,n. The classical likelihood estimator of x is given by ŷ=(1/n)∑i=1nyi. As we know that the unknown mean x has an exponential probability distribution with parameter θ(x~𝔼(θ)), therefore the joint density of the yi and x is proportional to
(5.1)∏i=1n12πδ2exp{-(yi-x)22δ2}θexp(-θx)π(θ),
where θexp(-θx) is the density of the unknown mean x and where π(θ)∝θ-1 is the Jeffrey's noninformative prior distribution for the parameter θ [5].In order to derive the Bayesian estimator, we need to get the posterior probability distribution forθ, which we do with the following Gibbs sampling scheme [7].(i)
Drawx~N(ŷ-θδ2/n,δ2/n)1x>0.(ii)
Drawθ~𝔼(x).Repeat this algorithm many times in order to obtain a large sample from the posterior distribution ofθ in order to obtain the posterior distribution of E(x)=1/θ. For our application, we simulate data with θ=1, which gives an expected value for x equal to E(x)=1.We get the histogram displayed in Figure3 for the estimations of E(x) after 5000 iterations when simulating data for θ=1.Histogram forE(x) with Bayes Method for different n.
(a)(b)(c)


## 5.4. The Maximum Likelihood Estimator

The problem of obtaining a ML estimator is complicated in this setup because data points are distributed like(5.2)fθ(t)=∫-∞tθe-θ(t-s)e-s2/2δ2ds(2πδ2),fθ(t)=θe-θt+(θδ)2/2ℙ(S<t),
where S~N(θδ2,δ2). Therefore, after observing t1,t2, and t3, we get the following likelihood that we maximize numerically:
(5.3)θ3e-θ∑i=13ti+3(θδ)2/2∏i=13ℙ(S<ti).
If we attempted to obtain the ML estimator analytically, we would need to solve
(5.4)nθ-∑j=1n∫-∞tjθe-θ(tj-s)e-s2/2δ2ds/(2πδ2)∫-∞tjθe-θ(tj-s)e-s2/2δ2ds/(2πδ2)=0.Notice that asδ→0 this equation tends to (n/θ)-∑j=1ntj=0 as expected. We can move forward a bit, and integrate by parts each numerator, and after some calculations we arrive to
(5.5)nθ-∑j=1ntj+nδ2θ-∑j=1nδe-tj2/2δ2∫-∞tjθe-θ(tj-s)e-s2/2δ2ds/(2πδ2)=0.Trying to solve this equation inθ is rather hopeless. That is the reason why we carried on a numerical maximization procedure on (5.3). To understand what happens when the noise is small, we drop the last term in the last equation and we are left with
(5.6)nθ-∑j=1ntj+nδ2θ
the solution of which is
(5.7)1θ*=12(ŷ+ŷ2-4δ2)
or θ*=2(ŷ+ŷ2-4δ2)-1, and we see that the effect of noise is to increase the ML estimator. In Figure 4 we plot the histogram of (1/θ)* obtained by numerically maximizing (5.3) for each simulated data point.Histogram forE(x) with the MLE for different n.
(a)(b)(c)The results are summarized in Tables1, 2, and 3.Table 1
Means and standard deviations for different methods whenn=3.
MethodMeanStandard deviationMaximum entropy1.31120.5193Bayesian1.02710.5701Maximum likelihood1.80852.4630Sample average1.09280.5904Table 2
Means and standard deviations for different methods whenn=5.
MethodMeanStandard deviationMaximum entropy1.30900.4034Bayesian1.05320.4561Maximum likelihood0.58170.2016Sample average1.10090.4596Table 3
Means and standard deviations for different methods whenn=10.
MethodMeanStandard deviationMaximum entropy1.30930.2825Bayesian1.08460.3239Maximum likelihood0.26140.0627Sample average1.10970.3230When simulating data forθ=1, the MEM, Maximum likelihood and Bayesian histograms are all skewed to the right and yield a mean under the three simulated histograms close to 1. As shown in the table, compiled for n=3, the MEM method yields a sample mean of 1.3112 with a sample standard deviation of 0.5193, the Bayesian yields a sample mean equal to 1.0271 and sample standard deviation of 0.5701, and the Maximum Likelihood method yields a sample mean of 1.808 with a sample standard deviation of 2.463. All the three methods produce right skewed histograms for E(x). The MEM and Bayesian method provide better and similar results and more accurate than the Maximum Likelihood method, but keep in mind that we carried out an approximate calculation.Note as well that forln~10 the variability in the (approximate) MLE decreases, but it is far away from the true value. This could be due to the numerical approximation that we used, whereas for n~25 the estimator improves considerably. We owe this observation to one of our referee's, who pointed out that for very small samples, the MEM estimator outperforms the MLE estimator because it is biased, and that this advantage disappears as n becomes large.


## 6. Concluding Remarks

We exhibited the usefulness of the method of maximum entropy of the mean for dealing with estimation problems in which the samples are small and contaminated by noise, thus adding and extra source of randomness which has to be filtered out. The problem we chose, while being real, it is a problem in which the Lagrange multiplierλ could be estimated analytically and the properties of the resulting estimators could be studied analytically as well. This possibility appears as well when the observed signal is Gaussian. In general, to obtain the filtered estimator, one has to determine the Lagrange multipliers numerically.On one hand, MEM backs up the intuitive belief, according to which, if theyi are all the data that you have, it is all right to compute your estimator of the mean for α=0. The MEM and Bayesian methods yield closer results to the true parameter value than the maximum likelihood estimator for small number of measurements.On the other hand, and this depends on your choice of priors, MEM provides us with a way of modifying those priors, and obtain representations likeŷ=x̂*+ê*; where of course x̂*=x̂*(ŷ). What we saw above, is that there is a choice of prior distributions such that x̂*=ŷ and ê*=0.The important thing is that this is actually true regardless of what the “true” probability describing thexi is.



---

*Source: 563281-2009-06-28.xml*