Talk:Maximum entropy probability distribution

message: Who was the first to show that the normal has maximum entropy among all distributions on the real line with specified mean μ and standard deviation σ? A reference for this would be a nice addition to the article. Btyner 21:21, 13 May 2006 (UTC)
thread_title:  History 

message: I am against merging these articles.  The princible of maximal entropy and the maximal extropy probability distributions are completely different; there will not be enough room on the maximal entropy page to treat this topic well.  It certainly needs its own article.  I am removing the tag. Agalmic (talk) 01:19, 4 July 2008 (UTC)
thread_title:  Merging 

message: I am wondering what the maximum entropy distribution would be for a variable in agular space. Note that for such a variable with support [-180°,180°], the points -180° and 180° are the same. So saying that the mean is mean μ=0°, μ=90° or any other value would be equivalent. I suspect that under no specification of standard deviation σ, the maxent distribution would be continuous uniform in angular space, and not a Gaussian. Do you have any references on this? It would be good to write something about the coordinate system dependence on this article. Jorgenumata (talk) 10:04, 29 March 2009 (UTC)
replies: text: OK, so I found out the answer. For a random variable with a circular distribution, given the mean and the (analog of) the variance, the von Mises distribution maximizes entropy. Jorgenumata (talk) 10:27, 28 April 2009 (UTC)
thread_title:  Maximum entropy in other coordinate systems 

message: I added the 'citation needed' because I'd really like some clarification on that point. Is it that in general there is no maximum entropy distribution given the first three moments, or is it that there's something particular about this combination (seems unlikely)?
This is also unclear (to me): "bounded above but there is no distribution which attains the maximal entropy"
Is it asymptotic? Then can't we just take the limit as you approach the bound to get the answer?
If not then can't we just look at all distributions that do exist and meet these conditions, and choose the one with the highest entropy (or the limit as you approach the highest entropy..)? because, there's probably something I misunderstand, if it's not asymptotic to the maximum entropy then to me it sounds like the article says:
#! python
assert all([f(x)<=1 for x in R]),"no f(x) is above 1"
assert max([f(x) for x in R]) == 1,"so the maximum value must be 1"
which is obviously wrong; its okay if the maximum turns out not to be right on the theoretical limit.
(of course my example is iterating over the real numbers... so who am I to complain about things seeming wrong)
Sukisuki (talk) 13:37, 10 April 2010 (UTC)
replies: text: I think I fixed this. PAR (talk) 04:57, 7 January 2022 (UTC)
thread_title:  Caveats 

message: Please include the Gamma distribution.  — Preceding unsigned comment added by 187.64.42.156 (talk) 00:30, 19 April 2012 (UTC)
replies: text: Initially empirical Zipf's distribution and its demonstrated form Mandelbrot's distribution (of which Zipf's is a special case) happen to be maximal entropy distributions too, if the constraint imposed is the order of magnitude (mean value of the log of rank for Zipf) and mean value of a more complicated expression involving usage cost and storage cost for Mandelbrot). I was shown that in 1974. I do not remember the demonstration very well, but they seemed pretty obvious at the time. Perhaps somebody can find a link to them ? 212.198.148.24 (talk) 19:06, 10 May 2013 (UTC)
thread_title:  Examples of maximum entropy distributions 

message: I removed this:
In physics, this occurs when gravity acts on a gas that is kept at constant pressure and temperature: if X describes the distance of a molecule from the bottom, then the variable X is exponentially distributed (which also means that the density of the gas depends on height proportional to the exponential distribution). The reason: X is clearly positive and its mean, which corresponds to the average potential energy, is fixed. Over time, the system will attain its maximum entropy configuration, according to the second law of thermodynamics.
You can't just conclude this from the purely mathematical example (given in the article above where I removed this) of a maximum-entropy distribution constrained to have a given mean; you have to do the physics ! It can be viewed as a coincidence that the two examples both produce an exponential distribution. 178.38.142.81 (talk) 00:11, 2 February 2015 (UTC)
thread_title:  conditions of example don't apply 

message: I find ME interesting, but I find it puzzling as to when, in the real world, it should be invoked. My concern is that the utility of ME is exaggerated, though I would be extremely happy to have this concern allayed. Let me elaborate on my thinking: Suppose I have a set of data. From these I might measure their variance, and with that number in hand, I could maximize entropy subject to the constraint that the result match the variance I have measured. I would then obtain the normal distribution. But, on this basis, am I really supposed to believe that the normal distribution is an appropriate description of my data? What if I chose to measure some other quality of the data, I don't know, maybe kurtosis or mean absolute deviation, or something, then invoking ME would yield a different distribution. Same data, same effect generating the data, but now, all of a sudden, based on what I have chosen to do, a different distribution seems to be implied. To me, this is not how my analyses usually work. If I have data, I can measure the variance, sure, but I would probably want to invoke the central limit theorem to justify use of the normal, not ME. So I'm left wondering, in what circumstances would one actually invoke the very attractive mathematics of Maximum entropy probability distribution? If a bit of discussion on this were added to the article by someone with more insight than I apparently have, then that would be very much appreciated. Isambard Kingdom (talk) 17:37, 30 May 2015 (UTC)
thread_title:  Examples motivating ME + constraint = classic distributions would be appreciated 

message: The truncated Gaussian fails to be the maximum entropy probability distribution for at least some cases:
for example if mean = standard deviation, the maximum entropy distribution is the exponential (it is when you fix only the mean, if you than set the variance to match the one of the exponential, the solution will stay the same), and the truncated Gaussian never has the shape of an exponential regardless of the choice of the parameters.
Numerical counterexample:
with mean 2 and standard deviation 2, the exponential has differential entropy around 1.7 while the truncated Gaussian has differential entropy roughly 1.6.  — Preceding unsigned comment added by Bdn96 (talk • contribs) 10:36, 3 December 2020 (UTC)
thread_title:  Truncated Gaussian maximum entropy for positive support and fixed mean and variance 

message: It seems to me there is a mistake in the section Maximum entropy probability distribution#Uniqueness of the maximum. One can read there « It follows that a distribution satisfying the expectation-constraints and maximising entropy must necessarily have full support », but there are examples of pdf with support on an interval, of maximal entropy among several natural sets of pdf, in particular among \{q\text{ pdf with finite entropy}:-\int q(x)f(x)dx=H(p)\} -using the same notation as in the text. For instance the Beta distribution or the uniform distribution on [0,1]. I believe there is also a mistake in the constraint equality « \int p(x)f(x)dx=-H » which should read \int q(x)f(x)dx=-H=-H(p) as we are introducing a new pdf, here denoted by the variable q, in general different from p. Note that -\int q(x)f(x)dx=H(q,p) is the cross-entropy of q relative to p, and the  Gibbs inequality implies that H(p)=H(q,p)\geq H(q), from which it follows that p is a maximal entropy distribution subject to the cross-entropy equality constraint -though equality of p and q, thus uniqueness of the maximal entropy function, in the case of equality of entropies, seems not to be a part of the usual statement of the Gibbs inequality.
Also, i believe uniqueness of maximum entropy distribution does not hold either for equal supports: for instance the normal distribution with standard deviation \sigma and the Cauchy distribution with \gamma=\frac{\sigma\sqrt{2\pi e}}{4\pi}=\frac{\sigma}{2\sqrt{2\pi /e }} have the same entropy \ln\left(\sigma\sqrt{2\pi e}\right) and are supported on the whole real line but are different. I think the text would be correct if instead of "full support" it asserted "supported Lebesgue-almost surely on supp(p)".
There is also a uniqueness criterion of med without assuming a priori equality of support but only an inequality of what we may call an "entropy integral constraint", see Lemma 4.2 and Theorem 4.3 in https://kconrad.math.uconn.edu/blurbs/analysis/entropypost.pdf . Actually, in this reference, Theorem 7.7 gives uniqueness but only within a convex set of pdf with finite entropy.
The statement « Every probability distribution is trivially a maximum entropy probability distribution under the constraint that the distribution has its own entropy. » is also false i believe, even clarifying "its own entropy" to "the same entropy": the constraint is an equality of a cross-entropy with an entropy. As the example above show, there are pdf p\neq q with H(p)=H(q) and both full support; for those we can deduce that they do not belong to a common convex set of pdf with finite entropy at most H(p).
Please anyone correct me if i am wrong. Plm203 (talk) 06:47, 25 October 2023 (UTC)
thread_title:  Nonfull support and nonuniqueness of maximum entropy distribution