Talk:Differential entropy

message: Regarding "Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not.[citation needed]" I propose https://www.crmarsh.com/static/pdf/Charles_Marsh_Continuous_Entropy.pdf (dead link) 167.191.202.243 (talk) 20:29, 23 July 2019 (UTC)
thread_title:  citation in first paragraph 

message: The normal distribution maximizes differential entropy for a distribution with a given mean and a given variance. Perhaps it should be emphasized that there are distributions for which the mean and variance do not exist, such as the Cauchy distribution.
--Kaba3 (talk) 11:27, 20 October 2014 (UTC)
thread_title:  Maximization of differential entropy 

message: i am not sure how familiar the avg information theorist is with measure theory and integration. the article does use language such as "almost everywhere", "random variable", etc, so it must not be foreign. in that case, the division between "differential" and "non-differential" entropy is unnecessary, and quite misleading.  one can simply use a single definition for entropy, for any random variable on a arbitrary measure space, and surely it has been done somewhere. Mct mht 03:53, 18 June 2006 (UTC)
replies: text: While not being an expert on the subject, I do not agree, see Information entropy#Extending_discrete_entropy_to_the_continuous_case:_differential_entropy - it is probably possible to give such a unique definition, but it would add unnecessary complication to the article; say, a Computer Science student does not normally need to study measure theory (I have a certain informal knowledge of it), as not all probability student need to know it. Beyond that, Shannon entropy is almost always used in the discrete version even because that's the natural application; beyond that, the two versions have different properties (the other article dismiss differential entropy as being of little use).--Blaisorblade 18:34, 10 September 2007 (UTC)
thread_title:  unnecessary treatment 

message: I believe there is an error in the differential entropy for multivariate Gaussian (it should be in minus sign). No access to tex to correct it.  — Preceding unsigned comment added by 93.173.58.105 (talk) 17:08, 6 October 2014 (UTC)
replies: text: I'm gonna enjoy watching you die, so let me it with my own eyes.(Darth Vader) 5.176.76.35 (talk) 09:23, 7 April 2024 (UTC)
thread_title: Error in Text

message: This article was automatically assessed because at least one WikiProject had rated the article as start, and the rating on other projects was brought up to start class. BetacommandBot 09:48, 10 November 2007 (UTC)
thread_title:  WikiProject class rating

message: Note that the continuous mutual information I(X;Y) has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of X and Y as these partitions become finer and finer. Thus it is invariant under quite general transformations of X and Y, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.
I actually added this statement some time ago to the article (anonymously under my IP address), and it was recently marked as needing a citation or reference.  I made the statement a little weaker by changing "quite general transformations" to "linear transformations", and added Reza's book as a reference.  However, it is still true that I(X;Y) is invariant under any bijective, continuous (and thus monotonic) transformations of the continuous spaces X and Y.  This fact needs a reference, though. Deepmath (talk) 06:57, 12 March 2008 (UTC)
thread_title:  Continuous mutual information 

message: <rant>
Hmm. At the top of this talk page, there is discussion of measure theory, but there is not a drop of measure theory to be found in this article. FWIW, at least some textbooks on probability *start* with defining measure theory (I own one such), so I'm not sure why this should be so scary.
Anyway, this article *completely* glosses over the hard problem of defining the continuous entropy.  Consider a digital electronic oscillator, which nominally outputs two voltages: 0 Volts and 1 Volt. Now, if the oscillator is running at very high speed, it typically won't hit exactly 0 and 1, it will have a (continuous) distribution of voltages around 0 and around 1. If the distribution is very sloppy, and I compute entropy with the integral formula, I get a nice positive entropy. But if I sharpen up the oscillator, so that it hits the voltages 0 and 1 more and more precisely (more and more sharply peaked at 0 and 1), instead of being sloppy, the "continuous entropy" will go negative -- more and more negative, the sharper the peaks get. This is very disconcerting if one was hoping to get the entropy resembling that of a discrete coin toss out of the system. 
This is an actual problem which does show up, and is partly solvable
by using bins, and doing bin-counting. But this hardly merits the la-dee-da treatment given here!
</rant> linas (talk) 15:21, 29 August 2008 (UTC)
thread_title:  WTF!  This is a mess. 

message: This property is given in the article, thought it would be valuable to add a full proof.
One proof could be made for smooth PDF's using Lagrange multipliers.
There is a more general proof:
Let g \left(x \right) be a Normal Distribution PDF.
Without loss of generality (See Differential Entropy Properties) we could assume g \left(x \right) to be centered with its second moment to be 1:
 \int_{- \infty}^{\infty}g \left(x \right)xdx=0, \ \int_{- \infty}^{\infty}g \left(x \right){x}^{2}dx=1
Let f \left(x \right) be any arbitrary PDF with the same statistical properties.
We'll apply on them the Kullback–Leibler divergence (Mind the Minus Sign):
-{D}_{KL} \left(f \left(x \right)\|g \left(x \right) \right)=
::=-\int_{- \infty}^{\infty}f \left(x \right) \log \left( \frac{f \left(x \right)}{g \left(x \right)} \right)dx
::=\int_{- \infty}^{\infty}f \left(x \right) \underset{\log \left(x \right) \leq x-1}{\underbrace{\log \left( \frac{g \left(x \right)}{f \left(x \right)} \right)}}dx
::\leq \int_{- \infty}^{\infty}f \left(x \right) \left( \frac{g \left(x \right)}{f \left(x \right)}-1 \right)dx
::=\int_{- \infty}^{\infty}g \left(x \right) - f \left(x \right) dx=0
Looking back on \int_{- \infty}^{\infty}f \left(x \right) \log \left( \frac{g \left(x \right)}{f \left(x \right)} \right)dx we get:
\int_{- \infty}^{\infty}f \left(x \right) \log \left( \frac{g \left(x \right)}{f \left(x \right)} \right)dx
::={h}_{f \left(x \right)} \left(X \right) + \underset{=-{h}_{g \left(x \right)} \left(X \right)}{ \underbrace{\int_{- \infty}^{\infty}f \left(x \right) \log \left(g \left(x \right) \right)dx)} }
::={h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right)
Applying the inequality we showed prviously we get: 
{h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \leq 0
With the Equality holds only for g \left(x \right)=f \left(x \right).
--Royi A (talk) 13:07, 30 January 2010 (UTC)
thread_title:  Normal Distribution Maximizes The Differential Entropy For a Given Variance 

message: In the properties section, it is stated that the entropy of a transformed RV can be calculated by adding the expected value of the log of the Jacobian to the entropy of the original RV. I believe, however, that this is only true for bijective transforms, but not for the general case.  —Preceding unsigned comment added by 129.27.140.190 (talk) 09:08, 19 July 2010 (UTC)
thread_title:  Transformations of Random Variables 

message: The current proof on that a normal distribution maximizes differential entropy seems erroneous to me. First, it does not use normality anywhere. Second, the last integral is actually not h_g.
--Kaba3 (talk) 15:52, 30 August 2010 (UTC)
thread_title:  Erroneous proof? 

message: The normal distribution maximizes the entropy for a fixed variance and MEAN. In the final step of the proof, it says that
:\int_{-\infty}^\infty f(x)\frac{(x-\mu)^2}{2 \sigma^2}=\frac{1}{2}
which is true only if \mu is also the mean of f(x). I will give an alternate proof based on Lagrange multipliers if no one has an objection. PAR (talk) 01:56, 27 July 2011 (UTC)
replies: text: \frac{\sigma_g}{2\sigma_f}
Maybe the requirement that the two have the same variance was dropped? If not, they won't cancel and you end up with the ratio of standard deviations in the answer. Or am I missing something?  — Preceding unsigned comment added by 2620:10D:C091:480:0:0:1:7F8F (talk) 20:39, 1 March 2023 (UTC)
thread_title:  Maximization in the normal distribution - Proof is incorrect 

message: I undid an edit of User:Portabili, but perhaps there is something to discuss, so I bring it up here.  The undone edit spoke of the limit of a normal distribution as the variance increases without bound and how that is similar to a uniform distribution and how the uniform distribution on the real line maximizes differential entropy.  Unfortunately, the uniform distribution on the real line is at best an improper probability distribution.  The article does not discuss these at all, and it is not at all clear from the present article how differential entropy would apply to an improper distribution.  If someone has a source that discusses this, it might merit adding material to the article.  𝕃eegrc (talk) 13:39, 13 January 2017 (UTC)
thread_title:  Application to improper probability distributions 

message: in section https://en.wikipedia.org/wiki/Differential_entropy#Proof
--2A02:560:4109:AE00:8868:8CDC:7CBA:B775 (talk) 21:04, 15 October 2018 (UTC)
thread_title:  MathML error