Talk:Information theory and measure theory

message: Where is this in Reza? I could not find it.
:pp. 106-108 198.145.196.71 20:43, 25 August 2007 (UTC)
thread_title: Entropy as a Measure Theoretic measure

message: Also, the Kullback–Leibler divergence should be explained here in a proper measure-theoretic framework. --130.94.162.64 21:27, 20 June 2006 (UTC)
thread_title: Kullback–Leibler divergence

message: I am deeply suspicious of the defintions given here; I think they're wrong.  Normally, given a collection of measureable sets \{X_i\}, the entropy is defined by
:S=-\sum_i \mu(X_i) \log \mu(X_i)
Its critical to have the logarithm in there, otherwise things like the partition function (statistical mechanics) and etc. just fail to work correctly. See, for example, information entropy. 
Also, signed measures are bizarre, and you shoul avoid using them if you cannot explain why they are needed in the theory.
Also, I assumed that when this article said "H(X)", I assumed it meant "entropy", but on second reading, I see that it does not actually say what the symbols H(X) and I(X) are, or what they mean. Without defining the terms, the article is unreadable/subject to (mis-)interpretation, as perhaps I'm doing ??
Finally, I'm unhappy that this article blatently contradicts the article on random variable as to the defintion of what a random variable is. This article states that a random variable is a "set", and that is most certainly not how I understand random variables. 
linas 23:54, 22 June 2006 (UTC)
replies: text: It's been over a year now and this article still needs a major rewrite to clear up the confusion.  I need some help here. -- 198.145.196.71 19:37, 25 August 2007 (UTC)

text: Specifically, the "Other measures in information theory" section is actually more fundamental than the sections that come before it, so that could be one source of confusion. 198.145.196.71 20:40, 25 August 2007 (UTC)
thread_title:  Mis-statement of entropy. 

message: Integration with respect to various measures is one of the main ideas of this article as it stands now.  It ties together differential entropy, discrete entropy, and K–L divergence.  The second main idea is from Reza pp. 106-108 where it is called a "set-theoretic" interpretation of mutual information, conditional entropy, joint entropy, and so forth.  (But Reza very clearly discusses measure there in roughly the way discussed in that section of this article.)  There might be yet more main ideas to discuss in this article as well as references to add and clarifications to make. 198.145.196.71 17:21, 7 September 2007 (UTC)
thread_title: Main ideas

message: I strongly suggest that R. Yeung, and not Fazlollah M. Reza be the primary reference cited.
Specifically, I recommend the reference
"A new outlook on Shannon's information measures", Information Theory, IEEE Transactions on, 1991 vol. 37 (3) pp. 466 - 474
First, in Yeung's paper pg 467, he says
"The use of diagrams to represent the relation among Shannon's information measures has been suggested by Reza [2], Abramson [3], Dyckman [5], and Papoulis [15]."
Yeung's paper constitutes a proof of (good) intuition by previous authors (including Reza).
Second, Reza pg 108, makes a serious misstatement that seems to indicate that the graphical interpretation of Shannon's measures was not completely thought out, and certainly not thought out in terms of a (signed) measure.
"When two variables $X_k$ and $X_j$ are independent, their representative sets will be mutually exclusive."
Yeung says on pg 469
"It was incorrectly pointed out in [2] [Reza] that when two random variables are independent, the corresponding set variables are disjoint."
On pg 470-471, Yeung cites a "classical example (see Gallager [6])" where three random variables, $X$, $Y$, $Z$, have vanishing mutual informations between each pair, and so are independent, yet cannot be represented as a trio without intersecting set variables. This is one of several illustrations that Yeung has mastered this material.
Further, Yeung discusses: 2 variables, 3 variables, Markov chains; bounds on certain quantities for some cases; the "non-intuitive" quantity I(X;Y;Z); and in a rigorous, yet transparent style.  —Preceding unsigned comment added by Mohnjahoney (talk • contribs) 22:00, 18 June 2009 (UTC)
thread_title:  Misattribution of credit? 

message: The article claimes Shannon entropy has the typical properties of a countably-additive measure. However, only the finite additivity property is ever touched upon. The difference between finite additivity (content) and countable additivity (measure) is very decisive in measure theory: almost nothing works with contents. Some justification needs to be given for the word "measure", or the entire section should be deleted. --87.146.22.162 (talk) 00:04, 13 May 2012 (UTC)
thread_title:  measure versus content 

message: This article claims that entropy is a measure in the sense of measure theory. There are two major issues with this claim that have already been mentioned here:
* There is no sigma-algebra.
* There is no proof that entropy is countably additive.
The comments that first raised these issues are very old. It is high time something is done to fix them. I don't doubt that you could get countable additivity somehow, but what is the sigma-algebra? It is total nonsense to say, as one user did, "Just pretend those sets exist."
Here is a possible direction we could go: Cover & Thomas tell us that the entropy-power, 2^{nH(X)} where the entropy is calculated with a base of 2, is the "effective support set size" of a random variable. This non-rigorous statement is based on the asymptotic equipartition property. (C&T also say the Fisher information is akin to the "effective surface area," although they explain it poorly.) If you could make this work, then your sigma-algebra is the same one from the probability space. This seems like the obvious way to go if you want a "set of information" and corresponding sigma-algebra. Maybe someone has already done this, and the article can be about that instead of what we have now. But if it doesn't work, and entropy isn't a measure, then this article has no reason to exist.
137.216.185.166 (talk) 22:04, 7 April 2021 (UTC)
thread_title:  Reasons for deletion 

message: While entropy is evidently not a measure in the measure theory sense, you can define a "measure" or "valuation" on a lattice, and some people have done so. That is one other direction this article could go. 66.17.111.151 (talk) 07:39, 10 April 2021 (UTC)
thread_title:  Alternative using lattices 

message: I thought about the lattice approach and I'm not convinced it would be a good idea. As for entropy being a measure of "effective support set size," I added some content to the Fisher information page related to that but I still don't see how entropy can become an actual measure that way. So I still think this page should be deleted. Here's why:
The article has three sections. The first section has two parts. The first part is a demonstration of how a Riemann integral and an infinite series can be turned into a Lebesgue integral. The second part is essentially the same idea but applied to Kullback-Leibler divergence. Nothing in this section is about the relationship between information theory and measure theory per se. If any of it should be retained, it would make more sense to move it to Entropy (information theory) or Kullback–Leibler divergence.
The second section centers around the untrue claim that "Shannon's 'measure' of information content satisfies all the postulates and basic properties of a formal signed measure over sets." This claim has been repeatedly disputed on the talk page and yet it has been part of the article since its creation nearly 15 years ago. In order for entropy to be a signed measure it must be countably additive and there must be a sigma-algebra of sets for it to measure. Entropy is a function of a probability distribution, not a set. No cited source offers any support for the claim that entropy is in fact a measure. This section seems to come from an analogy between sets, intersection, union, and measure on the one hand and concepts from information theory on the other hand. But you could just as well make an analogy with propositions, conjunction, disjunction, and probability. Anything that forms a lattice and has a kind of "lattice measure" with an inclusion-exclusion property would work. So again, no special relationship between information theory and measure theory is evident here.
The third section is a continuation of the second section and has the same issues. The parts of this section that should be retained are already part of Interaction information, where they belong. 66.17.111.151 (talk) 18:22, 14 April 2021 (UTC)
thread_title:  Updated reasons for deletion