Talk:Mutual information

message: instead
>> It should be noted that these definitions are ambiguous because the base of the log function is not specified. To disambiguate, the function I could be parameterized as I(X,Y,b) where b is the base. Alternatively, since the most common unit of measurement of mutual information is the bit, a base of 2 could be specified.
how about
The unit of information depends on the base of the log function. Most common are bases of 2, e, or 10, resulting in units of bits, nats and digits, respectively.
Internetexploder 08:15, 29 April 2007 (UTC)
replies: text: Noting :Category:Pages needing attention, I would say that, while someone may have thought that a good guideline, it is de facto incorrect (and not policy).  I, for one, do not agree with that guideline, because it hides the fact that the article needs attention from all those who can edit it and it disclaims to newbies that we know the article isn't as good as it could be. — 131.230.133.185 5 July 2005 19:23 (UTC)
[This article is] poorly explained. --Eequor 03:39, 22 Aug 2004 (UTC)
thread_title:  Unit of information? 

message: why not just say:
: I(X,Y) = \sum_{x,y} p(x,y) \times \log_2 \frac{p(x,y)}{p(x)\,p(y)}. \!
instead of all the confusing talk about what f and g are?  Please elaborate if there is a specific reason why it is done this way.  -- BAxelrod 02:08, 19 October 2005 (UTC)
replies: text: The definitions given in the article are correct.  They just happen to be highly formal.  Less formal definitions are given in the article on information theory (recently added by me, but I called it transinformation).  Whether this level of formality is appropriate for this article is a matter for debate.  I tend to think not, because in general, someone who is working at that level of formality is not going to be looking in Wikipedia for a definition, but on the other hand, it "simplifies" matters because then one definition suffices for both the discrete and continuous cases.  (i.e. integration over the counting measure is simply ordinary discrete summation.) -- 130.94.162.64 22:53, 2 December 2005 (UTC)
replies: text: O.K. Simplified the formula. -- 130.94.162.64 05:24, 3 December 2005 (UTC)

text: Another note:   I(X,Y)\,  is incorrect.   I(X;Y)\,  is the accepted usage.  Use a semicolon. -- 130.94.162.64 11:35, 4 December 2005 (UTC)
thread_title:  Simplify eq? 

message: How about adding the mutual information among multiple scalar random variables:
I(y_1;\ldots; y_m)=\sum^m_{i=1}H(y_i)-H(\mathbf{y})
:(In reply to unsigned comment above:) Apparently there isn't a single well-defined mutual information for three or more random variables.  It is sometimes defined recursively:
::I(Y_1; Y_2) = H(Y_1) - H(Y_1 | Y_2),\,
::I(Y_1; \ldots ; Y_m) =  I(Y_1; \ldots ; Y_{m-1}) - I(Y_1; \ldots ; Y_{m-1} | Y_m),\, m\geq 3 ,
:where I(Y_1; \ldots ; Y_{m-1} | Y_m)= \mathbb E _{Y_m}\{I((Y_1|y_m); \ldots;(Y_{m-1}|y_m))\}.
:This definition fits more along the lines of the interpretation of the mutual information as the measure of an intersection of sets, but it can become negative as well as positive for three or more random variables (in contrast to the definition in the comment above, which is always non-negative).
:--130.94.162.64 23:15, 19 May 2006 (UTC)
replies: text: This may or may not be useful to those of you wanting to extend mutual information, see Interaction_information.  Mouse7mouse9 20:30, 9 May 2016 (UTC)
thread_title:  Mutual information between <math>m</math> random variables 

message: The formula is from Shannon (1948). This should be written.
Who coined the term "mutual information"?
--Henri de Solages 18:41, 7 November 2005 (UTC)
thread_title:  Source 

message: The first reference, Cilibrasi and Vitanyi (2005), contains only two mentions of mutual information:
"Another recent offshoot based on our work is hierarchical clustering based on mutual information, [23]."
"[23] A. Kraskov, H. St¨ogbauer, R.G. Adrsejak, P. Grassberger, Hierarchical clustering based on mutual information, 2003, http://arxiv.org/abs/q-bio/0311039"
I suggest this reference be removed as it's not helpful.
--84.9.75.186 10:57, 3 September 2007 (UTC)
replies: text: The Kraskov & Stögbauer paper is an interesting one.  Is that the one you are referring to?  —Dfass 11:25, 3 September 2007 (UTC)
thread_title:  Remove irrelevant reference? 

message: Possible source: [].  Note that some of Scholarpedia is under copyright, so you can't just copy the content.  —Preceding unsigned comment added by Njerseyguy (talk • contribs) 04:45, 7 July 2009 (UTC)
thread_title:  Scholarpedia 

message: It seems to me that there is a possibility for joint probability function p(x,y) to return 0, for example, in such case when variable values x and y never occur together.
Can anyone explain, how it is possible to calculate mutual information, when there is a chance that p(x,y) for some x and y returns 0.
90.190.231.235 (talk) 12:15, 30 December 2009 (UTC)
replies: text: Priors or pseudo-counts are often used to iron out wrinkles like this and account for un-observed data. In this case you can define I(X;Y) := 0 if p(x,y)=0. This is relatively trivial to prove using L'Hôpital's rule. --Paul (talk) 21:49, 30 December 2009 (UTC)
thread_title:  p(x,y) = 0 ? 

message: The conditional mutual expectation is defined using the epxression
::
\mathbb E_Z \big(I(X;Y)|Z\big)
There are two problems with this expression. First, I(X;Y) is not a random variable. So taking its expectation is a no-op. Second, the expectation is conditioned on the random variable Z, which is not defined outside the scope of the expectation. This is how conditional expecatation is defined (see e.g. )
::
\mathbb E \big(X|Y)
= \sum_{x} p(x|Y) x
Indeed, this expectation is itself a random variable.
The same problem occurs in the definition of the multivariate mutual entropy.  —Preceding unsigned comment added by 128.114.60.41 (talk) 19:50, 9 March 2010 (UTC)
thread_title:  Bogus Expectations 

message: Would some knowledgeable person please add some material on the mutual information of stochastic processes? Thanx! Rinconsoleao (talk) 14:12, 22 December 2010 (UTC)
thread_title:  Stochastic processes 

message: Can anyone verify the equations in the multivariate mutual information section.
This equation
:I(X_1;\,...\,;X_n) = I(X_1;\,...\,;X_{n-1}) - I(X_1;\,...\,;X_{n-1}|X_n),
Does not reduce to any of the equations above for the basic 2-variable mutual information, if, for example, X_1=X and X_n=X_2=Y
Should the equation instead be this?
:I(X_1;\,...\,;X_n) = H(X_1;\,...\,;X_{n-1}) - H(X_1;\,...\,;X_{n-1}|X_n),
Willkeim (talk) 13:43, 8 April 2012 (UTC)
thread_title: Multivariate mutual information equations

message: The claim that the distance D(X,Y) is "universal" is pretty flimsy. Here is the text from the article "An information-based sequence distance and its application to whole mitochondrial genome phylogeny", Ming Li et al., referred to in the Wikipedia article:
:Now, consider any computable distance D. In order to exclude degenerate distances such as D(x, y) = 1/2 for all sequences x and y, we limit the number of sequences in a neighborhood of size d. Let us require for each x,
::|{y : |y| = n and D(x, y) ≤ d}| ≤ 2dn. (2) 
:Assuming equation (2), we prove the following theorem.
::THEOREM 2. For any computable distance D, there is a constant c < 2 such that, with probability 1, for all sequences x and y, d(x, y) ≤ cD(x, y).
In other words, the distance must be "computable" (which I expect is a very specific, application-dependent notion) and it must satisfy another very technical condition (2). So it is not very universal -- at least not without considerable qualification. "Qualified universal" is fairly like "adulterated pregnant" in my view. See also remarks at Talk:Variation_of_information. 129.132.211.9 (talk) 19:57, 22 November 2014 (UTC)
thread_title:  Distance is "universal" 

message: The figure giving the mutual information of various scatterings of data looks like nonsense. It gives positive values for distributions that are separable.  — Preceding unsigned comment added by 128.40.61.82 (talk) 12:24, 19 February 2016 (UTC)
replies: text: I don't see that it's nonsense. The figures represent normalized probability distributions P[x,y] on presumably the same scale, and the mutual information is calculated directly from that, no "estimators" required. I don't understand the statement " It gives positive values for distributions that are separable." or in what sense that implies a problem. I agree that an explanation would be good, particularly describing the scale, and someone should verify the numbers. PAR (talk) 17:06, 24 February 2016 (UTC)
replies: text:  It's explained via the code is included at https://en.wikipedia.org/wiki/File:Mutual_Information_Examples.svg - that it uses `mi.plugin` from the `entropy` R package, which uses a histogram-based approach to MI calculation. It's the equivalent figure to that used on the correlation page, and the distance covariance page. I don't really have any strong attachment to it. I make it to help me understand how MI worked on different datasets that I was already familiar with. naught101 (talk) 06:20, 25 February 2016 (UTC)
replies: text: Looking at the numbers, the distribution at the top center looks like a normal distribution in 2 dimensions, and that mutual information should be zero, so I think all of the numbers are suspect. I think it is a great idea, a great plot, to compare the different notions of variance and entropy, if only the numbers were right. If you would rework it to fix those numbers, it would be an excellent contribution. PAR (talk) 11:13, 25 February 2016 (UTC)
replies: text:  Probably because n=1000, which is pretty low, and histogram binning is always a bit biased. I'm probably not going to fix it, because that would require re-writing a new MI estimator in R, and I don't have the time for that. There's a non-biased python implementation of a k-nearest neighbours estimator, so it would be possible re-build the script in python around that, but again, time restraints. naught101 (talk) 04:26, 26 February 2016 (UTC)
thread_title:  Figure 

message: The quantity introduced here has been called many things in the literature, but no publication I can find has ever referred to it as "multivariate mutual information". Wikipedia should NOT be the primary source defining new terms. This is a real problem that introduces confusion to the field. 130.209.89.69 (talk) 16:21, 25 April 2016 (UTC)
thread_title:  Multivariate mutual information 

message: As written the article takes the nature of the model to be fixed. A consequence from fixing it is for increase of the mutual information to be made impossible. To increase the mutual information is, however, the aim of scientific research. This pitfall can be avoided by viewing the model as a procedure for making inferences and by optimizing these inferences information theoretically. A consequence is for the maximum possible mutual information to be created from fixed resources. Details are provided by Ronald Christensen in his "Entropy Minimax Source Book."Terry Oldberg (talk) 22:14, 10 July 2016 (UTC)
thread_title:  Creating the maximum possible mutual information 

message: The article states that the mutual information between two variables is 0 iff the two variables are independent and random, however I think there may be another case: when both variables may take only one and the same value (thus they are always the same value with 1 probability). I'm not sure if that's a reasonable edge case, but if it is, it ought to be noted.  — Preceding unsigned comment added by 192.80.95.230 (talk) 21:09, 11 January 2017 (UTC)
replies: text: The article says "I(X; Y) = 0 if and only if X and Y are independent random variables."  This is correct.  This is not the same as "... the two variables are independent and random", though "being random" is often interpreted in informal contexts to mean "being a random variable of uniform distribution" (or sometimes simply as "having an uncertain value").  These informal interpretations do not apply here.  What is meant by a "random variable" (duly linked in the article) is a quantity that has an associated distribution over its values.  The edge case described (fixed value) corresponds to a random variable in which the distribution has a form that constrains the value to a constant.  It is nevertheless correct to still call it a "random variable".  —Quondum 17:03, 6 June 2017 (UTC)
thread_title:  Zero mutual information between two variables 

message: In , I removed the following passage:
::In the special case where the number of states for both row and column variables is 2 (i, j = 1, 2), the degrees of freedom of the Pearson's chi-squared test is 1. Out of the four terms in the summation
::: \sum_{i,j } p_{ij} \log \frac{p_{ij}}{p_i p_j }
::only one is independent. It is the reason that mutual information function has an exact relationship with the correlation function  p_{X=1, Y=1}-p_{X=1}p_{Y=1} for binary sequences.
My reasons are as follows:
*The source describes a rather specific circumstance: a bit sequence, in which the mutual information between bits of the  sequence separated by a fixed offset is being considered.  This may too esoteric for the article, and may be illustrated by the amount of information that would be needed to outline the case under consideration.
*As assumption appears to be that the statistics are stationary (i.e. that the joint probability of any pair of bit positions depends only on their separation, not on their absolute position), and symmetric; these are natural assumptions in settings of interest to some, but should not be used to suggest that there is anything special about binary sequences in the general case.
*The way in which the statement was presented, it comes across as implying what is demonstrably untrue: that with an arbitrary joint probability distribution (or contingency table) of two bits, there is a direct relationship between at the stated correlation function (and others?) and the mutual information of the bits.
*I found (through calculation) that in the general setting I had to constrain the joint probability (which starts with three degrees of freedom, the only constraint being that the total probability of all four possible sequences is 1) to one degree of freedom artificially, to achieve the stated exact relationship.
—Quondum 16:40, 6 June 2017 (UTC)
thread_title:  Removed statement about exact relationship for binary data 

message: I am a novice with information theory (though not with probability) but there seems to be an uncommented glaring inconsistency of the jargon used between several  different wikipedia pages. Throughout page Kullback–Leibler_divergence , "of" is attached to the distribution that is immediately to the left of the || in the mathematical notation. And "from" is applied to the distribution that is to the right of the ||.
Whereas on pages Mutual_information#Relation_to_Kullback%E2%80%93Leibler_divergence and Information_gain_in_decision_trees this assignment of prepositions (to construct the accompanying jargon terminology) is precisely reversed. Can anyone please explain or comment (or correct if one of these pages is in error)? Is this just another instance of those in the theoretical know making a big effort to maximize the impenetrability of their subject matter to novices, or am I being too cynical? -Thanks in advance.  — Preceding unsigned comment added by 138.40.68.45 (talk) 15:47, 27 February 2019 (UTC)
thread_title:  Jargon: Inconsistent Usage of Prepositions "of" and "from" Needs Explaining (or at least acknowledging with a comment) 

message: In the section 'Relation to conditional and joint entropy' the description of the terms describes H(X) and H(Y) as 'marginal entropies' ("where H(X) and H(Y) are the marginal entropies"). But in the Venn diagram description, H(X) is described as the 'individual entropy' ("The circle on the left (red and violet) is the individual entropy H(X)"). It is confusing to someone not well versed in probability and statistics to use the two different labels 'marginal entropy' and 'individual entropy' because they won't know that the two terms refer to the same thing. I think H(X) and H(Y) should be referred to as 'marginal entropies' throughout, but a parenthetical such as '(The marginal entropy of X is the preferred name for the entropy of X, H(X), when contrasting it conditional entropies.)' should be added. If no one objects, I'm happy to add it. Nick (talk) 19:40, 25 June 2019 (UTC)
thread_title:  Marginal Entropy or Individual Entropy? 

message: It is a bit unclear why many of the equations are wrapped in an Equation box. Not only does it seem to be uncommon, it also has poor rendering in dark mode when read in one of the native apps on e.g. iPad or iPhone. Normal equations would do fine as well. Daniel Strobusch (talk) 09:39, 4 January 2021 (UTC)
thread_title:  Get Rid of Equation Boxes 

message: Article reads:
:The quantity was defined and analyzed by Claude Shannon..., although he did not call it "mutual information".
So what did Shannon call it? --Macrakis (talk) 19:11, 26 July 2021 (UTC)
thread_title: Shannon's name

message: I am thinking in particular of the section:
"In terms of PDFs for continuous distributions"
but the translation between continuous and discrete distributions
and so the effect quantization or binning data is not straight forward.
Can we add something?
Or is there a good existing article?  — Preceding unsigned comment added by W102102 (talk • contribs) 14:56, 21 May 2022 (UTC)
thread_title: Discrete v. Continuous distributions