Talk:Entropy (information theory)

message: I would like to remove the statement in the introduction that states that entropy is only defined for messages with an independent and identically distributed alphabet. This is not true, it is defined for any message, each letter of which may have different probabilities, and for which there may be correlations among the letters. For example, it is stated that in the English language, the pair "qu" occurs very frequently compared to other pairs beginning with "q". In other words, for the letter "q", the letter following it is highly correlated to it, not independent. Also, in English, the characters are not identically distributed, "e" is more probable than "x". For a message which is N letters long, with an alphabet of m letters, there are m^N possible messages, each with their own probability pi, with no restrictions on these probabilities other than that they are non-negative and their sum is equal to unity. The entropy of this set of messages is the sum of -pi log(pi over all m^N possible messages. All sorts of different letter frequencies and correlations may result from this set of m^N probabilities. PAR (talk) 11:50, 30 April 2011 (UTC)
replies: text: It is a long time since I thought about this topic, but I think that text in the lead is talking about the best possible system (the one which conveys most information with each "character" that is transmitted). It looks out of place, or perhaps poorly expressed. Johnuniq (talk) 03:56, 1 May 2011 (UTC)
thread_title:  Independent and identically distributed? 

message: If entropy is considered an equilibrium property as in energy physics, then it conflicts with the conservation of information. But the second law of thermodynamics may simply be a apparent effect of the conservation of information, that is, entropy is really the amount of information it takes to describe a system, each reaction creates new information but information cannot be destroyed. That means the second law of thermodynamics is not an independent law of physics at all, but just an apparent effect of the fact that information can be created but not destroyed. The arrow of time is thus not about destruction, but about the continuous creation of information. This explains how the same laws of physics can cause self-organization. Organized systems are not anyhow less chaotic than non-organized systems at all, and the spill heat life produces can, in an information-physical sense, be considered beings eliminated by evolution rather than a step towards equilibrium. It is possible that overload of information will cause the arrow of time to extract vacuum energy into usefulness rather than heat death.
217.28.207.226 (talk) 10:51, 23 August 2011 (UTC)
replies: text: This sounds interesting, but the details are not explained clearly. For example, information can be destroyed--we do it whenever we delete a file. And there is no reason given that "The arrow of time is...about the continuous creation of information." There is no basis to edit the article until these claims are explained more clearly (and with references). David Spector (talk) 11:31, 17 February 2012 (UTC)
thread_title:  Entropy as an apparent effect of conservation of information. 

message: The characterization section can probably be rewritten by a professional. If I am correct "A Characterization of Entropy in Terms of Information Loss" by Baez et al. only require functoriality, convex linearity, and continuity for the definition of Shannon's entropy. The current statements are mutually overlapping and probably not entirely true.
Andy (talk) 14:10, 2 July 2013 (UTC)
thread_title:  Characterization 

message: Two vaguenesses in the Definition section.
1. "When taken from a finite sample, the entropy can explicitly be written as...": here it sounds like the author means that the entropy is taken from a finite sample, which is certainly not the case.
2. The term n_i in the expanded form for H(X) is never defined.  Scorwin (talk) 17:03, 3 October 2013 (UTC)
thread_title:  Definition 

message: If you know about estimating entropy from a finite sample that is randomly drawn with replacement according to a probability distribution on a (possibly infinite) population, would you please add it to the article?  For example, drawing a sample of size 1 gives p=1 for the drawn value and p=0 for all other values, yielding an estimated entropy of 0 regardless of the actual probability distribution on the population.  More generally, the expected value of the entropy computed from a finite sample will be less than the actual entropy of the population.  And so on.  Leegrc (talk) 15:43, 9 March 2015 (UTC)
thread_title:  Estimating entropy from a sample 

message: I removed a paragraph in the lead which explained entropy as an amount of randomness. Indeed, entropy is greater for distributions which are more "random." For example, a uniform distribution has the maximum entropy over all possible discrete distributions. Gaussian distributions take that role for continuous distributions. However, when it comes to messages having the alphabet of say, the English alphabet, then it is not clear whether "aaaaaaaa" has more entropy than the message "alphabet." If it is assumed that all letters of the English language are equally likely then, in fact, these two messages are equally likely and thus have the same entropy. However, in reality, "a"'s are probably more likely to occur than other letters. In which case "aaaaaaaa" would have less entropy. However, this can be terribly misleading to relay to people who may not know the subject well to give them this general rule-of-thumb based off of perceived randomness. BourkeM Converse! 05:50, 17 October 2015 (UTC)
thread_title:  Lead text 

message: quote	question
...For instance, in case of a fair coin toss, heads provides  bit of information, which is approximately 0.693 nats or 0.631 trits. Because of additivity,  tosses provide  bits of information, which is approximately  nats or  trits.
Now, suppose we have a distribution where event  can happen with probability . Suppose we have sampled it  times and outcome  was, accordingly, seen  times. The total amount of information we have received is 
:\sum_i {n_i \mathrm{I}(p_i)} = \sum {N p_i \log(1/p_i)}. 
The average amount of information that we receive with every event is therefore
:\sum_i {p_i \log {1\over p_i}}.	Explain me somebody please why single event has entropy = 1 bit, but according last formula average entropy = 0,5 bit ?
95.132.143.157 (talk) 11:13, 11 November 2015 (UTC)
replies: text: Why summation is used for single toss?95.132.143.157 (talk) 04:18, 15 November 2015 (UTC)

text: "I(p) is monotonic – increases and decreases in the probability of an event produces increases and decreases in information, respectively."
shouldn't that be "increases and decreases in the probability of an event produces decreases and increases in information, respectively." since less probable events convey more information? In other words, I(p) is monotonic, but isn't it monotonic decreasing, rather than monotonic increasing, as the article implies?207.165.235.61 (talk) 16:46, 14 September 2016 (UTC)
thread_title: Rationale

message: The glorification of the field of information entropy as conceptually distinct and more fundamental, compared to statistical mechanics and thermostatistics, is quite oblivious of the pre-existing vastness and sophistication of this basic field (statistical mechanics) from whence the Shannon paper and subsequent efforts are merely offshoots.  The statements under this "comparison" are rife with patently false statements about what "entropy" was limited to previously in statistical mechanics (which, contrary to this silly paragraph, was defined by probability distributions, non-observably small fluctuations, and other such concepts that fundamentally define entropy!!).  "Information entropy" is a small conceptual contribution to a vast, existing body of development on entropy theory that is falsely distinguished from by this paragraph.  The author of this piece should be entirely better educated on the subject, rather than making this display of ignorance and presuming it corresponds to the pre-existing field.2602:306:CF87:A200:6D19:A65A:57A0:F49D (talk) 18:44, 7 January 2016 (UTC)
replies: text: If your general objections are valid, it's difficult to make corrections without a specific example or examples of what you object to. Can you state specifically the text that you object to? PAR (talk) 01:01, 8 January 2016 (UTC)
thread_title:  Posturing and Silliness 

message: Substituted at 14:34, 29 April 2016 (UTC)
thread_title: Assessment comment

message: Shouldn't the intro sentence read "the average **rate** of information produced", or something like that, rather than "the average **amount** of information produced"? I'm new to this, but I think information entropy is basically the expected self-information of each event (or outcome? measurement?). So it seems that it should be described as either the expected *amount per event/outcome/measurement*, or as the "rate" of information produced by the stochastic source. 203.63.183.112 (talk) 05:39, 25 January 2018 (UTC)
replies: text: Done. Loraof (talk) 16:08, 19 June 2018 (UTC)
thread_title:  (intro sentence) "average amount" vs "average rate" 

message: The statement:
"I(p) is monotonically decreasing in p – an increase in the probability of an event decreases the information from an observed event, and vice versa"
is wrong. This is obviously wrong, otherwise you could not have a maximum.
The correct statement is that I(p) is a continuous function. 
If there is agreement I will fix this to: "I(p) is a continuous function of p"
--Skater00 (talk) 14:08, 10 April 2018 (UTC)
replies: text: But reference 7, p. 18, says “I(p) is monotonic and continuous in p”. Loraof (talk) 16:07, 19 June 2018 (UTC)

text: It’s the entropy H(p), not the information I(p), that is non-monotonic in p, as shown in the graph in the Examples section. Loraof (talk) 18:28, 19 June 2018 (UTC)
thread_title:  Important Mistake 

message: The introduction reads 
"The measure of information entropy associated with each possible data value is the negative logarithm of the probability mass function for the value:"
But then displays a formula showing the *average* information content, i.e. the weighted sum of each possible data value.
Either the formula should be updated to
h(x=a_i) = - log(p(x=a_i))
OR the statement should be updated to 
"The measure of information entropy associated with a random variable X is a weighted sum of the negative logarithm of the probability mass function p(x):"
Maitchison (talk) 06:15, 11 September 2019 (UTC)
thread_title:  Introduction formula inconsistant 

message: OK, so the page on Shannon information redirects to this article. But someone who searches for "Shannon information" can have no idea why they have been served instead this article on entropy. And recourse to searching for "Shannon information" within this article can only render such readers even more confused. After all, the first of the string's two occurrences informs readers that
:"the thermodynamic entropy [emphasis added] is interpreted as being proportional to the amount of further Shannon information needed to define the detailed microscopic state of the system,"
but that makes it sound as though the thing that this article is describing—entropy—is being explained in terms of the very thing that our poor baffled readers had been wanting to learn about in the first place. This experience comes awfully close to circularity.
Anyway, to paraphrase Thomas Jefferson in the Declaration of Independence, when a redirect is created, a decent respect to the opinions of mankind requires that the redirector should declare the relationship which impels the redirection (See WP:R#ASTONISH).
And that's not even the worst of it, because the term at the heart of this as-yet-unexplained redirection may not even be well defined. To quote one source, "the concept of Shannon information is still a focus of much debate."
I would happily clean up all this mess in this article if only I had the necessary expertise. Would someone who's qualified please take the task on?—PaulTanenbaum (talk) 20:02, 7 November 2019 (UTC)
thread_title:  Relationship to Shannon information? 

message: It seems like the website directed by the "A java applet representing Shannon's Experiment to Calculate the Entropy of English" no longer has a working applet. Am I missing something? If not, how can this be fixed?
Eulslick (talk) 22:02, 26 April 2020 (UTC)
thread_title:  Entropy game in external links is defunct? 

message: There is a nice question here: What does entropy capture that variance does not?
The link doesn't provide a good answer, but I thought that the comparison of entropy with variance could be a good section to add to this article.
For example (from here): If you have a bimodal distribution with two peaks and allow the spacing between them to vary, the standard deviation would increase as the distance between the peaks increases. However, the entropy doesn't care about where the peaks are, so the entropy would be the same.
Another insight is that variance is easiest to interpret when we have a near normal distribution (in the sense of how much of data is X SD jumps from the mean). Whereas with entropy the interpretation is that if it goes up in one data set vs another it means that the one with the higher entropy is "closer" to a uniform distribution.
I'm sure there are references for these statements.
What do others think - would adding such a section to this article make sense?
Tal Galili (talk) 19:55, 21 May 2020 (UTC)
thread_title:  Using entropy vs using variance for measuring uncertainty in the results of a random sample 

message: I'm not sure what to make of this, but the efficiency section seems off: "A source alphabet with non-uniform distribution will have less entropy than if those symbols had uniform distribution (i.e. the "optimized alphabet")." Isn't that the point of entropy? Also, dividing by log(n) is a normalization by alphabet SIZE (since 0 <= H(x) <= log(n)), not anything about its distribution. Could somebody more knowledgeable than me have a look at this?  — Preceding unsigned comment added by 2001:16B8:2E50:3200:DDEA:47B9:21D0:B732 (talk) 08:51, 5 August 2020 (UTC)
thread_title:  Efficiency 

message: @Bilorv
In version https://en.wikipedia.org/w/index.php?title=Entropy_(information_theory)&oldid=968833507 you added "In practice, compression algorithms deliberately include some judicious redundancy in the form of checksums to protect against errors." I don't think that is correct, and none of the mentioned algorithms says anything about checksums (by searching for "checksum" in the equivalent wiki article). If that is really correct I think it deserves some references or just mention some examples of that. For compression algorithms there is no point in adding checksums.
I suggest to delete that sentence
--Jarl (talk) 06:06, 3 September 2020 (UTC)
replies: text: Hi . I didn't add the sentence—I just moved it from another part of the article when doing some reorganisation. I'm only familiar with the maths of information theory and not any of the details of compression algorithms commonly used today so I cannot verify whether the statement is true or false (either is plausible to me). As the statement is unsourced, you are free to remove it if you believe it should be removed. — Bilorv (talk) 07:07, 3 September 2020 (UTC)
thread_title:  Checksums is not part of coding algorithms 

message: "The Shannon entropy satisfies the following properties, for some of which it is useful to interpret entropy as the amount of information learned (or uncertainty eliminated) by revealing the value of a random variable X" is incorrect, but can be fixed by adding the word "expected": The Shannon entropy satisfies the following properties, for some of which it is useful to interpret entropy as the expected amount of information learned (or uncertainty eliminated) by revealing the value of a random variable X.
For example, consider the joint distribution of X and Y, where X has two values A and B with probabilities .99 and .01 respectively. Suppose that when X = A there are 2 equally likely values for Y, and when X = B there are 32 equally likely values for Y. Then H(X, Y) = 1.12 bits. If X is revealed to be A, the uncertainty decreases to 1 bit, but if X is revealed to be B, the uncertainty increases to 5 bits. Thus the expected value of the remaining uncertainty, aka H(Y|X), is 1.04 bits, so the expected amount of information learned is 1.12 - 1.04 = 0.08, which equals H(X). 
It might be counterintuitive that the uncertainty can increase when X is revealed, but I think this is quite common in real life. p(X = A) = 0.99 implies that we've lived through case A many times before, and we have a good understanding of what can happen. When X = B happens, we're not familiar with this situation, and we don't know what to expect. 2600:1700:6EC0:35CF:E8F8:B2A6:C652:5C13 (talk) 00:24, 3 April 2022 (UTC)
replies: text: An interesting point—if I have understood correctly, it is that uncertainty can increase conditional on particular r.v. values (in some cases, \Eta (Y|X=x)>\Eta (X,Y)), but the average uncertainty will never increase based on revealing a value of an r.v. (in all cases, \Eta(X,Y)=\Eta(X)+\Eta(Y|X) and each term is non-negative).I have added the word "expected" where you suggest. In future, you can boldly make such changes yourself, and use the talk page if a justification will not fit in the edit summary. — Bilorv (talk) 11:14, 4 April 2022 (UTC)
thread_title:  Error in first sentence of "Further properties" section 

message: It is the opposite of the second law of thermodynamics: while enthropy can only stay the same or increase, the information enthropy of a system tends to decrease. (source: scitechdaily).  — Preceding unsigned comment added by 151.18.144.86 (talk) 16:37, 7 September 2022 (UTC)
thread_title: Second law of information dynamics

message: The usual notation for measures of Entropy is a capital "S", would be more appropriate than the "X"es, wouldn't it? 2601:140:9182:4E50:6436:D263:C72E:89DA (talk) 01:28, 19 January 2023 (UTC)
replies: text: H is the usual symbol for Information Entropy. The X used in this article is for the random variable that the entropy is being calculated for. S seems to be sued in the section on Thermodynamic Entropy, as you suggest. David Malone (talk) 08:37, 19 January 2023 (UTC)
replies: text: X is typical for a random variable, and it's what I saw in a mathematical treatment of information theory; I'm not aware of where S might be preferred but I think X is fine. (Slight pedantry—it's actually a capital eta, \Eta, not the letter 'H'.) — Bilorv (talk) 11:41, 22 January 2023 (UTC)
thread_title:  "S" preferable to "X"? 

message: I think the proof in the second point of the section "Further properties" is incorrect. -log(x) (negative log) is a convex function so Jensen's inequality should have the opposite sign. This can be checked with a very simple example, e.g. p_0=3/4 and p_1=1/4. Moreover, note that the right-hand side of the first inequality is the Renyi-2 entropy and the left-hand side is the Shannon entropy and indeed the latter must be greater or equal than the former, not lesser or equal as stated in the article. Finally, I believe H(p) < log(n) is proved in a different way in the reference given (Cover and Thomas).  If what I am saying is correct, I suggest replacing this line with a proper proof, e.g., from the Cover and Thomas.  Qlodo (talk) 09:11, 20 September 2023 (UTC)
replies: text: checking your example I agree the text looks wrong. I've rephrased to omit a proof (and be slightly clearer about what n is). Are you able to add a correct proof, from Cover and Thomas or another text you can access? When you find a mistake on Wikipedia the etiquette is to fix it yourself. — Bilorv (talk) 21:14, 24 September 2023 (UTC)
thread_title:  Mistake in “Further properties”?