Talk:Entropy (information theory)/Archive 1

message: The term "cross-entropy" directs to this page, yet there is no discussion of cross-entropy.
:Fixed. There is now a separate article on cross entropy. --MarkSweep 20:13, 13 Apr 2005 (UTC)
thread_title:  Cross-entropy 

message: It might be better to move this page to Shannon entropy instead of redirecting from there to this page. That way, this page can talk about other formulations of entropy, such as the Rényi entropy, and naming/linking to those pages from this one.
I agree with Vegalabs on this. --V79 19:33, 3 October 2005 (UTC)
thread_title:  Move 

message: This page claims that the term "information entropy" was invented as the majority of people don't understand what is entropy and if one used it he/she would always have the advantage in the debates. If this is true, it should be included in the article. --Eleassar777 10:41, 14 May 2005 (UTC)
thread_title: Why the term was invented

message: I remember hearing that the amount of entropy in a message is related to the amount of information it carries. In otherwords, the higher the entropy of a message, the more information it has relative to the number of bits. (For example, would this mean that in evolution, the entropy of a string of base pairs actually increases over time?) Is there any truth to this? Keep in mind I flunked Calc II three times running, so please keep it simple and use short words. crazyeddie 06:58, 19 Jun 2005 (UTC)
thread_title: Layman with a question

message: The graph that opens this article is wrong: the max entropy should be ln2 not 1.0.  Lets work it out: for p=1/2, we have 
:H = -(p log p + (1-p) * log (1-p)) = -log(1/2) = log 2
linas 04:12, 9 October 2005 (UTC)
replies: text: But the logarithm used in the definition of the entropy is based 2, not e. Therefore log 2 = 1. Brona 01:42, 10 October 2005 (UTC)
thread_title:  Graph is wrong 

message: I am confused by the definition given in the article, and believe that parts of it are wrong:
:Claude E. Shannon defines entropy in terms of a discrete random event x, with possible states 1..n as:
::H(x)=\sum_{i=1}^np(i)\log_2 \left(\frac{1}{p(i)}\right)=-\sum_{i=1}^np(i)\log_2 p(i).\,\!
:That is, the entropy of the event x is the sum, over all possible outcomes i of x, of the product of the probability of outcome i times the log of the probability of i (which is also called s's surprisal - the entropy of x is the expected value of its outcome's surprisal). We can also apply this to a general probability distribution, rather than a discrete-valued event.
First, there's a reference to "s's surprisal", but the variable "s" has not been defined. I suspect that it is supposed to be "i", but I'm not familiar enough with the material to make the change.
Second, the way I read the definition, it doesn't matter what the actual outcome is, all that matters is the collection of possible outcomes. I'm pretty sure that this is wrong. I'm probably just confused by the terminology used, but in that case, someone who understands this topic should try to rewrite it in a way that is more understandable to a layman. AdamRetchless 18:10, 21 October 2005 (UTC)
replies: text: Indeed all that matters are the outcomes and the probabilities of them. The formula above intends to define the information generated by an experiment (for instance taking a coloured ball out of a vase that contains balls with several colours) before the experiment is actually performed. So the specific outcome is unknown. But what we do know is: if the outcome is i (which happens with probability p(i) \,) then the information that we get is \log_2 \left(\frac{1}{p(i)}\right). Bob.v.R 00:16, 13 September 2006 (UTC)
thread_title:  Problem with definition