Talk:Information theory/Archive 1

message: is a question well asked. -- 130.94.162.61 13:52, 19 January 2006 (UTC)
replies: text: Something got lost; I don't know how to do edit conflicts. Sorry about that. -- 130.94.162.61 14:00, 19 January 2006 (UTC)
thread_title: What's to hope for?

message: November's Monthly just arrived, with an important article by Dan Kalman: "Virtual Empirical Investigation: Concept Formation and Theory Justification".  The Monthly arrived over a month late, with a packing ticket and a returns policy.
:The theory got too deep for me, and I ended up in the looney bin.  I'll have to leave the article for the time being for others to edit while I recover my sanity.  Thank you.  -- 130.94.162.61 13:22, 15 January 2006 (UTC)
thread_title: Important Information Update

message: O.K., Jheald, logically you are right.  The material within the quotation marks should represent exactly what it says, without any extraneous punctuation.  The English language is not very logical in this respect, however.  Punctuation that terminates a clause, such as the comma, period, semicolon, exclamation mark, and question mark, actually belongs inside the quotation marks.  When a ? or ! appears just before a closing quotation mark, it is considered part of the quoted material, and no additional punctuation is necessary to terminate the clause.  Other terminal punctuation is either added or replaced if necessary, but always just inside the closing quotation mark.  Yes, this is complicated, totally illogical, and leads to slight ambiguity, but as far as I know, that's just how it is in the English language. -- 130.94.162.61 00:24, 17 December 2005 (UTC)
replies: text: That may be the unfortunate standard in some misguided parts of the world, but thankfully it's not the standard on Wikipedia :-)  -- Jheald 12:07, 17 December 2005 (UTC)
replies: text: O.K.  This is getting way off-topic here, but as long as you have a standard, of course you should follow it.  I'm sure I've mentioned before that I'm not that familiar with Wikipedia policy.  I wasn't even aware that the matter had been duly considered.  I know popular publishers all have their quirks, and the sheer volume of material they put out does set de facto standards, so maybe that's where I got the impression I did.  -- 130.94.162.64 21:41, 17 December 2005 (UTC)
thread_title: Comma inside Quotation mark

message: A wonderful introduction to information theory can be found in Jeremy Campbell's book Grammatical Man.
:Add a proper reference to it in "Other Books" if it's appropriate. -- 130.94.162.64 03:03, 16 December 2005 (UTC)
thread_title: Book recommendation 

message: O.K., I just did a major reorganization of this article and expanded it considerably.  I think it's somewhat more in line with Wiki guidelines than it was before, but I need some help from more experienced Wikipedians:  Please be bold and prettify the article, correct my capitalization, and wikify things I didn't do correctly.  I would like this article to be useful both to the layman and to the person with the requisite math knowledge (mainly probability theory) who wants to learn more about the topic.  There is still much room for improvement.  All the information that was in the article before was sound to the best of my knowledge, and i tried to include all of it in this revision.  The way it was organized, however, was an absolute mess.  Please make some specific, concrete suggestions here on how to further improve this article. -- 130.94.162.64 01:55, 2 December 2005 (UTC)
replies: text: Note:  Looks like somebody took those references out and reworded, which was actually the right thing to do under the circumstances.  Also looks like I'll either look it up myself, or (more probably) leave well enough alone.  Sorry for the confusion. -- 130.94.162.61 00:43, 15 December 2005 (UTC)

text: Nevertheless, I'm not going to dispute that classification (with all the K-junk, etc.).  It might be some kind of gov't standard or official ___information or something like that.  Must mean something to somebody. -- 130.94.162.64 01:30, 10 December 2005 (UTC)

text: Good work, though, and maybe a lot of it should stay.  You did explain things a lot more nicely than I did.  I'll try not to clobber your edits. -- 130.94.162.64 15:04, 11 December 2005 (UTC)
thread_title:  Major Revision of Article 

message: Here's a start (130.94.162.64 21:02, 2 December 2005 (UTC)
replies: text: Note:  Looks like somebody took those references out and reworded, which was actually the right thing to do under the circumstances.  Also looks like I'll either look it up myself, or (more probably) leave well enough alone.  Sorry for the confusion. -- 130.94.162.61 00:43, 15 December 2005 (UTC)

text: Nevertheless, I'm not going to dispute that classification (with all the K-junk, etc.).  It might be some kind of gov't standard or official ___information or something like that.  Must mean something to somebody. -- 130.94.162.64 01:30, 10 December 2005 (UTC)

text: Good work, though, and maybe a lot of it should stay.  You did explain things a lot more nicely than I did.  I'll try not to clobber your edits. -- 130.94.162.64 15:04, 11 December 2005 (UTC)
thread_title:  To Do 

message: This page has three different requests for attention on it.
* On the main page, User:Rvollmert put an attention sticker on it, but didn't (AFAIK) explain what specifically needed attention.
* On Wikipedia:Pages_needing_attention, User:Eequor said, "information theory is leaking information."  I don't understand what that means.
*:Why would it leak?  Because it has holes in it. Kevin Baas | talk 16:51, 2004 Sep 7 (UTC)
* On the Talk page(this page),User:Pcarbonn added a Todo list refering to Wikipedia:WikiProject Science, which, at least, has a clear set of things to do.  But what about the other two items?
If anyone knows anything about this, please comment. (or just fix it. :-) )
:I think it's really rather stubby, but I don't find anything grossly objectionable in the article.  It's only what's not in the article that is the problem. Michael Hardy 15:36, 7 Sep 2004 (UTC)
replies: text:  I agree that the article has nothing wrong in it. So I don't see why we would put the "page needing attention" alert in the face of the casual reader.  The to-do list is there to plan enhancements. That's why I removed the "Attention" tag in the article, but I'm still open to discussion... Pcarbonn 18:50, 7 Sep 2004 (UTC)
thread_title:  Multiple "need attention" requests 

message: In response to Michael Hardy: I don't think that information theory is only of interest to persons who study probability theory as you suggest.  Anyone from the general public who would like to understand what "Information theory" is about should be able to find a reasonable answer in this article.  So, yes, I believe that the introduction is not satisfactory as it is today.  It should clarify what issues information theory helps to understand, what it is used for, and everyday examples where information theory had a role (e.g. compression technologies ?).  I studied information theory a while ago, and I do not know all these answers anymore, but would like to refresh my mind...  Pcarbonn 18:50, 7 Sep 2004 (UTC)
thread_title:  What should the introduction say ?

message: The way several parts are properly linked to original sources is great, and perhaps an initiative should be made to encourage this across wikipedia.
I agree an introduction is necesary more accesisble to the general public. syats
Shannon's paper is copyrighted (Bell, i.e. now AT&T) thus should be removed from wikipedia, shouldn't it ?
--Henri de Solages 17:21, 7 November 2005 (UTC)
replies: text: As far as I can see, Shannon's paper isn't on the wikipedia; and the pdf linked at the bottom is to a Bell Labs site.  So what is it that is giving you copyright angst? Jheald 19:06, 7 November 2005 (UTC)
thread_title:  Great work on the references 

message: Correct me if I'm wrong, but I don't think Dembski uses information theory at all.  The "information" in his Complex Specified Information is just 1/probability (and even that is a kludge, since his notion of probability assumes the complete absense of any mechanisms). Unless he actually uses the Shannon notion of information somewhere, the section should either be removed or else have a clarification added. — B.Bryant 01:29, 2 October 2005 (UTC)
thread_title:  Dembski 

message: * In the very first equation mentioned, the equation determining the ammount of information in a message, the "propability of message m_i" is not defined. The article states only that p(m_i) is the "propability of message m_i" without saying how this number is arrived at. --203.91.90.238 07:29, 21 October 2005 
(UTC)
* It should be said in which sense Shannon's formula can be called a "measure".
:Shannon's formula for entropy does behave a lot like a set-theoretic measure.  It is additive for independent rvs. i.e. H(X,Y) = H(X) + H(Y).  Joint entropy behaves like the measure of a set union, equivocation behaves like the measure of a set difference, and transinformation like the measure of a set intersection.  And independent rvs behave like disjoint sets (wrt the information they carry).  But how does one nail down just what kind of "sets" one is "measuring" in information theory?  Or how can these ideas be added to the article? -- 130.94.162.64 08:04, 3 December 2005 (UTC)
replies: text: I added a section on measure theory per request.  It boils down to this: it walks like a duck, it looks like a duck, it quacks like a duck, it satisfies the postulates of duck theory.  Is it O.K. to call it a duck now? :-) 130.94.162.64 13:41, 3 December 2005 (UTC)
thread_title:  Needs Explanations 

message: Just curious on this quote:  "This has been applied to the paradox of Maxwell's demon which would need to process information to reverse thermodynamic entropy; but erasing that information, to begin again, exactly balances out the thermodynamic gain that the demon would otherwise achieve."...Does this imply that a Maxwell's demon could never erase information on the fly, passively using energy applied to the process of reading the information to erase as it goes?  From the wiki article on Maxwell's demon, it would seem that it's not the use of energy to erase that is the problem, so much as the act of erasure itself that increases entropy in the system.  Or am I missing something?
: I think that's right - at least that is what I was trying to convey, based on what I'd read elsewhere - but I confess I haven't read the paper by Bennett, so I don't know the detail of his argument; and I don't know of any reason to believe it.
: The most straightforward connection (IMO) between information entropy and thermodynamic entropy is Jaynes's point of view (see MaxEnt thermodynamics): simply that the thermodynamics entropy is the quantity of Shannon information which you don't know about a physical system; particularly, but not necessarily, the Shannon information you don't have if all you do have is a description in terms of the classic thermodynamic variables.
: Note that this entropy depends on what you know, rather than being a property of the system.
: The classic demonstration of this is Leo Szilard setting Maxwell's demon to work a Szilard engine (1929), which really ought to have a wikipedia entry of its own.  The idea is that you have a box at temperature T, with a single particle rattling around inside it.   The box has two plungers, one at each end.  If a demon can know that at an instant the particle is in the right hand side of the box, he can push in the left hand plunger without the particle hitting it - ie with no work.  Similarly, if the particle is in the left hand side, the demon can push in the right hand plunger for free.  The demon can then extract an energy (kT ln 2) from the particle isothermally pushing the plunger out again.
: The question then (supposing the demon can see through the walls of the box), is what stops the demon repeating the process again and again, essentially creating a perpetual energy pump.
: This I think is where Bennett tries to save the day by insisting the demon must spend some entropy to reset its memory.
: But I haven't mentioned any memory.  I think if the demon can see the particle through the walls of the box, it can quite happily go on extracting the energy -- because if the demon knows exactly where the particle is, it doesn't have any entropy, so the demon is free to spend the full entropy over-estimated by assuming all it knew was the classical description.
: However, as I said, I haven't read Bennett's paper, and I don't really know how his argument works.   Maybe somebody can tell us?   All best, Jheald 22:30, 3 November 2005 (UTC)
replies: text: Some of the Landauer and Bennett argument is in the reversible computing article; but maybe the page could use a sharpening.  As it stands, I'm still not 100% won over. Jheald 20:46, 4 November 2005 (UTC)
thread_title: Maxwell's Demon

message: I was looking for a article on “electrical signals” which is redirected to “Signal (information theory)”.  Is the “Signal (information theory)” subject matter related to this article and if so why is there no mention of signals in this article? Freestyle 20:48, 12 November 2005 (UTC)
replies: text: I did mention signal and noise in the article. -- 130.94.162.61 13:02, 19 January 2006 (UTC)
thread_title: "Signal (information theory)" Vs. "Information theory" articles

message: JA: I will put some time in over the next couple of weeks with the aim of constructing an "on-ramp" to the information autobahn that we have here presently.  I'll start with an intuitive conceptual framework that has informed my own thinking for lo! these many years.  It may be rough hitting the right expository level at first, so give me a few tries.  Jon Awbrey 14:54, 16 January 2006 (UTC)
replies: text: Sounds good to me.  Getting better and better.  There is more interest in the article now, so I think something good will eventually develop.  This article is getting rather long, though, and some of the more detailed sections could be moved or merged into linked articles, with a short blurb left here.  Like the template says, we don't want to remove the technical information, but make it more accessible.  Feel free to edit or move information to information entropy, joint entropy, mutual information, conditional entropy, etc., or create new articles, even one on channel capacity (communications), for example.  -- 130.94.162.61 13:15, 19 January 2006 (UTC)
thread_title: Once over quickly?

message: JA: Francis, please read the notice at the top of the talk page, and the referral to WP:WPSci.  I personally don't consider guidance like that as anything cast in stone, but it is at the very least worth considering and discussing.  The suggested structure is to put the dedicated or detailed history section toward the end.  I think that probably makes sense for this type of article, though I think it also makes sense to intersperse apt bits of historical sidelights throughout the article, as the occasion arises.  Jon Awbrey 04:20, 19 January 2006 (UTC)
replies: text: Hey, sorry if I trod on anyones toes :) - feel free to revert my edits, move it around. Perhaps a brief history could be included in the introduction? I actually came here to try and find out whether information theory could be used in evaluating machine translation. I'm still looking :) - FrancisTyers 13:42, 19 January 2006 (UTC)
thread_title: Pending tasks

message: JA: The distinction that was "catechized" on me is that the "reference section" (sometimes "endnotes") contains sources cited in the article, while the bibliography contains a more generic invitation to or exhaustion of the pertinent literature.  I was getting around to sorting the sources into those two bins, but there is only so much time.  Jon Awbrey 05:15, 19 January 2006 (UTC)
replies: text: Although that approach is a common one, it is not the only approach, and in fact, it is not the standard approach in most academic journals nor on Wikipedia.  Per WP Policy, the section entitled References is supposed to contain a list of bibliographic references, whereas the sources specifically cited within the article are listed in a separate section called Footnotes.  WP articles do not include a section called Bibliography.  See WP:CITE for more information.  -- Metacomet 05:23, 19 January 2006 (UTC)

text: Yes, they are two different types of lists, and each one is useful for a different purpose.  That is why WP policy supports both types of lists.  But the policy is quite clear as to the naming conventions for the section headers, as I mentioned above.  Moreover, I personally think that the WP policy in this case is effective in terms of providing information to the reader in a form that is easily accessible.  -- Metacomet 00:20, 20 January 2006 (UTC)

text: I don't mean to be harsh, but I think you are missing something fundamental here.  It really doesn't matter what you or I think about this issue.  WP has adopted a standard way of handling citations, and that standard is articulated as a guideline.  There is really nothing to argue about, unless you want to try to change the standard, which I would guess is pretty unlikely.  Again, if you don't believe me, please read the policy at WP:CITE.  In addition, take a look at virtually any WP article, and you will see that the terminology that virtually everyone is using is as I have described.  -- Metacomet 03:46, 20 January 2006 (UTC)
replies: text: Instead of all the non-constuctive criticism and bickering, it would be nice if someone would help edit the references section (and citations) in the article.  It is admittedly a mess, but even some help with the punctuation and italicization would be nice.  Much of the material in the article, especially the mathematical part, is (at least intended to be) fairly standard and can be found in virtually any textbook on the subject.  Such material is not usually cited, but I have listed most of the standard textbooks since this is a somewhat obscure subject.  Other material I have tried to cite more specifically, but I am not really familiar with Wikipedia standards for citations (I find them confusing) so any help with references will be appreciated.  Thank you. -- 130.94.162.64 17:45, 16 February 2006 (UTC)

text: I apologize if I was harsh or uncivil.  I came on too strong in my earlier posting.  I do, however, still maintain that the WP guideline on Citations does provide standards regarding the format and the naming of sections for references and footnotes.  I also disagree with the idea that it is okay to have varying standards from one article to the next.  -- Metacomet 02:16, 25 February 2006 (UTC)

text: One other thing:  I have moved on to other topics.  So feel free to do whatever you want.  -- Metacomet 02:16, 25 February 2006 (UTC)
thread_title: References & Bibliography

message: is a question well asked here and I believe it can be answered by a general statement of the noisy-channel coding theorem.  Thank you. -- 130.94.162.61 13:32, 19 January 2006 (UTC)
thread_title: What's to hope?

message: Why are some of the headings so nonstandard (probably not the right word)? I refer to the headings 1, 1.1, 1.2, and 2. I'm not sure what else it should be (maybe 1 should be "Introduction" 2.1 and 2.2 should be moved up one level and "Looking more closely" should be removed). Superdosh 22:27, 19 January 2006 (UTC)
replies: text:  I get that. I'm referring to the particular wording of the headings though. They don't sound quite encyclopedic? Superdosh 23:03, 20 January 2006 (UTC)

text: The concepts of self-information, conditional entropy or equivocation, joint entropy, and mutual information or transinformation are rather basic to information theory, in my opinion.  Not that they are easy or simple to understand, but the theory is based on them, so to speak.  This article is getting too long.  Don't deconstruct everything.  Please try to remember this is an encyclopedia article, not a textbook.  Nonetheless I did gloss over some even more basic concepts that you are covering now in great detail.  How about another article: Introduction to Information Theory?  That may not be quite standard for an encyclopedia article, either, I suppose.  Hmm... -- 130.94.162.61 21:05, 8 February 2006 (UTC)
thread_title: Headings...

message: * JA: Once again pursuant to the requested "attention", I added the adjective "advanced" in anticipation of having two separate sections, one for simple applications and one for more advanced applications, but we can let it ride till we get to that.  Jon Awbrey 04:04, 22 January 2006 (UTC)
replies: text: The concepts of self-information, conditional entropy or equivocation, joint entropy, and mutual information or transinformation are rather basic to information theory, in my opinion.  Not that they are easy or simple to understand, but the theory is based on them, so to speak.  This article is getting too long.  Don't deconstruct everything.  Please try to remember this is an encyclopedia article, not a textbook.  Nonetheless I did gloss over some even more basic concepts that you are covering now in great detail.  How about another article: Introduction to Information Theory?  That may not be quite standard for an encyclopedia article, either, I suppose.  Hmm... -- 130.94.162.61 21:05, 8 February 2006 (UTC)
thread_title: Advanced applications?

message: * JA: Having let it set out in the sun a while, I'll look at these sections again over the next few days and see if they can be KICAS-ified some more.  I was just trying to "comply" with the request to provide a bit of a leg-up.  Forgive me, I was so much younger and naiver then.  Insistence is futile.  Jon Awbrey 21:34, 8 February 2006 (UTC)
thread_title: Cultural horizons, motivational stuff

message: * JA: Mulling over recent discussions, along with the general impress of the various "attention" stickers/stigmata, it does begin to strike me that it might be a good idea to have a gateway, hub, introduction, roundhouse, or overview article to the effect of "Information theory (Overview)".  I just tried this out with a similar tangle of articles associated with Relational databases and it seems to be working pretty well so far.  But rather than make up a new article title for the overview, with the concomitant Uncertainty over whether to capitalize "overview" or not, and the chance that seekers would not think to look there first, it seems to be a better strategy to make the un-epitheted article, this one, the overview, leaving the introduction, motivation, and history bits here, and moving the rest to a more specific name.  So maybe we could think about this for a while.  Jon Awbrey 14:20, 10 February 2006 (UTC)
replies: text:  I agree with grubber's suggestion. Superdosh 19:55, 10 February 2006 (UTC)
thread_title: Proposal for an Information theory (overview) article

message: * JA: That's what I thought I heard from 1 contingent the 1st time.  In terms of the proposed reorg, you are saying this:
 1.  Hub (on-ramp to info autobahn) stays
 2.  Mathematical theory            >---> new article
     Channel capacity               >---> new section
     Shannon information            >---> new section
 3.  Applications                   >---> new article
 4.  History                        stays
 5.  General bibliography           stays
 6.  See also                       stays
 7.  External links                 stays
* JA: But Jheald seems to be saying something else.  (NB.  I personally will not touch the Information article any time soon — the last time I tried to help with a pop article like that, even with folks who are supposed to know what a function or a manifold is, it was sheer grief — better to tend our own garden first.)  Jon Awbrey 18:42, 15 February 2006 (UTC)
replies: text:  I really don't see Jheald saying anything substantially different. I agree with what he says. I'd say: if it's more appropriate elsewhere, trim it out.  For example, state what channel capacity is, but leave Channel capacity with the job to explain it thoroughly.  To be safe, if you cut something, make sure you link somehow to an article that has what you cut. I'll help you do all this too... Sounds like fun! - grubber 19:48, 15 February 2006 (UTC)
replies: text:  I like the extra stuff that you've added to this page, though I think they should be made into their own article so that we can trim this one up quite a bit. What would be a good name for a new article to cover this information? - grubber 19:57, 16 February 2006 (UTC)
thread_title: Should I go or should I stay

message: * JA: Do you mean the intro to the main page or something on the talk page?  I'm starting to think about absorbing some of the overview (what's it good for & what's information, etc.) into the article on Logic of information, as it fits in with a kind of "semiotic information theory" that C.S. Peirce was lecturing on in the 1860's -- it was at colonial backwaters like Harvard & Lowell, so of course nobody ever heard of it.  Jon Awbrey 20:08, 16 February 2006 (UTC)
replies: text:  Nice idea. I think that would be appropriate. - grubber 20:56, 16 February 2006 (UTC)
thread_title:  Extra stuff 

message: I've been working at shortening and condensing the main page... Take a look. It's tough to figure out what is essential -- and I'm sure that will change as we go. - grubber 01:05, 17 February 2006 (UTC)
replies: text: Looking better. -- 130.94.162.64 05:24, 17 February 2006 (UTC)

text: The three fundamental questions in the introduction deal somewhat more with coding theory than information theory per se.  Information theory is the foundation for coding theory, but it is not so easy to easy to boil it down to the magic number of three fundamental questions.  We need questions more like "What is information?" and "How does a random variable bear information?"  Someone, not so briefly, tried to deal with these issues, which are now being ignored.  Some of the information in the information entropy article perhaps belongs in the introduction here.  And the three fundamental questions fail to mention error correction or redundancy at all.  The channel capacity is a very important concept, but the questions of lossy and lossless compression could at least be combined and error correction added as one of the fundamental questions, if we wish to retain the focus on coding theory.  Also, many of the advances in information and coding theory and applications could be added to the history section of this article "Developments since 1948". -- 130.94.162.64 15:38, 17 February 2006 (UTC)
replies: text:  I'm not sure you are likely to find anything that cannot be categorized under those three statements. Lossless compression deals with what information is and how much  information something fundamentally contains; lossy compression deals with approximations and distortion-rate and rate-distortion; and channel capacity deals with all the coding issues (including error correction and redundancy - there are no errors if there is not a noisy medium, ie a channel). When I took graduate info theory, these three questions were the ones used to introduce me to the field, and as I've studied IT, those three questions seem to partition the field quite well. My aim was to give a very lay introduction to give people an idea of what IT is all about. You are right that there are many subtle ideas that are not apparent in such a terse overview. But I think that is the aim of the article, not the introduction. - grubber 18:10, 17 February 2006 (UTC)
replies: text: O.K., I'm conceding on this one.  I never had the benefit of a college course to introduce me to information theory, so I am quite lacking on how it should be introduced. -- 130.94.162.64 19:26, 18 February 2006 (UTC)
replies: text: After seeing one mistake from this author (the converse to capacity is a strict inequality), I randomly checked another alteration made by him/her, which also revealed a lack of understanding about information theory (confusing mutual information with relative entropy).  The former mistake was actually a correction to a significant mistake already in the articles on theShannon–Hartley theorem and the noisy channel coding theorem, so I would not want to discourage the author from contributing to Wikipedia.  However, I believe it would be better if the author put suggestions in the talk page for review before putting them on the article.  I don't mean to be rude or out of line, but two out of two mistakes should give some pause.  (Also, regarding the "three" question, I changed it to two with one having two subquestions, as in Elements of Information Theory.) Calbaer 21:01, 26 April 2006 (UTC)
replies: text: That second mistake looks like it has been corrected by adding the word "average" to the offending sentence in the section on mutual information in this article.  I think it is clear now (especially if one reads the linked article) what is being referred to, but perhaps it could be made a little more clear here in this article.  This article needs more major work, though, including cleanup and removal of anything (including, I am sure, some of my contributions) that is not encyclopedic and verifiable.  I've done a lot of work on this article in the past, but I'm getting out of my depth when it comes to these fine (but nonetheless important) details.  I hope Calbaer (who seems quite qualified) will continue to do some more work on this and related articles, and not be put off by my mistakes.  I know I've made a lot of mistakes, and I appreciate the corrections.  By the way, this talk page is getting too long and should be archived or something.  130.94.162.64 01:02, 17 May 2006 (UTC)
replies: text: I was probably hasty in calling it a "mistake," though I still feel it should be "corrected" (and, indeed, am about to change it after it was added back in).  Yes, mutual information and Kullback–Leibler divergence are related.  But it is not clear why "information gained" should be linked to Kullback–Leibler divergence, which is not — in my mind, let alone the mind of someone new to the subject — automatically associated with information gain; to me, the association with information gain is with mutual information.  It makes about as much sense to link to information entropy, since I(X;Y)= H(Y|X)-H(Y).  Besides, why should a word like "gained" link to a concept being explained?  (By the way, I won't be put off by mistakes.  Perhaps I will be by non-Wikipedia commitments, however....) Calbaer 01:37, 7 June 2006 (UTC)

text:  Well, although those are good partitions of the topic, maybe there's a better way to call them? - grubber 01:49, 19 February 2006 (UTC)
thread_title:  Terse-ifying 

message: :The three fundamental questions in the introduction deal somewhat more with coding theory than information theory per se.  Information theory is the foundation for coding theory, but it is not so easy to easy to boil it down to the magic number of three fundamental questions.  We need questions more like "What is information?" and "How does a random variable bear information?"  Someone, not so briefly, tried to deal with these issues, which are now being ignored.  Some of the information in the information entropy article perhaps belongs in the introduction here.  And the three fundamental questions fail to mention error correction or redundancy at all.  The channel capacity is a very important concept, but the questions of lossy and lossless compression could at least be combined and error correction added as one of the fundamental questions, if we wish to retain the focus on coding theory.  Also, many of the advances in information and coding theory and applications could be added to the history section of this article "Developments since 1948". -- 130.94.162.64 15:38, 17 February 2006 (UTC)
replies: text:  I'm not sure you are likely to find anything that cannot be categorized under those three statements. Lossless compression deals with what information is and how much  information something fundamentally contains; lossy compression deals with approximations and distortion-rate and rate-distortion; and channel capacity deals with all the coding issues (including error correction and redundancy - there are no errors if there is not a noisy medium, ie a channel). When I took graduate info theory, these three questions were the ones used to introduce me to the field, and as I've studied IT, those three questions seem to partition the field quite well. My aim was to give a very lay introduction to give people an idea of what IT is all about. You are right that there are many subtle ideas that are not apparent in such a terse overview. But I think that is the aim of the article, not the introduction. - grubber 18:10, 17 February 2006 (UTC)
replies: text: O.K., I'm conceding on this one.  I never had the benefit of a college course to introduce me to information theory, so I am quite lacking on how it should be introduced. -- 130.94.162.64 19:26, 18 February 2006 (UTC)
replies: text: After seeing one mistake from this author (the converse to capacity is a strict inequality), I randomly checked another alteration made by him/her, which also revealed a lack of understanding about information theory (confusing mutual information with relative entropy).  The former mistake was actually a correction to a significant mistake already in the articles on theShannon–Hartley theorem and the noisy channel coding theorem, so I would not want to discourage the author from contributing to Wikipedia.  However, I believe it would be better if the author put suggestions in the talk page for review before putting them on the article.  I don't mean to be rude or out of line, but two out of two mistakes should give some pause.  (Also, regarding the "three" question, I changed it to two with one having two subquestions, as in Elements of Information Theory.) Calbaer 21:01, 26 April 2006 (UTC)
replies: text: That second mistake looks like it has been corrected by adding the word "average" to the offending sentence in the section on mutual information in this article.  I think it is clear now (especially if one reads the linked article) what is being referred to, but perhaps it could be made a little more clear here in this article.  This article needs more major work, though, including cleanup and removal of anything (including, I am sure, some of my contributions) that is not encyclopedic and verifiable.  I've done a lot of work on this article in the past, but I'm getting out of my depth when it comes to these fine (but nonetheless important) details.  I hope Calbaer (who seems quite qualified) will continue to do some more work on this and related articles, and not be put off by my mistakes.  I know I've made a lot of mistakes, and I appreciate the corrections.  By the way, this talk page is getting too long and should be archived or something.  130.94.162.64 01:02, 17 May 2006 (UTC)
replies: text: I was probably hasty in calling it a "mistake," though I still feel it should be "corrected" (and, indeed, am about to change it after it was added back in).  Yes, mutual information and Kullback–Leibler divergence are related.  But it is not clear why "information gained" should be linked to Kullback–Leibler divergence, which is not — in my mind, let alone the mind of someone new to the subject — automatically associated with information gain; to me, the association with information gain is with mutual information.  It makes about as much sense to link to information entropy, since I(X;Y)= H(Y|X)-H(Y).  Besides, why should a word like "gained" link to a concept being explained?  (By the way, I won't be put off by mistakes.  Perhaps I will be by non-Wikipedia commitments, however....) Calbaer 01:37, 7 June 2006 (UTC)

text:  Well, although those are good partitions of the topic, maybe there's a better way to call them? - grubber 01:49, 19 February 2006 (UTC)
thread_title:  Not so terse, there! 

message: In the article both H_2 and H_b are used for the binary entropy function.  Some consistency would be desired here.  There is a very nice graph of this function at Information entropy, but not much explanation of the notation that should be used for this important function. 130.94.162.64 21:25, 15 May 2006 (UTC)
replies: text: Neither is correct.  In sources I've seen, H is used, as an entropy function on a single scalar can be assumed to be the binary entropy function; H_2 and H_b are bad because many information theorists would assume that these are expressions for R%C3%A9nyi entropy of order 2 or b, respectively.  If anyone has a respected yet conflicting source, please let me know before I change this. Calbaer 19:29, 30 May 2006 (UTC)
replies: text:  The H_2 notation seems to come from David J. C. MacKay's textbook.  I don't know where the H_b comes from.  But by all means do what you need to do to make things clear. 130.94.162.64 00:47, 31 May 2006 (UTC)
replies: text:  Ah, I see.  I replaced all instances with H_\mbox{b}, which doesn't make the b scriptsize in the PNGs (so it looks kind of odd), but at least can't be confused with R%C3%A9nyi entropy.  H would be better in theory, but might confuse first-time readers. Calbaer 00:23, 1 June 2006 (UTC)

text: \mathrm{Ergodic} \subset \mathrm{Stationary} \subset \mathrm{Stochastic\ processes} 
and also provides an example of a process that is stationary but not ergodic (on the same page).  This ought to be checked out more carefully. 130.94.162.64 20:29, 22 May 2006 (UTC)

text: The example in Reza's book is {X_t} = A \sin t + B \cos t, where A and B are i.i.d normal rvs.  This is stationary, but not ergodic.  On the other hand, the article Measurable function notes that "an ergodic process that is not stationary can, for example, be generated by running an ergodic Markov chain with an initial distribution other than its stationary distribution."  This is true, but any such ergodic process will  approach stationarity as the effects of the initial distribution fall off exponentially. 130.94.162.64 22:41, 25 May 2006 (UTC)
thread_title: Notational consistency & other errors in article

message: One thing that might be useful would be sort of a flow chart of mathematical concepts and tools that a person would need if they wanted to learn how to understand Shannon's theory.  That way those of us who think that it might be useful in our work, could carry out a study program that would allow us to utilize information theory.
I.e. something like...okay first you need Statistics...this/this/this concept, then you need X, then you need Y and so on.
:Information theory is mainly based on probability theory. 130.94.162.64 23:49, 9 May 2006 (UTC)
thread_title: Self Teaching

message: Here are how I interpret the pending tasks:
* Make sure intro and overview are understandable with daily life examples.  Making sure the reader  can subjectively tell what a "bit" is, for example, is a good start.  Discussion of elephant trumpets, whale songs, and black holes, do not contribute in this manner, although their mere mention (with links, say, to the no hair theorem) may be useful to the curious.
* Move applications up before mathematics (just after Overview) and make it independent.  I'm not  sure this is for the best, as Overview states some applications and, for the rest, it's useful to know some theory.  Also, we need to be careful not to conflate security and information theory, since information knowledge is necessary for, yet not sufficient for, breaking ciphers.  (For example, a Diffie-Hellman key exchange assumes that an adversary will have full knowledge, but the form of that knowledge assures that the adversary will not be able to use it within a reasonable amount of time.  Thus, although secure in practice, Diffie-Hellman key exchange is information-theoretically insecure).
* A separate section on the theory already exists (although it could use a mention of a few more concepts, e.g., the asymptotic equipartition property).
* Move history down to end (just before References).  Again, I'm not sure why this is best, and some "good" articles, like Thermodynamics, list history first.  If moved, we need to check that this is just as readable, but the reader doesn't need to know scientific history to understand scientific phenomena.  Clearly more post-1948 history is needed.  Most applications of information theory, even elementary ones, are post-1948 (Huffman codes, arithmetic codes, LDPC codes, Turbo codes), and much of the theory is too, e.g., information divergence, information inequalities, Fisher sufficient statistics, Kolmogorov complexity, network information theory, etc.  However, it might be best to move the history to its own article.  The current section is far too in-depth.
There's still a lot to do and a lot to add, and we might reasonably ask if certain topics should be shortened or omitted, e.g., Kullback–Leibler divergence (which, although important, can generally be omitted from elementary explanation), differential entropy (which is most important for transinformation, which can be interpreted as a limit of discrete transinformation), gambling (which most information theoretists I know love but is a somewhat fringe topic).  Again, the thermodynamics article is shorter and far less mathematical.  Do we want this article to be long with full mathematical explanations or medium-sized with general explanations and links to the math?  I don't know the right answer, but it might be nice to get some ideas of what people would like out of this.
A few final things to keep in mind:
*Diversions or examples absent of explanation confuse, not clarify.
*Sentences should be concise as possible, but no more so, for easy reading. 
*Proofread before (or at least immediately after) changes.
*Others might not read the same way as you.
Due to these factors, it's good to discuss significant changes on the talk page.  I find my "peer edited" sections are a lot better than my first attempts.  And too much unilateral change can result in an unstable, unreadable, and flat-out wrong article.  Calbaer 01:44, 15 June 2006 (UTC)
replies: text: I say keep the K–L divergence.  Not only is it important, but it is really useful in understanding the mutual information.  The treatment of other stuff like gambling, intelligence, and measure theory could be cut.  The measure theory section shows a useful mnemonic device, but the analogy is incomplete due to the failure of the transinformation to remain non-negative in the case of three or more random variables.  It could probably be cut out entirely.  History of information theory could indeed have its own article, but keep a short summary of history in this article here.  Also some mention of Gibbs' inequality would be nice as it shows the K–L divergence (and thus the bivariate mutual information) to be non-negative.  The AEP could be mentioned in the source theory section, since it is a property of certain sources.  Don't hesitate to add what you think should be added. --130.94.162.64 17:17, 20 June 2006 (UTC)
replies: text: K-L divergence is certainly fundamental, so it should stay.  Measure theory is also fundamental, but it is not necessary for a good understanding of information theory, so it could go.  History could be summarized and moved.  I'll get to this when I can, though if someone else wants to do so first, go ahead. Calbaer 18:36, 20 June 2006 (UTC)

text: the section that mentions measure theory should be kept, or moved somewhere else, rather than deleted. also, the following sentence in that section is misleading/incorrect: "it justifies, in a certain formal sense, the practice of calling Shannon's entropy a "measure" of information." entropy is not a measure, given an random variable f, one integrates f lnf against a suitable measure (the Lebesgue measure, the counting measure, etc) to get the entropy. Mct mht 18:52, 20 June 2006 (UTC)
replies: text:  I think that the section confuses more than it helps; most readers will not have a working knowledge of measure theory, and, as you say, the section needs at least a little reworking.  I'll delete it and add a link to Information theory and measure theory. Calbaer 19:47, 20 June 2006 (UTC)

text: Good point, and one that argues for the elimination of the following as all but links: Self-information, Kullback–Leibler divergence, differential entropy.  We shouldn't be scared to have a little math in the article, but regular, joint, and conditional entropy can be defined together and mutual entropy in terms of them.  That's enough for the idealizations of source and channel coding.  If people want to know why the math is as it is, there could be a definitions in information theory article or something like that.  Two things though: 1) "error correction" is mentioned three times in the article (and I don't think it's that important to specifically state FEC in a "starting point" article) and 2) are you volunteering to do this? Calbaer 06:13, 23 July 2006 (UTC)
thread_title: Pending tasks (what do we want?)