Talk:Entropy in thermodynamics and information theory

message: This article was created from content copied from the information theory article.  I believe this topic justifies its own article for the following reasons (130.94.162.64 02:29, 5 December 2005 (UTC)
thread_title: Entropy in thermodynamics and information theory

message: Does anyone want to write an article on this one?  I know we don't have an article on every Principle that comes along, (nor do we want to), but it looks like some researchers went to a lot of trouble to refute it.  I'm not sure how important it is.  Maybe someone can just explain it a little more in this article. -- 130.94.162.64 03:24, 5 December 2005 (UTC)
replies: text: On second thought, I do think some more explanation of Zeilinger's principle is in order.  I seem to recall it was popular a number of years ago. -- 130.94.162.61 18:55, 6 December 2005 (UTC)
thread_title: Zeilinger's Principle

message: I expanded the text on this subject in the Information theory article, then spotted this page.
So I've copied it all across to here.   Some editing may be needed to fit it into the evolving structure of the page here; it would then probably also be worth cutting down the treatment in the Information theory.
But I'm calling it a night just for now.  -- Jheald 00:01, 10 December 2005 (UTC)
replies: text: I had to make a slight correction.  We cannot speak of a "joint entropy" of two distributions that are not jointly observable.  The "joint distribution" formed by considering them as statistically independent random variables is a completely artificial (not to mention rather misleading) construction.  (It completely fails to take into account which variable we observe first!)  Maybe someone who knows more about this will expand on it.  All we can really include in the article is information from verifiable sources, such as Hirschman's paper.  If you have Original Research on this topic, you will have to write a paper on it so that we can refer to it here in the article.  But you might think obut the Difficulty of Research before you do this. :) -- 130.94.162.64 15:20, 20 December 2005 (UTC)
replies: text: We need an Expert on the subject to help with this article. -- 130.94.162.64 00:21, 19 March 2006 (UTC)
thread_title:  Updated, re-editing needed 

message: :H[f] = -\int_{-\infty}^{\infty} f(x) \log f(x)\, dx,\quad 
I see no difficulties in this representation as long as f(x) is a probability density function, p. d. f.. It is the mean value of information of f(x) and its exponential represents a volume of the region covered by a uniform disgtribution (in analogy to the cardinality in the discrete case). Why would it not represent the logarithm of volume covered by any p. d. f.? A Gaussian p. d. f. covers a volume equal to 
sqrt( 2 pi e variance )
Kjells 09:06, 29 March 2007 (UTC)
thread_title:  the coninuous case 

message: I think there need to be some explanition on the matter of units for the continuous case. 
:H[f] = -\int_{-\infty}^{\infty} f(x) \log_2 f(x)\, dx,\quad 
f(x) will have the unit 1/x. Unless x is dimmensionless the unit of entropy will inclue the log of a unit which is weird. This is a strong reason why it is more useful for the continuous case to use the relative entropy of a distribution, where the general form is the Kullback-Leibler divergence from the distribution to a reference measure m(x). It could be pointed out that a useful special case of the relative entropy is:
:H_{relative}[f] = -\int_{x_{min}}^{x_{max}} f(x) \log_2 (f(x)(x_{max}-x_{min}))\, dx,\quad 
which should corresponds to a rectangular distribution of m(x) between xmin and xmax. It is the entropy of a general bounded signal, and it gives the entropy in bits.
Petkr 13:38, 6 October 2007 (UTC)
thread_title:  Units in the continuous case

message: Paragraph cited below is very confusing and should be removed. Not only does it fail to clarify anything about the topic of the article it actually makes it much harder to understand by invoking a whole lot of concepts from the area of evolutionary algorithms. For readers with no knowledge from that area this paragraph will be completely incomprehensible. If an example is needed it should be as simple as possible.
:Average information may be maximized using Gaussian adaptation - one of the evolutionary algorithms - keeping the mean fitness - i. e. the probability of becoming a parent to new individuals in the population - constant (and without the need for any knowledge about average information as a criterion function). This is illustrated by the figure below, showing Gaussian adaptation climbing a mountain crest in a phenotypic landscape. The lines in the figure are part of a contour line enclosing a region of acceptability in the landscape. At the start the cluster of red points represents a very homogeneous population with small variances in the phenotypes. Evidently, even small environmental changes in the landscape, may cause the process to become extinct.
Enemyunknown (talk) 03:48, 17 December 2008 (UTC)
thread_title:  Paragraph about evolutionary algorithms should be removed 

message: has proposed that the content of the article be moved into History of thermodynamics.
I would oppose this merger, because:
*  This is a distinct topic, of current interest and current disagreement, important in a full current understanding of thermodynamics, and well worthy of a full discussion at the length presented here.  (See also the points made at the top of the page by the original anon who created this article).
*  A one-paragraph WP summary style mention would be appropriate in the history article; anything more would be excessive in that context.  But a one-paragraph overview, directing the interested reader here for the full story, would be good both for there and for here.  Jheald (talk) 09:04, 7 July 2009 (UTC)
replies: text: Upon reflection, I must agree. I will withdraw the proposal. FilipeS (talk) 11:24, 7 July 2009 (UTC)
thread_title:  Proposed merger 

message: I like this article, but the sentence
"This article explores what links there are between the two concepts..."
confuses me.
Isn't an exploration of links between two concepts just original research?  — Preceding unsigned comment added by 80.42.63.247 (talk) 10:21, 13 September 2013 (UTC)
thread_title:  Original research? 

message: This section needs work to clarify the disparity between what I think was Brillouin's initial over-broad belief (expressed in his 1953 book "Science and Information Theory") that any operation one bit of information had a thermodynamic cost of kT ln 2 (where k is Boltzmann's constant), and our current understanding (due largely to Rolf Landauer) that some data operations are thermodynamically reversible in principle, while others have a thermodynamic cost. This understanding was already implicit in Szilard's 1929 analysis of his one-molecule engine (nicely explained at the end of the preceding section) where he showed that the cost is associated not with any one step, but with the whole cycle of the engine's operation, comprising the acquisition, exploitation, and resetting of one bit of information about the molecule.CharlesHBennett (talk) 20:12, 16 February 2014 (UTC)
thread_title:  Negentropy 

message: From Shannon's booklet: "Thus when one meets the concept of entropy in communication theory, he has a right to be rather excited -  a right to suspect that he has hold of something that may turn out to be basic and important. ... for unless I am quite mistaken, it is an important aspect of the more general significance of this theory."  
H = - K*sum( pi*log2(pi) ) which corresponds directly to the classical general form of entropy S = - kB*sum( pi*ln(pi) ).  With K=1 this H is Shannon information in units of bits (the base 2).  S has units of kB=Joules/temperature. But temperature is a measure of the average kinetic energy per molecule, Joules/molecule, so the Joules in kB can cancel.  This means S has a more fundamental unit of "molecules", a count very much like "bits".  To precisely make the count in terms of molecules, the molecules would be counted in terms of the average kinetic energy. A slower-than-average molecule would get a count a little less than 1, a faster molecule would get > 1. The result would be converted to base 2 to make the count in bits. So I'm making a bit-wise count of physical entropy.  I don't know how to get absolute entropy, but it should work for entropy changes, i.e. ΔS=ΔH*ln(2), where H is counted before and after as described, and ln(2) is just a log base conversion. This is the converse of what Szilard said in 1929, 1 bit = kB*ln(2). But my method of counting bits obviates the need for kB and shows the ln(2) to be a log base conversion factor instead of being explained as 2 required states. Ywaz (talk) 21:49, 21 February 2015 (UTC)
thread_title:  Isn't there a more explicit deep connection? 

message: About negative entropy Erwin Schrödinger gave opinions in his book "What is Life" from 1944, recent edition 2004 Cambridge University Press pages 72 and 73, derived from work of Boltzmann and Gibbs. It begins with a familiar residual from the third law of thermodynamics derived from statistical mechanics.
   entropy = k * log D 
defined for natural logarithm base e, Boltzmann constant k and, disorder of the system D.
Then Schrödinger defined order of the system as (1/D) and gave a calculation for negative entropy.
 -(entropy) = k * log (1/D)
In modern terms the first D has become Ω the number of possible disordered states that can be randomly filled or emptied of energy. The order of the system (1/D) can be expressed as N the number of states that are prevented from being randomly filled or emptied of energy. Then net entropy is given.
   S = k * Ln (Ω/N)
Examples of physical objects that obey this rule are electronic diodes producing dark currrent from heat or vibration, and parabolic reflectors focusing radiant energy to a hot spot. Radiant energy is always present because of the microwave background in space
When the number of non random states is larger than the number of random states, locally negative entropy can occur like in the examples which are not in violation of the second law of thermodynamics because of the non random construction. The examples can be easily verified, although the power generated is small in both cases. Random and non random states can also be represented as degrees of freedom in a set of variables.
Unlike the claims of Szilard, the method of Schrödinger does not require intelligent intervention, although the information is essential in both cases. Not just any information is sufficient to change entropy. The information must be linked to a counting method of non random action on the system.
Astrojed (talk) 21:51, 28 June 2015 (UTC)
thread_title: About negative entropy

message: Its very confusing to figure out what Landauer said that Brilliuon did not already say, or that Szilard said even earlier.  So we have this sentence: In 1953, Brillouin derived a general equation[4] stating that the changing of an information bit value requires at least kT ln(2) energy. Leon Brillouin (1953), "The negentropy principle of information", J. Applied Physics 24, 1152-1163. which suggests that the claim that Landauer said it first is incorrect. Perhaps Landauer indpependelty obtained this result, as stated it in a way that engineers could undrstand, in an engineering journal? Certainly, very few comp-sci researchers would ever read Briliuon... or is it something else? 67.198.37.16 (talk) 21:18, 23 September 2015 (UTC)
thread_title:  Landauer, Szillard, Brilliuon 

message: Yes, Boltzmann's constant is expressed in units of energy per unit temperature, and the choice of temperature scale affects its particular value. As with any universal constant, various scales of measurement can be chosen such that its value is unity.  For a light beam, Δx=cΔt, but setting c=1 does not reveal that space and time are identical (Introducing relativity concepts here obscures the point, so perhaps its not the best analogy). Given that we are dealing with thermodynamics, thermodynamic entropy and information entropy are perfect measures of each other, but they are not identical, just as, given that we are dealing with a light beam, Δx and Δt are perfect measures of each other, but they are not identical. PAR (talk) 22:08, 26 March 2016 (UTC)
thread_title:  Boltzmann's constant 

message: The Szilard Engine is perpetual motion, and the linked Arxiv article claims that it has actually been built. This is obviously a fraud, and calls into question the entire theory.  — Preceding unsigned comment added by 107.77.194.190 (talk) 03:21, 3 October 2016 (UTC)
replies: text: No, it is not a perpetual motion machine.  It takes energy to reset at every stage.  See: https://www.scientificamerican.com/article/the-fundamental-physical-limits-of-computation/ 173.75.1.12 (talk) 02:26, 10 February 2017 (UTC)
thread_title:  Pseudoscience 

message: I can't put my finger on what's wrong with it. As a mathematical abstraction (which every physical model is) there is no material difference between information theoretic entropy and stat-mech entropy. Information theory doesn't care what constant you scale entropy by: That's why the distinction between nats and bits as units of entropy isn't important in information theory. As far as dimension theory goes, dimensions (like that of Boltzmann's constant) are just a heuristic to check if your equations are put together properly. You can't make a physical argument from it. I'm very close to deleting this section. --Svennik (talk) 17:49, 21 December 2021 (UTC)
replies: text: Deleting now. Feel free to undo and explain. --Svennik (talk) 20:58, 21 December 2021 (UTC)
replies: text: Good. I agree. Finally. kbrose (talk) 02:11, 22 December 2021 (UTC)
thread_title:  "Criticism" section looks like utter nonsense 

message: In information theory entropy is a property of a random variable, where such random variable holds a randomly chosen message, such as the most recent message to arrive on the channel.  Messages themselves have information, not entropy. The article does not say what it is that has the property of entropy. 88.127.1.197 (talk) 21:49, 13 December 2023 (UTC)
replies: text: I think it's the entropy of the microstate – which is unknown – given the macrostate – which is known. In a Bayesian framework, ignorance can be modelled using a probability distribution, which gives you the random variable you asked for. This answers your question. I don't know how easy it is to make this completely rigorous. --Svennik (talk) 15:55, 19 December 2023 (UTC)
thread_title:  Entropy of What?