Talk:Kullback–Leibler divergence

message: It says: A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P.
That's a great sentence, but I sort of think it might be worth saying what we mean by "excess".  We could add "By 'excess' we mean above the expected surprise when using P, since using the true distribution always minimizes expected surprise."
Or maybe that's obvious to most people.  I actually missed the word "excess" on my first reading and thought the sentence was wrong.  Maybe if we just bolded or italicized "excess" it would let people know it's an important word.  Maybe "excess expected surprise" would read better than "expected excess surprise", since "expected surprise" is a familiar phrase, being equal to entropy.  — Preceding unsigned comment added by 38.105.200.57 (talk) 21:23, 25 April 2024 (UTC)
thread_title: possible clarification in opening paragraph

message: I think it is not equivalent to turn around the P and Q distributions inside the logarithm,
and then get rid of the minus. If the measure-based definition is taken as the definition, then that should consistently 
induce all of the definitions for the special cases, such as the discrete and continuous cases. In particular, because
absolute continuity is only required to one direction, the switch is not legal. Therefore, I am being bold and changing 
the definition of the discrete KL-divergence to be consistent with the measure-based definition. --Kaba3 (talk) 22:19, 8 October 2012 (UTC)
replies: text: D_{\mathrm{KL}}(P\|Q) = \int_X \ln \frac{{\rm d}P}{{\rm d}Q} \,{\rm d}P,
if P is absolutely continuous with respect to Q (P << Q), and infinite otherwise. This is not equivalent to appending a minus, and switching to dQ/dP, 
since that would require Q << P. Fixing this makes everything consistent, which I'll now go do. --Kaba3 (talk) 23:13, 8 October 2012 (UTC)
thread_title: To minus or not to minus

message: I read the motivation, but I am not really sure what it means.  The first two sentences have nothing to do with the rest of the article.  Could someone make this clearer? —Preceding unsigned comment added by Forwardmeasure (talk • contribs) 03:38, 25 March 2007
: In fact, there is no "motivation" in there at all.  That section really should be renamed.  MisterSheik 03:54, 25 March 2007 (UTC)
thread_title:  motivation 

message: The link to f-divergence I placed in the opening section was removed and not placed anywhere else in the article. Is there a reason for this? This family of divergences is a fairly important generalisation and may lead readers who find the KL-divergence unsuitable to something more appropriate. Should I put it back in somewhere else? If not, a reason would be helpful as I can't understand the motivation to remove it completely. MDReid (talk) 11:26, 15 January 2008 (UTC)
thread_title:  f-divergence 

message: The "Motivations, properties, etc" is a hodgepodge of ideas. I actually added to it, to complement the standard "it's not a metric" boilerplate, but I don't think I made it any more confusing than it already was. The way ideas are ordered makes it almost trivial to add subtitles with minor reorganization, I just ran out of time today.  —Preceding unsigned comment added by Dnavarro (talk • contribs) 15:04, 29 April 2009 (UTC)
thread_title:  Reorganization 

message: Any good book recommendations on this topic?  Thanks  —Preceding unsigned comment added by 129.133.94.143 (talk) 22:05, 10 January 2010 (UTC)
thread_title:  Book recommendations 

message: At the very beginning of the section "Definition", D_KL(P||Q) is referred to as "the KL divergence of Q from P".
On the other hand, in the last paragraph of that same section, the wording "the divergence of P from Q" is given for D_KL(P||Q).
This is inconsistent and should be fixed. I guess, one of the places was missed when changing the wording as a result of the above discussion. As I myself am not sure which is right and which is wrong I won't change it myself. Please somebody of the contributors involved more deeply here do it!
Ernstkl (talk) 12:31, 7 January 2020 (UTC)
thread_title:  Inconsistency within the Article 

message: These two statements seem to contradict each other:
(1) The K-L divergence is only defined when P>0 and Q>0 for all values of i, and when P and Q both sum to 1.
(2) The self-information ... is the KL divergence of the probability distribution P(i) from a Kronecker delta...
They appear contradictory because a Kronecker delta puts probability zero on all events except one. So if (1) is correct, a KL divergence from a Kronecker delta should be undefined. Could someone knowledgeable please correct the page if necessary? (Or help me understand why no correction is necessary...) Thanks! Rinconsoleao (talk) 16:27, 15 August 2010 (UTC)
replies: text: The first requirement is unnecessarily strict.  It should be that Q(i)>0 for all i for which P(i)>0.  One then makes the usual information theory limit, that we can take 0 \; \log \; 0 to be zero. Jheald (talk) 17:41, 15 August 2010 (UTC)
replies: text: Thanks! I just edited the page for consistency with your comment. I will look for an appropriate page-specific reference, but if you can provide one that would be great. Rinconsoleao (talk) 12:41, 16 August 2010 (UTC)

text: Nothing is wrong.  I assume you're referring to the image captioned "Illustration of the Kullback–Leibler (KL) divergence for two normal Gaussian distributions" KL divergence is the sum of area under the curve shown.  And though parts of the curve may be negative, the SUM of the area will always be positive.  As mentioned in the article, this is gauranteed by Gibb's inequality. Kevin Baastalk 15:44, 8 September 2010 (UTC)
thread_title:  Question 

message: Any objections if I change "Kullback and Leibler themselves actually defined the divergence as"
to "Kullback and Leibler cite Jeffery's divergence as" 
(source: Kullback & Leibler p81 (1951)) Wrecktaste (talk) 17:59, 1 November 2011 (UTC)
thread_title:  Symmetrised divergence 

message: The lambda-divergence in the wikipedia article is given as an obvious generalisation of the KL. However, I struggled to find it anywhere else than in this particular document: https://camo.ici.ro/journal/vol11/v11d15.pdf which does not seem 100% worthy of full credibility (additionally that document doesn't cite anything, or prove anything when discussing this generalisation). The sentence below the definition in the wikipedia article is copied almost verbatim from that PDF too: "this signifies the gaining expectation of info about that X is obtained from P or Q, with respective probabilities p and q" (cf document, p 575). I don't find this either clear or convincing. Even though there may not in fact be an issue with using that as a divergence, I would feel more comfortable if there were more sources (ideally, more credible ones) discussing this generalisation, where it comes from and how it can be interpreted. In the mean time I'll flag it as citation needed. Thibaut Lienart (talk) 06:21, 15 May 2018 (UTC)
thread_title:  Symmetrised divergence: lambda-divergence 

message: In my opinion the mentioning of a random variable in the definition is superfluous and in some way misleading. The definition is just about two discrete  or two absolute continuous probability measures. Nijdam (talk) 17:51, 29 April 2012 (UTC)
thread_title:  Definitions 

message: I am just wondering if there are any accent marks on Kullback or Leiber. I would suspect, but do not know if there is supposed to be an umlaut over the 'u' in Kullback. Maybe there are no accent marks at all? I know this is trivial, but I would assume there are others like me who respect accent marks when possible.  — Preceding unsigned comment added by 68.228.41.185 (talk) 21:18, 5 April 2013 (UTC)
replies: text: There should be no accent marks. Solomon Kullback was American.  Jheald (talk) 08:21, 6 April 2013 (UTC)
thread_title:  Correct Accent Marks? 

message: Just a very basic question, I was confused by the equation after "This distribution has a new entropy", isn't it missing a minus?  — Preceding unsigned comment added by 130.88.91.196 (talk) 09:46, 14 May 2014 (UTC)
thread_title:  Entropy formula 

message: In the Bayesian updating section, a notation involving \cdot is used without any introduction, explanation, or link. Can someone that knows what's going on there fix that? Thanks.  — Preceding unsigned comment added by 174.101.176.129 (talk) 10:42, 11 March 2015 (UTC)
thread_title:  cdot Notation in Bayesian section 

message: A clearer description of relative entropy wrt data differencing could be written.  For absolute entropy the p(i) Log [p(i)] formula works pretty well in approximating a theoretical limit on file compression.  However I cannot see how the relative entropy formula places a theoretical limit on a patch size between two files.
I've taken two web cam images of the same scene under identical lighting conditions.  1.jpg is 64139 bytes long.  2.jpg is 64555 bytes, differing in what I assume to be image noise only.  The relative entropy formula then gives 330 bits as a limiting patch /delta size.  It seems low.  I've also created a patch using the xdelta open source utility.  It comes out as 60939 bytes.  This intuitively seems realistic.  This is several decades of magnitude different from the 330 bits.
Saying that relative entropy forms a background for determining minimum patch size causes confusion if the numbers cannot be made to support it.  Perhaps alternative language / a numerical example needs to be used?
(I'm another user:  Carlos Pinzón)
I think that the whole section is misleading because the diff rate corresponds to the conditional entropy H(Y|X) instead of the KL-divergence.
Without entering into formulas, you can see that KL-divergence does not correspond to the average length of the patch: the diff size depends strongly on the relationship (joint distribution) between the source (X) and the target (Y), but the KL-divergence is independent of the joint distribution as it only depends on the marginals P and Q.
Concretely, let the source code X be a random bit and let the target code Y be also a random bit. Then the diff length is H(Y) if they were independent, and is zero if X=Y or X=1-Y. This is inconsistent with KL-divergence because in both cases P and Q are the same, but the diff lengths are different.  — Preceding unsigned comment added by Caph1993 (talk • contribs) 00:20, 24 December 2021 (UTC)
thread_title:  Data differencing 

message: The section Kullback–Leibler_divergence#Relationship_to_available_work is completely incomprehensible and needs major reworking (or should be deleted). The jump from relatively simple mathematical definitions and statement to a physical application (using different symbols, terminology, and never explained termini like "available work") is too big to be comprehensible.  — Preceding unsigned comment added by 134.96.90.27 (talk) 08:25, 7 July 2016 (UTC)
thread_title:  Incomprehensible section needs reworking 

message: The figure caption says "Note the typical asymmetry for the Kullback–Leibler divergence is clearly visible.' I'm not sure that I see that in the current figure.  While the area to be integrated there is visually asymmetric, still for that example, D(P || Q) = D(Q || P).  Two Gaussians that differ only in their mean do not demonstrate the asymmetry. 
I'm not sure how to generate graphic examples that will show well on wiki pages, but the following is an example with two Gaussians that differ also in standard deviation, so that the D(P || Q) is interestingly different from D(Q || P):  
--Dancing figures (talk) 17:10, 6 August 2016 (UTC)
thread_title:  Asymmetry in the graphic illustration 

message: Please have a look at my non-expert attempt at revising the introduction. Ostracon (talk) 18:37, 8 May 2017 (UTC)
thread_title:  New Introduction 

message: The upper limit of the KL divergence is mentioned as 1 in the introduction. The following example gives a KL div > 1: p(1) = 0.99,
p(2) = 0.01, p(3) = 0, q(1) = 0.001, q(2) = 0.01, q(3) = 0.989. Should that be rectified?  — Preceding unsigned comment added by 169.234.233.68 (talk) 03:14, 20 February 2018 (UTC)
thread_title:  Incorrect upper limit in introduction? 

message: The Sentence in the Section Definition: 
In other words, it is the expectation of the logarithmic difference between the probabilities {\displaystyle P}P and {\displaystyle Q}Q, where the expectation is taken using the probabilities {\displaystyle P}P. 
is not clear to an unadvanced reader. Can anyone give an explanation or a citation, why this is the expectation? The regular definition of the expected value looks differenthttps://en.wikipedia.org/wiki/Expected_value#Countably_infinite_case. 
2001:16B8:2833:B900:C954:23B5:7E2C:F310 (talk) 04:10, 12 January 2020 (UTC)
thread_title:  Expectation unclear.  

message: User:Anthony Appleyard - I think this move is agreed upon by all the people in the discussion. Could you please help with making the move? Thanks. Tal Galili (talk) 08:17, 17 December 2020 (UTC)
replies: text:  User:Anthony Appleyard - there was a short discussion on moving Kullback–Leibler divergence to Relative entropy. This move was done quickly, and after the fact, a bunch of other editors suggested this move should not have taken place, and asked to move Relative entropy back to Kullback–Leibler divergence. You can see that in the discussion above we've reached a consensus for making this switch back. Tal Galili (talk) 13:44, 17 December 2020 (UTC)
thread_title: Requested move 28 November 2020 : Discussion part 2

message: The definition section has the following text, concerning the measure theoretic definition:
::More generally, if P and Q are probability measures on a measurable space {\mathcal{X}}, and P is absolutely continuous with respect to Q, then the relative entropy from Q to P is defined as
:::D_\text{KL}(P \parallel Q) = \int_{\mathcal{X}} \log\left(\frac{P(dx)}{Q(dx)}\right)\, P(dx),
::where \frac{P(dx)}{Q(dx)} is the Radon–Nikodym derivative of P with respect to Q,ie. the unique  Q-almost everywhere defined function r on \mathcal{X} such that P(dx) = r(x)Q(dx)  which exists because P is absolutely continuous with respect to Q. Also we assume the expression on the right-hand side exists.
I can't work out the meaning of that last sentence and I suspect it's superfluous. All the text before it seems to be about showing that the expression on the right-hand side of the equation does in fact exist, so stating it as an additional assumption seems a bit weird.
Or maybe I'm missing something and an additional assumption is needed beyond the one stated about absolute continuity? If that's the case, could it be reworded to make the extra assumption more explicit? If not, I suggest the extra sentence be removed.
Nathaniel Virgo (talk) 01:35, 13 January 2023 (UTC)
thread_title:  Weird sentence in definition section 

message: The article states « While it is a distance, it is not a metric, the most familiar type of distance: it is not symmetric in the two distributions (in contrast to variation of information), and does not satisfy the triangle inequality. » But this does not correspond to common mathematics. And wiki indeed contradicts the above claims: Distance#Mathematical formalization and Metric space provide exactly the same definition for "metric" and "distance", which requires symmetry and triangle inequality. -I am of course, like all the articles above, not refering to a "riemannian metric", which is a positive definite symmetric (0,2)-tensors, or equivalently a real inner product on a tangent bundle, with some differentiability requirement. Thus i recommend deleting the above comment -which is false according to common mathematics- and rather commenting that KL divergence satisfies some properties of a distance but not all. Plm203 (talk) 02:34, 30 October 2023 (UTC)
replies: text: I made edits to the article to address your point.  Please feel free to edit further or leave additional comments here.  —Quantling (talk | contribs) 13:34, 30 October 2023 (UTC)
thread_title:  Metric and distance are synonyms in mathematics, contrary to what the page states