Repository: pytorch/pytorch

Issue:
kldivloss doc issues
- not clearly stated that `y` should not be log probabilities
- `l(x,y)` is defined both as `L` and as `mean(L)`
- not defined that `l_n`, `y_n` and `x_n` are each tensors, associated with a single data example

(I basically had to search through discuss.pytorch.org for examples, in order to use it). Actually, in fact, what I ended up doing was write my own kl-divergence implementation, and comparing the output with that of kldivloss, until the kldivloss one matched the self-coded one :P


Discussion:

Resolving patch:
diff --git a/torch/_tensor_docs.py b/torch/_tensor_docs.py
--- a/torch/_tensor_docs.py
+++ b/torch/_tensor_docs.py
@@ -1684,8 +1684,8 @@ def callable(a, b) -> number
 This is the reverse operation of the manner described in :meth:`~Tensor.gather`.
 
 :attr:`self`, :attr:`index` and :attr:`src` should have same number of
-dimensions. It is also required that `index->size[d] <= src->size[d]` for all
-dimension `d`, and that `index->size[d] <= real->size[d]` for all dimensions
+dimensions. It is also required that `index.size(d) <= src.size(d)` for all
+dimensions `d`, and that `index.size(d) <= self.size(d)` for all dimensions
 `d != dim`.
 
 Moreover, as for :meth:`~Tensor.gather`, the values of :attr:`index` must be
diff --git a/torch/nn/modules/loss.py b/torch/nn/modules/loss.py
--- a/torch/nn/modules/loss.py
+++ b/torch/nn/modules/loss.py
@@ -262,20 +262,22 @@ class KLDivLoss(_Loss):
     and is often useful when performing direct regression over the space of
     (discretely sampled) continuous output distributions.
 
-    As with `NLLLoss`, the `input` given is expected to contain
-    *log-probabilities*, however unlike `ClassNLLLoss`, `input` is not
+    As with :class:`~torch.nn.NLLLoss`, the `input` given is expected to contain
+    *log-probabilities*. However, unlike :class:`~torch.nn.NLLLoss`, `input` is not
     restricted to a 2D Tensor, because the criterion is applied element-wise.
+    The targets are given as *probabilities* (i.e. without taking the logarithm).
 
     This criterion expects a `target` `Tensor` of the same size as the
     `input` `Tensor`.
 
-    The loss can be described as:
+    The unreduced (i.e. with :attr:`reduce` set to ``False``) loss can be described as:
 
     .. math::
-        \ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad
-        l_n = y_n \odot \left( \log y_n - x_n \right),
+        l(x,y) = L := \{ l_1,\dots,l_N \}, \quad
+        l_n = y_n \cdot \left( \log y_n - x_n \right),
 
-    where :math:`N` is the batch size. If reduce is ``True``, then:
+    where the index :math:`N` spans all dimensions of ``input`` and :math:`L` has the same
+    shape as ``input``. If :attr:`reduce` is ``True`` (the default), then:
 
     .. math::
         \ell(x, y) = \begin{cases}
@@ -309,7 +311,7 @@ class KLDivLoss(_Loss):
 
 
     Args:
-        size_average (bool, optional: By default, the losses are averaged
+        size_average (bool, optional): By default, the losses are averaged
             for each minibatch over observations **as well as** over
             dimensions. However, if ``False`` the losses are instead summed.
         reduce (bool, optional): By default, the losses are averaged
@@ -321,8 +323,8 @@ class KLDivLoss(_Loss):
         - input: :math:`(N, *)` where `*` means, any number of additional
           dimensions
         - target: :math:`(N, *)`, same shape as the input
-        - output: scalar. If `reduce` is ``True``, then :math:`(N, *)`,
-            same shape as the input
+        - output: scalar by default. If `reduce` is ``False``, then :math:`(N, *)`,
+            the same shape as the input
 
     """
     def __init__(self, size_average=True, reduce=True):
@@ -525,7 +527,7 @@ class HingeEmbeddingLoss(_Loss):
     dissimilar, e.g. using the L1 pairwise distance as `x`, and is typically
     used for learning nonlinear embeddings or semi-supervised learning::
 
-    The loss function for :math:`n`-th sample in the mini-batch is:
+    The loss function for :math:`n`-th sample in the mini-batch is
 
     .. math::
         l_n = \begin{cases}


Test patch: