Dealing with NaN's in gradients

username: dpernes
cooked: <p>I am trying to train a Mixture Density Network, by reproducing the results of the toy example of section 5 of Bishop’s paper, where MDNs were initially proposed (link to the paper: <a href="https://publications.aston.ac.uk/373/1/NCRG_94_004.pdf" rel="nofollow noopener">https://publications.aston.ac.uk/373/1/NCRG_94_004.pdf</a>).</p>
<p>I am using exactly the same network architecture proposed there (single hidden layer with 20 neurons and tanh activation). The only difference is that I am using SGD instead of BFGS as my optimization algorithm.</p>
<p>Unfortunately, after 2k or 3k iterations (where the loss reduces considerably), I start getting NaN’s as the loss value. After some intense debug, I finally found out where these NaN’s initially appear: they appear due to a 0/0 in the computation of the gradient of the loss w.r.t. the means of the gaussian.</p>
<p>I am using negative log-likelihood as the loss function, L=-sum(log(p_i)). Therefore, the gradient of L w.r.t. p_i is</p>
<p>dLdp= -1/p_i</p>
<p>However, the derivative of a Gaussian function w.r.t. its mean, mu, is:</p>
<p>dpdmu = p_i * (x_i-mu_i)/(sigma_i^2)</p>
<p>so it is proportional to p_i. Therefore, when p_i is close to 0, the derivative of the loss w.r.t. the mean is:</p>
<p>dLdmu =  dLdp * dpdmu = 0/0 -&gt; NaN</p>
<p>However, this indetermination is easy to eliminate, since the expression may be algebraically simplified to dLdmu = (mu_i - x_i)/sigma_i^2. Of course, all that PyTorch does is numeric computing, so it is not able to do this simplification. How can I deal with this issue? Can I at least replace NaN’s with something else (zeros, for instance), so that they do not propagate?</p>
<p>(In all my reasoning I have assumed a MDN with one single Gaussian kernel, which is kind of stupid, but similar results roughly apply if we consider multiple kernels.)</p>
<p>Thank you in advance.</p>

username: tom
cooked: <p>Hello <a class="mention" href="/u/dpernes">@dpernes</a></p>
<p>I think it is most common to give the expression for the log likelihood in “one go” in that you do not compute dLdp.<br>
The weights of your components will sum to one, so you can compute the per-component log likelihoods and then do the usual log-sum-exp stabilisation to safely get from per-component log likelihood to the full one.</p>
<p>Best regards</p>
<p>Thomas</p>

username: dpernes
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a>. Thank you for your reply!</p>
<p>Do you mean that I should write the expression for the log of p directly, instead of computing p and then applying log() to p? That seems a good idea, I will check if that works.</p>
<p>EDIT: After thinking a bit, that does not seem to be useful when I have more than one Gaussian component. Suppose that my Gaussian mixture has two components. Then, p is given by:</p>
<p>p = a0<em>exp(b0(x-u0)^2) + a1</em>exp(b1(x-u1)^2)</p>
<p>where a0, b0, a1 and b1 are some constants and u0 and u1 are the means. If we apply log() to p, there is no obvious way to simplify the expression, since we get the logarithm of a sum…</p>

username: tom
cooked: <p>Hello,</p>
<aside class="quote no-group" data-username="dpernes" data-post="3" data-topic="5529">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dpernes/48/16250_2.png" class="avatar"> dpernes:</div>
<blockquote>
<p>If we apply log() to p, there is no obvious way to simplify the expression, since we get the logarithm of a sum…</p>
</blockquote>
</aside>
<p>That is the log-sum-exp computation I mentioned.<br>
I have implemented something that sounds similar to what your description in the <code>GaussianMixture1d</code>’s <code>forward</code> method in this<br>
<a href="https://github.com/t-vi/pytorch-tvmisc/blob/master/misc/gaussian_process_regression_basic.ipynb" rel="noopener nofollow ugc">1d Mixture Density Network notebook</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: dpernes
cooked: <p>Got it, thank you!! <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: JethroJC
cooked: <p>Hi. I met the same problem as you. Could you please show me your modified code about loss computation?Thanks!</p>

username: dpernes
cooked: <p>Hi Jethro,</p>
<p>Unfortunately, I can’t find the source code (this was for a toy example that I might have deleted).<br>
However, scipy has an (open source) implementation of the log-sum-exp operation (see below), which is easy to adapt for PyTorch.</p>
<p>Moreover, the idea behind this function is very simple, so if you understand it you might not even need to look at the scipy code. Let me explain.</p>
<p>Log-sum-exp, like its name says, computes the logarithm of a sum of exponentials:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/39db4520d0d04bd03bc8398904cde311372f5265.png" alt="image" data-base62-sha1="8fOZYOI61juNAiFSeTnt7DLy901" width="169" height="55"></p>
<p>If the values inside the exponentials are all large negative numbers, the sum inside the log vanishes and we get the logarithm of 0, so we’re in trouble. To avoid this, the following simple trick is used:</p>
<ol>
<li>
<p>Find the maximum value among the a_i’s:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/7/7f4237ab1707d5c26aaf2bc60bae10e23832de4b.png" alt="image" data-base62-sha1="i9MvyygvCiC023X9JPWCar4rBG3" width="90" height="25"></p>
</li>
<li>
<p>Compute b using:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f81637e8b4a2e75e4956cef512dc827a02b828d1.png" alt="image" data-base62-sha1="zoG0X3eqSWAGhAZG0wCYJtUg9JD" width="248" height="55"></p>
</li>
</ol>
<p>Now, at least one of the values inside the exponentials will be 0, and exp(0) = 1, so we will certainly not get log(0). Thus, we now have a numerically robust implementation of the log-sum-exp operation.</p>
<p>Hope this explanation helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/scipy/scipy/blob/v0.19.1/scipy/special/_logsumexp.py#L8-L127">
  <header class="source">

      <a href="https://github.com/scipy/scipy/blob/v0.19.1/scipy/special/_logsumexp.py#L8-L127" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/scipy/scipy/blob/v0.19.1/scipy/special/_logsumexp.py#L8-L127" target="_blank" rel="noopener nofollow ugc">scipy/scipy/blob/v0.19.1/scipy/special/_logsumexp.py#L8-L127</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="8" style="counter-reset: li-counter 7 ;">
          <li>def logsumexp(a, axis=None, b=None, keepdims=False, return_sign=False):</li>
          <li>    """Compute the log of the sum of exponentials of input elements.</li>
          <li></li>
          <li>    Parameters</li>
          <li>    ----------</li>
          <li>    a : array_like</li>
          <li>        Input array.</li>
          <li>    axis : None or int or tuple of ints, optional</li>
          <li>        Axis or axes over which the sum is taken. By default `axis` is None,</li>
          <li>        and all elements are summed.</li>
          <li></li>
          <li>        .. versionadded:: 0.11.0</li>
          <li>    keepdims : bool, optional</li>
          <li>        If this is set to True, the axes which are reduced are left in the</li>
          <li>        result as dimensions with size one. With this option, the result</li>
          <li>        will broadcast correctly against the original array.</li>
          <li></li>
          <li>        .. versionadded:: 0.15.0</li>
          <li>    b : array-like, optional</li>
          <li>        Scaling factor for exp(`a`) must be of the same shape as `a` or</li>
      </ol>
    </code></pre>


  This file has been truncated. <a href="https://github.com/scipy/scipy/blob/v0.19.1/scipy/special/_logsumexp.py#L8-L127" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: JethroJC
cooked: <p>Thank you very much, your answer has given me a lot of help.</p>

username: dpernes
cooked: <p>UPDATE: At least from version 0.4.1, PyTorch has a built-in log-sum-exp function. See link below.</p>
<p><a href="https://pytorch.org/docs/stable/torch.html#torch.logsumexp" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.logsumexp</a></p>

username: whoab
cooked: <p>I am still getting nans for some reason, trying to train mixture of gaussians</p>

username: whoab
cooked: <p>Like <a class="mention" href="/u/dpernes">@dpernes</a> , the issue is with the sum of exponentials, in my case wrt the <strong>categorical variable</strong>. Applying his trick works. Also, see “get_mixture_coef” here: <a href="https://blog.otoro.net/2015/11/24/mixture-density-networks-with-tensorflow/" class="inline-onebox" rel="noopener nofollow ugc">Mixture Density Networks with TensorFlow | 大トロ</a></p>

username: whoab
cooked: <p>Still not fixed actually. Check this repo, the categorical logits are nan’ing out:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://pastebin.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://pastebin.com/KwUMs3eN" target="_blank" rel="noopener nofollow ugc">Pastebin</a>
  </header>
  <article class="onebox-body">
    <img src="https://pastebin.com/i/facebook.png" class="thumbnail" width="" height="">

<h3><a href="https://pastebin.com/KwUMs3eN" target="_blank" rel="noopener nofollow ugc">import torchfrom matplotlib import pyplot as pltimport numpy as npNSAM -...</a></h3>

<p>Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Okay, I fixed it by applying a tanh before the final linear layer. No more nans, the tanh chokes the magnitude of the final layer…</p>