Nan grads after few training steps

username: ricky123
cooked: <p>So, I am trying to design a model for unsupervised object detection. As of now, its task is to locate the digits in a multi-MNIST digit dataset and draw a bounding box around them. I am using a variational-autoencoder based architecture and it is a modification over <a href="https://research.fb.com/wp-content/uploads/2018/12/Spatially-Invariant-Unsupervised-Object-Detection-with-Convolutional-Neural-Networks.pdf" rel="nofollow noopener">this</a> paper.</p>
<p>Now I am facing a lot of problems in implementation. When I am running the code, after few batches, the weights are becoming Nan. Therefore I checked all the gradients of all the parameters and found that after a few steps the KL-divergence of the Z_pres variable is becoming Nan and moreover, the standard deviation of the gradient of the bias of glimpse_decoder and z_pres encoder are becoming Nan just after the first training batch. My questions are:</p>
<ol>
<li>
<p>Under what condition can these incidents happen? Does it have to do anything with network architecture? Can awry network lead to Nan gradients?</p>
</li>
<li>
<p>What does affine_grid and grid_sample exactly do? I read the documentation but could not exactly understand their purpose. I used them to take a glimpse from the image based on z_where. So, is its purpose similar to cropping a picture? Someone please explain the arguments of F.affine_grid, what do they actually expect in the translation arguments? Can this cause Nan ever?</p>
</li>
<li>
<p>Please take a look at my code, this single error has caused me lot of wastage of time. I will be grateful if someone figures out what is wrong in this.<br>
<a href="https://github.com/DEBADRIBASAK/Fun-Projects/blob/master/spair.py" rel="nofollow noopener">i)spair.py</a><br>
<a href="https://github.com/DEBADRIBASAK/Fun-Projects/blob/master/train_mnist.py" rel="nofollow noopener">ii)train_mnist.py</a></p>
</li>
</ol>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: MartinBagge
cooked: <p>Have you given exploding gradients a thought?<br>
I would think it is the most likely scenario for your nan gradients.<br>
You can try to reduce the learning rate and check whether that helps.</p>

username: G.M
cooked: <ol>
<li>Division by zero or reaching infinity somewhere(not limited to these cases) can cause Nan.</li>
<li>I don’t think these two can cause Nan. These 2 functions are generally used Spatial Transformer Networks. <code>affine_grid</code> apply linear transformation like rotating, translating, scaling(I think sheering is also part of the transformation but I’m not sure) to the input image.  <code>grid_sample</code> uses the coordinates generated by <code>affine_grid</code> to perform a bi-linear interpolation on the original image to form a result image. <a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html</a>
</li>
<li>Ur code is quite long <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:">  so I only took a glance. My suggestion is to check the part where u used <code>.exp()</code>, I had expierenced Nan using expontenial operations tho it may not be the case for u. If that’s the case, then the Nan in gradients can also be explained. U can also use <code>tc.any(tc.isnan(INPUT))</code> to see where did Nan show up.</li>
</ol>
<p>Hope this helps</p>

username: ricky123
cooked: <p>Thank you so much for your reply. Can you explain what does the argument of <code>F.affine_grid</code> represent? Like, it is a b<em>2</em>3 matrix where in that 2*3 matrices, it contains some affine transformation arguments. Can you explain which element is for scaling and which is for translation?</p>

username: G.M
cooked: <p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.2.8/images/icons/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://arxiv.org/abs/1506.02025" target="_blank" rel="nofollow noopener">arXiv.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://arxiv.org/abs/1506.02025" target="_blank" rel="nofollow noopener">Spatial Transformer Networks</a></h3>

<p>Convolutional Neural Networks define an exceptionally powerful class of
models, but are still limited by the lack of ability to be spatially invariant
to the input data in a computationally and parameter efficient manner. In this
work we introduce a...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I didn’t read the paper carefully so I might be wrong. For the 2*3 matrix, 1 value is scaling, 1 for x-translation, 1 for y-translation, 1 for rotating, 2 for sheering. I’m not sure about the ordering tho, u should take a look at the paper.</p>