NAN - PyTorch code diverging with Runtime Error

username: snsmssss
cooked: <p>Background</p>
<p>-I took a working Python code for sample neural net( analtically derive gradient, Victor Zhou Blog)<br>
-I converted it into PyTorch with autograd<br>
-My aim is to explain NN through Pytorch &amp; GPU/CUDA - to beginners<br>
-I have hard coded weights to make the compute deterministic<br>
-CPU version<br>
-Python VERSION: 3.8.3<br>
-pyTorch VERSION: 1.5.1</p>
<p>Problem<br>
-Though Python code converges, pytorch code terminates with Error &amp; NAN<br>
-Am I making any basic mistakes?( I am new to PyTorch)</p>
<p>-since pure Python version works fine, why torch code explodes with NAN?<br>
after running for 80 epochs</p>
<p>PyTorch code given below:<br>
</p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/snsmssss/0369052c6daf7a001660e51ba9da9e92" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/snsmssss/0369052c6daf7a001660e51ba9da9e92" target="_blank" rel="nofollow noopener">https://gist.github.com/snsmssss/0369052c6daf7a001660e51ba9da9e92</a></h4>
<h5>gistfile1.txt</h5>
<pre><code class="Text">import torch
def sigmoid(x):
  # Sigmoid activation function: f(x) = 1 / (1 + e^(-x))
  sx =   1.0 / (1.0 + torch.exp(-x))
  return sx 

# y_true and y_pred are numpy arrays of the same length.
def mse_loss(y_true, y_pred):
  return torch.mean((y_true - y_pred) ** 2)
</code></pre>
This file has been truncated. <a href="https://gist.github.com/snsmssss/0369052c6daf7a001660e51ba9da9e92" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>OUTPUT<br>
</p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/snsmssss/0e6f4e49de6af9d5ca2a8cb7bf2b0260" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/snsmssss/0e6f4e49de6af9d5ca2a8cb7bf2b0260" target="_blank" rel="nofollow noopener">https://gist.github.com/snsmssss/0e6f4e49de6af9d5ca2a8cb7bf2b0260</a></h4>
<h5>Output with Error message</h5>
<pre><code class="">Epoch 0 loss: 0.334
Epoch 10 loss: 0.305
Epoch 20 loss: 0.118
Epoch 30 loss: 0.072
Epoch 40 loss: 0.049
Epoch 50 loss: 0.047
Epoch 60 loss: 0.046
Epoch 70 loss: 0.037
Epoch 80 loss: 0.024
---------------------------------------------------------------------------</code></pre>
This file has been truncated. <a href="https://gist.github.com/snsmssss/0e6f4e49de6af9d5ca2a8cb7bf2b0260" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Kushaj
cooked: <p>Maybe normalizing the inputs can help.</p>

username: snsmssss
cooked: <p>I did that as well, still … problem persisits</p>

username: Kushaj
cooked: <pre><code class="lang-auto">data = (data - data.mean())/data.std()
</code></pre>
<p>Loss is converging</p>
<pre><code class="lang-auto">Epoch 0 loss: 0.326
Epoch 10 loss: 0.309
Epoch 20 loss: 0.263
Epoch 30 loss: 0.200
Epoch 40 loss: 0.166
Epoch 50 loss: 0.161
Epoch 60 loss: 0.145
Epoch 70 loss: 0.102
Epoch 80 loss: 0.051
Epoch 90 loss: 0.020
Epoch 100 loss: 0.010
Epoch 110 loss: 0.009
Epoch 120 loss: 0.009
Epoch 130 loss: 0.009
Epoch 140 loss: 0.009
Epoch 150 loss: 0.007
Epoch 160 loss: 0.006
Epoch 170 loss: 0.004
Epoch 180 loss: 0.003
Epoch 190 loss: 0.002
Epoch 200 loss: 0.001
Epoch 210 loss: 0.001
Epoch 220 loss: 0.001
Epoch 230 loss: 0.000
Epoch 240 loss: 0.000
Epoch 250 loss: 0.000
Epoch 260 loss: 0.000
Epoch 270 loss: 0.000
Epoch 280 loss: 0.000
Epoch 290 loss: 0.000
Epoch 300 loss: 0.000
Epoch 310 loss: 0.000
Epoch 320 loss: 0.000
Epoch 330 loss: 0.000
Epoch 340 loss: 0.000
Epoch 350 loss: 0.000
</code></pre>

username: snsmssss
cooked: <p>Kushaj: Thank you ! What did you find &amp; change.</p>
<p>I found one issue with the way I had defined in my code</p>
<p>all_y_trues = torch.tensor([<br>
1.,<br>
0.,<br>
0.,<br>
1.])<br>
I changed it to</p>
<p>all_y_trues = torch.tensor([<br>
[1.],<br>
[0.],<br>
[0.],<br>
[1.]])</p>
<p>it doesn’t explode now in the midway of epochs iteration</p>

username: Kushaj
cooked: <p>I only normalized the input data by adding this line <code>data = (data - data.mean())/data.std()</code>.</p>