How to fix when pytorch randomizing/sampling keeps yielding nan?

Martin_Nguyen: <p>Greetings! I am using Bayesian Neural Network and pytorch randomizing/sampling keeps yielding nan in some epoch. Does anyone know how to fix this? I have tried:</p>
<p>Trial 1: <code>torch.empty(120).uniform_(-.01, 0.01)</code><br>
Trial 2: <code>torch.randn(120).uniform_(-.01, 0.01)</code><br>
Trial 3: <code>torch.distributions.Uniform(-.01, 0.01).sample((1, 120))[0]</code><br>
Trial 4: <code>torch.torch.distributions.Uniform(-.01, 0.01).sample((1, 120))[0]</code></p>
<p>I also even tried like this for all the above trials <code>torch.nan_to_num(torch.torch.distributions.Uniform(-.01, 0.01).sample((1, out_features))[0], nan=torch.tensor(0.0037))</code>, but this is not gonna work in the training loop. In particular, it only took into account the part <code>torch.torch.distributions.Uniform(-.01, 0.01).sample((1, out_features))[0]</code> and still yielded <code>nan</code>.</p>
<p>The results I got with nan might not be reproducible with the examples of trials I gave here. Please let me know if you need to know deeper to reproduce the results with nan.</p>
<p>This is the result in some epoch I tried with <code>torch.torch.distributions.Uniform(-.01, 0.01).sample((1, 120))[0]</code> and yielded <code>nan</code></p>
<pre><code class="lang-auto">tensor([-1.4659,  0.3137,  0.1822,  0.1657,  0.2099, -0.1929,  0.2386,  0.4739,
 0.2335,  0.2515,  0.2949,  0.2874,  0.1460, -2.9367,  0.6829,  0.2574,
 0.4183,  0.3023,  0.4242, -1.0063,  0.3041,     nan,  0.2051,  0.1333,
 0.3006,  0.2397,  0.2347,  0.1516,  0.3593,  0.2641,  0.1562, -0.5621,
 -0.2820,  0.3090,  0.1144,  0.5003,  0.3033,  0.1153,  0.3229,  0.4727,
 0.0221,  0.2196,  0.3357,  0.1594,  0.3025, -0.3293,  0.0314,  0.1627,
 0.0971, -0.7528,  0.3589,  0.2749,  0.3002,  0.1839, -0.5211, -0.5340,
 0.2862,  0.3999,  0.1999,  0.0734,  0.3181,  0.1325, -0.6105,  0.2866,
 0.2635,  0.2449,  0.1772,  0.2520, -4.4859,  0.4437,  0.2694,  0.1909,
 0.4727,  0.2047,  0.3235,  0.4658, -1.2267,  0.1527,  0.0942,  0.5141,
 0.6169,  0.2492,  0.0831,  0.0781,  0.0855,  0.1781,  0.1907,  0.5520,
 0.3903,  0.1470,  0.3136,  0.3869, -7.8619,  0.1830,  0.1720,  0.3044,
 0.1460,  0.3055,  0.1487, -0.0105, -2.5014,  0.0465, -0.2051,  0.2039,
 -4.9786,  0.3331,  0.3496, -0.0930,  0.2421, -3.2148,  0.2964,  0.2464,
 0.3046,  0.2449,  0.2755,  0.3433,  0.3893,  0.3142,  0.0477,  0.1953])
</code></pre>
<p>The versions of packages relating to pytorch are here:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a679cdf2aa7ca749d997c509a9dc221568bfa1ad.png" alt="Pytorch asking" data-base62-sha1="nKIdWTkCj6o5ahSYpYsG3FgKGZf" width="528" height="99"></p>
<p>It has already taken me nearly a month to find solutions for this but still not successful! So I guess I should ask for help here. Please help me with this. Thank you very much.</p>

albanD: <p>Hi,</p>
<p>A quick clarifying question: is it the sampling function itself that yields nan or your network or your loss function?<br>
I would suggest to first figure out which op exactly is generating such nan by adding checks within the network.</p>

Martin_Nguyen: <p>Thank you <a class="mention" href="/u/alband">@albanD</a>. I did trace back to the source, it happened before I fed the data into forward step. The process is like this:</p>
<ul>
<li>The sampling line of code (one of 4 trials I listed) was used to have random tensors.</li>
<li>These random tensors were then fed into forward step (for weight and bias).</li>
</ul>
<p>Simply speaking: Sampling tensors for weight and bias using one of 4 trials I listed (<code>nan</code> appeared here) →  some process → F.linear(x, w, b)</p>

albanD: <p>This is quite surprising, running <code>torch.empty(120).uniform_(-.01, 0.01)</code> again and again locally never generates <code>nan</code>.</p>

Martin_Nguyen: <p>Thanks <a class="mention" href="/u/alband">@albanD</a>. Yea unfortunately, it cannot be reproduced easily like that. I have been trying to see if I can reproduce it. Otherwise, I will keep changing “set_seed” until there is a number of “set_seed” that does not yield out any <code>nan</code>.</p>