Weights turn into NaN's

jw1400: <p>Hello all,</p>
<p>running this code:</p>
<pre><code class="lang-auto">    def do_something(self, img):

        # Create mask with weak edges set to 1, else 0
        masked_img = img.clone()
        masked_img[torch.logical_not(img == 0.5)] = 0
        masked_img[img == 0.5] = 1

        # Calculate weak edges that are changed to strong edges
        changed_edges = img.clone()
        changed_edges[:] = 0
        changed_edges[((self.conv4(img) &gt; 1) &amp; (masked_img == 1))] = 1

        # Add changed edges to already good edges
        changed_edges[img == 1] = 1

        return changed_edges
</code></pre>
<p>results in:<br>
RuntimeError: Function ‘PowBackward0’ returned nan values in its 0th output.</p>
<p>And with anomaly detection set to false I can see that my kernel weights have turned to NaN’s.</p>
<p>If I delete the line “changed_edges[:] = 0” the network trains without problems. I’ve tried some different methods to cope with this problem but I am out of ideas here. Seems like the gradients become extremely big so that my network gets rekt during training? Help or some hints would be highly appreciated. Thanks.</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="jw1400" data-post="1" data-topic="151900">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jw1400/48/45317_2.png" class="avatar"> jw1400:</div>
<blockquote>
<pre><code class="lang-auto">        changed_edges[:] = 0
</code></pre>
</blockquote>
</aside>
<p>If you’re trying to fill a Tensor with zeros you can either do <code>.fill_(0)</code> or you could try, <code>changed_edges.zeros_()</code>. This <em>might</em> cause in-place operation issues for the gradient, but give it a go and see if it solves your problem. If not, could try something like, <code>changed_edges = torch.zeros_like(changed_edges)</code> to avoid in-place operations.</p>
<p>You could also try debugging your code with <code>torch.autograd.set_detect_anomaly</code>, more information about that command can be found in the documentation <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly" rel="noopener nofollow ugc">here</a></p>
<p><a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly</a></p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="2" data-topic="151900">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>You could also try debugging your code with <code>torch.autograd.set_detect_anomaly</code>, more information about that command can be found in the documentation <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly" rel="noopener nofollow ugc">here</a></p>
</blockquote>
</aside>
<p>Try running your code within a <code>torch.autograd.set_detect_anomaly</code> decorator and see where the error is. The RuntimeError states <code>PowBackward0</code> returned <code>NaN</code> in its 0th output. So look for any use of <code>**</code> or <code>.pow()</code> in your code and the gradient of that is causing the issue.</p>

MyCenturaHealth: <p>Thanks for sharing its solution</p>