Function 'PowBackward1' returned nan values in its 1th output

radek: <p>I am attempting to implement the following operation from this <a href="http://ceur-ws.org/Vol-2125/paper_181.pdf" rel="noopener nofollow ugc">paper</a>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/470feafc027b26575afd811158c63000f477a807.png" data-download-href="https://discuss.pytorch.org/uploads/default/470feafc027b26575afd811158c63000f477a807" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/470feafc027b26575afd811158c63000f477a807_2_690x166.png" alt="image" data-base62-sha1="a8E0gLDe6usbtTx5Vq0BhSxKuwL" width="690" height="166" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/470feafc027b26575afd811158c63000f477a807_2_690x166.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/470feafc027b26575afd811158c63000f477a807_2_1035x249.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/470feafc027b26575afd811158c63000f477a807_2_1380x332.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1402×339 74.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is what it looks like in code:</p>
<pre><code class="lang-auto">class FrontEnd(nn.Module):
    def __init__(self):
        super().__init__()
        self.bn = nn.BatchNorm1d(80, affine=False)
        self.register_parameter('alpha', torch.nn.Parameter(torch.tensor(0.5)))
        
    def forward(self, x):
        bs, im_num, ch, y_dim, x_dim = x.shape
        x = x ** torch.sigmoid(self.alpha) # &lt;----- line causing issues
        x = x.view(-1, y_dim, x_dim)
        x = self.bn(x)
        return x.view(bs, im_num, ch, y_dim, x_dim)
</code></pre>
<p>If I set the <code>alpha</code> parameter to not require grad, everything is fine. If I however make it learnable, loss turns to nan after a single iteration.</p>
<p>When running with anomaly detection, this is the message that I get:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-19-953c2962d9a9&gt; in &lt;module&gt;
     10         loss = criterion(outputs, labels)
     11         with autograd.detect_anomaly():
---&gt; 12             loss.backward()
     13     #         print(model.frontend.alpha.grad)
     14             optimizer.step()

/opt/conda/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    193                 products. Defaults to ``False``.
    194         """
--&gt; 195         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    196 
    197     def register_hook(self, hook):

/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 99         allow_unreachable=True)  # allow_unreachable flag
    100 
    101 

RuntimeError: Function 'PowBackward1' returned nan values in its 1th output.
</code></pre>
<p>If I register a backward hook on <code>frontend</code> as follows:</p>
<pre><code class="lang-auto">def printgradvals(self, grad_input, grad_output):
    print(grad_input[0].ne(grad_input[0]).any())
    print(grad_output[0].ne(grad_output[0]).any())
    print(grad_input[0].abs().mean())
    print(grad_output[0].abs().mean())
    print(grad_input[0].abs().min())
    print(grad_output[0].abs().min())

model.frontend.register_backward_hook(printgradvals)
</code></pre>
<p>I get the following output:</p>
<pre><code class="lang-auto">tensor(False, device='cuda:0')
tensor(False, device='cuda:0')
tensor(1.9707e-05, device='cuda:0')
tensor(1.9707e-05, device='cuda:0')
tensor(1.3642e-12, device='cuda:0')
tensor(1.3642e-12, device='cuda:0')
</code></pre>
<p>The gradients are very small. But why doesn’t input grad differ from output grad? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> And what could be causing the <code>nan</code>s?</p>
<p>Thank you very much for any help that you can give me on this <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
<p>EDIT: Just noticed I initialize <code>alpha</code> to 0.5 in the code that I copied - I tried with initializing it to zero, no changes. Getting same results.</p>

radek: <p>I’ve put together a small example hoping it could help find the issue, but everything works as expected:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c174ebf918e8c85ff922717bf008dd65764a1150.png" data-download-href="https://discuss.pytorch.org/uploads/default/c174ebf918e8c85ff922717bf008dd65764a1150" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c174ebf918e8c85ff922717bf008dd65764a1150_2_361x500.png" alt="image" data-base62-sha1="rBoEDRQ6TDJwoFiAZV9KFo9xfGw" width="361" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c174ebf918e8c85ff922717bf008dd65764a1150_2_361x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c174ebf918e8c85ff922717bf008dd65764a1150_2_541x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c174ebf918e8c85ff922717bf008dd65764a1150_2_722x1000.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">785×1086 88.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I created a simplified architecture, still no go:</p>
<pre><code class="lang-auto">class FrontEnd(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_parameter('alpha', torch.nn.Parameter(torch.tensor(0.)))
        
    def forward(self, x):
        x = x ** torch.sigmoid(self.alpha) # &lt;----- line causing issues
        return x

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.frontend = FrontEnd()
        self.lin = nn.Linear(508800, len(classes))
    
    def forward(self, x):
        bs, im_num, ch, y_dim, x_dim = x.shape
        x = self.frontend(x)
        return self.lin(x.view(16, -1))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f484e85e517bbf6c7a801e0f496617ba58051857.png" data-download-href="https://discuss.pytorch.org/uploads/default/f484e85e517bbf6c7a801e0f496617ba58051857" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f484e85e517bbf6c7a801e0f496617ba58051857_2_690x421.png" alt="image" data-base62-sha1="yT7fSPIOSHWHpoXKWTsB5P7zz0j" width="690" height="421" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f484e85e517bbf6c7a801e0f496617ba58051857_2_690x421.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f484e85e517bbf6c7a801e0f496617ba58051857_2_1035x631.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f484e85e517bbf6c7a801e0f496617ba58051857_2_1380x842.png 2x" data-dominant-color="F6F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1938×1184 221 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I wonder what I could try to fix this or troubleshoot this further? How can I pinpoint why <code>PowBackward1</code> returned nan values?</p>

ptrblck: <p>If you are initializing <code>self.alpha</code> as zero initially, <code>torch.sigmoid(self.alpha)</code> would have the value <code>0.5</code>.<br>
If the input <code>x</code> contains negative values, you would calculate the square root of these negative values, which would yield a NaN output:</p>
<pre><code class="lang-python">x = torch.tensor([[-1]])
print(x ** 0.5)
&gt; tensor([[nan]])
</code></pre>
<p>Did the paper mention something about a non-negative condition for the input to this layer?</p>

radek: <p>Thank you very much for your help <a class="mention" href="/u/ptrblck">@ptrblck</a>, really appreciate it.</p>
<p>All the inputs should be non-negative - added a check on line 7 to make sure that is the case:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9abd531ea4102bb5eec3725635fa5672b97549e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/9abd531ea4102bb5eec3725635fa5672b97549e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9abd531ea4102bb5eec3725635fa5672b97549e0_2_505x500.png" alt="image" data-base62-sha1="m4T9IqtLaQReLLyt5V7lWxKBTH2" width="505" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9abd531ea4102bb5eec3725635fa5672b97549e0_2_505x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9abd531ea4102bb5eec3725635fa5672b97549e0_2_757x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9abd531ea4102bb5eec3725635fa5672b97549e0_2_1010x1000.png 2x" data-dominant-color="F4F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1300×1286 81.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Even if I remove the sigmoid nonlinearity and initialize <code>alpha</code> to a value &gt; 1, the issue persists:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/312ae2cf7fe763f2ce89643217f80d2251603d5b.png" data-download-href="https://discuss.pytorch.org/uploads/default/312ae2cf7fe763f2ce89643217f80d2251603d5b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/312ae2cf7fe763f2ce89643217f80d2251603d5b_2_507x500.png" alt="image" data-base62-sha1="70XhhQNZ6orI2Xv8Vg7QkTM5KUr" width="507" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/312ae2cf7fe763f2ce89643217f80d2251603d5b_2_507x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/312ae2cf7fe763f2ce89643217f80d2251603d5b_2_760x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/312ae2cf7fe763f2ce89643217f80d2251603d5b_2_1014x1000.png 2x" data-dominant-color="F4F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1298×1280 81 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>That’s a bit weird. I cannot reproduce this issue using this code snippet:</p>
<pre><code class="lang-python">class FrontEnd(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_parameter('alpha', torch.nn.Parameter(torch.tensor(0.)))
        
    def forward(self, x):
        x = x ** torch.sigmoid(self.alpha) # &lt;----- line causing issues
        return x

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.frontend = FrontEnd()
        self.lin = nn.Linear(10, 2)
    
    def forward(self, x):
        x = self.frontend(x)
        return self.lin(x)


model = Model()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

x = torch.empty(10, 10).uniform_(0, 10)
target = torch.randint(0, 2, (10,))

for i in range(100):
    optimizer.zero_grad()
    out = model(x)
    print(torch.isfinite(out).all())
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()
</code></pre>
<p>I reduced the number of neurons a bit, but that shouldn’t make a difference.<br>
Could you run my code and check, if you are still getting NaNs?<br>
Also, do you see any difference between my and your code?</p>
<p>PS: It’s better to post code directly by wrapping it into three backticks ```, as I could simply copy-paste it <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

radek: <p>Thank you very much again for your help <a class="mention" href="/u/ptrblck">@ptrblck</a>, really, really appreciate it! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"> Very sorry regarding the images - my apologies!</p>
<p>I couldn’t reproduce the issue with synthetic data, but if I modify the example that you shared to work with my data, the issue appears.</p>
<p>I saved the data as follows (I uploaded it <a href="https://storage.googleapis.com/misc_share/data.pth" rel="nofollow noopener">here</a>, it is under 8MB, forums say I cannot upload it here):</p>
<pre><code class="lang-auto">it = iter(train_dl)
x1, t1 = next(it)
x2, t2 = next(it)

torch.save([[x1, t1], [x2, t2]], 'data.pth')
</code></pre>
<p>This is the code I run</p>
<pre><code class="lang-auto">data = torch.load('data.pth')

class FrontEnd(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_parameter('alpha', torch.nn.Parameter(torch.tensor(0.)))
        
    def forward(self, x):
        x = x ** torch.sigmoid(self.alpha) # &lt;----- line causing issues
        return x

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.frontend = FrontEnd()
        self.lin = nn.Linear(10, 264)
    
    def forward(self, x):
        x = self.frontend(x)
        x = x.mean((2,3,4))
        return self.lin(x)
    
model = Model()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.BCEWithLogitsLoss()

for x, target in data:
    optimizer.zero_grad()
    out = model(x)
    print(torch.isfinite(out).all())
    if not torch.isfinite(out).all(): break
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()
</code></pre>
<p>And this is the output I receive</p>
<pre><code class="lang-auto">tensor(True)
tensor(False)
</code></pre>
<p>The author of the paper <a href="https://github.com/f0k/birdclef2018/blob/745ee8b2b8994ad745edf0fd4e81fd727bd710fb/experiments/model.py#L476" rel="nofollow noopener">implemented this operation</a> using <code>lasagne</code> and <code>theano</code> - my data is different but I should be preprocessing it how he does in the paper.</p>

ptrblck: <p>Thanks for the data!<br>
I loaded it and executed the provided code snippet, however I’m getting all finite outputs (even with executing the training loop several times).</p>
<p>Which PyTorch version are you currently using?<br>
I tried to reproduce it with <code>1.6.0.dev20200611</code> and it’s working fine.</p>

radek: <p>Really appreciate all your help <a class="mention" href="/u/ptrblck">@ptrblck</a>! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"> I am using a GCP vm, their deep learning image. I was on pytorch version <code>1.4.0</code> and that is what was causing the issue! Upgrading to <code>1.5.1</code> ended up fixing the issue but causing slowness in dataloader, so I compiled using source from github (assuming this way I also got all the computation support libraries specific to the platform) and everything seems to be working fine now! <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>Apologies to have bothered you for something that ended up being a version issue. I wouldn’t have imagined that this could be the culprit and so it didn’t cross my mind to try a newer version (and searching didn’t turn up any results). Thanks again for all your help!!!</p>

ptrblck: <p>I’m glad it’s working and thanks for the code snippets and the debugging! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I’m surprised I haven’t seen this issue before (at least I cannot remember a similar error).</p>

xianqian: <p>Hi radek, (<a class="mention" href="/u/radek">@radek</a> )<br>
How did you find the “line causing the issue” from the anomaly detection error message “…Function: PowBackward1…”?</p>