Nan in backward pass for torch.square()

username: Alan_Wang
cooked: <p>When using <code>detect_anomoly</code>, I’m getting an nan in the backward pass of a squaring function. This confuses me because both the square and its derivative should not give nans at any point. I’ve checked that the nan arises in the backward pass and not the forward pass. Am I missing something here?</p>
<p>Here’s the full error:<br>
The forward pass call where the error occurs is</p>
<pre><code class="lang-auto">y_2 = torch.pow(diff, 2) 
</code></pre>
<p>and the specific error is</p>
<pre><code class="lang-auto">RuntimeError: Function 'PowBackward0' returned nan values in its 0th output.
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Does <code>diff</code> contain some pathological values?<br>
Does the gradient flowing back does? You can print it by doing:</p>
<pre><code class="lang-auto">y_2 = torch.pow(diff, 2) 
y_2.register_hook(print)  # This will print the gradient when it's computed.
</code></pre>

username: Alan_Wang
cooked: <p>Hi albanD, thanks for the response. Here’s the full code, I’m doing a simple total variation function that can be found here: <a href="https://www.wikiwand.com/en/Total_variation_denoising" rel="nofollow noopener">https://www.wikiwand.com/en/Total_variation_denoising</a>. I’ve checked for nans in the forward pass as well as using a hook for the backward pass, and for some reason there are no nans in either pass.</p>
<pre><code class="lang-auto">def get_tv(x, phase):                                                                                                                                                                                              
    findnan = lambda s, x: print(s, (x != x).any().item())

    x = x.float()
    befor = x[:, 0, :, :-1]
    after = x[:, 0, :, 1:] 
    diff = befor-after
    x_1 = torch.pow(diff, 2)
    findnan('x_1', x_1)
    x_1.register_hook(lambda grad: print('x_1 back', (grad != grad).any().item()))

    befor = x[:, 1, :, :-1]
    after = x[:, 1, :, 1:] 
    diff = befor-after
    x_2 = torch.pow(diff, 2)
    findnan('x_2', x_2)
    x_2.register_hook(lambda grad: print('x_2 back', (grad != grad).any().item()))

    x_sqrt = torch.sqrt(x_1 + x_2)
    tv_x = torch.sum(x_sqrt)

    befor = x[:, 0, :-1, :]
    after = x[:, 0, 1:, :]
    diff = befor-after
    y_1 = torch.pow(diff, 2)
    findnan('y_1', y_1)
    y_1.register_hook(lambda grad: print('y_1 back', (grad != grad).any().item()))

    befor = x[:, 1, :-1, :]
    after = x[:, 1, 1:, :]
    diff = befor-after
    y_2 = torch.pow(diff, 2)
    findnan('y_2', y_2)
    y_2.register_hook(lambda grad: print('y_2 back', (grad != grad).any().item()))

    y_sqrt = torch.sqrt(y_1 + y_2)
    tv_y = torch.sum(y_sqrt)
    return tv_x + tv_y

</code></pre>
<p>And here’s the full error output:</p>
<pre><code class="lang-auto">x_1 False
x_2 False
y_1 False
y_2 False
y_2 back False
/opt/conda/conda-bld/pytorch_1570711283072/work/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:
  File "run.py", line 84, in &lt;module&gt;
    train.train(xdata, gt_data, vars(args))
  ...
  File "...", line 119, in unsup_loss
    tv = get_tv(x_hat, phase)
  File "...", line 51, in get_tv
    y_2 = torch.pow(diff, 2)

Traceback (most recent call last):
  File "run.py", line 84, in &lt;module&gt;
    train.train(xdata, gt_data, vars(args))
  ...
  File "...", line 118, in trainer
    loss.backward()
  File "/nfs01/shared_software/anaconda3/envs/aw847-py37/lib/python3.7/site-packages/torch/tensor.py", line 150, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/nfs01/shared_software/anaconda3/envs/aw847-py37/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'PowBackward0' returned nan values in its 0th output.

</code></pre>

username: albanD
cooked: <p>Can you add a hook on the input of this <code>pow()</code> and see what the gradient actually is when you de-active anomaly mode?</p>

username: Alan_Wang
cooked: <p>I added some more hooks and prints to the function, which I pasted at the bottom. It seems that setting <code>set_detect_anomaly</code> on prevents any nan detection (all the outputs are False). Is this expected?</p>
<p>With <code>set_detect_anomaly(False)</code>, the output is</p>
<pre><code class="lang-auto">x_1 False
x_2 False
y_1 False
y_2 False
tv_y back False
y_sqrt back False
y_2 back False
diff_4 back True
y_1 back False
diff_3 back True
x_2 back False
diff_2 back False
x_1 back False
diff_1 back False

Epoch 211:
x_1 True
x_2 True
y_1 True
y_2 True
tv_y back False
y_sqrt back False
y_2 back True
diff_4 back True
y_1 back True
diff_3 back True
x_2 back True
diff_2 back True
x_1 back True
diff_1 back True
</code></pre>
<p>With <code>set_detect_anomaly(True)</code>, the output is</p>
<pre><code class="lang-auto">x_1 False
x_2 False
y_1 False
y_2 False
tv_y back False
y_sqrt back False
y_2 back False
diff_4 back False
y_1 back False
diff_3 back False
x_2 back False
diff_2 back False
x_1 back False
diff_1 back False

Epoch 211:
x_1 False
x_2 False
y_1 False
y_2 False
tv_y back False
y_sqrt back False
y_2 back False
/opt/conda/conda-bld/pytorch_1570711283072/work/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:
  File "run.py", line 84, in &lt;module&gt;
    train.train(xdata, gt_data, vars(args))
  ...
  File "...", line 119, in unsup_loss
    tv = get_tv(x_hat, phase)
  File "...", line 51, in get_tv
    y_2 = torch.pow(diff, 2)

Traceback (most recent call last):
  File "run.py", line 84, in &lt;module&gt;
    train.train(xdata, gt_data, vars(args))
  ...
  File "...", line 118, in trainer
    loss.backward()
  File "/nfs01/shared_software/anaconda3/envs/aw847-py37/lib/python3.7/site-packages/torch/tensor.py", line 150, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/nfs01/shared_software/anaconda3/envs/aw847-py37/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'PowBackward0' returned nan values in its 0th output.

</code></pre>
<p>For my problem, it seems that the error is when calculating the diff. Why is a subtraction operator giving an nan?</p>
<p>Full function:</p>
<pre><code class="lang-auto">def get_tv(x, phase):
    findnan = lambda s, x: print(s, (x != x).any().item())

    x = x.float()
    befor = x[:, 0, :, :-1]
    after = x[:, 0, :, 1:]
    diff = befor-after
    if phase == 'train':                                                                                                                                                                                                                                                        
        diff.register_hook(lambda grad: print('diff_1 back', (grad != grad).any().item()))
    x_1 = torch.pow(diff, 2)
    findnan('x_1', x_1)
    x_1.register_hook(lambda grad: print('x_1 back', (grad != grad).any().item()))

    befor = x[:, 1, :, :-1]
    after = x[:, 1, :, 1:]
    diff = befor-after
    diff.register_hook(lambda grad: print('diff_2 back', (grad != grad).any().item()))
    x_2 = torch.pow(diff, 2)
    findnan('x_2', x_2)
    x_2.register_hook(lambda grad: print('x_2 back', (grad != grad).any().item()))

    x_sqrt = torch.sqrt(x_1 + x_2)
    tv_x = torch.sum(x_sqrt)

    befor = x[:, 0, :-1, :]
    after = x[:, 0, 1:, :]
    diff = befor-after
    diff.register_hook(lambda grad: print('diff_3 back', (grad != grad).any().item()))
    y_1 = torch.pow(diff, 2)
    findnan('y_1', y_1)
    y_1.register_hook(lambda grad: print('y_1 back', (grad != grad).any().item()))

    befor = x[:, 1, :-1, :]
    after = x[:, 1, 1:, :]
    diff = befor-after
    diff.register_hook(lambda grad: print('diff_4 back', (grad != grad).any().item()))
    y_2 = torch.pow(diff, 2)
    findnan('y_2', y_2)
    y_2.register_hook(lambda grad: print('y_2 back', (grad != grad).any().item()))

    y_sqrt = torch.sqrt(y_1 + y_2)
    y_sqrt.register_hook(lambda grad: print('y_sqrt back', (grad != grad).any().item()))
    tv_y = torch.sum(y_sqrt)
    tv_y.register_hook(lambda grad: print('tv_y back', (grad != grad).any().item()))
    return tv_x + tv_y

</code></pre>

username: albanD
cooked: <aside class="quote no-group" data-username="Alan_Wang" data-post="5" data-topic="97489">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alan_wang/48/13886_2.png" class="avatar"> Alan_Wang:</div>
<blockquote>
<p>For my problem, it seems that the error is when calculating the diff. Why is a subtraction operator giving an nan?</p>
</blockquote>
</aside>
<p><code>inf - inf</code> would give nan IIRC.</p>
<blockquote>
<p>It seems that setting <code>set_detect_anomaly</code> on prevents any nan detection (all the outputs are False). Is this expected?</p>
</blockquote>
<p>No but maybe your code is “flaky” and depending on the run, the nan don’t appear at the same place?</p>

username: Alan_Wang
cooked: <p>Yes, <code>inf-inf</code> was exactly the issue! Thanks for the help.</p>