Question about how to use the result of detect_anomaly

username: KoalaSheep
cooked: <p>Hi, everyone.<br>
I’m having trouble with my custom loss due to Nan output after several epochs.<br>
I found a similar topic and a possible solution:</p><aside class="quote" data-post="8" data-topic="25929">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/getting-nan-after-first-iteration-with-custom-loss/25929/8">Getting Nan after first iteration with custom loss</a> 
  </div>
  <blockquote>
    As your script is quite complicated, you could try to build PyTorch from source and try out the <a href="https://pytorch.org/docs/master/autograd.html#anomaly-detection" rel="noopener nofollow ugc">anomaly detection</a>, which will try to get the method causing the NANs. 
You’ll find the build instructions <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">here</a>. 
Let me know, if you encounter any problems. 
Alternatively, you could create an executable code snippet and I could try to run it on my machine.
  </blockquote>
</aside>
<p>
I use it like this(GB is the name of my network which is a densenet169)</p>
<pre><code class="lang-auto">        for inputs, labels in dataloaders['train']:
            inputs = inputs.to(device)
            labels = labels.to(device).float()
            optimizers['GB'].zero_grad()
            with torch.set_grad_enabled(True):
                with torch.autograd.detect_anomaly():
                    outputs = GB(inputs)
                    loss = LOSS(outputs, labels)
#                    loss.register_hook(lambda grad : print(grad))
                    loss.backward()
</code></pre>
<details>
<summary>
detail of the traceback, it's too long to read so I try to make some summary below</summary>
<p>/opt/conda/conda-bld/pytorch_1565272271120/work/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/runpy.py”, line 193, in _run_module_as_main<br>
“<strong>main</strong>”, mod_spec)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/runpy.py”, line 85, in _run_code<br>
exec(code, run_globals)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/spyder_kernels/console/<strong>main</strong>.py”, line 11, in <br>
start.main()<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/spyder_kernels/console/start.py”, line 318, in main<br>
kernel.start()<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/ipykernel/kernelapp.py”, line 563, in start<br>
self.io_loop.start()<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/tornado/platform/asyncio.py”, line 148, in start<br>
self.asyncio_loop.run_forever()<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/asyncio/base_events.py”, line 534, in run_forever<br>
self._run_once()<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/asyncio/base_events.py”, line 1771, in _run_once<br>
handle._run()<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/asyncio/events.py”, line 88, in _run<br>
self._context.run(self._callback, *self._args)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/tornado/ioloop.py”, line 690, in <br>
lambda f: self._run_callback(functools.partial(callback, future))<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/tornado/ioloop.py”, line 743, in _run_callback<br>
ret = callback()<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/tornado/gen.py”, line 787, in inner<br>
self.run()<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/tornado/gen.py”, line 748, in run<br>
yielded = self.gen.send(value)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/ipykernel/kernelbase.py”, line 365, in process_one<br>
yield gen.maybe_future(dispatch(*args))<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/ipykernel/kernelbase.py”, line 272, in dispatch_shell<br>
yield gen.maybe_future(handler(stream, idents, msg))<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/ipykernel/kernelbase.py”, line 542, in execute_request<br>
user_expressions, allow_stdin,<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/ipykernel/ipkernel.py”, line 294, in do_execute<br>
res = shell.run_cell(code, store_history=store_history, silent=silent)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/ipykernel/zmqshell.py”, line 536, in run_cell<br>
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/IPython/core/interactiveshell.py”, line 2855, in run_cell<br>
raw_cell, store_history, silent, shell_futures)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/IPython/core/interactiveshell.py”, line 2881, in <em>run_cell<br>
return runner(coro)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/IPython/core/async_helpers.py”, line 68, in <em>pseudo_sync_runner<br>
coro.send(None)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/IPython/core/interactiveshell.py”, line 3058, in run_cell_async<br>
interactivity=interactivity, compiler=compiler, result=result)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/IPython/core/interactiveshell.py”, line 3249, in run_ast_nodes<br>
if (await self.run_code(code, result,  async</em>=asy)):<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/IPython/core/interactiveshell.py”, line 3326, in run_code<br>
exec(code_obj, self.user_global_ns, self.user_ns)<br>
File “”, line 1, in <br>
runfile(‘/home/koalasheep/YMY/pro2/EFF_512.py’, wdir=‘/home/koalasheep/YMY/pro2’)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py”, line 985, in runfile<br>
exec_code(file_code, filename, namespace)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py”, line 891, in exec_code<br>
exec(compile(code, filename, ‘exec’), namespace)<br>
File “/home/koalasheep/YMY/pro2/EFF_512.py”, line 260, in <br>
top(50, 12, 12, data_folder, 556, 512, CXR14_csv, train_val_list, test_list, device</em>)<br>
File “/home/koalasheep/YMY/pro2/EFF_512.py”, line 248, in top<br>
train(GB, optimizers, NUM_EPOCH, dataloaders, device, dataset_sizes)<br>
File “/home/koalasheep/YMY/pro2/EFF_512.py”, line 97, in train<br>
loss = LOSS(outputs, labels)<br>
File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/koalasheep/YMY/pro2/utils_pro2.py”, line 394, in forward<br>
temp_label = (1 - (targets[…, i] * inputs[…, i] + (1 - targets[…, i]) * (1 - inputs[…,i]))) ** gamma * temp</p>
<p>Traceback (most recent call last):</p>
<p>File “/home/koalasheep/YMY/pro2/EFF_512.py”, line 260, in <br>
top(50, 12, 12, data_folder, 556, 512, CXR14_csv, train_val_list, test_list, device_)</p>
<p>File “/home/koalasheep/YMY/pro2/EFF_512.py”, line 248, in top<br>
train(GB, optimizers, NUM_EPOCH, dataloaders, device, dataset_sizes)</p>
<p>File “/home/koalasheep/YMY/pro2/EFF_512.py”, line 99, in train<br>
loss.backward()</p>
<p>File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/torch/tensor.py”, line 118, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)</p>
<p>File “/home/koalasheep/anaconda3/envs/torch/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 93, in backward<br>
allow_unreachable=True)  # allow_unreachable flag</p>
<p>RuntimeError: Function ‘MulBackward0’ returned nan values in its 1th output.</p>
</details>
<p>traceback stops at loss.backward(), so I paste my custom loss here.</p>
<pre><code class="lang-auto">def weighted_BCE(sigmoid_x, targets):
    assert sigmoid_x.size() == targets.size()
    count_p = (targets == 1.0).sum() + 1
    count_n = (targets == 0.0).sum() + 1
    loss = -((targets * sigmoid_x.log()) * (1 / count_p.float())) - (((1 - targets) * (1 - sigmoid_x).log()) * (1 / count_n.float()))
    return loss.mean()

class WBCE(nn.Module): #this is my custom loss
    def __init__(self, weight = None, size_average = True):
        super(WBCE, self).__init__()
        
    def forward(self, inputs, targets, gamma = 0.5):
        assert inputs.size() == targets.size()
        L = inputs.size(1)
        loss = 0.0
        for i in range(L):
            temp = weighted_BCE(sigmoid_x = inputs[..., i], targets = targets[..., i])
            temp_label = (1 - (targets[..., i] * inputs[..., i] + (1 - targets[..., i]) * (1 - inputs[...,i]))) ** gamma * temp
# traceback stops here, at the calculation of temp_label.
            loss += temp_label
        loss = loss.mean()
        return loss
</code></pre>
<p>Here is the error message, Nan grad seems to be the reason.</p>
<pre><code class="lang-auto">RuntimeError: Function 'MulBackward0' returned nan values in its 1th output.
</code></pre>
<p>But I don’t know what to do next. My code is normal with nn.BCEloss(), and still runs several epochs with the custom loss before it outputs Nan.<br>
It’s my first time to use a custom loss, and any advice will be helpful.</p>

username: ptrblck
cooked: <p>Based on the output message I would assume <code>targets * sigmoid_x.log()</code> creates the NaN values.<br>
In particular, could you check the values of <code>sigmoid_x.log()</code>?<br>
If you are passing zeros in <code>sigmoid_x</code>, the <code>log()</code> operation will result in an <code>-inf</code> output.</p>

username: KoalaSheep
cooked: <p>Thank you for your advice.<br>
Based on your suggestion, I make a change to check it.<br>
Since sigmoid_x.log() only exists in the def of weighted_BCE(), I # the temp_label like this:</p>
<pre><code class="lang-auto">        for i in range(L):
            temp = weighted_BCE(sigmoid_x = inputs[..., i], targets = targets[..., i])
#            temp_label = (1 - (targets[..., i] * inputs[..., i] + (1 - targets[..., i]) * (1 - inputs[...,i]))) ** gamma * temp
#            loss += temp_label
            loss += temp
</code></pre>
<p>and all the rest remains the same.<br>
Now the loss will only calculate weighted_BCE without calculating temp_label, and until now the network works well(around 7 epochs, I set 30 epochs so I will tell you the result later).<br>
So I think sigmoid_x.log() may not be the reason, as the input is the output of nn.sigmoid().</p>