How do I understand PyTorch anomaly detection?

username: Kong
cooked: <p>I just started using PyTorch’s anomaly detection <a href="https://pytorch.org/docs/master/autograd.html#anomaly-detection" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch master documentation</a> and was hoping to get some help in reading the output.</p>
<p>Does the error message indicate that the derivative of the line below results in x being a nan of inf?</p>
<blockquote>
<p>return self.mu(x), torch.log(torch.exp(self.sigma(x))+1)</p>
</blockquote>
<h2><a name="error-messages-1" class="anchor" href="#error-messages-1"></a>Error messages</h2>
<p>Warning: NaN or Inf found in input tensor.</p>
<blockquote>
<p>sys:1: RuntimeWarning: Traceback of forward call that caused the error:<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/runpy.py”, line 193, in _run_module_as_main<br>
“<strong>main</strong>”, mod_spec)<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/runpy.py”, line 85, in _run_code<br>
exec(code, run_globals)<br>
File “/home/kong/.local/lib/python3.5/site-packages/ipykernel_launcher.py”, line 16, in <br>
app.launch_new_instance()<br>
File “/home/kong/.local/lib/python3.5/site-packages/traitlets/config/application.py”, line 658, in launch_instance<br>
app.start()<br>
File “/home/kong/.local/lib/python3.5/site-packages/ipykernel/kernelapp.py”, line 563, in start<br>
self.io_loop.start()<br>
File “/home/kong/.local/lib/python3.5/site-packages/tornado/platform/asyncio.py”, line 148, in start<br>
self.asyncio_loop.run_forever()<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/asyncio/base_events.py”, line 421, in run_forever<br>
self._run_once()<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/asyncio/base_events.py”, line 1425, in _run_once<br>
handle._run()<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/asyncio/events.py”, line 127, in _run<br>
self._callback(*self._args)<br>
File “/home/kong/.local/lib/python3.5/site-packages/tornado/ioloop.py”, line 690, in <br>
lambda f: self._run_callback(functools.partial(callback, future))<br>
File “/home/kong/.local/lib/python3.5/site-packages/tornado/ioloop.py”, line 743, in _run_callback<br>
ret = callback()<br>
File “/home/kong/.local/lib/python3.5/site-packages/tornado/gen.py”, line 787, in inner<br>
self.run()<br>
File “/home/kong/.local/lib/python3.5/site-packages/tornado/gen.py”, line 748, in run<br>
yielded = self.gen.send(value)<br>
File “/home/kong/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py”, line 365, in process_one<br>
yield gen.maybe_future(dispatch(*args))<br>
File “/home/kong/.local/lib/python3.5/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/kong/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py”, line 272, in dispatch_shell<br>
yield gen.maybe_future(handler(stream, idents, msg))<br>
File “/home/kong/.local/lib/python3.5/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/kong/.local/lib/python3.5/site-packages/ipykernel/kernelbase.py”, line 542, in execute_request<br>
user_expressions, allow_stdin,<br>
File “/home/kong/.local/lib/python3.5/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/kong/.local/lib/python3.5/site-packages/ipykernel/ipkernel.py”, line 294, in do_execute<br>
res = shell.run_cell(code, store_history=store_history, silent=silent)<br>
File “/home/kong/.local/lib/python3.5/site-packages/ipykernel/zmqshell.py”, line 536, in run_cell<br>
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)<br>
File “/home/kong/.local/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 2855, in run_cell<br>
raw_cell, store_history, silent, shell_futures)<br>
File “/home/kong/.local/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 2881, in _run_cell<br>
return runner(coro)<br>
File “/home/kong/.local/lib/python3.5/site-packages/IPython/core/async_helpers.py”, line 68, in <em>pseudo_sync_runner<br>
coro.send(None)<br>
File “/home/kong/.local/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 3058, in run_cell_async<br>
interactivity=interactivity, compiler=compiler, result=result)<br>
File “/home/kong/.local/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 3249, in run_ast_nodes<br>
if (await self.run_code(code, result,  async</em>=asy)):<br>
File “/home/kong/.local/lib/python3.5/site-packages/IPython/core/interactiveshell.py”, line 3326, in run_code<br>
exec(code_obj, self.user_global_ns, self.user_ns)<br>
File “”, line 114, in <br>
prior_mu, prior_sigma, posterior_mu, posterior_sigma, x_pred = net(x_true)<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/kong/Desktop/AST/scripts/experiments_block/vrnn_gru/model.py”, line 363, in forward<br>
f_posterior_mu, f_posterior_sigma = self.f_posterior(torch.cat((state_convlstm,self.CNN(x)),1))<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/kong/Desktop/AST/scripts/experiments_block/vrnn_gru/model.py”, line 179, in forward<br>
return self.mu(x), torch.log(torch.exp(self.sigma(x))+1)<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/site-packages/torch/nn/modules/module.py”, line 489, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/kong/anaconda3/envs/ffp/lib/python3.5/site-packages/torch/nn/modules/conv.py”, line 320, in forward<br>
self.padding, self.dilation, self.groups)</p>
</blockquote>
<pre><code class="lang-auto">-
RuntimeErrorTraceback (most recent call last)
&lt;ipython-input-1-a59af95c119f&gt; in &lt;module&gt;
    126 
    127             optimizer.zero_grad()
--&gt; 128             loss.backward()
    129             optimizer.step()
    130 

~/anaconda3/envs/ffp/lib/python3.5/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    100                 products. Defaults to ``False``.
    101         """
--&gt; 102         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    103 
    104     def register_hook(self, hook):

~/anaconda3/envs/ffp/lib/python3.5/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 90         allow_unreachable=True)  # allow_unreachable flag
     91 
     92 

RuntimeError: Function 'CudnnConvolutionBackward' returned nan values in its 0th output.
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>This means that the gradients computed by the convolution at this line (self.mu I guess?) returned gradients for its 0th input (x in this case) that contains <code>nan</code>.<br>
It’s not that <code>x</code> is <code>nan</code> but that its gradients contain <code>nan</code>.</p>