Function 'SvdHelperBackward' returned nan values in its 0th output

username: Kavita_Wagh
cooked: <p>I am using torch.svd() in my code. I am getting this error at backward() function call. I have read and tried many things. Nothing worked. I have written my own pinv() function and in that I am using torch.svd(). Please help! Following is the stack trace.</p>
<p>/home/kavita/anaconda3/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py:145: UserWarning: Error detected in SvdHelperBackward. Traceback of forward call that caused the error:<br>
File “/home/kavita/anaconda3/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/home/kavita/anaconda3/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/ipykernel_launcher.py”, line 16, in <br>
app.launch_new_instance()<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/traitlets/config/application.py”, line 845, in launch_instance<br>
app.start()<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/ipykernel/kernelapp.py”, line 612, in start<br>
self.io_loop.start()<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/tornado/platform/asyncio.py”, line 149, in start<br>
self.asyncio_loop.run_forever()<br>
File “/home/kavita/anaconda3/lib/python3.8/asyncio/base_events.py”, line 570, in run_forever<br>
self._run_once()<br>
File “/home/kavita/anaconda3/lib/python3.8/asyncio/base_events.py”, line 1859, in _run_once<br>
handle._run()<br>
File “/home/kavita/anaconda3/lib/python3.8/asyncio/events.py”, line 81, in _run<br>
self._context.run(self._callback, *self._args)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/tornado/ioloop.py”, line 690, in <br>
lambda f: self._run_callback(functools.partial(callback, future))<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/tornado/ioloop.py”, line 743, in _run_callback<br>
ret = callback()<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/tornado/gen.py”, line 787, in inner<br>
self.run()<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/tornado/gen.py”, line 748, in run<br>
yielded = self.gen.send(value)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/ipykernel/kernelbase.py”, line 365, in process_one<br>
yield gen.maybe_future(dispatch(*args))<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/ipykernel/kernelbase.py”, line 268, in dispatch_shell<br>
yield gen.maybe_future(handler(stream, idents, msg))<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/ipykernel/kernelbase.py”, line 543, in execute_request<br>
self.do_execute(<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/tornado/gen.py”, line 209, in wrapper<br>
yielded = next(result)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/ipykernel/ipkernel.py”, line 306, in do_execute<br>
res = shell.run_cell(code, store_history=store_history, silent=silent)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/ipykernel/zmqshell.py”, line 536, in run_cell<br>
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 2877, in run_cell<br>
result = self._run_cell(<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 2923, in _run_cell<br>
return runner(coro)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/IPython/core/async_helpers.py”, line 68, in <em>pseudo_sync_runner<br>
coro.send(None)<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 3146, in run_cell_async<br>
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 3338, in run_ast_nodes<br>
if (await self.run_code(code, result,  async</em>=asy)):<br>
File “/home/kavita/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py”, line 3418, in run_code<br>
exec(code_obj, self.user_global_ns, self.user_ns)<br>
File “”, line 38, in <br>
res2 = pinv2(torch.matmul(zt_U, zt_U_transp))<br>
File “”, line 18, in pinv2<br>
u, s, vt = LA.svd(a, full_matrices=True)<br>
(Triggered internally at  /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:104.)<br>
Variable._execution_engine.run_backward(</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
51         #  print(’$’, torch.isnan(param.data).int().sum())<br>
52         optimizer.zero_grad()<br>
—&gt; 53         total_loss.backward()<br>
54         <span class="hashtag">#print</span>(’$’)<br>
55</p>
<p>~/anaconda3/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)<br>
243                 create_graph=create_graph,<br>
244                 inputs=inputs)<br>
 → 245         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
246<br>
247     def register_hook(self, hook):</p>
<p>~/anaconda3/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)<br>
143         retain_graph = create_graph<br>
144<br>
 → 145     Variable.<em>execution_engine.run_backward(<br>
146         tensors, grad_tensors</em>, retain_graph, create_graph, inputs,<br>
147         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag</p>
<p>RuntimeError: Function ‘SvdHelperBackward’ returned nan values in its 0th output.</p>
<p>Following is the array of singular values I get from svd():<br>
[4.7464e-02, 2.9351e+03, 2.1174e+05, 2.7355e+05, 2.7720e+05, 3.3497e+05,<br>
3.8153e+05, 4.6172e+05, 5.1771e+05, 6.3530e+05, 7.1788e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05, 8.1157e+05,<br>
8.1157e+05, 8.8670e+05, 1.3213e+06, 1.3892e+06]</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can check the warning for the svd function: <a href="https://pytorch.org/docs/master/generated/torch.linalg.svd.html?highlight=svd#torch.linalg.svd" class="inline-onebox">torch.linalg.svd — PyTorch master documentation</a><br>
In particular it says that gradients are not well defined if there are repeated singular values.<br>
You might want to add a small noise to your matrix to avoid such repeated values I think.</p>