Softmax problem when BACKWARD is executed

username: Peter78
cooked: <p>Hello everyone,</p>
<p>below what I get at a certain episode using REINFORCE algorithm with Pytorch:</p>
<p>C:\Users\fusco\Anaconda3\lib\site-packages\torch\autograd_<em>init</em>_.py:154: UserWarning: Error detected in SoftmaxBackward0. Traceback of forward call that caused the error:<br>
File “C:\Users\fusco\Anaconda3\lib\runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “C:\Users\fusco\Anaconda3\lib\runpy.py”, line 87, in <em>run_code<br>
exec(code, run_globals)<br>
File "C:\Users\fusco\Anaconda3\lib\site-packages\spyder_kernels\console_<em>main</em></em>.py", line 23, in <br>
start.main()<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\spyder_kernels\console\start.py”, line 328, in main<br>
kernel.start()<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\ipykernel\kernelapp.py”, line 677, in start<br>
self.io_loop.start()<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\tornado\platform\asyncio.py”, line 199, in start<br>
self.asyncio_loop.run_forever()<br>
File “C:\Users\fusco\Anaconda3\lib\asyncio\base_events.py”, line 570, in run_forever<br>
self._run_once()<br>
File “C:\Users\fusco\Anaconda3\lib\asyncio\base_events.py”, line 1859, in _run_once<br>
handle._run()<br>
File “C:\Users\fusco\Anaconda3\lib\asyncio\events.py”, line 81, in _run<br>
self._context.run(self._callback, *self._args)<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\ipykernel\kernelbase.py”, line 457, in dispatch_queue<br>
await self.process_one()<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\ipykernel\kernelbase.py”, line 446, in process_one<br>
await dispatch(*args)<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\ipykernel\kernelbase.py”, line 353, in dispatch_shell<br>
await result<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\ipykernel\kernelbase.py”, line 648, in execute_request<br>
reply_content = await reply_content<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\ipykernel\ipkernel.py”, line 353, in do_execute<br>
res = shell.run_cell(code, store_history=store_history, silent=silent)<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\ipykernel\zmqshell.py”, line 533, in run_cell<br>
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 2901, in run_cell<br>
result = self._run_cell(<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 2947, in _run_cell<br>
return runner(coro)<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\IPython\core\async_helpers.py”, line 68, in <em>pseudo_sync_runner<br>
coro.send(None)<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 3172, in run_cell_async<br>
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 3364, in run_ast_nodes<br>
if (await self.run_code(code, result,  async</em>=asy)):<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py”, line 3444, in run_code<br>
exec(code_obj, self.user_global_ns, self.user_ns)<br>
File “C:\Users\fusco\AppData\Local\Temp/ipykernel_17840/2349141103.py”, line 1, in <br>
debugfile(‘C:/Users/fusco/Desktop/Ph.D/Reinforcement_Learning/Smart_Grid/REINFORCEMENT_LEARNING/SARL/Policy_Based_Methods/REINFORCE_BASELINE/Test_4/SARL_Main_REINFORCE_BASELINE_MRKI.py’, wdir=‘C:/Users/fusco/Desktop/Ph.D/Reinforcement_Learning/Smart_Grid/REINFORCEMENT_LEARNING/SARL/Policy_Based_Methods/REINFORCE_BASELINE/Test_4’)<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 607, in debugfile<br>
enter_debugger(<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\spyder_kernels\customize\spyderpdb.py”, line 776, in enter_debugger<br>
debugger.run(code)<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\spyder_kernels\customize\spyderpdb.py”, line 717, in run<br>
super(SpyderPdb, self).run(cmd, globals, locals)<br>
File “C:\Users\fusco\Anaconda3\lib\bdb.py”, line 580, in run<br>
exec(cmd, globals, locals)<br>
File “”, line 1, in <br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 577, in runfile<br>
exec_code(file_code, filename, ns_globals, ns_locals,<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 465, in exec_code<br>
exec(compiled, ns_globals, ns_locals)<br>
File “c:\users\fusco\desktop\ph.d\reinforcement_learning\smart_grid\reinforcement_learning\sarl\policy_based_methods\reinforce_baseline\test_4\sarl_main_reinforce_baseline_mrki.py”, line 184, in <br>
Manager.run()<br>
File “C:\Users\fusco\Desktop\Ph.D\Reinforcement_Learning\Smart_Grid\REINFORCEMENT_LEARNING\SARL\Policy_Based_Methods\REINFORCE_BASELINE\Test_4\SARL_System_Manager_REINFORCE_BASELINE_MRKI.py”, line 123, in run<br>
policy_loss, value_loss = self.agent.train( state_list, action_list, reward_list, episode )<br>
File “C:\Users\fusco\Desktop\Ph.D\Reinforcement_Learning\Smart_Grid\REINFORCEMENT_LEARNING\SARL\Policy_Based_Methods\REINFORCE_BASELINE\Test_4\SARL_Agent_REINFORCE_BASELINE_MRKI.py”, line 223, in train<br>
selected_action_probs = self.policy_network(state_t).gather(1, action_t.long())<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\fusco\Desktop\Ph.D\Reinforcement_Learning\Smart_Grid\REINFORCEMENT_LEARNING\SARL\Policy_Based_Methods\REINFORCE_BASELINE\Test_4\SARL_Agent_REINFORCE_BASELINE_MRKI.py”, line 52, in forward<br>
out_data =  torch.nn.functional.softmax(out_data, dim = -1) <span class="hashtag">#-1</span> to take softmax of last dimension<br>
File “C:\Users\fusco\Anaconda3\lib\site-packages\torch\nn\functional.py”, line 1680, in softmax<br>
ret = input.softmax(dim)<br>
(Triggered internally at  …\torch\csrc\autograd\python_anomaly_mode.cpp:104.)<br>
Variable._execution_engine.run_backward(</p>
<p>It looks like there is any problem with the SOFTMAX when BACKWARD is executed.</p>
<p>Could anyone give me a hint?</p>
<p>Thanks in advance.</p>
<p>Regards.</p>

username: ptrblck
cooked: <p>The actual error message seems to be missing in the posted stacktrace. Are you running your code with anomaly detection enabled and could an operation create invalid values in the forward pass?</p>

username: Peter78
cooked: <p>Dear ptrblck,<br>
thank you for your feedback.</p>
<p>I also noticed that in the error message something is missed.<br>
I am running my code with torch.autograd.set_detect_anomaly(True).</p>
<p>This is my forward pass:</p>
<pre><code>        out_data = torch.nn.functional.relu( self.dense_layer_1(inp_data) )
        out_data = torch.nn.functional.relu( self.dense_layer_2(out_data) )
        out_data = self.output(out_data)
        
        out_data =  torch.nn.functional.softmax(out_data, dim = -1) #-1 to take softmax of last dimension
</code></pre>
<p>To be honest, I cannot understand what is happening.</p>
<p>Regards,</p>
<p>P.</p>
<p>P.S.</p>
<p>I am using Pytorch 1.8.0</p>

username: ptrblck
cooked: <p>Check which operation creates invalid outputs in the forward pass which would explain the failure in the backward. I would also recommend updating to a newer release as 1.8 is quite old by now.</p>