DDPG ounoise inplace operation error

username: jack70703350
cooked: <p>Hi everyone, I’m working on two link robot arm control using DDPG. My custom environment is like reacher in Mujoco environment and I’m trying to train my robot arm fingertip to follow specific trajectory (i.e. circle). I’m having problems with convergence to good policy, so I’m using something like DDPGfD, I extracted wanted state trajectories and actions from some other file using Lagrange - Euler method. The problem is that I’m having a inplace operation error.</p>
<p>C:\Users\user\Desktop\project\venv\Lib\site-packages\torch\autograd_<em>init</em>_.py:266: UserWarning: Error detected in MmBackward0. Traceback of forward call that caused the error:<br>
File “c:\Users\user\Desktop\project\sym_robot_arm\sample_two_link\04-30 DDPG\twolink_DDPG_trial.py”, line 332, in <br>
main()<br>
File “c:\Users\user\Desktop\project\sym_robot_arm\sample_two_link\04-30 DDPG\twolink_DDPG_trial.py”, line 260, in main<br>
action = get_action(actor, state)<br>
File “c:\Users\user\Desktop\project\sym_robot_arm\sample_two_link\04-30 DDPG\twolink_DDPG_trial.py”, line 181, in get_action<br>
action = _actor(state)<br>
File “C:\Users\user\Desktop\project\venv\Lib\site-packages\torch\nn\modules\module.py”, line 1511, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “C:\Users\user\Desktop\project\venv\Lib\site-packages\torch\nn\modules\module.py”, line 1520, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “c:\Users\user\Desktop\project\sym_robot_arm\sample_two_link\04-30 DDPG\twolink_DDPG_trial.py”, line 153, in forward<br>
out = 10.0 * F.tanh(self.fc_out(out))<br>
File “C:\Users\user\Desktop\project\venv\Lib\site-packages\torch\nn\modules\module.py”, line 1511, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “C:\Users\user\Desktop\project\venv\Lib\site-packages\torch\nn\modules\module.py”, line 1520, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “C:\Users\user\Desktop\project\venv\Lib\site-packages\torch\nn\modules\linear.py”, line 116, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
(Triggered internally at …\torch\csrc\autograd\python_anomaly_mode.cpp:118.)<br>
Variable.<em>execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
Traceback (most recent call last):<br>
File “c:\Users\user\Desktop\project\sym_robot_arm\sample_two_link\04-30 DDPG\twolink_DDPG_trial.py”, line 332, in <br>
main()<br>
File “c:\Users\user\Desktop\project\sym_robot_arm\sample_two_link\04-30 DDPG\twolink_DDPG_trial.py”, line 298, in main<br>
update_critic(state_batch, action_batch, Q_target_batch)<br>
torch.autograd.backward(<br>
File "C:\Users\user\Desktop\project\venv\Lib\site-packages\torch\autograd_<em>init</em></em>.py", line 266, in backward<br>
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [32, 2]], which is output 0 of AsStridedBackward0, is at version 219897; expected version 219896 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>I know what inplace operation is, I’ve encountered them already when I was trying to implement DDPG. But what makes me frustrated is that, it didn’t make inplace operation error when I was adding noise to the action.</p>
<p>action = get_action(actor, state)<br>
if episode &lt; cfg.max_explore_eps:<br>
p = episode / cfg.max_explore_eps<br>
action = action.detach() + 1.5 * (1 - p) * next(noise)</p>
<p>My action selection code is like this, and I kind of separated episodes into exploration_eps and exploitation_eps. In exploration_eps agents can learn some rules in dynamics, some basic policies to follow the target trajectory, and I wanted to finetune or shape this policy to accurate, good one in exploitation_eps. But after exploration_eps, pytorch gives me this kind of error. The only thing changed is elmination of ounoise to action, but this gives me inplace operation error. Does anyone know why?</p>
<p>Also, I plotted some rewards and rendered my robot arm to see if it is learning well. I found out in early stages the average total reward during eps incrementally increased, but after half exploration_eps, (when noise scale decreases) the performance decrease dramatically. Sometime good score, some times really bad score. This kind of result makes me think that noise is actually helping agent to give good performance. Does anyone can tell me why?</p>
<p>For anyone who want to see whole code, here’s my github repository. Main code is twolink_DDPG_trial</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/Parksanghu/Sym_robot_arm">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/Parksanghu/Sym_robot_arm" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/18e1dc4a3c8acec9c1bea2f79d6a5227783fc0730977db2a555b0cc5b7ccefd7/Parksanghu/Sym_robot_arm" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/Parksanghu/Sym_robot_arm" target="_blank" rel="noopener nofollow ugc">GitHub - Parksanghu/Sym_robot_arm</a></h3>

  <p>Contribute to Parksanghu/Sym_robot_arm development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Also I would appreciate it if you can recommend me some other algorithm to control two link robot arm.</p>