Backward error in DDPG

username: yxz77777
cooked: <p><strong>My algorithm is DDPG, and my actor network is unable to backpropagate. How should I handle this issue?</strong><br>
class ActorNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(ActorNet, self).<strong>init</strong>()<br>
init_w = 1e-3<br>
self.input_size = 3<br>
self.output_size = 1 + 1<br>
self.fc1 = nn.Linear(self.input_size, HIDDEN_SIZE_1)<br>
self.fc2 = nn.Linear(HIDDEN_SIZE_1, HIDDEN_SIZE_2)<br>
self.fc3 = nn.Linear(HIDDEN_SIZE_2, self.output_size)</p>
<pre><code>    init.kaiming_uniform_(self.fc1.weight)
    init.kaiming_uniform_(self.fc2.weight)
    init.kaiming_uniform_(self.fc3.weight)


def forward(self, x):
    x = self.fc1(x)
    x = torch.relu(x)
    x = self.fc2(x)
    x = torch.relu(x)
    x = self.fc3(x)
    x = torch.relu(x)
    return x
</code></pre>
<p><strong>this is my actor-net frame,and the update is below</strong><br>
state, action, reward, next_state = self.memory.sample(self.batch_size)<br>
state_batch = torch.FloatTensor(np.array(state))<br>
action_batch = torch.FloatTensor(np.array(action))<br>
reward_batch = torch.FloatTensor(reward).unsqueeze(1)<br>
next_state_batch = torch.FloatTensor(np.array(next_state))<br>
state_actor_batch = torch.cat((state_batch, action_batch), 1)<br>
policy_Q = torch.mean(self.critic(state_actor_batch))<br>
actor_loss = -policy_Q<br>
self.actor_optimizer.zero_grad()<br>
torch.nn.utils.clip_grad_norm_(self.actor.parameters(), 1)<br>
actor_loss.backward(retain_graph=True)<br>
self.actor_optimizer.step()<br>
and my optim is<br>
self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=1e-3, weight_decay=1e-5)</p>

username: ptrblck
cooked: <p>I don’t know what the error is but often using <code>retain_graph=True</code> is wrong and causes issues trying to calculate gradients from stale forward activations, so could you explain why this argument is used?</p>

username: yxz77777
cooked: <p>without using that argument ,it will rasing this error<br>
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
<p>And what is important is that the weight and bias still stay none after the actor_loss.backward(),this causes my actor-net can’t update and the weight and bias does not change after the backward.I don’t know how this question.</p>

username: yxz77777
cooked: <p>I have solved the error of retain_graph=True.I use the policy_Q to calculate both actorloss and criticloss which causes the error.But I still have  problem of weight and bias keeps none after backward().I don’t know how to handle this question.This is the key problem.</p>

username: ptrblck
cooked: <p>I assume you have removed <code>retain_graph=True</code> and fixed the originally raised error?<br>
Trainable parameters will get a valid gradient assigned to their <code>.grad</code> attribute during the <code>backward</code> pass if they were used in the associated computation graph. If some parameters show a <code>None</code> gradient after calling <code>loss.backward()</code> they weren’t used to compute <code>loss</code> and are thus not in the computation graph or they were detached from the graph.</p>

username: yxz77777
cooked: <p>I am sorry that my question hasn’t been solved.<br>
so is that the weight and bias in actornet haven’t been used to compute the loss?that’s why they always keeps none gradient?<br>
thx so much for your help,I have been confused for a few weeks.I will check train frame,and reply you later.<br>
thank you very much</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="yxz77777" data-post="6" data-topic="190201">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/6bbea6/48.png" class="avatar"> yxz77777:</div>
<blockquote>
<p>so is that the weight and bias in actornet haven’t been used to compute the loss?that’s why they always keeps none gradient?</p>
</blockquote>
</aside>
<p>Yes, as you can see here:</p>
<pre data-code-wrap="python"><code class="lang-python">state_batch = torch.FloatTensor(np.array(state))
action_batch = torch.FloatTensor(np.array(action))
state_actor_batch = torch.cat((state_batch, action_batch), 1)
policy_Q = torch.mean(self.critic(state_actor_batch))
actor_loss = -policy_Q
actor_loss.backward()
</code></pre>
<p><code>actor_loss</code> uses the <code>state_/action_batch</code> inputs, which are not coming from the <code>actor</code> and then calls into <code>self.critic</code>. <code>ActorNet</code> is thus never used to calculate <code>actor_loss</code>.</p>

username: yxz77777
cooked: <p>wow,i got it.thank you so much for your help!</p>