Can not create a new calculation graph after the backward propagation

hcl: <p>Background：I use DQN and DDPG to solve two tasks simultaneously. The <code>state(input)</code> of DQN and DDPG are both two parts. One part is the states of the environment, and the other one is the states abstracted from the environment by CNN+LSTM. The two parts are concatenate in <code>forward_dqn()</code> , <code>forward_actor()</code> and <code>forward_critic()</code> respectively.</p>
<p>Question1： I backward propagate the <code>loss_dqn</code> , <code>loss_ddpg_actor</code> , and <code>loss_ddpg_critic</code> in sequence and get the error “Trying to backward through the graph a second time, but the buffers have already been freed.” in the backward propagation of <code>loss_ddpg_actor</code> . Since after the backward propagation of loss_dqn, the computational graph has been freed, so I have forward propagated CNN+LSTM again to calculate the loss_ddpg_actor. Why the computational graph cannot be created again? Thanks.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/8/836209a982ecec4c1f2d463470e86104570a6779.png" alt="%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_20190512164242" data-base62-sha1="iKgANvoyZVwKO1Z12Y95IwTahDz" width="607" height="323"><br>
Model: (env: environment)</p>
<pre><code class="lang-auto">output_cnnlstm = cnnlstm.forward(env)
DQN_output = dqn.forward(cat(output_cnnlstm, state_env))
Actor_output = actor.forward(cat(output_cnnlstm, state_env))
Critic_output = critic.forward(cat(output_cnnlstm, state_env))
</code></pre>
<p>Code 1 (Q1):</p>
<pre><code class="lang-auto">    # dqn
    # forward: cnnlstm
    s_cnnlstm_out, _, _ = self.model.forward_cnnlstm(s_cnnlstm, flag_optim=True)
    # forward: dqn
    q_eval_dqn = self.model.forward_dqn_eval(s_dqn, s_cnnlstm_out).gather(1, a_dqn)
    q_next_dqn = self.model.forward_dqn_target(s_dqn_next, s_cnnlstm_out).detach()
    q_target_dqn = r + GAMMA_DQN * q_next_dqn.max(dim=1)[0].reshape(SIZE_BATCH * SIZE_TRANSACTION, 1)
    # optimzie: dqn
    loss_dqn = self.loss_dqn(q_eval_dqn, q_target_dqn)
    self.optimizer_cnnlstm.zero_grad()
    self.optimizer_dqn.zero_grad()
    loss_dqn.backward()
    self.optimizer_cnnlstm.step()
    self.optimizer_dqn.step()
    loss_dqn = loss_dqn.detach().numpy()
    # ddpg
    # actor
    # forward: cnnlstm
    s_cnnlstm_out, _, _ = self.model.forward_cnnlstm(s_cnnlstm, flag_optim=True)
    # forward: ddpg: actor
    a_eval_ddpg = self.model.forward_actor_eval(s_ddpg, s_cnnlstm_out)
    # optimze: ddpg: cnnlstm + actor
    loss_ddpg_actor = - self.model.forward_cirtic_eval(s_ddpg, a_eval_ddpg, s_cnnlstm_out).mean()
    self.optimizer_cnnlstm.zero_grad()
    self.optimizer_actor.zero_grad()
    loss_ddpg_actor.backward()
    self.optimizer_cnnlstm.step()
    self.optimizer_actor.step()
    loss_ddpg_actor = loss_ddpg_actor.detach().numpy()
</code></pre>
<p>Question2: I write a demo to test the propagation process and the demo seems to work well since the loss descends normally and the test error is low. So I want to ask the difference between the two codes and models.</p>
<p>Model:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f310c3ba08a8cddf70fd0e75134182cb470171f0.png" alt="%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_20190512164247" data-base62-sha1="yGfWtJbEqN5DGYUYaUgMSaUuHLi" width="467" height="430"></p>
<pre><code class="lang-auto">output_model1 = model1.forward(x)
output_model21 = model21.forward(cat(output_model1, x1))
output_model22 = model221.forward(cat(output_model1, x2))
</code></pre>
<p>compared with the model of Q1, output_model1 ~ cnnlstm, output_model21 ~ DQN, output_model22 ~ Actor</p>
<p>Question3： I set breakpoint in the demo after <code>loss1.backward()</code> and before <code>optimizer1.step()</code> . However, on the one hand, the weight of the linear layer of Model21 changes with the optimization. On the other hand, <code>x._grad</code> is a gradient value tensor, while <code>x1._grad</code> is <code>None</code> . So I wonder the parameters of Model21 are optimized whether or not and why <code>x1._grad</code> is None.</p>
<p>Code 2 (Q2 and Q3):</p>
<pre><code class="lang-auto">for i in range(NUM_OPTIM):
    # optimize task 1
    y1_pred = self.model.forward_task1(x, x1)
    loss1 = self.loss_21(y1_pred, y1)
    self.optimizer1.zero_grad()
    self.optimizer21.zero_grad()
    loss1.backward()
    self.optimizer1.step()
    self.optimizer21.step(
    # optimze task 2
    y2_pred = self.model.forward_task2(x, x2)
    loss2 = self.loss_22(y2_pred, y2)
    self.optimizer1.zero_grad()
    self.optimizer22.zero_grad()
    loss2.backward()
    self.optimizer1.step()
    self.optimizer22.step()
</code></pre>

hcl: <p>My fault. I forget to <code>detach()</code> the hidden state and the cell state of LSTM from the  computational graph in <code>self.model.forward_cnnlstm</code>.</p>