RuntimeError: Trying to backward through the graph a second time... and other errors

ajoq: <p>I’d appreciate it if someone could shed some light on this.</p>
<p>I’ve encountered this problem when trying to train my model (policy gradient for asset allocation. Hence the terms below <em>weights</em> and <em>rebalance</em> and <code>pv</code> for <em>portfolio value</em>), but anytime I try to solve the issue I stumble upon another error:</p>
<p>If it is of any help:</p>
<ul>
<li>
<code>data_loader</code> is a custom class instance for retrieving batches.</li>
<li>
<code>pvm</code> is buffer (torch tensor) where the weights vectors <code>w</code> are saved. At each iteration, I retrieve some of them, and then replace them by the output of the model.</li>
<li><code>env.get_rewards(pv_prev, pv_next) = torch.log(pv_next / pv_prev)</code></li>
</ul>
<p>Training loop:</p>
<pre><code class="lang-auto">for t in range(batch_size + n_batches, train_data_length):
    for t_last in range(t-n_batches, t):
        
        opt.zero_grad()
        
        X_batch, y_batch = data_loader.get_batch(t_last)
        
        w_prev = pvm.get_weights(t_last)
        
        w_opt = model(X_batch, w_prev)
        
        pvm.update_weights(w_opt, t_last)
        
        pv_prev, pv_next = env.rebalance(y_batch, w_prev, w_opt, t_last)
        
        rewards = env.get_rewards(pv_prev, pv_next)
        
        loss = -torch.mean(rewards)
        
        print(loss.data.numpy())
        
        loss.backward()
        
        opt.step()
</code></pre>
<p>As it is, at the second iteration it throws <code>RuntimeError: Trying to backward through the graph a second time...</code></p>
<p>There some posts suggesting to:</p>
<ol>
<li>change <code>loss.backward()</code> to <code>loss.backward(retain_graph=True)</code> (<a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795">here</a>)</li>
</ol>
<ol start="2">
<li>use <code>.detach</code> on the output of the network (if I’m not mistaken), that is <code>model(X_batch, w_prev).detach()</code> (<a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time-while-using-custom-loss-function/12360/2">here</a>)</li>
</ol>
<p>If I use the first approach, I get <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 21, 1, 1]] is at version 2...</code>. The only tensor having <code>21</code> in one of its dimensions is the first <code>concat</code> variable of the network in the forward pass:</p>
<pre><code class="lang-auto">def forward(self, X_batch, w_batch):
        
        w_stocks = w_batch[:, 1:, :]
        
        out = F.relu(self.cnn1(X_batch))
        out = F.relu(self.cnn2(out))
        concat = torch.cat([out, w_stocks.unsqueeze(dim=1)], dim=1)
        out = self.cnn3(concat).squeeze(dim=1)
        
        cash_bias = torch.zeros(self.batch_size, 1, 1)
        
        concat = torch.cat([cash_bias, out], dim=1)
        
        out = F.softmax(concat, dim=1)
        return out
</code></pre>
<p>if I use the second one, then the backward pass will stop working.</p>

vainaijr: <p>maybe use,</p>
<pre><code class="lang-auto">loss.backward(create_graph=True)
</code></pre>