I only backward once.Why this happen?RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward()

yupeng_chou: <p>I only backward once.Why this happen?If I don’t add detach on radical_template,this code can not run.</p>
<pre><code class="lang-auto"> radical_template = []
            for step,img in enumerate(radical_dataloader):
                backbone1.train()
                img = torch.unsqueeze(img,dim=1)
                img = img.to(device)
                feature = backbone1(img)
                radical_template.append(feature)
            radical_template = torch.cat(radical_template,dim=0)
            for step, (imgs, lbs) in enumerate(train_dataloader):
                backbone.train(), classifier.train(),backbone1.train()
                character_template = self.template_manager.sample_random_template(self.opt.training.num_templates).to(device)
                imgs, lbs = imgs.to(device), lbs.to(device)
                out = backbone(imgs)
                out = out.permute(0, 2, 3, 1)
                out = out.view(out.size(0), 64, 128)  # 128 * 16 * 128
                sim = torch.matmul(out, radical_template.transpose(0, 1).detach())  # 128 * 16 * 260
                sim = sim.transpose(1, 2)  # 128 * 260 * 16
                sim = torch.softmax(sim, dim=2)
                output = torch.matmul(sim, out)  # 128 * 260 * 128
                output = torch.sum(output, dim=1)  # 128 * 128
                template = backbone1(character_template, dropout=False,template=True)
                logits,temp,temp1 = classifier(output, label=lbs, template=template)
                loss = torch.nn.functional.cross_entropy(logits, lbs)
                loss = loss + temp + temp1

                accelerator.backward(loss)
</code></pre>

KFrank: <p>Hi Yupeng!</p>
<aside class="quote no-group" data-username="yupeng_chou" data-post="1" data-topic="195432" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yupeng_chou/48/65482_2.png" class="avatar"> yupeng_chou:</div>
<blockquote>
<pre><code class="lang-auto"> radical_template = []
            for step,img in enumerate(radical_dataloader):
                backbone1.train()
                img = torch.unsqueeze(img,dim=1)
                img = img.to(device)
                feature = backbone1(img)
                radical_template.append(feature)
            radical_template = torch.cat(radical_template,dim=0)
</code></pre>
</blockquote>
</aside>
<p>I assume that <code>backbone1</code> is a model with trainable parameters.  If so, at<br>
this point <code>radical_template</code> depends on <code>backbone1</code> and is attached to<br>
a computation graph that leads back to those parameters.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            for step, (imgs, lbs) in enumerate(train_dataloader):
                ...
                out = backbone(imgs)
                ...
                sim = torch.matmul(out, radical_template.transpose(0, 1).detach())  # 128 * 16 * 260
</code></pre>
</blockquote>
</aside>
<p>At this point, <code>sim</code> has been <code>detach()</code>ed from any and all computation graphs,<br>
so it connects back to neither <code>backbone1</code> (through <code>radical_template</code>) nor<br>
<code>backbone</code> (through <code>out</code>).</p>
<p>But without the <code>.detach()</code>, <code>sim</code> will be connected to <code>backbone1</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">                ...
                template = backbone1(character_template, dropout=False,template=True)
                logits,temp,temp1 = classifier(output, label=lbs, template=template)
                loss = torch.nn.functional.cross_entropy(logits, lbs)
                loss = loss + temp + temp1)
</code></pre>
</blockquote>
</aside>
<p>Without the <code>.detach()</code>, <code>loss</code> is connected to <code>backbone1</code> through <code>output</code><br>
and thus <code>sim</code>.  (It is separately connected to <code>backbone1</code> through <code>template</code>,<br>
but that is not the cause of your problem.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">                accelerator.backward(loss)
</code></pre>
</blockquote>
</aside>
<p>This <code>backward()</code> call – inside the <code>for step</code> loop – frees the full computation<br>
graph attached to <code>loss</code>.  Without the <code>.detach()</code>, this includes the portion of<br>
the graph connects <code>radical_template</code> to <code>backbone1</code>.  However, the<br>
computation of <code>radical_template</code> occurs outside of the loop, so that part<br>
of the graph is not rebuilt when you execute the loop a second time.  Therefore<br>
when you call <code>accelerator.backward (loss)</code> a second time, you will try to<br>
backpropagate through the <code>radical_template</code> graph that has already been<br>
freed (and not rebuilt).  Hence the error.</p>
<p>How to fix this will depend on what you are trying to do.  (You don’t show<br>
your optimization step.)  Are you trying to optimize <code>backbone1</code> via <code>loss</code><br>
through the connection of <code>radical_template</code> to <code>backbone1</code>?  In that case<br>
you should move the computation of <code>radical_tensor</code> inside of the<br>
<code>for step</code> loop in order to rebuild the graph.  (<code>backbone1</code> will have changed<br>
because you have optimized it, so recomputing the graph would make sense.)</p>
<p>If you are not optimizing <code>backbone1</code>, you should probably <code>detach()</code> both<br>
<code>radical_template</code> and <code>template</code> so that no graph connects back to<br>
<code>backbone1</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

yupeng_chou: <p>Hi K. Frank!<br>
Thanks for your time!<br>
I have changed the code to solve the problem you mentioned.And it do work.<br>
I still have a question about this sentence.</p>
<blockquote>
<p>At this point, <code>sim</code> has been <code>detach()</code> ed from any and all computation graphs,<br>
so it connects back to neither <code>backbone1</code> (through <code>radical_template</code> ) nor<br>
<code>backbone</code> (through <code>out</code> ).</p>
</blockquote>
<p>Why sim is detached from out ?</p>
<p>Looking forward to your reply.</p>
<p>Best.</p>
<p>Zhou Yupeng</p>

KFrank: <p>Hi Yupeng!</p>
<aside class="quote no-group quote-modified" data-username="yupeng_chou" data-post="3" data-topic="195432" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yupeng_chou/48/65482_2.png" class="avatar"> yupeng_chou:</div>
<blockquote>
<blockquote>
<p>At this point, <code>sim</code> has been <code>detach()</code> ed from any and all computation graphs,<br>
so it connects back to neither <code>backbone1</code> (through <code>radical_template</code> ) nor<br>
<code>backbone</code> (through <code>out</code> ).</p>
</blockquote>
<p>Why sim is detached from out ?</p>
</blockquote>
</aside>
<p>That’s my mistake – <code>sim</code> is <em>not</em> detached from <code>out</code>.</p>
<p>I misread the location of some of the parentheses in this line of code:</p>
<pre data-code-wrap="python"><code class="lang-python">sim = torch.matmul(out, radical_template.transpose(0, 1).detach())
</code></pre>
<p>The second argument to <code>matmul()</code> is detached from its connection to<br>
<code>backbone1</code>, so <code>sim</code> is not connected to <code>backbone1</code>.  But <code>sim</code> is, indeed,<br>
connected to <code>out</code> and hence to <code>backbone</code>.</p>
<p>Sorry for the confusion.</p>
<p>Best.</p>
<p>K. Frank</p>

yupeng_chou: <p>Thanks for your valuable time again!</p>
<p>Best.</p>
<p>Zhou Yupeng</p>