Why am I getting this error about a Leaf Variable?

username: polm23
cooked: <p>I’m trying to train some entity embeddings, but when doing a backward pass I get an error about Leaf Variables being moved into the graph interior. I saw there are other threads where people had that issue and it’s due to in-place operations or assigning to tensors, but I’m not doing anything like that so I’m unsure what’s causing it.</p>
<p>My code is based on the code <a href="http://bytepawn.com/hacker-news-embeddings-with-pytorch.html" rel="nofollow noopener">here</a>; here’s my network:</p>
<pre><code class="lang-auto">class GameEmbedding(torch.nn.Module):
    def __init__(self, num_games, num_links, embedding_dim=64):
        super(GameEmbedding, self).__init__()
        self.game_embedding = torch.nn.Embedding(num_games, embedding_dim, max_norm=1.0)
        self.link_embedding = torch.nn.Embedding(num_links, embedding_dim, max_norm=1.0)
        self.embedding_dim = embedding_dim

    def forward(self, batch):
        # in the batch each input is [game, link, label]
        # label is 1 (true) or -1 (false)
        t1 = self.game_embedding(torch.LongTensor([v[0] for v in batch]))
        t2 = self.link_embedding(torch.LongTensor([v[1] for v in batch]))
        dot_products = torch.bmm(
            t1.contiguous().view(len(batch), 1, self.embedding_dim),
            t2.contiguous().view(len(batch), self.embedding_dim, 1)
        )
        return dot_products.contiguous().view(len(batch))
</code></pre>
<p>Here’s my training, skipping initialization params:</p>
<pre><code class="lang-auto">for i in range(num_epochs):
    for j in range(num_steps_per_epoch):
        optimizer.zero_grad()
        minibatch = build_minibatch(num_positives, num_negatives)
        y = model.forward(minibatch)
        target = torch.FloatTensor([v[2] for v in minibatch])
        loss = loss_function(y, target)
        if i == 0 and j == 0:
            print('r: loss = %.3f' % float(loss))
        loss.backward(retain_graph=True)
        optimizer.step()
    print('%s: loss = %.3f' % (i, float(loss)))
</code></pre>
<p>And here’s detailed error output:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 88, in &lt;module&gt;
    loss.backward(retain_graph=True)
  File "/mnt/pool/code/gamesearch/env/lib/python3.7/site-packages/torch/tensor.py", line 150, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/mnt/pool/code/gamesearch/env/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: leaf variable has been moved into the graph interior
</code></pre>
<p>Sorry I can’t ask a more specific question, but after reading up on other cases of this error and how leaf variables work I have no idea why it’s happening here.</p>

username: dejanbatanjac
cooked: <aside class="quote no-group" data-username="polm23" data-post="1" data-topic="58468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/polm23/48/16768_2.png" class="avatar"> polm23:</div>
<blockquote>
<p>retain_graph=True</p>
</blockquote>
</aside>
<p>This part looks like a “beep” to me.<br>
If you check out the <a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.backward" rel="noopener nofollow ugc">official docs</a> you will try to avoid <code>retain_graph</code>.</p>
<p>Will you try backward w/o the <code>retain_graph=False</code> and let us know. Also, is your train.py somewhere for preview?</p>

username: polm23
cooked: <p>My whole <code>train.py</code> is <a href="https://gist.github.com/polm/ef59bd501a35d75a3a3110a185c52059" rel="nofollow noopener">here</a>.</p>
<p><code>retain_graph=True</code> is from the code I’m using as a base. I tried removing it but my error didn’t change.</p>

username: dejanbatanjac
cooked: <p>Sorry, I meant to say <code>retain_graph=False</code>.</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you can try to get a small code sample that reproduces the issue, that would help a lot to find out what the problem is.<br>
My approach for this is usually, in a new file:</p>
<ul>
<li>copy/paste your modules definitions</li>
<li>copy/paste your training inner loop</li>
<li>add code that generate one random input</li>
</ul>
<p>If the error does not appear any-more, add back more things from your model initialization / data preprocessing.<br>
If the error still occurs, start removing stuff from this new file until it disappears.<br>
Once you have such a minimal example, could you post it here?</p>

username: polm23
cooked: <p>So I worked on making a minimal example of this, and in fiddling with the parameters I found that reducing the minibatch size makes the error go away. I’m glad it’s fixed but this doesn’t make sense to me - should changing the minibatch size change my computation graph? Based on the error message I would never have guessed it was related to my batch size.</p>
<p>The simplified version of the code I made is <a href="https://gist.github.com/polm/f512451edc0a670a334c76ddef99815c" rel="nofollow noopener">here</a>, though the fix is the same as in the original.</p>

username: albanD
cooked: <p>That’s interesting. I can run your code sample with no issue locally. (Even after increasing the batch size drastically to (5000, 5000).<br>
Which version of pytorch are you using?<br>
It running this code consistently give you the error above?</p>

username: polm23
cooked: <p>I ran the minimal version’s code several more times and at 500 it gives me the error every time. At 499 it works.</p>
<p>I’m using torch 1.3.0, installed via pip, on Linux.</p>

username: albanD
cooked: <p>Ok running with 1.3 on linux I can repro.<br>
Also I am very worried as I reduced your example even further to:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

# &lt;= 1000 works
# &gt; 1000 does not work
val = 1001

mod = torch.nn.Embedding(100, 64, max_norm=1.0)
v0 = torch.rand(val).mul(100).long()

out = mod(v0)
print(out.grad_fn)
print(out.grad_fn.next_functions) # This should show an AccumulateGrad. If it's a CopySlices, it will fail below
print(out.sum().backward())
</code></pre>
<p>Could you check what is the breakpoint value for you? Is it the same?</p>

username: polm23
cooked: <p>Your code works for me at 999 but fails at 1000 as well as 1001. As you suggest in your comment, at 1000 it’s a CopySlices, but at 999 it’s an AccumulateGrad.</p>
<p>I guess it’s safe to say this is a bug?</p>

username: albanD
cooked: <p>Yes !<br>
I opened an issue there: <a href="https://github.com/pytorch/pytorch/issues/28370" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/28370</a></p>