Leaf variable has been moved into the graph interior

liangbright: <p>I am using PyTorch 0.4</p>
<p>import torch<br>
X=torch.randn((100,3))<br>
Y=torch.randn((100))<br>
w1=torch.tensor(0.1, requires_grad=True)<br>
w2=torch.tensor(0.1, requires_grad=True)<br>
w3=torch.tensor(0.1, requires_grad=True)<br>
W=torch.tensor([0.1, 0.1, 0.1], requires_grad=True)<br>
W[0]=w1 * w2;  W[1]=w2 * w3;  W[2]=w3 * w1<br>
#W=torch.cat([w1.view(1),w2.view(1),w3.view(1)])<br>
Yp=torch.sum(X*W, dim=1)<br>
loss = torch.nn.MSELoss()(Yp, Y)<br>
loss.backward()</p>
<p>run the code and I got:<br>
RuntimeError: leaf variable has been moved into the graph interior</p>
<p>uncomment  <span class="hashtag">#W</span>, and it is fine</p>
<p>use case:<br>
w1, w2, w3,…, are many many … tensors from outputs of some modules<br>
then we can assemble them to a big tensor via (1) or (2) below:<br>
(1) use torch.cat<br>
(2) create a tensor W, and assign w1, w2, w3 to subsections of W.  It is easier to control where the w1/w2/w3 should be put into the W</p>

ptrblck: <p>I don’t think you need to initialize <code>W</code> with data and <code>requires_grad=True</code>, if you are overwriting the values in the next line.</p>
<p>Would this work for you?</p>
<pre><code class="lang-auto">w1=torch.tensor(0.1, requires_grad=True)
w2=torch.tensor(0.1, requires_grad=True)
w3=torch.tensor(0.1, requires_grad=True)
W=torch.empty(3, requires_grad=False)
W[0]=w1 * w2; W[1]=w2 * w3; W[2]=w3 * w1
Yp=torch.sum(X*W, dim=1)
loss = torch.nn.MSELoss()(Yp, Y)
loss.backward()
print(w1.grad)
</code></pre>

liangbright: <p>It works. Thank you!</p>
<p>I’m curious, why torch.empty works ?</p>

ptrblck: <p><code>torch.empty</code> just creates a <code>tensor</code> with uninitialized values.<br>
Try to print it after the init and you will see, that the values are quite random.</p>
<p>You could also create it with <code>torch.zeros</code> or skip the initialization completely:</p>
<pre><code class="lang-auto">w1=torch.tensor([0.1], requires_grad=True)
w2=torch.tensor([0.1], requires_grad=True)
w3=torch.tensor([0.1], requires_grad=True)
W=torch.cat((w1 * w2, w2 * w3, w3 * w1))
</code></pre>

liangbright: <p>I tried the following, all of them are fine</p>
<p>W=torch.empty(3, requires_grad=False)<br>
W=torch.zeros(3, requires_grad=False)<br>
W=torch.ones(3, requires_grad=False)<br>
W=torch.randn(3, requires_grad=False); W[0]=0.1; W[1]=0.1;W[2]=0.1<br>
W_numpy=numpy.array([0.1,0.1,0.1],dtype=‘float32’); W=torch.from_numpy(W_numpy)<br>
W=torch.arange(0,3)<br>
W=torch.linspace(0,1,3)<br>
W=torch.logspace(-10, 10,3)<br>
W=torch.full((3,), 3.141592)<br>
W=torch.tensor([0.1, 0.1, 0.1])</p>
<p>If I use the option “requires_grad=True” with any of the above expressions, then I got<br>
“RuntimeError: leaf variable has been moved into the graph interior”</p>
<p>Then I run:</p>
<p>W=torch.tensor([0.1, 0.1, 0.1])<br>
W.requires_grad<br>
Out[17]: False<br>
W.is_leaf<br>
Out[18]: True<br>
W[0]=w1 * w2;  W[1]=w2 * w3;  W[2]=w3 * w1<br>
W.is_leaf<br>
Out[19]: False</p>
<p>W=torch.tensor([0.1, 0.1, 0.1], requires_grad=True)<br>
W.requires_grad<br>
Out[20]: True<br>
W.is_leaf<br>
Out[21]: True<br>
W[0]=w1 * w2;  W[1]=w2 * w3;  W[2]=w3 * w1<br>
W.is_leaf<br>
Out[22]: False</p>
<p>So, what the error message really means?</p>

ptrblck: <p>You can read a nice explanation of leaf variables in <a href="https://discuss.pytorch.org/t/leaf-variable-was-used-in-an-inplace-operation/308/2?u=ptrblck">this post</a>.</p>
<p>Usually you have to avoid modifying a leaf variable with an in-place operation.<br>
I haven’t seen your error message yet, but it seems it related to an in-place modification, because you are re-assigning the values of <code>W</code> in this line:</p>
<pre><code>W[0]=w1 * w2; W[1]=w2 * w3; W[2]=w3 * w1</code></pre>

donkeysaddle: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="18679">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I haven’t seen your error message yet, but it seems it related to an in-place modification, because you are re-assigning the values of <code>W</code> in this line:</p>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>Is that an in-place modification though?</p>
<pre><code class="lang-auto">
X=torch.randn((100,3))
Y=torch.randn((100))

w1=torch.tensor(0.1, requires_grad=True)
w2=torch.tensor(0.1, requires_grad=True)
w3=torch.tensor(0.1, requires_grad=True)

W=torch.tensor([0.1, 0.1, 0.1])
print(id(W[0]),id(W[1]),id(W[2]))
W[0]=w1 * w2; W[1]=w2 * w3; W[2]=w3 * w1
print(id(W[0]),id(W[1]),id(W[2]))

4778466184 4778466184 4778466184
4778467120 4778467120 4778467120
</code></pre>
<p>The addresses are different. In fact, I’m not even sure why each of the elements of the tensor, W, don’t have unique addresses, at least prior to being assigned values based on the w’s. Once tensor elements are assigned values, particularly values based on tensors with requires_grad=True, it seems that they all are given the same address. Is that correct?</p>

ptrblck: <p>I’m not sure which <code>id</code> is shown if you access the tensor, so I would compare the <code>ids</code> of the complete tensor <code>W</code> which stay the same.</p>

marwa: <p>I am using PyTorch 0.4.1<br>
I am trying to create a cost-sensitive Loss by modifying the code source of cross entropy loss.<br>
I wrote new Log_Softmax function as described below</p>
<p>def LogSoftmax(input,labels,ksi):<br>
# input = the output layer of my model (nn.Linear(N,nb_classes)<br>
# labels = target classes<br>
# ksi = cost sensitive matrix<br>
layer = torch.empty(input.shape,requires_grad=True)<br>
for j in range(input.shape[0]):<br>
for i in range(input.shape[1]):<br>
layer[j,i] = log(ksi[labels[i],i]*exp(input[j,i])) - log(sum([ksi[labels[i],k]*exp(input[j,k]) for k in range(len(ksi))]))<br>
return layer</p>
<p>input, labels and ski are tensors and have requires_grad set to True<br>
I got error message for loss.backward() during training phase.<br>
For my function’s return ‘layer’:<br>
If requires_grad = True, I got the following error msg: RuntimeError: leaf variable has been moved into the graph interior.<br>
If requires_grad = False, RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn.</p>
<p>Thank you</p>

ptrblck: <p>I think your <code>input</code> should require gradients as it’s most likely a result of some other model operation.<br>
If so, I assume you don’t need to set <code>required_grad=True</code> for <code>layer</code>.<br>
Could you post some random inputs for all tensors, so that I could debug it?</p>
<p>Also, I’m not completely sure how your code works, but maybe the nested loop might be avoided using matrix operations.</p>

marwa: <p>Thank you for your reply.<br>
My input is in fact the output of my model ie the last fully connected layer (nn.Linear(…,nb_classes)). It works well when using criterion = nn.CrossEntropyLoss(). The problem appears only when using the loss function I have implemented.<br>
If I give my model a batch composed of 2 images for instance, I obtain the following :<br>
input = tensor([[-0.4151,  0.0702, -0.4210,  0.4268,  0.0183,  0.1305, -0.5303, -0.1792,<br>
-0.6745, -0.0481, -0.2302],<br>
[-0.0338, -0.3070, -0.8314,  0.3735,  0.4717, -0.2299, -0.4984, -0.1337,<br>
-0.3467,  0.1117, -0.2985]], grad_fn=&lt;ThAddmmBackward))<br>
labels = tensor([0,10])<br>
ksi = = torch.tensor([[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.],<br>
[1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.]],requires_grad = True)<br>
At the moment, my cost sensitive matrix is ones matrix because I just want to debug the code first.<br>
I have  slightly modified my LogSoftmax function:<br>
def LogSoftmax(input,labels,ksi):<br>
layer = torch.empty(input.shape)<br>
for j in range(input.shape[0]):<br>
for i in range(input.shape[1]):<br>
layer[j,i] = log(ksi[labels[j],i]*exp(input[j,i])) - log(sum([ksi[labels[j],k]*exp(input[j,k]) for k in range(len(ksi))]))<br>
return layer</p>

marwa: <p>I have resolved the problem this way</p>
<p>def LogSoftmax(input,labels,ksi):<br>
layer = torch.autograd.Variable(torch.empty(input.shape), requires_grad = True)<br>
layer1 = layer.clone()<br>
for j in range(input.shape[0]):<br>
for i in range(input.shape[1]):<br>
layer1[j,i] = torch.log(ksi[labels[j],i]*torch.exp(input[j,i])) - torch.log(sum([ksi[labels[j],k]*torch.exp(input[j,k]) for k in range(len(ksi))]))<br>
return layer1</p>

saba: <p>Hi Ptrblck,</p>
<p>Would you please help me with that error regarding backward.() ("RuntimeError: leaf variable has been moved into the graph interior<br>
" )</p>
<pre><code class="lang-auto">      bbb=fake.squeeze(1)
        bbb1=MASKGaussy.squeeze(1)
        
        zzz1=torch.zeros(bbb.shape,requires_grad=True)
        zzz=torch.zeros(bbb1.shape,requires_grad=True)

        for ii in range(bbb.shape[0]):
            for ii1 in range(bbb.shape[1]):
                for ii2 in range(bbb.shape[2]):
                    if bbb[ii,ii1,ii2]&gt;=0.99:
                       zzz[ii,ii1,ii2]=1
                    elif  bbb[ii,ii1,ii2]&lt;0.99:
                       zzz[ii,ii1,ii2]=0
            
        for ii in range(bbb1.shape[0]):
            for ii1 in range(bbb1.shape[1]):
                for ii2 in range(bbb1.shape[2]):
                    if bbb1[ii,ii1,ii2]&gt;=0.99:
                       zzz1[ii,ii1,ii2]=1
                    elif  bbb1[ii,ii1,ii2]&lt;0.99:
                        zzz1[ii,ii1,ii2]=0
                        
        L1=nn.MSELoss()(zzz,zzz1)
        loss2=L1
        loss2.backward()
</code></pre>