Leaf variable was used in an inplace operation

OswinLG: 

colesbury: Loosely, tensors you create directly are leaf variables. Tensors that are the result of a differentiable operation are not leaf variables For example: <SCODE>w = torch.tensor([1.0, 2.0, 3.0]) # leaf variable
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True) # also leaf variable
y = x + 1  # not a leaf variable
<ECODE> An in-place operation is something which modifies the data of a variable. For example: <SCODE>x += 1  # in-place
y = x + 1 # not in place
<ECODE> <SCODE>x2 = x.clone()  # clone the variable
x2 += 1  # in-place operation
<ECODE> <SCODE>with torch.no_grad():
    x += 1
<ECODE>

henrye: Is it bad practice to get around pytorch disallowing in-place operations by assigning to Variable().data?

tom: Best regards Thomas

henrye: Yes, I remember now, messing with backprop and autograd was why I was running into problems with in-place assignment before. Using .data as I am currently for initialising word embeddings seems ok then.

Yoni_Keren: Hi, Trying to read the code for optim (I want to implement something a bit differently) and your previous example/explanation of what is a leaf Variable doesn’t seem to be valid anymore. <SCODE>y = x + 1  # not a leaf variable
<ECODE> Well, here’s output from my termial for the code which you have mentioned: <SCODE>>>> x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]))
>>> x.is_leaf
True
>>> y = x + 1
>>> y.is_leaf
True
>>> y
Variable containing:
 2
 3
 4
 5
[torch.FloatTensor of size (4,)]
<ECODE> So, can someone please explain what is a leaf Variable, and what is not a leaf variable? Clearly a non-leaf-variable cannot be optimized, but what is it?

Ed_Beeching: <SCODE>Came across a similar issue. Reason is requires grad.

x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4]), requires_grad=True)
x.is_leaf    
#True
y = x + 1
y.is_leaf
#False<ECODE>

11185: Hi, I used to create leaf variable like: y = torch.autograd.Variable(torch.zeros([batch_size, c, h, w]), requires_grad=True) Then I want to assign value to indexed parts of y like below,(y_local is a Variable computed based on other variables and I want to assign the value of y_local to part of the y and ensure that the gradients from y can flow to the y_local.) y.data[:,:,local_x[i]:local_x[i+1],local_y[i]:local_y[i+1]] = y_local.data I am wondering such operation supports the normal gradient backward for the y_local varible?

shivangi: I am also facing similar issue. Please let me know how were you able to resolve it

dreamyun: leaf variable, in essence, is a variable, or a tensor with requires_grad=True. So, if a tensor with requires_grad=False, it does not belong to the variable, let alone leaf variable.

klory: 

klory: Could you please explain why it’s not correct?

danmoller: 

nima_rafiee: I had a similar case. I used: <SCODE>y = torch.zeros([batch_size, c, h, w]), requires_grad=False)
<ECODE> then I update the value of y according to the value of the network output and then apply a loss function on y and it worked for me.

pinocchio: Is the reason this is not “usually correct” because we could have just initialized it directly with the data that we wanted in the first place instead of doing a in-place op?

pinocchio: I also agree it shouldn’t be a leaf but Pytorch disagrees with us…why?

colesbury: 

pinocchio: <SCODE>def inplace_playground():
    import torch

    x = torch.tensor([1,2,3.], requires_grad=True)
    y = x + 1
    print(f'x.is_leaf = {x.is_leaf}')
    print(f'y.is_leaf = {y.is_leaf}')
    x += 1
<ECODE> output: <SCODE>x.is_leaf = True
y.is_leaf = False
<ECODE> Thanks!

pinocchio: Why not? What were the competing semantics? What’s the difficulty in defining the semantics for leafs + in-place ops?

jeremysun1224: Perhaps, you could try adding the following code before updating the gradient: <SCODE>with torch.no_grad():
<ECODE> and it works: <SCODE>N, D_in, H, D_out = 64, 1000, 100, 10

X = torch.randn(N, D_in).cuda()
y = torch.randn(N, D_out).cuda()

# device = torch.cuda.device('cuda:0')
device = torch.device('cuda:0')
w1 = torch.randn(D_in, H, requires_grad=True, device=device)
w2 = torch.randn(H, D_out, requires_grad=True, device=device)

learning_rate = 1e-6
for it in range(10):
    # 1.forward pass
    y_pred = X.mm(w1).clamp(min=0).mm(w2).cuda()
    # 2.compute loss
    loss = (y_pred - y).pow(2).sum().cuda()
    print(f'iter {it}, loss {loss}')
    # Backward pass
    loss.backward()
    # update weights of w1 and w2
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad
        w1.grad.zero_()
        w2.grad.zero_()
<ECODE>