ValueError: can't optimize a non-leaf Tensor?

dragen: <p>Dear all:<br>
i know when</p>
<pre data-code-wrap="python"><code class="lang-python">  x_cuda = x_cpu.to(device)
</code></pre>
<p>It will trigger error:</p>
<blockquote>
<p>ValueError: can’t optimize a non-leaf Tensor</p>
</blockquote>
<p>when you use <code>optimizer = optim.Adam([x_cuda])</code>. The right way may be <code>optimizer = optim.Adam([x_cpu])</code>. That’s to way, we need keep both reference of <code>x_cpu</code> and <code>x_cuda</code>.</p>
<p>Since in most case, our program will only keep a reference of the <code>cuda</code> version of tensor, such as :</p>
<pre data-code-wrap="python"><code class="lang-python">		self.vars = [
			# [28*28, 512]
			torch.ones(512, 2 * 28 * 28, requires_grad=True).to(device),
			torch.zeros(512, requires_grad=True).to(device),
			# [512, 256]
			torch.ones(256, 512, requires_grad=True).to(device),
			torch.zeros(256, requires_grad=True).to(device),
			# [256, n]
			torch.ones(n_class, 256, requires_grad=True).to(device),
			torch.zeros(n_class, requires_grad=True).to(device)
		]
</code></pre>
<p>So i wonder how to pass the parameters to <code>optimizer</code> when We dnt want to keep the reference of <code>cpu</code> version of tensor?</p>

albanD: <p>Hi,</p>
<p>A leaf Variable is a variable that is at the beginning of the graph. That means that no operation tracked by the autograd engine created it.<br>
This is what you want when you optimize neural networks as it is usually your weights or input.</p>
<p>So to be able to give weights to the optimizer, they should follow the definition of leaf variable above.</p>
<pre><code class="lang-auto">a = torch.rand(10, requires_grad=True) # a is a leaf variable
a = torch.rand(10, requires_grad=True).double() # a is NOT a leaf variable as it was created by the operation that cast a float tensor into a double tensor
a = torch.rand(10).requires_grad_().double() # equivalent to the formulation just above: not a leaf variable
a = torch.rand(10).double() # a does not require gradients and has not operation creating it (tracked by the autograd engine).
a = torch.rand(10).doube().requires_grad_() # a requires gradients and has no operations creating it: it's a leaf variable and can be given to an optimizer.
a = torch.rand(10, requires_grad=True, device="cuda") # a requires grad, has not operation creating it: it's a leaf variable as well and can be given to an optimizer
</code></pre>
<p>So in your case, you want to use the last line.</p>

dragen: <p><a class="mention" href="/u/alband">@albanD</a> Thanks, you solved my problem in another approach.<br>
However, If i must write tensor as <code>a = torch.tensor(..).to(device)</code>, like the following linear regression example:</p>
<pre data-code-wrap="python"><code class="lang-python">def main2():

	device = torch.device('cuda')
	lr = 5e-2

	w_ = torch.tensor(1., requires_grad=True)
	b_ = torch.tensor(0.1, requires_grad=True)
	w = w_.to(device)
	b = b_.to(device)


	criteon = nn.MSELoss()
	optimizer = optim.Adam([w_, b_], lr=lr)

	for i in range(500):
		x = torch.rand(1).to(device)[0]


		pred = w * x + b
		y = 2 * x + 3
		loss = criteon(pred, y)

		grads = torch.autograd.grad(loss, [w_, b_])
		w_.grad.fill_(grads[0])
		b_.grad.fill_(grads[1])
		optimizer.step()

		print(w_, b_)
</code></pre>
<p>It print the Error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “/home/i/ncrs/aaai/fsgan/test2.py”, line 94, in <br>
main2()<br>
File “/home/i/ncrs/aaai/fsgan/test2.py”, line 60, in main2<br>
w_.grad.fill_(grads[0])<br>
AttributeError: ‘NoneType’ object has no attribute ‘fill_’</p>
</blockquote>
<p>What should I do? How to make the above episode code work as normal?</p>

albanD: <p>The <code>.grad</code> field are allocated lazily. So if they are <code>None</code>, you can do <code>w_.grad = grads[0]</code> I think.<br>
When they already exist, you don’t want this copy to be tracked by the autograd actualy so you should do:</p>
<pre><code class="lang-auto">grads = torch.autograd.grad(loss, [w_, b_])
with torch.no_grad():
    w_.grad.fill_(grads[0])
    b_.grad.fill_(grads[1])
optimizer.step()
</code></pre>

dragen: <p><a class="mention" href="/u/alband">@albanD</a> Thanks so much.<br>
One more question: I dnt understand why it’s nessary to use <code>with torch.no_grad()</code>?<br>
Could u kindly explain some case going wrong if no use <code>torch.no_grad()</code>?</p>

albanD: <p>Thinking about it, <code>w_.grad = grads[0]</code> is not a great idea either <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"> You should do <code>w_.grad = grads[0].detach()</code>.<br>
Otherwise, same as if you were not using <code>torch.no_grad()</code> you would create a memory leak if <code>grads.requires_grad</code> is True.</p>
<p>Basically, a computational graph is kept in memory as long as it could be used for a backward.<br>
For example if you use a net to compute a scalar <code>loss</code>. Then as long as you have your <code>loss</code> that exist, all the computational graph and intermediary results associated will stay in memory.<br>
If you do <code>w_.grad.fill_(grads[0])</code>, then <code>w_.grad</code> becomes reponsible for the whole graph. And since <code>w_.grad</code> is never deleted, the graph will remain forever.</p>
<p>In your case, <code>grads</code> should not require gradient as you don’t pass the corresponding flags to <code>autograd.grad</code>.</p>
<p>In general, I like it as it makes the code more readable (you are sure this block is not part of your model) and future proof: if one day you change something that makes <code>grads</code> require grad, it will still work and not create a memory leak.</p>

Baka_Moon: <p>How to do so when loading a PIL image?</p>
<p>In my case I am loading a PIL image and then using pre-configured transform object:</p>
<pre><code class="lang-auto">img = Image.open(pretrained_img_path)

        noise = transform(img).unsqueeze(0)

        noise.requires_grad = True

        print(noise.requires_grad)

        noise = noise.to(device)
</code></pre>

albanD: <p>If you want the final <code>noise</code> to be the leaf, you want to do <code>noise.requires_grad = True</code> after moving it to the device with <code> noise = noise.to(device)</code>.</p>

Baka_Moon: <p>Ah. Is there a more “Pytorch” way to do this when loading an image from PIL. Something possible in one line, where I can assign <code>requires grad = True</code> as I do when creating new variables. I couldn’t find anything within the docs here:</p>
<p><a href="https://pytorch.org/docs/stable/torchvision/transforms.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/torchvision/transforms.html</a></p>
<p>I am currently using this transform:</p>
<pre><code class="lang-auto">transform = transforms.Compose([transforms.Resize((img_dim, img_dim)),
                                    transforms.ToTensor(), 
                                    transforms.Normalize([0.485, 0.456, 0.406],
                                                         [0.229, 0.224, 0.225])])
    
</code></pre>
<p>Currently my way of trying to load a PIL image as a trackable tensor seems very sloppy.</p>

sophia_k: <p><a class="mention" href="/u/alband">@albanD</a>  Thank you for the solution in the last line <code>a = torch.rand(10, requires_grad=True, device="cuda")</code> !<br>
How does this work for a <code>torch.nn.Linear(in_features: int, out_features: int, bias: bool = True)</code> when I want to create a tensor with the linear transformation?</p>
<p>Could you explain what should be altered in this example here to create the linear transformation tensor on CUDA?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; m = nn.Linear(20, 30)
&gt;&gt;&gt; input = torch.randn(128, 20)
&gt;&gt;&gt; output = m(input)
</code></pre>

albanD: <p>Hi,</p>
<p>For the input, you can do the same thing: <code>input = torch.randn(128, 20, device="cuda")</code> But for the <code>nn.Module</code> you will have to first create it and then send it to the GPU:</p>
<pre><code class="lang-auto">m = nn.Linear(20, 30)
m.cuda()
</code></pre>