Getting an error while optimizing for a parameter: can't optimize a non-leaf Tensor

ArshadIram: <p>I am trying to optimize over the value z and netG is trained GAN model.</p>
<p>Here is the code. I am getting an error. I want to min the loss between the generated images x and the z. something like this:</p>
<pre><code class="lang-auto">maxEpochs = 100
z = nn.Parameter(torch.rand(64,100,1,1), requires_grad=True).to(device)
opt = torch.optim.Adam([z]).to(device)

for e in range(maxEpochs):
    netG.eval()
    g_z = netG(z)
    generatedImages = fakeimages.to(device)
    loss = nn.MSELoss()(generatedImages, g_z)
    # calculate gradient
    opt.zero_grad()
    loss.backward()
    opt.step()

    losses['rec'].append(loss)
    if e % 100 == 0:
        print( '[%d] epoch: %0.5f, Loss: %0.5f' % (e, loss))

I am getting the error     

 ValueError("can't optimize a non-leaf Tensor")
Can anyone help me out in resolving the error?</code></pre>

KFrank: <p>Hi Iram!</p>
<p>Let’s start with the first three lines of the code you posted:</p>
<aside class="quote no-group" data-username="ArshadIram" data-post="1" data-topic="128959" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arshadiram/48/32886_2.png" class="avatar"> ArshadIram:</div>
<blockquote>
<pre><code class="lang-auto">maxEpochs = 100
z = nn.Parameter(torch.rand(64,100,1,1), requires_grad=True).to(device)
opt = torch.optim.Adam([z]).to(device)
</code></pre>
</blockquote>
</aside>
<p>Let me assume that <code>device</code> refers to a gpu.</p>
<p>Your second line of code creates a tensor on the cpu and then<br>
“moves” it to the gpu by creating a gpu copy of the original cpu<br>
tensor.  Although the original cpu tensor is a “leaf” tensor, the<br>
copy-to-gpu operation counts as a “computation” so that the<br>
gpu tensor is <em>not</em> a leaf tensor, hence the error you report:</p>
<aside class="quote no-group">
<blockquote>
<p>I am getting the error</p>
<p>ValueError(“can’t optimize a non-leaf Tensor”)</p>
</blockquote>
</aside>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; x = torch.nn.Parameter(torch.rand(64,100,1,1), requires_grad=True)
&gt;&gt;&gt; x.is_cuda
False
&gt;&gt;&gt; x.is_leaf
True
&gt;&gt;&gt; y = torch.nn.Parameter(torch.rand(64,100,1,1, device = 'cuda'), requires_grad=True)
&gt;&gt;&gt; y.is_cuda
True
&gt;&gt;&gt; y.is_leaf
True
&gt;&gt;&gt; z = torch.nn.Parameter(torch.rand(64,100,1,1), requires_grad=True).to ('cuda')
&gt;&gt;&gt; z.is_cuda
True
&gt;&gt;&gt; z.is_leaf
False
</code></pre>
<p>As an aside, your third line of code, as posted, is fully bogus, and<br>
will throw an error, even if you try to construct your <code>Adam</code> optimizer<br>
with a leaf tensor.  (In general, a pytorch <code>Optimizer</code> doesn’t have a<br>
<code>.to()</code> method.)</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; opt = torch.optim.Adam([x]).to ('cuda')
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: 'Adam' object has no attribute 'to'
</code></pre>
<p>Construct the <code>Parameter</code> you wish to optimize directly on the gpu, as<br>
is done with <code>y</code> in the example I gave above, and then use it to construct<br>
the <code>Optimizer</code> (but don’t attempt to “move” the <code>Optimizer</code>):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; opt = torch.optim.Adam([y])
&gt;&gt;&gt; opt.param_groups[0]['params'][0].shape
torch.Size([64, 100, 1, 1])
&gt;&gt;&gt; opt.param_groups[0]['params'][0].is_cuda
True
&gt;&gt;&gt; opt.param_groups[0]['params'][0].is_leaf
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

ArshadIram: <p>Yes, thank you so much for your reply. I am trying to learn it. Thanks again for your great help. Can you please answer this query as well?</p>
<aside class="quote quote-modified" data-post="2" data-topic="128336">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-solve-this-optimization-problem/128336/2">How to solve this optimization problem?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Iram! 

The equation you posted an image of means that you are to find 
the values of z[i] that minimize MSELoss (x[?], G (z)), while 
holding the parameters that define G constant. 
(I believe that the use of i as the index on both sides of the 
equation is a notational inconsistency – that’s why I used ? as 
the index for x – but please explain if you believe otherwise.) 
To use pytorch to do this, you would use pytorch’s autograd and 
gradient-descent-optimization machinery to minimize you…
  </blockquote>
</aside>


ArshadIram: <p>Can you explain me about param_groups[0][‘params’][0] ?</p>

KFrank: <p>Hi Iram!</p>
<aside class="quote no-group" data-username="ArshadIram" data-post="4" data-topic="128959" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arshadiram/48/32886_2.png" class="avatar"> ArshadIram:</div>
<blockquote>
<p>Can you explain me about param_groups[0][‘params’][0] ?</p>
</blockquote>
</aside>
<p>Looking at an object’s <code>__dict__</code> property is often a good way to<br>
probe all of its properties.  So start with:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; opt = torch.optim.Adam ([torch.randn (3, requires_grad = True)])
&gt;&gt;&gt; opt.__dict__
{'defaults': {'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False}, '_zero_grad_profile_name': 'Optimizer.zero_grad#Adam.zero_grad', 'state': defaultdict(&lt;class 'dict'&gt;, {}), 'param_groups': [{'params': [tensor([0.2824, 0.7790, 0.3820], requires_grad=True)], 'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False}]}
</code></pre>
<p>and drill down deeper from there.</p>
<p>Best.</p>
<p>K. Frank</p>

Harshtherocking: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="128959">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Although the original cpu tensor is a “leaf” tensor, the<br>
copy-to-gpu operation counts as a “computation” so that the<br>
gpu tensor is <em>not</em> a leaf tensor</p>
</blockquote>
</aside>
<p>Thanks. I was having the same problem with the copied tensor not being considered a leaf node.</p>