ValueError: can't optimize a non-leaf Tensor

username: ProGamerGov
cooked: <p>The following code meant to create an image, and allow for backpropagation on the method used to create the image.</p>
<p>The code is meant to be used with a pretrained CNN like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8.png" data-download-href="https://discuss.pytorch.org/uploads/default/418ea0fe461d50a6ab99a757f786b6bc45fc4db8" title="example"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8_2_517x166.png" alt="example" data-base62-sha1="9lWCdZXbkx479Xx3Z99t6YLXhoc" width="517" height="166" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8_2_517x166.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8_2_775x249.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8.png 2x" data-dominant-color="EDE8E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">example</span><span class="informations">854×275 46.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I run the code below:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

def create_input(input_size, mode=0):
    if mode == 0:
        if type(input_size) is not tuple and type(input_size) is not list:
            input_size = (input_size, input_size) 
        w = torch.arange(0, input_size[1])
        h = torch.arange(0, input_size[0]) 
        w_exp = w.unsqueeze(1).expand((input_size[1], input_size[0])).true_divide(input_size[0]) - 0.5
        h_exp = h.unsqueeze(0).expand((input_size[1], input_size[0])).true_divide(input_size[1]) - 0.5
        return torch.stack((w_exp, h_exp), -1).permute(2,1,0).unsqueeze(0)
    elif mode == 1: # TensorFlow/Lucid Creation method
        if type(input_size) is tuple or type(input_size) is list:
            input_size = input_size[0]
        r = 3.0**0.5
        coord_range = torch.linspace(-r, r, input_size)
        y, x = torch.meshgrid(coord_range, coord_range)
        tensor = torch.stack((x, y), -1).unsqueeze(0).permute(0,3,1,2)
        return tensor

def cppn_image(size, num_channels=16, num_layers=9):
    tensor = nn.Parameter(create_input(size))
    weight_val = nn.Parameter(torch.randn(num_channels,2,1,1))
    tensor = F.conv2d(tensor, weight_val).tanh()
    for i in range(num_layers):
        weight_val = nn.Parameter(torch.randn(num_channels,num_channels,1,1))
        tensor = F.conv2d(tensor, weight_val).tanh()
    weight_val = nn.Parameter(torch.randn(3,num_channels,1,1))
    tensor = F.conv2d(tensor, weight_val).sigmoid()
    return tensor


img = cppn_image((512,460))
optimizer = optim.Adam([img])
</code></pre>
<p>I get this error message:</p>
<pre><code class="lang-auto">    raise ValueError("can't optimize a non-leaf Tensor")
ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>What am I doing wrong here? And how can I fix it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Why do you pass the image to an optimizer?<br>
Keep in mind that optimizer’s steps are not differentiable and no gradients will be able to flow back.</p>

username: ProGamerGov
cooked: <p><a class="mention" href="/u/alband">@albanD</a></p>
<blockquote>
<p>Why do you pass the image to an optimizer?</p>
</blockquote>
<p>I’m using this setup for things like neural style transfer and DeepDream visualization, which requires a single image:</p>
<pre><code class="lang-auto">    img = nn.Parameter(img)
    num_iterations = 1000

    num_calls = [0]
    def feval():
        num_calls[0] += 1
        optimizer.zero_grad()
        net(img) # Run input through CNN
        loss = 0

        for mod in loss_list:
            loss += mod.loss # Get loss from layers with nn.MSELoss() loss variable

        loss.backward()

        print('Iteration', num_calls[0], 'loss', loss.item())
        return loss

    optimizer = optim.Adam([img], lr = learning_rate)
    while num_calls[0] &lt;= num_iterations:
         optimizer.step(feval)
</code></pre>
<p>This strategy was done a lot in the original Torch7, and “feval” was what the “closure” function was called back then.</p>
<blockquote>
<p>Keep in mind that optimizer’s steps are not differentiable and no gradients will be able to flow back.</p>
</blockquote>
<p>So, how would I make this mini network that I created using Functional.Conv2d work then?</p>
<pre><code class="lang-auto">    weight_val = nn.Parameter(torch.randn(num_channels,2,1,1))
    tensor = F.conv2d(tensor, weight_val).tanh()
    for i in range(num_layers):
        weight_val = nn.Parameter(torch.randn(num_channels,num_channels,1,1))
        tensor = F.conv2d(tensor, weight_val).tanh()
    weight_val = nn.Parameter(torch.randn(3,num_channels,1,1))
    tensor = F.conv2d(tensor, weight_val).sigmoid()
</code></pre>
<p>I was originally using a class (with <code>nn.Conv2d</code>), but that didn’t seem to work.</p>

username: ProGamerGov
cooked: <p>These two classes do the same thing as the <code>cppn_image()</code> function above, but I don’t think that I can use them for what I am trying to do?</p>
<pre><code class="lang-auto">class CPPN_Conv(nn.Module):

    def __init__(self, size, num_channels=16, num_layers=6, activ_func=nn.Tanh()):
        super(CPPN_Conv, self).__init__()
        self.size = size
        self.net = self.create_net(num_channels, num_layers, activ_func) 		
				
    def create_net(self, num_channels, num_layers, activ_func):
        net = nn.Sequential()
        net.add_module(str(len(net)), nn.Conv2d(in_channels=2, out_channels=num_channels, kernel_size=1))
        net.add_module(str(len(net)), activ_func)
        for l in range(num_layers - 1):
            net.add_module(str(len(net)), nn.Conv2d(in_channels=num_channels, out_channels=num_channels, kernel_size=1))
            net.add_module(str(len(net)), activ_func)
        net.add_module(str(len(net)), nn.Conv2d(in_channels=num_channels, out_channels=3, kernel_size=1))
        net.add_module(str(len(net)), nn.Sigmoid())
        net.apply(self.cppn_normal)
        return net

    def cppn_normal(self, l):
        if type(l) == nn.Conv2d:        
            nn.init.normal_(l.weight)
			
    def forward(self):
        input = create_input(self.size)
        return self.net(input)
		

cppn_c = CPPN_Conv((512,460))
img = cppn_c()



class CPPN_Linear(nn.Module):

    def __init__(self, size, num_channels=16, num_layers=6, activ_func=nn.Tanh()):
        super(CPPN_Linear, self).__init__()
        if type(size) is not tuple:
            size = (size, size)
        self.input_size = size
        self.net = self.create_net(num_channels, num_layers, activ_func)
                
    def create_net(self, num_channels, num_layers, activ_func):
        net = nn.Sequential()
        net.add_module(str(len(net)), nn.Linear(2, num_channels, bias=True))
        net.add_module(str(len(net)), activ_func)
        for l in range(num_layers - 1):
            net.add_module(str(len(net)), nn.Linear(num_channels, num_channels, bias=False))
            net.add_module(str(len(net)), activ_func)
        net.add_module(str(len(net)), nn.Linear(num_channels, 3, bias=False))
        net.add_module(str(len(net)), nn.Sigmoid())
        net.apply(self.cppn_normal)
        return net
        
    def cppn_normal(self, l):
        if type(l) == nn.Linear:        
            nn.init.normal_(l.weight)

    def create_input(self, input_size): 
        w = torch.arange(0, input_size[1])
        h = torch.arange(0, input_size[0]) 
        w_exp = w.unsqueeze(1).expand((input_size[1], input_size[0])).true_divide(input_size[0]) - 0.5
        h_exp = h.unsqueeze(0).expand((input_size[1], input_size[0])).true_divide(input_size[1]) - 0.5
        return torch.stack((w_exp, h_exp), -1).reshape(input_size[1] * input_size[0], 2) # nn.Linear requires a different input shape

    def forward(self):
        input = self.create_input(self.input_size)
        return self.net(input).reshape(self.input_size[1], self.input_size[0], 3).permute(2,1,0).unsqueeze(0)
        
      
cppn_l = CPPN_Linear((512,460))
img = cppn_l()
</code></pre>

username: albanD
cooked: <p>You can do the same as the example you show here.<br>
In this example, <code>img</code> is a leaf (it was not created in a differentiable manner, it is just a Tensor that you optmize over) so it will also work with pytorch if you do that.</p>