How to use network as an input parameter instead of a tensor for an optimizer?

username: ProGamerGov
cooked: <p>According to <a href="https://distill.pub/2018/differentiable-parameterizations/" rel="nofollow noopener">this post using TensorFlow</a> (<a href="https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/differentiable-parameterizations/xy2rgb.ipynb#scrollTo=GS4kqcEJuy5R" rel="nofollow noopener">code here</a>), you can use a simple Compositional Pattern Producing Network (CPPN) as an input parameter for a classification network. By using the CPPN as the input parameter, you can create really smooth looking visualizations based on neuron activation.</p>
<p>I tried using the following in PyTorch, and while the loss seems to go down, nothing else changes:</p>
<pre><code>       optimizer = optim.Adam([
       {'params': cppn_net.parameters()}], lr = learning_rate)  # Replaced image tensor
</code></pre>
<p>This is what <code>cppn_net</code> looks like:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

		
def cppn_normal(l):
    if type(l) == nn.Linear:        
        nn.init.normal_(l.weight)

		
# Regular CPPN         
class CPPN(nn.Module):

    def __init__(self, size, input_range=1, num_channels=16, num_layers=6, activ_func=nn.Tanh()):
        super(CPPN, self).__init__()
        if type(size) is not tuple:
            size = (size, size)
        self.input_size = size
        self.input_range = input_range
        self.net = nn.Sequential()
        self.net.add_module(str(len(self.net)), nn.Linear(2, num_channels, bias=True))
        self.net.add_module(str(len(self.net)), activ_func)
        for l in range(num_layers - 1):
             self.net.add_module(str(len(self.net)), nn.Linear(num_channels, num_channels, bias=False))
             self.net.add_module(str(len(self.net)), activ_func)
        self.net.add_module(str(len(self.net)), nn.Linear(num_channels, 3, bias=False))
        self.net.add_module(str(len(self.net)), nn.Sigmoid())
        
    def create_input(self): 
        w = torch.arange(0, self.input_size[1])
        h = torch.arange(0, self.input_size[0]) 
        w_exp = w.unsqueeze(1).expand((self.input_size[1], self.input_size[0])).true_divide(self.input_size[0]) - 0.5
        h_exp = h.unsqueeze(0).expand((self.input_size[1], self.input_size[0])).true_divide(self.input_size[1]) - 0.5
        return torch.stack((w_exp, h_exp), -1).reshape(self.input_size[1] * self.input_size[0], 2)

    def forward(self):
        input = self.create_input()
        input.requires_grad= True
        return (self.net(input).reshape(self.input_size[1], self.input_size[0], 3).permute(2,1,0) * self.input_range).unsqueeze(0)
		
		
cppn_net = CPPN(size=(512, 512)).apply(cppn_normal)
</code></pre>
<p>I’m not sure if I need to setup the CPPN network differently, change how I’m integrating it with the main network, or do something else entirely?</p>

username: Ge0rges
cooked: <p>It is possible to subclass any given optimizer.</p>

username: ProGamerGov
cooked: <p>So, I have to make changes to the optimizer class for it to work?</p>

username: Ge0rges
cooked: <p>I’m unsure if there’s a built-in way to do this, but I’m sure you can make it work by subclassing.</p>

username: ProGamerGov
cooked: <p>So, looking at the example image, I think that I could probably use a CPPN as a function instead of a nn.Sequential.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8.png" data-download-href="https://discuss.pytorch.org/uploads/default/418ea0fe461d50a6ab99a757f786b6bc45fc4db8" title="example"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8_2_690x222.png" alt="example" data-base62-sha1="9lWCdZXbkx479Xx3Z99t6YLXhoc" width="690" height="222" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8_2_690x222.png, https://discuss.pytorch.org/uploads/default/original/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/1/418ea0fe461d50a6ab99a757f786b6bc45fc4db8.png 2x" data-dominant-color="EDE8E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">example</span><span class="informations">854×275 46.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Though I am not sure if creating Conv2d weights in a loop like this is a good idea, or if I need to do it a different way?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

def create_input(input_size, mode=0):
    if mode == 0:
        if type(input_size) is not tuple and type(input_size) is not list:
            input_size = (input_size, input_size) 
        w = torch.arange(0, input_size[1])
        h = torch.arange(0, input_size[0]) 
        w_exp = w.unsqueeze(1).expand((input_size[1], input_size[0])).true_divide(input_size[0]) - 0.5
        h_exp = h.unsqueeze(0).expand((input_size[1], input_size[0])).true_divide(input_size[1]) - 0.5
        return torch.stack((w_exp, h_exp), -1).permute(2,1,0).unsqueeze(0)
    elif mode == 1: # TensorFlow/Lucid Creation method
        if type(input_size) is tuple or type(input_size) is list:
            input_size = input_size[0]
        r = 3.0**0.5
        coord_range = torch.linspace(-r, r, input_size)
        y, x = torch.meshgrid(coord_range, coord_range)
        tensor = torch.stack((x, y), -1).unsqueeze(0).permute(0,3,1,2)
        return tensor

def cppn_image(size, num_channels=16, num_layers=9):
    tensor = create_input(size)
    weight_val = nn.Parameter(torch.randn(num_channels,2,1,1))
    tensor = F.conv2d(tensor, weight_val).tanh()
    for i in range(num_layers):
        weight_val = nn.Parameter(torch.randn(num_channels,num_channels,1,1))
        tensor = F.conv2d(tensor, weight_val).tanh()
    weight_val = nn.Parameter(torch.randn(3,num_channels,1,1))
    tensor = F.conv2d(tensor, weight_val).sigmoid()
    return tensor


output = cppn_image((512,460))
</code></pre>