Creating a separate optimizer for a new class object

avalon1511: <p>Hi, I have a generator class and a class for something that is doing a simple math operation on the output of one of the intermediate outputs from the generator layer</p>
<p>So what this looks like is:</p>
<pre><code class="lang-auto">First few generator layers
.
.
.
math operation
.
.
other generator layers
</code></pre>
<p>When I create the optimizer for the generator, the parameters for my math operation object are passed as one of the generator parameters.</p>
<p>I want to be able to create a separate optimizer for this math operation layer and call that during training. I tried applying an instance of the math object within the generator definition but that gave me an error.</p>
<p>Since the generator is wrapped in a <code>Sequential</code> container, I can’t assign any values inside the definition -</p>
<pre><code class="lang-auto">self.mathlayer1 = x + b
</code></pre>
<p>gives me an error.</p>
<p>So how can I define the generator in a way that will let me create a different optimizer for the novel layer?</p>

gphilip: <p>If you could provide executable code which shows what you want, then it would be easier to figure out the issue. If you could make the code as simple as you can while still illustrating the question, that would help, too.</p>
<p>In the example code you could, for instance, have the math operation sandwiched between just two generator layers (I am not sure what these are, from your description …), if such a setup will also illustrate the issue.</p>

avalon1511: <p>Yes of course, my apologies!</p>
<p>My generator code is as follows:</p>
<pre><code class="lang-auto">class Generator(nn.Module):
    def __init__(self, ngpu, b_val):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            ShiftLayer(b_val),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )
    
    def forward(self, input):
        return self.main(input)
</code></pre>
<p>The layer called <code>ShiftLayer</code> is defined as:</p>
<pre><code class="lang-auto">class ShiftLayer(nn.Module):
    def __init__(self, b):
        super().__init__()
        self.b = nn.Parameter(torch.tensor(1.))
        
    def forward(self, x):
        return x + self.b
</code></pre>
<p>During training, I have an optimizer for the generator. Since <code>b</code> is a parameter defined within the generator, when I execute</p>
<p><code>optimizerG = optim.Adam(generator.parameters(), lr=lr, betas=(beta1, 0.999))</code></p>
<p><code>b</code> is returned as one of these <code>generator.parameters()</code>. I want to create a separate optimizer for optimizing<code> b</code>  so that I can do something like:</p>
<pre><code class="lang-auto">optimizerb = optim.SGD(ShiftLayer.parameters(), . . .)
</code></pre>
<p>How can I do this?</p>
<p>(Sorry about the long post, I am not sure how to make it more concise without getting too abstract)</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for the tag, would you have any idea?</p>

gphilip: <p>Your <code>ShiftLayer</code> code just ignores the value of <code>b</code> passed in to its constructor. Is this what you want to do?</p>

avalon1511: <p>No, I want to create an optimizer for <code>b</code> only while using <code>ShiftLayer</code> in my generator</p>

gphilip: <p>As your code currently is:</p>
<pre><code class="lang-auto">class ShiftLayer(nn.Module):
    def __init__(self, b):
        super().__init__()
        self.b = nn.Parameter(torch.tensor(1.))
        
    def forward(self, x):
        return x + self.b
</code></pre>
<p>the value of <code>b</code> passed in to <code>___init___</code> is not used at all. So it is as good as not passing in this value. Perhaps you meant to say</p>
<pre><code class="lang-auto">self.b = nn.Parameter(torch.tensor(b))
</code></pre>
<p>?</p>

gphilip: <p>What are some valid values for the shapes of <code>nz</code> , <code>ngf</code>, and <code>nc</code> ? If you tell me these, I could try figuring out how to optimize for <code>b</code>.</p>

avalon1511: <pre><code class="lang-auto">nz = 100
ngf = 64
nc = 3
</code></pre>
<p>I am just trying to create a built in optimizer for b using<code> torch.optim</code>, from the feedback in my previous posts, for <code>b</code> to be optimized, I had to add <code>self.b = nn.Parameter(torch.tensor(b))</code> in my definition of <code>ShiftLayer()</code>.</p>
<p>As my code is right now, <code>b</code> is being passed as a parameter when I instantiate my generator and then call</p>
<p><code>optimizerG = optim.Adam(generator.parameters(), lr=lr, betas=(beta1, 0.999))</code></p>
<p>All I’m trying to do is instead of optimizing <code>b</code> through the Generator’s optimizer, create an optimizer for itself using <code>torch.optim</code></p>
<p>Hope that makes sense.</p>

gphilip: <p>See if passing the <code>b_param</code> retrieved as follows, to the optimizer, works:</p>
<pre><code class="lang-auto">x = torch.randn(1,10) # Dummy value!
ngpu = True # Dummy value!
nz = 100
ngf = 64
nc = 3
myGen = Generator(True, x)
shiftLayer = myGen.main[11] # I got the 11 by looking at the output of print(myGen) 
_, b_param = next(shiftLayer.named_parameters())
print(b_param)
</code></pre>

avalon1511: <p>Seems like i am getting an error:</p>
<pre><code class="lang-auto"> File "mnist_film.py", line 299, in &lt;module&gt;              
 _, b_param = next(shiftLayer.named_parameters())                                                                               
StopIteration   
</code></pre>

gphilip: <p><a href="https://docs.python.org/3/library/exceptions.html#StopIteration" rel="noopener nofollow ugc"><code>StopIteration</code> means there are no more things for <code>next</code> to return from its argument.</a></p>
<p>Are you running this line within a loop? This won’t work the second time, since <code>shiftLayer</code> has only one named parameter.</p>
<p>Edited to add: If you need to have this line within a loop, you can replace it with</p>
<pre><code class="lang-auto">_, b_param = list(shiftLayer.named_parameters())[0]
</code></pre>

avalon1511: <p>Nope it is not inside a loop! I think for some reason <code>shiftlayer.named_parameters()</code> is not returning a value</p>
<p>Edit: I printed all of the parameters of the generator object and it seems like <code>b</code> is not considered a parameter. Confused about what’s going on</p>

ptrblck: <aside class="quote no-group" data-username="avalon1511" data-post="12" data-topic="132000">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/avalon1511/48/45884_2.png" class="avatar"> avalon1511:</div>
<blockquote>
<p>I printed all of the parameters of the generator object and it seems like <code>b</code> is not considered a parameter. Confused about what’s going on</p>
</blockquote>
</aside>
<p>I don’t know why this would happen, as <code>b</code> is returned as a parameter in all of these implementations:</p>
<pre data-code-wrap="python"><code class="lang-python">class ShiftLayer(nn.Module):
    def __init__(self, b):
        super().__init__()
        self.b = nn.Parameter(torch.tensor(b))
        
    def forward(self, x):
        return x + self.b
    
shift = ShiftLayer(1.)
print(list(shift.parameters()))
print(dict(shift.named_parameters()))

model = nn.Sequential(
    nn.Linear(1, 1),
    ShiftLayer(2.)
)
print(list(model.parameters()))
print(dict(model.named_parameters()))



class Generator(nn.Module):
    def __init__(self, b_val):
        super(Generator, self).__init__()
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            ShiftLayer(b_val),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )
    
    def forward(self, input):
        return self.main(input)

nz = 100
ngf = 64
nc = 3
gen = Generator(10.)

for name, param in gen.named_parameters():
    if 'b' in name and 'bias' not in name:
        print(name, param)
&gt; main.11.b Parameter containing:
  tensor(10., requires_grad=True)
</code></pre>

avalon1511: <p>That’s what I am thinking. I am going to double check my code again, maybe I left something there by mistake like last time</p>

avalon1511: <p>Hey <a class="mention" href="/u/gphilip">@gphilip</a> , seems like I’ve fixed my issues. This is my final solution:</p>
<pre><code class="lang-auto">shift_layer = generator.main[11]
optimizer_b = optim.SGD(shift_layer.parameters(), lr = ...)
</code></pre>
<p>and in the training loop I just call <code>optimizer_b.step()</code> for every epoch.</p>
<p>So far this seems to be working! Thanks to you both for your help. Appreciate it!</p>