Multiply feature map by a learnable scalar

username: Alexey_Chernyavskiy
cooked: <p>I’d like to know if there is a way to multiply the output of a convolutional layer (a set of N feature maps) by N learnable multipliers. Or, similarly, how to multiply all feature maps in a stack by one single learnable parameter. What layer/function should I use? My case is that I have the outputs of two parallel CNN branches, A and B, with same sizes and number of feature maps, and I want to make a new output C = alpha<em>A + beta</em>B, where alpha and beta are learnable parameters. Thanks!</p>

username: fmassa
cooked: <p>Make your scalar a Variable containing a 1D tensor, and use the <code>expand_as</code> function.</p>
<pre><code class="lang-python">matrix = Variable(torch.rand(3,3))
scalar = Variable(torch.rand(1), requires_grad=True)
output = matrix * scalar.expand_as(matrix)
</code></pre>

username: Alexey_Chernyavskiy
cooked: <p>Thank <a class="mention" href="/u/fmassa">@fmassa</a>, I have another question though.<br>
How do I perform training with such a multiplier?<br>
Right now, I have this:</p>
<p>in my Model definition:</p>
<pre><code>    self.multip = torch.autograd.Variable(torch.rand(1).cuda(), requires_grad=True)
    self.multip = self.multip.cuda()
</code></pre>
<p>in my Model forward:</p>
<pre><code>def forward(self, x):
  x1 = self.relu(self.conv1(x))
  x2 = self.relu(self.conv2(x))
  x1 = x1 * self.multip.expand_as(x1) # multiply x1 output by learnable parameter "multip"
  x = torch.add(x1, x2)
  return x
</code></pre>
<p>and in my training code:<br>
…<br>
optimizer.zero_grad()<br>
loss = criterion(model(input), target)<br>
loss.backward()<br>
optimizer.step()</p>
<p>As you see, nothing out of the ordinary. But when I print the value of model.multip.data[0], I see that the initial value of multip remains unchanged (all other params in my Model do change). I deduced that probably I messed somewhere and did not apply the gradients to self.multip<br>
Am I right?</p>

username: fmassa
cooked: <p>If you are using a <code>nn.Module</code> and the multiplier is inside the network, you need to make it a <code>nn.Parameter</code> so that it is registered as a parameter for you call <code>model.parameters()</code>.<br>
So instead of using a <code>Variable</code> to encapsulate your multiplier, you should use a <code>nn.Parameter</code>.</p>
<p>Quick example:</p>
<pre><code class="lang-python">class Model1(nn.Module):
    def __init__(self):
        super(Model1, self).__init__()
        self.multp = Variable(torch.rand(1), requires_grad=True)

class Model2(nn.Module):
    def __init__(self):
        super(Model2, self).__init__()
        self.multp = nn.Parameter(torch.rand(1)) # requires_grad is True by default for Parameter

m1 = Model1()
m2 = Model2()

print('m1', list(m1.parameters()))
print('m2', list(m2.parameters()))
</code></pre>

username: Alexey_Chernyavskiy
cooked: <p>Thank you. Alright, I can now see nonzero values in<br>
m2.multp.grad.data<br>
after I do the optimizer steps, but the value of multp still doesn’t change during training. Will investigate. I think it might have something to do with the learning rate.</p>
<p>Now, how can I specify a special learning rate for this new param? I used to do something like this for my convolutional layers to make a specific smaller learning rate for layer ‘conv2’.</p>
<p>optimizer = optim.Adam([{‘params’: model.conv2.parameters(), ‘lr’: <a href="http://opt.lr" rel="nofollow noopener">opt.lr</a>*0.1}], <a href="http://lr=opt.lr" rel="nofollow noopener">lr=opt.lr</a>)</p>
<p>Now if I do this:<br>
optimizer = optim.Adam([{‘params’: model.<strong>multp</strong>.parameters(), ‘lr’: <a href="http://opt.lr" rel="nofollow noopener">opt.lr</a>*0.1}], <a href="http://lr=opt.lr" rel="nofollow noopener">lr=opt.lr</a>)</p>
<p>I get an error File “/home/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py”, line 63, in <strong>getattr</strong><br>
raise AttributeError(name)<br>
AttributeError: parameters</p>
<p>UPD: I replaced my ‘custom’ Adam (see above) that had different learning rates by a regular Adam which has the same learning rate for all the layers, and it worked - my multp started to become updated during the course of training (it also worked with SGD). However my question with setting a special learning rate for the nn.Parameter remains.</p>

username: fmassa
cooked: <p>You can simply do something like</p>
<pre><code class="lang-python">optimizer = optim.Adam([{'params':[model.multp], 'lr':opt.lr*0.1}], lr=opt.lr)
</code></pre>

username: Alexey_Chernyavskiy
cooked: <p>Great, it worked! Thank you.</p>

username: Andrefmds
cooked: <p>Hey, I was wondering if is possible to mutiply each feature map by a different scalar?</p>

username: Chenhongyi_Yang
cooked: <p>Maybe using a 1x1 conv with group=n_channels?</p>

username: mbutt
cooked: <p>Hi did you work it out? I also need to do something similar.</p>