Different learning rate for each weight

Blaze: <p>The docs here:  <a href="http://pytorch.org/docs/optim.html" rel="noopener nofollow ugc"> http://pytorch.org/docs/optim.html</a> explain how to set different learning rates for each layer in the model. However, I have a slightly different problem - I need to be able to set different learning rates for each weight within a layer. More specifically, I would like the weights connected to earlier output nodes to have a higher learning rate than those connected to later nodes, as is perhaps better explained by this picture:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/98709985ea049a7c0781738a81b61472e7b33e84.png" data-download-href="https://discuss.pytorch.org/uploads/default/98709985ea049a7c0781738a81b61472e7b33e84" title="expw"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/98709985ea049a7c0781738a81b61472e7b33e84_2_690x345.png" alt="expw" data-base62-sha1="lKxONcI2j4ner0feA5fHPEiZqHW" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/98709985ea049a7c0781738a81b61472e7b33e84_2_690x345.png, https://discuss.pytorch.org/uploads/default/original/2X/9/98709985ea049a7c0781738a81b61472e7b33e84.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/9/98709985ea049a7c0781738a81b61472e7b33e84.png 2x" data-dominant-color="E2E5E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">expw</span><span class="informations">1000×500 47.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> Any ideas as to how I might go about implementing this?</p>

tom: <p>The <code>parameters()</code> in the <a href="http://pytorch.org/docs/master/optim.html#per-parameter-options" rel="nofollow noopener">per-parameter options example</a> be any iterables for the parameters, so if you had a list of <code>parameters_per_step</code> and a list of <code>learning_rates_per_step</code>, you can do <code>optim.SGD([{'params': p, 'lr': l} for p,l in zip(parameters_per_timestep, learning_rates_per_step)])</code> or somesuch.</p>
<p>Best regards</p>
<p>Thomas</p>

Blaze: <p>Many thanks, but I’m not sure I entirely understand. The <code>parameters()</code> include the weights and biases of the network, both of which are multidimensional, so how would this translate into the <code>parameters_per_step</code>?</p>

tom: <p>If you have them as separate instances of <code>Parameter</code>, you can arrange them however you want manually.<br>
If you only have one large multi-dimension <code>Parameter</code>, you could multiply the parameters <code>.grad</code> by an appropriate weight after the backward and before the optimizer call to get something similar to a different learning rate.</p>
<p>Best regards</p>
<p>Thomas</p>

Blaze: <p>So how would I extract each of the weights from a <code>torch.nn.Linear</code> layer as a separate instance of <code>Parameter</code>?</p>

tom: <p>No, in this case, you could go for the workaround described above (multiply the .grad with something (say <code>torch.exp(-delta*torch.arange(0,4)).unsqueeze(1)</code> if you have the example from your other thread). For SGD this will be the same as having learning rates smaller for the later rows of the weight.</p>
<p>Or you could first define separate tensors and <code>torch.cat</code>/<code>torch.stack</code> them.</p>
<p>Best regards</p>
<p>Thomas</p>

Nadedic: <p>Hello,<br>
please consider the following code:</p>
<h3>There are multiple layers before this affine operation and activation layer</h3>
<pre><code>    self.fc1 = nn.Linear(13824, 4096) 
    torch.nn.init.kaiming_normal_(self.fc1.weight) 
    self.batchnorm_fc1 = nn.BatchNorm1d(4096)
</code></pre>
<h3>I start creating multiple heads here as the output, however YH[0] for instance has a product of numbers (you might call it weights) around 10k while YL has product of numbers of around ~400. I would like to treat each of those layers (and their weight) differently when it comes to their learning rates, otherwise the gradient is propagating through all those 400+10000+… numbers which overlooks the significance of YL sub-band output which is the most significant in my case. What do you recommend?!</h3>
<pre><code>    #YL
    self.fc3 = nn.Linear(4096,  3* 11* 13)
    torch.nn.init.kaiming_normal_(self.fc3.weight)
    self.batchnorm_fc3 = nn.BatchNorm1d( 3* 11* 13)
    
    #YH[0]
    self.fc4 = nn.Linear(4096,   3* 3* 29* 39)
    torch.nn.init.kaiming_normal_(self.fc4.weight)
    self.batchnorm_fc4 = nn.BatchNorm1d( 3* 3* 29* 39)
    
    #YH[1] there are two more layers here 
    #YH[2] 
</code></pre>
<p>with the forward:</p>
<pre><code>    x = F.leaky_relu( self.batchnorm_fc1(self.fc1(x)) )
    ###############

           
    xl = self.fc3(x)
    xl  = xl .view(-1, 3, 11, 13)
    
    xh=[]
    xh.append(self.fc4(x))
    xh[0] = xh[0].view(-1,3, 3, 29, 39)
    ##same done with the xh[1] and xh[2]

      
    # return the output
    return xl,xh
</code></pre>
<p>I am using those 4 output heads for constructing an inverse wavelet signal (2d) but that isn’t so important right now.</p>