How to set a different learning rate for a single layer in a network

username: partially_observed
cooked: <p>Hi,</p>
<p>I am trying to change the learning rate for any arbitrary single layer (which is part of a nn.Sequential block). For example, I use a VGG16 network and wish to control the learning rate of one of the fully connected layers in the classifier.</p>
<p>Going by this link: <a href="https://pytorch.org/docs/0.3.0/optim.html#per-parameter-options" rel="nofollow noopener">https://pytorch.org/docs/0.3.0/optim.html#per-parameter-options</a>, we can specify the learning rate like this -</p>
<p><code>optim.SGD([ {'params': model.base.parameters()}, {'params': model.classifier.parameters(), 'lr': 1e-3} ], lr=1e-2, momentum=0.9)</code></p>
<p>But here, both base and classifier are entire blocks. In the VGG16 network for example, I want to change the learning rate for classifier[0] / classifier[3] / classifier[6], which are linear layers. Any ideas as to how that can be accomplished?</p>
<p>VGG16 network:<br>
VGG(<br>
(features): Sequential(<br>
(0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(1): ReLU(inplace)<br>
(2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(3): ReLU(inplace)<br>
(4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(6): ReLU(inplace)<br>
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(8): ReLU(inplace)<br>
(9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(11): ReLU(inplace)<br>
(12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(13): ReLU(inplace)<br>
(14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(15): ReLU(inplace)<br>
(16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(18): ReLU(inplace)<br>
(19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(20): ReLU(inplace)<br>
(21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(22): ReLU(inplace)<br>
(23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(25): ReLU(inplace)<br>
(26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(27): ReLU(inplace)<br>
(28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(29): ReLU(inplace)<br>
(30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=(7, 7))<br>
(classifier): Sequential(<br>
(0): Linear(in_features=25088, out_features=4096, bias=True)<br>
(1): ReLU(inplace)<br>
(2): Dropout(p=0.5)<br>
(3): Linear(in_features=4096, out_features=4096, bias=True)<br>
(4): ReLU(inplace)<br>
(5): Dropout(p=0.75)<br>
(6): Linear(in_features=4096, out_features=10, bias=True)<br>
(7): Softmax()<br>
)<br>
)</p>

username: JuanFMontesinos
cooked: <p>You just need to create more groups as you did.<br>
run</p>
<pre><code class="lang-auto">for name,param in model.named_parameters():
     filter them out and create a list of dicts
optim.SGD(list)
</code></pre>
<p>The only constrain is you cannot repeat parameters, thus, if you decompose classifier parameters you will have to assign them all by this method.</p>

username: partially_observed
cooked: <p>Thank you for the response. Here is what I did:</p>
<pre><code class="lang-auto">my_list = ['classifier.3.weight', 'classifier.3.bias']
params = list(filter(lambda kv: kv[0] in my_list, vgg16.named_parameters()))
base_params = list(filter(lambda kv: kv[0] not in my_list, vgg16.named_parameters()))
</code></pre>
<p>And then defined the optimizer:</p>
<pre><code class="lang-auto">optimizer = SGD([{'params': base_params}, {'params': params, 'lr': '1e-4'}], lr=3e-6, momentum=0.9)
</code></pre>
<p>However, I get the following error:</p>
<pre><code class="lang-auto">TypeError: optimizer can only optimize Tensors, but one of the params is tuple
</code></pre>
<p>I get the same error when I try this as well:</p>
<pre><code class="lang-auto">optimizer = SGD([{'params': base_params, 'lr': 3e-6, 'momentum': 0.9 }, {'params': params, 'lr': 1e-4, 'momentum': 0.9}])
</code></pre>
<p>I am not entirely sure what I need to change. Any ideas?</p>

username: sai_tharun
cooked: <p>Try this, hope it helps!</p>
<pre><code class="lang-auto">optimizer = SGD([{'params': model.classifier[0].parameters(), 'lr': 3e-6, 'momentum': 0.9 }, 

                 {'params': m.classifier[1].parameters(), 'lr': 1e-4, 'momentum': 0.9}])
</code></pre>

username: partially_observed
cooked: <p>Please correct me if I am wrong, but here the learning rates have been set only for <em>two</em> layers of the network: classifier[0] and classifier[1]. The rest of the network doesn’t have learning rates associated with them.</p>
<p>What I wish to accomplish is to change the learning rate for a single layer only (in a Sequential block), and have a common learning rate for the rest of the layers.</p>

username: sai_tharun
cooked: <p>try this</p>
<pre><code class="lang-auto">optimizer = SGD([{'params': model.classifier[0].parameters(), 'lr': 3e-6, 'momentum': 0.9 }], 
                 model.parameters,lr=1e-2 ,momentum=0.9
)
</code></pre>

username: partially_observed
cooked: <p>Hi, when I try this, that returns the following error:</p>
<pre><code class="lang-auto">TypeError: __init__() got multiple values for argument 'lr'
</code></pre>
<p>I am not quite sure what change I need to make. As <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> mentioned, I think I need to specify separate parameter lists for each learning rate. Though I don’t know how to do that, given the error I mentioned earlier:</p>
<pre><code class="lang-auto">TypeError: optimizer can only optimize Tensors, but one of the params is tuple
</code></pre>
<p>Any ideas?</p>

username: JuanFMontesinos
cooked: <p>I’m afraid you can only set each parameter once. The way <a class="mention" href="/u/sai_tharun">@sai_tharun</a> mentioned you are passing parameter 0 twice.</p>
<p>The problem is once you set parameter for classifier you need to set all the parameters of classifier .<br>
As I aforementioned you need to do for name,param in model.classifier.named_parameters():<br>
If name == yourlayer_name:<br>
List.append(…)<br>
Else:<br>
Others.append(…)</p>
<p>Ofc you have to pass the rest of the network which is not model.pclassifier as you were doing</p>

username: partially_observed
cooked: <p>I am not quite sure what you mean.</p>
<p>As you can see, using the following code (similar to what <a class="mention" href="/u/ptrblck">@ptrblck</a> details in <a href="https://discuss.pytorch.org/t/different-learning-rate-for-a-specific-layer/33670/4">link</a>):</p>
<pre><code class="lang-auto">my_list = ['classifier.3.weight', 'classifier.3.bias']
params = list(filter(lambda kv: kv[0] in my_list, vgg16.named_parameters()))
base_params = list(filter(lambda kv: kv[0] not in my_list, vgg16.named_parameters()))
</code></pre>
<p>returns two lists - the layer(s) that requires a different learning rate (classifier[3] in this case), and the rest of the network. I am having trouble passing these to the optimizer like so:</p>
<pre><code class="lang-auto">optimizer = SGD([{'params': base_params}, {'params': params, 'lr': '1e-4'}], lr=3e-6, momentum=0.9)
</code></pre>
<p>I think is incorrect since it gives me errors. Any thoughts <a class="mention" href="/u/ptrblck">@ptrblck</a>, <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>?</p>

username: JuanFMontesinos
cooked: <aside class="quote no-group quote-modified" data-username="partially_observed" data-post="9" data-topic="48552">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/cab0a1/48.png" class="avatar"> partially_observed:</div>
<blockquote>
<p>my_list = [‘classifier.3.weight’, ‘classifier.3.bias’] params = list(filter(lambda kv: kv[0] in my_list, vgg16.named_parameters()))</p>
</blockquote>
</aside>
<p>Hi,<br>
You are passing also the name that way as params contains tuples of (name,parameters)<br>
could you please use</p>
<pre><code class="lang-auto">from torchvision.models import vgg16
from torch.optim import SGD
model = vgg16()
my_list = ['classifier.3.weight', 'classifier.3.bias']
params = list(map(lambda x: x[1],list(filter(lambda kv: kv[0] in my_list, model.named_parameters()))))
base_params = list(map(lambda x: x[1],list(filter(lambda kv: kv[0] not in my_list, model.named_parameters()))))
optimizer = SGD([{'params': base_params}, {'params': params, 'lr': '1e-4'}], lr=3e-6, momentum=0.9)
</code></pre>
<p>This is ok, I promise <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: partially_observed
cooked: <p>This works! Thank you very much!</p>

username: CharviVitthal
cooked: <p>Hi, I tried this but it is giving me the error: “<strong>TypeError: SGD() got multiple values for argument ‘lr’</strong>”. Do you have any suggestion on what the problem could be and how to solve it?<br>
Thanks and regards,<br>
Charvi</p>

username: JuanFMontesinos
cooked: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="10" data-topic="48552">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p><code>optimizer = SGD([{'params': base_params}, {'params': params, 'lr': '1e-4'}], lr=3e-6, momentum=0.9)</code></p>
</blockquote>
</aside>
<pre><code class="lang-auto">optimizer = SGD([{'params': base_params}, {'params': params, 'lr': '1e-4'}], lr=3e-6, momentum=0.9)
</code></pre>
<p>is it possible you are repeating the ‘lr’ argument in any of the dictionaries you are passing?</p>

username: CharviVitthal
cooked: <p>Does not look like that. I have used in exactly the same way as</p>
<pre><code class="lang-auto">my_list = ['classifier.3.weight', 'classifier.3.bias']
params = list(map(lambda x: x[1],list(filter(lambda kv: kv[0] in my_list, model.named_parameters()))))
base_params = list(map(lambda x: x[1],list(filter(lambda kv: kv[0] not in my_list, model.named_parameters()))))
optimizer = SGD([{'params': base_params}, {'params': params, 'lr': '1e-4'}], lr=3e-6, momentum=0.9)
</code></pre>
<p><code>params</code> and <code>base_params</code> does not contain the names of the parameters involved but I printed their shapes and they seem to be distinct.</p>
<p>I also tried<br>
<code>optimizer = SGD([{'params': base_params, 'lr':'3e-6'}, {'params': params, 'lr': '1e-4'}], momentum=0.9)</code><br>
this gave me the error<br>
<code>TypeError: '&lt;' not supported between instances of 'list' and 'float'</code> at the start of the training loop.</p>
<pre><code class="lang-auto">File "xxx.py", line 251, in main
    with Trainer(model, optimizer, F.cross_entropy, scheduler=scheduler, callbacks=_callbacks) as trainer:
  File "/usr/local/lib/python3.8/site-packages/homura/trainers.py", line 519, in __init__
    super(SupervisedTrainer, self).__init__(model, optimizer, loss_f, callbacks=callbacks, scheduler=scheduler,
  File "/usr/local/lib/python3.8/site-packages/homura/trainers.py", line 121, in __init__
    self.set_optimizer()
  File "/usr/local/lib/python3.8/site-packages/homura/trainers.py", line 422, in set_optimizer
    self.optimizer = optimizer(self.model.parameters())
  File "/usr/local/lib/python3.8/site-packages/torch/optim/sgd.py", line 57, in __init__
    if lr is not required and lr &lt; 0.0:
TypeError: '&lt;' not supported between instances of 'list' and 'float'
</code></pre>
<p>Any suggestions are welcome and deeply appreciated! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: JuanFMontesinos
cooked: <p>Sorry for the late replay but without code it seems everything is about a mistake.<br>
Can you post some standalone code?</p>

username: CharviVitthal
cooked: <p>Hi, the problem got solved.<br>
I was using a github repository as my base code and the problem was that I was using the optim function defined by them instead of the torch.optim. I changed it to torch.optim and followed your method. It works very well. Sorry for the confusion and thanks for the help.</p>