Different learning rate for a specific layer

username: thematrixduo
cooked: <p>I want to change the learning rate of only one layer of my neural nets to a smaller value. I am aware that one can have per-layer learning rate according to this:<br>
<a href="https://pytorch.org/docs/0.3.0/optim.html#per-parameter-options" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/0.3.0/optim.html#per-parameter-options</a></p>
<p>However if I have a lot of layers, it is quite tedious to specific learning rate for each of them. Is there a more convenient way to specify one lr for just a specific layer and another lr for all other layers? Many thanks!</p>

username: ptrblck
cooked: <p>Yes, as you can see in the example of the docs you’ve linked, <code>model.base.parameters()</code> will use the default learning rate, while the learning rate is explicitly specified for <code>model.classifier.parameters()</code>.</p>
<p>In your use case, you could filter out the specific layer and use the same approach.</p>

username: thematrixduo
cooked: <p>Thanks. So i there a convenient way of filter out a specific layer? Cuz I have been searching for a while and could not fine one</p>

username: ptrblck
cooked: <p>You could use <code>filter</code> to get the base and “special” parameters:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 3, 1, 1)
        self.pool1 = nn.MaxPool2d(2)
        self.conv2 = nn.Conv2d(6, 12, 3, 1, 1)
        self.pool2 = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(12*56*56, 128)
        self.fc2 = nn.Linear(128, 10)
        
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.pool1(x)
        x = F.relu(self.conv2(x))
        x = self.pool2(x)
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

        
model = MyModel()

my_list = ['fc1.weight', 'fc1.bias']
params = list(filter(lambda kv: kv[0] in my_list, model.named_parameters()))
base_params = list(filter(lambda kv: kv[0] not in my_list, model.named_parameters()))
</code></pre>
<p>Depending on your model definition (custom vs. <code>nn.Sequential</code> etc.) some other code snippets might be prettier. Let me know, if that works for you.</p>

username: thematrixduo
cooked: <p>Thanks a lot! this works for me.</p>

username: Nadedic
cooked: <p>Hello,<br>
please consider the following code:</p>
<h3>There are multiple layers before this affine operation and activation layer</h3>
<pre><code class="lang-auto">    self.fc1 = nn.Linear(13824, 4096) 
    torch.nn.init.kaiming_normal_(self.fc1.weight) 
    self.batchnorm_fc1 = nn.BatchNorm1d(4096)
</code></pre>
<pre><code class="lang-auto">    #YL
    self.fc3 = nn.Linear(4096,  3* 11* 13)
    torch.nn.init.kaiming_normal_(self.fc3.weight)
    self.batchnorm_fc3 = nn.BatchNorm1d( 3* 11* 13)
    
    #YH[0]
    self.fc4 = nn.Linear(4096,   3* 3* 29* 39)
    torch.nn.init.kaiming_normal_(self.fc4.weight)
    self.batchnorm_fc4 = nn.BatchNorm1d( 3* 3* 29* 39)
    
    #YH[1] there are two more layers here 
    #YH[2] 
</code></pre>
<p>I start creating multiple heads here as the output, however YH[0] for instance has a product of numbers (you might call it weights) around 10k while YL has product of numbers of around ~400. I would like to treat each of those layers (and their weight) differently when it comes to their learning rates, otherwise the gradient is propagating through all those 400+10000+… numbers which overlooks the significance of YL sub-band output which is the most significant in my case. What do you recommend?!</p>
<p>with the forward propagation:</p>
<pre><code class="lang-auto">    x = F.leaky_relu( self.batchnorm_fc1(self.fc1(x)) )
    ###############

           
    xl = self.fc3(x)
    xl  = xl .view(-1, 3, 11, 13)
    
    xh=[]
    xh.append(self.fc4(x))
    xh[0] = xh[0].view(-1,3, 3, 29, 39)
    ##same done with the xh[1] and xh[2]

      
    # return the output
    return xl,xh
</code></pre>
<p>Could you guide me on how to handle xl, and xh’s layers weights differently?!</p>
<p>I am using those 4 output heads for constructing an inverse wavelet transform(2d) but that isn’t so important right now.</p>

username: peter_wills
cooked: <p>This stuff all worked great - thanks for the post!</p>
<p>I will point out something: If you want to use this with a scheduler (e.g. CyclicLR), then the lr’s of the parameter groups should also be passed to the scheduler constructor as a list of floats. Otherwise, the group dependent lr’s are lost.</p>
<p>Edit: note also that, for example, base_parms will not be a list of torch tensors (which it needs to be). It will be a list of tuples and one must use only the second component of each tuple.</p>

username: madmaxiwnl
cooked: <aside class="quote no-group" data-username="peter_wills" data-post="7" data-topic="33670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peter_wills/48/13661_2.png" class="avatar"> peter_wills:</div>
<blockquote>
<p>If you want to use this with a scheduler (e.g. CyclicLR), then the lr’s of the parameter groups should also be passed to the scheduler constructor as a list of floats.</p>
</blockquote>
</aside>
<p>Is this also required for LambdaLR and such? If so, can you please show how to implement it?</p>

username: CuriousCat-7
cooked: <p>Hello, it gets error when I resume the optimizer.<br>
I define the optimizer like that:</p>
<pre><code class="lang-python">     # 削减公用层learning_rate, 这个很重要
     all_parameters = set(model.parameters())
     nas_layers_params = []
     for m in model.modules():
         if isinstance(m, BlockSwitch):
             nas_layers_params += list(m.parameters())
     nas_layers_params = set(nas_layers_params)
     comm_layers_params = all_parameters - nas_layers_params
     nas_layers_params = list(nas_layers_params)
     comm_layers_params = list(comm_layers_params)

     optimizer = torch.optim.Adam(
         [{"params": nas_layers_params},
          {"params": comm_layers_params, "lr": args.learning_rate/model.num_blocks_per_layer}  # 公用层learning_rate应取平均
         ],
         args.learning_rate,
         #momentum=args.momentum,
         weight_decay=args.weight_decay)
     scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
         optimizer, float(args.epochs), eta_min=args.learning_rate_min, last_epoch=-1)

</code></pre>
<p>Save model like that:</p>
<pre><code class="lang-python">         # save the states of this epoch
         state = {
             'epoch': epoch,
             'args': args,
             'optimizer_state': optimizer.state_dict(),
             'supernet_state': model.state_dict(),
             'scheduler_state': scheduler.state_dict()
         }
         path = './super_train/{}/super_train_states.pt.tar'.format(args.exp_name)
         torch.save(state, path)
</code></pre>
<p>And load optimizer like that:</p>
<pre><code class="lang-python">     if args.resume:
         resume_path = './super_train/{}/super_train_states.pt.tar'.format(args.exp_name)
         if os.path.isfile(resume_path):
             print("Loading checkpoint '{}'".format(resume_path))
             checkpoint = torch.load(resume_path)

             start_epoch = checkpoint['epoch']
             model.load_state_dict(checkpoint['supernet_state'])
             optimizer.load_state_dict(checkpoint['optimizer_state'])
             scheduler.load_state_dict(checkpoint['scheduler_state'])
         else:
             raise ValueError("No checkpoint found at '{}'".format(resume_path))
</code></pre>
<p>But get the error:</p>
<pre><code class="lang-bash">  File "train.py", line 197, in main
    train(args, epoch, train_data, device, model, criterion=criterion, optimizer=optimizer, my_choice=choice)
  File "train.py", line 77, in train
    optimizer.step()
  File "/data/limingyao/miniconda3/envs/py38/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 66, in wrapper
    return wrapped(*args, **kwargs)
  File "/data/limingyao/miniconda3/envs/py38/lib/python3.8/site-packages/torch/optim/adam.py", line 95, in step
    exp_avg.mul_(beta1).add_(1 - beta1, grad)
RuntimeError: The size of tensor a (80) must match the size of tensor b (240) at non-singleton dimension 0
</code></pre>
<p>Is the way I load optimizer.state_dict wrong?<br>
Thank you</p>

username: CuriousCat-7
cooked: <aside class="quote quote-modified" data-post="9" data-topic="33670">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/curiouscat-7/48/14227_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/different-learning-rate-for-a-specific-layer/33670/9">Different learning rate for a specific layer</a> 
  </div>
  <blockquote>
    Hello, it gets error when I resume the optimizer. 
I define the optimizer like that: 
     # 削减公用层learning_rate, 这个很重要
     all_parameters = set(model.parameters())
     nas_layers_params = []
     for m in model.modules():
         if isinstance(m, BlockSwitch):
             nas_layers_params += list(m.parameters())
     nas_layers_params = set(nas_layers_params)
     comm_layers_params = all_parameters - nas_layers_params
     nas_layers_params = list(nas_layers_params)
     comm_layers_params…
  </blockquote>
</aside>


username: faizan
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>’s solution requires you to specify full names of names parameters for changed lr.<br>
I wrote a recursive solution which lets you use just the submodule path.</p>
<p>The code:</p>
<pre><code class="lang-python">from pprint import pprint
from typing import Dict
from torchvision import models


def group_wise_lr(model, group_lr_conf: Dict, path=""):
    """
    Refer https://pytorch.org/docs/master/optim.html#per-parameter-options


    torch.optim.SGD([
        {'params': model.base.parameters()},
        {'params': model.classifier.parameters(), 'lr': 1e-3}
    ], lr=1e-2, momentum=0.9)


    to


    cfg = {"classifier": {"lr": 1e-3},
           "lr":1e-2, "momentum"=0.9}
    confs, names = group_wise_lr(model, cfg)
    torch.optim.SGD([confs], lr=1e-2, momentum=0.9)



    :param model:
    :param group_lr_conf:
    :return:
    """
    assert type(group_lr_conf) == dict
    confs = []
    nms = []
    for kl, vl in group_lr_conf.items():
        assert type(kl) == str
        assert type(vl) == dict or type(vl) == float or type(vl) == int

        if type(vl) == dict:
            assert hasattr(model, kl)
            cfs, names = group_wise_lr(getattr(model, kl), vl, path=path + kl + ".")
            confs.extend(cfs)
            names = list(map(lambda n: kl + "." + n, names))
            nms.extend(names)

    primitives = {kk: vk for kk, vk in group_lr_conf.items() if type(vk) == float or type(vk) == int}
    remaining_params = [(k, p) for k, p in model.named_parameters() if k not in nms]
    if len(remaining_params) &gt; 0:
        names, params = zip(*remaining_params)
        conf = dict(params=params, **primitives)
        confs.append(conf)
        nms.extend(names)

    plen = sum([len(list(c["params"])) for c in confs])
    assert len(list(model.parameters())) == plen
    assert set(list(zip(*model.named_parameters()))[0]) == set(nms)
    assert plen == len(nms)
    if path == "":
        for c in confs:
            c["params"] = (n for n in c["params"])
    return confs, nms


if __name__ == "__main__":
    model = models.resnet18(pretrained=True)

    test_configs = [
        # Give same Lr to all model params
        {"lr": 0.3},

        # For the below 3 cases, you will need to pass the optimiser overall optimiser params for remaining model params.
        # This is because we did not specify optimiser params for all top-level submodules, so defaults need to be supplied
        # Refer https://pytorch.org/docs/master/optim.html#per-parameter-options

        # Give same Lr to layer4 only
        {"layer4": {"lr": 0.3}},

        # Give one LR to layer4 and another to rest of model. We can do this recursively too.
        {"layer4": {"lr": 0.3},
         "lr": 0.5},

        # Give one LR to layer4.0 and another to rest of layer4
        {"layer4": {"0": {"lr": 0.001},
                    "lr": 0.3}},

        # More examples
        {"layer4": {"lr": 0.3,
                    "0": {"lr": 0.001}}},

        {"layer3": {"0": {"conv2": {"lr": 0.001}},
                    "1": {"lr": 0.003}}},

        {"layer4": {"lr": 0.3},
         "layer3": {"0": {"conv2": {"lr": 0.001}},
                    "lr": 0.003},
         "lr": 0.001}
    ]

    for cfg in test_configs:
        confs, names = group_wise_lr(model, cfg)
        print("#" * 140)
        pprint(cfg)
        print("-" * 80)
        pprint(confs)
        print("#" * 140)

</code></pre>
<p>Usage:<br>
Suppose you have the <code>resnet18</code> model from torchvision. Now you want to change LR of <code>layer4</code> and <code>layer3.1</code> and <code>layer3.0.conv2</code>:</p>
<pre><code class="lang-python">from torchvision import models
model = models.resnet18(pretrained=True)
confs, names = group_wise_lr(model, {"layer4": {"lr": 0.3},
                                     "layer3": {"0": {"conv2": {"lr": 0.001}},
                                                "1": {"lr": 0.003}}})
# Notice we write in hierarchical structure, we go down the hierarchy till we need, not more.

This generates confs as 
```python
[{'lr': 0.3,
  'params': &lt;generator object group_wise_lr.&lt;locals&gt;.&lt;genexpr&gt; at 0x11f4a84a0&gt;},
 {'lr': 0.001,
  'params': &lt;generator object group_wise_lr.&lt;locals&gt;.&lt;genexpr&gt; at 0x11f4a8510&gt;},
 {'params': &lt;generator object group_wise_lr.&lt;locals&gt;.&lt;genexpr&gt; at 0x11f4a8580&gt;},
 {'lr': 0.003,
  'params': &lt;generator object group_wise_lr.&lt;locals&gt;.&lt;genexpr&gt; at 0x11f4a85f0&gt;},
 {'params': &lt;generator object group_wise_lr.&lt;locals&gt;.&lt;genexpr&gt; at 0x11f4a87b0&gt;}]
</code></pre>
<p>the last <code>{"params": &lt;generator&gt;}</code> has all the model params we did not specify any lr.<br>
the third <code>{"params": &lt;generator&gt;}</code> has all params of <code>layer3.0</code> which aren’t in submodule <code>conv2</code><br>
this allows you to pass this <code>confs</code> to an optimiser and do fine-grained lr tuning in a hierarchical manner.</p>
<pre><code class="lang-auto"></code></pre>

username: karl7
cooked: <p>i think these two api: <code>model.base.parameters() </code>, <code>model.classifier.parameters()</code> are made for the standard model classes, how about self defined models?</p>
<p>May i ask how to output the parameters of each layer for self defined models? such as each layer’s learning rate? It would be important in finetuning when the user want to check if the learing rate is correctly set, worked or not.</p>

username: ptrblck
cooked: <p>You can use the same syntax for custom modules to get the parameters, e.g.:</p>
<pre><code class="lang-python">my_custom_model.my_custom_layer.parameters()
</code></pre>
<p>That being said, modules do not contain the learning rate, which is set in the creation of the optimizer.<br>
If you want to use layer-specific learning rates, you could use the <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">per-parameter options</a>.</p>

username: Omar_AlSuwaidi
cooked: <p>Hey thanks a lot for this, I found this very helpful. Though is it possible to to apply a LR scheduler on one parameter group but not the other (meaning the other parameter group has a constant LR)?</p>
<p>I couldn’t figure it out, since the scheduler gets applied on all the parameter groups defined in your optimizer; so do you have any idea?</p>

username: Hedwig_Huang
cooked: <p>Hey there, I face the same problem here, I only need to change a little block’s learning rate, but the mostly of the model stay the same learning rate. How can I filter the parameters? I try defined a list and search in <code>.named_paramters()</code> like <a class="mention" href="/u/ptrblck">@ptrblck</a> of 4th said,  but I don’t know how to assert the new params list in <code>torch.optim.SGD</code>.</p>
<p>Update:<br>
I fixed this problem by using list(filter()) method to flit the named_paramters and define SGD with specified block<code>.paramters()</code> , problem fixed. So, the key is to find out how the blocks defined in your model. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

username: melzohbi
cooked: <p>I want to do the same thing but couldn’t figure it out. Did you find an answer for this?</p>

username: Jack_Shi
cooked: <p>Once having the <code>params</code> and <code>base_params</code>, can we simply define the optimizer as follows?</p>
<pre><code class="lang-auto">optimizer = optim.Adam([
                            {'params': [temp[1] for temp in base_params]},
                            {'params': [temp[1] for temp in params],
                                        'lr': 1e-4}
                        ], lr=2e-3, betas=(0.9, 0.999),weight_decay=0)
</code></pre>

username: ptrblck
cooked: <p>Your idea should generally work, but note that you would need to use <code>param[1]</code> in the <code>list</code> instead of <code>temp[1]</code> as it’s undefined.</p>

username: Jack_Shi
cooked: <p>Typo, corrected. Thanks!</p>

username: leibyj
cooked: <p>I believe you can just create two separate optimizers and pass whichever one to the scheduler.</p>
<p><a href="https://stackoverflow.com/questions/65342928/how-to-use-different-optimizers-for-each-model-layer-pytorch" rel="noopener nofollow ugc">stackover example</a></p>