Two learning rate schedulers one optimizer

barakb: <p>Hi, I want to adjust the learning rate of part of my model, let’s call it PartA using lr_schedulerA<br>
And  PartB using lr_schedulerB.<br>
I didn’t find a way to do this, the only solution I found is to duplicate my optimizer, and put the parameters of each part in the corresponding optimizer:</p>
<pre><code class="lang-auto">    optimizerA = torch.optim.SGD(parametersA, args.lr,
                                momentum=args.sgd_momentum,
                                weight_decay=args.weight_decay)

   optimizerB = torch.optim.SGD(parametersB, args.lr,
                                    momentum=args.sgd_momentum,
                                    weight_decay=args.weight_decay)


</code></pre>
<p>And then :</p>
<pre><code class="lang-auto">        lr_schedulerA = torch.optim.lr_scheduler.MultiStepLR(optimizerA,
                                                        milestones=[100, 150], last_epoch=args.start_epoch - 1)
        lr_schedulerB = torch.optim.lr_scheduler.ExponentialLR(optimizerB,gamma=0.99)
</code></pre>
<p>Anyone holds a better idea to share with me?</p>
<p>Thanks!</p>

ptrblck: <p>Your approach looks reasonable as you are not duplicating the optimizer but rather use two different optimizers and schedulers for different parts of the model.<br>
I think this use case looks clean and is quite easy to understand so I would stick to it and not hack around <code>param_groups</code> of the optimizer to pass them to different schedulers.</p>

NoobCoder: <p>Hello,</p>
<p>I have two parts in my model let’s say ‘feature’ and ‘classifier’. In optimizer, I have defined two learning rates for these parts. I am not sure how I can set up the schedular for these two?</p>
<pre><code class="lang-auto">optim.SGD([
{‘params’: model.feature.parameters(), ‘lr’: 0.1},
{‘params’: model.classifier.parameters(), ‘lr’: 1}
], lr=1e-2, momentum=0.9)
</code></pre>
<p>I want to use <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.OneCycleLR" rel="noopener nofollow ugc">OneCycleLR</a> which needs <code>lr</code> in its parametrs. How should I pass two learning rates?</p>

NoobCoder: <p>Oh, that function accepts a list for each parameter group. I tested it, and it works fine.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8eb71ab3cd148f45f90a1014009e212e45a3dd22.png" data-download-href="https://discuss.pytorch.org/uploads/default/8eb71ab3cd148f45f90a1014009e212e45a3dd22" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8eb71ab3cd148f45f90a1014009e212e45a3dd22_2_690x396.png" alt="image" data-base62-sha1="kmw6chHmH0SCarAOQ4hjVSyuDCy" width="690" height="396" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8eb71ab3cd148f45f90a1014009e212e45a3dd22_2_690x396.png, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8eb71ab3cd148f45f90a1014009e212e45a3dd22.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8eb71ab3cd148f45f90a1014009e212e45a3dd22.png 2x" data-dominant-color="FAF9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">901×518 30.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Just one note that when you use OneCycleLR, the learning rate in the optimizer does not matter. What you put in <code>max_lr</code> determines your learning rate!</p>

jameslahm: <p><a class="mention" href="/u/noobcoder">@NoobCoder</a>  Hi, I encounter this problem too. Could you share how you set up the OneCycleLR for two different parameter groups with different learning rate? Thanks a lot!</p>