Using LR-Scheduler with param groups of different LR's

Alexander_Riedel: <p>Hey,<br>
I have defined the following optimizer with different learning rates for each parameter group:</p>
<pre><code class="lang-auto">  optimizer = optim.SGD([
          {'params': param_groups[0], 'lr': CFG.lr, 'weight_decay': CFG.weight_decay},
          {'params': param_groups[1], 'lr': 2*CFG.lr, 'weight_decay': 0},
          {'params': param_groups[2], 'lr': 10*CFG.lr, 'weight_decay': CFG.weight_decay},
          {'params': param_groups[3], 'lr': 20*CFG.lr, 'weight_decay': 0},
      ], lr=CFG.lr, momentum=0.9, weight_decay=CFG.weight_decay, nesterov=CFG.nesterov)
</code></pre>
<p>Now I want to use a LR-Scheduler to update all the learning rates and not only the first one, because by deafult, a scheduler would only update the param_groups[0]?</p>
<pre><code class="lang-auto">scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=5, T_mult=2, eta_min=CFG.min_lr, last_epoch=-1, verbose=True)
</code></pre>
<p>Giving me:</p>
<pre><code class="lang-auto">Parameter Group 0
    dampening: 0
    initial_lr: 0.001
    lr: 0.0009999603905218616
    momentum: 0.9
    nesterov: True
    weight_decay: 0.0001

Parameter Group 1
    dampening: 0
    initial_lr: 0.002
    lr: 0.002
    momentum: 0.9
    nesterov: True
    weight_decay: 0

Parameter Group 2
    dampening: 0
    initial_lr: 0.01
    lr: 0.01
    momentum: 0.9
    nesterov: True
    weight_decay: 0.0001

Parameter Group 3
    dampening: 0
    initial_lr: 0.02
    lr: 0.02
    momentum: 0.9
    nesterov: True
    weight_decay: 0
)
</code></pre>
<p>after on update.</p>
<p>Any idea how to update all the learning rates with a scheduler?</p>

Vignesh_Baskaran: <p>Hi <a class="mention" href="/u/alexander_riedel">@Alexander_Riedel</a>, did you find a way to do this?</p>

Alexander_Riedel: <p>Didn’t work on it anymore but you can find some more hints here: <a href="https://stackoverflow.com/questions/66711210/pytorch-using-lr-scheduler-with-param-groups-of-different-lrs" class="inline-onebox" rel="noopener nofollow ugc">python - PyTorch using LR-Scheduler with param groups of different LR's - Stack Overflow</a></p>

Kirtikumar_Pandya: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you please look into this!!</p>
<p>Thanking you.</p>

ptrblck: <p>I don’t think this statement:</p>
<aside class="quote no-group" data-username="Alexander_Riedel" data-post="1" data-topic="115356">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexander_riedel/48/36027_2.png" class="avatar"> Alexander_Riedel:</div>
<blockquote>
<p>Now I want to use a LR-Scheduler to update all the learning rates and not only the first one, because by deafult, a scheduler would only update the param_groups[0]?</p>
</blockquote>
</aside>
<p>is true, since the <code>CosineAnnealingWarmRestarts</code> would iterate all <code>.param_groups</code> as seen <a href="https://github.com/pytorch/pytorch/blob/728dfeee486fbd965710ccbb225fd275dd7bd35c/torch/optim/lr_scheduler.py#L1456">here</a>. Let me know if I’m missing something.</p>

Kirtikumar_Pandya: <pre><code class="lang-auto">opt = AdamW([
            {'params':[*self.block.parameters()], 'lr':1e-3}, 
            {'params':self.bert.parameters(), 'lr':1e-4}
        ])
sch = CosineAnnealingLR(opt, T_max=len_train_df/(batch_size*n_epochs), verbose=True)
</code></pre>
<p>In this case it will update all parameter groups. I cannot set different schedulers for different param groups. Right!!!</p>

ptrch_c_m: <p>Any update on this?</p>
<p>What I am trying to do is to use a scheduler only on the first dictionary passed to the definition of the optimizer, in other words, I want to use a scheduler only on <code>param_group 0</code>.</p>
<p>Is this possible?</p>