AttributeError: 'LambdaLR' object has no attribute 'param_groups' in deberta-v3-large

username: Shubham_Singh1
cooked: <p>AttributeError                            Traceback (most recent call last)<br>
 in ()<br>
1 for fold in range(5):<br>
----&gt; 2   training_loop(train_df.iloc[0:500,:],fold)<br>
3</p>
<p>4 frames<br>
 in training_loop(df, fold)<br>
30     for epoch in range(Cfg.epochs):<br>
31         criterion=nn.CrossEntropyLoss(weight=weights.to(torch.float16).to(device),reduction=‘none’)<br>
—&gt; 32         loss,preds,labels=train_fn(model,train_loader,scheduler,optimizer,criterion)<br>
33         print(f’training loss for {fold}th fold and {epoch+1} epoch is {loss}’)<br>
34         train_score=pearsonr(preds,labels)[0]</p>
<p> in train_fn(model, train_loader, optimizer, scheduler, criterion)<br>
24         scaler.scale(loss).backward()<br>
25         if (batch_idx+1)%Cfg.accumulation_steps==0 or (batch_idx+1)==len(train_loader):<br>
—&gt; 26             scaler.step(optimizer)<br>
27             scaler.update()<br>
28             optimizer.zero_grad()</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/cuda/amp/grad_scaler.py in step(self, optimizer, *args, **kwargs)<br>
332<br>
333         if optimizer_state[“stage”] is OptState.READY:<br>
 → 334             self.unscale_(optimizer)<br>
335<br>
336         assert len(optimizer_state[“found_inf_per_device”]) &gt; 0, “No inf checks were recorded for this optimizer.”</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/cuda/amp/grad_scaler.py in unscale_(self, optimizer)<br>
277         found_inf = torch.full((1,), 0.0, dtype=torch.float32, device=self._scale.device)<br>
278<br>
 → 279         optimizer_state[“found_inf_per_device”] = self.<em>unscale_grads</em>(optimizer, inv_scale, found_inf, False)<br>
280         optimizer_state[“stage”] = OptState.UNSCALED<br>
281</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/cuda/amp/grad_scaler.py in <em>unscale_grads</em>(self, optimizer, inv_scale, found_inf, allow_fp16)<br>
200         per_device_and_dtype_grads = defaultdict(lambda: defaultdict(list))  # type: ignore[var-annotated]<br>
201         with torch.no_grad():<br>
 → 202             for group in optimizer.param_groups:<br>
203                 for param in group[“params”]:<br>
204                     if param.grad is None:</p>
<p>AttributeError: ‘LambdaLR’ object has no attribute ‘param_groups’</p>

username: ptrblck
cooked: <p>Based on the error message it seems you are passing a scheduler to <code>scaler.step()</code> instead of an optimizer.</p>

username: Shubham_Singh1
cooked: <aside class="quote no-group" data-username="Shubham_Singh1" data-post="1" data-topic="150190">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shubham_singh1/48/48889_2.png" class="avatar"> Shubham_Singh1:</div>
<blockquote>
<p>26 scaler.step(optimizer)</p>
</blockquote>
</aside>
<p>thanks for reply!<br>
you can see the code i am passing scaler.step(optimizer) at line 26</p>

username: ptrblck
cooked: <p>The code doesn’t show what <code>optimizer</code> is, so I guess you are overriding it as seen here:</p>
<pre><code class="lang-python"># works
p = nn.Parameter(torch.randn(1).cuda())
optimizer = torch.optim.Adam([p], lr=1e-3)

scaler = torch.cuda.amp.GradScaler()

loss = p * 2
scaler.scale(loss).backward()
scaler.step(optimizer) # works

# breaks
p = nn.Parameter(torch.randn(1).cuda())
optimizer = torch.optim.Adam([p], lr=1e-3)
optimizer = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda epoch: 0.95 ** epoch)

scaler = torch.cuda.amp.GradScaler()

loss = p * 2
scaler.scale(loss).backward()
scaler.step(optimizer)
# AttributeError: 'LambdaLR' object has no attribute 'param_groups'
</code></pre>

username: Shubham_Singh1
cooked: <p>thanks for reply!</p>
<p>sorry if i misunderstood your comment ‘’ The code doesn’t show what <code>optimizer</code> is’’ are you asking which optimizer i am using or you are referring to something else.<br>
i am sure that i am not confusing scheduler with optimizer as you mentioned in your comment here  ‘optimizer = torch.optim.Adam([p], lr=1e-3)<br>
optimizer = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda epoch: 0.95 ** epoch)’’<br>
fyi i am using AdamW optimizer, with get_cosine_schedule_with_warmup(),</p>

username: ptrblck
cooked: <p>That’s interesting. Could you post a minimal, executable code snippet which would reproduce the issue, please?</p>