optimizer:No inf checks were recorded for this optimizer

username: StawEndl
cooked: <p>i want to get mask by yolov5, so make some change in yolov5. but get the error like this:</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 713, in <br>
main(opt)<br>
File “train.py”, line 610, in main<br>
train(opt.hyp, opt, device, callbacks)<br>
File “train.py”, line 411, in train<br>
scaler_mask.step(optimizer_mask)  # optimizer.step<br>
File “/home/mopanzhong/.conda/envs/rope_detection/lib/python3.7/site-packages/torch/cuda/amp/grad_scaler.py”, line 318, in step<br>
assert len(optimizer_state[“found_inf_per_device”]) &gt; 0, “No inf checks were recorded for this optimizer.”</p>
<p>train.py:</p>
<p>optimizer.zero_grad()<br>
optimizer_mask.zero_grad()<br>
for …:<br>
…<br>
with amp.autocast(enabled=cuda):<br>
pred, mask_res = model(imgs, get_mask_res=True)  # forward<br>
loss, loss_mask, loss_items = compute_loss(pred, targets.to(device), mask_res=mask_res,img_masks=img_masks)  # loss scaled by batch_size</p>
<pre><code>        # Backward
        print("$"*20, loss, loss_mask)  #$$$$$$$$$$$$$$$$$$$$ tensor([3.15543], device='cuda:0', grad_fn=&lt;MulBackward0&gt;) tensor(0.74285, device='cuda:0', grad_fn=&lt;MeanBackward0&gt;)
        scaler.scale(loss).backward()
        scaler_mask.scale(loss_mask).backward()

        # Optimize
        if ni - last_opt_step &gt;= accumulate:
            print("%"*20,optimizer)
            print("%" * 20, optimizer_mask)
</code></pre>
<h3><a name="h-1" class="anchor" href="#h-1"></a></h3>
<p>print:<br>
%%%%%%%%%%%%%%%%%%%% SGD (<br>
Parameter Group 0<br>
dampening: 0<br>
initial_lr: 0.01<br>
lr: 0.0<br>
momentum: 0.8<br>
nesterov: True<br>
weight_decay: 0</p>
<p>Parameter Group 1<br>
dampening: 0<br>
initial_lr: 0.01<br>
lr: 0.0<br>
momentum: 0.8<br>
nesterov: True<br>
weight_decay: 0.0005</p>
<p>Parameter Group 2<br>
dampening: 0<br>
initial_lr: 0.01<br>
lr: 0.1<br>
momentum: 0.8<br>
nesterov: True<br>
weight_decay: 0<br>
)<br>
%%%%%%%%%%%%%%%%%%%% SGD (<br>
Parameter Group 0<br>
dampening: 0<br>
initial_lr: 0.01<br>
lr: 0.0<br>
momentum: 0.8<br>
nesterov: True<br>
weight_decay: 0</p>
<p>Parameter Group 1<br>
dampening: 0<br>
initial_lr: 0.01<br>
lr: 0.0<br>
momentum: 0.8<br>
nesterov: True<br>
weight_decay: 0.0005</p>
<p>Parameter Group 2<br>
dampening: 0<br>
initial_lr: 0.01<br>
lr: 0.1<br>
momentum: 0.8<br>
nesterov: True<br>
weight_decay: 0<br>
)</p>
<h3><a name="h-2" class="anchor" href="#h-2"></a></h3>
<pre><code>            scaler.step(optimizer)  # optimizer.step
            scaler.update()
            scaler_mask.step(optimizer_mask)  # optimizer.step
            scaler_mask.update()
            optimizer.zero_grad()
            optimizer_mask.zero_grad()
            if ema:
                ema.update(model)
            last_opt_step = ni

.....
</code></pre>
<p>and compute_loss:<br>
…<br>
# mask_losses is a list, like [1,2,3,4,5]<br>
mask_loss = torch.tensor(mask_losses, device=device, requires_grad=True)<br>
mask_losses = torch.tensor(mask_losses, device=device).mean(0,keepdim=True)<br>
print(mask_losses)<br>
print(lbox, lobj, lcls)<br>
<span class="hashtag">#print</span>(torch.cat((lbox, lobj, lcls)).detach())<br>
return (lbox + lobj + lcls) * bs, mask_loss.mean(), torch.cat((lbox, lobj, lcls, mask_losses)).detach()</p>