Half precision training

username: eslam_fouda
cooked: <p>I would like to know if these lines of code arranged correctly:</p>
<pre><code>lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, len(dataloader))               
for batch_i, (_, imgs, targets) in enumerate(dataloader):
    optimizer.zero_grad() 
    with torch.cuda.amp.autocast(enabled=use_amp):
        imgs = Variable(imgs.type(Tensor))
        targets = Variable(targets.type(Tensor), requires_grad=False)
        loss = model(imgs, targets)
    
    scaler.scale(loss).backward()

    # accumulate gradient for x batches before optimizing
    if ((batch_i + 1) % accumulated_batches == 0) or (batch_i == len(dataloader) - 1):
        # Unscales the gradients of optimizer's assigned params in-place
        scaler.unscale_(optimizer)

        # Since the gradients of optimizer's assigned params are now unscaled, clips as usual.
        # You may use the same value for max_norm here as you would without gradient scaling.
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=0.1)
        scaler.step(optimizer)
        lr_scheduler.step()
        scaler.update()
        optimizer.zero_grad() # set_to_none=True here can modestly improve performance
</code></pre>
<p>Thanks</p>

username: ptrblck
cooked: <p>The amp usage from your code snippet matches the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation">AMP - Gradient accumulation example</a> so I think it’s correct from this point of view.</p>
<p>However, from a general perspective, I think you should remove the first <code>zero_grad()</code> operation, as it would remove the accumulated gradients.<br>
Also, <code>Variables</code> are deprecated since PyTorch <code>0.4</code>, so you can use tensors now.</p>

username: eslam_fouda
cooked: <p>Thanks for your reply<br>
I would also like to know when I resume training algorithm like yolov3 with darknet backbone should I use model.load_weights(darknet weights) or use model.load_state_dict(checkpoint[‘model’]) and what this should do with this piece of code:</p>
<h1>Freeze darknet53.conv.74 layers for first some epochs</h1>
<pre><code>if freeze_backbone:
    if epoch &lt; 20:
        for i, (name, p) in enumerate(model.named_parameters()):
            if int(name.split('.')[1]) &lt; 75:  # if layer &lt; 75
                p.requires_grad = False
    elif epoch &gt;= 20:
        for i, (name, p) in enumerate(model.named_parameters()):
            if int(name.split('.')[1]) &lt; 75:  # if layer &lt; 75
                p.requires_grad = True
</code></pre>
<p>Also when I wrote  imgs = torch.tensor(imgs, requires_grad=True) it said its recommended to use imgs.clone().detach().requires_grad_(True) then I got:<br>
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="eslam_fouda" data-post="3" data-topic="113920">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eslam_fouda/48/30589_2.png" class="avatar"> eslam_fouda:</div>
<blockquote>
<p>I would also like to know when I resume training algorithm like yolov3 with darknet backbone should I use model.load_weights(darknet weights) or use model.load_state_dict(checkpoint[‘model’])</p>
</blockquote>
</aside>
<p>I don’t know, how <code>load_weights</code> is defined, as it’s not an <code>nn.Module</code> method so I assume it’s a custom darknet method?</p>
<aside class="quote no-group" data-username="eslam_fouda" data-post="3" data-topic="113920">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eslam_fouda/48/30589_2.png" class="avatar"> eslam_fouda:</div>
<blockquote>
<p>and what this should do with this piece of code:</p>
</blockquote>
</aside>
<p>This code snippet freezes some weights for 20 epochs. I would guess you would have to keep these weights frozen until the model was trained for 20 epochs. Depending when the model was saved (before or after reaching 20 epochs) you might need to add it.</p>
<aside class="quote no-group" data-username="eslam_fouda" data-post="3" data-topic="113920">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eslam_fouda/48/30589_2.png" class="avatar"> eslam_fouda:</div>
<blockquote>
<p>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same</p>
</blockquote>
</aside>
<p>This error points to a device mismatch, so make sure the input tensor is on the GPU via:</p>
<pre data-code-wrap="python"><code class="lang-python">input = input.to('cuda:0') # change 'cuda:id' to the desired GPUid
</code></pre>

username: eslam_fouda
cooked: <p>Thanks for your reply<br>
when reaching 20 epochs with frozen weights, should I stop using darknet weights and use model.load_state_dict(checkpoint[‘model’])</p>

username: ptrblck
cooked: <p>If “darknet weights” refer to a pretrained model, and you’ve fine-tuned it for 20 epochs, I would assume you want to load your fine-tuned weights instead. So yes, I guess you might want to use <code>model.load_state_dict(your_fientuned_checkpoint)</code>, but it depends on your actual use case.</p>