Lower accuracy after speeding up training process

hyman9090: <p>I am training an object detection model (draw bounding box around the target object and tell me what class it is) with a very common training structure like this below:</p>
<pre><code class="lang-auto">for i, (inputs, labels) in enumerate(data_loader):

    optimizer.zero_grad()

    inputs = inputs.to(device)
    labels = labels.to(device)

    preds = model(inputs)
    loss  = criterion(preds, labels)

    loss.backward() 

    optimizer.step()
    
</code></pre>
<p>The training is very time-consuming.<br>
And due to hardware limitations, I found some strategies that can speed up my training process so as to reduce training time. I have tried it but the accuracy is worse than that of before implementing it.<br>
(Before implementation, though the predicted class in incorrect, the bounding box can draw at an approximately correct position. But after implementation, both are incorrect.)</p>
<p>These are the strategies I tried to implement – 1) enable <code>num_workers</code>, 2) set <code>pin_memory=True</code>, 3) use cuDNN Autotuner, 4) use Automatic Mixed Precision (AMP), 5) use gradient accumulation</p>
<p>Below is the modified code:</p>
<pre><code class="lang-auto">accumulation_steps = 4
torch.backends.cudnn.benchmark = True

data_loader = DataLoader(..., num_workers=num_workers, pin_memory=True)

for i, (inputs, labels) in enumerate(data_loader):

    optimizer.zero_grad()

    inputs = inputs.to(device)
    labels = labels.to(device)

    with autocast():
        preds = model(inputs)
        loss  = criterion(preds, labels)
        loss = loss/accumulation_steps

    scaler.scale(loss).backward()

    if ((i+1) % accumulation_steps == 0) or (i+ 1 == len(data_loader)):
        scaler.step(optimizer)
        scaler.update()
    
</code></pre>
<p>The strategies are only to speed up the training process, it should have no impact on the accuracy of the model. But I really have no idea why the performance is getting so worse after that. Can someone help me out with this?</p>

ptrblck: <p>Could you try to use these implementations separately and see how the results would change?</p>

hyman9090: <p>I am trying to take out the implementation one by one to see the result.</p>

hyman9090: <p>Found that there mentioned the AMP and gradient accumulation.<br>
<a href="https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation</a></p>
<p>The problem might have come from the cuDNN. I will try it out to disable it.</p>

oke-aditya: <p>Can you try training with torchvision reference scripts? They should have a proper recipe to get good results in decent time.</p>
<aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/vision/tree/main/references/detection">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/vision/tree/main/references/detection" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/vision/tree/main/references/detection" target="_blank" rel="noopener nofollow ugc">vision/references/detection at main · pytorch/vision</a></h3>

  <p><a href="https://github.com/pytorch/vision/tree/main/references/detection" target="_blank" rel="noopener nofollow ugc">main/references/detection</a></p>

  <p><span class="label1">Datasets, Transforms and Models specific to Computer Vision</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>