Torch DDP Multi-GPU gives low accuracy metric

username: grid_world
cooked: <p>I am trying Multi-GPU, single machine DDP training in PyTorch (CIFAR-10 + ResNet-18 setup). You can refer to the model architecture code <a href="https://github.com/arjun-majumdar/CNN_Classifications/blob/master/ResNet18_swish_torch.py" rel="noopener nofollow ugc">here</a> and the full training code <a href="https://github.com/arjun-majumdar/CNN_Classifications/blob/master/multigpu_singlemachine_ddp_torch.py" rel="noopener nofollow ugc">here</a>.</p>
<p>Within main() function, the training loop is:</p>
<pre><code>for epoch in range(1, num_epochs + 1):
        # Initialize metric for metric computation, for each epoch-
        running_loss = 0.0
        running_corrects = 0.0
    
        model.train()

        # Inform DistributedSampler about current epoch-
        train_loader.sampler.set_epoch(epoch)

        # One epoch of training-
        for batch_idx, (images, labels) in enumerate(train_loader):
            images = images.to(rank)
            labels = labels.to(rank)

            # Get model predictions-
            outputs = model(images)
              
            # Compute loss-
            J = loss(outputs, labels)
                
            # Empty accumulated gradients-
            optimizer.zero_grad()
                
            # Perform backprop-
            J.backward()
                
            # Update parameters-
            optimizer.step()

            '''                
            global step
            optimizer.param_groups[0]['lr'] = custom_lr_scheduler.get_lr(step)
            step += 1
            ''' 
            
            # Compute model's performance statistics-
            running_loss += J.item() * images.size(0)
            _, predicted = torch.max(outputs, 1)
            running_corrects += torch.sum(predicted == labels.data)
            
        train_loss = running_loss / len(train_dataset)
        train_acc = (running_corrects.double() / len(train_dataset)) * 100
        print(f"GPU: {rank}, epoch = {epoch}; train loss = {train_loss:.4f} &amp; train accuracy = {train_acc:.2f}%")
</code></pre>
<p>The problem is that the train accuracy being computed in this way is very low (say only 7.44% on average) across 8 GPUs. But, when I obtain the saved model and test its accuracy with the following code:</p>
<pre><code>def test_model_progress(model, test_loader, test_dataset):
    total = 0.0
    correct = 0.0
    running_loss_val = 0.0

    with torch.no_grad():
        with tqdm(test_loader, unit = 'batch') as tepoch:
            for images, labels in tepoch:
                tepoch.set_description(f"Validation: ")
                
                images = images.to(device)
                labels = labels.to(device)
                
                # Set model to evaluation mode-
                model.eval()
            
                # Predict using trained model-
                outputs = model(images)
                _, y_pred = torch.max(outputs, 1)
                
                # Compute validation loss-
                J_val = loss(outputs, labels)
                
                running_loss_val += J_val.item() * labels.size(0)
    
                # Total number of labels-
                total += labels.size(0)

                # Total number of correct predictions-
                correct += (y_pred == labels).sum()
                
                tepoch.set_postfix(
                    val_loss = running_loss_val / len(test_dataset),
                    val_acc = 100 * (correct.cpu().numpy() / total)
                )
            
        
    # return (running_loss_val, correct, total)
    val_loss = running_loss_val / len(test_dataset)
    val_acc = (correct / total) * 100

    return val_loss, val_acc.cpu().numpy()

test_loss, test_acc = test_model_progress(trained_model, test_loader, test_dataset)

print(f"ResNet-18 (multi-gpu DDP) test metrics; loss = {test_loss:.4f} &amp; acc = {test_acc:.2f}%")
# ResNet-18 (multi-gpu DDP) test metrics; loss = 1.1924 &amp; acc = 59.88%
</code></pre>
<p>Why is there this discrepancy? What am I missing?</p>

username: fduwjj
cooked: <aside class="quote no-group" data-username="grid_world" data-post="1" data-topic="189430">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/grid_world/48/33204_2.png" class="avatar"> grid_world:</div>
<blockquote>
<p><code>running_loss</code></p>
</blockquote>
</aside>
<p>If train your model without DDP, did you see something similar?</p>

username: grid_world
cooked: <p>What do you mean by something similar?</p>
<p>On training with a single GPU, the metrics don’t need to be aggregated. And, the more worrisome point seems to be that with everything else remaining same, single GPU training gives a train accuracy of about 81%, while multi-GPU is stuck at under 60%</p>
<p>Why is there a huge gap in performance for multi-GPU training with DDP?</p>

username: fduwjj
cooked: <p>OK, sounds like training accuracy of training on a single GPU is not same as that trained on DDP. How about test accuracy of single GPU vs DDP? Are the same or not?</p>

username: grid_world
cooked: <p>They are not the same: test accuracy after 50 epochs of multi-GPU training = 57.28% (approx), while for single GPU, the test accuracy = 79.75% (approx). So, there seems to be a delta of 22% (approx) which is huge!</p>

username: fduwjj
cooked: <p>I see… Interesting… So have you try to reduce the batch size on rank GPU when using DDP? The way DDP works is that we give different input on each GPU while the model is the same (copied) across all GPUs. We do a gradient average in BWD so I am not sure if using original batch size can affect model learning negatively.</p>
<p>After reading through your training code, I have a guess here.<br>
You might want to check whether the way you set lr in the optimizer is expected or not. Because the param groups’ FQN might be changed after wrapping with DDP. The rest part looks good to me.</p>

username: grid_world
cooked: <p>Can you elaborate on the optimizer part? Why will param_group’s FQN change due to DDP?</p>

username: maxim.yudayev
cooked: <p>Your <code>running_loss</code> and <code>dataloader</code> are local to each GPU in the DDP setup. Calling <code>len(dataloader)</code> will give you the size of the local shard of the original dataset (size of the full dataset divided by the number of GPUs you are using), since the distributed data sampler shards the dataset into equal portions across GPUs. The contents of <code>running_loss</code> is a local variable in each separate process (rank): to obtain numerically accurate results, you will have to aggregate the running losses on the master node by calling <code>reduce(ReduceOp.SUM)</code> operation from the torch.distributed package.<br>
After you do that, you can investigate if you need to change the batch size by dividing by the number of GPUs or to alter the learning rate, to achieve the same numerical result as when training on a single GPU.</p>
<p>TLDR; I am fairly confident the models are nearly identical, you simply forgot to reduce (exchange) local metrics across the ranks and to use the full dataset length in your accuracy calculation (<code>len(dataloader.__dataset__)</code> or <code>len(dataloader) * world_size</code>).</p>

username: Ed_Moman
cooked: <p>I do not know if this could be relevant:</p>
<p><a href="https://torchmetrics.readthedocs.io/en/stable/pages/overview.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://torchmetrics.readthedocs.io/en/stable/pages/overview.html</a></p>
<blockquote>
<h2>Metrics in Distributed Data Parallel (DDP) mode</h2>
<p>When using metrics in <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">Distributed Data Parallel (DDP)</a> mode, one should be aware that DDP will add additional samples to your dataset if the size of your dataset is not equally divisible by <code>batch_size * num_processors</code>. The added samples will always be replicates of datapoints already in your dataset. This is done to secure an equal load for all processes. However, this has the consequence that the calculated metric value will be slightly biased towards those replicated samples, leading to a wrong result.</p>
<p>During training and/or validation this may not be important, however it is highly recommended when evaluating the test dataset to only run on a single gpu or use a <a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel.join" rel="noopener nofollow ugc">join</a> context in conjunction with DDP to prevent this behaviour.</p>
</blockquote>

username: grid_world
cooked: <p>Is there any example code I can refer to?</p>

username: maxim.yudayev
cooked: <p><a href="https://pytorch.org/docs/stable/distributed.html#collective-functions" rel="noopener nofollow ugc">Here is</a> an exhaustive list of available PyTorch communication primitives. For a sample code, I took the liberty in adapting your excerpt (the idea is identical for your test script)</p>
<pre><code class="lang-python">for epoch in range(1, num_epochs + 1):
    # Initialize metric for metric computation, for each epoch-
    running_loss = 0.0
    running_corrects = 0.0
    
    model.train()
    
    train_loader.sampler.set_epoch(epoch)

    # One epoch of training-
    for batch_idx, (images, labels) in enumerate(train_loader):
        images = images.to(rank)
        labels = labels.to(rank)

        # Get model predictions-
        outputs = model(images)
              
        # Compute loss-
        J = loss(outputs, labels)
                
        # Empty accumulated gradients-
        optimizer.zero_grad()
                
        # Perform backprop-
        J.backward()
                
        # Update parameters-
        optimizer.step()

        '''                
        global step
        optimizer.param_groups[0]['lr'] = custom_lr_scheduler.get_lr(step)
        step += 1
        ''' 
            
        # Compute model's performance statistics on each rank
        running_loss += J.item() * images.size(0)
        _, predicted = torch.max(outputs, 1)
        running_corrects += torch.sum(predicted == labels.data)

    # to globally reduce local metrics across ranks, they should be Tensors
    running_loss = torch.tensor([running_loss], device=self.rank)         
    running_corrects = torch.tensor([running_corrects], device=self.rank)

    if torch.cuda.is_available():
        reduce(running_loss, dst=0, op=torch.distributed.ReduceOp.SUM)
        reduce(running_corrects, dst=0, op=torch.distributed.ReduceOp.SUM)

    # will log the aggregated metrics only on the 0th GPU. Make sure "train_dataset" is of type Dataset and not DataLoader to get the size of the full dataset and not of the local shard
    if rank==0:
        train_loss = running_loss / len(train_dataset)
        train_acc = (running_corrects.double() / len(train_dataset)) * 100     
        print(f"GPU: {rank}, epoch = {epoch}; train loss = {train_loss:.4f} &amp; train accuracy = {train_acc:.2f}%")
</code></pre>