(Distributed) Why all gpus are giving the same output?

Wayne_Mai: <p>Hi, I am a new comer to PyTorch and I’m confused when I am running the official example of <code>torch.distributed</code><br>
at <a href="https://github.com/pytorch/examples/blob/d587b53f3604b029764f8c864b6831d0ab269008/imagenet/main.py#L304" rel="nofollow noopener">PyTorch ImageNet main.py L304</a>.</p>
<p>I have made some small modification on the evaluation part of the source code like below:</p>
<pre><code class="lang-auto">model.eval()
    with torch.no_grad():
        end = time.time()
        for i, (images, target, image_ids) in enumerate(val_loader):
            if args.gpu is not None:
                images = images.cuda(args.gpu, non_blocking=True)

            target = target.cuda(args.gpu, non_blocking=True)
            image_ids = image_ids.data.cpu().numpy()
            output = model(images)
            loss = criterion(output, target)

            # Get acc1, acc5 and update
            acc1, acc5 = accuracy(output, target, topk=(1, 5))
            losses.update(loss.item(), images.size(0))
            top1.update(acc1[0], images.size(0))
            top1.update(acc1[0], images.size(0))
            top5.update(acc5[0], images.size(0))
            
            # print at i-th batch of images only
            dist.barrier()
            if i==0:
                if args.gpu==0:
                    print("gpu 0",acc1,output.shape)
                if args.gpu==1:
                    print("gpu 1",acc1,output.shape)
                if args.gpu==2:
                    print("gpu 2",acc1,output.shape)
                if args.gpu==3:
                    print("gpu 3",acc1,output.shape)
</code></pre>
<p>And above code gives the following output:</p>
<pre><code class="lang-auto">Use GPU: 0 for training
Use GPU: 1 for training
Use GPU: 3 for training
Use GPU: 2 for training
=&gt; loading checkpoint model_best.pth.tar'
...
gpu 3 tensor([75.], device='cuda:3') torch.Size([32, 200])
gpu 2 tensor([75.], device='cuda:2') torch.Size([32, 200])
gpu 1 tensor([75.], device='cuda:1') torch.Size([32, 200])
gpu 0 tensor([75.], device='cuda:0') torch.Size([32, 200])
</code></pre>
<p>As I am using <strong>4 GPU</strong> with a batch size of <strong>128</strong>, I think 128 images have been divided and fed into 4 GPU respectively. So all the four GPU have <code>output.shape[0]=32</code><em>(where 200 is num_classes).</em></p>
<p>But what has really confused me is that, all the 4 GPU are showing the same <code>acc1</code>. In my understanding, as 4 GPUs are taking different input portion (32 images respectively), they should also give different output and accuracy corresponding to their input respectively. However, in my print test, these GPU are showing the same output and accuracy. And I don’t know why, shouldn’t they be different ?</p>
<p>Looking for help. Thank you in advance !</p>

Wayne_Mai: <p>Okay, I think that maybe I have found the answer at its Github issues.<a href="https://github.com/pytorch/examples/issues/461" rel="nofollow noopener">distributed eval to be done</a></p>

pietern: <p>That’s correct. Without a distributed sampler for the evaluation dataset, the different processes end up processing the same evaluation inputs, and correspondingly give the same accuracy.</p>