Correct accuracy calculation with DistributedDataParallel

username: numpee
cooked: <p>Hi,<br>
I’m using DistributedDataParallel to train a simple classification model. I have some experience with distributed training, but I can’t seem to wrap my head around one specific detail.</p>
<p>Let me refer you to an example provided by PyTorch: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L338" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at master · pytorch/examples · GitHub</a><br>
Here, you will see that the accuracy is calculated by a <code>accuracy()</code> function, and the average accuracy is updated using the <code>AverageMeter</code> in the following lines.<br>
From my understanding, this calculates the accuracy for the samples that <strong>each GPU</strong> receives, <strong>not the accuracy of samples across all GPUs</strong>. So, this function returns <code>top1.avg</code>, and now we head over to <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L248" rel="noopener nofollow ugc">L248</a>, where the model is saved <strong>if the accuracy from GPU rank 0 is larger than the best accuracy</strong>.</p>
<p>Am I going crazy, or is this intended behavior? Are we assuming that all GPUs receive the same samples, or that the accuracy on GPU0 is somehow representative of entire accuracy?</p>
<p>To show that my interpretation is correct, I wrote a short sandbox code that mimics the code attached above. The <code>AverageMeter</code> and <code>accuracy()</code> functions were all copied from the linked code base. This assumes a 2-class classification scenario, with <code>batch_size=5</code>, and I ran it on 4 GPUs:</p>
<pre><code class="lang-python">acc_meter = AverageMeter()
model = nn.Linear(10, 2).cuda(gpu)
model = DistributedDataParallel(model, device_ids=[gpu])
a = torch.randn(5, 10).cuda(gpu)
gt = torch.randint(0, 2, (5, 1)).cuda(gpu)
outputs = model(a)
acc = accuracy(outputs, gt, topk=(1,))
acc_meter.update(acc[0], a.size(0))
print("Avg: ", acc_meter.avg)
print("Sum: ", acc_meter.sum)
print("Count: ", acc_meter.count)
return acc_meter.avg
</code></pre>
<p>There are two issues:</p>
<ol>
<li>As suspected, returning <code>acc.avg</code> will only return the accuracy for current GPU. This means that saving a checkpoint or logging from <code>rank=0</code> will only checkpoint or log the accuracy from <code>rank=0</code>.</li>
<li>The accuracy calculation is wrong. The <code>accuracy()</code> function divides by the <code>batch_size</code>, so returning <code>acc_meter.avg</code> divides by the <code>batch_size</code> again. The return value should be <code>acc_meter.sum</code>.</li>
</ol>
<p>Ultimately, I would like to write code that uses <code>DistributedDataParallel</code> but can compute the accuracy correctly. For now, I have resorted to the following method:</p>
<ol>
<li>Compute <code>num_correct</code> for all GPUs</li>
<li>all reduce <code>num_correct</code>, as well as <code>num_samples</code> as such: <code>dist.all_reduce(num_correct), dist.all_reduce(num_samples)</code>. For this step, <code>num_samples</code> must be cast to GPU first.</li>
<li>Cast back CPU, then update the average meter.</li>
</ol>
<p>To me, this does not seem like an elegant solution. Perhaps this could mean creating an <code>AverageMeter</code> that can handle Distributed updates? In search of a more elegant solution, I’ve looked at multiple code bases, but they all seem to do it incorrectly, as shown above. Am I completely missing something big here? If anyone has suggestions/solutions, I would love to hear from you.</p>

username: rvarm1
cooked: <p>Thanks for your question!</p>
<p>In the code you linked, a <code>DistributedSampler</code> is used to ensure that each GPU is fed different data which is the typical scenario in DDP training.</p>
<blockquote>
<p>or that the accuracy on GPU0 is somehow representative of entire accuracy<br>
In general, we take the model on an arbitrary GPU and use that to measure the accuracy, in this case, GPU 0. The actual model is the same across all GPUs though since gradients are synchronized.</p>
</blockquote>
<blockquote>
<p>The <code>accuracy()</code> function divides by the <code>batch_size</code> , so returning <code>acc_meter.avg</code> divides by the <code>batch_size</code> again. The return value should be <code>acc_meter.sum</code> .</p>
</blockquote>
<p>Dones’t <code>acc_meter.avg</code> take the average accuracy across all the updates? Or does it actually also divide by the batch size? If so, this seems incorrect and an issue should be filed in the pytorch/examples repo.</p>
<p>Your approach if you would like to get the accuracy across all ranks seems correct. In general, maybe it would be good to have something like <code>DistributedAverageMeter</code>. Although in most training scenarios I’ve come across, it’s enough to evaluate the model on only one rank during training and then the entire dataset during eval. This of course assumes the data is shuffled properly and one rank doesn’t get all positive/negative samples, for example.</p>

username: numpee
cooked: <p>Hi, thanks for the reply.</p>
<p>Regarding <code>DistributedSampler</code>:<br>
What you mentioned is exactly the reason why I raised the question. Since each GPU is fed different data, the accuracy on GPU0 is not necessarily the accuracy on the entire dataset. But as you mentioned, if it is enough to evaluate the model on a single rank, this shouldn’t be an issue.</p>
<p>Regarding <code>AverageMeter</code>:<br>
The  <code>acc_meter.avg</code> should take the average accuracy across all updates, but it actually divides by the batch size again. If you look on <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L340" rel="noopener nofollow ugc">L340</a>, <code>images.size(0)</code> is passed as a second parameter. On the <code>update()</code> function at <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L376" rel="noopener nofollow ugc">L376</a>, the variable <code>n</code> represents the number of values (the count), and the <code>sum</code> is divided by the <code>count</code> to compute the average. Thus, we end up dividing by the batch size twice: once in the accuracy function, and once in the average meter. The second parameter in L340 should be left blank. But again, this is assuming that all batch sizes are same (sometimes the last batch is smaller).</p>
<p>I’ve raised an <a href="https://github.com/pytorch/examples/issues/905" rel="noopener nofollow ugc">issue</a> on the repo. I could open a pull request as well. The accuracy issue looks like a minor fix, but adding a <code>DistributedAverageMeter</code> may need some work <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>