How to calculate F1 score, Precision in DDP

username: Saurav_Gupta1
cooked: <p>Hi, I am new to the concept of DDP. I am currently training my model on two GPUs.</p>
<ol>
<li>
<p>If I train on a single GPU with a batch size of <code>b</code>, do I need to divide this batch size by the number of GPUs available for training in DDP.</p>
</li>
<li>
<p>How can I calculate F1 score, Precision, Recall for a model being trained in DDP?</p>
</li>
<li>
<p>If I store local loss of two GPUs in two arrays. Is okay if I add them and divide by number of GPUs to get an average?</p>
</li>
</ol>

username: mrshenli
cooked: <blockquote>
<ol>
<li>If I train on a single GPU with a batch size of <code>b</code>, do I need to divide this batch size by the number of GPUs available for training in DDP.</li>
</ol>
</blockquote>
<p>Yep, you can start with dividing the batch size. But depending on the loss function and whether each process is consuming the same number of samples per iteration, DDP may or may not give you the exactly same result as local training. See this discussion: <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769" class="inline-onebox">Should we split batch_size according to ngpu_per_node when DistributedDataparallel</a></p>
<blockquote>
<ol start="2">
<li>How can I calculate F1 score, Precision, Recall for a model being trained in DDP?</li>
</ol>
</blockquote>
<p>DDP does not change the behavior of the forward pass. So, these metrics can be calculated similar to local training. But since now the outputs and loss locate on multiple GPUs, you might need to <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.gather" rel="noopener nofollow ugc">gather</a>/<a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather" rel="noopener nofollow ugc">allgather</a> them first if you need global numbers.</p>
<blockquote>
<ol start="3">
<li>If I store local loss of two GPUs in two arrays. Is okay if I add them and divide by number of GPUs to get an average?</li>
</ol>
</blockquote>
<p>Similar to the 1st bullet, this depends on your loss function. If it’s sth like MSE, then yes, the average of two local loss should be the sane of the global one. But other loss functions might not have this property.</p>

username: Saurav_Gupta1
cooked: <p>Since I am using multiple GPUs with nccl backend <strong>gather</strong> does not work so I tried using <strong>all_gather</strong> and below is a part of my code</p>
<p>code:</p>
<pre><code class="lang-auto">loss = criterion(output, targets)
tmp = [torch.empty_like(loss).cuda(rank) for _ in range(2)]
dist.all_gather(tmp, loss)
print(tmp[0].data.item())
loss.backward()
optimizer.step()
print(f'Model on GPU {rank}, Epoch: {epoch}--{i}/{len(dataloader)}],  Loss: {loss.data.item()}')
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">0.9214748740196228
0.9214748740196228
Model on GPU 0, Epoch: 1--0/28125],  Loss: 0.9214748740196228,  Batch_time_Average - 4.840
Model on GPU 1, Epoch: 1--0/28125],  Loss: 0.9064291715621948,  Batch_time_Average - 4.781
0.7848501801490784
0.7848501801490784
Model on GPU 0, Epoch: 1--1/28125],  Loss: 0.7848501801490784,  Batch_time_Average - 6.893
Model on GPU 1, Epoch: 1--1/28125],  Loss: 0.6567432880401611,  Batch_time_Average - 6.798
0.7931838035583496
0.7931838035583496
Model on GPU 0, Epoch: 1--2/28125],  Loss: 0.7931838035583496,  Batch_time_Average - 8.924
Model on GPU 1, Epoch: 1--2/28125],  Loss: 0.825346052646637,  Batch_time_Average - 8.835
1.0175780057907104
1.0175780057907104
Model on GPU 0, Epoch: 1--3/28125],  Loss: 1.0175780057907104,  Batch_time_Average - 10.966
Model on GPU 1, Epoch: 1--3/28125],  Loss: 0.5258045196533203,  Batch_time_Average - 10.868
</code></pre>
<p>tmp is just printing loss value at GPU 0. I am using Categorical cross entropy. I am not sure if this is the required output.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/saurav_gupta1">@Saurav_Gupta1</a>, the usage on <code>torch.distributed.all_gather</code> looks correct to me. One thing I wanna mention is that <code>torch.distributed.all_gather</code> is not an autograd function, so running backward on gathered tensors (i.e., <code>tmp</code> in your code) won’t reach the autograd graph prior to the <code>all_gather</code> operation. If you really need autograd to extend beyond comm ops, you can try this experimental autograd-powered <a href="https://github.com/pytorch/pytorch/blob/7a8c64da4d93fef5986aee5ada59289b5387cf8e/torch/distributed/nn/functional.py#L82" rel="noopener nofollow ugc"><code>all_gather</code></a>.</p>

username: Saurav_Gupta1
cooked: <p>My intension here is only to gather information regarding loss, accuracy and calculate F1 score, precision. I don’t want to run backward on gathered data. Am I gathering loss correctly? I am asking because in the above example tmp contains loss from GPU:0 but in some other run tmp contains loss of GPU:1</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="Saurav_Gupta1" data-post="5" data-topic="110065">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saurav_gupta1/48/32817_2.png" class="avatar"> Saurav_Gupta1:</div>
<blockquote>
<p>Am I gathering loss correctly?</p>
</blockquote>
</aside>
<p>It looks correct to me.</p>
<blockquote>
<p>I am asking because in the above example tmp contains loss from GPU:0 but in some other run tmp contains loss of GPU:1</p>
</blockquote>
<p>Could you please share a repro where <code>tmp[0]</code> contains loss from rank 1? The above output seems only shows loss from rank 0?</p>

username: Saurav_Gupta1
cooked: <p>Code:</p>
<pre><code class="lang-auto">loss = criterion(output, targets)
tmp = [torch.empty_like(loss).cuda(rank) for _ in range(2)]
dist.all_gather(tmp, loss)
print(tmp)
loss.backward()
optimizer.step()
print(f'Model on GPU {rank}, Epoch: {epoch}--{i}/{len(dataloader)}],  Loss: {loss.data.item()}')
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">[tensor(0.6518, device='cuda:0'), tensor(0.7940, device='cuda:0')]
[tensor(0.6518, device='cuda:1'), tensor(0.7940, device='cuda:1')]
Model on GPU 0, Epoch: 1--0/28125],  Loss: 0.6518099904060364
Model on GPU 1, Epoch: 1--0/28125],  Loss: 0.7939583659172058
[tensor(0.7865, device='cuda:1'), tensor(0.7719, device='cuda:1')]
[tensor(0.7865, device='cuda:0'), tensor(0.7719, device='cuda:0')]
Model on GPU 0, Epoch: 1--1/28125],  Loss: 0.7865331172943115
Model on GPU 1, Epoch: 1--1/28125],  Loss: 0.7718786001205444
[tensor(0.7348, device='cuda:1'), tensor(0.8238, device='cuda:1')]
[tensor(0.7348, device='cuda:0'), tensor(0.8238, device='cuda:0')]
</code></pre>
<p>tmp right now contains two tensors(loss from each GPU).</p>
<p>On a single GPU I train with a batch size of 32. Since, I have a very deep model and huge dataset I thought of using multiple GPUs in order to increase training speed. After reading <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769">this </a> I came to know I need to divide my batch size and train model with a batch size of 16 for two GPUs. Gradient is computed on batch of 16 on each GPU and average of gradient is applied to the models which gives an effect as in one iteration a batch of 32 is processed by GPUs and gradient is applied. I also want to gather the loss which will be equivalent of this batch of 32.</p>

username: mrshenli
cooked: <p>The output looks as-expected to me: the <code>tmp[0]</code> contains rank0 loss and <code>tmp[1]</code> contains rank1 loss. Did I miss anything?</p>

username: Saurav_Gupta1
cooked: <p>Okay Let me try to rephrase my question, gather has given me loss calculated by each GPU. My batch size for model being trained on 2 GPUs is 16 which means at a time 32 images will be processed. The gradient step that was taken was with respect to 32 images. <code>torch.distributed.gather</code> returns loss with respect to 16 images on each GPU in a list. I want to get the loss with respect to 32 images that were processed in one iteration in total.</p>

username: mrshenli
cooked: <p>I see. In that case, DDP alone won’t be sufficient, as DDP’s output and loss are local to each process. If you only need to calculate the globally loss, one option is to gather the outputs instead of loss, and then calculated loss on the gathered outputs. If you also need back propagation from the global loss, there are at least two options:</p>
<ol>
<li>combine DDP with RPC: Use DDP to compute local output and use RPC to collect them into one process and compute global loss, and then use distributed autograd to start the backward pass from global loss. This <a href="https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html" rel="noopener nofollow ugc">tutorial</a> shows how to combine DDP with RPC.</li>
<li>Use the autograd-enabled collective communications in <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributed/nn/functional.py" rel="noopener nofollow ugc"><code>torch.distributed.nn.functional</code></a>. Tests in <a href="https://github.com/pytorch/pytorch/pull/40762/files" rel="noopener nofollow ugc">this PR</a> can serve as examples. (This feature is not officially released yet.)</li>
</ol>

username: Jacfger
cooked: <p>I have another question is that what would be the best way to do all_gather when the batch size isn’t fixed? E.g. when drop_last=False in the Dataloader so the last batch size will be different?</p>