Training freezes before forward pass when number of GPUs are increased in distributed training

amogh112: <p>I made some changes to the model’s forward pass in <a href="https://github.com/jackroos/VL-BERT" rel="noopener nofollow ugc">VL-BERT repository</a>.<br>
I was able to successfully run my script for training over multiple (7) GPUs. However after some time, suddenly, my code freezes when I increase the number of GPUs to more than 2 GPUs. I did not make any changes to the way model is passed for distributed training.</p>
<p>This is how I increase the number of GPUs used:</p>
<p><code>CUDA_VISIBLE_DEVICES=1,2,3,4 ./scripts/dist_run_single.sh 4 pretrain/train_end2end.py ./cfgs/contrastive_pretrain/base_prec_random_movienet_images_4x16G_fp32.yaml ./checkpoints_debugcv04</code><br>
<strong>Gets stuck if GPUs are more than 2</strong></p>
<p>instead of</p>
<p><code>CUDA_VISIBLE_DEVICES=1,2 ./scripts/dist_run_single.sh 2 pretrain/train_end2end.py ./cfgs/contrastive_pretrain/base_prec_random_movienet_images_4x16G_fp32.yaml ./checkpoints_debugcv04 </code><br>
<strong>Starts training successfully</strong></p>
<p>The model is loaded on each of the local ranks successfully.<br>
The script also enters the train function on each rank : <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/trainer.py#L56" rel="noopener nofollow ugc">https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/trainer.py#L56</a></p>
<p>However, the forward pass doesn’t proceed.<br>
I am using the latest version of PyTorch 1.7.0<br>
What might be going wrong here? I assume some synchronization problems might be occurring with more than 2 GPUs<br>
Thanks</p>

osalpekar: <blockquote>
<p>However after some time, suddenly, my code freezes when I increase the number of GPUs to more than 2 GPUs.</p>
</blockquote>
<p>At what point does the training get stuck? Do you have any logs outputted until the point the training gets stuck (ideally with <code>NCCL_DEBUG</code>=WARN)?</p>
<p>Also how many GPUs does this host have? Do you run into the same issue on other multi-GPU hosts as well?</p>
<blockquote>
<p>I assume some synchronization problems might be occurring with more than 2 GPUs</p>
</blockquote>
<p>This usually only becomes a major issue at much larger numbers of GPUs, so it should be able to handle more than 2. Are there any factors that could cause synchronization issues, such as some GPUs being significantly slower than others/other jobs or processes using those GPUs while you were training?</p>

amogh112: <p>Hi <a class="mention" href="/u/osalpekar">@osalpekar</a>, thanks for your response</p>
<p>I have figured out that the training runs fine when I remove the <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/pretrain/function/train.py#L259" rel="noopener nofollow ugc">metric logger for MLMAccuracy</a><br>
The place where it gets stuck is when the <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/metrics/eval_metric.py#L38" rel="noopener nofollow ugc">get() function</a> is called in  MLMAccuracy when writing to <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/trainer.py#L171" rel="noopener nofollow ugc">tensorboard</a> and <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/callbacks/batch_end_callbacks/speedometer.py#L64" rel="noopener nofollow ugc">logger</a>. Somehow it is not able to do <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/metrics/eval_metric.py#L51" rel="noopener nofollow ugc">all_reduce here for <code>sum_metric</code></a>  for this particular metric when more number of GPUs are there.</p>
<p><strong>In case it makes sense to see the log files as you suggested:</strong><br>
I have posted the output with NCCL_DEBUG=WARN in 2 files here for 2 cases (freezes and works_fine): <a href="https://gist.github.com/amogh112/84a27280e69b983ea88497892e3855cb" rel="noopener nofollow ugc">https://gist.github.com/amogh112/84a27280e69b983ea88497892e3855cb</a><br>
And the comment at the end shows the output for NCCL_DEBUG=INFO for different cases with 1,2,6 GPUs</p>

pritamdamania87: <aside class="quote no-group" data-username="amogh112" data-post="3" data-topic="104419">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amogh112/48/5145_2.png" class="avatar"> amogh112:</div>
<blockquote>
<p>The place where it gets stuck is when the <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/metrics/eval_metric.py#L38" rel="noopener nofollow ugc">get() function</a> is called in MLMAccuracy when writing to <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/trainer.py#L171" rel="noopener nofollow ugc">tensorboard</a> and <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/callbacks/batch_end_callbacks/speedometer.py#L64" rel="noopener nofollow ugc">logger</a>. Somehow it is not able to do <a href="https://github.com/jackroos/VL-BERT/blob/4373674cbf2bcd6c09a2c26abfdb6705b870e3be/common/metrics/eval_metric.py#L51" rel="noopener nofollow ugc">all_reduce here for <code>sum_metric</code> </a> for this particular metric when more number of GPUs are there.</p>
</blockquote>
</aside>
<p>Can you verify that this allreduce gets called on all ranks? One possibility could be that some ranks don’t invoke this allreduce which would result in a freeze.</p>