Distributed training got stuck every few seconds

fanl: <p>Hi, everyone<br>
When I train my model with DDP, I observe that my training process got stuck every few seconds. The device information is shown in the following figure when it is stuck.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da188fea4e3933dd55269ec47daf32b42afbb613.png" data-download-href="https://discuss.pytorch.org/uploads/default/da188fea4e3933dd55269ec47daf32b42afbb613" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da188fea4e3933dd55269ec47daf32b42afbb613.png" alt="image" data-base62-sha1="v7mIbRbTEWXtZZlQr0wDOC36XVp" width="495" height="500" data-dominant-color="2B2B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">637×643 26.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
There seems always one GPU got stuck whose utilization is 0%, and the others are waiting for it to synchronizing.<br>
This issue disappears after switching to another server (with the same image).<br>
Though it is solved, I am curious about the reasons. I will appreciate it if someone shares the ideas.</p>

AndreaSottana: <p>Hi,<br>
My experience with distributed  training so far is quite limited so others might have better answers. But what I notice straight away is that some GPUs (e.g. number 3) are almost at full capacity, i.e. 11004/11019 and this might be causing problems. I had similar issues when running so close to capacity and found that reducing the batch size (perhaps together with using gradient accumulation) helped.<br>
The reason could be some deadlock, where the code is waiting for data to come from all GPUs before moving onto the next step but one is unresponsive. Although it’s hard to guess without looking at the code. If reducing batch size (or increasing GPUs memory) doesn’t help, feel free to give more details and ideally attach a reproducible example and someone with more experience will take a look.</p>

fanl: <p>Hi Andrea,<br>
Thanks for your reply.</p>
<ol>
<li>The causes seem not about large memory cost, because this issue happened to several models with different sizes.</li>
<li>I think this issue is not related to my code, giving some code snippets might be a misleading instead. I use the standard code configuration which works fine several days ago and it indeed works fine in another server.</li>
</ol>
<p>It’s really tricky to figure out the reasons for such a problem without applicable running environment. Thank you anyway.<br>
Any other comments are definitely welcome!</p>

wayi: <blockquote>
<p>There seems always one GPU got stuck whose utilization is 0%, and the others are waiting for it to synchronizing.</p>
</blockquote>
<p>Do you have uneven inputs? For example, you run one epoch every few seconds, and one partition on a rank has much fewer batches to process.</p>
<blockquote>
<p>This issue disappears after switching to another server (with the same image).</p>
</blockquote>
<p>Does the “image” here means <code>cuda:6</code> still has 0 GPU utilization for some time, but you don’t feel it’s get stuck?</p>

fanl: <p>Thanks for reply.</p>
<ol>
<li>My data is uniformly partitioned to every GPUs.</li>
<li>Sorry for the unclarity.  “Image” here refer to as the docker image. Two machines share the same docker image, while only one of them works fine.</li>
</ol>

wayi: <blockquote>
<p>“Image” here refer to as the docker image. Two machines share the same docker image, while only one of them works fine.</p>
</blockquote>
<p>That’s really weird, so the software setup should be exactly the same, and you also ran exactly the same code. Have you tried any different programs that can run on multiple GPUs? For example, another DDP training use case or just running some collective communication APIs like <code>torch.distributed.all_reduce</code>?</p>
<p>I am afraid that I may not be able to reproduce your issue on my own setup.</p>

fanl: <p>Thanks for your suggestions, and I will have a try.<br>
I think this issue has something to do with my environment, where I observed some strange things. For example, another <a href="https://discuss.pytorch.org/t/out-of-memory-when-running-model-cuda/132480/4">cuda error</a> occurs in the same machine. Your comments about this cuda error are definitely welcome !!</p>

fanl: <p>The problem got more serious: several dataloaders got completely stuck, and the others are waiting for them.<br>
I try print some information once the <code>__getitem__</code> function is called.</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        print(f'rank {torch.distributed.get_rank()}, fetch sample {idx}')
        # my custom transformations ...
</code></pre>
<p>After several iterations I got following information:</p>
<pre><code class="lang-auto">rank 4, fetch sample 1469
rank 2, fetch sample 3282
rank 5, fetch sample 2757
rank 1, fetch sample 1355
rank 3, fetch sample 279
rank 0, fetch sample 4107
rank 7, fetch sample 2834
</code></pre>
<p>Rank 6 is missing, and the GPU information is as follows.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6eaef4732bc76313c5a333d5ff6ea4b5aff8ee75.png" alt="image" data-base62-sha1="fN9mn87uqJytFVmrcDbaGzWM1Kd" width="648" height="480"></p>
<p>After sending an interrupt, the traceback is:</p>
<pre><code class="lang-auto">  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/mmdet/apis/train.py", line 170, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 47, in train
    for i, data_batch in enumerate(self.data_loader):
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
    idx, data = self._get_data()
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data
    success, data = self._try_get_data()
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/root/.pyenv/versions/3.6.8/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt

</code></pre>
<p>Anyone has any idea? <a class="mention" href="/u/wayi">@wayi</a> <a class="mention" href="/u/andreasottana">@AndreaSottana</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

wayi: <p>Looks like uneven inputs for me.</p>
<p>Can you add <code>no_sync</code> context manager to disable allreduce in DDP?<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel.no_sync" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel.no_sync</a></p>
<p>This way you shouldn’t have any gradient synchronization, and you can check if this issue still occurs.</p>

wayi: <p>Another suggestion is enabling debug mode by:<br>
<code>os.environ["TORCH_DISTRIBUTED_DEBUG"] = "DETAIL"</code></p>

111414: <p>Hi, I encounter a similar problem. Does yours be fixed?</p>

Will_Chi: <p>Hi <a class="mention" href="/u/fanl">@fanl</a> , I also met this issue. Have you addressed it? I really look forward to your help!</p>

fanl: <p>It is a very complicated problem. For me, the issue disappeared when I abandoned VSCode in the server, but it may not be the real cause. Hope it helps.</p>

Will_Chi: <p>Very thanks for your reply! I am also using the server through VSCode. Let me have a try.</p>

trancelestial: <p>I have the same exact issue happening. Has anyone figured out how to solve this problem?</p>