Distributed data parallel freezes without error message

Xinqiang_Ding: <p>Hello,</p>
<p>I’m trying to use the distributed data parallel to train a resnet model on mulitple GPU on multiple nodes. The script is adapted from the <a href="https://github.com/pytorch/examples/tree/master/imagenet" rel="noopener nofollow ugc">ImageNet example code</a>. After the script is started,  it builds the module on all the GPUs, but it freezes when it tries to copy the data onto GPUs.</p>
<p>During the freezing time, all the GPUs has been allocated memories for the model, but the GPU utilization is 0% and stays at 0% for a long time. In the output file, it complaints<br>
<code>srun: Job step aborted: Waiting up to 32 seconds for job step to finish.</code></p>
<p>Here are the scirpts and the corresponding output:</p>
<h2><a name="h-1-testing-the-distributed-package-1" class="anchor" href="#h-1-testing-the-distributed-package-1"></a>1. Testing the distributed package:</h2>
<blockquote>
<pre><code>&gt; import torch
&gt; import torch.distributed as dist
&gt; dist.init_process_group(backend='gloo',
&gt;                               init_method = 'file:///home/xqding/tmp/pytorch_dist/shared_file',
&gt;                               world_size = 4)
&gt; print('Hello from process {} (out of {})!'.format(
&gt; 	torch.distributed.get_rank(), torch.distributed.get_world_size()))
&gt; x = torch.Tensor([torch.distributed.get_rank()])
&gt; torch.distributed.all_reduce(x)
&gt; print("value of x: {}".format(x))
</code></pre>
</blockquote>
<p>This script works fine and this is the output:</p>
<blockquote>
<p>Hello from process 3 (out of 4)!<br>
Hello from process 2 (out of 4)!<br>
value of x:<br>
6<br>
[torch.FloatTensor of size 1]</p>
<p>value of x:<br>
6<br>
[torch.FloatTensor of size 1]</p>
<p>Hello from process 1 (out of 4)!<br>
value of x:<br>
6<br>
[torch.FloatTensor of size 1]</p>
<p>Hello from process 0 (out of 4)!<br>
value of x:<br>
6<br>
[torch.FloatTensor of size 1]</p>
</blockquote>
<h2><a name="h-2-resnet-model-using-distributed-data-parallel-2" class="anchor" href="#h-2-resnet-model-using-distributed-data-parallel-2"></a>2. Resnet model using distributed data parallel</h2>
<pre><code>&gt; import torch.nn as nn
&gt; import torch.optim as optim
&gt; from torch.autograd import Variable
&gt; from read_data import *
&gt; from torch.utils.data import Dataset, DataLoader
&gt; from torch.utils.data.distributed import DistributedSampler
&gt; from torch.nn.parallel import DistributedDataParallel
&gt; from torchvision import transforms, utils
&gt; 
&gt; dist = torch.distributed.init_process_group(backend = 'gloo',
&gt;                                             init_method = 'file:///home/xin/shared_file',
&gt;                                             world_size = 3)
&gt; print("Rank:", torch.distributed.get_rank())
&gt; 
&gt; net = models.ResNet(models.resnet.BasicBlock, [3, 4, 6, 3],
&gt;                       num_classes = 100)
&gt; net = torch.nn.parallel.DistributedDataParallel(net.cuda())
&gt; criterion = nn.CrossEntropyLoss().cuda()
&gt; optimizer = optim.SGD(net.parameters(), lr=0.1, momentum=0.9)
&gt; 
&gt; train_data = Dataset(.....)
&gt; train_sampler = DistributedSampler(train_data, num_replicas= 3, rank = torch.distributed.get_rank())
&gt; train_loader = DataLoader(train_data, batch_size = 300,
&gt;                           sampler = train_sampler,
&gt;                           num_workers=5, pin_memory = True)
&gt; num_epoches = 5
&gt; for epoch in range(num_epoches):
&gt;     train_sampler.set_epoch(epoch)
&gt;     running_loss = 0.0
&gt;     for i, data in enumerate(train_loader, 0):
&gt;         print("Step:", i)
&gt;         # get the inputs
&gt;         inputs = data['image']
&gt;         labels = data['labels']
&gt;         print("Here 1")
&gt; 
&gt;         # wrap them in Variable
&gt;         labels = labels.cuda(async = True)
&gt;         print("Here 2")
&gt;         input_var = torch.autograd.Variable(inputs)
&gt;         print("Here 3")
&gt;         labels_var = torch.autograd.Vairable(labels)
&gt; 
&gt;         # zero the parameter gradients
&gt;         optimizer.zero_grad()
&gt;         print("Here 4")
&gt;         # forward + backward + optimize
&gt;         outputs = net(inputs_var)
&gt;         print("Here 5")
&gt;         loss=criterion(outputs,labels_var)
&gt;         loss.backward()
&gt;         optimizer.step()
</code></pre>
<p>The output is like this:</p>
<p>Rank: 0<br>
Rank: 1<br>
Rann: 2<br>
Step: 0<br>
Step: 0<br>
Step: 0<br>
Here 1<br>
Here 1<br>
Here 1</p>
<p>It never reaches the <code>print('Here 2')</code>, which means it is freezing at the command <code>labels = labels.cuda(async = True)</code>.</p>
<p>While it is freezing, the GPU status on the corresponding nodes is</p>
<pre><code>Thu Sep 28 12:44:50 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.69                 Driver Version: 384.69                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980     On   | 00000000:02:00.0 Off |                  N/A |
| 26%   37C    P2    41W / 180W |    420MiB /  4038MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 980     On   | 00000000:82:00.0 Off |                  N/A |
| 26%   30C    P2    41W / 180W |    354MiB /  4038MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     19947    C   /home/xin/apps/anaconda3/bin/python         409MiB |
|    1     19947    C   /home/xin/apps/anaconda3/bin/python         343MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>Does anyone have some idea what is going on here?<br>
Thanks.</p>

Xinqiang_Ding: <p>I found where the problem is.<br>
Before running labels = labels.cuda(async = True), labels has to been converted into torch vairable labels = torch.autograd.Variable(labels).</p>

smth: <p>there might be a NCCL deadlock happening in the distributed setting (which is why you saw a freeze). We’ve identified this last week. I am issuing fixes for this.</p>

ailzhang: <p>Hi here is my 2 cents. The deadlock actually seldom happened to me when I use Ethernet(only once actually). And for me the hang happened in loss.backward() I think. Could you add flush=True to your print statement and see it still hangs at the same place when it happens next time?</p>

Evangelos_Kazakos: <p>Hi, I also have the same problem! Is there any solution to this? Thanks!</p>

GenkunAbe: <p>Is there any update for this?</p>
<p>I have the similar problem. I added flush=True to print statement and found that the hang actually happened in loss.backward(). But the strange thing is that it didn’t happen at the first iteration.</p>
<p>My training code:</p>
<pre><code class="lang-auto">    for epoch in range(args.epoch):
        log('EPOCH %d' % epoch)
        if args.distributed:
            train_sampler.set_epoch(epoch)
        for i, data in enumerate(trainloader):
            log('ITER %d' % i)
            net.train()
            inputs, labels = Variable(data[0]), Variable(data[1])
            if args.use_gpu:
                inputs, labels = inputs.cuda(), labels.cuda()
            log('ITER %d DATA LOADED' % i)

            outputs = net.forward(inputs)
            loss = criterion(outputs, labels)
            log('ITER %d FORWARDED' % i)

            optimizer.zero_grad()
            log('ITER %d ZERO_GRAD' % i)

            loss.backward()
            log('ITER %d BACKWARDED' % i)

            optimizer.step()
            log('ITER %d STEP' % i)
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">2018-04-17_10:19:18 EPOCH 0
2018-04-17_10:19:21 ITER 0
2018-04-17_10:19:21 ITER 0 DATA LOADED
2018-04-17_10:19:30 ITER 0 FORWARDED
2018-04-17_10:19:30 ITER 0 ZERO_GRAD
2018-04-17_10:19:32 ITER 0 BACKWARDED
2018-04-17_10:19:32 ITER 0 STEP
2018-04-17_10:19:32 ITER 1
2018-04-17_10:19:32 ITER 1 DATA LOADED
2018-04-17_10:19:33 ITER 1 FORWARDED
2018-04-17_10:19:33 ITER 1 ZERO_GRAD
</code></pre>

Mr.Z: <p>You can try to change your batchsize, my distributed training works with batchsize 48, and freezes with batch size 32.</p>

Shoaib_Ahmed_Siddiqu: <p>I am facing a similar sort of an issue, therefore, opened up an issue on PyTorch repo regarding this. If you are still facing the problem, it would be nice to contribute to the discussion there so that the PyTorch maintainers are aware of this problem with DDP.<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/shoaibahmed" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/3884843?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">Issue: Script freezes with no output when using DistributedDataParallel</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/shoaibahmed" target="_blank" rel="nofollow noopener">shoaibahmed</a>
	on <a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">2019-07-13</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
I was trying to evaluate the performance of the system with static data but different models, batch sizes and AMP...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: distributed</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

gemfield: <p>Env:</p>
<ul>
<li>Ubuntu 18.04</li>
<li>Pytorch 1.6.0</li>
<li>CUDA 10.1</li>
</ul>
<p>Actually, I am using Docker image gemfield/pytorch:1.6.0-devel which stated in <a href="https://github.com/DeepVAC/deepvac" rel="nofollow noopener">https://github.com/DeepVAC/deepvac</a> (same with above env), and use PyTorch DDP (by use the class DeepvacDDP in <a href="https://github.com/DeepVAC/deepvac/blob/master/deepvac/syszux_deepvac.py" rel="nofollow noopener">https://github.com/DeepVAC/deepvac/blob/master/deepvac/syszux_deepvac.py</a>) to train my model, which the code worked perfect yesterday. But today when I launch the train program again, the DDP is stucked in loss.backward()， with cpu 100% and GPU 100%。<br>
There has no code change and docker container change since yesterday, except the Ubuntu host got a system update today:</p>
<pre><code class="lang-bash">gemfield@ai03:~$ cat /var/log/apt/history.log | grep -C 3 nvidia

Start-Date: 2020-09-03  06:44:01
Commandline: /usr/bin/unattended-upgrade
Install: linux-modules-nvidia-440-5.4.0-45-generic:amd64 (5.4.0-45.49, automatic)
Upgrade: linux-modules-nvidia-440-generic-hwe-20.04:amd64 (5.4.0-42.46, 5.4.0-45.49)
End-Date: 2020-09-03  06:44:33
</code></pre>
<p>Obviously, the nvidia driver got update from 440.64 to 440.100, and I think these info may be useful for somebody.</p>

iamshnik: <p><a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> was this issue ever solved. I am facing NCCL deadlock issues in DistributedDataParallel.</p>
<p>Also, I face this issue with some particular architectures only and I don’t understand what does the architecture has to do with the NCCL deadlock?</p>

ptrblck: <p>Could you create a new topic including an executable code snippet to reproduce the issue as well as information about your setup (PyTorch, CUDA, cudnn, NCCL version, used GPU, OS etc.)?</p>

iamshnik: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, the problem is that this issue is not reproducible. It is totally random whether the training will face a deadlock or not. For example, I had three network architectures, and the day before yesterday, two of the three were suffering from NCCL deadlock while the other one was not. Yesterday when I tried it again, I had this issue with only one of the architectures and not with the other. I didn’t change a single line in the code.</p>
<p>I am using Pytorch 1.5.0 on Ubuntu 18.04. Also,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.nccl.version()
2408
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.backends.cudnn.version()
7605
</code></pre>
<p>I am training my models on RTX 2080ti and in the multi-gpu set up, I have tried using 2,4 and 8 GPUs but the deadlock issue persists.</p>
<p>I am training image classification models and the training freezes on the line</p>
<pre><code class="lang-auto">images = images.cuda(non_blocking=True)
</code></pre>
<p>I have already experimented with <code>non_blocking=False</code> and I am sure that it is not the problem.</p>

ptrblck: <p>You could try to use the nightly binary with NCCL 2.7.6 and see if you are still facing this issue.</p>

iamshnik: <p>I installed the pytorch-nightly using the following command</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.2 -c pytorch-nightly
</code></pre>
<p>The NCCL version is 2.7.6 now.</p>
<p>The training is working fine so far. Thanks for the help <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>However, I want to know what was the issue that was triggering the NCCL deadlock.</p>

ptrblck: <aside class="quote no-group" data-username="iamshnik" data-post="14" data-topic="8009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iamshnik/48/28740_2.png" class="avatar"> iamshnik:</div>
<blockquote>
<p>However, I want to know what was the issue that was triggering the NCCL deadlock.</p>
</blockquote>
</aside>
<p>It’s unclear to me that NCCL caused the deadlock. Without a code snippet to reproduce it, I cannot be very helpful in isolating the issue.</p>

Feywell: <p>Hi~ I have the same problem. So It need to update nccl to 2.7.6 and install pytorch-nightly.<br>
But I use conda to install pytorch-nightly, The nccl’s version is also 2.4.8.</p>

iamshnik: <p>Hi <a class="mention" href="/u/feywell">@Feywell</a>, I did not quite get what issue are you facing.</p>
<p>If you have the same problem (i.e, training freezes because of some deadlock) then try upgrading your pytorch to the latest pytorch-nightly. It uses the NCCL submodule version 2.7.6</p>
<p>To install pytorch-nightly from conda, refer to <a href="https://pytorch.org/" rel="nofollow noopener">pytorch official website</a>.</p>
<p>If the issue is that you installed pytorch-nightly using conda but the nccl version is still 2.4.8 then can you please mention the command that you are using to install pytorch-nightly from conda?</p>

Feywell: <aside class="quote no-group" data-username="iamshnik" data-post="12" data-topic="8009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iamshnik/48/28740_2.png" class="avatar"> iamshnik:</div>
<blockquote>
<p>deadlock</p>
</blockquote>
</aside>
<p>Thanks for your reply.<br>
I update pytorch to 1.7 sucessfully. And ncc version is 2.7.6 now. But my code is also dead without errors.<br>
It is weired</p>

Feywell: <p><a class="mention" href="/u/iamshnik">@iamshnik</a> <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
This is my problem in detail<br>
Is it the same problem?</p><aside class="quote" data-post="1" data-topic="96229">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/feywell/48/7936_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/distributed-training-failed-without-errors/96229">Distributed training failed without errors</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    I just change 

<a href="https://github.com/ZPdesu/SEAN" rel="noopener nofollow ugc">https://github.com/ZPdesu/SEAN</a> to distributed training. 

And I get a weird error. 
It is no error，when I use one gpu. 
But, It will stop without printing errors when I use multi gpus. 
It will stop after Every 13 epoches by two gpus. 
What could be the cause of such a problem？ 
my environments: 
ubuntu: 16.04 
gpu：nvidia-2080ti 
cuda: 10.1 / 10.2 
pytorch: 1.6.0 / 1.7.0 
nccl: 2.4.8 / 2.7.6 
python:3.6 / 3.7
  </blockquote>
</aside>


iamshnik: <p><a class="mention" href="/u/feywell">@Feywell</a> it seems like you are facing the same issue. In my case NCCL 2.7.6 resolved the issue and I was able to train my models. Infact, I had almost similar system settings:</p>
<p>Ubuntu 18.04<br>
2080ti<br>
CUDA 10.2<br>
pytorch-nightly 1.7<br>
python 3.7</p>
<p>Are you using pytorch-nightly or just pytorch? If you are not using pytorch-nightly, please try using that.</p>