RuntimeError: Stop_waiting response is expected

yangkz: <p>When I use two gpus to train my model, I got RuntimeError below:</p>
<p>Process SpawnProcess-2:<br>
Traceback (most recent call last):<br>
File “/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap<br>
self.run()<br>
File “/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/multiprocessing/process.py”, line 93, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/ubuntu/ogb/ogb/graphproppred/m.py”, line 190, in run<br>
main(rank, dev_id, args)<br>
File “/home/ubuntu/ogb/ogb/graphproppred/m.py”, line 149, in main<br>
train(args[‘gnn’], model, device, train_loader, criterion, optimizer, args[‘num_devices’], rank)<br>
File “/home/ubuntu/ogb/ogb/graphproppred/m.py”, line 41, in train<br>
optimizer.backward_and_step(loss)<br>
File “/home/ubuntu/ogb/ogb/graphproppred/utils.py”, line 146, in backward_and_step<br>
self._sync_gradient()<br>
File “/home/ubuntu/ogb/ogb/graphproppred/utils.py”, line 127, in _sync_gradient<br>
dist.all_reduce(p.grad.data, op=dist.ReduceOp.SUM)<br>
File “/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py”, line 902, in all_reduce<br>
work = _default_pg.allreduce([tensor], opts)<br>
RuntimeError: Stop_waiting response is expected</p>
<p>Process SpawnProcess-1:<br>
Traceback (most recent call last):<br>
File “/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap<br>
self.run()<br>
File “/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/multiprocessing/process.py”, line 93, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/ubuntu/ogb/ogb/graphproppred/m.py”, line 190, in run<br>
main(rank, dev_id, args)<br>
File “/home/ubuntu/ogb/ogb/graphproppred/m.py”, line 149, in main<br>
train(args[‘gnn’], model, device, train_loader, criterion, optimizer, args[‘num_devices’], rank)<br>
File “/home/ubuntu/ogb/ogb/graphproppred/m.py”, line 41, in train<br>
optimizer.backward_and_step(loss)<br>
File “/home/ubuntu/ogb/ogb/graphproppred/utils.py”, line 146, in backward_and_step<br>
self._sync_gradient()<br>
File “/home/ubuntu/ogb/ogb/graphproppred/utils.py”, line 127, in _sync_gradient<br>
dist.all_reduce(p.grad.data, op=dist.ReduceOp.SUM)<br>
File “/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py”, line 902, in all_reduce<br>
work = _default_pg.allreduce([tensor], opts)<br>
RuntimeError: Stop_waiting response is expected</p>
<p>Here is the code where the error occurred:</p>
<pre><code>def _sync_gradient(self):
    """Average gradients across all subprocesses."""
    for param_group in self.optimizer.param_groups:
        for p in param_group['params']:
            if p.requires_grad and p.grad is not None:
                # print(p.grad.data.shape, p.grad.data.device)
                dist.all_reduce(p.grad.data, op=dist.ReduceOp.SUM)
                p.grad.data /= self.n_processes
</code></pre>
<p>Ps. When I do “print(p.grad.data.shape, p.grad.data.device)”, I find the grads are normal and has the same shape [1,300] on 2 different gpus. So I’m confused why it stopped here.</p>

mrshenli: <p>Is the result of <code>p.requires_grad and p.grad is not None</code> always the same across all process and all parameters? If allreduce ops on different processes could run into desync.</p>
<p>Which backend are you using (NCCL/Gloo/MPI) and which PyTorch version are you using? It will be helpful to have a min repro of this error.</p>

yangkz: <p>Thanks for your reply!<br>
Here is my short code:</p>
<pre><code class="lang-auto">def main(rank, dev_id, args):
    torch.distributed.init_process_group(backend="nccl",
                                         init_method='tcp://localhost:22',
                                         world_size=args['num_devices'],
                                         rank=dev_id)
    model = mymodel.to(dev_id)
    optimizer = optim.Adam(model.parameters(), lr=args['lr'])
    for epochs:
        pred = model(inputs)
        loss = criterion(pred, label)
        optimizer.zero_grad()
        loss.backward()

        for param_group in optimizer.param_groups:
            for p in param_group['params']:
                if p.requires_grad and p.grad is not None:
                    # print(p.grad.data.shape, p.grad.data.device) Ps. We can get grad information here
                    dist.all_reduce(p.grad.data, op=dist.ReduceOp.SUM)
                    p.grad.data /= n_processes
        optimizer.step()
    torch.distributed.barrier()

mp = torch.multiprocessing.get_context('spawn')
for id, device_id in enumerate(devices):
        procs.append(mp.Process(target=main, args=(id, device_id, args), daemon=True))
        procs[-1].start()
for p in procs:
    p.join()
</code></pre>
<p>The pytorch version I’m using is 1.4.0 and all my codes are running in AWS instance.<br>
Ps. The port of init_method can only be 22 which is strange otherwise it got Runtime Error like this:</p>
<p>File “/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/rendezvous.py”, line 120, in _tcp_rendezvous_handler<br>
store = TCPStore(result.hostname, result.port, world_size, start_daemon)<br>
RuntimeError: connect() timed out.</p>
<p>Thanks for your time and reading!</p>

yangkz: <p>The error has been fixed.<br>
‘Stop_waiting response is expected’ error occurred in TCPStore.cpp. So it was actually the communication problem. It works finally when I reinstalled NCCL: <a href="https://github.com/NVIDIA/nccl.git" rel="nofollow noopener">https://github.com/NVIDIA/nccl.git</a></p>