Distributed training gives nan loss but single GPU training is fine

Beinan_Wang: <p>When I train my network with a single GPU, the training process terminates successfully after 120 epochs. However, if I use two GPUs, I get nan loss after a dozen epochs. The only thing I change is the batch size. For single GPU I use a batch size of 2 and for 2 GPUs I use a batch size of 1 for each GPU. The other parameters are exactly the same. I also replace every batchnorm2d layer with a syncbatchnorm layer. Strangely, syncbatchnorm gives higher loss. What could be the possible reasons?</p>

mrshenli: <p>Could you please paste a code snippet to reproduce? Are you using DataParallel or DistributedDataParallel?</p>

Beinan_Wang: <p>I use DDP. I enabled anomaly detection. Below is the message I get</p>
<p>/pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:<br>
File “”, line 1, in <br>
File “/usr/lib/python3.6/multiprocessing/spawn.py”, line 105, in spawn_main<br>
exitcode = _main(fd)<br>
File “/usr/lib/python3.6/multiprocessing/spawn.py”, line 118, in _main<br>
return self._bootstrap()<br>
File “/usr/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap<br>
self.run()<br>
File “/usr/lib/python3.6/multiprocessing/process.py”, line 93, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap<br>
fn(i, *args)<br>
File “/home/beinan/Desktop/pytorch-bcn/jupyter/train.py”, line 153, in main_worker<br>
train(train_loader, model, criterion, optimizer, epoch, args)<br>
File “/home/beinan/Desktop/pytorch-bcn/jupyter/train.py”, line 199, in train<br>
output = model(images)<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/beinan/.local/lib/python3.6/site-packages/apex/parallel/distributed.py”, line 560, in forward<br>
result = self.module(*inputs, **kwargs)<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “…/bcn/models/semantic/resnet34.py”, line 295, in forward<br>
x, encoder_features, encoder_feature = self.encoder(x)<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “…/bcn/models/semantic/resnet34.py”, line 224, in forward<br>
x = self.bn(self.conv(x))<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “…/bcn/layers/conv.py”, line 38, in forward<br>
groups=self.groups<br>
File “/home/beinan/.local/lib/python3.6/site-packages/apex/amp/wrap.py”, line 28, in wrapper<br>
return orig_fn(*new_args, **kwargs)</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 314, in <br>
main()<br>
File “train.py”, line 66, in main<br>
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 171, in spawn<br>
while not spawn_context.join():<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 118, in join<br>
raise Exception(msg)<br>
Exception:</p>
<p>– Process 1 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap<br>
fn(i, *args)<br>
File “/home/beinan/Desktop/pytorch-bcn/jupyter/train.py”, line 153, in main_worker<br>
train(train_loader, model, criterion, optimizer, epoch, args)<br>
File “/home/beinan/Desktop/pytorch-bcn/jupyter/train.py”, line 208, in train<br>
scaled_loss.backward()<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/tensor.py”, line 166, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/beinan/.local/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: Function ‘CudnnConvolutionBackward’ returned nan values in its 1th output.</p>
<p>This consistently happens after 90+ epochs but only if I use DDP. Single GPU single node does not have this problem. BTW, I train with fp16 precision. Is it possible that fp16 + DDP + SyncBatchNorm somehow leads to this?</p>

Beinan_Wang: <p>Is it possible for the transformation below to cause any problem?</p>
<pre><code class="lang-auto">class RandomResizeCrop(object):

    def __init__(self, min_scale, max_scale, scale_step, output_size):

        self.scales = np.arange(min_scale, max_scale, scale_step)
        self.output_height, self.output_width = output_size

    def __call__(self, image, annotation):

        scale = np.random.choice(self.scales)
        
        image = cv2.resize(image, (0,0), fx=scale, fy=scale, interpolation=cv2.INTER_LINEAR)
        annotation = cv2.resize(annotation, (0,0), fx=scale, fy=scale, interpolation=cv2.INTER_NEAREST)

        input_height, input_width = image.shape[:2]

        row_pads = max(self.output_height - input_height, 0)
        col_pads = max(self.output_width - input_width, 0)

        top_pads = randint(0, row_pads)
        bot_pads = row_pads - top_pads

        left_pads = randint(0, col_pads)
        right_pads = col_pads - left_pads

        image = np.pad(image, ((top_pads,bot_pads),(left_pads,right_pads),(0,0)), mode='constant', constant_values=0)
        annotation = np.pad(annotation, ((top_pads,bot_pads),(left_pads,right_pads)), mode='constant', constant_values=255)

        y1 = randint(0, max(input_height - self.output_height, 0))
        y2 = y1 + self.output_height

        x1 = randint(0, max(input_width - self.output_width, 0))
        x2 = x1 + self.output_width

        return image[y1:y2,x1:x2], annotation[y1:y2,x1:x2]</code></pre>

Beinan_Wang: <p>It looks like the first convolution (operation) in resnet is causing nan. Is there any way some values of an image become nan after transformation?</p>

royve: <p>I ran into the exact same problem.<br>
Any chance that you have eventually found what was the problem?</p>
<p>Thanks!</p>

paganpasta: <p><a class="mention" href="/u/royve">@royve</a> In my experience fp16 can lead to nans due to limited precision.</p>
<p>Can you try without fp16?</p>

royve: <p>I train with fp32.<br>
I believe my problem was that I used an LBFGS optimizer. When I switched back to Adam the NaNs were gone. Maybe the learning rate for LBFGS should be substantially decreased, I didn’t try it though.</p>