Runtime execption when using DataParallel on Nvidia A100, Pytoch 1.12

SM19: <p>Hi,</p>
<p>I get this error when trying to use 2 GPUs - A100s</p>
<p>File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py”, line 168, in forward<br>
outputs = self.parallel_apply(replicas, inputs, kwargs)<br>
File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py”, line 178, in parallel_apply<br>
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])<br>
File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py”, line 86, in parallel_apply<br>
output.reraise()<br>
File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/_utils.py”, line 461, in reraise<br>
raise exception<br>
RuntimeError: Caught RuntimeError in replica 1 on device 2.<br>
Original Traceback (most recent call last):<br>
File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker<br>
output = module(*input, **kwargs)<br>
File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/usr/scratch4/samo4615/Documents/codeworks/lidar_mtl/kitti_3d_det/model_custom.py”, line 657, in forward<br>
x_rot = self.bn2_rot(self.a2_rot(self.c2_rot(x_rot)))<br>
File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/nn/modules/conv.py”, line 457, in forward<br>
return self._conv_forward(input, self.weight, self.bias)<br>
File “/usr/scratch4/samo4615/miniconda3/envs/torchcon38/lib/python3.8/site-packages/torch/nn/modules/conv.py”, line 453, in _conv_forward<br>
return F.conv2d(input, weight, bias, self.stride,<br>
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED<br>
You can try to repro this exception using the following code snippet. If that doesn’t trigger the error, please include your original repro script when reporting this issue.</p>
<p>import torch<br>
torch.backends.cuda.matmul.allow_tf32 = False<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = False<br>
torch.backends.cudnn.allow_tf32 = True<br>
data = torch.randn([4, 64, 576, 384], dtype=torch.float, device=‘cuda’, requires_grad=True).to(memory_format=torch.channels_last)<br>
net = torch.nn.Conv2d(64, 64, kernel_size=[3, 3], padding=[2, 2], stride=[1, 1], dilation=[2, 2], groups=1)<br>
net = net.cuda().float().to(memory_format=torch.channels_last)<br>
out = net(data)<br>
out.backward(torch.randn_like(out))<br>
torch.cuda.synchronize()</p>
<p>ConvolutionParams<br>
memory_format = ChannelsLast<br>
data_type = CUDNN_DATA_FLOAT<br>
padding = [2, 2, 0]<br>
stride = [1, 1, 0]<br>
dilation = [2, 2, 0]<br>
groups = 1<br>
deterministic = false<br>
allow_tf32 = true<br>
input: TensorDescriptor 0x7f10015e9880<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 4, 64, 576, 384,<br>
strideA = 14155776, 1, 24576, 64,<br>
output: TensorDescriptor 0x7f10015eb3d0<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 4, 64, 576, 384,<br>
strideA = 14155776, 1, 24576, 64,<br>
weight: FilterDescriptor 0x7f100165a1f0<br>
type = CUDNN_DATA_FLOAT<br>
tensor_format = CUDNN_TENSOR_NHWC<br>
nbDims = 4<br>
dimA = 64, 64, 3, 3,<br>
Pointer addresses:<br>
input: 0x7f0e29800000<br>
output: 0x7f0dd5800000<br>
weight: 0x7f14efa1a600<br>
Forward algorithm: 1</p>
<p>The model runs fine on 1 GPU.</p>
<p>Please help.<br>
Input shape - 16x3x576x320</p>

ptrblck: <p>Could you check if you are running out of memory and if this issue is still reproducible in the latest PyTorch release?<br>
I cannot reproduce the cuDNN error using the posted code snippet on an A100.</p>