Data Parallel throws Error: replica 1 on device 5 and CUDNN_STATUS_EXECUTION_FAILED

weight_theta: <p>Currently trying to use pytorch data parallel  to up batchsize as shown in code, but get 1) RuntimeError: Caught RuntimeError in replica 1 on device 5. and 2) RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED.</p>
<p>**Anyone who has worked with dataparallel before and had similar issues ? **</p>
<p><strong>Code:</strong></p>
<pre data-code-wrap="python"><code class="lang-python">device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
device_ids = [1, 5]
model = YoloV4_EfficentNet(nclasses = arguments['nclasses']).to(device)
model = nn.DataParallel(model, device_ids = device_ids)

def trainyolov4(arguments, train_loader, model, optimizer, scheduler, loss_f, scaled_anchors, scaler, mode = 'iou'):
    model.train()
    for batch_idx, (x, y) in enumerate(train_loader):
        x = x.permute(0, 3, 1, 2) #.to(device)
        y0, y1, y2 = (y[0], y[1], y[2]) #.to(device)
        # x shape :-: (batchsize, channels, height, width)
        with autocast():
            preds = model(x)
            loss_val = (
                loss_f(preds[0], y0, scaled_anchors[0], mode = mode)
                + loss_f(preds[1], y1, scaled_anchors[1], mode = mode)
                + loss_f(preds[2], y2, scaled_anchors[2], mode = mode))
        class_acc, noobj_acc, obj_acc = class_accuracy(preds, y, arguments["conf_thresh"])
        optimizer.zero_grad()
        scaler.scale(loss_val).backward()
        scaler.step(optimizer)
        scaler.update()
        if arguments["one_cycle"] == True:
            scheduler.step()
    return (float(loss_val.item()), float(class_acc),float(noobj_acc), float(obj_acc))
</code></pre>
<p><strong>Error :</strong></p>
<ol>
<li>RuntimeError: Caught RuntimeError in replica 1 on device 5. and 2) RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED.</li>
</ol>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 54, in &lt;module&gt;
    train_model_with_args()
  File "main.py", line 52, in train_model_with_args
    initialize_with_args(arguments)
  File "/home/thesis/train/train.py", line 214, in initialize_with_args
    main(arguments)
  File "/home/thesis/train/train.py", line 695, in main
    train_loss_val, train_class_acc, train_noobj_acc, train_obj_acc = trainyolov4(arguments, train_loader, model, optimizer, scheduler, loss_f, scaled_anchors, scaler, mode = 'ciou')
  File "/home/thesis/train/train.py", line 376, in trainyolov4
    preds = model(x)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
    output.reraise()
  File "/home/.local/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 5.
Original Traceback (most recent call last):
  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
    output = module(*input, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/thesis/models/yolov4.py", line 206, in forward
    sclaed_pred2 = self.yolov4head[1](panet_scale2)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/thesis/models/yolov4.py", line 64, in forward
    out = self.scaled_pred(x)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/container.py", line 204, in forward
    input = module(input)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/thesis/models/yolov4.py", line 19, in forward
    out = self.bn(out)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 171, in forward
    return F.batch_norm(
  File "/home/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 2450, in batch_norm
    return torch.batch_norm(
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>
<p><strong>Tried:</strong></p>
<ol>
<li>Explicitly setting cuda visible devices, 2) set cuda launch blocking = 1 for debug, 3) moving x, y to device vs not doing it as data parallel should do this for us , 4) made sure CUDA version 11.6 on the labs servers matches pytorch version and 5) checked that GPUs are reconized.</li>
</ol>

weight_theta: <p>So I gave torch a fresh install, and it now works but only if I disable the backend <code>torch.backends.cudnn.enabled = False</code><br>
If I use CUDA_LAUNCH_BLOCKING=1, the process freezes and needs to be manually killed using <code>kill -9 PID</code>. But if I set If I set the flag to True <code>torch.backends.cudnn.enabled = True</code>, then I get the following error:</p>
<p><strong>Error</strong></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 54, in &lt;module&gt;
    train_model_with_args()
  File "main.py", line 52, in train_model_with_args
    initialize_with_args(arguments)
  File "/home/thesis/train/train.py", line 214, in initialize_with_args
    main(arguments)
  File "/home/thesis/train/train.py", line 699, in main
    train_loss_val, train_class_acc, train_noobj_acc, train_obj_acc = trainyolov4(arguments, train_loader, model, optimizer, scheduler, loss_f, scaled_anchors, scaler, mode = 'ciou')
  File "/home/thesis/train/train.py", line 378, in trainyolov4
    preds = model(x)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 185, in forward
    outputs = self.parallel_apply(replicas, inputs, module_kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 200, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 108, in parallel_apply
    output.reraise()
  File "/home/.local/lib/python3.8/site-packages/torch/_utils.py", line 705, in reraise
    raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/home/.local/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in _worker
    output = module(*input, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/thesis/models/yolov4.py", line 206, in forward
    sclaed_pred2 = self.yolov4head[1](panet_scale2)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/thesis/models/yolov4.py", line 64, in forward
    out = self.scaled_pred(x)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/container.py", line 217, in forward
    input = module(input)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/thesis/models/yolov4.py", line 21, in forward
    out = self.activation(out)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 790, in forward
    return F.leaky_relu(input, self.negative_slope, self.inplace)
  File "/home/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 1675, in leaky_relu
    result = torch._C._nn.leaky_relu(input, negative_slope)
RuntimeError: CUDA error: misaligned address
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> any ideas?</p>
<p>If I disable the backend, I am essentially using torches own implementations ? Based on GPU memory usage, utility, and training process, the model is training even if the backend flag is set to false. Does this affect data parallel or is it taking effect ?</p>

ptrblck: <p>Are you able to reproduce the issue without <code>DataParallel</code> or only with it?<br>
Also, are you using the latest stable or nightly release? If so, could you post a minimal and executable code snippet reproducing the issue?</p>

weight_theta: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I abandoned the DP approach in favor of a <code>DDP</code> approach, which seems to work. (See code below.)<br>
However, I was wondering whether it is common to use the <code>DistributedSampler</code> together with DDP ?<br>
I am experiencing serious memory overhead, when trying to using the <code>DistributedSampler</code>  together with <code>DDP</code>. I could fit a batchsize of 30 before on a single gpu without DDP and now I can only fit a batchsize of 16 on two gpus.<br>
If I try to fit 30 on 2,3 or 4 gpus I obtain a <code>CUDA OUT of MEMORY ERROR</code>.</p>
<p>I am using the newest torch version, and have 8 RTX 3080 available.<br>
Should I opt out of using the <code>DistributedSampler</code> and only use <code>DDP</code> ?</p>
<pre><code class="lang-auto">def setup(rank, world_size):
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "12355"

    # initialize the process group
    dist.init_process_group("nccl", rank=rank, world_size=world_size)


def cleanup():
    dist.destroy_process_group()


def main(rank, world_size):
    image_size = 608
    batch_size = 16
    setup(rank, world_size)
    # yolo anchors rescaled between 0,1
    # yolo scales and anchors for image size 608, 608
    S = [19, 38, 76]
    anchors = [
        [(0.23, 0.18), (0.32, 0.40), (0.75, 0.66)],
        [(0.06, 0.12), (0.12, 0.09), (0.12, 0.24)],
        [(0.02, 0.03), (0.03, 0.06), (0.07, 0.05)],
    ]

    train_dataset = CoCoDataset(
        "data/coco/train.csv",
        "data/coco/images/",
        "data/coco/labels/",
        S=S,
        anchors=anchors,
        image_size=image_size,
        mode="train",
    )

    test_dataset = test_dataset = CoCoDataset(
        "data/coco/test.csv",
        "data/coco/images/",
        "data/coco/labels/",
        S=S,
        anchors=anchors,
        image_size=image_size,
        mode="test",
    )

    scaled_anchors = (torch.tensor(anchors) * torch.tensor(S).unsqueeze(1).unsqueeze(1).repeat(1, 3, 2)).to(rank)
    loss_f = YoloV4Loss2()

    sampler_train = DistributedSampler(train_dataset, shuffle=True)
    sampler_test = DistributedSampler(test_dataset, shuffle=False)

    # we drop the last batch to ensure each batch has the same size
    train_loader = DataLoader(
        dataset=train_dataset,
        num_workers=2,
        batch_size=batch_size,
        drop_last=False,
        sampler=sampler_train,
    )

    test_loader = DataLoader(
        dataset=test_dataset,
        num_workers=2,
        batch_size=batch_size,
        drop_last=False,
        sampler=sampler_test,
    )
    model = YoloV4_EfficentNet(nclasses=80).to(rank)

    ddp_model = DDP(model, device_ids=[rank], gradient_as_bucket_view = True)
    optimizer = optim.Adam(
        ddp_model.parameters(), lr=1e-4, weight_decay=0.0005, betas=(0.937, 0.999)
    )
    scheduler = optim.lr_scheduler.OneCycleLR(
        optimizer,
        max_lr=1e-4,
        epochs=300,
        steps_per_epoch=len(train_loader),
        anneal_strategy="cos",
    )

    for epoch in range(300):
        # train
        print(f"Rank:{rank}")
        scaler = GradScaler()
        train_loss_val, train_class_acc, train_noobj_acc, train_obj_acc = trainyolov4(
            rank,
            train_loader,
            model,
            optimizer,
            scheduler,
            loss_f,
            scaled_anchors,
            scaler,
            conf_thresh=0.8,
            mode="ciou",
        )

        print(f"Epoch:{epoch + 1}  Train[Loss:{train_loss_val} Class Acc:{train_class_acc} NoObj acc:{train_noobj_acc} Obj Acc:{train_obj&gt;

if __name__ == "__main__":
    mp.spawn(
        main,
        args=(2,),
        nprocs=2,
        join=True,
    )
</code></pre>

ptrblck: <p>Using the <code>DistributedSampler</code> is the common approach as it splits the dataset into shards for each rank making sure samples are not repeated on all ranks (with the exception of repeating samples to make the number of samples divisible by the batch size and ranks).<br>
Note that <code>DP</code> will shard the input batch in <code>dim0</code> from the default rank so are you sure every GPU used a batch size of 30 or was it shared? In the former case your batch size would be <code>30 * num_gpus</code>.</p>

weight_theta: <p>If <code>DP</code> shards batch in dim0, then each gpu should have <code>batchsize/n_gpus</code>, right? This would mean for my case tath <code>batchsize/n_gpus = 30 / 2 = 15</code> also tested with <code>batchsize/n_gpus = int(30/4) = 7</code> on each gpu. The GPU memory load reaches its limits and then throws a cuda out of memory error.</p>
<p>Since it works without DP, I suspect the problem is related to either 2. or 3. How would I proceed to debug?</p>
<ol>
<li></li>
</ol>
<pre><code class="lang-auto">train_dataset = CoCoDataset( "data/coco/train.csv",
        "data/coco/images/",
        "data/coco/labels/",
        S=S,
        anchors=anchors,
        image_size=image_size,
        mode="train",
    )
</code></pre>
<ol start="2">
<li></li>
</ol>
<pre><code class="lang-auto">sampler_train = DistributedSampler(train_dataset, shuffle=True)
</code></pre>
<ol start="3">
<li></li>
</ol>
<pre><code class="lang-auto">    train_loader = DataLoader(
        dataset=train_dataset,
        num_workers=2,
        batch_size=32,
        drop_last=False,
        sampler=sampler_train,
    )
</code></pre>
<ol start="4">
<li>Snipped after:</li>
</ol>
<pre><code class="lang-auto">    model = YoloV4_EfficentNet(nclasses=80).to(rank)

    ddp_model = DDP(model, device_ids=[rank], gradient_as_bucket_view = True)
    optimizer = optim.Adam(
        ddp_model.parameters(), lr=1e-4, weight_decay=0.0005, betas=(0.937, 0.999)
    )
    scheduler = optim.lr_scheduler.OneCycleLR(
        optimizer,
        max_lr=1e-4,
        epochs=300,
        steps_per_epoch=len(train_loader),
        anneal_strategy="cos",
    )
</code></pre>

ptrblck: <aside class="quote no-group" data-username="weight_theta" data-post="8" data-topic="201768">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/d26b3c/48.png" class="avatar"> weight_theta:</div>
<blockquote>
<p>This would mean for my case tath <code>batchsize/n_gpus = 30 / 2 = 15</code> also tested with <code>batchsize/n_gpus = int(30/4) = 7</code> on each gpu. The GPU memory load reaches its limits and then throws a cuda out of memory error</p>
</blockquote>
</aside>
<p>Just to make sure we are talking about the same configs and expectations.</p>
<p>The deprecated <code>DataParallel</code> (<code>DP</code>) module allocates a single batch on the default device and shards it in the forward pass in <code>dim0</code>. The output is collected again on the default device and will thus create a memory imbalance. I.e. if you are using 8 GPUs and a batch size of 32, each device will use 32/8=4 only (with the exception of the default device where the entire input and output are collected).</p>
<p><code>DDP</code> on the other hand uses a single process per device and the defined batch size is also the used one on each device (or rank). If you want to use the same batch size as before with <code>DP</code>, you would need to decrease it and divide by the number of ranks. <code>DDP</code> is considered superior as it’s not causing a memory imbalance and reduces the communication between devices.</p>
<p>I’m currently unsure which config worked when, but you should double check what the actual batch size was.</p>
<p>Based on your code snippets it seems the batch size on <em>each rank</em> is set to 30, so you are increasing it in the <code>DDP</code> use case.</p>

weight_theta: <p>I abondoned the <code>DP</code> approached infavor of <code>DDP</code>, after reading up on the <code>DP</code> vs <code>DDP</code> and what is the standard way to do distributed parallel. I was not entirely clear on that <code>DDP</code> did not shard the dataset at <code>dim0</code>, but run its  own process on each rank or GPU.</p>
<p>So clarifiy, what you describe then means that:</p>
<ol>
<li><code>DP</code> shards the dataset at <code>dim0</code> and allocating a batchsize of batchsize / n_gpus to each gpu</li>
<li><code>DDP</code> on the other hand does not shard the dataset but runs a single process on each gpu, meaning that it will run a batchsize of <code>batchsize * 1</code> on each gpu or a<code> total batchsize = batch size * n_gpus</code></li>
</ol>
<p>So in my case as I am using DDP, we given <code>n_gpus = 2</code>, and a <code>batchsize = 30</code>, each GPU has a batchsize of <code>30</code>, and thus the total batchsize is <code>30* 2</code>. This is too large, I can however fit a batchsize of <code>16</code>, meaning <code>16</code> on each GPU process and thus, <code>32 = 16 * 2</code> in total.</p>

ptrblck: <p>Yes, your explanation is correct. In addition <code>DDP</code> uses a <code>DistributedSampler</code> to make sure each rank only loads and processes its corresponding samples to avoid duplications. This was not needed in the (deprecated) <code>DP</code> approach since only a single process was loading all samples.</p>