SIGBUS error with DistributedDataParallel (NCCL)

username: ThiagoPorto
cooked: <p>Hi, I’m trying to run a code to test DistributedDataParallel with nccl backend, but I’m receiving a SIGBUS error, if I change to DataParallel it works. Trying to run o 4 V100 GPUs.</p>
<p>Edit:<br>
Running with NCCL_DEBUG, TORCH_CPP_LOG_LEVEL, and TORCH_DISTRIBUTED_DEBUG, I got Something like this:</p>
<pre><code class="lang-auto">NCCL INFO Failed to open libibverbs.so[.1]
 [2] NCCL INFO NET/Socket : Using [0]eth0:10.224.0.70&lt;0&gt;
 [2] NCCL INFO Using network Socket
 [1] NCCL INFO Setting affinity for GPU 1 to fff000
 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
 [0] NCCL INFO Setting affinity for GPU 0 to fff000
 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
 [0] NCCL INFO Channel 00/02 :    0   1   2   3
 [0] NCCL INFO Channel 01/02 :    0   1   2   3
 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1
 [0] NCCL INFO Channel 00 : 0[100000] -&gt; 1[200000] via SHM/direct/direct
 [0] NCCL INFO Channel 01 : 0[100000] -&gt; 1[200000] via SHM/direct/direct
 [2] NCCL INFO Trees [0] 3/-1/-1-&gt;2-&gt;1 [1] 3/-1/-1-&gt;2-&gt;1
 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0 [1] 2/-1/-1-&gt;1-&gt;0
 [3] NCCL INFO Trees [0] -1/-1/-1-&gt;3-&gt;2 [1] -1/-1/-1-&gt;3-&gt;2
"python3.9/site-packages/torch/multiprocessing/spawn.py", line 140, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 3 terminated with signal SIGBUS
</code></pre>

username: fduwjj
cooked: <p>Do you have a code example so that we can repro?</p>

username: ThiagoPorto
cooked: <p>Thank you for responding. Sure, I’ll provide the entire script since it’s small. Additionally, when I switch the backend to “gloo,” it works. I’m currently running it in JupyterLab with Azure. Perhaps there’s an issue with the instance.</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from datautils import MyTrainDataset

import torch.multiprocessing as mp
from torch.utils.data.distributed import DistributedSampler
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.distributed import init_process_group, destroy_process_group
import os


def ddp_setup(rank, world_size):
    """
    Args:
        rank: Unique identifier of each process
        world_size: Total number of processes
    """
    os.environ["MASTER_ADDR"] = "localhost"
    os.environ["MASTER_PORT"] = "12355"
    init_process_group(backend="nccl", rank=rank, world_size=world_size)

class Trainer:
    def __init__(
        self,
        model: torch.nn.Module,
        train_data: DataLoader,
        optimizer: torch.optim.Optimizer,
        gpu_id: int,
        save_every: int,
    ) -&gt; None:
        self.gpu_id = gpu_id
        self.model = model.to(gpu_id)
        self.train_data = train_data
        self.optimizer = optimizer
        self.save_every = save_every
        self.model = DDP(model, device_ids=[gpu_id])

    def _run_batch(self, source, targets):
        self.optimizer.zero_grad()
        output = self.model(source)
        loss = F.cross_entropy(output, targets)
        loss.backward()
        self.optimizer.step()

    def _run_epoch(self, epoch):
        b_sz = len(next(iter(self.train_data))[0])
        print(f"[GPU{self.gpu_id}] Epoch {epoch} | Batchsize: {b_sz} | Steps: {len(self.train_data)}")
        self.train_data.sampler.set_epoch(epoch)
        for source, targets in self.train_data:
            source = source.to(self.gpu_id)
            targets = targets.to(self.gpu_id)
            self._run_batch(source, targets)

    def _save_checkpoint(self, epoch):
        ckp = self.model.module.state_dict()
        PATH = "checkpoint.pt"
        torch.save(ckp, PATH)
        print(f"Epoch {epoch} | Training checkpoint saved at {PATH}")

    def train(self, max_epochs: int):
        for epoch in range(max_epochs):
            self._run_epoch(epoch)
            if self.gpu_id == 0 and epoch % self.save_every == 0:
                self._save_checkpoint(epoch)


def load_train_objs():
    train_set = MyTrainDataset(2048)  # load your dataset
    model = torch.nn.Linear(20, 1)  # load your model
    optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
    return train_set, model, optimizer


def prepare_dataloader(dataset: Dataset, batch_size: int):
    return DataLoader(
        dataset,
        batch_size=batch_size,
        pin_memory=True,
        shuffle=False,
        sampler=DistributedSampler(dataset)
    )


def main(rank: int, world_size: int, save_every: int, total_epochs: int, batch_size: int):
    ddp_setup(rank, world_size)
    dataset, model, optimizer = load_train_objs()
    train_data = prepare_dataloader(dataset, batch_size)
    trainer = Trainer(model, train_data, optimizer, rank, save_every)
    trainer.train(total_epochs)
    destroy_process_group()


if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description='simple distributed training job')
    parser.add_argument('total_epochs', type=int, help='Total epochs to train the model')
    parser.add_argument('save_every', type=int, help='How often to save a snapshot')
    parser.add_argument('--batch_size', default=32, type=int, help='Input batch size on each device (default: 32)')
    args = parser.parse_args()
    
    world_size = torch.cuda.device_count()
    mp.spawn(main, args=(world_size, args.save_every, args.total_epochs, args.batch_size), nprocs=world_size)
</code></pre>
<pre><code class="lang-python">import torch
from torch.utils.data import Dataset

class MyTrainDataset(Dataset):
    def __init__(self, size):
        self.size = size
        self.data = [(torch.rand(20), torch.rand(1)) for _ in range(size)]

    def __len__(self):
        return self.size
    
    def __getitem__(self, index):
        return self.data[index]
</code></pre>

username: xsf
cooked: <p>Hi,Did you solve this problem?I hit the same question when i try to run training in Kubernetes.Also when use gloo and it works</p>

username: ThiagoPorto
cooked: <p>I also was trying to run on Kubernetes, and it’s related with the shared memory of the container.</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/43373463/how-to-increase-shm-size-of-a-kubernetes-container-shm-size-equivalent-of-doc/47921100#47921100">
  <header class="source">

      <a href="https://stackoverflow.com/questions/43373463/how-to-increase-shm-size-of-a-kubernetes-container-shm-size-equivalent-of-doc/47921100#47921100" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1165797/glenn-vandamme" target="_blank" rel="noopener nofollow ugc">
    <img alt="Glenn Vandamme" src="https://www.gravatar.com/avatar/e82030b383c11a50fbf18d29fb8fcada?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/43373463/how-to-increase-shm-size-of-a-kubernetes-container-shm-size-equivalent-of-doc/47921100#47921100" target="_blank" rel="noopener nofollow ugc">How to increase shm size of a kubernetes container (--shm-size equivalent of docker)</a>
</h4>

<div class="tags">
  <strong>docker, kubernetes</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/1165797/glenn-vandamme" target="_blank" rel="noopener nofollow ugc">
    Glenn Vandamme
  </a>
  on <a href="https://stackoverflow.com/questions/43373463/how-to-increase-shm-size-of-a-kubernetes-container-shm-size-equivalent-of-doc/47921100#47921100" target="_blank" rel="noopener nofollow ugc">08:41AM - 21 Dec 17 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine/46434614#46434614">
  <header class="source">

      <a href="https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine/46434614#46434614" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/939701/posit-labs" target="_blank" rel="noopener nofollow ugc">
    <img alt="posit labs" src="https://www.gravatar.com/avatar/40daf1700004c187763959bd3055bb9e?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine/46434614#46434614" target="_blank" rel="noopener nofollow ugc">Define size for /dev/shm on container engine</a>
</h4>

<div class="tags">
  <strong>docker, google-cloud-platform, google-kubernetes-engine, kubectl</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/939701/posit-labs" target="_blank" rel="noopener nofollow ugc">
    posit labs
  </a>
  on <a href="https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine/46434614#46434614" target="_blank" rel="noopener nofollow ugc">07:46PM - 26 Sep 17 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>