How to fix a SIGSEGV in pytorch when using distributed training (e.g. DDP)?

Brando_Miranda: <p>I keep getting this issue when running DDP in pytorch:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 330, in &lt;module&gt;
    main_distributed()
  File "ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 230, in main_distributed
    mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 105, in join
    raise Exception(
Exception: process 1 terminated with signal SIGSEGV
</code></pre>
<p>but this error is rather uninformative (dones’t tell me what process or what it was trying to access for example) so I am unsure what I need to do do solve it.</p>
<p><a href="https://www.suse.com/support/kb/doc/?id=000016786#:~:text=On%20a%20Unix%20operating%20system,memory%20address%20that%20does%20not" rel="noopener nofollow ugc">Some research</a> takes you that:</p>
<blockquote>
<p>SIGSEGV: On a Unix operating system such as Linux, a “segmentation violation” (also known as “signal 11”, “SIGSEGV”, “segmentation fault” or, abbreviated, “sig11” or “segfault”) is a signal sent by the kernel to a process when the system has detected that the process was attempting to access a memory address that does not belong to it. Typically, this results in the offending process being terminated.</p>
</blockquote>
<p>yes I do have multiprocessing code as the usual <code>mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)</code> requires.</p>
<p>First I read the docs on <a href="https://pytorch.org/docs/stable/multiprocessing.html#sharing-strategies" rel="noopener nofollow ugc">sharing strategies</a> which talks about how tensors are shared in pytorch:</p>
<blockquote>
<p>Note that it applies only to CPU tensor - CUDA tensors will always use the CUDA API, as that’s the only way they can be shared.</p>
</blockquote>
<p>I was using the <a href="https://pytorch.org/docs/stable/multiprocessing.html#file-system-file-system" rel="noopener nofollow ugc">file system sharing memory</a> since it seemed to give me less issue when I needed lots of processes but I went down to only 2 processes and 2 gpus and to the share strategy being <a href="https://pytorch.org/docs/stable/multiprocessing.html#file-system-file-system" rel="noopener nofollow ugc">file descriptor</a>. I thought that perhaps if the processes had their own cached file descriptor then there wouldn’t be issues.</p>
<p>I did check the cuda devices availabe:</p>
<pre><code class="lang-auto"> $ echo $CUDA_VISIBLE_DEVICES
1,3
</code></pre>
<p>all seems fine.</p>
<p>I am unsure what might be causing the issue. There are possible issues like:</p>
<ul>
<li>two processes are trying to checkpoint at the same time but I always only let <code>rank=0</code> do the checkpointing so that doesn’t make sense.</li>
<li>two processes are writing to tensorboard but I also only allow <code>rank=0</code> to do the logging (or any of the printing).</li>
</ul>
<p>So I am unsure what could be causing the issue. It could be that I have my dataset concatenated all 1 single json file causing the issue, but that wasn’t causing issues yesterday with multiple gpus…though, if that is the case it would be hard to fix since <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc">DDP (distributed data parallel)</a> uses the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler" rel="noopener nofollow ugc">DistributedSampler</a> which doesn’t place any restriction like that on my data-set or dataloaders…or at least as far as I know (afaik).</p>
<p>Last thing is that yesterday I was getting weird error too and somehow it occurred to me to check the gpu type. I was quetting an issue because I was using a k40 gpu. I made sure that was not the case. Yesterday I was using a <code>Quadro 6000 RTX</code>, today it seems these are the GPUs I got:</p>
<pre><code class="lang-auto">$ nvidia-smi
Tue Mar  2 12:15:04 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06    Driver Version: 450.36.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 00000000:02:00.0 Off |                  N/A |
| 22%   37C    P0    56W / 250W |      0MiB / 12196MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    Off  | 00000000:03:00.0 Off |                  N/A |
| 24%   39C    P0    56W / 250W |      0MiB / 12196MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    Off  | 00000000:82:00.0 Off |                  N/A |
| 53%   84C    P2   244W / 250W |  11935MiB / 12196MiB |     57%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  TITAN X (Pascal)    Off  | 00000000:83:00.0 Off |                  N/A |
| 25%   39C    P0    56W / 250W |      0MiB / 12196MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    2   N/A  N/A     31809      C   python                          11933MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>not sure if that is causing the issue but it’s not always realistic to get the Quadro’s so I want it to work for the Titan’s too (and anything that isn’t the k40s since the k40s seem to not be supported by pytorch anymore).</p>
<p>There are a few pytorch discussion forum posts and gitissues but none seems very helpful (to me at lest - not clear what they did to solve things despite end of discussion):</p>
<ul>
<li>
<p><a href="https://discuss.pytorch.org/t/multiprocessing-using-torch-multiprocessing/11029/3" class="inline-onebox">Multiprocessing using torch.multiprocessing - #3 by Brando_Miranda</a></p>
</li>
<li>
<p><a href="https://discuss.pytorch.org/t/using-torch-tensor-over-multiprocessing-queue-process-fails/2847/12" class="inline-onebox">Using torch.Tensor over multiprocessing.Queue + Process fails - #12 by eacousineau</a></p>
</li>
<li>
<p><a href="https://github.com/pytorch/fairseq/issues/1720" class="inline-onebox" rel="noopener nofollow ugc">Process 3 terminated with signal SIGSEGV · Issue #1720 · facebookresearch/fairseq · GitHub</a></p>
</li>
<li>
<p><a href="https://github.com/pytorch/fairseq/issues/1308" class="inline-onebox" rel="noopener nofollow ugc">SIGSEGV while running train.py on a multi GPU setup · Issue #1308 · facebookresearch/fairseq · GitHub</a></p>
</li>
<li>
<p><a href="https://github.com/facebookresearch/SlowFast/issues/118" class="inline-onebox" rel="noopener nofollow ugc">Exception: process 0 terminated with signal SIGSEGV · Issue #118 · facebookresearch/SlowFast · GitHub</a></p>
</li>
<li>
<p><a href="https://discuss.pytorch.org/t/segmentation-fault/23489" class="inline-onebox">Segmentation fault</a></p>
</li>
</ul>
<hr>
<p>crossposted:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/66444927/how-to-fix-a-sigsegv-in-pytorch-when-using-distributed-training-e-g-ddp" class="inline-onebox" rel="noopener nofollow ugc">python - How to fix a SIGSEGV in pytorch when using distributed training (e.g. DDP)? - Stack Overflow</a></li>
<li><a href="https://www.reddit.com/r/pytorch/comments/lwbb72/how_to_fix_a_sigsegv_in_pytorch_when_using/" class="inline-onebox" rel="noopener nofollow ugc">Reddit - Dive into anything</a></li>
</ul>

dsethz: <p>Hey <a class="mention" href="/u/brando_miranda">@Brando_Miranda</a>,</p>
<p>I have a very similar if not the same issue (difficult to say). Have you found a solution to this problem? In my case this issue also occurs rather infrequently. Running on the same server (same GPUs, environment, etc.) training my model sometimes is successful and sometimes ends with <code>SIGSEGV</code>.</p>
<p>Cheers</p>
<p>Edit: If it is of any help, I posted my code <a href="https://discuss.pytorch.org/t/how-to-fix-sigsegv-in-distributed-training-i-e-ddp-2/120571">here</a>.</p>

Brando_Miranda: <p>Hi Dsethcz. I have not been able to solve the weird memory errors. However, I noticed that it happened only at the end of my training script (i.e. once destroying the dist group happened). So my model in theory seemed to have trained to the end and the issue is something when shutting down the distributed set up. From skimming your posts it seems your having the same behaviour. Did you just try checkpointing your model or doing whatever you had to do up to the point you have to destroy the dist group?</p>
<p>For me I just collect all the process with some wait call and then allow them to crash (so that the same error occurs) and it seems to give the same error. I think it must be something with my server because in the dgx I have access I don’t think I’ve ever happened. I recommend to ignore it if you fall in my scenario.</p>
<p>best of luck!</p>

dsethz: <p>Hmm that is not an optimal solution…</p>
<p>Have you tested if the trains correctly until the crash occurs (by for example comparing to single GPU or <code>DataParallel</code> training)?</p>

Brando_Miranda: <p>single gpu works fine. Dgx machine works fine. I can’t see a pattern on which gpu is crashing on me.</p>
<p>I don’t use <code>DataParallel</code> so no.</p>
<p>Yea I know it’s suboptimal but sometimes due to the laws of diminishing returns the last tiny gain (which is just that my script doesn’t print an errort) isn’t worth the (already days/weeks of effort) I put into solving it. It’s also not really my job to solve pytorch not work. So I am happy with my current solution as long as it trains to completion as it is now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

dsethz: <p>Not really a solution, but a workaround that seems to work: I downgraded from <code>python 3.9.1</code> to <code>3.8.7</code>. A more detailed report can be found <a href="https://discuss.pytorch.org/t/how-to-fix-sigsegv-in-distributed-training-i-e-ddp-2/120571">here</a>.</p>

Brando_Miranda: <p>I also play around with version of stuff. Python 3.8 pytorch 1.7.1 and cuda 10.2 (but have driver of at leat 11.0) seemed to work consistently for me.</p>
<p>Glad your python version change worked.</p>

Richard_Jenkins: <p>I had the same problem and solution. I downgraded python to 3.8.7 and to PyTorch 1.7.  I was using torchrun and ddp in PyTorch 1.10, but torchrun doesn’t work w PyTorch 1.7 so I had to stop using torchrun and use torch.distributed.launch instead. Now it works smoothly and no sigsegv errors.</p>

PalaashAgrawal: <p>This worked for me</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/NVlabs/stylegan2-ada-pytorch/issues/190#issuecomment-953676470">
  <header class="source">

      <a href="https://github.com/NVlabs/stylegan2-ada-pytorch/issues/190#issuecomment-953676470" target="_blank" rel="noopener nofollow ugc">github.com/NVlabs/stylegan2-ada-pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/NVlabs/stylegan2-ada-pytorch/issues/190#issuecomment-953676470" target="_blank" rel="noopener nofollow ugc">raise ProcessExitedException( torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGSEGV</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-10-22" data-time="14:32:44" data-timezone="UTC">02:32PM - 22 Oct 21 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2021-10-29" data-time="09:13:05" data-timezone="UTC">09:13AM - 29 Oct 21 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/zhanjiahui" target="_blank" rel="noopener nofollow ugc">
          <img alt="zhanjiahui" src="https://avatars.githubusercontent.com/u/36075322?v=4" class="onebox-avatar-inline" width="20" height="20">
          zhanjiahui
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">**Describe the bug**
``` python
Evaluating metrics...
/mnt/lab/zjh/anaconda3/<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">envs/pytorch_gpu/lib/python3.8/site-packages/torch/nn/modules/module.py:1051: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return forward_call(*input, **kwargs)
/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/torch/nn/modules/module.py:1051: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return forward_call(*input, **kwargs)
/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/torch/nn/modules/module.py:1051: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return forward_call(*input, **kwargs)
/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/torch/nn/modules/module.py:1051: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
  return forward_call(*input, **kwargs)
Traceback (most recent call last):
  File "train.py", line 538, in &lt;module&gt;
    main() # pylint: disable=no-value-for-parameter
  File "/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "train.py", line 533, in main
    torch.multiprocessing.spawn(fn=subprocess_fn, args=(args, temp_dir), nprocs=args.num_gpus)
  File "/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 130, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGSEGV

/mnt/lab/zjh/anaconda3/envs/pytorch_gpu/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 68 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

```

**To Reproduce**
python train.py --outdir=training_runs --data=anime_trainB.zip --gpus=4



**Server**
 - OS: Linux Ubuntu 18.04.1
 - PyTorch version pytorch 1.9.0
 - CUDA toolkit version CUDA 11.4
 - NVIDIA driver version 470.57.02
 - GPU four RTX 3090

**Additional context**</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Apparantly you need to downgrade your python version to 3.7 and pytorch version to 1.7.1. If you are using cuda 11.0, then upgrade to 1.7.1+cu110</p>

pix1: <p>Replacing <code>mp.spawn</code> with the <code>start</code> and <code>join</code> methods can solve this problem. The issue is likely caused by a faulty implementation of <code>spawn</code> in PyTorch, which leads to incorrect mapping of shared memory between processes. Using <code>start</code> and <code>join</code> avoids this problem and prevents segmentation faults.</p>
<pre><code class="lang-auto"># mp.spawn(run, args=(world_size, q), nprocs=world_size, join=True)
children = []
for i in range(world_size):
    subproc = mp.Process(target=run, args=(i, world_size, q))
    children.append(subproc)
    subproc.start()

for i in range(world_size):
    children[i].join()
</code></pre>
<p>With modifications made like this, the code should work properly.</p>

Felipe_Cordeiro: <p>Even with this modification my training didn’t start. See my code below:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import (
    DistributedSampler
)
from torch.distributed import init_process_group, destroy_process_group

def ddp_setup(rank: int, world_size: int):
        """
        Args:
            Rank: Unique identifier of each process
            world_size: Total number of process
        """
        os.environ["MASTER_ADDR"] = "localhost"
        os.environ["MASTER_PORT"] = "3000"  

        init_process_group(backend="nccl", rank=rank, world_size=world_size)

def prepare_dataloader(
        trainset: Dataset,
        bs: int,
    ) -&gt; DataLoader:
        sampler_train = DistributedSampler(trainset)
        
        trainloader = DataLoader(
            trainset, 
            batch_size=bs, 
            pin_memory=True,
            shuffle=False, 
            sampler=sampler_train
        )
        return trainloader

class Trainer:
    def __init__(
        self,
        model: AutoModelForUniversalSegmentation,
        train_data: DataLoader,
        optimizer: torch.optim,
        gpu_id: int,
        save_every: int
    ) -&gt; None:
        self.gpu_id = gpu_id
        self.model = model.to(gpu_id)
        self.train_data = train_data
        self.optimizer = optimizer
        self.save_every = save_every
        self.model = DDP(self.model, device_ids=[self.gpu_id])

    def _run_epoch(self, epoch):
        b_sz = len(next(iter(self.train_data))[0])
        print(f"[GPU{self.gpu_id}] Epoch {epoch} | Batchsize: {b_sz}")

        for i, batch in enumerate(self.train_data):
            # zero the parameter gradients
            self.optimizer.zero_grad()

            batch = {k:v.to(self.gpu_id) for k,v in batch.items()}
            outputs = model(**batch)
            
            # backward pass + optimize
            loss = outputs.loss
            loss.backward()
            self.optimizer.step()   

    def _save_checkpoint(self, epoch):
        ckp = self.model.module.state_dict()
        torch.save(ckp, f"Checkpoint_{epoch}.pt")
        print(f"Epoch {epoch} | Training checkpoint save at checkpoint_{epoch}.pt")

    def train(self, epochs:int):
        for epoch in range(epochs):
            self._run_epoch(epoch)
            if self.gpu_id == 0 and epoch % self.save_every == 0:
                self._save_checkpoint(epoch)

from torch.optim import AdamW

def main(rank: int, world_size:int, epochs:int, save_every:int):
    torch.cuda.empty_cache()
    ddp_setup(rank, world_size)
    adamw = AdamW(model.parameters(), lr=1e-3)
    train_data = prepare_dataloader(dataset, bs=16)
    trainer = Trainer(model, train_data, adamw, rank, save_every)
    trainer.train(epochs)
    destroy_process_group()

world_size = torch.cuda.device_count()
total_epochs = 300
save_every = 10
children = []
#mp.spawn(main, args=(world_size, total_epochs, save_every,), nprocs=world_size)

for i in range(world_size):
    subproc = mp.Process(target=main, args=(i, world_size, total_epochs, save_every))
    children.append(subproc)
    subproc.start()

for i in range(world_size):
    children[i].join()</code></pre>

isalia99: <p>I have found that lowering num_workers in the DataLoader gets rid of this error, I had a very high value of 24 and by lowering it to 4, it helped it not run in this error</p>

Octav_Florescu: <p>my issue was the torch.onnx.export … this had appeared after setting the torch default device to cuda… it seems that torch.onnx.export depends on that. So to make it work, i have changed the default device to cpu before the export, then changed it back to cuda after the export. Hope this helps.</p>