Unexpected segmentation fault encountered in worker when loading dataset

SHM: <p>I encounter the following error when using DataLoader workers to load data.<br>
I am using <a href="https://github.com/pyg-team/pytorch_geometric/blob/master/torch_geometric/loader/neighbor_sampler.py" rel="noopener nofollow ugc">NeighborSampler in PyG</a> as “loader” in run_main.py line 152 to load custom dataset, and use <code>num_workers</code> of <code>os.cpu_count()</code>.</p>
<pre><code class="lang-auto">ERROR: Unexpected segmentation fault encountered in worker.
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1134, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 107, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 424, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/home/user/.local/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 1096707) is killed by signal: Segmentation fault.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "run_main.py", line 152, in train
    for step, _ in enumerate(loader):
  File "/home/user/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 652, in __next__
    data = self._next_data()
  File "/home/user/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1330, in _next_data
    idx, data = self._get_data()
  File "/home/user/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1296, in _get_data
    success, data = self._try_get_data()
  File "/home/user/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1147, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 1096707) exited unexpectedly
</code></pre>
<p>I am using Pytorch 1.12.0+cu116, one NVIDIA TITAN Xp GPU, CUDA version of 11.6, and Python version of 3.8.10.</p>
<p>I’ve searched a lot for this error, and found following solutions. However, they were all unhelpful.</p>
<ul>
<li>Using <code>num_workers</code> of 0 or 1. When I run with <code>num_workers</code> of 0, it results in “corrupted double-linked list” error. When I set <code>num_workers</code> as 1, the same error (Unexpected segmentation fault encountered in worker) still occurs. Actually, I don’t want to lessen <code>num_workers</code>, because I am working on a pretty large dataset and lessening <code>num_workers</code> is a much slower option.</li>
<li>Increasing shared memory size. I’ve done this by adding <code>none /dev/shm tmpfs defaults,size=MY_SIZEG 0 0</code> line in <code>/etc/fstab</code> and running <code>mount -o remount /dev/shm</code>. I’ve set MY_SIZE exactly as the size of main memory (which was previously 50% of main memory).</li>
<li>Changing Python version to &lt;= 3.6.9. I’ve tried this, but the same error still occurs.</li>
<li>Checking that Python and dataset are mounted on the same disk. I’ve already verified that they are mounted on the same disk.</li>
</ul>
<p>I’ve been struggling to fix this issue for several days, but I can’t find the right solution, and it really makes me frustrated. Could you please help me out?</p>

ptrblck: <aside class="quote no-group" data-username="SHM" data-post="1" data-topic="163947">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/58f4c7/48.png" class="avatar"> SHM:</div>
<blockquote>
<p>When I run with <code>num_workers</code> of 0, it results in “corrupted double-linked list” error.</p>
</blockquote>
</aside>
<p>Try to solve this issue first before trying to debug the one raised my multiple workers.</p>

SHM: <p>I will try to fix this issue ASAP and share the results. Thanks.</p>

SHM: <p>I’ve figured this out. I was using a customized, synthesized dataset expanded from an existing dataset. I’ve re-synthesized the dataset and the error disappeared when applied on this new dataset. I’m not sure why, but it seems that the previous dataset was corrupted in some way. Thanks for your advice.</p>

ptrblck: <p>Thanks for the follow up and good to hear you’ve solved the issue. I assume the error raised by multiple workers is also gone now?</p>

SHM: <p>Yes, those symptoms were all gone now.</p>

r00bi: <aside class="quote no-group" data-username="SHM" data-post="4" data-topic="163947">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/58f4c7/48.png" class="avatar"> SHM:</div>
<blockquote>
<p>red this out. I was using a customized, synthesized dataset expanded from an existing dataset. I’ve re-synthesized the dataset and the error disappeared when applied on this new dataset. I’m not sure why, but it seems that the previous dataset was corrupted in some way. Thanks for your advice.</p>
</blockquote>
</aside>
<p>I am getting the ERROR: Unexpected segmentation fault encountered in worker.<br>
but cannot figure out why when it gets to load validation set after first epoch to calculate the performance of training set, it returns this error:<br>
Epoch 0:  95%|████████████████ERROR: Unexpected segmentation fault encountered in worker.6&lt;00:11,  4.30it/s, loss=nan]<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
Traceback (most recent call last):<br>
File “/home/se/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1120, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/home/lib/python3.10/queue.py”, line 180, in get<br>
self.not_empty.wait(remaining)<br>
File “/home/lib/python3.10/threading.py”, line 324, in wait<br>
gotit = waiter.acquire(True, timeout)<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 1202763) is killed by signal: Segmentation fault.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py”, line 38, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 650, in _fit_impl<br>
self._run(model, ckpt_path=self.ckpt_path)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1103, in _run<br>
results = self._run_stage()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1182, in _run_stage<br>
self._run_train()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1205, in _run_train<br>
self.fit_loop.run()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py”, line 267, in advance<br>
self._outputs = self.epoch_loop.run(self._data_fetcher)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 200, in run<br>
self.on_advance_end()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 250, in on_advance_end<br>
self._run_validation()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 308, in _run_validation<br>
self.val_loop.run()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py”, line 152, in advance<br>
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py”, line 121, in advance<br>
batch = next(data_fetcher)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 184, in <strong>next</strong><br>
return self.fetching_function()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 265, in fetching_function<br>
self._fetch_next_batch(self.dataloader_iter)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 280, in _fetch_next_batch<br>
batch = next(iterator)<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 628, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1316, in _next_data<br>
idx, data = self._get_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1272, in _get_data<br>
success, data = self._try_get_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1133, in _try_get_data<br>
raise RuntimeError(‘DataLoader worker (pid(s) {}) exited unexpectedly’.format(pids_str)) from e<br>
RuntimeError: DataLoader worker (pid(s) 1202763) exited unexpectedly</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>Traceback (most recent call last):<br>
File “/home/bin/casanovo”, line 8, in <br>
sys.exit(main())<br>
File “/home/lib/python3.10/site-packages/click/core.py”, line 1130, in <strong>call</strong><br>
return self.main(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/click/core.py”, line 1055, in main<br>
rv = self.invoke(ctx)<br>
File “/home/lib/python3.10/site-packages/click/core.py”, line 1404, in invoke<br>
return ctx.invoke(self.callback, **ctx.params)<br>
File “/home/lib/python3.10/site-packages/click/core.py”, line 760, in invoke<br>
return __callback(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/casanovo/casanovo.py”, line 256, in main<br>
model_runner.train(peak_path, peak_feature, peak_path_val, peak_feature_val, model, config)<br>
File “/home/lib/python3.10/site-packages/casanovo/denovo/model_runner.py”, line 320, in train<br>
trainer.fit(<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 608, in fit<br>
call._call_and_handle_interrupt(<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py”, line 59, in _call_and_handle_interrupt<br>
trainer.strategy.reconciliate_processes(traceback.format_exc())<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py”, line 460, in reconciliate_processes<br>
raise DeadlockDetectedException(f"DeadLock detected from rank: {self.global_rank} \n {trace}")<br>
pytorch_lightning.utilities.exceptions.DeadlockDetectedException: DeadLock detected from rank: 1<br>
Traceback (most recent call last):<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1120, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/home/lib/python3.10/queue.py”, line 180, in get<br>
self.not_empty.wait(remaining)<br>
File “/home/lib/python3.10/threading.py”, line 324, in wait<br>
gotit = waiter.acquire(True, timeout)<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 1202763) is killed by signal: Segmentation fault.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py”, line 38, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 650, in _fit_impl<br>
self._run(model, ckpt_path=self.ckpt_path)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1103, in _run<br>
results = self._run_stage()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1182, in _run_stage<br>
self._run_train()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1205, in _run_train<br>
self.fit_loop.run()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py”, line 267, in advance<br>
self._outputs = self.epoch_loop.run(self._data_fetcher)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 200, in run<br>
self.on_advance_end()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 250, in on_advance_end<br>
self._run_validation()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 308, in _run_validation<br>
self.val_loop.run()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py”, line 152, in advance<br>
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py”, line 121, in advance<br>
batch = next(data_fetcher)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 184, in <strong>next</strong><br>
return self.fetching_function()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 265, in fetching_function<br>
self._fetch_next_batch(self.dataloader_iter)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 280, in _fetch_next_batch<br>
batch = next(iterator)<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 628, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1316, in _next_data<br>
idx, data = self._get_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1272, in _get_data<br>
success, data = self._try_get_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1133, in _try_get_data<br>
raise RuntimeError(‘DataLoader worker (pid(s) {}) exited unexpectedly’.format(pids_str)) from e<br>
RuntimeError: DataLoader worker (pid(s) 1202763) exited unexpectedly</p>
<p>Killed</p>

mohammad_moghadas: <p>I have the same error!</p>

Gregor: <p><a class="mention" href="/u/r00bi">@r00bi</a> Any updates how you solved it?</p>