ERROR: Unexpected segmentation fault encountered in worker

r00bi: <p>I got this error when it gets to 95% of Epoch 0 to load validation set to test the performance of training set.<br>
Epoch 0:  95%|████████████████ERROR: Unexpected segmentation fault encountered in worker.6&lt;00:11,  4.30it/s, loss=nan]<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
ERROR: Unexpected segmentation fault encountered in worker.<br>
Traceback (most recent call last):<br>
File “/home/se/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1120, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/home/lib/python3.10/queue.py”, line 180, in get<br>
self.not_empty.wait(remaining)<br>
File “/home/lib/python3.10/threading.py”, line 324, in wait<br>
gotit = waiter.acquire(True, timeout)<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 1202763) is killed by signal: Segmentation fault.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py”, line 38, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 650, in _fit_impl<br>
self._run(model, ckpt_path=self.ckpt_path)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1103, in _run<br>
results = self._run_stage()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1182, in _run_stage<br>
self._run_train()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1205, in _run_train<br>
self.fit_loop.run()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py”, line 267, in advance<br>
self._outputs = self.epoch_loop.run(self._data_fetcher)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 200, in run<br>
self.on_advance_end()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 250, in on_advance_end<br>
self._run_validation()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 308, in _run_validation<br>
self.val_loop.run()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py”, line 152, in advance<br>
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py”, line 121, in advance<br>
batch = next(data_fetcher)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 184, in <strong>next</strong><br>
return self.fetching_function()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 265, in fetching_function<br>
self._fetch_next_batch(self.dataloader_iter)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 280, in _fetch_next_batch<br>
batch = next(iterator)<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 628, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1316, in _next_data<br>
idx, data = self._get_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1272, in _get_data<br>
success, data = self._try_get_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1133, in _try_get_data<br>
raise RuntimeError(‘DataLoader worker (pid(s) {}) exited unexpectedly’.format(pids_str)) from e<br>
RuntimeError: DataLoader worker (pid(s) 1202763) exited unexpectedly</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>Traceback (most recent call last):<br>
File “/home/bin/casanovo”, line 8, in <br>
sys.exit(main())<br>
File “/home/lib/python3.10/site-packages/click/core.py”, line 1130, in <strong>call</strong><br>
return self.main(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/click/core.py”, line 1055, in main<br>
rv = self.invoke(ctx)<br>
File “/home/lib/python3.10/site-packages/click/core.py”, line 1404, in invoke<br>
return ctx.invoke(self.callback, **ctx.params)<br>
File “/home/lib/python3.10/site-packages/click/core.py”, line 760, in invoke<br>
return __callback(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/casanovo/casanovo.py”, line 256, in main<br>
model_runner.train(peak_path, peak_feature, peak_path_val, peak_feature_val, model, config)<br>
File “/home/lib/python3.10/site-packages/casanovo/denovo/model_runner.py”, line 320, in train<br>
trainer.fit(<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 608, in fit<br>
call._call_and_handle_interrupt(<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py”, line 59, in _call_and_handle_interrupt<br>
trainer.strategy.reconciliate_processes(traceback.format_exc())<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py”, line 460, in reconciliate_processes<br>
raise DeadlockDetectedException(f"DeadLock detected from rank: {self.global_rank} \n {trace}")<br>
pytorch_lightning.utilities.exceptions.DeadlockDetectedException: DeadLock detected from rank: 1<br>
Traceback (most recent call last):<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1120, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/home/lib/python3.10/queue.py”, line 180, in get<br>
self.not_empty.wait(remaining)<br>
File “/home/lib/python3.10/threading.py”, line 324, in wait<br>
gotit = waiter.acquire(True, timeout)<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 1202763) is killed by signal: Segmentation fault.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py”, line 38, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 650, in _fit_impl<br>
self._run(model, ckpt_path=self.ckpt_path)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1103, in _run<br>
results = self._run_stage()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1182, in _run_stage<br>
self._run_train()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py”, line 1205, in _run_train<br>
self.fit_loop.run()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py”, line 267, in advance<br>
self._outputs = self.epoch_loop.run(self._data_fetcher)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 200, in run<br>
self.on_advance_end()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 250, in on_advance_end<br>
self._run_validation()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py”, line 308, in _run_validation<br>
self.val_loop.run()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py”, line 152, in advance<br>
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py”, line 199, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py”, line 121, in advance<br>
batch = next(data_fetcher)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 184, in <strong>next</strong><br>
return self.fetching_function()<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 265, in fetching_function<br>
self._fetch_next_batch(self.dataloader_iter)<br>
File “/home/lib/python3.10/site-packages/pytorch_lightning/utilities/fetching.py”, line 280, in _fetch_next_batch<br>
batch = next(iterator)<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 628, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1316, in _next_data<br>
idx, data = self._get_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1272, in _get_data<br>
success, data = self._try_get_data()<br>
File “/home/lib/python3.10/site-packages/torch/utils/data/dataloader.py”, line 1133, in _try_get_data<br>
raise RuntimeError(‘DataLoader worker (pid(s) {}) exited unexpectedly’.format(pids_str)) from e<br>
RuntimeError: DataLoader worker (pid(s) 1202763) exited unexpectedly</p>
<p>Killed<br>
Surprisingly, after getting this error <strong>all gpus will be invisible</strong> so torch.cuda.is_available() returns False and cannot find the gpus anymore.</p>

r00bi: <aside class="quote no-group" data-username="fduwjj" data-post="4" data-topic="180080" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fduwjj/48/47855_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/runtimeerror-no-gpus-available/180080/4">RuntimeError("No GPUs available.")</a></div>
<blockquote>
<p>How did you start your training job on 8GPUs?</p>
</blockquote>
</aside>
<p>There are 8 16G GPUS:<br>
trainer = pl.Trainer(<br>
accelerator=“auto”,<br>
auto_select_gpus=True,<br>
callbacks=callbacks,<br>
devices=_get_devices(),<br>
max_epochs=config[“max_epochs”],<br>
strategy=_get_strategy(),<br>
)<br>
# Train the model.<br>
trainer.fit(<br>
model, train_loader.train_dataloader(), val_loader.val_dataloader()<br>
)</p>
<p>I learned that why the gpus will be invisible please read this:</p><aside class="quote quote-modified" data-post="1" data-topic="180187">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/r00bi/48/47334_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/error-unexpected-segmentation-fault-encountered-in-worker/180187">ERROR: Unexpected segmentation fault encountered in worker</a> <a class="badge-category__wrapper " href="/c/data/37"><span data-category-id="37" data-drop-close="true" class="badge-category " title="Topics related to DataLoader, Dataset, torch.utils.data, pytorch/data, and TorchArrow."><span class="badge-category__name">data</span></span></a>
  </div>
  <blockquote>
    I got this error when it gets to 95% of Epoch 0 to load validation set to test the performance of training set. 
Epoch 0:  95%|████████████████ERROR: Unexpected segmentation fault encountered in worker.6&lt;00:11,  4.30it/s, loss=nan] 
ERROR: Unexpected segmentation fault encountered in worker. 
ERROR: Unexpected segmentation fault encountered in worker. 
ERROR: Unexpected segmentation fault encountered in worker. 
ERROR: Unexpected segmentation fault encountered in worker. 
ERROR: Unexpected segme…
  </blockquote>
</aside>

<p>Thanks!</p>

r00bi: <p>I learned this error happened because of using Cython, so I had to remove that part of the code and replace it with python. I don’t why it happened after calling validation set for calculating training performance because the cython functions called during the training part as well. Anyway it was fixed by removing replacing cython code.</p>