DataLoader worker (pid(s) 2245) exited unexpectedly with Ubuntu on Windows

Khoa_112: <p>I have this error with dataloader when I try to run this python script on Ubuntu/Windows Terminal.</p>
<p>ERROR: Unexpected segmentation fault encountered in worker.<br>
Traceback (most recent call last):<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 990, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/queue.py”, line 179, in get<br>
self.not_empty.wait(remaining)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/threading.py”, line 306, in wait<br>
gotit = waiter.acquire(True, timeout)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 2245) is killed by signal: Segmentation fault.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “unet3d-lightning-tabular.py”, line 462, in <br>
trainer.fit(model)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 740, in fit<br>
self._call_and_handle_interrupt(<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 685, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 777, in _fit_impl<br>
self._run(model, ckpt_path=ckpt_path)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 1199, in _run<br>
self._dispatch()<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 1279, in _dispatch<br>
self.training_type_plugin.start_training(self)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py”, line 202, in start_training<br>
self._results = trainer.run_stage()<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 1289, in run_stage<br>
return self._run_train()<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 1311, in _run_train<br>
self._run_sanity_check(self.lightning_module)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py”, line 1375, in _run_sanity_check<br>
self._evaluation_loop.run()<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/loops/base.py”, line 145, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py”, line 110, in advance<br>
dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/loops/base.py”, line 140, in run<br>
self.on_run_start(*args, **kwargs)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py”, line 86, in on_run_start<br>
self._dataloader_iter = _update_dataloader_iter(data_fetcher, self.batch_progress.current.ready)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/loops/utilities.py”, line 121, in _update_dataloader_iter<br>
dataloader_iter = enumerate(data_fetcher, batch_idx)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py”, line 199, in <strong>iter</strong><br>
self.prefetching(self.prefetch_batches)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py”, line 258, in prefetching<br>
self._fetch_next_batch()<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py”, line 300, in _fetch_next_batch<br>
batch = next(self.dataloader_iter)<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 521, in <strong>next</strong><br>
data = self._next_data()<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 1186, in _next_data<br>
idx, data = self._get_data()<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 1142, in _get_data<br>
success, data = self._try_get_data()<br>
File “/home/khoatruong1412/miniconda3/envs/tensorflow/lib/python3.8/site-packages/torch/utils/data/dataloader.py”, line 1003, in _try_get_data<br>
raise RuntimeError(‘DataLoader worker (pid(s) {}) exited unexpectedly’.format(pids_str)) from e<br>
RuntimeError: DataLoader worker (pid(s) 2245) exited unexpectedly</p>
<p>I have tried many solutions such as changing num_workers to 4,8,16, wraping the model under if <strong>name</strong> == ‘<strong>main</strong>’:, changing the batch size, but so far nothing works.<br>
My computer specs are:<br>
Ryzen 7 3700x<br>
32gb ram<br>
rtx 3090</p>
<p>If you want to see the files and folders to have a better understanding of what is going on, please let me know. Thank you for your time!</p>

ejguan: <p>I would say you need to check if your <code>Dataset</code> instance is correct.<br>
Does it work using <code>num_worker=0</code>?</p>

Khoa_112: <p>Hi ejguan, I just fixed it a few days ago. The issue was the way I defined the dataset. Thanks for viewing my post!</p>