Dataloader stop working ? Deadlock?

Gitoa: <p>When I use pytorch to finetune ResNet, it runs well at the begining, but it stop running after several epoch.<br>
I check nvidia-smi, about half memory is occupied, but GPU is not working, while CPU is almost 100%. It seems like that GPU is waiting for the data from Dataloader which is preprocessed by CPU. I interrupt with CTRL-C, it return some information, can anyong tell me what had happend and how to solve this problem. smaller batch_size, fewer num_workers?  Any advice will be grateful…Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Gitoa: <p>After I interrupt with CTRL-C</p>
<p>epoch:  17  lr:  0.05  loss:  5.825265407562256  acc_rate:  0.65875  total_num:  1600<br>
^CProcess Process-70:<br>
Process Process-72:<br>
Traceback (most recent call last):<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/process.py”, line 249, in _bootstrap<br>
self.run()<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/process.py”, line 93, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/gitoo/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 50, in _worker_loop<br>
r = index_queue.get()<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/queues.py”, line 341, in get<br>
with self._rlock:<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/synchronize.py”, line 96, in <strong>enter</strong><br>
return self._semlock.<strong>enter</strong>()<br>
KeyboardInterrupt</p>

Gitoa: <p>KeyboardInterrupt<br>
Traceback (most recent call last):<br>
File “train.py”, line 143, in <br>
Process Process-69:<br>
Traceback (most recent call last):<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/process.py”, line 249, in _bootstrap<br>
self.run()<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/process.py”, line 93, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “/home/gitoo/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 50, in _worker_loop<br>
r = index_queue.get()<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/queues.py”, line 341, in get<br>
with self._rlock:<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/synchronize.py”, line 96, in <strong>enter</strong><br>
return self._semlock.<strong>enter</strong>()<br>
KeyboardInterrupt<br>
main(args)<br>
File “train.py”, line 70, in main<br>
for i_batch, sample_batched in enumerate(dataloader):<br>
File “/home/gitoo/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 275, in <strong>next</strong><br>
idx, batch = self._get_batch()<br>
File “/home/gitoo/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 254, in _get_batch<br>
return self.data_queue.get()<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/queues.py”, line 342, in get<br>
res = self._reader.recv_bytes()<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/connection.py”, line 216, in recv_bytes<br>
buf = self._recv_bytes(maxlength)<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/connection.py”, line 407, in _recv_bytes<br>
buf = self._recv(4)<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/connection.py”, line 379, in _recv<br>
chunk = read(handle, remaining)<br>
KeyboardInterrupt<br>
Error in atexit._run_exitfuncs:<br>
Traceback (most recent call last):<br>
File “/home/gitoo/anaconda3/lib/python3.6/multiprocessing/popen_fork.py”, line 29, in poll<br>
pid, sts = os.waitpid(self.pid, flag)<br>
File “/home/gitoo/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 175, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 15761) exited unexpectedly with exit code 1.</p>

luxuan_li: <p>Have you solved the problem? I’ve met the almost same one.</p>

maralm: <p>I have the same problem. Any solution?</p>