Evaluation on validation dataset stuck

i_m_a_q: <p>I am working on <strong><a href="https://github.com/facebookresearch/mmf/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/mmf: A modular framework for vision &amp; language multimodal research from Facebook AI Research (FAIR)</a></strong> and using grid features from resnet-50 on coco dataset</p>
<p><strong>hardware details</strong><br>
2 gpus each with 11 gb memory<br>
16 gb RAM in total</p>
<p><strong>other details</strong><br>
gloo backend for training on 2 gpus<br>
8 batch size<br>
num_workers=2</p>
<p><strong>I am training MOVIE_MCAN model with the following command</strong><br>
CUDA_VISIBLE_DEVICES=0,1 mmf_run config=projects/movie_mcan/configs/vqa2/defaults.yaml \model=movie_mcan \dataset=vqa2 \run_type=train \training.num_workers=2</p>
<p><strong>it starts training:</strong></p>
<p>2021-03-12T15:46:12 | mmf.utils.general: Total Parameters: 254918110. Trained Parameters: 254918110<br>
2021-03-12T15:46:12 | mmf.trainers.core.training_loop: Starting training…<br>
2021-03-12T15:52:30 | mmf.trainers.callbacks.logistics: progress: 5100/236000, train/vqa2/triple_logit_bce: 20.1549, train/vqa2/triple_logit_bce/avg: 20.1549, train/total_loss: 20.1549, train/total_loss/avg: 20.1549, max mem: 9993.0, experiment: run, epoch: 1, num_updates: 100, iterations: 100, max_updates: 236000, lr: 0.00001, ups: 0.27, <strong>time: 06m 17s</strong> 344ms, time_since_start: 06m 51s 100ms, eta: 1863h 06m 21s 108ms<br>
2021-03-12T15:57:56 | mmf.trainers.callbacks.logistics: progress: 5200/236000, train/vqa2/triple_logit_bce: 20.1549, train/vqa2/triple_logit_bce/avg: 21.8194, train/total_loss: 20.1549, train/total_loss/avg: 21.8194, max mem: 9993.0, experiment: run, epoch: 2, num_updates: 200, iterations: 200, max_updates: 236000, lr: 0.00001, ups: 0.31, <strong>time: 05m 26s</strong> 277ms, time_since_start: 12m 17s 392ms, eta: 1610h 16m 12s 756ms</p>
<p><strong>after every 1000 iterations it evaluates the model on Validation_set of coco dataset</strong></p>
<p><strong>2021-03-12T16:41:25 | mmf.trainers.callbacks.checkpoint: Checkpoint time. Saving a checkpoint.</strong><br>
2021-03-12T16:42:04 | mmf.trainers.callbacks.logistics: progress: 1000/236000, train/vqa2/triple_logit_bce: 20.1549, train/vqa2/triple_logit_bce/avg: 20.0570, train/total_loss: 20.1549, train/total_loss/avg: 20.0570, max mem: 9993.0, experiment: run, epoch: 2, num_updates: 1000, iterations: 1000, max_updates: 236000, lr: 0.00001, ups: 0.28, <strong>time: 05m 51s</strong> 002ms, time_since_start: 56m 23s 258ms, eta: 1726h 17m 24s 444ms<br>
<strong>2021-03-12T16:42:05 | mmf.trainers.core.training_loop: Evaluation time. Running on full validation set…</strong></p>
<p><strong>and here the model seems to be stuck. I have waited for 10 hours but no proceeding. Upon interruption it gives me following details</strong></p>
<p>^CTraceback (most recent call last):<br>
File “/home/anaconda3/envs/mmf/bin/mmf_run”, line 33, in<br>
sys.exit(load_entry_point(‘mmf’, ‘console_scripts’, ‘mmf_run’)())<br>
File “/home/new-mmf/mmf-master/mmf_cli/run.py”, line 118, in run<br>
nprocs=config.distributed.world_size,<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 200, in spawn<br>
return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 158, in start_processes<br>
while not context.join():<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 78, in join<br>
timeout=timeout,<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/multiprocessing/connection.py”, line 911, in wait<br>
ready = selector.select(timeout)<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/selectors.py”, line 376, in select<br>
fd_event_list = self._poll.poll(timeout)<br>
KeyboardInterrupt<br>
^CError in atexit._run_exitfuncs:<br>
Traceback (most recent call last):<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/multiprocessing/popen_fork.py”, line 28, in poll<br>
Exception ignored in: &lt;bound method _MultiProcessingDataLoaderIter.del of &lt;torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7f387d4815f8&gt;&gt;<br>
Traceback (most recent call last):<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 1101, in del<br>
pid, sts = os.waitpid(self.pid, flag)<br>
KeyboardInterrupt<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 1075, in _shutdown_workers<br>
w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/multiprocessing/process.py”, line 124, in join<br>
res = self._popen.wait(timeout)<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/multiprocessing/popen_fork.py”, line 47, in wait<br>
if not wait([self.sentinel], timeout):<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/multiprocessing/connection.py”, line 911, in wait<br>
ready = selector.select(timeout)<br>
File “/home/anaconda3/envs/mmf/lib/python3.6/selectors.py”, line 376, in select<br>
fd_event_list = self._poll.poll(timeout)<br>
KeyboardInterrupt:</p>
<p><strong>Can you tell me what is causing this to be stuck at this point. I have tried with python 3.6,7,8 but nothing worked. Also i checked the gpu utilization through “nvidia-smi” and mostly the gpu utilization is at 0% and other times it shows some number like 80% or 90%. Could you please help me regarding what is going wrong here.</strong></p>
<p><strong>python -m torch.utils.collect_env</strong><br>
Collecting environment information…<br>
PyTorch version: 1.6.0+cu101<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.1</p>
<p>OS: Ubuntu 16.04.7 LTS<br>
GCC version: (Ubuntu 5.5.0-12ubuntu1~16.04) 5.5.0 20171010<br>
CMake version: version 3.18.2</p>
<p>Python version: 3.6<br>
Is CUDA available: Yes<br>
CUDA runtime version: 10.1.243<br>
GPU models and configuration:<br>
GPU 0: Tesla K80<br>
GPU 1: Tesla K80</p>
<p>Nvidia driver version: 430.64<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.2<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.2<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.2<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.2<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.2<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.2<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.2<br>
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.2<br>
[pip3] pytorch-lightning==1.1.6<br>
[pip3] torch==1.6.0+cu101<br>
[pip3] torchtext==0.5.0<br>
[pip3] torchvision==0.7.0+cu101<br>
[conda] blas 1.0 mkl<br>
[conda] cudatoolkit 10.0.130 0<br>
[conda] mkl 2020.2 256<br>
[conda] mkl-service 2.3.0 py36he8ac12f_0<br>
[conda] mkl_fft 1.2.0 py36h23d657b_0<br>
[conda] mkl_random 1.1.1 py36h0573a6f_0<br>
[conda] numpy 1.19.2 py36h54aff64_0<br>
[conda] numpy-base 1.19.2 py36hfa32c7d_0<br>
[conda] pytorch-lightning 1.1.6 pypi_0 pypi<br>
[conda] torch 1.4.0+cu100 pypi_0 pypi<br>
[conda] torchtext 0.5.0 pypi_0 pypi<br>
[conda] torchvision 0.7.0+cu101 pypi_0 pypi</p>