Mask RCNN predicting no BB during training

username: subramanian
cooked: <p>I am training biomedical images to detect a trace using Masked RCNN. However, during training, for some batch of images, the MRCNN does not output any bounding box proposal regions and this is causing the loss to be Nan. Since I am training on GPU, the training stops in between with below error,<br>
" RuntimeError: CUDA error: an illegal memory access was encountered</p>
<p>CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect."</p>
<p>When I traced back the error, I could see the loss values to be Nan which in turn is because there are no bounding boxes predicted by the MRCNN.</p>
<p>Would be great if someone can help me understand this error better and how to fix it.</p>

username: ptrblck
cooked: <p>Could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1</code> and check the stacktrace to see which operation fails?<br>
I’m currently unsure if the illegal memory access is created by using invalid values (e.g. NaNs) in a method which is not checking the input tensors for these values.</p>

username: subramanian
cooked: <p>I re ran the code with CUDA_LAUNCH_BLOCKING=1 and I get below stacktrace.</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 27, in main<br>
return train(config)<br>
File “/home/haicu/harshavardhan.subramanian/AIS_DL/AIS_DL/src/ml_pipeline_template/training_pipeline.py”, line 87, in train<br>
trainer.fit(model=model, datamodule=datamodule)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 741, in fit<br>
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 685, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 777, in _fit_impl<br>
self._run(model, ckpt_path=ckpt_path)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1199, in _run<br>
self._dispatch()<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1279, in _dispatch<br>
self.training_type_plugin.start_training(self)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py”, line 202, in start_training<br>
self._results = trainer.run_stage()<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1289, in run_stage<br>
return self._run_train()<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1311, in _run_train<br>
self._run_sanity_check(self.lightning_module)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1375, in _run_sanity_check<br>
self._evaluation_loop.run()<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/loops/base.py”, line 145, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py”, line 110, in advance<br>
dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/loops/base.py”, line 145, in run<br>
self.advance(*args, **kwargs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py”, line 122, in advance<br>
output = self._evaluation_step(batch, batch_idx, dataloader_idx)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py”, line 217, in _evaluation_step<br>
output = self.trainer.accelerator.validation_step(step_kwargs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py”, line 239, in validation_step<br>
return self.training_type_plugin.validation_step(*step_kwargs.values())<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py”, line 219, in validation_step<br>
return self.model.validation_step(*args, **kwargs)<br>
File “/home/haicu/harshavardhan.subramanian/AIS_DL/AIS_DL/src/ml_pipeline_template/models/AIS_UNetModule.py”, line 106, in validation_step<br>
loss, loss_mask, loss_box_reg, loss_classifier = self.step(batch)<br>
File “/home/haicu/harshavardhan.subramanian/AIS_DL/AIS_DL/src/ml_pipeline_template/models/AIS_UNetModule.py”, line 55, in step<br>
loss_dict = self.forward(images, targets)<br>
File “/home/haicu/harshavardhan.subramanian/AIS_DL/AIS_DL/src/ml_pipeline_template/models/AIS_UNetModule.py”, line 40, in forward<br>
return <a href="http://self.net" rel="noopener nofollow ugc">self.net</a>(x,y)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/haicu/harshavardhan.subramanian/AIS_DL/AIS_DL/src/ml_pipeline_template/models/components/MRCNN.py”, line 51, in forward<br>
return self.model(x,y)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py”, line 99, in forward<br>
detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py”, line 803, in forward<br>
rcnn_loss_mask = maskrcnn_loss(mask_logits, mask_proposals, gt_masks, gt_labels, pos_matched_idxs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py”, line 113, in maskrcnn_loss<br>
project_masks_on_boxes(m, p, i, discretization_size) for m, p, i in zip(gt_masks, proposals, mask_matched_idxs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py”, line 113, in <br>
project_masks_on_boxes(m, p, i, discretization_size) for m, p, i in zip(gt_masks, proposals, mask_matched_idxs)<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py”, line 95, in project_masks_on_boxes<br>
return roi_align(gt_masks, rois, (M, M), 1.0)[:, 0]<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torchvision/ops/roi_align.py”, line 62, in roi_align<br>
input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned<br>
RuntimeError: CUDA error: an illegal memory access was encountered</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="subramanian" data-post="3" data-topic="160288">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/85f322/48.png" class="avatar"> subramanian:</div>
<blockquote>
<p>return roi_align(gt_masks, rois, (M, M), 1.0)[:, 0]<br>
File “/home/haicu/harshavardhan.subramanian/miniconda3/envs/ml_template_env/lib/python3.7/site-packages/torchvision/ops/roi_align.py”, line 62, in roi_align<br>
input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned<br>
RuntimeError: CUDA error: an illegal memory access was encountered</p>
</blockquote>
</aside>
<p>It seems the <code>roi_align</code> call is running into the memory violation. Could you check the shapes and values of all tensors to see if some of them might be invalid and if the layer is missing checks?</p>

username: subramanian
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="160288">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CUDA_LAUNCH_BLOCKING=1</p>
</blockquote>
</aside>
<p>I checked the input values and shape of all the tensors at each step. But the roi values outputs empty list for certain images in a batch and when the roi_align() method tries to  return torch.ops.torchvision.roi_align(<br>
input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned<br>
) with empty list of rois then I get the above CUDA error.</p>

username: ptrblck
cooked: <p>Thanks for the update!<br>
Are you seeing the illegal memory access using a similar code like this?</p>
<pre><code class="lang-python">input = torch.randn(2, 3, 224, 224, device='cuda')
torchvision.ops.roi_align(input, [], output_size=(100, 100))
</code></pre>

username: subramanian
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="160288">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CUDA_LAUNCH_BLOCKING=1</p>
</blockquote>
</aside>
<p>No I am not seeing the error when I run this code. It is very weird that at some steps though the roi is an empty list, "torch.ops.torchvision.roi_align(<br>
input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned<br>
) " returns an empty list and I do not get any error.<br>
I am printing input, roi at every step and it is getting hard to debug where exactly the model is failing.<br>
But the error is same and stack trace is same.<br>
Another interesting thing is, I do not get the error if I use SGD optimizer, but I get the loss as Nan at the end of each epoch and I get this prompt in my slurm output file, “Trainer was signaled to stop but required minimum epochs (200) or minimum steps (None) has not been met. Training will continue…%1B[0m”.</p>

username: ptrblck
cooked: <p>I guess the “trainer” output is coming from Lightning and don’t know how to interpret it.<br>
Maybe this module is getting some invalid values and then misses to check them before executing the kernel. I’ll try to rerun it with different invalid inputs and see how <code>roi_align</code> would behave.</p>