CUDA out of memory only using model.eval() even with empty cache

Nathb26: <p>Hello,</p>
<p>I’m having the same issue since 2 weeks. I’m callling a faster r cnn model, I trained a model with my GPU (only one image by batches because of “memory”). Then I saved the model and now I can call it. But when I only load the model and I’m trying to take a look on the predictions, I use model.eval() and then model([image]) … But it works for 2 images but with 3 images I have a cuda error. But I tried a lot of things : I call torch.cuda.empty_cache() after every prediction, but it didn’t work. Then I add  gc.collect() but still doesn’t work. I show you my error :</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-4-cb3f5f560ce4&gt; in &lt;module&gt;
----&gt; 1 list_images, list_predictions = inference_gpu([27612, 45125, 31654 ], path_to_video = path_to_video_lab7, draw_bbox = True, threshold = 0.9)

&lt;ipython-input-1-37a3b7b4f6bf&gt; in inference_gpu(list_path_inf, path_to_video, draw_bbox, threshold)
     72             model.cuda()
     73             model.eval()
---&gt; 74             out = model([image])
     75             list_images.append(image)
     76             list_predictions.append(out)

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/lib/python3.8/site-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
     94         if isinstance(features, torch.Tensor):
     95             features = OrderedDict([('0', features)])
---&gt; 96         proposals, proposal_losses = self.rpn(images, features, targets)
     97         detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
     98         detections = self.transform.postprocess(detections, images.image_sizes, original_image_sizes)

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/lib/python3.8/site-packages/torchvision/models/detection/rpn.py in forward(self, images, features, targets)
    341         # RPN uses all feature maps that are available
    342         features = list(features.values())
--&gt; 343         objectness, pred_bbox_deltas = self.head(features)
    344         anchors = self.anchor_generator(images, features)
    345 

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/anaconda3/lib/python3.8/site-packages/torchvision/models/detection/rpn.py in forward(self, x)
     55         bbox_reg = []
     56         for feature in x:
---&gt; 57             t = F.relu(self.conv(feature))
     58             logits.append(self.cls_logits(t))
     59             bbox_reg.append(self.bbox_pred(t))

~/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py in relu(input, inplace)
   1296         result = torch.relu_(input)
   1297     else:
-&gt; 1298         result = torch.relu(input)
   1299     return result
   1300 

RuntimeError: CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 5.81 GiB total capacity; 4.44 GiB already allocated; 32.38 MiB free; 4.47 GiB reserved in total by PyTorch)

</code></pre>
<p>My image’s shape is something like (1060,1020, 3).</p>
<p>Is there a solution to apply inference with a gpu model ?</p>
<p>Thanks !</p>