RuntimeError: CUDA out of memory ONLY for validation but NOT for training

Nathane_Berrebi: <p>Hello, I’m trying to train a pre-trained Faster R-CNN in Google Collab Pro. My function for the training is OK and everything works very well.</p>
<p>But I tried to launch a validation step using the exact same function except that i don’t use the backpropagation when I computed the loss.<br>
But when I skip the backpropagation step, I get an error of CUDA MEMORY, whilst when I apply the backpropagation (during the training) I don’t have any error of CUDA MEMORY even without using the   <code>torch.cuda.empty_cache()</code>. This is my code :</p>
<pre><code class="lang-auto">def train_one_epoch(model, optimizer, data_loader, device, validation):    
    model.train()
    for i, values in enumerate(data_loader):
        images, targets = values
        images = list(image.to(device) for image in images)
        targets = [{'boxes' : t['boxes'] , 'labels' : t['labels'].to(device)} for t in targets ]
        loss_dict = model(images, targets)
        losses = sum(loss for loss in loss_dict.values())
        optimizer.zero_grad()

        if validation == False :               
            losses.backward()
            optimizer.step()

        torch.cuda.empty_cache()
</code></pre>
<p>The only thing that changes between training and validation is that during training I set validation = False, so I enter in the if condition. And during validation i set the validation variable = True, so I don’t ente rin the if condition.</p>
<p>So, setting validation to False gave me this error :</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 15.90 GiB total capacity; 14.53 GiB already allocated; 25.75 MiB free; 14.86 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>Does someone have an idea to fix it please ?</p>

JosephC: <p>I think if you wanted to disable the gradients for validation you might want to do <code>torch.no_grad()</code> for one of the steps rather than just conditionally skipping the calls to losses.backwards() and optimizer.step.  Or, alternatively, set <code>model.eval()</code> before running the evaluation step and then set <code>model.train()</code> when you’re done.</p>

Nathane_Berrebi: <p>Hello <a class="mention" href="/u/josephc">@JosephC</a> thanks for your answer ! But if I run <code>model.eval()</code> I will not be allowed to put the target in the command <code>model(images, targets)</code>.</p>
<p>Also If you know how to do it, I would like during the training mode to get the prediction and the loss. But when I do <code>model(images, targets)</code> I only get the loss.</p>
<p>Thanks for your help !</p>

mMagmer: <p>Because torch auto grad wants to (and can)  compute grad on multiple pass,<br>
as <a class="mention" href="/u/josephc">@JosephC</a>  said, you should always use</p>
<pre><code class="lang-auto">with torch.no_grad():
metric = evaluate(model)
</code></pre>
<p>In forward pass because you call backward, torch removes all data required for computing grad after computing it.</p>

JosephC: <p>Without seeing more of the code it’s a little hard to say for sure, but hearing that model is taking both the inputs and the targets is a little unusual.  Normally, we want the model to only take the inputs.  The loss function should be something separate.</p>
<p>This is some <strong>pseudo-code</strong> with a little more of the best practice.  Note that the loss_fn is separate from the model.  The reason for this is we don’t really care about keeping the loss function when storing the model – we just want to use it.</p>
<p>Example:</p>
<pre><code class="lang-auto">def train_one_epoch(model, optimizer, data_loader, device, validation):    
    model.train()
    loss_fn = torch.nn.BCELoss()
    for i, values in enumerate(data_loader):
        images, targets = values
        images = list(image.to(device) for image in images)
        targets = [{'boxes' : t['boxes'] , 'labels' : t['labels'].to(device)} for t in targets ]

        if validation:
            torch.cuda.empty_cache()
            model.eval()  # Optional, or use the torch.no_grad()
            with torch.no_grad():
                predictions = model(images)
                loss_dict = loss_fn(images, targets)
                losses = sum(loss for loss in loss_dict.values())
                # Do something with losses.
            model.train()
        else:
            optimizer.zero_grad()
            predictions = model(images)
            loss_dict = loss_fn(images, targets)
            losses = sum(loss for loss in loss_dict.values())
            losses.backward()
            optimizer.step()
</code></pre>
<p>That being said, you may be able to work around this issue with only a few code changes by doing any of the following:</p>
<ul>
<li>Reduce your batch size to something smaller.</li>
<li>Perform a validation on a smaller set of items.</li>
</ul>
<p>Those would be band-aids, but they’re something to try.  I’d do the no_grad thing first.</p>

Nathane_Berrebi: <p>Thank you very much <a class="mention" href="/u/josephc">@JosephC</a> ! But when I runned this code in the training mode (<code>validation = False</code>), I got this error :</p>
<pre><code class="lang-auto">2 frames
&lt;ipython-input-19-3c77990fe1dc&gt; in train_one_epoch_forum_pytorch_(model, optimizer, data_loader, device, validation)
     20         else:
     21             optimizer.zero_grad()
---&gt; 22             predictions = model(images)
     23             print(predictions)
     24             loss_dict = loss_fn(images, targets)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
     55         """
     56         if self.training and targets is None:
---&gt; 57             raise ValueError("In training mode, targets should be passed")
     58         if self.training:
     59             assert targets is not None

ValueError: In training mode, targets should be passed
</code></pre>
<p>How should I do to extract the prediction in the train mode ?</p>

JosephC: <p><a class="mention" href="/u/nathane_berrebi">@Nathane_Berrebi</a>, looking at the error:</p>
<pre><code class="lang-auto">/usr/local/lib/python3.7/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
     55         """
     56         if self.training and targets is None:
---&gt; 57             raise ValueError("In training mode, targets should be passed")
     58         if self.training:
     59             assert targets is not None

ValueError: In training mode, targets should be passed
</code></pre>
<p>If self.training is True, then targets cannot be None. We want targets to be None, so self.training must be set to false.  To do this, we must call <code>model.eval()</code> at the start of the validation section.</p>

my3bikaht: <p>Just an opinion here. I prefer to code validation loop aside from training loop. Maybe resulting code isn’t as compact as combined loop, but it is much easier to see what goes where.</p>