How to fix RuntimeError: CUDA out of memory

zyh3826: <p>I got this error when the program running 3 batches, sometimes 33 batches, even I set the <code>batch_size=1</code>. I read some other topics but still don’t know how to fix it, any one can help me? thanks a lot.<br>
my model is something like this:</p>
<pre><code class="lang-auto">def forward(self, input_id: torch.LongTensor,
                token_type_id: torch.LongTensor,
                attention_mask: torch.LongTensor) -&gt; torch.FloatTensor:
        print("mem free: {} Mb".format(get_gpu_memory(self.cuda_id)))
        out = self.bert(input_ids=input_id,
                        token_type_ids=token_type_id,
                        attention_mask=attention_mask) # huggingface's transformers
        if self.use_bert_predictions:
            out = out[0]
        else:
            out = out[1]
        out = self.encoder(out.unsqueeze(0))  # it's a LSTM
        out = out[:, -1, :]
        out = self.fc1(out)
        out = F.relu(out)
        out = self.fc2(out)
        return out
</code></pre>
<p>my trainer is like this:</p>
<pre><code class="lang-auto">for step, trains in enumerate(train_iter):
            encodes, label = trains
            input_ids = torch.stack([item['input_ids']
                                     for item in encodes[0]]).squeeze(1)
            token_type_ids = torch.stack(
                [item['token_type_ids'] for item in encodes[0]]).squeeze(1)
            attention_masks = torch.stack(
                [item['attention_mask'] for item in encodes[0]]).squeeze(1)
            # outputs = []
            logit = model(input_ids.cuda(config.Commom.cuda_id),
                          token_type_ids.cuda(config.Commom.cuda_id),
                          attention_masks.cuda(config.Commom.cuda_id))
            loss = loss_fct(logit, label.cuda(config.Commom.cuda_id))
            # outputs.append(logit.cpu().tolist())
            loss /= config.Train.grad_accumulate_step
            loss.backward()
            if (step + 1) % config.Train.grad_accumulate_step == 0:
                optimizer.step()
                optimizer.zero_grad()
           .....
</code></pre>
<p>and my error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2e32b37ab52c4b63852deda90077fa7cfd6400b.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2e32b37ab52c4b63852deda90077fa7cfd6400b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2e32b37ab52c4b63852deda90077fa7cfd6400b.png" alt="image" data-base62-sha1="u5AYuKMJNiZnVeNqtpeotosPwWn" width="690" height="281" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1453×593 35.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
before the error, the GPU still have 1527Mb free, but can’t allocate 28.00 MiB， don’t know why</p>

ptrblck: <p>If this error seems to be raised “randomly”, this might point to e.g. a specifically large input batch.<br>
If you are dealing with a variable sequence length, you might want to truncate the samples to a fixed size.<br>
Also make sure you are not storing any tensors, which are still attached to the computation graph, during the training.</p>

zyh3826: <p>Thanks ptrblck. In my machine, it’s always 3 batches, but in another machine that has the same hardware, it’s 33 batches. Today, I change the model.py and then turns to 40 batches in my machine. I haven’t store any tensors</p>
<pre><code class="lang-auto">def forward(self, input_id: torch.LongTensor,
                token_type_id: torch.LongTensor,
                attention_mask: torch.LongTensor,
                length: Tuple[int]) -&gt; torch.FloatTensor:
        if self.use_bert_predictions:
            out, _ = self.bert(input_ids=input_id,
                               token_type_ids=token_type_id,
                               attention_mask=attention_mask)
        else:
            _, out = self.bert(input_ids=input_id,
                               token_type_ids=token_type_id,
                               attention_mask=attention_mask)
        del _
        out = out.split_with_sizes(length)
        seq_lengths = torch.LongTensor([x for x in map(len, out)])
        out = torch.nn.utils.rnn.pad_sequence(
                            out, batch_first=True)
        out = torch.nn.utils.rnn.pack_padded_sequence(
                        out, seq_lengths.cpu().numpy(), batch_first=True, enforce_sorted=False)
        out, _ = self.encoder(out)
        del _
        del seq_lengths
        out, index = torch.nn.utils.rnn.pad_packed_sequence(out, batch_first=True)
        assert len(out) == len(index)
        temp = []
        for i in range(len(out)):
            temp.append(out[i][index[i]-1])
        out = torch.stack(temp)
        del temp
        gc.collect()
        out = self.fc1(out)
        out = F.relu(out)
        out = self.fc2(out)
        return out
</code></pre>

zyh3826: <p>I fix the sequence with 200 tokens, at another mission, use the same GPU, it’s well. My GPU is RTX 2080Ti, even I use DDP with 2 or 3 RTX 2080Ti, I also get the same error.</p>

zyh3826: <p>Hi ptrblck, I have another question. The size of data that feed in my model may be different at each batch, can this cause the problem above? Thank you very much</p>

ptrblck: <p>Yes, this might cause a memory spike and thus raise the out of memory issue, so try to make sure to keep the input shapes at a “reasonable” value.</p>