RuntimeError: CUDA out of memory after many epochs

antran96: <p>My model size is not so big (2M parameters). However, it throws the following error after several epochs (sometimes it could train for &gt;20 epochs and throw this error).</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 31.75 GiB total capacity; 16.29 GiB already allocated; 1.69 MiB free; 14.46 GiB cached)</code></p>
<p>Here is my model:</p>
<pre><code class="lang-auto">class WaveNetEncoder9(Module):

    def __init__(self,
                 in_channels: int,
                 out_waveblock: List,
                 kernel_size: int,
                 dilation_rates: List,
                 inner_kernel_size: int,
                 inner_padding: int,
                 last_dim: int,
                 mode: str,
                 ) \
            -&gt; None:
        super(WaveNetEncoder9, self).__init__()

        assert len(dilation_rates) == len(out_waveblock), "length of dilation rates must match out channels"
        self.in_channels: int = in_channels
        self.out_waveblock = out_waveblock
        self.kernel_size = kernel_size
        self.dilation_rates = dilation_rates

        self.wave_blocks = ModuleList() 
        self.bn_relu = ModuleList()
        self.wave_blocks.append(WaveBlock(
            in_channels=self.in_channels,
            out_channels=self.out_waveblock[0],
            dilation_rates=self.dilation_rates[0],
            kernel_size=self.kernel_size
        ))
        
        for i in range(1,len(self.dilation_rates)):
            self.wave_blocks.append(
                WaveBlock(
                    in_channels=self.out_waveblock[i-1],
                    out_channels=self.out_waveblock[i],
                    dilation_rates=self.dilation_rates[i],
                    kernel_size=self.kernel_size
                )
            )
        
        for i in range(len(self.dilation_rates)):
            self.bn_relu.append(
                Sequential(
                    BatchNorm1d(self.out_waveblock[i]),
                    ReLU(),
                )
            )

        if mode == "max":
            self.dnn = DepthWiseSeparableDNN(  # this is 3 blocks of conv2d+relu+batchnorm2d and avgpool2d layers
                cnn_channels=out_waveblock[-1],
                cnn_dropout=0.2,
                inner_kernel_size=inner_kernel_size,
                inner_padding=inner_padding
            )
        elif mode == 'avg':
            self.dnn = DepthWiseSeparableDNN2(
                cnn_channels=out_waveblock[-1],
                cnn_dropout=0.2,
                inner_kernel_size=inner_kernel_size,
                inner_padding=inner_padding
            )
        else:
            raise Exception('Incorrect argument')
        
        self.point_wise: Module = Conv2d(
            in_channels=2, out_channels=1,
            kernel_size=inner_kernel_size, stride=1,
            padding=1, dilation=1,
            groups=1, bias=True, padding_mode='zeros')

        self.fc_audioset = Linear(last_dim,last_dim, bias=True)

        
    def forward(self,
                x: Tensor) \
            -&gt; Tensor:
        """Forward pass of the encoder.

        :param x: Input to the encoder.
        :type x: torch.Tensor
        :return: Output of the encoder.
        :rtype: torch.Tensor
        """
        bs, time_step, mel_bins = x.size()
        x1 = x.permute(0,2,1)
        x2 = x

        # WaveNet
        for i in range(len(self.wave_blocks)):
            x1 = self.wave_blocks[i](x1)
            x1 = self.bn_relu[i](x1) 
        
        x1 = x1.unsqueeze(1).transpose(2,3)
        # CNN
        x2 = self.dnn(x2)
        x2 = x2.transpose(1,3) # (bs, 1, time_steps, 128)
        x2 = x2.reshape(x2.shape[0],1,time_step,-1)
        x = torch.cat((x2, x1), dim=1)
        x = self.point_wise(x) 

        x = x.reshape(bs,time_step,-1)
        x = F.relu_(self.fc_audioset(x))
        return x

# EOF
</code></pre>
<p>I also empty cached memory at the begining of each iteration by</p>
<pre><code class="lang-auto">torch.cuda.empty_cache()
gc.collect()
</code></pre>
<p>use <code>torch.no_grad()</code> in evaluation and detach loss <code>loss.item()</code>.</p>

ptrblck: <p>Are you seeing an increase in memory usage during the training?<br>
If not, I would recommend to lower the batch size as it seems your code runs close to the GPU memory limit.</p>

antran96: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="96563">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>.</p>
</blockquote>
</aside>
<p>When I run <code>torch.cuda.memory_cached()</code> after the end of each epoch, my memory cached is unchanged at 3.04GB (like every digit is the same), which is weird to me but I still get CUDA out of memory and the cached memory is &gt;10GB?</p>

ptrblck: <p>Your script might be already hitting OOM issues and would call <code>empty_cache</code> internally.<br>
You can check it via <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.memory_stats"><code>torch.cuda.memory_stats()</code></a>.<br>
If you see that OOMs were detected, lower the batch size as suggested.</p>

antran96: <p>Yes, seems like decreasing the batch size resolve the issue. But is there a way to train with batch size unchanged? I feel like the memory is not emptied and it keeps aggregating in every epoch</p>

banikr: <p>Have you found any other solution without changing the batch size? I have encountered the same issue after 193 epochs.</p>

AndreyBocharnikov: <p>Same, I encountered cuda out of memory after 342 epochs and <code>drop_last</code> in <code>Dataloader</code> is set to <code>False</code>, so all samples are used during a single epoch, and what makes it even more strange is that this error often occurs at the first epoch or on the 9th epoch, but some times with the same data and architecture it can train hundreds of epochs. Any ideas what can cause it and how can it be fixed? <span class="mention">@ptrblck_de</span> Maybe you know why it might be happening? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

SaharNasser: <p>I am facing the same problem, were you able to solve it?</p>