Help needed - Cuda out of memory

mark_eu: <p>Hi,</p>
<p>I need help. I have 32Gb of GPU and facing the problem of “Cuda out of memory.”<br>
I tried to improve my train routine:</p>
<pre><code class="lang-auto">import gc
def train(model, iterator, optimizer, criterion):
    
    epoch_loss = 0.0
    epoch_acc = 0.0
    
    model.train()
    
    for batch in iterator:
        
        optimizer.zero_grad()
        
        predictions = model(batch.posts) 
        
        loss = criterion(predictions, batch.type_numeric.long().to(device))
        
        acc = categorical_accuracy(predictions, batch.type_numeric.long().to(device))
        
        loss.backward()
        
        optimizer.step()
        
        epoch_loss += loss.item()
        epoch_acc += acc.item()
        #####################
        #the following part really slows the code execution
        del predictions
        del loss
        del acc
        del batch
        gc.collect()
        torch.cuda.empty_cache()
        #####################
    return epoch_loss / len(iterator), epoch_acc / len(iterator)
</code></pre>
<p>and also, I lowered the batch size to have small values:</p>
<pre><code class="lang-auto">train_batch_size      =   4
valid_batch_size      =   4
test_batch_size       =   4
</code></pre>
<p>but still, I get the error Cuda out of memory:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4a31ea8449487adb299b17a6b54dd4a7be6bd2f.jpeg" alt="error2" data-base62-sha1="nurMPBcwywuxkgrLcRVQOqtDPeD" width="635" height="266"></p>
<p>My task is an NLP multiclass classification task on a dataset that has, after preprocessing, 1 Gb. I tried to reduce MAX_VOCAB_SIZE to 300. I use the CNN model with the parameters:</p>
<pre><code class="lang-auto">INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 300
N_FILTERS = 100
FILTER_SIZES = [2,3,4]
OUTPUT_DIM = len(LABEL.vocab)
DROPOUT = 0.2
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT, PAD_IDX)

optimizer = optim.Adam(model.parameters())

criterion1 = nn.CrossEntropyLoss()

model = model.to(device)
criterion1 = criterion1.to(device)
</code></pre>
<p>Could you please help me with this - my deadline is approaching, and I have no other ideas on how to solve this issue.</p>
<p>Thank you</p>

eqy: <p>It’s a bit difficult to pinpoint this without more context on the model (e.g., how many parameters, layers, etc?) and is the OOM happening on the first iteration or after many iterations due to gradually increasing memory consumption?</p>
<p>If it is happening in the first few iterations you might want to check if the model size itself needs to be reduced, otherwise it might be worthwhile to see which operation is gradually increasing the total memory consumed over many iterations (you can check this by removing operations until the memory used stops increasing).</p>

mark_eu: <p><a class="mention" href="/u/eqy">@eqy</a> Thank you!<br>
My model has the following structure:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F

class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, 
                 dropout, pad_idx):
        
        super().__init__()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, 
                                              out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):
        
        text = text.permute(1, 0)
                
        embedded = self.embedding(text)
                
        embedded = embedded.unsqueeze(1)
        
        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
            
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]
  
        cat = self.dropout(torch.cat(pooled, dim = 1)) 
        return self.fc(cat)
</code></pre>
<p>The model has 365,716 trainable parameters (in a ‘reduced’ version of MAX_VOCAB_SIZE = 300 instead of 25000 and batch sizes of 4, instead of 256/64/64). The OOM happens on the first iteration.</p>
<p>Do you have any suggestions for solving this problem? I also hope that <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/alband">@albanD</a> could give some advice(s). The problem does not exist with the same model on the small dataset (70MB)<br>
Thank you</p>

mark_eu: <p>It seems to me that the problem appears in :<br>
<code>conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs</code><br>
but I have no idea how to solve it.</p>
<p>Also, for text vocabulary I use:</p>
<pre><code class="lang-auto">TEXT.build_vocab(train_ds, 
                 min_freq   =  3,
                 max_size   =  MAX_VOCAB_SIZE,
                 vectors    = 'fasttext.en.300d',
                 unk_init   =  torch.Tensor.normal_
                 )
</code></pre>

ptrblck: <p>You are currently storing all activation outputs of <code>self.convs</code> in the <code>conved</code> list and are then applying the pooling on each of them.<br>
Could you combine these lines of code and see if the memory footprint would go down?</p>

mark_eu: <p>Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a>, for your advice.<br>
I changed the model into:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim,
                 dropout, pad_idx):

        super().__init__()

        self.embedding = nn.Embedding(vocab_size, embedding_dim, pad_idx)
        '''
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1,
                                              out_channels = n_filters,
                                              kernel_size = (fs, embedding_dim))
                                    for fs in filter_sizes
                                    ])
        '''
        self.conv_0 = nn.Conv2d(in_channels = 1,
                                out_channels = n_filters,
                                kernel_size = (filter_sizes[0], embedding_dim))

        self.conv_1 = nn.Conv2d(in_channels = 1,
                                out_channels = n_filters,
                                kernel_size = (filter_sizes[1], embedding_dim))

        self.conv_2 = nn.Conv2d(in_channels = 1,
                                out_channels = n_filters,
                                kernel_size = (filter_sizes[2], embedding_dim))
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, text):
        text = text.permute(1, 0) 
        embedded = self.embedding(text)
        embedded = embedded.unsqueeze(1)

        conved_0 = F.relu(self.conv_0(embedded).squeeze(3))
        conved_1 = F.relu(self.conv_1(embedded).squeeze(3))
        conved_2 = F.relu(self.conv_2(embedded).squeeze(3))
        '''
        conved_n = [batch size, n_filters, sent len - filter_sizes[n] + 1]
        '''
        pooled_0 = F.max_pool1d(conved_0, conved_0.shape[2]).squeeze(2)
        pooled_1 = F.max_pool1d(conved_1, conved_1.shape[2]).squeeze(2)
        pooled_2 = F.max_pool1d(conved_2, conved_2.shape[2]).squeeze(2)
        '''
        pooled_n = [batch size, n_filters]
       '''
        cat = self.dropout(torch.cat((pooled_0, pooled_1, pooled_2), dim = 1))


        return self.fc(cat)
</code></pre>
<p>However, I still get CUDA out of memory error. Do you have any other suggestions?<br>
Thank you</p>

ptrblck: <p>There wouldn’t be any difference in your code as you are still storing all intermediate activations.<br>
Try this:</p>
<pre><code class="lang-python">pooled_0 = F.max_pool1d(F.relu(self.conv_0(embedded).squeeze(3)))
</code></pre>

mark_eu: <p>Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a>, for additional help.<br>
My model now looks like this:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F

class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, 
                 dropout, pad_idx):
        
        super().__init__()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim, pad_idx)
        '''
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, 
                                              out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        '''
        self.conv_0 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[0], embedding_dim))
        
        self.conv_1 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[1], embedding_dim))
        
        self.conv_2 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[2], embedding_dim))
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):

        text = text.permute(1, 0)
        embedded = self.embedding(text)
                
        
        embedded = embedded.unsqueeze(1)
        
        
        conved_0 = F.relu(self.conv_0(embedded).squeeze(3))
        conved_1 = F.relu(self.conv_1(embedded).squeeze(3))
        conved_2 = F.relu(self.conv_2(embedded).squeeze(3))
            
        '''
        pooled_0 = F.max_pool1d(conved_0, conved_0.shape[2]).squeeze(2)
        pooled_1 = F.max_pool1d(conved_1, conved_1.shape[2]).squeeze(2)
        pooled_2 = F.max_pool1d(conved_2, conved_2.shape[2]).squeeze(2)
        '''
        pooled_0 = F.max_pool1d(F.relu(self.conv_0(embedded).squeeze(3)))
        pooled_1 = F.max_pool1d(F.relu(self.conv_1(embedded).squeeze(3)))
        pooled_2 = F.max_pool1d(F.relu(self.conv_2(embedded).squeeze(3)))
      
        
        cat = self.dropout(torch.cat((pooled_0, pooled_1, pooled_2), dim = 1))

            
        return self.fc(cat)

</code></pre>
<p>Now, I don’t know if I get Cuda out of memory since I get the error:</p>
<pre><code class="lang-auto">TypeError: _max_pool1d() missing 1 required positional argument: 'kernel_size'
</code></pre>
<p>Do you have a suggestion on how to solve it?</p>
<p>Thank you</p>

ptrblck: <p>The <code>F.max_pool1d</code> operations are missing the kernel size and you are also unnecessarily computing and storing <code>conved_X</code>.</p>

mark_eu: <p>Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I hope this could be that improvement:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, 
                 dropout, pad_idx):
        
        super().__init__()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim, pad_idx)
        '''
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, 
                                              out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        '''
        self.conv_0 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[0], embedding_dim))
        
        self.conv_1 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[1], embedding_dim))
        
        self.conv_2 = nn.Conv2d(in_channels = 1, 
                                out_channels = n_filters, 
                                kernel_size = (filter_sizes[2], embedding_dim))
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):

        text = text.permute(1, 0) 
        embedded = self.embedding(text)
               
        embedded = embedded.unsqueeze(1)


        conved_0 = F.relu(self.conv_0(embedded).squeeze(3))
        conved_1 = F.relu(self.conv_1(embedded).squeeze(3))
        conved_2 = F.relu(self.conv_2(embedded).squeeze(3))
            
        
        pooled_0 = F.max_pool1d(conved_0, conved_0.shape[2]).squeeze(2)
        pooled_1 = F.max_pool1d(conved_1, conved_1.shape[2]).squeeze(2)
        pooled_2 = F.max_pool1d(conved_2, conved_2.shape[2]).squeeze(2)

        
        cat = self.dropout(torch.cat((pooled_0, pooled_1, pooled_2), dim = 1))

            
        return self.fc(cat)

</code></pre>
<p>I again get the error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be759fa4347382271e7fc75e405ff5acc05a9587.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/be759fa4347382271e7fc75e405ff5acc05a9587" title="convolutions - error cuda"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/be759fa4347382271e7fc75e405ff5acc05a9587_2_690x454.jpeg" alt="convolutions - error cuda" data-base62-sha1="raSJ3L7DrtPdGEzDDLTGDUxLb03" width="690" height="454" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/be759fa4347382271e7fc75e405ff5acc05a9587_2_690x454.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/e/be759fa4347382271e7fc75e405ff5acc05a9587.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/e/be759fa4347382271e7fc75e405ff5acc05a9587.jpeg 2x" data-dominant-color="F3F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">convolutions - error cuda</span><span class="informations">1007×664 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I lowered batches to 1, but that didn’t help. <a class="mention" href="/u/ptrblck">@ptrblck</a>, do you have any idea what else I could do to solve this problem?</p>
<p>Thank you</p>

mark_eu: <p>Does anyone have any additional idea on how to solve this issue? <a class="mention" href="/u/ptrblck">@ptrblck</a>, do you have any further suggestions?</p>
<p>Thank you</p>