Memory error while training a variable sequence length LSTM

Venkatesh_Ragavan: <p>CUDA out of memory. Tried to allocate 17179869176.57 GiB (GPU 0; 15.90 GiB total capacity; 8.57 GiB already allocated; 6.67 GiB free; 8.58 GiB reserved in total by PyTorch)</p>
<p>I am working with a  text dataset with 50 to 60 data points. Each sequence has about 200K tokens on an average. The maximum length sequence has about 500K tokens.  GPU Memory is about 16 GB. Hence, it’s throwing a memory error. Any suggestions on how to circumvent this issue?</p>

ptrblck: <aside class="quote no-group" data-username="Venkatesh_Ragavan" data-post="1" data-topic="83539">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/venkatesh_ragavan/48/23260_2.png" class="avatar"> Venkatesh_Ragavan:</div>
<blockquote>
<p>17179869176.57 GiB</p>
</blockquote>
</aside>
<p>is quite large, so that it won’t fit on the device <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
Could you post the complete shape of your inputs?</p>
<p>If this memory footprint is expected and is not raised by a bug, then you would have to reduce the input shape and model significantly.</p>

Venkatesh_Ragavan: <p>Yeah <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self,vocab_len,input_size):
        super(Model,self).__init__()
        self.embed=nn.Embedding(num_embeddings=vocab_len,embedding_dim=128,)
        self.lstm=nn.LSTM(input_size=input_size,hidden_size=1024,num_layers=1)
        self.fc=nn.Linear(in_features=1024,out_features=1)
    def forward(self,x):
        x=self.embed(x)
        o,(ht,ct)=self.lstm(x)
        out=self.fc(ct)
        return nn.Sigmoid(out)
</code></pre>
<p>This is the model.<br>
The input shape is (2,560874)[bs,seq_len).<br>
Am I making a blinder?</p>

ptrblck: <p>The input tensor itself will just use ~4.2MB and the linear layer is also quite small.<br>
How large is <code>vocal_len</code> and is <code>input_size=560874</code>?</p>

Venkatesh_Ragavan: <p>Yes. The vocab_len is 513</p>

ptrblck: <p>Thanks for the information.<br>
It seems the <code>nn.LSTM</code> might use too much memory for your device, as it would need ~9GB.<br>
However, the error message is wrong and yields a wrong number. We’ve recently fixed a similar issue for convolutions, so I’ll check it with the latest master build.</p>

Venkatesh_Ragavan: <p>I figured it out. The text preprocessing was improper. Moving on to my next question.</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self,emb_sz,hidden_sz,num_c):
        super(Model,self).__init__()
        self.embed=nn.Embedding(num_embeddings=len(TEXT.vocab),embedding_dim=emb_sz,)
        self.lstm=nn.LSTM(input_size=emb_sz,hidden_size=hidden_sz,num_layers=1)
        self.fc=nn.Linear(in_features=hidden_sz,out_features=num_c)
    def forward(self,x):
        #print(x.size())
        x=x.permute(1,0)
        x=self.embed(x)
        #print(f'After embedding size {x.size()}')
        o,_=self.lstm(x)
        shape=o.size()
        o=o[:,-1,:]
        #o=o.view(shape[0],-1)
        #print(f'After LSTM size {o.size()}')
        out=self.fc(o)
        #print(f'After FC size {out.size()}')
        return F.sigmoid(out)
</code></pre>
<p>This my model.</p>
<pre><code class="lang-auto">def train(model,loss,opt,num_epochs,train_dl,valid_dl):
    model=model.cuda()
    for i in tqdm(range(num_epochs)):
        print('*'*5+f'Training epoch {i+1}'+'*'*5)
        for (x,y) in train_dl:
            model.train()
            x=Variable(x).cuda()
            y=torch.FloatTensor(y)
            y=Variable(y).cuda()
            pred=model(x)
            error=loss(pred,y)
            opt.zero_grad()
            error.backward()
            opt.step()
        for (x1,y1) in valid_dl:
            with torch.no_grad():
                x1=x1.cuda()
                y1=y1.cuda()
                model.eval()
                pred1=model(x1)
                error1=loss(pred1,y1)
            #print(pred1.size())
                preds=list(map(prediction,pred1))
                accuracy= accuracy_score(y1.cpu(),preds)
                print(f'Valid loss: {error1}\n Accuracy Score: {accuracy}')
        torch.save(model.state_dict,f'model{i+1}.pth')
</code></pre>
<p>This is my training loop.</p>
<p>I am getting a “CUDA out of memory. Tried to allocate 4.43 GiB (GPU 0; 15.90 GiB total capacity; 6.42 GiB already allocated; 4.42 GiB free; 10.83 GiB reserved in total by PyTorch) (malloc at /opt/conda/conda-bld/pytorch_1587428398394/work/c10/cuda/CUDACachingAllocator.cpp:289)<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x7f14c4533b5e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)”  <strong>Error</strong>  after running about 7 epochs.<br>
Any insights would be appreciated…</p>