Model using too much memory when initialising

Walter_White: <p>I am trying to make a headline generator. But when I initialise my model it keeps on crashing. I checked my memory usage and saw that it was using 50GB of memory.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.o2o = nn.Linear(hidden_size + output_size, output_size)
        self.dropout = nn.Dropout(0.1)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        input_combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(input_combined)
        output = self.i2o(input_combined)
        output_combined = torch.cat((hidden, output), 1)
        output = self.o2o(output_combined)
        output = self.dropout(output)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)
</code></pre>
<p>Here is how I initialise it</p>
<pre><code class="lang-auto">rnn = RNN(len(vocab.get_itos()), 128, len(vocab.get_itos()))
</code></pre>
<p>The length of the vocab is 102,577. I don’t know if that’s too much or if my computer is just bad  but if it is too big what should I do to reduce it</p>

bloos: <p>What exceptions or errors do you get? What are your system specs?</p>

vdw: <aside class="quote no-group" data-username="Walter_White" data-post="1" data-topic="189151">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/walter_white/48/64287_2.png" class="avatar"> Walter_White:</div>
<blockquote>
<p><code>rnn = RNN(len(vocab.get_itos()), 128, len(vocab.get_itos()))</code></p>
</blockquote>
</aside>
<p>What happens if you try just for testing:</p>
<pre><code class="lang-auto">rnn = RNN(100, 128, 100)
</code></pre>

Walter_White: <p>I dont know how to check on a jupyter notebook but I am on an intel 2017 MacBook air with 8GB ram</p>

Walter_White: <p>It runs instantly no errors. As I said i think it might be the large vocab. How can I decrease it</p>

bloos: <p>with a vocab size of 102577 you have <code>(102577+128)*128</code> parameters per linear layer.</p>
<p>this gives a total amount of parameters of <code>3*(102,577+128)*128 = 39438720</code>. Assuming you are training with 32 bit precision this should take a total of 150 mb. This is much, but acceptable, even for 8GB of RAM. There are networks out there, that are MUCH bigger.<br>
What is your batch size?</p>

bloos: <aside class="quote no-group" data-username="Walter_White" data-post="5" data-topic="189151">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/walter_white/48/64287_2.png" class="avatar"> Walter_White:</div>
<blockquote>
<p>How can I decrease it</p>
</blockquote>
</aside>
<p>you could train only on lower case letters. This should bring down your vocabulary by a bit. But I would recommend you to use a proper <a href="https://huggingface.co/docs/transformers/main_classes/tokenizer" class="inline-onebox" rel="noopener nofollow ugc">Tokenizer</a></p>

vdw: <p>There are many common preprocessing techniques, e.g.;</p>
<ul>
<li>case-folding</li>
<li>stemming/lemmatization</li>
<li>normalization (e.g., removal of numbers, emoji, emoticons, punctuation marks)</li>
<li>removal of rare words (optional: replacing with a special “unknown” token)</li>
<li>subword tokenization</li>
</ul>
<p>Which steps are appropriate depends on your tasks. Since you are trying to generate text, stemming/lemmatization is properly out. Maybe this <a href="https://github.com/chrisvdweth/nus-cs4246x/blob/master/3-neural-nlp/Section%201.3%20-%20Data%20Preparation%20(LM%2BSA).ipynb" rel="noopener nofollow ugc">notebook</a> gives some ideas.</p>

Walter_White: <p>For me the kernel only crashes with the code above</p>

Walter_White: <p>I uploaded the code to <a href="https://www.kaggle.com/code/utahjose/notebook0a21c33888" rel="noopener nofollow ugc">kaggle</a></p>

vdw: <p>You can try to convert your notebook to a <code>.py</code> script and run it. Jupyter might have settings to restrict the memory a notebook/kernel is allowed to use:</p>
<pre><code class="lang-auto">jupyter nbconvert --to script mynotebook.ipynb
</code></pre>
<p>And then run</p>
<pre><code class="lang-auto">python mynotebook.py
</code></pre>
<p>I as also wondering what you’re trying to do. From your notebook it looks like you want to predict the headline give a short description of a news article. Is this correct? Because if so, I wondering about this snippet:</p>
<pre><code class="lang-auto">    for i in range(input_line_tensor.size(0)):
        output, hidden = rnn(input_line_tensor[i], hidden)
        l = criterion(output, target_line_tensor[i])
        loss += l
</code></pre>
<p>As this assume that <code>input_line_tnesor</code> and <code>target_line_tensor</code> have the same length.</p>
<p>Your network is suitable for a sequence-labeling task. However, if you want to generate headline from a news article as input, then this is a sequence-to-sequence architecture, and you need a encoder-decoder architecture.</p>

Walter_White: <p>I’m pretty new to machine learning and replacing <a href="https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html" rel="noopener nofollow ugc">this tutorial</a> with my own code. The aim is to generate a headline given the description of the article. The code you showed was copied from the tutorial and I had not edited it yet.</p>
<p>I run the notebook as a script and after reaching 55GB of ram it gave me an error of</p>
<pre><code class="lang-auto">Killed: 9
</code></pre>
<p>I feel like it might be a memory leak because as <a class="mention" href="/u/bloos">@bloos</a> said it shouldn’t be using that much ram</p>
<pre><code class="lang-auto"></code></pre>

vdw: <p>This is a very different task that is done here. Look how the input and target of a single training sample looks like:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e54ca75ce4f87e9852e513e94d8b4bd7d0f63786.png" alt="character-rnn" data-base62-sha1="wItznBY86DTaugYkEiyJ5fydWkK" width="594" height="95"></p>
<p>The input is a name, and the target is the same name shifted by one letter to the left. In short, this network’s goal is to train a language model which will then generate a name given a start sequence of letters.</p>
<p>What you want is a <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">Seq2Seq model</a> similar to Machine Translation, where the input is a text (e.g., a short summary of a news article) and the output is a new text (e.g., a headline).</p>

Walter_White: <p>Oh, I thought I could change it from letters to words and generate words. Anyways I’ll try the Seq2Seq model</p>