RNN not learning for predicting the next word

marco_zaror: <p>I’ve been working on a simple RNN model to predict the next word, I manage to make the model but for some reason is it not learning (the loss is roughly the same at every iteration). Anyone can spot some bug in my code?</p>
<p>class RNN(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, output_size):<br>
super(RNN, self).<strong>init</strong>()</p>
<pre><code>    self.hidden_size = hidden_size
    self.x2h = nn.Linear(input_size, hidden_size)
    self.h2h = nn.Linear(hidden_size, hidden_size)
    self.h2o = nn.Linear(hidden_size, output_size)
    self.softmax = nn.Softmax(dim=1)

def forward(self, input, hidden):

    hidden1 = self.x2h(input)
    hidden2 = self.h2h(hidden)
    hidden = hidden1 + hidden2
    output = self.h2o(hidden)
    output = self.softmax(output)
    
    return output, hidden

def initHidden(self):
    return torch.zeros(1, self.hidden_size)
</code></pre>
<p>rnn_1 = RNN(n_vocab, n_hidden, n_vocab)<br>
optimizer_1 = torch.optim.Adam(rnn_1.parameters(), lr=learning_rate)<br>
criterion = nn.CrossEntropyLoss()</p>
<p>def train(text_x_tensor,label): <span class="hashtag">#text_x_tensor</span> = [4,1,2721]<br>
hidden = rnn.initHidden()<br>
rnn_1.zero_grad()<br>
for tensor in text_x_tensor: <span class="hashtag">#Loop</span> over the 4 context words<br>
output_1, hidden_1 = rnn_1(tensor, hidden_1) <span class="hashtag">#output_1</span> = [1,2721]</p>
<pre><code>loss = criterion(output_1,label)
loss.backward()
optimizer_1.step()
return output_1,loss.item(), hidden_1
</code></pre>
<p>n_epochs = 3<br>
for epoch in range(n_epochs):<br>
<span class="hashtag">#Shuffle</span> batches<br>
random.shuffle(batches)<br>
for i,batch in enumerate(batches):<br>
output_1, loss, hidden_1 = train(batch[0], batch[1])</p>

vdw: <p>Is there any reason why you don’t use a out-of-the-box LSTM or GRU layer?</p>

TinfoilHat0: <p>Did you check <a href="https://github.com/pytorch/examples/tree/master/word_language_model" rel="nofollow noopener">this pytorch tutorial?</a></p>
<p>I’ve coded a next-word predictor myself by slightly modifying this code a few days ago.</p>

marco_zaror: <p>Hi <a class="mention" href="/u/vdw">@vdw</a>,</p>
<p>Thank you very much, the only reason is because I want to start from the basics to understand it well. I spot an important thing, the loss decreases within each epoch, but it doesn’t  between epochs. In other words, when a new epoch starts, the model forget everything.<br>
I’m attaching an image that shows this effect. Do you have any idea what in my code is generating this?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44e134a26dd1182bb391f75ce662f3353a86654e.png" alt="loss" data-base62-sha1="9PkY3Xbm84jySbPeLE2d47A90js" width="350" height="250"></p>

marco_zaror: <p>Thank you <a class="mention" href="/u/tinfoilhat0">@TinfoilHat0</a>!<br>
It is good to have it to keep learning, but I think it is a little advance for me for now. I’m trying to see structural differences in the training part between the tutorial and my code and I can’t find anything. Can you spot some key differences? (or maybe a silly mistake in my code or something)<br>
Thanks again!</p>