Character level RNN loss fluctuating widely

username: anteater
cooked: <p>I’m writing a basic character level RNN trained on some English text. The model does reach very close to 0 loss when trained on a very very small training set of a few sentences, but it takes many epochs (~500). When I try it on a more complex dataset (~2-3 MB) the loss fluctuates widely even after training for several hours and the predictions are extremely repetitive, repeating the same word over and over.</p>
<p>Example of loss graph after running for 10 epochs:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/310702ddd52792d87eef679b9cc1e7476a9a7d0b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/310702ddd52792d87eef679b9cc1e7476a9a7d0b" title="Screenshot 2023-06-07 at 10.13.10 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/310702ddd52792d87eef679b9cc1e7476a9a7d0b_2_690x474.jpeg" alt="Screenshot 2023-06-07 at 10.13.10 PM" data-base62-sha1="6ZIpSLxwG85yHFKtYq1NUXYjx9h" width="690" height="474" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/310702ddd52792d87eef679b9cc1e7476a9a7d0b_2_690x474.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/310702ddd52792d87eef679b9cc1e7476a9a7d0b_2_1035x711.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/1/310702ddd52792d87eef679b9cc1e7476a9a7d0b.jpeg 2x" data-dominant-color="FAFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-06-07 at 10.13.10 PM</span><span class="informations">1210×832 51.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And the model itself:</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
  def __init__(self,input_size, hidden_size):
    super(LSTMModel,self).__init__()
    self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
    self.h2o = nn.Linear(hidden_size,input_size)
    self.input_size = input_size
    self.hidden = None
  
  def forward(self, input):
    input = torch.nn.functional.one_hot(input, num_classes=self.input_size).type(torch.FloatTensor).unsqueeze(0)
    input = input.to(device)
    if self.hidden == None:
      l_output, self.hidden = self.lstm(input)
    else:
      l_output, self.hidden = self.lstm(input,self.hidden)

    self.hidden = (self.hidden[0].detach(), self.hidden[1].detach())

    return self.h2o(l_output)


</code></pre>
<p>And the training loop:</p>
<pre><code class="lang-auto">dict_size = len(char_to_in)
hidden_size = 512
lr = 0.001


model = LSTMModel(dict_size, hidden_size)
model.to(device)
torch.nn.utils.clip_grad_norm_(model.parameters(), 1)

losses = []

optimizer = torch.optim.Adam(model.parameters(),amsgrad=True,lr=lr)
def run_epoch():
    i = 0
    while i &lt; len(text)-40:
      seq_len = random.randint(10,40)
      input, target = text[i:i+seq_len], text[i+1:i+1+seq_len]

      i += seq_len
      # forward pass
      output = model(input)
      loss = F.cross_entropy(output.squeeze()[-1:], torch.Tensor(target[-1:]).long()) 
      # compute gradients
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()

    print("Loss:",loss.item())
    losses.append(loss.item())
</code></pre>
<p>I’m basing this off of the following tutorial, with some changes, who achieved much better results with a similar structure: <a href="https://jvns.ca/blog/2020/11/30/implement-char-rnn-in-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">An attempt at implementing char-rnn with PyTorch</a><br>
I am very much a beginner at creating these sorts of models, so apologies if I’m doing anything basic wrong, and thanks everyone for the help.</p>

username: vdw
cooked: <p>Hm, I cannot spot anything obvious wrong, but I think you can optimize your training quite a lot. Look at this line</p>
<pre><code class="lang-auto">loss = F.cross_entropy(output.squeeze()[-1:], torch.Tensor(target[-1:]).long()) 
</code></pre>
<p>If I understand this line correctly – please correct me if I’m wrong – you calculate the loss only based on the <strong>last</strong> character. However, you can calculate the loss for <strong>each</strong> character. For example, if your sequence is of length 20, even the prediction for the character at time step 4 gives you useful information. So what you can do is to accumulate all the losses at all time steps, roughly like</p>
<pre><code class="lang-auto">target = torch.Tensor(target).long().to(device)
...
loss = 0
for j in range(seq_len):
    loss += F.cross_entropy(output.squeeze()[j:], target[j:])
</code></pre>
<p>This can be implemented without the loop, but it’s easier to see the idea here.</p>