Is the code correct for character level generation in lstm?

username: Sanjayvarma11
cooked: <p>I started learning nlp on my own.Intially i started with movie review sentiment classification and it is working fine.Next I started working on text generation from shakespeare text.But it is not training at all.For every epoch i am printing the predicted results and they are same everytime.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p><strong>All about dimensions</strong><br>
I took an entire corpus and divided into a fixed length of say 100.Each character is replaced by it’s index in dictionary.So input and target looks like this</p>
<pre><code class="lang-auto">input:
[38, 32, 38, 50, 56, 28, 5, 41, 38, 5, 67, 52, 33, 4, 32, 67, 52, 67, 8, 28, 62, 66, 38, 59, 71, 59, 50, 28, 31, 44, 27, 28, 66, 56, 15, 50, 50, 28, 28, 38, 50, 28, 53, 80, 37, 57, 1, 28, 38, 42]
target:
[32, 38, 50, 56, 28, 5, 41, 38, 5, 67, 52, 33, 4, 32, 67, 52, 67, 8, 28, 62, 66, 38, 59, 71, 59, 50, 28, 31, 44, 27, 28, 66, 56, 15, 50, 50, 28, 28, 38, 50, 28, 53, 80, 37, 57, 1, 28, 38, 42, 82]

</code></pre>
<p>let us say batch size is32<br>
let us say fixed length is 100</p>
<p>input to model:[32,100]</p>
<p><strong>For enabling to use batches we have to take respective words of each sentence into a row</strong><br>
So applying transpose<br>
then shape becomes [100,32]</p>
<p>Then sending it to nn.Embedding layer(output shape now becomes [100,32,emb_dim]) and again sending it to lstm layer(output becomes [100,32,hidden_dim]).<br>
If you have doubt plz take a look at my model.</p>
<p><strong>Model implementation</strong></p>
<pre><code class="lang-auto">class CharRNN(nn.Module):
  def __init__(self,n_hidden=50,n_layers=2,drop_prob=0.5,vocab_dim=50,emb_size=50,batch_size=64,device="cuda"):
      super().__init__()
      self.drop_prob = drop_prob
      self.n_layers = n_layers
      self.n_hidden = n_hidden
      self.batch_size=batch_size
      self.device=device
      self.embedding=nn.Embedding(vocab_dim,emb_size)
      self.lstm=nn.LSTM(emb_size,n_hidden,n_layers,dropout=self.drop_prob)
      
  

  def init_hidden(self):
        """Set initial hidden states."""
        h0 = torch.zeros(
            self.n_layers,
            self.batch_size,
            self.n_hidden,
        )
        c0 = torch.zeros(
            self.n_layers,
            self.batch_size,
            self.n_hidden,
        )
        
        h0 = h0.to(self.device)
        c0 = c0.to(self.device)

        return h0, c0    


  def apply_rnn(self, embedding_out):
        
        activations, (hn,cn) = self.lstm(embedding_out, self.init_hidden())
        
        return activations

  def forward(self, inputs, return_activations=False):
        self.batch_size = len(inputs)
    
       inputs = torch.LongTensor(inputs).to(self.device)
        inputs=inputs.transpose(0,1)
        
        # Get embeddings
        embedding_out = self.embedding(inputs)
        
        activations = self.apply_rnn(embedding_out)
        
        out = torch.sigmoid(activations)

        # Put the output back in correct order
        return out
</code></pre>
<p>Now out output is of shape [100,32,emb_dim].Now to apply cross entropy initally my target is of shape [32,100].So firstly i have to transpose it again and permute it.So after transposing shape becomes [32,100,emb_dim] and after permuting the shape becomes [32,emb_dim,100].After that i am finding loss.<br>
<strong>Here emb_dim is the output produced and i am using it as one hot vector by sending it through sigmid and finding most probable word in it.</strong></p>
<p><strong>Here is my training code</strong></p>
<pre><code class="lang-auto">

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(
    filter(lambda p: p.requires_grad, model.parameters()),
    lr=0.1,
)


def train_epoch(model, optimizer, train_loader):
    model.train()
    total_loss = total = 0
    
    progress_bar = tqdm_notebook(train_loader, desc='Training', leave=False)
    for inputs,targets in progress_bar:
        targets=torch.LongTensor(targets)
        target = targets.to(device)  # targets shape[batch_size,fixed_length]
        #target=target.transpose(0,1)
        # Clean old gradients
        optimizer.zero_grad()
        
        # Forwards pass
        output = model(inputs) # inputs shape [batch_size,fixed_length] outputs shape [fixed_length,batch_size,hidden_dim]
        outputs=output.transpose(0,1) # outputs shape [batch_size,fixed_length,hidden_dim]
        outputs=outputs.permute(0,2,1 # outputs shape [batch_size,hidden_dim,fixed_length])
        loss = nn.CrossEntropyLoss()(outputs, target)

        # Perform gradient descent, backwards pass
        loss.backward()

        # Take a step in the right direction
        optimizer.step()
        #scheduler.step()

        # Record metrics
        total_loss += loss.item()
        total += len(target)
        progress_bar.set_description(
        f'train_loss: {loss:.2e}'
        f'\tavg_loss: {total_loss/total:.2e}\n',
      )

    return total_loss / total


</code></pre>
<p><strong>I also printed results during my 9th and 10th epoch.Plz take a look at them and see what went wrong</strong></p>
<pre><code class="lang-auto">predicted_text=
c&amp;aeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

Parameter containing:
tensor([[-0.0848,  1.7742, -0.7627,  ...,  0.2818,  0.3630,  0.6283],
        [ 0.1355, -0.4772,  0.0499,  ...,  0.6572, -1.6990,  0.6295],
        [ 1.3463, -1.8206, -0.1466,  ..., -0.4201,  1.1724, -1.2711],
        ...,
        [ 0.7217,  0.2917, -0.4138,  ..., -0.9130, -0.3257,  0.7373],
        [ 1.9201,  1.0811,  0.0864,  ..., -1.5404, -0.4448, -1.3606],
        [-0.3362,  0.4130,  0.4206,  ...,  1.8701,  1.0428, -0.9026]],
       device='cuda:0', requires_grad=True)
Gradient containing:
tensor([[-6.0252e-05,  2.3833e-05, -9.2437e-05,  ..., -4.4102e-05,
          6.2251e-05,  3.9563e-05],
        [-1.2471e-04,  1.6429e-04, -1.6832e-06,  ..., -4.8308e-05,
          2.9626e-05,  6.5522e-05],
        [-2.6410e-05,  6.6710e-05, -7.8888e-05,  ..., -1.4033e-05,
         -2.6348e-05,  2.2501e-05],
        ...,
        [-8.3676e-06, -4.0126e-05, -5.0642e-05,  ...,  1.2275e-05,
         -3.3493e-05,  1.0394e-06],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00]], device='cuda:0')
epoch #  9	train_loss: 1.33e-01	valid_loss: 2.17e-01


predicted_text=
c&amp;aeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

Parameter containing:
tensor([[-0.0848,  1.7740, -0.7626,  ...,  0.2818,  0.3629,  0.6283],
        [ 0.1355, -0.4778,  0.0503,  ...,  0.6571, -1.6992,  0.6297],
        [ 1.3463, -1.8207, -0.1465,  ..., -0.4201,  1.1724, -1.2711],
        ...,
        [ 0.7217,  0.2913, -0.4136,  ..., -0.9129, -0.3258,  0.7374],
        [ 1.9201,  1.0810,  0.0864,  ..., -1.5403, -0.4448, -1.3605],
        [-0.3362,  0.4130,  0.4206,  ...,  1.8701,  1.0428, -0.9026]],
       device='cuda:0', requires_grad=True)
Gradient Containing:
tensor([[-5.8078e-05,  3.2291e-05, -8.6524e-05,  ..., -4.5590e-05,
          7.4338e-05,  3.1892e-05],
        [-1.0505e-04,  1.4412e-04,  7.3077e-06,  ..., -4.6723e-05,
          3.1852e-05,  5.1921e-05],
        [-2.7566e-05,  6.2100e-05, -7.5454e-05,  ..., -1.2124e-05,
         -2.9602e-05,  2.2030e-05],
        ...,
        [-7.7157e-06, -3.8574e-05, -4.5693e-05,  ...,  1.3392e-05,
         -3.1313e-05, -9.7651e-07],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,
          0.0000e+00,  0.0000e+00]], device='cuda:0')
epoch # 10	train_loss: 1.33e-01	valid_loss: 2.16e-01

</code></pre>
<p>Thank you.</p>

username: Abhilash_Srivastava
cooked: <aside class="quote no-group" data-username="Sanjayvarma11" data-post="1" data-topic="99580">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sanjayvarma11/48/20339_2.png" class="avatar"> Sanjayvarma11:</div>
<blockquote>
<p><code>out = torch.sigmoid(activations)</code></p>
</blockquote>
</aside>
<p>Should you be using a <code>nn.Linear</code> layer here instead of sigmoid activation?<br>
I see that <code>CrossEntropyLoss</code> being used. So, simply using the logit outputs from the linear should be fine.</p>

username: Sanjayvarma11
cooked: <p>Sir. I didn’t get it. So after getting outputs from the lstm I have to pass them through linear layer and then again i have to pass them through sigmoid right??<br>
Before this one i did a movie sentiment classification where it worked fine just by sending hidden outputs.Okay sir i will try that out.<br>
Also sir is my implementation correct?</p>

username: Abhilash_Srivastava
cooked: <p>What was the sentiment classification problem like? Was it a binary classification problem (Positive vs Negative)? In that case, sigmoid activation would work. But this is a multi-class classification. So, my guess is that you shouldn’t be using sigmoid here. Instead, replace it with a Linear layer (size = num_classes, input_dim).</p>

username: Sanjayvarma11
cooked: <p>Yeah sir.I confused the softmax with sigmoid but cross entropy actually has softmax.So i think i dont need to use that.Thank you sir.My loss is decreasing now but the prediction is not changing at all.Can you please take a look at it.</p>
<p><strong>Updated model definition</strong></p>
<pre><code class="lang-auto">class CharRNN(nn.Module):
  def __init__(self,n_hidden=50,n_layers=2,drop_prob=0.5,vocab_dim=50,emb_size=50,batch_size=64,device="cuda"):
      super().__init__()
      self.drop_prob = drop_prob
      self.n_layers = n_layers
      self.n_hidden = n_hidden
      self.batch_size=batch_size
      self.device=device
      self.embedding=nn.Embedding(vocab_dim,emb_size)
      self.lstm=nn.LSTM(emb_size,n_hidden,n_layers,dropout=self.drop_prob)
      self.linear=nn.Linear(n_hidden,n_hidden)
  

  def init_hidden(self):
        """Set initial hidden states."""
        h0 = torch.zeros(
            self.n_layers,
            self.batch_size,
            self.n_hidden,
        )
        c0 = torch.zeros(
            self.n_layers,
            self.batch_size,
            self.n_hidden,
        )
        
        h0 = h0.to(self.device)
        c0 = c0.to(self.device)

        return h0, c0    


  def apply_rnn(self, embedding_out):
        
        activations, (hn,cn) = self.lstm(embedding_out, self.init_hidden())
        
        return activations

  def forward(self, inputs, return_activations=False):
        self.batch_size = len(inputs)
    
        inputs = torch.LongTensor(inputs).to(self.device)
        inputs=inputs.transpose(0,1)
        
        # Get embeddings
        embedding_out = self.embedding(inputs)
        
        activations = self.apply_rnn(embedding_out)
        
        activations=activations.transpose(0,1) # output shape [batchSize,fixed_length,n_hidden]
        
        activations=self.linear(activations)
        

       return activations    

</code></pre>
<p><strong>My model results.It is predicting spaces again and again</strong></p>
<pre><code class="lang-auto">epoch #  6	train_loss: 1.05e-01	valid_loss: 1.07e-01



prediction: c                                                   

epoch #  7	train_loss: 1.05e-01	valid_loss: 1.07e-01



prediction:c                                                   

epoch #  8	train_loss: 1.05e-01	valid_loss: 1.07e-01



prediction:c                                                   

epoch #  9	train_loss: 1.05e-01	valid_loss: 1.06e-01



predicition:c                                                   

epoch # 10	train_loss: 1.04e-01	valid_loss: 1.06e-01

</code></pre>
<p><strong>Code for printing prediction</strong></p>
<pre><code class="lang-auto">def print_re(text):
  
  print(text,end=' ')   #printing first character
  model.eval()          #setting to eval mode
  with torch.no_grad():
        # Forwards pass
        tokens=torch.LongTensor([[char2int[text]]]).to(device)  #coonverting  first char to token
        for i in range(0,50):    #i considered fix len as 50
         
          embeddings=model.embedding(tokens)   #accessing layers through (.) operator.Is it correct?? 
                                                                            #output shape[1,1,emb_len]
          
          hidden_output,(h,c)=model.lstm(embeddings,hiddens) # getting output for that single token
                                                                                                 #output shape[1,1,n_hidden]
                                 
          hidden_output=model.linear(hidden_output)
                                                                                               #output shape[1,1,n_hidden]        
  
          
          hidden_output=hidden_output[0].squeeze(0).squeeze(0)
          
        
          prediction = hidden_output.cpu().numpy()
          
          prediction = np.argmax(prediction, axis=0)
          
          print(int2char[prediction],end='')
          
          tokens=torch.LongTensor([[prediction]]).to(device)
</code></pre>
<p>It is printing just spaces for prediction.Try to see if there is any problem sir.Thank you</p>

username: Sanjayvarma11
cooked: <p>Sir i also tried to print predictions of classes in  sorted order and except first one which i gave everytime prediciton is same for nn.linear.Everytime the sorted prediction is same and there are not changing.</p>
<pre><code class="lang-auto">predicted character:c

 predicted_classes in sorted order:
[63 76  2 18 59 51 57 13  3 41 10 68 28 82 19 23 46  0 70 54 80 47 11 69
 31 67 32 12 30 35 43 33 39  6 60  7 38 56  5 65 61 42 29 62 22 14  4 21
 44 17 81 36 55 77 53 78 27 74 75 45 16 71 26 48 37 72 49 40 73 25 64 20
 79 34 52 66 58  8 24 15 50  1  9]

output: tensor([[76]], device='cuda:0') #i choose second highest vale but the predicted sorted order is always same

[63 76  2 18 51 57 13 59  3 41 10 68 28 82  0 23 80 19 54 46 31 70 30 67
 39 69 11 12 47 33 35 32 38 61 43  6 56 65 60 42 29  7  5 21 22 16 44 81
 62 14  4 73 40 36 49 17 48 25 78 72 77 71 79 66 75  8 45 15 58 55 26 74
 53 20  1 64  9 50 37 34 27 24 52]

tensor([[76]], device='cuda:0')

[63 76  2 18 51 57 13 59  3 41 10 68 28 82  0 23 80 19 54 46 31 70 30 67
 39 69 11 12 47 33 35 32 38 61 43  6 56 65 60 42 29  7  5 21 22 16 44 81
 62 14  4 73 40 36 49 17 48 25 78 72 77 71 79 66 75  8 45 15 58 55 26 74
 53 20  1 64  9 50 37 34 27 24 52]
tensor([[76]], device='cuda:0')
</code></pre>
<p><strong>code for the printing prediction</strong></p>
<pre><code class="lang-auto">def print_re(text,hiddens):
  
  print(text,end=' ')
  model.eval()
  with torch.no_grad():
        # Forwards pass
        tokens=torch.LongTensor([[char2int[text]]]).to(device)
        for i in range(0,10):
         
          embeddings=model.embedding(tokens)
          #print(embeddings.shape)
          hidden_output,(h,c)=model.lstm(embeddings,hiddens)
          #print(hidden_output.shape)
          hidden_output=model.linear(hidden_output)
          #print(hidden_output.shape)
          
          #hidden_output = torch.sigmoid(hidden_output)
          
          hidden_output=hidden_output[0].squeeze(0).squeeze(0)
          
        
          prediction = hidden_output.cpu().numpy()
          #print(prediction)
          prediction = np.argsort(-prediction, axis=0)
          print(prediction)
          print(int2char[prediction[1]],end='')
          
          tokens=torch.LongTensor([[prediction[1]]]).to(device)
          print(tokens)
</code></pre>

username: Abhilash_Srivastava
cooked: <p>For the training part:</p>
<ol>
<li>Try using <code>optim.Adam</code> instead of <code>optim.SGD</code>.</li>
<li>Add a <code>nn.dropout</code> (dropout = 0.1) layer before Linear layer in your model.</li>
<li>Assuming you have sufficient high-quality data, make sure training is happening correctly (loss values decrease and converge). Plot the loss values.</li>
</ol>
<p>For the inference part:</p>
<ol>
<li>Simply pass the input to <code>model</code> and simply use <code>np.argmax</code> to pick the next best char. You don’t need to use the <code>(.) operators</code> functions for embeddings etc.</li>
</ol>

username: Sanjayvarma11
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7a091b085c9518d41cba41ad03a70ec0416f4f3.png" alt="download" data-base62-sha1="uLwFMiNPb0yqYgtCrpLqHkPnTur" width="398" height="278"></p>
<p>I applied all your suggestions sir but the predictions are same.They are prinitng spaces again and again.</p>
<p><strong>printing code</strong></p>
<pre><code class="lang-auto">def print_re(text,hiddens):
  
  print(text,end=' ')
  model.eval()
  with torch.no_grad():
        # Forwards pass
        tokens=[[char2int[text]]]
        #print(tokens)
        for i in range(0,10):
          output = model(tokens)
          #outputs=output.permute(0,2,1)
          #print(output.shape)
          hidden_output=output[0].squeeze(0).squeeze(0)
          
          prediction = hidden_output.cpu().numpy()
          #print(prediction)
          prediction = np.argsort(-prediction, axis=0)
          #print(prediction)
          print(int2char[prediction[0]],end='')
          
          tokens=[[prediction[0]]]
          #print(tokens)
</code></pre>
<p><strong>Model code</strong></p>
<pre><code class="lang-auto"> def forward(self, inputs, return_activations=False):
        self.batch_size = len(inputs)
    
        inputs = torch.LongTensor(inputs).to(self.device)
        inputs=inputs.transpose(0,1)
        
        # Get embeddings
        embedding_out = self.embedding(inputs)
        
        activations = self.apply_rnn(embedding_out)
        
        activations=activations.transpose(0,1) # output shape [batchSize,fixed_length,emb_length]
        activations=self.dropout(activations)
        activations=self.linear(activations)
        
        #out = torch.sigmoid(activations)

        # Put the output back in correct order
        return activations    
</code></pre>
<p>Thank you sir</p>

username: Abhilash_Srivastava
cooked: <p>Most likely your network is not learning. A few more things to try:</p>
<ol>
<li>Remove the <code>init_hidden </code> in LSTM. Zero initilizers are usually not a good idea.</li>
<li>Maybe your dataset has way too many spaces, hence the model is simply overfitting to spaces.</li>
<li>Try overfitting to a very small dataset (say just 1 batch of 15 chars), and train for a few epochs, to bring the loss value ~ 0 (say &lt; e-05). This is a nice way to test your model is actually learning.</li>
</ol>

username: Sanjayvarma11
cooked: <p>Sir i don’t think it can overfit to space bcz for every few chars there will be a single space.So i think it is not a possibility<br>
<strong>Sir replacing torch.zeros with torch.randn really helps in prediction.SO it is one of the soution</strong></p>
<p>But i tried various ones.Results are below.I saw your linkedin and your career is so good so i assume you are busy.So Thank you for answering to my queries.</p>
<p>Sir i did it with batch size 1 and 15 chars and it converged to 0 loss. The prediction is also good.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c6575af9faec92b0238538d8783283344e2e8e9.png" alt="download (1)" data-base62-sha1="6kKnSqPbGbkAJI8z0FhdDTm8sNr" width="376" height="278"></p>
<p>After that I tried with batch size 32 and results are again bad.</p>
<p>So this time i tried with 1000 chars and <strong>batch size 1</strong> whose length is 30 by <strong>removing spaces</strong> and results are as follows:</p>
<pre><code class="lang-auto">ch #285	train_loss: 8.89e-02	valid_loss: 0.00e+00


predicted_text=cdtmseteseeetehhnaelcibeaeaelce
epoch #286	train_loss: 7.01e-02	valid_loss: 0.00e+00


cieaeteelmteeh,wWshselueaeeeshs
epoch #287	train_loss: 5.59e-02	valid_loss: 0.00e+00


celceSeaehshhshh,tmteshslowhsee
epoch #288	train_loss: 5.67e-02	valid_loss: 0.00e+00


celyslueteaeteshnaesewlueseelse
epoch #289	train_loss: 4.86e-02	valid_loss: 0.00e+00
</code></pre>
<p>It didn’t converged to 0.but predicted text is TEXT not repeating a char<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b506d3423b3890ad478bcad11739b37e8d04aff1.png" alt="download (2)" data-base62-sha1="pPr2NvAr8VOjmJe7Ax878jgFWxP" width="392" height="278"></p>
<p>SO lets try with spaces sir.<br>
So this time i tried with 1000 chars and <strong>batch size 1</strong> whose length is 30 by <strong>not removing spaces</strong> and results are as follows:</p>
<pre><code class="lang-auto">predicted_text=yt o oryuaTsy sye o-ssst o eb,
epoch #384	train_loss: 8.89e-02	valid_loss: 0.00e+00


cht syt,t o o eb,t,t s o sso ol
epoch #385	train_loss: 7.21e-02	valid_loss: 0.00e+00


c ot,tret e olebe o olyt,t,t,t 
epoch #386	train_loss: 7.94e-02	valid_loss: 0.00e+00

train_loss: 1.09e-01 avg_loss: 9.11e-02 : 100%


ci o yt solyeb,t,t onle e syeb,
epoch #387	train_loss: 9.11e-02	valid_loss: 0.00e+00
</code></pre>
<p>It will converge to 0.but predicted text is TEXT not repeating a char.So i think it is not repeating spaces at all.So i think it is good</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/841d57264d3f39d3c606f6a07bcb898afda61ce3.png" alt="download (3)" data-base62-sha1="iQJT2edYoWozAlj3iJ5ReLyXUzx" width="376" height="278"></p>
<p><strong>So i am getting doubt in culprit batch size.So i will increase the batch size to 5</strong><br>
I overfit the data for 750 epochs<br>
Here are the results:</p>
<pre><code class="lang-auto">cta madraol mck  t eulkthlTvonh
epoch #496	train_loss: 2.02e-01	valid_loss: 0.00e+00


cii slmal eeta o m yt&lt;EOS&gt;mrs  a  h
epoch #497	train_loss: 1.98e-01	valid_loss: 0.00e+00


cneewntn Bee hew o kl    a o  h
epoch #498	train_loss: 1.96e-01	valid_loss: 0.00e+00


co&lt;EOS&gt;f yniimwy6  chm'hgrted oese 
epoch #499	train_loss: 1.96e-01	valid_loss: 0.00e+00


coadkth at  teadsrs  ho w:ognsk
epoch #500	train_loss: 1.94e-01	valid_loss: 0.00e+00
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d1f6ded93d1aaf3e5a9af1a89a174d317bc1c806.png" alt="download (4)" data-base62-sha1="tXqHX9dUb5qTEZrwBOksp0wG30a" width="392" height="278"></p>
<p>The results are not bad like there is no repeating spaces.For first 10 epochs it is repeating spaces but after 50 epochs it starts to print other chars slowly</p>
<p><strong>So So i will increase the batch size to 20</strong></p>
<pre><code class="lang-auto">c     s  Ce &lt;EOS&gt;9f            eoh 
epoch #496	train_loss: 1.33e-01	valid_loss: 0.00e+00


cd  s s s    e      ee         
epoch #497	train_loss: 1.34e-01	valid_loss: 0.00e+00


c     e o  e        ere r   rt 
epoch #498	train_loss: 1.34e-01	valid_loss: 0.00e+00


c-   f          h r e   s e   e
epoch #499	train_loss: 1.33e-01	valid_loss: 0.00e+00


c    e  e    ee ree     th Bae 
epoch #500	train_loss: 1.33e-01	valid_loss: 0.00e+00
</code></pre>
<p><strong>For 200 epochs it is printing only spaces but after that it started to print results like above</strong><br>
**Graph is like below<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bfb90c24eb4d53d8dc2eda4448ef0659ba83bdca.png" alt="download (5)" data-base62-sha1="rm3ERPe1nRQ4zHJdLzhOdwDUwum" width="398" height="278"> **</p>
<p><strong>Thank you sir.</strong><br>
Conclusions:<br>
1)It is good for batch size 1,5<br>
2)It is okay for batch size 20<br>
3)It is bad like printing spaces again and again for batch size 32</p>
<p>Waiting for your reply sir.</p>

username: Abhilash_Srivastava
cooked: <p>Good to see that the network is learning now.<br>
A few things:</p>
<ol>
<li>What is your train data like and what output are you expecting (after the model is trained)? Also, how big is the train and valid data sizes?</li>
<li>If you’re sure that the model is training correctly (loss is decreasing, backpropagation happens correctly), you can start playing with the hyperparameters:<br>
a. Change the dim sizes, eg: n_hidden = 128, emb_size = 128 etc.<br>
b. Also, for the line below, I think the Linear dims should be (n_hidden, n_classes). Use distinct values to avoid confusion.</li>
</ol>
<aside class="quote no-group" data-username="Sanjayvarma11" data-post="5" data-topic="99580">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sanjayvarma11/48/20339_2.png" class="avatar"> Sanjayvarma11:</div>
<blockquote>
<p><code>self.linear=nn.Linear(n_hidden,n_hidden)</code></p>
</blockquote>
</aside>
<p>c. Modify the learning rate: eg: lr = 0.01 etc.</p>

username: Sanjayvarma11
cooked: <p>Sir the model is learning pretty good.Thank you so much sir.I will continue from here.Thank you so much sir.</p>

username: Komil_Parmar
cooked: <p>I am trying to train a LSTM too except, I am trying to implement the LSTM model (i.e. without nn.LSTM). I thought it might be a good idea to revive this discussion as it is very much related.</p>
<p>I trained the model using nn.LSTM with the very same hyper parameters as my implementation and it runs well, generates human like movie reviews (I am feeding IMDB movie reviews) but the one I implemented keeps generating some phrase repeatedly like: “One of thnother starts of the <em><strong>movie that is a great movie that is a great movie that is a great movie that i</strong></em>”. Here, “One of th” is the input.</p>
<p>This is my model code:</p>
<pre><code class="lang-auto">class LSTMCell(nn.Module):
    def __init__(self, hidden_size):
        super(LSTMCell, self).__init__()
        self.hidden_size = hidden_size

        self.forget_gate = nn.Linear(hidden_size + hidden_size, hidden_size)
        self.input_gate = nn.Linear(hidden_size + hidden_size, hidden_size)
        self.output_gate = nn.Linear(hidden_size + hidden_size, hidden_size)
        self.cell_gate = nn.Linear(hidden_size + hidden_size, hidden_size)

    def forward(self, x: torch.Tensor, h: torch.Tensor, c: torch.Tensor) -&gt; Tuple[torch.Tensor, torch.Tensor]:
        x = torch.cat((x, h), dim=1)
        f_t = torch.sigmoid(self.forget_gate(x))
        c = c * f_t

        i_t = torch.sigmoid(self.input_gate(x))
        c_t = torch.tanh(self.cell_gate(x))
        c = c + (i_t * c_t)

        o_t = torch.sigmoid(self.output_gate(x))
        h = o_t * torch.tanh(c)

        return h, c


class LSTM(nn.Module):
    def __init__(self, input_size, hidden_size, index_to_char):
        super(LSTM, self).__init__()
        self.hidden_size = hidden_size
        self.input_size = input_size
        self.index_to_char = index_to_char

        self.embedding = nn.Embedding(input_size, hidden_size, device='cuda')
        self.dropout = nn.Dropout(p=0.7)

        self.lstm_cell = LSTMCell(hidden_size)

        self.fc = nn.Linear(hidden_size, input_size, device='cuda')

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        batch_size, sequence_len = x.size(0), x.size(1)
        h, c = self.init_hidden(batch_size)
        outputs = torch.zeros(sequence_len, x.size(0), self.input_size, device='cuda')

        for t in range(sequence_len):
            x_t = x[:, t]
            x_t = self.embedding(x_t)
            # x_t = self.dropout(x_t) if self.training else x_t
            h, c = self.lstm_cell(x_t, h, c)
            # h = self.dropout(h) if self.training else h
            output = self.fc(h)
            outputs[t] = output

        outputs = outputs.permute(1, 0, 2)
        outputs = outputs.reshape(-1, self.input_size)
        return outputs

    @torch.jit.export
    def generate(self, x: torch.Tensor, sequence_len: int) -&gt; str:
        h, c = self.init_hidden(x.size(0))
        outputs = self.forward(x[:, :-1])
        generated_indices = torch.zeros(sequence_len, dtype=torch.long, device='cpu')
        last_char = torch.unsqueeze(torch.argmax(outputs[-1].long()), 0)

        for t in range(1, sequence_len):
            x_t = self.embedding(last_char)
            h, c = self.lstm_cell(x_t, h, c)

            logits = self.fc(h)
            probabilities = torch.softmax(logits / 1., dim=1)

            char_index = torch.argmax(logits, dim=1)
            generated_indices[t] = char_index.item()
            last_char = char_index.reshape(last_char.shape)

        generated_text = ''.join(self.index_to_char[idx] for idx in generated_indices)

        return generated_text

    def init_hidden(self, batch_size: int) -&gt; Tuple[torch.Tensor, torch.Tensor]:
        h = torch.zeros(batch_size, self.hidden_size, device='cuda')
        c = torch.zeros(batch_size, self.hidden_size, device='cuda')
        return h, c
</code></pre>
<p>I am not putting the training loop code since the same code trains the nn.LSTM model perfectly.<br>
Some important points:-</p>
<ol>
<li>Using the dropout layer which is commented out in the above code does solve the problem to some extent but still the output is not as human-like as generated by nn.LSTM. Also, nn.LSTM does not use dropout by default.</li>
<li>I am clipping the gradients as follows: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=5) for both the model.</li>
<li>Using some temperature to generate text does help but then, the words are not any from the English dictionary. To do this I used the below function:</li>
</ol>
<pre><code class="lang-auto">def sample_from_output(logits, temperature:float=1.0):
    probabilities = torch.softmax(logits / temperature, dim=1)
    return torch.multinomial(probabilities, num_samples=1)
</code></pre>
<p>The characters are so random that it clearly indicates that the probabilities are very close to each other. Atleast that is what I take out from the observation.<br>
4. The generated text is the result after training the model for 10 epochs on 50,000 movie reviews.<br>
5. Hyperparameters are as follows:-</p>
<pre><code class="lang-auto"># Hyperparameters
    INPUT_SIZE = len(train_dataset.chars)
    HIDDEN_SIZE = 256
    NUM_LAYERS = 1
    SEQUENCE_LEN = 50
    BATCH_SIZE = 64
</code></pre>
<ol start="6">
<li>The loss is CrossEntropy.</li>
<li>The validation loss for both the models is identical (around 8.0) and so is the training loss (around 1.02). However, the training loss from nn.LSTM have a huge variance which also results in losses like 0.2 and sometimes above 2.0 while loss from my implementation is very very stable.</li>
<li>Some outputs from the nn.LSTM model with same hyperparameters and same amount of training.</li>
</ol>
<ul>
<li>One of the best movies I have ever seen. I was a fan of the film about a movie that is a good movie about a mo</li>
<li>One of the worst movies I have ever seen. I was a fan of the first time I watched this movie and I was surpris</li>
<li>One of the most comment on the screening of the story of the story is a great movie and I have seen this movie</li>
<li>One of the worst movies I have ever seen in the first time I have ever seen in a lot of good action for the fi<br>
It also generates some phrases repeatedly like my implementation but it is very rare to notice (about once every 10 generations).</li>
<li>One of the most people who has the story of the film is a film that is a film that is a film that is a film th<br>
I highly appreciate your help. Even if you do not get time to look into the code, I still appreciate if you can put some advices or suggestions. Thankyou!</li>
</ul>