Proper way to combine linear layer after LSTM

JohnCwok: <p>Hello,</p>
<p>I have implemented a simple word generating network using a <code>LSTMCell</code> coupled with a <code>Linear</code> layer which works perfectly. I now want to use the <code>LSTM</code> class to be able to process the data in batches in order to go faster.</p>
<p>The same architecture with an <code>LSTM</code> object instance + <code>Linear</code> output layer produces outer nonsense. I figured out that this might be due to the fact that LSTM expects the arguments in the order <code>(seq_lenth, batch_dim, input_dim)</code> whereas the linear layers want <code>(batch_dim,seq_lengnth,input_length)</code> and changed the lstm in my code to <code>batch_first = True</code> but still get the same results.<br>
Could it be that the LSTM still outputs in the format <code>seq_len,batch_dim,input_size</code> even if we pass <code>batch_first = True</code> as an argument ?</p>
<p>I’ve searched a bit on the forum and the answers always suggest to reshape the output of the LSTM before passing it to the linear layer, which I find cumbersome but maybe there is no way around in PyTorch.</p>

Prerna_Dhareshwar: <p>If you’ve specified <code>batch_first = True</code>, then the output will be of the format <code>(batch, seq_length, num_directions * hidden_size)</code>.</p>
<p>From the documentation -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/031c0ad26df0b6564cfd42f88060dd17bf9caa3b.png" data-download-href="https://discuss.pytorch.org/uploads/default/031c0ad26df0b6564cfd42f88060dd17bf9caa3b" title="13%20AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/031c0ad26df0b6564cfd42f88060dd17bf9caa3b_2_689x55.png" alt="13%20AM" data-base62-sha1="rvvMPXAGCC3Xr1pvbhoYTcJXsf" width="689" height="55" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/031c0ad26df0b6564cfd42f88060dd17bf9caa3b_2_689x55.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/031c0ad26df0b6564cfd42f88060dd17bf9caa3b_2_1033x82.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/031c0ad26df0b6564cfd42f88060dd17bf9caa3b.png 2x" data-dominant-color="F0F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">13%20AM</span><span class="informations">1346×108 13 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JohnCwok: <p>Yes that is what I thought … I also controlled by looking carefully at the dimension of each forward step. Then I don’t know what is wrong. Do you see any flaw in my design ? Here is the model :</p>
<pre><code class="lang-auto">class pytorchLSTM(nn.Module):
    def __init__(self,input_size,hidden_size):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size, batch_first = True)
        self.output_layer = nn.Linear(hidden_size,input_size)
        self.tanh = nn.Tanh()
        self.softmax = nn.LogSoftmax(dim = 2)
        
    def forward(self, input, hidden = None):
        if hidden == None:
            hidden = (torch.zeros(1,1,self.hidden_size),torch.zeros(1,1,self.hidden_size))
            out, hidden = self.lstm(input,hidden)
            out = self.tanh(out)
            out = self.output_layer(out)
            out = self.softmax(out)
        else:
            out, hidden = self.lstm(input,hidden)
            out = self.tanh(out)
            out = self.output_layer(out)
            out = self.softmax(out)
        return out, hidden
</code></pre>
<p>the inputs are <code>(1 x seq_length x input_length)</code> tensors corresponding to the one-hot-encoded letters of a word. same for the target. There is of course a start and end token.<br>
Here is the training loop :</p>
<pre><code class="lang-auto">def train_rnn(model):
    criterion = nn.NLLLoss()
    optimizer = torch.optim.Adam(model.parameters())
    n_iters = 10000

    for iter in range(1,n_iters+1):
        #chooses a word randomly in the data
        word = randomChoice(words)
        #transforms the word into a (1 x seq_length x input_length) tensor of one-hot encoded vectors.
        input_tensor = inputTensor(word)
        #target is the same word as input but one step after.
        target_tensor = targetTensor(word).unsqueeze(-1)
        optimizer.zero_grad()       
        loss = 0
        output, hidden = model(input_tensor)
        
        for i in range(input_tensor.size(1)):
            l = criterion(output[0][i].unsqueeze(0), target_tensor[i])
            loss += l
        loss.backward()
        optimizer.step()  
</code></pre>

Prerna_Dhareshwar: <p>This looks correct to me, what errors are you getting?</p>
<p>Edit : Also you dont need to define hidden when you have no previous hidden state and cell state, when they aren’t provided, they default to 0.</p>

JohnCwok: <p>I am not getting an error but I am getting nonsensical results. Here is an example of words the I get with this network :</p>
<pre><code class="lang-auto">ReBGlormodw
FonFanbLeBMG
CBGDBLS
PBBavetharStFEmieYou
DeanVBDTCRe.ChelJaSH
MmiYSvew
BozFaveYPNontLPBLLap
SamonTFoSThox
CLy
DFaneroiboBuD'To
</code></pre>
<p>This is in stark contrast with this model (which has excactly the same architecture) based on <code>LSTMCell</code> :</p>
<pre><code class="lang-auto">class lstmCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.cell = nn.LSTMCell(input_size,hidden_size)
        self.softmax = nn.LogSoftmax(dim = 1)
        self.tanh = nn.Tanh()
        self.output_layer = nn.Linear(hidden_size,input_size)
    
    def forward(self, input, hidden):
        hidden, context = self.cell(input, hidden)
        out = self.tanh(hidden)
        out = self.softmax(self.output_layer(out))
        return out, (hidden, context)
</code></pre>
<p>where i get output like (in this case it was trained with names) :</p>
<pre><code class="lang-auto">Lane
Branis
Rome
Bondelon
</code></pre>

Prerna_Dhareshwar: <p>In the loss calculation you’re comparing output to <code>target_line_tensor</code> instead of <code>target_tensor</code>, is that intentional? What is <code>target _line _tensor</code>?</p>

JohnCwok: <p>woops, I made a small mistake in transferring the code here. It is actually <code>target_tensor</code>.</p>

Prerna_Dhareshwar: <p>I would recommend  the following-</p>
<ol>
<li>Printing out values of out and target tensor and ensure that you are comparing the right values.</li>
<li>Trying to overfit to one (or a few) training example.</li>
</ol>
<p>If you’re not able to overfit to a few examples, there’s probably something wrong with the code.</p>

JohnCwok: <p>okay I will try that tomorrow. Thanks for the tips</p>