LSTMCell autoencoder input shape problems

Wesley_Neill: <p>I need some help. I’m having some problems setting up an basic LSTM autoencoder (without attention or anything fancy).</p>
<p>This implementation follows a paper that uses this implementation:</p>
<p><strong>Encoder:</strong> Standard LSTM layer. Input sequence is encoded in the final hidden state.<br>
<strong>Decoder:</strong> Reconstruct the sequence one element at a time, starting with the last element <code>x[N]</code>.</p>
<p>Decoder algorithm is as follows for a sequence of length <code>N</code>:</p>
<ol>
<li>Get Decoder initial hidden state `hs[N]: Just use encoder final hidden state.</li>
<li>Reconstruct last element in the sequence: <code>x[N]= w.dot(hs[N]) + b</code>.</li>
<li>Same pattern for other elements: <code>x[i]= w.dot(hs[i]) + b</code>
</li>
<li>use <code>x[i]</code> and <code>hs[i]</code> as inputs to LSTM to get <code>x[i-1]</code> and <code>hs[i-1]</code>
</li>
</ol>
<p>The problem is that at each time step, <code>x[i]</code> and <code>hs[i]</code> must be the same shape in order to be input to an LSTM cell, but they are not. That’s because <code>x</code> is the shape of the original sequence data, where <code>hs</code> is the shape of the embedded/encoded vector. There’s no way to make them the same…</p>
<p>What gives?</p>
<p>Encoder:</p>
<pre><code class="lang-auto">class SeqEncoderLSTM(nn.Module):
    def __init__(self, n_features, hidden_size):
        super(SeqEncoderLSTM, self).__init__()
        
        self.lstm = nn.LSTM(
            n_features, 
            hidden_size, 
            batch_first=True)
        
    def forward(self, x):
        _, hs = self.lstm(x)
        return hs
</code></pre>
<p>Decoder:</p>
<pre><code class="lang-auto">class SeqDecoderLSTM(nn.Module):
    def __init__(self, enc_hs, emb_size, n_features, seq_len):
        super(SeqDecoderLSTM, self).__init__()
        
        self.seq_len = seq_len
        
        self.cell = nn.LSTMCell(emb_size, emb_size)
        self.dense = nn.Linear(emb_size, n_features)
        
    def forward(self,x, hs_0):
        
        # add each point to the sequence as it's reconstructed
        x = torch.tensor([])
        
        # Initial hidden state of decoder is final hidden state of encoder
        hs_i, cs_i = hs_0
        
        # reconstruct first (last) element
        x_i = self.dense(hs_i)
        x = torch.cat([x, x_i])
        
        # reconstruct remaining elements
        for i in range(1, self.seq_len):
            print("last x: ", x.shape)
            print("hs shape: ", hs_i.shape)
            print("cs shape: ", cs_i.shape)
            hs_i, cs_i = self.cell(x_i, (hs_i, cs_i))
            x_i = self.dense(hs)
            torch.cat([x, x_i])
</code></pre>
<p>Bringing it all together:</p>
<pre><code class="lang-auto">
class LSTMEncoderDecoder(nn.Module):
    def __init__(self, n_features, emb_size, seq_len):
        super(LSTMEncoderDecoder, self).__init__()
        self.n_features = n_features
        self.hidden_size = emb_size

        self.encoder = SeqEncoderLSTM(n_features, emb_size)
        self.decoder = SeqDecoderLSTM(emb_size, emb_size, n_features, seq_len)
    
    def forward(self, x):
        hs = self.encoder(x)
        hs = tuple([h.squeeze(0) for h in hs])
        out = self.decoder(x, hs)
        return out
        
</code></pre>

vdw: <p>I feel the issue is with</p>
<aside class="quote no-group" data-username="Wesley_Neill" data-post="1" data-topic="105590">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wesley_neill/48/25709_2.png" class="avatar"> Wesley_Neill:</div>
<blockquote>
<p><code>self.cell = nn.LSTMCell(emb_size, emb_size)</code></p>
</blockquote>
</aside>
<p>which should be <code>self.cell = nn.LSTMCell(n_features, emb_size)</code>.</p>
<p>I assume(!), looking at</p>
<pre><code>3. Same pattern for other elements: x[i]= w.dot(hs[i]) + b 
</code></pre>
<p>that <code>self.dense</code> in your decoder converts the first hidden state into input. However, I can’t see what</p>
<pre><code class="lang-auto">x = torch.cat([x, x_i])
</code></pre>
<p>is doing.</p>

Wesley_Neill: <p>Good Morning, and thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>The whole time I was thinking with my fingers: “The input to the LSTM cell is the embedding from the encoder”, even though I knew that the input to the LSTM cell is the last reconstructed input. I’ve been staring at that diagram for two days!</p>
<p>to your question about <code>torch.cat()</code>, I’m collecting each reconstructed element of the sequence back into a sequence…</p>
<p>Is that bad/wrong?</p>

vdw: <aside class="quote no-group" data-username="Wesley_Neill" data-post="3" data-topic="105590">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wesley_neill/48/25709_2.png" class="avatar"> Wesley_Neill:</div>
<blockquote>
<p>Is that bad/wrong?</p>
</blockquote>
</aside>
<p>With neural networks, that’s always a tricky question :).</p>
<p>Really complaining are networks only when the shapes of the tensors to workout as inputs for the layers. That’s essentially the only case where PyTorch throws errors. And since this seemed to be your concern here, I merely tried to get a sense of what’s going on based on the shapes.</p>
<p>If a network does not throw any error, one can sometimes quickly spot that something is off. For example, a couple of weeks ago, someone tried to figure out why his RNN-based classifier wasn’t working, and it turnout that he used the first and not the last hidden state for further processing.</p>
<p>Beyond that, if something is bad or wrong depends on the data, the task, and whatnot.</p>

Wesley_Neill: <p>Well feed forward and training is happening… But it appears that the model just converges to predicting the sequence average value… a flat line. Bummer.</p>
<p>But, I’ve marked your answer as my solution and I’ll ask a separate question about that!</p>