Simple Timeseries Auto-encoder

HmmmmML: <p>Hi! I’m implementing a basic time-series autoencoder in PyTorch, according to a tutorial in Keras, and would appreciate guidance on a PyTorch interpretation.  I think this would also be useful for other people looking through this tutorial. Thanks all! HL.</p>
<p>In the <a href="https://rickyhan.com/jekyll/update/2017/09/14/autoencoders.html" rel="noopener nofollow ugc">tutorial</a>, pairs of short segments of sin waves (10 time steps each) are fed through a simple autoencoder (LSTM/Repeat/LSTM) in order to forecast 10 time steps. The latent space chosen is 2 parameters, as the experiment attempts to learn a notion of phase.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3d4a7b284b25a72e8098aa2d716e268c71d830c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d4a7b284b25a72e8098aa2d716e268c71d830c5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3d4a7b284b25a72e8098aa2d716e268c71d830c5_2_300x200.png" alt="image" data-base62-sha1="8KcHrqnC167ObKpsElBOPrHD4Zn" width="300" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3d4a7b284b25a72e8098aa2d716e268c71d830c5_2_300x200.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3d4a7b284b25a72e8098aa2d716e268c71d830c5_2_450x300.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3d4a7b284b25a72e8098aa2d716e268c71d830c5_2_600x400.png 2x" data-dominant-color="F2F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">774×558 68 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Keras implementation from the website:</p>
<pre data-code-wrap="from"><code class="lang-from">from keras.layers import LSTM, RepeatVector
...
m = Sequential()
m.add(LSTM(2, input_shape=(10, 2)))
m.add(RepeatVector(10))
m.add(LSTM(2, return_sequences=True))
...
</code></pre>
<p>I think the below is along the right lines for a PyTorch equivalent - but I have some questions inline:</p>
<pre><code class="lang-auto">seq_len = 10
input_dim = 2
latent_dim = 2
batches = 32 # Arbitrary
...
class AE(nn.Module):
    ...
    def __init__(self, input_dim, latent_dim):
        ...
        self.enc = nn.LSTM(input_dim, latent_dim) 
        self.dec = nn.LSTM(latent_dim, input_dim)
        ...
    def forward(self, x_batch):
        # From [batches x seq_len x input_dim] to [seq_len x batches x input_dim]
        x_batch = x_batch.permute(1,0,2)
        # Extract last hidden, the 'bottleneck'.
        _, (prev_hidden, _) = self.enc(x_batch)
        # (Q) Is this repeat correct? I also wonder if this will fail for multiple layers.
        encoded = prev_hidden.repeat(x_batch.shape)
        # Pass through decoder for output [seq_len x batches x input_dim]
        out, _ = self.dec(encoded)
        # (Q)  Maybe, at this stage, there should be a Linear layer instead of just
        # returning the hidden state from the decoding LSTM?
        return out
    ....
</code></pre>
<p>I suspect it should be easy enough to adapt the encoder above (<code>self.enc</code>) into some (multi-)layered combination of <code>Linear</code>/<code>Conv1D</code> layers; but the <code>LSTM</code> interaction is confusing for me, and I suspect its quite simple.</p>