GRU Time Series Autoencoder

R.Giskard: <p>Hi to all,<br>
<strong>Issue:</strong><br>
I’m trying to implement a <em><strong>working</strong></em> GRU Autoencoder (AE) for biosignal time series from Keras to PyTorch without succes.</p>
<p>The model has 2 layers of GRU.<br>
The 1st is bidirectional.<br>
The 2nd is not.<br>
I take the ouput of the 2dn and repeat it “<em><strong>seq_len</strong></em>” times when is passed to the decoder.<br>
The decoder ends with linear layer and relu activation ( samples are normalized [0-1])</p>
<p>I already try:</p>
<ul>
<li>Use the hidden of the 2d layer and pass it to the decoder and not the output.</li>
<li>Quit and preserve the grad_clip functions</li>
<li>Quit the 1st layer in decoder (gru_dec1)</li>
</ul>
<p>I welcome any suggestions/advice</p>
<blockquote>
<p><strong>Model Keras</strong>:<br>
inputs = Input(shape=(t, in_channels))<br>
encoded = Bidirectional(GRU(256,return_sequences=True))(inputs)<br>
encoded = GRU(32)(encoded)<br>
decoded = RepeatVector(Signal_Len)(encoded)<br>
decoded = GRU(32,return_sequences=True)(decoded)<br>
decoded = Bidirectional(layers.GRU(256,return_sequences=True))(decoded)<br>
decoded = TimeDistributed(layers.Dense(in_channels,activation=tf.nn.relu,<br>
bias_initializer = b_init))(decoded)<br>
<strong>Optimizer</strong><br>
opt = tf.keras.optimizers.RMSprop()<br>
<strong>Loss</strong><br>
model_autoencoder.compile(optimizer=opt, loss=‘mse’, metrics=[‘mae’,‘mse’])</p>
</blockquote>
<p><strong>MODEL PYTORCH</strong><br>
<strong>Encoder</strong></p>
<blockquote>
<pre><code>class EncoderRNN(nn.Module)
    def __init__(self, n_features, latent_dim, hidden_size):
    super(EncoderRNN, self).__init__()

    self.n_features = n_features
    self.hidden_size = hidden_size
    self.latent_dim = latent_dim

    self.gru_enc = nn.GRU(n_features, hidden_size,
                      batch_first = True,dropout=0,
                      bidirectional=True)
    
    self.lat_layer = nn.GRU(hidden_size*2, latent_dim,
                      batch_first = True, dropout=0,
                      bidirectional = False)
def forward(self, x):
    x, _  = self.gru_enc(x)
    x , h = self.lat_layer(x)
      return x[:,-1].unsqueeze(1)
</code></pre>
</blockquote>
<p><strong>Decoder</strong></p>
<blockquote>
<pre><code>class EncoderRNN(nn.Module):
  def __init__(self, seq_len, n_features , latent_dim , hidden_size):
    super(DecoderRNN, self).__init__()

    self.seq_len = seq_len
    self.n_features = n_features
    self.latent_dim = latent_dim
    self.hidden_size = hidden_size

    self.gru_dec1 = nn.GRU(latent_dim, latent_dim,
                     batch_first = True, dropout=0,
                     bidirectional= False)

    self.gru_dec2 = nn.GRU(latent_dim, hidden_size,
                      batch_first = True, dropout=0,
                      bidirectional= True)

    self.output_layer = nn.Linear(self.hidden_size*2, n_features,bias=True)
    self.act = nn.ReLU()

  def forward(self, x):
   x = x.repeat(1,self.seq_len, 1)
   x, _ = self.gru_dec1(x)
   x, _ = self.gru_dec2(x)
     return self.act(self.output_layer(x))
</code></pre>
</blockquote>
<blockquote>
<pre><code>class AERNN(nn.Module):
  def __init__(self, seq_len, n_features, latent_dim , hidden_size):
    super(AERNN, self).__init__()

    self.seq_len = seq_len
    self.encoder = EncoderRNN(n_features, latent_dim, hidden_size).to(device)
    self.decoder = DecoderRNN(seq_len, n_features, latent_dim, hidden_size).to(device)

  def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
     return x
</code></pre>
</blockquote>
<blockquote>
<pre><code>model = AERNN(seq_len, n_features, latent_dim , hidden_size)
model.apply(bias_init)
model = model.to(device)

optimizer = torch.optim.RMSprop(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss().to(device)
</code></pre>
</blockquote>
<p><strong>TRAIN</strong></p>
<blockquote>
<pre><code>    for epoch in range(100):
        #Train
        model.train()
        train_loss = 0
        for x,_  in train_dl:

            x = x.cuda()
            optimizer.zero_grad()
            recon = model(x)
            loss = loss_fn(recon[:,:,0], x[:,:,0])
            train_loss += loss.item()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), clip_norm)
            torch.nn.utils.clip_grad_value_(model.parameters(), clip_value)
            
            optimizer.step()
</code></pre>
</blockquote>

vdw: <p>It’s not quite clear what your asking. What is not working? Do you get errors? Does the loss not go down? Are you generally not happy with the accuracy? Is the accuracy worse compared to the Keras model?</p>
<p>When I build my autoencoders, I usually start with the most basic setup, see if it works (no errors, loss goes down, able to overtrain it in a small dataset, etc), and then step by step add complexity to the model and check again each time if still works. Getting no errors is usually the easy part, but that doesn’t mean it’s correct.</p>
<p>I’m not saying the model is wrong, but it’s definitely not the classic RNN-based encoder-decoder model. There the encoder, well, encodes your sequence to some latent representation (typically the last hidden state) which is that the “seed” hidden state for the decoder. The decoder the step by step generates the next output item and next hidden state using the current hidden state. In code that usually involves some loop.</p>
<p>In your code, you copy/repeat the last hidden state (I ignore the linear layer for simplicity) and give that sequence to your decoder GRU. This can’t make sense, since that sequence has the same items at each time step.</p>
<p>You may want to look at my code for an <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/autoencoder/textrnnae.py" rel="nofollow noopener">autoencoder</a> and <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/autoencoder/textrnnvae.py" rel="nofollow noopener">variational autoencoder</a> (VAE). The context is text (NLP), but that doesn’t matter. I essentially started with the basic <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">machine translaten / seq2seq model</a>, only that input sentence and output sentence are the same. And then I just tweaked some stuff. They both train fine, with the VAE inherently much more difficult to train.</p>

R.Giskard: <p>Hi <a class="mention" href="/u/vdw">@vdw</a>, thanks for your reply.</p>
<p>Sorry for the lack of details that your ramarked me,</p>
<p>To be more clear:</p>
<ul>
<li><strong>About the loss:</strong> It doesn’t decrese, remains constant</li>
</ul>
<blockquote>
<div class="md-table">
<table>
<thead>
<tr>
<th>Epoch:0</th>
<th>Patience: 0</th>
<th>Train_Loss: 0.00085809</th>
<th>Val_Loss: 0.00086563</th>
</tr>
</thead>
<tbody>
<tr>
<td>Epoch:1</td>
<td>Patience: 0</td>
<td>Train_Loss: 0.00082885</td>
<td>Val_Loss: 0.00086676</td>
</tr>
<tr>
<td>Epoch:2</td>
<td>Patience: 0</td>
<td>Train_Loss: 0.00082884</td>
<td>Val_Loss: 0.00086649</td>
</tr>
<tr>
<td>Epoch:3</td>
<td>Patience: 0</td>
<td>Train_Loss: 0.00082886</td>
<td>Val_Loss: 0.00086577</td>
</tr>
<tr>
<td>Epoch:4</td>
<td>Patience: 0</td>
<td>Train_Loss: 0.00082882</td>
<td>Val_Loss: 0.00086545</td>
</tr>
<tr>
<td>Epoch:5</td>
<td>Patience: 0</td>
<td>Train_Loss: 0.00082893</td>
<td>Val_Loss: 0.00086574</td>
</tr>
</tbody>
</table>
</div></blockquote>
<p><strong>To give more context:</strong></p>
<p>I’m working with bio-signal in a steady state<br>
I decided to use “repeat” thinking that the hole signal could be represented in the output of the encoder (a compressed representation of it). Then, the decoder, though the hiden_state and the last output of the decoder itself, could do the rest.</p>
<p>See below random example of reconstruction with Keras during training epochs (1,20,50,100 and 180 respectively). Its seems to be in order to me.</p>
<p>Anyway … thank you your links, I’ll check it and I’ll continue traying to obtain similar result.</p>
<blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2f5c964e5bef4441ad6e116974152fdca10c667.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e2f5c964e5bef4441ad6e116974152fdca10c667" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2f5c964e5bef4441ad6e116974152fdca10c667_2_345x119.jpeg" alt="1" data-base62-sha1="wnMveWM3ioLXbNHafN1lV1n12Cz" width="345" height="119" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2f5c964e5bef4441ad6e116974152fdca10c667_2_345x119.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2f5c964e5bef4441ad6e116974152fdca10c667_2_517x178.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2f5c964e5bef4441ad6e116974152fdca10c667_2_690x238.jpeg 2x" data-dominant-color="F8F6F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">874×303 32 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/8881aa0203a1bc048259d2cb6beda11b2afc8055.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8881aa0203a1bc048259d2cb6beda11b2afc8055" title="20"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8881aa0203a1bc048259d2cb6beda11b2afc8055_2_345x119.jpeg" alt="20" data-base62-sha1="jtAJSDolZd4JnzPKfjAoDpvMJLL" width="345" height="119" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8881aa0203a1bc048259d2cb6beda11b2afc8055_2_345x119.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8881aa0203a1bc048259d2cb6beda11b2afc8055_2_517x178.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8881aa0203a1bc048259d2cb6beda11b2afc8055_2_690x238.jpeg 2x" data-dominant-color="F7F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20</span><span class="informations">874×303 51 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed3818982b1f58112bc1e8e4a7fcf52d45c8d426.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ed3818982b1f58112bc1e8e4a7fcf52d45c8d426" title="50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed3818982b1f58112bc1e8e4a7fcf52d45c8d426_2_345x119.jpeg" alt="50" data-base62-sha1="xQxlra9fKnCzrjU91NpS1kr3m62" width="345" height="119" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed3818982b1f58112bc1e8e4a7fcf52d45c8d426_2_345x119.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed3818982b1f58112bc1e8e4a7fcf52d45c8d426_2_517x178.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed3818982b1f58112bc1e8e4a7fcf52d45c8d426_2_690x238.jpeg 2x" data-dominant-color="F8F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">50</span><span class="informations">874×303 45.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/aff3dc6a642e97d2f1d0db26ae5f422d523293b4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/aff3dc6a642e97d2f1d0db26ae5f422d523293b4" title="100"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/aff3dc6a642e97d2f1d0db26ae5f422d523293b4_2_345x119.jpeg" alt="100" data-base62-sha1="p6y1GcBLM5HG1hm0KirtPNQt24Y" width="345" height="119" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/aff3dc6a642e97d2f1d0db26ae5f422d523293b4_2_345x119.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/aff3dc6a642e97d2f1d0db26ae5f422d523293b4_2_517x178.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/aff3dc6a642e97d2f1d0db26ae5f422d523293b4_2_690x238.jpeg 2x" data-dominant-color="F9F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">100</span><span class="informations">874×303 50.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b4f7fe7b9e9998cd2e50d0c6003e4f63b40be562.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b4f7fe7b9e9998cd2e50d0c6003e4f63b40be562" title="180"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4f7fe7b9e9998cd2e50d0c6003e4f63b40be562_2_345x119.jpeg" alt="180" data-base62-sha1="pOVgI5n5Sai3opet8I4pqjv8Zxw" width="345" height="119" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4f7fe7b9e9998cd2e50d0c6003e4f63b40be562_2_345x119.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4f7fe7b9e9998cd2e50d0c6003e4f63b40be562_2_517x178.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4f7fe7b9e9998cd2e50d0c6003e4f63b40be562_2_690x238.jpeg 2x" data-dominant-color="FAF7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">180</span><span class="informations">874×303 41.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> !</p>
</blockquote>