Need advice on how to modify the GRU to made it work with raw vibration data

Mystify2823: <p>Hello,</p>
<p>I am trying to feed raw vibration to a GRU layer. The idea of feeding these vibration raw data, is so the GRU layer extract the features of the raw vibration data, so it can optimize the feature extraction to get better prediction.</p>
<p>This is the GRU layer I have:</p>
<pre><code class="lang-auto">class GruPeakFR(nn.Module):
    def __init__(self, num_inputs, num_hiddens=1, sigma=0.01):
        super().__init__()
        self.num_hiddens = num_hiddens
        init_weight = lambda * shape: nn.Parameter(torch.randn(*shape) * sigma)
        
        # Calculate the size of the rfft output
        self.rfft_size = num_inputs # // 2 + 1
        
        triple = lambda: (init_weight(num_inputs, num_hiddens),
                          init_weight(num_hiddens, num_hiddens),
                          nn.Parameter(torch.zeros(num_hiddens)))
        self.W_xz, self.W_hz, self.b_z = triple()  # Update gate
        self.W_xr, self.W_hr, self.b_r = triple()  # Reset gate
        self.W_xh, self.W_hh, self.b_h = triple()  # Candidate hidden state
        self.input_dropout = nn.Dropout(0.0)
        self.input_dropout = nn.Dropout(0.0)

    def forward(self, inputs, H=None):
        if H is None:
            H = torch.zeros((inputs.shape[1], self.num_hiddens), device=inputs.device)
        outputs = []
        for X in inputs:
            fft_x = torch.fft.fft(X).real
            update_gate = (torch.matmul(fft_x, self.W_xz) +
                            torch.matmul(H, self.W_hz) + self.b_z)
            Z_fft = torch.sigmoid(update_gate)
            Z = torch.fft.ifft(Z_fft).real
            Z_peak = calculate_the_peak(Z)
            reset_gate = (torch.matmul(fft_x, self.W_xr) +
                            torch.matmul(H, self.W_hr) + self.b_r)
            R_fft = torch.sigmoid(reset_gate)
            R = torch.fft.ifft(R_fft).real
            R_peak = calculate_the_peak(R)
            
            # Z = torch.fft.irfft(Z_fft, n=X.size(-1))

            candidate_hidden_gate = torch.matmul(fft_x, self.W_xh) + torch.matmul(R* H, self.W_hh) + self.b_h
            candidate_hidden_gate = torch.fft.ifft(candidate_hidden_gate).real
            # candidate_hidden_gate = calculate_the_peak(candidate_hidden_gate)
            
            H_tilde = torch.tanh(candidate_hidden_gate)
            H_tilde = calculate_the_peak(H_tilde)
            H = Z_peak * H + (1 - Z_peak) * H_tilde
            # print(H)
            outputs.append(H)
            
            # print("Z shape:", Z.shape)
            # print("R shape:", R.shape)
            # print("H shape:", H.shape)
            # print("R * H shape:", (R * H).shape)
            # print("self.W_hh shape:", self.W_hh.shape)
            
        outputs = torch.stack(outputs, dim=0)
        if outputs.isnan().any():
            print("Gru Peak giving NaN values")
        return outputs, H
</code></pre>
<p>As the sigmoid function destroy the vibration signal, I use the FFT real values to prevent this. I also tried by getting the absolute value (the magnitude), After the sigmoid and tanh function, I extract the max value of the recreated vibration signal with the ifft.</p>
<p>It does train, however it fails on the test set. I was thinking that this layer increase the risk of overfitting and tried different methods to prevent overfitting, but with no avail. I’m still new with working with RNN and trying to change their way how their work. If there are people more used to GRU that can give some tips about creating your own custom GRU layers to tailor it to my task.</p>
<p>One possibility is to change the layer by a Convolutional and changing the kernel, but I would like to give this GRU one more shoot before giving up on it. If you have also resources regarding GRU that can help me understand better how to custom them, I am all ears.</p>
<p>Thanks !</p>

anantguptadbl: <aside class="quote no-group" data-username="Mystify2823" data-post="1" data-topic="216261">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mystify2823/48/61564_2.png" class="avatar"> Mystify2823:</div>
<blockquote>
<p><code>fft_x = torch.fft.fft(X).real</code></p>
</blockquote>
</aside>
<p>Instead of a matmul on real values, can you try masking a percentage of the data with 0 and 1, because there are papers that talk about the efficacy of masking data this way</p>

Mystify2823: <p>O interesting, I will look into this masking. Do you have links for these papers so I can read them ?</p>
<p>Is it this one : <a href="https://ieeexplore.ieee.org/document/8404456" class="inline-onebox" rel="noopener nofollow ugc">Online additive updates with FFT-IFFT operator on the GRU neural networks | IEEE Conference Publication | IEEE Xplore</a> ?</p>

anantguptadbl: <p><a class="mention" href="/u/mystify2823">@Mystify2823</a>  there are many papers and articles on this</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.ml-science.com/masking">
  <header class="source">
      <img src="https://images.squarespace-cdn.com/content/v1/5acbdd3a25bf024c12f4c8b4/1600024348668-O52EU3A2Y2J0LZUUTBMZ/favicon.ico?format=100w" class="site-icon" width="100" height="97">

      <a href="https://www.ml-science.com/masking" target="_blank" rel="noopener nofollow ugc">The Science of Machine Learning &amp; AI</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:110/107;"><img src="http://static1.squarespace.com/static/5acbdd3a25bf024c12f4c8b4/t/5f88da482db31f49c18e2fff/1602804300698/SML+Favicon.png?format=1500w" class="thumbnail" width="110" height="107"></div>

<h3><a href="https://www.ml-science.com/masking" target="_blank" rel="noopener nofollow ugc">Masking — The Science of Machine Learning &amp; AI</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It is used in places where you want the model to learn patterns without overfitting</p>