What is the correct way of implmenting a simple subsampling RNN network?

knoriy: <p>I’m trying to implement an RNN with Hierarchical Subsampling as shown <a href="https://www.cs.toronto.edu/~graves/preprint.pdf" rel="noopener nofollow ugc">here</a> [chapter 9.0]</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9af0ff3d95d4e0dca807bc47faa02fd389946a2f.png" data-download-href="https://discuss.pytorch.org/uploads/default/9af0ff3d95d4e0dca807bc47faa02fd389946a2f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9af0ff3d95d4e0dca807bc47faa02fd389946a2f_2_690x499.png" alt="image" data-base62-sha1="m6FRA1VDI5qjl72ClrfL6HC9Wzl" width="690" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9af0ff3d95d4e0dca807bc47faa02fd389946a2f_2_690x499.png, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9af0ff3d95d4e0dca807bc47faa02fd389946a2f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9af0ff3d95d4e0dca807bc47faa02fd389946a2f.png 2x" data-dominant-color="F5F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">920×666 82.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Is my understanding of how to implement this correct? I feel that this is too simple</strong></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class subsampleRNN(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()

        self.lstm1 = nn.LSTM(512, 256)
        self.lstm2 = nn.LSTM(256, 128)
        self.lstm3 = nn.LSTM(128, 64)
        
        self.relu = nn.ReLU()
        
         
    def forward(self, x):
        x, h0 = self.lstm1(x)
        x = self.relu(x)
        x, h0 = self.lstm2(x)
        x = self.relu(x)
        x, h0 = self.lstm3(x)
        
        return x
    
    
model = subsampleRNN()
input = torch.rand(1,1,512)
model(input).shape
</code></pre>

googlebot: <p>No, described subsampling is in time dimension, it should be the same to:</p>
<p>lstm1 = nn.LSTM(512, 256)<br>
lstm2 = nn.LSTM(<strong>512</strong>, arbitrary_width)</p>
<p>if you permute+reshape (time=2,width=256) windows → (1,512) between layers</p>

knoriy: <p>Thank you for your answer <a class="mention" href="/u/googlebot">@googlebot</a>. I’m not sure if I understand fully. Can you kindly expand on your previous post?</p>

googlebot: <p>So, you have lstm1=LSTM(I,H1) that maps shapes as (T,B,I) → (T,B,H1), where hidden size H1 is arbitrary, (T,B,I) can be a shape of network input.</p>
<p>Now you want to process shorter sequences in lstm2. You have options:</p>
<ol>
<li>Drop some information by using stride&gt;1, e.g. x[::2,:,:]</li>
<li>Use pooling or something similar</li>
<li>Do what your text suggests and use weighted combinations (if I understood correctly from quick skimming). So, you reshape data as (T,B,H1) → (T/window_size,B,H1*window_size) and do bigger steps in lstm2. To concatenate chunks in T dimension, you can instead .reshape() a permuted (B,T,H1) tensor and permute it back (as time-major lstm2 may be faster).</li>
</ol>