CNN with LSTM input shapes

shakeel608: <p>I am trying to combine CNN and LSTM for the audio data.<br>
Let us say the output of my CNN model is <code>torch.Size([8, 1, 10, 10]</code> which is [B X C_out X Frequency X Time ]<br>
and the LSTM requires [L X B X InputSize].</p>
<p>My question is what is the inputSize in LSTM and how shall I feed the output of CNN to the LSTM</p>
<p>Please help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <aside class="quote no-group" data-username="shakeel608" data-post="1" data-topic="86910">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a587f6/48.png" class="avatar"> shakeel608:</div>
<blockquote>
<p>My question is what is the inputSize in LSTM</p>
</blockquote>
</aside>
<p>The mentioned <code>inputSize</code> in your shape information would correspond to the “feature” dimension.</p>
<p>Since your CNN output is 4-dimensional, you would have to decide which dimensions are corresponding to the temporal dimensions and which to the features.</p>
<p>Assuming you would like to use <code>C_out</code> and <code>Fequency</code> as the features, you could use:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(8, 1, 10, 10)
x = x.view(x.size(0), -1, x.size(3)) # [batch_size, features=channels*height, seq_len=width]
x = x.permute(2, 0, 1) # [seq_len, batch_size, features]
</code></pre>
<p>and pass it to the RNN.</p>
<p>PS: Please don’t tag certain people, as this might discourage others to post a solution <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

shakeel608: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, This is what I was looking for .</p>
<p>Suppose after feeding the data input to the CNN model, it outputs variable length like the example below.</p>
<pre><code class="lang-auto">CNN Shape==========&gt; torch.Size([2, 128, 5, 28])
CNN Shape==========&gt; torch.Size([2, 128, 9, 28])
</code></pre>
<p>Now When I am feeding this to the LSTM after performing the below mentioned operations</p>
<pre><code class="lang-auto">x = x.view(x.size(0), -1, x.size(3)) # [batch_size, features=channels*height, seq_len=width]
x = x.permute(2, 0, 1) # [seq_len, batch_size, features]
</code></pre>
<p>It is giving the error<br>
<code>input.size(-1)</code> must be equal to input_size.</p>
<p><em><strong>How the variable length from the CNNs is handled before feeding it the LSTM</strong></em></p>
<p>Sure I will be careful in future.<br>
Actually your explanations are very clear and to the point and I really enjoy those.</p>

ptrblck: <p>In an RNN the temporal dimension is variable, not the feature dim.<br>
You could use the channels (dim1) as the feature dimension and the <code>height*width</code> as the temporal dimension as a workaround.<br>
However, based on your description you would like to use the width as the time dim: <code>[B X C_out X Frequency X Time ]</code>.</p>

shakeel608: <p>Sorry<br>
you are right<br>
the temporal dimension is variable as</p>
<pre><code class="lang-auto">CNN Shape==========&gt; torch.Size([16, 128, 40, 21])
CNN Shape==========&gt; torch.Size([16, 128, 40, 28])
</code></pre>
<p>This is what I get output from the CNN.<br>
Now how do we handle this variable length in RNNs (this is on the fly training)</p>

ptrblck: <p>Same as before: make sure to pass the inputs to the RNN as <code>[seq_len, batch_size, features]</code>.<br>
If dim3 is now the time dimension and (dim1+dim2) are the features:</p>
<pre><code class="lang-python">x = x.view(x.size(0), -1, x.size(3))
x = x.permute(2, 0, 1)
</code></pre>

ananda2020: <p>Hi ptrblck,<br>
During permutation I had to put (0, 2, 1) to match with <code>batch_size, seq_length, out_channels</code> in my case.<br>
For video classification, my variable factor is batch_size so changing batch_size I can control the temporal part of a video. seq_length is coming from the previous block as a part of feature vector which I can’t change. I am little confused here. As temporal part should be controlled by seq_length not by the batch_size.<br>
Please let me know. Thank you once again.</p>
<p>Regards,<br>
ananda2020</p>

ptrblck: <p>While the batch size can very, it doesn’t represent the temporal dimension, but just how many samples you are processing at once. If your <code>seq_length</code> is static, you are not working with a variable temporal dimension.</p>
<p>Make sure to permute the input to the expected dimensions. By default RNNs expect an input of <code>[seq_len, batch_size, features]</code>. With <code>batch_first=True</code> the input should be <code>[batch_size, seq_len, features]</code>.</p>

ananda2020: <p>I am extracting frames from videos. Then each frame is fed into the CNN to get the features and the output from CNN is fed into LSTM. So how can I change the seq_length?</p>
<p>Thanks in advance.<br>
Regards,<br>
ananda2020</p>

ptrblck: <p>I assume your CNN creates features in the shape <code>[batch_size, features]</code> and you would like to use the batch size as the temporal dimension, since you made sure that the ordering of the input images is appropriate for the use case.<br>
If that’s the case, just <code>unsqueeze</code> a fake batch dimension in dim1 and pass the outputs to the RNN.</p>

ananda2020: <p>Thank you ptrblck. Yes, I named the frames such way that they are sequenced. Thank you once again.</p>
<p>Regards,<br>
Alakananda</p>

shakeel608: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , it worked as per the requirement</p>

ananda2020: <p>Hi ptrblck,<br>
I was wrong. My dataloader was not taking sequenced data. So I added a sample to get the sequence.<br>
Thanks for posting the sampler code in another thread.</p>
<p>Regards,<br>
ananda2020</p>

shakeel608: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have question regarding the hidden units of LSTM</p>
<pre><code class="lang-auto">self.lstm = nn.LSTM(
                    input_size = 64,
                    hidden_size = 128,
                    num_layers  = 2)
</code></pre>
<p>Since the hidden units of LSTM are fixed 128. how does it handle the variable length inputs. It is bit confusing ?<br>
Every time the input takes the batch, its input sequence length changes. so how does it handle it ?</p>

ptrblck: <p>The hidden size is defining the “feature dimension” of the input and is thus unrelated to the temporal dimension.<br>
<a href="http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture10.pdf">This lecture on RNNs</a> gives you a good overview how these shapes are used.</p>

shakeel608: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
this was very helpful</p>