Input shape to GRU layer

talhaanwarch: <p>Greetings. I am getting confused about the input shape to GRU layer.<br>
I have one batch of 128 images and I extracted 9 features from each images.<br>
So now my shape is <code>(1,128,9)</code>.<br>
Here 1 is batch, 128 images in each batch and 9 features of each images. The images is in sequence, for example 128 frame of a video. So simply one batch represent one video.</p>
<p>This is the GRU layer</p>
<pre><code class="lang-auto">gru=torch.nn.GRU(input_size=128,hidden_size=8,batch_first=True)
</code></pre>
<p><strong>Question 1: Is the <code>input_size=128</code> correctly defined</strong>?<br>
Here is the code of forward function</p>
<pre><code class="lang-auto">def forward(features):
    features=features.permute(0,2,1)#[1, 9, 128]
    x2,_=self.gru(features)
</code></pre>
<p><strong>Question 2: Is the code in forward function is correctly defined</strong>?</p>
<p><strong>Question 3: In shape <code>(1,128,9)</code>, 9 is sequence length and 128 is the input. Is this correct or its the opposite way</strong>?</p>
<p>Thanks</p>

J_Johnson: <p>You also need a hidden state of size (num_layers, batch_size, output_size), which can start out as zeros and evolve at each time step.</p>
<p>See here for an example:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.GRU.html</a></p>

J_Johnson: <p>I saw you updated your question. This is the example given on the docs page:</p>
<pre><code class="lang-auto">rnn = nn.GRU(10, 20, 2)
input = torch.randn(5, 3, 10)
h0 = torch.randn(2, 3, 20)
output, hn = rnn(input, h0)
</code></pre>
<p>Note that you’re still missing the hidden state <code>h0</code> in your code.</p>
<p>The GRU layer expects <strong>two</strong> inputs. The hidden input is how you convey contextual information to the model between timesteps.</p>
<p>Now onto your questions.</p>
<p>Q1. No, that is incorrect to have batch size at <code>dim=1</code> when you have clearly set <code>batch_first=True</code>. You have to choose how you want to send your data in and get it out, then set that argument accordingly. If you set <code>batch_first=True</code>, your input size should be (128, 1, 9).</p>
<p>Q2. No. You’re not giving the model a hidden state.</p>
<p>Q3. The order can be N, L, H with <code>batch_first=True</code> or L, N, H if <code>False</code> where L is length, N is batch size and H is number of features.</p>
<p>Last issue, when you instantiated the GRU class, you put the input size as 128, which you have indicated to be your batch size. That is incorrect. Input size should equal the number of features, in this case, 9.</p>

vdw: <p><code>input_size</code> refers to the number of features, in your case 9. You don’t specify the sequence length when defining an <code>nn.GRU</code> or <code>nn.LSTM</code>. Recurrent Neural Networks go through each item in your sequence no matter how long.</p>
<p><code>hidden_size</code> is the size of the internal state of your <code>nn.GRU</code>. You probably want to set it to a larger value than 9 :), is it might have to capture a lot of information.</p>
<p>So you should define your recurrent layer as follows:</p>
<pre><code class="lang-auto">gru=torch.nn.GRU(input_size=9, hidden_size=256, batch_first=True) # The 256 is just a suggestion
</code></pre>
<p>Since your set <code>batch_first=True</code>, you <code>gru</code> layer expects an input of <code>(batch_size, seq_len, input_size)</code>, which in your example would be <code>(1, 128, 9)</code>. As this seems already to be your shape, there should be no need to permute the dimensions.</p>

vdw: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> You don’t have to explicitly give an initial hidden state as input – although it’s probably a good practice to always do so.</p>
<p>If you don’t specify an initial hidden state, it gets initialized with a 0-vector. You can check the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#GRU" rel="noopener nofollow ugc">source code</a>. The signature for the <code>forward()</code> method is</p>
<pre><code class="lang-auto">def forward(self, input: Tensor, hx: Optional[Tensor] = None) -&gt; Tuple[Tensor, Tensor]:
</code></pre>
<p>So <code>hx</code> is optional. If it is <code>None</code>, then it gets initialized directly in the <code>forward()</code> method as follows: <code>hx = torch.zeros(...)</code></p>

J_Johnson: <p>I just assume someone making use of a GRU with a sequence length of 1 intends to pass in later timesteps via their model’s forward method. So while that might not give any errors, it sure would negate any benefit of using an RNN class layer.</p>

vdw: <p>I thought 1 is the <code>batch_size</code> is the <code>seq_len</code>. But, yeah, no reading “I have a batch of 128 images” again, I’m not sure sure anymore <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

talhaanwarch: <p>Hi, Thanks for the detailed answer. sorry my bad. actually my batch size is 1, each batch has 128 images, and each image gas 9 features.<br>
Second thing if we dont defined hidden state, wont it take it automatically, because it did not throw any error? thanks</p>

talhaanwarch: <p>apology my side. my batch size is 1, each batch has 128 images and each image has 9 features. for example if i have 2 batch size then the shape would be <code>(2,128,9)</code></p>