Dimensionslity in RNN

username: aldeka12
cooked: <p>For instance I have this rnn:</p>
<pre><code class="lang-auto">rnn = torch.nn.RNN(2, 2, 1, batch_first = True)
</code></pre>
<p>and I have input of:<br>
<code>x = torch.tensor([[[1,1], [2,2], [3,3]], [[2,2], [3,3], [4,4]], [[4,4], [5,5], [6,6]]])</code><br>
My model shows that it have this type of parameters:</p>
<pre><code class="lang-auto">weight_ih_l0 tensor([[-0.1641, -0.6958],
        [ 0.1889,  0.4084]])
weight_hh_l0 tensor([[ 0.0063, -0.5073],
        [-0.2890, -0.5403]])
bias_ih_l0 tensor([-0.0039, -0.2850])
bias_hh_l0 tensor([ 0.5279, -0.1149])
</code></pre>
<p>If hidden is initialized as none my output will be like:</p>
<pre><code class="lang-auto">hidden = None 
out, hidden = rnn(x.float(), hidden)
out
tensor([[[-0.3238,  0.1948],
         [-0.8609,  0.6544],
         [-0.9835,  0.8584]],

        [[-0.8324,  0.6611],
         [-0.9836,  0.8553],
         [-0.9976,  0.9480]],

        [[-0.9941,  0.9633],
         [-0.9996,  0.9821],
         [-0.9999,  0.9945]]]
</code></pre>
<p>So this RNN hidden state have this formula:<br>
h_t = tanh(w_{ih} * x_t + b_{ih} + w_{hh} * h_{(t-1)} + b_{hh})<br>
but w_{ih} will have shape 2X2 and x_t will have shape 1X2 (for instance first input [1,1]), how then they multiple and work? Or is it work differently in pytorch?<br>
I am asking this question because I tried to recreate this in numpy, but this call error due to inappropriate dimensions in this part, which is right from lin. alg. point of view.</p>
<pre><code class="lang-auto">h_t = numpy.tanh(hh@hidden + ih@numpy.matrix([1,1]) + bias_ih + bias_hh)
</code></pre>

username: aldeka12
cooked: <p>When I tried recreated in numpy I just copied for instance like this:</p>
<pre><code class="lang-auto">weight_ih_l0 tensor([[-0.1641, -0.6958],
       [ 0.1889,  0.4084]])

then
ih = numpy.matrix([[-0.1641, -0.6958],
       [ 0.1889,  0.4084]])

bias_ih = ([-0.0039, -0.2850])
and so on
</code></pre>
<p>But when I reshaped my bias and my input like this:</p>
<pre><code class="lang-auto">bias_ih = numpy.matrix([[-0.0039],  [-0.2850])
(and another bias)
h_t = numpy.tanh(hh@hidden + ih@numpy.matrix([[1],[1]]) + bias_ih + bias_hh) 
</code></pre>
<p>it is started to work correctly and match pytorch outputs, so is there auto-reshaper of some kind or pytorch have some kind of different printed shape(sorry for reinventing words :/)?</p>

username: AbdulsalamBande
cooked: <p>Okay aldeka, starting from the creation of your instance, it means <code>rnn = torch.nn.RNN(2, 2, 1, batch_first = True)</code>, starting from the left, it means that that your input size =2 (meaning each sequence has to values), 2 meaning that each of your input sequence will have an output that 2 values long and 1 means you just have 1 layer in your network. So the dimension of your input is [3,3,2] starting from the right 2 = feature size (meaning each of your input sequences has 2 features), 3 =  number of sequences in each batch, 3 = batch_size(and each batch has 3 sequences which are all 2 vectors long). Just for your information each batch will have its hidden vector. So lets try and calculate the output of the the first input  of the first batch using the formula from the pytorch documentation. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e44b30919ed2057f74f1d7f0d5faae8fa567b0ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/e44b30919ed2057f74f1d7f0d5faae8fa567b0ef" title="Screen Shot 2020-03-23 at 16.19.05"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e44b30919ed2057f74f1d7f0d5faae8fa567b0ef_2_690x81.png" alt="Screen Shot 2020-03-23 at 16.19.05" data-base62-sha1="wzzXiMrAf4nfiAvYUR2VnPfYsHB" width="690" height="81" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e44b30919ed2057f74f1d7f0d5faae8fa567b0ef_2_690x81.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e44b30919ed2057f74f1d7f0d5faae8fa567b0ef_2_1035x121.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e44b30919ed2057f74f1d7f0d5faae8fa567b0ef.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-03-23 at 16.19.05</span><span class="informations">1250×148 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>W_{ih} = <code>torch.tensor([[ 0.0063, -0.5073],         [-0.2890, -0.5403]])</code></p>
<ul>
<li>The dimension is [2,2]. It has 2 rows because the output_size is 2 and it has 2 columns because the every input has has 2 features. So in general, the dimension of the weight of an input is [output_size, input_feature_size].</li>
</ul>
<p>X{t} = <code>torch.tensor([[[1, 1]]])</code></p>
<ul>
<li>This is the dimension is <code>[1,1,2]</code> , but we would have to reshape it to <code>[2,1]</code> to obey the  matrix dot rule. we can reshape by first saying <code> x= x.squeeze(0)</code> and saying <code>x = x.t()</code> , the latter one is to get the transpose.</li>
<li>Finally it will become <code>tensor([[1],       [1]])</code></li>
</ul>
<p>b{ih} = <code>torch.tensor([-0.0039, -0.2850])</code></p>
<ul>
<li>The dimension of the bias of the input is just [2], so you would have to reshape it to <code>[2,1]</code> by saying <code>b = b.reshape(2,1</code>).</li>
</ul>
<p>W{hh} = <code>torch.tensor([[ 0.0063, -0.5073],         [-0.2890, -0.5403]])</code></p>
<ul>
<li>The dimension of this is<code> [2,2]</code>, this is always a square matrix. And in general its dimension is <code> [hidden_size,hidden_size]</code></li>
</ul>
<p>h{t_1} = In your case you did not create the hidden yourself but the dimension of hidden is always <code>[number_of_layers,batch_size, hidden_size]</code></p>
<ul>
<li>By the way each batch of input will have its own hidden vector and each hidden vector size will depend on the number of output features you have. In your case, instead of saying the hidden to be <code>none</code> , you can create it as <code>hidden = torch.rand(1,3,2)</code>. Starting from the left , 1 = number of layers, 3 = number of batches from input , and lastly each of the hidden vector of each batch is has a size of 2.</li>
<li>So lets assume it is <code>torch.tensor([[[0.8888, 0.8303],        [0.9467, 0.4717],        [0.1862, 0.6796]]]) </code> , Meaning the hidden <code>[0.8888, 0.8303]</code> for the first batch ,  the hidden <code>[0.9467, 0.4717]</code> is for the second batch and the hidden <code>  [0.1862, 0.6796]</code> is for the 3rd batch.</li>
</ul>
<p>b{hh} = <code>torch.tensor([ 0.5279, -0.1149])</code></p>
<ul>
<li>The dimension of the bias of the hidden is just [2], so you would have to reshape it to <code>[2,1]</code> by using this code <code> b = b.reshape(2,1)</code></li>
</ul>
<p><strong>Finally,</strong> using the above formula, the dimension would be</p>
<ul>
<li>
<p>``( [2,2] * [2,1] )+ [2,1] + ([ 2,2 ] * [2,1] )+ [2,1])</p>
</li>
<li>
<p><code>( torch.tensor([[ 0.0063, -0.5073],      [-0.2890, -0.5403]]) * torch.tensor([[1],      [1]]) )</code>.   —&gt; which is  w_{ih} * x  .    Lets call this <strong>a1</strong></p>
</li>
<li>
<p><code>torch.tensor([-0.0039, -0.2850])</code> – which is b{ih} – call this <strong>a2</strong></p>
</li>
<li>
<p><code>torch.tensor(torch.tensor([[ 0.0063, -0.5073],     [-0.2890, -0.5403]]) *[0.8888, 0.8303] ) </code> → w{hh}* h{t-1} – call this <strong>a3</strong></p>
</li>
<li>
<p><code>torch.tensor([ 0.5279, -0.1149])</code> → b{hh} --call this <strong>a3</strong></p>
</li>
</ul>
<p><strong>FINAL RESULT IS</strong>  <code>a1+a2+a3+a4</code></p>