Pytorch rnn error

username: saurabhvyas
cooked: <p>I am getting the following error , trying to create a simple rnn for implementing ctc based neural net   <code>matrices expected, got 3D, 2D tensors</code></p>
<pre><code class="lang-python">import torch 
import torch.nn as nn
import torchvision.datasets as dsets
import torchvision.transforms as transforms
from torch.autograd import Variable
import torch.optim as optim
import numpy as np
import matplotlib
from __future__ import print_function

# Hyper Parameters
sequence_length = 2
input_size = 13
hidden_size = 4
num_layers = 2
num_classes = 3
batch_size = 1
num_epochs = 2
learning_rate = 0.01

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.rnn = nn.RNN(input_size, hidden_size, num_layers)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        # Set initial states 
        h0 = Variable(torch.zeros(self.num_layers, 1 ,1, self.hidden_size)) 
        
        
        # Forward propagate RNN
        out, out2 = self.rnn(x, h0)  
        
        # Decode hidden state of last time step
       # out = self.fc(out[:, -1, :])  
        return out

rnn = RNN(input_size, hidden_size, num_layers, num_classes)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(rnn.parameters(), lr=learning_rate)

input = Variable(torch.randn(sequence_length, 1, 13))
h0 = Variable(torch.randn(2, 1, num_classes))
print(input)
output = rnn.forward(input)

</code></pre>

username: alexis-jacq
cooked: <p>your variable <code>out</code> has 3 dimensions (sequence length, batch, hidden_size), and you must feed your linear layer with a 2 dimensions tensor.</p>
<p>You should, for example, reshape your output like this:</p>
<pre><code class="lang-auto">out = out.view(-1, hidden_size) 
</code></pre>

username: saurabhvyas
cooked: <p>Thanks for your reply , but I dont think thats the problem , that line is commented out , so it should atleast result intermediate out variable , when I try printing that out I get this error</p>

username: sinhasam
cooked: <p>can you post the error message stack?</p>

username: santi-pdp
cooked: <p>I think you got your hidden state Variable with wrong dimensions. I haven’t executed your code, but I think it should be:<br>
<code>h0 = Variable(torch.zeros(self.num_layers, 1 ,self.hidden_size))</code> as it stands for <code>num_layers, batch_size, hidden_size</code>.</p>
<p>Regards</p>

username: saurabhvyas
cooked: <p>This is the error message stack</p>
<p>`    RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
3 h0 = Variable(torch.randn(2, 1, num_classes))<br>
4<br>
----&gt; 5 print( rnn.forward(input))</p>
<p> in forward(self, x)<br>
13<br>
14         # Forward propagate RNN<br>
—&gt; 15         out, out2 = self.rnn(x, h0)<br>
16<br>
17         # Decode hidden state of last time step</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in <strong>call</strong>(self, *input, **kwargs)<br>
204<br>
205     def <strong>call</strong>(self, *input, **kwargs):<br>
–&gt; 206         result = self.forward(*input, **kwargs)<br>
207         for hook in self._forward_hooks.values():<br>
208             hook_result = hook(self, input, result)</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)<br>
89             dropout_state=self.dropout_state<br>
90         )<br>
—&gt; 91         output, hidden = func(input, self.all_weights, hx)<br>
92         if is_packed:<br>
93             output = PackedSequence(output, batch_sizes)</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py in forward(input, *fargs, **fkwargs)<br>
341         else:<br>
342             func = AutogradRNN(*args, **kwargs)<br>
–&gt; 343         return func(input, *fargs, **fkwargs)<br>
344<br>
345     return forward</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py in forward(input, weight, hidden)<br>
241             input = input.transpose(0, 1)<br>
242<br>
–&gt; 243         nexth, output = func(input, hidden, weight)<br>
244<br>
245         if batch_first and batch_sizes is None:</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py in forward(input, hidden, weight)<br>
81                 l = i * num_directions + j<br>
82<br>
—&gt; 83                 hy, output = inner(input, hidden[l], weight[l])<br>
84                 next_hidden.append(hy)<br>
85                 all_output.append(output)</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py in forward(input, hidden, weight)<br>
110         steps = range(input.size(0) - 1, -1, -1) if reverse else range(input.size(0))<br>
111         for i in steps:<br>
–&gt; 112             hidden = inner(input[i], hidden, *weight)<br>
113             # hack to handle LSTM<br>
114             output.append(isinstance(hidden, tuple) and hidden[0] or hidden)</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/_functions/rnn.py in RNNTanhCell(input, hidden, w_ih, w_hh, b_ih, b_hh)<br>
16<br>
17 def RNNTanhCell(input, hidden, w_ih, w_hh, b_ih=None, b_hh=None):<br>
—&gt; 18     hy = F.tanh(F.linear(input, w_ih, b_ih) + F.linear(hidden, w_hh, b_hh))<br>
19     return hy<br>
20</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in linear(input, weight, bias)<br>
447 def linear(input, weight, bias=None):<br>
448     state = _functions.linear.Linear()<br>
–&gt; 449     return state(input, weight) if bias is None else state(input, weight, bias)<br>
450<br>
451</p>
<p>/usr/local/lib/python3.5/dist-packages/torch/nn/<em>functions/linear.py in forward(self, input, weight, bias)<br>
8         self.save_for_backward(input, weight, bias)<br>
9         output = input.new(input.size(0), weight.size(0))<br>
—&gt; 10         output.addmm</em>(0, 1, input, weight.t())<br>
11         if bias is not None:<br>
12             # cuBLAS doesn’t support 0 strides in sger, so we can’t use expand</p>
<p>RuntimeError: matrices expected, got 3D, 2D tensors at /b/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:1232    `</p>

username: saurabhvyas
cooked: <p><a class="mention" href="/u/santi-pdp">@santi-pdp</a> Thanks it worked , I am new to pytorch and deeplearning framework trying hard to learn <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I found out what I did wrong</p>
<p>from docs<br>
" h_0 (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for each element in the batch. "</p>
<p>I got confused between num_layers * num_directions and treated them as two params instead of one , Thanks for help again</p>

username: Will_Zhang
cooked: <p><a class="mention" href="/u/saurabhvyas">@saurabhvyas</a> I’m having similar issues, can you point me to where in the docs you saw num_layers * num_directions? Specifically, what does num_directions refer to here?</p>
<p>thanks.</p>

username: Will_Zhang
cooked: <p>I found the source <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#71" rel="nofollow noopener">here</a>. num_directions == 2 if it’s bi-directional.</p>

username: shirui-japina
cooked: <p>So can the num_directions be 3 or more? for multh bi-directional rnn model?</p>