RNN for sequence prediction

moskomule: <p>Hello,</p>
<p>Previously I used keras for CNN and so I am a newbie on both PyTorch and RNN. In keras you can write a script for an RNN for sequence prediction like,</p>
<pre><code class="lang-python">in_out_neurons = 1
hidden_neurons = 300

model = Sequential()  
model.add(LSTM(hidden_neurons, batch_input_shape=(None, length_of_sequences, in_out_neurons), return_sequences=False))  
model.add(Dense(in_out_neurons))  
model.add(Activation("linear"))  
</code></pre>
<p>but when it comes to PyTorch I don’t know how to implement it. I directly translate the code above into below, but it doesn’t work.</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=seq_len,
                            hidden_size=128,
                            num_layers=1)
        self.dense1 = nn.Linear(128, 1)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = self.dense1(x)
        return x, hidden

    def init_hidden(self, batch_size):
        weight = next(self.parameters()).data
        return Variable(weight.new(128, batch_size, 1).zero_())
</code></pre>
<p>how can I implement something like the keras code? thank you.</p>

apaszke: <p>The <code>input_size</code> argument to any RNN says how many features will there be for each step in a sequence, not what it’s length is going to be. Keras uses static graphs, so it needs to know the length of the sequence upfront, PyTorch has dynamic autodifferentiation so it doesn’t care about the sequence length - you can use a different one every iteration.</p>
<p>See <a href="http://pytorch.org/docs/nn.html#torch.nn.GRU">the GRU docs</a> for more details on the arguments.</p>
<p>Apart from this, your module looks good to me!</p>

moskomule: <p>Thank you for your quick response, but the word <em>features</em> in a context of RNN is still unclear to me. The GRU doc says,</p>
<blockquote>
<p>input : A (seq_len x batch x input_size) tensor containing the features of the input sequence.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>input_size – The number of expected features in the input x</p>
</blockquote>
<p>For example, if you input a sequence</p>
<pre><code class="lang-auto">[[[ 0.1,  0.2]],
 [[ 0.1,  0.2]],
 [[ 0.3,  0.1]]]
</code></pre>
<p>, then <code>seq_len</code> is 3, <code>batch</code> is 1 and <code>input_size</code> i.e. <code>features</code> is 2?</p>

apaszke: <p>Correct. <code>features</code> or <code>input_size</code> says how many dimensions each data point has.</p>

moskomule: <p>Thanks a lot for your help, finally the code below works,</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.autograd import Variable

features = 1
seq_len = 10
hidden_size = 128
batch_size = 32

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=features,
                            hidden_size=hidden_size,
                            num_layers=1)
        self.dense1 = nn.Linear(hidden_size, 1)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(1, seq_len-1).contiguous()
        x = x.view(-1, hidden_size)
        x = self.dense1(x)
        return x, hidden

    def init_hidden(self):
        weight = next(self.parameters()).data
        return Variable(weight.new(1, batch_size, hidden_size).zero_())

model = Net()
model.cuda()
hidden = model.init_hidden()

X_train_1 = X_train[0:batch_size].reshape(seq_len,batch_size,features)
y_train_1 = y_train[0:batch_size]
model.zero_grad()
T = torch.Tensor
X_train_1, y_train_1 = T(X_train_1), T(y_train_1)
X_train_1, y_train_1 = Variable(X_train_1).cuda(), Variable(y_train_1).cuda()

output, hidden = model(X_train_1, Variable(hidden.data))
</code></pre>

moskomule: <p>Thanks for your helping, like I wrote above the script works, “literally” but the loss doesn’t decrease over the epochs, so give me some advice. I think the related parts are,</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self, features, cls_size):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=features,
                            hidden_size=hidden_size,
                            num_layers=1)
        self.dense1 = nn.Linear(hidden_size, cls_size)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(0, maxlen-1).contiguous()
        x = x.view(-1, hidden_size)
        x = F.softmax(self.dense1(x))
        return x, hidden

    def init_hidden(self, batch_size=batch_size):
        weight = next(self.parameters()).data
        return Variable(weight.new(1, batch_size, hidden_size).zero_())

def var(x):
    x = Variable(x)
    if cuda:
        return x.cuda()
    else:
        return x

model = Net(features=features, cls_size=len(chars))
if cuda:
    model.cuda()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)

def train():
    model.train()
    hidden = model.init_hidden()
    for epoch in range(len(sentences) // batch_size):
        X_batch = var(torch.FloatTensor(X[:, epoch*batch_size: (epoch+1)*batch_size, :]))
        y_batch = var(torch.LongTensor(y[epoch*batch_size: (epoch+1)*batch_size]))
        model.zero_grad()
        output, hidden = model(X_batch, var(hidden.data))
        loss = criterion(output, y_batch)
        loss.backward()
        optimizer.step()

for epoch in range(nb_epochs):
    train()
</code></pre>
<p>the input is “one-hot” vector and I tried changing its learning rate but the result is the same.</p>

apaszke: <p>I’m not sure, it’s hard to spot bugs in code that you can’t run. Why do you do this:</p>
<pre><code class="lang-python">x = x.select(0, maxlen-1).contiguous()
</code></pre>
<p>Don’t you want to return predictions for the whole sequence? It seems to me that you’re only taking the last output.</p>

moskomule: <p>In fact I’m trying to (re)implement <a href="https://github.com/fchollet/keras/blob/master/examples/lstm_text_generation.py" rel="nofollow noopener">keras’s text generation example</a> with PyTorch. In keras’s Recurrent layers, there is</p>
<ul>
<li>
<code>return_sequences</code>: Boolean. Whether to return the last output in the output sequence, or the full sequence.</li>
</ul>
<p>and in the example, this is <code>false</code>, so I think taking only the last output is needed.</p>

apaszke: <p>I’m not sure, I don’t know Keras. I’m just pointing it out (it might be easier to do <code>x[-1]</code> to achieve the same thing).</p>
<p>If you have the full code available somewhere I can take a look.</p>

moskomule: <p>OK, thanks. Does</p>
<p><code>x = x[-1]</code> i.e. <code>x = x.select(0, maxlen-1).contiguous()</code></p>
<p>interfere in back propagation?</p>
<p>I uploaded my code <a href="https://github.com/moskomule/pytorch_learning/blob/master/simple/rnn_text_gen.py" rel="nofollow noopener">here</a></p>

apaszke: <p>How would they interfere? They both should be ok.</p>

moskomule: <p>I’m not certain but I use only the last output and so I think this may give bad influence on back prop.<br>
I’ll check the keras again. Thank you.</p>

moskomule: <p>Finally I found that I misused the loss function <code>torch.nn.CrossEntropyLoss</code>. I changed the loss function to <code>nn.NLLLoss(log_softmax(output), target)</code> then the loss decreases as expected.</p>

apaszke: <p>And you removed the <code>softmax</code> from the module, right?</p>

moskomule: <p>Right. So now,</p>
<pre><code class="lang-auto">class Net(nn.module):
    ...
    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(0, maxlen-1).contiguous()
        x = x.view(-1, hidden_size)
        x = F.relu(self.dense1(x))
        x = F.log_softmax(self.dense2(x))
        return x, hidden
...
criterion = nn.NLLLoss()
...
def train():
    model.train()
    hidden = model.init_hidden()
    for epoch in range(len(sentences) // batch_size):
        X_batch = var(torch.FloatTensor(X[:, epoch*batch_size: (epoch+1)*batch_size, :]))
        y_batch = var(torch.LongTensor(y[epoch*batch_size: (epoch+1)*batch_size]))
        model.zero_grad()
        output, hidden = model(X_batch, var_pair(hidden))
        loss = criterion(output, y_batch)
        loss.backward()
        optimizer.step()
</code></pre>

apaszke: <p>Yup, that looks good! Note that you can now pass in <code>hidden = None</code> in the first iteration. The RNN will initialize a zero-filled hidden state for you. You might need to update pytorch though.</p>

CodePothunter: <p>I have a question about the the number of parameters in RNN. I defined a RNN layer and get its paramters. I thought the number of parameters in a RNN layer should differ from different input lengths. However, when I use <code>parameters()</code> to get its parameters, the number of parameters seemed similar to that of the RNN layer with only one time steps.</p>
<p>How to understand this fact? Thank you!</p>

Atcold: <p>Your model is going to be the same, whatever is the length of your input.<br>
In Torch we used to clone the model as many times as the time steps while sharing the parameters, because it is the <strong>same</strong> model, just over time.<br>
The number of parameters will change when your input dimensionality will change (the size of <code>x[t]</code>, for a given <code>t = 1, ..., T</code>), and not when <code>T</code> changes.</p>
<p>If it is still not clear, you can go over my lectures on RNNs (<a href="https://github.com/Atcold/torch-Video-Tutorials#4---recurrent-neural-networks" rel="nofollow noopener">ref.</a>).<br>
And if it is still confusing, wait for the PyTorch video tutorials I’m currently working on.</p>

CodePothunter: <p>I see. Thank you very much!</p>

osm3000: <p>Hi,</p>
<p>Sorry for reopening this topic. I also just moved to PyTorch from Keras, and I am super confused about how RNN works.<br>
I am confused about:</p>
<ol>
<li>I don’t understand what is the ‘batch’ mean in the context of PyTorch</li>
<li>Since RNN can accept variable length sequences, can someone please make a small example about this?</li>
<li>What is the difference between RNN cell and RNN?<br>
<a href="http://pytorch.org/docs/nn.html#torch.nn.RNNCell" rel="nofollow noopener">http://pytorch.org/docs/nn.html#torch.nn.RNNCell</a><br>
<a href="http://pytorch.org/docs/nn.html#rnn" rel="nofollow noopener">http://pytorch.org/docs/nn.html#rnn</a>
</li>
<li>In RNN cell, why the documentation says the input is <code>input (batch, input_size)</code> , while in the example given in the documentation, the input is <code>input = Variable(torch.randn(6, 3, 10))</code> ?</li>
</ol>
<p>Thank you</p>