RNN for sequence prediction

moskomule: Hello, Previously I used keras for CNN and so I am a newbie on both PyTorch and RNN. In keras you can write a script for an RNN for sequence prediction like, <SCODE>in_out_neurons = 1
hidden_neurons = 300

model = Sequential()  
model.add(LSTM(hidden_neurons, batch_input_shape=(None, length_of_sequences, in_out_neurons), return_sequences=False))  
model.add(Dense(in_out_neurons))  
model.add(Activation("linear"))  
<ECODE> but when it comes to PyTorch I don’t know how to implement it. I directly translate the code above into below, but it doesn’t work. <SCODE>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=seq_len,
                            hidden_size=128,
                            num_layers=1)
        self.dense1 = nn.Linear(128, 1)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = self.dense1(x)
        return x, hidden

    def init_hidden(self, batch_size):
        weight = next(self.parameters()).data
        return Variable(weight.new(128, batch_size, 1).zero_())
<ECODE> how can I implement something like the keras code? thank you.

apaszke: Apart from this, your module looks good to me!

moskomule: and For example, if you input a sequence <SCODE>[[[ 0.1,  0.2]],
 [[ 0.1,  0.2]],
 [[ 0.3,  0.1]]]
<ECODE>

apaszke: 

moskomule: Thanks a lot for your help, finally the code below works, <SCODE>import torch
import torch.nn as nn
from torch.autograd import Variable

features = 1
seq_len = 10
hidden_size = 128
batch_size = 32

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=features,
                            hidden_size=hidden_size,
                            num_layers=1)
        self.dense1 = nn.Linear(hidden_size, 1)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(1, seq_len-1).contiguous()
        x = x.view(-1, hidden_size)
        x = self.dense1(x)
        return x, hidden

    def init_hidden(self):
        weight = next(self.parameters()).data
        return Variable(weight.new(1, batch_size, hidden_size).zero_())

model = Net()
model.cuda()
hidden = model.init_hidden()

X_train_1 = X_train[0:batch_size].reshape(seq_len,batch_size,features)
y_train_1 = y_train[0:batch_size]
model.zero_grad()
T = torch.Tensor
X_train_1, y_train_1 = T(X_train_1), T(y_train_1)
X_train_1, y_train_1 = Variable(X_train_1).cuda(), Variable(y_train_1).cuda()

output, hidden = model(X_train_1, Variable(hidden.data))
<ECODE>

moskomule: Thanks for your helping, like I wrote above the script works, “literally” but the loss doesn’t decrease over the epochs, so give me some advice. I think the related parts are, <SCODE>class Net(nn.Module):
    def __init__(self, features, cls_size):
        super(Net, self).__init__()
        self.rnn1 = nn.GRU(input_size=features,
                            hidden_size=hidden_size,
                            num_layers=1)
        self.dense1 = nn.Linear(hidden_size, cls_size)

    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(0, maxlen-1).contiguous()
        x = x.view(-1, hidden_size)
        x = F.softmax(self.dense1(x))
        return x, hidden

    def init_hidden(self, batch_size=batch_size):
        weight = next(self.parameters()).data
        return Variable(weight.new(1, batch_size, hidden_size).zero_())

def var(x):
    x = Variable(x)
    if cuda:
        return x.cuda()
    else:
        return x

model = Net(features=features, cls_size=len(chars))
if cuda:
    model.cuda()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)

def train():
    model.train()
    hidden = model.init_hidden()
    for epoch in range(len(sentences) // batch_size):
        X_batch = var(torch.FloatTensor(X[:, epoch*batch_size: (epoch+1)*batch_size, :]))
        y_batch = var(torch.LongTensor(y[epoch*batch_size: (epoch+1)*batch_size]))
        model.zero_grad()
        output, hidden = model(X_batch, var(hidden.data))
        loss = criterion(output, y_batch)
        loss.backward()
        optimizer.step()

for epoch in range(nb_epochs):
    train()
<ECODE> the input is “one-hot” vector and I tried changing its learning rate but the result is the same.

apaszke: I’m not sure, it’s hard to spot bugs in code that you can’t run. Why do you do this: <SCODE>x = x.select(0, maxlen-1).contiguous()
<ECODE> Don’t you want to return predictions for the whole sequence? It seems to me that you’re only taking the last output.

moskomule: 
return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence.

apaszke: If you have the full code available somewhere I can take a look.

moskomule: OK, thanks. Does interfere in back propagation?

apaszke: How would they interfere? They both should be ok.

moskomule: 

moskomule: 

apaszke: 

moskomule: Right. So now, <SCODE>class Net(nn.module):
    ...
    def forward(self, x, hidden):
        x, hidden = self.rnn1(x, hidden)
        x = x.select(0, maxlen-1).contiguous()
        x = x.view(-1, hidden_size)
        x = F.relu(self.dense1(x))
        x = F.log_softmax(self.dense2(x))
        return x, hidden
...
criterion = nn.NLLLoss()
...
def train():
    model.train()
    hidden = model.init_hidden()
    for epoch in range(len(sentences) // batch_size):
        X_batch = var(torch.FloatTensor(X[:, epoch*batch_size: (epoch+1)*batch_size, :]))
        y_batch = var(torch.LongTensor(y[epoch*batch_size: (epoch+1)*batch_size]))
        model.zero_grad()
        output, hidden = model(X_batch, var_pair(hidden))
        loss = criterion(output, y_batch)
        loss.backward()
        optimizer.step()
<ECODE>

apaszke: 

CodePothunter: How to understand this fact? Thank you!

Atcold: 

CodePothunter: I see. Thank you very much!

osm3000: Hi, Thank you