Linear layers on top of LSTM

douwe: <SCODE>local dec = nn.Sequential()
dec:add(nn.LookupTable(opt.vocabSize, opt.hiddenSize))
dec:add(nn.Sequencer(nn.LSTM(opt.hiddenSize, opt.hiddenSize))
dec:add(nn.Sequencer(nn.Linear(opt.hiddenSize, opt.vocabSize)))
dec:add(nn.Sequencer(nn.LogSoftMax()))
<ECODE> <SCODE>class Net(nn.Module):
    def __init__(self, vocabSz, hiddenSz):
        super(Net, self).__init__()
        self.emb = nn.Embedding(vocabSz, hiddenSz)
        self.dec = nn.LSTM(hiddenSz, hiddenSz)
        self.lin = nn.Linear(hiddenSz, vocabSz)
    def forward(self, input, hidden):
        embeddings = self.emb(input)
        out, hidden = self.dec(embeddings, hidden)
        # do the linear here for each lstm output.. what's the best way?
        return F.softmax(out), hidden<ECODE>

apaszke: I’d do this: <SCODE>out, hidden = self.dec(embeddings, hidden)
out = self.lin(out.view(-1, out.size(2))
return F.softmax(out)
<ECODE>

Atcold: I think you need <SCODE>return F.softmax(out), hidden
<ECODE>

douwe: edited example code – typo when transcribing, thanks.

csarofeen: 

mfa: 

simono: Yes, it just basically folds together the first two dimensions, the third will stay unaffected. Example output from my code: <SCODE>out
Variable containing:
( 0 ,.,.) = 
  0.0426 -0.2047 -0.0125  ...  -0.0409  0.1487  0.0376
 -0.0744 -0.0927  0.0185  ...  -0.1620  0.1348 -0.2179
  0.0111 -0.1565 -0.0192  ...  -0.1247  0.0352 -0.0625

( 1 ,.,.) = 
  0.0726 -0.1911  0.3049  ...  -0.1031  0.1991  0.0659
 -0.2233  0.1622  0.0794  ...  -0.1720  0.1020 -0.0430
  0.0600 -0.3536  0.0201  ...  -0.0990 -0.1203 -0.1207

( 2 ,.,.) = 
  0.0000  0.0000  0.0000  ...   0.0000  0.0000  0.0000
 -0.1225  0.1387  0.1205  ...  -0.1052 -0.1638 -0.2314
  0.0336 -0.3169  0.0090  ...  -0.0343  0.0635 -0.0280

( 3 ,.,.) = 
  0.0000  0.0000  0.0000  ...   0.0000  0.0000  0.0000
  0.0001 -0.1604  0.1111  ...  -0.0778 -0.1291 -0.0852
  0.0000  0.0000  0.0000  ...   0.0000  0.0000  0.0000
[torch.FloatTensor of size 4x3x128]

out.view(-1, out.size(2))
Variable containing:
 0.0426 -0.2047 -0.0125  ...  -0.0409  0.1487  0.0376
-0.0744 -0.0927  0.0185  ...  -0.1620  0.1348 -0.2179
 0.0111 -0.1565 -0.0192  ...  -0.1247  0.0352 -0.0625
          ...             ⋱             ...          
 0.0000  0.0000  0.0000  ...   0.0000  0.0000  0.0000
 0.0001 -0.1604  0.1111  ...  -0.0778 -0.1291 -0.0852
 0.0000  0.0000  0.0000  ...   0.0000  0.0000  0.0000
[torch.FloatTensor of size 12x128]

<ECODE>

live-wire: 

Shisho_Sama: If your inputs are independent(like for generating baby names where each name is a separate thing) then yes, there is no need to do this, however, when the the inputs are related, you’ll need to forward the states as well. for example you want to produce Shakespearean text, or something to to that extend, you’ll benefit from doing so.

shahensha: <SCODE>voc_size = 100
n_labels = 3
emb_dim = 16
rnn_size = 32
embedding = nn.Embedding(voc_size, emb_dim)
rnn = nn.LSTM(input_size=emb_dim, hidden_size=rnn_size, bidirectional=True, num_layers=1)
top_layer = nn.Linear(2 * rnn_size, n_labels)

sentences = torch.randint(high=voc_size, size=(10, 4))
print(sentences.shape)

embedded = embedding(sentences)
print(embedded.shape)

rnn_out, _ = rnn(embedded)
print(rnn_out.shape)

out = top_layer(rnn_out)
print(out.shape)
<ECODE> The output is as follows: <SCODE>torch.Size([10, 4])
torch.Size([10, 4, 16])
torch.Size([10, 4, 64])
torch.Size([10, 4, 3])
<ECODE>