Implementation of Seq2Seq models for sentiment classification

akuysal: <p>I’m trying to use Seq2Seq models for sentiment classification. I can execute an implementation using torchtext with FastText model on IMDB dataset. It works as expected and I can obtain predictions. However, when I try to replace FastText model with a BiRNN model , I’m getting an error about input sizes as below. Can anyone explain how to correct the error about input sizes? If you can provide a suitable implementation of a Seq2Seq model that I can adapt, it will be ok for me also. I provide the implementation of these two models as below. Thanks.</p>
<p>raise ValueError(“Target size ({}) must be the same as input size ({})”.format(target.size(), input.size()))<br>
ValueError: Target size (torch.Size([64])) must be the same as input size (torch.Size([1631]))</p>
<p>class FastText(nn.Module):<br>
def <strong>init</strong>(self,<br>
vocab_size,<br>
embedding_dim,<br>
output_dim,<br>
pad_idx):<br>
super().<strong>init</strong>()<br>
self.embedding = nn.Embedding(<br>
vocab_size,<br>
embedding_dim,<br>
padding_idx=pad_idx)<br>
self.fc = nn.Linear(embedding_dim,<br>
output_dim)</p>
<pre><code>def forward(self, text):
    embedded = self.embedding(text)
    embedded = embedded.permute(1, 0, 2)
    pooled = F.avg_pool2d(
        embedded,
        (embedded.shape[1], 1)).squeeze(1)
    return self.fc(pooled)
</code></pre>
<p>class BiRNN(nn.Module):<br>
def <strong>init</strong>(self, vocab, embed_size, num_hiddens, num_layers):<br>
super(BiRNN, self).<strong>init</strong>()<br>
self.embedding = nn.Embedding(len(vocab), embed_size)</p>
<pre><code>    self.encoder = nn.LSTM(input_size=embed_size,
                           hidden_size=num_hiddens,
                           num_layers=num_layers,
                           bidirectional=True)
    self.decoder = nn.Linear(4 * num_hiddens, 1)

def forward(self, inputs):
    embeddings = self.embedding(inputs.permute(1, 0))
    outputs, _ = self.encoder(embeddings)
    encoding = torch.cat((outputs[0], outputs[-1]), -1)
    outs = self.decoder(encoding)  # (batch_size, 2)
    return outs
</code></pre>

vdw: <p>You can check the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">docs</a>: The shape of <code>outputs</code> is <code>(L, N, D*H_out​) </code> where <code>L</code> is the sequence length, <code>N</code> is the batch size, and <code>D</code> is the number of directions (1 or 2). This means that</p>
<pre><code class="lang-auto">(outputs[0], outputs[-1])
</code></pre>
<p>does not give exactly what you want. For example, <code>outputs[0]</code> gives you the first hidden state w.r.t. the forward direction and the last hidden state w.r.t. the backward direction; vice verse for <code>outputs[-1]</code>.</p>
<p>I would recommend writing</p>
<pre><code class="lang-auto">outputs, hidden = self.encoder(embeddings)
</code></pre>
<p>and use <code>hidden</code> for further processing. It makes life easier for <code>bidirectional=True</code>, I think. The complete snippet would be</p>
<pre><code class="lang-auto">outputs, hidden = self.encoder(embeddings)
# Extract last hidden state (w.r.t. number of layers)
hidden = hidden[0].view(self.num_layers, self.num_directions, batch_size, self.hidden_size)[-1]
# Handle directions (to easily switch between uni and bidirectional)
if self.num_directions == 1:
    hidden = hidden.squeeze(0)
elif self.num_directions == 2:
    h_1, h_2 = hidden[0], hidden[1]    # Get forward and backward direction
    hidden = torch.cat((h_1, h_2), 1)  # Concatenate both states
outs = self.decoder(hidden)
</code></pre>
<p>The only thing needed is to adjust the <code>__init__()</code> method a bit; something like</p>
<pre><code class="lang-auto">...
# Calculate number of directions
self.num_directions = 2 if self.bidirectional == True else 1
...
# Define last linear layer
self.decoder = nn.Linear(self.num_directions * hidden_size, 1)
...
</code></pre>