Simple IMDB Binary classification not working

username: awsgcp
cooked: <p>Tried to switch from tensorflow(keras) to pytorch. Testing basic binary classification model with embedding layer for IMDB movie review data. in tensorflow, everything is working fine, easy to build the model and converge very fast. Tried to replicate the same model in pytorch, struggled to make it work.</p>
<p>i searched online, did not see anyone used sigmoid for binary classificaiton, some used log_softmax, not sure why.</p>
<p>Is there anything done wrong? please help, thank you very much!</p>
<pre><code class="lang-auto">import torch
import torchtext
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

TEXT = torchtext.data.Field(tokenize = 'spacy', fix_length=40)
LABEL = torchtext.data.LabelField(dtype = torch.float)

train_dataset, test_dataset = torchtext.datasets.IMDB.splits(TEXT, LABEL)

MAX_VOCAB_SIZE = 10000

TEXT.build_vocab(train_dataset, max_size = MAX_VOCAB_SIZE, vectors = "glove.6B.100d")
LABEL.build_vocab(train_dataset)

BATCH_SIZE = 32
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

train_loader, test_loader = torchtext.data.BucketIterator.splits(
    (train_dataset, test_dataset),
    batch_size = BATCH_SIZE,
    device = device)




class RNN(nn.Module):
    def __init__(self):
        super(RNN, self).__init__()

        self.embedding = nn.Embedding(len(TEXT.vocab), 100, padding_idx=TEXT.vocab.stoi[TEXT.pad_token])
        self.lstm = nn.LSTM(100, 100, batch_first = True)
        self.fc1 = nn.Linear(100, 1)


    def forward(self, x):
        x = self.embedding(x)
        x, hidden = self.lstm(x)
        x = x[-1,:,:]
        x = torch.sigmoid(self.fc1(x))
        return x


net = RNN()
net.embedding.weight.data.copy_(TEXT.vocab.vectors)

for param in net.embedding.parameters():
  param.requires_grad = False

net = net.to(device)
criterion = nn.BCELoss()
optimizer = optim.Adam(net.parameters()) 

for epoch in range(2): 
    print("epoch: {}".format(epoch+1))
    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs.squeeze(1), labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 200 == 199:
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 200))
            running_loss = 0.0
print('Finished Training')

correct = 0
total = 0
with torch.no_grad():
    for data in test_loader:
        images, labels = data
        outputs = net(images)
        predicted = torch.round(outputs).reshape(-1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the test data: {}'.format(100 * correct / total))
</code></pre>

username: ptrblck
cooked: <p>The code looks generally alright.<br>
We recommend using raw logits and <code>nn.BCEWithLogitsLoss</code>, as the numerical stability is better than <code>sigmoid</code> + <code>nn.BCELoss</code>.</p>
<p><code>F.log_softmax</code> is used together with e.g. <code>nn.NLLLoss</code> for a multi-classification use case.<br>
You could rewrite the binary classification to a 2 class multi-class classification, but your approach should work.<br>
If you want to try to reproduce your TensorFlow model, I would recommend to also check the parameter initializations, as they might differ for the used layers.</p>

username: awsgcp
cooked: <p>Thanks ptrblck, i wish there is something wrong with the code. but seems you also agree there is no major problem. i thought for this simple example it would converge very fast since embedding should be powerful enough for simple text classification. but the above code never gets the loss below 0.6 after many epochs. accuracy is just a bit above 50%.<br>
anyway, I will keep looking into other example scripts online to see if I can reproduce the expected good result.<br>
thanks again for the quick reply! ptrblck</p>

username: ptrblck
cooked: <p>Oh, wait a moment.<br>
<code>x = x[-1,:,:]</code> looks fishy.<br>
Since you are using <code>batch_first=True</code>, you would slice the tensor in the batch dimension, thus only using the last sample?<br>
If you want to use the last output in the <code>seq</code> dimension, you would have to index the tensor in dim1.</p>
<p>Could you print the shape of <code>output</code>, please?</p>

username: awsgcp
cooked: <p>sure, no problem.</p>
<p>before x = x[-1,:,:],  x.shape = torch.Size([40, 32, 100])<br>
after x = x[-1,:,:],  x.shape = torch.Size([32, 100])<br>
after fully connected layer, x.shape = torch.Size([32, 1])</p>
<p>if the slice is in the batch dimension, the code should throw error. since 40(max sentence length) != 32. thanks.</p>

username: ptrblck
cooked: <p>Thanks for the information.<br>
It seems as if the output is returned as <code>[seq_len, batch_size, features]</code>.<br>
Could you check the shape of the <code>input</code> also?<br>
As you can see, I’m not very familiar with <code>torchtext</code>, but since you are using <code>batch_first=True</code> in yyour <code>nn.LSTM</code>, the batch dimension shoule be in dim0, which is apparently not the case here.</p>

username: awsgcp
cooked: <p>the input shape of the model is: torch.Size([40, 32])<br>
yes, you are right, batch_first seems to be the cause of the problem!<br>
thank you very much ptrblck!</p>