Sentiment analysis using LSTM on imbalanced citation dataset

username: VikasRajashekar
cooked: <p>I have an extremely unbalanced dataset. <a href="https://cl.awaisathar.com/citation-sentiment-corpus/" rel="nofollow noopener">https://cl.awaisathar.com/citation-sentiment-corpus/</a><br>
Class POSITIVE:829<br>
Class NEGATIVE:280<br>
Class NEUTRAL: 7627</p>
<p>Here is my network:</p>
<pre><code class="lang-auto">import torch.nn as nn

class Sentiment_LSTM(nn.Module):
    """
    We are training the embedded layers along with LSTM for the sentiment analysis
    """

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        """
        Settin up the parameters.
        """
        super(Sentiment_LSTM, self).__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        
        # embedding layer and LSTM layers 
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, 
                            dropout=drop_prob, batch_first=True)
        
        # dropout layer to avoida over fitting
        self.dropout = nn.Dropout(0.5)
        
        # linear and sigmoid layers
        self.fc = nn.Linear(hidden_dim, output_size)
        self.sig = nn.Sigmoid()
        

    def forward(self, x):
        """
        Perform a forward pass

        """
        batch_size = x.size(0)

        x = x.long()
        embeds = self.embedding(x)

        lstm_out, hidden = self.lstm(embeds)

    
        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)


        out = self.dropout(lstm_out)
        out = self.fc(out)

        # sigmoid function
        sig_out = self.sig(out)
        
        # reshape to be batch_size first
        sig_out = sig_out.view(batch_size, -1,3)
        #print("sig_out",sig_out.shape)
        sig_out = sig_out[:, -1,:] # get last batch of labels
        
        # return last sigmoid output and hidden state
        return sig_out
</code></pre>
<p>Loss function:</p>
<pre><code class="lang-auto">lr=0.001

criterion = nn.BCELoss()
optimizer = torch.optim.Adam(net.parameters(), lr=lr)
</code></pre>
<p>My accuracy is low on the small classes. How can i improve it futher?</p>

username: JamesTrick
cooked: <p>Hi!</p>
<p>Are you trying to predict all three classes? Positive, neutral, negative?</p>
<p>If so, applying a sigmoid function probably isn’t the way to as that’s designed for Binary cases. Using a Softmax function, with NLLLoss is better - or you can pass the raw logits (from the linear layer) to CrossEntropyLoss which combines the softmax + NLLLoss.</p>

username: VikasRajashekar
cooked: <p>Hello , Thanks for the input.<br>
I modified the network as below.</p>
<p>import torch.nn as nn</p>
<pre><code class="lang-auto">class Sentiment_LSTM(nn.Module):
    """
    We are training the embedded layers along with LSTM for the sentiment analysis
    """

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        """
        Settin up the parameters.
        """
        super(Sentiment_LSTM, self).__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        
        # embedding layer and LSTM layers 
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, 
                            dropout=drop_prob, batch_first=True)
        
        # dropout layer to avoida over fitting
        self.dropout = nn.Dropout(0.5)
        
        # linear and sigmoid layers
        self.fc = nn.Linear(hidden_dim, output_size)
        

    def forward(self, x):
        """
        Perform a forward pass

        """
        batch_size = x.size(0)

        x = x.long()
        embeds = self.embedding(x)

        lstm_out, hidden = self.lstm(embeds)

    
        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)


        out = self.dropout(lstm_out)
        out = self.fc(out)


        
        # reshape to be batch_size first
        out = out.view(batch_size, -1,3)
        #print("sig_out",sig_out.shape)
        out = out[:, -1,:] # get last batch of labels
        
        # return last sigmoid output and hidden state
        return out
</code></pre>
<p>Loss as follows:</p>
<pre><code class="lang-auto"># loss and optimization functions
lr=0.001

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=lr)
</code></pre>
<p>Is my understanding correct?</p>

username: zhangguanheng66
cooked: <p>PyTorch has a tutorial for text classification analysis <a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html" rel="nofollow noopener">here</a>. Consider to replace Bag-of-Word model with LSTM for your case. Other parts should be same, including <a href="https://pytorch.org/docs/stable/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss" rel="nofollow noopener">CrossEntropyLoss</a>.</p>