Significantly Different Performance Scores between Pytorch and Sklearn Model

username: eager.tf
cooked: <p>I am trying to build a basic text classifier. My benchmark model using a simple logistic regression w/ TF-IDF Vectorization on Sklearn is yielding me these performance results:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/3875f3883f93cc8a42506a153ccca5abc659d0c9.png" data-download-href="https://discuss.pytorch.org/uploads/default/3875f3883f93cc8a42506a153ccca5abc659d0c9" title="Screen Shot 2020-08-06 at 3.31.31 AM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/3875f3883f93cc8a42506a153ccca5abc659d0c9.png" alt="Screen Shot 2020-08-06 at 3.31.31 AM" data-base62-sha1="83trLtDjV5s7Z0A44IRtkTAcUCR" width="690" height="239" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-08-06 at 3.31.31 AM</span><span class="informations">882×306 12.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
However, utilizing a word embedding based RNN, has given me very poor performance results:<br>
precision    recall  f1-score   support</p>
<pre><code>       0       0.86      0.96      0.91      2400
       1       0.13      0.04      0.06       399

accuracy                           0.83      2799
</code></pre>
<p>I understand the performance would be different, but not to this degree. Any advice or pointers?</p>
<pre><code class="lang-auto">class TextClassificationRNN(nn.Module):
    def __init__(self,batch,vocab_size,w2v_dim,hidden_size,num_layers, num_classes):
        super(TextClassificationRNN, self).__init__()
        
        self.batch=batch
        self.hidden_size=hidden_size
        self.num_layers=num_layers
        self.num_classes=num_classes
        self.vocab_size=vocab_size
        self.w2v_dim=w2v_dim
        
        self.emb = nn.Embedding(self.vocab_size,self.w2v_dim)
        self.rnn= nn.LSTM(self.w2v_dim,self.hidden_size,self.num_layers)
        self.init_state=self.init_hidden()
        self.dense=nn.Linear(self.hidden_size,int(self.hidden_size/2))
        self.dense2class=nn.Linear(int(self.hidden_size/2),2)
        self.dropout=nn.Dropout(0.5)

    def init_hidden(self):
        h0 = torch.randn(self.num_layers, self.batch, self.hidden_size,device=device)
        c0 = torch.randn(self.num_layers, self.batch, self.hidden_size,device=device)
        return (h0,c0)
    
    
    def forward(self, x, seq_len_list):
        x=self.emb(x)
        x=x.view(x.shape[1],x.shape[0],-1)
        packed_data=torch.nn.utils.rnn.pack_padded_sequence(x,seq_len_list,enforce_sorted=False)
        output,hn=self.rnn(packed_data,self.init_state)
        unpacked_data, _ = torch.nn.utils.rnn.pad_packed_sequence(output)
        dense_output=self.dense(unpacked_data[-1])
        dense_output=self.dropout(dense_output)
        dense2class=self.dense2class(dense_output)
        return dense2class

objective=nn.BCEWithLogitsLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-1, momentum=0.9)
model=TextClassificationRNN(128,encoder.vocab_size,200,256,1,2)
if torch.cuda.is_available():
    model.cuda()
</code></pre>

username: vdw
cooked: <p>Two things to consider</p>
<ul>
<li>
<p>(Deep) Neural Networks require large amount of data to yield good predictions. According to the table, you have 2.8k sentences, which is not much. I assume you show the scores for the test data. How about on the training data?</p>
</li>
<li>
<p>Without specifically having tested that code, I’m pretty sure that the line <code>x=x.view(x.shape[1],x.shape[0],-1)</code> causes a problem; <a href="https://discuss.pytorch.org/t/for-beginners-do-not-use-view-or-reshape-to-swap-dimensions-of-tensors/75524">please read this older post of mine</a>. Try <code>x = x.transpose(0,1)</code>.</p>
</li>
</ul>

username: eager.tf
cooked: <p>Wow that definitely made a difference. Thank you! Not quite the same score as the LGR but I think I can figure out the next steps. I didn’t realize view would mess with the tensor like that.</p>
<pre><code class="lang-auto"> precision    recall  f1-score   support

           0       0.92      1.00      0.96      2400
           1       0.97      0.45      0.62       399

    accuracy                           0.92      2799
   macro avg       0.94      0.73      0.79      2799
weighted avg       0.92      0.92      0.91      2799

</code></pre>

username: vdw
cooked: <p>I saw a lot of code were people use <code>view()</code> or <code>reshape()</code> “carelessly”. This is way that line jumped out to me immediately. I didn’t check the rest of your code much, but it looks alright. Then again, for small datasets, basic classifiers like Naive Bayes often show better results then Neural Networks.</p>