Help for converting sklearn model to PyTorch model

Mugdha: <p>I want to implement a multi-class classifier for categorizing sentences in class labels 0, 1 and 2 using PyTorch. The input to the neural network will be an average of the word emeddings (vectors of 300 dimensions) of  all the words that form a sentence. Hence the input size is 300.</p>
<p>This is the sklearn code for the same which I found here: <a href="https://github.com/mdvu15/CS488-Senior-Capstone/blob/master/classifierTrain.py" rel="nofollow noopener">https://github.com/mdvu15/CS488-Senior-Capstone/blob/master/classifierTrain.py</a></p>
<p>(typing out the code snippet)<br>
X_train, X_test, y_train, y_test = train_test_split(V, y, test_size = 0.25) <span class="hashtag">#25</span>% of data set aside for testing</p>
<p>mlp = MLPClassifier(hidden_layer_sizes=(500, 20, 20, 20), max_iter=1000, batch_size=32, <br>
warm_start=True, early_stopping= True)  <span class="hashtag">#Classifier</span> object</p>
<p>mlp.fit(X_train, y_train)</p>
<p>This is the PyTorch code I have so far for this-</p>
<p>class Linear_Model(torch.nn.Module):</p>
<p>def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)</p>
<pre><code>self.fc1 = torch.nn.Linear(300, 500).to(device)
self.fc2 = torch.nn.Linear(500, 20).to(device)
self.fc3 = torch.nn.Linear(20, 20).to(device)
self.fc4 = torch.nn.Linear(20, 20).to(device)
self.fc5 = torch.nn.Linear(20, 3).to(device)

# activation functions

self.relu = torch.nn.ReLU().to(device)
self.softmax = torch.nn.Softmax(dim = 1).to(device)
</code></pre>
<p>def forward (self, input, flag):<br>
fc_out = self.relu(self.fc1(input))<br>
fc_out = self.relu(self.fc2(fc_out))<br>
fc_out = self.relu(self.fc3(fc_out))<br>
fc_out = self.relu(self.fc4(fc_out))<br>
fc_out = self.softmax(self.fc5(fc_out))</p>
<pre><code>return fc_out
</code></pre>
<p>net = Linear_Model()</p>
<p>criterion = torch.nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adam(net.parameters(), lr = 0.001)</p>
<p><span class="hashtag">#Training</span></p>
<p>for epoch in range(n_epochs):<br>
train_accuracy = []  <span class="hashtag">#for</span> one epoch<br>
train_losses = []      <span class="hashtag">#for</span> one epoch<br>
net.train()<br>
for inputs, labels in trainloader:<br>
inputs, labels = inputs.to(device), labels.to(device)<br>
output = net(inputs.float(), 0)<br>
loss = criterion(output.squeeze(), labels)<br>
train_losses.append(loss.item())<br>
loss.backward()<br>
optimizer.step()       <span class="hashtag">#Stochastic</span> gradient descent on mini batches<br>
optimizer.zero_grad()</p>
<pre><code># calculate training accuracy
pred = torch.argmax(output.squeeze(), axis = 1)
print(pred)
correct_tensor = pred.eq(labels.view_as(pred))
correct = np.squeeze(correct_tensor.cpu().numpy()) if torch.cuda.is_available() else np.squeeze(correct_tensor.numpy())
num_correct = np.sum(correct)
train_accuracy += [num_correct / batch_size]
</code></pre>
<p>Is this code correct? If not, then can the corrections (also suggestions if any) be pointed out? Thanks!</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits, as internally <code>F.log_softmax</code> will be used, so remove the softmax and rerun the code again. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Mugdha: <p>Thank you so much for your reply! I will do that. Although how do I calculate training accuracy from raw logits? Because the 3 values that will be returned can be either both positive or negative (I guess).</p>

ptrblck: <p>The outputs will represent logits, so <code>torch.argmax(output, 1)</code> will yield the same prediction as you would get using softmax.<br>
The higher the logit, the higher the probability.</p>

Mugdha: <p>Done. The model is converging much faster now. Thank you so much!</p>