Poor results with pytorch compared to NN written from scratch

jarvico: <p>Hi,</p>
<p>I have an NN model which is written from stratch.This model has 2 neurons input layer, 1 hidden layer with 5 neurons and 2 output neurons. In hidden layer there is tanh activation. Cross entropy loss is being used. Weight 2 regularization aplied with lambda 0.01. Also weights are initialized with Xavier initialization. Learning rate is: 0.001. After 1000 epoch, this model has gives a Loss value of 0.093, and accuracy of 0.970 with below dataset.</p>
<p>x,y = sklearn.datasets.make_moons(n_samples=200,noise=0.05)</p>
<p>I decided to build the same model using Pytorch. However, I got very poor results in Pytorch. It seemed to be quiet strange.</p>
<p>Can you please check my Pytorch code and comment on why did I got poor results?</p>
<p>input_neurons = 2<br>
hidden_neurons = 5<br>
output_neurons = 2<br>
learning_rate = 0.001<br>
lambda_reg = 0.01</p>
<p>x,y = sklearn.datasets.make_moons(n_samples=200,noise=0.05)</p>
<p>x = torch.FloatTensor(x)<br>
y = torch.LongTensor(y)<br>
class FeedForward(torch.nn.Module):<br>
def <strong>init</strong>(self, input_neurons, hidden_neurons, output_neurons):<br>
super(FeedForward,self).<strong>init</strong>()<br>
self.hidden = nn.Linear(input_neurons, hidden_neurons)<br>
self.out = nn.Linear(hidden_neurons,output_neurons)<br>
def forward(self, x):<br>
x = self.hidden(x)<br>
x = F.tanh(x)<br>
x = self.out(x)<br>
return x</p>
<p>def init_weights(m):<br>
if type(m) == nn.Linear:<br>
torch.nn.init.xavier_uniform(m.weight)<br>
<span class="hashtag">#m</span>.bias.data.fill_(0.01)</p>
<p>network = FeedForward(input_neurons = input_neurons, hidden_neurons = hidden_neurons, output_neurons = output_neurons)<br>
network.apply(init_weights)</p>
<p>optimizer = torch.optim.SGD(network.parameters(), lr = learning_rate,weight_decay=lambda_reg)<br>
loss_function = torch.nn.CrossEntropyLoss()</p>
<p>for epoch in range(1000):<br>
out = network(x)<br>
loss = loss_function(out, y)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<pre><code>if epoch % 50 == 0:
    max_value, prediction = torch.max(out, 1)
    predicted_y = prediction.data.numpy()
    target_y = y.data.numpy()
    accuracy = (predicted_y == target_y).sum() / target_y.size
    print('Accuracy = {:.2f} Loss = {:.2f}'.format(accuracy,loss))</code></pre>

KFrank: <p>Hi Ömer!</p>
<aside class="quote no-group" data-username="jarvico" data-post="1" data-topic="95678" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>I decided to build the same model using Pytorch. However, I got very poor results in Pytorch.</p>
</blockquote>
</aside>
<p>You don’t show your scikit-learn code nor your pytorch results, so its<br>
hard to know the the difference might lie.</p>
<p>However, I don’t see anything wrong with your code (other than it being<br>
garbled by the forum because you didn’t enclose it in a triple-backtick<br>
code block).</p>
<p>You code seems to run fine for me, and gives good results <em>if</em> I increase<br>
the learning rate, get rid of weight-decay (l2 regularization), and add<br>
“momentum” to the <code>SGD</code> optimizer.</p>
<p>Note, I am using pytorch version 0.3.0, so I had to wrap the <code>Tensor</code>s<br>
in <code>autograd.Variable</code>s.</p>
<p>Here is your code, as slightly modified by me:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F
torch.__version__

import numpy as np
import sklearn.datasets

torch.manual_seed (2020)
np.random.seed (2020)

x,y = sklearn.datasets.make_moons(n_samples=200,noise=0.05)

x = torch.autograd.Variable (torch.FloatTensor(x))   # using pytorch version 0.3.0
y = torch.autograd.Variable (torch.LongTensor(y))    # using pytorch version 0.3.0

input_neurons = 2
hidden_neurons = 5
output_neurons = 2
# learning_rate = 0.001
learning_rate = 0.1   # increase learning rate
# lambda_reg = 0.01
lambda_reg = 0.00     # turn off weight decay

momentum = 0.9        # add momentum to SGD

class FeedForward(torch.nn.Module):
    def __init__(self, input_neurons, hidden_neurons, output_neurons):
        super(FeedForward,self).__init__()
        self.hidden = nn.Linear(input_neurons, hidden_neurons)
        self.out = nn.Linear(hidden_neurons,output_neurons)
    def forward(self, x):
        x = self.hidden(x)
        x = F.tanh(x)
        x = self.out(x)
        return x

def init_weights(m):
    if type(m) == nn.Linear:
        torch.nn.init.xavier_uniform(m.weight)
        # m.bias.data.fill_(0.01)

network = FeedForward(input_neurons = input_neurons, hidden_neurons = hidden_neurons, output_neurons = output_neurons)
network.apply(init_weights)

# optimizer = torch.optim.SGD(network.parameters(), lr = learning_rate,weight_decay=lambda_reg)
optimizer = torch.optim.SGD(network.parameters(), lr = learning_rate, weight_decay = lambda_reg, momentum = momentum)
loss_function = torch.nn.CrossEntropyLoss()

for epoch in range(1000):
    out = network(x)
    loss = loss_function(out, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    if epoch % 50 == 0:
        max_value, prediction = torch.max(out, 1)
        predicted_y = prediction.data.numpy()
        target_y = y.data.numpy()
        accuracy = (predicted_y == target_y).sum() / target_y.size
        print('Accuracy = {:.3f} Loss = {:.6f}'.format(accuracy,loss.data[0]))

</code></pre>
<p>Here is the output, showing rapid convergence to 100% accuracy:</p>
<pre><code class="lang-plaintext">Accuracy = 0.500 Loss = 0.761427
Accuracy = 0.885 Loss = 0.255683
Accuracy = 0.885 Loss = 0.241214
Accuracy = 0.905 Loss = 0.218220
Accuracy = 0.955 Loss = 0.119875
Accuracy = 0.990 Loss = 0.048679
Accuracy = 1.000 Loss = 0.027161
Accuracy = 1.000 Loss = 0.018272
Accuracy = 1.000 Loss = 0.013567
Accuracy = 1.000 Loss = 0.010696
Accuracy = 1.000 Loss = 0.008776
Accuracy = 1.000 Loss = 0.007410
Accuracy = 1.000 Loss = 0.006392
Accuracy = 1.000 Loss = 0.005607
Accuracy = 1.000 Loss = 0.004983
Accuracy = 1.000 Loss = 0.004478
Accuracy = 1.000 Loss = 0.004060
Accuracy = 1.000 Loss = 0.003710
Accuracy = 1.000 Loss = 0.003412
Accuracy = 1.000 Loss = 0.003156
</code></pre>
<p>Good luck.</p>
<p>K. Frank</p>

jarvico: <p>Thank you Frank! Appreciated…</p>
<p>After setting weight decay parameter to 0, altering learning rate to 0.1 and adding momentum, I got similar results as yours which is pretty good.</p>
<p>The thing I still did not understand is that in pure python implementation, I used a learning rate of 0.001 and lambda value of 0.01 for L2 weight normalization. And no momentum applied. How come pure python implementation resulted in a much better performance than my first Pytorch model? I really don’t know.</p>
<p>By the way below is the pure python implementation code:</p>
<p>import numpy as np<br>
import matplotlib.pyplot as plt<br>
import sklearn.datasets</p>
<p>np.random.seed(3)<br>
x,y = sklearn.datasets.make_moons(n_samples=200,noise=0.05)</p>
<p>input_neurons = 2<br>
hidden_neurons = 5<br>
output_neurons = 2<br>
samples = x.shape[0]<br>
learning_rate = 0.001<br>
lambda_reg = 0.01</p>
<h1>Define Initial Weights</h1>
<p>def init_network(input_dim, hidden_dim, output_dim):<br>
model = {}<br>
# Xavier Initialization<br>
W1 = np.random.randn(input_dim, hidden_dim) / np.sqrt(input_dim)<br>
b1 = np.zeros((1, hidden_dim))<br>
W2 = np.random.randn(hidden_dim, output_dim) / np.sqrt(hidden_dim)<br>
b2 = np.zeros((1, output_dim))<br>
model[‘W1’] = W1<br>
model[‘b1’] = b1<br>
model[‘W2’] = W2<br>
model[‘b2’] = b2<br>
return model</p>
<p>def retreive(model_dict):<br>
W1 = model_dict[‘W1’]<br>
b1 = model_dict[‘b1’]<br>
W2 = model_dict[‘W2’]<br>
b2 = model_dict[‘b2’]<br>
return W1, b1, W2, b2</p>
<p>def forward(x, model_dict):<br>
W1, b1, W2, b2 = retreive(model_dict)<br>
z1 = x.dot(W1) + b1<br>
a1 = np.tanh(z1)<br>
z2 = a1.dot(W2) + b2<br>
exp_scores = np.exp(z2)<br>
softmax = exp_scores / np.sum(exp_scores, axis = 1, keepdims = True)<br>
return z1, a1, softmax</p>
<p>def loss(softmax, y, model_dict):<br>
W1, b1, W2, b2 = retreive(model_dict)<br>
m = np.zeros(200)<br>
for i,correct_index in enumerate(y):<br>
predicted = softmax[i][correct_index]<br>
m[i] = predicted<br>
log_prob = -np.log(m)<br>
loss = np.sum(log_prob)<br>
reg_loss = lambda_reg / 2 * (np.sum(np.square(W1)) + np.sum(np.square(W2)))<br>
loss+= reg_loss<br>
return float(loss / y.shape[0])</p>
<p>def backpropagation(x, y, model_dict, epochs):<br>
for i in range(epochs):<br>
W1, b1, W2, b2 = retreive(model_dict)<br>
z1, a1, probs = forward(x, model_dict)  # a1: (200,3), probs: (200,2)<br>
delta3 = np.copy(probs)<br>
delta3[range(x.shape[0]), y] -= 1  # (200,2)<br>
dW2 = (a1.T).dot(delta3)  # (3,2)<br>
db2 = np.sum(delta3, axis=0, keepdims=True)  # (1,2)<br>
delta2 = delta3.dot(W2.T) * (1 - np.power(np.tanh(z1), 2))<br>
dW1 = np.dot(x.T, delta2)<br>
db1 = np.sum(delta2, axis=0)<br>
# Add regularization terms<br>
dW2 += lambda_reg * np.sum(W2)<br>
dW1 += lambda_reg * np.sum(W1)<br>
# Update Weights: W = W + (-lr<em>gradient) = W - lr</em>gradient<br>
W1 += -learning_rate * dW1<br>
b1 += -learning_rate * db1<br>
W2 += -learning_rate * dW2<br>
b2 += -learning_rate * db2<br>
# Update the model dictionary<br>
model_dict = {‘W1’: W1, ‘b1’: b1, ‘W2’: W2, ‘b2’: b2}<br>
# Print the loss every 50 epochs<br>
if i % 50 == 0:<br>
prediction = np.argmax(probs, 1)<br>
accuracy = (prediction == y).sum() / y.size<br>
print(“Loss at epoch {} is: {:.3f}, accuracy is {:.3f}”.format(i, loss(probs, y, model_dict),accuracy))</p>
<pre><code>return model_dict
</code></pre>
<p>def predict(model_dict, x):<br>
W1, b1, W2, b2 = retreive(model_dict)<br>
z1 = x.dot(W1) + b1<br>
a1 = np.tanh(z1)<br>
z2 = a1.dot(W2) + b2<br>
exp_scores = np.exp(z2)<br>
softmax = exp_scores / np.sum(exp_scores, axis = 1, keepdims = True)   # (200,2)<br>
return np.argmax(softmax, axis = 1)    # (200,)</p>
<p>model_dict = init_network(input_dim = input_neurons , hidden_dim = hidden_neurons, output_dim = output_neurons)<br>
model = backpropagation(x, y, model_dict, 1000)</p>

KFrank: <p>Hi Ömer!</p>
<aside class="quote no-group" data-username="jarvico" data-post="3" data-topic="95678" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>How come pure python implementation resulted in a much better performance than my first Pytorch model? I really don’t know.</p>
</blockquote>
</aside>
<p>I don’t know either.</p>
<p>But somewhere your pytorch code and numpy code are doing<br>
something different.</p>
<p>I would suggest looking at a single forward and backward pass,<br>
and see where things first differ.  Make sure you feed the same<br>
<code>sklearn.datasets</code> into both your pytorch and numpy networks.<br>
Make sure both networks are initialized with the same weights<br>
and biases.</p>
<p>Run a batch of inputs through both networks.  Do you get the same<br>
predictions?  If not, compare intermediate results until you localize<br>
the difference.</p>
<p>If the predictions are the same, do you get the same loss value?</p>
<p>Run your backward pass.  Do you get the same gradients?  If not<br>
look at intermediate results, etc.</p>
<p>There is no reason you shouldn’t be able to get the same results<br>
with pytorch and numpy (up to reasonable round-off error, which<br>
isn’t your issue here).</p>
<p>(As an aside, I don’t see your numpy code backpropagating through<br>
the softmax calculation in your model’s <code>forward()</code> function.)</p>
<p>Good luck.</p>
<p>K. Frank</p>