Struggle comparing PyTorch and Python results for a simple architecture

erm: <p>I built the following simple architecture in Pytorch, to try to predict cat vs. non cat from Coursera.</p>
<pre><code class="lang-auto">class Net(nn.Module): 
    def __init__(self):
        super(Net,self).__init__()  
        self.input_layer = nn.Linear(dim,7)  
        self.hidden_layer = nn.Linear(7,1) 
        self.output_layer = nn.Linear(1,2) # 2 classes 
           
    def forward(self, x):
        x = x.view(-1,dim) 
        x = F.relu(self.input_layer(x)) 
        x = F.relu(self.hidden_layer(x))
        x = self.output_layer(x) 
        return F.log_softmax(x, dim=1) 
    
net= Net()
</code></pre>
<pre><code class="lang-auto">#Training 
optimizer = optim.SGD(net.parameters(), lr=0.0075) 

iterations=0
losses = np.array([])
for e in range(epochs):

    for data in (train_loader): 

        X,y = data
        net.zero_grad()
        output = net(X.view(-1, dim)) 
        loss = F.nll_loss(output, y)   
        loss.backward() #calc. gradients
        optimizer.step() #update weights
        iterations +=1
        losses = np.append(losses, loss.item())
        print(loss)

print ('final loss:', loss)   
print (losses)
</code></pre>
<p>I have the expected results from Coursera, so I used the same lr=0.0075 and same number of nodes per layer BUT my results are really different and not good:</p>
<ul>
<li>
<ol>
<li>using the same learning rate but SGD instead of gradient descent, the loss is ridiculous noisy, no matter how much I increase the batch size to soften the curve. The situation only improves reducing the lr one order…</li>
</ol>
</li>
<li>the probabilities predict only one of the classes and accuracy is low.</li>
</ul>
<p>I explored the train and test datasets, and they happen to be unbalanced (train dataset has <strong>65% of 0, 35% of 1</strong> over 209 examples, while the test set consists 50 images: <strong>34% of 0 and 64% of 1</strong>, thus, unbalanced in favor of the opposite class of the trainset)<br>
This unbalance does not pose a problem for Coursera example, but seems that my architecture is not dealing in the same manner?<br>
Is it possible that SGD is way more sensitive than GD for treating this unbalanced datasets? Or should I put my eye on trying to find another source of error?<br>
Besides, they use random initialization for weights, multiplying by 0.01, and zero for the bias in each layer… Is it possible that using the default Kaiming initialization for nn.Linear changes the results significantly?</p>
<p>P.S.:</p>
<ul>
<li>I am using dataloader with shuffle = True, to avoid uploading batches that accidentally consist of only label 0 or only label 1.</li>
<li>Since dataloader accepts the format [bs, C, H, W] and the original dataset is [n_examples, H, W, C] , I do a swap operation on the initial numpy 4D array as follow, before building the tensors.</li>
</ul>
<pre><code class="lang-auto">Images_dataset = np.swapaxes(images_dataset,1,3)
</code></pre>

ptrblck: <p>The <code>swapaxes</code> operation will create an input of <code>[N, C, W, H]</code>, while <code>[N, C, H, W]</code> is expected.<br>
I would recommend to use <code>x = x.permute(0, 3, 1, 2)</code> on the tensor.<br>
This shouldn’t make a huge difference if you are flattening the tensor anyway.</p>
<aside class="quote no-group" data-username="erm" data-post="1" data-topic="78791">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/erm/48/18230_2.png" class="avatar"> erm:</div>
<blockquote>
<p>Is it possible that using the default Kaiming initialization for nn.Linear changes the results significantly?</p>
</blockquote>
</aside>
<p>Yes, that could make a huge difference.<br>
Could you apply the same initialization as is used in the Coursera code?</p>
<p>I also assume that the model architecture is equal to the one from the course?<br>
The single output feature of <code>hidden</code> looks a bit “odd”, but might still work.</p>

erm: <p>If the architecture has good prediction power, shouldn’t it also be good at predicting 90 degree rotated images? I mean, if I made the mistake of feeding WH instead of HW for the whole trainset and test set, shouldn’t it be able to extract the features anyway? Or is this “rotation” any different in 4D? I would love to conceptually understand this.<br>
I will do then the operation in PyTorch using permute and change the default initialization to see if something improves.<br>
Yes, they oddly use only one node at the end. (They have another deeper architecture with 4 layers but i face same issues with that one)<br>
Last thing:  Sigmoid activation on the last layer plus cross entropy loss is equivalent to NLL plus Softmax, I am right?<br>
Thanks!!</p>

ptrblck: <aside class="quote no-group" data-username="erm" data-post="3" data-topic="78791">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/erm/48/18230_2.png" class="avatar"> erm:</div>
<blockquote>
<p>If the architecture has good prediction power, shouldn’t it also be good at predicting 90 degree rotated images? I mean, if I made the mistake of feeding WH instead of HW for the whole trainset and test set, shouldn’t it be able to extract the features anyway? Or is this “rotation” any different in 4D? I would love to conceptually understand this.</p>
</blockquote>
</aside>
<p>I think it shouldn’t matter, if you are retraining the model and also flattening the tensor for this particular model.</p>
<aside class="quote no-group" data-username="erm" data-post="3" data-topic="78791">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/erm/48/18230_2.png" class="avatar"> erm:</div>
<blockquote>
<p>Sigmoid activation on the last layer plus cross entropy loss is equivalent to NLL plus Softmax, I am right?</p>
</blockquote>
</aside>
<p>No, <code>nn.CrossEntropyLoss</code> expects raw logits (no activation at the end), while <code>nn.NLLLoss</code> expects log probabilities (<code>F.log_softmax</code> at the end).<br>
Since you are using the second approach, it should be fine.</p>