Pytorch not converge but keras did

Diaa_elsayed_ziada: <p>i’m work on real time gender, expression detection<br>
i created model using pytorch but el maximum accuracy the model reach &lt; 50% but the same model with same data and same configuration in keras model reach  99% accuracy  what is the reason for this<br>
this is link for colab notebook contains both models an the training steps<br>
<a href="https://colab.research.google.com/drive/1knoJrk-P3I0V2ZRfEV40CWC2lvZ-oEFR" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1knoJrk-P3I0V2ZRfEV40CWC2lvZ-oEFR</a></p>

jackparrySG: <p>Are the hyperparameters exactly the same for each model? It’s quite hard to parse the training logs - it’d be nice if you showed a graph of the loss, accuracy etc over time for each model.</p>
<p>From what I can see though, your keras model converges to almost 0 training loss, whereas the PyTorch model seems to increase in training loss. It looks as though the learning rate in PyTorch is 0.1 wheras you use 0.001 in Keras, this could be the issue. For training loss to increase, it’s usually because your learning rate is too high.</p>

Diaa_elsayed_ziada: <p>models are totally separated form each other they just have same cfgs ,data and model stricture</p>
<p>i tried both with 0.001 and 0.01 and 0.1 gives same situation what use see is just experiment<br>
and accuracy in pytorch model spot increasing after it reach 51% and some times after this the accuracy decrease to</p>

jackparrySG: <p>Does the training loss continue to decrease, and approach 0? If not, you have a problem.</p>

Diaa_elsayed_ziada: <p>no it’s most of time increasing and at any epoch validation loss become less than el minimum validation loss of training process but the accuracy still low and it return again to the higher accuracy an start to increase and so on, look like training process work for first epochs and el process be come random after this</p>

jackparrySG: <p>If the loss is increasing then perhaps even 0.001 is too high for the learning rate. Accuracy is not a great metric to look at to debug the code. Basically any neural network should be able to fully learn the training data if you train it for long enough, otherwise you dont have enough parameters in the network. So you should focus on trying to get the training loss to converge to 0. If the loss isnt decreasing, your model isnt training correctly.</p>

Diaa_elsayed_ziada: <p>so why 0.001 lr works well in keras</p>

Diaa_elsayed_ziada: <p>what is your advice for my to do?</p>

jackparrySG: <blockquote>
<p>Blockquote<br>
so why 0.001 lr works well in keras</p>
</blockquote>
<p>It could be any number of factors. You’d have to investigate the source code for every layer to see if it is implemented exactly the same as in PyTorch. Maybe something simple like a learning rate decay is implemented by default in Keras.</p>
<blockquote>
<p>Blockquote<br>
what is your advice for my to do?</p>
</blockquote>
<p>Try training a pytorch model with a lower learning rate, until you actually observe a consistent decrease in training loss throughout training. Don’t worry about valid accuracy for now. If train loss does not decrease with a lower LR, then something is wrong with either the data, the labels, the model architecture or the training script. I can’t really help much for debugging these. Just remember that PyTorch will not necessarily do everything in exactly the same way as Keras by default.</p>

Diaa_elsayed_ziada: <p>I tried some experiments and discovered that<br>
All models in pytorch works will with same data and training steps and any configurations except resnet model either my implementation or the pytorch implementation<br>
I tried to remove normalizing transformation, freezing batch norm layer,used sgd or adam optimizer, and use lower leaning rate 0.0000001<br>
Nothing work<br>
Is pytorch has problem with resnet architecture</p>

jackparrySG: <blockquote>
<p>Blockquote<br>
Is pytorch has problem with resnet architecture</p>
</blockquote>
<p>Nope there are plenty of examples of resnet implemented correctly in PyTorch.</p>
<p>Are you passing network outputs through a softmax before computing the loss with nn.CrossEntropyLoss()? If you read the documentation you’ll notice that CrossEntropyLoss() actually takes logits as input.</p>

Diaa_elsayed_ziada: <p>no i’m taking the output directly form the resnet to el  CrossEntropyLoss() without apply any thing</p>

jackparrySG: <p>Well in the code you posted, the model is comprised of a feature extractor and a classifier. The classifier has x = self.softmax(x) as the final function. Without seeing the actual code you’re using I can’t help much, and I’m not going to debug your entire script</p>

Diaa_elsayed_ziada: <p>no this old code i’m now using pytorch model<br>
sorry about that</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=False)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 10)
net = model_ft
</code></pre>

Diaa_elsayed_ziada: <p>This is the actual and full code that i’m using</p>
<p>data:</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

</code></pre>
<p>optimizer and loss</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
</code></pre>
<p>model:</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=False)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 10)
net = model_ft
</code></pre>
<p>train:</p>
<pre><code class="lang-auto">net.cuda()
for epoch in range(15):  # loop over the dataset multiple times
    acc= 0
    train_acc=0
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        
        inputs, labels = inputs.cuda(), labels.cuda()
        # zero the parameter gradients
        optimizer.zero_grad()
        labels = labels.squeeze()
        # forward + backward + optimize
        outputs = net(inputs)
        
#         outputs = outputs.view(outputs.shape[0],outputs.shape[1])
#         labels = labels.squeeze()
#         print(outputs.shape,labels.shape)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        for i in range(len(labels)):
            if labels[i] == torch.argmax(outputs[i]):
                acc+=1
        # print statistics
        running_loss += loss.item()
        ps = torch.exp(outputs)
        top_p, top_class = ps.topk(1, dim=1)
        equals = top_class == labels.view(*top_class.shape)
        train_acc += torch.mean(equals.type(torch.FloatTensor))
        
#         if i % 2000 == 1999:    # print every 2000 mini-batches
    print('[%d, %5d] loss: %.3f acc: %.3f ' %
          (epoch + 1, i + 1, running_loss,train_acc ))
    print(acc)
    acc=0
    running_loss = 0.0
    train_acc = 0
            
print('Finished Training')
</code></pre>

jackparrySG: <p>You should initialise the optimiser after creating the model, and after sending the model to cuda</p>

Diaa_elsayed_ziada: <p>it’s working now<br>
i think my model was suffering from many problems not one<br>
as you say<br>
1- final layer contains softmax and it makes train much slower<br>
2- order of initializing model and it’s optimizer, it was train on model but optimizer steps takes on other model<br>
3- i was using batched gd and keras model was used sgd so that keras converged much faster</p>
<p>i don’t know how could thank you for your help and your patience<br>
thanks a lot  <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:"><img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:"><br>
i hope for your great future and good life</p>

Saeed_Alahamri: <p>I’m have the same issue with Unet. In keras it converge nicely with no issue, but in pytorch it is not. it diverge after 4 to 5 epochs. Although I have checked the architecture implementations, and I don’t see any differences between the keras and pytorch implementation and the number of trainable parameters. How to solve the divergence of pytorch model? any idea why this is happening?</p>
<pre><code class="lang-auto">100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:18&lt;00:00,  1.91it/s]
epoch 1/20, loss 0.11540429561613606, dice 0.5524536604077167
saving the model ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:20&lt;00:00,  1.89it/s]
epoch 2/20, loss 0.030815164163202097, dice 0.7417304706613025
saving the model ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:19&lt;00:00,  1.90it/s]
epoch 3/20, loss 0.025856498361659385, dice 0.7701306981134968
saving the model ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:20&lt;00:00,  1.89it/s]
epoch 4/20, loss 0.0235783447655131, dice 0.7951608874509188
saving the model ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:20&lt;00:00,  1.90it/s]
epoch 5/20, loss 0.022023522508196273, dice 0.8102755615762612
saving the model ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:16&lt;00:00,  1.93it/s]
epoch 6/20, loss 1.5251818179503502, dice 0.4822449768164751
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:14&lt;00:00,  1.94it/s]
epoch 7/20, loss 2.747159472943143, dice 0.1791747296810478
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:12&lt;00:00,  1.95it/s]
epoch 8/20, loss 2.7447210459369136, dice 0.17914873635318582
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 603/603 [05:12&lt;00:00,  1.96it/s]
epoch 9/20, loss 2.744714410941597, dice 0.17913664069644042

</code></pre>

Diaa_elsayed_ziada: <p>Try to use ten crop transformation</p>

Saeed_Alahamri: <p>I found the issue. I will mention it here for anyone who come across this issue later.<br>
I’m using Adam where the default eps in Keras is 1e-7, whereas the default value in Pytorch is 1e-8… Also the default Conv2d initialization is different from Keras.<br>
After fixing both eps and initialization. It converged nicely like what it did in Keras.</p>