Keras to Pytorch model not working as expected

username: Padmaksha_Roy
cooked: <p>hi there! I converted a keras CNN classifier model to Pytorch. But the model training is not working as expected.</p>
<p>Keras code:</p>
<pre><code class="lang-auto">def model():
    model = Sequential()
    model.add(Conv1D(filters=64, kernel_size=6, activation='relu', 
                    padding='same', input_shape=(4, 1)))
    model.add(BatchNormalization())
    
    # adding a pooling layer
    model.add(MaxPooling1D(pool_size=(3), strides=2, padding='same'))
    
    model.add(Conv1D(filters=64, kernel_size=6, activation='relu', 
                    padding='same', input_shape=(4, 1)))
    model.add(BatchNormalization())
    model.add(MaxPooling1D(pool_size=(3), strides=2, padding='same'))
    
    model.add(Conv1D(filters=64, kernel_size=6, activation='relu', 
                    padding='same', input_shape=(4, 1)))
    model.add(BatchNormalization())
    model.add(MaxPooling1D(pool_size=(3), strides=2, padding='same'))
    
    model.add(Flatten())
    model.add(Dense(64, activation='relu'))
    model.add(Dense(64, activation='relu'))
    model.add(Dense(4, activation='softmax'))
    
    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model
logger = CSVLogger('logs.csv', append=True)
his = model.fit(X_train, y_train, epochs=30, batch_size=32, 
          validation_data=(X_test, y_test), callbacks=[logger])  
</code></pre>
<p>Converted pytorch code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.conv1 = nn.Conv1d(4,32, kernel_size=2, padding=2)
        self.bn1 = nn.BatchNorm1d(32)
        self.pool1 = nn.MaxPool1d(kernel_size=2, stride=2, padding=1)
        
        self.conv2 = nn.Conv1d(32, 32, kernel_size=2, padding=2)
        self.bn2 = nn.BatchNorm1d(32)
        self.pool2 = nn.MaxPool1d(kernel_size=2, stride=2, padding=1)
        
        self.conv3 = nn.Conv1d(32, 32, kernel_size=2, padding=2)
        self.bn3 = nn.BatchNorm1d(32)
        self.pool3 = nn.MaxPool1d(kernel_size=2, stride=2, padding=1)
        
        self.fc1 = nn.Linear(32 * 4, 32)
        self.fc2 = nn.Linear(32, 32)
        self.fc3 = nn.Linear(32, 3)

        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.pool1(x)
        
        x = self.conv2(x)
        x = self.bn2(x)
        x = self.relu(x)
        x = self.pool2(x)
        
        x = self.conv3(x)
        x = self.bn3(x)
        x = self.relu(x)
        x = self.pool3(x)
        
        x = torch.flatten(x, 1)
        
        x = self.fc1(x)
        x = self.relu(x)
        
        x = self.fc2(x)
        x = self.relu(x)
        
        x = self.fc3(x)
        x = self.softmax(x)
        return x

import torch.optim as optim

criterion = nn.CrossEntropyLoss()

optimizer = optim.Adam(model.parameters(), lr=0.01)

for epoch in range(20):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        #print (inputs.shape)
        #print (labels.shape)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        #if i % 2000 == 1999:    # print every 2000 mini-batches
        print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.6f}')
        #running_loss = 0.0

print('Finished Training')
</code></pre>
<p>The keras model trains very well and gives very good accuracy on test data. But, in case of the Pytorch model the loss goes on increasing during training on the same data split. Please let me know what I am doing wrong here.</p>

username: sebastian-sz
cooked: <p>Hi <a class="mention" href="/u/padmaksha_roy">@Padmaksha_Roy</a><br>
I could spot a few issues with the torch version:</p>
<p>Issue 1: There are quite a few differences between the Keras model and Torch model.</p>
<p>For example, the first conv layer has 64 filters in Keras but only 32 in Pytorch. The kernel_size also differs.<br>
Another example: the first dense layer <code>self.fc1</code> has 64 units in Keras but only 32 units in Pytorch.</p>
<p>This model is rather compact - it is possible (and it would be beneficial) to go over each layer and make sure that it’s correctly replicated.</p>
<p>Issue 2: Loss function expects raw logits.<br>
In Pytorch, the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">Cross Entopy Loss</a> expects raw logits, while you are providing probabilities:</p>
<pre><code class="lang-python">x = self.fc3(x)
x = self.softmax(x)
return x
</code></pre>
<p>It would help to remove the softmax, if you plan to use CrossEntropy.<br>
Alternatively, you can keep the softmax and use <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html" rel="noopener nofollow ugc">negative log likelihood</a> which expects log-probabilities: <code>criterion(outputs.log(), labels)</code></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sebastian-sz" data-post="2" data-topic="178145">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sebastian-sz/48/48336_2.png" class="avatar"> sebastian-sz:</div>
<blockquote>
<p>Alternatively, you can keep the softmax and use <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html">negative log likelihood </a> which expects log-probabilities: <code>criterion(outputs.log(), labels)</code></p>
</blockquote>
</aside>
<p>NIY: for a better numerical stability you should use <code>F.log_softmax</code> instead and pass it to <code>nn.NLLLoss</code>.</p>
<p><a class="mention" href="/u/padmaksha_roy">@Padmaksha_Roy</a> additionally to the differences <a class="mention" href="/u/sebastian-sz">@sebastian-sz</a> already pointed out you are also applying the <code>relu</code> after the conv in Keras but after the batchnorm layers in PyTorch.</p>