Performance of keras and Pytorch comparison

Hassan_Imani: <p>I converted my code from keras to pytorch. It is a regression task with 3D CNNs.<br>
I used just one thread in keras, but in pytorch I am using 16 threads.<br>
Can this issue be the reason for the increase of my loss?<br>
I am using SGD optimizer and MSE loss function.</p>

ptrblck: <p>No, the loss shouldn’t depend on the number of threads (inter-op/intra-op).<br>
I would recommend to check the model architecture as well as the initialization of all random parameters as they differ between the frameworks.</p>
<p>If the loss is still diverging in PyTorch, you could load the Keras parameters into your PyTorch model and make sure to get the same output.</p>

Hassan_Imani: <p>Thank you so much <a class="mention" href="/u/ptrblck">@ptrblck</a>  who I learned pytorch with your comments here.</p>
<p>Actually my model in keras (for stereoscopic video processing) is like bellow:</p>
<p>…<br>
some code<br>
.<br>
.<br>
.<br>
combined = concatenate([left.output, right.output])</p>
<pre><code>combined = Conv3D(128, (3, 3, 3), activation='relu', strides=1, kernel_initializer='he_uniform' , padding='same')(combined)

combined = BatchNormalization()(combined)

combined = Conv3D(256, (3, 3, 3), activation='relu', strides=1 , kernel_initializer='he_uniform' , padding='same')(combined)

combined = BatchNormalization()(combined)

combined = MaxPooling3D(pool_size=(2, 2, 2))(combined)

combined = Flatten()(combined)

# apply a FC layer and then a regression prediction on the

# combined outputs

z = Dense(128, activation="relu")(combined)

z = BatchNormalization()(z)

z = Dropout(0.5)(z)

z = Dense(32, activation="relu")(z)

z = Dense(1, activation="linear")(z)

model = Model(inputs=[left.input, right.input], outputs=z)

return model
</code></pre>
<p>What I did not change in pytorch is the “kernel_initializer=‘he_uniform’”<br>
I dont know how to set it in pytorch.<br>
Do you think it can be the source of my big loss?<br>
Regards</p>

ptrblck: <p>It could be the reason for an initially different loss value, but I’m not sure if it could also explain the same loss at the beginning which diverges later.</p>
<p>You can use the <code>torch.nn.init</code> methods via <code>model.apply</code> as e.g. shown <a href="https://discuss.pytorch.org/t/how-are-layer-weights-and-biases-initialized-by-default/13073/25">here</a>.</p>

Hassan_Imani: <p>I did the initialization but still my code in pytorch is not the performance of keras code<br>
My pytorch class is as bellow:</p>
<p>class combineNet(nn.Module):</p>
<pre><code>def __init__(self, modelA, modelB):

    super(combineNet, self).__init__()

    self.modelA = modelA

    self.modelB = modelB

    

    # conv

    self.conv1 = nn.Conv3d(2048, 128, kernel_size=(3,3,3), stride=1,  padding=( 0,1,2))

    self.BN1 = nn.BatchNorm3d(128, eps=0.001, momentum=0.99)

    self.conv2 = nn.Conv3d(128, 256, kernel_size=(3,3,3), stride=1,  padding=(2,1,0))

    self.BN2 = nn.BatchNorm3d(256, eps=0.001, momentum=0.99)

    self.MP = nn.MaxPool3d(2)

    self.fc1 = nn.Linear(1024, 256, bias=False)#

    self.BN = nn.BatchNorm1d(256, eps=0.001, momentum=0.99)

    self.dr = nn.Dropout(0.5)

    self.classifier1 = nn.Linear(256, 128, bias=False)#

    self.classifier2 = nn.Linear(128, 1, bias=False)#

    for m in self.modules():

            if isinstance(m, nn.Conv3d):

                m.weight = nn.init.kaiming_normal(m.weight, mode='fan_out')

            elif isinstance(m, nn.BatchNorm3d) or isinstance(m, nn.BatchNorm1d):

                m.weight.data.fill_(1)

                m.bias.data.zero_()

def forward(self, x1, x2):

    x1 = self.modelA(x1)

    x2 = self.modelB(x2)

    x = torch.cat((x1, x2), 1)#, dim=1

    x = x.view(x.size(0),2048,4,4,2)

    x = F.relu(self.conv1(x))

    x = self.BN1(x)

    x = F.relu(self.conv2(x))

    x = self.BN2(x)

    x = self.MP(x)

    x = x.view(x.size(0), -1)

    x = self.BN(F.relu(self.fc1(x))) #self.pool 

    x = self.dr(x)

    x = F.relu(self.classifier1(x))

    x = self.classifier2(x)

    return x
</code></pre>
<p>Any help is appreciated.</p>

ptrblck: <p>If the initialization adaption didn’t help, you could try to load the Keras parameters into the PyTorch model and make sure the same (random) input creates the same outputs.<br>
If that’s the case, the difference should come from the preprocessing.</p>

Hassan_Imani: <p>My problem still persists!</p>
<p>Actually the loss in keras constantly decreases to the minimum loss(about 0.2).<br>
But in pytorch the loss starts from about 5.0 and decreases to aboout 0.8 abd stays in 0.8±0.1 forever…<br>
I changed learning rate from 0.1 to 0.0000001 and the same results!</p>
<p>The model and the initializations are the same. The loss function and SGD are the same.<br>
The number of trainable parameters are same for both keras and pytorch.</p>
<p>I think something really wrong with pytorch.<br>
Any help…</p>

Hassan_Imani: <p>I have another question related to my loss.<br>
My batch size is 128, and my target and model outputs are with size 128.<br>
I want to optimize the model with all 128 entries, not their mean or sum values.<br>
How can I do that?</p>
<p>I used reduction=none but I dont know how to apply this loss</p>

Hassan_Imani: <p>Initialization solved most part of my loss.<br>
Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a></p>