Big difference in keras and pytorch results

username: talhaanwarch
cooked: <p>Here is the keras architecture</p>
<pre><code class="lang-auto">model=Sequential()

model.add(Dense(units=255,activation='relu',input_shape=[21]))

model.add(Dense(units=128,activation='relu'))

model.add(Dense(units=64,activation='relu'))

model.add(Dense(units=32,activation='relu'))

model.add(Dense(units=10,activation='softmax'))
</code></pre>
<p>Here is the pytorch architecture</p>
<pre><code class="lang-auto">class nonseqV2(torch.nn.Module):

    def __init__(self):

        super(nonseqV2,self).__init__()

        self.fcLayer1=torch.nn.Linear(in_features=21, out_features=256) 

        self.fcLayer2=torch.nn.Linear(in_features=256, out_features=128)

        self.fcLayer3=torch.nn.Linear(in_features=128, out_features=64)

        self.fcLayer4=torch.nn.Linear(in_features=64, out_features=10)

        self.relu=torch.nn.ReLU()

        self.softmax=torch.nn.Softmax(dim=1)

        

    def forward(self,x):

        x=self.fcLayer1(x)

        x=self.relu(x)

        x=self.fcLayer2(x)

        x=self.relu(x)

        x=self.fcLayer3(x)

        x=self.relu(x)

        x=self.fcLayer4(x)

        x=self.softmax(x)

        return x
</code></pre>
<p>Keras compiling and training</p>
<pre><code class="lang-auto">model.compile(optimizer=Adam(0.0001),loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(x=train_modeling,y=train_target_binary,batch_size=200,epochs=50,verbose=0,validation_data=(valid_modeling,valid_target_binary))
</code></pre>
<p>Pytoch training</p>
<pre><code class="lang-auto">for ep in range(50):

    for xtb,ytb in train_loader:

        xtb,ytb=xtb.cuda(),ytb.cuda()

        tpred=model(xtb)

        loss=crossentropy(tpred,ytb.long())

        opt.zero_grad()

        loss.backward()

        opt.step()
</code></pre>
<p>Keras loss<br>
<code>log_loss(valid_target_binary,y_pred)</code><br>
Pytorch Loss<br>
<code>log_loss(val_labels,yprd.detach().cpu().numpy())</code></p>
<p>As pytorch accept 1D label, so the variable names are different, but essentially they have same data.<br>
This is how data feeded to keras is converted to pytorch tensor</p>
<pre><code class="lang-auto">val_features=torch.Tensor(valid_modeling)

val_labels=torch.Tensor(np.argmax(valid_target_binary,axis=1))
</code></pre>
<p>But loss obtained from Keras is <strong>1.4</strong> and PyTorch its <strong>10.43</strong>. I know because of initilization factor, result should be bit different, but it should not be much different.</p>

username: ptrblck
cooked: <p>Your Keras model seems to use 5 layers, while your PyTorch model only defines 4, so you might want to add the missing layer.</p>
<p>Also, <code>nn.CrossEntropyLoss</code> expects the raw logits as the model output, so you should remove the <code>softmax</code> in your PyTorch model at the end of the <code>forward</code>.</p>

username: talhaanwarch
cooked: <p>I have chnaged the keras model and pytorch model and also incorporated the chnages you have suggested<br>
<span class="hashtag-raw">#create</span> a keras model</p>
<pre><code class="lang-auto">model=Sequential()

model.add(Dense(units=256,activation='relu',input_shape=[21])) 

model.add(Dense(units=128,activation='relu'))

model.add(Dense(units=64,activation='relu')) 

model.add(Dense(units=32,activation='relu'))

model.add(Dense(units=10,activation='softmax')) 
</code></pre>
<p>Here is keras summary<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74ebef0e3dcdeea6065851a0d8987e49aba210b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/74ebef0e3dcdeea6065851a0d8987e49aba210b3" title="keras"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74ebef0e3dcdeea6065851a0d8987e49aba210b3.png" alt="keras" data-base62-sha1="gGkS5CJEaBQldiDPTYadsUG4Cxt" width="517" height="304" data-dominant-color="414141"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">keras</span><span class="informations">948×559 17.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is pytorch model<br>
<span class="hashtag-raw">#we</span> have create the same architecure in pytorch as we did in keras</p>
<pre><code class="lang-auto">class nonseqV2(torch.nn.Module):

    def __init__(self):

        super(nonseqV2,self).__init__()

        self.fcLayer1=torch.nn.Linear(in_features=21, out_features=256) 

        self.fcLayer2=torch.nn.Linear(in_features=256, out_features=128)

        self.fcLayer3=torch.nn.Linear(in_features=128, out_features=64)

        self.fcLayer4=torch.nn.Linear(in_features=64, out_features=32)

        self.fcLayer5=torch.nn.Linear(in_features=32, out_features=10)

        self.relu=torch.nn.ReLU()

        self.softmax=torch.nn.Softmax(dim=1)

        

    def forward(self,x):

        x=self.fcLayer1(x)

        x=self.relu(x)

        x=self.fcLayer2(x)

        x=self.relu(x)

        x=self.fcLayer3(x)

        x=self.relu(x)

        x=self.fcLayer4(x)

        x=self.relu(x)

        x=self.fcLayer5(x)

        #x=self.softmax(x)

        return x
</code></pre>
<p>Pytorch summary is in the attached image<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24755ea2db554578818ea2494265710d6aeb4a51.png" data-download-href="https://discuss.pytorch.org/uploads/default/24755ea2db554578818ea2494265710d6aeb4a51" title="pytorch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24755ea2db554578818ea2494265710d6aeb4a51.png" alt="pytorch" data-base62-sha1="5cwErrilgwk7axd1ZOcR9OiGi2Z" width="517" height="341" data-dominant-color="3F3F3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch</span><span class="informations">993×655 21 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The loss obtained using Keras is <strong>1.4</strong> and in PyTorch its <strong>4.0</strong>.<br>
I am expecting Pytorch output to be closer to keras output</p>

username: talhaanwarch
cooked: <p>I have made the changes, but still the difference is significant, please check notebook <a href="https://colab.research.google.com/drive/1svb65eoEypT0aIcvgeBroAjUtYFjmrN2?usp=sharing" rel="nofollow noopener">here</a></p>

username: ptrblck
cooked: <p>Once the model architecture is fixed, I would recommend to check the weight initializations, as they might be different between both frameworks.<br>
Some PyTorch init methods were directly taken from Torch7 (Lua), which might not be the state of the art anymore. I don’t know, if Keras changes the default inits regularly.</p>

username: KaneFu
cooked: <p>Any updates?  Thanks.<br>
Recently I also find pytorch gets higher loss than keras even in the simplest setting. I init the networks in the same way, the problem is still there…</p>