Need help on Autoencoder implementation for Credit Card Fraud Detection

username: vimalkansal
cooked: <p>I am trying to reproduce the Credit Card Fraud Detection using autoencoders done in this git repo (<a href="https://github.com/curiousily/Credit-Card-Fraud-Detection-using-Autoencoders-in-Keras" rel="nofollow noopener">https://github.com/curiousily/Credit-Card-Fraud-Detection-using-Autoencoders-in-Keras</a>) and my network looks like :<br>
class Autoencoder(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Autoencoder, self).<strong>init</strong>()<br>
self.encoder = nn.Sequential(<br>
nn.Linear(29, 14),<br>
nn.Tanh(),<br>
nn.Linear(14, 7),<br>
nn.ReLU())</p>
<pre><code>    self.decoder = nn.Sequential(
        nn.Linear(7, 14),
        nn.Tanh(),
        nn.Linear(14, 29),
        nn.ReLU())

def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x
</code></pre>
<p>My training loop looks like</p>
<p>for epoch in range(num_epochs):<br>
for data in train_loader:<br>
<span class="hashtag">#print</span>(type(data))<br>
<span class="hashtag">#data</span> = Variable(data).cpu()<br>
<span class="hashtag">#print</span>(type(data))<br>
# ===================forward=====================<br>
output = model(data)<br>
loss = criterion(output, data)</p>
<pre><code>    # ===================backward====================
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
# ===================log========================
print('epoch [{}/{}], loss:{:.4f}'
      .format(epoch + 1, num_epochs, loss.item()))
history['train_loss'].append(loss.item())
</code></pre>
<p>torch.save(model.state_dict(), ‘./credit_card_model.pth’)</p>
<p>Yet my training loss behaviour is completely different to what has been achieved using Keras. In my case it looks like model is not learning at all, whereas in case of Keras, there is a smooth degradation of training loss. Can somebody have a look at what I am doing wrong. All other steps like data prep, preprocessing etc are exactly same.</p>
<p>Thx</p>
<p>Vimal</p>

username: ptrblck
cooked: <p>Based on this Keras model definition:</p>
<pre><code class="lang-python">input_layer = Input(shape=(input_dim, ))

encoder = Dense(encoding_dim, activation="tanh", 
                activity_regularizer=regularizers.l1(10e-5))(input_layer)
encoder = Dense(int(encoding_dim / 2), activation="relu")(encoder)

decoder = Dense(int(encoding_dim / 2), activation='tanh')(encoder)
decoder = Dense(input_dim, activation='relu')(decoder)

autoencoder = Model(inputs=input_layer, outputs=decoder)
</code></pre>
<p>if looks like your <code>decoder</code> is wrong and should be:</p>
<pre><code class="lang-python">self.decoder = nn.Sequential(
    nn.Linear(7, 7)
    nn.Tanh(),
    nn.Linear(7, 29)
    nn.ReLU()
)
</code></pre>
<p>Could you check that and let me know, if I’m misunderstanding the Keras code.</p>

username: vimalkansal
cooked: <p>Trouble is even with that I see lot of divergence in the first few epochs itself <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c8aaeb6003aa6c5f0fdf8a2939ae37f31cdf2c49.png" alt="image" data-base62-sha1="sDbGBVWLB6gg9XPVNl5s7ZivSSl" width="606" height="414"></p>
<p>whereas with keras I get this<br>
Epoch 1/100<br>
227451/227451 [==============================] - 12s 53us/step - loss: 0.8129 - acc: 0.5749 - val_loss: 0.7844 - val_acc: 0.6528<br>
Epoch 2/100<br>
227451/227451 [==============================] - 11s 50us/step - loss: 0.7395 - acc: 0.6744 - val_loss: 0.7644 - val_acc: 0.6766<br>
Epoch 3/100<br>
227451/227451 [==============================] - 11s 50us/step - loss: 0.7279 - acc: 0.6842 - val_loss: 0.7599 - val_acc: 0.6891<br>
Epoch 4/100<br>
227451/227451 [==============================] - 11s 50us/step - loss: 0.7217 - acc: 0.6943 - val_loss: 0.7530 - val_acc: 0.6989<br>
Epoch 5/100<br>
227451/227451 [==============================] - 11s 50us/step - loss: 0.7169 - acc: 0.7050 - val_loss: 0.7488 - val_acc: 0.7073<br>
Epoch 6/100<br>
227451/227451 [==============================] - 11s 50us/step - loss: 0.7129 - acc: 0.7113 - val_loss: 0.7442 - val_acc: 0.7064<br>
Epoch 7/100<br>
227451/227451 [==============================] - 11s 50us/step - loss: 0.7105 - acc: 0.7130 - val_loss: 0.7432 - val_acc: 0.7029<br>
Epoch 8/100<br>
227451/227451 [==============================] - 11s 50us/step - loss: 0.7086 - acc: 0.7162 - val_loss: 0.7413 - val_acc: 0.7142<br>
Epoch 9/100<br>
227451/227451 [==============================] - 12s 51us/step - loss: 0.7074 - acc: 0.7172 - val_loss: 0.7398 - val_acc: 0.7151<br>
Epoch 10/100<br>
227451/227451 [==============================] - 11s 50us/step - loss: 0.7065 - acc: 0.7165 - val_loss: 0.7400 - val_acc: 0.7114</p>
<p>So with Keras, I amseeing a nice smooth fall in training loss.</p>

username: vimalkansal
cooked: <p>Training is about to finish I will post the loss curve for both</p>

username: ptrblck
cooked: <p>Is Keras printing the training loss for the current batch or is the average taken?<br>
The loss in your PyTorch model also seems to go down, but note that you are only printing the loss of the last seen training batch.<br>
The usage of an <code>AverageMeter</code>from the <a href="https://github.com/pytorch/examples/blob/1de2ff9338bacaaffa123d03ce53d7522d5dcc2e/imagenet/main.py#L287" rel="nofollow noopener">ImageNet example</a> might smooth the values out a bit.</p>

username: vimalkansal
cooked: <p>Curve from Keras training (look at the blue color one :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/002a23744e7cd5cad35853c1885aedc1fcce51ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/002a23744e7cd5cad35853c1885aedc1fcce51ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/002a23744e7cd5cad35853c1885aedc1fcce51ee_2_690x425.png" alt="image" data-base62-sha1="1shrl3ARdlJ78xsi50wCR0xFee" width="690" height="425" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/002a23744e7cd5cad35853c1885aedc1fcce51ee_2_690x425.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/002a23744e7cd5cad35853c1885aedc1fcce51ee_2_1035x637.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/002a23744e7cd5cad35853c1885aedc1fcce51ee.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1060×654 85.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: vimalkansal
cooked: <p>And the one from PyTorch :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/296e0c6e3ba8e1da9b79e4bd13910fa0e202e914.png" data-download-href="https://discuss.pytorch.org/uploads/default/296e0c6e3ba8e1da9b79e4bd13910fa0e202e914" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/296e0c6e3ba8e1da9b79e4bd13910fa0e202e914_2_690x498.png" alt="image" data-base62-sha1="5UvlXFZ3iKJtIH0JwghbQ79XUR6" width="690" height="498" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/296e0c6e3ba8e1da9b79e4bd13910fa0e202e914_2_690x498.png, https://discuss.pytorch.org/uploads/default/original/2X/2/296e0c6e3ba8e1da9b79e4bd13910fa0e202e914.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/296e0c6e3ba8e1da9b79e4bd13910fa0e202e914.png 2x" data-dominant-color="F5F7FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">920×664 92.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: vimalkansal
cooked: <p>Keras documentation says :</p>
<p>"history object records training loss values  at successive epochs "   So given this definition, I believe it is not the average. I am really wondering now that why should the behaviour of 2 simple networks represented in two different frameworks be so different.</p>

username: vimalkansal
cooked: <p>I think may be you are right. Keras does seem to do some funky stuff in its model.fit metod - I just had a cursory glance. I will dig it up tomorrow morning . In the mean time,</p>

username: ptrblck
cooked: <p>The differences might come from a lot of sources:</p>
<ul>
<li>different model architectur</li>
<li>different parameter initialization</li>
<li>regularization</li>
<li>optimizer</li>
</ul>
<p>I would start by comparing the initialization methods and write a custom init method for your PyTorch model if necessary.<br>
Also, I’m not sure what <code>activity_regularizer=regularizers.l1(10e-5)</code> means in your Keras model.<br>
Does it take the L1-norm of the output activations and add it to the loss?</p>

username: vimalkansal
cooked: <p>Actually in Keras code I set activity_regularizer to None, just to make it equivalent to PyTorch implementation . Didn’t see much difference in terms of behaviour i.e Keras implementation gave me a smoothly decaying loss graph as compared to PyTorch</p>

username: ptrblck
cooked: <p>That’s a good start.<br>
Did you also change the initialization?<br>
Apparently Keras’ Dense layer uses Xavier init for the weights and zeros for the bias, while PyTorch uses [these] init methods.</p>

username: vimalkansal
cooked: <p>Model architecture is exactly same, after you pointed out the decoder architecture<br>
parameter initialization also I have kept same<br>
I removed regularization from Keras implementation<br>
optimizer is same Adam.</p>

username: ptrblck
cooked: <p>Could you post your current PyTorch code so that we could also have a look and debug it?</p>

username: vimalkansal
cooked: <p>Here you go : <a href="https://github.com/vimalkansal/credit_card_fraud_detection" rel="nofollow noopener">https://github.com/vimalkansal/credit_card_fraud_detection</a></p>

username: vimalkansal
cooked: <p>Since yesterday I have made some great progress. It  now compares very well with Keras version</p>

username: ptrblck
cooked: <p>Good to hear!<br>
What changed did you make to get similar results?</p>

username: vimalkansal
cooked: <p>Very similar results ! So after making change to the model I was still getting the same weird loss graph. Then I realised within an epoch, I was updating the loss with the loss at the end of every batch. So at the end of the epoch the loop was retaining the loss at the end of the last batch of an epoch. So I just retained all the losses during an epoch and then took a mean of those losses as the loss for the epoch.</p>
<p>Thanks a lot man, for being agreat help - I wouldn’t have figured out that decoder layer issue.</p>

username: vimalkansal
cooked: <p>Now I have the issue of my Reconstruction error  (farud vs non Fraud) histograms, ROC and Precision-Recall looking totally different from what I saw in this Keras implementation.  My source code is at <a href="https://github.com/vimalkansal/credit_card_fraud_detection" rel="nofollow noopener">https://github.com/vimalkansal/credit_card_fraud_detection</a></p>