Significant differences between Keras/TensorFlow and Torch

RubixCube: <p>Hi all,</p>
<p>After several years of applying Deep Learning using Keras/TensorFlow, I recently tried to convert a rather simple image classification task from TensorFlow/Keras to PyTorch/Lightning. Basically, everything works, however Torch is not hitting the same accuracy as Keras does. After spending about two weeks of comparing and analyzing - mostly based on topics I found here - without resolving the issue, I decided to ask here. I’m trying to explain my application step by step, without missing anything important as well as skipping all trivial stuff.</p>
<p>In Keras, the images are loaded / processed using a custom class derived from <code>tf.keras.utils.Sequence</code>. In Torch, I created an equivalent <code>torch.utils.data.Dataset</code> class that is passed to a <code>torch.utils.data.DataLoader</code> instance. The batches are loaded in both ways using the same code (wrapped in the respective class) such that data loading and processing is the same. Additionally, correct loading of the images was verified by showing the images after loading. Before forwarding by the model, they are scaled to [0,1] by a mutliplication with 1/255 in either case.</p>
<p>The model used is a simple CNN consisting only of Conv2D, ReLU, MaxPool and fully connected layers. The conversion is relatively straightforward:</p>
<pre><code class="lang-auto">tf.keras.layers.Conv2D(channels, kernel_size, activation=tf.relu),
tf.keras.layers.MaxPool2D((2,2)),
</code></pre>
<p>translates to</p>
<pre><code class="lang-auto">nn.Conv2d(channels_in, channels_out, kernel_size),
nn.ReLU(),
nn.MaxPool2d((2,2))
</code></pre>
<p>and similarly after flatten the output after the convolution part,</p>
<pre><code class="lang-auto">tf.keras.layers.Dense(size, activation=tf.nn.relu)
</code></pre>
<p>translates to</p>
<pre><code class="lang-auto">nn.Linear(size_in, size_out),
nn.ReLU()
</code></pre>
<p>both used with a <code>tf.keras.models.Sequential</code> and <code>nn.Sequential</code> model, respectively.<br>
Verifying both models using model.summary() and torchinfo.summary() shows the same layer structure, i.e. same shapes with an identical number of parameters.</p>
<p>Here, I noticed two significant differences between Keras and Torch that were a bit tricky: In Keras, the final softmax classification layer is included in the model and the loss computation, whereas in Torch, the loss computation expects unsoftmaxed logits. Additionally, the layer weights and biases are initialized differently. Keras uses zeros for the biases and Xavier uniform for the weights, the Torch equivalents are</p>
<pre><code class="lang-auto">torch.nn.init.zeros_(layer.bias)
torch.nn.init.xavier_uniform_(layer.weight)
</code></pre>
<p>applied to all layers that contain parameters (i.e. convolution and linear).</p>
<p>In Keras, the model was compiled (<code>model.compile()</code>) using a default Adam optimizer and trained using <code>model.fit()</code>.</p>
<p>To do the same in Torch, I’ve implemented a small lightning.LightningModule class. The optimizer is the same, Adam uses <code>eps=1e-8</code> in Torch be default, changed this to <code>1e-7</code> as in Keras, everything else is the same. In particular, learning rate is the same in both settings (0.001), the same holds for the batch size. Passing this configuration to a <code>lightning.Trainer()</code> yields pretty much the same training functionality as Keras.</p>
<p>The image data used consitis of ~100k images, randomly stratified partitionied using sklearn’s <code>train_test_split</code> into 80% training / 20% validation data. In Keras, this yields the following losses and accuracies during the first ten epochs (i.e. running loss and accuracy during each epoch)</p>
<pre><code class="lang-auto">loss: 0.1162 - accuracy: 0.9580
loss: 0.0507 - accuracy: 0.9823
loss: 0.0402 - accuracy: 0.9858
loss: 0.0345 - accuracy: 0.9880
loss: 0.0304 - accuracy: 0.9891
loss: 0.0276 - accuracy: 0.9904
loss: 0.0251 - accuracy: 0.9909
loss: 0.0242 - accuracy: 0.9914
loss: 0.0229 - accuracy: 0.9917
loss: 0.0213 - accuracy: 0.9924
</code></pre>
<p>whereas Torch yields (step outputs removed):</p>
<pre><code class="lang-auto">[acc=0.934, loss=0.134]
[acc=0.975, loss=0.050]
[acc=0.981, loss=0.0386]
[acc=0.982, loss=0.0332]
[acc=0.984, loss=0.0299]
[acc=0.985, loss=0.0285]
[acc=0.986, loss=0.0262]
[acc=0.987, loss=0.024]
[acc=0.987, loss=0.0236]
[acc=0.988, loss=0.0225]
</code></pre>
<p>Obviously, the task is solved well in both cases, but Keras yields about 0.5% higher accuracy. The same holds for the validation error after 20-40 epochs. Keras yields ~99.5 % validation accuracy, Torch ~99 % “only”. I mean, 99 % accuracy is still fine, but nevertheless 99.5 % is half the error rate, so more than random. Most importantly, these results are reproducible even with repeated trainings, i.e. randomization effects exist (train / test split, image shuffling during epoch, etc.), still the loss / accuracy results are relatively stable. In particular, Keras keeps outperforming Torch by 0.5 % validation accuracy (99.5 % vs. 99.0 %).</p>
<p>Furthermore, even using Torch’s default initialization (i.e. removing explicit Xavier uniform and zeros initialization of weights and biases) does not change anything here. To me, there is something significantly different during training with Keras and Torch. I read that Keras applies an internal learning rate decay during each epoch (cf. <a href="https://pyimagesearch.com/2019/07/22/keras-learning-rate-schedules-and-decay/" class="inline-onebox" rel="noopener nofollow ugc">Keras learning rate schedules and decay - PyImageSearch</a>) - not to confuse with a learning rate scheduler <em>after</em> each epoch that easily can be implemented in both frameworks - I suspect that this is one of the most significant differences. Or does Torch perform the same learning rate updates during each epoch? And if not, how can this ideally be implemented in Torch? Or are there any other significant differences one should be aware of?</p>
<p>Appreciating any comment and thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="RubixCube" data-post="1" data-topic="181932">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/8e7dd6/48.png" class="avatar"> RubixCube:</div>
<blockquote>
<p>Or does Torch perform the same learning rate updates during each epoch?</p>
</blockquote>
</aside>
<p>No, PyTorch itself won’t apply learning rate updates behind your back. I’m unsure if Lightning has such a functionality.</p>
<p>I would recommend tryin to load the Keras parameters into the PyTorch model directly (instead of trying to initialize both models in the same way) which would allow you to compare the outputs for the same samples to narrow down where the difference is coming from.</p>

RubixCube: <p>Thanks for your comment! Indeed, using Keras’ weights in Torch is a good idea. I tested this, however only the first epoch’s training accuracy increases to the same level as Keras. In particular, Keras yields about 95-96 % running training accuracy during the first epoch, whereas Torch only yields about 93.0-93.5 %. Copying Keras initialization weights, Torch yields 95 % training accuracy after the first epoch. However, the remaining accuracies are comparable to using Torch’s initialization. Thus, Keras seems to better initialize the weights than Torch (even though they should be equivalent…?) but this only boosts the first epoch’s accuracy, while the convergence stays pretty much the same. Similarly, Torch’ yields ~99% validation accuracy after 20 epochs.</p>
<p>This brings me back to my previous guess, in particular because Torch does not apply it: I need to implement a learning rate decay during each epoch, i.e. somewhere in my lightning module instance. What’s the right or most elegant way to implement this?</p>

ptrblck: <aside class="quote no-group" data-username="RubixCube" data-post="1" data-topic="181932">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/8e7dd6/48.png" class="avatar"> RubixCube:</div>
<blockquote>
<p>I read that Keras applies an internal learning rate decay during each epoch (cf. <a href="https://pyimagesearch.com/2019/07/22/keras-learning-rate-schedules-and-decay/">Keras learning rate schedules and decay - PyImageSearch</a>) - not to confuse with a learning rate scheduler <em>after</em> each epoch that easily can be implemented in both frameworks</p>
</blockquote>
</aside>
<p>I’m unsure what the main difference with this approach to a standard learning rate scheduler would be as you could just call <code>sceduler.step()</code> in each iteration instead each epoch.<br>
Assuming you can easily implement the decay formula in a custom scheduler (or use an available one) I wouldn’t assume to see any special treatment.</p>

RubixCube: <p>Thanks for your comment! I tried to implement this in Torch and while inspecting the parameters Keras uses, I noticed that they are zero. Thus, also Keras should not change the learning rate during the epoch <em>unless this is explicitly enabled by the user</em>. In particular, there has to be a different reason for my issue.</p>
<p>Going one step back to your previous suggestion, I will try to initialize the models in both frameworks the same and eliminate any randomization during training (shuffles, train/test splits, no data augmentation). In this way, the results should be almost identical - same data, same model, same initialization, same optimizer, no randomization - or is there any internal randomization that needs to be controlled as well? I hope this way I can find the difference.</p>

RubixCube: <p>After further analyzing the issue, I found an issue with the accuracy computation in Torch / Lightning. For the moment I don’t know whether this completely resolves my issue. However, it yields the following results:</p>
<pre><code class="lang-auto">import torch
from torchmetrics.classification import MulticlassAccuracy
from torchmetrics import Accuracy

acc1 = MulticlassAccuracy(num_classes=4)
acc2 = Accuracy(task='multiclass', num_classes=4)
preds = torch.Tensor([[1, 0, 0, 0], [1, 0, 0, 0]])
targets = torch.Tensor([0, 0])

acc1(preds, targets) # 0.25 -&gt; ???
acc2(preds, targets) # 1    -&gt; correct
</code></pre>
<p>For some reason, <code>MulticlassAccuracy</code> computes something wrong here. I don’t want to claim that this is a bug in Torch, maybe the class has to be initialized differently, but at least it is an easy pitfall.</p>
<p>Due to this issue, my previously computed accuracies were incorrect. After using <code>Accuracy</code> instead, the accuracies are significantly improved. Further tests to follow…</p>

ptrblck: <p>Great finding!<br>
By default it seems <code>MulticlassAccuracy</code> will use the <code>average='macro'</code> setting, which will:</p>
<blockquote>
<ul>
<li><code>macro</code>: Calculate statistics for each label and average them</li>
</ul>
</blockquote>
<p>according to the <a href="https://torchmetrics.readthedocs.io/en/stable/classification/accuracy.html?highlight=multiclassaccuracy#multiclassaccuracy">docs</a>.<br>
Also take a look at their example which shows:</p>
<pre data-code-wrap="python"><code class="lang-python">from torchmetrics.classification import MulticlassAccuracy
target = torch.tensor([2, 1, 0, 0])
preds = torch.tensor([2, 1, 0, 1])
metric = MulticlassAccuracy(num_classes=3)
metric(preds, target)

mca = MulticlassAccuracy(num_classes=3, average=None)
mca(preds, target)
</code></pre>
<p>In this example <code>3</code> classes are used where each prediction indicates the class label.<br>
You can also use a floating point tensor representing logits or probabilities. Internally <code>torch.argmax</code> will then be used to create the targets.<br>
Based on your code I assume your <code>preds</code> tensor indicates if a class is “active” or not.<br>
If you pass it as a floating point tensor into the unreduced metric you would get:</p>
<pre data-code-wrap="python"><code class="lang-python">acc1 = MulticlassAccuracy(num_classes=4, average=None)
preds = torch.tensor([[1., 0, 0, 0], [1, 0, 0, 0]])
targets = torch.Tensor([0, 0])

acc1(preds, targets)
#  tensor([1., 0., 0., 0.])
</code></pre>
<p>which corresponds to a perfect accuracy for <code>class0</code> and a zero accuracy for all other classes as these are missing.<br>
The <code>macro</code> average will then average the accuracies of all classes yielding <code>0.25</code>.<br>
If you use <code>micro</code>:</p>
<blockquote>
<ul>
<li><code>micro</code>: Sum statistics over all labels</li>
</ul>
</blockquote>
<p>the result should match your other approach and will return <code>tensor(1.)</code>.</p>
<p>At least that’s how I understand the docs.</p>

RubixCube: <p>Indeed, you are right, I checked the documentation, too, but I don’t understand the difference between <code>average='micro'</code> and <code>average='weighted'</code> so far. Presumably, I can supply weights for the latter somewhere, but both alternatives seem to compute the correct accuracy - in contrast to the default <code>'macro'</code>. However, it is unclear to me why this is used as a default… In particular, since <code>Accuracy(task='multiclass')</code> appears to be equvialent but actually computes something different (in fact the correct value). In my opinion, the default of <code>MulticlassAccuracy</code> should be changed and I find it rather surprising that nobody else ran into this issue before (or maybe people did without really noticing?). Or what do you think about this?</p>
<p>With regard to my issue, the results of Torch and Keras are reasonably close to each other using correct accuracy computation, as expected. I did not finish all my tests yet, still I guess that my issue is mostly resolved. It seems as there never really was a real difference, only my accuracy statistics showed me something wrong.</p>

ptrblck: <p>I don’t know how these default arguments were selected, but note that both metrics are “correct” from their respective perspective. Indeed it might be confusing and you could certainly ask the library authors why the <code>macro</code> average was selected as the default (maybe in their GitHub repository).</p>

RubixCube: <p>Yes agreed, I don’t want to claim it’s a bug as I wrote before, still it is a very easy pitfall. Besides this, so far all my experiments confirmed comparable results between Torch and Keras, i.e. my issue most likely was caused by incorrect accuracy computation. In summary, I think these are the most important aspects while converting Keras code to Torch code, just if someone else might run into similar issues:</p>
<ol>
<li>Do not add a softmax prediction layer, this is included in the cross entropy loss function.</li>
<li>Use <code>Accuracy(task='multiclass')</code> instead of <code>MulticlassAccuracy()</code> or at least verify that the latter computes the correct accuracy, i.e. use the right <code>'average'</code> option.</li>
<li>Layer initialization is different: Keras uses Xavier uniform for weights and zero for the biases whereas Torch uses Kaiming uniform.</li>
<li>Default parameters are sometimes slightly different (e.g. Adam optimizer’s <code>eps</code> parameter).</li>
</ol>
<p>Besides this, converting TensorFlow/Keras to PyTorch/Lightning is relatively straightforward. So thanks for your support <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>