Different results on CUDA and CPU

AresEkb: <p>Here is a sample notebook: <a href="https://www.kaggle.com/code/aresekb/mnist-using-pytorch-cnn-32-16" class="inline-onebox" rel="noopener nofollow ugc">MNIST using PyTorch CNN 32,16 | Kaggle</a><br>
It’s a MNIST digit recognizer based on CNN.<br>
It works good enough on CUDA locally. But it seems that it gives worse results on CPU (either locally or on the Kagle server).</p>
<p>You can see results of last code block in the notebook. Predicted number for several first images is 5. Also you can see full results: <a href="https://www.kaggle.com/code/aresekb/mnist-using-pytorch-cnn-32-16/data?select=submission.csv" class="inline-onebox" rel="noopener nofollow ugc">MNIST using PyTorch CNN 32,16 | Kaggle</a> The prediction for first 32 images is 5. There are more differences in results between cuda and cpu, but results for first 32 images looks very weird.</p>
<p>Maybe something is wrong with my model and it can be fixed somehow? Or it’s ok, just a difference between devices?</p>

ptrblck: <p>Numerical differences between devices are expected and correspond to the used <code>dtype</code>. E.g. for <code>float32</code> you would expect to see a relative error of <code>~1e-6</code> (this also depends on the used operations etc., but might be a good first estimate). While some predictions could change your description might point to a valid issue, since it seems that all 32 predictions are returning the same class on the CPU, which differs from the GPU predictions.<br>
I also assume that you are using the same pretrained model and data for this comparison. If so, could you calculate the training and validation accuracy on both devices and check the gap between these?</p>

AresEkb: <p>Yes, I use the same input data. And don’t use pretrained model, it’s trained in the notebook. I split the input data to training (90%) and validation sets (10%).</p>
<p>The results are very close:</p>
<p>CUDA:<br>
train_accuracy=0.985<br>
train_loss=0.0519<br>
validation_accuracy=0.981<br>
validation_loss=0.057<br>
kaggle_accuracy=0.97978</p>
<p>CPU:<br>
train_accuracy=0.986<br>
train_loss=0.0474<br>
validation_accuracy=0.981<br>
validation_loss=0.0609<br>
kaggle_accuracy=0.97771</p>
<p>kaggle_accuracy is calculated for file containing prediction ‘5’ for first 32 images on CPU. kaggle_accuracy is not very different for CPU and CUDA. But anyway first predictions are very weird.</p>

ptrblck: <p>What’s the difference between <code>train_accuracy</code> and <code>kaggle_accuracy</code>?</p>

AresEkb: <p>train_accuracy is calculated on 90% of input data with known classes used for model training (the calculation is done in my notebook)</p>
<p>validation_accuracy is calculated on 10% of input data with known classes used for model validation (the calculation is done in my notebook)</p>
<p>kaggle_accuracy is calculated on data with <strong>unknown classes</strong> (the calculation is done by Kaggle server, because I don’t know right classes)</p>

AresEkb: <p>It seems that the problem is related to prediction (not to model training):</p>
<ol>
<li>I trained the model on CUDA</li>
<li>Saved the model to file</li>
<li>Loaded it on CPU</li>
<li>And I see the same problem: class ‘5’ for first images</li>
</ol>

AresEkb: <p>I think I’ve found a bug in my notebook. I guess it’s not related to model itself, but to transfer of tensors between devices. I’ll try to locate it and clarify the question.</p>

AresEkb: <p>Sorry for spam. The problem was caused by inplace modification of input tensor.</p>
<p>The forward() method of my model contained line: x /= 255</p>
<p>It works fine after I replaced it by: x = x / 255</p>
<p>Here is the simplified version of my original notebook:</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

# device = torch.device('cuda')
device = torch.device('cpu')

torch.manual_seed(42)

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, dtype=torch.float32)
        self.conv2 = nn.Conv2d(32, 16, kernel_size=3, dtype=torch.float32)
        self.linear1 = nn.Linear(400, 10, dtype=torch.float32)

    def forward(self, x):
        # x = x / 255
        x /= 255 # The problem was here
        x = (F.max_pool2d(self.conv1(x), 2))
        x = (F.max_pool2d(self.conv2(x), 2))
        x = x.view(-1, 400)
        x = F.log_softmax(self.linear1(x), dim=1)
        return x

model = Model().to(device)

X_test = torch.rand(20, 1, 28, 28, dtype=torch.float32) * 255
test_dataset = torch.utils.data.TensorDataset(X_test)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=100)

# The following code calls Model::forward() method
# and modifies X_test in the following line
# x /= 255
with torch.no_grad():
    for i in range(0, 10):
        model(X_test[i].to(device))

with torch.no_grad():
    for [data] in test_loader:
        data = data.to(device)
        output = model(data)
        print(output.data.max(1)[1])
</code></pre>
<p>If you ran it on CPU, first 10 results will be always the same.</p>
<p>When I ran the model on CUDA, the input tensor was modified on CUDA. And the original input tensor on CPU stayed unmodified. But when I ran the model on CPU, the forward() method modified the original input tensor.</p>
<p>I think it’s a newbie problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Maybe it will be helpful for someone. It’s better to avoid inplace tensor modifications.</p>

ptrblck: <p>Great debugging and good to hear you’ve isolated the issue. Thanks for sharing the root cause!</p>