Losing accuracy

username: Ali_Elhenidy20222
cooked: <p>Hi, when train my model on a GPU and save weights.<br>
Loading the saved weights on CPU, it lacks accuracy and doing so bad.<br>
Is it normal?</p>

username: ptrblck
cooked: <p>No, that’s not expected.<br>
Comapre the outputs of a static tensor (e.g. <code>torch.ones</code>) on the GPU vs. CPU after calling <code>model.eval()</code> and before saving the <code>state_dict</code>. If these are close (up to floating point precision), compare the outputs again after loading the model.</p>

username: Ali_Elhenidy20222
cooked: <p>wright now, I am training a sign language model on mu own created dataset<br>
the traing accuracy is like that<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce406c17417b62354e638ff972e88dae9174c49f.png" alt="image" data-base62-sha1="tqAnJSBrca3ZitdeSKtmqq7hi91" width="646" height="404"></p>
<p>the validation accuracy is like the same</p>
<p>but in testing code it is so bad<br>
any hint</p>
<p>and this the testing code</p>
<p><span class="hashtag-raw">#test</span><br>
img = cv2.imread(r"C:\Users\Ali\Sign_Language_Project\test\20211229_234757.jpg")<br>
img = cv2.resize(img , (512,512))<br>
plt.imshow(img)<br>
img_ten = torch.from_numpy(img)<br>
img_ten = img_ten.to(device)<br>
img_ten = torch.permute(img_ten , (2,1,0))<br>
img_ten = torch.unsqueeze(img_ten,0)<br>
img_ten = img_ten.type(torch.cuda.FloatTensor)<br>
pred = m(img_ten)</p>
<p>y_pred_softmax = torch.log_softmax(pred, dim = 1)<br>
_, y_pred_tags = torch.max(y_pred_softmax, dim = 1)<br>
print(y_pred_softmax)<br>
print(y_pred_tags)<br>
print("prediction is: " , class_names[y_pred_tags])</p>

username: Ali_Elhenidy20222
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="144701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.ones</p>
</blockquote>
</aside>
<p>could explain more<br>
I pass a tensor of one to the model then comapre results?</p>

username: ptrblck
cooked: <p>Yes, exactly. Passing a static input such as <code>torch.ones</code> using the GPU as well as the CPU and comparing the results before and after saving could give us a hint where the issue in your code might be.<br>
E.g. if the results are equal before saving but differ after loading the model, it could point to an issue of the model loading itself.</p>

username: Ali_Elhenidy20222
cooked: <p>on CPU before loading<br>
tensor([[  1.8160,  -6.7850,   5.1841,  -3.4524,  -7.2334,  -1.0573,  -8.7593,<br>
-7.5111,  -5.3923,  -8.3448,  -6.9471, -11.9200,  -5.1619,  -1.3001,<br>
-9.2441, -16.4653,  -6.6711,  -6.5694, -10.5172,  -4.6673,  -8.1703,<br>
-8.2364, -14.3871, -14.1084,  -0.4286,  -5.9760,  -8.5350,  -9.5102,<br>
-5.8201, -13.8306,  -8.8035,  -4.9221,  -4.7478,  -7.7635,  -2.6563,<br>
-6.3988,  -3.0070,  -3.6580]], grad_fn=)<br>
on GPU before loading<br>
tensor([[ -5.4878,  -7.7286,  -4.0034,  -3.4907,  -7.5340,  -6.5319, -15.8143,<br>
-7.1648, -12.7269, -14.6478, -14.2912, -14.9882, -18.0514,  -5.6059,<br>
-6.8539, -18.3004,  -9.7909,  -5.0261, -11.3465,  -3.6464,  -8.3862,<br>
-12.2398, -18.5062, -15.1331,  -2.9380,  -6.7910, -14.1230,  -8.9430,<br>
-15.2894, -10.7991,  -0.5428,  -7.2093,  -4.6537,  -6.5283,  -7.3082,<br>
-17.5702,  -5.5608,  -7.3001]], device=‘cuda:0’,<br>
grad_fn=)</p>

username: ptrblck
cooked: <p>Could you post the model definition as well as the output of <code>python -m torch.utils.collect_env</code>, please?<br>
Also, I assume you’ve called <code>model.eval()</code> before running the comparison as previously described?</p>

username: Ali_Elhenidy20222
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="144701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>python -m torch.utils.collect_env</p>
</blockquote>
</aside>
<p>Python platform: Windows-10-10.0.19042-SP0<br>
Is CUDA available: True<br>
CUDA runtime version: 10.2.89<br>
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2060<br>
Nvidia driver version: 466.11<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>

username: ptrblck
cooked: <p>Thanks, we would still need the model definition, please.</p>

username: Ali_Elhenidy20222
cooked: <p>m = models.resnet50(pretrained=True)<br>
num_ftrs = m.fc.in_features<br>
m.fc = nn.Linear(num_ftrs, 38)</p>

username: ptrblck
cooked: <p>Thanks. I cannot reproduce the issue using <code>1.10.2+cu113</code> on a <code>TitanV</code> and <code>RTX2080Ti</code> using this code snippet:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torchvision.models as models

m = models.resnet50(pretrained=True).eval()
num_ftrs = m.fc.in_features
m.fc = nn.Linear(num_ftrs, 38)

x = torch.ones(10, 3, 224, 224)
out = m(x)

m.cuda()
x = x.cuda()
out_gpu = m(x)
print((out_gpu.cpu() - out).abs().max())
</code></pre>
<p>and get abs. errors of <code>~1e-6</code>.</p>