Loading and Evaluating Model

ssalome: <p>I have trained a model using resnet18 from scratch. I save a model with a minimum loss to a .pth file. The training accuracy came around 90 % and testing accuracy around 15%.<br>
On loading the file and calling evaluation(testloader, resent) or evaluation(trainloader, resnet) the percentage keeps varying each time I load the model example on calling evaluation(trainloader, resnet) i would get values like 98% 97% etc. So why does this happen?</p>

ptrblck: <p>Are you calling <code>model.eval()</code> on the model before running the evaluation?<br>
This would make sure to use the running stats in batchnorm layers (and not update them anymore) and disable dropout layers.</p>

ssalome: <p>no i wasnt calling model.eval()</p>

ssalome: <p>Also have this doubt<br>
Hi<br>
I’m currently working on a classification model whose input are x-ray images, i need to classify it into the respective disease categories. I have trained resnet 18 (not pretrained )on a data-set of 100 images per class.</p>
<p>I have 1493 images out of which 1045 for training and 448 images for testing (70:30 split).</p>
<p>On running for 50 epochs the training accuracy goes to 97.99 percent, but the testing accuracy isn’t good at all it is just about 30 %</p>
<p>this is plotting loss per epoch<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bd898fbf92c3c2b15cad0343c1b99cc24d55a90.png" alt="Screenshot (1377)" data-base62-sha1="8xqec8kx49l8cdl6F1q5Jsg6Zaw" width="498" height="346"></p>
<p>The testing accuracy is low since there arent sufficient images?</p>

ssalome: <p>I have grey scale images and i have down-scaled them to 224 x 224 so image shape is [1,224,224].<br>
original image size 1024x1024</p>
<p>I have modified the first conv2d layer to accept grey scale images and last layer of resnet to output the required number of classes.</p>

ptrblck: <p>The general workflow sounds reasonable.<br>
However, I doubt you’ll be able to train a model from scratch with just 1000 images.<br>
You could try to use a pretrained model (even though the pretraining used data from another domain) and compare the results.</p>

ssalome: <p>Oh okay will try that</p>

ssalome: <p>Also while plotting confusion matrix there’ s a problem i have only 1493 images in total and 1045 go for training and 448 for testing…but the confusion matrix rows sum up to 5100…what can be the error.</p>
<blockquote>
<p><span class="hashtag-raw">#preds</span> = torch.tensor(<span class="chcklst-box fa fa-square-o fa-fw"></span>)<br>
confusion_matrix = torch.zeros(num_classes, num_classes)<br>
<strong>def evaluation(dataloader, model):</strong><br>
model added as argument - 1 change from lenet<br>
total, correct = 0,0<br>
for data in dataloader:<br>
inputs, labels, imgindex  = data<br>
inputs, labels, imgindex = inputs.to(device), labels.to(device), imgindex<br>
# inputs are put in the gpu<br>
outputs = model(inputs)<br>
# from net to model - 2 change from lenet<br>
_, pred = torch.max(outputs.data,1)<br>
for t, p in zip(labels.view(-1), pred.view(-1)):<br>
confusion_matrix[t.long(), p.long()] +=1<br>
total += labels.size(0)<br>
correct +=(pred==labels).sum().item()<br>
return 100 * correct/total</p>
</blockquote>
<p>this is the evaluation function<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e6e3ea8bea52cc06707adb25c6a36e4e23c54b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/9e6e3ea8bea52cc06707adb25c6a36e4e23c54b8" title="Screenshot (1379)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e6e3ea8bea52cc06707adb25c6a36e4e23c54b8.png" alt="Screenshot (1379)" data-base62-sha1="mBxDAWY38pHuIvb8kvh0YKQisdO" width="640" height="500" data-dominant-color="F6F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (1379)</span><span class="informations">903×705 24.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ssalome: <p>Also what should be the weight and SD for grey scale images for using pretrained resnet 18?</p>

ptrblck: <p>Not sure, what’s causing the higher number of samples in your confusion matrix, but the code seems to be fine generally:</p>
<pre data-code-wrap="python"><code class="lang-python">m = torch.zeros(10, 10)

pred = torch.randint(0, 10, (100,))
target = torch.randint(0, 10, (100,))

for t, p in zip(target.view(-1), pred.view(-1)):
    m[t.long(), p.long()] += 1

print(m.sum())
&gt; tensor(100.)
</code></pre>
<aside class="quote no-group" data-username="ssalome" data-post="9" data-topic="71762">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/43a26b/48.png" class="avatar"> ssalome:</div>
<blockquote>
<p>weight and SD for grey scale images for using pretrained resnet 18?</p>
</blockquote>
</aside>
<p>Do you mean the mean and standard deviation for normalization?<br>
If so, you could either calculate if from your training dataset or alternatively start with <code>0.5</code> for both values.</p>

ssalome: <p>Okay . Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/+1/4.png?v=9" title=":+1:t4:" class="emoji" alt=":+1:t4:"></p>

ssalome: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Hi<br>
On importing torch, I’m getting the following error. Can you help me with this?</p>
<p>File “”, line 14, in<br>
import torch</p>
<p>File “C:\Users\SALOME\Anaconda3\lib\site-packages\torch_ <em>init</em> _.py”, line 83, in<br>
<strong>all</strong>  += [name for name in dir(_C)</p>
<p>NameError: name ‘_C’ is not defined</p>

ptrblck: <p>Have you changed / updated anything and are you using the right virtual environment (if used at all)?</p>

ssalome: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
not using virtual environment<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a4366ed62507af9c67d62a2159b42000ad9491c.png" data-download-href="https://discuss.pytorch.org/uploads/default/9a4366ed62507af9c67d62a2159b42000ad9491c" title="Screenshot (1391)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a4366ed62507af9c67d62a2159b42000ad9491c_2_690x388.png" alt="Screenshot (1391)" data-base62-sha1="m0FWcly7ZBQblRMVvgyMpgQrVLK" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a4366ed62507af9c67d62a2159b42000ad9491c_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a4366ed62507af9c67d62a2159b42000ad9491c_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a4366ed62507af9c67d62a2159b42000ad9491c_2_1380x776.png 2x" data-dominant-color="969697"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (1391)</span><span class="informations">1920×1080 258 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>since i got that error i tried reinstalling pytorch cpu version but the error still persisted</p>

ssalome: <p>This issue occurs while importing torch on spyder</p>

ssalome: <p>(base) C:\Users\SALOME&gt;python<br>
Python 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32<br>
Type “help”, “copyright”, “credits” or “license” for more information.<br>
import torch<br>
goes to next line</p>
<p>it runs fine on using the anaconda prompt but throws an error when i import it in Spyder</p>

ssalome: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0af83b8d7fa5d881708a7b6e12ef3567f9e1d565.png" data-download-href="https://discuss.pytorch.org/uploads/default/0af83b8d7fa5d881708a7b6e12ef3567f9e1d565" title="Screenshot (1392)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0af83b8d7fa5d881708a7b6e12ef3567f9e1d565.png" alt="Screenshot (1392)" data-base62-sha1="1z2BRmCQat9lSAc5mphNKUT6YxT" width="690" height="269" data-dominant-color="F0F0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (1392)</span><span class="informations">708×277 12.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>some error on line 81 in <strong>init</strong>.py</p>

ptrblck: <p>Not sure, what is creating this error, as I’m using Spyder myself without ever seeing this error.<br>
Could you check your working directory and make sure it’s not inside the PyTorch dir?</p>
<p>Also, could you create a new virtual environment and install PyTorch again with Spyder there?</p>