Problem in Loading the Saved model

Soumyajit_Das: <p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
import torch.nn.functional as F</p>
<p>import torchvision<br>
import torchvision.transforms as transforms</p>
<p>import numpy as np<br>
from torch.utils.data.dataset import Dataset<br>
ckpt_path=’/home/students/soumyajit_po/hsd_cnn_code/SOUMYA/checkpoint/vggCifar10TEST.pth’<br>
checkpoint=torch.load(ckpt_path)<br>
print(“Successfully Loaded”)</p>
<p>This code is giving me the following error:</p>
<p>Traceback (most recent call last):<br>
File “Checking.py”, line 16, in <br>
checkpoint=torch.load(ckpt_path)<br>
File “/home/anaconda2/envs/pytorch_v36/lib/python3.6/site-packages/torch/serialization.py”, line 387, in load<br>
return _load(f, map_location, pickle_module, **pickle_load_args)<br>
File “/home/anaconda2/envs/pytorch_v36/lib/python3.6/site-packages/torch/serialization.py”, line 574, in _load<br>
result = unpickler.load()<br>
AttributeError: Can’t get attribute ‘CifarVggnet’ on &lt;module ‘<strong>main</strong>’ from ‘Checking.py’&gt;</p>
<p>While Saving I did:<br>
if(test_acc&gt;best_accuracy):<br>
best_accuracy=test_acc<br>
print(‘Saving…’)<br>
if not os.path.isdir(‘checkpoint’):<br>
os.mkdir(‘checkpoint’)<br>
torch.save(model,PATH)</p>
<p>Please tell me where I am going wrong as I am following the instruction given in <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training</a></p>

Soumyajit_Das: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/smth">@smth</a> please help</p>

ptrblck: <p>Did you store the complete model using <code>torch.save(model)</code> or just the <code>state_dict</code> (the latter approach is recommended)?<br>
Based on the error message it looks like you’ve used the former approach.<br>
If so, you would need to restore the file structures in your current working directly as they have been while saving the model.</p>
<p>PS: Please don’t tag certain people, as this might discourage others to post an answer. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Soumyajit_Das: <p>You are right I used torch.save(model),previously. But now I did the following:-</p>
<p>I need to save 3 parameters as- <em><strong>epoch,best_acc and the model itself</strong></em>.<br>
I did it this way:-<br>
<strong>state={‘net’:model, ‘epoch’:epoch,‘accuracy’:best_acc}</strong><br>
<strong>torch.save(state,PATH)</strong></p>
<p>but when I’m loading using    <strong>checkpoint=torch.load(saved_path)</strong>,<br>
I am getting an <strong>Attribute  Error as Can’t get attribute ‘CifarVggnet’ on &lt;module ‘ <strong>main</strong> ’ from ‘Checking.py’&gt;</strong><br>
Here, <strong>CifarVggnet is the name of the model</strong> that I’m saving. And I’m <strong>loading it from Checking.py</strong>.</p>
<p>To avoid this error then I defined the model as:<br>
<strong>class CifarVggnet(nn.Module):</strong><br>
<strong>def <strong>init</strong>(self):</strong><br>
<strong>super(CifarVggnet,self).<strong>init</strong>()</strong></p>
<p>I defined this in my Checking.py from where I’m loading the file.</p>
<p>Now I can load the attributes of the saved model as <em>checkpoint[‘net’],checkpoint[‘epoch’] and checkpoint[‘accuracy’]</em>. I printed then and they are correctly printing.</p>
<p>But now I got a new problem:-<br>
The problem is that I can print the model. But I cannot access the feature part of the model.<br>
i.e., when I do <strong>print(net.features)</strong>, the error I am getting is:</p>
<p><em>Traceback (most recent call last):</em></p>
<ul>
<li>File “Checking.py”, line 27, in *</li>
<li>print(net.features)*</li>
<li>File “/home/anaconda2/envs/pytorch_v36/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 539, in <strong>getattr</strong>*</li>
<li>type(self).<strong>name</strong>, name))*<br>
<em>AttributeError: ‘CifarVggnet’ object has no attribute ‘features’</em>
</li>
</ul>
<p>How do I solve this now…Am I doing it the wrong way?</p>

ptrblck: <p>You would have to restore the complete scripts in your current working directory, i.e. with the complete model definition as explained <a href="https://pytorch.org/docs/stable/notes/serialization.html" rel="nofollow noopener">here</a>.<br>
It seems you are just creating a dummy model using</p>
<pre><code class="lang-python">class CifarVggnet(nn.Module):
    def __init__(self):
        super(CifarVggnet,self).__init__()
</code></pre>
<p>, while you would need the original model definition.</p>

Soumyajit_Das: <p>Ok thank you very much for your reply,I’m doing it.<br>
Also one more thing, I do need to save the accuracy and epoch along the model.<br>
Will it be saved by <strong>torch.save(the_model.state_dict(), PATH)</strong> itself. If so then how to access that.</p>

ptrblck: <p>I would recommend to store a <code>dict</code> with all necessary values, e.g. as given in the <a href="https://github.com/pytorch/examples/blob/99a83a26b33622945eaed0d20614943a16f45a43/imagenet/main.py#L249" rel="nofollow noopener">ImageNet example</a>.</p>

Soumyajit_Das: <p>model = CifarVggnet(*args, **kwargs)<br>
model.load_state_dict(torch.load(ckpt_path))<br>
print(model)<br>
I did that , but during loading I get error:</p>
<p>File “Checking.py”, line 17, in <br>
model = CifarVggnet(*args, **kwargs)<br>
NameError: name ‘CifarVggnet’ is not defined</p>

ptrblck: <p>Where did you define <code>CifarVggnet</code>? It looks like this error is actually thrown in:</p>
<pre><code class="lang-python">model = CifarVggnet(*args, **kwargs)
</code></pre>
<p>not in loading the <code>state_dict</code>.</p>

Soumyajit_Das: <p>Checking.py</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
import torch.nn.functional as F</p>
<p>import torchvision<br>
import torchvision.transforms as transforms</p>
<p>import numpy as np<br>
from torch.utils.data.dataset import Dataset<br>
ckpt_path=’/home/students/soumyajit_po/hsd_cnn_code/SOUMYA/checkpoint/vggCif    ar10TEST.pth’<br>
model = CifarVggnet(*args, **kwargs)<br>
model.load_state_dict(torch.load(ckpt_path))<br>
print(model)</p>
<p>I am using Checking.py to load the model saved in ckpt_path.</p>

ptrblck: <p>The definition of <code>CifarVggnet</code> is still missing.<br>
You should have some code like:</p>
<pre><code class="lang-python">class CifarVggnet(nn.Module):
    def __init__(self):
        super(CifarVggnet, self).__init__()
        self....

    def forward(self, x):
        ...
</code></pre>
<p>Either define it in the same file before creating an instance of this model or import it from another file, where it was defined.</p>

Soumyajit_Das: <p>import torch<br>
import torchvision<br>
import torchvision.transforms as transforms<br>
import os</p>
<p>transform_train = transforms.Compose([<br>
transforms.RandomCrop(32, padding=4),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),<br>
])<br>
trainset=torchvision.datasets.CIFAR10(root=’./data’,train=True,download=True,transform=transform_train)<br>
trainloader=torch.utils.data.DataLoader(trainset,batch_size=4,shuffle=True,num_workers=2)</p>
<p>transform_test = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),<br>
])<br>
testset=torchvision.datasets.CIFAR10(root=’./data’,train=False,download=True,transform=transform_test)<br>
testloader=torch.utils.data.DataLoader(testset,batch_size=4,shuffle=True,num_workers=2)</p>
<p>classes=(‘plane’,‘car’,‘bird’,‘cat’,‘deer’,‘dog’,‘frog’,‘horse’,‘ship’,‘truck’)</p>
<p>import torch.nn as nn<br>
import torch.optim as optim<br>
import torch.nn.functional as F<br>
device=torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)</p>
<p>import torchvision.models as models<br>
vggnet=models.vgg16_bn(pretrained=True)</p>
<p>import matplotlib.pyplot as plt</p>
<p>classifier=list(vggnet.classifier.children())[:-6]<br>
features=list(vggnet.features.children())</p>
<p>features.extend(nn.Sequential(nn.AvgPool2d(1, stride=1, padding=0, ceil_mode=False, count_include_pad=True)))<br>
avgpool=list(vggnet.avgpool.children())[:-1]</p>
<p>vggnet.features=nn.Sequential(*list(features))<br>
vggnet.classifier=nn.Sequential(*list(classifier))<br>
vggnet.classifier[0]=nn.Linear(512,10)<br>
vggnet.avgpool=nn.Sequential(*list(avgpool))</p>
<p>class CifarVggnet(nn.Module):<br>
def <strong>init</strong>(self,vggnet):<br>
super(CifarVggnet,self).<strong>init</strong>()<br>
self.vggnet=vggnet</p>
<pre><code>def forward(self,x):
    return self.vggnet(x)
</code></pre>
<p>model=CifarVggnet(vggnet)<br>
print(model)</p>
<p>PATH=’./checkpoint/vggCifar10TEST.pth’<br>
batch_size=128<br>
num_epochs=1<br>
model.to(device)<br>
criterion=nn.CrossEntropyLoss()<br>
optimizer=optim.Adam(model.parameters(),lr=0.0001)</p>
<p>torch.cuda.empty_cache()<br>
globaliter=-1<br>
best_accuracy=0<br>
trainLOSS=[]<br>
testACC=[]<br>
testLOSS=[]<br>
avg_train=0<br>
avg_test_loss=0<br>
for epoch in range(num_epochs):<br>
globaliter+=1<br>
running_loss=0<br>
total=0<br>
avg_train=0<br>
correct_classified=0<br>
model.train()<br>
for i,data in enumerate(trainloader):<br>
inputs,labels=data<br>
inputs,labels=inputs.to(device),labels.to(device)<br>
optimizer.zero_grad()<br>
outputs=model(inputs)<br>
loss=criterion(outputs,labels)<br>
loss.backward()<br>
optimizer.step()<br>
predicted=torch.argmax(outputs,1)<br>
total+=labels.size(0)<br>
correct_classified+=(predicted==labels).sum().item()<br>
running_loss+=loss.item()<br>
if i%200==199:<br>
print('Epoch:[%d, %5d] ’ % (epoch+1,i+1))</p>
<pre><code>train_acc=100*(correct_classified/total)
avg_train=float(running_loss)/total    
trainLOSS.append(avg_train)    
    
print('Train Accuracy:%.3f'%(train_acc))
print('Train Average Loss:%.4f'%(avg_train))

c=0
total=0
globaliter=-1
l=0
j=-1
model.eval()
with torch.no_grad():
  for data in testloader:
    globaliter+=1
    j=j+1
    inputs,labels=data
    inputs,labels=inputs.to(device),labels.to(device)
    optimizer.zero_grad()
    outputs=model(inputs)
    loss=criterion(outputs,labels)
    l=l+loss.item()
    predicted=torch.argmax(outputs,1)
    total+=labels.size(0)
    c+=(predicted==labels).sum().item()
    
test_acc=(100*c/total)
if(test_acc&gt;best_accuracy):
    best_accuracy=test_acc
    print('Saving...')
    state={'net':model,'acc':best_accuracy,'epoch':epoch}
    if not os.path.isdir('checkpoint'):
       os.mkdir('checkpoint')
   # torch.save(state,PATH)
   # torch.save(model,PATH)
    torch.save(model.state_dict(),PATH)
print('Accuracy of the network on test images:%.3f %%' % test_acc)

testACC.append(test_acc)
avg_test_loss=float(l)/total
testLOSS.append(avg_test_loss)
fig1 = plt.figure(1)        
plt.plot(range(epoch+1),trainLOSS,'r-',label='train loss')
plt.plot(range(epoch+1),testLOSS,'g-',label='test loss')
if epoch==0:
  plt.legend(loc='upper left')
  plt.xlabel('Epochs')
  plt.ylabel('Loss')
fig2 = plt.figure(2)        
plt.plot(range(epoch+1),testACC,'g-',label='test')        
if epoch==0:
  plt.legend(loc='upper left')
  plt.xlabel('Epochs')
  plt.ylabel('Testing Accuracy')    
</code></pre>
<p>print(‘Best Accuracy of the network on test images:%.3f %%’% best_accuracy)<br>
fig1.savefig(‘trainloss_vs_epoch1.png’)<br>
fig2.savefig(‘testacc_vs_epoch1.png’)</p>
<p>This is the code where I’m actually training and saving. For checking purpose I did set num_epoch=1.</p>

Soumyajit_Das: <p>As you said I made a separate class as vgg.py and kept it in a folder model</p>
<p>import torch.nn as nn<br>
import torch.optim as optim<br>
import torch.nn.functional as F<br>
import torchvision.models as models<br>
vggnet=models.vgg16_bn(pretrained=True)<br>
classifier=list(vggnet.classifier.children())[:-6]<br>
features=list(vggnet.features.children())<br>
features.extend(nn.Sequential(nn.AvgPool2d(1, stride=1, padding=0, ceil_mode    =False, count_include_pad=True)))<br>
avgpool=list(vggnet.avgpool.children())[:-1]<br>
vggnet.features=nn.Sequential(*list(features))<br>
vggnet.classifier=nn.Sequential(*list(classifier))<br>
vggnet.classifier[0]=nn.Linear(512,10)<br>
vggnet.avgpool=nn.Sequential(*list(avgpool))<br>
class CifarVggnet(nn.Module):<br>
def <strong>init</strong>(self,vggnet):<br>
super(CifarVggnet,self).<strong>init</strong>()<br>
self.vggnet=vggnet<br>
def forward(self,x):<br>
return self.vggnet(x)</p>
<p>model=CifarVggnet(vggnet)</p>
<p><strong>-----------------------------------------------------------------------------------------------------------------------------------------</strong><br>
From a separate file as Checking.py I did— from model import *</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
import torch.nn.functional as F</p>
<p>import torchvision<br>
import torchvision.transforms as transforms<br>
from model import *<br>
import numpy as np<br>
from torch.utils.data.dataset import Dataset</p>
<p>ckpt_path=’/home/students/soumyajit_po/hsd_cnn_code/SOUMYA/checkpoint/vggCif    ar10TEST.pth.tar’<br>
checkpoint=torch.load(ckpt_path,map_location=lambda storage,loc:storage)</p>
<p><strong>---------------------------------------------------------------------------------------------------------------------------------------</strong></p>
<p>The error I get is-<br>
File “Checking.py”, line 15, in <br>
checkpoint=torch.load(ckpt_path,map_location=lambda storage,loc:storage)<br>
File “/home/anaconda2/envs/pytorch_v36/lib/python3.6/site-packages/torch/serialization.py”, line 387, in load<br>
return _load(f, map_location, pickle_module, **pickle_load_args)<br>
File “/home/anaconda2/envs/pytorch_v36/lib/python3.6/site-packages/torch/serialization.py”, line 574, in _load<br>
result = unpickler.load()<br>
AttributeError: Can’t get attribute ‘CifarVggnet’ on &lt;module ‘<strong>main</strong>’ from ‘Checking.py’&gt;</p>
<p>I do not understand this error!!!<br>
CifarVggnet the model I saved , should be accessible now as I imported the model .</p>

ptrblck: <p>Are you able to create a new instance using</p>
<pre><code class="lang-python">model = CifarVggnet()
</code></pre>
<p>in <code>Checking.py</code>?</p>
<p>PS: You can post code snippets using three backticks ``` <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Soumyajit_Das: <p>No I am not able to do so.<br>
Doing that gives me the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "Checking.py", line 12, in &lt;module&gt;
    model=CifarVggnet()
NameError: name 'CifarVggnet' is not defined</code></pre>

ptrblck: <p>So apparently your <code>from model import *</code> does not import <code>CifarVggnet</code>.</p>
<p>I would recommend to avoid using the asterisk for now and try to import the module directly.<br>
If you manage to import the module and create a new instance, try to run your restoring code again.</p>

Soumyajit_Das: <p>I tried it now, but it is giving me the same error.</p>
<pre><code class="lang-auto">  1 import torch                                                                
  2 import torch.nn as nn
  3 import torch.optim as optim
  4 import torch.nn.functional as F
  5 
  6 import torchvision
  7 import torchvision.transforms as transforms
  8 from model import vgg
  9 import numpy as np
 10 from torch.utils.data.dataset import Dataset
 11 
 12 net=CifarVggnet()
 13 ckpt_path='/home/students/soumyajit_po/hsd_cnn_code/SOUMYA/checkpoint/vggCif    ar10TEST.pth.tar'
 14 #model=vgg.model
 15 #print(model)
 16 checkpoint=torch.load(ckpt_path,map_location=lambda storage,loc:storage)
</code></pre>
<p>And the error is</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "Checking.py", line 12, in &lt;module&gt;
    net=CifarVggnet()
NameError: name 'CifarVggnet' is not defined
</code></pre>
<p>This is my vgg.py inside model folder</p>
<pre><code class="lang-auto"> import torch.nn as nn                                                       
 import torch.optim as optim
 import torch.nn.functional as F
 import torchvision.models as models
 vggnet=models.vgg16_bn(pretrained=True)
 
 classifier=list(vggnet.classifier.children())[:-6] 
 features=list(vggnet.features.children())
 features.extend(nn.Sequential(nn.AvgPool2d(1, stride=1, padding=0, ceil_mode    =False, count_include_pad=True)))
 avgpool=list(vggnet.avgpool.children())[:-1]
 vggnet.features=nn.Sequential(*list(features))
 vggnet.classifier=nn.Sequential(*list(classifier))
 vggnet.classifier[0]=nn.Linear(512,10)
 vggnet.avgpool=nn.Sequential(*list(avgpool))
 
 class CifarVggnet(nn.Module):
     def __init__(self,vggnet):
         super(CifarVggnet,self).__init__()
         self.vggnet=vggnet
     
     def forward(self,x):
         return self.vggnet(x)
 
 model=CifarVggnet(vggnet)
</code></pre>

ptrblck: <p>Your script should already thrown an error in <code>from model import vgg</code>, since <code>model</code> doesn’t seem to be a valid module.<br>
I would recommend to look into <a href="https://docs.python.org/3/tutorial/modules.html" rel="nofollow noopener">Python modules</a> to get an idea how your model should be defined inside <code>vgg.py</code>.</p>
<p>That being said, I would recommend to save and load the <code>state_dict</code> approach generally.<br>
Using this approach your files could look like:</p>
<pre><code class="lang-python"># models.py
import torch
import torch.nn as nn

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc = nn.Linear(1, 1)
        
    def forward(self, x):
        x = self.fc(x)
        return x
</code></pre>
<pre><code class="lang-python"># main.py
import torch
import torch.nn as nn

from models import MyModel

model = MyModel
model.load_state_dict(torch.load('STATE_DICT_PATH'))
</code></pre>

Soumyajit_Das: <p>Thank you very much for this help. I could actually solve the problem of attribute error.<br>
I read the link you gave.<br>
Then I understood the meaning of the error that was being given.</p>
<pre><code class="lang-auto">AttributeError: Can’t get attribute ‘CifarVggnet’ on &lt;module ‘main’ from ‘Checking.py’&gt;
</code></pre>
<p>It turns out that may be the Checking.py file may not be successful in importing the model definition from vgg.py<br>
To make it work I did the following:</p>
<pre><code class="lang-auto">from model.vgg import *
</code></pre>
<p>instead of:</p>
<pre><code class="lang-auto">from model import *
</code></pre>
<p>and</p>
<pre><code class="lang-auto">from model import vgg
</code></pre>
<p>The problem got resolved due to this line.<br>
Thankyou very much <a class="mention" href="/u/ptrblck">@ptrblck</a> for bearing my non-stop comments and replies.</p>