Save a pytorch model, reload it for transfer learning, but evaluation result is bad

username: Leonmac
cooked: <p>Dear community,<br>
I am a newbie for Pytorch so forgive whatever mistake I could make. here I meet some problem that hopefully, somebody can give some advice,</p>
<p>I am using the Cifar10 data-set and using a partitioner tool to partition the train-loader, I am pretty sure that I am using the same train-loader/test-loader for all the tasks below.<br>
This is what I did:</p>
<ol>
<li>train the model for, say 10 epochs, I get an evaluation accuracy of about 40%. Then I saved the model;</li>
<li>next,  load the model from the previously saved datafile, start the training again. On principle, I expect to start with an evaluation accuracy of ~40%. However, instead, I only get something like 15%…<br>
Note if I train the model from scratch, I would get ~10% evaluation accuracy for the 1st epoch – this is under expectation, but this also indicates my loadfile indeed helps a little, better than a totally random start model.</li>
</ol>
<ul>
<li>I wonder if  I made mistake with the model save/reload? The model save/load functions are pretty simple and follow the PyTorch standard method, as below;</li>
<li>The model is trained on the GPU (cuda), after loading the model from the file, I do have the code like the model.to(device)</li>
<li>I wrap up the model save function into my model class,</li>
</ul>
<pre><code class="lang-auto">import torch

def import_parameters_from_pt_file(file: str, model):
    '''import a model from python pt file on disk, file_name shall include path'''
    #print(f"load the file: {file}")
    model.load_state_dict(torch.load(file))
    model.eval()
    return model

def save_parameters_to_pt_file(file: str, model):
    '''save a model to python pt file on disk, file_name shall include path'''
    torch.save(model.state_dict(), file)
    print(f"saving the file: {file}")

#in my cifar10 class wrap the model save method:
class cifar10():
...
    def save_model_to_file(self, file_name: str):
        utl.save_parameters_to_pt_file(file_name, self.model)

# this is how I call the model save at end of training:
&lt;...training completed...&gt;
    file_name = f"./{local.device}-fiinal.pth"
    cifar10.save_model_to_file(file_name)
</code></pre>

username: Andrei_Cristea
cooked: <p>Greetings, try saving your optimizer’s state dict per the docs <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" rel="noopener nofollow ugc">here</a>, and see if that helps.</p>
<pre><code class="lang-auto">torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),  # &lt;-- this part
            'loss': loss,
            ...
            }, PATH)
</code></pre>
<p>If you’re using an optimizer like Adam, it is learning stuff along the way and might benefit from not being reset.</p>

username: Leonmac
cooked: <p>I am using SGD as the optimizer.<br>
I am trying to follow your suggestion. I warp a new method as below:</p>
<pre><code class="lang-auto">    def save_fullmodel_to_file(self, file_name: str):
        model_dict = {
                    #'epoch': epoch,
                    'model_state_dict':     self.model.state_dict(),
                    'optimizer_state_dict': self.optim.state_dict(),
                    #'loss': loss,
                    }
        utl.save_full_model(file_name, model_dict)
</code></pre>
<p>But when I call this method I get error report as:</p>
<pre><code class="lang-auto">    'optimizer_state_dict': self.optim.state_dict(),
TypeError: state_dict() missing 1 required positional argument: 'self'
</code></pre>
<p>the self.optim is an attribute in the cifar10 class, it is not a class, so I just don’t understand this error complain…</p>

username: Andrei_Cristea
cooked: <p>Could you show where you define <code>self.optim</code> and also the definition of <code>utl.save_full_model</code>?</p>

username: Leonmac
cooked: <p>optimiz is one of the components of the cifar10 class, which is initialized to None in <strong>init</strong>, but assigned before the  train as the pass in parameter:</p>
<pre><code class="lang-auto">self.optimz  = optim.SGD
</code></pre>
<p>and please forget the utl.save_full_model, now I simply use torch.save as below:</p>
<pre><code class="lang-auto">torch.save({self.model.state_dict(), self.optimz.state_dict()}, file_name)
</code></pre>
<p>the reported error keep the same:</p>
<pre><code class="lang-auto">   torch.save({self.model.state_dict(), self.optimz.state_dict()}, file_name)
TypeError: state_dict() missing 1 required positional argument: 'self'


</code></pre>

username: Andrei_Cristea
cooked: <p>Looks like you never initialized the optimizer, your <code>self.optimz</code> is merely a pointer to the (un-instantiated) <code>optim.SGD</code> class.</p>
<p>Try this instead and see if this helps:</p>
<pre><code class="lang-auto">self.optimz = optim.SGD(params=self.model.parameters(), lr=1e-4)
</code></pre>
<p>Also for the save, I think you should save a dict, not a set. Something like this:</p>
<pre><code class="lang-auto">torch.save({"model": self.model.state_dict(), "optim": self.optimz.state_dict()}, file_name)
</code></pre>

username: Leonmac
cooked: <p>Great, I am able to save the model + optimizer state_dict, many thanks.<br>
next I am trying to load and then check if I can get aligned performance.<br>
&lt;…&gt;<br>
Update: Yes follow the suggestion, I think my problem is solved.<br>
remain questions:</p>
<ol>
<li>Could you please help to explain a little bit: why the optimizer matter? Originally I think a model file is all…</li>
<li>If I am just interested in the inference but not transfer-learning, loading a model file would be enough–correct? because a pure test does not involve the optimizer at all…</li>
</ol>

username: Andrei_Cristea
cooked: <p>Hi Leonmac, great to hear this solved your issue.</p>
<ol>
<li>If you were using a very basic optimizer, it may not matter. However, a “stateful” optimizer learns things about the model / data as the training process unfolds, and applies that to take better optimization steps. If you restart the training process and don’t save / reload the optimizer state the optimizer loses all of its learnings, and has to start over from scratch, which means it’s going to take some awkward steps at first, damaging performance. Adam, for example, keeps track of a couple of coefficients per parameter, in essence learning the appropriate learning rate to use for each model parameter (based on moments of the gradient, detail <a href="https://arxiv.org/abs/1412.6980" rel="noopener nofollow ugc">here</a>) so it’s helpful not to lose that.</li>
<li>That’s right: load the model file, set the model to eval(), and you’re good to go.</li>
</ol>