Performance decreases after saving and reloading the model

kenmikanmi: <p>Hi, I  trained the following model:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch

class LambdaBase(nn.Sequential):
    def __init__(self, fn, *args):
        super(LambdaBase, self).__init__(*args)
        self.lambda_func = fn

    def forward_prepare(self, input):
        output = []
        for module in self._modules.values():
            output.append(module(input))
        return output if output else input

class Lambda(LambdaBase):
    def forward(self, input):
        return self.lambda_func(self.forward_prepare(input))

model = nn.Sequential( # Sequential,
	nn.Conv2d(3,64,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Conv2d(64,64,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Dropout(0.25),
	nn.MaxPool2d((4, 4),(4, 4)),
	nn.BatchNorm2d(64,0.001,0.9,True),
	nn.Conv2d(64,128,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Conv2d(128,128,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Dropout(0.25),
	nn.MaxPool2d((4, 4),(4, 4)),
	nn.BatchNorm2d(128,0.001,0.9,True),
	nn.Conv2d(128,256,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Conv2d(256,256,(3, 3),(1, 1),(1, 1)),
	nn.ReLU(),
	nn.Dropout(0.25),
	nn.MaxPool2d((4, 4),(4, 4)),
	nn.BatchNorm2d(256,0.001,0.9,True),
	nn.Conv2d(256,128,(1, 1)),
	nn.ReLU(),
	Lambda(lambda x: x.view(x.size(0),-1)), # Reshape,
	nn.Sequential(Lambda(lambda x: x.view(1,-1) if 1==len(x.size()) else x ),nn.Linear(3072,128)), # Linear,
    #nn.Linear(3072,128)
)

</code></pre>
<p><strong>Then I load the parameters and train the model like this:</strong></p>
<pre><code class="lang-python">model.load_state_dict(torch.load("pretrained_model.pth"))

for i in epochs:
    if(i % 100 == 0):
        model.train(False)
        ...validation process...
        if(current_score &gt; best_score):
            torch.save(model.state_dict(), "best_model.pth")

    model.train(True)
    ...training process...
     
</code></pre>
<p>But, when I reload the saved model <code>best_model.pth</code>, it shows <strong>low performance</strong> as model before training, though it showed best performance when training.</p>
<p><strong>Following is how performance changes:</strong></p>
<ul>
<li>before training・・・<strong>70% accuracy</strong>
</li>
<li>after training・・・<strong>87% accuracy</strong>  =&gt; OK, that’s best score and I save the model as <code>"best_model.pth"</code>
</li>
<li>after loading <code>"best_model.pth"</code>(expected to have 87% accuracy)・・・<strong>70% accuracy</strong>
</li>
</ul>
<p>Do you know why this thing occurs?</p>
<p>Thanks.</p>

ptrblck: <p>How did you measure the performance?<br>
Have you set the model to <code>.eval()</code> before calculating the performance?</p>

kenmikanmi: <p>I used <code>model.train(False)</code> instead, then evaluated and saved the model by <code>torch.save(model.state_dict, "weights.pth")</code>.</p>
<p>Is <code>model.eval()</code> needed before saving?</p>
<p>Thanks</p>

ptrblck: <p>No, you should use it for calculating the accuracy of the validation set. Did you set it again after loading the model?</p>

kenmikanmi: <blockquote>
<p>No, you should use it for calculating the accuracy of the validation set.</p>
</blockquote>
<p>I got it. I’ll do this way.</p>
<blockquote>
<p>Did you set it again after loading the model?</p>
</blockquote>
<p>Could you tell me what “it” mean please?<br>
When I loaded saved model, I did following way:</p>
<pre><code class="lang-auto">model = MyModel()
model.load_state_dict...
model.train(True)
...learning...
model.train(False)
...evaluating...
</code></pre>

ptrblck: <p>Sorry for being not clear enough.<br>
I mean setting your model to <code>eval</code>.</p>
<p>So let me understand your workflow.<br>
You train your model and the accuracy is good for the training set.<br>
You evaluate this model, the accuracy is still good, and you save it.<br>
After loading the model you train it again.<br>
How is the training accuracy then?<br>
When evaluating again the accuracy is bad.</p>
<p>Did you use an adaptive optimizer (Adam, etc.)?<br>
If so, did you take care of lowering the learning rate?<br>
Did you also save the optimizer and reloaded it?</p>

kenmikanmi: <p>Following is answer for your reply:</p>
<hr>
<blockquote>
<p>You train your model and the accuracy is good for the training set.  <strong>=&gt; Yes</strong><br>
You evaluate this model, the accuracy is still good, and you save it.   <strong>=&gt; Yes(achieved 87% accuracy)</strong><br>
After loading the model you train it again. <strong>=&gt; No, I only loaded the model</strong><br>
When evaluating again the accuracy is bad. <strong>=&gt; Yes(70% accuracy)</strong></p>
</blockquote>
<hr>
<blockquote>
<p>Did you use an adaptive optimizer (Adam, etc.)?<br>
If so, did you take care of lowering the learning rate?<br>
Did you also save the optimizer and reloaded it?</p>
</blockquote>
<p>I used Adadelta for optimization, but I wanted to use the saved model as feature extractor, so I don’t retrain the model.</p>
<hr>

Diego: <p>Once you loaded the model did you make sure to set it in eval mode?<br>
e.g: model.eval() or model.train(False)</p>

kenmikanmi: <p>Hi, thanks for dealing with it.<br>
Yes, I set <code>model.eval()</code> before computing accuracy.</p>

0bff83efac608c536648: <p>I met the same problem on pytorch v1.0, I got auc of 0.73 on validation set but when I load the model, I only got auc of 0.51 on the same data. and I also called model.eval() before calculate the auc metric. I still don’t know why.</p>

ptrblck: <p>Could you post a (small) executable code snippet so that we could have a look at this issue?</p>

0bff83efac608c536648: <p>Sorry, when I retrain the model and restore it, the bug doesn’t appear any more. If it appears again, I will post the code.</p>
<p>ptrblck via PyTorch Forums <a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a> 于2019年5月29日周三 下午6:51写道：</p>

0bff83efac608c536648: <p>hi ptrblck, I got the reason why my model got worse when I load the model.It was because that I didn’t save the word2idx dict, so when I loaded the model, the word2idx dict are not persistent with the word2idx when model was trained. So the problem is solved when I save the word2idx also. Thanks, it was my fault.</p>

ptrblck: <p>Great it’s working and thanks for getting back! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

0bff83efac608c536648: <p>hi ptrblck,  could you help me to take a look at the topic <a href="https://discuss.pytorch.org/t/deploy-pytorch-model-on-spark/48361" class="inline-onebox">Deploy pytorch model on spark</a> . thanks very much.</p>

ptrblck: <p>No, sorry, unfortunately I’m inexperienced in Spark.</p>

0bff83efac608c536648: <p>thanks all the same~</p>

arod40: <p>I know this is an old post, but it turns Im experiencing the same issue now. It is not clear to me what you refer with word2idx dict, is this some kind of embedding in your model?</p>

0bff83efac608c536648: <p>it’s a dict of word and it’s index for putting it into embedding layer. like {‘trump’:0, ‘is’:1, ‘shit’:2,}</p>
<p>Alejandro Rodriguez Perez via PyTorch Forums <a href="mailto:noreply@discuss.pytorch.org">noreply@discuss.pytorch.org</a> 于2020年3月19日周四 上午2:42写道：</p>

arod40: <p>Thanks for your response. I actually figured out the reason for the undeterministic behaviour. It was the dataset, some indices I was creating to feed a char-embedding layer were being constructed non-deterministically.</p>