Loading fine-tuned model

_joker: <p>I am using pre-trained resnet for fine-tuning it to adversarial examples in this way;</p>
<pre><code class="lang-auto">model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, NUM_CLASSES)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=100)
model_ft.save_state_dict('fine_tuned_best_model.pt')

</code></pre>
<p>My confusion is, when I load my model <code>fine_tuned_best_model.pt</code> back for testing, should I just give the resnet18 definition like this;</p>
<pre><code class="lang-auto">classifier = resnet18()
model_para = torch.load(CLASSIFIER_PATH)
classifier.load_state_dict(model_para['net'])
classifier = nn.DataParallel(classifier) if torch.cuda.device_count() &gt; 1 else classifier
classifier.to(device)
</code></pre>
<p>or do I need that last layer which I added for training?</p>

ptrblck: <p>You would need to restore the model in the same way, including the last layer change, to be able to properly load the <code>state_dict</code>. Otherwise you’ll get a shape mismatch for this layer in <code>load_state_dict</code>.</p>

_joker: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> Can you show me a code snippet. As I have a pre-trained resnet for CIFAR10 and I am using that to load for training with the same resnet18 definition as <code>torchvision.models.resnet18</code>. My model doesn’t learn anything and has only 10% accuracy.</p>
<p>But when I print the model, all the <code>biases=False</code> except for the last fc layer.<br>
Now when I try training the whole pre-trained model over my new data which is adversary of CIFAR10, it gives me 76% test accuracy but the original accuracy on CIFAR10 decreases from 93.33% to 87%</p>
<p>I am not sure where am I going wrong and I am unsure which one of the above methods would be good for me if I want my model to be trained for both the data (CIFAR10 and adversary of CIFAR10).</p>

ptrblck: <p>I’m not sure what code snippet you are looking for.</p>
<aside class="quote no-group" data-username="_joker" data-post="3" data-topic="91644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_joker/48/37415_2.png" class="avatar"> _joker:</div>
<blockquote>
<p>As I have a pre-trained resnet for CIFAR10 and I am using that to load for training with the same resnet18 definition as <code>torchvision.models.resnet18</code> . My model doesn’t learn anything and has only 10% accuracy.</p>
</blockquote>
</aside>
<p>I would generally recommend to try to overfit a small data sample as a quick test to check for potential bugs in the code.</p>
<aside class="quote no-group" data-username="_joker" data-post="3" data-topic="91644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_joker/48/37415_2.png" class="avatar"> _joker:</div>
<blockquote>
<p>But when I print the model, all the <code>biases=False</code> except for the last fc layer.</p>
</blockquote>
</aside>
<p>This is expected, as all <code>bias</code> parameters in the conv layers are deactivated, since they are followed by batchnorm layers and thus scaled.</p>
<aside class="quote no-group" data-username="_joker" data-post="3" data-topic="91644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/_joker/48/37415_2.png" class="avatar"> _joker:</div>
<blockquote>
<p>Now when I try training the whole pre-trained model over my new data which is adversary of CIFAR10, it gives me 76% test accuracy but the original accuracy on CIFAR10 decreases from 93.33% to 87%</p>
</blockquote>
</aside>
<p>How was the original accuracy on CIFAR10 calculated, if you can only achieve 10%?</p>

vsant: <p>Hello,</p>
<p>Regarding this point, as per <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc"> SAVING AND LOADING MODELS</a>:</p>
<blockquote>
<p>If you only plan to keep the best performing model (according to the acquired validation loss), … You must serialize <code>best_model_state</code> or use <code>best_model_state = deepcopy(model.state_dict())</code> otherwise your best <code>best_model_state</code> will keep getting updated by the subsequent training iterations. As a result, the final model state will be the state of the overfitted model.</p>
</blockquote>
<p>However, I have done something like this:</p>
<pre><code class="lang-auto">def train_model(model, ...):
       ...
       if validation_loss improves:
            delete previous best model
            torch.save(model.state_dict(), best_model_path)
       else:
             ....
        ...
        return model

 def test_model(model, best_model_path, ...):
     model.load_state_dict(torch.load(best_model_path))
     model.eval()
     ...

...

my_model = train_model(my_model, ...)

test_model(my_model, my_path, ...)
</code></pre>
<p>In other words, the model returned by the training phase is the final one which is likely to present overfitting. But since I saved the best model during training, I have no problem during the test/inference phase because I load it during testing.</p>
<p>Is something wrong with this solution?</p>
<p>Thanks.</p>