Using a pretrained model on another dataset

username: snowe
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a pretrained model that I trained on ImageNet data. I stored the <code>model.state_dict()</code> and the <code>optimizer.state_dict()</code> in a checkpoint file.</p>
<p>Now I would like to train a model on VGGFace2 data with the pretrained weights of the model trained on ImageNet to speed up training. This means that I have to change some of the layers in my model. I have seen that I can add the <code>strict=False</code> flag when I load the <code>model.state_dict()</code> to only add the weights to the layers that are present in both the old and the new model. But there is no such option for the <code>optimizer.state_dict()</code>.</p>
<p>How can I resume training with both the <code>model.state_dict()</code> and <code>optimizer.state_dict()</code> on another model than they were trained on?</p>
<p>Any help is very much appreciated!</p>
<p>All the best,<br>
snowe</p>

username: CedricLy
cooked: <p>Have you checked<br>
optimizer.load_state_dict(pathload[])?</p>
<p>I think for most of the optimizer you don’t need to load their state.</p>

username: snowe
cooked: <p>Hi <a class="mention" href="/u/cedricly">@CedricLy</a>, thank you for your response!</p>
<p>What do you mean with <code>optimizer.load_state_dict(pathload[])</code>?</p>
<p>But as discussed <a href="https://discuss.pytorch.org/t/why-save-optimizer-state-dict/108185">here</a>, I think you are right. You only need to load the optimizer state if you want to continue training on the same data with the same network. Otherwise the <code>model.state_dict()</code> is sufficient.</p>

username: CedricLy
cooked: <pre><code class="lang-auto">model = TheModelClass(*args, **kwargs)
optimizer = TheOptimizerClass(*args, **kwargs)

checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

model.eval()
# - or -
model.train()
</code></pre>
<p>This is from a tutorial how to load if you want to continue a training.<br>
Check this link for more details.<br>
<a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training" rel="noopener nofollow ugc">Link</a><br>
Does this already solve your issue?</p>

username: snowe
cooked: <p>I am familiar with how to resume training and loading the <code>model.state_dict()</code> and the <code>optimizer.state_dict()</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>My problem was that I had to load the <code>state_dicts</code> into another network. Whereas for the <code>model.state_dict()</code> you can pass the <code>strict=False</code> flag to tell PyTorch to only load the parameters of the layers that are present in both the old and the new network, there is no such option for the <code>optimizer.state_dict()</code>. However, apparently this is not a problem since you don’t need the <code>optimizer.state_dict()</code> if you don’t continue on the same network.</p>
<p>Essentially, my use case does not really fall under the category ‘resume training’ but more so ‘start training from scratch’ but with better initial parameters compared to random initialisation.</p>
<p>Hope that makes sense!</p>

username: CedricLy
cooked: <p>Ah I see.</p>
<p>So theoretically it would be interesting to see what the “optimizer state” consists of and if the state is dependent on the parameters, which it tries to train.<br>
If the optimizer does not depent on the parameters there will be no issue with loading the current state with different parameters.<br>
If it is dependent, it is questionable if the dependence is separated for each parameter.<br>
In this case it may be possible to load the state of the optimizer without all parameters.</p>
<p>In the case of the optimizer having the second derivative saved in its state, it will not be possible. Or at least I would not know how .</p>