Meaningless results after loading a saved model with scheduler

username: avalon1511
cooked: <p>Hi, I trained a GAN on a dataset and then saved it using <code>torch.save()</code> . However when I load this saved model and use the generator to generate results, they are completely meaningless.</p>
<p>While loading I am using</p>
<pre><code class="lang-auto">generator.load_state_dict(path['basic_celeba_gen'], strict = False)
</code></pre>
<p>I had to set this flag to “False” because while training the generator I am using a Scheduler that updates the weights at certain epochs and if I don’t use the False flag, I am getting an error. However, I think this is what is causing the generation to fail after loading the model.</p>
<p>How do I fix this?</p>

username: ptrblck
cooked: <p>Using <code>strict=False</code> ignores missing or unexpected keys in the <code>state_dict</code> and should not be triggered by the learning rate scheduler, so I also think that using this argument is causing the issue.<br>
Could you post the error message and explain why you think the lr scheduler is causing it?</p>

username: avalon1511
cooked: <p>Hi, thanks for the response. This is my error message:</p>
<pre><code class="lang-auto">RuntimeError: Error(s) in loading state_dict for Generator:                                                                                                                                                                                          Missing key(s) in state_dict: 
"main.0.weight", "main.1.weight", "main.1.bias", "main.1.running_mean", "main.1.running_var", 
"main.3.weight", "main.4.weight", "main.4.bias", "main.4.running_mean", "main.4.running_var", 
"main.6.weight", "main.7.weight", "main.7.bias", "main.7.running_mean", "main.7.running_var", 
"main.9.weight", "main.10.weight", "main.10.bias", "main.10.running_mean", "main.10.running_var", 
"main.12.weight".                                               
Unexpected key(s) in state_dict: "module.main.0.weight", "module.main.1.weight", 
"module.main.1.bias", "module.main.1.running_mean", "module.main.1.running_var", 
"module.main.1.num_batches_tracked", "module.main.3.weight", "module.main.4.weight", 
"module.main.4.bias", "module.main.4.running_mean", "module.main.4.running_var", 
"module.main.4.num_batches_tracked", "module.main.6.weight", "module.main.7.weight", 
"module.main.7.bias", "module.main.7.running_mean", "module.main.7.running_var", 
"module.main.7.num_batches_tracked", "module.main.9.weight", "module.main.10.weight", 
"module.main.10.bias", "module.main.10.running_mean", "module.main.10.running_var", 
"module.main.10.num_batches_tracked", "module.main.12.weight". 
</code></pre>
<p>I think the scheduler is causing my outputs to be meaningless because I had run the same training previously without the scheduler and didn’t have this error when loading the saved model and my outputs were what I would expect. I looked up a different thread on this forum by someone who had the same error and the suggested solution was to use <code>strict = False</code> to fix this error. But as you said, I think this argument is causing issues.</p>
<p>Alternatively, I am not sure how to account for missing weights when I load my trained model.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="avalon1511" data-post="3" data-topic="126371">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/avalon1511/48/45884_2.png" class="avatar"> avalon1511:</div>
<blockquote>
<p>and the suggested solution was to use <code>strict = False</code> to fix this error</p>
</blockquote>
</aside>
<p>Yeah, you should be a bit careful about using this argument, as it’s ignoring the (valid) error and is thus not used by default.</p>
<p>Your issue seems to be caused by <code>nn.DataParallel</code>, which will add the <code>.module</code> tag to each parameter and buffer.<br>
You could thus either remove the <code>.module</code> tags manually from the <code>state_dict</code>, wrap the model into <code>nn.DataParallel</code> before loading the <code>state_dict</code>, or store the <code>state_dict</code> from the underlying model using <code>torch.save(model.module.state_dict(), PATH)</code>.</p>

username: avalon1511
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="126371">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Yeah, you should be a bit careful about using this argument, as it’s ignoring the (valid) error and is thus not used by default.</p>
</blockquote>
</aside>
<p>Thanks, will keep this in mind!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="126371">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>or store the <code>state_dict</code> from the underlying model using <code>torch.save(model.module.state_dict(), PATH)</code>.</p>
</blockquote>
</aside>
<p>I am not sure what you mean by <code>.module</code> here, do you mean to pass <code>torch.save()</code> as</p>
<pre><code class="lang-auto">torch.save(m.module.state_dcit(), PATH)
</code></pre>
<p>where <code>m</code> is an instance of my model class?</p>

username: ptrblck
cooked: <p>Yes, since the <code>.module</code> keywords will be added by <code>nn.DataParallel</code> you could store the internal model’s <code>state_dict</code> by accessing <code>m.module.state_dict()</code> where <code>m</code> is the <code>nn.DataParallel</code> object:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1)
print(model.state_dict())
&gt; OrderedDict([('weight', tensor([[-0.8219]])), ('bias', tensor([-0.6302]))])

model = nn.DataParallel(model)
print(model.state_dict())
&gt; OrderedDict([('module.weight', tensor([[-0.8219]], device='cuda:0')), ('module.bias', tensor([-0.6302], device='cuda:0'))])

print(model.module.state_dict())
&gt; OrderedDict([('weight', tensor([[-0.8219]], device='cuda:0')), ('bias', tensor([-0.6302], device='cuda:0'))])
</code></pre>

username: avalon1511
cooked: <p>Thanks for clarifying! Following your advice I  was getting an error trying to do what you suggested:</p>
<pre><code class="lang-auto">AttributeError: 'Generator' object has no attribute 'module'
</code></pre>
<p>And when I printed my <code>state_dict()</code> using <code>print(Generator.state_dict())</code> I got this:</p>
<pre><code class="lang-auto"> ('main.10.weight', tensor(&lt;size&gt;),  device='cuda:0')
</code></pre>
<p>So there was no <code>module</code> term in my weights.</p>
<p>But, I think I fixed it!</p>
<p>Checked my model class definition and found I had the condition:</p>
<pre><code class="lang-auto">if (device.type == 'cuda') and ngpu &gt; 1:

    netG = nn.DataParallel(netG, list(range(ngpu)))
</code></pre>
<p>So I simply removed the condition of <code>ngpu &gt; 1</code> and it works fine now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks again!</p>