Does torch.load() affect another model?

username: stnoh
cooked: <p>Greetings,<br>
I recently tried to save/load several parts (defined as nn.Sequential() ) of the model, like:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, ):
        super(MyModel, self).__init__()
        self.sequential_1 = nn.Sequential( ... )
        ...
        self.sequential_N = nn.Sequential( ... )
        
    def forward(self, x):
        x1 = self.sequential_1(x)
        ...
        z = self.sequential_N(xn)
        return z

model = MyModel()
</code></pre>
<p>and save this like this:</p>
<pre><code class="lang-auto">torch.save(model.sequential_1.state_dict(), './data/sequential_1.pth')
...
torch.save(model.sequential_N.state_dict(), './data/sequential_N.pth')
</code></pre>
<p>I’m not fully sure the save process above is valid, so I also saved my whole model as a backup additionally.</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), './data/model.pth')
</code></pre>
<p>Once I got those .pth files, I tried to reload the previous status, like:</p>
<pre><code class="lang-auto">model_2 = MyModel() ## for checking
model_2.sequential_1.load_state_dict(torch.load('./data/sequential_1.pth'))
...
model_2.sequential_N.load_state_dict(torch.load('./data/sequential_N.pth'))
</code></pre>
<p>but this <code>model_2</code> doesn’t work as I expected.<br>
So, I had no choice and tried to reload the whole model in a conventional way, like:</p>
<pre><code class="lang-auto">model.load_state_dict(torch.load('./data/model.pth'))
</code></pre>
<p>This <code>model</code> works as I expected. However, the weird problem starts from here.<br>
I ran <code>model_2</code> because I wanted to double-check its behavior, but I saw this works PROPERLY now <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>My questions are followings:</p>
<ol>
<li>Is this an expected behavior?</li>
<li>What is the proper way to save/load the part of the model with torch.save/load?</li>
</ol>
<p>I am also trying to reproduce this with a small-sized toy example…</p>

username: JuanFMontesinos
cooked: <p>It’s probably about optimizer’s state dict rather than model’s. Unless you forgot to save some parameter not contained in sequential_N.</p>
<p>Realize some optimizers compute online statistics which are loss. If you train to resume training but lost those statistics you will see a peak in the loss plot.</p>

username: stnoh
cooked: <p>Hello Juan,</p>
<p>I only saved the model’s information <code>torch.save(model.state_dict(), ...)</code> in the first session, and there is no optimizer in the second (load) session since I just evaluate the model.</p>

username: stnoh
cooked: <p>ok… I figured out what was happening. Basically, most problems came from my misusing.</p>
<p>In my model definition, some parts were already instantiated model, like:</p>
<pre><code class="lang-auto">class Sequential_X(nn.Module):
        ...

sequential_x = Sequential_X()
sequential_x.requires_grad = False ## instantiated in the global scope
...

class MyModel(nn.Module):
    def __init__(self, ):
        super(MyModel, self).__init__()
        ...
        self.sequential_x = sequential_x ## &lt;- it was the source of problem :(
   ...
</code></pre>
<p>I thought that <code>sequential_x</code> was fixed by <code>sequential_x.requires_grad=False</code>, but it was trained anyway (maybe I forgot to set <code>sequential_x.requires_grad=False</code> before the training session and changed the code later?) so I only saved/loaded some modules that I thought trained at this session. Therefore, the first result was not that I expected.</p>
<p>After this, I loaded the whole model by <code>model.load_state_dict(torch.load('./data/model.pth'))</code> , it also updates the globally defined <code>sequential_x </code>, so both <code>model</code> and <code>model_2</code> work fine.</p>