Does deepcopying optimizer of one model works across the model? or should I create new optimizer every time?

bhushans23: <p>I am trying to deep copy models</p>
<p><a href="https://discuss.pytorch.org/t/are-there-any-recommended-methods-to-clone-a-model/483/6">Are there any recommended methods to clone a model?</a> and <a href="https://discuss.pytorch.org/t/copying-weights-from-one-net-to-another/1492">Copying weights from one net to another</a><br>
recommends to use copy.deepcopy() which works, but shouldn’t I need to deep copy optimizer as well?</p>
<p>I am confused between following tow -</p>
<pre><code class="lang-auto">model2 = copy.deepcopy(model1)
opt2 = torch.optim.Adam(model2.parameters(), lr = 0.0001)
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">model2 = copy.deepcopy(model1)
opt2 = copy.deepcopy(opt1)   # we are deepcopying model1 so, deep copy it's optimizer
</code></pre>
<p>Which approach is better? or incorrect?</p>
<p>Thanks.</p>

jpeg729: <p>deepcopying the optimiser won’t work because it will either give you another optimiser for <code>model1</code>, or more likely simply break the references to the model parameters.</p>
<p>Here is what I would do…</p>
<pre><code class="lang-auto">model2 = Model() # get new instance
model2.load_state_dict(model1.state_dict()) # copy state
opt2 = torch.optim.Adam(model2.parameters(), lr = 0.0001) # get new optimiser
opt2.load_state_dict(opt1.state_dict()) # copy state
</code></pre>

Konpat_Ta_Preechakul: <p>Does this mean <code>opt2</code> will be just another optimizer for <code>model1</code> because it gets the <code>param_group</code> from <code>opt1</code>?</p>

bhushans23: <p>Yes.<br>
Opt2 will be different set of parameters copied at the checkpoint.</p>

hoangcuong2011: <p>Let us assume we want to copy a model and then fine-tune a model as normal after that. Let us also assume that we don’t need to resume the optimizer that was used to train the model before. We just create a totally new new optimizer and start training as in normal fine-tuning.</p>
<p>With that I am very curious what is wrong with deepcopy and why we need to use load_state_dict instead?</p>
<p>I read some discussion on <a href="http://pytorch.org" rel="nofollow noopener">pytorch.org</a> and some well-known people (e.g. <a class="mention" href="/u/ptrblck">@ptrblck</a>, <a class="mention" href="/u/alband">@albanD</a>, <a class="mention" href="/u/apaszke">@apaszke</a>, to name a few) suggest to use copy.deepcopy. But I heard some concern about that as in this post. Could someone elaborate more on this? Thanks!</p>

ptrblck: <p><code>deepcopy</code> should work in your use case, since you are not trying to copy the optimizer.<br>
I personally prefer the explicit approach in creating new objects and load the <code>state_dict</code>s I want.<br>
As you can see from the previous posts copying the model and optimizer might create new objects, but the parameter references in the optimizer wouldn’t be updated to the new model.</p>

johannes-lee: <p>For anyone who wants to copy a model (<code>model1</code>) and its optimizer (<code>optimizer1</code>) without knowing beforehand which type of optimizer it is, I use:</p>
<pre><code class="lang-auto">model2 = copy.deepcopy(model1)
optimizer2 = type(optimizer1)(model2.parameters())
optimizer2.load_state_dict(optimizer1.state_dict())
</code></pre>

johannes-lee: <p>Correction:</p>
<pre><code class="lang-auto">model2 = copy.deepcopy(model1)
optimizer2 = type(optimizer1)(model2.parameters(), lr=optimizer1.defaults['lr'])
optimizer2.load_state_dict(optimizer1.state_dict())
</code></pre>

Nikola_Andro: <p>Would we do the same if we wanted to make a copy of a scheduler?</p>
<pre><code class="lang-auto">scheduler2 = type(scheduler1)(optimizer2)
scheduler2.load_state_dict(schedduler1.state_dict())
</code></pre>