Training progressive growing network

alwynmathew: <p>I’m trying to dynamically keep adding new modules to my network after N steps as in <a href="https://arxiv.org/abs/1710.10196" rel="noopener nofollow ugc">Progressive Growing of GAN</a>.</p>
<p>But I came across this info from <a class="mention" href="/u/carl">@Carl</a> about PyTorch from <a href="https://discuss.pytorch.org/t/i-have-questions-about-training-networks-progressively-in-pytorch/21645/2">here</a>:</p>
<blockquote>
<p>Dynamically adding/removing modules is relatively easy in Tensorflow/Keras, since a graph of the model is available on the python side. In PyTorch, you cannot traverse the graph of your model to insert modules.</p>
</blockquote>
<p><a class="mention" href="/u/carl">@Carl</a> also mention that we can make:</p>
<blockquote>
<p>layers act like the identity function</p>
</blockquote>
<p>and train the network.</p>
<p><strong>Questions</strong>:</p>
<ul>
<li>How to dynamically add modules to a pre-exsisting model?</li>
<li>How to update optimizer (<code>state_dict</code>) without losing old info?</li>
<li>How can I use initially <code>nn.Identity</code> in my model and later replace it with <code>Conv2d</code> after N steps?</li>
</ul>
<p>Any help is appreciated. Thank you.</p>

alwynmathew: <p>Dynamically adding new params to optimizer seems to be harder that I though in pytorch. Even though, I found a solution to do it, <a href="https://stackoverflow.com/a/55766749" rel="noopener nofollow ugc">here</a>. This post author itself doesn’t recommend using it.</p>
<blockquote>
<p>However, here’s the reason why you should probably <em>never</em> update your optimizer like this, but should instead re-initialize from scratch, and just accept the loss of state information</p>
</blockquote>
<blockquote>
<p>So - to sum it up: I’d really recommend to try to keep it simple, and to only change a parameter as conservatively as possible, and not to touch the optimizer.</p>
</blockquote>
<p>I tried re-initialize optimizer from scratch whenever a new module is added to the network and I saw <strong>huge drop</strong> in validation which I cant sacrifice. Is there a better way to solve this problem?</p>
<p>cc: <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <aside class="quote no-group" data-username="alwynmathew" data-post="1" data-topic="105121">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alwynmathew/48/8684_2.png" class="avatar"> alwynmathew:</div>
<blockquote>
<p>How to dynamically add modules to a pre-exsisting model?</p>
</blockquote>
</aside>
<p>You can use <code>model.add_module</code>, but would also need to change the <code>forward</code> method or use this new module manually.</p>
<aside class="quote no-group" data-username="alwynmathew" data-post="1" data-topic="105121">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alwynmathew/48/8684_2.png" class="avatar"> alwynmathew:</div>
<blockquote>
<p>How to update optimizer (<code>state_dict</code>) without losing old info?</p>
</blockquote>
</aside>
<p><code>optimizer.add_param_group</code> can be used to add new parameters.</p>
<aside class="quote no-group" data-username="alwynmathew" data-post="1" data-topic="105121">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alwynmathew/48/8684_2.png" class="avatar"> alwynmathew:</div>
<blockquote>
<p>How can I use initially <code>nn.Identity</code> in my model and later replace it with <code>Conv2d</code> after N steps?</p>
</blockquote>
</aside>
<p>This might work, but I would recommend to double check, if this layer is really used (e.g. via forward hooks) and make sure the parameters are added to the optimizer.</p>

alwynmathew: <p>Thank you. This is how I want to add new blocks to the network dynamically. Can I use <code>add_module</code> as shown below?</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.modules = [convblock1, convblock2, convblock3]  # some list of modules
        self.convblock0 = self.conv(...)
        self.add_module("convblock1", self.modules[0]) # this doesn't look like the recommend way to use it 
        self.add_module("convblock2", self.modules[1])
        self.add_module("convblock3", self.modules[2])

    def forward(self,x, epoch_num):
        x = self.convblock0(x)
        if epoch_num &gt;= 2:
            x = self.convblock1(x)
        if epoch_num &gt;= 4:
            x = self.convblock2(x)
        if epoch_num &gt;= 6:
            x = self.convblock3(x)
        return x
</code></pre>
<p>OR</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.modules = [convblock1, convblock2, convblock3]  # some list of modules
        self.convblock0 = self.conv(...)

    def forward(self,x, epoch_num):
        x = self.convblock0(x)
        if epoch_num == 2: # can I use add_module inside forward?
            self.add_module("convblock1", self.modules[0])
        if epoch_num &gt;= 2:      
            x = self.convblock1(x)
        if epoch_num == 4:
            self.add_module("convblock2", self.modules[1])
        if epoch_num &gt;= 4:
            x = self.convblock2(x)
        if epoch_num == 6:
            self.add_module("convblock3", self.modules[2])
        if epoch_num &gt;= 6:
            x = self.convblock3(x)
        return x
</code></pre>
<p>Is this how I should update the optimizer after added new module to the model?</p>
<pre><code class="lang-auto">model = MyModel()
optimizer = torch.optim.Adam(model.convblock0.parameters(), learning_rate)
for epoch in range(10):
    if epoch == 2:
        new_par_dict = dict()
        new_par = model.modules[0].parameters()
        new_par_dict["params"] = new_par
        optimizer.add_param_group(new_par_dict)
    if epoch == 4:
        new_par_dict = dict()
        new_par = model.modules[1].parameters()
        new_par_dict["params"] = new_par
        optimizer.add_param_group(new_par_dict)
    if epoch == 6:
        new_par_dict = dict()
        new_par = model.modules[2].parameters()
        new_par_dict["params"] = new_par
        optimizer.add_param_group(new_par_dict)
    output = model(input, epoch)
    loss = loss_fun(output)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>But <code>add_param_group</code> expect <code>dict</code>. Is there a faster way to get param group dict of a specific layer? And also does this preserve previous state of the optimizer?</p>
<p>cc: <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

alwynmathew: <p>Why can’t I just append my new parameter tensor list to the current optimizer’s <code>state_dict</code>?</p>
<pre><code class="lang-auto">model = MyModel()
optimizer = torch.optim.Adam(model.convblock0.parameters(), learning_rate)
state_dict = optimizer.state_dict()
for epoch in range(10):
    if epoch == 2:
        new_par = model.modules[0].parameters()
        state_dict['param_groups'][0]['params'].append(new_params)
...
</code></pre>
<p>Does this preserve previous state of the optimizer? Any help is appreciated. Thank you.</p>
<p>cc: <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/smth">@smth</a></p>