Layer by layer self-supervised learning

nik1: <p>Hello Everyone,</p>
<p>I have a question regarding training the model one layer at a time in a self-supervised manner. For example, I need to train Alexnet first conv layer, then save the model. After that, load the model, freeze the weights and add another conv layer and keep on repeating the process until it trains on all the five conv layers.</p>
<p>I am not sure how to approach this task. I can think of saving the entire model instead of just saving the state_dict, and then pass the model to another class that is having one more conv layer. So, In this way, I am supposed to create 5 classes for each conv layer. In the final step, I need to load the last model contains all five layers and train it in a supervised way.<br>
Is there any other better way to do this?.</p>
<p>Could anyone help me with this?.</p>
<p>Thanks in advance.</p>
<p>Regards,</p>
<p>Nikita</p>

Eta_C: <p>I have never try that but I think it may work</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.body = nn.ModuleList() 

    def add_layer(self, layer):
        self.body.append(layer)

    def forward(self, x):
        return self.body(x)


pool = [nn.Conv2d(3, 32, 1), nn.Conv2d(32, 64, 1), nn.Conv2d(64, 128, 1)]
net = Net()
for i, layer in enumerate(pool):
    # 1. load state dict
    if i &gt; 0:
        state_dict = torch.load("{}.pt".format(i - 1))
        print(state_dict.keys())
        net.load_state_dict(state_dict)

    # 2. add layer
    net.add_layer(layer)

    # 2. init optimizer(freeze weights), scheduler, and train
    pass 

    # 3. save dict
    torch.save(net.state_dict(), "{}.pt".format(i))
</code></pre>

Eta_C: <p>By the way, PGGAN is one of the “growing layer by layer” network. Here’s its implemetation, using <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html" rel="noopener nofollow ugc">add_module</a></p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/nashory/pggan-pytorch/blob/070cc23c7f1ba57122d0140cbe129f13f8021c1d/network.py#L126-L151" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/nashory/pggan-pytorch/blob/070cc23c7f1ba57122d0140cbe129f13f8021c1d/network.py#L126-L151" target="_blank" rel="noopener nofollow ugc">nashory/pggan-pytorch/blob/070cc23c7f1ba57122d0140cbe129f13f8021c1d/network.py#L126-L151</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="126" style="counter-reset: li-counter 125 ;">
<li>def grow_network(self, resl):</li><li>    # we make new network since pytorch does not support remove_module()</li><li>    new_model = nn.Sequential()</li><li>    names = get_module_names(self.model)</li><li>    for name, module in self.model.named_children():</li><li>        if not name=='to_rgb_block':</li><li>            new_model.add_module(name, module)                      # make new structure and,</li><li>            new_model[-1].load_state_dict(module.state_dict())      # copy pretrained weights</li><li>        </li><li>    if resl &gt;= 3 and resl &lt;= 9:</li><li>        print('growing network[{}x{} to {}x{}]. It may take few seconds...'.format(int(pow(2,resl-1)), int(pow(2,resl-1)), int(pow(2,resl)), int(pow(2,resl))))</li><li>        low_resl_to_rgb = deepcopy_module(self.model, 'to_rgb_block')</li><li>        prev_block = nn.Sequential()</li><li>        prev_block.add_module('low_resl_upsample', nn.Upsample(scale_factor=2, mode='nearest'))</li><li>        prev_block.add_module('low_resl_to_rgb', low_resl_to_rgb)</li><li></li><li>        inter_block, ndim, self.layer_name = self.intermediate_block(resl)</li><li>        next_block = nn.Sequential()</li><li>        next_block.add_module('high_resl_block', inter_block)</li><li>        next_block.add_module('high_resl_to_rgb', self.to_rgb_block(ndim))</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/nashory/pggan-pytorch/blob/070cc23c7f1ba57122d0140cbe129f13f8021c1d/network.py#L126-L151" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


nik1: <p>Thank You so much for the idea. I would try this approach.</p>