Duplicate fully connected layers and train model with new duplicated layers only

awais980: <p>I am trying to make two branches in the network as shown in picture below. L1 are layers in the original model. And L2 are cloned layers from L1. These are fully connected layers. I can add layers by <code>nn.Sequential</code>? Moreover, how I can train only L2 layers in the network while L1 layers should not be trained? It is like how I can train network with either L2 branch or L1 branch separately.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b7943f109ed003a35be6d361578a3f1ce1d9186.png" data-download-href="https://discuss.pytorch.org/uploads/default/2b7943f109ed003a35be6d361578a3f1ce1d9186" title="2020-07-06T22_01_11"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b7943f109ed003a35be6d361578a3f1ce1d9186_2_281x500.png" alt="2020-07-06T22_01_11" data-base62-sha1="6cAl9m3pP2YM1CWzBZTIy0TeFCK" width="281" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b7943f109ed003a35be6d361578a3f1ce1d9186_2_281x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b7943f109ed003a35be6d361578a3f1ce1d9186_2_421x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b7943f109ed003a35be6d361578a3f1ce1d9186_2_562x1000.png 2x" data-dominant-color="9F9A94"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2020-07-06T22_01_11</span><span class="informations">562×1000 450 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

alpha_omega_alpha: <p>To add layers in the <code>nn.Sequential</code>, you can create a class that inherits <code>nn.Sequential</code>, or you can pass in the modules directly as parameters (like here: <a href="https://pytorch.org/docs/stable/nn.html#sequential" rel="nofollow noopener">torch.nn.Sequential</a>).<br>
Re your second question, I think one approach is to set the L1 layers to be <code>.requires_grad=False</code>.</p>

awais980: <p>I have tried to put two different classifiers like this:</p>
<pre><code class="lang-auto">(classifier): Sequential(
    (0): Sequential(
      (0): Linear(in_features=1280, out_features=512, bias=True)
      (1): Tanh()
      (2): Linear(in_features=512, out_features=256, bias=True)
      (3): Tanh()
      (4): Linear(in_features=256, out_features=128, bias=True)
      (5): Tanh()
      (6): Linear(in_features=128, out_features=200, bias=True)
    )
    (1): Sequential(
      (0): Linear(in_features=1280, out_features=512, bias=True)
      (1): Tanh()
      (2): Linear(in_features=512, out_features=256, bias=True)
      (3): Tanh()
      (4): Linear(in_features=256, out_features=128, bias=True)
      (5): Tanh()
      (6): Linear(in_features=128, out_features=29, bias=True)
    )
  )
</code></pre>
<p>How I can skip  <code>classifier[0]</code>  or  <code>classifier[1]</code>  during training the model?</p>

alpha_omega_alpha: <p>I’m sure there is a better way, but I thought you could try separating them, then pas in a pointer in the <code>forward</code> method.</p>
<p>For example:</p>
<pre><code class="lang-auto">import torch

class NeuralNetwork(torch.nn.Module):
  def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.add_module('L1', torch.nn.Sequential(
      torch.nn.Linear(1280, 512, True),
      torch.nn.Tanh(),
      torch.nn.Linear(512, 256, True),
      torch.nn.Tanh(),
      torch.nn.Linear(256, 128, True),
      torch.nn.Tanh(),
      torch.nn.Linear(128, 200, True)
    )
    self.add_module('L2', torch.nn.Sequential(
      torch.nn.Linear(1280, 512, True),
      torch.nn.Tanh(),
      torch.nn.Linear(512, 256, True),
      torch.nn.Tanh(),
      torch.nn.Linear(256, 128, True),
      torch.nn.Tanh(),
      torch.nn.Linear(128, 29, True)
    )
  def forward(self, x, classifier):
    if classifier == 0:
      return self.L1(x)
    elif classifier == 1:
      return self.L2(x)

</code></pre>
<p>Then in your training loop, you could just flag in the classifier into the <code>model.(x, classifier)</code>.</p>