Transfer learning in parallel NNs

Mari: <p>I am doing parallel NN in PINN and I need to use transfer learning but I do not know how I can use it? Can someone please guide me on how to use different pre-trained networks in parallel NN?</p>
<p>``</p>
<pre><code>class Net2_kc(nn.Module): 

    def __init__(self):
        super(Net2_kc, self).__init__()

         
        self.k3 = nn.Sequential(
            nn.Linear(input_n,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,1),
     
        )


        self.k4 = nn.Sequential(
            nn.Linear(input_n,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),             
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,1),
     
        )


        self.k5 = nn.Sequential(
            nn.Linear(input_n,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),             
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,1),
     
        )
        
        
        self.k6 = nn.Sequential(
            nn.Linear(input_n,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),             
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,1),
     
        )
        
        self.k7 = nn.Sequential(
            nn.Linear(input_n,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),             
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,hidden_dim),
            Swish(),
            # nn.ReLU(),
            nn.Linear(hidden_dim,1),
     
        )


       
        self.final = nn.Sequential(
            #nn.Linear(2,1),  #Doing just 2,1 gives very bad results!!!!!!!!!
            nn.Linear(2,50),
            nn.Linear(50,1),

        )



   
    def forward(self,x):  
        k0 = self.k3(x)
        k1 = self.k4(x)
        k2 = self.k5(x)
        k3 = self.k6(x)
        k4 = self.k5(x)

        output = torch.cat(k0,k1,k2,k3,k4), dim=1 )
        output2 = self.final(output)
        
        return  output2
</code></pre>
<p>``</p>
<p>How can i use pretrained network for k3,k4,k5,k6, and k7 separately?</p>

Mari: <p>Do i need to do s.t. like this:</p>
<p>net2_kc.k3.load_state_dict(torch.load(path))<br>
net2_kc.k4.load_state_dict(torch.load(path))</p>

ptrblck: <p>Yes, you could index the internal submodules and load the pre-trained <code>state_dict</code>s.</p>

Mari: <p>Can you please give me an example?</p>

ptrblck: <p>Your posted code already would be a minimal example assuming you have stored the pre-trained <code>state_dict</code>s for each submodule before.</p>

Mari: <p>net2_kc.k3.load_state_dict(torch.load(path))<br>
net2_kc.k4.load_state_dict(torch.load(path))</p>
<p>This does not work. Giving below error.</p>
<p>RuntimeError: Error(s) in loading state_dict for Sequential: Missing key(s) in state_dict: “0.weight”, “0.bias”, “2.weight”, “2.bias”, “4.weight”, “4.bias”, “6.weight”, “6.bias”, “8.weight”, “8.bias”. Unexpected key(s) in state_dict: “main.0.weight”, “main.0.bias”, “main.2.weight”, “main.2.bias”, “main.4.weight”, “main.4.bias”, “main.6.weight”, “main.6.bias”, “main.8.weight”, “main.8.bias”.</p>

Mari: <p>ptrblk,</p>
<p>What is this error for?</p>

ptrblck: <p>The error is raised since your current <code>k3</code> or <code>k4</code> modules use a different structure (i.e. they are <code>nn.Sequential</code> containers) while the <code>state_dict</code> contains a <code>main</code> module, which seems to use an <code>nn.Sequential</code> container internally.<br>
You could either change the <code>state_dict</code> keys to match the newly create submodule or change the module to use the <code>main</code> attribute.</p>

Mari: <p>The pre-trained network size maches the submodule.<br>
The pre-trained network k3 is like this:<br>
``<br>
class k3(nn.Module):</p>
<pre><code>    def __init__(self):
        super(k3, self).__init__()
        self.main = nn.Sequential(
            nn.Linear(input_n,h_nk),
            
            Swish(),
            nn.Linear(h_nk,h_nk),
            
            Swish(),
           
            nn.Linear(h_nk,h_nk),
            
            Swish(),
           
            nn.Linear(h_nk,h_nk),
            
            Swish(),
            

            nn.Linear(h_nk,1),
        )
    
    def forward(self,x):
        output = self.main(x)
        return  output
</code></pre>
<p>``<br>
here hn_k is equal to hidden_dim.<br>
I have the same NN for k4…k7. For each network, results are saved seperately and are used in state_dicts.</p>

ptrblck: <p>As you can see in the posted code snippet, the difference is that your new model uses:</p>
<pre><code class="lang-python">self.k3 = nn.Sequential(...
</code></pre>
<p>while the old module was defined as a custom <code>nn.Module</code> <code>k3</code> with a <code>self.main</code> attribute containing the <code>nn.Sequential</code>.<br>
If you want to directly load a <code>state_dict</code>, make sure the architectures match exactly.<br>
Otherwise you would have to manipulate e.g. the <code>state_dict</code> keys.</p>

Mari: <p>Right. But here if i want to train all pre-trained networks seperately how can i dfine the pre-trained nn to match the submodule?</p>

ptrblck: <p>In <code>Net2_kc</code> use <code>self.k3 = k3(...)</code>.</p>

Mari: <p>Sorry, i did not understand. How k3 is defined in <code>Net2_kc</code>, <code>self.k3 = k3(...)</code> ?</p>

ptrblck: <p>Here is a minimal code snippet showing how to load the previously stored <code>k3</code> <code>state_dict</code>:</p>
<pre><code class="lang-python">class k3(nn.Module):
    def __init__(self):
        super(k3, self).__init__()
        self.main = nn.Sequential(
            nn.Linear(10, 10)
        )
    
    def forward(self,x):
        output = self.main(x)
        return  output


class Net2_kc(nn.Module): 
    def __init__(self):
        super(Net2_kc, self).__init__()
        self.k3 = k3()

    def forward(self,x):  
        k0 = self.k3(x)        
        return  k0

# store state_dict from original k3 layer
model = k3()
sd = model.state_dict()

# load into new model
model = Net2_kc()
model.k3.load_state_dict(sd)
# &gt; &lt;All keys matched successfully&gt;
</code></pre>

Mari: <p>Thanks ptrblck,</p>
<p>The reason why i was confused about <code>k3()</code> it was that i did not define class K3 in my current code. By defining it, it is working now.</p>
<p>Thanks again for your help.</p>