How can I connect a new neural network after a trained neural network and optimize them together

cat-loves-donuts: <p>Hello, I am new in Pytorch and this question makes me waste a couple of days. I am trying to connect two different neural networks together. The model one is a trained NN which I have already saved as a .pth file. Then I want to put another NN with a totally different architecture after it. Then I combining those two models and train them together. How can I connect two models? I mean simply use the output from module one as the input of model two but use one optimize function to train them. Because I may put a dozen of models after them.</p>

ptrblck: <aside class="quote no-group" data-username="cat-loves-donuts" data-post="1" data-topic="48293">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cat-loves-donuts/48/13770_2.png" class="avatar"> cat-loves-donuts:</div>
<blockquote>
<p>I mean simply use the output from module one as the input of model two</p>
</blockquote>
</aside>
<p>Yes, that would just work.<br>
You can basically treat each model as a “layer” passing one output to the next layer/model.</p>
<p>If you would like to pass all parameters to the optimizer, just concatenate the lists together:</p>
<pre data-code-wrap="python"><code class="lang-python">model1 = nn.Linear(1, 1)
model2 = nn.Linear(2, 2)

optimizer = torch.optim.SGD(
    list(model1.parameters()) + list(model2.parameters()),
    lr=1e-3
)
</code></pre>

cat-loves-donuts: <p>Thank you so much. This method can solve the problem. But I want to combine the two models as one model and train and optimize them together. The reason why I want to do this is that I will connect a dozen of models one by one. So do you have any idea of how to combine different models? Thank you so much.</p>

ptrblck: <p>You can just combine them in the training loop:</p>
<pre><code class="lang-python">for data, target in train_loader:
    output = model1(data)
    output = model2(output)
    output = model3(output)

   # or use a loop instead
</code></pre>
<p>or just wrap them in another <code>nn.Module</code> and treat them as layers:</p>
<pre><code class="lang-python">class MyHugeModel(nn.Module):
    def __init__(self, models):
        super(MyHugeModel, self).__init__()
        self.models = models
        
    def forward(self, x):
        for model in self.models[:-1]:
            x = F.relu(model(x))
        x = self.models[-1](x)  # don't use relu for last model
        return x
            
class MySmallModel(nn.Module):
    def __init__(self):
        super(MySmallModel, self).__init__()
        self.lin = nn.Linear(10, 10)
        
    def forward(self, x):
        x = self.lin(x)
        return x


models = nn.ModuleList()
for _ in range(10):
    models.append(MySmallModel())

model = MyHugeModel(models)
x = torch.randn(1, 10)
output = model(x)
</code></pre>
<p>Using this approach, you can pass all parameters as <code>model.parameters()</code> to your optimizer.</p>

cat-loves-donuts: <p>Thank you so much, this is exactly what I want. Thank you.<br>
By the way, may I ask another question please? Well, I tried to make the architectures of the neural networks more complex so, in some models, some layers will skip connecting with other layers. How can I write this skip connection function? Should I do this in the forward function?</p>

ptrblck: <p>Yes, the <code>forward</code> method is the right place to add skip connections.<br>
Have a look at the <a href="https://github.com/pytorch/vision/blob/3b8a14038cf615f5e96a5c85a09c2dfdb6351746/torchvision/models/resnet.py#L65" rel="nofollow noopener">resnet implementation</a> for an example.</p>

cat-loves-donuts: <p>Thank you so much. But I faced another question. I let the function randomly choose different convolution layers with different kernel sizes. The question is I used skip connect in this function. So when it skips connecting, the size of the tensor is different and cannot add each other. I mean I tried to sum different layers output when they are skipping connected. But the size of the tensors are different and the pytorch gave me an error:</p>
<p>The size of tensor a (26) must match the size of tensor b (18) at non-singleton dimension 3</p>
<p>I know in this error, it said the size of the tensor is 18X18, but another one is 26X26. So, could you please tell me some methods to change the size of the output.</p>

ptrblck: <p>You could use an interpolation (or cropping) to create equally sized activations in a similar manner as is currently being done with <code>downsample</code> in the resnet example.<br>
Note that the optional downsampling  changed the number of channel, not the spatial size, so that you cannot copy the code directly.</p>

cat-loves-donuts: <p>Thank you so much. That is really helpful.<img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=9" title=":clap:" class="emoji" alt=":clap:"><img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=9" title=":clap:" class="emoji" alt=":clap:"><img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=9" title=":clap:" class="emoji" alt=":clap:"> Can I ask one more question, please? Well, I wrote a code which allowed the system to randomly choose different convolution layers with different kernel sizes and the kernel size is the only difference between each convolution layers. For example, 3 conv layers: 3X3, 1X1,5X5. But how can I get the information of the kernel size and save them in an array or a string after I create this model?</p>

ptrblck: <p>What would you want to do with this information?<br>
Is your model choosing one of these conv layers randomly for each batch or only once at initialization?</p>

cat-loves-donuts: <p>Well, I am trying to work on NAS(Neural Architecture Search). And my prototype will choose different types of convolution layer when I create a new model. I need to save that information and find the best performance one.</p>
<p>I wrote an independent file which defines different convolution layers in it. And this function will return a random layer which have a random kernel size. And this function will be used in the definition file of the model. I mean, there is a loop to add the layers into the model and a special forward function which allows skipping connection between layers in the definition file.</p>

ptrblck: <p>Ah OK, thanks for the information.<br>
You can get the kernel size using <code>conv.kernel_size</code> and store it in your config file.</p>

cat-loves-donuts: <p>Oh, thank you so much. I never thinking of one line code can finish this work. Anyway, thank you so much for answering me so many questions. You are so kind. Thank you so much.</p>

GLV: <p>Hello ptrblock<br>
I am trying to connect 3 different GNC together<br>
but the solution you propose here does not work:<br>
here is my code:<br>
a very simple GNC</p>
<p>class Net(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = GCNConv(300,256)<br>
self.conv2 = GCNConv(256, 1)</p>
<pre><code>def forward(self,x, edge_index):
    x = self.conv1(x, edge_index)
    x = x.relu()
    x = self.conv2(x, edge_index)
    return x
</code></pre>
<p>then</p>
<h1>
<a name="here-we-instantiate-all-of-the-variables-necessary-to-train-the-model-1" class="anchor" href="#here-we-instantiate-all-of-the-variables-necessary-to-train-the-model-1"></a>here we instantiate all of the variables necessary to train the model</h1>
<p>model1 = Net().cpu()<br>
model2 = Net().cpu()<br>
model3 = Net().cpu()</p>
<h1>
<a name="set-hyperparameters-2" class="anchor" href="#set-hyperparameters-2"></a>set hyperparameters</h1>
<p>optimizer = torch.optim.Adam(model1.parameters(), lr = 0.01)<br>
criterion1 = MSELoss()</p>
<h1>
<a name="loading-subsets-for-each-graph-layer-3" class="anchor" href="#loading-subsets-for-each-graph-layer-3"></a>loading subsets for each graph layer</h1>
<p>data_L3 = data[‘graph1’]<br>
data_L4 = data[‘graph2’]<br>
data_L5 = data[‘graph3’]</p>
<h1>
<a name="create-different-subsets-for-reach-graph-layer-4" class="anchor" href="#create-different-subsets-for-reach-graph-layer-4"></a>create different subsets for reach graph layer</h1>
<p>loader_L3 = RandomNodeSampler(data_L3,2)<br>
loader_L4 = RandomNodeSampler(data_L4,2)<br>
loader_L5 = RandomNodeSampler(data_L5,2)</p>
<p>then</p>
<p>def train(loader,model):<br>
for epoch in range(2):<br>
running_loss = 0.0<br>
model.train()<br>
for data in loader:<br>
data = data.cpu()<br>
# change to float to match with the weights of the class model<br>
data.x = data.x.float()<br>
logits = model1(data.x, data.edge_index)<br>
loss = criterion1(logits, data.y)<br>
optimizer.step()<br>
running_loss += loss.item()<br>
# we print out here if we want to see the loss<br>
print(epoch+1,running_loss/2000)<br>
running_loss = 0.0<br>
return model</p>
<p>Then train 3 models</p>
<h1>
<a name="train-each-model-on-correct-data-subset-5" class="anchor" href="#train-each-model-on-correct-data-subset-5"></a>Train each model on correct data subset</h1>
<p>train(loader_L3,model1)<br>
train(loader_L4,model2)<br>
train(loader_L5,model3)</p>
<p>How I can combine into one single model<br>
I try to use<br>
torch.stack([self.GNC1.weight, self.GNC2.weight, self.GNC2.weight], dim=0)<br>
with no success</p>
<p>Can you help, please</p>

ptrblck: <p>Could you post the error message you were seeing or what exactly is not working, please?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Aileen: <p>Could you please explain this a bit? I am relatively new to PyTorch and I did quite catch the part where we are looping this in</p>

ptrblck: <p>I’m unsure which part of my post is unclear as the first one applies the models in a sequential way while the second one iterates the smaller models inside a larger one. Could you post the code that is unclear?</p>

Aileen: <p>My bad, should have specified this earlier, I was curious about the implementation of the smaller model iterating inside the larger ones. The code is very clear but the part where I was unclear was on:</p>
<p><strong>models = nn.ModuleList()</strong><br>
for _ in range(10):<br>
models.append(MySmallModel())</p>
<p>model = MyHugeModel(models)<br>
x = torch.randn(1, 10)<br>
output = model(x)</p>
<p>I did not understand the ModuleList().<br>
Sorry may be the very basics here, just trying to understand</p>

ptrblck: <p>The <code>nn.ModuleList</code> is used to store the 10 <code>MySmallModel</code>s and is then passed to <code>MyHugeModel</code> where it will be registered as submodules.<br>
You could also create the 10 <code>MySmallModel</code>s inside <code>MyHugeModel.__init__</code> and would also use the <code>nn.ModuleList</code> to register them as submodules. If you would use a plain Python <code>list</code> these submodules will not be registered inside the parent class and thus won’t show up in its <code>state_dict</code> or <code>.parameters()</code> call.</p>