Merging 3 models

the_coder: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41dd1d94cbf25409616bc6c533fb702f5bf428de.png" data-download-href="https://discuss.pytorch.org/uploads/default/41dd1d94cbf25409616bc6c533fb702f5bf428de" title="Bildschirmfoto 2020-01-09 um 23.20.03"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41dd1d94cbf25409616bc6c533fb702f5bf428de_2_690x379.png" alt="Bildschirmfoto 2020-01-09 um 23.20.03" data-base62-sha1="9oELXc5WZsYuFSIHvWGb7jjY8mW" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41dd1d94cbf25409616bc6c533fb702f5bf428de_2_690x379.png, https://discuss.pytorch.org/uploads/default/original/3X/4/1/41dd1d94cbf25409616bc6c533fb702f5bf428de.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/1/41dd1d94cbf25409616bc6c533fb702f5bf428de.png 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Bildschirmfoto 2020-01-09 um 23.20.03</span><span class="informations">1021×562 231 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am trying to implemnt this model. I have the following code:</p>
<pre><code class="lang-auto">class MyModelA(nn.Module):
    def __init__(self):
        super(MyModelA, self).__init__()
        self.conv1 = nn.Sequential(
                    nn.Conv1d(in_channels=256, out_channels=252, kernel_size=5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                    nn.Dropout(p=0.25))  
        self.conv2 = nn.Conv1d(in_channels=126, out_channels=122, kernel_size= 5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                     nn.Dropout(p=0.25)
        
    def forward(self, x):
        x = self.fc1(x)
        return x
    

    class MyModelB(nn.Module):
    def __init__(self):
        super(MyModelB, self).__init__()
        self.conv1 = nn.Sequential(
                    nn.Conv1d(in_channels=256, out_channels=252, kernel_size=5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                    nn.Dropout(p=0.25)) 
        self.conv2 = nn.Conv1d(in_channels=126, out_channels=122, kernel_size= 5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                     nn.Dropout(p=0.25)
        
    def forward(self, x):
        x = self.fc1(x)
        return x
    
     class MyModelC(nn.Module):
        def __init__(self):
        super(MyModelC, self).__init__()
        self.conv1 = nn.Sequential(
                    nn.Conv1d(in_channels=256, out_channels=252, kernel_size=5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                    nn.Dropout(p=0.25))  
        self.conv2 = nn.Conv1d(in_channels=126, out_channels=122, kernel_size= 5),
                    nn.softmax(),
                    nn.Avgpool1d(kernel_size= 2, stride=2),
                     nn.Dropout(p=0.25)

    class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
</code></pre>
<p>Now i have to merge this three classes (MyModelA, MyModelB and MyModelC) into one. Do i use<br>
<code>tensor.cat(MyModelA, MyModelB and MyModelC)</code> or is there something else that i should do to merge this 3 models ?</p>
<p>Please note the code is not complete, i am editing still. Thanks to <a class="mention" href="/u/ptrblck">@ptrblck</a> for the model converging cocept code.</p>

ptrblck: <p>You would merge the model output activations inside <code>MyEnsemble</code>.<br>
E.g. this code snippet removes the last linear layer of both passed models, combines their activation output, and passes them to a new linear layer, which now acts as the classification output:</p>
<pre><code class="lang-python">class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB, nb_classes=10):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        # Remove last linear layer, if necessary
        self.modelA.fc = nn.Identity()
        self.modelB.fc = nn.Identity()
        
        # Create new classifier
        self.classifier = nn.Linear(2048+512, nb_classes)
        
    def forward(self, x):
        x1 = self.modelA(x.clone())  # clone to make sure x is not changed by inplace methods
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        
        x = self.classifier(F.relu(x))
        return x
</code></pre>
<p>Your code contains some errors, e.g. undefined <code>self.fc</code> etc., but since it’s work in progress, I assume you are still fixing these. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

the_coder: <p>Thanks for your reply, <a class="mention" href="/u/ptrblck">@ptrblck</a>. As you can see i have three models to concatenate to the fc layer. So at forward propagation i add another x3,  is that correct ? Can <code>torch.cat</code> concatenate more than 2 models? Also in</p>
<pre><code class="lang-auto">self.classifier = nn.Linear(2048+512, nb_classes)
</code></pre>
<p>How you determined the 2048+512  ? If i add my third model what will be the number for this classifier?<br>
Also I have defined the activaiton function inside my model, is it necessary to add the F.relu() again at the end?</p>

ptrblck: <aside class="quote no-group" data-username="the_coder" data-post="3" data-topic="66230">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<p>Can <code>torch.cat</code> concatenate more than 2 models?</p>
</blockquote>
</aside>
<p><code>torch.cat</code> concatenates tensors. In my example the output activations.<br>
You can pass a list of tensors (also more than two) to concatenate them in a specified dimension.</p>
<p>The number of input features is determined by the feature size of each model output.<br>
E.g. if you have 3 models and each returns an activation of <code>[batch_size, 100]</code>, you would use <code>in_features=300</code> in the next layer after concatenating these tensors.</p>