How to ensemble two model in pytorch?

sampa: <p>I want to ensemble Mode1A and Model1B, But there is a run time error<br>
<strong>Expected 4-dimensional input for 4-dimensional weight 8 3, but got 2-dimensional input of size [1, 25088] instead</strong><br>
Please help me</p>
<pre><code class="lang-auto">class MyModelA(nn.Module):
    def __init__(self):
        super(MyModelA, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=8, kernel_size=3,stride=1, padding=1),
            nn.BatchNorm2d(8),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(in_channels=8, out_channels=16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2)) 
        self.layer3 = nn.Sequential(
            nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2))     
        self.fc = nn.Linear(25088, 2)
        
    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        #out = self.layer4(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out
class MyModelB(nn.Module):
..
class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        #self.classifier = nn.Linear(4, 2)
        
    def forward(self, x1, x2):
        head1a, head1b = self.modelA(x1)
        head2 = self.modelB(head1a)
        x = torch.cat((head1b, head2), dim=1)
        return x

# Create models and load state_dicts    
modelA = MyModelA()
modelB = MyModelB()
# Load state dicts
modelA.load_state_dict(torch.load('checkpoint1.pt'))
modelB.load_state_dict(torch.load('checkpoint2.pt'))

model = MyEnsemble(modelA, modelB)
x1, x2 = torch.randn(1,25088), torch.randn(1, 25088)
output = model(x1, x2)

</code></pre>

ptrblck: <p><code>MyModelA</code> uses a <code>nn.Conv2d</code> layer as the first layer, which expects a 4-dimensional input with the shape <code>[batch_size, channels, height, width]</code>.<br>
In your code snippet, you are defining <code>x1</code> as a 2-dimensional tensor, which will throw this error.<br>
Based on the shape, it seems as if you would only want to use <code>self.fc</code> on this input?</p>

sampa: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I saw your Ensemble code and try to ensemble my model with this code. I want to combine the classifier only. Can I use this for resent model?  If yes, how to use?</p>

ptrblck: <p>Would you like to combine the penultimate activations and train a new classifier on top or did I misunderstand your use case?</p>

sampa: <p>I have 2 similar dataset A and B . So I want to train the model in A and B and save the model in checkpoint1 and checkpoint2. Now I want to ensemble this two model. My question is how to combine this two model?</p>

sampa: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="65348">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>e penultimate activations and train a new classifier</p>
</blockquote>
</aside>
<p>Yes…you are right.I want to combine the penultimate activations and train a new classifier.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">My example</a> should do exactly this. Would that work for you?</p>

sampa: <p>Thank you… I will try this &amp; Let you know</p>

Ajay_krishan_gairola: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Please check: is this correct?<br>
‘’’<br>
class MyEnsemble(nn.Module):<br>
def <strong>init</strong>(self, modelA, modelB, nb_classes=2):<br>
super(MyEnsemble, self).<strong>init</strong>()<br>
self.modelA = modelA<br>
self.modelB = modelB</p>
<pre><code>    self.modelA.features[10] = nn.Identity() 
    model_ft.features=model_ft.features[:--11]
    
     self.modelB.features[10] = nn.Identity() 
    model_ft1.features=model_ft1.features[:--11]

    
    self.features = nn.Sequential(nn.Conv2d(256+256, 256+256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)), nn.ReLU(inplace=True), nn.MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False))
   
    self.classifier = nn.Sequential(   #this add multple fc layers
        nn.Dropout(p=0.5, inplace=False),
        nn.Linear(in_features=9216, out_features=4096, bias=True),
        nn.ReLU(inplace=True),
        nn.Dropout(p=0.5, inplace=False),
        nn.Linear(in_features=4096, out_features=4096, bias=True),
        nn.ReLU(inplace=True),
        nn.Linear(in_features=4096, out_features=1000, bias=True)
    )

    
def forward(self, x):
    x1 = self.modelA(x.clone())          
  x1 = x1.view(x1.size(0), -1)
    x2 = self.modelB(x)
    x2 = x2.view(x2.size(0), -1)
    x = torch.cat((x1, x2), dim=1)
    
    x = self.features(F.relu(x))
    return x
</code></pre>
<p>‘’’</p>

ptrblck: <p>It’s unclear how <code>model_ft</code> is defines, but <code>features[:--11]</code> looks wrong as you could also just remove the double minus signs or fix the typo.</p>
<p>Besides that <code>self.classifier</code> is never used.</p>

Ajay_krishan_gairola: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply.</p>
<p>This is how model look:</p>
<p>‘’’<br>
MyEnsemble(<br>
(modelA): AlexNet(<br>
(features): Sequential(<br>
(0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))<br>
(1): ReLU(inplace=True)<br>
(2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))<br>
(4): ReLU(inplace=True)<br>
(5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(7): ReLU(inplace=True)<br>
(8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(9): ReLU(inplace=True)<br>
(10): Identity()<br>
)<br>
)<br>
(modelB): AlexNet(<br>
(features): Sequential(<br>
(0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))<br>
(1): ReLU(inplace=True)<br>
(2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))<br>
(4): ReLU(inplace=True)<br>
(5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(7): ReLU(inplace=True)<br>
(8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(9): ReLU(inplace=True)<br>
(10): Identity()<br>
)<br>
)<br>
(features): Sequential(<br>
(0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(1): ReLU(inplace=True)<br>
(2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
)<br>
(classifier): Sequential(<br>
(0): Dropout(p=0.5, inplace=False)<br>
(1): Linear(in_features=9216, out_features=4096, bias=True)<br>
(2): ReLU(inplace=True)<br>
(3): Dropout(p=0.5, inplace=False)<br>
(4): Linear(in_features=4096, out_features=4096, bias=True)<br>
(5): ReLU(inplace=True)<br>
(6): Linear(in_features=4096, out_features=1000, bias=True)<br>
)<br>
)</p>
<p>‘’‘’<br>
Please help to concatenate the two alexnet model in last convolution layer. After this concatenate i added this convolution layer and all linear layers.<br>
rest code is:</p>
<p>‘’’<br>
model_ft = models.alexnet(pretrained=True)</p>
<p>model_ft1 = models.alexnet(pretrained=True)</p>
<p>del model_ft.classifier <span class="hashtag">#delete</span> all fully connected layer</p>
<p>del model_ft.avgpool <span class="hashtag">#delete</span> AdaptiveAvgPool2d</p>
<p>del model_ft1.classifier <span class="hashtag">#delete</span> all fully connected layer</p>
<p>del model_ft1.avgpool <span class="hashtag">#delete</span> AdaptiveAvgPool2d</p>
<p>num_ftrs = model_ft.features[10].get_parameter</p>
<p>num_ftrs1 = model_ft1.features[10].get_parameter</p>
<p>model_ft.features[10]= nn.Conv2d(256, 256, 2)</p>
<p>model_ft1.features[10]= nn.Conv2d(256, 256, 2)</p>
<p>model = MyEnsemble(model_ft,model_ft1).to(device)</p>
<p>criterion = nn.CrossEntropyLoss()</p>
<p>model = MyEnsemble(model_ft, model_ft1)</p>
<p>optimizer_ft = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)</p>
<p>exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=20, gamma=0.1)</p>
<p>print(model)</p>
<p>‘’’</p>

Ajay_krishan_gairola: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
model_ft.features=model_ft.features[:–11]<br>
This line removes 11 and 12th layer of 1st alexnet model features, i.e. ReLu and MaxPool2d layer.<br>
I used self.classifier to add multple fc layers after concatenation.</p>

Ajay_krishan_gairola: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
please tell me how to resolve this error?</p>
<p>usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)<br>
457                             weight, bias, self.stride,<br>
458                             _pair(0), self.dilation, self.groups)<br>
 → 459         return F.conv2d(input, weight, bias, self.stride,<br>
460                         self.padding, self.dilation, self.groups)<br>
461</p>
<p>RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [16, 86528]</p>

ptrblck: <p>You are passing a 2D tensor to <code>nn.Conv2d</code> while either a batched 4D tensor in the shape <code>[batch_size, channels, height, width]</code> or an unbatched 3D tensor in the shape <code>[channels, height, width]</code> is expected.</p>

Ajay_krishan_gairola: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> .<br>
This is the code I am passing:<br>
‘’’<br>
self.features = nn.Conv2d(in_channels=256+256, out_channels=256, kernel_size=3)<br>
‘’</p>

Ajay_krishan_gairola: <p>and <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>how to initialize it:<br>
‘’’<br>
def <strong>init</strong>(self, modelA, modelB, out_channels=256, kernel_size=3):<br>
‘’’<br>
correct??</p>