How to ensemble different CNN models when use the same dataset

username: 111186
cooked: <p>I am doing classification challenge,  i want to ensemble different CNN models to improve the accuracy of the classification. i have see the similar question <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024">https://discuss.pytorch.org/t/custom-ensemble-approach/52024</a> but i stil feel confuse.</p>
<p>this is one of the answer：</p>
<pre><code class="lang-auto">class MyModelA(nn.Module):
    def __init__(self):
        super(MyModelA, self).__init__()
        self.fc1 = nn.Linear(10, 2)
        
    def forward(self, x):
        x = self.fc1(x)
        return x
    

class MyModelB(nn.Module):
    def __init__(self):
        super(MyModelB, self).__init__()
        self.fc1 = nn.Linear(20, 2)
        
    def forward(self, x):
        x = self.fc1(x)
        return x


class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        self.classifier = nn.Linear(4, 2)
        
    def forward(self, x1, x2):
        x1 = self.modelA(x1)
        x2 = self.modelB(x2)
        x = torch.cat((x1, x2), dim=1)
        x = self.classifier(F.relu(x))
        return x

# Create models and load state_dicts    
modelA = MyModelA()
modelB = MyModelB()
# Load state dicts
modelA.load_state_dict(torch.load(PATH))
modelB.load_state_dict(torch.load(PATH))

model = MyEnsemble(modelA, modelB)
x1, x2 = torch.randn(1, 10), torch.randn(1, 20)
output = model(x1, x2)
</code></pre>
<p>another answer：<a href="https://discuss.pytorch.org/t/how-to-train-an-ensemble-of-two-cnns-and-a-classifier/3026/8">https://discuss.pytorch.org/t/how-to-train-an-ensemble-of-two-cnns-and-a-classifier/3026/8</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/0218a0647e114648d4eda8061247aa643c1efbb4.png" data-download-href="https://discuss.pytorch.org/uploads/default/0218a0647e114648d4eda8061247aa643c1efbb4" title="QQ截图20200801123618"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/0218a0647e114648d4eda8061247aa643c1efbb4_2_690x219.png" alt="QQ截图20200801123618" data-base62-sha1="ixIsjZ7foY4hdLcbUusifScOKE" width="690" height="219" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/0218a0647e114648d4eda8061247aa643c1efbb4_2_690x219.png, https://discuss.pytorch.org/uploads/default/original/3X/0/2/0218a0647e114648d4eda8061247aa643c1efbb4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/2/0218a0647e114648d4eda8061247aa643c1efbb4.png 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ截图20200801123618</span><span class="informations">966×307 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>one of is to use the torch.cat() . but another to use the “+”, which answer is more suitable for my classification.</p>
<p>this is part of my code.</p>
<pre><code class="lang-auto">if opt.model == 'VGG19':
    net = VGG('VGG19')
elif opt.model  == 'Resnet18':
    net = ResNet18()

if opt.resume:
    # Load checkpoint.
    print('==&gt; Resuming from checkpoint..')
    assert os.path.isdir(path), 'Error: no checkpoint directory found!'
    checkpoint = torch.load(os.path.join(path,'PrivateTest_model.t7'))

    net.load_state_dict(checkpoint['net'])
    best_PublicTest_acc = checkpoint['best_PublicTest_acc']
    best_PrivateTest_acc = checkpoint['best_PrivateTest_acc']
    best_PrivateTest_acc_epoch = checkpoint['best_PublicTest_acc_epoch']
    best_PrivateTest_acc_epoch = checkpoint['best_PrivateTest_acc_epoch']
    start_epoch = checkpoint['best_PrivateTest_acc_epoch'] + 1
else:
    print('==&gt; Building model..')

if use_cuda:
    net.cuda()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=opt.lr, momentum=0.9, weight_decay=5e-4)

# Training
def train(epoch):
    print('\nEpoch: %d' % epoch)
    global Train_acc
    net.train()
    train_loss = 0
    correct = 0
    total = 0

    if epoch &gt; learning_rate_decay_start and learning_rate_decay_start &gt;= 0:
        frac = (epoch - learning_rate_decay_start) // learning_rate_decay_every
        decay_factor = learning_rate_decay_rate ** frac
        current_lr = opt.lr * decay_factor
        utils.set_lr(optimizer, current_lr)  # set the decayed rate
    else:
        current_lr = opt.lr
    print('learning_rate: %s' % str(current_lr))

    for batch_idx, (inputs, targets) in enumerate(trainloader):
        if use_cuda:
            inputs, targets = inputs.cuda(), targets.cuda()
        optimizer.zero_grad()
        inputs, targets = Variable(inputs), Variable(targets)
        outputs = net(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        utils.clip_gradient(optimizer, 0.1)
        optimizer.step()
        train_loss += loss.data[0]
        _, predicted = torch.max(outputs.data, 1)
        total += targets.size(0)
        correct += predicted.eq(targets.data).cpu().sum()

        utils.progress_bar(batch_idx, len(trainloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
            % (train_loss/(batch_idx+1), 100.*correct/total, correct, total))

    Train_acc = 100.*correct/total

</code></pre>
<p>Please help me， thanks so much</p>

username: Kushaj
cooked: <p>If you concatenate the outputs, then you want to train a classifier on those which then makes the prediction.</p>
<p>If you add the outputs, you can directly make the prediction.</p>
<p>These are two different techniques of ensembling. I have only tried the second one. The logic behind adding the outputs is that the errors will cancel out.</p>

username: 111186
cooked: <p>thank you so much. but if in the second method.</p>
<pre><code class="lang-auto">out_resnet = resnet(input)
out_densenet = densenet(input)
out = out_resnet + out_densenet
</code></pre>
<p>if out_resnet = 0.7,  out_densenet  = 0.8<br>
and out_resnet + out_densenet &gt; 1<br>
so do we need to change the code like that?:</p>
<pre><code class="lang-auto">out = (out_resnet + out_densenet) / 2
</code></pre>
<p>Is there a full blown example of how to ensemble two models for example- resnet18 and densenet19 for prediction on the same dataset? I would really appreciate it.</p>

username: Kushaj
cooked: <h3>Unweighted average</h3>
<p>This can be done as follows</p>
<pre><code class="lang-python"># here out1, out2 are after softmax
out1 = model1(input)
out2 = model2(input)

out = (out1+out2)/2
</code></pre>
<p>The idea behind this unweighted average is the models typically have high variance and low bias, so we can combine these models by simply adding their outputs to reduce the variance.</p>
<p>You should average the output of softmax layer rather than raw scores because they may be on different scales.</p>
<p>Unweighted average is a good idea when both the models are similar i.e. it is not the case that model1 is a lot better than model2.</p>
<h3>Weighted average</h3>
<pre><code class="lang-auto">out1 = model1(input)
out2 = model2(input)

out = out1*weight1 + out2*weight2
# weight1+weight2 = 1
</code></pre>
<p>Here you can train a linear classifier to learn the optimal values of weight1 and weight2 after your original models (model1, model2) have converged. Be careful not to overfit.</p>

username: 111186
cooked: <p>thanks so much， I have implement the first method， but I don’t know how to implement the second method ，can you give  a full blown example， that will be very grateful. or you can teach me how to change my code, to achieve the second method.<br>
this is  my code:</p>
<pre><code class="lang-auto"># Model
if opt.model == 'VGG19':
    net = VGG('VGG19')
elif opt.model  == 'Resnet18':
    net = ResNet18()

if opt.resume:
    # Load checkpoint.
    print('==&gt; Resuming from checkpoint..')
    assert os.path.isdir(path), 'Error: no checkpoint directory found!'
    checkpoint = torch.load(os.path.join(path,'PrivateTest_model.t7'))

    net.load_state_dict(checkpoint['net'])
    best_PublicTest_acc = checkpoint['best_PublicTest_acc']
    best_PrivateTest_acc = checkpoint['best_PrivateTest_acc']
    best_PrivateTest_acc_epoch = checkpoint['best_PublicTest_acc_epoch']
    best_PrivateTest_acc_epoch = checkpoint['best_PrivateTest_acc_epoch']
    start_epoch = checkpoint['best_PrivateTest_acc_epoch'] + 1
else:
    print('==&gt; Building model..')

if use_cuda:
    net.cuda()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=opt.lr, momentum=0.9, weight_decay=5e-4)

# Training
def train(epoch):
    print('\nEpoch: %d' % epoch)
    global Train_acc
    net.train()
    train_loss = 0
    correct = 0
    total = 0

    if epoch &gt; learning_rate_decay_start and learning_rate_decay_start &gt;= 0:
        frac = (epoch - learning_rate_decay_start) // learning_rate_decay_every
        decay_factor = learning_rate_decay_rate ** frac
        current_lr = opt.lr * decay_factor
        utils.set_lr(optimizer, current_lr)  # set the decayed rate
    else:
        current_lr = opt.lr
    print('learning_rate: %s' % str(current_lr))

    for batch_idx, (inputs, targets) in enumerate(trainloader):
        if use_cuda:
            inputs, targets = inputs.cuda(), targets.cuda()
        optimizer.zero_grad()
        inputs, targets = Variable(inputs), Variable(targets)
        outputs = net(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        utils.clip_gradient(optimizer, 0.1)
        optimizer.step()
        train_loss += loss.data[0]
        _, predicted = torch.max(outputs.data, 1)
        total += targets.size(0)
        correct += predicted.eq(targets.data).cpu().sum()

        utils.progress_bar(batch_idx, len(trainloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
            % (train_loss/(batch_idx+1), 100.*correct/total, correct, total))

    Train_acc = 100.*correct/total

def PublicTest(epoch):
    global PublicTest_acc
    global best_PublicTest_acc
    global best_PublicTest_acc_epoch
    net.eval()
    PublicTest_loss = 0
    correct = 0
    total = 0
    for batch_idx, (inputs, targets) in enumerate(PublicTestloader):
        bs, ncrops, c, h, w = np.shape(inputs)
        inputs = inputs.view(-1, c, h, w)
        if use_cuda:
            inputs, targets = inputs.cuda(), targets.cuda()
        inputs, targets = Variable(inputs, volatile=True), Variable(targets)
        outputs = net(inputs)
        outputs_avg = outputs.view(bs, ncrops, -1).mean(1)  # avg over crops
        loss = criterion(outputs_avg, targets)
        PublicTest_loss += loss.data[0]
        _, predicted = torch.max(outputs_avg.data, 1)
        total += targets.size(0)
        correct += predicted.eq(targets.data).cpu().sum()

        utils.progress_bar(batch_idx, len(PublicTestloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
                           % (PublicTest_loss / (batch_idx + 1), 100. * correct / total, correct, total))

    # Save checkpoint.
    PublicTest_acc = 100.*correct/total
    if PublicTest_acc &gt; best_PublicTest_acc:
        print('Saving..')
        print("best_PublicTest_acc: %0.3f" % PublicTest_acc)
        state = {
            'net': net.state_dict() if use_cuda else net,
            'acc': PublicTest_acc,
            'epoch': epoch,
        }
        if not os.path.isdir(path):
            os.mkdir(path)
        torch.save(state, os.path.join(path,'PublicTest_model.t7'))
        best_PublicTest_acc = PublicTest_acc
        best_PublicTest_acc_epoch = epoch

def PrivateTest(epoch):
    global PrivateTest_acc
    global best_PrivateTest_acc
    global best_PrivateTest_acc_epoch
    net.eval()
    PrivateTest_loss = 0
    correct = 0
    total = 0
    for batch_idx, (inputs, targets) in enumerate(PrivateTestloader):
        bs, ncrops, c, h, w = np.shape(inputs)
        inputs = inputs.view(-1, c, h, w)
        if use_cuda:
            inputs, targets = inputs.cuda(), targets.cuda()
        inputs, targets = Variable(inputs, volatile=True), Variable(targets)
        outputs = net(inputs)
        outputs_avg = outputs.view(bs, ncrops, -1).mean(1)  # avg over crops
        loss = criterion(outputs_avg, targets)
        PrivateTest_loss += loss.data[0]
        _, predicted = torch.max(outputs_avg.data, 1)
        total += targets.size(0)
        correct += predicted.eq(targets.data).cpu().sum()

        utils.progress_bar(batch_idx, len(PublicTestloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
            % (PrivateTest_loss / (batch_idx + 1), 100. * correct / total, correct, total))
    # Save checkpoint.
    PrivateTest_acc = 100.*correct/total

    if PrivateTest_acc &gt; best_PrivateTest_acc:
        print('Saving..')
        print("best_PrivateTest_acc: %0.3f" % PrivateTest_acc)
        state = {
            'net': net.state_dict() if use_cuda else net,
	        'best_PublicTest_acc': best_PublicTest_acc,
            'best_PrivateTest_acc': PrivateTest_acc,
    	    'best_PublicTest_acc_epoch': best_PublicTest_acc_epoch,
            'best_PrivateTest_acc_epoch': epoch,
        }
        if not os.path.isdir(path):
            os.mkdir(path)
        torch.save(state, os.path.join(path,'PrivateTest_model.t7'))
        best_PrivateTest_acc = PrivateTest_acc
        best_PrivateTest_acc_epoch = epoch
</code></pre>

username: Kushaj
cooked: <p>I have not done much ensembling so I don’t know all the nuances of the best implementation. One simple way of how I would go about implementing this, is to first store the output of the two models (after softmax, so I can iterate faster). Then do a random search from 0 to 1, until I find a good weight value.</p>
<p>How I would go about implementing it. You can use a smaller dataset to get the value of weight, as you goal is to generalize, not to memorize the training set.</p>
<pre><code class="lang-python">model1_out = # torch.tensor of shape [num_images, num_classes]
model2_out = # torch.tensor of shape [num_images, num_classes]

low = 0.0
high = 1.0
check_vals = low + torch.rand(10)*(high-low)

for val in check_vals:
    out = model1_out*val + model2_out*(1-val)
    # compute your metric
    
# change the values of `low`, `high`, depending upon the best value
# that you got for your metric, until you are satisfied.
</code></pre>

username: 111186
cooked: <p>thanks again，recently I am learning the ensemble. ensemble including Bagging，Boosting and Stacking<br>
could you tell me the method 1 and method 2 belong which kind of method？</p>

username: Kushaj
cooked: <p>I used this article to learn about ensembling. <a href="https://mlwave.com/kaggle-ensembling-guide/" rel="nofollow noopener">This</a> can answer a lot of your questions.</p>

username: 111186
cooked: <p>thank you so much ，best wish for you</p>