How to train an ensemble of two CNNs and a classifier?

username: steeltrooper17
cooked: <p>I have two CNNs where each CNN runs on a different dataset. The output feature vectors of these CNNs (basically without the last fc layers) are concatenated and given to a classifier. How do I train such a network?</p>

username: smth
cooked: <blockquote>
<p>How do I train such a network?</p>
</blockquote>
<p>what are you exactly looking for? a pointer or a full blown example?<br>
Generally, you can write your autograd expressions in free form and call backward on the final loss of the classifier.</p>

username: steeltrooper17
cooked: <p>I didn’t realize that my Net class could take 2 inputs and have two parallel networks inside it in free form. Thank you for the help xD</p>

username: panovr
cooked: <p>I also want to ensemble two CNNs for jointly training. May you share some code examples?<br>
Thanks!</p>

username: steeltrooper17
cooked: <p>Here’s a basic example: (sorry for the late reply, was out of town)</p>
<pre><code>class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.pool = nn.MaxPool2d(2, 2)

    #1st net
    self.conv1 = nn.Conv2d(3, 64, 3, padding = 1)
    self.conv2 = nn.Conv2d(64, 128, 3, padding = 1)
    self.conv3 = nn.Conv2d(128, 256, 3, padding = 1)
    self.conv4 = nn.Conv2d(256, 256, 3, padding = 1)
    self.conv5 = nn.Conv2d(256, 512, 3, padding = 1)
    self.conv6 = nn.Conv2d(512, 512, 3, padding = 1)
    self.conv7 = nn.Conv2d(512, 512, 3, padding = 1)
    self.conv8 = nn.Conv2d(512, 512, 3, padding = 1)

    self.fc1 = nn.Linear(512*9, 4096)
    self.fc2 = nn.Linear(4096, 4096)

    #2nd net
    self.conv1o = nn.Conv2d(1, 3, 2)
    self.conv2o = nn.Conv2d(3, 3, 2)
    self.fc1o = nn.Linear(16 * 5 * 5, 120)
    self.fc2o = nn.Linear(120, 84)

  def forward(self, x, y):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = F.relu(self.conv3(x))
    x = self.pool(F.relu(self.conv4(x)))
    x = F.relu(self.conv5(x))
    x = self.pool(F.relu(self.conv6(x)))
    x = F.relu(self.conv7(x))
    x = self.pool(F.relu(self.conv8(x)))

    x = x.view(-1, 512*9) #can also do x.view(-1, 1)

    y = F.relu(self.conv1o(y))
    y = F.relu(self.conv2o(y))
    y = x.view(-1, 1)
    y = F.relu(self.fc1o(y))
    y = F.relu(self.fc2o(y))

    x = F.relu(self.fc1(x))
    x = F.dropout(x, training = self.training)
    x = F.relu(self.fc2(x))
    x = F.dropout(x, training = self.training)

    x = torch.cat((x, y))

    return F.log_softmax(x)</code></pre>

username: zhihe_lu
cooked: <p>Your advice is so instructive to me. However, I wonder about how to copy with the input training set because of the two inputs. Please reply me when you are not  busy. Thanks for your attention.</p>

username: steeltrooper17
cooked: <p>I’m sorry, I didn’t understand your question. Did you mean how to give two inputs into the model? I use two datasets here, one for each sub model.</p>

username: Arjan_Hada
cooked: <p>Is there a full blown example of how to ensemble two models for example- resnet152 and densenet161 for prediction on the same dataset? I would really appreciate it.</p>

username: smth
cooked: <p>I dont have a full-blown example, but really it’s as simple as something like:</p>
<pre><code class="lang-auto">for input, targets in test_loader:
    out_resnet = resnet(input)
    out_densenet = densenet(input)
    out = out_resnet + out_densenet
    loss = loss_fn(out, target)
    total_loss += loss.item()
</code></pre>

username: Arjan_Hada
cooked: <p>Thank you so much <a class="mention" href="/u/smth">@smth</a>. Appreciate your quick response. Is it possible to save these two models in one checkpoint for submission? I am participating in a challenge that requires me to load a saved checkpoint as a part of the submission.</p>

username: smth
cooked: <p>Sure, something like this:</p>
<pre><code class="lang-python">torch.save([densenet.state_dict(), resnet.state_dict()], 'checkpoint.pt')
densenet_state, resnet_state = torch.load('checkpoint.pt')
densenet = DenseNet(...).load_state_dict(densenet_state)
resnet = Resnet(...).load_state_dict(resnet_state)
</code></pre>

username: Arjan_Hada
cooked: <p>Thanks again <a class="mention" href="/u/smth">@smth</a>. Can we do something like in this post - <a href="https://discuss.pytorch.org/t/combining-trained-models-in-pytorch/28383/2?u=arjan_hada" class="inline-onebox">Combining Trained Models in PyTorch</a> but for let’s say resnet152 and densenet161? An example would help. Appreciate your humble support.</p>

username: Anamul_Hasan_Dhusor
cooked: <p>is this for testing ?? what about trainning?</p>