Pre-Trained Model Feature Fusion

Ghasem_Abdi: <p>Hi,<br>
I have two different image data-set but related to a same class. I am going to use pre-trained net like alexnet for both to detect features and then, concatenate those features into a classifier (and optimize this classifier not the whole models). I would appreciate if you can help me within this context. Here is the pseudo code:</p>
<ol>
<li>Alexnet(Image I) --&gt; Feature I</li>
<li>Alexnet(Image II) --&gt; Feature II</li>
<li>Concatenate (Feature I, Feature II) --&gt; Feature</li>
<li>Feature --&gt; Classifier</li>
<li>Optimize Step 4</li>
</ol>
<p>Thanks in advance,</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">This code snippet</a> could be a good starter for your use case.<br>
Let me know, if that would work for you.</p>

Ghasem_Abdi: <p>Many thanks for your prompt action.<br>
As I described, I should transfer one image to first model and second image to another model (while your code transfers an image to both models). The main problem here is that how data loader shuffles same on both dataset and how can do the feature fusion.<br>
Thanks in advance,</p>

ptrblck: <p>In that case you could pass both image tensors to your <code>forward</code> method as:</p>
<pre><code class="lang-python">    def forward(self, x1, x2):
        x1 = self.modelA(x1)  
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x2)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        
        x = self.classifier(F.relu(x))
        return x
</code></pre>
<p>The output features will be concatenated in the <code>torch.cat</code> call.<br>
Would that work for your “fusion” or are you thinking about another method?</p>
<p>Could you explain your concerns about the <code>DataLoader</code>?<br>
How are you defining both <code>Datasets</code> currently and how would you like to sample each image?</p>
<p>If you want to use the same <code>index</code> to get the samples from both <code>Datasets</code>, I think the easiest approach would be to wrap these <code>Datasets</code> in another one and just index them:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, ds1, ds2):
        self.ds1 = ds1
        self.ds2 = ds2
        
    def __getitem__(self, index):
        x1, y1 = self.ds1[index]
        x2, y2 = self,ds2[index]
        return x1, y1, x2 ,y2
    
    def __len__(self):
        return len(self.ds1) # assume both datasets have same length
</code></pre>

Ghasem_Abdi: <p>Could you please advice me??</p>

ptrblck: <p><code>self.features_i</code> and <code>self.features_ii</code> will point to the same module and will thus reuse the parameters.<br>
If you want to create two separate modules, you could use <code>deepcopy</code> as:</p>
<pre><code class="lang-python">self.features_ii = deepcopy(model.features)
</code></pre>

ptrblck: <p>Could you post the code here, please?</p>

ptrblck: <p>The code looks alright from what I could see.<br>
PS: it’s generally better to post code snippets directly by wrapping them in three backticks ```. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

STK: <p>Hi,<br>
I have three different image data-sets (say X, Y and Z) but related to a same class. I am going to use pre-trained net like resnet50 and resnet18 for both to detect features and then, concatenate those features into a classifier. Here I want two classifiers to predict output based on 1st and 2nd dataset, based on 1st and 3rd dataset. I have reused your code of ensemble.<br>
I would appreciate if you can help me within this context. Please suggest if there are any corrections. Thanks in advance,</p>
<pre><code class="lang-auto">class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB, nb_classes=2):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        
        # Remove last linear layer
        self.modelA.fc = nn.Identity()
        self.modelB.fc = nn.Identity()
        
        # Create new classifier
        self.classifier = nn.Linear( 2048+ 512, nb_classes)
        
    def forward(self, x1, x2):
        x1 = self.modelA(x1)  
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x2)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        
        x = self.classifier(F.relu(x))
        return x

class Conv(nn.Module):
    def __init__(self):
        super(Conv, self).__init__()
        # Train your separate models
        # ...
        # We use pretrained torchvision models here
        self.modelA = models.resnet50(pretrained=True)       #model to extract common feature
        self.modelB = models.resnet18(pretrained=True)
        self.modelA_1 = copy(self.modelA)
        self.modelC = deepcopy(self.modelB)

        # Freeze these models
        for param in self.modelA.parameters():
            param.requires_grad_(False)

        for param in self.modelB.parameters():
            param.requires_grad_(False)

        for param in self.modelC.parameters():
            param.requires_grad_(False)

        self.model1 = MyEnsemble(self.modelA, self.modelB)
        self.model2 = MyEnsemble(self.modelA_1, self.modelC)

    def forward(self, d1, d2, tgt):
        # Create ensemble model
        output1 = self.model1(d1, tgt)
        output2 = self.model2(d2, tgt)
        return output1, output2


x = torch.randn(1, 3, 224, 224)
y = torch.randn(1, 3, 224, 224)
z = torch.randn(1, 3, 224, 224)
model = Conv()
out1, out2 = model(x, y, z)
</code></pre>

Ajay_krishan_gairola: <p>hey, how to fuse two different CNN models in googlecolab using pytorch? Also want classification report results?</p>