Ensemble two features

prog_asker: <p>I need to ensemble the features only that help in classification which I can extract it from two different models . So I need to remove the last layer first from each model then start to concatenate them … how can the ensemble method will be ?</p>

ptrblck: <p>You could replace the last linear layers with <code>nn.Identity</code> modules and create the ensemble as described in <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">this post</a>.</p>

prog_asker: <p>Thanks a lot . I will try it . But I shouldn’t write the line for new classifier</p>
<blockquote>
<p>self.classifier = nn.Linear(2048+512, nb_classes)</p>
</blockquote>
<p>and this too</p>
<blockquote>
<p>x = self.classifier(F.relu(x))</p>
</blockquote>
<p>Right ?</p>

ptrblck: <p>Yes, you could add a new classifier using the output features of both feature extractors.</p>

prog_asker: <p>Excuse me do you mean that I need to write those lines too . As I thought that I concatenate the features I got from both model only not to make a classifier from them …</p>

ptrblck: <p>You don’t need to use a classifier on top of it as it depends on your use case.<br>
If you want to return the features only, don’t use the additional classifier. If you want to train another stage on top of the pretrained feature extractors, add the classifier.</p>

prog_asker: <pre><code class="lang-auto">class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        # Remove last linear layer
        self.modelA.fc = nn.Identity()
        self.modelB.fc = nn.Identity()
        
        # Create new classifier
       # self.classifier = nn.Linear(2048+512, nb_classes)
        
    def forward(self, x):
        x1 = self.modelA(x.clone())  # clone to make sure x is not changed by inplace methods
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
        
     #   x = self.classifier(F.relu(x))
        return x

# Create models and load state_dicts    
modelA= mode1()
# Load state dicts
PATH1 = 'model1.pth'
modelA.load_state_dict(torch.load(PATH1))
modelB = torch.hub.load("model2")
model = MyEnsemble(modelA, modelB)
</code></pre>
<p>but got<br>
<code>TypeError: forward() takes 2 positional arguments but 3 were given</code></p>

ptrblck: <p>Your code works fine and I don’t know where the error is raised, as you are not calling the <code>forward</code> method in the posted code snippet:</p>
<pre><code class="lang-python">modelA = nn.Linear(1, 1)
modelB = nn.Linear(1, 1)
model = MyEnsemble(modelA, modelB)

x = torch.randn(1, 1)
out = model(x)
</code></pre>

prog_asker: <p>Thanks a lot for your time and helping … excuse me if i have two models one of them i should remove the last layer or classification layer and the other model need to implement it as those line</p>
<pre><code class="lang-auto">modelB =  ResNet50()
modelB.layers.pop()
modelB = Model(inputs=model.inputs, outputs=model.layers[-2].output)
</code></pre>
<p>then the ensemble should be like that</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x

class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB
        # Remove last linear layer
        self.modelA.fc = nn.Identity()
             
    def forward(self, x):
        x1 = self.modelA(x.clone())  # clone to make sure x is not changed by inplace methods
        x1 = x1.view(x1.size(0), -1)
        x2 = self.modelB(x)
        x2 = x2.view(x2.size(0), -1)
        x = torch.cat((x1, x2), dim=1)
       
        return x

# Create models and load state_dicts    
modelA= model1()
# Load state dicts
PATH1 = 'model1.pth'
modelA.load_state_dict(torch.load(PATH1))
modelB =  ResNet50()
modelB.layers.pop()
modelB = Model(inputs=model.inputs, outputs=model.layers[-2].output)

model = MyEnsemble(modelA, modelB)
</code></pre>

ptrblck: <p>You should be able to use different architectures as long as the <code>torch.cat</code> operation is able to concatenate the output tensors from both models. I don’t know how your new model would be interpreted as it’s getting features from one model and predictions from another, but technically it should work.<br>
Are you seeing any issues using your code?</p>

prog_asker: <p>i got this error <code>CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 5.80 GiB total capacity; 191.43 MiB already allocated; 16.56 MiB free; 202.00 MiB reserved in total by PyTorch)</code></p>

ptrblck: <p>This error points out that your GPU doesn’t have enough memory for the current use case and you would thus need to decrease the memory usage e.g. via lowering the batch size.</p>

ptrblck: <p>If a single sample already causes the OOM issue, it might be easier to call the models separately to reduce the memory usage and concatenate (and store) the features afterwards.<br>
If you don’t want to train the ensemble, but only want to get the features, you should also wrap the code into <code>with torch.no_grad()</code> to save additional memory.</p>

prog_asker: <p>Thanks a lot but how can I concatenate the files… I’m already getting the features for both in two separate files but don’t know how can I merge them together . how can I start, please?</p>

ptrblck: <p>After storing the features, you could <code>torch.load</code> them and use <code>torch.cat</code> in the same way as in your ensemble, which is currently running out of memory.</p>

prog_asker: <p>Thanks for your help. I saved the features in a pickle file using dump but got problem when I loaded it using torch.load</p>
<blockquote>
<p>torch.load pickle runtime error: invalid magic number; corrupt file ?</p>
</blockquote>
<p>Is there any other way to concatenate the pickles, please</p>
<p>Appreciate your time and help</p>

ptrblck: <p>The file might be corrupt, so try to save it again (and maybe load it directly afterwards to verify that you can indeed load the file properly).</p>