Fusion/ensembel of two MLP models

username: Omnia_Al-wazzan
cooked: <p>Hi</p>
<p>I’m trying to combine the features of two MLP models.</p>
<p>I have two tabular datasets of the same class that I want to fuse to improve accuracy. The straightforward approach is currently considering concatenation.</p>
<p>Here is a summary of my two models; both models are identical except for the input features.</p>
<p>MulticlassClassificationA(<br>
(layer_1): Linear(in_features=4, out_features=1500, bias=True)<br>
(layer_2): Linear(in_features=1500, out_features=1000, bias=True)<br>
(layer_3): Linear(in_features=1000, out_features=500, bias=True)<br>
(layer_out): Linear(in_features=500, out_features=3, bias=True)<br>
(relu): ReLU()<br>
(dropout): Dropout(p=0.3, inplace=False)<br>
(batchnorm1): BatchNorm1d(1500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(batchnorm2): BatchNorm1d(1000, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(batchnorm3): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
MulticlassClassificationB(<br>
(layer_1): Linear(in_features=78, out_features=1500, bias=True)<br>
(layer_2): Linear(in_features=1500, out_features=1000, bias=True)<br>
(layer_3): Linear(in_features=1000, out_features=500, bias=True)<br>
(layer_out): Linear(in_features=500, out_features=3, bias=True)<br>
(relu): ReLU()<br>
(dropout): Dropout(p=0.3, inplace=False)<br>
(batchnorm1): BatchNorm1d(1500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(batchnorm2): BatchNorm1d(1000, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(batchnorm3): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)</p>
<p>I tried the code posted by <a class="mention" href="/u/ptrblck">@ptrblck</a>, the ensembele model, but it did not work.</p>
<p>class MyEnsemble(nn.Module):<br>
def <strong>init</strong>(self, model_EHR, model_G, nb_classes=3):<br>
super(MyEnsemble, self).<strong>init</strong>()<br>
self.model_EHR = model_EHR<br>
self.model_G = model_G<br>
# Remove last linear layer<br>
self.model_EHR.layer_out = nn.Identity()<br>
self.model_G.layer_out = nn.Identity()</p>
<pre><code>    # Create new classifier
    self.classifier = nn.Linear(500+500, nb_classes)
    
def forward(self,x):
    x1 = self.model_EHR(x.clone())  # clone to make sure x is not changed by inplace methods
    x1 = x1.view(x1.size(0), -1)
    x2 = self.model_G(x2)
    x2 = x2.view(x2.size(0), -1)
    x = torch.cat((x1, x2), dim=1)
    
    x = self.classifier(F.relu(x))
    return x
</code></pre>
<p>for param in model_EHR.parameters():<br>
param.requires_grad_(False)</p>
<p>for param in model_G.parameters():<br>
param.requires_grad_(False)</p>
<h1>
<a name="create-ensemble-model-1" class="anchor" href="#create-ensemble-model-1"></a>Create ensemble model</h1>
<p>model = MyEnsemble(model_EHR, model_G)</p>
<p>another question: what input will I give the model when I do the ensemble model?</p>
<p>Is it a batch from dataset A’s loader and a batch from dataset B’s loader?</p>
<p>I’m not sure how I’ll approach the prediction after concatenation?</p>
<p>Any help is greatly appreciated!</p>

username: ptrblck
cooked: <p>In my <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">example</a> you can see that the same input is passed to both models so that the features created by these models are then concatenated before being passed to a final classifier.<br>
In your code snippet you are passing an unknown <code>x2</code> tensor to <code>self.model_G</code> so I guess this might be a typo?</p>

username: Omnia_Al-wazzan
cooked: <p>Thanks a lot for the quick response, you are right! it was a typo. Yet It didn’t work.</p>
<p>this is the error:</p>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
7             X_batch = X_batch.to(device)<br>
8             X_batch_G = X_batch_G.to(device)<br>
----&gt; 9             y_test_pred_F = model(X_batch,X_batch_G)<br>
10         <span class="hashtag">#probs</span> = torch.nn.functional.softmax(y_test_pred, dim=1)<br>
11         #y_prob=y_prob.np.append(probs.detach().cpu().numpy())</p>
<p>~/.local/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1109                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1110             return forward_call(*input, **kwargs)<br>
1111         # Do not call functions when jit is used<br>
1112         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>TypeError: forward() takes 2 positional arguments but 3 were given</p>
<p>Let me tell you about my dataset a bit . I have a CVS file for the EHR model which has 4 features (col) and it is  my x_train  (Features:age, gender, censor,survive)  the table is of 742 records (rows)</p>
<p>The other CSV file for training the model_G has 78 features which is my x_train_G, this table again is of 742 records</p>
<p>the label for both models are the same</p>
<p>As I mentioned previously the two models are identical but have different input shapes</p>
<p>this is the code snippet that brings the error</p>
<p>y_pred_list_Fusion = []</p>
<p>with torch.no_grad():<br>
model.eval() # Ensembel model<br>
for X_batch, _ in test_loader: # test loader of EHR data<br>
for X_batch_G, _ in test_loader_G: # test loader of Genes data<br>
X_batch = X_batch.to(device)<br>
X_batch_G = X_batch_G.to(device)<br>
y_test_pred_F = model(X_batch,X_batch_G)<br>
_, y_pred_tags = torch.max(y_test_pred_F, dim = 1)<br>
y_pred_list_Fusion.append(y_pred_tags.cpu().numpy())</p>
<p>y_pred_list_Fusion = [a.squeeze().tolist() for a in y_pred_list_Fusion]</p>
<p>I would be happy to have your kind feedback!</p>

username: ptrblck
cooked: <p>You re passing two inputs to the model:</p>
<pre><code class="lang-python">y_test_pred_F = model(X_batch,X_batch_G)
</code></pre>
<p>while a single one is expected:</p>
<pre><code class="lang-python">def forward(self,x):
</code></pre>
<p>so either pass a single input or change the <code>forward</code> method.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>

username: Omnia_Al-wazzan
cooked: <p>Again thanks for your prompt response!</p>
<p>Here is my confusion. When I send the data for the forward training (using ensemble model) at this point</p>
<pre><code class="lang-auto">y_test_pred_F = model(X_batch,X_batch_G)
</code></pre>
<p>am I doing something wrong?</p>
<p>I want to send the data from both (CSV) files that’s why I sent two patches, please correct if I am wrong.</p>
<p>When did you say pass one input, how to do that in my case? I do need 2 inputs from two datasets?</p>
<p>How to change the forward method, please?</p>
<p>I am sorry I am very new to PyTorch and still learning <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Since you want to pass both inputs to the model, I guess your <code>forward</code> method should be:</p>
<pre><code class="lang-python">def forward(self, x1, x2):
    x1 = self.model_EHR(x1)
    x1 = x1.view(x1.size(0), -1)

    x2 = self.model_G(x2)
    x2 = x2.view(x2.size(0), -1)
    x = torch.cat((x1, x2), dim=1)
    
    x = self.classifier(F.relu(x))
    return x
</code></pre>
<p>as this will now pass the two different inputs to the two internal models.</p>

username: Omnia_Al-wazzan
cooked: <p>Many thanks for your prompt response!</p>
<p>It worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>