Pytorch MultiModel

Aishuvenkat09: <p>I’m trying to implement multimodel for an Image regression task using MONAI, a PyTorch based framework. Adapted from this approach<br>
<a href="https://github.com/nvahmadi/MONAI_multimodal/blob/main/demo_monai_multimodal_crisismmd.ipynb" rel="noopener nofollow ugc">MONAI_multimodal/demo_monai_multimodal_crisismmd.ipynb at main · nvahmadi/MONAI_multimodal (github.com)</a></p>
<p>However, I’m getting this error.</p>
<p><code>RuntimeError: Function AddmmBackward returned an invalid gradient at index 1 - got [32, 272] but expected shape compatible with [32, 1040]</code></p>
<p>Here’s the approach I used.</p>
<pre><code class="lang-auto">
class MultimodalNet(nn.Module):
    def __init__(self, n_classes, hidden_size_fc_image=256, hidden_size=256):
        super().__init__()
        # same vision model as above, again implicitly containing an ebmedding layer fc1
        
        
        self.cnn = TorchVisionFCModel("resnet50", num_classes=hidden_size_fc_image, use_conv=True, pretrained=True)
        self.cnn.features[0]=nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)

        
        for param in self.cnn.parameters():
            param.requires_grad = False 
        for param in self.cnn.fc.parameters():
            param.requires_grad = True
        nr_genders = 2
        
        # Fully Connected layer for gender
        self.gen_fc_1 = nn.Linear(1,16)
        
        
        # Feature Concatenation Layer
        self.fc1 = nn.Linear(16+hidden_size_fc_image,hidden_size)
        
        
        # Final Fully Connected Layer
        self.fc2 = nn.Linear(hidden_size,n_classes)
        
    def forward(self, images, gender):
        
        x = F.relu(self.cnn(images))
        x = x.view(x.size(0), -1)
        print(x.shape)   #torch.Size([32, 1024])
        
        # =============================================================================
        #       Gender Fully Connected Layer
        # =============================================================================
        y = F.relu(self.gen_fc_1(gender))
        y = y.view(y.size(0), -1)
        print(y.shape)   #torch.Size([32, 16])

        
        # =============================================================================
        #       Feature Concatenation Layer
        # =============================================================================
    
        z = torch.cat((x,y),dim = 1)
        x = F.relu(self.fc1(z))
        x = self.fc2(x)
        return x
</code></pre>
<p>I’m using L1Loss(). Not sure where I’m doing wrong.</p>

ptrblck: <p>Could you post the missing definitions (e.g. <code>TorchVisionFCModel</code>) and make this code executable so that we could try to reproduce the issue, please?</p>

Aishuvenkat09: <p>TorchVisionFCModel is used to Customize the fully connected layer of (pretrained) TorchVision model or replace it by convolutional layer provided by MONAI. Here’s the <a href="https://docs.monai.io/en/stable/networks.html#monai.networks.nets.TorchVisionFCModel" rel="noopener nofollow ugc">documentation</a>.</p>
<p><code>from monai.networks.nets import resnet50, TorchVisionFCModel</code></p>

ptrblck: <p>Thanks for providing the information.<br>
I’m unable to reproduce the issue using:</p>
<pre><code class="lang-python">model = MultimodalNet(10)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.L1Loss()

data = torch.randn(2, 1, 224, 224)
gender = torch.randn(2, 1)
target = torch.randn(2, 10)

for _ in range(10):
    optimizer.zero_grad()
    output = model(data, gender)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print("loss {:.3f}".format(loss.item()))
</code></pre>

J_Johnson: <p>Simple mistake on your definition for <code>fc1</code>.<br>
Try with this:</p>
<pre><code class="lang-auto">self.fc1 = nn.Linear(16+1024,hidden_size)
</code></pre>
<p>Alternatively(NOT both), you could add a <code>maxpool2d</code> layer here:</p>
<pre><code class="lang-auto">def forward(self, images, gender):
        x = F.relu(self.cnn(images))
        x = F.maxpool2d(x, 2, 2) # &lt;---- new line
        x = x.view(x.size(0), -1)
</code></pre>
<p>That will reduce the <code>dim=1</code> size of <code>x</code> by a factor of 4.</p>

Aishuvenkat09: <p>modifying definition of fc1 worked. Thank you!</p>