Classification model with mixed input types

Anna_Inberg: <p>Good afternoon,</p>
<p>I’m building a binary classification model based on 2 inputs: images and numeric data.<br>
I’ve encountered a PyTorch tutorial on building such a model and built my model in a similar way:</p>
<pre><code class="lang-auto">class MixedNetwork(nn.Module):
    def __init__(self):
        super(MixedNetwork, self).__init__()
        
        image_modules = list(models.resnet50().children())[:-1]
        self.image_features = nn.Sequential(*image_modules)

        self.landmark_features = nn.Sequential(
            nn.Linear(in_features=96, out_features=192,bias=False), 
            nn.ReLU(inplace=True), 
            nn.Dropout(p=0.25),
            nn.Linear(in_features=192,out_features=1000,bias=False), 
            nn.ReLU(inplace=True), 
            nn.Dropout(p=0.25))
        
        self.combined_features = nn.Sequential(
            nn.Linear(3048, 512),
            nn.ReLU(),
            nn.Linear(512, 32),
            nn.ReLU(),
            nn.Linear(32,1))
        
    def forward(self, image, landmarks):
        a = self.image_features(image)
        # print("shape of a", a.shape)
        b = self.landmark_features(landmarks)
        # print("shape of b", b.shape)
        x = torch.cat((a.view(a.size(0), -1), b.view(b.size(0), -1)), dim=1)
        x = self.combined_features(x)
        x = torch.sigmoid(x)
        return x
</code></pre>
<p>How can I make sure that the model is using pre-trained resnet50 weights?</p>

ptrblck: <p>Specify the <code>weights</code> in <code>models.resnet50</code> to the desired set of pretrained weights or use the older <code>pretrained=True</code> argument to use the pretrained resnet.</p>

Anna_Inberg: <p>Thanks a lot!</p>
<p>One more question - I tried another model, vgg16, but there was the following mistake:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-17-8fdc3c23039e&gt; in &lt;module&gt;
     11 
     12 trainer = Trainer(criterion,optimizer)
---&gt; 13 trainer.fit(model, train_dataloader, val_dataloader,epochs = epochs)

7 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py in forward(self, input)
    112 
    113     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 114         return F.linear(input, self.weight, self.bias)
    115 
    116     def extra_repr(self) -&gt; str:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x26088 and 3048x32)
</code></pre>
<p>What needs to be changed to make it work?</p>

ptrblck: <p><code>torchvision.models.vgg16</code> uses an adaptive pooling layer to avoid shape mismatch errors in the linear layer. Could you describe if and what you’ve changed in the model which raises the shape mismatch?</p>

Anna_Inberg: <p>I didn’t change anything but the model name.<br>
The line:</p>
<pre><code class="lang-auto">image_modules = list(models.resnet50().children())[:-1]
</code></pre>
<p>just put vgg16 instead of resnet50, tried to run it without any changes.</p>
<p>Data dimensions are:<br>
images = torch.Size([1, 3, 224, 224])<br>
numeric data = torch.float32 torch.Size([1, 96])</p>

ptrblck: <p>It’s unclear to me how you are executing the model, so please post a minimal and executable code snippet to reproduce the error.<br>
Based on the already posted code snippet I would assume you are breaking the <code>forward</code> pass by trying to wrap some modules into an <code>nn.Sequential</code> container without making sure the functional API calls from the original model’s <code>forward</code> method are also used.</p>

Anna_Inberg: <p>Here’s the training functions:</p>
<pre><code class="lang-auto">class Trainer():
    
    def __init__(self,criterion = None,optimizer = None,schedular = None):
        
        self.criterion = criterion
        self.optimizer = optimizer
        self.schedular = schedular
    
    def train_batch_loop(self,model,train_dataloader):
        
        train_loss = 0.0
        train_acc = 0.0
        metric = BinaryAccuracy().to(device)
                
        for sample in train_dataloader:
            # move the data to CPU
            images = sample["image"].type(torch.cuda.DoubleTensor).to(device)
            landmarks = sample["landmarks"].type(torch.cuda.DoubleTensor).to(device)
            labels = sample["labels"].type(torch.cuda.DoubleTensor).to(device)
                    
            self.optimizer.zero_grad()
            logits = model(images, landmarks)
            loss = self.criterion(logits, labels)
            
            loss.backward()
            self.optimizer.step()
            
            train_loss += loss.item()
            # train_acc += accuracy(logits, labels)
            train_acc += metric(logits, labels)
            
        return train_loss / len(train_dataloader), train_acc / len(train_dataloader) 

    
    def valid_batch_loop(self,model,val_dataloader):
        
        valid_loss = 0.0
        valid_acc = 0.0
        metric = BinaryAccuracy().to(device)
        
        for sample in val_dataloader:
            # move the data to CPU
            images = sample["image"].type(torch.cuda.DoubleTensor).to(device)
            landmarks = sample["landmarks"].type(torch.cuda.DoubleTensor).to(device)
            labels = sample["labels"].type(torch.cuda.DoubleTensor).to(device)

            self.optimizer.zero_grad()
            logits = model(images, landmarks)
            loss = self.criterion(logits,labels)
            
            valid_loss += loss.item()
            valid_acc += metric(logits, labels)
            
        return valid_loss / len(val_dataloader), valid_acc / len(val_dataloader)
            
        
    def fit(self,model,trainloader,validloader,epochs):
        
        valid_min_loss = np.Inf 
        
        for i in range(epochs):
            
            model.train() # this turn on dropout
            avg_train_loss, avg_train_acc = self.train_batch_loop(model,trainloader) ###
            # print("Epoch : {} Train Loss : {:.6f} Train Acc : {:.6f}".format(i+1, avg_train_loss, avg_train_acc))


            model.eval()  # this turns off the dropout lapyer and batch norm
            avg_valid_loss, avg_valid_acc = self.valid_batch_loop(model,validloader) ###
            
            if avg_valid_loss &lt;= valid_min_loss :
                print("Valid_loss decreased {} --&gt; {}".format(valid_min_loss,avg_valid_loss))
                torch.save(model.state_dict(),'ColabCatPainModel.pt')
                valid_min_loss = avg_valid_loss

                
            print("Epoch : {} Train Loss : {:.6f} Train Acc : {:.6f}".format(i+1, avg_train_loss, avg_train_acc))
            print("Epoch : {} Valid Loss : {:.6f} Valid Acc : {:.6f}".format(i+1, avg_valid_loss, avg_valid_acc))
</code></pre>
<p>Execution:</p>
<pre><code class="lang-auto">model = MixedNetwork()
model.double()
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = model.to(device)
print(device)

criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters() , lr = 0.0001)
epochs = 10
batch_size = 10

trainer = Trainer(criterion,optimizer)
trainer.fit(model, train_dataloader, val_dataloader,epochs = epochs)
</code></pre>

Anna_Inberg: <p>@<a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a></p>

ptrblck: <p>Your code is unfortunately not executable, so please update it so that I could run it without any dataset dependencies to reproduce the issue and debug it further.</p>