Loss stuck for regression model

dejhost: <p>I’m training a model that returns 2 parameters. These two parameters are used for classical image processing:</p>
<ol>
<li>a threshold for the  kirsch-operator</li>
<li>the number of iterations for billateral filter.</li>
</ol>
<p>The model trains using 300 representative images, along with both parameters that were manually determined.</p>
<ul>
<li>I am currently using resnet18. A convolutional regression model.</li>
<li>The fully connected layer is changed to output 2 nodes.</li>
<li>As loss function I’ve chosen is the mean squared loss.</li>
<li>Reducelronplateau is used as a learning rate scheduler to minimize validation loss.</li>
</ul>
<p>Unfortunately, my validation-loss is stuck. It settles inbetween 2000 and 3000.</p>
<p>Here are some of the things I tried:</p>
<ol>
<li>Experiment with different models, including resnet36 resnet50 vgg16 mobilenet.</li>
<li>Resized and changed the batchsize.</li>
<li>Multiple heads after the renset18 feature layer for both output and calculated loss, seperately for iteration and thresholding</li>
<li>Ssingle output model for each threshold and iterations separately.</li>
<li>Replaced RGB-images tried HSV-images.</li>
</ol>
<p>I’d really appreaciate suggestions on how to succeed. Thank you very much.</p>

JuanFMontesinos: <p>No really clue but some suggestions:</p>
<ul>
<li>Do you know that the task is feasible? Can you estimate those numbers your self just from the image?</li>
<li>Which kind of images are you using? 300 are very few images. Toy datasets (mnist or cifar) have already 60k images. Something more adequate could be using neural networks are feature extractors and then use classic tools like trees or regression models. Even training only some fully connected layers on top of that. However, retraining a res net seems overkilling it (and it’s probably overfitting)</li>
<li>networks are not good at all for predicting “numbers”. For example, classification problems are not trained to predict an ID rather but to generate a probability vector of all the possible values.  Segmentation predicts per-class binary masks, landmark estimation predicts gaussian distributions per node etcetera…  Also, predicting unbounded values makes the training more unstable as losses can vary a lot. if your network predicts 100 but the result was 1000 the MSE is so huge and the lr is not adjusted for such big gradients compared to predicting 100 and 150.</li>
</ul>
<p>Hope it helps</p>

dejhost: <p>Hello Juan,</p>
<p>thank you for your elaboration - most appreciated!<br>
Yes, the task is feasable. After some execising, I can estimate the numbers myself. Of course: there is always room for improvement after the first guess.</p>
<p>The images come from high-resolution cameras (mirror-cameras). All taken under water. The two parameters are supposed the adapted the processing methods to the visibility conditions.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a54b7bffe1325eb97a1f92d2e9dd7b8c969ee0d2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a54b7bffe1325eb97a1f92d2e9dd7b8c969ee0d2" title="0008_DSC05002"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a54b7bffe1325eb97a1f92d2e9dd7b8c969ee0d2_2_690x460.jpeg" alt="0008_DSC05002" data-base62-sha1="nAgvrtvi0ObZoq2E3RF6Jkoed7s" width="690" height="460" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a54b7bffe1325eb97a1f92d2e9dd7b8c969ee0d2_2_690x460.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a54b7bffe1325eb97a1f92d2e9dd7b8c969ee0d2_2_1035x690.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a54b7bffe1325eb97a1f92d2e9dd7b8c969ee0d2_2_1380x920.jpeg 2x" data-dominant-color="609D98"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">0008_DSC05002</span><span class="informations">1920×1281 24.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Other than that, there is little variation in the content of the images. That’s why I believe that 300 (representative) images should be sufficient.</p>
<p>I will do some more testing based on your input.<br>
Would it make sense to post the source-code?</p>
<p>Thank you once again for your response.</p>
<p>Adding the code here:</p>
<pre data-code-wrap="from"><code class="lang-from">import cv2
from torch.utils.data import Dataset
import pandas as pd
import os
import glob
from tqdm import tqdm
from PIL import Image, ImageFilter
import random
import torch
import torch.nn as nn
from torch.nn import MSELoss
from torch.optim.lr_scheduler import ReduceLROnPlateau
from torch.utils.data import DataLoader, random_split
import torchvision.transforms as transforms
import numpy as np
import torchvision.models as models
# from torchsummary import summary
from itertools import product, combinations
from random import randint
import torch.nn.functional as F
torch.manual_seed(0)
torch.cuda.empty_cache()
extensions = [".jpg", ".jpeg", ".JPG"]


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# device = 'cpu'

class CardDataset(Dataset):
    def __init__(self, root_dir, data_file_path, transform=None):
      self.Df = pd.read_csv(data_file_path)
      print(self.Df)
      self.root_dir = root_dir
      self.transform = transform

    def __len__(self):
        return len(self.Df)

    def __getitem__(self, index):
        #img_path = os.path.join(self.root_dir, self.Df['Image Name'][index]+'.JPG')        
        #print('type img_path orig: ', type(img_path))
        #print('img_path orig: ', img_path)

        img_path = [f for f in glob.glob(os.path.join(self.root_dir, self.Df['Image Name'][index]+'*'), recursive=True) if os.path.splitext(f)[1] in extensions]
        img_path = str(img_path)[2:-2]
        #print('type img_path new: ', type(img_path))
        #print('img_path new: ', img_path)

        img = Image.open(img_path)
        y_label = torch.tensor([float(self.Df['Iterations'][index]), float(self.Df['Threshold'][index])])
        # y_label = torch.tensor( [float(self.Df['Threshold'][index])])
        # img = Image.new('RGB',(400,200))
        # img.paste(im1,(0,0))
        # img.paste(im2,(200,0))
        # label = y_label.item()
        # print(label)
        if self.transform is not None:
            img = self.transform(img)

            # print(img.size)

        return img, y_label

transform_train = transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Resize((1024, 1024)),
            #transforms.RandomCrop(1024),
            # transforms.RandomRotation(180),
            # transforms.RandomPerspective(),            
            # transforms.ColorJitter(saturation=(0.8, 1.3), contrast=(0.8, 1.4), brightness=(0.8, 1.25)),
            # transforms.RandomPerspective(distortion_scale=0.6, p=1.0)
            # transforms.Resize(300),
            # transforms.Normalize(mean=[0.485, 0.456, 0.406],
            #                      std=[0.229, 0.224, 0.225]),
        ]
    )


num_epochs = 600
learning_rate = 0.001
train_CNN = False
batch_size = 16
shuffle = True
pin_memory = True
num_workers = 2
print(learning_rate)
print(num_epochs)

train_set = CardDataset('all_days','all_days.csv',transform=transform_train)
train_set, validation_set = random_split(train_set, [int(0.85*len(train_set)), len(train_set)-int(0.85*len(train_set))])
#validation_set = CardDataset("hologram_classifier_dataset_splitted",'validation',transform=transform_val)
# train_set, validation_set = torch.utils.data.random_split(dataset,[train_size, val_size], generator=torch.Generator().manual_seed(2))
train_loader = DataLoader(dataset=train_set, shuffle=shuffle, batch_size=batch_size, num_workers=num_workers)
validation_loader = DataLoader(dataset=validation_set, shuffle=shuffle, batch_size=batch_size, num_workers=num_workers)
print(len(train_set))

class parameter_model(torch.nn.Module):
  def __init__(self):
    super().__init__()
    self.Conv1 = torch.nn.Conv2d(3, 16, 3, padding = 'same')
    self.Conv2 = torch.nn.Conv2d(16, 32, 3, padding = 'same')
    self.Conv3 = torch.nn.Conv2d(32, 64, 3, padding = 'same')
    self.Conv4 = torch.nn.Conv2d(64, 128, 3, padding = 'same')
    self.pool = torch.nn.MaxPool2d(2,2)
    self.fc1 = torch.nn.Linear(in_features = 128, out_features = 64)
    self.fc2 = torch.nn.Linear(64, 2)
    self.adaptive_pool  = torch.nn.AdaptiveAvgPool2d(output_size=(1, 1))

  def forward(self, x):
    x = self.pool(F.relu(self.Conv1(x)))
    x = self.pool(F.relu(self.Conv2(x)))
    x = self.pool(F.relu(self.Conv3(x)))
    x = self.pool(F.relu(self.Conv4(x)))
    x = self.adaptive_pool(x)
    x = torch.flatten(x, 1)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    return x




# # ct =0
# # # childern = model.children()
# # # print(childern)
# # for child in model.children():
# #   ct += 1
# #   # print(child)
# #   if ct &lt; 10:
# #       for param in child.parameters():
# #           param.requires_grad = False
# # # model.to(device)
# # print(ct)
model = parameter_model()
# print(model)
# from torch.nn.modules.conv import Conv2d
# model = models.resnet18(pretrained=False)
# model.fc = nn.Linear(in_features=512, out_features=1, bias=True)
# print(model)


if torch.cuda.is_available():
    model.to(device)
# loss_weight = torch.Tensor([0.5, 1])
# loss_weight = loss_weight.to(device)
criterion = MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
scheduler = ReduceLROnPlateau(optimizer, 'min', patience=15)

def train():
    # % % time
    # keeping-track-of-losses
    
    for epoch in range(1, num_epochs + 1):
        # keep-track-of-training-and-validation-loss
        # train_loss = 0.0

        # training-the-model
        model.train()
        # i= 0
        total_loss = []

        for data, target in tqdm(train_loader):
            # print('yes')
            # i = i+1
            # print(i)
            # move-tensors-to-GPU
            optimizer.zero_grad()
            data = data.to(device)
            target = target.to(device)

            # target = target.long()
            # clear-the-gradients-of-all-optimized-variables
           
            # forward-pass: compute-predicted-outputs-by-passing-inputs-to-the-model
            output = model(data)
            
            # results = torch.max(output, 1).indices
   
            # calculate-the-batch-loss
            loss = criterion(output, target)
            # backward-pass: compute-gradient-of-the-loss-wrt-model-parameters
            loss.backward()
            # perform-a-ingle-optimization-step (parameter-update)
            optimizer.step()
            #perform learning rate scheduler step
            total_loss.append(loss)
            # update-training-loss
            # train_loss += loss.item() * data.size(0)
        val_loss = 0
        model.eval()
        with torch.no_grad():
            for data, target in tqdm(validation_loader):
                # print('yes')
                # i = i+1
                # print(i)
                # move-tensors-to-GPU
                
                data = data.to(device)
                target = target.to(device)

                # target = target.long()
                # clear-the-gradients-of-all-optimized-variables
            
                # forward-pass: compute-predicted-outputs-by-passing-inputs-to-the-model
                output = model(data)
                
                # results = torch.max(output, 1).indices
    
                # calculate-the-batch-loss
                loss = criterion(output, target)
                val_loss = val_loss + loss
        
        scheduler.step(val_loss) 

        print(f'Epoch: {epoch}\t Train Loss: {sum(total_loss)/len(train_loader)}\t Validation Loss: {val_loss/len(validation_loader)}')
        torch.save(model.state_dict(), "model_checkpoints/checkpoint_" + str(epoch) + ".pth")
print(learning_rate)
train()```</code></pre>

JuanFMontesinos: <p>Hmmm the thing is I don’t know how difficult the task is.<br>
I would tell you to try to find a network pretrained on similar images so that you can use the features (there should be challenges in kaggle or so).<br>
In the worst case use at least vision networks pretrained on imagenet.</p>
<p>It would be nice if you could pose the whole problem in pytorch and apply the loss over the images directly. But seems diff</p>