Model loss not decreasing via transfer learning

MasayoMusic: <p>I tried playing around with learning rates, <code>.01, .001, .0001</code><br>
however my model loss and val loss are not decreasing.</p>
<p>I am using Densenet from Pytorch models, and have copied most of the code from the Pytorch transfer learning tutorial<br>
which some few minor changes to print out <code>val accurac</code>y every x amount of batches. I think my minor changes could be affecting training somehow, but I am not sure what I am doing wrong.</p>
<p>Here is how my training looks even after first epoch</p>
<h2><a name="training-loss-85898-acc-00102-epoch-12-1" class="anchor" href="#training-loss-85898-acc-00102-epoch-12-1"></a>Training Loss: 8.5898 Acc: 0.0102<br>
Epoch 1/2</h2>
<p>val_loss  8.5163 val_acc: 0.0191<br>
val_loss  8.9667 val_acc: 0.0255<br>
val_loss  9.3078 val_acc: 0.0127<br>
val_loss  8.7855 val_acc: 0.0318<br>
val_loss  8.6217 val_acc: 0.0255<br>
val_loss  9.0631 val_acc: 0.0191<br>
val_loss  8.5167 val_acc: 0.0255<br>
val_loss  9.0499 val_acc: 0.0191<br>
val_loss  9.0549 val_acc: 0.0255<br>
val_loss  8.8373 val_acc: 0.0191<br>
val_loss  8.9288 val_acc: 0.0191<br>
val_loss  8.9968 val_acc: 0.0127<br>
val_loss  9.2790 val_acc: 0.0127<br>
val_loss  9.4389 val_acc: 0.0191<br>
val_loss  8.6907 val_acc: 0.0318<br>
val_loss  9.0903 val_acc: 0.0191<br>
val_loss  9.0093 val_acc: 0.0191<br>
val_loss  9.4387 val_acc: 0.0127<br>
val_loss  9.1059 val_acc: 0.0191<br>
val_loss  9.3480 val_acc: 0.0127<br>
val_loss  8.9435 val_acc: 0.0191<br>
val_loss  8.4412 val_acc: 0.0318<br>
val_loss  8.8712 val_acc: 0.0382<br>
val_loss  8.9125 val_acc: 0.0191<br>
val_loss  9.3815 val_acc: 0.0127<br>
val_loss  9.0214 val_acc: 0.0191<br>
val_loss  9.4234 val_acc: 0.0127<br>
val_loss  9.1625 val_acc: 0.0191</p>
<p>Here is the code (I can’t seem to fix formatting on this site)</p>
<blockquote>
<p>from <strong>future</strong> import print_function, division</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
from torch.optim import lr_scheduler<br>
import numpy as np<br>
import torchvision<br>
from torchvision import datasets, models, transforms<br>
import matplotlib.pyplot as plt<br>
import time<br>
import os<br>
import copy</p>
<p>def prepare_dataset_from_folder(data_dir, size, batch_size):</p>
</blockquote>
<pre><code> """prepare datasets into dataloaders
</code></pre>
<blockquote>
<pre><code>args
------
data_dir = root_dir holding 'train_dir' and 'val_dir'

size = size of image

batch_size = batch_size
"""

data_transforms = {
    'training': transforms.Compose([
       transforms.Resize((size,size)),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'validation': transforms.Compose([
        transforms.Resize((size,size)),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}


image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['training', 'validation']}


dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size= batch_size,
                                             shuffle=True, num_workers=4) #num_workers leads to errors
              for x in ['training', 'validation']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['training', 'validation']}
class_names = image_datasets['training'].classes

return dataloaders, dataset_sizes, class_names
</code></pre>
</blockquote>
<blockquote>
<p>def train_model(model, criterion, optimizer, scheduler, num_epochs=3, best_acc = 80.0, batch_size = 5):<br>
since = time.time()<br>
dataloaders, dataset_sizes, class_names =  prepare_dataset_from_folder(data_dir, size, batch_size)</p>
<pre><code>for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)

    running_loss = 0.0
    running_corrects = 0

    for num, data in enumerate(dataloaders["training"]):
       
        if num % 100 == 0:
               val_running_loss = 0.0
               val_running_corrects = 0
               model.eval()

               for val_num, val_data in enumerate(dataloaders["validation"]):
                   inputs, labels = val_data
                   inputs = inputs.to(device)
                   labels = labels.to(device)
                   with torch.set_grad_enabled(False):
                       outputs = model(inputs)
                       _, preds = torch.max(outputs, 1)
                       loss = criterion(outputs, labels)

                   val_running_loss += loss.item() * inputs.size(0)
                   val_running_corrects += torch.sum(preds == labels.data)

               val_loss = ( val_running_loss/dataset_sizes['validation'])
               val_acc = (val_running_corrects.double() / dataset_sizes['validation'])#.cpu().numpy()
               print('val_loss  {:.4f} val_acc: {:.4f}'.format(val_loss, val_acc))
               if val_acc &gt; best_acc:
                  print("Saving due to high val accuracy")
                  x = datetime.datetime.now()
                  time_stamp_str = str(x).split(' ')[0] + "_" 
                  save_path = os.path.join(SAVE_DIR, f"val_acc_{val_acc}.pt")
                  torch.save(model, save_path)


        model.train()
        inputs, labels = data
        inputs = inputs.to(device)
        labels = labels.to(device)
        scheduler.step()  
       
        optimizer.zero_grad()

        with torch.set_grad_enabled(True):
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)
                loss.backward()
                optimizer.step()

        running_loss += loss.item() * inputs.size(0)
        running_corrects += torch.sum(preds == labels.data)

    epoch_loss = running_loss / dataset_sizes['training']
    epoch_acc = running_corrects.double() / dataset_sizes['training']


    print('Training Loss: {:.4f} Acc: {:.4f}'.format(epoch_loss, epoch_acc))

time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(
time_elapsed // 60, time_elapsed % 60))


return model
</code></pre>
<p>if <strong>name</strong>==“<strong>main</strong>”:</p>
<pre><code>data_dir = "TEST 10"
device = torch.device("cuda:0")
PATH =" "
size = 224
batch_size = 5



# pick your model
#model_ft = torch.load(PATH)
#model_ft = models.resnet18(pretrained=True)
model_ft = models.densenet201(pretrained=True)



#num_ftrs = model_ft.fc.in_features #in features
num_ftrs = 1920 #densenet 201
#https://discuss.pytorch.org/t/what-does-the-fc-in-feature-mean/4889


model_ft.fc = nn.Linear(num_ftrs, 2)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=2, gamma=0.1)


model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=3, batch_size = batch_size)
</code></pre>
</blockquote>
<p>Strangely if I use original script from transfer learning with minimal changes, my model begins to learn:</p>
<blockquote>
<blockquote>
<p><span class="hashtag-raw">#original</span> script</p>
</blockquote>
<p>from <strong>future</strong> import print_function, division<br>
import pretrainedmodels as ptmodels</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
from torch.optim import lr_scheduler<br>
import numpy as np<br>
import torchvision<br>
from torchvision import datasets, models, transforms</p>
<p>import matplotlib.pyplot as plt<br>
import time<br>
import os<br>
import copy</p>
<p>def train_model(model, criterion, optimizer, scheduler, num_epochs=25):<br>
since = time.time()</p>
<pre><code>best_model_wts = copy.deepcopy(model.state_dict())
best_acc = 0.0

for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)

    # Each epoch has a training and validation phase
    for phase in ['training', 'validation']:
        if phase == 'training':
            scheduler.step()  #what is this
            model.train()  # Set model to training mode
        else:
            model.eval()   # Set model to evaluate mode

        running_loss = 0.0
        running_corrects = 0

        # Iterate over data.
        for num, data in enumerate(dataloaders[phase]):
            inputs, labels = data
            inputs = inputs.to(device)
            labels = labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'training'):
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'training':
                    loss.backward()
                    optimizer.step()

            # statistics
            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)

            if num % 100 == 0:
                print("500 batches")
                temp_running_corrects = (running_corrects.double() / ( len(inputs) *  num + 1)).cpu().numpy()
                print(temp_running_corrects)
                #print('running_corrects', running_corrects/( len(inputs) *  num + 1).numpy())

        epoch_loss = running_loss / dataset_sizes[phase]
        epoch_acc = running_corrects.double() / dataset_sizes[phase]

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(
            phase, epoch_loss, epoch_acc))

        # deep copy the model
        if phase == 'validation' and epoch_acc &gt; best_acc:
            best_acc = epoch_acc
            best_model_wts = copy.deepcopy(model.state_dict())

    print()

time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(
    time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))

# load best model weights
model.load_state_dict(best_model_wts)
save_path = os.path.join(SAVE_DIR, f"val_acc_{val_acc}.pt")
torch.save(model, save_path)
</code></pre>
</blockquote>
<p>Training Loss:</p>
<h2><a name="epoch-02-2" class="anchor" href="#epoch-02-2"></a>Epoch 0/2</h2>
<p>500 batches<br>
0.0<br>
500 batches<br>
0.681592039800995<br>
500 batches<br>
0.6359102244389028<br>
500 batches<br>
0.6422628951747088<br>
500 batches<br>
0.6541822721598003<br>
500 batches<br>
0.6763236763236763<br>
500 batches<br>
0.694421315570358<br>
500 batches<br>
0.7059243397573162<br>
500 batches<br>
0.7108057464084947<br>
500 batches<br>
0.7096057745696835<br>
500 batches<br>
0.7126436781609196<br>
500 batches<br>
0.7142208087233075<br>
500 batches<br>
0.7251145356101625<br>
500 batches<br>
0.7293348712033834<br>
500 batches<br>
0.729382363441628<br>
500 batches<br>
0.7340886371209597<br>
500 batches<br>
0.7363323961262105<br>
500 batches<br>
0.7391943546015877<br>
500 batches<br>
0.7428492085531797<br>
500 batches<br>
0.7432254669823731<br>
500 batches<br>
0.74656335916021<br>
500 batches<br>
0.7493453939538205<br>
500 batches<br>
0.7521017950465804</p>

god_sp33d: <p>Hey,</p>
<p>The code seems ok to me(training loop).<br>
why don’t you first freeze the feature extractor of the densenet and train only the classifier? If that works then you can tune the features too ? After you declare the model</p>
<pre><code class="lang-auto">for param in model_ft.features.parameters():
    param.requires_grad = False
</code></pre>
<p>Now in the optimizer just train the classifier by modifying optimizer to train only classifier parameters.</p>
<pre><code class="lang-auto">optimizer = optim.SGD(filter(lambda p: p.requires_grad=True, model_ft.parameters()), lr=0.001)
</code></pre>
<p>Also evaluate for fewer iterations to see if it is learning. If this works better then I would suggest you to train classifier with same learning rate but feature extractor with lower lr like 5e-4 or 1e-4 may be ?</p>

MasayoMusic: <p>Thank you for your post.</p>
<p>I have noticed,</p>
<ol>
<li>If I use the original transfer learning script (from tutorial), my model seems to be learning I will post the script in the OP after this post. I am not sure what I am doing wrong in my revision</li>
</ol>
<p>As for you post, thank you! If I wanted to train at different parts of the model at<br>
different rates, do I create multiple optimizers?</p>

god_sp33d: <p>Did you get the transfer learning tutorial working ? what were you doing wrong ?</p>
<p>For your question, You can do it either way creating 2 optimizers and updating individually or creating groups and setting values in the groups like</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD([{'params': filter(lambda p: p.requires_grad, model.features.parameters()), 'lr':1e-4, },
                             {'params': filter(lambda p: p.requires_grad, model.classifier.parameters())}], lr=1e-3, momentum=0.95, weight_decay=1e-5)
</code></pre>
<p>you can set parameters specific to every part of your model like above. And the values outside the list are like default values, will be used if not specifically set.</p>

MasayoMusic: <p>Ah I see! . I haven’t gotten it working yet. I haven’t figured out why original code is working and not my revised code. There must be something wrong with my revised code. I will play around with a little more and see if I can figure it out.</p>
<p>EDIT:</p>
<p>Even turning almost all layers for training:</p>
<pre><code class="lang-auto">for param in model_ft.features.parameters():
    param.requires_grad = False
</code></pre>
<p>not resulting in improvements.</p>

preethamam: <p><a class="mention" href="/u/masayomusic">@MasayoMusic</a> the problem is with the <code>scheduler.step()</code>, it is inside the <code>for loop</code> of the data enumerator. It has to be with <code>epochs for loop</code>. So every time the scheduler in the optimizer was making the weights <code>zeros</code> and thus every epoch has the same loss (as the first epoch).</p>
<p>I hope this helps someone with the same tiny mistake!</p>

ShinyDemon: <p><a class="mention" href="/u/preethamam">@preethamam</a> That depends on the scheduler. Some schedulers (<a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.OneCycleLR.html#torch.optim.lr_scheduler.OneCycleLR" rel="noopener nofollow ugc">OneCycleLR</a>, <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CyclicLR.html#torch.optim.lr_scheduler.CyclicLR" rel="noopener nofollow ugc">CyclicLR</a>) should be called on every step. The specific scheduler used here (StepLR) should indeed be called every epoch.</p>