Transfer Learning - Long Training Times || Loss not converging - GoogLeNet

learner13: <p>Hi Community and thanks in advance for the help.</p>
<p>I am working on transfer learning - specifically GoogLeNet model with the Food101 Dataset. Code is below. I think everything is in order from data preprocessing through to training but the training times are incredibly long and the loss doesn’t seem to be converging. I was running it on CPU initially to make sure it was functioning fine before xfering to GPU to run. Code is below. Any advice or assistance would be greatly appreciated.</p>
<p>Model:</p>
<pre><code class="lang-auto">from torchinfo import summary 
for params in google_net.parameters():
    params.requires_grad = False
## Change the number of output params in the fc layer
num_classes = 101 ## output classes
google_net.fc = nn.Linear(1024, num_classes)

# Check Model Summary
sum = summary(google_net, 
        input_size=(128, 3, 256, 256), # make sure this is (batch_size, color_channels, height, width)
        verbose=0,
        col_names=["input_size", "output_size", "num_params", "trainable"],
        col_width=20,
        row_settings=["var_names"]
)
print(sum)
</code></pre>
<p>Only trainable layer in the output of above is the .fc layer at the end. Output below:</p>
<p>Total params: 5,703,429<br>
Trainable params: 103,525 ## .fc layer only<br>
Non-trainable params: 5,599,904<br>
Total mult-adds (G): 47.92</p>
<p>Input size (MB): 19.27<br>
Forward/backward pass size (MB): 1651.82<br>
Params size (MB): 22.81<br>
Estimated Total Size (MB): 1693.90</p>
<p>Data PreProcess:</p>
<pre><code class="lang-auto">T_form_gog = transforms.Compose([transforms.CenterCrop(224),
                                transforms.Resize((256,256)),
                                transforms.ToTensor(), #standardise to 0-1
                                transforms.Normalize(
                                    mean = [0.485, 0.456, 0.406],
                                    std = [0.229, 0.224, 0.225])]) ## Taken from pytorch googlenet documentation 

trg_101_gog = torchvision.datasets.Food101(root = './data',
                                          split = 'train',
                                          transform = T_form_gog,
                                          download = True)

val_101_gog = torchvision.datasets.Food101(root = './data',
                                          split = 'test',
                                          transform = T_form_gog,
                                          download = True)
</code></pre>
<p>Data Loaders:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader
Batch_size = 128

trg_load_gog = DataLoader(trg_101_gog,
                          batch_size  = Batch_size,
                          shuffle = True,
                          )
val_load_gog = DataLoader(val_101_gog,
                          batch_size = Batch_size,
                          shuffle = True)
</code></pre>
<p>Params:</p>
<pre><code class="lang-auto">from torch import nn 
loss_fn = nn.CrossEntropyLoss()
optim = torch.optim.Adam(google_net.parameters(), lr = 0.001)
</code></pre>
<p>Train Loop:</p>
<pre><code class="lang-auto">from datetime import datetime as DT
NUM_EPOCHS = 30
train_losses = []

for epoch in range(NUM_EPOCHS):
    train_loss = 0
    test_loss = 0
    tic = DT.now()
    for batch, (img, label) in enumerate(trg_load_gog):

        ## zero the grads
        optim.zero_grad()

        ## forward 
        output = google_net(img)

        ## calc losses 
        loss = loss_fn(output.squeeze(), label.long())

        ## prop losses
        loss.backward()

        ## update weights
        optim.step()

        ## update loss 
        train_loss += loss.item()
    toc = DT.now()
    r_time = toc-tic
    train_losses.append(train_loss)
    print(f'epoch: {epoch}, train loss: {train_loss}, start time: {tic}, run time:{r_time}')     
</code></pre>
<p>output is:</p>
<p>epoch: 0, train loss: 1524.6963007450104,  run time:1:02:26.554898<br>
epoch: 1, train loss: 1454.7811387777328, run time:1:07:53.805809<br>
epoch: 2, train loss: 1419.673243880272,  run time:1:07:17.671551<br>
epoch: 3, train loss: 1395.2064596414566,  run time:1:07:22.943297<br>
epoch: 4, train loss: 1380.3496369123459,  run time:1:07:15.002554<br>
epoch: 5, train loss: 1365.1366027593613,  run time:1:07:17.721393<br>
Paused…</p>
<p>My initial thoughts are on the .fc. Should I add in other linear layers to make this more efficient/ effective. I was trying to use the original architecture as much as possible.</p>
<p>Any advice and/or assistance would be greatly appreciated.</p>
<p>Thanks again</p>