Model is not learning/training

username: Laura_Arakaki
cooked: <p>Here is my training loop code:</p>
<pre><code class="lang-auto">def train(model, train_dl, valid_dl, loss_fn, optimizer, scheduler, acc_fn, epochs=50):
  
    start = time.time()

    model.cuda()

    train_loss, valid_loss = [], []

    train_acc, valid_acc = [], []

    best_acc = 0.0

    for epoch in range(epochs):
        print('Epoch {}/{}'.format(epoch, epochs - 1))
        print('-' * 10)

        time_epoch_start = time.time()

        for phase in ['train', 'valid']:
            if phase == 'train':
             
                model.train(True)  # Set trainind mode = true
                dataloader = train_dl

            else:
                model.train(False)  # Set model to evaluate mode
                dataloader = valid_dl

            running_loss = 0.0
            running_acc = 0.0

            step = 0

            for x, y, _ in dataloader:

                x = x.cuda()
                y = y.cuda()

                step += 1

               
                if phase == 'train':
                    optimizer.zero_grad()

                    outputs = model(x)

                    loss = loss_fn(outputs, y)

                    
                    loss.backward()
                    optimizer.step()

                    if scheduler is not None:
                        scheduler.step()

                else:
                    with torch.no_grad():
                        outputs = model(x)
                        loss = loss_fn(outputs, y.long())


             
                acc = acc_fn(outputs, y)

                running_acc  += acc * dataloader.batch_size
                running_loss += loss * dataloader.batch_size 

                if step % 100 == 0:
                    print('Current step: {}  Loss: {}  Acc: {}  AllocMem (Mb): {}'.format(step, loss, acc, torch.cuda.memory_allocated()/1024/1024))

            epoch_loss = running_loss / len(dataloader.dataset)
            epoch_acc = running_acc / len(dataloader.dataset)

            train_loss.append(epoch_loss) if phase=='train' else valid_loss.append(epoch_loss)
            train_acc.append(epoch_acc) if phase=='train' else valid_acc.append(epoch_acc)

            if phase=='train':
                train_loss_print = epoch_loss
                train_acc_print = epoch_acc
            else:
                valid_loss_print = epoch_loss
                valid_acc_print = epoch_acc

           

       
        time_epoch = time.time() - time_epoch_start

        print('Epoch {}/{} - TRAIN Loss: {:.4f} TRAIN Acc: {:.4f} - VAL. Loss: {:.4f} VAL. Acc: {:.4f} ({:.4f} seconds - {:.4f} Mb)'.format(epoch, epochs - 1, train_loss_print, train_acc_print, valid_loss_print, valid_acc_print, time_epoch, torch.cuda.memory_allocated()/1024/1024))

    time_elapsed = time.time() - start
    print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))    
    
    return model, train_loss, valid_loss, train_acc, valid_acc
</code></pre>
<p>Here I define the optimizer and learning rate</p>
<pre><code class="lang-auto">opt = torch.optim.Adam(modelo.parameters(), lr=hp_lr, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)

lr_scheduler = lr_scheduler.StepLR(opt, step_size=10, gamma=0.1) 


modelo_trained, train_loss, valid_loss, train_acc, valid_acc = train(modelo, train_dl, valid_dl, loss_fn, opt, lr_scheduler, acc_metric, epochs=num_epochs) 
</code></pre>
<p>However, when training, here is my training loss</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c2435561064354a80af0872f5ba50e8af7242d07.png" data-download-href="https://discuss.pytorch.org/uploads/default/c2435561064354a80af0872f5ba50e8af7242d07" title="training_report_loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2435561064354a80af0872f5ba50e8af7242d07_2_647x500.png" alt="training_report_loss" data-base62-sha1="rIwTdEZrLPUY2yIZKnbhGQ9tFWL" width="647" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2435561064354a80af0872f5ba50e8af7242d07_2_647x500.png, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c2435561064354a80af0872f5ba50e8af7242d07.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/2/c2435561064354a80af0872f5ba50e8af7242d07.png 2x" data-dominant-color="FDFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">training_report_loss</span><span class="informations">853×659 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: eqy
cooked: <p>What is the starting learning rate in this case?</p>
<p>I would also check the the model parameters are indeed changing by e.g., doing a sum over the model parameters after each optimizer step.</p>