`scheduler` gives me worse results

jrreda: <p>I tried to add a scheduler to my network for better convergence but I get worse results</p>
<p>without any schedulers:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/dff43055c68f75fed1241f177293112f0c605891.png" data-download-href="https://discuss.pytorch.org/uploads/default/dff43055c68f75fed1241f177293112f0c605891" title="Screenshot 2023-02-22 at 22-00-58 Denoising Autoencoders (DAE) in PT &amp; PT⚡"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/dff43055c68f75fed1241f177293112f0c605891.png" alt="Screenshot 2023-02-22 at 22-00-58 Denoising Autoencoders (DAE) in PT &amp; PT⚡" data-base62-sha1="vXbEaZmtoSozHJBsVSteQRb8Fxv" width="641" height="500" data-dominant-color="F5F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-02-22 at 22-00-58 Denoising Autoencoders (DAE) in PT &amp; PT⚡</span><span class="informations">665×518 7.27 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>with schedulers on the same network:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/516a9a27cfd6a3df90fae812935d521dea45b093.png" data-download-href="https://discuss.pytorch.org/uploads/default/516a9a27cfd6a3df90fae812935d521dea45b093" title="Screenshot 2023-02-22 at 22-07-56 Denoising Autoencoders (DAE) in PT &amp; PT⚡ Kaggle"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/516a9a27cfd6a3df90fae812935d521dea45b093.png" alt="Screenshot 2023-02-22 at 22-07-56 Denoising Autoencoders (DAE) in PT &amp; PT⚡ Kaggle" data-base62-sha1="bCf4i0PbjsEdRR89ft647yhf7Xl" width="447" height="500" data-dominant-color="2E3132"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-02-22 at 22-07-56 Denoising Autoencoders (DAE) in PT &amp; PT⚡ Kaggle</span><span class="informations">569×636 13.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>all code: <a href="https://www.kaggle.com/code/jrreda/denoising-autoencoders-dae-in-pt-pt/notebook#Denoising-Linear-Autoencoder" class="inline-onebox" rel="noopener nofollow ugc">Denoising Autoencoders (DAE) in PT &amp; PT⚡ | Kaggle</a></p>
<p>what is my mistake?</p>

eqy: <p>Are you calling <code>scheduler.step()</code>? (In case there is a learning rate warmup and the learning rate is never changed from the initial value)</p>

jrreda: <p>yes, I do as I see in the documentation :</p>
<pre><code class="lang-auto">
# tqdm progress bar
pbar = tqdm(total=epochs * len(dataloader), file=sys.stdout, colour='green')

model.train()
for epoch in range(1, epochs+1):
    # monitor training loss
    train_loss = 0.0
    
    # Loops over our dataset in the batches the data loader creates for us
    for data in dataloader:
        # Get the data
        images, labels = data
        # flatten images, keeps the three channels and merges all the remaining dimensions into one,
        # figuring out the appropriate size.
        images = images.view(images.size(0), -1)
        
        # add noise to the images
        noisy_images = images + 0.5 * torch.randn(*images.shape) # 0.5: noise_factor
        noisy_images = np.clip(noisy_images, 0., 1.)
        
        # place all the tensors on the same device
        images, noisy_images = images.to(device), noisy_images.to(device)
            
        # forward pass, Feeds a batch through our model
        outputs = model(noisy_images)
        # calculate the loss - measure how far (wrong) the noisy from the original image
        loss = criterion(outputs, images)
        
        # clear the gradients of all optimized variables from the last round
        optimizer.zero_grad()
        # backward pass, Propagate the loss signal backward
        loss.backward()
        # perform a single optimization step (update model parameters)
        optimizer.step()
        # Step the learning rate scheduler
        scheduler.step(loss)
        
        # update running training loss
        # transform the loss to a Python number with .item(), to escape the gradients.
        train_loss += loss.item() * images.size(0)
        
        # update tqdm progress bar
        pbar.update(1)
        pbar.refresh()
     
    train_loss = train_loss / len(dataloader)
    print(f'Epoch: {epoch} \tTraining Loss: {train_loss:0.6f}')
    
# close tqdm progress bar
pbar.close()
</code></pre>

eqy: <p>Could you share how you are creating the scheduler (e.g., how the optimizer is being passed, etc.) and check if the state dict  and learning rate of the optimizer (e.g., via <code>.state_dict()</code> and <code>.get_lr()</code> ) makes sense?</p>

jrreda: <p>that’s how i instantiate the optimizer &amp; scheduler:</p>
<pre><code class="lang-auto">lr = 1e-3
optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=1e-5)  # weight decay: L2 penalty
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=5)
</code></pre>

jrreda: <p>I found the problem: moving the line <code>scheduler.step(loss)</code> outside the inner loop after calculating <code>train_loss</code>. and passing in <code>train_loss</code> instead of <code>loss</code> to the scheduler’s step method.</p>
<p>My mistake <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=12" title=":man_facepalming:" class="emoji" alt=":man_facepalming:" loading="lazy" width="20" height="20"></p>