Optimization error Adam.step()

username: Chihoub_Chiheb_Eddin
cooked: <p>I am trying to fine-tune a model on my machine, I have prepared my dataset images of 256*256. However, once I load the checkpoints file, I get the error shown below. Ps: I used the same file to test and it is working, I can retrain the model from epoch 0 normal and I don’t need to make any changes.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa4667c6dfaa026df2cb9e7795a2c2fe00a0d772.png" data-download-href="https://discuss.pytorch.org/uploads/default/aa4667c6dfaa026df2cb9e7795a2c2fe00a0d772" title="error optim"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa4667c6dfaa026df2cb9e7795a2c2fe00a0d772_2_401x500.png" alt="error optim" data-base62-sha1="oik0NdtYhJn4drWFq2av5MlAaKS" width="401" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa4667c6dfaa026df2cb9e7795a2c2fe00a0d772_2_401x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa4667c6dfaa026df2cb9e7795a2c2fe00a0d772_2_601x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aa4667c6dfaa026df2cb9e7795a2c2fe00a0d772_2_802x1000.png 2x" data-dominant-color="242525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error optim</span><span class="informations">1369×1706 185 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet to reproduce the issue?</p>

username: Chihoub_Chiheb_Eddin
cooked: <p>of course here you are. here some more details. I am using wsl ubuntu 22.04, and I am using the compressai lib. The first time, after installation, it’s worked perfectly, however I had to chmod the home repository because a warning about no permission to write and it worked.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/052eb0b52a900f317ba74fdf561f5915112ef0d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/052eb0b52a900f317ba74fdf561f5915112ef0d1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/052eb0b52a900f317ba74fdf561f5915112ef0d1_2_690x388.png" alt="image" data-base62-sha1="JQq5ud1p9TbyWUIKuwuvryLt9D" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/052eb0b52a900f317ba74fdf561f5915112ef0d1_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/052eb0b52a900f317ba74fdf561f5915112ef0d1_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/052eb0b52a900f317ba74fdf561f5915112ef0d1_2_1380x776.png 2x" data-dominant-color="1E1E1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2560×1440 373 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/chihoub_chiheb_eddin">@Chihoub_Chiheb_Eddin</a>,</p>
<p>You need to copy and paste in the code and wrap it in three backticks ```, posting screenshots doesn’t work.</p>

username: Chihoub_Chiheb_Eddin
cooked: <pre data-code-wrap="import"><code class="lang-import">import torch
import torch.nn as nn
import torch.optim as optim

load the dataset, split into input (X) and output (y) variables
dataset = np.loadtxt(‘pima-indians-diabetes.csv’, delimiter=‘,’)

X = dataset[:,0:8]

y = dataset[:,8]
X = torch.tensor(X, dtype=torch.float32)

y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1)
model = nn.Sequential(
nn.Linear(8, 12),
nn.ReLU(),
nn.Linear(12, 8),
nn.ReLU(),
nn.Linear(8, 1),
nn.Sigmoid()
)
loss_fn = nn.BCELoss() # binary cross entropy
optimizer = optim.Adam(model.parameters(), lr=0.001)
n_epochs = 100
batch_size = 10

for epoch in range(n_epochs):
for i in range(0, len(X), batch_size):
Xbatch = X[i:i+batch_size]
y_pred = model(Xbatch)
ybatch = y[i:i+batch_size]
loss = loss_fn(y_pred, ybatch)
optimizer.zero_grad()
loss.backward()
optimizer.step()
print(f’Finished epoch {epoch}, latest loss {loss}')
</code></pre>

username: ptrblck
cooked: <p>Your code is not executable as it depends on specific data. It’s also not formatted properly and does not load a checkpoint, which seems to trigger the error.</p>

username: Chihoub_Chiheb_Eddin
cooked: <p>alright, thank you. and this case:</p>
<pre><code class="lang-auto"> net = MLICPlusPlus(config=config)
    if args.cuda and torch.cuda.device_count() &gt; 1:
        net = CustomDataParallel(net)
    net = net.to(device)
    #optimizer, aux_optimizer = configure_optimizers(net, args)
    parameters = set(p for n, p in net.named_parameters() if not n.endswith(".quantiles"))
    aux_parameters = [p for n, p in net.named_parameters() if n.endswith(".quantiles")]
    optimizer = optim.Adam(parameters, lr=1e-4)
    aux_optimizer = optim.Adam(aux_parameters, lr=1e-3)
    lr_scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=[80, 100], gamma=0.1)
    criterion = RateDistortionLoss(lmbda=args.lmbda, metrics=args.metrics)

    if args.checkpoint != None:
        checkpoint = torch.load(args.checkpoint)
        # new_ckpt = modify_checkpoint(checkpoint['state_dict'])
        net.load_state_dict(checkpoint['state_dict'])
        optimizer.load_state_dict(checkpoint['optimizer'])
        aux_optimizer.load_state_dict(checkpoint['aux_optimizer'])
        # lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])
        lr_scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=[450,550], gamma=0.1)
        # lr_scheduler._step_count = checkpoint['lr_scheduler']['_step_count']
        # lr_scheduler.last_epoch = checkpoint['lr_scheduler']['last_epoch']
        # print(lr_scheduler.state_dict())
        start_epoch = checkpoint['epoch']
        best_loss = checkpoint['loss']
        current_step = start_epoch * math.ceil(len(train_dataloader.dataset) / args.batch_size)
        checkpoint = None
    else:
        start_epoch = 0
        best_loss = 1e10
        current_step = 0

    # start_epoch = 0
    # best_loss = 1e10
    # current_step = 0

    logger_train.info(args)
    logger_train.info(config)
    logger_train.info(net)
    logger_train.info(optimizer)
    optimizer.param_groups[0]['lr'] = args.learning_rate
    for epoch in range(start_epoch, args.epochs):
        logger_train.info(f"Learning rate: {optimizer.param_groups[0]['lr']}")
        current_step = train_one_epoch(
            net,
            criterion,
            train_dataloader,
            optimizer,
            aux_optimizer,
            epoch,
            args.clip_max_norm,
            logger_train,
            tb_logger,
            current_step
        )

        save_dir = os.path.join('./experiments', args.experiment, 'val_images', '%03d' % (epoch + 1))
        loss = test_one_epoch(epoch, test_dataloader, net, criterion, save_dir, logger_val, tb_logger)

        lr_scheduler.step()
        is_best = loss &lt; best_loss
        best_loss = min(loss, best_loss)

        net.update(force=True)
        if args.save:
            save_checkpoint(
                {
                    "epoch": epoch + 1,
                    "state_dict": net.state_dict(),
                    "loss": loss,
                    "optimizer": optimizer.state_dict(),
                    "aux_optimizer": aux_optimizer.state_dict(),
                    "lr_scheduler": lr_scheduler.state_dict(),
                },
                is_best,
                os.path.join('./experiments', args.experiment, 'checkpoints', "checkpoint_%03d.pth.tar" % (epoch + 1))
            )
            if is_best:
                logger_val.info('best checkpoint saved.')

</code></pre>