How to apply backward by summing multiple losses?

ysleer: <p>How to apply backward by summing multiple losses?</p>
<p>Hi.</p>
<p>I am training a model using multiple data loaders.</p>
<p>First, the example code is as follows:</p>
<pre><code class="lang-auto">loss_list = list()
    for epoch in range(cfg.start_epoch, cfg.max_epoch):
        batch_time = AverageMeter()
        data_time = AverageMeter()
        losses = AverageMeter()
        losses_exist = AverageMeter()

        print("Train Epoch: {}".format(epoch))
        model.train()

        end = time.time()
        for i in range(len(train_loader_list)):
            train_loader = train_loader_list[i]
            for batch_idx, sample in enumerate(train_loader):
                data_time.update(time.time() - end)

                optimizer.zero_grad()

                input = sample['image'].cuda()
                target = sample['label'].cuda()
                target_exist = sample['exist'].cuda()

                output, output_exist = model(input)  # output_mid
                loss_seg = criterion(torch.nn.functional.log_softmax(output, dim=1), target)
                loss_exist = criterion_exist(output_exist, target_exist)
                loss = loss_seg + loss_exist * 0.1

                losses.update(loss.data.item(), input.size(0))
                losses_exist.update(loss_exist.item(), input.size(0))

                # optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                # scheduler.step()

                # measure elapsed time
                batch_time.update(time.time() - end)
                end = time.time()

                if (batch_idx + 1) % cfg.print_freq == 0:
                    print((
                        'dataloader: {0} input_shape: {1} Epoch: [{2}][{3}/{4}], lr: {lr:.5f}\t' 'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t' 'Data {data_time.val:.3f} ({data_time.avg:.3f})\t' 'Loss {loss.val:.4f} ({loss.avg:.4f})\t' 'Loss_exist {loss_exist.val:.4f} ({loss_exist.avg:.4f})\t'.format(
                            i, input.shape[2:], epoch, batch_idx, len(train_loader), batch_time=batch_time,
                            data_time=data_time,
                            loss=losses,
                            loss_exist=losses_exist, lr=optimizer.param_groups[-1]['lr'])))
                    batch_time.reset()
                    data_time.reset()
                    losses.reset()

                if batch_idx == len(train_loader) - 1:
                    loss_list.append(loss)

        for loss_sample in loss_list:
            loss += loss_sample

        loss_tot = loss / len(loss_list)

        loss_tot.backward()
        optimizer.step()
</code></pre>
<p>As a result, several losses are inserted into the loss_list and finally the average of the losses is calculated.</p>
<p>Then, I want to backward the average loss to the model.</p>
<p>However, the following error occurs during the backward process.<br>
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.</p>
<p>Is there a way to backward the average loss again?</p>
<p>Thank you.</p>

ptrblck: <p>In side the <code>train_loader</code> loop you are already calling <code>loss.backward()</code>, which will calculate the gradients and will free the intermediate activations, which are needed for a second backward pass using this loss.<br>
Later in the same loop you are appending <code>loss</code> to <code>loss_list</code> and try to call <code>backward</code> again on the sum of all losses, which will raise this issue. Besides the already freed intermediate activations, you are also already updating the parameters via <code>optimizer.step()</code> inside the <code>train_loader</code> loop, which would then raise another error, since the forward activations (kept via <code>retain_graph=True</code>) would be stale.</p>

ysleer: <p>Thank you for your polite response.</p>
<p>If so, how can I fix it?</p>

ptrblck: <p>It depends a bit on your use case and what you are trying to achieve?<br>
I.e. since you are already updating the model in the inner loop, the outer <code>backward</code> call should be removed.<br>
Alternatively, you could also remove the inner <code>backward</code> and <code>step</code> call and update the model in the outer loop.</p>