Why testing memory costs similar with training phrase

Liang: <p>When I  train CNN on my dataset, it takes about 7 GB GPU memory in (epoch 0, training) phrase. However, it takes 11 GB since (epoch0, testing) phrase, and it keeps 11 GB from then on. It seems that some memory are not released when switching between training and testing.</p>
<p>Due to GPU limits, can I release unused memory to allow larger batch_size?</p>
<p>Here is my training process (from official example):</p>
<pre><code class="lang-auto">for epoch in range(args.start_epoch, args.epochs):
        if args.distributed:
            train_sampler.set_epoch(epoch)
        adjust_learning_rate(optimizer, epoch)

        # train for one epoch
        train(train_loader, model, criterion, optimizer, epoch)

        # evaluate on validation set
        prec1 = validate(val_loader, model, criterion)

        # remember best prec@1 and save checkpoint
        is_best = False #prec1 &gt; best_prec1
        best_prec1 = max(prec1, best_prec1)
        if epoch % args.save_freq == 0:
            save_checkpoint({
                'epoch': epoch + 1,
                'arch': args.arch,
                'state_dict': model.state_dict(),
                'best_prec1': best_prec1,
                'optimizer' : optimizer.state_dict(),
                'loss_acc1':plot_statistic,
            }, is_best)
</code></pre>

Liang: <p>I guess it’s pytorch’s problem. The version I used is 0.4.0 at present. When I downgrade it to 0.3.0.post4, the training memory usage is about 7 GB, and the testing usage is about 2 GB. Anyone who knows the difference between these two versions?</p>
<p>I replace <code>volatile=True</code> with <code>with torch.no_grad():</code>, and <code>loss.data[0]</code> with <code>loss.item()</code>. But testing still costs similar memory.</p>

ptrblck: <p>Are you running out of memory? Maybe the memory is just cached and looks like it’s used in <code>nvidia-smi</code>.<br>
You can find more information about the memory management <a href="https://pytorch.org/docs/stable/notes/cuda.html#memory-management">here</a>.<br>
If you want to release the cached memory to the OS, you could call <code>torch.cuda.empty_cache()</code>.</p>

54c41629a13193bc3c1f: <p>Have you solved this problem? I have been confused with this, too.</p>

lxtGH: <p><a class="mention" href="/u/liang">@Liang</a> <a class="mention" href="/u/ptrblck">@ptrblck</a>  Hi，I met the same problem,  Have you solved this ??</p>

ptrblck: <p>Do you run out of memory or just see a high memory usage in <code>nvidia-smi</code>?<br>
If so, it might be just the cache.</p>

lxtGH: <p>A high memory usage and it will run out of memory. but it seems in torch-0.4.1 it doesn’t happen</p>

solsol: <p>I’m facing such this problem. for testing it run out of memory.</p>
<p>Any suggestion?</p>

Liang: <p>Problem solved. I migrated from 0.3.0 to 0.4.0, so I add  <code>with torch.no_grad():</code> myself (in a wrong place). That is like:</p>
<pre><code class="lang-auto">def validate(val_loader, model, criterion):
    batch_time = AverageMeter()
    losses = AverageMeter()
    top1 = AverageMeter()
    top5 = AverageMeter()

    # switch to evaluate mode
    model.eval()

    end = time.time()
    for i, (inputs, target) in enumerate(val_loader):

        with torch.no_grad():
            input_var = torch.autograd.Variable(inputs.cuda())
            target_var = torch.autograd.Variable(target.cuda())

        # compute output
        output = model(input_var)
        loss = criterion[0](output, target_var)
        ...
</code></pre>
<p>Recently, I read the example imagenet classification code of pytorch0.4.0, and find that it puts <code>torch.no_grad()</code> in a different place. That is like:</p>
<pre><code class="lang-auto">def validate(val_loader, model, criterion):
    batch_time = AverageMeter()
    losses = AverageMeter()
    top1 = AverageMeter()
    top5 = AverageMeter()

    # switch to evaluate mode
    model.eval()
    
    with torch.no_grad():
        end = time.time()
        for i, (inputs, target) in enumerate(val_loader):
    
            input_var = torch.autograd.Variable(inputs.cuda())
            target_var = torch.autograd.Variable(target.cuda())
    
            # compute output
            output = model(input_var)
            loss = criterion[0](output, target_var)
</code></pre>
<p>IT WORKS ! Although I don’t know why.</p>
<p>I think this problem might easily come to PyTorch old-users, especially those migrate from 0.3.0 to the above.<a class="mention" href="/u/ptrblck">@ptrblck</a> @ <a href="https://discuss.pytorch.org/u/54c41629a13193bc3c1f">54c41629a13193bc3c1f</a> <a class="mention" href="/u/lxtgh">@lxtGH</a> <a class="mention" href="/u/solsol">@solsol</a></p>