RuntimeError: CUDA out of memory saving model predictions

username: Niki
cooked: <p>Hi,</p>
<p>I am looking for saving model predictions and later using them for calculating accuracy. The dataset has 20000 samples, I was trying to use</p>
<pre><code class="lang-auto">prediction_list.append(prediction)
</code></pre>
<p>And then using <code>torch.save</code>  to save them. But this gives this error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 31.72 GiB total capacity; 30.73 GiB already allocated; 6.12 MiB free; 19.83 MiB cached)
</code></pre>
<p>Is there any methods to save them, not facing this error?</p>

username: ptrblck
cooked: <p>If you are not running the code in a <code>with torch.no_grad()</code> block, you will store the whole computation graph in the list for each prediction.<br>
Use <code>prediction_list.append(prediction.detach())</code> to store the tensor only (and use the <code>no_grad()</code> guard to save more memory, if you don’t need to calculate the gradients later).</p>

username: Niki
cooked: <p>Thank you very much, <a class="mention" href="/u/ptrblck">@ptrblck</a>. It worked.</p>

username: Niki
cooked: <p>I think I am missing something in the saving and loading files by torch. I saved prediction, labels, and later I called torch.load to retrieve these values,</p>
<pre><code class="lang-auto">prediction_load = torch.load(ME_DIR + 'prediction' + '.torch')         
labels_load = torch.load(ME_DIR + 'labels' + '.torch')

for i in range(len(prediction_load)):
    _, predicted = torch.max(prediction_load[i].data, 0)
    correct += predicted.eq(labels_load[i].data).cpu().sum().  ###error in this line
 
    total += labels_load[i].size(0)


accuracy = 100.*correct.float()/total
</code></pre>
<p>But it gives this error in the line for calculating correct :</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (10) must match the size of tensor b (2) at non-singleton dimension 1.
</code></pre>

username: ptrblck
cooked: <p><code>torch.max(tensor, 0)</code> will apply the max operation in the batch dimension.<br>
If your output has the shape <code>[batch_size, nb_classes]</code>, you should use <code>dim=1</code>.</p>

username: Niki
cooked: <p>Ok. Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

username: Zaid_Rjoub
cooked: <pre><code class="lang-auto">for epoch in range(150):
    train, valid = random_split(train_nn, [850000, 50000])
    
    
    trainloader = DataLoader(train, batch_size=BATCH_SIZE)
    validloader = DataLoader(valid, batch_size=BATCH_SIZE) 
    
    for i in range(len(train_nn)//BATCH_SIZE):
            train_data = next(iter(trainloader))
            validation_data = next(iter(validloader))
            
            train_x = train_data[:,0:-1]
            train_y = train_data[:,-1]
            
            validation_x = validation_data[:,0:-1]
            validation_y = validation_data[:,-1]


            y_hat = model(train_x).reshape((BATCH_SIZE,))

            cost = LOSS(y_hat, train_y)

            cost.backward()

            OPTIMIZER.step()
            OPTIMIZER.zero_grad()
            with torch.no_grad():
                roc_auc_values_validation.append(roc_auc_score(validation_y.cpu(), model(validation_x).detach().cpu().numpy()))

                roc_auc_values_train.append(roc_auc_score(train_y.cpu(), model(train_x).detach().cpu().numpy()))

    with torch.no_grad():
            y_sub.append(model(test_nn).detach().cpu().numpy())
        
    torch.cuda.empty_cache()
    print(f'Epoch: {epoch+1}:',f"cost: {cost}")
    print(f"auc under the ROC curve for the validation set is: {roc_auc_values_validation[-1]}")
    print(f"auc under the ROC curve for the training set is: {roc_auc_values_train[-1]}")
            
</code></pre>
<p>I have tried what you said but nothing is working</p>

username: ptrblck
cooked: <p>I assume you are observing an increasing memory usage and tried to fix this by detaching tensors?<br>
If so, could you post a minimal, executable code snippet reproducing the issue, please?</p>