Illegal memory access when trying to clear cache

jasperhyp: <p>Hi everyone,  I encountered an strange error today…  A <code>RuntimeError: CUDA error: an illegal memory access was encountered</code> pops up at <code>torch.cuda.empty_cache()</code>.</p>
<p>Even more peculiarly, this issue comes out at the 39th epoch of a training session…  How could that be?</p>
<p>Info:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "build_model_and_train.py", line 206, in &lt;module&gt;
    train_loss, train_acc = train()
  File "build_model_and_train.py", line 105, in train
    torch.cuda.empty_cache()
  File "/public/workspace/z/miniconda3/envs/ST-Torch/lib/python3.7/site-packages/torch/cuda/memory.py", line 35, in empty_cache
    torch._C._cuda_emptyCache()
</code></pre>
<p>Code snippet:</p>
<pre><code class="lang-auto">def train():  
    model.train()
    loss_all=0
    correct = 0
    for i, data in enumerate(train_loader, 0):
        torch.cuda.empty_cache()   # This is where the issue lies!
        data = data.to(device)
        optimizer.zero_grad()
        output = model(data.x, data.batch)
        torch.cuda.empty_cache()  # This is not where the issue lies.
        label = data.y.to(device)
        loss = loss_func(output, label)
        loss.backward()
        loss_all += loss.item()
        
        output = output.detach().cpu().numpy().squeeze()
        label = label.detach().cpu().numpy().squeeze()        
        correct += (abs(output-label)&lt;0.5).sum()
        
        optimizer.step()
        
    return loss_all / len(train_dataset), correct / len(train_dataset)

## Start training
# Omitted dataset generation, DataLoader, and initial setup
os.environ["CUDA_VISIBLE_DEVICES"] = '2'
......
device = torch.device('cuda')
edge_index = edge_index.to(device)
model = GCN().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay = l2_reg)
loss_func = torch.nn.BCELoss()  # binary cross-entropy
train_loader = DataLoader(train_dataset, batch_size=batch_size, drop_last = True)
val_loader = DataLoader(val_dataset, batch_size=batch_size)
for epoch in range(num_epochs):
    gc.collect()
    torch.cuda.empty_cache()  # Yes I put this everywhere because I am suffering from OOMs...
    train_loss, train_acc = train()</code></pre>

ptrblck: <p>CUDA operations are executed asynchronously, so the illegal memory access error would most likely happen before you are calling <code>empty_cache()</code>. To isolate it, you could run the script via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and check the stacktrace, which should point to the failing operation. Also, in case you are using an older PyTorch release, I would recommend to update it to the latest stable or nightly release.</p>

jasperhyp: <p>Thanks for your reply!  I am using the latest stable version actually.  Will check the failing operations.</p>

jinluyang: <p>Dude, have you solved the problem ? I have the same problem and I am 90% sure that this error is caused by torch.cuda.empty_cache().  I call empty_cache every 1000 epochs, and every time the error comes out, its stack has empty_cache.  My pytorch version is 1.11, my cuda toolkit is 11.3, driver is 470.</p>

jinluyang: <p>And I have also tried using CUDA_LAUNCH_BLOCKING=1,  the first thread of mine that raised that error has also torch.cuda.empty_cache() in its back trace.  The error occurs ramdomly. I am using tensorrt in my threads, so probably they manage GPU memory differently than pytorch and may have some contradicts? Seems difficult to debug…</p>

ptrblck: <p>I doubt <code>empty_cache()</code> is causing the illegal memory access, but most likely an operation used before this call as already described. You could try to narrow it down further via e.g. <code>cuda-gdb</code> or <code>compute-sanitizer</code>.</p>