Using debugpy with DDP results in driver leaking GPU memory

BSchneider: <p>When debugging DDP code with debugpy, if I use the debugger to stop the process during training the driver (sporactically) does not free the GPU memory (see Nvidia-smi output below on gpus 6,7). I have not had issue with the driver failing to clean up after the debugger in the past. I am launching the processes using torchrun in vscode. <strong>Is it expected behaviour that the driver might leak memory if <code>dist.destroy_process_group()</code> is not reached / the process is interrupted?</strong> The problem is quite painful because I have to reset the server to free the memory (nvidia-smi –reset-gpu is not supported on SXM A100s). Sadly, I have not managed to create a more minimal repro script than my full training <a href="https://github.com/Toolbox-HQ/pokeagent-speedrun/blob/main/train/train_idm.py" rel="noopener nofollow ugc">loop</a>. <strong>What are the suggested workflows to prevent the driver from leaking memory when using a debugger?</strong></p>
<h2><a name="p-475918-vscode-launch-script-1" class="anchor" href="#p-475918-vscode-launch-script-1"></a>.vscode launch script</h2>
<pre><code class="lang-auto">    {
        "name": "Debug Training",
        "type": "debugpy",
        "request": "launch",
        "program": "./.venv/bin/torchrun",
        "args": [
            "--nnodes",
            "1",
            "--nproc-per-node",
            "2",
            "--master_addr",
            "localhost",
            "--master_port",
            "6575",
            "./train/train_idm.py",
            "--config",
            "./config/idm.yaml"
        ],
        "console": "integratedTerminal",
        "justMyCode": false,
        "env": {
            "CUDA_VISIBLE_DEVICES": "6,7",
            "WANDB_MODE": "offline",
            "LD_LIBRARY_PATH": "/scratch/b3schnei/anaconda3/envs/ffmpeg/lib:$LD_LIBRARY_PATH",
            "TMPDIR": "/scratch/b3schnei/tmp"
        }
    }
</code></pre>
<h2><a name="p-475918-h-2" class="anchor" href="#p-475918-h-2"></a></h2>
<h2><a name="p-475918-nvidia-smi-output-3" class="anchor" href="#p-475918-nvidia-smi-output-3"></a>Nvidia-smi Output</h2>
<p>GPUs 6,7 have remaining memory with no associated processes.</p>
<pre><code class="lang-auto">Thu Oct  2 21:34:46 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.07              Driver Version: 580.82.07      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  |   00000000:07:00.0 Off |                    0 |
| N/A   31C    P0             60W /  400W |       4MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-80GB          On  |   00000000:0B:00.0 Off |                    0 |
| N/A   30C    P0             58W /  400W |       4MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-80GB          On  |   00000000:48:00.0 Off |                    0 |
| N/A   28C    P0             59W /  400W |       4MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-80GB          On  |   00000000:4C:00.0 Off |                    0 |
| N/A   30C    P0             58W /  400W |       4MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A100-SXM4-80GB          On  |   00000000:88:00.0 Off |                    0 |
| N/A   61C    P0            339W /  400W |   60985MiB /  81920MiB |     89%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A100-SXM4-80GB          On  |   00000000:8B:00.0 Off |                    0 |
| N/A   79C    P0            432W /  400W |   60985MiB /  81920MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A100-SXM4-80GB          On  |   00000000:C8:00.0 Off |                    0 |
| N/A   31C    P0             62W /  400W |   50517MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A100-SXM4-80GB          On  |   00000000:CB:00.0 Off |                    0 |
| N/A   32C    P0             66W /  400W |   50499MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    4   N/A  N/A         2718452      C   python                                60976MiB |
|    5   N/A  N/A         2724137      C   python                                60976MiB |
+-----------------------------------------------------------------------------------------+

</code></pre>

ptrblck: <p>The CUDA Runtime will clear all memory on process exit. Did you try to find the (dead) processes and kill them?</p>