Low-level errors when retrying training after OOMs

username: prayingforpnl
cooked: <p>I am using Fabric to run DDP on a single node. I have an OOM retry mechanism via reducing batch size that looks something like this (omitted some details):</p>
<pre data-code-wrap="python"><code class="lang-python">while True:
    fabric = Fabric(
        accelerator="cuda" if on_gpu else "cpu",
        devices=num_ddp_workers,
        strategy=DDPStrategy(
            process_group_backend="nccl" if on_gpu else "gloo",
            find_unused_parameters=optimizer.module.has_unused_params(),
            start_method="spawn",
        )
        if num_ddp_workers &gt; 1
        else "auto",
        precision="bf16-mixed",
    )
    try:
        fabric.launch(train_loop_fn)
    except (torch.OutOfMemoryError, mp.ProcessRaisedException) as e:
        if isinstance(
            e,
            mp.ProcessRaisedException,  # type:ignore
        ):
            if not ("torch.OutOfMemoryError" in str(e) or "CUBLAS_STATUS_ALLOC_FAILED" in str(e)):
                raise
        if not params.retry_ooms_using_gradient_accumulation:
            raise
        _LOGGER.warning(f"Training OOMed with {batch_size=}:")
        traceback.print_exc()
        if batch_size == 1:
            raise
        batch_size = math.ceil(batch_size / 2)
        _LOGGER.warning(f"Retrying training with {batch_size=}")
        if on_gpu:
            torch.cuda.synchronize()
            torch.cuda.empty_cache()
            torch.cuda.ipc_collect()
        if torch.distributed.is_initialized():
            torch.distributed.destroy_process_group()
        gc.collect()
</code></pre>
<p>After a couple OOMs (which always happen on the first batch) I am getting errors that look like this (they look like transient errors but this happens consistently on the first batch after 2 retries):</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /home/conda/feedstock_root/build_artifacts/libtorch_1741562950230/work/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0xae (0x78f158fe6d1e in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0xd3 (0x78f158f88f11 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x41b (0x78f15939fb8b in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libc10_cuda.so)
frame #3: &lt;unknown function&gt; + 0xf67ab4 (0x78f15a35cab4 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libtorch_cuda.so)
frame #4: &lt;unknown function&gt; + 0xf69c26 (0x78f15a35ec26 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libtorch_cuda.so)
frame #5: &lt;unknown function&gt; + 0xf6c294 (0x78f15a361294 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libtorch_cuda.so)
frame #6: &lt;unknown function&gt; + 0x4e5de1 (0x78f1bc2c3de1 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../../libtorch_python.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0xa (0x78f158fc0d1a in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libc10.so)
frame #8: &lt;unknown function&gt; + 0x7e73b8 (0x78f1bc5c53b8 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../../libtorch_python.so)
frame #9: THPVariable_subclass_dealloc(_object*) + 0x378 (0x78f1bc5c5798 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../../libtorch_python.so)
frame #10: &lt;unknown function&gt; + 0x1d4db2 (0x600f4c55cdb2 in /envy/research/69/5f111dee627277/bin/python)
frame #11: &lt;unknown function&gt; + 0x210935 (0x600f4c598935 in /envy/research/69/5f111dee627277/bin/python)
frame #12: &lt;unknown function&gt; + 0x20f1e7 (0x600f4c5971e7 in /envy/research/69/5f111dee627277/bin/python)
frame #13: &lt;unknown function&gt; + 0x2ae5dc (0x600f4c6365dc in /envy/research/69/5f111dee627277/bin/python)
frame #14: &lt;unknown function&gt; + 0x2263a5 (0x600f4c5ae3a5 in /envy/research/69/5f111dee627277/bin/python)
frame #15: &lt;unknown function&gt; + 0x2d9dfc (0x600f4c661dfc in /envy/research/69/5f111dee627277/bin/python)
frame #16: _PyEval_EvalFrameDefault + 0x78b8 (0x600f4c572ba8 in /envy/research/69/5f111dee627277/bin/python)
frame #17: _PyFunction_Vectorcall + 0x17f (0x600f4c59063f in /envy/research/69/5f111dee627277/bin/python)
frame #18: PyObject_Call + 0x134 (0x600f4c59a434 in /envy/research/69/5f111dee627277/bin/python)
frame #19: &lt;unknown function&gt; + 0x2dcb7d (0x600f4c664b7d in /envy/research/69/5f111dee627277/bin/python)
frame #20: _PyObject_MakeTpCall + 0x29b (0x600f4c55e38b in /envy/research/69/5f111dee627277/bin/python)
frame #21: _PyEval_EvalFrameDefault + 0x4acb (0x600f4c56fdbb in /envy/research/69/5f111dee627277/bin/python)
frame #22: &lt;unknown function&gt; + 0x226ca8 (0x600f4c5aeca8 in /envy/research/69/5f111dee627277/bin/python)
frame #23: &lt;unknown function&gt; + 0x1d1f14 (0x600f4c559f14 in /envy/research/69/5f111dee627277/bin/python)
frame #24: &lt;unknown function&gt; + 0x2dcabb (0x600f4c664abb in /envy/research/69/5f111dee627277/bin/python)
frame #25: _PyEval_EvalFrameDefault + 0x4acb (0x600f4c56fdbb in /envy/research/69/5f111dee627277/bin/python)
frame #26: &lt;unknown function&gt; + 0x226ca8 (0x600f4c5aeca8 in /envy/research/69/5f111dee627277/bin/python)
frame #27: _PyEval_EvalFrameDefault + 0x4acb (0x600f4c56fdbb in /envy/research/69/5f111dee627277/bin/python)
frame #28: _PyFunction_Vectorcall + 0x17f (0x600f4c59063f in /envy/research/69/5f111dee627277/bin/python)
frame #29: _PyEval_EvalFrameDefault + 0x4acb (0x600f4c56fdbb in /envy/research/69/5f111dee627277/bin/python)
frame #30: &lt;unknown function&gt; + 0x29ab0d (0x600f4c622b0d in /envy/research/69/5f111dee627277/bin/python)
frame #31: PyEval_EvalCode + 0x9f (0x600f4c62224f in /envy/research/69/5f111dee627277/bin/python)
frame #32: &lt;unknown function&gt; + 0x2b81aa (0x600f4c6401aa in /envy/research/69/5f111dee627277/bin/python)
frame #33: &lt;unknown function&gt; + 0x2b3e33 (0x600f4c63be33 in /envy/research/69/5f111dee627277/bin/python)
frame #34: PyRun_StringFlags + 0x62 (0x600f4c630852 in /envy/research/69/5f111dee627277/bin/python)
frame #35: PyRun_SimpleStringFlags + 0x3c (0x600f4c63060c in /envy/research/69/5f111dee627277/bin/python)
frame #36: Py_RunMain + 0x24c (0x600f4c64af8c in /envy/research/69/5f111dee627277/bin/python)
frame #37: Py_BytesMain + 0x37 (0x600f4c6124c7 in /envy/research/69/5f111dee627277/bin/python)
frame #38: &lt;unknown function&gt; + 0x2a1ca (0x78f1c882a1ca in /lib/x86_64-linux-gnu/libc.so.6)
frame #39: __libc_start_main + 0x8b (0x78f1c882a28b in /lib/x86_64-linux-gnu/libc.so.6)
frame #40: &lt;unknown function&gt; + 0x28a37a (0x600f4c61237a in /envy/research/69/5f111dee627277/bin/python)
</code></pre>
<p>I have also seen this one:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">[rank1]:[E908 19:10:10.067872007 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: Invalid access of peer GPU memory over nvlink or a hardware error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /home/conda/feedstock_root/build_artifacts/libtorch_1741562950230/work/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0xae (0x7a1a0b7e6d1e in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0xd3 (0x7a1a0b788f11 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x41b (0x7a1a0bb9fb8b in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x52 (0x7a1a0cd0a8d2 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xc9 (0x7a1a0cd33f29 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x666 (0x7a1a0cd36126 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x169 (0x7a1a0cd37219 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/torch/../../.././libtorch_cuda.so)
frame #7: &lt;unknown function&gt; + 0xd3e11 (0x7a1a786d5e11 in /envy/research/69/5f111dee627277/lib/python3.11/site-packages/ray/../../../libstdc++.so.6)
frame #8: &lt;unknown function&gt; + 0x9caa4 (0x7a1a7b09caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: &lt;unknown function&gt; + 0x129c3c (0x7a1a7b129c3c in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<p>These look like the distributed OOMing is resulting in some bad state (which I tried unsuccessfully to fix by adding <code>torch.cuda.synchronize()</code> and friends), but it’s weird that it’s so reproducible (always happens after 2 retries). Any ideas for what’s going on or what I can try next?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="prayingforpnl" data-post="1" data-topic="222953">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prayingforpnl/48/78177_2.png" class="avatar"> prayingforpnl:</div>
<blockquote>
<p>These look like the distributed OOMing is resulting in some bad state</p>
</blockquote>
</aside>
<p>I doubt it and guess you are running into real errors, e.g. a memory violation in a broken kernel and/or an NVLink/HW error.</p>
<p>I would recommend trying to narrow down if a kernel is indeed broken, e.g. via running your code with <code>compute-sanitizer</code> or <code>cuda-gdb</code>. Additionally, you could also run standalone <code>nccl-tests</code> to check if the NVLink error is real.</p>

username: prayingforpnl
cooked: <p>Thanks for the suggestions! Is that likely to happen even if I’m only using torch builtins?</p>

username: ptrblck
cooked: <p>I’m unsure I understand the question correctly, but PyTorch kernels can also of course have bugs.</p>