akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Weird cuda oom: missing memory

sbelharbi: <p>hi,<br>
any explanation to this error?<br>
it happens during validation.<br>
where did the 31gb go?</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 392.00 MiB (GPU 0; 31.75 GiB total capacity; 394.86 MiB already allocated; 53.00 MiB free; 424.00 MiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</code></p>
<p>pytorch 1.10.0 installation:<br>
<code>pip install torch==1.10.0 -f https://download.pytorch.org/whl/cu111/torch-1.10.0%2Bcu111-cp37-cp37m-linux_x86_64.whl</code></p>
<p>thanks</p>

ptrblck: <p>Could you check <code>nvidia-smi</code> and see if other processes are using the device and are thus allocating memory?</p>

sbelharbi: <p>it cant be the case unless there is an issue with slurm that failed to clear previous job, i think.<br>
the job runs on a cluster managed by slurm.<br>
when asked for a gpu, only free gpus are allocated. once allocated, only that job is allowed to run on that gpu.</p>
<p>i’ll add <code>nvidia-smi</code> to the code to run at the beginning of the code and store it. hopefully to catch this error when it happens again.<br>
this type of error does not happen frequently, but i got 4 jobs in a row failed because of this.</p>
<p>i had the same issue once. admins said that it could be a driver bug. i didnt get the follow up of their investigation.<br>
this error happens only on clusters with slurm.<br>
never happens on simple machine.</p>
<p>thanks</p>