High RAM utilization during training

username: Muhammad_Faizan
cooked: <p>Hi, we are using 1M images to train and validate. We are using yolov5m, and while training system RAM is increasing and reaches to it’s maximum limit. While it’s approaching to it’s maximum limit, the GPU utilization plummet to a very low value and it fluctuates constantly. When we kill some processes run by python by resuming training, it creates some space in the RAM; then, again GPU utilization gets stable. Please see screenshots for further insight. My question is, what to do to keep RAM in a safe limit, so GPU utilization does not fall down to a low value? Is there any code or change in code that keeps it in an optimal range? if there are any other steps please let me know since we have to resume training again and again to make some space in the RAM by killing processes run by python. After it starts using swap memory, during the swap memory use, it’s gpu usage is still low.</p>

username: ptrblck
cooked: <p>Based on your description it seems your script is increasing the memory usage by e.g. storing tensors (or references) which might still be attached to a computation graph.<br>
You didn’t post any code snippets, so I don’t know what exactly is going on in your script, but try to narrow down which part of the code is increasing the memory by adding debug print statements.</p>

username: Muhammad_Faizan
cooked: <p>I am using yolov5 repository. <a href="https://github.com/ultralytics/yolov5/blob/master/train.py" class="inline-onebox" rel="noopener nofollow ugc">yolov5/train.py at master · ultralytics/yolov5 · GitHub</a></p>

username: DFK1991
cooked: <p>Hi both Muhammad and Piotr,</p>
<p>I experience exactly the same issue while using non-custom training (.train() method of YOLO model object), I turned off image caching in RAM, put nbs=1 (nominal batch size, values &gt; than batch size might induce additional gradient accumulation I suggest), but no success. There is a thread on github <a href="https://github.com/ultralytics/ultralytics/issues/3745" class="inline-onebox" rel="noopener nofollow ugc">increase memory while training yolov8l · Issue #3745 · ultralytics/ultralytics · GitHub</a> , but they suggest that it is possible only to completely switch off gradient accumulation when launching training through terminal.</p>
<p>This issue is annoying as it limits the number of epochs you could set during a single run as otherwise you will eventually hit the VRAM Limit. I do not provide the code because I am not doing anything custom - only generating a proper YOLO dataset and yaml file, all the rest is built-in ultralytics methods. However, below I attach the output, and as you could see, the GPU memory usage goes up with every epoch. In there I put nbs=1, and cache images on SSD.</p>
<p>Would be happy to discuss how to solve that :^). I found the “solution” by training models in stages - 20-30 epochs, then 30 more, and so on. Ultralytics version is 8.3.185, not the latest but close to it, the latest sometimes crashes during training on my laptop for some reason.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/227597dfb3d976d238017482c733fea084108ff9.png" data-download-href="https://discuss.pytorch.org/uploads/default/227597dfb3d976d238017482c733fea084108ff9" title="Снимок экрана 2025-09-09 в 15.03.12"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/227597dfb3d976d238017482c733fea084108ff9_2_690x431.png" alt="Снимок экрана 2025-09-09 в 15.03.12" data-base62-sha1="4UQaW1V3QF0EpX53aZmf2LxIALT" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/227597dfb3d976d238017482c733fea084108ff9_2_690x431.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/227597dfb3d976d238017482c733fea084108ff9_2_1035x646.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/227597dfb3d976d238017482c733fea084108ff9_2_1380x862.png 2x" data-dominant-color="F1EDEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Снимок экрана 2025-09-09 в 15.03.12</span><span class="informations">1816×1136 275 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>As previously described your script is most likely storing references disallowing PyTorch to delete these tensors potentially including the computation graph with all intermediate activations.</p>
<p>You could check if any losses or outputs are stored in e.g. a <code>list</code> without detaching them as this is the common issue.</p>

username: DFK1991
cooked: <p>Thank you for a rapid response!</p>
<p>I am not using anything custom in training, it is a built-in ultralytics’ training method. All I do is I set up the config with training parameters, prepare the dataset (proper folders and annotations), all the rest is out of the box. I am discussing it right now on ultralytics forum as some other users do also experience the same problem - slight memory leakage from epoch to epoch, which limits the total number of epochs for training as you eventually hit the maximum.</p>
<p>The solution was to use gc.collect(), emptying memory cache after each epoch, and not to use pin_memory=True in their built-in dataloader as it somehow prevents memory release in certain set-ups.</p>
<p>For NVIDIA GPU users it seemed to help, but I am training a lightweight model right on the Macbook with Apple Silicon, and the problem persists on this machine no matter what I do.</p>
<p>I tried to ask LLM’s how to track what’s happening with the memory in details, but all I was able to see so far is that there are gazillions of lists being created along the way ( not surprisingly :^) ) and nothing too informative. I do not think it is really pytorch-related, but rather some small issue with ultralytics’ training scripts that somehow fails on Macs specifically, because as I mentioned above - when working NVIDIA’s GPU’s it is possible to fix with custom callback functions and altering some parameters.</p>
<p>If I will find the way to fix it for Macs - I will come back and post a comment.</p>