Memory Leak - DataLoader

username: kubslo
cooked: <p>Hi,<br>
I’ve got a problem with memory leak during training. I suspect the main cause of that problem is Dataset created by using torchvision.datasets.ImageFolder, (when I used torchvision.datasets.CIFAR10 instead of my dataset the problem does not occur) . I’ve tried to find a solution on similar topics.<br>
Here is my dataset<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/32a7ce2e959b84b04a993bd86adcf2ca45b2e6e3.png" data-download-href="https://discuss.pytorch.org/uploads/default/32a7ce2e959b84b04a993bd86adcf2ca45b2e6e3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/32a7ce2e959b84b04a993bd86adcf2ca45b2e6e3.png" alt="image" data-base62-sha1="7e7ou6Wf7sqDw5dibe9rAw5g22f" width="690" height="179" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">691×180 5.88 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and dataloader:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92eec291ce51b441a262cdbc88cb6a95b584dee8.png" data-download-href="https://discuss.pytorch.org/uploads/default/92eec291ce51b441a262cdbc88cb6a95b584dee8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92eec291ce51b441a262cdbc88cb6a95b584dee8.png" alt="image" data-base62-sha1="kXPfByFbO73MxUXHRaPa6zOVAso" width="690" height="150" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">988×216 9.44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you have any idea what could be a reason of a memory leak ?</p>
<p>Thanks in an advance</p>

username: albanD
cooked: <p>Hi,</p>
<p>Could you give more details on how you measure the memory?<br>
Also do you have a small code sample I could run locally to reproduce this?</p>

username: kubslo
cooked: <p>To be honest I checked it on task manager. On linux I get runtime error during training (I suspected that it was related with small ram capacity of my gpu, but when I use CPU my RAM fills up during  a training )</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f55cdc9f2d6ce7577542e6d7daedaba3a839b784.png" data-download-href="https://discuss.pytorch.org/uploads/default/f55cdc9f2d6ce7577542e6d7daedaba3a839b784" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f55cdc9f2d6ce7577542e6d7daedaba3a839b784.png" alt="image" data-base62-sha1="z0zW138RLw1e4scPdmQ8UVevYIA" width="690" height="129" data-dominant-color="F5F1F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">871×164 3.94 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Code :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90f234662198de24377e02e722b019e79eb261a2.png" data-download-href="https://discuss.pytorch.org/uploads/default/90f234662198de24377e02e722b019e79eb261a2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90f234662198de24377e02e722b019e79eb261a2.png" alt="image" data-base62-sha1="kGfFXWNasBg7bXYyU1H14sP3vp0" width="485" height="500" data-dominant-color="EEEFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">555×572 19.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9caea96c736d61c9b547d9f5039039f2d4caff6b.png" alt="image" data-base62-sha1="mm4HbHXAuU9bMr0dyroTnvanc83" width="574" height="409"></p>

username: albanD
cooked: <p>In your screenshot, it seems like there is no leak, just high usage.<br>
If you use many workers or if your dataset has larger images or larger batchsize, then the memory needed to load the data will be bigger.</p>

username: kubslo
cooked: <p>I see now, that was stupid question from me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Thanks for help</p>