Regular increase in RAM over a period of Time

Abhi_rathi: <p>Hello,</p>
<p>I am training CNN model using pytorch on GPU. At initial stage the RAM usage is 25% but during training the memory consumption get increas slowly. After 12 hours 90 % of memory get consumed and OOM error get encounter. I check the memory usage it shows that 80% of memory get consumed by Page Pool and Non-Page pool.<br>
My PC have only one GPU of 16GB. And other configurations are as follows:</p>
<ol>
<li>Batch size: 256</li>
<li>RAM : 32 GB</li>
<li>GPU: 16GB</li>
<li>processor: i7 11th gen</li>
</ol>
<p>Pytorch config :</p>
<p>Number of workers: 4<br>
Training device : Cuda<br>
Number of iterations per epoch : 12000(approx)</p>

bfss: <p>Do you use no_grad() when you evaluate your dataset?</p>

ptrblck: <p>Are you running out of host RAM or GPU memory?</p>

Abhi_rathi: <p>I am running out of RAM</p>

Abhi_rathi: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79413a8033e21dd3e4b526c92ffed66c81fb8ce7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/79413a8033e21dd3e4b526c92ffed66c81fb8ce7" title="IMG_20230630_143924"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79413a8033e21dd3e4b526c92ffed66c81fb8ce7_2_690x413.jpeg" alt="IMG_20230630_143924" data-base62-sha1="hiFwzz2yJ0p7oFZC2qWUOFxSwjd" width="690" height="413" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79413a8033e21dd3e4b526c92ffed66c81fb8ce7_2_690x413.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79413a8033e21dd3e4b526c92ffed66c81fb8ce7_2_1035x619.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79413a8033e21dd3e4b526c92ffed66c81fb8ce7_2_1380x826.jpeg 2x" data-dominant-color="424340"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20230630_143924</span><span class="informations">1920×1151 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The training code is in the image file.</p>

bfss: <p><a class="mention" href="/u/abhi_rathi">@Abhi_rathi</a></p>
<p>emmm… I mean, when your train is over, after call model.eval(), you should use torch.no_grad() to avoid store grad during eval or inference. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"></p>

Abhi_rathi: <p><a class="mention" href="/u/bfss">@bfss</a><br>
Again I am facing the same issue. The increase in RAM consumption is during the training period. As the the iterations in a single epoch increase the page pool and non page pool memory is getting increase over the time. At initial stage the memory occupied by these pool are few MBs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df582eabf145289be1d195ef72de09be9e8b6244.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/df582eabf145289be1d195ef72de09be9e8b6244" title="16881283794384784742370395743504"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df582eabf145289be1d195ef72de09be9e8b6244_2_666x500.jpeg" alt="16881283794384784742370395743504" data-base62-sha1="vRNp9hsY0YxZa5sfS0hAJzh9R3e" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df582eabf145289be1d195ef72de09be9e8b6244_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df582eabf145289be1d195ef72de09be9e8b6244_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df582eabf145289be1d195ef72de09be9e8b6244_2_1332x1000.jpeg 2x" data-dominant-color="9F9B9C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">16881283794384784742370395743504</span><span class="informations">1920×1440 251 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

bfss: <p><a class="mention" href="/u/abhi_rathi">@Abhi_rathi</a></p>
<p>Sorry, I have no idea about the memory usage. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"></p>
<p>Generally speaking pytorch will use GPU memory.</p>