Why anomalib PyTorch almost maxed out RAM but underutilize GPU

jasonrichdarmawan: <p>My goal is to train anomalib model (CFlow) <a href="https://github.com/openvinotoolkit/anomalib/tree/main/anomalib" rel="noopener nofollow ugc">anomalib/anomalib at main · openvinotoolkit/anomalib (github.com)</a>. The problem is that it took a very long time just to pass 1 epoch.</p>
<p>I checked the <code>htop</code> and <code>nvidia-smi -l</code>.</p>
<p>The problem is anomalib PyTorch underutilize GPU (2.8 GB out of 15.3 GB) while almost maxed out RAM (18.6 GB out of 29.4 GB).</p>
<p>My expectation was that it only use little RAM and maxed out GPU.</p>
<p>So, I tried to use YOLOv4 as comparison while identical data set. YOLOv4 almost maxed GPU (9.5 GB out of 15.3 GB) and can sit comfortably with 3.75 GB of RAM.</p>
<p><strong>The screenshot for anomalib PyTorch</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27229c9db47045107d4d6c25bcec48660bcba2cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/27229c9db47045107d4d6c25bcec48660bcba2cb" title="Screen Shot 2023-02-16 at 00.46.29"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27229c9db47045107d4d6c25bcec48660bcba2cb_2_690x347.png" alt="Screen Shot 2023-02-16 at 00.46.29" data-base62-sha1="5AcM9OAz7qwCM56Hcs3TJ1SMtsf" width="690" height="347" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27229c9db47045107d4d6c25bcec48660bcba2cb_2_690x347.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27229c9db47045107d4d6c25bcec48660bcba2cb_2_1035x520.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27229c9db47045107d4d6c25bcec48660bcba2cb_2_1380x694.png 2x" data-dominant-color="323633"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-02-16 at 00.46.29</span><span class="informations">1920×966 412 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jasonrichdarmawan: <p><strong>The screenshot for YOLOv4</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d65aa03be94f069d1310d8ac7085d5df13332a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d65aa03be94f069d1310d8ac7085d5df13332a6" title="Screen Shot 2023-02-16 at 00.46.52"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d65aa03be94f069d1310d8ac7085d5df13332a6_2_690x345.png" alt="Screen Shot 2023-02-16 at 00.46.52" data-base62-sha1="fBLReEvvDYFNdDhMgP4UKykHx42" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d65aa03be94f069d1310d8ac7085d5df13332a6_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d65aa03be94f069d1310d8ac7085d5df13332a6_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d65aa03be94f069d1310d8ac7085d5df13332a6_2_1380x690.png 2x" data-dominant-color="323534"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-02-16 at 00.46.52</span><span class="informations">1920×962 395 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>