Profiling memory usage and training performance

jcbrouwer: <p>Hello, I’m working on analyzing the bottlenecks in some training code. It’s a fairly complicated task: StyleGAN2-ADA training with distributed data-parallel training and quite a few other bells and whistles (<a href="https://github.com/JCBrouwer/maua-stylegan2/blob/master/train.py" rel="noopener nofollow ugc">the training code can be found here</a>).</p>
<p>I’m wondering what the best way to approach profiling it is. I’ve noticed that in some cases, as training progresses, more memory is allocated. I don’t believe it’s a memory leak as sometimes things get deallocated, and I’ve never had it run out of memory even when running for more than a week.</p>
<p>I wasn’t too worried about it, but recently I’ve noticed that some runs don’t have these weird allocations and that those run tend to have a lower average tick duration (see graphs below). I have been tweaking some hyperparameter settings between runs, but I don’t understand how those hyperparameters could be affecting the amount of allocated memory (they’re just reweighting loss terms).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e882b615486435b11114d7c506270082a277c9db.png" data-download-href="https://discuss.pytorch.org/uploads/default/e882b615486435b11114d7c506270082a277c9db" title="mem_alloc"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e882b615486435b11114d7c506270082a277c9db_2_690x375.png" alt="mem_alloc" data-base62-sha1="xaSOSTpySlHbcAU6AojiVOQKTG3" width="690" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e882b615486435b11114d7c506270082a277c9db_2_690x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e882b615486435b11114d7c506270082a277c9db_2_1035x562.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e882b615486435b11114d7c506270082a277c9db_2_1380x750.png 2x" data-dominant-color="FBFAF8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mem_alloc</span><span class="informations">1597×870 146 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b7b1fa5d9928174b88a58dbe0dc04ba9ef19710.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b7b1fa5d9928174b88a58dbe0dc04ba9ef19710" title="tick_length"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b7b1fa5d9928174b88a58dbe0dc04ba9ef19710_2_690x373.png" alt="tick_length" data-base62-sha1="1Dz2Gxf1tsXf7vzg21AMUutTc6k" width="690" height="373" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b7b1fa5d9928174b88a58dbe0dc04ba9ef19710_2_690x373.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b7b1fa5d9928174b88a58dbe0dc04ba9ef19710_2_1035x559.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b7b1fa5d9928174b88a58dbe0dc04ba9ef19710_2_1380x746.png 2x" data-dominant-color="FBFAF7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tick_length</span><span class="informations">1595×863 140 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To try and understand what’s going on, first I added some torch.cuda.nvtx.ranges to the training code and profiled it with nsys (NVIDIA Nsight’s profiling CLI). I’m having a hard time interpreting the results and I don’t think it’s profiled the memory as I’m getting the error: <code>unrecognised option '--cuda-memory-usage=true'</code> despite the option being listed in <a href="https://docs.nvidia.com/nsight-systems/UserGuide/index.html#cli-profile-command-switch-options" rel="noopener nofollow ugc">the documentation</a>.</p>
<p>(Here’s the trace for reference, maybe I’m just dumb and reading over useful information)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9877e6614707d0f61ce9e6357cfc53f92b8af53.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9877e6614707d0f61ce9e6357cfc53f92b8af53" title="nsight_trace"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9877e6614707d0f61ce9e6357cfc53f92b8af53_2_690x386.png" alt="nsight_trace" data-base62-sha1="obIZaanxyVbGU8xoc0DrSZBmlkD" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9877e6614707d0f61ce9e6357cfc53f92b8af53_2_690x386.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9877e6614707d0f61ce9e6357cfc53f92b8af53_2_1035x579.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9877e6614707d0f61ce9e6357cfc53f92b8af53_2_1380x772.png 2x" data-dominant-color="D0DED3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nsight_trace</span><span class="informations">1668×934 177 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve also tried using the autograd profiler, but am also having trouble interpreting the results to identify my problem. Is there an easy way to store the profiling object and analyze it later? Just printing out the tables and averages doesn’t seem to give me the allocation information over time as I want and the exported chrome trace doesn’t seem to have any memory information (despite profiling with use_cuda=True, record_shapes=True, profile_memory=True, and with_stack=True).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44ae9be7d0b09871114e22c22445e423f227675f.png" data-download-href="https://discuss.pytorch.org/uploads/default/44ae9be7d0b09871114e22c22445e423f227675f" title="cuda_mem_table"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44ae9be7d0b09871114e22c22445e423f227675f_2_690x194.png" alt="cuda_mem_table" data-base62-sha1="9NAz5W4brOkmXJEWYa9zizcyUBV" width="690" height="194" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44ae9be7d0b09871114e22c22445e423f227675f_2_690x194.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44ae9be7d0b09871114e22c22445e423f227675f_2_1035x291.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44ae9be7d0b09871114e22c22445e423f227675f_2_1380x388.png 2x" data-dominant-color="070707"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cuda_mem_table</span><span class="informations">3840×1080 312 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66fe48755cc35204ff96aa75f8463e81600ee462.png" data-download-href="https://discuss.pytorch.org/uploads/default/66fe48755cc35204ff96aa75f8463e81600ee462" title="autograd_profile_trace"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66fe48755cc35204ff96aa75f8463e81600ee462_2_690x169.png" alt="autograd_profile_trace" data-base62-sha1="eH7v3pFzi0JPhuFPGNo5pb5uXzY" width="690" height="169" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66fe48755cc35204ff96aa75f8463e81600ee462_2_690x169.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66fe48755cc35204ff96aa75f8463e81600ee462_2_1035x253.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66fe48755cc35204ff96aa75f8463e81600ee462_2_1380x338.png 2x" data-dominant-color="DDD4EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">autograd_profile_trace</span><span class="informations">1920×471 57.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’d love to hear any advice on how best to profile this issue or if there are any resources which go a bit deeper into using nsight, nvprof, or the autograd profiler to analyze memory usage. Thanks!</p>