Find bottleneck in DL training

username: shrutee
cooked: <p>Hi, I am training a deep learning model that performs transformations on the fly. However, training is very slow and I am trying to figure out the <strong>bottleneck</strong> - Disk I/O, CPU or/and GPU.</p>
<p>I am running a <code>16CPU</code>, <code>P100</code> GPU, <code>200gb</code> persistent disk on GCP.</p>
<p>This is the output of <code>htop</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1800c42e19ae0635bb3aa652e3f61fd3c11d158.png" data-download-href="https://discuss.pytorch.org/uploads/default/b1800c42e19ae0635bb3aa652e3f61fd3c11d158" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1800c42e19ae0635bb3aa652e3f61fd3c11d158_2_690x387.png" alt="1" data-base62-sha1="pkeQVjDR4bXUXxZJols3nibYWM0" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1800c42e19ae0635bb3aa652e3f61fd3c11d158_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b1800c42e19ae0635bb3aa652e3f61fd3c11d158_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b1800c42e19ae0635bb3aa652e3f61fd3c11d158.png 2x" data-dominant-color="3F2A37"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1366×768 271 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is the output of <code>iotop</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/362d4de8ca6b2445c27a95e53d12df177b8d7622.png" data-download-href="https://discuss.pytorch.org/uploads/default/362d4de8ca6b2445c27a95e53d12df177b8d7622" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/362d4de8ca6b2445c27a95e53d12df177b8d7622_2_690x387.png" alt="2" data-base62-sha1="7JgQxKMTBbhpHryxFSRebt4o1XQ" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/362d4de8ca6b2445c27a95e53d12df177b8d7622_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/362d4de8ca6b2445c27a95e53d12df177b8d7622_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/6/362d4de8ca6b2445c27a95e53d12df177b8d7622.png 2x" data-dominant-color="442539"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1366×768 182 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone help me understand what the bottleneck is?<br>
From the htop output - CPU usage seems ~100%, and from the <code>iotop</code> output I/O doesn’t seem to be the bottleneck.</p>
<p>I ran <code>cprofile</code> on a smaller dataset. <code>Dataloader</code> seems to be taking up 80% of the time.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f36c2758dc83aaccf16674972e7cff468c668a3.png" data-download-href="https://discuss.pytorch.org/uploads/default/1f36c2758dc83aaccf16674972e7cff468c668a3" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f36c2758dc83aaccf16674972e7cff468c668a3_2_690x387.png" alt="3" data-base62-sha1="4s87EJjr1w5uDecNdu0B8bsuPon" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f36c2758dc83aaccf16674972e7cff468c668a3_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f36c2758dc83aaccf16674972e7cff468c668a3_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f36c2758dc83aaccf16674972e7cff468c668a3.png 2x" data-dominant-color="C3BFC0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">1366×768 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, what exactly is the bottleneck here?</p>

username: ptrblck
cooked: <p>I would recommend to time the <code>DataLoader</code> using the <code>AverageMeter</code> from the <a href="https://github.com/pytorch/examples/blob/5551061414d3bcf202de520d20e8163f58eb664a/imagenet/main.py#L280">ImageNet example</a>.<br>
I’m not sure, if <code>cprofile</code> is giving the right profiles as CUDA operations are asynchronous and this it might accumulate the time on the blocking operation on the CPU side.</p>