Majority of training time spent copying to mps unit

pederlh: <p>When trying to train a custom classifier (4 output features) based on the <code>torchvision.models.resnet18</code> model on an M1 MacBook Air, I notice that the majority of the time logged when profiling is spent copying the image batches (standard 224x224 3-channel ImageNet resolution) to the mps device. This happens regardless of whether the model is compiled or not. Is this expected behavior?</p>
<p>Using <code>torch.profiler</code> with both a wait+skip step, a typical cycle takes about 165 ms. The table and picture below shows how this time is distributed with a batch size of 16 images.</p>
<pre><code class="lang-plaintext">Process                                         |   Time (ms)| Cycle ratio (%)
------------------------------------------------|------------|----------------
DataLoader (read img, uint8-&gt;float32, normalize)|    24.8 ms |           15 %
Copy to GPU                                     |   105.2 ms |           64 %
Forward pass (incl. loss)                       |    16.7 ms |           10 %
Backprop (incl. use_grad)                       |    17.9 ms |           11 %
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16b8455a6075377609d05578ea662ee89f7ca291.png" data-download-href="https://discuss.pytorch.org/uploads/default/16b8455a6075377609d05578ea662ee89f7ca291" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16b8455a6075377609d05578ea662ee89f7ca291_2_690x307.png" alt="image" data-base62-sha1="3eZiMyvMlFGSCluvzeXF5ySCajv" width="690" height="307" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16b8455a6075377609d05578ea662ee89f7ca291_2_690x307.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16b8455a6075377609d05578ea662ee89f7ca291_2_1035x460.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/16b8455a6075377609d05578ea662ee89f7ca291_2_1380x614.png 2x" data-dominant-color="E0E3E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2180×972 242 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As shown, almost 2/3 of the training consists of the <code>aten__::copy_</code> operator, which I find pretty strange.</p>

marksaroufim: <p>Hard to say without seeing an exact repro but I believe this is actually expected behavior for a small model like resnet18, I would try something much larger and try a larger batch size and see if the cycle ration for copy reduces</p>

pederlh: <p>In the case of the ResNet-18 model, increasing the batch size from 16 to 128 actually increases the the copy time ratio from 64 % to 83 % (while the total time per sample per epoch goes up from 10.3 ms to 12.9 ms).</p>
<p>Trying to plug in a Visual Transformer (ViT_B_16, batch size 16) instead, the cycle ratio does indeed decrease. If the batch size is increased (to 64), the profiling tool causes the Mac to deplete the whole 16 GB of RAM, making it hard to obtain any reliable results. Although mps gives a nice speed boost compared to running on just the bare cpu, one might suspect that the MacBook Air may not be the right machine for heavy model training. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-plaintext">Process                                         |   Time (ms)| Cycle ratio (%)
------------------------------------------------|------------|----------------
DataLoader (read img, uint8-&gt;float32, normalize)|      29 ms |          &lt; 2 %
Copy to GPU                                     |     270 ms |           13 %
Forward pass (incl. loss)                       |     692 ms |           34 %
Backprop (incl. use_grad)                       |    1031 ms |           51 %
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d28bc12901d84a61d500c24652dd3398f89d15c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/d28bc12901d84a61d500c24652dd3398f89d15c4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d28bc12901d84a61d500c24652dd3398f89d15c4_2_690x343.png" alt="image" data-base62-sha1="u2zGS7mDp9c6pRQ0Tq4ih36agiU" width="690" height="343" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d28bc12901d84a61d500c24652dd3398f89d15c4_2_690x343.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d28bc12901d84a61d500c24652dd3398f89d15c4_2_1035x514.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d28bc12901d84a61d500c24652dd3398f89d15c4_2_1380x686.png 2x" data-dominant-color="DAD2D6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1983×987 166 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>