Backward time consumption linearly increases with batch_size

15271887443: <p>When I am training a simple resnet which involves convolutional network forward and backward on celebA dataset, I find out that the loss.backward() (MSE loss) whose time consumption increases linearly with batch_size. A simple profile is as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/070913c051ab1cc4581f8bad31b5f072ac1cc34f.png" data-download-href="https://discuss.pytorch.org/uploads/default/070913c051ab1cc4581f8bad31b5f072ac1cc34f" title="Screenshot from 2024-03-26 14-39-08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/070913c051ab1cc4581f8bad31b5f072ac1cc34f_2_690x121.png" alt="Screenshot from 2024-03-26 14-39-08" data-base62-sha1="10eMYMazI6DB7V5B9HF1fkKuwtp" width="690" height="121" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/070913c051ab1cc4581f8bad31b5f072ac1cc34f_2_690x121.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/070913c051ab1cc4581f8bad31b5f072ac1cc34f_2_1035x181.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/070913c051ab1cc4581f8bad31b5f072ac1cc34f_2_1380x242.png 2x" data-dominant-color="1F2020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-03-26 14-39-08</span><span class="informations">1680×296 37.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ul>
<li>Here is a part of backward profiling. The batch_size is set to 128, and GPU consumption is about 1.5GB/12GB.</li>
</ul>

15271887443: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/056e4c085acd92c1efd1000cbe5975ff1eddb3fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/056e4c085acd92c1efd1000cbe5975ff1eddb3fb" title="Screenshot from 2024-03-26 14-42-02"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/056e4c085acd92c1efd1000cbe5975ff1eddb3fb_2_690x154.png" alt="Screenshot from 2024-03-26 14-42-02" data-base62-sha1="M2HfxhnvexWiGeooz0LNAEUx63" width="690" height="154" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/056e4c085acd92c1efd1000cbe5975ff1eddb3fb_2_690x154.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/056e4c085acd92c1efd1000cbe5975ff1eddb3fb_2_1035x231.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/056e4c085acd92c1efd1000cbe5975ff1eddb3fb_2_1380x308.png 2x" data-dominant-color="1F2020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-03-26 14-42-02</span><span class="informations">1669×374 47.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ul>
<li>Here is another backward profiling. The batch_size is set to 512, and GPU ram consumption is about 4.7GB/12GB.</li>
</ul>
<p>I am sure the bottleneck is the loss.backward(). However, neither the CPU or GPU is fully used, it is weird why the backward time consumptio gets a linear increment. The parallel processing seems not working here.</p>

YH_04: <p>did you figure out the solution? im running into the same problem…</p>

15271887443: <p>Not yet <img src="https://discuss.pytorch.org/images/emoji/apple/person_shrugging.png?v=12" title=":person_shrugging:" class="emoji" alt=":person_shrugging:" loading="lazy" width="20" height="20">. I guess it is related to how CUDA do parallel computation for tensor convolution, but have no idea how to identify it.</p>