Why kernels different streams can't in parallel

username: shadowshadow
cooked: <p>I have a image batch that each of them has a different shape. So I can’t batch them and have to use a for-loop instead.<br>
my code is like below:</p>
<pre data-code-wrap="python"><code class="lang-python">for image in image_batch:
    out = encoder(image)   
</code></pre>
<p>I find this encode process only used 1/4 of my GPU memory. Since I can’t increase batch_size, so I want to use multi-stream to dispatch my kernels and hope they can run in parallel. Unfortunately, the total time when dispatching samples to two stream(7.3s) is just.slightly faster than dispatching them to one stream(7.75s)</p>
<p>Here is my timeline when running in one stream:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff20ec5488f7548c57f81427e2ca0e83062f5ee9.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff20ec5488f7548c57f81427e2ca0e83062f5ee9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff20ec5488f7548c57f81427e2ca0e83062f5ee9_2_639x500.png" alt="image" data-base62-sha1="AoYi8rHJU63xWADyiCwWgrXwXb3" width="639" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff20ec5488f7548c57f81427e2ca0e83062f5ee9_2_639x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff20ec5488f7548c57f81427e2ca0e83062f5ee9_2_958x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff20ec5488f7548c57f81427e2ca0e83062f5ee9_2_1278x1000.png 2x" data-dominant-color="DCE5E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1300×1017 129 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Here is timeline when running in two stream<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e80bb442472756d3e57b8e3e8cd3725e8ca1d1b.png" data-download-href="https://discuss.pytorch.org/uploads/default/0e80bb442472756d3e57b8e3e8cd3725e8ca1d1b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e80bb442472756d3e57b8e3e8cd3725e8ca1d1b_2_355x500.png" alt="image" data-base62-sha1="24iupKSaRAnQBE9ESwMQBNSi7p1" width="355" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e80bb442472756d3e57b8e3e8cd3725e8ca1d1b_2_355x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/e/0e80bb442472756d3e57b8e3e8cd3725e8ca1d1b_2_532x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e80bb442472756d3e57b8e3e8cd3725e8ca1d1b.png 2x" data-dominant-color="DAE3E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">679×954 92.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It seems two stream version overlaps very well, but if I zoom in, I find only one kernel is running a lot time, but there are also some overlap cases</p>
<p>for example, element-wise kernels seems unable to run in parallel<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c7971ab74df7e56c5d658fd1bb80e80337f5995.png" data-download-href="https://discuss.pytorch.org/uploads/default/1c7971ab74df7e56c5d658fd1bb80e80337f5995" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c7971ab74df7e56c5d658fd1bb80e80337f5995_2_690x76.png" alt="image" data-base62-sha1="43TyPwspwHVbYvD1XB26sa8IGBn" width="690" height="76" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c7971ab74df7e56c5d658fd1bb80e80337f5995_2_690x76.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c7971ab74df7e56c5d658fd1bb80e80337f5995_2_1035x114.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c7971ab74df7e56c5d658fd1bb80e80337f5995_2_1380x152.png 2x" data-dominant-color="CFDADA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2115×235 47.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I check their kernel config, find they often have many blocks, like grid[172800,1,1], block[128,1,1]<br>
blocks per SM 1309, warps per SM 5236.</p>
<p>there are also some cases that kernels can overlap.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/474e1ed083edf6cf0ead31ba6a456add9284ee5f.png" data-download-href="https://discuss.pytorch.org/uploads/default/474e1ed083edf6cf0ead31ba6a456add9284ee5f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/474e1ed083edf6cf0ead31ba6a456add9284ee5f_2_690x126.png" alt="image" data-base62-sha1="aaNgTznkDPKQ9MgaqYjeDKcipAj" width="690" height="126" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/474e1ed083edf6cf0ead31ba6a456add9284ee5f_2_690x126.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/474e1ed083edf6cf0ead31ba6a456add9284ee5f_2_1035x189.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/474e1ed083edf6cf0ead31ba6a456add9284ee5f_2_1380x252.png 2x" data-dominant-color="CFDCDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1425×261 51.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So，my questions are</p>
<ul>
<li>
<p><strong>So does cuda doesn’t run kernel in parallel because one kernel is able to saturate the whole gpu? But I think element-wise kernel are memory bound. Are there any way for me to make each block compute more elements and save some SMs for another kernel?</strong></p>
</li>
<li>
<p><strong>How to check how many SMs a kernel use(e.g. nsys or ncu). And when gpu will run kernel in differnent streams sequentially or in parallel</strong></p>
</li>
<li>
<p><strong>Are there other best practices for my problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></strong></p>
</li>
</ul>
<p>I found some posts with the same issue as mine, but they can’t fully solve my problem.</p>
<ol>
<li><a href="https://github.com/pytorch/pytorch/issues/59692" class="inline-onebox" rel="noopener nofollow ugc">cuda streams run sequentially, expected to run parallel · Issue #59692 · pytorch/pytorch · GitHub</a></li>
<li><a href="https://forums.developer.nvidia.com/t/why-kernel-executions-in-different-streams-are-not-parallel/73850/5" class="inline-onebox" rel="noopener nofollow ugc">Why kernel executions in different streams are not parallel? - #5 by bit_drinkwater - CUDA Programming and Performance - NVIDIA Developer Forums</a></li>
</ol>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/cuda-stream-not-working/155749/4">This post</a> including the linked talk helps.</p>