Question about bmm cuda runtime

username: wukon007
cooked: <p>Here, I use torch profile on my model. and this is a part of results. Now I have a doubt: the bmmbackward’(aten::bmm)s input size is A(32, 1310763), and B(1310763, 64) but the uderlying cuda op(cutlass::Kernel&lt;cutlass_80_tensorop_s1688gemm_64x64_32x4_tn_align1)‘s block size is(64,1,1) ,grid is (1,1,2). I think the number of input’ parameter should match the block * grid, but now there is a big gap between them.   Meanwhile, it‘s time-consuming, this one op takes about 30ms.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5b21811ed786530644ba6c5778ff5bee79014ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5b21811ed786530644ba6c5778ff5bee79014ac" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5b21811ed786530644ba6c5778ff5bee79014ac_2_690x448.png" alt="image" data-base62-sha1="z3wxMLP7MQrMbDlHX7k7csAIJpa" width="690" height="448" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5b21811ed786530644ba6c5778ff5bee79014ac_2_690x448.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5b21811ed786530644ba6c5778ff5bee79014ac_2_1035x672.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5b21811ed786530644ba6c5778ff5bee79014ac.png 2x" data-dominant-color="E9EAEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1038×674 50.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61c27fd529b77ac0db938b37c4824fe90d53564c.png" data-download-href="https://discuss.pytorch.org/uploads/default/61c27fd529b77ac0db938b37c4824fe90d53564c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61c27fd529b77ac0db938b37c4824fe90d53564c_2_690x448.png" alt="image" data-base62-sha1="dWP1HWCC9VARHitJAyBdYLwXmd6" width="690" height="448" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61c27fd529b77ac0db938b37c4824fe90d53564c_2_690x448.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61c27fd529b77ac0db938b37c4824fe90d53564c_2_1035x672.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/1/61c27fd529b77ac0db938b37c4824fe90d53564c.png 2x" data-dominant-color="F2F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1038×674 52.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: eqy
cooked: <p>I am not sure what you mean by “number of input parameter,” and how that should match kernel launch bounds without knowing the parallelization strategy used by the kernel. As a trivial example, if you include e.g., 1310763 as a factor in your parameter count, it wouldn’t make sense to increase the launch bounds according to this value as this dimension is reduced as part of the matmul.</p>
<p>If you think the achieved FLOPs is far below what is expected given your GPU, a script that reproduces this kernel call in PyTorch would be useful for debugging.</p>