When matmul will be dispatched to cutlass gemm kernel?

acmore: <p>When I use nsys to profile the matmul op, the kernel is <code>cutlass::Kernel2&lt;cutlass_80_tensorop_s1688gemm_256x128_32x3_nn_align4&gt;(T1::Params)</code>, the python code is just a simple call:</p>
<pre><code class="lang-auto">import torch

torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

m = 512
n = 1536
k = 21504

a = torch.randn(m, k, dtype=torch.float32, device='cuda')
b = torch.randn(k, n, dtype=torch.float32, device='cuda')

with torch.no_grad():
    c = torch.matmul(a, b)
</code></pre>
<p>But when I check the code, seems like if no compile, no other optimization, it will call cublas gemm</p>

ptrblck: <p>cuBLAS itself can call into CUTLASS kernels.</p>

xuehuanran: <p>hello, I have a similar question, the gemm kernel may be dispatched to cutlass kernel (cutlass::Kernel2&lt;cutlass_75_tensorop_bf16_s1688gemm_bf16_256x128_tn_align1&gt;) or cublas kernel (sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize128x128x64_warpgroupsize1x1x1_execute_segment_k_off_kernel__5x_cublas)</p>
<p>Can I use some args, like environment_variable to control the dispatched kernel of cublas?</p>

dddong: <p>What’s the mechanisim behind it? I use <code>cuobjdump -sass</code> to disassemble libcublasLt.so, I observe that <code>void cutlass::Kernel&lt;cutlass_80_wmma_tensorop_f16_s161616gemm_f16_32x32_32x1_nn_align2&gt;(T1::Params)</code> only has few lines sass code:</p>
<pre><code class="lang-auto">        /*0000*/                   MOV R1, c[0x0][0x28] ;  /* 0x00000a0000017a02 */
                                                           /* 0x000fc40000000f00 */
        /*0010*/                   EXIT ;                  /* 0x000000000000794d */
                                                           /* 0x000fea0003800000 */
        /*0020*/                   BRA 0x20;               /* 0xfffffff000007947 */
                                                           /* 0x000fc0000383ffff */
        /*0030*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*0040*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*0050*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*0060*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*0070*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*0080*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*0090*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*00a0*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*00b0*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*00c0*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*00d0*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*00e0*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
        /*00f0*/                   NOP;                    /* 0x0000000000007918 */
                                                           /* 0x000fc00000000000 */
</code></pre>
<p>It looks abnormal. I use cuda-gdb to debug this, make my program break on kernel launch, and use <code>x/100i </code> to print the sass code, the code looks normal and correct.<br>
Is there JIT compiling in cublas?</p>