Cuda kernel 'volta_sgemm_128x32_nn' means what?

hohohihi: <p>I am studying the nvidia torch matmul function.</p>
<pre><code class="lang-auto">### variable creation
a = torch.randn(size=(1,128,3),dtype=torch.float32).to(cuda)
b = torch.randn(size=(1,3,32),dtype=torch.float32).to(cuda)

### execution
c = torch.matmul(a,b)
</code></pre>
<p>I profiled this code using pyprof and this gives me the result below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c418cc9fbf745f45cf315f16607cbb6174f48cf.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c418cc9fbf745f45cf315f16607cbb6174f48cf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c418cc9fbf745f45cf315f16607cbb6174f48cf.png" alt="image" data-base62-sha1="da8mMlEv8nNjb1aXHSgfUHsXFJZ" width="690" height="86" data-dominant-color="F4F3F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1691×211 15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I cannot understand many things in there.</p>
<ol>
<li>what is sgemm_128_32 means? I see the ‘s’ in sgemm stands for single precision and ‘gemm’ means general matrix multiplication. But i don’t know the 128_32 means. My output matrix dimension is 128 by 32. But I know that cutlass optimizes the sgemm using outer product. (i will give you the link, ref 1) Actually i cannot understand the link.</li>
</ol>
<p>(1)Does 128_32 means simply the output matrix’s dimension? (2)Is there any way how my output matrix(c, in my code) is actually calculated? (for example, there are total 128*32 threads. And each thread is responsible for one output element using inner product way)</p>
<ol start="2">
<li>Why the Grid and Block have 3 dimension each and how the grid and block is used for sgemm_128_32? Grid consists of x, y, z. And Block consists of x, y, z. (1) Why do you need 3 dimension? I see that (in the picture above) block X has 256 thread. (2) is this true? And Grid Y is 4. so this means that there is 4 blocks in Grid Y. (3) is this true?</li>
<li>By using that pyprof result, can i figure out how many SMs are used? how many warps are activated in that SM?</li>
</ol>
<p>Thank you.</p>
<p>ref 1 : <a href="https://developer.nvidia.com/blog/cutlass-linear-algebra-cuda/" rel="noopener nofollow ugc">https://developer.nvidia.com/blog/cutlass-linear-algebra-cuda/</a></p>

ptrblck: <p>As you’ve already partly exaplained:</p>
<ul>
<li>volta: GPU architecture</li>
<li>s = accumulator type: single precision in this case</li>
<li>gemm: kernel type: matrix multiplication in this case</li>
<li>128: number of elements per CTA in <code>M</code> dimension of the <code>C</code> matrix</li>
<li>32: number of elements per CTA in <code>N</code> dimension of the <code>C</code> matrix</li>
<li>nn: storage mode for <code>A</code> and <code>B</code> matrices, respectively: “normal” or “no-transpose” (column-major) in your case</li>
</ul>
<aside class="quote no-group" data-username="hohohihi" data-post="1" data-topic="112686">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/bbce88/48.png" class="avatar"> hohohihi:</div>
<blockquote>
<p>By using that pyprof result, can i figure out how many SMs are used? how many warps are activated in that SM?</p>
</blockquote>
</aside>
<p>You could probably use Nsight Compute to profile this kernel in more detail.</p>

hohohihi: <p>Thank you for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I cannot understand the meaning of " 128: number of elements per CTA in <code>M</code> dimension of the <code>C</code> matrix".</p>
<p>CTA means the thread block. The thread block(CTA) has 128 elements because of ‘M’ dimension of the C matrix.</p>
<p>Do you mean ‘128 elements’ as 128 threads in the thread block(CTA)?<br>
So are there total 128*32 threads in the CTA?</p>
<p>Thank you.</p>

Ther-nullptr: <p>BTW, where is the operator called from? CUBLAS, CUDNN, or CUTLASS?</p>