Optimal matrix sizes for utilizing tensor cores when running Transformer-decoder type models?

username: vgoklani
cooked: <p>What are the optimal matrix sizes for utilizing tensor cores when running <code>transformer-decoder</code> type models (e.g. Mistral7B)?  Should both the sequence length and batch-size be a multiple of 8, 64 or 256? The vocab-size is 32000 which already divides 256.</p>
<p>Is there a reference for this setting? I’m looking specifically for the Ada Lovelace chipset (we are running on the A6000 Ada Lovelace cards) but I also need this value for the H100 (Hopper chipset).</p>
<p>Enabling tensor-cores in pytorch requires this to be set:</p>
<pre><code>torch.set_float32_matmul_precision("high")
</code></pre>
<p>but I can’t tell if the tensor-cores are actually being used. Is there a way of checking - some sort of debugging/logging that we can expose?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Are all of these points generated by ChatGPT or did you verify them? E.g. I don’t understand how blocking launches would help here and <code>nvprof</code> was deprecated a long time ago.</p>

username: vgoklani
cooked: <p>More info here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc">
  <header class="source">
      <img src="https://docs.nvidia.com/favicon-32x32.png" class="site-icon" width="32" height="32">

      <a href="https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc" target="_blank" rel="noopener nofollow ugc">NVIDIA Docs</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc" target="_blank" rel="noopener nofollow ugc">Matrix Multiplication Background User's Guide - NVIDIA Docs</a></h3>

  <p>GPUs accelerate machine learning operations by performing calculations in parallel. Many operations, especially those representable as matrix multipliers will see good acceleration right out of the box. Even better performance can be achieved by...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code>Tensor Cores can be used for... 

cuBLAS version ≥ 11.0
cuDNN version ≥ 7.6.3

INT8    Always but most efficient with multiples of 16; on A100, multiples of 128.
FP16    Always but most efficient with multiples of 8; on A100, multiples of 64.
TF32    Always but most efficient with multiples of 4; on A100, multiples of 32.
FP64    Always but most efficient with multiples of 2; on A100, multiples of 16.
</code></pre>
<p>Presumably bfloat16 would match up with the FP16 recommendations here (?)</p>
<p>I wish there was more info on the recommendations; where do the Ada Lovelace chips fall into this?</p>
<p>I probably just need to figure out how to use the PyTorch profiler and figure this out via trial and error…</p>

username: vgoklani
cooked: <p>And of course there is this famous example</p>
<p><a href="https://twitter.com/karpathy/status/1621578354024677377" class="onebox" target="_blank" rel="noopener nofollow ugc">https://twitter.com/karpathy/status/1621578354024677377</a></p>