How do I know that Tensor Cores used in PyTorch (for FP16, bFloat16, INT8)?

username: Michelvl92
cooked: <p>From PyTorch documentation it is very to know if a model is using Tensor Cores or not (for FP16, bFloat16, INT8)?.</p>
<p>What I know so far:</p>
<ul>
<li>FP32 will not run on Tensor Cores, since it is not supported</li>
<li>Enabling TF32 for PyTorch will run your model in TF32 on Tensor Cores</li>
<li>Running (Automatic) Mixed Precision will run on Tensor Cores</li>
<li><em><strong>Converting a model to FP16, bfloat16 it is unclear if it is/will using Tensor Cores or not! According to Pytorch forums:</strong></em></li>
</ul>
<blockquote>
<p>PyTorch is using Tensor Cores on volta GPU as long as your inputs are in fp16 and the dimensions of your gemms/convolutions satisfy conditions for using Tensor Cores (basically, gemm dimensions are multiple of 8, or, for convolutions, batch size and input and output number of channels is multiple of 8). For Ampere and newer, fp16, bf16 should use Tensor Cores for common ops and fp32 for convs (via TF32). So how do I know if CUDA cores are not used?</p>
</blockquote>
<ul>
<li>What about INT8?</li>
</ul>

username: rinkujadhav2013
cooked: <p>This blog post from NVIDIA explains how to use Tensor cores. This should answer your question partially as to when will tensor cores be used:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/">
  <header class="source">
      <img src="https://developer-blogs.nvidia.com/wp-content/themes/nvidia/dist/images/favicon_300a1064.ico" class="site-icon" width="48" height="48">

      <a href="https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/" target="_blank" rel="noopener nofollow ugc" title="09:29AM - 17 October 2017">NVIDIA Technical Blog – 17 Oct 17</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:334/500;"><img src="https://developer-blogs.nvidia.com/wp-content/uploads/2017/12/tensor_cube_white-624x934.png" class="thumbnail" width="334" height="500"></div>

<h3><a href="https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/" target="_blank" rel="noopener nofollow ugc">Programming Tensor Cores in CUDA 9 | NVIDIA Technical Blog</a></h3>

  <p>A defining feature of the new Volta GPU Architecture is its Tensor Cores, which give the Tesla V100 accelerator a peak throughput 12 times the 32-bit floating point throughput of the previous…</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
If you wanna ensure that tensor cores were used then you can use NVPROF as mentioned here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://forums.developer.nvidia.com/t/how-to-confirm-whether-tensor-core-is-working-or-not/70263/6">
  <header class="source">
      <img src="https://global.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://forums.developer.nvidia.com/t/how-to-confirm-whether-tensor-core-is-working-or-not/70263/6" target="_blank" rel="noopener nofollow ugc" title="06:27PM - 13 February 2019">NVIDIA Developer Forums – 13 Feb 19</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:150/80;"><img src="https://global.discourse-cdn.com/nvidia/original/3X/e/4/e41d27491ee72562cf68c340597ca95f587879bd.png" class="thumbnail" width="150" height="80"></div>

<h3><a href="https://forums.developer.nvidia.com/t/how-to-confirm-whether-tensor-core-is-working-or-not/70263/6" target="_blank" rel="noopener nofollow ugc">How to confirm whether Tensor Core is working or not.</a></h3>

  <p>Thanks Kaka, I understand now.  You can use the nvprof CUDA profiler tool to capture the Tensor Core usage while your application runs.  nvprof supports two metrics for Tensor Core utilization:    tensor_precision_fu_utilization:  The utilization...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>