Bfloat16 native support

stas: <p>I have a few questions about <code>bfloat16</code></p>
<ol>
<li>how can I tell via pytorch if the gpu it’s running on supports <code>bf16</code> natively? I tried:</li>
</ol>
<pre><code class="lang-auto">$ python -c "import torch; print(torch.tensor(1).cuda().bfloat16().type())"
torch.cuda.BFloat16Tensor
</code></pre>
<p>and it works on any card, whether it’s supported natively or not.</p>
<p>non-pytorch way will do too. I wasn’t able to find any.</p>
<ol start="2">
<li>What’s the cost/overheard - how does pytorch handle bf16 on gpus that don’t have native support for it?</li>
</ol>
<p>e.g. I’m trying to check whether rtx-3090 supports bf16 natively. The information is inconsistent - the Ampere arch supports bf16 but some comments I found suggest that the non-high end cards may have it disabled.</p>
<p>Thank you!</p>

tom: <p>The <a href="https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/ampere/pdf/NVIDIA-ampere-GA102-GPU-Architecture-Whitepaper-V1.pdf" rel="noopener nofollow ugc">GA102 whitepaper</a> seems to indicate that the RTX cards do support bf16 natively (in particular p23 where they also state that GA102 doesn’t have fp64 tensor core support in contrast to GA100).</p>
<p>So in my limited understanding there are broadly three ways how PyTorch might use the GPU capabilities:</p>
<ul>
<li>Use backend functions (like cuDNN, cuBlas) and hopefully they use all the latest and greatest.</li>
<li>When using <a href="https://docs.nvidia.com/cuda/cuda-math-api/group__CUDA__MATH__INTRINSIC__BFLOAT16.html#group__CUDA__MATH__INTRINSIC__BFLOAT16" rel="noopener nofollow ugc">intrinsics directly</a> conventional wisdom (see <a href="https://www.comp.nus.edu.sg/~wongwf/papers/HPEC2017.pdf" rel="noopener nofollow ugc">this 2017 paper discussing half vs. half2 performance</a>) seems to say that bfloat162 will offer better performance over using bfloat16 unless the compiler has learned lots of new tricks. But I am not aware if we actually use that a lot in PyTorch.</li>
</ul>
<p>Again, I’m still looking into how get most fp16 performance, so take this with a grain of salt.</p>
<p>Best regards</p>
<p>Thomas</p>

stas: <p>Thank you for the links, <a class="mention" href="/u/tom">@tom</a>, and attempting to answer my query.</p>
<p>I read that paper but also read comments on reddit where users suggested some features might have been disabled. I don’t want to propagate rumours of something I don’t really know first hand, hence trying to find a practical way to test whether a given gpu has a native support for a specific datatype.</p>

tom: <p>I think the hardware whitepaper is probably the most official documentation.<br>
If you wanted to make experiments to verify:</p>
<ul>
<li>You can look at the generated kernels to see if the expected instructions are in there as a first step,</li>
<li>you can benchmark versus expectations (either comparing to A100 relative spee-up or to some theoretical promise of relative speed-up).</li>
</ul>
<p>But both of them rely on you taking the right steps to unlock the speedups (see eg the requirements to use Tensor Cores), so it is tricky. If you are interested in ops supported by official NVIDIA libs, that might be a good way to test and here maybe (to my mind at least) <a href="https://github.com/NVIDIA/cutlass" rel="noopener nofollow ugc">CUTLASS</a> is the thing that sticks out for being open source and very much at the cutting edge.</p>
<p>Best regards</p>
<p>Thomas</p>

stas: <p>This approaches sounds doable, but complicated. I was hoping pytorch could provide users an API to query that, e.g. like it does with the current compute capacity of the gpu card, but I guess if CUDA doesn’t provide that API, then pytorch can’t either.</p>
<p>I found this compilation,</p>
<p><a href="https://machine-learning-note.readthedocs.io/en/latest/blog/cheatsheets/Nvidia_GPU.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://machine-learning-note.readthedocs.io/en/latest/blog/cheatsheets/Nvidia_GPU.html</a></p>
<p>that also includes a spreedsheet with performance numbers for the different operations. So it contains the rtx-30* series too:</p>
<aside class="onebox googledocs">
  <header class="source">
      <a href="https://docs.google.com/spreadsheets/d/1AAe7gB0vfOtU_RMjE_WNcjrSmB6_LLgUIPj9cfR3KFw/edit#gid=0" target="_blank" rel="noopener nofollow ugc">docs.google.com</a>
  </header>
  <article class="onebox-body">
    <a href="https://docs.google.com/spreadsheets/d/1AAe7gB0vfOtU_RMjE_WNcjrSmB6_LLgUIPj9cfR3KFw/edit#gid=0" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-sheets-logo"></span></a>

<h3><a href="https://docs.google.com/spreadsheets/d/1AAe7gB0vfOtU_RMjE_WNcjrSmB6_LLgUIPj9cfR3KFw/edit#gid=0" target="_blank" rel="noopener nofollow ugc">Nvidia GPU</a></h3>

<p>GPU summary table

Model,Series,Architecture,Processing Power (Gflops), underlined value is estimated from other value with some ratio; note that probably not all operations could be benefit from Tensor Core so usually boost might be around x2~3...</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>So it has TF32 numbers for Ampere cards but not bf16 yet.</p>

AlbertZeyer: <p>Probably the <a href="https://machine-learning-note.readthedocs.io/en/latest/blog/cheatsheets/Nvidia_GPU.html" rel="noopener nofollow ugc">last link</a> was updated recently. It states that the Nvidia 30* series (Ampere) does support bfloat16.</p>
<p>You can also find the same <a href="https://en.wikipedia.org/wiki/GeForce_30_series" rel="noopener nofollow ugc">information on Wikipedia</a>.</p>

stas: <p>Thanks for the update, Albert.</p>
<p>A lot of water has passed under the bridge since this discussion. We have integrated bf16 support in HF Transformers and even have benchmarks to show its performance under RTX-3090 and A100</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/huggingface/transformers/issues/14608">
  <header class="source">

      <a href="https://github.com/huggingface/transformers/issues/14608" target="_blank" rel="noopener nofollow ugc">github.com/huggingface/transformers</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/huggingface/transformers/issues/14608" target="_blank" rel="noopener nofollow ugc">[Benchmark] HF Trainer on RTX-3090</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-12-03" data-time="05:56:53" data-timezone="UTC">05:56AM - 03 Dec 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/stas00" target="_blank" rel="noopener nofollow ugc">
          <img alt="stas00" src="https://avatars.githubusercontent.com/u/10676103?v=4" class="onebox-avatar-inline" width="20" height="20">
          stas00
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          Benchmarks
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          WIP
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container"># 🖥 Benchmarking RTX-3090 `transformers` w/ HF Trainer:

We are going to use a<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden"> special benchmarking tool that will do all the work for us. https://github.com/huggingface/transformers/pull/14934

This is the index post and specific benchmarks are in their own posts below:

1. [fp16 vs bf16 vs tf32 vs fp32](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004390803)
2. [gradient accumulation steps](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)
3. [gradient checkpointing](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004422281)
4. [batch size](https://github.com/huggingface/transformers/issues/14608#issuecomment-1004392537)
5. [optimizers](https://github.com/huggingface/transformers/pull/14744#issuecomment-1002398095)

TODO:
- redo optimizers once https://github.com/huggingface/transformers/pull/14744 gets merged
- a combined benchmark with the slowest vs fastest configs
- other suggestions?</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubissue" data-onebox-src="https://github.com/huggingface/transformers/issues/15026">
  <header class="source">

      <a href="https://github.com/huggingface/transformers/issues/15026" target="_blank" rel="noopener nofollow ugc">github.com/huggingface/transformers</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/huggingface/transformers/issues/15026" target="_blank" rel="noopener nofollow ugc">[WIP] [Benchmark] HF Trainer on A100</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-01-04" data-time="05:41:33" data-timezone="UTC">05:41AM - 04 Jan 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/stas00" target="_blank" rel="noopener nofollow ugc">
          <img alt="stas00" src="https://avatars.githubusercontent.com/u/10676103?v=4" class="onebox-avatar-inline" width="20" height="20">
          stas00
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container"># 🖥 Benchmarking A100 `transformers` w/ HF Trainer:

We are going to use a spe<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">cial benchmarking tool that will do all the work for us. https://github.com/huggingface/transformers/pull/14934

This is the index post and specific benchmarks are in their own posts below:

Please ignore for now as I'm gathering results.


This is the index post and specific benchmarks are in their own posts below:

1. [fp16 vs bf16 vs tf32 vs fp32](https://github.com/huggingface/transformers/issues/15026#issuecomment-1004543189)
2. ...</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


lessw2020: <aside class="quote no-group" data-username="stas" data-post="5" data-topic="117155">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stas/48/8409_2.png" class="avatar"> stas:</div>
<blockquote>
<p>I was hoping pytorch could provide users an API to query that, e.g. like it does with the current compute capacity of the gpu card,</p>
</blockquote>
</aside>
<p>There is an api now - just found out about it today:</p>
<pre><code class="lang-auto">torch.cuda.is_bf16_supported()
</code></pre>

stas: <p>Thank you, Less</p>
<p>It was indeed added 8 months ago:</p><aside class="onebox githubcommit" data-onebox-src="https://github.com/pytorch/pytorch/commit/6324d98e9e736214ee7a161482b02aaf8cebbc9d#diff-9189d159e01ea5173c234ead87f40f4a99cb1aa01264a3259b99a162fee4248cR81-R88">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/commit/6324d98e9e736214ee7a161482b02aaf8cebbc9d#diff-9189d159e01ea5173c234ead87f40f4a99cb1aa01264a3259b99a162fee4248cR81-R88" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Commit">
    <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M10.86 7c-.45-1.72-2-3-3.86-3-1.86 0-3.41 1.28-3.86 3H0v2h3.14c.45 1.72 2 3 3.86 3 1.86 0 3.41-1.28 3.86-3H14V7h-3.14zM7 10.2c-1.22 0-2.2-.98-2.2-2.2 0-1.22.98-2.2 2.2-2.2 1.22 0 2.2.98 2.2 2.2 0 1.22-.98 2.2-2.2 2.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/commit/6324d98e9e736214ee7a161482b02aaf8cebbc9d" target="_blank" rel="noopener nofollow ugc">bf16 Error message cleanup as well as addition of is_bf16_supported (#63798)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        committed <span class="discourse-local-date" data-format="ll" data-date="2021-08-25" data-time="16:59:59" data-timezone="UTC">04:59PM - 25 Aug 21 UTC</span>
      </div>

      <div class="user">
        <a href="" rel="noopener">
          <img alt="" src="" class="onebox-avatar-inline" width="20" height="20">
          
        </a>
      </div>

      <div class="lines" title="changed 4 files with 18 additions and 10 deletions">
        <a href="https://github.com/pytorch/pytorch/commit/6324d98e9e736214ee7a161482b02aaf8cebbc9d" target="_blank" rel="noopener nofollow ugc">
          <span class="added">+18</span>
          <span class="removed">-10</span>
        </a>
      </div>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Summary:
ngimel

Pull Request resolved: https://github.com/pytorch/pytorch/pull/<span class="show-more-container"><a href="https://github.com/pytorch/pytorch/commit/6324d98e9e736214ee7a161482b02aaf8cebbc9d" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">63798

Reviewed By: heitorschueroff

Differential Revision: D30526187

Pulled By: ngimel

fbshipit-source-id: c484aec14638097c96c720095d3491249b6b2d14</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>but with it not being documented, it’s hard to tell if it’s public or not.</p>
<p>at HF Transformers meanwhile we just copied it and extended it to our needs.</p>
<p><a href="https://github.com/huggingface/transformers/blob/d91841315aab55cf1347f4eb59332858525fad0f/src/transformers/utils/import_utils.py#L275-L303" class="inline-onebox" rel="noopener nofollow ugc">transformers/import_utils.py at d91841315aab55cf1347f4eb59332858525fad0f · huggingface/transformers · GitHub</a></p>

chiragjn: <p>I know this thread is quite old, but I have a similar confusion.</p>
<p>I am able to run</p>
<pre><code class="lang-auto">model.to(torch.bfloat16)
</code></pre>
<p>on older GPUs like V100, T4 and it does not complain even though bfloat16 is only supported on Ampere and above. The memory consumption is in line with half-precision, so then what does it mean when the docs say bfloat16 is only supported on Ampere and above?</p>
<p>How does it work? Does it mean on older GPUs the kernels copy tensors to fp32 for each operation?</p>
<p>Can I then also train (albeit slowly) in bfloat16 on older GPUs?</p>

sjrhl: <p>Any update on this? I’m having a similar confusion. I seem to be able to load models from transformers using torch.bfloat16 and have them run inference with no problem on T4 GPUs. I also checked some of the underlying torch tensors and they at least report to be of dtype=torch.bfloat16.</p>

tom: <p>Even if there is no native hardware support, with recent CUDA you get “emulated” bf16 support where the internal computations are with fp32.<br>
Regarding speed, you have offsetting effects:</p>
<ul>
<li>generally, fallback and emulation is slow,</li>
<li>optimized kernels (e.g. cublas) are possibly missing,</li>
<li>you might still get an advantage from reduced memory transfer from main to local/cache memory.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

sjrhl: <p>Ahh okay, that makes a lot of sense. Thanks for the info!</p>

victorx: <p><a class="mention" href="/u/tom">@tom</a> I’m curious where you found this information. Do you mind sharing some link? I was trying to search online with "bf16 cuda emulation’, but could not find any relevant entries.</p>

Klubokok: <p>python -c “import torch; import time; x = torch.randn(4096, 4096).cuda(); x_bf16 = x.to(torch.bfloat16); start = time.time(); y_bf16 = x_bf16 @ x_bf16; bf16_time = time.time()-start; x_fp32 = x.to(torch.float32); start = time.time(); y_fp32 = x_fp32 @ x_fp32; fp32_time = time.time()-start; print(f’BF16: {bf16_time:.3f}s | FP32: {fp32_time:.3f}s | Ratio: {fp32_time/bf16_time:.1f}x’)”<br>
BF16: 0.027s | FP32: 0.055s | Ratio: 2.0x</p>
<p>1080ti pytorch2.6.0+cu124</p>