`sm_89` not listed in the `torch.cuda.get_arch_list()`

vgoklani: <p>We’re getting the docker image from here: <code>nvcr.io/nvidia/pytorch:24.12-py3</code></p>
<p>and when we get the arch list:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.cuda.get_arch_list()

# ['sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90', 'compute_90']

torch.version.cuda

# '12.6'
</code></pre>
<p><code>sm_89</code> is not listed. We are running 4x NVIDIA RTX 6000 Ada cards.</p>
<p>We also tried installing the latest version of torch via pip, but still don’t see <code>sm_89</code> listed. Do we have to do something special to enable it?</p>
<p>Here is the CUDA information if needed:</p>
<pre data-code-wrap="bash"><code class="lang-bash">root@~ $ nvidia-smi --version
NVIDIA-SMI version  : 565.57.01
NVML version        : 565.57
DRIVER version      : 565.57.01
CUDA Version        : 12.7
</code></pre>
<p>Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="vgoklani" data-post="1" data-topic="215827">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vgoklani/48/1971_2.png" class="avatar"> vgoklani:</div>
<blockquote>
<p>Do we have to do something special to enable it?</p>
</blockquote>
</aside>
<p>No, you don’t need to build for <code>sm_89</code>as it’s binary compatible with <code>sm_86/sm_80</code>. Your device is thus supported in all of our builds.</p>

vgoklani: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> so there shouldn’t be any issues running FP8 specific kernels in pytorch? Those aren’t part of sm_86…</p>
<p>More specifically I’m unable to utilize row-wise scaling in FP8</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/ao/issues/1057#issuecomment-2610796124">
  <header class="source">

      <a href="https://github.com/pytorch/ao/issues/1057#issuecomment-2610796124" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/ao</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue" data-github-private-repo="false">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/ao/issues/1057#issuecomment-2610796124" target="_blank" rel="noopener nofollow ugc">How to use float8 with SM89 hardware - i.e. NVIDIA A6000 ADA?</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2024-10-11" data-time="14:40:38" data-timezone="UTC">02:40PM - 11 Oct 24 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2025-01-22" data-time="17:53:38" data-timezone="UTC">05:53PM - 22 Jan 25 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/vgoklani" target="_blank" rel="noopener nofollow ugc">
          <img alt="" src="https://avatars.githubusercontent.com/u/180487?v=4" class="onebox-avatar-inline" width="20" height="20">
          vgoklani
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          question
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          float8
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">I am running torchao: 0.5 and torch: '2.5.0a0+b465a5843b.nv24.09' on an NVIDIA A<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">6000 ADA card (sm89) which supports FP8.

I ran the generate.py code from the benchmark:

    python generate.py --checkpoint_path $CHECKPOINT_PATH --compile --compile_prefill --write_result /root/benchmark_results__baseline.txt

&gt; Average tokens/sec: 57.01
&gt; Average Bandwidth: 855.74 GB/s
&gt; Peak Memory Usage: 16.19 GB
&gt; Model Size: 15.01 GB

&gt; 20241011143042, tok/s= 57.01, mem/s= 855.74 GB/s, peak_mem=16.19 GB, model_size=15.01 GB quant: None, mod: Meta-Llama-3-8B, kv_quant: False, compile: True, compile_prefill: True, dtype: torch.bfloat16, device: cuda repro: python generate.py --checkpoint_path /models/Meta-Llama-3-8B/consolidated.00.pth --device cuda --precision torch.bfloat16 --compile --compile_prefill --num_samples 5 --max_new_tokens 200 --top_k 200 --temperature 0.8

    python generate.py --checkpoint_path $CHECKPOINT_PATH --compile --compile_prefill --quantization float8wo --write_result /root/benchmark_results__float8wo.txt`

&gt; Average tokens/sec: 57.00
&gt; Average Bandwidth: 855.62 GB/s
&gt; Peak Memory Usage: 16.19 GB
&gt; Model Size: 15.01 GB

&gt; 20241011143316, tok/s= 57.00, mem/s= 855.62 GB/s, peak_mem=16.19 GB, model_size=15.01 GB quant: float8wo, mod: Meta-Llama-3-8B, kv_quant: False, compile: True, compile_prefill: True, dtype: torch.bfloat16, device: cuda repro: python generate.py
--quantization float8wo --checkpoint_path /models/Meta-Llama-3-8B/consolidated.00.pth --device cuda --precision torch.bfloat16 --compile --compile_prefill --num_samples 5 --max_new_tokens 200 --top_k 200 --temperature 0.8

The `float8wo` flag does not appear to be doing anything. Am I missing a step? Thanks!</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The response from the thread was that sm_89 should be listed in the arch_list.</p>
<p>Thanks!</p>

ptrblck: <p>I don’t know if the current row-wise scaling kernel implementation is compatible with <code>sm_89</code> and we are explicitly building them for <code>sm_90a</code> (arch-conditional) <a href="https://github.com/pytorch/pytorch/blob/0f051eaf66b5045fe0131e75966be2cb7645687b/cmake/Codegen.cmake#L80-L92">here</a>.</p>

vgoklani: <p>There was a recent PR - <a href="https://github.com/pytorch/pytorch/pull/144348" class="inline-onebox" rel="noopener nofollow ugc">Add SM89 support for f8f8bf16_rowwise() by alexsamardzic · Pull Request #144348 · pytorch/pytorch · GitHub</a></p>
<p>which “introduced support for <code>_scaled_mm</code> operator with FP8 inputs on SM89 architecture. The support is based on CUTLASS library, that is header-only C++ library, so this new functionality gets fully built along with PyTorch build; however, it will get built only in case the build includes SM89 among targets.”</p>
<p>Unfortunately this requires that <code>sm_89</code> is on the list of targets.</p>
<p>I just opened a ticket <a href="https://github.com/pytorch/pytorch/issues/145632" class="inline-onebox" rel="noopener nofollow ugc">Support `sm_89` in Stable/Nightly/Docker Images · Issue #145632 · pytorch/pytorch · GitHub</a></p>
<p>Does this make sense?</p>

ptrblck: <p>No, we should not add <code>sm_89</code> directly as it will waste space with no benefits besides FP8 support. Instead, we should add <code>sm_89</code> to the one file supporting FP8 as explained in my comment on GitHub.</p>