4090 25% slower than 3090 for pytorch transformers?

StevenCooko: <p>Is anyone else having severe performance issues on RTX 4090 cards with pytorch and transformers?</p>
<p>Using nvidia ncg docker images 22.11-py3 didn’t help a bit. Also tried pytorch 2.0/nightly. Nothing works. Pure cuda benchmarks shows 4090 can scale to 450w on cuda workload and perf is as advertised of almost 2x vs 3090. But, on my pytorch transformer  workload using huggingspaces translation pipeline, 3090 is consistently 25% faster.</p>
<aside class="onebox twitterstatus" data-onebox-src="https://twitter.com/StevenCooko/status/1602902518895296513">
  <header class="source">

      <a href="https://twitter.com/StevenCooko/status/1602902518895296513" target="_blank" rel="noopener nofollow ugc">twitter.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://pbs.twimg.com/profile_images/1465875287560704002/DAqDu50v_400x400.jpg" class="thumbnail onebox-avatar" width="400" height="400">
<h4><a href="https://twitter.com/StevenCooko/status/1602902518895296513" target="_blank" rel="noopener nofollow ugc">| Steven Lee ⟩ ⚛️🧸🇺🇦</a></h4>
<div class="twitter-screen-name"><a href="https://twitter.com/StevenCooko/status/1602902518895296513" target="_blank" rel="noopener nofollow ugc">@StevenCooko</a></div>

<div class="tweet">
  <span class="tweet-description">For those that use Linux pytorch 1.13 and transformers, you may want to hold off buying 4090. Depending on your pytorch workload, 4090 barely beats 3080 &amp; 25% slower than 3090. I don't think cuda 11.8 is optimized for the new cuda cores. <a target="_blank" href="https://twitter.com/StevenCooko/status/1602330371437170688" rel="noopener nofollow ugc">twitter.com/StevenCooko/st…</a></span>
    <div class="quoted">
      <a class="quoted-link" href="https://twitter.com/StevenCooko/status/1602330371437170688" rel="noopener nofollow ugc">
        <p class="quoted-title">| Steven Lee ⟩ ⚛️🧸🇺🇦 <span>@StevenCooko</span></p>
      </a>

      <div>Does anyone know if RTX 4090 operating in pcie x8 mode is gimped to 145w tdp? I am experiencing a rather perplexing bug where on AMD B550 5950x platform and 4090 via pcie x8 can only scale up to 145w making ML/compute slower than 3090 and barely beats 3080.</div>
    </div>
</div>

<div class="date">
  <a href="https://twitter.com/StevenCooko/status/1602902518895296513" class="timestamp" target="_blank" rel="noopener nofollow ugc">5:45 AM - 14 Dec 2022</a>


</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thank s a bunch.</p>

rashmi: <p>Pytorch 2.0 with cuda 11.8 worked for me -</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch-nightly -c nvidia
</code></pre>
<p>I also had similar issues with Pytorch 1.13 and I tried different cuda versions 11.6, 11.7 - these were slower than 3090.</p>

JamesDickens: <p>Any update to this? Did you try Cuda 11.8? I’m curious because I have a machine with two 4090s and I want to find the best environment.</p>

eqy: <p>CUDA 11.8 is recommended because it is the first version that includes 4090 (SM 8.9) support. If you want something more bleeding edge, CUDA 12.1 nightlies are available, e.g., via pip3 install --pre torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/nightly/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu121</a>.</p>

JamesDickens: <p>If you don’t mind elaborating, what can be expected with SM 8.9 support?</p>

eqy: <p>The RTX 4090 is compute capability 8.9, so CUDA 11.8 having SM 8.9 support means it is the earliest toolkit version that can compile code specific to it. In theory, SM 8.9 can run kernels compiled for SM 8.6, the performance could be worse as you may have already experienced.</p>

ptrblck: <p>Small addition: compiling natively for sm_89 will not improve the performance but indeed using a newer CUDA toolkit and cuDNN should boost the performance again. <a class="mention" href="/u/eqy">@eqy</a>’s recommendation is correct and you should use the PyTorch binaries shipping with 11.8+.</p>