Amazing, code runs much faster when pytorch updates from 1.2 to pytorch 2.0

Zhaoyi-Yan: <p>When running my code on pytorch 1.2, it takes 305s per epoch, it speeds significantly and costs only 220s on pytorch 2.0.</p>
<p>I do not use <code>torch.compile</code> since the cuda is only 11.1 in my environment.</p>
<p>It is really a big surprise for me.</p>
<p>I wonder how it could run so fast  ?</p>

ptrblck: <aside class="quote no-group" data-username="Zhaoyi-Yan" data-post="1" data-topic="171373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhaoyi-yan/48/8480_2.png" class="avatar"> Zhaoyi-Yan:</div>
<blockquote>
<p>I do not use <code>torch.compile</code> since the cuda is only 11.1 in my environment.</p>
</blockquote>
</aside>
<p>Are you building PyTorch from source using your locally installed 11.1 CUDA toolkit?<br>
If not, then note that the PyTorch binaries will ship with their own CUDA runtime (as well as cuDNN, cuBLAS, NCCL etc.) which you can specify in the <a href="https://pytorch.org/get-started/locally/">install instructions</a>.<br>
Since you are not using <code>torch.compile</code> do you see the speedup in the plain eager mode or are you <code>torch.jit.script</code>ing the model?</p>

Zhaoyi-Yan: <p>The version of my local CUDA toolkit is 11.1.</p>
<p>I install pt2.0 with <code>conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch-nightly -c nvidia</code>.   And <code>print(torch.version.cuda)</code> gives the result <code>11.7</code>.</p>
<p>However, when I use <code>torch.compile(model)</code>, it errors with the info <code>torch._dynamo_exc.BackendCompilerFailed: compile_fn raised RuntimeError: Triton requires CUDA 11.4+</code>.</p>
<p>I do not use <code>torch.jit.scripting</code>.</p>

ptrblck: <p><code>torch.compile</code> with the <code>pytorch-triton</code> backend uses your locally installed CUDA toolkit to compile the kernels during its code-generation step (in particular the locally installed <code>ptxas</code>) and depends on CUDA&gt;=11.4. In future releases the needed <code>ptxas</code> should be packaged into the binaries so that your local CUDA toolkit would not be needed anymore.<br>
Until then you would need to update your 11.1 CUDA toolkit to 11.4 or newer.</p>
<p>Since you are seeing a general speedup without scripting the model or using <code>torch.compile</code> I would guess some CUDA math libraries could perform better using the PyTorch binaries with CUDA 11.7 such as cuBLAS or cuDNN for your GPU (assuming your previous release used an older CUDA runtime).</p>

Zhaoyi-Yan: <p>Hello, thank you for your prompt response. I have one more question: will the released pt 2.0 include ‘pytorch-trition’?</p>

ptrblck: <p>Yes, <code>pytorch-triton</code> is already installed as a dependency in the nightly releases.<br>
We also recently discussed how to package <code>ptxas</code>, so the dependency on your locally installed CUDA toolkit should also disappear.</p>

Zhaoyi-Yan: <p>I installed pytorch nightly (2.0.0.dev20230213+cu117).   It errors:<br>
name, asm, shared_mem = _triton.code_gen.compile_ttir(backend, module, device, num_warps, num_stages, extern_libs, cc)<br>
RuntimeError: Triton requires CUDA 11.4+</p>

ptrblck: <p>That’s expected for now and you would need to install a local CUDA toolkit as explained before.</p>