Slow A100, cudnn problem?

username: 0x97
cooked: <p>Hi, I’ve just switched to a cluster with an A100 GPU, but I’m seeing worse performances than what I had on the previous card I was using (i.e. V100). By looking at other discussions, I believe it could be a cudnn version related issue.<br>
I’m working with an installation of pytorch in a conda environment, with the following specifications:</p>
<pre><code class="lang-auto">torch.__version__ = '1.13.1'
torch.cuda.get_device_name = 'NVIDIA A100 80GB PCIe'
torch.version.cuda = '11.6'
torch.backends.cudnn.version() = 8302
</code></pre>
<p>I’ve read online that the CUDA version should be 11.x, so there should not be any problem, since the one installed is 11.6.<br>
Are there any recommended cudnn versions (or torch versions) for working with an A100? Can the problem be solved via a conda install?</p>

username: ptrblck
cooked: <p>In case you are using the default <code>float32</code> for your model training you might consider enabling <code>TF32</code> for cuBLAS operations via: <code>torch.backends.cuda.matmul.allow_tf32 = True</code> and see if this would give you the desired speedup.<br>
Also, I would recommend updating to the latest PyTorch release with the latest CUDA runtime.</p>

username: 0x97
cooked: <p>Sorry for the late reply, but I didn’t have access to the cluster this week.<br>
After running some tests, and despite using the command you recommended, I get the following times for multiplying two 1000x1000 matrices:</p>
<p>cpu time 0.00585627555847168<br>
gpu time 0.8221950531005859</p>
<p>I have also reinstalled the new version and also installed the cudatoolkit-dev, because I saw that the nvcc command was missing, but nothing.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="0x97" data-post="3" data-topic="175308">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/0/ecc23a/48.png" class="avatar"> 0x97:</div>
<blockquote>
<p>After running some tests, and despite using the command you recommended, I get the following times for multiplying two 1000x1000 matrices:</p>
<p>cpu time 0.00585627555847168<br>
gpu time 0.8221950531005859</p>
</blockquote>
</aside>
<p>I cannot reproduce a slowdown and get:</p>
<pre data-code-wrap="python"><code class="lang-python">GPU: 23456.076499351133iters/s, 4.263287596404552e-05s/iter
CPU: 544.8127996382856iters/s, 0.0018354928530752658s/iter
</code></pre>
<p>on an A100-SXM4-40GB vs. AMD EPYC 7742 using:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import time

x = torch.rand(1000, 1000, device="cuda")

# warmup
for _ in range(10):
    y = torch.matmul(x, x)

nb_iters = 1000
torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(nb_iters):
    y = torch.matmul(x, x)
torch.cuda.synchronize()
t1 = time.perf_counter()
print("GPU: {}iters/s, {}s/iter".format(nb_iters/(t1 - t0), (t1 - t0)/nb_iters))

x = torch.randn(1000, 1000)
# warmup
for _ in range(10):
    y = torch.matmul(x, x)

t0 = time.perf_counter()
for _ in range(nb_iters):
    y = torch.matmul(x, x)
t1 = time.perf_counter()
print("CPU: {}iters/s, {}s/iter".format(nb_iters/(t1 - t0), (t1 - t0)/nb_iters))
</code></pre>
<p>EDIT:<br>
The nsys profile also shows the same corresponding runtime reported via my manual profiling approach:</p>
<pre data-code-wrap="python"><code class="lang-python"> Time (%)  Total Time (ns)  Instances  Avg (ns)  Med (ns)  Min (ns)  Max (ns)  StdDev (ns)                                                  Name                                                
 --------  ---------------  ---------  --------  --------  --------  --------  -----------  ----------------------------------------------------------------------------------------------------
    100.0         41990974       1010   41575.2   41568.0     41248     42112        113.6  void cutlass::Kernel&lt;cutlass_80_tensorop_s1688gemm_128x128_32x5_nn_align4&gt;(T1::Params)              
      0.0            10752          1   10752.0   10752.0     10752     10752          0.0  void at::native::&lt;unnamed&gt;::distribution_elementwise_grid_stride_kernel&lt;float, (int)4, void at::nat…
</code></pre>

username: 0x97
cooked: <p>Ok, just tried and got something good, but not as good as yours:</p>
<pre><code>GPU: 8068.225570173726iters/s, 0.0001239429898560047s/iter
CPU: 1316.47006870968iters/s, 0.0007596070915460586s/iter
</code></pre>
<p>(A100-80GB)</p>

username: ptrblck
cooked: <p>I guess you didn’t enable TF32 for cuBLAS operations as previously mentioned.<br>
With pure FP32 I get <code>~GPU: 5977.826972784215iters/s, 0.00016728486865758897s/iter</code> as this kernel is used:</p>
<pre><code class="lang-python"> Time (%)  Total Time (ns)  Instances  Avg (ns)  Med (ns)  Min (ns)  Max (ns)  StdDev (ns)                                                  Name                                                
 --------  ---------------  ---------  --------  --------  --------  --------  -----------  ----------------------------------------------------------------------------------------------------
    100.0        161114759       1010  159519.6  159743.0    156352    160383        835.4  ampere_sgemm_128x64_nn                                                                              
      0.0            11040          1   11040.0   11040.0     11040     11040          0.0  void at::native::&lt;unnamed&gt;::distribution_elementwise_grid_stride_kernel&lt;float, (int)4, void at::nat…
</code></pre>