torch.cudnn.benchmark=True picks slower algorithm

nkem: <p>I am getting worse profiling results with <code>cudnn.benchmark = True</code> on my toy example (see below), and was wondering if this is user error or a bug / incompatible build?<br>
My understanding is that, esp. in combination with <code>cudnn.benchmark_limit = 0</code>, cudnn should brute-force through absolutely all available choices and pick the fastest one?</p>
<p><strong>Other slowdown-related topics (but no match)</strong></p>
<ul>
<li><a href="https://discuss.pytorch.org/t/cudnn-benchmark-slowing-execution-down/31762" class="inline-onebox">Cudnn.benchmark slowing execution down</a> - OP included warmup time in measurement; second user possibly ran into repeated benchmarks due to varying input shapes</li>
</ul>
<p><strong>System information:</strong></p>
<ul>
<li>GPU: 1x NVIDIA L4</li>
<li>Driver Version: 560.35.03</li>
<li>Ubuntu 24.04 (Docker)</li>
<li>Python 3.11.11</li>
<li>Pytorch 2.5.1 (built from source):
<ul>
<li>CUDA 12.6</li>
<li>cuDNN 9.6.0.74-1</li>
</ul>
</li>
</ul>
<p><strong>Pytorch details:</strong></p>
<pre><code class="lang-auto">print(torch.__config__.show())
PyTorch built with:
  - GCC 13.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2025.0.1-Product Build 20241031 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.6
  - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_89,code=sm_89
  - CuDNN 90.6
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.6, CUDNN_VERSION=9.6.0, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS=-O2 -pipe -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.0, USE_CUDA=1, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,
</code></pre>
<p><strong>profile.py</strong></p>
<pre><code class="lang-auto">import torch
from torch.backends import cudnn

kernel = torch.rand(64, 3, 3, 3, device='cuda')

cudnn.benchmark = False
cudnn.benchmark_limit = 0
torch.cuda.synchronize()

for _ in range(50):
    data = torch.rand(64,3,224,224, device='cuda')
    _ = torch.nn.functional.conv2d(data, kernel)
torch.cuda.synchronize()

with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA],
) as prof:
        for _ in range(100):
            data = torch.rand(64,3,224,224, device='cuda')
            _ = torch.nn.functional.conv2d(data, kernel)

print(prof.key_averages().table(sort_by='self_cuda_time_total', row_limit=10))
</code></pre>
<p><strong>Output (cudnn.benchmark=False):</strong></p>
<pre><code class="lang-auto">python cudnn_bench.py
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                aten::cudnn_convolution         0.58%       2.199ms         0.92%       3.488ms      34.876us     362.741ms        95.90%     362.741ms       3.627ms           100  
_5x_cudnn_ampere_scudnn_128x64_relu_xregs_large_nn_v...         0.00%       0.000us         0.00%       0.000us       0.000us     362.427ms        95.82%     362.427ms       3.624ms           100  
                                         aten::uniform_         0.28%       1.048ms         0.50%       1.914ms      19.137us      15.509ms         4.10%      15.509ms     155.086us           100  
void at::native::(anonymous namespace)::distribution...         0.00%       0.000us         0.00%       0.000us       0.000us      15.509ms         4.10%      15.509ms     155.086us           100  
void cask__5x_cudnn::computeOffsetsKernel&lt;false, fal...         0.00%       0.000us         0.00%       0.000us       0.000us     313.564us         0.08%     313.564us       3.136us           100  
                                             aten::rand         0.10%     382.634us         1.42%       5.401ms      54.012us       0.000us         0.00%      15.509ms     155.086us           100  
                                            aten::empty         0.82%       3.105ms         0.82%       3.105ms      31.049us       0.000us         0.00%       0.000us       0.000us           100  
                                  cudaStreamIsCapturing         0.03%     119.870us         0.03%     119.870us       0.599us       0.000us         0.00%       0.000us       0.000us           200  
                                       cudaLaunchKernel         0.52%       1.984ms         0.52%       1.984ms       6.615us       0.000us         0.00%       0.000us       0.000us           300  
                                           aten::conv2d         0.04%     134.365us         1.11%       4.220ms      42.204us       0.000us         0.00%     362.741ms       3.627ms           100  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 379.873ms
Self CUDA time total: 378.249ms
</code></pre>
<p><strong>Output (cudnn.benchmark=True and cudnn.benchmark_limit=0)</strong></p>
<pre><code class="lang-auto">python cudnn_bench.py
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                                   Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
                                aten::cudnn_convolution         0.40%       2.042ms         0.65%       3.291ms      32.912us     492.278ms        97.28%     492.278ms       4.923ms           100  
       _5x_cudnn_ampere_scudnn_128x128_relu_small_nn_v1         0.00%       0.000us         0.00%       0.000us       0.000us     491.961ms        97.22%     491.961ms       4.920ms           100  
                                         aten::uniform_         0.20%       1.018ms         0.37%       1.885ms      18.845us      13.759ms         2.72%      13.759ms     137.587us           100  
void at::native::(anonymous namespace)::distribution...         0.00%       0.000us         0.00%       0.000us       0.000us      13.759ms         2.72%      13.759ms     137.587us           100  
void cask__5x_cudnn::computeOffsetsKernel&lt;false, fal...         0.00%       0.000us         0.00%       0.000us       0.000us     317.248us         0.06%     317.248us       3.172us           100  
                                             aten::rand         0.06%     295.520us         1.04%       5.258ms      52.584us       0.000us         0.00%      13.759ms     137.587us           100  
                                            aten::empty         0.61%       3.078ms         0.61%       3.078ms      30.784us       0.000us         0.00%       0.000us       0.000us           100  
                                  cudaStreamIsCapturing         0.02%     118.718us         0.02%     118.718us       0.594us       0.000us         0.00%       0.000us       0.000us           200  
                                       cudaLaunchKernel         0.38%       1.945ms         0.38%       1.945ms       6.484us       0.000us         0.00%       0.000us       0.000us           300  
                                           aten::conv2d         0.02%     123.625us         0.79%       3.994ms      39.936us       0.000us         0.00%     492.278ms       4.923ms           100  
-------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 507.724ms
Self CUDA time total: 506.037ms
</code></pre>
<p><strong>Additional info:</strong><br>
I noticed that I get the fast algorithm in both cases if I add one more call of this convolution at the beginning of the script, before setting <code>cudnn.benchmark = True</code>.<br>
Does the benchmark itself perhaps also require some sort of warmup? I.e. did it ignore the first (fastest) choice because some benchmark-related setup “polluted” that measurement?</p>

ptrblck: <p>Which GPU are you using?</p>

nkem: <p>Oops, thanks for pointing that out. It’s a GCP g2-standard-8 instance with <strong>1x NVIDIA L4</strong> - added it to the System information at the top.</p>

nkem: <p>OK, looking at the nsys profile results for the benchmark section, I can see what happened - but not <em>why</em>:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8be8cc542bc322ca3567e58a0f14af85487c331b.png" data-download-href="https://discuss.pytorch.org/uploads/default/8be8cc542bc322ca3567e58a0f14af85487c331b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8be8cc542bc322ca3567e58a0f14af85487c331b.png" alt="image" data-base62-sha1="jXH8rm0n8oLQqXafQFEApYfSfVh" width="690" height="136" data-dominant-color="CFDDE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1543×306 31.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b9cca42479f62287af67981425423b114245483.png" data-download-href="https://discuss.pytorch.org/uploads/default/8b9cca42479f62287af67981425423b114245483" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b9cca42479f62287af67981425423b114245483_2_220x500.png" alt="image" data-base62-sha1="jV4hX8a0BILZkeoYAjtMpVf2K3h" width="220" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b9cca42479f62287af67981425423b114245483_2_220x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b9cca42479f62287af67981425423b114245483_2_330x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b9cca42479f62287af67981425423b114245483_2_440x1000.png 2x" data-dominant-color="E8E8E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">541×1228 51.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Each algorithm only gets measured twice and based on the first two measurements of each algorithm, <code>relu_small_nn_v1</code> was definitely better:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>#</th>
<th>Name</th>
<th>Start</th>
<th>Duration</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>_5x_cudnn_ampere_scudnn_128x64_relu_xregs_large_nn_v1</td>
<td>2.35358s</td>
<td>3.611 ms</td>
</tr>
<tr>
<td>2</td>
<td>_5x_cudnn_ampere_scudnn_128x64_relu_xregs_large_nn_v1</td>
<td>2.35735s</td>
<td>3.609 ms</td>
</tr>
<tr>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>1</td>
<td>_5x_cudnn_ampere_scudnn_128x128_relu_small_nn_v1</td>
<td>2.36117s</td>
<td>3.527 ms</td>
</tr>
<tr>
<td>2</td>
<td>_5x_cudnn_ampere_scudnn_128x128_relu_small_nn_v1</td>
<td>2.36479s</td>
<td><strong>3.494 ms</strong></td>
</tr>
<tr>
<td>49</td>
<td>_5x_cudnn_ampere_scudnn_128x128_relu_small_nn_v1</td>
<td>3.49518s</td>
<td>5.771 ms</td>
</tr>
<tr>
<td>50</td>
<td>_5x_cudnn_ampere_scudnn_128x128_relu_small_nn_v1</td>
<td>3.5011s</td>
<td>5.770 ms</td>
</tr>
</tbody>
</table>
</div><p>So now I have even more questions:</p>
<ul>
<li>
Why does the time vary so much for the first ~30 iterations - it happens every time I run the script, not some occasional, temporary slowdown
</li>
<li>
Is there a way to increase the number of tries cudnn.benchmark will perform?
</li>
</ul>
<p><strong>Update:</strong><br>
Yes, there is a way to change the cuDNN benchmark behavior, found in:</p><aside class="onebox githubcommit" data-onebox-src="https://github.com/pytorch/pytorch/commit/8f6951cf552ccf13f74c80fe078e0f7381f45439">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/commit/8f6951cf552ccf13f74c80fe078e0f7381f45439" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Commit">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M10.86 7c-.45-1.72-2-3-3.86-3-1.86 0-3.41 1.28-3.86 3H0v2h3.14c.45 1.72 2 3 3.86 3 1.86 0 3.41-1.28 3.86-3H14V7h-3.14zM7 10.2c-1.22 0-2.2-.98-2.2-2.2 0-1.22.98-2.2 2.2-2.2 1.22 0 2.2.98 2.2 2.2 0 1.22-.98 2.2-2.2 2.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/commit/8f6951cf552ccf13f74c80fe078e0f7381f45439" target="_blank" rel="noopener nofollow ugc">[cuDNN][cuDNN V8 frontend API] Clean up `time_sorted_plan` workaround for cuDNN v8 API (#100287)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        committed <span class="discourse-local-date" data-format="ll" data-date="2023-05-03" data-time="22:37:16" data-timezone="UTC">10:37PM - 03 May 23 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/eqy" target="_blank" rel="noopener nofollow ugc">
          <img alt="" src="https://avatars.githubusercontent.com/u/2239616?v=4" class="onebox-avatar-inline" width="20" height="20">
          eqy
        </a>
      </div>

      <div class="lines" title="changed 1 files with 4 additions and 83 deletions">
        <a href="https://github.com/pytorch/pytorch/commit/8f6951cf552ccf13f74c80fe078e0f7381f45439" target="_blank" rel="noopener nofollow ugc">
          <span class="added">+4</span>
          <span class="removed">-83</span>
        </a>
      </div>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">`cudnn-frontend` (bumped in #99674) has added support for limiting the number of<span class="show-more-container"><a href="https://github.com/pytorch/pytorch/commit/8f6951cf552ccf13f74c80fe078e0f7381f45439" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden"> kernels to benchmark, so we can remove the workaround introduced in #91032.

CC @ngimel @ptrblck

Pull Request resolved: https://github.com/pytorch/pytorch/pull/100287
Approved by: https://github.com/ngimel</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Currently, the sampling technique is hardcoded to <code>CUDNN_FIND_SAMPLE_ONCE</code>, but there are also <code>CUDNN_FIND_SAMPLE_MEDIAN_OF_THREE</code> and <code>CUDNN_FIND_SAMPLE_TILL_STABLE</code>.<br>
Also, I take the existence of the latter as confirmation that the wobbly performance in the first 30 iterations is not unexpected, either…</p>

eqy: <p>I’m not sure the root cause of the observed slowdown is actually benchmarking <em>inaccuracy</em>–it’s a bit unusual of a scenario but it seems that the current script in conjunction on L4 (being only 72W power limit?) might be throttling back and slowing down more in the benchmarking case.</p>
<p>Namely I observe that reducing the number of “warmup” iterations in the script along with the number of timing iterations brings the <code>benchmark=True</code> and <code>benchmark=False</code> cases much closer together. I suspect that this is because benchmark=True is more liable to saturate the power and/or thermal limits of the GPU (due to the additional kernels being run) such that the actual profiled iterations are slowed down. Additionally note that cuDNN will only benchmark/autotune in the very first warmup iteration, the following warmup iterations will not affect the kernel <em>selection</em> itself.</p>
<p>Finally for future experimentation I did open a PR to make the cuDNN benchmarking technique user-controllable: <a href="https://github.com/pytorch/pytorch/pull/145779" rel="noopener nofollow ugc">[CUDNN][CUDNN V8 API] Allow user-specified CUDNN V8 API benchmarking technique by eqy · Pull Request #145779 · pytorch/pytorch</a></p>

nkem: <aside class="quote no-group" data-username="eqy" data-post="5" data-topic="215906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>I suspect that this is because benchmark=True is more liable to saturate the power and/or thermal limits of the GPU (due to the additional kernels being run) such that the actual profiled iterations are slowed down.</p>
</blockquote>
</aside>
<p>It looks like that is exactly what is happening. I added a 100ms sleep after each iteration to prevent any power/temperature related throttling and now the measurements for each iteration are consistent (and <code>cudnn.benchmark=true</code> picks the best choice assuming no external throttling):</p>
<div class="md-table">
<table>
<thead>
<tr>
<th><code>cudnn.benchmark=</code></th>
<th>Instances</th>
<th>Avg</th>
<th>Med</th>
<th>Min</th>
<th>Max</th>
<th>StdDev</th>
<th>Name</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>true</code></td>
<td>54</td>
<td>3,523 ms</td>
<td>3,511 ms</td>
<td>3,482 ms</td>
<td>4,101 ms</td>
<td>80,590 μs</td>
<td><code>_5x_cudnn_ampere_scudnn_128x128_relu_small_nn_v1</code></td>
</tr>
<tr>
<td><code>false</code></td>
<td>51</td>
<td>3,638 ms</td>
<td>3,638 ms</td>
<td>3,606 ms</td>
<td>3,646 ms</td>
<td>6,006 μs</td>
<td><code>_5x_cudnn_ampere_scudnn_128x64_relu_xregs_large_nn_v1</code></td>
</tr>
</tbody>
</table>
</div><p>Next, I want to check if I also can get benchmark to pick the best choice for the throttled case. Will update later.</p>
<p>Thanks so much, <a class="mention" href="/u/eqy">@eqy</a>!</p>
<p><strong>Update</strong><br>
Getting benchmark to pick a more “heat-resistant” algorithm works, but only if I run my ~200 warmup iterations before enabling <code>cudnn.benchmark</code> with a slightly differently shaped input tensor, otherwise nothing happens, i.e. new iterations with that convolution won’t trigger sampling of new algorithms, it just continues to use the default one… maybe a bug.</p>
<p>Manually throttling the GPU should give more consistent results?</p>

nkem: <p>Maybe there is still something else going on than just power/temperature related throttling? I manually reduced the power limit of that L4 from 72W to 40W and still see the same pattern. <code>nvidia-smi</code> reports idle temperatures of 76°C. I saw a short spike to 80°C at 72W, and 77°C at 40W.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c442d4f7a20f064effdc4ea4eb50078f7d5b37aa.png" data-download-href="https://discuss.pytorch.org/uploads/default/c442d4f7a20f064effdc4ea4eb50078f7d5b37aa" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c442d4f7a20f064effdc4ea4eb50078f7d5b37aa.png" alt="image" data-base62-sha1="s0cLNlja9XXqVrI2VSKRZVreuGC" width="511" height="500" data-dominant-color="E9E9EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">527×515 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Repeated the run a few hours later when I saw idle temperature was down to ~55°C and the measurments jumped from 4ms to 22ms… I don’t know what it is, but I am now fairly sure it’s not Pytorch related.</p>