How cudnn.deterministc actually works？

username: jiangzz
cooked: <p>Recently we’ ve been working on storing the cache of benchmark and deterministic.<br>
We test on a inside detection model, whose input shape varies a lot.</p>
<h2>
<a name="testing-environment-1" class="anchor" href="#testing-environment-1"></a>Testing Environment:</h2>
<ul>
<li><strong>pytorch 1.8.1 and cuda 11.2 on Tesla PG503-216.</strong></li>
<li>seeds are fixed to 0</li>
</ul>
<p>we ran (<strong>benchmark=False, deterministic=False</strong>) and (<strong>benchmark=True, deterministic=False</strong>) and analyzed the saved cache, (key,value) pairs stored in BenchmarkCace struct.</p>
<p>During our test, we observe:</p>
<ol>
<li>
<p>we ran (<strong>benchmark=False, deterministic=False</strong>)  twice, for same ConvolutionParams we get the same conv algorithm, however (<strong>benchmark=True, deterministic=False</strong>) gives different results.</p>
</li>
<li>
<p>It seems not definite that the algorithm chosen by <strong>cudnnFindConvolutionForwardAlgorithmEx</strong> give better results than <strong>cudnnGetConvolutionForwardAlgorithm_v7</strong>.<br>
<em>here we load the cache at the beginning, the algorithm id is directly provided.</em></p>
</li>
</ol>
<p>So we are wondering:</p>
<ol>
<li>how <strong>cudnnGetConvolutionForwardAlgorithm_v7</strong> actually works, what kind of information it utilizes to offer the algorithm, we noticed that the perf struct’s time = -1 (e.g. cudnnConvolutionFwdAlgoPerf_t)</li>
<li>the time which has actually value inside the perf struct returned by <strong>cudnnFindConvolutionForwardAlgorithmEx</strong> is measured.</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p><code>benchmark=True</code> will profile kernels and select the fastest one, which might not be deterministic as it would depend on your system load. <code>benchmark=False</code> will use internal heuristics and try to select the fastest kernel given the setup (input shape, conv arguments, <code>dtype</code> etc.). <code>deterministic=True</code> will force cuDNN to select deterministic kernels (i.e. their computation is deterministic).</p>
</li>
<li>
<p>The returned time is created internally in <code>cudnnFind</code> during the kernel profiling.</p>
</li>
</ol>

username: Jack47
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks<br>
for you reply. During jiangzz’s test, we found in detection model, <code>determinstic=False</code> behaves on par with <code>benchmark=True</code>, so we are wondering why deterministic performs so well? Before that we believe benchmark have more algorithms to choose and should found the fastest one.</p>

username: ptrblck
cooked: <p>I don’t know what your exact use case is, but did you check which actual kernel is being executed?<br>
Is the deterministic one the fastest or is <code>benchmark=True</code> selecting another one with the same execution speed?<br>
Deterministic kernels are not forced to be slower by default, but <em>can</em> decrease the performance.</p>

username: jiangzz
cooked: <p>During Test, we compare the Benchmark cache of cudnn operator selected by <strong>cudnnGetConvolutionForwardAlgorithm_v7</strong> and <strong>cudnnFindConvolutionForwardAlgorithmEx</strong>, some of them are different, some are same.<br>
Among the different choices, some operators selected by <strong>cudnnFindConvolutionForwardAlgorithmEx</strong> are not faster.<br>
We did the time test like this:</p>
<pre><code class="lang-auto">struct TimeMeasure{
  float _t;
  cudaEvent_t start, stop;
  TimeMeasure(){
    cudaEventCreate(&amp;start);
    cudaEventCreate(&amp;stop);
  }
  ~TimeMeasure(){
    cudaEventDestroy(start);
	  cudaEventDestroy(stop);
  }
};
TimeMeasure tm;
if(flag){
        cudaEventRecord(tm-&gt;start);
      }
      AT_CUDNN_CHECK_WITH_SHAPES(cudnnConvolutionForward(
          args.handle,
          &amp;one, args.idesc.desc(), input.data_ptr(),
          args.wdesc.desc(), weight.data_ptr(),
          args.cdesc.desc(), fwdAlgPerf.algo, workspace.data_ptr(), fwdAlgPerf.memory,
          &amp;zero, args.odesc.desc(), output.data_ptr()),
        args, "Forward algorithm: ", static_cast&lt;int&gt;(fwdAlgPerf.algo), "\n");
      if(flag){
        cudaEventRecord(tm-&gt;stop);
        cudaEventSynchronize(tm-&gt;stop);
        float time;
        cudaEventElapsedTime(&amp;time, tm-&gt;start, tm-&gt;stop);
        tm-&gt;_t = time;
}
</code></pre>