Finding source code for cudnn.benchmark

Rasmus_Hoier: <p>Hi,<br>
I would like to read the source code for <a href="https://pytorch.org/docs/stable/backends.html#torch.backends.cudnn.benchmark" rel="noopener nofollow ugc">cudnn.benchmark</a>, but there is no link to it in the docs, and I couldn’t find it by googling. Where can I find it?</p>

ptrblck: <p>The <code>benchmark</code> flag will be passed eventually to <a href="https://github.com/pytorch/pytorch/blob/92a9c0e3e050ac9e1972914d6040c9dd0c69cbdb/aten/src/ATen/native/cudnn/Conv_v7.cpp#L258-L329"><code>algorithm_search</code></a> where either <code>cudnnGet</code> or <code>cudnnFind</code> will be called in the v7 API. A similar approach is used for the experimental v8 API.</p>

Rasmus_Hoier: <p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Are these the various convolution implementations being benchmarked?: <a href="https://github.com/pytorch/pytorch/blob/92a9c0e3e050ac9e1972914d6040c9dd0c69cbdb/aten/src/ATen/native/cudnn/Conv_v7.cpp#L267-L276" rel="noopener nofollow ugc">link</a><br>
Where are these algorithms defined?</p>

ptrblck: <p>The algorithms are closed source and shipped in cuDNN.</p>

Rasmus_Hoier: <p>Okay,<br>
Thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

Wang_Albert: <p>Hi,</p>
<p>In v7 API, it will use cudnnFindConvolutionForwardAlgorithmEx() to get the lowest compute time convolution algorithm, according to <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cudnn/Conv_v7.cpp#L265-L336" rel="noopener nofollow ugc">algorithm_search</a><br>
But in v8, I can’t understand or find the corresponding part similar to it. Is <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cudnn/Conv_v8.cpp#L403-L429" rel="noopener nofollow ugc">get_plans_from_find</a> the approach in v8 to select which convolution algorithm to run? Could you help me understanding this part?<br>
Thanks!</p>

ptrblck: <p><code>cudnn_frontend::time_sorted_plan</code> should contain the execution plans sorted by their run time, which is then moved to <code>sorted_plans</code> and returned.<br>
<a class="mention" href="/u/eqy">@eqy</a> would know the details about this implementation as he is the code owner.</p>

eqy: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is right, the corresponding function is <code>get_plans_from_find</code>, but the actual timing is done by <code>time_sorted_plan</code> which is called by <code>get_plans_from_find</code>.</p>

Wang_Albert: <p>Thanks,</p>
<p>After tracing the corresponding part in cudnn frontend API, now I understand that <code>time_sorted_plan</code> can find the the fastest plan.<br>
But how does each plan correspond to a different convolution algorithm?<br>
Is that because the cudnnBackendDescriptorType_t, which be passed in to <code>get_plans_from_find</code>, is <code>CUDNN_BACKEND_OPERATION_CONVOLUTION_FORWARD_DESCRIPTOR</code>?</p>

eqy: <p>Plans are derived from engine configs, which can be thought of as a combination of algorithm + specific knob settings. Benchmarking is done to find the best choice of algorithm + settings.</p>