cuDNN exception during engine config selection

F10w101: <p>Note: My understanding of cuDNN internals are limited, so if you realize that my question solely stems from wrong udnerstanding, please point it out so I can learn.</p>
<p>In <code>aten/src/ATen/native/cudnn/Conv_v8.cpp</code> (<a href="https://github.com/pytorch/pytorch/blob/19e93b85b91dba5c2d04e97f82ec764f404a4c10/aten/src/ATen/native/cudnn/Conv_v8.cpp#L564" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/19e93b85b91dba5c2d04e97f82ec764f404a4c10/aten/src/ATen/native/cudnn/Conv_v8.cpp#L564</a>) the <code>try_configs</code> function tries to create a cudnn_frontend engine, catching exceptions when a certain config fails. Reading the code around that function and <a href="https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html" rel="noopener nofollow ugc">the cuDNN docs</a> these exceptions are of no concern as long as one Engine can be created succesfully.</p>
<p>These exceptions however always show up in my output window in visual studio and my debugger always breaks at those. I am also wondering about the design choice in general, as there seems to be a cuDNN internal function that does the same job, but probably doesn’t throw any exceptions internally. <a href="https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#heuristics" rel="noopener nofollow ugc">From the docs</a>:</p>
<blockquote>
<p>You can “auto-tune”, that is, iterate over the list and time for each engine config and choose the best one for a particular problem on a particular device. The cuDNN frontend API provides a convenient function, <code>cudnnFindPlan()</code>, which does this.</p>
</blockquote>
<p>Is there</p>
<ul>
<li>a) A reason why choosing the first working config from the list was implemented in this form?</li>
<li>b) A way to hide this exception from showing up in visual studios output window? I like to keep it clean to be able to pinpoint actual errors more quickly</li>
</ul>

ptrblck: <p>Could you show what your Visual Studio output prints for these exceptions, please?<br>
CC <a class="mention" href="/u/eqy">@eqy</a></p>

F10w101: <p>I get</p>
<pre><code>Exception thrown at 0x00007FFA6AC3CF19 in foo.exe: Microsoft C++ exception: cudnn_frontend::cudnnException at memory location 0x00000049E82EE7D0.
</code></pre>
<p>it is not very descriptive in itself. With the debugger I could see that it is thrown from <code>pytorch\third_party\cudnn_frontend\include\cudnn_frontend_utils.h</code> in <code>set_error_and_throw_exception</code> which itself is called from <code>pytorch\third_party\cudnn_frontend\include\cudnn_frontend_ExecutionPlan.h</code> in the lines (</p>
<pre><code class="lang-auto">if (status != CUDNN_STATUS_SUCCESS) {
            set_error_and_throw_exception(
                &amp;m_execution_plan, status, "CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed");
            return std::move(m_execution_plan);
</code></pre>
<p>In pytorch, the calling function is <code>try_configs</code> in <code>pytorch\aten\src\ATen\native\cudnn\Conv_v8.cpp</code> which catches and ignores the exception:</p>
<pre><code class="lang-auto">bool try_configs(cudnn_frontend::EngineConfigList&amp; configs, const std::string&amp; opgraph_tag, const CacheKey&amp; key, const cudnnHandle_t handle, const Tensor&amp; x, const Tensor&amp; y, const Tensor&amp; w) {
  for (auto &amp; config : configs) {
    try {
      auto plan = cudnn_frontend::ExecutionPlanBuilder()
                    .setHandle(handle)
                    .setEngineConfig(config, opgraph_tag)
                    .build();
      if (plan_errata_exception(handle, plan.getTag())) {
        continue;
      }
      run_conv_plan(handle, x, y, w, plan);
      benchmark_cache.emplace(key, plan);
      return true;
    } catch (cudnn_frontend::cudnnException &amp;e) {} catch(CuDNNError &amp;e) {}
      catch (c10::OutOfMemoryError &amp;e) {
        cudaGetLastError(); // clear CUDA error
    }
  }
  return false;
}
</code></pre>

eqy: <p>Sorry for the delayed response, but for a) my understanding is that this is a historical limitation of cuDNN where the guarantee of a given <code>EngineConfig</code> working after being returned by heuristics is not super strong, and this exception handling until the first working config is found is just a CYA solution (which unfortunately still seems to be needed based on your experience).</p>