Manually set cudnn convolution algorithm

gabrieldernbach: <p>From other threads I found that,<br>
<code>&gt; `cudnn.benchmark=True` will try different convolution algorithms for each input shape.</code><br>
So I believe that torch can set the algorithms specifically for each layer individually.</p>
<p>Is there a possibility to set this manually?<br>
From <code>https://core.ac.uk/download/pdf/224976536.pdf</code> I see that we have <code>GEMM</code>, <code>FFT</code> and <code>Winograd</code>.</p>

ptrblck: <p>No, you cannot set the algorithm manually and the benchmark mode will select the fastest found algo.<br>
What’s your use case you want to specify the algorithms manually?</p>

gabrieldernbach: <p>Thank you for the quick answer.</p>
<p>My use case is as follows:</p>
<p>I have a model that needs to act on variable input sizes. Therefore benchmark mode can not infer a single best option.</p>
<p>I know that the vast majority of cases lie within a small range of input sizes. So I could run a benchmark on the mode of the distribution and thereby get the fasted algorithm for most of the cases.</p>
<p>I would then manually fix the algorithm choice before going to variable-sized inputs.</p>

ptrblck: <p>Thanks for this information.<br>
At the moment this would be possible by writing a custom CUDA extension and specifying the algo there.<br>
We are currently working on enabling the cudnnV8 API, so feel free to post a feature request on GitHub for it so that we can discuss it there further.</p>

eduardo4jesus: <p>Would you have an example of how to call cudnn functions from a custom CUDA extension? I was looking for how <code>aten::cudnn_convolution</code> is implemented and I couldn’t find anything under the <code>aten</code> folder.</p>

ptrblck: <p>A general cuDNN extension example can be found <a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/test/cpp_extensions/cudnn_extension.cpp">here</a> and <a href="https://discuss.pytorch.org/t/implementing-a-custom-convolution-using-conv2d-input-and-conv2d-weight/18556/24">this post</a> might also be useful.</p>

eduardo4jesus: <p>Thank you for sharing these references, on the second link there is the call for the following function.</p>
<pre><code class="lang-auto">at::Tensor convolution(
    const at::Tensor&amp; input,
    const at::Tensor&amp; weight,
    const at::Tensor&amp; bias,
    c10::ArrayRef&lt;int64_t&gt; stride,
    c10::ArrayRef&lt;int64_t&gt; padding,
    c10::ArrayRef&lt;int64_t&gt; dilation,
    int64_t groups,
    bool benchmark,
    bool deterministic) {

    return at::cudnn_convolution(
        input,
        weight,
        bias,
        padding,
        stride,
        dilation,
        groups,
        benchmark,
        deterministic); 
}
</code></pre>
<p>I am still trying to identify how to select the algorithm used by the <code>cudnn</code> convolution. None of these arguments seems to do so.</p>
<p>I am to be able to select among (a) GEMM, (b) FFT and (c) Winograd.</p>

ptrblck: <p>For the algo selection, you would need to down a layer deeper and call the raw cuDNN calls as seen <a href="https://github.com/pytorch/pytorch/blob/8d08b103be936d78d5d4ed90c0547aeccb8ce166/aten/src/ATen/native/cudnn/Conv_v7.cpp#L625">here</a>.</p>

eduardo4jesus: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. I am trying to do a wrap to work if the <code>cudnn</code> functions and set the algorithm. As of now I did the forward pass, as in <a href="https://github.com/eduardo4jesus/PyTorch-cuDNN-Convolution/blob/master/cudnn_convolution.cpp" rel="noopener nofollow ugc">this file</a>. It seems it is working fine for a given algorithm.</p>
<p>However, when I try find the fastest algorithm by calling the <a href="https://github.com/eduardo4jesus/PyTorch-cuDNN-Convolution/blob/6bb042d1aff370031d69ffa22d5f67cb43461df1/cudnn_convolution.cpp#L123" rel="noopener nofollow ugc"><code>cudnnFindConvolutionForwardAlgorithm</code></a>, half of the times the program crashes, giving <code>Segmentation fault (core dumped)</code>.</p>
<p>Any tip on how to investigate the reason for that?</p>
<hr>
<pre><code class="lang-auto">(base) eduardoj@Worksmart:~/Repo/eduardo4jesus/PyTorch-cuDNN-Convolution$ python example.py 
Using /home/eduardoj/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/build.ninja...
Building extension module cudnn_convolution...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cudnn_convolution...
Compiled and Loaded!
Trying all
Algorithm: CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM
	Status: CUDNN_STATUS_SUCCESS
	Time: 0.26624
	Memory: 0
	MathType: CUDNN_DEFAULT_MATH

Algorithm: CUDNN_CONVOLUTION_FWD_ALGO_GEMM
	Status: CUDNN_STATUS_SUCCESS
	Time: 0.535424
	Memory: 55795728
	MathType: CUDNN_DEFAULT_MATH

Algorithm: CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM
	Status: CUDNN_STATUS_SUCCESS
	Time: 92.4836
	Memory: 1592080
	MathType: CUDNN_DEFAULT_MATH

Algorithm: CUDNN_CONVOLUTION_FWD_ALGO_DIRECT
	Status: CUDNN_STATUS_NOT_SUPPORTED
	Time: -1
	Memory: 0
	MathType: CUDNN_DEFAULT_MATH

Algorithm: CUDNN_CONVOLUTION_FWD_ALGO_FFT
	Status: CUDNN_STATUS_NOT_SUPPORTED
	Time: -1
	Memory: 0
	MathType: CUDNN_DEFAULT_MATH

Algorithm: CUDNN_CONVOLUTION_FWD_ALGO_FFT_TILING
	Status: CUDNN_STATUS_NOT_SUPPORTED
	Time: -1
	Memory: 0
	MathType: CUDNN_DEFAULT_MATH

Algorithm: CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD
	Status: CUDNN_STATUS_NOT_SUPPORTED
	Time: -1
	Memory: 0
	MathType: CUDNN_DEFAULT_MATH

Algorithm: CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD_NONFUSED
	Status: CUDNN_STATUS_INTERNAL_ERROR
	Time: -1
	Memory: 92887808
	MathType: CUDNN_DEFAULT_MATH

Allocating Workspace
Workspace size: 0MB
Elapsed Time: 0.278528 ms
Done!
(base) eduardoj@Worksmart:~/Repo/eduardo4jesus/PyTorch-cuDNN-Convolution$ python example.py 
Using /home/eduardoj/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/build.ninja...
Building extension module cudnn_convolution...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cudnn_convolution...
Compiled and Loaded!
Trying all
Segmentation fault (core dumped)
(base) eduardoj@Worksmart:~/Repo/eduardo4jesus/PyTorch-cuDNN-Convolution$ 

</code></pre>

ptrblck: <p>Could you check the backtrace of the seg fault and see what exactly is causing it?</p>

eduardo4jesus: <p>Here is the backtrace output.</p>
<pre><code class="lang-auto">(base) eduardoj@Worksmart:~/Repo/eduardo4jesus/PyTorch-cuDNN-Convolution$ gdb python
gdb: /home/eduardoj/miniconda3/lib/libtinfo.so.6: no version information available (required by gdb)
gdb: /home/eduardoj/miniconda3/lib/libncursesw.so.6: no version information available (required by gdb)
gdb: /home/eduardoj/miniconda3/lib/libncursesw.so.6: no version information available (required by gdb)
gdb: /home/eduardoj/miniconda3/lib/libncursesw.so.6: no version information available (required by gdb)
GNU gdb (Ubuntu 12.0.90-0ubuntu1) 12.0.90
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
&lt;https://www.gnu.org/software/gdb/bugs/&gt;.
Find the GDB manual and other documentation resources online at:
    &lt;http://www.gnu.org/software/gdb/documentation/&gt;.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
(gdb) run example.py
Starting program: /home/eduardoj/miniconda3/bin/python example.py
/bin/bash: /home/eduardoj/miniconda3/lib/libtinfo.so.6: no version information available (required by /bin/bash)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 8480]
[New Thread 0x7fff17cdd640 (LWP 8486)]
Using /home/eduardoj/.cache/torch_extensions/py39_cu113 as PyTorch extensions root...
[Detaching after fork from child process 8487]
[Detaching after fork from child process 8488]
[Detaching after fork from child process 8489]
Detected CUDA files, patching ldflags
Emitting ninja build file /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/build.ninja...
Building extension module cudnn_convolution...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[Detaching after fork from child process 8490]
ninja: no work to do.
Loading extension module cudnn_convolution...
Compiled and Loaded!
[New Thread 0x7fff161fdac0 (LWP 8491)]
[New Thread 0x7fff159fbb40 (LWP 8492)]
[New Thread 0x7fff151f9bc0 (LWP 8493)]
[New Thread 0x7fff149f8640 (LWP 8494)]
[New Thread 0x7fff11fff640 (LWP 8495)]
Trying all

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffd73c7eb1 in cublasGetMathMode () from /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/lib/../../../../libcublas.so.11
(gdb) backtrace
#0  0x00007fffd73c7eb1 in cublasGetMathMode () from /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/lib/../../../../libcublas.so.11
#1  0x00007ffe2c0b9014 in ?? () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#2  0x00007ffe2c0beff3 in cudnn::winograd_nonfused::conv2dForward(cudnnContext*, CUstream_st*, void const*, cudnnTensor4dStruct const*, void const*, cudnnFilter4dStruct const*, void const*, cudnnConvolutionStruct const*, void*, unsigned long, bool, void const*, void const*, void const*, cudnnTensor4dStruct const*, void*, bool) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#3  0x00007ffe2bde96b9 in cudnn::cnn::WinogradNonfusedEngine&lt;true&gt;::execute_internal_impl(cudnn::backend::VariantPack const&amp;, CUstream_st*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#4  0x00007ffe2b9d3e71 in cudnn::cnn::EngineInterface::execute(cudnn::backend::VariantPack const&amp;, CUstream_st*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#5  0x00007ffe2bb46b1e in cudnn::cnn::AutoTransformationExecutor::execute_pipeline(cudnn::cnn::EngineInterface&amp;, cudnn::backend::VariantPack const&amp;, CUstream_st*) const () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#6  0x00007ffe2bb46c23 in cudnn::cnn::BatchPartitionExecutor::operator()(cudnn::cnn::EngineInterface&amp;, cudnn::cnn::EngineInterface*, cudnn::backend::VariantPack const&amp;, CUstream_st*) const () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#7  0x00007ffe2bb62466 in cudnn::cnn::GeneralizedConvolutionEngine&lt;cudnn::cnn::WinogradNonfusedEngine&lt;true&gt; &gt;::execute_internal_impl(cudnn::backend::VariantPack const&amp;, CUstream_st*) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#8  0x00007ffe2b9d3e71 in cudnn::cnn::EngineInterface::execute(cudnn::backend::VariantPack const&amp;, CUstream_st*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#9  0x00007ffe2b9ef9e8 in cudnn::backend::execute(cudnnContext*, cudnn::backend::ExecutionPlan&amp;, cudnn::backend::VariantPack&amp;) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#10 0x00007ffe2bcd3a19 in cudnn::backend::EnginesAlgoMap&lt;cudnnConvolutionFwdAlgo_t, 8&gt;::execute_wrapper(cudnnContext*, cudnnConvolutionFwdAlgo_t, cudnn::backend::ExecutionPlan&amp;, cudnn::backend::VariantPack&amp;) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#11 0x00007ffe2bcc6aa8 in cudnn::backend::convolutionForward(cudnnContext*, void const*, cudnnTensorStruct const*, void const*, cudnnFilterStruct const*, void const*, cudnnConvolutionStruct const*, cudnnConvolutionFwdAlgo_t, void*, unsigned long, bool, void const*, void const*, void const*, cudnnActivationStruct const*, cudnnTensorStruct const*, void*) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#12 0x00007ffe2be2d839 in cudnn::cnn::convolutionForward(cudnnContext*, void const*, cudnnTensorStruct const*, void const*, cudnnFilterStruct const*, void const*, cudnnConvolutionStruct const*, cudnnConvolutionFwdAlgo_t, void*, unsigned long, void const*, cudnnTensorStruct const*, void*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#13 0x00007ffe2be3f426 in ?? () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#14 0x00007ffe2be3fe34 in cudnnStatus_t findAlgorithm&lt;find_get_conv_params, cudnnConvolutionStruct, cudnnConvolutionFwdAlgo_t, cudnnConvolutionFwdAlgoPerfStruct, 8, true&gt;(find_get_conv_params, int, int*, cudnnConvolutionFwdAlgoPerfStruct*) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#15 0x00007ffe2be2dbc9 in cudnnFindConvolutionForwardAlgorithm () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#16 0x00007fff17480e87 in convolution(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool) () from /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/cudnn_convolution.so
#17 0x00007fff174abce8 in at::Tensor pybind11::detail::argument_loader&lt;int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool&gt;::call_impl&lt;at::Tensor, at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool), 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, pybind11::detail::void_type&gt;(at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool), std::integer_sequence&lt;unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul&gt;, pybind11::detail::void_type&amp;&amp;) &amp;&amp; () from /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/cudnn_convolution.so
--Type &lt;RET&gt; for more, q to quit, c to continue without paging--c
#18 0x00007fff174a7213 in std::enable_if&lt;!std::is_void&lt;at::Tensor&gt;::value, at::Tensor&gt;::type pybind11::detail::argument_loader&lt;int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool&gt;::call&lt;at::Tensor, pybind11::detail::void_type, at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool)&gt;(at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool)) &amp;&amp; () from /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/cudnn_convolution.so
#19 0x00007fff174a1bac in pybind11::cpp_function::initialize&lt;at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool), at::Tensor, int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool, pybind11::name, pybind11::scope, pybind11::sibling, char [12]&gt;(at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool), at::Tensor (*)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool), pybind11::name const&amp;, pybind11::scope const&amp;, pybind11::sibling const&amp;, char const (&amp;) [12])::{lambda(pybind11::detail::function_call&amp;)#3}::operator()(pybind11::detail::function_call&amp;) const () from /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/cudnn_convolution.so
#20 0x00007fff174a2238 in pybind11::cpp_function::initialize&lt;at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool), at::Tensor, int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool, pybind11::name, pybind11::scope, pybind11::sibling, char [12]&gt;(at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool), at::Tensor (*)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, bool), pybind11::name const&amp;, pybind11::scope const&amp;, pybind11::sibling const&amp;, char const (&amp;) [12])::{lambda(pybind11::detail::function_call&amp;)#3}::_FUN(pybind11::detail::function_call&amp;) () from /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/cudnn_convolution.so
#21 0x00007fff1748faae in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/cudnn_convolution.so
#22 0x00005555556a290c in cfunction_call (func=0x7fff1df514f0, args=&lt;optimized out&gt;, kwargs=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.9.13/Objects/methodobject.c:543
#23 0x0000555555688fa7 in _PyObject_MakeTpCall (tstate=0x55555590fd50, callable=0x7fff1df514f0, args=&lt;optimized out&gt;, nargs=&lt;optimized out&gt;, keywords=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.9.13/Objects/call.c:191
#24 0x0000555555684d5f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=&lt;optimized out&gt;, args=&lt;optimized out&gt;, callable=0x7fff1df514f0, tstate=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:116
#25 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=&lt;optimized out&gt;, args=0x55555596a8c8, callable=0x7fff1df514f0, tstate=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:103
#26 PyObject_Vectorcall (kwnames=0x0, nargsf=&lt;optimized out&gt;, args=0x55555596a8c8, callable=0x7fff1df514f0) at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127
#27 call_function (kwnames=0x0, oparg=&lt;optimized out&gt;, pp_stack=&lt;synthetic pointer&gt;, tstate=0x55555590fd50) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077
#28 _PyEval_EvalFrameDefault (tstate=&lt;optimized out&gt;, f=&lt;optimized out&gt;, throwflag=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3489
#29 0x000055555567ea17 in _PyEval_EvalFrame (throwflag=0, f=0x55555596a750, tstate=0x55555590fd50) at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40
#30 _PyEval_EvalCode (tstate=&lt;optimized out&gt;, _co=0x7ffff6dd2870, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;, kwargs=&lt;optimized out&gt;, kwcount=&lt;optimized out&gt;, kwstep=&lt;optimized out&gt;, defs=&lt;optimized out&gt;, defcount=&lt;optimized out&gt;, kwdefs=&lt;optimized out&gt;, closure=&lt;optimized out&gt;, name=&lt;optimized out&gt;, qualname=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
#31 0x000055555567e6d7 in _PyEval_EvalCodeWithName (_co=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4361
#32 0x000055555567e689 in PyEval_EvalCodeEx (_co=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kws=&lt;optimized out&gt;, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4377
#33 0x0000555555739e3b in PyEval_EvalCode (co=co@entry=0x7ffff6dd2870, globals=globals@entry=0x7ffff6dc7b00, locals=locals@entry=0x7ffff6dc7b00) at /usr/local/src/conda/python-3.9.13/Python/ceval.c:828
#34 0x00005555557684a9 in run_eval_code_obj (tstate=0x55555590fd50, co=0x7ffff6dd2870, globals=0x7ffff6dc7b00, locals=0x7ffff6dc7b00) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1221
#35 0x0000555555764694 in run_mod (mod=&lt;optimized out&gt;, filename=&lt;optimized out&gt;, globals=0x7ffff6dc7b00, locals=0x7ffff6dc7b00, flags=&lt;optimized out&gt;, arena=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1242
#36 0x00005555555e96d2 in pyrun_file (fp=0x55555590d4f0, filename=0x7ffff6d69eb0, start=&lt;optimized out&gt;, globals=0x7ffff6dc7b00, locals=0x7ffff6dc7b00, closeit=1, flags=0x7fffffffdf38) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:1140
#37 0x000055555575e1f2 in pyrun_simple_file (flags=0x7fffffffdf38, closeit=1, filename=0x7ffff6d69eb0, fp=0x55555590d4f0) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:450
#38 PyRun_SimpleFileExFlags (fp=0x55555590d4f0, filename=&lt;optimized out&gt;, closeit=1, flags=0x7fffffffdf38) at /usr/local/src/conda/python-3.9.13/Python/pythonrun.c:483
#39 0x000055555575b533 in pymain_run_file (cf=0x7fffffffdf38, config=0x55555590e490) at /usr/local/src/conda/python-3.9.13/Modules/main.c:377
#40 pymain_run_python (exitcode=0x7fffffffdf30) at /usr/local/src/conda/python-3.9.13/Modules/main.c:602
#41 Py_RunMain () at /usr/local/src/conda/python-3.9.13/Modules/main.c:681
#42 0x000055555572db79 in Py_BytesMain (argc=&lt;optimized out&gt;, argv=&lt;optimized out&gt;) at /usr/local/src/conda/python-3.9.13/Modules/main.c:1101
#43 0x00007ffff7cabd90 in __libc_start_call_main (main=main@entry=0x55555572db30 &lt;main&gt;, argc=argc@entry=2, argv=argv@entry=0x7fffffffe168) at ../sysdeps/nptl/libc_start_call_main.h:58
#44 0x00007ffff7cabe40 in __libc_start_main_impl (main=0x55555572db30 &lt;main&gt;, argc=2, argv=0x7fffffffe168, init=&lt;optimized out&gt;, fini=&lt;optimized out&gt;, rtld_fini=&lt;optimized out&gt;, stack_end=0x7fffffffe158) at ../csu/libc-start.c:392
#45 0x000055555572da81 in _start ()
</code></pre>

ptrblck: <p>Thanks for sharing it. It seems cublas is failing via cuDNN.<br>
Could you post an executable code to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>?</p>

eduardo4jesus: <p>Thank you for taking a look at this.</p>
<p>Here is the output of <code>python -m torch.utils.collect_env</code>:</p>
<pre><code class="lang-auto">(base) eduardoj@Worksmart:~/Repo/eduardo4jesus/PyTorch-cuDNN-Convolution$ python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04 LTS (x86_64)
GCC version: (Ubuntu 10.3.0-15ubuntu1) 10.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-37-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.64
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3070 Laptop GPU
Nvidia driver version: 515.48.07
cuDNN version: Probably one of the following:
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn.so.8.4.1
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.4.1
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.4.1
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.4.1
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.4.1
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.4.1
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.22.3
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py39h7f8727e_0  
[conda] mkl_fft                   1.3.1            py39hd3c417c_0  
[conda] mkl_random                1.2.2            py39h51133e4_0  
[conda] numpy                     1.22.3           py39he7a7128_0  
[conda] numpy-base                1.22.3           py39hf524024_0  
[conda] pytorch                   1.11.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.11.0               py39_cu113    pytorch
[conda] torchvision               0.12.0               py39_cu113    pytorch
(base) eduardoj@Worksmart:~/Repo/eduardo4jesus/PyTorch-cuDNN-Convolution$ 
</code></pre>
<p>You can find the code I am executing <a href="https://github.com/eduardo4jesus/PyTorch-cuDNN-Convolution/blob/master/cudnn_convolution.cpp" rel="noopener nofollow ugc">here</a>. But, I am also posting parts of it bellow:</p>
<p><strong>example.py</strong></p>
<pre><code class="lang-auto">import torch
from torch.utils.cpp_extension import load

# load the PyTorch extension
cudnn_convolution = load(
  name="cudnn_convolution",
  sources=["cudnn_convolution.cpp"],
  extra_ldflags = ["-lcudnn"],
  with_cuda=True,
  verbose=True
)
print("Compiled and Loaded!")


# create dummy input, convolutional weights and bias
B, F, C = 8, 32, 3
N, K, O = 32, 5, 28
input  = torch.zeros(B, C, N, N).to('cuda')
weight = torch.zeros(F, C, K, K).to('cuda')
output = torch.zeros(B, F, O, O).to('cuda')

stride   = (1, 1)
padding  = (0, 0)
dilation = (1, 1)
groups   = 1

output = cudnn_convolution.convolution(
  -1, # FWD_ALGO. (-1: Tries all methods and pick the fastest)
  input, weight, output, stride, padding, dilation, groups, True
  )

print("Done!")
</code></pre>
<p><strong>cudnn_convolution.cpp</strong></p>
<pre><code class="lang-auto">/**
 * The #include&lt;ATen/cudnn/*.h&gt; needs guards as pointed in
 * https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/cudnn
 */
#include &lt;ATen/cuda/CUDAConfig.h&gt; // for the definition of AT_CUDNN_ENABLED
#if AT_CUDNN_ENABLED()

#include &lt;cudnn.h&gt;
#include &lt;torch/extension.h&gt;
#include &lt;ATen/cudnn/Handle.h&gt; // for getCudnnHandle
#include &lt;ATen/cudnn/Types.h&gt;      // for getCudnnDataType

#define checkCUDNN(expression)                               \
  {                                                          \
    cudnnStatus_t status = (expression);                     \
    if (status != CUDNN_STATUS_SUCCESS)                      \
    {                                                        \
      std::cerr &lt;&lt; "Error on line " &lt;&lt; __LINE__ &lt;&lt; ": "      \
                &lt;&lt; cudnnGetErrorString(status) &lt;&lt; std::endl; \
      std::exit(EXIT_FAILURE);                               \
    }                                                        \
  }

std::ostream&amp; operator&lt;&lt;(std::ostream &amp;out, const cudnnConvolutionFwdAlgoPerf_t &amp;fwdAlgoPert);
std::ostream&amp; operator&lt;&lt;(std::ostream &amp;out, const cudnnConvolutionBwdFilterAlgoPerf_t &amp;bwdFilterAlgoPerf);
std::ostream&amp; operator&lt;&lt;(std::ostream &amp;out, const cudnnConvolutionBwdDataAlgoPerf_t &amp;bwdDataAlgoPerf);

typedef struct _cudnnDescriptors_t_
{
  cudnnTensorDescriptor_t input, output;
  cudnnFilterDescriptor_t weight;
  cudnnConvolutionDescriptor_t convolution;

  virtual ~_cudnnDescriptors_t_()
  {
    cudnnDestroyTensorDescriptor(input);
    cudnnDestroyTensorDescriptor(output);
    cudnnDestroyFilterDescriptor(weight);
    cudnnDestroyConvolutionDescriptor(convolution);
  }
} cudnnDescriptors_t;

void initialize_descriptors(const at::Tensor &amp;input, const at::Tensor &amp;weight, const at::Tensor &amp;output,
                                          c10::ArrayRef&lt;int64_t&gt; &amp;stride,
                                          c10::ArrayRef&lt;int64_t&gt; &amp;padding,
                                          c10::ArrayRef&lt;int64_t&gt; &amp;dilation,
                                          cudnnDescriptors_t &amp;descriptors);


at::Tensor convolution(const int fwdAlgo,
                       const at::Tensor &amp;input, const at::Tensor &amp;weight, const at::Tensor &amp;output,
                       c10::ArrayRef&lt;int64_t&gt; stride, c10::ArrayRef&lt;int64_t&gt; padding,
                       c10::ArrayRef&lt;int64_t&gt; dilation, int64_t groups, bool verbose)
{
  const cudnnHandle_t cudnn = at::native::getCudnnHandle();
  cudnnDescriptors_t desc;
  initialize_descriptors(input, weight, output, stride, padding, dilation, desc);

  /*****************************************************************************
   * 2. Setting FWD Convolution Algo
   ****************************************************************************/
  cudnnConvolutionFwdAlgoPerf_t convolution_algorithm[CUDNN_CONVOLUTION_FWD_ALGO_COUNT];
  int returnedAlgoCount;

  /**
   * TODO: I frequently get segmentation fault when finding the convolution
   * algorithms. I am not sure how to fix it.
   */
  if (fwdAlgo == -1)
  {
    std::cout &lt;&lt; "Trying all" &lt;&lt; std::endl;
    checkCUDNN(
        cudnnFindConvolutionForwardAlgorithm(/*handle*/ cudnn,
                                             /*xDesc*/ desc.input,
                                             /*wDesc*/ desc.weight,
                                             /*convDesc*/ desc.convolution,
                                             /*yDesc*/ desc.output,
                                             /*requestedAlgoCount*/ CUDNN_CONVOLUTION_FWD_ALGO_COUNT,
                                             /*returnedAlgoCount*/ &amp;returnedAlgoCount,
                                             /*perfResults*/ convolution_algorithm));
    if (verbose)
      for (int i = 0; i &lt; returnedAlgoCount; i++)
        std::cout &lt;&lt; convolution_algorithm[i] &lt;&lt; std::endl;
  }
  else
  {
    convolution_algorithm[0].algo = static_cast&lt;cudnnConvolutionFwdAlgo_t&gt;(fwdAlgo);
    convolution_algorithm[0].status = static_cast&lt;cudnnStatus_t&gt;(0);
    convolution_algorithm[0].time = -1;
    convolution_algorithm[0].memory = 0;
    convolution_algorithm[0].determinism = static_cast&lt;cudnnDeterminism_t&gt;(-1);
    convolution_algorithm[0].mathType = static_cast&lt;cudnnMathType_t&gt;(0);
    if (verbose)
    {
      std::cout &lt;&lt; "Attempt with defined Algo:" &lt;&lt; std::endl;
      std::cout &lt;&lt; convolution_algorithm[0] &lt;&lt; std::endl;
    }
  }

  /*****************************************************************************
   * 3. Get and Allocate Memory for Workspace
   ****************************************************************************/
  if (verbose)
    std::cout &lt;&lt; "Allocating Workspace" &lt;&lt; std::endl;

  size_t workspace_bytes{0};
  checkCUDNN(cudnnGetConvolutionForwardWorkspaceSize(cudnn,
                                                     /*xDesc*/ desc.input,
                                                     /*wDesc*/ desc.weight,
                                                     /*convDesc*/ desc.convolution,
                                                     /*yDesc*/ desc.output,
                                                     /*algo*/ convolution_algorithm[0].algo,
                                                     /*sizeInBytes*/ &amp;workspace_bytes));

  if (verbose)
    std::cout &lt;&lt; "Workspace size: " &lt;&lt; (workspace_bytes) &lt;&lt; " Bytes" &lt;&lt; std::endl;

  void *d_workspace{nullptr};
  cudaMalloc(&amp;d_workspace, workspace_bytes);

  /*****************************************************************************
   * 4. Get and Allocate Memory for Workspace
   ****************************************************************************/
  cudaEvent_t start, stop;
  cudaEventCreate(&amp;start);
  cudaEventCreate(&amp;stop);

  cudaEventRecord(start);
  const float alpha = 1.0f, beta = 0.0f;
  checkCUDNN(cudnnConvolutionForward(cudnn,
                                     /*alpha*/ &amp;alpha,
                                     /*xDesc*/ desc.input,
                                     /*x*/ input.data_ptr(),
                                     /*wDesc*/ desc.weight,
                                     /*w*/ weight.data_ptr(),
                                     /*convDesc*/ desc.convolution,
                                     /*algo*/ convolution_algorithm[0].algo,
                                     /*workSpace*/ d_workspace,
                                     /*workSpaceSizeInBytes*/ workspace_bytes,
                                     /*beta*/ &amp;beta,
                                     /*yDesc*/ desc.output,
                                     /*y*/ output.data_ptr()));
  cudaEventRecord(stop);
  cudaEventSynchronize(stop);
  float milliseconds{0};
  cudaEventElapsedTime(&amp;milliseconds, start, stop);
  if (verbose)
    std::cout &lt;&lt; "Elapsed Time: " &lt;&lt; milliseconds &lt;&lt; " ms" &lt;&lt; std::endl;

  /*****************************************************************************
   * 5. Freeing variables
   ****************************************************************************/
  cudaFree(d_workspace);
  return output;
}


std::ostream &amp;operator&lt;&lt;(std::ostream &amp;out, const cudnnConvolutionFwdAlgo_t &amp;algo)
{
  out &lt;&lt; "FWD Algorithm: ";
  switch (algo)
  {
  case CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM";
    break;
  case CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM";
    break;
  case CUDNN_CONVOLUTION_FWD_ALGO_GEMM:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_GEMM";
    break;
  case CUDNN_CONVOLUTION_FWD_ALGO_DIRECT:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_DIRECT";
    break;
  case CUDNN_CONVOLUTION_FWD_ALGO_FFT:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_FFT";
    break;
  case CUDNN_CONVOLUTION_FWD_ALGO_FFT_TILING:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_FFT_TILING";
    break;
  case CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD";
    break;
  case CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD_NONFUSED:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD_NONFUSED";
    break;
  case CUDNN_CONVOLUTION_FWD_ALGO_COUNT:
    out &lt;&lt; "CUDNN_CONVOLUTION_FWD_ALGO_COUNT";
  default:
    std::cerr &lt;&lt; "Invalid value FWD Algorithm" &lt;&lt; std::endl;
    exit(1);
  }
  return out;
}

std::ostream &amp;operator&lt;&lt;(std::ostream &amp;out, const cudnnStatus_t &amp;status) {
  out &lt;&lt; "Status: ";
  switch (status)
  {
    case CUDNN_STATUS_SUCCESS:
      out &lt;&lt; "CUDNN_STATUS_SUCCESS";
      break;
    case CUDNN_STATUS_NOT_INITIALIZED:
      out &lt;&lt; "CUDNN_STATUS_NOT_INITIALIZED";
      break;
    case CUDNN_STATUS_ALLOC_FAILED:
      out &lt;&lt; "CUDNN_STATUS_ALLOC_FAILED";
      break;
    case CUDNN_STATUS_BAD_PARAM:
      out &lt;&lt; "CUDNN_STATUS_BAD_PARAM";
      break;
    case CUDNN_STATUS_INTERNAL_ERROR:
      out &lt;&lt; "CUDNN_STATUS_INTERNAL_ERROR";
      break;
    case CUDNN_STATUS_INVALID_VALUE:
      out &lt;&lt; "CUDNN_STATUS_INVALID_VALUE";
      break;
    case CUDNN_STATUS_ARCH_MISMATCH:
      out &lt;&lt; "CUDNN_STATUS_ARCH_MISMATCH";
      break;
    case CUDNN_STATUS_MAPPING_ERROR:
      out &lt;&lt; "CUDNN_STATUS_MAPPING_ERROR";
      break;
    case CUDNN_STATUS_EXECUTION_FAILED:
      out &lt;&lt; "CUDNN_STATUS_EXECUTION_FAILED";
      break;
    case CUDNN_STATUS_NOT_SUPPORTED:
      out &lt;&lt; "CUDNN_STATUS_NOT_SUPPORTED";
      break;
    case CUDNN_STATUS_LICENSE_ERROR:
      out &lt;&lt; "CUDNN_STATUS_LICENSE_ERROR";
      break;
    case CUDNN_STATUS_RUNTIME_PREREQUISITE_MISSING:
      out &lt;&lt; "CUDNN_STATUS_RUNTIME_PREREQUISITE_MISSING";
      break;
    case CUDNN_STATUS_RUNTIME_IN_PROGRESS:
      out &lt;&lt; "CUDNN_STATUS_RUNTIME_IN_PROGRESS";
      break;
    case CUDNN_STATUS_RUNTIME_FP_OVERFLOW:
      out &lt;&lt; "CUDNN_STATUS_RUNTIME_FP_OVERFLOW";
      break;
    case CUDNN_STATUS_VERSION_MISMATCH:
      out &lt;&lt; "CUDNN_STATUS_VERSION_MISMATCH";
      break;
    default:
      std::cerr &lt;&lt; "Invalid value Status Value" &lt;&lt; std::endl;
      exit(1);
  }
  return out;
}

std::ostream &amp;operator&lt;&lt;(std::ostream &amp;out, const cudnnDeterminism_t &amp;determinism) {
  out &lt;&lt; "Determinism: ";
  switch (determinism)
  {
    case CUDNN_NON_DETERMINISTIC:
      out &lt;&lt; "CUDNN_NON_DETERMINISTIC";
      break;
    case CUDNN_DETERMINISTIC:
      out &lt;&lt; "CUDNN_DETERMINISTIC";
      break;
    default:
      std::cerr &lt;&lt; "Underfined Value: " &lt;&lt; static_cast&lt;int&gt;(determinism);
  }
  return out;
}

std::ostream &amp;operator&lt;&lt;(std::ostream &amp;out, const cudnnMathType_t &amp;mathType) {
  out &lt;&lt; "MathType: ";
  switch (mathType)
  {
  case CUDNN_DEFAULT_MATH:
    out &lt;&lt; "CUDNN_DEFAULT_MATH";
    break;
  case CUDNN_TENSOR_OP_MATH:
    out &lt;&lt; "CUDNN_TENSOR_OP_MATH";
    break;
  case CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION:
    out &lt;&lt; "CUDNN_TENSOR_OP_MATH_ALLOW_CONVERSION";
    break;
  case CUDNN_FMA_MATH:
    out &lt;&lt; "CUDNN_FMA_MATH";
    break;
  default:
      std::cerr &lt;&lt; "Invalid (" &lt;&lt; mathType 
      &lt;&lt; ")value Algorithm Memory Type" &lt;&lt; std::endl;
      exit(1);
  }
  return out;
}

std::ostream&amp; operator&lt;&lt;(std::ostream &amp;out, const cudnnConvolutionFwdAlgoPerf_t &amp;fwdAlgoPert) {
  out &lt;&lt; fwdAlgoPert.algo;
  out &lt;&lt; "\n\t" &lt;&lt; fwdAlgoPert.status;
  out &lt;&lt; "\n\tTime: " &lt;&lt; fwdAlgoPert.time;
  out &lt;&lt; "\n\tMemory: " &lt;&lt; fwdAlgoPert.memory;
  out &lt;&lt; "\n\t" &lt;&lt; fwdAlgoPert.determinism;
  out &lt;&lt; "\n\t" &lt;&lt; fwdAlgoPert.mathType;
  out &lt;&lt; std::endl;
  return out;
}

std::ostream&amp; operator&lt;&lt;(std::ostream &amp;out, const cudnnConvolutionBwdFilterAlgoPerf_t &amp;bwdFilterAlgoPerf) {
  out &lt;&lt; bwdFilterAlgoPerf.algo;
  out &lt;&lt; "\n\t" &lt;&lt; bwdFilterAlgoPerf.status;
  out &lt;&lt; "\n\tTime: " &lt;&lt; bwdFilterAlgoPerf.time;
  out &lt;&lt; "\n\tMemory: " &lt;&lt; bwdFilterAlgoPerf.memory;
  out &lt;&lt; "\n\t" &lt;&lt; bwdFilterAlgoPerf.determinism;
  out &lt;&lt; "\n\t" &lt;&lt; bwdFilterAlgoPerf.mathType;
  out &lt;&lt; std::endl;
  return out;
}

std::ostream&amp; operator&lt;&lt;(std::ostream &amp;out, const cudnnConvolutionBwdDataAlgoPerf_t &amp;bwdDataAlgoPerf) {
  out &lt;&lt; bwdDataAlgoPerf.algo;
  out &lt;&lt; "\n\t" &lt;&lt; bwdDataAlgoPerf.status;
  out &lt;&lt; "\n\tTime: " &lt;&lt; bwdDataAlgoPerf.time;
  out &lt;&lt; "\n\tMemory: " &lt;&lt; bwdDataAlgoPerf.memory;
  out &lt;&lt; "\n\t" &lt;&lt; bwdDataAlgoPerf.determinism;
  out &lt;&lt; "\n\t" &lt;&lt; bwdDataAlgoPerf.mathType;
  out &lt;&lt; std::endl;
  return out;
}

void initialize_descriptors(const at::Tensor &amp;input, const at::Tensor &amp;weight, const at::Tensor &amp;output,
                            c10::ArrayRef&lt;int64_t&gt; &amp;stride,
                            c10::ArrayRef&lt;int64_t&gt; &amp;padding,
                            c10::ArrayRef&lt;int64_t&gt; &amp;dilation,
                            cudnnDescriptors_t &amp;desc)
{
  /*****************************************************************************
   * 1. Initializing Descriptors
   ****************************************************************************/
  assert(input.dim() == 4);
  checkCUDNN(cudnnCreateTensorDescriptor(&amp;desc.input));
  checkCUDNN(cudnnSetTensor4dDescriptor(desc.input,
                                        /*format=*/CUDNN_TENSOR_NCHW,
                                        /*dataType=*/at::native::getCudnnDataTypeFromScalarType(input.scalar_type()),
                                        /*batch_size=*/input.size(0),
                                        /*channels=*/input.size(1),
                                        /*image_height=*/input.size(2),
                                        /*image_width=*/input.size(3)));

  assert(weight.dim() == 4);
  checkCUDNN(cudnnCreateFilterDescriptor(&amp;desc.weight));
  checkCUDNN(cudnnSetFilter4dDescriptor(desc.weight,
                                        /*dataType=*/at::native::getCudnnDataTypeFromScalarType(weight.scalar_type()),
                                        /*format=*/CUDNN_TENSOR_NCHW,
                                        /*out_channels=*/weight.size(0),
                                        /*in_channels=*/weight.size(1),
                                        /*kernel_height=*/weight.size(2),
                                        /*kernel_width=*/weight.size(3)));

  checkCUDNN(cudnnCreateConvolutionDescriptor(&amp;desc.convolution));
  checkCUDNN(cudnnSetConvolution2dDescriptor(desc.convolution,
                                             /*pad_height=*/padding[0],
                                             /*pad_width=*/padding[1],
                                             /*vertical_stride=*/stride[0],
                                             /*horizontal_stride=*/stride[1],
                                             /*dilation_height=*/dilation[0],
                                             /*dilation_width=*/dilation[1],
                                             /*mode=*/CUDNN_CROSS_CORRELATION,
                                             /*computeType=*/at::native::getCudnnDataTypeFromScalarType(output.scalar_type())));

  int batch_size{0}, channels{0}, height{0}, width{0};
  checkCUDNN(cudnnGetConvolution2dForwardOutputDim(desc.convolution,
                                                   desc.input,
                                                   desc.weight,
                                                   &amp;batch_size,
                                                   &amp;channels,
                                                   &amp;height,
                                                   &amp;width));

  assert(batch_size == output.size(0) &amp;&amp; channels == output.size(1) &amp;&amp;
    height == output.size(2) &amp;&amp; width == output.size(3));

  checkCUDNN(cudnnCreateTensorDescriptor(&amp;desc.output));
  checkCUDNN(cudnnSetTensor4dDescriptor(desc.output,
                                        /*format=*/CUDNN_TENSOR_NCHW,
                                        /*dataType=*/at::native::getCudnnDataTypeFromScalarType(output.scalar_type()),
                                        /*batch_size=*/batch_size,
                                        /*channels=*/channels,
                                        /*image_height=*/height,
                                        /*image_width=*/width));
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m)
{
  m.def("convolution", &amp;convolution, "convolution");
}

#endif
</code></pre>
<p>I noticed that I had the wrong format for some of the tensors. Now, I have all of them as <code>CUDNN_TENSOR_NCHW</code>. It did show <code>CUDNN_CONVOLUTION_FWD_ALGO_FFT</code> as supported now, however, I am still getting a <code>Segmentation fault (core dumped)</code> error.</p>
<p>The backtracking is now different though:</p>
<p><strong><code>gdb</code> backtracing:</strong>*</p>
<pre><code class="lang-auto">Thread 1 "python" received signal SIGSEGV, Segmentation fault.
___pthread_mutex_lock (mutex=0x0) at ./nptl/pthread_mutex_lock.c:80
80	./nptl/pthread_mutex_lock.c: No such file or directory.
(gdb) br
Breakpoint 1 at 0x7ffff7d19f74: file ./nptl/pthread_mutex_lock.c, line 80.
(gdb) bt
#0  ___pthread_mutex_lock (mutex=0x0) at ./nptl/pthread_mutex_lock.c:80
#1  0x00007ffe6065cb1e in cudnn::ops::GetInternalStreams(cudnnContext*, int, CUstream_st**) () from /usr/local/cuda/lib64/libcudnn_ops_infer.so.8
#2  0x00007ffe35df22c5 in cudnn::cnn::Fft_2d_Non_Tiled&lt;float&gt;::execute_internal_impl(cudnn::backend::VariantPack const&amp;, CUstream_st*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#3  0x00007ffe359d3e71 in cudnn::cnn::EngineInterface::execute(cudnn::backend::VariantPack const&amp;, CUstream_st*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#4  0x00007ffe359fcb75 in cudnn::cnn::EngineContainer&lt;(cudnnBackendEngineName_t)3, 4096ul&gt;::execute_internal_impl(cudnn::backend::VariantPack const&amp;, CUstream_st*) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#5  0x00007ffe359d3e71 in cudnn::cnn::EngineInterface::execute(cudnn::backend::VariantPack const&amp;, CUstream_st*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#6  0x00007ffe35b46b1e in cudnn::cnn::AutoTransformationExecutor::execute_pipeline(cudnn::cnn::EngineInterface&amp;, cudnn::backend::VariantPack const&amp;, CUstream_st*) const () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#7  0x00007ffe35b46c23 in cudnn::cnn::BatchPartitionExecutor::operator()(cudnn::cnn::EngineInterface&amp;, cudnn::cnn::EngineInterface*, cudnn::backend::VariantPack const&amp;, CUstream_st*) const () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#8  0x00007ffe35b59d66 in cudnn::cnn::GeneralizedConvolutionEngine&lt;cudnn::cnn::EngineContainer&lt;(cudnnBackendEngineName_t)3, 4096ul&gt; &gt;::execute_internal_impl(cudnn::backend::VariantPack const&amp;, CUstream_st*) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#9  0x00007ffe359d3e71 in cudnn::cnn::EngineInterface::execute(cudnn::backend::VariantPack const&amp;, CUstream_st*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#10 0x00007ffe359ef9e8 in cudnn::backend::execute(cudnnContext*, cudnn::backend::ExecutionPlan&amp;, cudnn::backend::VariantPack&amp;) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#11 0x00007ffe35cd3a19 in cudnn::backend::EnginesAlgoMap&lt;cudnnConvolutionFwdAlgo_t, 8&gt;::execute_wrapper(cudnnContext*, cudnnConvolutionFwdAlgo_t, cudnn::backend::ExecutionPlan&amp;, cudnn::backend::VariantPack&amp;) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#12 0x00007ffe35cc6aa8 in cudnn::backend::convolutionForward(cudnnContext*, void const*, cudnnTensorStruct const*, void const*, cudnnFilterStruct const*, void const*, cudnnConvolutionStruct const*, cudnnConvolutionFwdAlgo_t, void*, unsigned long, bool, void const*, void const*, void const*, cudnnActivationStruct const*, cudnnTensorStruct const*, void*) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#13 0x00007ffe35e2d839 in cudnn::cnn::convolutionForward(cudnnContext*, void const*, cudnnTensorStruct const*, void const*, cudnnFilterStruct const*, void const*, cudnnConvolutionStruct const*, cudnnConvolutionFwdAlgo_t, void*, unsigned long, void const*, cudnnTensorStruct const*, void*) ()
   from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#14 0x00007ffe35e3f426 in ?? () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#15 0x00007ffe35e3ff1c in cudnnStatus_t findAlgorithm&lt;find_get_conv_params, cudnnConvolutionStruct, cudnnConvolutionFwdAlgo_t, cudnnConvolutionFwdAlgoPerfStruct, 8, true&gt;(find_get_conv_params, int, int*, cudnnConvolutionFwdAlgoPerfStruct*) () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#16 0x00007ffe35e2dbc9 in cudnnFindConvolutionForwardAlgorithm () from /usr/local/cuda/lib64/libcudnn_cnn_infer.so.8
#17 0x00007fff17488d75 in convolution(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool) () from /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/cudnn_convolution.so
#18 0x00007fff174b3b26 in at::Tensor pybind11::detail::argument_loader&lt;int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool&gt;::call_impl&lt;at::Tensor, at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool), 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, pybind11::detail::void_type&gt;(at::Tensor (*&amp;)(int, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool), std::integer_sequence&lt;unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul&gt;, pybind11::detail::void_type&amp;&amp;) &amp;&amp; ()
   from /home/eduardoj/.cache/torch_extensions/py39_cu113/cudnn_convolution/cudnn_convolution.so
#19 0x00007fff174af5bf in std::enable_if&lt;!std::is_void&lt;at::Tensor&gt;::value, at::Tensor&gt;::type pybind11::detail::argument_loader&lt;int, at::Tensor const&amp;, at::Te--Type &lt;RET&gt; for more, q to quit, c to continue without paging--
</code></pre>

eduardo4jesus: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I am still facing this Segmentation Fault back tracing when using <code>CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD_NONFUSED</code>.</p>
<p>I noticed that I have two different versions of the <code>libcublas</code>. It appears <code>cudnn</code> is invoking the one used by <code>torch</code>, instead of the one installed with <code>cudnn</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8efaa17a4ac8685027a2562581a373e1489d1893.png" data-download-href="https://discuss.pytorch.org/uploads/default/8efaa17a4ac8685027a2562581a373e1489d1893" title="Screen Shot 2022-06-14 at 11.28.46 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8efaa17a4ac8685027a2562581a373e1489d1893_2_690x437.png" alt="Screen Shot 2022-06-14 at 11.28.46 PM" data-base62-sha1="koQM2afWcWzCF4GpeiHYFOnUxi3" width="690" height="437" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8efaa17a4ac8685027a2562581a373e1489d1893_2_690x437.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8efaa17a4ac8685027a2562581a373e1489d1893_2_1035x655.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8efaa17a4ac8685027a2562581a373e1489d1893_2_1380x874.png 2x" data-dominant-color="F7F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-06-14 at 11.28.46 PM</span><span class="informations">1388×881 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there a way to have <code>cudnn</code> pointing to <code>/usr/local/cuda/lib64/libcublas.so.11</code> instead of <code>~/miniconda3/lib/libcublas.so.11</code>?</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="eduardo4jesus" data-post="14" data-topic="101596">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eduardo4jesus/48/20203_2.png" class="avatar"> eduardo4jesus:</div>
<blockquote>
<p>It appears <code>cudnn</code> is invoking the one used by <code>torch</code>, instead of the one installed with <code>cudnn</code>.</p>
<p>Is there a way to have <code>cudnn</code> pointing to <code>/usr/local/cuda/lib64/libcublas.so.11</code> instead of <code>~/miniconda3/lib/libcublas.so.11</code>?</p>
</blockquote>
</aside>
<p>I think it might depend how you are building the extension and which libraries are found/used.<br>
Given it seems you are hitting a cuDNN issue, it might be easier to rip out the code and write a standalone cuDNN version first using their docs and examples to make sure you are not mixing up different issues.</p>

eduardo4jesus: <p>I did have standalone cuDNN code (<a href="https://gist.github.com/eduardo4jesus/33ef6d8696e8af70a3046e9f364a65f8" rel="noopener nofollow ugc">in here</a>) that works just fine, including for <code>CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD</code>.</p>
<p>At this  point I am looking for a way to change this</p>
<pre><code class="lang-auto">cudnn_convolution = load(
  name="cudnn_convolution",
  sources=["cudnn_convolution.cpp", "cudnn_utils.cpp"],
  extra_ldflags = ["-lcudnn"],
  with_cuda=True,
  verbose=True
)
</code></pre>
<p>So that, when compiling, I wouldn’t have <code>miniconda/libs</code> preceding <code>cuda/libs64</code>.</p>
<pre><code class="lang-auto">[1/3] c++ -MMD -MF cudnn_utils.o.d -DTORCH_EXTENSION_NAME=cudnn_convolution -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/include -isystem /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/include/TH -isystem /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/eduardoj/miniconda3/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/eduardoj/Repo/eduardo4jesus/PyTorch-cuFFT-Convolution/cudnn_utils.cpp -o cudnn_utils.o 
[2/3] c++ -MMD -MF cudnn_convolution.o.d -DTORCH_EXTENSION_NAME=cudnn_convolution -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/include -isystem /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/include/TH -isystem /home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/eduardoj/miniconda3/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/eduardoj/Repo/eduardo4jesus/PyTorch-cuFFT-Convolution/cudnn_convolution.cpp -o cudnn_convolution.o 
[3/3] c++ cudnn_convolution.o cudnn_utils.o -shared -lcudnn -L/home/eduardoj/miniconda3/lib/python3.9/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda_cu -ltorch_cuda_cpp -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o cudnn_convolution.so
</code></pre>
<p>I am hoping that by changing the order the issue would get fixed.</p>

ptrblck: <p>Could you run a quick check by rebuilding PyTorch from source using your system CUDA toolkit and cuDNN? This would then dynamically link to these libraries and the extension shouldn’t be able to use any libraries shipped in the binaries (as they are not installed).<br>
Maybe a docker container could work as it would already come with the properly installed libraries and you could then rebuild PyTorch there.<br>
If this works, it could indeed point to a library mismatch.</p>

eduardo4jesus: <p>Despite not being a quick building, that solved my issue! Many thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
<p>The only issue I am having now is on why <code>CUDNN_CONVOLUTION_FWD_ALGO_FFT</code> yields <code>CUDNN_STATUS_NOT_SUPPORTED</code> for the following configuration</p>
<pre><code class="lang-auto">B, F, C = 256, 512, 512
N, K, O = 32, 5, 32
padding  = (2, 2)

# create dummy input, convolutional weights and bias
input  = torch.zeros(B, C, N, N).to('cuda')
weight = torch.zeros(F, C, K, K).to('cuda')
output = torch.zeros(B, F, O, O).to('cuda')
stride   = (1, 1)
dilation = (1, 1)
groups   = 1
</code></pre>
<p>According to Table 22 off <a href="https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnConvolutionForward" rel="noopener nofollow ugc">the documentation</a>, it should work just fine.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69219dafd418034b765205de21f8db8c26d7b1bd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/69219dafd418034b765205de21f8db8c26d7b1bd" title="Screen Shot 2022-06-15 at 4.09.07 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69219dafd418034b765205de21f8db8c26d7b1bd_2_690x438.jpeg" alt="Screen Shot 2022-06-15 at 4.09.07 PM" data-base62-sha1="f029GH65tvXPSNWeL1Sy3gwO1Y1" width="690" height="438" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69219dafd418034b765205de21f8db8c26d7b1bd_2_690x438.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69219dafd418034b765205de21f8db8c26d7b1bd_2_1035x657.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69219dafd418034b765205de21f8db8c26d7b1bd_2_1380x876.jpeg 2x" data-dominant-color="EDEEED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-06-15 at 4.09.07 PM</span><span class="informations">3584×2278 743 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/ballot_box_with_check.png?v=12" title=":ballot_box_with_check:" class="emoji" alt=":ballot_box_with_check:" loading="lazy" width="20" height="20"> <code>xDesc</code> (height and width) + 2*<code>padding</code> &lt; 256</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/ballot_box_with_check.png?v=12" title=":ballot_box_with_check:" class="emoji" alt=":ballot_box_with_check:" loading="lazy" width="20" height="20"> <code>stride</code> == 1</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/ballot_box_with_check.png?v=12" title=":ballot_box_with_check:" class="emoji" alt=":ballot_box_with_check:" loading="lazy" width="20" height="20"> <code>wDesk</code> (height and width) &gt; <code>padding</code></p>

ptrblck: <p>OK, great to hear it’s working with the matching libraries.</p>
<p>Could you forward me the cuDNN logs for the <code>CUDNN_STATUS_NOT_SUPPORTED</code> use case so that I could take a look at them (with our cuDNN team), please?<br>
You can find the instructions <a href="https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#api-logging">here</a> with the needed env variables.</p>

eduardo4jesus: <p>Thank you for the link.</p>
<p>Here is the log file.</p>
<p><strong>$cat cuDNN.log</strong></p>
<pre><code class="lang-auto">
I! CuDNN (v8401) function cudnnCreate() called:
i!     handle: location=host; addr=0x55bcb07b1950;
i! Time: 2022-06-16T13:10:01.736287 (0d+0h+0m+0s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnSetStream() called:
i!     handle: type=cudnnHandle_t; streamId=(nil) (defaultStream);
i!     streamId: type=cudaStream_t; streamId=(nil) (defaultStream);
i! Time: 2022-06-16T13:10:02.406407 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=0; Handle=0x55bce02ec740; StreamId=(nil) (defaultStream).


I! CuDNN (v8401) function cudnnCreateTensorDescriptor() called:
i! Time: 2022-06-16T13:10:02.406455 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnSetTensor4dDescriptor() called:
i!     format: type=cudnnTensorFormat_t; val=CUDNN_TENSOR_NCHW (0);
i!     dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!     n: type=int; val=256;
i!     c: type=int; val=512;
i!     h: type=int; val=32;
i!     w: type=int; val=32;
i! Time: 2022-06-16T13:10:02.406468 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnSetTensor4dDescriptorEx() called:
i!     dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!     n: type=int; val=256;
i!     c: type=int; val=512;
i!     h: type=int; val=32;
i!     w: type=int; val=32;
i!     nStride: type=int; val=524288;
i!     cStride: type=int; val=1024;
i!     hStride: type=int; val=32;
i!     wStride: type=int; val=1;
i! Time: 2022-06-16T13:10:02.406473 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnCreateFilterDescriptor() called:
i! Time: 2022-06-16T13:10:02.406485 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnSetFilter4dDescriptor() called:
i!     dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!     format: type=cudnnTensorFormat_t; val=CUDNN_TENSOR_NCHW (0);
i!     k: type=int; val=512;
i!     c: type=int; val=512;
i!     h: type=int; val=5;
i!     w: type=int; val=5;
i! Time: 2022-06-16T13:10:02.406490 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnCreateConvolutionDescriptor() called:
i!     convDesc: location=host; addr=0x7ffdc81c24d0;
i! Time: 2022-06-16T13:10:02.426075 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnSetConvolution2dDescriptor() called:
i!     convDesc: location=host; addr=0x55bd1e659f50;
i!     pad_h: type=int; val=2;
i!     pad_w: type=int; val=2;
i!     u: type=int; val=1;
i!     v: type=int; val=1;
i!     dilation_h: type=int; val=1;
i!     dilation_w: type=int; val=1;
i!     mode: type=cudnnConvolutionMode_t; val=CUDNN_CROSS_CORRELATION (1);
i!     dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i! Time: 2022-06-16T13:10:02.426160 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnGetConvolution2dForwardOutputDim() called:
i!     convDesc: type=cudnnConvolutionDescriptor_t:
i!         mode: type=cudnnConvolutionMode_t; val=CUDNN_CROSS_CORRELATION (1);
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         mathType: type=cudnnMathType_t; val=CUDNN_DEFAULT_MATH (0);
i!         reorderType: type=int; val=0;
i!         arrayLength: type=int; val=2;
i!         padA: type=int; val=[2,2];
i!         strideA: type=int; val=[1,1];
i!         dilationA: type=int; val=[1,1];
i!         groupCount: type=int; val=1;
i!     inputTensorDesc: type=cudnnTensorDescriptor_t:
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         nbDims: type=int; val=4;
i!         dimA: type=int; val=[256,512,32,32];
i!         strideA: type=int; val=[524288,1024,32,1];
i!     filterDesc: type=cudnnFilterDescriptor_t:
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         vect: type=int; val=0;
i!         nbDims: type=int; val=4;
i!         dimA: type=int; val=[512,512,5,5];
i!         format: type=cudnnTensorFormat_t; val=CUDNN_TENSOR_NCHW (0);
i!     n: location=host; addr=0x7ffdc81c2384;
i!     c: location=host; addr=0x7ffdc81c2388;
i!     h: location=host; addr=0x7ffdc81c238c;
i!     w: location=host; addr=0x7ffdc81c2390;
i! Time: 2022-06-16T13:10:02.426175 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnCreateTensorDescriptor() called:
i! Time: 2022-06-16T13:10:02.426191 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnSetTensor4dDescriptor() called:
i!     format: type=cudnnTensorFormat_t; val=CUDNN_TENSOR_NCHW (0);
i!     dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!     n: type=int; val=256;
i!     c: type=int; val=512;
i!     h: type=int; val=32;
i!     w: type=int; val=32;
i! Time: 2022-06-16T13:10:02.426197 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnSetTensor4dDescriptorEx() called:
i!     dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!     n: type=int; val=256;
i!     c: type=int; val=512;
i!     h: type=int; val=32;
i!     w: type=int; val=32;
i!     nStride: type=int; val=524288;
i!     cStride: type=int; val=1024;
i!     hStride: type=int; val=32;
i!     wStride: type=int; val=1;
i! Time: 2022-06-16T13:10:02.426202 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


I! CuDNN (v8401) function cudnnGetConvolutionForwardWorkspaceSize() called:
i!     handle: type=cudnnHandle_t; streamId=(nil) (defaultStream);
i!     xDesc: type=cudnnTensorDescriptor_t:
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         nbDims: type=int; val=4;
i!         dimA: type=int; val=[256,512,32,32];
i!         strideA: type=int; val=[524288,1024,32,1];
i!     wDesc: type=cudnnFilterDescriptor_t:
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         vect: type=int; val=0;
i!         nbDims: type=int; val=4;
i!         dimA: type=int; val=[512,512,5,5];
i!         format: type=cudnnTensorFormat_t; val=CUDNN_TENSOR_NCHW (0);
i!     convDesc: type=cudnnConvolutionDescriptor_t:
i!         mode: type=cudnnConvolutionMode_t; val=CUDNN_CROSS_CORRELATION (1);
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         mathType: type=cudnnMathType_t; val=CUDNN_DEFAULT_MATH (0);
i!         reorderType: type=int; val=0;
i!         arrayLength: type=int; val=2;
i!         padA: type=int; val=[2,2];
i!         strideA: type=int; val=[1,1];
i!         dilationA: type=int; val=[1,1];
i!         groupCount: type=int; val=1;
i!     yDesc: type=cudnnTensorDescriptor_t:
i!         dataType: type=cudnnDataType_t; val=CUDNN_DATA_FLOAT (0);
i!         nbDims: type=int; val=4;
i!         dimA: type=int; val=[256,512,32,32];
i!         strideA: type=int; val=[524288,1024,32,1];
i!     algo: type=cudnnConvolutionFwdAlgo_t; val=CUDNN_CONVOLUTION_FWD_ALGO_FFT (4);
i!     sizeInBytes: location=host; addr=0x7ffdc81c2488;
i! Time: 2022-06-16T13:10:02.426253 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=0; Handle=0x55bce02ec740; StreamId=(nil) (defaultStream).


I! CuDNN (v8401) function cudnnGetErrorString() called:
i!     status: type=int; val=9;
i! Time: 2022-06-16T13:10:02.426424 (0d+0h+0m+1s since start)
i! Process=4107; Thread=4107; GPU=NULL; Handle=NULL; StreamId=NULL.


</code></pre>
<p>Following the instructions you sent, the log above was generated after executing these instructions:</p>
<pre><code class="lang-auto">export CUDNN_LOGINFO_DBG=1 CUDNN_LOGWARN_DBG=1 CUDNN_LOGERR_DBG=1 CUDNN_LOGDEST_DBG=cuDNN.log
python example.py
</code></pre>