Cudnn Error with cudnn convolution backward function

zhryang: <p>Hi, everyone,<br>
When I writing cpp extension using cudnn_convolution_backward, I meet Cudnn Error. My cpp extension looks like this:</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#include &lt;ATen/Config.h&gt;
#include &lt;cstdio&gt;
#include &lt;iostream&gt;
#include &lt;array&gt;


at::Tensor conv2d(
    const at::Tensor&amp; input,
    const at::Tensor&amp; weight,
    const at::Tensor&amp; bias,
    int64_t stride_weight,
    int64_t stride_height,
    int64_t padding_weight,
    int64_t padding_height,
    int64_t dilation_weight,
    int64_t dilation_height,
    int64_t groups,
    bool benchmark,
    bool deterministic) {

    std::array&lt;int64_t, 2&gt; arr_stride{stride_weight, stride_height};
    std::array&lt;int64_t, 2&gt; arr_padding{padding_weight, padding_height};
    std::array&lt;int64_t, 2&gt; arr_dilation{dilation_weight, dilation_height};

    c10::IntArrayRef stride(arr_stride);
    c10::IntArrayRef padding(arr_padding);
    c10::IntArrayRef dilation(arr_dilation);

    // printf("--------------------------------hello conv2d----------------------------------------\n");

    return at::cudnn_convolution(
        input,
        weight,
        bias,
        padding,
        stride,
        dilation,
        groups,
        benchmark,
        deterministic);
}

std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt; conv2d_backward(
    const at::Tensor&amp; self,
    const at::Tensor&amp; grad_output,
    const at::Tensor&amp; weight,
    int64_t stride_width,
    int64_t stride_height,
    int64_t padding_width,
    int64_t padding_height,
    int64_t dilation_width,
    int64_t dilation_height,
    int64_t groups,
    bool benchmark,
    bool deterministic,
    std::array&lt;bool, 3&gt; output_mask) {

    // printf("--------------------------------hello conv2d backward----------------------------------------\n");
    
    std::array&lt;int64_t, 2&gt; arr_stride{stride_width, stride_height};
    std::array&lt;int64_t, 2&gt; arr_padding{padding_width, padding_height};
    std::array&lt;int64_t, 2&gt; arr_dilation{dilation_width, dilation_height};

    c10::IntArrayRef stride(arr_stride);
    c10::IntArrayRef padding(arr_padding);
    c10::IntArrayRef dilation(arr_dilation);

    return at::cudnn_convolution_backward(
        self,
        grad_output,
        weight,
        padding,
        stride,
        dilation,
        groups,
        benchmark,
        deterministic,
        output_mask);
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
    m.def("conv2d", &amp;conv2d, "2d convolution");
    m.def("conv2d_backward", &amp;conv2d_backward, "2d convolution backward");
}
</code></pre>
<p>I test extensions like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
from torch.autograd import Function
import math
import cudnn_conv2d
from torch.utils.cpp_extension import load

cudnn_convolution = load(name="cudnn_convolution", sources=["cudnn_convolution.cpp"], verbose=True)

input  = torch.zeros(1, 1, 4, 4).to('cuda')
weight = torch.zeros(1, 1, 3, 3).to('cuda')
bias   = torch.zeros(1).to('cuda')

stride   = (1, 1)
padding  = (0, 0)
dilation = (1, 1)
groups   = 1

# # compute the result of convolution
output = cudnn_convolution.conv2d(input, weight, bias, stride[0], stride[1], padding[0], padding[1], dilation[0], dilation[1], groups, False, False)

grad_output = torch.zeros(1, 1, 2, 2).to('cuda')

grad_i, grad_w = cudnn_convolution.conv2d_backward(input, grad_output, weight, padding[0], padding[1], stride[0], stride[1], dilation[0], dilation[1], groups, False, False, [True, True, False])

print(grad_weight.shape)
print(grad_input.shape)
</code></pre>
<p>I receive the following error:</p>
<pre><code class="lang-auto">Loading extension module cudnn_convolution...
Traceback (most recent call last):
  File "example.py", line 30, in &lt;module&gt;
    grad_i, grad_w = cudnn_convolution.conv2d_backward(input, grad_output, weight, padding[0], padding[1], stride[0], stride[1], dilation[0], dilation[1], groups, False, False, [True, True, False])
RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM (set at /pytorch/aten/src/ATen/cudnn/Descriptors.h:157)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x33 (0x7ff9e40fe813 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0x4151585 (0x7ff983527585 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #2: &lt;unknown function&gt; + 0x414a4b5 (0x7ff9835204b5 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #3: &lt;unknown function&gt; + 0x414b5bb (0x7ff9835215bb in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #4: at::native::cudnn_convolution_backward_input(c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool) + 0xf7 (0x7ff983521ba7 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #5: &lt;unknown function&gt; + 0x41a9805 (0x7ff98357f805 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #6: &lt;unknown function&gt; + 0x3918558 (0x7ff982cee558 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #7: at::native::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt;) + 0x579 (0x7ff98351da99 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #8: &lt;unknown function&gt; + 0x41a9b0c (0x7ff98357fb0c in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #9: &lt;unknown function&gt; + 0x3918505 (0x7ff982cee505 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #10: &lt;unknown function&gt; + 0x3bd6e11 (0x7ff982face11 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #11: &lt;unknown function&gt; + 0x3918505 (0x7ff982cee505 in /home/yangkan/.local/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #12: std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt; c10::KernelFunction::callUnboxedOnly&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt; &gt;(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt;) const + 0x1ac (0x7ff977ad1d74 in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #13: c10::impl::OperatorEntry::callUnboxedOnly&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt; &gt;(c10::TensorTypeId, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt;) const::{lambda(c10::DispatchTable const&amp;)#1}::operator()(c10::DispatchTable const&amp;) const + 0x162 (0x7ff977acdd52 in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #14: std::result_of&lt;c10::impl::OperatorEntry::callUnboxedOnly&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt; &gt;(c10::TensorTypeId, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt;) const::{lambda(c10::DispatchTable const&amp;)#1} (c10::DispatchTable const&amp;)&gt;::type c10::LeftRight&lt;c10::DispatchTable&gt;::read&lt;c10::impl::OperatorEntry::callUnboxedOnly&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt; &gt;(c10::TensorTypeId, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt;) const::{lambda(c10::DispatchTable const&amp;)#1}&gt;(c10::impl::OperatorEntry::callUnboxedOnly&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt; &gt;(c10::TensorTypeId, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt;) const::{lambda(c10::DispatchTable const&amp;)#1}&amp;&amp;) const + 0x128 (0x7ff977ad1fdc in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #15: std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt; c10::impl::OperatorEntry::callUnboxedOnly&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt; &gt;(c10::TensorTypeId, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt;) const + 0xc6 (0x7ff977acde44 in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #16: std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt; c10::Dispatcher::callUnboxedOnly&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor&gt;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt; &gt;(c10::OperatorHandle const&amp;, c10::TensorTypeId, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 3ul&gt;) const + 0x188 (0x7ff977ac9f1c in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #17: &lt;unknown function&gt; + 0x2c1be (0x7ff977abb1be in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #18: conv2d_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, long, long, long, long, long, long, long, bool, bool, std::array&lt;bool, 3ul&gt;) + 0x133 (0x7ff977abb524 in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #19: &lt;unknown function&gt; + 0x4f085 (0x7ff977ade085 in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #20: &lt;unknown function&gt; + 0x4b41d (0x7ff977ada41d in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #21: &lt;unknown function&gt; + 0x46fbb (0x7ff977ad5fbb in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #22: &lt;unknown function&gt; + 0x473d4 (0x7ff977ad63d4 in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #23: &lt;unknown function&gt; + 0x37b97 (0x7ff977ac6b97 in /tmp/torch_extensions/cudnn_convolution/cudnn_convolution.so)
frame #24: python3() [0x50ac25]
&lt;omitting python frames&gt;
frame #26: python3() [0x508245]
frame #28: python3() [0x635222]
frame #33: __libc_start_main + 0xe7 (0x7ff9ea18cb97 in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<p>I can not find bug in the code. Does anyboby know how to debug?</p>

zhryang: <p>cudnn_convolution function is correct. I can use cudnn_convolution function to compute the result of convolution. Error occur in the backward.</p>

ptrblck: <p>There are a couple of issues in the code:</p>
<ul>
<li>
<p>the <code>CUDNN_STATUS_BAD_PARAM</code> error is raised, since you are passing <code>padding[0]</code> and <code>padding[1]</code> before the strides, while your <code>conv2d_backward</code> expects the opposite</p>
</li>
<li>
<p>You are returning an <code>std::tuple</code> of three <code>at::Tensor</code>s, while <code>at::cudnn_convolution_backward</code> returns two</p>
</li>
<li>
<p><code>output_mask</code> is defined as <code>std::array&lt;bool,2&gt;</code>, while you are passing three bool values</p>
</li>
</ul>

zhryang: <p>Thanks, when I pass padding[0] ans padding[1] after the stride. The program is correct.</p>

zhryang: <p>I use cudnn_convolution_backward in ATen/NativeFunctions.h, that return an std::tuple of three at::Tensors, output_mask is defined as std::array&lt;bool, 3&gt;.<br>
Is the version of my pytorch different from yours? The version of pytorch is 1.4.</p>

ptrblck: <p>Not sure, but I used the definition in <a href="https://github.com/pytorch/pytorch/blob/b38d505e425fb32f95b7f6dbf79be93417624431/aten/src/ATen/native/cudnn/Conv.cpp#L1064-L1089">cudnn/Conv.cpp</a>, which seemed to work.<br>
If your code is working fine, ignore my comment. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

zhryang: <p>My code is flawed.When I use the code, I must pass bias, So I don’t set bias=False.I think your comment is perfect to solve this problem.I reconfirm the version of pytorch is 1.3.1. But I can not find ATen/native/cudnn/Conv.cpp in the installation path of pytorch(I use pip3 install torch torchvision --user). How do I solve this problem. Meanwhile, I meet a new Error, <a href="https://discuss.pytorch.org/t/runtimeerror-expected-tensor-s-dynamic-type-to-be-variable-not-tensor/76622">RuntimeError: Expected tensor’s dynamic type to be Variable, not Tensor </a>. I am confused and looking for a solution.<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Why is it flawed? It seems your other thread was answered already.</p>