CUDNN_STATUS_NOT_SUPPORTED error occurs when apply autograd.grad to compute high-order differentiation

username: Esparami
cooked: <p>Hi, I’m working on my project involving the great high-order differentiation operator. It works fine under CPU stride, however, when I try to work on GPU strides, I encountered the following error</p>
<p><strong>RuntimeError: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.</strong></p>
<p>I’m working with <code>python3.6.1</code>, <code>torch 0.2.0_4</code>, <code>CUDA Version 8.0.44</code> and <code>cudnn 6021</code>, which I think are all updated. I track back the error, and it shows the error shows up whenever I use <strong>conv2d</strong> layer.</p>
<p>I tried <code>torch.backends.cudnn.enabled=False</code> to get rid of the error but then no acceleration happens seems.</p>
<p>Some other solution like <code>torch.backends.cudnn.benchmark=True</code> is not working in my case.</p>
<p>As for now, I’m not able to find a way to resolve this, would anyone help me to figure it out? thank you in advance,</p>
<p>I created the following demo code to show my issue, please let me know if you need anything more from me <img src="//discuss.pytorch.org/images/emoji/apple/blush.png?v=5" title=":blush:" class="emoji" alt=":blush:"></p>
<pre><code class="lang-python">import torch
import torch.nn as nn
from torch.autograd import Variable, grad
import torch.utils.data as Data
import torchvision

train_data = torchvision.datasets.MNIST(
    root='./mnist/',
    train=True,
    transform=torchvision.transforms.ToTensor(),
    download=False,
)

train_loader = Data.DataLoader(
    dataset=train_data, batch_size=50, shuffle=True, num_workers=2)


class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(nn.Conv2d(1, 16, 5, 1, 2))
        self.out = nn.Linear(16 * 28 * 28, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = x.view(x.size(0), -1)
        output = self.out(x)
        return output, x


cnn = CNN()
cnn.cuda()

loss_func = nn.CrossEntropyLoss()

for step, (data, label) in enumerate(train_loader):
    input = Variable(data).cuda()
    target = Variable(label).cuda()

    output = cnn(input)[0]
    loss = loss_func(output, target)

    params = cnn.parameters()
    g = grad(loss, params, create_graph=True)

    g_sum = 0
    for g_para in g:
        g_sum += g_para.sum()

    params = cnn.parameters()
    hv = grad(g_sum, params, create_graph=True)

    break
</code></pre>
<p>The error message is as following in my setting:</p>
<pre><code class="lang-bash">Traceback (most recent call last):
  File "gpu_demo.py", line 67, in &lt;module&gt;
    hv = grad(g_sum, params, create_graph=True)
  File "/root/miniconda3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 153, in grad
    inputs, only_inputs)
RuntimeError: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.
</code></pre>

username: smth
cooked: <p>Looks like a bug, we’ll get this fixed. Opened an issue here to track this: <a href="https://github.com/pytorch/pytorch/issues/3064">https://github.com/pytorch/pytorch/issues/3064</a></p>

username: Hsi-Wei_Hsieh
cooked: <p>Hi, this issue also happens for Conv1d, is that also fixed?</p>

username: ptrblck
cooked: <p>Could you post the <code>nn.Conv1d</code> setup, the input shape, as well as the CUDA and cudnn version so that we could reproduce it, please?</p>