CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input

davidlee: <p>Hi all,<br>
Im training CNN and error message like below comes up<br>
<strong>"CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input."</strong><br>
How can I resolve it?<br>
Is there any way that I can fix it by not using data loader?</p>
<p>Total Input : (117000, 51, 51), where batch size is 32.</p>
<p>Dataloader</p>
<p><code>trainloader = DataLoader(dataset=train_data, batch_size=32, num_workers=0, shuffle=True, drop_last=True)</code></p>
<p>Thank you very much in advance !!</p>

ptrblck: <p>Could you post the model definition so that we can reproduce this error?</p>

davidlee: <p>Hi ptrblck, Please refer to the below.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.cnn = nn.Sequential(

        # (51,51)
        torch.nn.Conv2d(1, 64, kernel_size=3, stride=1, padding=1, bias=True),
        torch.nn.BatchNorm2d(64),    
        torch.nn.ReLU(),
        torch.nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1, bias=True),
        torch.nn.BatchNorm2d(64),    
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=2, stride=2), # (25,25)

        torch.nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1, bias=True),
        torch.nn.BatchNorm2d(128),    
        torch.nn.ReLU(),
        torch.nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=True),
        torch.nn.BatchNorm2d(128),    
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=2, stride=2), # (12,12)

        torch.nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1, bias=True),
        torch.nn.BatchNorm2d(256),    
        torch.nn.ReLU(),
        torch.nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=True),
        torch.nn.BatchNorm2d(256),    
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=2, stride=2), # (6,6)
            
        torch.nn.Conv2d(256, 512, kernel_size=3, stride=1, padding=1, bias=True),
        torch.nn.BatchNorm2d(512),    
        torch.nn.ReLU(),
        torch.nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=True),
        torch.nn.BatchNorm2d(512),    
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=2, stride=2),
            
        )
        # (batch, 128,6,6)
        self.linear = nn.Sequential(
        torch.nn.Linear(512*3*3, 2048),
        torch.nn.ReLU(),
        torch.nn.Linear(2048, 1024),
        torch.nn.ReLU(),
        torch.nn.Linear(1024, 512),
        torch.nn.ReLU(),
        torch.nn.Linear(512, 256),
        torch.nn.ReLU(),
        torch.nn.Linear(256,3)
        )
    def forward(self, x):
        out = self.cnn(x)
        out = self.linear(out.view(-1, 128*6*6))
        
        return out</code></pre>

ptrblck: <p>Thanks for the code! Which PyTorch, CUDA and cudnn versions are you using?</p>

davidlee: <p>Thanks for quick reponse!<br>
I’m using CUDA!</p>

davidlee: <pre><code class="lang-auto">device = 'cuda:4' if torch.cuda.is_available() else 'cpu'
</code></pre>
<p>btw,<br>
If I check torch version(torch.version.cuda), it says ‘9.2.148’<br>
and also it says ‘7603’ if I check with torch.backends.cudnn.version()</p>

davidlee: <p>May I get any clues about this problem?<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I used the following input shape to reproduce this issue:</p>
<pre><code class="lang-python">model = Net().cuda()
x = torch.randn(32, 1, 51, 51).cuda()
output = model(x)
</code></pre>
<p>with PyTorch <code>1.4.0</code> using the CUDA10.1, CUDA10.0, and CUDA9.2 binaries and all run successful on my device.<br>
Could you post some information about your machine, e.g. which GPU you are using so that we can try to reproduce it again?</p>

davidlee: <p>Hi ptrblck,</p>
<p>Thank you very much for your response.<br>
Here is my code link contains this error (<a href="https://github.com/leesays92/NN-in-general/blob/master/Error1.ipynb" rel="nofollow noopener">https://github.com/leesays92/NN-in-general/blob/master/Error1.ipynb</a>). I would really appreciate it if you could look through it.</p>
<p>I’m currently using GPU for RTX 2080 Ti.<br>
I upgraded to PyTorch 1.4.0 (Which was originally 1.3.1) and it doesn’t work as well.<br>
The funny thing is that… If i reduce the total amount of data (11,7000 to 35,000), it works.</p>
<p>Thanks in advance!</p>

ptrblck: <p>Thanks for the code.<br>
It is indeed an out of memory issue.<br>
You are currently using a batch size of the complete validation dataset via:</p>
<pre><code class="lang-python">val_data = ValDataset()
valloader = DataLoader(dataset=val_data, batch_size=len(val_data), num_workers=0, shuffle=False, drop_last=False)
</code></pre>
<p>Based on the previous outputs of your notebook, the length of the validation dataset should be <code>130000 - 117000 = 13000</code>, which will use too much memory.<br>
I’ve tested different shapes and it seems a batch size of ~5000 would be the limit using approx. 44GB of your GPU.</p>

davidlee: <p>Oh, i thought it is different from OOM issue.<br>
Thank you very much for your kind help!!</p>