Error when using CUDNN

maplewizard: <SCODE> export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH
<ECODE> How can I find and solve the problem. <SCODE>Traceback (most recent call last):
  File "main.py", line 157, in <module>
    train()
  File "main.py", line 131, in train
    output, hidden = model(data, hidden)
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/disk1/workbench/learn/pytorch/examples/word_language_model/model.py", line 28, in forward
    output, hidden = self.rnn(emb, hidden)
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/modules/rnn.py", line 81, in forward
    return func(input, self.all_weights, hx)
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 235, in forward
    return func(input, *fargs, **fkwargs)
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/autograd/function.py", line 201, in _do_forward
    flat_output = super(NestedIOFunction, self)._do_forward(*flat_input)
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/autograd/function.py", line 223, in forward
    result = self.forward_extended(*nested_tensors)
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 180, in forward_extended
    cudnn.rnn.forward(self, input, hx, weight, output, hy)
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/backends/cudnn/rnn.py", line 184, in forward
    handle = cudnn.get_handle()
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/backends/cudnn/__init__.py", line 337, in get_handle
    handle = CuDNNHandle()
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/backends/cudnn/__init__.py", line 128, in __init__
    check_error(lib.cudnnCreate(ctypes.byref(ptr)))
  File "/data/disk1/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/backends/cudnn/__init__.py", line 324, in check_error
    raise CuDNNError(status)
torch.backends.cudnn.CuDNNError: 6: CUDNN_STATUS_ARCH_MISMATCH
Exception ctypes.ArgumentError: "argument 1: <type 'exceptions.TypeError'>: Don't know how to convert parameter 1" in <bound method CuDNNHandle.__del__ of <torch.backends.cudnn.CuDNNHandle instance at 0x7fa7707dd5f0>> ignored
<ECODE>

smth: are you sure you have the correct cudnn version? it needs to be R5 or R6

maplewizard: <SCODE>nvcc  --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17
<ECODE> How can I find more details about the cudnn?

apaszke: 

maplewizard: <SCODE>torch.backends.cudnn.version()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-de1bb2d5285f> in <module>()
----> 1 torch.backends.cudnn.version()

/global-hadoop/home/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/backends/cudnn/__init__.pyc in version()
     73 def version():
     74     if not lib:
---> 75         raise RuntimeError("cuDNN not initialized")
     76     if len(__cudnn_version) == 0:
     77         __cudnn_version.append(lib.cudnnGetVersion())

RuntimeError: cuDNN not initialized
<ECODE>

apaszke: Right, sorry. Do this please: <SCODE>print(torch.backends.cudnn.is_acceptable(torch.cuda.FloatTensor(1)))
print(torch.backends.cudnn.version())
<ECODE>

maplewizard: <SCODE>In [1]: import torch

In [2]: print(torch.backends.cudnn.is_acceptable(torch.cuda.FloatTensor(1)))
   ...: print(torch.backends.cudnn.version())
   ...:
True
5005
<ECODE> Now the error code change to: <SCODE>torch.backends.cudnn.CuDNNError: 6: CUDNN_STATUS_ARCH_MISMATCH
Exception ctypes.ArgumentError: "argument 1: <type 'exceptions.TypeError'>: Don't know how to convert parameter 1" in <bound method CuDNNHandle.__del__ of <torch.backends.cudnn.CuDNNHandle instance at 0x7f4b9099a320>> ignored
<ECODE> I just found that my gpu is Tesla M2075, I searched a similar issue in caffe, saying that cudnn require higher version than pure cuda. Is it not supported in Tesla? Can I run the sample code with only cuda instead of cudnn?

ngimel: M2075 is Fermi architecture card, cudnn is not supported on it. You can disable cudnn by setting torch.backend.cudnn.enabled=False. But you can expect only very modest speed-ups with such an old card.

maplewizard: <SCODE>THCudaCheck FAIL file=/data/users/soumith/miniconda2/conda-bld/pytorch-0.1.9_1487343590888/work/torch/lib/THC/generated/../generic/THCTensorMathPointwise.cu line=246 error=8 : invalid device function
Traceback (most recent call last):
  File "main.py", line 157, in <module>
    train()
  File "main.py", line 131, in train
    output, hidden = model(data, hidden)
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/disk1/ckyn/workbench/learn/pytorch/examples/word_language_model/model.py", line 28, in forward
    output, hidden = self.rnn(emb, hidden)
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 202, in __call__
    result = self.forward(*input, **kwargs)
ons/rnn.py", line 138, in forward
    nexth, output = func(input, hidden, weight)
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 67, in forward
    hy, output = inner(input, hidden[l], weight[l])
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 96, in forward
    hidden = inner(input[i], hidden, *weight)
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 22, in LSTMCell
    gates = F.linear(input, w_ih, b_ih) + F.linear(hx, w_hh, b_hh)
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py", line 752, in __add__
    return self.add(other)
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py", line 292, in add
    return self._add(other, False)
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/autograd/variable.py", line 286, in _add
    return Add(inplace)(self, other)
  File "/data/disk1/ckyn/ProgramFiles/anaconda2/lib/python2.7/site-packages/torch/autograd/_functions/basic_ops.py", line 13, in forward
    return a.add(b)
RuntimeError: cuda runtime error (8) : invalid device function at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.9_1487343590888/work/torch/lib/THC/generated/../generic/THCTensorMathPointwise.cu:246
<ECODE> Is there any idea about this?

ngimel: 

Chun_Li: 

apaszke: 

ngimel: The problem is not ctypes (it looks in ld cache) and not ld cache per se. The problem is that ld cache typically contains libname.so.MAJOR (verify this with ldconfig -p), and for cudnn pytorch tries to load libcudnn.so.MAJOR.MINOR.PATCH. Try adding libcudnn.so.MAJOR.MINOR.PATCH to your ld cache (ldconfig -l may be?)

Chun_Li: 

EthanZhangYi: <SCODE>UserWarning: PyTorch was compiled without cuDNN support. To use cuDNN, rebuild PyTorch making sure the library is visible to the build system.
  "PyTorch was compiled without cuDNN support. To use cuDNN, rebuild "
<ECODE> <SCODE>export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64/cuDNNv5:$LD_LIBRARY_PATH
<ECODE> I build the pytorch from source code. <SCODE>cd pytorch-root/ & python setup.py install
<ECODE>

RayAnteku: Hello , I meet the same problem , can you tell me how to handle this?