Repository: chainer/chainer

Issue:
cupy.cuda.cudnn.CuDNNError in NStepLSTM
I'm trying to use NStepLSTM like as follows. But if n_layer is more than 2 (n_layer>=2), I get error 'CUDNN_STATUS_BAD_PARAM'. When n_layer==1, my code make success. Is there anything wrong in my code?

Source code is [here](https://github.com/zacapa/chainer-nsteplstm).

``` python
class RNN(chainer.Chain):

    def __init__(self, n_layer, n_vocab, n_units, dropout, cudnn):
        super(RNN, self).__init__(
            embed=L.EmbedID(n_vocab, n_units),
            l1=L.NStepLSTM(n_layer, n_units, n_units,
                           dropout, use_cudnn=cudnn),
            l2=L.Linear(n_units, n_vocab),
        )

    def __call__(self, hx, cx, xs, train=True):
        xs = [self.embed(item) for item in xs]
        hy, cy, ys = self.l1(hx, cx, xs, train=train)
        y = [self.l2(item) for item in ys]
        return y
```

``` bash
$ python ptb_nslstm.py -g 0 -l 2 --cudnn
File "/anaconda3/lib/python3.5/site-packages/chainer/links/connection/n_step_lstm.py", line 96, in __call__
    train=train, use_cudnn=self.use_cudnn)
  File "/anaconda3/lib/python3.5/site-packages/chainer/functions/connection/n_step_lstm.py", line 468, in n_step_lstm
    ret = rnn(*inputs)
  File "/anaconda3/lib/python3.5/site-packages/chainer/function.py", line 197, in __call__
    outputs = self.forward(in_data)
  File "/anaconda3/lib/python3.5/site-packages/chainer/functions/connection/n_step_lstm.py", line 267, in forward
    self.reserve_space.data.ptr, reserve_size)
  File "cupy/cuda/cudnn.pyx", line 983, in cupy.cuda.cudnn.RNNForwardTraining (cupy/cuda/cudnn.cpp:13268)
  File "cupy/cuda/cudnn.pyx", line 1002, in cupy.cuda.cudnn.RNNForwardTraining (cupy/cuda/cudnn.cpp:13008)
  File "cupy/cuda/cudnn.pyx", line 321, in cupy.cuda.cudnn.check_status (cupy/cuda/cudnn.cpp:1718)
cupy.cuda.cudnn.CuDNNError: CUDNN_STATUS_BAD_PARAM: b'CUDNN_STATUS_BAD_PARAM'
```



Discussion:
Thank you for reporting a problem. Which version of CUDA and cuDNN do you use?

I have 2 GPUPC. Both of GPUPC have same error.

There are version of CUDA and cuDNN in GPUPC1. 
- CUDA: 8.0

``` bash
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Wed_May__4_21:01:56_CDT_2016
Cuda compilation tools, release 8.0, V8.0.26
```
- cuDNN: 5.1

```
$ python
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import chainer.cuda as cuda
>>> cuda.cudnn.cudnn.getVersion()
5103
```

There are version of CUDA and cuDNN in GPUPC2.
- CUDA: 7.5

``` bash
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17 
```
- cuDNN: 5.1

```
$ python
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import chainer.cuda as cuda
>>> cuda.cudnn.cudnn.getVersion()
5103
```

Thank you. I can reproduce this error. I'll fix it. Please wait.


Resolving patch:
diff --git a/chainer/functions/connection/n_step_lstm.py b/chainer/functions/connection/n_step_lstm.py
--- a/chainer/functions/connection/n_step_lstm.py
+++ b/chainer/functions/connection/n_step_lstm.py
@@ -63,17 +63,16 @@ def __init__(self, states, desc):
         self.states = states
         self.desc = desc
 
+    def set_dropout_ratio(self, handle, dropout):
+        cudnn.set_dropout_descriptor(self.desc, handle, dropout)
+
     @staticmethod
-    def create(handle, states, dropout, seed):
+    def create(handle, dropout, seed):
+        states = cudnn.create_dropout_states(handle)
         desc = cudnn.create_dropout_descriptor(
             handle, dropout, states.data.ptr, states.size, seed)
         return DropoutStates(states, desc)
 
-    @staticmethod
-    def from_states(handle, states, dropout):
-        desc = cudnn.create_dropout_descriptor(handle, dropout, 0, 0, 0)
-        return DropoutStates(states, desc)
-
 
 class DropoutRandomStates(object):
 
@@ -94,11 +93,11 @@ def __init__(self, seed):
     def create_dropout_states(self, dropout):
         handle = cudnn.get_handle()
         if self._states is None:
-            self._states = cudnn.create_dropout_states(handle)
-            return DropoutStates.create(
-                handle, self._states, dropout, self._seed)
+            self._states = DropoutStates.create(handle, dropout, self._seed)
         else:
-            return DropoutStates.from_states(handle, self._states, dropout)
+            self._states.set_dropout_ratio(handle, dropout)
+
+        return self._states
 
 
 _random_states = {}
diff --git a/cupy/cudnn.py b/cupy/cudnn.py
--- a/cupy/cudnn.py
+++ b/cupy/cudnn.py
@@ -235,6 +235,11 @@ def create_dropout_descriptor(
     return desc
 
 
+def set_dropout_descriptor(desc, handle, dropout):
+    # When the fourth argument is NULL, random state is not updated.
+    cudnn.setDropoutDescriptor(desc.value, handle, dropout, 0, 0, 0)
+
+
 def create_rnn_descriptor(hidden_size, num_layers, dropout_desc,
                           input_mode, direction, mode, data_type):
     desc = Descriptor(cudnn.createRNNDescriptor(),


Test patch: