Repository: chainer/chainer

Issue:
NStepLSTM fails to backward when chainer.config.train=False and cuDNN is enabled.
NStepLSTM fails to backward when chainer.config.train=False and cuDNN is enabled.

* Conditions
  - OS/Platform: Colaboratory 
  - Chainer version: 5.0.0
  - CuPy version: 5.0.0
  - CUDA version: 9.2

* Code to reproduce
```
import chainer
import chainer.links as L
import chainer.functions as F
import numpy as np
from chainer import cuda

gpu=0

lstm=L.NStepLSTM(1,2,2,0.0)
if gpu >= 0:
    chainer.cuda.get_device(gpu).use()
    lstm.to_gpu()
xp = cuda.cupy if gpu>=0 else np
x=[chainer.Variable(xp.random.random((3,2)).astype(xp.float32))]
with chainer.using_config('train', False):
    _,_,y=lstm(None,None,x)
    F.sum(y[0]).backward()
    print(x[0].grad)
```
* Error messages, stack traces, or logs
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-11-bcaf56bfc39a> in <module>()
     15 with chainer.using_config('train', False):
     16     _,_,y=lstm(None,None,x)
---> 17     F.sum(y[0]).backward()
     18     print(x[0].grad)

/usr/local/lib/python3.6/dist-packages/chainer/variable.py in backward(self, retain_grad, enable_double_backprop, loss_scale)
    961         """
    962         with chainer.using_config('enable_backprop', enable_double_backprop):
--> 963             self._backward_main(retain_grad, loss_scale)
    964 
    965     def _backward_main(self, retain_grad, loss_scale):

/usr/local/lib/python3.6/dist-packages/chainer/variable.py in _backward_main(self, retain_grad, loss_scale)
   1038 
   1039                 _backprop_utils.backprop_step(
-> 1040                     func, target_input_indexes, out_grad, in_grad)
   1041 
   1042                 for hook in hooks:

/usr/local/lib/python3.6/dist-packages/chainer/_backprop_utils.py in backprop_step(func, target_input_indexes, grad_outputs, grad_inputs)
    104     else:  # otherwise, backward should be overridden
    105         gxs = func.backward(
--> 106             target_input_indexes, grad_outputs)
    107 
    108         if is_debug:

/usr/local/lib/python3.6/dist-packages/chainer/function.py in backward(self, target_input_indexes, grad_outputs)
    141 
    142         with cuda.get_device_from_array(*(in_data + grad_out_data)):
--> 143             gxs = self._function.backward(in_data, grad_out_data)
    144         for x, gx in six.moves.zip(self.inputs, gxs):
    145             variable._check_grad_type(self, x, gx)

/usr/local/lib/python3.6/dist-packages/chainer/functions/connection/n_step_rnn.py in backward(self, inputs, grads)
    456             c_dx_descs.data, dxs.data.ptr, dhx_desc.value, dhx.data.ptr,
    457             dcx_desc_value, dcx_data_ptr, workspace.data.ptr, work_size,
--> 458             self.reserve_space.data.ptr, self.reserve_space.size)
    459 
    460         dw = cuda.cupy.zeros_like(w)

AttributeError: 'NStepLSTM' object has no attribute 'reserve_space'
```

The error seems to occur because "reserve_space" in n_step_rnn.py is not defined when chainer.config.train=False. 

How can I fix the error? Or the operation above is not supported?






Discussion:
I will try to reproduce it.
I did reproduce the case on Colaboratory as well as my local environment with Chainer 6.0.0b1 and CuPy 6.6.0b1. I'll look into its detail.
I suppose that when `chainer.config.train=False` Chainer runs in testing, or inference, mode so you usually do not need to perform backward computation. Can I ask you what you intend in the code above?
Thank you for your reply.
I want to obtain Jacobian matrix of loss function in testing mode. As Guided Backpropagation and GradCAM use it, I think it's useful to analyze the model behavior.

I understand. I'd like to conclude that the operation is not supported in Chainer as cuDNN itself does 
 not look to support backward computation in training mode. `cudnnRNNBackwardData` and `cudnnRNNBackwardWeights` require `reserveSpace` parameter which must be generated by `cudnnRNNForwardTraining`.

https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#cudnnRNNBackwardData

Anyway, I think an appropriate error should be raised in this case. 


Resolving patch:
diff --git a/chainer/functions/connection/n_step_rnn.py b/chainer/functions/connection/n_step_rnn.py
--- a/chainer/functions/connection/n_step_rnn.py
+++ b/chainer/functions/connection/n_step_rnn.py
@@ -295,6 +295,10 @@ def forward(self, inputs):
             return hy, ys
 
     def backward(self, inputs, grads):
+        if not configuration.config.train:
+            raise RuntimeError('cuDNN does not support backward computation '
+                               'of RNN in testing mode')
+
         if self.use_cell:
             # LSTM
             hx, cx, w, xs = inputs


Test patch:
diff --git a/tests/chainer_tests/functions_tests/connection_tests/test_n_step_rnn.py b/tests/chainer_tests/functions_tests/connection_tests/test_n_step_rnn.py
--- a/tests/chainer_tests/functions_tests/connection_tests/test_n_step_rnn.py
+++ b/tests/chainer_tests/functions_tests/connection_tests/test_n_step_rnn.py
@@ -227,7 +227,7 @@ def test_call_cudnn_forward_inference(self):
         self.check_call_cudnn_forward_inference('never')
         self.check_call_cudnn_forward_inference('auto')
 
-    def check_call_cudnn_backward(self, use_cudnn):
+    def check_call_cudnn_backward_training(self, use_cudnn):
         with chainer.using_config('use_cudnn', use_cudnn):
             expect = chainer.should_use_cudnn('>=auto', 5000)
             hy, ys = self.call_forward(True)
@@ -237,10 +237,36 @@ def check_call_cudnn_backward(self, use_cudnn):
             assert func.called == expect
 
     @attr.cudnn
-    def test_call_cudnn_backward(self):
-        self.check_call_cudnn_backward('always')
-        self.check_call_cudnn_backward('never')
-        self.check_call_cudnn_backward('auto')
+    def test_call_cudnn_backward_training(self):
+        self.check_call_cudnn_backward_training('always')
+        self.check_call_cudnn_backward_training('never')
+        self.check_call_cudnn_backward_training('auto')
+
+    def check_call_cudnn_backward_inference(self, use_cudnn):
+        with chainer.using_config('use_cudnn', use_cudnn), \
+                chainer.using_config('train', False):
+            hx = _wrap_variable(_to_gpu(self.hx))
+            xs = _wrap_variable(_to_gpu(self.xs))
+            ws = _wrap_variable(_to_gpu(self.ws))
+            bs = _wrap_variable(_to_gpu(self.bs))
+            hy, ys = functions.n_step_rnn(
+                self.n_layers, self.dropout, hx, ws, bs, xs)
+
+            hy.grad = _to_gpu(self.dhy)
+            if chainer.should_use_cudnn('>=auto', 5000):
+                with self.assertRaises(RuntimeError):
+                    hy.backward()
+            else:
+                with testing.patch('cupy.cudnn.rnn_backward_weights') as func:
+                    hy.backward()
+                assert not func.called
+
+    @attr.cudnn
+    def test_call_cudnn_backward_inference(self):
+        # see chainer/5943
+        self.check_call_cudnn_backward_inference('always')
+        self.check_call_cudnn_backward_inference('never')
+        self.check_call_cudnn_backward_inference('auto')
 
 
 @testing.parameterize(*testing.product({