Repository: chainer/chainer

Issue:
TestLSTM's `check_backward` fails when output size is 1
LSTM function's backward test seems to fail when output size is 1, probably because `variable.backward` sets 1 to grad when the size is 1.

See also: #3435

* Conditions
  - Chainer version: chainer@9f94cea8db8ff3f686476100767db6f909e83703

* Code to reproduce
https://github.com/toslunar/chainer/commit/d8aff2785179b24687a25a8f1518935a5cd99dd9

* Error messages, stack traces, or logs
`12 failed, 102 passed, 114 skipped in 2.18 seconds`

All the failures are from gradient_check.  Here I put only the first failure.
```
% CHAINER_TEST_GPU_LIMIT=0 pytest -m "not slow and not cudnn" chainer_tests/functions_tests/activation_tests/test_lstm.py  -x
================================================= test session starts ==================================================
platform darwin -- Python 3.6.2, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
rootdir: /Users/tos/Documents/chainer, inifile: setup.cfg
collected 228 items

chainer_tests/functions_tests/activation_tests/test_lstm.py .s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s [ 22%]
.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.s.sF

======================================================= FAILURES =======================================================
____________________________________ TestLSTM1_param_0.test_flat_no_gc_backward_cpu ____________________________________

self = <chainer.testing.parameterized.TestLSTM1_param_0 testMethod=test_flat_no_gc_backward_cpu>

    def test_flat_no_gc_backward_cpu(self):
        self.flat()
>       self.test_no_gc_backward_cpu()

chainer_tests/functions_tests/activation_tests/test_lstm.py:111:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../chainer/testing/parameterized.py:58: in wrap
    six.raise_from(e_new.with_traceback(e.__traceback__), None)
<string>:2: in raise_from
    ???
../chainer/testing/parameterized.py:43: in wrap
    return method(*args, **kwargs)
chainer_tests/functions_tests/activation_tests/test_lstm.py:107: in test_no_gc_backward_cpu
    self.check_backward(self.c_prev, self.x, None, self.gh)
chainer_tests/functions_tests/activation_tests/test_lstm.py:97: in check_backward
    dtype=numpy.float64, **self.check_backward_options)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

func = <function lstm at 0x1083100d0>
x_data = (array([[ 0.08680988]], dtype=float32), array([[-0.44326124, -0.15096481,  0.68955225, -0.99056226]], dtype=float32))
y_grad = (None, array([[ 0.34149817]], dtype=float32)), params = (), eps = 0.001, atol = 1e-05, rtol = 0.0001
no_grads = [False, False], dtype = <class 'numpy.float64'>, detect_nondifferentiable = False

    def check_backward(
            func, x_data, y_grad, params=(),
            eps=1e-3, atol=1e-5, rtol=1e-4, no_grads=None, dtype=None,
            detect_nondifferentiable=False):
        """Test backward procedure of a given function.

        This function automatically checks backward-process of a given function.
        For example, when you have a :class:`~chainer.Function` class ``MyFunc``,
        that gets two arguments and returns one value, you can make its test like
        this::

        >> def test_my_func(self):
        >>   func = MyFunc()
        >>   x1_data = xp.array(...)
        >>   x2_data = xp.array(...)
        >>   gy_data = xp.array(...)
        >>   check_backward(func, (x1_data, x2_data), gy_data)

        This method creates :class:`~chainer.Variable` objects with ``x_data``
        and calls ``func`` with the :class:`~chainer.Variable`\\ s to get its
        result as :class:`~chainer.Variable`.
        Then, it sets ``y_grad`` array to ``grad`` attribute of the result and
        calls ``backward`` method to get gradients of the inputs.
        To check correctness of the gradients, the function calls
        :func:`numerical_grad` to calculate numerically the gradients and compares
        the types of gradients with :func:`chainer.testing.assert_allclose`.

        To reduce computational time, it uses directional derivative along a
        random vector. A function
        :math:`g: \\mathbb{R} \\rightarrow \\mathbb{R}^n` is defined as
        :math:`g(\\delta) = f(x + \\delta r)`, where
        :math:`\\delta \\in \\mathbb{R}`, :math:`r \\in \\mathbb{R}^n`
        is a random vector
        and :math:`f` is a function which you want to test.
        Its gradient is

        .. math::
           g'(\\delta) = f'(x + \\delta r) \\cdot r.

        Therefore, :math:`g'(0) = f'(x) \\cdot r`.
        So we can check the correctness of back propagation of :math:`f` indirectly
        by comparing this equation with the gradient of :math:`g` numerically
        calculated and that of :math:`f` computed by backprop.
        If :math:`r` is chosen from uniform distribution, we can conclude with
        high probability that the gradient of :math:`f` itself is correct.

        If input objects (``x1_data`` or/and ``x2_data`` in this example) represent
        integer variables, their gradients are ignored.

        You can simplify a test when ``MyFunc`` gets only one argument::

        >>   check_backward(func, x1_data, gy_data)

        If ``MyFunc`` is a loss function which returns a zero-dimensional
        array, pass ``None`` to ``gy_data``. In this case, it sets ``1`` to
        ``grad`` attribute of the result::

        >>   check_backward(my_loss_func, (x1_data, x2_data), None)

        If ``MyFunc`` returns multiple outputs, pass all gradients for outputs
        as a tuple::

        >>   gy1_data = xp.array(...)
        >>   gy2_data = xp.array(...)
        >>   check_backward(func, x1_data, (gy1_data, gy2_data))

        You can also test a :class:`~chainer.Link`.
        To check gradients of parameters of the link, set a tuple of the parameters
        to ``params`` arguments::

        >>   check_backward(my_link, (x1_data, x2_data), gy_data,
        >>                  (my_link.W, my_link.b))

        Note that ``params`` are not ``ndarray``\\ s,
        but :class:`~chainer.Variables`\\ s.

        Function objects are acceptable as ``func`` argument::

        >>   check_backward(lambda x1, x2: f(x1, x2),
        >>                  (x1_data, x2_data), gy_data)

        .. note::

           ``func`` is called many times to get numerical gradients for all inputs.
           This function doesn't work correctly when ``func`` behaves randomly as
           it gets different gradients.


        Args:
            func (callable): A function which gets :class:`~chainer.Variable`\\ s
                and returns :class:`~chainer.Variable`\\ s. ``func`` must returns
                a tuple of :class:`~chainer.Variable`\\ s or one
                :class:`~chainer.Variable`. You can use :class:`~chainer.Function`
                object, :class:`~chainer.Link` object or a function satisfying the
                condition.
            x_data (ndarray or tuple of ndarrays): A set of ``ndarray``\\ s to be
                passed to ``func``. If ``x_data`` is one ``ndarray`` object, it is
                treated as ``(x_data,)``.
            y_grad (ndarray or tuple of ndarrays or None):
                A set of ``ndarray``\\ s representing gradients of return-values of
                ``func``. If ``y_grad`` is one ``ndarray`` object, it is
                treated as ``(y_grad,)``. If ``func`` is a loss-function,
                ``y_grad`` should be set to ``None``.
            params (~chainer.Variable or tuple of ~chainder.Variable):
                A set of :class:`~chainer.Variable`\\ s whose gradients are
                checked. When ``func`` is a :class:`~chainer.Link` object,
                set its parameters as ``params``.
                If ``params`` is one :class:`~chainer.Variable` object,
                it is treated as ``(params,)``.
            eps (float): Epsilon value to be passed to :func:`numerical_grad`.
            atol (float): Absolute tolerance to be passed to
                :func:`chainer.testing.assert_allclose`.
            rtol (float): Relative tolerance to be passed to
                :func:`chainer.testing.assert_allclose`.
            no_grads (list of bool): Flag to skip variable for gradient assertion.
                It should be same length as ``x_data``.
            dtype (~numpy.dtype): ``x_data``, ``y_grad`` and ``params`` are casted
                to this dtype when calculating numerical gradients. Only float
                types and ``None`` are allowed.
            detect_nondifferentiable (bool):
                If ``True``, check for non-differentiable inputs is enabled.
                If ``func`` is non-differentiable at ``x_data``, ``check_backward``
                raises :class:`~chainer.gradient_check.NondifferentiableError`.

        .. seealso::
           :func:`numerical_grad`
        """
        if dtype is not None and numpy.dtype(dtype).kind != 'f':
            raise ValueError('`dtype` is allowed only float type')

        x_data = _as_tuple(x_data)
        if y_grad is not None:
            y_grad = _as_tuple(y_grad)
        params = _as_tuple(params)

        xs = [variable.Variable(x) for x in x_data]
        y = func(*xs)
        y = _as_tuple(y)
        y0_data = [_.data for _ in y]

        # All creators of `y` need to be the same because we only call
        # `y[0].backward` to call `backward` method of the creator.
        # To do so we need to insert a dummy function `Ident` to the
        # computational graph.
        # Note that `func` may not be a `Function` object.
        y = identity.Identity().apply(y)

        y_grad = _set_y_grad(y, y_grad)

        # Clear gradients which may exist if func calls backward inside of itself.
        _clear_grads(xs)
        _clear_grads(params)

        # We only need to call `backward` for one result `Variable`.
        # `Variable.backward` method calls `Function.backward` of its creator.
        y[0].backward()

        if no_grads is None:
            no_grads = [x.dtype.kind != 'f' for x in xs]
        else:
            if len(no_grads) != len(xs):
                raise ValueError(
                    'Length of no_grads param and xs should be same.\n'
                    'Actual: {0} != {1}'.format(len(no_grads), len(xs)))

        for skip, x in six.moves.zip(no_grads, xs):
            if skip:
                if x.grad is not None:
                    raise RuntimeError(
                        'gradient of int variable must be None')
            else:
                if x.grad is None:
                    raise RuntimeError(
                        'gradients of some arguments are not calculated')

        if len(xs) - no_grads.count(True) + len(params) == 0:
            # When there is no float variables, we need not to check gradient
            # values
            return

        variables = _filter_list(xs, no_grads) + list(params)
        # Keep the gradient arrays of params which may be overwritten by func
        grads = [x.grad for x in variables]

        if dtype is None:
            casted_data = [x.data for x in variables]
        else:
            if numpy.dtype(dtype).kind != 'f':
                raise ValueError('`dtype` is allowed only float type')
            casted_data = [x.data.astype(dtype, copy=False) for x in variables]

            # Even skipped variable must have the same dtype.
            for x, skip in six.moves.zip(xs, no_grads):
                if skip and x.data.dtype.kind == 'f':
                    x.data = x.data.astype(dtype, copy=False)

        xp = cuda.get_array_module(*xs)
        directions = [xp.random.normal(size=x.shape) for x in variables]
        # Use unit vector
        norm = math.sqrt(sum([xp.square(d).sum() for d in directions]))
        if norm != 0:
            # norm could be zero if input arrays are 0-sized.
            scale = 1. / norm
            directions = [d * scale for d in directions]

        delta = xp.array(0., 'd')

        def g():
            # This functions is called twice in `numerical_grad`.
            # `delta` is `epsilon` or `-epsilon` in these calls.
            # See the document of `numerical_grad`.
            for x, data, direction in six.moves.zip(
                    variables, casted_data, directions):
                # astype is require to store data with the given type
                data = (data.astype('d') +
                        delta * direction).astype(data.dtype)
                if numpy.isscalar(data):
                    data = xp.array(data)
                x.data = data

            # Clear gradients to support func that calls backward inside of itself.
            _clear_grads(xs)
            _clear_grads(params)

            ys = func(*xs)
            ys = _as_tuple(ys)
            ys_data = tuple(y.data for y in ys)
            for x, data in six.moves.zip(variables, casted_data):
                x.data = data
            return ys_data

        gx, = numerical_grad(
            g, (delta,), y_grad, eps=eps,
            detect_nondifferentiable=detect_nondifferentiable,
            center_outputs=y0_data)
        gx_accum = 0
        for g, direction in six.moves.zip(grads, directions):
            gx_accum += (g.astype('d') * direction).sum()

        try:
            testing.assert_allclose(gx, gx_accum, atol=atol, rtol=rtol)
        except AssertionError as e:
            f = six.StringIO()
            f.write('check_backward failed (eps={} atol={} rtol={})\n'.format(
                eps, atol, rtol))
            for i, x_ in enumerate(xs):
                f.write('inputs[{}]:\n'.format(i))
                f.write('{}\n'.format(x_))
            for i, gy_ in enumerate(y_grad):
                f.write('grad_outputs[{}]:\n'.format(i))
                f.write('{}\n'.format(gy_))
            f.write('gradients (numeric):  {}\n'.format(gx))
            f.write('gradients (backward): {}\n'.format(gx_accum))
            f.write('\n')
            f.write(str(e))
>           raise AssertionError(f.getvalue())
E           AssertionError: Parameterized test failed.
E
E           Base test method: TestLSTM1.test_flat_no_gc_backward_cpu
E           Test parameters:
E             batch: 1
E             dtype: <class 'numpy.float32'>
E
E           AssertionError: Parameterized test failed.
E
E           Base test method: TestLSTM1.test_no_gc_backward_cpu
E           Test parameters:
E             batch: 1
E             dtype: <class 'numpy.float32'>
E
E           AssertionError: check_backward failed (eps=0.001 atol=1e-05 rtol=0.0001)
E           inputs[0]:
E           variable([[ 0.08680988]])
E           inputs[1]:
E           variable([[-0.44326124 -0.15096481  0.68955225 -0.99056226]])
E           grad_outputs[0]:
E           None
E           grad_outputs[1]:
E           [[ 0.34149817]]
E           gradients (numeric):  -0.008615916456262186
E           gradients (backward): -0.054444917690953915
E
E
E           Not equal to tolerance rtol=0.0001, atol=1e-05
E
E           (mismatch 100.0%)
E            x: array(-0.008615916456262186)
E            y: array(-0.054444917690953915)
E
E           assert_allclose failed:
E             shape: () ()
E             dtype: float64 float64
E             i: (0,)
E             x[i]: -0.008615916456262186
E             y[i]: -0.054444917690953915
E             err[i]: 0.04582900123469173
E           x: -0.008615916456262186
E           y: -0.054444917690953915

../chainer/gradient_check.py:557: AssertionError
=================================== 1 failed, 57 passed, 57 skipped in 1.31 seconds ====================================
```


Discussion:

Resolving patch:
diff --git a/chainer/gradient_check.py b/chainer/gradient_check.py
--- a/chainer/gradient_check.py
+++ b/chainer/gradient_check.py
@@ -6,7 +6,7 @@
 
 from chainer.backends import cuda
 from chainer import configuration
-from chainer.functions.math import identity
+from chainer import FunctionNode
 from chainer import testing
 from chainer import variable
 
@@ -447,12 +447,11 @@ def check_backward(
 
     # All creators of `y` need to be the same because we only call
     # `y[0].backward` to call `backward` method of the creator.
-    # To do so we need to insert a dummy function `Ident` to the
+    # To do so we need to insert a dummy function `_GradientSetter` to the
     # computational graph.
     # Note that `func` may not be a `Function` object.
-    y = identity.Identity().apply(y)
 
-    y_grad = _set_y_grad(y, y_grad)
+    y, y_grad = _set_y_grad(y, y_grad)
 
     # Clear gradients which may exist if func calls backward inside of itself.
     _clear_grads(xs)
@@ -460,7 +459,7 @@ def check_backward(
 
     # We only need to call `backward` for one result `Variable`.
     # `Variable.backward` method calls `Function.backward` of its creator.
-    y[0].backward()
+    y.backward()
 
     if no_grads is None:
         no_grads = [x.dtype.kind != 'f' for x in xs]
@@ -609,12 +608,12 @@ def first_order_grad(*inputs):
         gys = inputs[n_x:]
 
         y = _as_tuple(func(*xs))
+
         # Let all elements of y share the same creator.
         # See the comment in check_backward.
-        y = identity.Identity().apply(y)
+        y, _ = _set_y_grad(y, gys)
 
-        _set_y_grad(y, gys)
-        y[0].backward(enable_double_backprop=True)
+        y.backward(enable_double_backprop=True)
 
         return tuple([x.grad_var for x in xs] + [p.grad_var for p in params])
 
@@ -646,6 +645,31 @@ def first_order_grad(*inputs):
         raise AssertionError(f.getvalue())
 
 
+class _GradientSetter(FunctionNode):
+    def __init__(self, grad):
+        self.grad = grad
+
+    def forward(self, inputs):
+        xp = cuda.get_array_module(inputs[0])
+
+        if self.grad is None:
+            y0, = inputs
+            gy0 = xp.ones_like(y0)
+            assert gy0.size == 1
+
+            self.grad = (gy0,)
+
+        # output a 0-sized 1-dim array like inputs
+        return xp.empty((0,), dtype=inputs[0].dtype),
+
+    def backward(self, inputs, grad_outputs):
+        grad = self.grad
+
+        return tuple(
+            None if g is None else variable.as_variable(g)
+            for g in grad)
+
+
 def _set_y_grad(y, y_grad):
     if y_grad is not None:
         if len(y) != len(y_grad):
@@ -653,19 +677,16 @@ def _set_y_grad(y, y_grad):
                 'Upstream gradients must contain equally many elements as '
                 'number of output elements.\n'
                 'Actual: {} != {}'.format(len(y), len(y_grad)))
-        for iy, igy in six.moves.zip(y, y_grad):
-            if isinstance(igy, variable.Variable):
-                iy.grad_var = igy
-            else:
-                iy.grad = igy
+        y, = _GradientSetter(y_grad).apply(y)
     else:
         if len(y) != 1:
             raise ValueError(
                 'Function must return a zero-dimensional array of length 1 '
                 'if the upstream gradient is `None`.\n'
                 'Actual: {} != 1'.format(len(y)))
+        y, = _GradientSetter(None).apply(y)
         y_grad = (1,)
-    return y_grad
+    return y, y_grad
 
 
 def _clear_grads(xs):


Test patch:
diff --git a/tests/chainer_tests/test_gradient_check.py b/tests/chainer_tests/test_gradient_check.py
--- a/tests/chainer_tests/test_gradient_check.py
+++ b/tests/chainer_tests/test_gradient_check.py
@@ -604,6 +604,45 @@ def f(x, y):
                                       no_grads=[False, True], dtype=self.dtype)
 
 
+class IdentNoneIsZero(chainer.Function):
+    """Identity function but following None-grad convention for RNNs"""
+
+    def forward(self, inputs):
+        return inputs
+
+    def backward(self, inputs, grads):
+        return tuple(
+            numpy.zeros_like(x) if g is None else g
+            for x, g in zip(inputs, grads)
+        )
+
+
+@testing.parameterize(*testing.product({
+    'dtype': [None, numpy.float32, numpy.float64],
+    'size': [3, 1]
+}))
+class TestCheckBackwardNoneConvention(unittest.TestCase):
+    dtype = numpy.float64
+
+    def test_multiple_output(self):
+        size = self.size
+        x1 = numpy.arange(size).astype('f')
+        x2 = numpy.arange(size).astype('f')
+        g1 = numpy.ones(size, dtype='f')
+        g2 = numpy.ones(size, dtype='f')
+
+        def f(x, y):
+            s, t = IdentNoneIsZero()(x, y)
+            return s, t
+
+        gradient_check.check_backward(
+            f, (x1, x2), (g1, g2), dtype=self.dtype, atol=1e-4, rtol=1e-3)
+        gradient_check.check_backward(
+            f, (x1, x2), (g1, None), dtype=self.dtype, atol=1e-4, rtol=1e-3)
+        gradient_check.check_backward(
+            f, (x1, x2), (None, g2), dtype=self.dtype, atol=1e-4, rtol=1e-3)
+
+
 class TestCheckBackwardFailure(unittest.TestCase):
 
     def _broken_func_1(self):