Repository: chainer/chainer

Issue:
2nd order derivative of ELU should not give NaN
The 2nd order derivative of ELU gives off NaN about once out of hundreds of iterations.  Then the entire network will instantly contaminated with NaN.

I tracked the cause and the following code in backward() of class ELUGrad in chainer/chainer/functions/activation/elu.py is the cause
```
        if 1 in indexes:
            ret.append(ggxgx / gy)
```

It is natural that this division will give NaN if some element of gy is zero.  Zero will occur when the single-precision floating point subtraction underflows.

For your information, I am circumventing the issue by making minor modification to elu.py as in the attached file, which uses the same way to calculate derivative as the mathematical functions like F.exp.
[elu.py.txt](https://github.com/chainer/chainer/files/1683829/elu.py.txt)


How to reproduce:
I am using Chainer 3.2.0, but that part of the ELU source code is not different in v4.0, therefore I think this issue persists over the versions.
```
>>> import chainer
>>> import numpy as np
>>> x = chainer.Variable(np.array([[0, 0]],dtype=np.float32))
>>> y = chainer.functions.elu(x)
>>> y
variable([[ 0.,  0.]])
>>> 
>>> y.grad = (np.array([[0, 1e-30]],dtype=np.float32))
>>> y.backward(enable_double_backprop=True)
>>> 
>>> x.grad_var.grad = np.array([[1, 1]],dtype=np.float32)
>>> x.grad_var.backward()
/home/mogami/.pyenv/versions/anaconda3-4.2.0/lib/python3.5/site-packages/chainer/functions/math/basic_math.py:322: RuntimeWarning: invalid value encountered in true_divide
  return utils.force_array(x[0] / x[1]),
>>> y.grad_var
variable([[  0.00000000e+00,   1.00000000e-30]])
>>> y.grad_var.grad
array([[ nan,   1.]], dtype=float32)
```
The first element is nan, though it should be 1.0 in this case.  This example may seem silly when considering ELU only, but having zero for some of elements often happens when dy is back propagated from somewhere else because of underflow.


Discussion:

Resolving patch:
diff --git a/chainer/functions/activation/elu.py b/chainer/functions/activation/elu.py
--- a/chainer/functions/activation/elu.py
+++ b/chainer/functions/activation/elu.py
@@ -22,14 +22,14 @@ def forward_cpu(self, x):
         self.retain_inputs((0,))
         y = x[0].copy()
         neg_indices = x[0] < 0
-        y[neg_indices] = self.alpha * (numpy.exp(y[neg_indices]) - 1)
+        y[neg_indices] = self.alpha * (numpy.expm1(y[neg_indices]))
         return y,
 
     def forward_gpu(self, x):
         self.retain_inputs((0,))
         y = cuda.elementwise(
             'T x, T alpha', 'T y',
-            'y = x >= 0 ? x : (T)(alpha * (exp(x) - 1))',
+            'y = x >= 0 ? x : (T)(alpha * expm1(x))',
             'elu_fwd')(
                 x[0], self.alpha)
         return y,
@@ -37,7 +37,7 @@ def forward_gpu(self, x):
     def backward(self, indexes, grad_outputs):
         x, = self.get_retained_inputs()
         gy, = grad_outputs
-        return ELUGrad(self.alpha).apply((x, gy))
+        return ELUGrad(self.alpha).apply((x,))[0] * gy,
 
 
 class ELUGrad(function_node.FunctionNode):
@@ -48,43 +48,34 @@ def __init__(self, alpha):
         self.alpha = alpha
 
     def check_type_forward(self, in_types):
-        type_check.expect(in_types.size() == 2)
+        type_check.expect(in_types.size() == 1)
         type_check.expect(in_types[0].dtype.kind == 'f')
-        type_check.expect(in_types[1].dtype.kind == 'f')
 
     def forward_cpu(self, inputs):
-        x, gy = inputs
-        gx = gy.copy()
+        x, = inputs
+        gx = numpy.ones_like(x)
         neg_indices = x < 0
         gx[neg_indices] *= self.alpha * numpy.exp(x[neg_indices])
-        self.retain_inputs((0, 1))
+        self.retain_inputs((0,))
         self.retain_outputs((0,))
         return gx,
 
     def forward_gpu(self, inputs):
-        x, gy = inputs
+        x, = inputs
         gx = cuda.elementwise(
-            'T x, T gy, T alpha', 'T gx',
-            'gx = x >= 0 ? gy : (T)(gy * alpha * exp(x))',
+            'T x, T alpha', 'T gx',
+            'gx = x >= 0 ? (T)1 : (T)(alpha * exp(x))',
             'elu_bwd')(
-                x, gy, self.alpha)
-        self.retain_inputs((0, 1))
+                x, self.alpha)
+        self.retain_inputs((0,))
         self.retain_outputs((0,))
         return gx,
 
     def backward(self, indexes, grad_outputs):
-        x, gy = self.get_retained_inputs()
+        x, = self.get_retained_inputs()
         gx, = self.get_retained_outputs()
         ggx, = grad_outputs
-        ggxgx = ggx * gx
-
-        ret = []
-        if 0 in indexes:
-            ret.append(ggxgx * (x.data < 0))
-        if 1 in indexes:
-            ret.append(ggxgx / gy)
-
-        return ret
+        return ggx * gx * (x.data < 0),
 
 
 def elu(x, alpha=1.0):


Test patch: