Repository: chainer/chainer

Issue:
numerical_grad caution
I was implementing a differentiable Transpose function. 

```
class Transpose(Function):
    def forward(self, inputs):
        x = inputs[0]
        return x.transpose(),
    def backward(self, inputs, grads):
        return grads[0].transpose(),
```

While testing the gradient with numerical_grad, 

```
    def test_numerical_grad_cpu(self):
        x = np.random.randn(1, 10)
        x_var = Variable(x)
        y_var = self.f(x_var)
        y_var.grad = np.random.rand(10, 1)
        y_var.backward()
        cl = lambda: self.f.forward((x,))
        gx, = gradient_check.numerical_grad(cl, (x,), (y_var.grad,))
        gradient_check.assert_allclose(gx, x_var.grad)
```

(here `self.f = Transpose()`)
the numerical gradient `gx` keeps coming back as 0. After much frustration, I finally figured out that I was returning a view of `x` in the above code, and in `numerical_grad_cpu`,

```
            flat_x[i] = orig + eps
            ys1 = f()
            flat_x[i] = orig - eps
            ys2 = f()
            flat_x[i] = orig
```

`ys1` and `ys2` end up being equal after the last line resetting `flat_x[i]` to the original value. I solved my problem by changing `cl = lambda: self.f.forward((x,))` to `cl = lambda: np.copy(self.f.forward((x,)))`.

I'm not sure how frequent this phenomenon could occur outside of transpose, but I just wanted to put this out here so that there could be a discussion. Perhaps a passing note in the documentation suffices here. Or doing `ys1 = np.copy(f())` instead might work as well.



Discussion:
Yes, I ran into exactly the same issue while implementing exactly the same function (see #480). This will happen for all reshaping functions and generally all functions that re-use the memory of the input. There is a way to check if two arrays are sharing memory (http://stackoverflow.com/questions/11286864/is-there-a-way-to-check-if-numpy-arrays-share-the-same-data), perhaps numerical_grad should check this.

Thank you for the report! I have not noticed it. We should add a copy on numerical_grad. Copying without check might be enough, since small arrays are used for gradient checking. If it prohibitively slows down the whole tests, we should think about checking the memory share.


Resolving patch:
diff --git a/chainer/gradient_check.py b/chainer/gradient_check.py
--- a/chainer/gradient_check.py
+++ b/chainer/gradient_check.py
@@ -5,6 +5,11 @@
 from chainer import utils
 
 
+def _copy_arrays(xs):
+    xp = cuda.get_array_module(*xs)
+    return tuple(xp.copy(x) for x in xs)
+
+
 def numerical_grad_cpu(f, inputs, grad_outputs, eps=1e-3):
     grads = tuple(numpy.zeros_like(x) for x in inputs)
     for x, gx in zip(inputs, grads):
@@ -13,9 +18,9 @@ def numerical_grad_cpu(f, inputs, grad_outputs, eps=1e-3):
         for i in six.moves.range(flat_x.size):
             orig = flat_x[i]
             flat_x[i] = orig + eps
-            ys1 = f()
+            ys1 = _copy_arrays(f())
             flat_x[i] = orig - eps
-            ys2 = f()
+            ys2 = _copy_arrays(f())
             flat_x[i] = orig
 
             for y1, y2, gy in zip(ys1, ys2, grad_outputs):
@@ -37,10 +42,10 @@ def numerical_grad_gpu(f, inputs, grad_outputs, eps=1e-3):
             orig = x_cpu[i]
             x_cpu[i] = orig + eps
             x.set(x_cpu)
-            ys1 = f()
+            ys1 = _copy_arrays(f())
             x_cpu[i] = orig - eps
             x.set(x_cpu)
-            ys2 = f()
+            ys2 = _copy_arrays(f())
             x_cpu[i] = orig
             x.set(x_cpu)
 


Test patch:
diff --git a/tests/chainer_tests/test_gradient_check.py b/tests/chainer_tests/test_gradient_check.py
--- a/tests/chainer_tests/test_gradient_check.py
+++ b/tests/chainer_tests/test_gradient_check.py
@@ -142,6 +142,26 @@ def setUp(self):
         self.gys = (None,)
 
 
+class NumericalGradientReferenceTest(unittest.TestCase):
+
+    def setUp(self):
+        self.x = _uniform(2, 3)
+
+    def check_reference(self, x):
+        # A returned value and an input refers the same memory.
+        # See issue #488
+        func = lambda: (x,)
+        gx, = gradient_check.numerical_grad(func, (x,), (1,))
+        gradient_check.assert_allclose(cuda.to_cpu(gx), 1)
+
+    def test_reference_cpu(self):
+        self.check_reference(self.x)
+
+    @attr.gpu
+    def test_reference_gpu(self):
+        self.check_reference(cuda.to_gpu(self.x))
+
+
 class NumericalGradientInvalidEps(NumericalGradientTest):
 
     def check_invalid_eps(self, xs, gys, eps):