Repository: chainer/chainer

Issue:
Type error after backward_cpu of SigmoidCrossEntropy
SigmoidCrossEntropy's backward_cpu returns an array with dtype float64, which causes an error in Variable's backward method.

```
TypeError: Dtype of data and grad mismatch: float32 != float64

home/petko/sfenv/lib/python2.7/site-packages/chainer/variable.pyc in backward(self, retain_grad)
    186                         y.grad = None
    187             for x, gx in zip(func.inputs, gxs):
--> 188                 x.grad = gx
    189                 if gx is not None:  # skip if gradient does not flow
    190                     add_cand(x.creator)

/home/petko/sfenv/lib/python2.7/site-packages/chainer/variable.pyc in grad(self, g)
    103             if g.dtype != self.data.dtype:
    104                 raise TypeError('Dtype of data and grad mismatch: %s != %s%s'
--> 105                                 % (self.data.dtype, g.dtype, error_msg))
    106             if g.shape != self.data.shape:
    107                 raise ValueError('Shape of data and grad mismatch: %s != %s%s'
```

The problem is caused  by the division in

https://github.com/pfnet/chainer/blob/master/chainer/functions/loss/sigmoid_cross_entropy.py#L46



Discussion:
Thank you for your issue report.

This issue is due to typecasting rule of numpy. 

``` python
>>> import numpy
>>> a = numpy.zeros(10, dtype=numpy.float32)
>>> a / 65535
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], dtype=float32)
>>> a / 65536
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
```

What size `t.shape[0]`?

Thanks for looking at this.

We were using very large batches when the error occured `t.shape[0] ~ 200000`.


Resolving patch:
diff --git a/chainer/functions/loss/sigmoid_cross_entropy.py b/chainer/functions/loss/sigmoid_cross_entropy.py
--- a/chainer/functions/loss/sigmoid_cross_entropy.py
+++ b/chainer/functions/loss/sigmoid_cross_entropy.py
@@ -43,7 +43,8 @@ def forward_gpu(self, inputs):
 
     def backward_cpu(self, inputs, grad_outputs):
         t, gloss = inputs[1], grad_outputs[0]
-        gx = gloss * (self.y - t.astype(self.y.dtype)) / t.shape[0]
+        dtype = self.y.dtype
+        gx = gloss * (self.y - t.astype(dtype)) / dtype.type(t.shape[0])
         return gx, None
 
     def backward_gpu(self, inputs, grad_outputs):


Test patch:
diff --git a/tests/chainer_tests/functions_tests/loss_tests/test_sigmoid_cross_entropy.py b/tests/chainer_tests/functions_tests/loss_tests/test_sigmoid_cross_entropy.py
--- a/tests/chainer_tests/functions_tests/loss_tests/test_sigmoid_cross_entropy.py
+++ b/tests/chainer_tests/functions_tests/loss_tests/test_sigmoid_cross_entropy.py
@@ -13,11 +13,15 @@
 from chainer.testing import condition
 
 
+@testing.parameterize(
+    {'shape': (4, 3)},
+    {'shape': (65536, 1)},  # too large shape causes int32 -> float64 issue
+)
 class TestSigmoidCrossEntropy(unittest.TestCase):
 
     def setUp(self):
-        self.x = numpy.random.uniform(-1, 1, (4, 3)).astype(numpy.float32)
-        self.t = numpy.random.randint(0, 2, (4, 3)).astype(numpy.int32)
+        self.x = numpy.random.uniform(-1, 1, self.shape).astype(numpy.float32)
+        self.t = numpy.random.randint(0, 2, self.shape).astype(numpy.int32)
 
     def check_forward(self, x_data, t_data, use_cudnn=True):
         x_val = chainer.Variable(x_data)
@@ -58,6 +62,10 @@ def check_backward(self, x_data, t_data, use_cudnn=True):
         loss.backward()
         self.assertEqual(None, t.grad)
 
+        # Skip too large case. That requires a long time.
+        if self.shape[0] == 65536:
+            return
+
         func = loss.creator
         f = lambda: func.forward((x.data, t.data))
         gx, = gradient_check.numerical_grad(f, (x.data,), (1,), eps=0.01)