Repository: chainer/chainer

Issue:
ChainerMN `test_create_mnbn_model.py` fails
**Error**
ChainerMN `test_create_mnbn_model.py` fails with following messages
```
=================================== FAILURES ===================================
_____________ TestCreateMnBnModel.test_create_mnbn_model_chain_gpu _____________

self = <test_create_mnbn_model.TestCreateMnBnModel testMethod=test_create_mnbn_model_chain_gpu>

    @chainer.testing.attr.gpu
    def test_create_mnbn_model_chain_gpu(self):
>       self.check_create_mnbn_model_chain(gpu=True)

chainermn_tests/links_tests/test_create_mnbn_model.py:107:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
chainermn_tests/links_tests/test_create_mnbn_model.py:56: in check_create_mnbn_model_chain
    mnbn_model(x)
../chainer/link.py:294: in __call__
    out = forward(*args, **kwargs)
chainermn_tests/links_tests/test_create_mnbn_model.py:19: in forward
    return chainer.functions.relu(self.bn(self.conv(x)))
../chainermn/links/batch_normalization.py:111: in __call__
    ret = func.apply((x, gamma, beta))[0]
../chainer/function_node.py:325: in apply
    outputs = self.forward(in_data)
../chainer/functions/normalization/batch_normalization.py:401: in forward
    running_var=self.running_var)
../chainer/functions/normalization/batch_normalization.py:80: in forward
    running_mean, running_var)
cupy/core/_kernel.pyx:539: in cupy.core._kernel.ElementwiseKernel.__call__
    ???
cupy/core/_kernel.pyx:582: in cupy.core._kernel.ElementwiseKernel._decide_params_type
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   TypeError: Type is mismatched. mean <class 'numpy.float64'> <class 'numpy.float32'> U

cupy/core/_kernel.pyx:292: TypeError
====================== 1 failed, 1 passed in 3.63 seconds ======================
```


**Main Cause**
`mean` and `var` have a different dtype `<numpy.float64>` from others `<numpy.float32> `, which violate the assumption that all the variables share the same dtype at https://github.com/chainer/chainer/blob/23f1fdbe7c534dc48538eaaff660c376a151049e/chainer/functions/normalization/batch_normalization.py#L66

The cupy elementwise code was from old chainermn implementation, and chainermn got the dtype of `mean` and `var` only from `gamma` https://github.com/chainer/chainer/blob/b02a7c402bef7f912c2d7109f07bda9438c3d57f/chainermn/functions/batch_normalization.py#L39, which makes all the dtype the same.
While chainer has changed to promote the dtype of `gamma` with `x` (<numpy.float64>) ,https://github.com/chainer/chainer/blob/b02a7c402bef7f912c2d7109f07bda9438c3d57f/chainer/functions/normalization/batch_normalization.py#L222, which makes them `float64`, and causes this bug.


Discussion:

Resolving patch:
diff --git a/chainer/functions/normalization/batch_normalization.py b/chainer/functions/normalization/batch_normalization.py
--- a/chainer/functions/normalization/batch_normalization.py
+++ b/chainer/functions/normalization/batch_normalization.py
@@ -64,7 +64,7 @@ def forward(self, axis, gamma, x, xp, expander,
                 running_var += (1 - decay) * adjust * var
             else:
                 cuda.elementwise(
-                    'U mean, U var, U decay, U adjust',
+                    'T mean, T var, U decay, U adjust',
                     'U r_mean, U r_var',
                     '''
                     r_mean = r_mean * decay + mean * (1 - decay);


Test patch: