Repository: chainer/chainer

Issue:
F.mean_absolute_error numerically unstable with float16 arrays
In #5053, float16 support has been enabled for [F.mean_absolute_error](https://github.com/chainer/chainer/blob/master/chainer/functions/loss/mean_absolute_error.py), but it seems to produce NaN values quite easily. Usually this happens when using big batch sizes and/or if the absolute error difference is large.

The calculation is done by summing over all the absolute differences, and then dividing by the number of elements in the array. However, it appears that the summing can produce large numbers outside the possible range for `float16`. The current implementation can be broken down as follows:

```python
def forward_cpu(self, inputs):
    x0, x1 = inputs
    self.diff = x0 - x1
    diff = self.diff.ravel()
    abs_diff = abs(diff)
    summed_abs_diff = abs_diff.sum()  # numerically unstable, can result in inf
    mean_abs_error = np.array(summed_abs_diff / diff.size, dtype=diff.dtype)
    return mean_abs_error
```

Code to reproduce error:

```python
import chainer.functions as F
import numpy as np

a = np.full(shape=(64,1,16,16), fill_value=2, dtype=np.float16)
b = np.full(shape=(64,1,16,16), fill_value=-2, dtype=np.float16)

loss = F.mean_absolute_error(a,b)
# /home/user/.local/share/virtualenvs/.../lib/python3.6/site-packages/numpy/core/_methods.py:36: RuntimeWarning: overflow encountered in reduce
#  return umr_sum(a, axis, dtype, out, keepdims, initial)
# variable(inf)

loss = F.mean_absolute_error(a.astype("float32"), b.astype("float32"))
# variable(4.)
```

Note that the actual loss (4) would still be valid in the float16 range, it is just that summing over many values results in an `inf`, which cannot then be divided to get a proper number.

Workaround ideas:

I've noticed the new `mixed16` mode that was implemented in #6456, and was wondering if there might be a similar way to do the intermediate calculations in `float32`, and cast the result back into `float16`? Thoughts?

System info:
```
Platform: Linux-4.15.0-46-generic-x86_64-with-debian-buster-sid
Chainer: 6.0.0b3
NumPy: 1.16.2
CuPy:
  CuPy Version          : 6.0.0b3
  CUDA Root             : /usr/local/cuda
  CUDA Build Version    : 10000
  CUDA Driver Version   : 10010
  CUDA Runtime Version  : 10000
  cuDNN Build Version   : 7402
  cuDNN Version         : 7402
  NCCL Build Version    : 2307
  NCCL Runtime Version  : 2307
iDeep: Not Available
```


Discussion:
I found that `numpy.mean` implicitly casted input to float32 before averaging if the input was fp16.
See NOTE of https://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html.

This seems to resolve this to some extent.
Thanks for your report and suggestion. To use a higher precision intermediate dtype should make sense and is actually applied in #6366 and #6442. I'll fix `F.mean_absolute_error` with the same approach.


Resolving patch:
diff --git a/chainer/functions/loss/mean_absolute_error.py b/chainer/functions/loss/mean_absolute_error.py
--- a/chainer/functions/loss/mean_absolute_error.py
+++ b/chainer/functions/loss/mean_absolute_error.py
@@ -6,6 +6,15 @@
 from chainer.utils import type_check
 
 
+def _get_intermediate_dtype(dtype):
+    # Returns the dtype for intermediate calculation.
+    # For float16 input, float32 is used.
+    # Otherwise the same dtype as the parameter is used.
+    if dtype == numpy.float16:
+        return numpy.float32
+    return dtype
+
+
 class MeanAbsoluteError(function_node.FunctionNode):
 
     """Mean absolute error function."""
@@ -21,14 +30,19 @@ def check_type_forward(self, in_types):
     def forward_cpu(self, inputs):
         x0, x1 = inputs
         self.diff = x0 - x1
-        diff = self.diff.ravel()
-        return numpy.array(abs(diff).sum() / diff.size, dtype=diff.dtype),
+        orig_dtype = self.diff.dtype
+        dtype = _get_intermediate_dtype(orig_dtype)
+        diff = self.diff.ravel().astype(dtype, copy=False)
+        return numpy.array(abs(diff).sum() / diff.size, dtype=orig_dtype),
 
     def forward_gpu(self, inputs):
         x0, x1 = inputs
         self.diff = x0 - x1
-        diff = self.diff.ravel()
-        return abs(diff).sum() / diff.dtype.type(diff.size),
+        orig_dtype = self.diff.dtype
+        dtype = _get_intermediate_dtype(orig_dtype)
+        diff = self.diff.ravel().astype(dtype, copy=False)
+        return (abs(diff).sum() / diff.dtype.type(diff.size)).astype(
+            orig_dtype, copy=False),
 
     def backward(self, indexes, grad_outputs):
         gy, = grad_outputs


Test patch:
diff --git a/tests/chainer_tests/functions_tests/loss_tests/test_mean_absolute_error.py b/tests/chainer_tests/functions_tests/loss_tests/test_mean_absolute_error.py
--- a/tests/chainer_tests/functions_tests/loss_tests/test_mean_absolute_error.py
+++ b/tests/chainer_tests/functions_tests/loss_tests/test_mean_absolute_error.py
@@ -3,9 +3,11 @@
 import numpy
 
 import chainer
+from chainer.backends import cuda
 from chainer import functions
 from chainer import testing
 from chainer import utils
+from chainer.testing import attr
 from chainer.utils import type_check
 
 
@@ -83,4 +85,23 @@ def test_invalid_dtype2(self):
             functions.mean_absolute_error(x0, x1)
 
 
+# See chainer#6702.
+class TestMeanAbsoluteErrorFP16Overflow(unittest.TestCase):
+
+    def check_fp16_overflow(self, xp):
+        x0 = chainer.Variable(xp.full(
+            shape=(64, 1, 16, 16), fill_value=2, dtype=xp.float16))
+        x1 = chainer.Variable(xp.full(
+            shape=(64, 1, 16, 16), fill_value=-2, dtype=xp.float16))
+        loss = functions.mean_absolute_error(x0, x1)
+        self.assertFalse(xp.isinf(loss.array))
+
+    def test_fp16_overflow_cpu(self):
+        self.check_fp16_overflow(numpy)
+
+    @attr.gpu
+    def test_fp16_overflow_gpu(self):
+        self.check_fp16_overflow(cuda.cupy)
+
+
 testing.run_module(__name__, __file__)