Repository: chainer/chainer

Issue:
extensions.ParameterStatistics cannot work with default parameter in GPU mode
`extensions.ParameterStatistics` raise error in GPU mode at `numpy.count_nonzero` or `numpy.percentile` at https://github.com/chainer/chainer/blob/master/chainer/training/extensions/parameter_statistics.py#L106

The type of `link.param` is cupy.ndarray but these functions cannot take this type.
Other functions (like numpy.mean) use the instance method (like hoge.mean).


Discussion:
Thank you for reporting!
Can you provide a reproduction code and error log?
I've ran into the same problem with chainer 1.24.0.

The error was the following:

```
Traceback (most recent call last):
  File "/home/koreyou/.pyenv/versions/2.7.12/lib/python2.7/pdb.py", line 1314, in main
    pdb._runscript(mainpyfile)
  File "/home/koreyou/.pyenv/versions/2.7.12/lib/python2.7/pdb.py", line 1233, in _runscript
    self.run(statement)
  File "/home/koreyou/.pyenv/versions/2.7.12/lib/python2.7/bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "bin/my_training_script.py", line 12, in <module>
    """
  File "/home/koreyou/.pyenv/versions/myprogram/lib/python2.7/site-packages/chainer/training/trainer.py", line 298, in run
    entry.extension(self)
  File "/home/koreyou/.pyenv/versions/myprogram/lib/python2.7/site-packages/chainer/training/extensions/parameter_statistics.py", line 106, in __call__
    value = function(params)
  File "/home/koreyou/.pyenv/versions/myprogram/lib/python2.7/site-packages/chainer/training/extensions/parameter_statistics.py", line 57, in <lambda>
    99.87))
  File "/home/koreyou/.pyenv/versions/myprogram/lib/python2.7/site-packages/numpy/lib/function_base.py", line 4116, in percentile
    else:
  File "/home/koreyou/.pyenv/versions/myprogram/lib/python2.7/site-packages/numpy/lib/function_base.py", line 3828, in _ureduce
    >>> A = fft(window, 2048) / 25.5
  File "/home/koreyou/.pyenv/versions/myprogram/lib/python2.7/site-packages/numpy/core/numeric.py", line 583, in asanyarray
    return array(a, dtype, copy=False, order=order, subok=True)
ValueError: object __array__ method not producing an array
```

You can reproduce the bug in following code:

```python
import chainer
import chainer.functions as F
import chainer.links as L
from chainer import training
from chainer.training import extensions

# Network definition
class MLP(chainer.Chain):
    def __init__(self, n_out):
        super(MLP, self).__init__()
        with self.init_scope():
            self.l1 = L.Linear(None, n_out)

    def __call__(self, x):
        return self.l1(x)

# Set up a neural network to train
# Classifier reports softmax cross entropy loss and accuracy at every
# iteration, which will be used by the PrintReport extension below.
model = L.Classifier(MLP(10))
# Make a specified GPU current
chainer.cuda.get_device_from_id(0).use()
model.to_gpu()  # Copy the model to the GPU

# Setup an optimizer
optimizer = chainer.optimizers.Adam()
optimizer.setup(model)

# Load the MNIST dataset
train, _ = chainer.datasets.get_mnist()

train_iter = chainer.iterators.SerialIterator(train, 32)

# Set up a trainer
updater = training.StandardUpdater(train_iter, optimizer, device=0)
trainer = training.Trainer(updater, (5, 'epoch'), out='result')

trainer.extend(extensions.ParameterStatistics(model))

# Write a log of evaluation statistics for each epoch
trainer.extend(extensions.LogReport())

# Run the training
trainer.run()
```

The `a` on the error code was `cupy.core.core.ndarray`.
I can see the reason behind this problem with following code:

```python
import cupy
import numpy as np

a = cupy.ones(3)
print a.__array__()  # this is actually fine
np.asanyarray(a)  # this raises the error
```
I am not sure this is an efficient way, but in my environment, the issue might be solved by simply wrapping params (at line 106 in parameter_statistics.py) by chainer.cuda.to_cpu, i.e., rewriting the line into 
`value = function(chainer.cuda.to_cpu(params))`
with
`import chainer`

Thanks for reporting this issue. 

The extension constructor accepts a `statistics` argument which is expected to be a `dict` where the keys are log entry keys and values are functions such as `numpy.max` or in the case of GPUs `cupy.max`. 

This issue occurs because the default values are all numpy functions. If you know beforehand that you'll be using the GPU, you may want to pass a different `dict` where the values are `cupy` functions.

Does this solve your problem?
I see. You mean that we pass a different `statistics` to `ParameterStatistics`?
Below is an example:
```
gpu_statistics = { 
        'mean': cupy.mean,
        'std': cupy.std,
        'min': cupy.min,
        'max': cupy.max }
trainer.extend(extensions.ParameterStatistics(model, statistics=gpu_statistics))
```

I did not try it, but it sounds good. It would be an alternative way to use `to_cpu`.
If there are any other way to handle `model` regardless of CPU or GPU, it would be great, but the current solutions are also fine for me at this point.
Yes exactly. In most cases, I think `cupy` functions are faster. If you have multiple devices, you might want to wrap the functions above where you get the device from the input, e.g. `lambda x: xp = cuda.get_array_module(x); xp...` . 
@Hiroshiba @t-sakai-kure Is it fine to close this issue?
It's fine with me. If you can update the code so that the default behavior automatically chooses numpy or cupy, it would be useful for other users. 

Resolving patch:
diff --git a/chainer/training/extensions/parameter_statistics.py b/chainer/training/extensions/parameter_statistics.py
--- a/chainer/training/extensions/parameter_statistics.py
+++ b/chainer/training/extensions/parameter_statistics.py
@@ -1,6 +1,8 @@
 import numpy
 import six
 
+import chainer
+from chainer.backends import cuda
 from chainer import reporter
 from chainer.training import extension
 from chainer.training import trigger as trigger_module
@@ -38,6 +40,10 @@ class ParameterStatistics(extension.Extension):
         prefix (str): Optional prefix to prepend to the report keys.
         trigger: Trigger that decides when to aggregate the results and report
             the values.
+        skip_nan_params (bool): If ``True``, statistics are not computed for
+            parameters including NaNs and a single NaN value is immediately
+            reported instead. Otherwise, this extension will simply try to
+            compute the statistics without performing any checks for NaNs.
     """
     default_name = 'parameter_statistics'
     priority = extension.PRIORITY_WRITER
@@ -47,24 +53,25 @@ class ParameterStatistics(extension.Extension):
                            '{function_name}')
 
     default_statistics = {
-        'mean': numpy.mean,
-        'std': numpy.std,
-        'min': numpy.min,
-        'max': numpy.max,
-        'zeros': lambda x: numpy.count_nonzero(x == 0),
-        'percentile': lambda x: numpy.percentile(x, (0.13, 2.28, 15.87,
-                                                     50, 84.13, 97.72,
-                                                     99.87))
+        'mean': lambda x: cuda.get_array_module(x).mean(x),
+        'std': lambda x: cuda.get_array_module(x).std(x),
+        'min': lambda x: cuda.get_array_module(x).min(x),
+        'max': lambda x: cuda.get_array_module(x).max(x),
+        'zeros': lambda x: cuda.get_array_module(x).count_nonzero(x == 0),
+        'percentile': lambda x: cuda.get_array_module(x).percentile(
+            x, (0.13, 2.28, 15.87, 50, 84.13, 97.72, 99.87))
     }
 
     def __init__(self, links, statistics=default_statistics,
                  report_params=True, report_grads=True, prefix=None,
-                 trigger=(1, 'epoch')):
+                 trigger=(1, 'epoch'), skip_nan_params=False):
 
         if not isinstance(links, (list, tuple)):
             links = links,
         self._links = links
 
+        if statistics is None:
+            statistics = {}
         self._statistics = statistics
 
         attrs = []
@@ -77,6 +84,7 @@ def __init__(self, links, statistics=default_statistics,
         self._prefix = prefix
         self._trigger = trigger_module.get_trigger(trigger)
         self._summary = reporter.DictSummary()
+        self._skip_nan_params = skip_nan_params
 
     def __call__(self, trainer):
         """Execute the statistics extension.
@@ -103,7 +111,12 @@ def __call__(self, trainer):
                         # since the statistics function should make no
                         # assumption about the axes
                         params = getattr(param, attr_name).ravel()
-                        value = function(params)
+                        if (self._skip_nan_params
+                            and (cuda.get_array_module(params).isnan(params)
+                                 .any())):
+                            value = numpy.nan
+                        else:
+                            value = function(params)
                         key = self.report_key_template.format(
                             prefix=self._prefix + '/' if self._prefix else '',
                             link_name=link_name,
@@ -111,7 +124,8 @@ def __call__(self, trainer):
                             attr_name=attr_name,
                             function_name=function_name
                         )
-                        if hasattr(value, '__iter__'):
+                        if (isinstance(value, chainer.get_array_types())
+                                and value.size > 1):
                             # Append integer indices to the keys if the
                             # statistic function return multiple values
                             statistics.update({'{}/{}'.format(key, i): v for


Test patch:
diff --git a/tests/chainer_tests/training_tests/extensions_tests/test_parameter_statistics.py b/tests/chainer_tests/training_tests/extensions_tests/test_parameter_statistics.py
--- a/tests/chainer_tests/training_tests/extensions_tests/test_parameter_statistics.py
+++ b/tests/chainer_tests/training_tests/extensions_tests/test_parameter_statistics.py
@@ -3,68 +3,168 @@
 import unittest
 
 import mock
-import numpy
 import six
 
 import chainer
+from chainer.backends import cuda
 from chainer import testing
 from chainer import training
 from chainer.training import extensions
 
 
+def _get_mocked_trainer(links, stop_trigger=(10, 'iteration')):
+    updater = mock.Mock()
+    optimizer = mock.Mock()
+    target = mock.Mock()
+    target.namedlinks.return_value = [
+        (str(i), link) for i, link in enumerate(links)]
+
+    optimizer.target = target
+    updater.get_all_optimizers.return_value = {'optimizer_name': optimizer}
+    updater.iteration = 0
+    updater.epoch = 0
+    updater.epoch_detail = 0
+    updater.is_new_epoch = True
+    iter_per_epoch = 10
+
+    def update():
+        time.sleep(0.001)
+        updater.iteration += 1
+        updater.epoch = updater.iteration // iter_per_epoch
+        updater.epoch_detail = updater.iteration / iter_per_epoch
+        updater.is_new_epoch = updater.epoch == updater.epoch_detail
+
+    updater.update = update
+
+    return training.Trainer(updater, stop_trigger)
+
+
 class TestParameterStatisticsBase(object):
 
     def setUp(self):
-        self.trainer = _get_mocked_trainer()
+        self.trainer = _get_mocked_trainer(self.links)
 
-    @property
-    def links(self):
-        links = []
-        for optimizer in six.itervalues(
-                self.trainer.updater.get_all_optimizers()):
-            for _, link in optimizer.target.namedlinks():
-                links.append(link)
-        return links
+    def create_extension(self, skip_statistics=False):
+        kwargs = {
+            'statistics': self.statistics if not skip_statistics else None,
+            'report_params': self.report_params,
+            'report_grads': self.report_grads,
+            'prefix': self.prefix,
+            'skip_nan_params': True  # avoid warnings when grads are nan
+        }
+
+        return extensions.ParameterStatistics(self.links, **kwargs)
 
 
 @testing.parameterize(
-    {'statistics': {'min': numpy.min}, 'expect': 4},
-    {'statistics': {'mean': numpy.mean, 'std': numpy.std}, 'expect': 8},
-    {'statistics': extensions.ParameterStatistics.default_statistics,
-     'expect': 48},
-    {'statistics': {}, 'expect': 0}
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': {'min': lambda x: cuda.get_array_module(x).min(x)},
+        'report_params': True,
+        'report_grads': True,
+        'prefix': None,
+        'expect': 4
+    },
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': {'min': lambda x: cuda.get_array_module(x).min(x)},
+        'report_params': False,
+        'report_grads': True,
+        'prefix': 'test',
+        'expect': 2
+    },
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': {'min': lambda x: cuda.get_array_module(x).min(x)},
+        'report_params': True,
+        'report_grads': False,
+        'prefix': None,
+        'expect': 2
+    },
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': {'min': lambda x: cuda.get_array_module(x).min(x)},
+        'report_params': False,
+        'report_grads': False,
+        'prefix': 'test',
+        'expect': 0
+    },
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': extensions.ParameterStatistics.default_statistics,
+        'report_params': True,
+        'report_grads': True,
+        'prefix': None,
+        'expect': 36
+    },
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': extensions.ParameterStatistics.default_statistics,
+        'report_params': True,
+        'report_grads': False,
+        'prefix': 'test',
+        'expect': 24
+    },
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': extensions.ParameterStatistics.default_statistics,
+        'report_params': False,
+        'report_grads': True,
+        'prefix': None,
+        'expect': 12
+    },
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': extensions.ParameterStatistics.default_statistics,
+        'report_params': False,
+        'report_grads': False,
+        'prefix': 'test',
+        'expect': 0
+    }
 )
 class TestParameterStatistics(TestParameterStatisticsBase, unittest.TestCase):
 
     def test_report(self):
-        extension = extensions.ParameterStatistics(self.links,
-                                                   statistics=self.statistics)
-        self.trainer.extend(extension)
+        self.trainer.extend(self.create_extension())
         self.trainer.run()
-
         self.assertEqual(len(self.trainer.observation), self.expect)
 
     def test_report_late_register(self):
-        extension = extensions.ParameterStatistics(self.links, statistics={})
+        extension = self.create_extension(skip_statistics=True)
         for name, function in six.iteritems(self.statistics):
             extension.register_statistics(name, function)
         self.trainer.extend(extension)
         self.trainer.run()
-
         self.assertEqual(len(self.trainer.observation), self.expect)
 
     def test_report_key_pattern(self):
-        extension = extensions.ParameterStatistics(self.links)
-        self.trainer.extend(extension)
+        self.trainer.extend(self.create_extension())
         self.trainer.run()
 
         pattern = r'^(.+/){2,}(data|grad)/.+[^/]$'
         for name in six.iterkeys(self.trainer.observation):
-            self.assertTrue(re.match(pattern, name))
+            if self.prefix is not None:
+                assert name.startswith(self.prefix)
+
+            match = re.match(pattern, name)
+            assert match is not None
+            if self.report_params and self.report_grads:
+                pass
+            elif self.report_params:
+                assert 'data' == match.group(2)
+            elif self.report_grads:
+                assert 'grad' == match.group(2)
 
 
 @testing.parameterize(
-    {'statistics': {'one': lambda x: 1.0}, 'expect': 1.0}
+    {
+        'links': [chainer.links.Linear(3, 2)],
+        'statistics': {'one': lambda x: 1.0},
+        'report_params': True,
+        'report_grads': True,
+        'prefix': 'test',
+        'expect': 1.0
+    }
 )
 class TestParameterStatisticsCustomFunction(TestParameterStatisticsBase,
                                             unittest.TestCase):
@@ -79,66 +179,4 @@ def test_custom_function(self):
             self.assertEqual(value, self.expect)
 
 
-@testing.parameterize(
-    {'statistics': extensions.ParameterStatistics.default_statistics}
-)
-class TestParameterStatisticsArguments(TestParameterStatisticsBase,
-                                       unittest.TestCase):
-
-    def test_skip_params(self):
-        extension = extensions.ParameterStatistics(
-            self.links, statistics=self.statistics, report_params=False)
-        self.trainer.extend(extension)
-        self.trainer.run()
-
-        for name in six.iterkeys(self.trainer.observation):
-            self.assertIn('grad', name)
-            self.assertNotIn('data', name)
-
-    def test_skip_grads(self):
-        extension = extensions.ParameterStatistics(
-            self.links, statistics=self.statistics, report_grads=False)
-        self.trainer.extend(extension)
-        self.trainer.run()
-
-        for name in six.iterkeys(self.trainer.observation):
-            self.assertIn('data', name)
-            self.assertNotIn('grad', name)
-
-    def test_report_key_prefix(self):
-        extension = extensions.ParameterStatistics(
-            self.links, statistics=self.statistics, prefix='prefix')
-        self.trainer.extend(extension)
-        self.trainer.run()
-
-        for name in six.iterkeys(self.trainer.observation):
-            self.assertTrue(name.startswith('prefix'))
-
-
-def _get_mocked_trainer(stop_trigger=(10, 'iteration')):
-    updater = mock.Mock()
-    optimizer = mock.Mock()
-    target = mock.Mock()
-    target.namedlinks.return_value = ('link_name',
-                                      chainer.links.Linear(10, 10)),
-    optimizer.target = target
-    updater.get_all_optimizers.return_value = {'optimizer_name': optimizer}
-    updater.iteration = 0
-    updater.epoch = 0
-    updater.epoch_detail = 0
-    updater.is_new_epoch = True
-    iter_per_epoch = 10
-
-    def update():
-        time.sleep(0.001)
-        updater.iteration += 1
-        updater.epoch = updater.iteration // iter_per_epoch
-        updater.epoch_detail = updater.iteration / iter_per_epoch
-        updater.is_new_epoch = updater.epoch == updater.epoch_detail
-
-    updater.update = update
-
-    return training.Trainer(updater, stop_trigger)
-
-
 testing.run_module(__name__, __file__)