Repository: chainer/chainer

Issue:
Unable to run mnist dual parallel example
Python version: 3.7
OS/Platform: Red Hat 7.6

I am unable to run the chainermn dual parallel example (chainer/examples/chainermn/mnist/train_mnist_dual_parallel.py) with chainer version `7.1.0`. I was able to run this example previously with chainer version `6.2.0`.

The error I get is

```
  File "chainer/chainermn/mnist/train_mnist_dual_parallel.py", line 156, in <module>
    main()
  File "chainer/chainermn/mnist/train_mnist_dual_parallel.py", line 152, in main
    trainer.run()
  File "/opt/chainer/chainer/training/trainer.py", line 376, in run
    six.reraise(*exc_info)
  File "/opt/chainer/six.py", line 703, in reraise
    raise value
  File "/opt/chainer/chainer/training/trainer.py", line 346, in run
    entry.extension(self)
  File "/opt/chainer/chainer/training/extensions/evaluator.py", line 180, in __call__
    result = self.evaluate()
  File "/opt/chainer/chainermn/extensions/multi_node_evaluator.py", line 229, in new_evaluate
    array0 = list(local_mean_dict.values())[0]
IndexError: list index out of range
```

I noticed that since release 6.2.0 some modification has been made to multi_node_evaluator.py, specifically the addition of the following lines
```
        # ChainerX support:
        # We need convert chainerx ndarray to Native array because
        #   (1) allreduce_obj is used to compute global mean values, since
        #       a simple allreduce operation cannot be applied in evaluation.
        #   (2) allreduce_obj calls mpi4py.allreduce, which pickles the object
        #   (3) chainerx.ndarray preserves CUDA device internally when pickled
        #   (4) An error will occur when an ndarray is unpickled in another
        #       process
        array0 = list(local_mean_dict.values())[0]
        xp = backend.get_array_module(array0)
        if xp == chx and array0.device.backend.name == 'cuda':
            # Results of evaluation is fairly small, so
            # the ndarray is transferred to CPU and allreduce()-ed.
            # NOTE: Matrices for evaluation are transferred to the host memory
            # and sent via MPI instead of NCCL. Although evaluation matrices
            # are small in most cases, this is a potential performance issue.
            local_mean_dict = {
                name: chx.to_numpy(value)
                for name, value in local_mean_dict.items()
            }
```
I have also installed ChainerX  but I still get the same error. The Data Parallel (`train_mnist.py`) and Model Parallel (`train_mnist_model_parallel.py`) examples are able to run successfully.


Discussion:
Following up on this issue. The `train_mnist_dual_parallel.py` example no longer runs into the  `IndexError: list index out of range` error when I replace the following https://github.com/chainer/chainer/blob/v7.1.0/chainermn/extensions/multi_node_evaluator.py#L229-L248 (copied below)
```
        array0 = list(local_mean_dict.values())[0]
        xp = backend.get_array_module(array0)
        if xp == chx and array0.device.backend.name == 'cuda':
            # Results of evaluation is fairly small, so
            # the ndarray is transferred to CPU and allreduce()-ed.
            # NOTE: Matrices for evaluation are transferred to the host memory
            # and sent via MPI instead of NCCL. Although evaluation matrices
            # are small in most cases, this is a potential performance issue.
            local_mean_dict = {
                name: chx.to_numpy(value)
                for name, value in local_mean_dict.items()
            }

        global_mean_dict = {
            name:
            self._mn_communicator.allreduce_obj(
                value) / self._mn_communicator.size
            for name, value in sorted(local_mean_dict.items())
        }
        return global_mean_dict
```
with what it previously was in the 6.2.0 release (https://github.com/chainer/chainer/blob/v6.2.0/chainermn/extensions/multi_node_evaluator.py#L29-L35)
```
        global_mean_dict = {
            name:
            self._mn_communicator.allreduce_obj(
                value) / self._mn_communicator.size
            for name, value in sorted(local_mean_dict.items())
        }
        return global_mean_dict
```
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs. Thank you for your contributions.

I could eproduce this error with 7.4.0

Resolving patch:
diff --git a/chainermn/extensions/multi_node_evaluator.py b/chainermn/extensions/multi_node_evaluator.py
--- a/chainermn/extensions/multi_node_evaluator.py
+++ b/chainermn/extensions/multi_node_evaluator.py
@@ -226,18 +226,21 @@ def new_evaluate(self):
         #   (3) chainerx.ndarray preserves CUDA device internally when pickled
         #   (4) An error will occur when an ndarray is unpickled in another
         #       process
-        array0 = list(local_mean_dict.values())[0]
-        xp = backend.get_array_module(array0)
-        if xp == chx and array0.device.backend.name == 'cuda':
-            # Results of evaluation is fairly small, so
-            # the ndarray is transferred to CPU and allreduce()-ed.
-            # NOTE: Matrices for evaluation are transferred to the host memory
-            # and sent via MPI instead of NCCL. Although evaluation matrices
-            # are small in most cases, this is a potential performance issue.
-            local_mean_dict = {
-                name: chx.to_numpy(value)
-                for name, value in local_mean_dict.items()
-            }
+        arrays = list(local_mean_dict.values())
+        if len(arrays) > 0:
+            array0 = list(local_mean_dict.values())[0]
+            xp = backend.get_array_module(array0)
+            if xp == chx and array0.device.backend.name == 'cuda':
+                # Results of evaluation is fairly small, so
+                # the ndarray is transferred to CPU and allreduce()-ed.
+                # NOTE: Matrices for evaluation are transferred to the
+                # host memory and sent via MPI instead of NCCL.
+                # Although evaluation matrices are small in most cases,
+                # this is a potential performance issue.
+                local_mean_dict = {
+                    name: chx.to_numpy(value)
+                    for name, value in local_mean_dict.items()
+                }
 
         global_mean_dict = {
             name:


Test patch: