Repository: optuna/optuna

Issue:
Occasional test failure with `ChainerMNStudy` with Python 3.5.
## Expected behavior

Tests should pass.

## Error messages, stack traces, or logs

https://app.circleci.com/pipelines/github/optuna/optuna/1171/workflows/41e626df-8501-46ca-a724-9b1c27079253/jobs/45897

## Additional context (optional)

Seems to be specific to Python 3.5 and only happen randomly.


Discussion:
I also encountered the failure in Python 3.7.

https://app.circleci.com/pipelines/github/optuna/optuna/1195/workflows/eccc4813-e4b0-443f-acba-90992ac0b738/jobs/46106/steps
Also in Python 3.8. Python version seems to be irrelevant. https://app.circleci.com/pipelines/github/optuna/optuna/1202/workflows/77a93dea-69d5-48d1-a911-fcad5878609e/jobs/46167/steps
We can reproduce the error by running the following command multiple times:

```
docker run -v $(pwd):/workspaces -e OMP_NUM_THREADS=1 optuna/optuna:py3.7-dev mpirun -n 2 --allow-run-as-root -- pytest tests/integration_tests/test_chainermn.py -k test_failure -vv --showlocals
```
We should preferable deal with this as soon as possible and I tried several times but couldn't reproduce it locally. Did you encounter it frequently? 

Note: Some discussions offline that https://github.com/optuna/optuna/pull/1380 could be related but I'm not sure.
@hvy Can you modify `test_chainermn.py` as below and run the command again?

```diff
class TestChainerMNStudy(object):
    ...
    @staticmethod
+   @pytest.mark.parametrize("x", range(10))  # run this test 10 times
    @pytest.mark.parametrize("storage_mode", STORAGE_MODES)
-   def test_failure(storage_mode, comm):
+   def test_failure(storage_mode, x, comm):
        # type: (str, CommunicatorBase) -> None
        with MultiNodeStorageSupplier(storage_mode, comm) as storage:
```
@harupy Thank you for sharing the command. I can reproduce the failure without the hack in https://github.com/optuna/optuna/issues/1395#issuecomment-647304646.

The actual error seems to be the last trial (Trial 20), which is still recognized as `RUNNING`.

```console
 FrozenTrial(number=20, value=None, datetime_start=datetime.datetime(2020, 6, 22, 11, 40, 58, 415896), datetime_complete=None, params={}, distributions={}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=21, state=TrialState.RUNNING)]
```

Resolving patch:
diff --git a/optuna/integration/chainermn.py b/optuna/integration/chainermn.py
--- a/optuna/integration/chainermn.py
+++ b/optuna/integration/chainermn.py
@@ -118,11 +118,13 @@ def optimize(
 
         if self.comm.rank == 0:
             func_mn = _ChainerMNObjectiveFunc(func, self.comm)
-            self.delegate.optimize(func_mn, n_trials=n_trials, timeout=timeout, catch=catch)
-            self.comm.mpi_comm.bcast(False)
+            try:
+                self.delegate.optimize(func_mn, n_trials=n_trials, timeout=timeout, catch=catch)
+            finally:
+                self.comm.mpi_comm.bcast(False)
         else:
+            has_next_trial = self.comm.mpi_comm.bcast(None)
             while True:
-                has_next_trial = self.comm.mpi_comm.bcast(None)
                 if not has_next_trial:
                     break
                 try:
@@ -137,6 +139,8 @@ def optimize(
                     pass
                 except catch:
                     pass
+                finally:
+                    has_next_trial = self.comm.mpi_comm.bcast(None)
 
     def __getattr__(self, attr_name):
         # type: (str) -> Any


Test patch: