Repository: ray-project/ray

Issue:
[tune] BOHB with DQN and MountainCar-v0 fails
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->


### What is the problem?
I am using BOHB to optimize the hyperparameters for the DQN algorithm in order to solve the MountainCar-v0 problem.
I alway run into the following issue (even if I use different values for num_samples): 
```python
== Status ==
Memory usage on this node: 7.0/15.6 GiB
Using HyperBand: num_stopped=832 total_brackets=3
Round #0:
  None
  Bracket(Max Size (n)=2, Milestone (r)=1458, completed=100.0%): {RUNNING: 1, TERMINATED: 833} 
  Bracket(Max Size (n)=324, Milestone (r)=8, completed=47.3%): {PAUSED: 166} 
Resources requested: 4/32 CPUs, 0/0 GPUs, 0.0/8.69 GiB heap, 0.0/2.98 GiB objects
Result logdir: /home/dl-user/ray_results/MCv0_DQN_BOHB
Number of trials: 1000 (166 PAUSED, 1 RUNNING, 833 TERMINATED)
+-----------------------------+------------+----------------------+-------------------+-------------+--------------------+--------+------------------+--------+----------+
| Trial name                  | status     | loc                  | batch_mode        |          lr |   train_batch_size |   iter |   total time (s) |     ts |   reward |
|-----------------------------+------------+----------------------+-------------------+-------------+--------------------+--------+------------------+--------+----------|
| DQN_MountainCar-v0_0428be42 | PAUSED     |                      | truncate_episodes | 1.99095e-05 |                408 |      2 |         25.6885  |   4032 |  -200    |
| DQN_MountainCar-v0_0428be45 | PAUSED     |                      | truncate_episodes | 0.000382289 |                211 |      2 |         24.7536  |   5040 |  -200    |
| DQN_MountainCar-v0_0428be48 | PAUSED     |                      | truncate_episodes | 0.000324929 |                233 |      2 |         25.5532  |   5040 |  -200    |
| DQN_MountainCar-v0_0747e5f2 | PAUSED     |                      | truncate_episodes | 0.000114766 |                 38 |      2 |         23.8492  |   7056 |  -200    |
| DQN_MountainCar-v0_0747e5f5 | PAUSED     |                      | truncate_episodes | 9.1226e-05  |                200 |      2 |         24.2349  |   5040 |  -200    |
| DQN_MountainCar-v0_08218bf0 | PAUSED     |                      | truncate_episodes | 0.000284028 |                 69 |      2 |         25.3671  |   7056 |  -200    |
| DQN_MountainCar-v0_093c0b8c | PAUSED     |                      | truncate_episodes | 0.00237606  |                114 |      2 |         23.3935  |   6048 |  -200    |
| DQN_MountainCar-v0_0a55eae6 | PAUSED     |                      | truncate_episodes | 0.000417829 |                111 |      2 |         23.4849  |   6048 |  -200    |
| DQN_MountainCar-v0_0b307d56 | PAUSED     |                      | truncate_episodes | 0.000196047 |                 59 |      2 |         23.1338  |   7056 |  -200    |
| DQN_MountainCar-v0_0eedea91 | PAUSED     |                      | truncate_episodes | 6.58278e-05 |                 59 |      2 |         24.0254  |   7056 |  -200    |
| DQN_MountainCar-v0_1fcd888b | RUNNING    | 172.16.160.219:47910 | truncate_episodes | 0.000237864 |                751 |     88 |       1638.34    | 199584 |  -122.05 |
| DQN_MountainCar-v0_0023f4f6 | TERMINATED |                      | truncate_episodes | 0.000255833 |                158 |      1 |          5.56779 |   1008 |  -200    |
| DQN_MountainCar-v0_0023f4f9 | TERMINATED |                      | complete_episodes | 0.000262904 |                156 |      1 |          5.43817 |   1200 |  -200    |
| DQN_MountainCar-v0_0023f4fc | TERMINATED |                      | complete_episodes | 0.0002605   |                260 |      1 |          5.33452 |   1200 |  -200    |
| DQN_MountainCar-v0_0108428e | TERMINATED |                      | truncate_episodes | 3.89327e-05 |                732 |      4 |         36.2218  |   5040 |  -200    |
| DQN_MountainCar-v0_01084291 | TERMINATED |                      | truncate_episodes | 2.39745e-05 |                714 |      4 |         36.2585  |   5040 |  -200    |
| DQN_MountainCar-v0_01084294 | TERMINATED |                      | truncate_episodes | 4.9252e-05  |                808 |      4 |         38.4182  |   5040 |  -200    |
| DQN_MountainCar-v0_01084297 | TERMINATED |                      | truncate_episodes | 7.42384e-05 |                804 |      4 |         38.0425  |   5040 |  -200    |
| DQN_MountainCar-v0_014223c0 | TERMINATED |                      | truncate_episodes | 0.0520328   |                 71 |      1 |          6.21906 |   1008 |  -200    |
| DQN_MountainCar-v0_01939ac4 | TERMINATED |                      | complete_episodes | 8.34678e-05 |                124 |      1 |          5.37302 |   1200 |  -200    |
| DQN_MountainCar-v0_01a4cc45 | TERMINATED |                      | complete_episodes | 0.00973094  |                373 |      3 |         27.2147  |  24000 |  -200    |
+-----------------------------+------------+----------------------+-------------------+-------------+--------------------+--------+------------------+--------+----------+
... 980 more trials not shown (156 PAUSED, 823 TERMINATED)


Traceback (most recent call last):
  File "/home/dl-user/python-code/modularized_version_ray/ray_BOHB.py", line 123, in <module>
    verbose=1,
  File "/home/dl-user/.local/lib/python3.7/site-packages/ray/tune/tune.py", line 327, in run
    runner.step()
  File "/home/dl-user/.local/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 342, in step
    self.trial_executor.on_no_available_trials(self)
  File "/home/dl-user/.local/lib/python3.7/site-packages/ray/tune/trial_executor.py", line 177, in on_no_available_trials
    raise TuneError("There are paused trials, but no more pending "
ray.tune.error.TuneError: There are paused trials, but no more pending trials with sufficient resources.

Process finished with exit code 1
```

By the way, why is the first bracket ``none``?
And also it seems that the whole process was pretty slow.
Is there any systematic way to find out the bottlenecks?

You guys are doing a great job with this framework! Thanks in advance for your help!

*Ray version and other system information (Python version, TensorFlow version, OS):*
OS: Ubuntu 16.04
Python Version: 3.7.8
Ray Version: 0.8.6

### Reproduction (REQUIRED)
```python
import ray
from ray import tune
from ray.tune.suggest.bohb import TuneBOHB
from ray.tune.schedulers.hb_bohb import HyperBandForBOHB
import ConfigSpace as CS

ray.init()

config_space = CS.ConfigurationSpace()
config_space.add_hyperparameter(CS.UniformFloatHyperparameter("lr", lower=0.00001, upper=0.1, log=True))
config_space.add_hyperparameter(CS.UniformIntegerHyperparameter("train_batch_size", lower=32, upper=1024, log=False))
config_space.add_hyperparameter(CS.CategoricalHyperparameter("batch_mode", choices=['truncate_episodes', 'complete_episodes']))

bohb_search = TuneBOHB(
    space=config_space,
    bohb_config=None,
    max_concurrent=32,
    metric='episode_reward_mean',
    mode='max'
)

bohb_hyperband = HyperBandForBOHB(
    time_attr='episodes_total',
    metric='episode_reward_mean',
    mode='max',
    max_t=2000,
    reduction_factor=3,
)

config = {
    "env": "MountainCar-v0",
    "min_iter_time_s": 15,
    "num_gpus": 0,
    "num_workers": 1,
    "double_q": True,
    "n_step": 3,
    "target_network_update_freq": 1000,
    "buffer_size": 20000,
    # "prioritzed_replay": True,
    "learning_starts": 1000,
    "log_level": "ERROR"
}

analysis = tune.run(
    run_or_experiment="DQN",
    name='MCv0_DQN_BOHB',
    config=config,
    num_samples=1000,
    checkpoint_at_end=True,
    search_alg=bohb_search,
    scheduler=bohb_hyperband,
    verbose=1,
)
```

Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

If we cannot run your script, we cannot fix your issue.

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:
Hey - I have a very similar issue with BOHB, with very similar code to above.

It seems to work when you use "training_iteration" but not when you use episodes or timesteps, with batch size as a hyperparameter. It seems to break when the number of update steps is not uniform across agents.

Not sure if this helps someone resolve the issue. It would be great if it can be resolved!

Resolving patch:
diff --git a/python/ray/tune/schedulers/hb_bohb.py b/python/ray/tune/schedulers/hb_bohb.py
--- a/python/ray/tune/schedulers/hb_bohb.py
+++ b/python/ray/tune/schedulers/hb_bohb.py
@@ -93,7 +93,7 @@ def _unpause_trial(self, trial_runner, trial):
         trial_runner.trial_executor.unpause_trial(trial)
         trial_runner._search_alg.searcher.on_unpause(trial.trial_id)
 
-    def choose_trial_to_run(self, trial_runner):
+    def choose_trial_to_run(self, trial_runner, allow_recurse=True):
         """Fair scheduling within iteration by completion percentage.
 
         List of trials not used since all trials are tracked as state
@@ -117,8 +117,17 @@ def choose_trial_to_run(self, trial_runner):
                 for bracket in hyperband:
                     if bracket and any(trial.status == Trial.PAUSED
                                        for trial in bracket.current_trials()):
-                        # This will change the trial state and let the
-                        # trial runner retry.
+                        # This will change the trial state
                         self._process_bracket(trial_runner, bracket)
+
+                        # If there are pending trials now, suggest one.
+                        # This is because there might be both PENDING and
+                        # PAUSED trials now, and PAUSED trials will raise
+                        # an error before the trial runner tries again.
+                        if allow_recurse and any(
+                                trial.status == Trial.PENDING
+                                for trial in bracket.current_trials()):
+                            return self.choose_trial_to_run(
+                                trial_runner, allow_recurse=False)
         # MAIN CHANGE HERE!
         return None
diff --git a/python/ray/tune/schedulers/hyperband.py b/python/ray/tune/schedulers/hyperband.py
--- a/python/ray/tune/schedulers/hyperband.py
+++ b/python/ray/tune/schedulers/hyperband.py
@@ -282,7 +282,8 @@ def debug_string(self):
         for i, band in enumerate(self._hyperbands):
             out += "\nRound #{}:".format(i)
             for bracket in band:
-                out += "\n  {}".format(bracket)
+                if bracket:
+                    out += "\n  {}".format(bracket)
         return out
 
     def state(self):


Test patch:
diff --git a/python/ray/tune/tests/test_trial_scheduler.py b/python/ray/tune/tests/test_trial_scheduler.py
--- a/python/ray/tune/tests/test_trial_scheduler.py
+++ b/python/ray/tune/tests/test_trial_scheduler.py
@@ -689,6 +689,30 @@ def result(score, ts):
         self.assertTrue("hyperband_info" in spy_result)
         self.assertEquals(spy_result["hyperband_info"]["budget"], 1)
 
+    def testPauseResumeChooseTrial(self):
+        def result(score, ts):
+            return {"episode_reward_mean": score, TRAINING_ITERATION: ts}
+
+        sched = HyperBandForBOHB(max_t=10, reduction_factor=3, mode="min")
+        runner = _MockTrialRunner(sched)
+        runner._search_alg = MagicMock()
+        runner._search_alg.searcher = MagicMock()
+        trials = [Trial("__fake") for i in range(3)]
+        for t in trials:
+            runner.add_trial(t)
+            runner._launch_trial(t)
+
+        all_results = [result(1, 5), result(2, 1), result(3, 5)]
+        for trial, trial_result in zip(trials, all_results):
+            decision = sched.on_trial_result(runner, trial, trial_result)
+            self.assertEqual(decision, TrialScheduler.PAUSE)
+            runner._pause_trial(trial)
+
+        run_trial = sched.choose_trial_to_run(runner)
+        self.assertEqual(run_trial, trials[1])
+        self.assertSequenceEqual([t.status for t in trials],
+                                 [Trial.PAUSED, Trial.PENDING, Trial.PAUSED])
+
 
 class _MockTrial(Trial):
     def __init__(self, i, config):