Repository: ray-project/ray

Issue:
[tune] Performance collapses with thousands of queued jobs
### System information
- **Ray version**: 0.6.2 to at least March 12th nightly

When there are thousands of trials queued (not even running), the master node performance is very poor and it processes results from the nodes very slowly.  The nodes cpu utilization is very low.

Here is an example where this problem happens:
```
import time

from ray import tune
import ray.tune.util
from ray.tune import trainable


class IdleTrainable(trainable.Trainable):

    def _setup(self, config):
        self.name = config['name']
        self.timeout = config['timeout']

    def _train(self):
        time.sleep(self.timeout)
        string = f"hello from {self.name}"
        print(f"done: {self.name}")
        return {'done': True, 'message': string}

timeout = 3

def build_experiment(i):
    name = f"exp{i}"
    return tune.Experiment(
                run=IdleTrainable,
                name=name,
                config={"name": name,
                        "timeout": timeout},
                #num_samples=1, # We are using repeats instead.
                checkpoint_at_end=False,
                local_dir="/tmp/rayidle",
                # It seems this means a thread and not a full core.
                resources_per_trial={'cpu': 1})

ray.init()
n_jobs = 2500
experiments = [build_experiment(i) for i in range(n_jobs)]

tune.run_experiments(experiments, verbose=False)
```
You can run this locally and see that the jobs (trials) are processed very slowly.  However if you change `n_jobs = 100`, than the _rate_ at which the jobs are processed is much faster.  So the this bug is directly related to the length of the list of jobs.  I explain why below.

And here is the associated stack trace:
```
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   2352/1    0.074    0.000   66.300   66.300 {built-in method builtins.exec}
        1    0.000    0.000   66.299   66.299 junk_ray.py:1(<module>)
        1    0.000    0.000   62.646   62.646 tune.py:55(run_experiments)
       35    0.002    0.000   62.637    1.790 trial_runner.py:218(step)
       35    0.000    0.000   43.330    1.238 trial_runner.py:377(_get_next_trial)
       35    0.055    0.002   40.574    1.159 trial_scheduler.py:86(choose_trial_to_run)
    33375    0.028    0.000   40.519    0.001 trial_runner.py:373(has_resources)
    33375    0.089    0.000   40.491    0.001 ray_trial_executor.py:320(has_resources)
    33411    0.345    0.000   40.011    0.001 ray_trial_executor.py:292(_update_avail_resources)
   167051    0.558    0.000   39.222    0.000 state.py:750(cluster_resources)
   167051    0.232    0.000   38.638    0.000 state.py:390(client_table)
   167051    2.000    0.000   38.356    0.000 state.py:20(parse_client_table)
       34    0.004    0.000   17.156    0.505 trial_runner.py:127(checkpoint)
       68    3.035    0.045   17.107    0.252 __init__.py:120(dump)
```

The issue is the performance of method `FIFOScheduler.choose_trial_to_run(trial_runner)` in tune/scheduler/trial_scheduler.py when there are no resources available.  This method loops over the complete list of trials to see if any one of them is small enough to run given the current available resources.  But checking each trial ends up calling `parse_client_table(redis_client)` in tune/experimental/state.py which is very slow (> 1 ms) because it communicates with the redis server.  

So if the list of trials contains thousands of items, it will take *many seconds* to iterate over the whole list to realize that not a single job can run when there are 0 cpus available.  This is why the master has very poor performance and can't process results from the nodes.

Note that in the main master loop, `FIFOScheduler.choose_trial_to_run(trial_runner)` is basically called in two different situations: right after a job completed and right after a job started.  In the first case, `FIFOScheduler.choose_trial_to_run(trial_runner)` will run very quickly because it will find a job to run given that some resources were just freed by the completed job.  However, when a new job just started, it is likely that there are no resources left (unless it's at the very beginning or the user just added new resources) and it is in those cases that `FIFOScheduler.choose_trial_to_run(trial_runner)` iterates over the full list of jobs and finds no runnable job and occupies the master thread for many seconds.  

So every time a new result comes in and a new job starts, the master thread basically freezes for a few seconds.



Discussion:
Yep, looks like self.has_resources is much more expensive than it needs to be: https://github.com/ray-project/ray/blob/d5f46983056642ad0a2e4dff2a0030845f5c3f03/python/ray/tune/ray_trial_executor.py#L359

A reasonable fix would to be to avoid calling `_update_avail_resources` unless there is actually a state change in the cluster.
But then it's not clear how the trial executor ever becomes aware that the user added resources.  

Currently every single pending trial calls `_update_avail_resources`, but maybe it could only be called once instead.
I see, it makes sense to call it once in a while. Perhaps at the beginning of `choose_trial_to_run`?
Actually, I think it would be OK if we just call it on `TrialExecutor.on_step_begin` as already done and remove it from `has_resources`. This is probably fine as the gap between updating the resources and making the admission control decision (in choose_next_trial_to_run) is very small.

Resolving patch:
diff --git a/python/ray/tune/ray_trial_executor.py b/python/ray/tune/ray_trial_executor.py
--- a/python/ray/tune/ray_trial_executor.py
+++ b/python/ray/tune/ray_trial_executor.py
@@ -18,6 +18,7 @@
 
 logger = logging.getLogger(__name__)
 
+RESOURCE_REFRESH_PERIOD = 0.5  # Refresh resources every 500 ms
 BOTTLENECK_WARN_PERIOD_S = 60
 NONTRIVIAL_WAIT_TIME_THRESHOLD_S = 1e-3
 
@@ -34,18 +35,24 @@ def unwrap(self):
 class RayTrialExecutor(TrialExecutor):
     """An implemention of TrialExecutor based on Ray."""
 
-    def __init__(self, queue_trials=False, reuse_actors=False):
+    def __init__(self,
+                 queue_trials=False,
+                 reuse_actors=False,
+                 refresh_period=RESOURCE_REFRESH_PERIOD):
         super(RayTrialExecutor, self).__init__(queue_trials)
         self._running = {}
         # Since trial resume after paused should not run
         # trial.train.remote(), thus no more new remote object id generated.
         # We use self._paused to store paused trials here.
         self._paused = {}
+        self._reuse_actors = reuse_actors
+        self._cached_actor = None
+
         self._avail_resources = Resources(cpu=0, gpu=0)
         self._committed_resources = Resources(cpu=0, gpu=0)
         self._resources_initialized = False
-        self._reuse_actors = reuse_actors
-        self._cached_actor = None
+        self._refresh_period = refresh_period
+        self._last_resource_refresh = float("-inf")
         self._last_nontrivial_wait = time.time()
         if ray.is_initialized():
             self._update_avail_resources()
@@ -370,11 +377,19 @@ def _update_avail_resources(self, num_retries=5):
 
         self._avail_resources = Resources(
             int(num_cpus), int(num_gpus), custom_resources=custom_resources)
+        self._last_resource_refresh = time.time()
         self._resources_initialized = True
 
     def has_resources(self, resources):
-        """Returns whether this runner has at least the specified resources."""
-        self._update_avail_resources()
+        """Returns whether this runner has at least the specified resources.
+
+        This refreshes the Ray cluster resources if the time since last update
+        has exceeded self._refresh_period. This also assumes that the
+        cluster is not resizing very frequently.
+        """
+        if time.time() - self._last_resource_refresh > self._refresh_period:
+            self._update_avail_resources()
+
         currently_available = Resources.subtract(self._avail_resources,
                                                  self._committed_resources)
 
@@ -445,7 +460,6 @@ def resource_string(self):
 
     def on_step_begin(self):
         """Before step() called, update the available resources."""
-
         self._update_avail_resources()
 
     def save(self, trial, storage=Checkpoint.DISK):


Test patch:
diff --git a/python/ray/tune/tests/test_actor_reuse.py b/python/ray/tune/tests/test_actor_reuse.py
--- a/python/ray/tune/tests/test_actor_reuse.py
+++ b/python/ray/tune/tests/test_actor_reuse.py
@@ -15,27 +15,30 @@ def on_trial_result(self, trial_runner, trial, result):
         return TrialScheduler.PAUSE
 
 
-class MyResettableClass(Trainable):
-    def _setup(self, config):
-        self.config = config
-        self.num_resets = 0
-        self.iter = 0
+def create_resettable_class():
+    class MyResettableClass(Trainable):
+        def _setup(self, config):
+            self.config = config
+            self.num_resets = 0
+            self.iter = 0
 
-    def _train(self):
-        self.iter += 1
-        return {"num_resets": self.num_resets, "done": self.iter > 1}
+        def _train(self):
+            self.iter += 1
+            return {"num_resets": self.num_resets, "done": self.iter > 1}
 
-    def _save(self, chkpt_dir):
-        return {"iter": self.iter}
+        def _save(self, chkpt_dir):
+            return {"iter": self.iter}
 
-    def _restore(self, item):
-        self.iter = item["iter"]
+        def _restore(self, item):
+            self.iter = item["iter"]
 
-    def reset_config(self, new_config):
-        if "fake_reset_not_supported" in self.config:
-            return False
-        self.num_resets += 1
-        return True
+        def reset_config(self, new_config):
+            if "fake_reset_not_supported" in self.config:
+                return False
+            self.num_resets += 1
+            return True
+
+    return MyResettableClass
 
 
 class ActorReuseTest(unittest.TestCase):
@@ -49,7 +52,7 @@ def testTrialReuseDisabled(self):
         trials = run_experiments(
             {
                 "foo": {
-                    "run": MyResettableClass,
+                    "run": create_resettable_class(),
                     "num_samples": 4,
                     "config": {},
                 }
@@ -63,7 +66,7 @@ def testTrialReuseEnabled(self):
         trials = run_experiments(
             {
                 "foo": {
-                    "run": MyResettableClass,
+                    "run": create_resettable_class(),
                     "num_samples": 4,
                     "config": {},
                 }
@@ -78,7 +81,7 @@ def run():
             run_experiments(
                 {
                     "foo": {
-                        "run": MyResettableClass,
+                        "run": create_resettable_class(),
                         "max_failures": 1,
                         "num_samples": 4,
                         "config": {