Repository: ray-project/ray

Issue:
[tune] stopping criterion to propagate to other trials
We use ray.tune to do automated hyperparameter tuning (yay 🙌). To check we have done everything correctly in our projects, we often have an integration test that runs tune and checks if an Experiment can reach a very achievable accuracy in a certain time.

Currently it seems that the stopping criterion that you pass to a ray.tune.Experiment is only applied per trial (that is if one trial reaches it it stops, but the other worse trials keep running). Since we are satisfied if any trial reaches the stopping criterion, my question: 

Is there a way to stop the entire Experiment (and get a result from ray.tune.run)  once any trial reaches the stopping criterion?

Thanks in advance! And thanks for building `ray` 🧡


Discussion:
Ah, I see; I'll get to the programmatic stopping condition PR this weekend. That should allow you to do what you want.


Resolving patch:
diff --git a/python/ray/tune/experiment.py b/python/ray/tune/experiment.py
--- a/python/ray/tune/experiment.py
+++ b/python/ray/tune/experiment.py
@@ -3,6 +3,7 @@
 from __future__ import print_function
 
 import copy
+import inspect
 import logging
 import os
 import six
@@ -87,11 +88,19 @@ def __init__(self,
             _raise_deprecation_note(
                 "sync_function", "sync_to_driver", soft=False)
 
+        stop = stop or {}
+        if not isinstance(stop, dict) and not callable(stop):
+            raise ValueError("Invalid stop criteria: {}. Must be a callable "
+                             "or dict".format(stop))
+        if callable(stop) and len(inspect.getargspec(stop).args) != 2:
+            raise ValueError("Invalid stop criteria: {}. Callable criteria "
+                             "must take exactly 2 parameters.".format(stop))
+
         config = config or {}
         run_identifier = Experiment._register_if_needed(run)
         spec = {
             "run": run_identifier,
-            "stop": stop or {},
+            "stop": stop,
             "config": config,
             "resources_per_trial": resources_per_trial,
             "num_samples": num_samples,
diff --git a/python/ray/tune/trial.py b/python/ray/tune/trial.py
--- a/python/ray/tune/trial.py
+++ b/python/ray/tune/trial.py
@@ -284,6 +284,9 @@ def should_stop(self, result):
         if result.get(DONE):
             return True
 
+        if callable(self.stopping_criterion):
+            return self.stopping_criterion(self.trial_id, result)
+
         for criteria, stop_value in self.stopping_criterion.items():
             if criteria not in result:
                 raise TuneError(
diff --git a/python/ray/tune/tune.py b/python/ray/tune/tune.py
--- a/python/ray/tune/tune.py
+++ b/python/ray/tune/tune.py
@@ -80,9 +80,11 @@ def run(run_or_experiment,
             If Experiment, then Tune will execute training based on
             Experiment.spec.
         name (str): Name of experiment.
-        stop (dict): The stopping criteria. The keys may be any field in
-            the return result of 'train()', whichever is reached first.
-            Defaults to empty dict.
+        stop (dict|func): The stopping criteria. If dict, the keys may be
+            any field in the return result of 'train()', whichever is
+            reached first. If function, it must take (trial_id, result) as
+            arguments and return a boolean (True if trial should be stopped,
+            False otherwise).
         config (dict): Algorithm-specific configuration for Tune variant
             generation (e.g. env, hyperparams). Defaults to empty dict.
             Custom search algorithms may ignore this.


Test patch:
diff --git a/python/ray/tune/tests/test_trial_runner.py b/python/ray/tune/tests/test_trial_runner.py
--- a/python/ray/tune/tests/test_trial_runner.py
+++ b/python/ray/tune/tests/test_trial_runner.py
@@ -453,6 +453,17 @@ def train(config, reporter):
         [trial] = tune.run(train, stop={"test/test1/test2": 6}).trials
         self.assertEqual(trial.last_result["training_iteration"], 7)
 
+    def testStoppingFunction(self):
+        def train(config, reporter):
+            for i in range(10):
+                reporter(test=i)
+
+        def stop(trial_id, result):
+            return result["test"] > 6
+
+        [trial] = tune.run(train, stop=stop).trials
+        self.assertEqual(trial.last_result["training_iteration"], 8)
+
     def testEarlyReturn(self):
         def train(config, reporter):
             reporter(timesteps_total=100, done=True)