Repository: ray-project/ray

Issue:
[tune] MedianStoppingRule dependent only on complete trials
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.
Questions about how to use Ray should be asked on
[StackOverflow](https://stackoverflow.com/questions/tagged/ray).

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: RHEL 7.6
- **Ray installed from (source or binary)**: source
- **Ray version**: 0.8
- **Python version**: 3.6.8
- **Exact command to reproduce**:

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
I'm looking to implement some of the different schedulers.. and the MedianStoppingRule by the docs seems like it should implement an early stopping rule... but when running it, the median_result seems to be dependent on values from a list of trials which only get added "on_trial_complete"... which doesn't really make sense to me, as shouldn't the median be updating all along with iteration results?  


### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
Forget it... i I understand what it's doing... I actually think the version I'm thinking of might be useful as well, more like PBT updating in parallel and dropping trials that are below a median value... in this case I guess it's more of a sequential process of running trials, rather than in parallel.
I think this is actually supposed to be addressed in https://github.com/ray-project/ray/pull/4119 but we never merged it because we haven't gotten around to fixing the tests.

You're welcome to try that one out though!
That's excellent Richard, thank you... I created a new class and incorporated those changes.  I also added to it the PBT style pausing/selecting of trials to allow all trials to stay in step, and the combination of those two gives me exactly what I'm looking for... I also added an extra argument to control the evaluation interval, similar to "perturbation_interval" as well as adding an optional parameter to take from the end of the list of results, thus the median calculated can be from a list of means constructed of tail_length... rather than the entire training history to reflect more recent performance.

A question I have is, when running PBT or MedianStopping using APEX DQN... when a trial is PAUSED... what happens to the replay buffer?  I'm trying to understand what the limitations might be regarding running multiple trials in a distributed fashion, which I have a decent idea when just outright training, but not sure if one is to PAUSE & CONTINUE along the way with this type of scheduling.

here is what my current version looks like... it seems to be doing what I expected...

class MedianTrialState(object):

    def __init__(self, trial):
        self.orig_tag = trial.experiment_tag
        self.last_eval_time = 0

    def __repr__(self):
        return str((self.last_eval_time,))

class MedianStoppingResult(FIFOScheduler):

    def __init__(self,
                 time_attr="time_total_s",
                 reward_attr=None,
                 metric="episode_reward_mean",
                 mode="max",
                 grace_period=60.0,
                 eval_interval=600.0,
                 min_samples_required=3,
                 hard_stop=True,
                 verbose=True,
                 tail_length= None):
        assert mode in ["min", "max"], "`mode` must be 'min' or 'max'!"

        if reward_attr is not None:
            mode = "max"
            metric = reward_attr
            logger.warning(
                "`reward_attr` is deprecated and will be removed in a future "
                "version of Tune. "
                "Setting `metric={}` and `mode=max`.".format(reward_attr))

        FIFOScheduler.__init__(self)
        self._stopped_trials = set()
        self._completed_trials = set()
        self._grace_period = grace_period
        self._eval_interval = eval_interval
        self._min_samples_required = min_samples_required
        self._metric = metric
        if mode == "max":
            self._metric_op = 1.
        elif mode == "min":
            self._metric_op = -1.
        self._time_attr = time_attr
        self._hard_stop = hard_stop
        self._verbose = verbose
        self._tail_length = tail_length

        self._trial_state = {}
        self._results = collections.defaultdict(list)

    @property
    def _trials_beyond_grace_period(self):
        trials = [
            trial for trial in self._results if (trial.last_result.get(
                self._time_attr, -float('inf')) > self._grace_period)
        ]
        return trials

    def on_trial_add(self, trial_runner, trial):
        self._trial_state[trial] = MedianTrialState(trial)

    def on_trial_result(self, trial_runner, trial, result):
        """Callback for early stopping.

        This stopping rule stops a running trial if the trial's best objective
        value by step `t` is strictly worse than the median of the running
        averages of all completed trials' objectives reported up to step `t`.
        """

        if trial in self._stopped_trials:
            assert not self._hard_stop
            return TrialScheduler.CONTINUE  # fall back to FIFO

        state = self._trial_state[trial]
        time = result[self._time_attr]
        self._results[trial].append(result)
        
        if time - state.last_eval_time < self._eval_interval:
            return TrialScheduler.CONTINUE  # avoid overhead

        state.last_eval_time = time
        median_result = self._get_median_result(time)
        best_result = self._best_result(trial)
        if self._verbose:
            logger.info("Trial {} best res={} vs median res={} at t={}".format(
                trial, best_result, median_result, time))
        
        if best_result < median_result and time > self._grace_period:
            if self._verbose:
                logger.info("MedianStoppingResult: "
                            "early stopping {}".format(trial))
            self._stopped_trials.add(trial)
            if self._hard_stop:
                return TrialScheduler.STOP
            else:
                return TrialScheduler.PAUSE
        else:
            for _trial in trial_runner.get_trials():
                if _trial.status in [Trial.PENDING, Trial.PAUSED]:
                    return TrialScheduler.PAUSE  # yield time to other trials

            return TrialScheduler.CONTINUE



    def on_trial_complete(self, trial_runner, trial, result):
        self._results[trial].append(result)
        self._completed_trials.add(trial)

    def on_trial_remove(self, trial_runner, trial):
        """Marks trial as completed if it is paused and has previously ran."""
        if trial.status is Trial.PAUSED and trial in self._results:
            self._completed_trials.add(trial)

    def debug_string(self):
        return "Using MedianStoppingResult: num_stopped={}.".format(
            len(self._stopped_trials))

    def _get_median_result(self, time):
        scores = []
        for trial in self._trials_beyond_grace_period:
            scores.append(self._running_result(trial, time))
        if len(scores) >= self._min_samples_required:
            return np.median(scores)
        else:
            return float("-inf")

    def _running_result(self, trial, t_max=float("inf")):
        results = self._results[trial]
        if self._tail_length is not None:
            results = results[-self._tail_length:]
        return self._metric_op * np.mean(
            [flatten_dict(r)[self._metric] for r in results \
                if flatten_dict(r)[self._time_attr] <= t_max])


    def _best_result(self, trial):
        results = self._results[trial]
        if self._tail_length is not None:
            results = results[-self._tail_length:]        
        return max([self._metric_op * flatten_dict(r)[self._metric] for r in results])

    def choose_trial_to_run(self, trial_runner):

        candidates = []
        for trial in trial_runner.get_trials():
            if trial.status in [Trial.PENDING, Trial.PAUSED] and \
                    trial_runner.has_resources(trial.resources):
                candidates.append(trial)
        candidates.sort(
            key=lambda trial: self._trial_state[trial].last_eval_time)
        return candidates[0] if candidates else None
I think the entire replay buffer gets serialized.

On Thu, Aug 1, 2019 at 11:06 AM waldroje <notifications@github.com> wrote:

> That's excellent Richard, thank you... I created a new class and
> incorporated those changes. I also added to it the PBT style
> pausing/selecting of trials to allow all trials to stay in step, and the
> combination of those two gives me exactly what I'm looking for... I also
> added an extra argument to control the evaluation interval, similar to
> "perturbation_interval" as well as adding an optional parameter to take
> from the end of the list of results, thus the median calculated can be from
> a list of means constructed of tail_length... rather than the entire
> training history to reflect more recent performance.
>
> A question I have is, when running PBT or MedianStopping using APEX DQN...
> when a trial is PAUSED... what happens to the replay buffer? I'm trying to
> understand what the limitations might be regarding running multiple trials
> in a distributed fashion, which I have a decent idea when just outright
> training, but not sure if one is to PAUSE & CONTINUE along the way with
> this type of scheduling.
>
> here is what my current version looks like... it seems to be doing what I
> expected...
>
> class MedianTrialState(object):
>
> def __init__(self, trial):
>     self.orig_tag = trial.experiment_tag
>     self.last_eval_time = 0
>
> def __repr__(self):
>     return str((self.last_eval_time,))
>
> class MedianStoppingResult(FIFOScheduler):
>
> def __init__(self,
>              time_attr="time_total_s",
>              reward_attr=None,
>              metric="episode_reward_mean",
>              mode="max",
>              grace_period=60.0,
>              eval_interval=600.0,
>              min_samples_required=3,
>              hard_stop=True,
>              verbose=True,
>              tail_length= None):
>     assert mode in ["min", "max"], "`mode` must be 'min' or 'max'!"
>
>     if reward_attr is not None:
>         mode = "max"
>         metric = reward_attr
>         logger.warning(
>             "`reward_attr` is deprecated and will be removed in a future "
>             "version of Tune. "
>             "Setting `metric={}` and `mode=max`.".format(reward_attr))
>
>     FIFOScheduler.__init__(self)
>     self._stopped_trials = set()
>     self._completed_trials = set()
>     self._grace_period = grace_period
>     self._eval_interval = eval_interval
>     self._min_samples_required = min_samples_required
>     self._metric = metric
>     if mode == "max":
>         self._metric_op = 1.
>     elif mode == "min":
>         self._metric_op = -1.
>     self._time_attr = time_attr
>     self._hard_stop = hard_stop
>     self._verbose = verbose
>     self._tail_length = tail_length
>
>     self._trial_state = {}
>     self._results = collections.defaultdict(list)
>
> @property
> def _trials_beyond_grace_period(self):
>     trials = [
>         trial for trial in self._results if (trial.last_result.get(
>             self._time_attr, -float('inf')) > self._grace_period)
>     ]
>     return trials
>
> def on_trial_add(self, trial_runner, trial):
>     self._trial_state[trial] = MedianTrialState(trial)
>
> def on_trial_result(self, trial_runner, trial, result):
>     """Callback for early stopping.
>
>     This stopping rule stops a running trial if the trial's best objective
>     value by step `t` is strictly worse than the median of the running
>     averages of all completed trials' objectives reported up to step `t`.
>     """
>
>     if trial in self._stopped_trials:
>         assert not self._hard_stop
>         return TrialScheduler.CONTINUE  # fall back to FIFO
>
>     state = self._trial_state[trial]
>     time = result[self._time_attr]
>     self._results[trial].append(result)
>
>     if time - state.last_eval_time < self._eval_interval:
>         return TrialScheduler.CONTINUE  # avoid overhead
>
>     state.last_eval_time = time
>     median_result = self._get_median_result(time)
>     best_result = self._best_result(trial)
>     if self._verbose:
>         logger.info("Trial {} best res={} vs median res={} at t={}".format(
>             trial, best_result, median_result, time))
>
>     if best_result < median_result and time > self._grace_period:
>         if self._verbose:
>             logger.info("MedianStoppingResult: "
>                         "early stopping {}".format(trial))
>         self._stopped_trials.add(trial)
>         if self._hard_stop:
>             return TrialScheduler.STOP
>         else:
>             return TrialScheduler.PAUSE
>     else:
>         for _trial in trial_runner.get_trials():
>             if _trial.status in [Trial.PENDING, Trial.PAUSED]:
>                 return TrialScheduler.PAUSE  # yield time to other trials
>
>         return TrialScheduler.CONTINUE
>
>
>
> def on_trial_complete(self, trial_runner, trial, result):
>     self._results[trial].append(result)
>     self._completed_trials.add(trial)
>
> def on_trial_remove(self, trial_runner, trial):
>     """Marks trial as completed if it is paused and has previously ran."""
>     if trial.status is Trial.PAUSED and trial in self._results:
>         self._completed_trials.add(trial)
>
> def debug_string(self):
>     return "Using MedianStoppingResult: num_stopped={}.".format(
>         len(self._stopped_trials))
>
> def _get_median_result(self, time):
>     scores = []
>     for trial in self._trials_beyond_grace_period:
>         scores.append(self._running_result(trial, time))
>     if len(scores) >= self._min_samples_required:
>         return np.median(scores)
>     else:
>         return float("-inf")
>
> def _running_result(self, trial, t_max=float("inf")):
>     results = self._results[trial]
>     if self._tail_length is not None:
>         results = results[-self._tail_length:]
>     return self._metric_op * np.mean(
>         [flatten_dict(r)[self._metric] for r in results \
>             if flatten_dict(r)[self._time_attr] <= t_max])
>
>
> def _best_result(self, trial):
>     results = self._results[trial]
>     if self._tail_length is not None:
>         results = results[-self._tail_length:]
>     return max([self._metric_op * flatten_dict(r)[self._metric] for r in results])
>
> def choose_trial_to_run(self, trial_runner):
>
>     candidates = []
>     for trial in trial_runner.get_trials():
>         if trial.status in [Trial.PENDING, Trial.PAUSED] and \
>                 trial_runner.has_resources(trial.resources):
>             candidates.append(trial)
>     candidates.sort(
>         key=lambda trial: self._trial_state[trial].last_eval_time)
>     return candidates[0] if candidates else None
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/ray-project/ray/issues/5335?email_source=notifications&email_token=ABCRZZLH3VWP4RHEWQAOZQDQCMQ2HA5CNFSM4IILBDB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3LNVUQ#issuecomment-517397202>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABCRZZL4AUAEHEN6ILTQQO3QCMQ2HANCNFSM4IILBDBQ>
> .
>

BTW, this change looks really neat - would you be interested in creating a
PR with these changes?

On Thu, Aug 1, 2019 at 12:42 PM Richard Liaw <rich.liaw@gmail.com> wrote:

> I think the entire replay buffer gets serialized.
>
> On Thu, Aug 1, 2019 at 11:06 AM waldroje <notifications@github.com> wrote:
>
>> That's excellent Richard, thank you... I created a new class and
>> incorporated those changes. I also added to it the PBT style
>> pausing/selecting of trials to allow all trials to stay in step, and the
>> combination of those two gives me exactly what I'm looking for... I also
>> added an extra argument to control the evaluation interval, similar to
>> "perturbation_interval" as well as adding an optional parameter to take
>> from the end of the list of results, thus the median calculated can be from
>> a list of means constructed of tail_length... rather than the entire
>> training history to reflect more recent performance.
>>
>> A question I have is, when running PBT or MedianStopping using APEX
>> DQN... when a trial is PAUSED... what happens to the replay buffer? I'm
>> trying to understand what the limitations might be regarding running
>> multiple trials in a distributed fashion, which I have a decent idea when
>> just outright training, but not sure if one is to PAUSE & CONTINUE along
>> the way with this type of scheduling.
>>
>> here is what my current version looks like... it seems to be doing what I
>> expected...
>>
>> class MedianTrialState(object):
>>
>> def __init__(self, trial):
>>     self.orig_tag = trial.experiment_tag
>>     self.last_eval_time = 0
>>
>> def __repr__(self):
>>     return str((self.last_eval_time,))
>>
>> class MedianStoppingResult(FIFOScheduler):
>>
>> def __init__(self,
>>              time_attr="time_total_s",
>>              reward_attr=None,
>>              metric="episode_reward_mean",
>>              mode="max",
>>              grace_period=60.0,
>>              eval_interval=600.0,
>>              min_samples_required=3,
>>              hard_stop=True,
>>              verbose=True,
>>              tail_length= None):
>>     assert mode in ["min", "max"], "`mode` must be 'min' or 'max'!"
>>
>>     if reward_attr is not None:
>>         mode = "max"
>>         metric = reward_attr
>>         logger.warning(
>>             "`reward_attr` is deprecated and will be removed in a future "
>>             "version of Tune. "
>>             "Setting `metric={}` and `mode=max`.".format(reward_attr))
>>
>>     FIFOScheduler.__init__(self)
>>     self._stopped_trials = set()
>>     self._completed_trials = set()
>>     self._grace_period = grace_period
>>     self._eval_interval = eval_interval
>>     self._min_samples_required = min_samples_required
>>     self._metric = metric
>>     if mode == "max":
>>         self._metric_op = 1.
>>     elif mode == "min":
>>         self._metric_op = -1.
>>     self._time_attr = time_attr
>>     self._hard_stop = hard_stop
>>     self._verbose = verbose
>>     self._tail_length = tail_length
>>
>>     self._trial_state = {}
>>     self._results = collections.defaultdict(list)
>>
>> @property
>> def _trials_beyond_grace_period(self):
>>     trials = [
>>         trial for trial in self._results if (trial.last_result.get(
>>             self._time_attr, -float('inf')) > self._grace_period)
>>     ]
>>     return trials
>>
>> def on_trial_add(self, trial_runner, trial):
>>     self._trial_state[trial] = MedianTrialState(trial)
>>
>> def on_trial_result(self, trial_runner, trial, result):
>>     """Callback for early stopping.
>>
>>     This stopping rule stops a running trial if the trial's best objective
>>     value by step `t` is strictly worse than the median of the running
>>     averages of all completed trials' objectives reported up to step `t`.
>>     """
>>
>>     if trial in self._stopped_trials:
>>         assert not self._hard_stop
>>         return TrialScheduler.CONTINUE  # fall back to FIFO
>>
>>     state = self._trial_state[trial]
>>     time = result[self._time_attr]
>>     self._results[trial].append(result)
>>
>>     if time - state.last_eval_time < self._eval_interval:
>>         return TrialScheduler.CONTINUE  # avoid overhead
>>
>>     state.last_eval_time = time
>>     median_result = self._get_median_result(time)
>>     best_result = self._best_result(trial)
>>     if self._verbose:
>>         logger.info("Trial {} best res={} vs median res={} at t={}".format(
>>             trial, best_result, median_result, time))
>>
>>     if best_result < median_result and time > self._grace_period:
>>         if self._verbose:
>>             logger.info("MedianStoppingResult: "
>>                         "early stopping {}".format(trial))
>>         self._stopped_trials.add(trial)
>>         if self._hard_stop:
>>             return TrialScheduler.STOP
>>         else:
>>             return TrialScheduler.PAUSE
>>     else:
>>         for _trial in trial_runner.get_trials():
>>             if _trial.status in [Trial.PENDING, Trial.PAUSED]:
>>                 return TrialScheduler.PAUSE  # yield time to other trials
>>
>>         return TrialScheduler.CONTINUE
>>
>>
>>
>> def on_trial_complete(self, trial_runner, trial, result):
>>     self._results[trial].append(result)
>>     self._completed_trials.add(trial)
>>
>> def on_trial_remove(self, trial_runner, trial):
>>     """Marks trial as completed if it is paused and has previously ran."""
>>     if trial.status is Trial.PAUSED and trial in self._results:
>>         self._completed_trials.add(trial)
>>
>> def debug_string(self):
>>     return "Using MedianStoppingResult: num_stopped={}.".format(
>>         len(self._stopped_trials))
>>
>> def _get_median_result(self, time):
>>     scores = []
>>     for trial in self._trials_beyond_grace_period:
>>         scores.append(self._running_result(trial, time))
>>     if len(scores) >= self._min_samples_required:
>>         return np.median(scores)
>>     else:
>>         return float("-inf")
>>
>> def _running_result(self, trial, t_max=float("inf")):
>>     results = self._results[trial]
>>     if self._tail_length is not None:
>>         results = results[-self._tail_length:]
>>     return self._metric_op * np.mean(
>>         [flatten_dict(r)[self._metric] for r in results \
>>             if flatten_dict(r)[self._time_attr] <= t_max])
>>
>>
>> def _best_result(self, trial):
>>     results = self._results[trial]
>>     if self._tail_length is not None:
>>         results = results[-self._tail_length:]
>>     return max([self._metric_op * flatten_dict(r)[self._metric] for r in results])
>>
>> def choose_trial_to_run(self, trial_runner):
>>
>>     candidates = []
>>     for trial in trial_runner.get_trials():
>>         if trial.status in [Trial.PENDING, Trial.PAUSED] and \
>>                 trial_runner.has_resources(trial.resources):
>>             candidates.append(trial)
>>     candidates.sort(
>>         key=lambda trial: self._trial_state[trial].last_eval_time)
>>     return candidates[0] if candidates else None
>>
>> —
>> You are receiving this because you commented.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/ray-project/ray/issues/5335?email_source=notifications&email_token=ABCRZZLH3VWP4RHEWQAOZQDQCMQ2HA5CNFSM4IILBDB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3LNVUQ#issuecomment-517397202>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/ABCRZZL4AUAEHEN6ILTQQO3QCMQ2HANCNFSM4IILBDBQ>
>> .
>>
>

I’m happy to give it a shot... I should have some time later today, early tomorrow...

Sent from my iPhone

> On Aug 1, 2019, at 3:43 PM, Richard Liaw <notifications@github.com> wrote:
> 
> BTW, this change looks really neat - would you be interested in creating a
> PR with these changes?
> 
> On Thu, Aug 1, 2019 at 12:42 PM Richard Liaw <rich.liaw@gmail.com> wrote:
> 
> > I think the entire replay buffer gets serialized.
> >
> > On Thu, Aug 1, 2019 at 11:06 AM waldroje <notifications@github.com> wrote:
> >
> >> That's excellent Richard, thank you... I created a new class and
> >> incorporated those changes. I also added to it the PBT style
> >> pausing/selecting of trials to allow all trials to stay in step, and the
> >> combination of those two gives me exactly what I'm looking for... I also
> >> added an extra argument to control the evaluation interval, similar to
> >> "perturbation_interval" as well as adding an optional parameter to take
> >> from the end of the list of results, thus the median calculated can be from
> >> a list of means constructed of tail_length... rather than the entire
> >> training history to reflect more recent performance.
> >>
> >> A question I have is, when running PBT or MedianStopping using APEX
> >> DQN... when a trial is PAUSED... what happens to the replay buffer? I'm
> >> trying to understand what the limitations might be regarding running
> >> multiple trials in a distributed fashion, which I have a decent idea when
> >> just outright training, but not sure if one is to PAUSE & CONTINUE along
> >> the way with this type of scheduling.
> >>
> >> here is what my current version looks like... it seems to be doing what I
> >> expected...
> >>
> >> class MedianTrialState(object):
> >>
> >> def __init__(self, trial):
> >> self.orig_tag = trial.experiment_tag
> >> self.last_eval_time = 0
> >>
> >> def __repr__(self):
> >> return str((self.last_eval_time,))
> >>
> >> class MedianStoppingResult(FIFOScheduler):
> >>
> >> def __init__(self,
> >> time_attr="time_total_s",
> >> reward_attr=None,
> >> metric="episode_reward_mean",
> >> mode="max",
> >> grace_period=60.0,
> >> eval_interval=600.0,
> >> min_samples_required=3,
> >> hard_stop=True,
> >> verbose=True,
> >> tail_length= None):
> >> assert mode in ["min", "max"], "`mode` must be 'min' or 'max'!"
> >>
> >> if reward_attr is not None:
> >> mode = "max"
> >> metric = reward_attr
> >> logger.warning(
> >> "`reward_attr` is deprecated and will be removed in a future "
> >> "version of Tune. "
> >> "Setting `metric={}` and `mode=max`.".format(reward_attr))
> >>
> >> FIFOScheduler.__init__(self)
> >> self._stopped_trials = set()
> >> self._completed_trials = set()
> >> self._grace_period = grace_period
> >> self._eval_interval = eval_interval
> >> self._min_samples_required = min_samples_required
> >> self._metric = metric
> >> if mode == "max":
> >> self._metric_op = 1.
> >> elif mode == "min":
> >> self._metric_op = -1.
> >> self._time_attr = time_attr
> >> self._hard_stop = hard_stop
> >> self._verbose = verbose
> >> self._tail_length = tail_length
> >>
> >> self._trial_state = {}
> >> self._results = collections.defaultdict(list)
> >>
> >> @property
> >> def _trials_beyond_grace_period(self):
> >> trials = [
> >> trial for trial in self._results if (trial.last_result.get(
> >> self._time_attr, -float('inf')) > self._grace_period)
> >> ]
> >> return trials
> >>
> >> def on_trial_add(self, trial_runner, trial):
> >> self._trial_state[trial] = MedianTrialState(trial)
> >>
> >> def on_trial_result(self, trial_runner, trial, result):
> >> """Callback for early stopping.
> >>
> >> This stopping rule stops a running trial if the trial's best objective
> >> value by step `t` is strictly worse than the median of the running
> >> averages of all completed trials' objectives reported up to step `t`.
> >> """
> >>
> >> if trial in self._stopped_trials:
> >> assert not self._hard_stop
> >> return TrialScheduler.CONTINUE # fall back to FIFO
> >>
> >> state = self._trial_state[trial]
> >> time = result[self._time_attr]
> >> self._results[trial].append(result)
> >>
> >> if time - state.last_eval_time < self._eval_interval:
> >> return TrialScheduler.CONTINUE # avoid overhead
> >>
> >> state.last_eval_time = time
> >> median_result = self._get_median_result(time)
> >> best_result = self._best_result(trial)
> >> if self._verbose:
> >> logger.info("Trial {} best res={} vs median res={} at t={}".format(
> >> trial, best_result, median_result, time))
> >>
> >> if best_result < median_result and time > self._grace_period:
> >> if self._verbose:
> >> logger.info("MedianStoppingResult: "
> >> "early stopping {}".format(trial))
> >> self._stopped_trials.add(trial)
> >> if self._hard_stop:
> >> return TrialScheduler.STOP
> >> else:
> >> return TrialScheduler.PAUSE
> >> else:
> >> for _trial in trial_runner.get_trials():
> >> if _trial.status in [Trial.PENDING, Trial.PAUSED]:
> >> return TrialScheduler.PAUSE # yield time to other trials
> >>
> >> return TrialScheduler.CONTINUE
> >>
> >>
> >>
> >> def on_trial_complete(self, trial_runner, trial, result):
> >> self._results[trial].append(result)
> >> self._completed_trials.add(trial)
> >>
> >> def on_trial_remove(self, trial_runner, trial):
> >> """Marks trial as completed if it is paused and has previously ran."""
> >> if trial.status is Trial.PAUSED and trial in self._results:
> >> self._completed_trials.add(trial)
> >>
> >> def debug_string(self):
> >> return "Using MedianStoppingResult: num_stopped={}.".format(
> >> len(self._stopped_trials))
> >>
> >> def _get_median_result(self, time):
> >> scores = []
> >> for trial in self._trials_beyond_grace_period:
> >> scores.append(self._running_result(trial, time))
> >> if len(scores) >= self._min_samples_required:
> >> return np.median(scores)
> >> else:
> >> return float("-inf")
> >>
> >> def _running_result(self, trial, t_max=float("inf")):
> >> results = self._results[trial]
> >> if self._tail_length is not None:
> >> results = results[-self._tail_length:]
> >> return self._metric_op * np.mean(
> >> [flatten_dict(r)[self._metric] for r in results \
> >> if flatten_dict(r)[self._time_attr] <= t_max])
> >>
> >>
> >> def _best_result(self, trial):
> >> results = self._results[trial]
> >> if self._tail_length is not None:
> >> results = results[-self._tail_length:]
> >> return max([self._metric_op * flatten_dict(r)[self._metric] for r in results])
> >>
> >> def choose_trial_to_run(self, trial_runner):
> >>
> >> candidates = []
> >> for trial in trial_runner.get_trials():
> >> if trial.status in [Trial.PENDING, Trial.PAUSED] and \
> >> trial_runner.has_resources(trial.resources):
> >> candidates.append(trial)
> >> candidates.sort(
> >> key=lambda trial: self._trial_state[trial].last_eval_time)
> >> return candidates[0] if candidates else None
> >>
> >> —
> >> You are receiving this because you commented.
> >> Reply to this email directly, view it on GitHub
> >> <https://github.com/ray-project/ray/issues/5335?email_source=notifications&email_token=ABCRZZLH3VWP4RHEWQAOZQDQCMQ2HA5CNFSM4IILBDB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3LNVUQ#issuecomment-517397202>,
> >> or mute the thread
> >> <https://github.com/notifications/unsubscribe-auth/ABCRZZL4AUAEHEN6ILTQQO3QCMQ2HANCNFSM4IILBDBQ>
> >> .
> >>
> >
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or mute the thread.

@waldroje did you end up getting around to this?
Not yet, been tied up, haven’t ever done a PR, so was looking into linting and tests... I have a few days that I’m free this week, was hoping to get to it.

Sent from my iPhone

> On Aug 5, 2019, at 11:48 PM, Richard Liaw <notifications@github.com> wrote:
> 
> @waldroje did you end up getting around to this?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.


Resolving patch:
diff --git a/python/ray/tune/schedulers/median_stopping_rule.py b/python/ray/tune/schedulers/median_stopping_rule.py
--- a/python/ray/tune/schedulers/median_stopping_rule.py
+++ b/python/ray/tune/schedulers/median_stopping_rule.py
@@ -27,13 +27,18 @@ class MedianStoppingRule(FIFOScheduler):
         mode (str): One of {min, max}. Determines whether objective is
             minimizing or maximizing the metric attribute.
         grace_period (float): Only stop trials at least this old in time.
+            The mean will only be computed from this time onwards. The units
+            are the same as the attribute named by `time_attr`.
+        min_samples_required (int): Minimum number of trials to compute median
+            over.
+        min_time_slice (float): Each trial runs at least this long before
+            yielding (assuming it isn't stopped). Note: trials ONLY yield if
+            there are not enough samples to evaluate performance for the
+            current result AND there are other trials waiting to run.
             The units are the same as the attribute named by `time_attr`.
-        min_samples_required (int): Min samples to compute median over.
         hard_stop (bool): If False, pauses trials instead of stopping
             them. When all other trials are complete, paused trials will be
             resumed and allowed to run FIFO.
-        verbose (bool): If True, will output the median and best result each
-            time a trial reports. Defaults to True.
     """
 
     def __init__(self,
@@ -43,10 +48,9 @@ def __init__(self,
                  mode="max",
                  grace_period=60.0,
                  min_samples_required=3,
-                 hard_stop=True,
-                 verbose=True):
+                 min_time_slice=0,
+                 hard_stop=True):
         assert mode in ["min", "max"], "`mode` must be 'min' or 'max'!"
-
         if reward_attr is not None:
             mode = "max"
             metric = reward_attr
@@ -54,21 +58,20 @@ def __init__(self,
                 "`reward_attr` is deprecated and will be removed in a future "
                 "version of Tune. "
                 "Setting `metric={}` and `mode=max`.".format(reward_attr))
-
         FIFOScheduler.__init__(self)
         self._stopped_trials = set()
-        self._completed_trials = set()
-        self._results = collections.defaultdict(list)
         self._grace_period = grace_period
         self._min_samples_required = min_samples_required
+        self._min_time_slice = min_time_slice
         self._metric = metric
-        if mode == "max":
-            self._metric_op = 1.
-        elif mode == "min":
-            self._metric_op = -1.
+        assert mode in {"min", "max"}, "`mode` must be 'min' or 'max'."
+        self._worst = float("-inf") if mode == "max" else float("inf")
+        self._compare_op = max if mode == "max" else min
         self._time_attr = time_attr
         self._hard_stop = hard_stop
-        self._verbose = verbose
+        self._trial_state = {}
+        self._last_pause = collections.defaultdict(lambda: float("-inf"))
+        self._results = collections.defaultdict(list)
 
     def on_trial_result(self, trial_runner, trial, result):
         """Callback for early stopping.
@@ -82,19 +85,38 @@ def on_trial_result(self, trial_runner, trial, result):
 
         if trial in self._stopped_trials:
             assert not self._hard_stop
-            return TrialScheduler.CONTINUE  # fall back to FIFO
+            # Fall back to FIFO
+            return TrialScheduler.CONTINUE
 
         time = result[self._time_attr]
         self._results[trial].append(result)
-        median_result = self._get_median_result(time)
+
+        if time < self._grace_period:
+            return TrialScheduler.CONTINUE
+
+        trials = self._trials_beyond_time(time)
+        trials.remove(trial)
+
+        if len(trials) < self._min_samples_required:
+            action = self._on_insufficient_samples(trial_runner, trial, time)
+            if action == TrialScheduler.PAUSE:
+                self._last_pause[trial] = time
+                action_str = "Yielding time to other trials."
+            else:
+                action_str = "Continuing anyways."
+            logger.debug(
+                "MedianStoppingRule: insufficient samples={} to evaluate "
+                "trial {} at t={}. {}".format(
+                    len(trials), trial.trial_id, time, action_str))
+            return action
+
+        median_result = self._median_result(trials, time)
         best_result = self._best_result(trial)
-        if self._verbose:
-            logger.info("Trial {} best res={} vs median res={} at t={}".format(
-                trial, best_result, median_result, time))
-        if best_result < median_result and time > self._grace_period:
-            if self._verbose:
-                logger.info("MedianStoppingRule: "
-                            "early stopping {}".format(trial))
+        logger.debug("Trial {} best res={} vs median res={} at t={}".format(
+            trial, best_result, median_result, time))
+
+        if self._compare_op(median_result, best_result) != best_result:
+            logger.debug("MedianStoppingRule: early stopping {}".format(trial))
             self._stopped_trials.add(trial)
             if self._hard_stop:
                 return TrialScheduler.STOP
@@ -105,33 +127,39 @@ def on_trial_result(self, trial_runner, trial, result):
 
     def on_trial_complete(self, trial_runner, trial, result):
         self._results[trial].append(result)
-        self._completed_trials.add(trial)
-
-    def on_trial_remove(self, trial_runner, trial):
-        """Marks trial as completed if it is paused and has previously ran."""
-        if trial.status is Trial.PAUSED and trial in self._results:
-            self._completed_trials.add(trial)
 
     def debug_string(self):
         return "Using MedianStoppingRule: num_stopped={}.".format(
             len(self._stopped_trials))
 
-    def _get_median_result(self, time):
-        scores = []
-        for trial in self._completed_trials:
-            scores.append(self._running_result(trial, time))
-        if len(scores) >= self._min_samples_required:
-            return np.median(scores)
-        else:
-            return float("-inf")
-
-    def _running_result(self, trial, t_max=float("inf")):
+    def _on_insufficient_samples(self, trial_runner, trial, time):
+        pause = time - self._last_pause[trial] > self._min_time_slice
+        pause = pause and [
+            t for t in trial_runner.get_trials()
+            if t.status in (Trial.PENDING, Trial.PAUSED)
+        ]
+        return TrialScheduler.PAUSE if pause else TrialScheduler.CONTINUE
+
+    def _trials_beyond_time(self, time):
+        trials = [
+            trial for trial in self._results
+            if self._results[trial][-1][self._time_attr] >= time
+        ]
+        return trials
+
+    def _median_result(self, trials, time):
+        return np.median([self._running_mean(trial, time) for trial in trials])
+
+    def _running_mean(self, trial, time):
         results = self._results[trial]
         # TODO(ekl) we could do interpolation to be more precise, but for now
         # assume len(results) is large and the time diffs are roughly equal
-        return self._metric_op * np.mean(
-            [r[self._metric] for r in results if r[self._time_attr] <= t_max])
+        scoped_results = [
+            r for r in results
+            if self._grace_period <= r[self._time_attr] <= time
+        ]
+        return np.mean([r[self._metric] for r in scoped_results])
 
     def _best_result(self, trial):
         results = self._results[trial]
-        return max(self._metric_op * r[self._metric] for r in results)
+        return self._compare_op([r[self._metric] for r in results])


Test patch:
diff --git a/python/ray/tune/tests/test_trial_scheduler.py b/python/ray/tune/tests/test_trial_scheduler.py
--- a/python/ray/tune/tests/test_trial_scheduler.py
+++ b/python/ray/tune/tests/test_trial_scheduler.py
@@ -36,6 +36,12 @@ def result(t, rew):
         time_total_s=t, episode_reward_mean=rew, training_iteration=int(t))
 
 
+def mock_trial_runner(trials=None):
+    trial_runner = MagicMock()
+    trial_runner.get_trials.return_value = trials or []
+    return trial_runner
+
+
 class EarlyStoppingSuite(unittest.TestCase):
     def setUp(self):
         ray.init()
@@ -47,93 +53,105 @@ def tearDown(self):
     def basicSetup(self, rule):
         t1 = Trial("PPO")  # mean is 450, max 900, t_max=10
         t2 = Trial("PPO")  # mean is 450, max 450, t_max=5
+        runner = mock_trial_runner()
         for i in range(10):
+            r1 = result(i, i * 100)
+            print("basicSetup:", i)
             self.assertEqual(
-                rule.on_trial_result(None, t1, result(i, i * 100)),
-                TrialScheduler.CONTINUE)
+                rule.on_trial_result(runner, t1, r1), TrialScheduler.CONTINUE)
         for i in range(5):
+            r2 = result(i, 450)
             self.assertEqual(
-                rule.on_trial_result(None, t2, result(i, 450)),
-                TrialScheduler.CONTINUE)
+                rule.on_trial_result(runner, t2, r2), TrialScheduler.CONTINUE)
         return t1, t2
 
     def testMedianStoppingConstantPerf(self):
         rule = MedianStoppingRule(grace_period=0, min_samples_required=1)
         t1, t2 = self.basicSetup(rule)
-        rule.on_trial_complete(None, t1, result(10, 1000))
+        runner = mock_trial_runner()
+        rule.on_trial_complete(runner, t1, result(10, 1000))
         self.assertEqual(
-            rule.on_trial_result(None, t2, result(5, 450)),
+            rule.on_trial_result(runner, t2, result(5, 450)),
             TrialScheduler.CONTINUE)
         self.assertEqual(
-            rule.on_trial_result(None, t2, result(6, 0)),
+            rule.on_trial_result(runner, t2, result(6, 0)),
             TrialScheduler.CONTINUE)
         self.assertEqual(
-            rule.on_trial_result(None, t2, result(10, 450)),
+            rule.on_trial_result(runner, t2, result(10, 450)),
             TrialScheduler.STOP)
 
     def testMedianStoppingOnCompleteOnly(self):
         rule = MedianStoppingRule(grace_period=0, min_samples_required=1)
         t1, t2 = self.basicSetup(rule)
+        runner = mock_trial_runner()
         self.assertEqual(
-            rule.on_trial_result(None, t2, result(100, 0)),
+            rule.on_trial_result(runner, t2, result(100, 0)),
             TrialScheduler.CONTINUE)
-        rule.on_trial_complete(None, t1, result(10, 1000))
+        rule.on_trial_complete(runner, t1, result(101, 1000))
         self.assertEqual(
-            rule.on_trial_result(None, t2, result(101, 0)),
+            rule.on_trial_result(runner, t2, result(101, 0)),
             TrialScheduler.STOP)
 
     def testMedianStoppingGracePeriod(self):
         rule = MedianStoppingRule(grace_period=2.5, min_samples_required=1)
         t1, t2 = self.basicSetup(rule)
-        rule.on_trial_complete(None, t1, result(10, 1000))
-        rule.on_trial_complete(None, t2, result(10, 1000))
+        runner = mock_trial_runner()
+        rule.on_trial_complete(runner, t1, result(10, 1000))
+        rule.on_trial_complete(runner, t2, result(10, 1000))
         t3 = Trial("PPO")
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(1, 10)),
+            rule.on_trial_result(runner, t3, result(1, 10)),
             TrialScheduler.CONTINUE)
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(2, 10)),
+            rule.on_trial_result(runner, t3, result(2, 10)),
             TrialScheduler.CONTINUE)
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(3, 10)), TrialScheduler.STOP)
+            rule.on_trial_result(runner, t3, result(3, 10)),
+            TrialScheduler.STOP)
 
     def testMedianStoppingMinSamples(self):
         rule = MedianStoppingRule(grace_period=0, min_samples_required=2)
         t1, t2 = self.basicSetup(rule)
-        rule.on_trial_complete(None, t1, result(10, 1000))
+        runner = mock_trial_runner()
+        rule.on_trial_complete(runner, t1, result(10, 1000))
         t3 = Trial("PPO")
+        # Insufficient samples to evaluate t3
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(3, 10)),
+            rule.on_trial_result(runner, t3, result(5, 10)),
             TrialScheduler.CONTINUE)
-        rule.on_trial_complete(None, t2, result(10, 1000))
+        rule.on_trial_complete(runner, t2, result(5, 1000))
+        # Sufficient samples to evaluate t3
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(3, 10)), TrialScheduler.STOP)
+            rule.on_trial_result(runner, t3, result(5, 10)),
+            TrialScheduler.STOP)
 
     def testMedianStoppingUsesMedian(self):
         rule = MedianStoppingRule(grace_period=0, min_samples_required=1)
         t1, t2 = self.basicSetup(rule)
-        rule.on_trial_complete(None, t1, result(10, 1000))
-        rule.on_trial_complete(None, t2, result(10, 1000))
+        runner = mock_trial_runner()
+        rule.on_trial_complete(runner, t1, result(10, 1000))
+        rule.on_trial_complete(runner, t2, result(10, 1000))
         t3 = Trial("PPO")
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(1, 260)),
+            rule.on_trial_result(runner, t3, result(1, 260)),
             TrialScheduler.CONTINUE)
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(2, 260)),
+            rule.on_trial_result(runner, t3, result(2, 260)),
             TrialScheduler.STOP)
 
     def testMedianStoppingSoftStop(self):
         rule = MedianStoppingRule(
             grace_period=0, min_samples_required=1, hard_stop=False)
         t1, t2 = self.basicSetup(rule)
-        rule.on_trial_complete(None, t1, result(10, 1000))
-        rule.on_trial_complete(None, t2, result(10, 1000))
+        runner = mock_trial_runner()
+        rule.on_trial_complete(runner, t1, result(10, 1000))
+        rule.on_trial_complete(runner, t2, result(10, 1000))
         t3 = Trial("PPO")
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(1, 260)),
+            rule.on_trial_result(runner, t3, result(1, 260)),
             TrialScheduler.CONTINUE)
         self.assertEqual(
-            rule.on_trial_result(None, t3, result(2, 260)),
+            rule.on_trial_result(runner, t3, result(2, 260)),
             TrialScheduler.PAUSE)
 
     def _test_metrics(self, result_func, metric, mode):
@@ -145,20 +163,21 @@ def _test_metrics(self, result_func, metric, mode):
             mode=mode)
         t1 = Trial("PPO")  # mean is 450, max 900, t_max=10
         t2 = Trial("PPO")  # mean is 450, max 450, t_max=5
+        runner = mock_trial_runner()
         for i in range(10):
             self.assertEqual(
-                rule.on_trial_result(None, t1, result_func(i, i * 100)),
+                rule.on_trial_result(runner, t1, result_func(i, i * 100)),
                 TrialScheduler.CONTINUE)
         for i in range(5):
             self.assertEqual(
-                rule.on_trial_result(None, t2, result_func(i, 450)),
+                rule.on_trial_result(runner, t2, result_func(i, 450)),
                 TrialScheduler.CONTINUE)
-        rule.on_trial_complete(None, t1, result_func(10, 1000))
+        rule.on_trial_complete(runner, t1, result_func(10, 1000))
         self.assertEqual(
-            rule.on_trial_result(None, t2, result_func(5, 450)),
+            rule.on_trial_result(runner, t2, result_func(5, 450)),
             TrialScheduler.CONTINUE)
         self.assertEqual(
-            rule.on_trial_result(None, t2, result_func(6, 0)),
+            rule.on_trial_result(runner, t2, result_func(6, 0)),
             TrialScheduler.CONTINUE)
 
     def testAlternateMetrics(self):