Repository: freqtrade/freqtrade
Pull request #11736: add early stopping for hyperopt
Author: login: viotemp1
Labels: name: Enhancement
description: Enhancements to the bot. Get lower priority than bugs by default.
name: Hyperopt
description: Hyperopt related issues and pull requests
Created: 2025-05-10T16:40:07Z | Merged: 2025-05-20T04:45:15Z

<!-- Thank you for sending your pull request. But first, have you included
unit tests, and is your code PEP8 conformant? [More details](https://github.com/freqtrade/freqtrade/blob/develop/CONTRIBUTING.md)
-->
## Summary

<!-- Explain in one sentence the goal of this PR -->
Add early stopping for hyperopt

## Quick changelog

- Add early stopping for hyperopt

## What's new?

<!-- Explain in details what this PR solve or improve. You can include visuals. -->
Will shorten the hyperopt time when no improvement in Objective


Diff:
diff --git a/docs/commands/hyperopt.md b/docs/commands/hyperopt.md
index e7a05bedf92..f1c8b28b5d0 100644
--- a/docs/commands/hyperopt.md
+++ b/docs/commands/hyperopt.md
@@ -16,6 +16,7 @@ usage: freqtrade hyperopt [-h] [-v] [--no-color] [--logfile FILE] [-V]
                           [--random-state INT] [--min-trades INT]
                           [--hyperopt-loss NAME] [--disable-param-export]
                           [--ignore-missing-spaces] [--analyze-per-epoch]
+                          [--early-stop INT]
 
 options:
   -h, --help            show this help message and exit
@@ -87,6 +88,8 @@ options:
                         Suppress errors for any requested Hyperopt spaces that
                         do not contain any parameters.
   --analyze-per-epoch   Run populate_indicators once per epoch.
+  --early-stop INT      Early stop hyperopt if no improvement after (default:
+                        0) epochs.
 
 Common arguments:
   -v, --verbose         Verbose mode (-vv for more, -vvv to get all messages).
diff --git a/docs/hyperopt.md b/docs/hyperopt.md
index 3a8100972e3..af5ea9d2cb3 100644
--- a/docs/hyperopt.md
+++ b/docs/hyperopt.md
@@ -490,6 +490,8 @@ freqtrade hyperopt --config config.json --hyperopt-loss <hyperoptlossname> --str
 ```
 
 The `-e` option will set how many evaluations hyperopt will do. Since hyperopt uses Bayesian search, running too many epochs at once may not produce greater results. Experience has shown that best results are usually not improving much after 500-1000 epochs.  
+The `--early-stop` option will set after how many epochs with no improvements hyperopt will stop. A good value is 20-30% of the total epochs. Any value greater than 0 and lower than 20 it will be replaced by 20. Early stop is by default disabled (`--early-stop=0`)
+
 Doing multiple runs (executions) with a few 1000 epochs and different random state will most likely produce different results.
 
 The `--spaces all` option determines that all possible parameters should be optimized. Possibilities are listed below.
diff --git a/freqtrade/commands/arguments.py b/freqtrade/commands/arguments.py
index c42e46711d4..52beb3dfaa9 100755
--- a/freqtrade/commands/arguments.py
+++ b/freqtrade/commands/arguments.py
@@ -78,6 +78,7 @@
     "disableparamexport",
     "hyperopt_ignore_missing_space",
     "analyze_per_epoch",
+    "early_stop",
 ]
 
 ARGS_EDGE = [*ARGS_COMMON_OPTIMIZE, "stoploss_range"]
diff --git a/freqtrade/commands/cli_options.py b/freqtrade/commands/cli_options.py
index f05907808c2..c5d0ad49b59 100755
--- a/freqtrade/commands/cli_options.py
+++ b/freqtrade/commands/cli_options.py
@@ -262,6 +262,13 @@ def __init__(self, *args, **kwargs):
         metavar="INT",
         default=constants.HYPEROPT_EPOCH,
     ),
+    "early_stop": Arg(
+        "--early-stop",
+        help="Early stop hyperopt if no improvement after (default: %(default)d) epochs.",
+        type=check_int_positive,
+        metavar="INT",
+        default=0,  # 0 to disable by default
+    ),
     "spaces": Arg(
         "--spaces",
         help="Specify which parameters to hyperopt. Space-separated list.",
diff --git a/freqtrade/configuration/configuration.py b/freqtrade/configuration/configuration.py
index 11aff879f41..264d114b8f8 100644
--- a/freqtrade/configuration/configuration.py
+++ b/freqtrade/configuration/configuration.py
@@ -334,6 +334,19 @@ def _process_optimize_options(self, config: Config) -> None:
             ("print_all", "Parameter --print-all detected ..."),
         ]
         self._args_to_config_loop(config, configurations)
+        es_epochs = self.args.get("early_stop", 0)
+        if es_epochs > 0:
+            if es_epochs < 20:
+                logger.warning(
+                    f"Early stop epochs {es_epochs} lower than 20. It will be replaced with 20."
+                )
+                config.update({"early_stop": 20})
+            else:
+                config.update({"early_stop": self.args["early_stop"]})
+            logger.info(
+                f"Parameter --early-stop detected ... Will early stop hyperopt if no improvement "
+                f"after {config.get('early_stop')} epochs ..."
+            )
 
         configurations = [
             ("print_json", "Parameter --print-json detected ..."),
diff --git a/freqtrade/optimize/hyperopt/hyperopt.py b/freqtrade/optimize/hyperopt/hyperopt.py
index 3db5ec9fe04..926aaaf8c6c 100644
--- a/freqtrade/optimize/hyperopt/hyperopt.py
+++ b/freqtrade/optimize/hyperopt/hyperopt.py
@@ -317,6 +317,13 @@ def start(self) -> None:
                         logging_mp_handle(log_queue)
                         gc.collect()
 
+                        if (
+                            self.hyperopter.es_epochs > 0
+                            and self.hyperopter.es_terminator.should_terminate(self.opt)
+                        ):
+                            logger.info(f"Early stopping after {(i + 1) * jobs} epochs")
+                            break
+
         except KeyboardInterrupt:
             print("User interrupted..")
 
diff --git a/freqtrade/optimize/hyperopt/hyperopt_optimizer.py b/freqtrade/optimize/hyperopt/hyperopt_optimizer.py
index 99e81e4b3b2..d0346f23a83 100644
--- a/freqtrade/optimize/hyperopt/hyperopt_optimizer.py
+++ b/freqtrade/optimize/hyperopt/hyperopt_optimizer.py
@@ -14,6 +14,7 @@
 from joblib import delayed, dump, load, wrap_non_picklable_objects
 from joblib.externals import cloudpickle
 from optuna.exceptions import ExperimentalWarning
+from optuna.terminator import BestValueStagnationEvaluator, Terminator
 from pandas import DataFrame
 
 from freqtrade.constants import DATETIME_PRINT_FORMAT, Config
@@ -104,6 +105,10 @@ def __init__(self, config: Config, data_pickle_file: Path) -> None:
 
         self.market_change = 0.0
 
+        self.es_epochs = config.get("early_stop", 0)
+        if self.es_epochs > 0 and self.es_epochs < 0.2 * config.get("epochs", 0):
+            logger.warning(f"Early stop epochs {self.es_epochs} lower than 20% of total epochs")
+
         if HyperoptTools.has_space(self.config, "sell"):
             # Make sure use_exit_signal is enabled
             self.config["use_exit_signal"] = True
@@ -424,6 +429,11 @@ def get_optimizer(
         else:
             sampler = o_sampler
 
+        if self.es_epochs > 0:
+            with warnings.catch_warnings():
+                warnings.filterwarnings(action="ignore", category=ExperimentalWarning)
+                self.es_terminator = Terminator(BestValueStagnationEvaluator(self.es_epochs))
+
         logger.info(f"Using optuna sampler {o_sampler}.")
         return optuna.create_study(sampler=sampler, direction="minimize")
 
diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py
index c22323bdf29..4d9d68eae3f 100644
--- a/tests/optimize/test_hyperopt.py
+++ b/tests/optimize/test_hyperopt.py
@@ -170,6 +170,48 @@ def test_setup_hyperopt_configuration_stake_amount(mocker, default_conf) -> None
         setup_optimize_configuration(get_args(args), RunMode.HYPEROPT)
 
 
+def test_setup_hyperopt_early_stop_setup(mocker, default_conf, caplog) -> None:
+    patched_configuration_load_config_file(mocker, default_conf)
+
+    args = [
+        "hyperopt",
+        "--config",
+        "config.json",
+        "--strategy",
+        "HyperoptableStrategy",
+        "--early-stop",
+        "1",
+    ]
+    conf = setup_optimize_configuration(get_args(args), RunMode.HYPEROPT)
+    assert isinstance(conf, dict)
+    assert conf["early_stop"] == 20
+    msg = (
+        r"Parameter --early-stop detected ... "
+        r"Will early stop hyperopt if no improvement after (20|25) epochs ..."
+    )
+    msg_adjust = r"Early stop epochs .* lower than 20. It will be replaced with 20."
+    assert log_has_re(msg_adjust, caplog)
+    assert log_has_re(msg, caplog)
+
+    caplog.clear()
+
+    args = [
+        "hyperopt",
+        "--config",
+        "config.json",
+        "--strategy",
+        CURRENT_TEST_STRATEGY,
+        "--early-stop",
+        "25",
+    ]
+    conf1 = setup_optimize_configuration(get_args(args), RunMode.HYPEROPT)
+    assert isinstance(conf1, dict)
+
+    assert conf1["early_stop"] == 25
+    assert not log_has_re(msg_adjust, caplog)
+    assert log_has_re(msg, caplog)
+
+
 def test_start_not_installed(mocker, default_conf, import_fails) -> None:
     start_mock = MagicMock()
     patched_configuration_load_config_file(mocker, default_conf)


Reviews:
user: login: xmatthias
body: I'm sorry, i've somehow completely missed this PR.

For the CI, you'll need to run `build_helpers/create_command_partials.py` and commit the result - it'll auto-update all doc descriptions to keep (at least this part of) the docs in sync with reality.

i assume in terms of testing, this is best tested with a very small search space, so it exhausts pretty quickly?
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T05:04:31Z
user: login: viotemp1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T22:11:26Z
user: login: viotemp1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T22:17:20Z
user: login: xmatthias
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-17T07:40:41Z
user: login: viotemp1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-17T08:43:00Z
user: login: xmatthias
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-17T09:47:05Z
user: login: xmatthias
body: Great enhancement :+1: 
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-20T04:45:13Z

Review comments:
diff_hunk: @@ -262,6 +262,14 @@ def __init__(self, *args, **kwargs):
         metavar="INT",
         default=constants.HYPEROPT_EPOCH,
     ),
+    "early_stop": Arg(
+        "-es",
path: freqtrade/commands/cli_options.py
user: login: xmatthias
body: I'd prefer not to have 2 letter "shortcuts" (they're no longer short - and can cause pretty much ambiguity, too).
do we really need a shortcut here, or is `--early-stop` more expressive? 
created_at: 2025-05-16T05:02:06Z
author_association: MEMBER
diff_hunk: @@ -262,6 +262,14 @@ def __init__(self, *args, **kwargs):
         metavar="INT",
         default=constants.HYPEROPT_EPOCH,
     ),
+    "early_stop": Arg(
+        "-es",
path: freqtrade/commands/cli_options.py
user: login: viotemp1
body: ok
created_at: 2025-05-16T22:11:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -262,6 +262,14 @@ def __init__(self, *args, **kwargs):
         metavar="INT",
         default=constants.HYPEROPT_EPOCH,
     ),
+    "early_stop": Arg(
+        "-es",
path: freqtrade/commands/cli_options.py
user: login: viotemp1
body: fixed docs for CI. 
the test can be small so it can finish fast. something like a hyperopt with 10 epochs with early stop of 3.
The idea is that it will stop hyperopt after x epochs with no improvement in loss, not when space is exhausted.
created_at: 2025-05-16T22:17:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -262,6 +262,14 @@ def __init__(self, *args, **kwargs):
         metavar="INT",
         default=constants.HYPEROPT_EPOCH,
     ),
+    "early_stop": Arg(
+        "-es",
path: freqtrade/commands/cli_options.py
user: login: xmatthias
body: Doesn't seem to work, then ... 
i did a test (limited to sell space) - and got seemingly the same result for 20 epochs.
`--early-stop 3` didn't stop it - but it continued through through all 20 epochs

I'd expect this to stop at latest at epoch 9 or 10 (i used -j 3 - so parallelism will have it "finish" that batch).

it did load it correctly based on the logs 

```
2025-05-17 09:35:48,781 - freqtrade.configuration.configuration - INFO - Parameter --early-stop detected ... Will early stop hyperopt if no improvement after 3 epochs ...  
```

but didn't stop.
![image](https://github.com/user-attachments/assets/d6b072f1-8d89-44e4-8f6d-4e260c992eea)

created_at: 2025-05-17T07:40:41Z
author_association: MEMBER
diff_hunk: @@ -262,6 +262,14 @@ def __init__(self, *args, **kwargs):
         metavar="INT",
         default=constants.HYPEROPT_EPOCH,
     ),
+    "early_stop": Arg(
+        "-es",
path: freqtrade/commands/cli_options.py
user: login: viotemp1
body: Setting a higher number of epochs (100 let's say) will work.
<img width="800" alt="Screenshot 2025-05-17 at 11 38 00" src="https://github.com/user-attachments/assets/06ee67a3-ee74-4d0b-8333-99de6263c48b" />
Do you have the latest version? I fixed something yesterday
created_at: 2025-05-17T08:43:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -262,6 +262,14 @@ def __init__(self, *args, **kwargs):
         metavar="INT",
         default=constants.HYPEROPT_EPOCH,
     ),
+    "early_stop": Arg(
+        "-es",
path: freqtrade/commands/cli_options.py
user: login: xmatthias
body: Yes i do, and yes, with more epochs it works :+1: 
created_at: 2025-05-17T09:47:05Z
author_association: MEMBER

PR comments:
user: login: coveralls
created_at: 2025-05-16T22:28:06Z
body: 
[![Coverage Status](https://coveralls.io/builds/73727321/badge)](https://coveralls.io/builds/73727321)

coverage: 94.349% (-0.01%) from 94.362%
when pulling **5a2b3d9d87e4c1b6bcfb3ede405cfc9117cc5bfc on viotemp1:optuna_addons**
into **c3f6aa17ee8b6ba2e0e17407ea376d05e5fd7d86 on freqtrade:develop**.

author_association: NONE
user: login: xmatthias
created_at: 2025-05-17T10:02:49Z
body: @viotemp1 i don't currently trust this functionality ...

with `--epochs 100, --early-stop 5` - it stops after 21 epochs (same with --early-stop=2) - but the objective function is clearly still changing - significantly at that (used with `--print-all` to see all results)

![image](https://github.com/user-attachments/assets/8244c332-6db7-4bb4-8309-d4a7e6c1a0aa)

author_association: MEMBER
user: login: viotemp1
created_at: 2025-05-17T10:15:23Z
body: I looked at their source code and it should work.

The default of min_n_trials is 20 in optuna.terminator.Terminator(improvement_evaluator=None, error_evaluator=None, min_n_trials=20). 
We can add a warning message when lower than 20 or change the default.

author_association: CONTRIBUTOR
user: login: xmatthias
created_at: 2025-05-17T11:40:31Z
body: well i think even enforcing a minimum of 20 (or ignoring values between 0 and 20) is reasonable.
if your result doesn't improve after 5 epochs - that won't tell you much :laughing: 
author_association: MEMBER