Repository: freqtrade/freqtrade

Issue:
RF is much better for hyperopt
I use freqtrade in/at devel.
I do use hyperopt .. and i know the endless discussion about how it finds its minima.

So .. i experimented with the Optimizer from skopt.
Using RF = Random Forest makes to optimization much bettter, and faster.

So .. please change the default, or add an option to choose the estimator for the Optimizer.

```
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@@ -375,7 +375,7 @@ class Hyperopt:
     def get_optimizer(self, dimensions: List[Dimension], cpu_count) -> Optimizer:
         return Optimizer(
             dimensions,
-            base_estimator="ET",
+            base_estimator="RF",
             acq_optimizer="auto",
             n_initial_points=INITIAL_POINTS,
             acq_optimizer_kwargs={'n_jobs': cpu_count},

```





Discussion:
It will depend on your searchspace and parameters.

In my tests, ET took 22m, while RF took 24 minutes for the same optimization (same strategy, same space, same number of epochs).

ET also got the better results than RF - so the results you see are not correct, or (more likely) only apply to your specific case / searchspace, but not in general.

I'm also not the first to come to this conclusion - if you search issues long enough through issues here you'll discover that others did similar tests, and came to the same conclusion.

That's true, in my search space I use a lot categorical parameters, for which ET never reaches any stable point. But a switch for selecting would be nice.Am 28.08.2021 13:55 schrieb Matthias ***@***.***>:
It will depend on your searchspace and parameters.
In my tests, ET took 22m, while RF took 24 minutes for the same optimization.
ET also got the better results than RF - so the results you see are not correct, or only apply to your specific case.

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

[
{
***@***.***": "http://schema.org",
***@***.***": "EmailMessage",
"potentialAction": {
***@***.***": "ViewAction",
"target": "https://github.com/freqtrade/freqtrade/issues/5492#issuecomment-907616067",
"url": "https://github.com/freqtrade/freqtrade/issues/5492#issuecomment-907616067",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
***@***.***": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
Feel free to submit a pull request for that if you think it's necessary (i think it's not so i'll not spend my time on this, but i'll surely not reject a well made PR introducing this).

Please also include documentation on when to include which estimator - so users can make an informed decision.
In my side, I always change it to 'GBRT', because I 'feel' it converging faster, with `-e` set to a lower value
It's certainly more a 'feeling' than a real optimization, but can be an option

Resolving patch:
diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@@ -45,7 +45,7 @@
 logger = logging.getLogger(__name__)
 
 
-INITIAL_POINTS = 30
+INITIAL_POINTS = 5
 
 # Keep no more than SKOPT_MODEL_QUEUE_SIZE models
 # in the skopt model queue, to optimize memory consumption
@@ -365,10 +365,20 @@ def _get_results_dict(self, backtesting_results, min_date, max_date,
         }
 
     def get_optimizer(self, dimensions: List[Dimension], cpu_count) -> Optimizer:
+        estimator = self.custom_hyperopt.generate_estimator()
+
+        acq_optimizer = "sampling"
+        if isinstance(estimator, str):
+            if estimator not in ("GP", "RF", "ET", "GBRT"):
+                raise OperationalException(f"Estimator {estimator} not supported.")
+            else:
+                acq_optimizer = "auto"
+
+        logger.info(f"Using estimator {estimator}.")
         return Optimizer(
             dimensions,
-            base_estimator="ET",
-            acq_optimizer="auto",
+            base_estimator=estimator,
+            acq_optimizer=acq_optimizer,
             n_initial_points=INITIAL_POINTS,
             acq_optimizer_kwargs={'n_jobs': cpu_count},
             random_state=self.random_state,
diff --git a/freqtrade/optimize/hyperopt_auto.py b/freqtrade/optimize/hyperopt_auto.py
--- a/freqtrade/optimize/hyperopt_auto.py
+++ b/freqtrade/optimize/hyperopt_auto.py
@@ -12,7 +12,7 @@
 with suppress(ImportError):
     from skopt.space import Dimension
 
-from freqtrade.optimize.hyperopt_interface import IHyperOpt
+from freqtrade.optimize.hyperopt_interface import EstimatorType, IHyperOpt
 
 
 def _format_exception_message(space: str) -> str:
@@ -79,3 +79,6 @@ def generate_trailing_params(self, params: Dict) -> Dict:
 
     def trailing_space(self) -> List['Dimension']:
         return self._get_func('trailing_space')()
+
+    def generate_estimator(self) -> EstimatorType:
+        return self._get_func('generate_estimator')()
diff --git a/freqtrade/optimize/hyperopt_interface.py b/freqtrade/optimize/hyperopt_interface.py
--- a/freqtrade/optimize/hyperopt_interface.py
+++ b/freqtrade/optimize/hyperopt_interface.py
@@ -5,8 +5,9 @@
 import logging
 import math
 from abc import ABC
-from typing import Dict, List
+from typing import Dict, List, Union
 
+from sklearn.base import RegressorMixin
 from skopt.space import Categorical, Dimension, Integer
 
 from freqtrade.exchange import timeframe_to_minutes
@@ -17,6 +18,8 @@
 
 logger = logging.getLogger(__name__)
 
+EstimatorType = Union[RegressorMixin, str]
+
 
 class IHyperOpt(ABC):
     """
@@ -37,6 +40,14 @@ def __init__(self, config: dict) -> None:
         IHyperOpt.ticker_interval = str(config['timeframe'])  # DEPRECATED
         IHyperOpt.timeframe = str(config['timeframe'])
 
+    def generate_estimator(self) -> EstimatorType:
+        """
+        Return base_estimator.
+        Can be any of "GP", "RF", "ET", "GBRT" or an instance of a class
+        inheriting from RegressorMixin (from sklearn).
+        """
+        return 'ET'
+
     def generate_roi_table(self, params: Dict) -> Dict[int, float]:
         """
         Create a ROI table.


Test patch:
diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py
--- a/tests/optimize/test_hyperopt.py
+++ b/tests/optimize/test_hyperopt.py
@@ -884,6 +884,10 @@ def test_in_strategy_auto_hyperopt(mocker, hyperopt_conf, tmpdir, fee) -> None:
     assert hyperopt.backtesting.strategy.buy_rsi.value != 35
     assert hyperopt.backtesting.strategy.sell_rsi.value != 74
 
+    hyperopt.custom_hyperopt.generate_estimator = lambda *args, **kwargs: 'ET1'
+    with pytest.raises(OperationalException, match="Estimator ET1 not supported."):
+        hyperopt.get_optimizer([], 2)
+
 
 def test_SKDecimal():
     space = SKDecimal(1, 2, decimals=2)