Repository: freqtrade/freqtrade

Issue:
"--spaces trailing" - Hyper-Opt Results Are Not Matching Back Test Results.
<!-- 
Have you searched for similar issues before posting it?

If you have discovered a bug in the bot, please [search our issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue). 
If it hasn't been reported, please create a new issue.

Please do not use bug reports to request new features.
-->

## Describe your environment

  * Operating system: UBUNTU 18.04
  * Python Version: 3.6.9 (`python -V`)
  * CCXT version: ccxt==1.33.72 (`pip freeze | grep ccxt`)
  * Freqtrade Version: develop-10c5b230 (`freqtrade -V` or `docker-compose run --rm freqtrade -V` for Freqtrade running in docker)
  
Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

## Describe the problem:

I am having trouble using the trailing stoploss in hyperopt. I cannot seem to match the results it gives me with the backtest results in freqtrade. 

### Steps to reproduce:

  1. I start off my hyperopt by using the "--spaces default" parameter. I pick the best hyperopt result, and run it in freqtrade backtester. It will provide me with matching hyperopt and backtesting results at this point.

  2. Next, I run the trailing hyperopt with "--spaces trailing" parameter. This is when I get hyperopt results that don't match backtest results.

  
### Observed Results:

From what I noticed is that every time I use "--spaces trailing" I get inaccurate backtest results in freqtrade. I am not sure if this is a bug and if I should continue to use trailing stoploss in my strategies anymore.



Discussion:
Hyperopt is just running backtesting under the hood, so if the parameters are transferred correctly, then results will match 100%. There is no technical base for results to not match if all conditions are in the strategy as defined in the hyperopt file / hyperopt output.

One common gotcha:
Make sure to read the backtesting log messages VERY carefully it's very likely that you have something in the config overwriting a setting you set in the strategy - making that setting irrelevant.

This is not done when hyperopting that specific parameter, so the same configuration that sets `"trailing_stop": false` will produce different results for backtesting when hyperopting for the trailing space.

Explicit disabling used to be in the configuration examples [up to recently](https://github.com/freqtrade/freqtrade/commit/dff0ac276803e480c03ba6fb33620b7179307824) - so i suspect this is the reason for your misalignment.
Oh okay, that's good to know. I'll take a look at the configuration file and logging to see what might be causing my problem. It definitely sounds like that could be the issue.
> Hyperopt is just running backtesting under the hood, so if the parameters are transferred correctly, then results will match 100%. There is no technical base for results to not match if all conditions are in the strategy as defined in the hyperopt file / hyperopt output.
> 
> One common gotcha:
> Make sure to read the backtesting log messages VERY carefully it's very likely that you have something in the config overwriting a setting you set in the strategy - making that setting irrelevant.
> 
> This is not done when hyperopting that specific parameter, so the same configuration that sets `"trailing_stop": false` will produce different results for backtesting when hyperopting for the trailing space.
> 
> Explicit disabling used to be in the configuration examples [up to recently](https://github.com/freqtrade/freqtrade/commit/dff0ac276803e480c03ba6fb33620b7179307824) - so i suspect this is the reason for your misalignment.

I took a look at the logging during backtest and I found out the problem. It was because the trailing stoploss wasn't being used in the strategy file for some reason. It's weird because I did have the parameters in the strategy file.

`
freqtrade.resolvers.strategy_resolver - INFO - Strategy using timeframe: 5m
freqtrade.resolvers.strategy_resolver - INFO - Strategy using minimal_roi: {'0': 0.10002, '13': 0.06943, '40': 0.03326, '140': 0}
freqtrade.resolvers.strategy_resolver - INFO - Strategy using stoploss: -0.32554
freqtrade.resolvers.strategy_resolver - INFO - Strategy using trailing_only_offset_is_reached: False
freqtrade.resolvers.strategy_resolver - INFO - Strategy using trailing_stop: False
freqtrade.resolvers.strategy_resolver - INFO - Strategy using trailing_stop_positive_offset: 0.0

`

So what I decided to try was moving the trailing stoploss parameters into the config file instead. The backtester did notice it in config file this time and was able to use the trailing stoploss. Which leads me to another question, I think I might be inputing trailing parameters incorrectly on the strategy file. I was wondering if you can take a look at my format? I'm not sure what I could be typing wrong.
`
class STRATEGY(IStrategy):

    ticker_interval = '5m'

    # ROI table:
    minimal_roi = {
        "0": 0.10002,
        "13": 0.06943,
        "40": 0.03326,
        "140": 0
    }

    # Stoploss:
    stoploss = -0.32554

    # Trailing stop:
    trailing_params = {
        'trailing_only_offset_is_reached': True,
        'trailing_stop': True,
        'trailing_stop_positive': 0.01,
        'trailing_stop_positive_offset': 0.04137
    }
`
Well you're using the wrong syntax for the parameters (that's why i said - if it's done correctly, it'll match).

each setting needs to be a top level setting - so the correct way in a strategy would be:

``` python
    trailing_only_offset_is_reached: True,
    trailing_stop: True,
    trailing_stop_positive: 0.01,
    trailing_stop_positive_offset: 0.04137
```

Oh okay cool, I'll make sure to use that format. I was just copy any pasting the results from hyperopt thinking that was the correct way. Do you think I can contribute by fixing that format on the hyperopt file?
Yeah sure

changing the format of something is usually a good way to start contributing :+1: 
Hey xmatthias, I had a quick question that's not in regards to this post but I just don't want to open up another issue since it's not really an issue. 

So I'm thinking of buying the new Ryzen 5900x CPU coming out soon or Ryzen threadripper 1950x. I also want to buy 64Gb of RAM. Do you really  think it will make hyperopt run better than my current build or I'm i just wasting money? Right now,  I have a Ryzen 2600 12 core CPU and 24Gb RAM. 

I really want to hyperopt at least 365 days of data on 5 minute time frame but my build is having trouble hyperopting 150 days of data. On the system monitor, I see about every 5 minutes all my cpu cores go to 100%, and my 24Gb of RAM is 100% used. I also see my CPU temp is around 55C which is good but I'm not sure why my computer freezes if I try to hyperopt 365 days of data.
because you run out of memory.

I'm not into the "high powered machines" anymore (not since about 15 years i think) - so i can't really recommend.
Now i'm not going to look into the differences between the 2 CPU's - but assuming both have 12 cores, then a slightly faster CPU will not give you much benefits, more cores on the other hand might - but the really limiting factor will be RAM, as each core requires the full datset in memory. You should however be albe to get around that by defining a biggish swap-file (at least avoid freezing).


Awesome, I went ahead and adjusted my swap file size and I see a good difference. Thanks for your advice.
For everyone trying to work on this:
A good place to start working on this is `_params_pretty_print()` in `hyperopt.py`.

Once you got a result in hyperopt, further testing can be done using `freqtrade hyperopt-show -n xx` - this prevents having to run hyperopt 200ds of times.



Resolving patch:
diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@@ -262,6 +262,11 @@ def _params_pretty_print(params, space: str, header: str) -> None:
                     ),
                     default=str, indent=4, number_mode=rapidjson.NM_NATIVE)
                 params_result += f"minimal_roi = {minimal_roi_result}"
+            elif space == 'trailing':
+
+                for k, v in space_params.items():
+                    params_result += f'{k} = {v}\n'
+
             else:
                 params_result += f"{space}_params = {pformat(space_params, indent=4)}"
                 params_result = params_result.replace("}", "\n}").replace("{", "{\n ")


Test patch:
diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py
--- a/tests/optimize/test_hyperopt.py
+++ b/tests/optimize/test_hyperopt.py
@@ -1,6 +1,7 @@
 # pragma pylint: disable=missing-docstring,W0212,C0103
 import locale
 import logging
+import re
 from copy import deepcopy
 from datetime import datetime
 from pathlib import Path
@@ -1230,3 +1231,39 @@ def test_simplified_interface_failed(mocker, hyperopt_conf, method, space) -> No
 
     with pytest.raises(OperationalException, match=f"The '{space}' space is included into *"):
         hyperopt.start()
+
+
+def test_print_epoch_details(capsys):
+    test_result = {
+        'params_details': {
+            'trailing': {
+                'trailing_stop': True,
+                'trailing_stop_positive': 0.02,
+                'trailing_stop_positive_offset': 0.04,
+                'trailing_only_offset_is_reached': True
+            },
+            'roi': {
+                0: 0.18,
+                90: 0.14,
+                225: 0.05,
+                430: 0},
+        },
+        'results_explanation': 'foo result',
+        'is_initial_point': False,
+        'total_profit': 0,
+        'current_epoch': 2,  # This starts from 1 (in a human-friendly manner)
+        'is_best': True
+    }
+
+    Hyperopt.print_epoch_details(test_result, 5, False, no_header=True)
+    captured = capsys.readouterr()
+    assert '# Trailing stop:' in captured.out
+    # re.match(r"Pairs for .*", captured.out)
+    assert re.search(r'^\s+trailing_stop = True$', captured.out, re.MULTILINE)
+    assert re.search(r'^\s+trailing_stop_positive = 0.02$', captured.out, re.MULTILINE)
+    assert re.search(r'^\s+trailing_stop_positive_offset = 0.04$', captured.out, re.MULTILINE)
+    assert re.search(r'^\s+trailing_only_offset_is_reached = True$', captured.out, re.MULTILINE)
+
+    assert '# ROI table:' in captured.out
+    assert re.search(r'^\s+minimal_roi = \{$', captured.out, re.MULTILINE)
+    assert re.search(r'^\s+\"90\"\:\s0.14,\s*$', captured.out, re.MULTILINE)