Repository: freqtrade/freqtrade

Issue:
Mode to allow startup candles sent to populate_*_trend()
Would it be possible to have a user selectable mode that sends startup candle data in the dataframe that is passed to populate_trend() methods during hyperoptimising? This would allow indicators to be calculated from within those routines, and that would mean you could use  .values passed by hyperopt in the indicators without nested for loops or creating a large number of columns. A simple conditional inclusion of indicators in populate_indicators() or populate_trend() depending on whether hyperopting or not could allow normal startup candle free dataframes in other run modes. It could make it a lot easier for strategies that, for example, have multiple parameters in indicators like MACD to be hyperotable. I realise this may slow things down somewhat, but simplifies things. Interested in your thoughts. 



Discussion:
Considering that we have the `.range` attributes now (in develop for now) for all parameters (except for `RealParameter`) - i don't see why this should be necessary. 
Having this possibility will only lead to bad practices.

Think about a simple case - you have "a few" parameters to optimize (say 3).
One of these is RSI between 20 and 40 (so a space of 20 parameters).

By calculating the 20 columns in `populate_indicators()` - you calculate each of the RSI's exactly once - no matter the epochs.

Now by moving this to the `populate_buy()`, you'd calculate it once every epoch. 
Assuming you run it for just a few epochs (say 100) - you'd be calculation 100x RSI (instead of 20x). Of these 100, there'll be a LOT of duplicate calculations - maybe because hyperopt finds rsi30 to be a sweet-spot - and does 60 of these epochs with an RSI of 30, modifying the other 2 parameters in the space.
Thanks Matthias. The analysis is good for indicators that have single parameters, but not for things like MACD which has 3. if the space sizes were say 20, 20 and 10 for such an indicator it would need 3 nested for loops and 4000 calculations, the number of calculations scales geometrically with the number of parameters used in the indicators and a very large proportion of those wouldn't get used at all. In that case calculating indicators in the trend methods becomes more efficient as that scales arithmetically with the number of epochs. 

Just a thought, someone has already posted a solution that involves adding the startup candles to the dataframe passed to the trend method, calculating the indicators, stripping the startup candles out again then doing the trend buy and sell calculations before passing the dataframe back to Freqtrade, I was just wondering whether it's worth natively supporting that.

Resolving patch:
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -15,7 +15,7 @@
 from freqtrade.constants import DATETIME_PRINT_FORMAT
 from freqtrade.data import history
 from freqtrade.data.btanalysis import trade_list_to_dataframe
-from freqtrade.data.converter import trim_dataframes
+from freqtrade.data.converter import trim_dataframe, trim_dataframes
 from freqtrade.data.dataprovider import DataProvider
 from freqtrade.enums import BacktestState, SellType
 from freqtrade.exceptions import DependencyException, OperationalException
@@ -116,6 +116,9 @@ def __init__(self, config: Dict[str, Any]) -> None:
 
         self.wallets = Wallets(self.config, self.exchange, log=False)
 
+        self.timerange = TimeRange.parse_timerange(
+            None if self.config.get('timerange') is None else str(self.config.get('timerange')))
+
         # Get maximum required startup period
         self.required_startup = max([strat.startup_candle_count for strat in self.strategylist])
         self.exchange.validate_required_startup_candles(self.required_startup, self.timeframe)
@@ -154,14 +157,11 @@ def load_bt_data(self) -> Tuple[Dict[str, DataFrame], TimeRange]:
         """
         self.progress.init_step(BacktestState.DATALOAD, 1)
 
-        timerange = TimeRange.parse_timerange(None if self.config.get(
-            'timerange') is None else str(self.config.get('timerange')))
-
         data = history.load_data(
             datadir=self.config['datadir'],
             pairs=self.pairlists.whitelist,
             timeframe=self.timeframe,
-            timerange=timerange,
+            timerange=self.timerange,
             startup_candles=self.required_startup,
             fail_without_data=True,
             data_format=self.config.get('dataformat_ohlcv', 'json'),
@@ -174,11 +174,11 @@ def load_bt_data(self) -> Tuple[Dict[str, DataFrame], TimeRange]:
                     f'({(max_date - min_date).days} days).')
 
         # Adjust startts forward if not enough data is available
-        timerange.adjust_start_if_necessary(timeframe_to_seconds(self.timeframe),
-                                            self.required_startup, min_date)
+        self.timerange.adjust_start_if_necessary(timeframe_to_seconds(self.timeframe),
+                                                 self.required_startup, min_date)
 
         self.progress.set_new_value(1)
-        return data, timerange
+        return data, self.timerange
 
     def prepare_backtest(self, enable_protections):
         """
@@ -223,7 +223,9 @@ def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tupl
 
             df_analyzed = self.strategy.advise_sell(
                 self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy()
-
+            # Trim startup period from analyzed dataframe
+            df_analyzed = trim_dataframe(df_analyzed, self.timerange,
+                                         startup_candles=self.required_startup)
             # To avoid using data from future, we use buy/sell signals shifted
             # from the previous candle
             df_analyzed.loc[:, 'buy'] = df_analyzed.loc[:, 'buy'].shift(1)
@@ -537,14 +539,15 @@ def backtest_one_strategy(self, strat: IStrategy, data: Dict[str, Any], timerang
         preprocessed = self.strategy.ohlcvdata_to_dataframe(data)
 
         # Trim startup period from analyzed dataframe
-        preprocessed = trim_dataframes(preprocessed, timerange, self.required_startup)
+        preprocessed_tmp = trim_dataframes(preprocessed, timerange, self.required_startup)
 
-        if not preprocessed:
+        if not preprocessed_tmp:
             raise OperationalException(
                 "No data left after adjusting for startup candles.")
 
-        min_date, max_date = history.get_timerange(preprocessed)
-
+        # Use preprocessed_tmp for date generation (the trimmed dataframe).
+        # Backtesting will re-trim the dataframes after buy/sell signal generation.
+        min_date, max_date = history.get_timerange(preprocessed_tmp)
         logger.info(f'Backtesting with data from {min_date.strftime(DATETIME_PRINT_FORMAT)} '
                     f'up to {max_date.strftime(DATETIME_PRINT_FORMAT)} '
                     f'({(max_date - min_date).days} days).')
diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@@ -378,16 +378,15 @@ def prepare_hyperopt_data(self) -> None:
 
         preprocessed = self.backtesting.strategy.ohlcvdata_to_dataframe(data)
 
-        # Trim startup period from analyzed dataframe
+        # Trim startup period from analyzed dataframe to get correct dates for output.
         processed = trim_dataframes(preprocessed, timerange, self.backtesting.required_startup)
-
         self.min_date, self.max_date = get_timerange(processed)
 
         logger.info(f'Hyperopting with data from {self.min_date.strftime(DATETIME_PRINT_FORMAT)} '
                     f'up to {self.max_date.strftime(DATETIME_PRINT_FORMAT)} '
                     f'({(self.max_date - self.min_date).days} days)..')
-
-        dump(processed, self.data_pickle_file)
+        # Store non-trimmed data - will be trimmed after signal generation.
+        dump(preprocessed, self.data_pickle_file)
 
     def start(self) -> None:
         self.random_state = self._set_random_state(self.config.get('hyperopt_random_state', None))


Test patch:
diff --git a/tests/optimize/test_backtest_detail.py b/tests/optimize/test_backtest_detail.py
--- a/tests/optimize/test_backtest_detail.py
+++ b/tests/optimize/test_backtest_detail.py
@@ -575,6 +575,7 @@ def test_backtest_results(default_conf, fee, mocker, caplog, data) -> None:
     frame = _build_backtest_dataframe(data.data)
     backtesting = Backtesting(default_conf)
     backtesting._set_strategy(backtesting.strategylist[0])
+    backtesting.required_startup = 0
     backtesting.strategy.advise_buy = lambda a, m: frame
     backtesting.strategy.advise_sell = lambda a, m: frame
     backtesting.strategy.use_custom_stoploss = data.use_custom_stoploss
diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py
--- a/tests/optimize/test_backtesting.py
+++ b/tests/optimize/test_backtesting.py
@@ -727,6 +727,7 @@ def test_backtest_alternate_buy_sell(default_conf, fee, mocker, testdatadir):
                                         pair='UNITTEST/BTC', datadir=testdatadir)
     default_conf['timeframe'] = '1m'
     backtesting = Backtesting(default_conf)
+    backtesting.required_startup = 0
     backtesting._set_strategy(backtesting.strategylist[0])
     backtesting.strategy.advise_buy = _trend_alternate  # Override
     backtesting.strategy.advise_sell = _trend_alternate  # Override