Repository: freqtrade/freqtrade

Issue:
handling of startup period
<!-- 
Have you searched for similar issues before posting it?

If you have discovered a bug in the bot, please [search our issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue). 
If it hasn't been reported, please create a new issue.

Please do not use bug reports to request new features.
-->

## Describe your environment

  * Operating system: ____
  * Python Version: Python 3.8.0 (`python -V`)
  * CCXT version: _____ (`pip freeze | grep ccxt`)
  * Freqtrade Version: freqtrade develop-4c349342
  
Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

## Describe the problem:

*Explain the problem you have encountered*

### Steps to reproduce:

- exchange:
 binance
- timerange: 
20190301-20190501
- pairlist: 
BTC/USDT,ETH/USDT,XMR/USDT
- startup_period: 
anything >0 like 20
- run a backtest

### Observed Results:

  * What happened?
XMR/USDT is not trimmed
  * What did you expect to happen?
all pairs to get clipped by startup period
### Relevant code exceptions or logs

Startup period is trimmed by reducing the timerange that includes all the pairs by the required length
the the timerange is made of the union, not the intersection, so trimming by date on data where that period is missing won't trim anything.
I just make sure that the length of the df after the trim by timerange is at least short of the `required_startup` 

Rare case: after that adjustment a df can be left empty, so I was removing those because some operations might error out if applied on an empty df.

During optimization, if you are generating indicators within pop buy/sell you have to account for startup yourself. 

To fix for all the cases maybe do not add the required_startup to the timerange trim here:
https://github.com/freqtrade/freqtrade/blob/4c34934258457ffeb53f57c187e4639091c8966b/freqtrade/optimize/backtesting.py#L146

instead exclude the first rows by index right before executing the 
backtest (so within the backtest function), maybe here
https://github.com/freqtrade/freqtrade/blob/4c34934258457ffeb53f57c187e4639091c8966b/freqtrade/optimize/backtesting.py#L314




Discussion:
It's unclear what did happen from the above, as it's missing crucial information (like logs!).

First of all - Please include the full log of the run. Log messages often include a lot of useful information - and are almost always required to diagnose anything.


For this particular case, it's also important to know, from when to when  (start / end date) do you have data for each of the pairs available.


Resolving patch:
diff --git a/freqtrade/data/converter.py b/freqtrade/data/converter.py
--- a/freqtrade/data/converter.py
+++ b/freqtrade/data/converter.py
@@ -115,17 +115,23 @@ def ohlcv_fill_up_missing_data(dataframe: DataFrame, timeframe: str, pair: str)
     return df
 
 
-def trim_dataframe(df: DataFrame, timerange, df_date_col: str = 'date') -> DataFrame:
+def trim_dataframe(df: DataFrame, timerange, df_date_col: str = 'date',
+                   startup_candles: int = 0) -> DataFrame:
     """
     Trim dataframe based on given timerange
     :param df: Dataframe to trim
     :param timerange: timerange (use start and end date if available)
-    :param: df_date_col: Column in the dataframe to use as Date column
+    :param df_date_col: Column in the dataframe to use as Date column
+    :param startup_candles: When not 0, is used instead the timerange start date
     :return: trimmed dataframe
     """
-    if timerange.starttype == 'date':
-        start = datetime.fromtimestamp(timerange.startts, tz=timezone.utc)
-        df = df.loc[df[df_date_col] >= start, :]
+    if startup_candles:
+        # Trim candles instead of timeframe in case of given startup_candle count
+        df = df.iloc[startup_candles:, :]
+    else:
+        if timerange.starttype == 'date':
+            start = datetime.fromtimestamp(timerange.startts, tz=timezone.utc)
+            df = df.loc[df[df_date_col] >= start, :]
     if timerange.stoptype == 'date':
         stop = datetime.fromtimestamp(timerange.stopts, tz=timezone.utc)
         df = df.loc[df[df_date_col] <= stop, :]
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -443,7 +443,8 @@ def backtest_one_strategy(self, strat: IStrategy, data: Dict[str, Any], timerang
 
         # Trim startup period from analyzed dataframe
         for pair, df in preprocessed.items():
-            preprocessed[pair] = trim_dataframe(df, timerange)
+            preprocessed[pair] = trim_dataframe(df, timerange,
+                                                startup_candles=self.required_startup)
         min_date, max_date = history.get_timerange(preprocessed)
 
         logger.info(f'Backtesting with data from {min_date.strftime(DATETIME_PRINT_FORMAT)} '
diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@@ -379,7 +379,8 @@ def start(self) -> None:
 
         # Trim startup period from analyzed dataframe
         for pair, df in preprocessed.items():
-            preprocessed[pair] = trim_dataframe(df, timerange)
+            preprocessed[pair] = trim_dataframe(df, timerange,
+                                                startup_candles=self.backtesting.required_startup)
         min_date, max_date = get_timerange(preprocessed)
 
         logger.info(f'Hyperopting with data from {min_date.strftime(DATETIME_PRINT_FORMAT)} '


Test patch:
diff --git a/tests/data/test_converter.py b/tests/data/test_converter.py
--- a/tests/data/test_converter.py
+++ b/tests/data/test_converter.py
@@ -197,6 +197,16 @@ def test_trim_dataframe(testdatadir) -> None:
     assert all(data_modify.iloc[-1] == data.iloc[-1])
     assert all(data_modify.iloc[0] == data.iloc[30])
 
+    data_modify = data.copy()
+    tr = TimeRange('date', None, min_date + 1800, 0)
+    # Remove first 20 candles - ignores min date
+    data_modify = trim_dataframe(data_modify, tr, startup_candles=20)
+    assert not data_modify.equals(data)
+    assert len(data_modify) < len(data)
+    assert len(data_modify) == len(data) - 20
+    assert all(data_modify.iloc[-1] == data.iloc[-1])
+    assert all(data_modify.iloc[0] == data.iloc[20])
+
     data_modify = data.copy()
     # Remove last 30 minutes (1800 s)
     tr = TimeRange(None, 'date', 0, max_date - 1800)