Repository: freqtrade/freqtrade

Issue:
get_analyzed_dataframe does not provide history prior to the defined `--timerange`
## Describe your environment

  * Operating system: Ubuntu 22.04.1 LTS
  * Python Version: 3.10.4
  * CCXT version: 1.93.3
  * Freqtrade Version: 939fb7acb338202dfa2bf945ee51d95e37e14e97

## Describe the problem:

The first time you enter `custom_stoploss` in backtesting: you may get only a few rows of DataFrame from `get_analyzed_dataframe` and then it keeps increasing as you loop back to `custom_stoploss`. But you may need more rows  from the first time you enter `custom_stoploss` (in backtesting). It is not because of buy signals at the first few candles because the indicators tested need way more than that.

I think I might know how the quirk appears. It looks back before `--timerange` in order to populate indicators (possibly because of a long `startup_candle_count`) but it may then enter a trade the very first day of `--timerange` at the start of it (it does give a longer DataFrame on other trades later so it's only if you get "lucky" at the start of the first day of `--timerange` that may cause it). 

It appears the code should respect `startup_candle_count` and return a longer DataFrame with `get_analyzed_dataframe` in early buy signals. A user workaround might be to produce needed indicators before entering `custom_stoploss` (or similar functions that need `get_analyzed_dataframe` I guess), but that's counter-intuitive and it defies part of `get_analyzed_dataframe`'s purpose.

### Steps to reproduce:

  1. Have a strategy that will produce a trade signal at the very start of `--timerange`
  2. It needs indicators that may be populated from data before `--timerange`
  3. `get_analyzed_dataframe` returns very few rows  at the start
  



Discussion:
to be honest, I see this as intended behavior.

Let's assume a timerange of `--timerange 20220201-20220401` (end date is irrelevant ...), a timeframe of 1h - and you have 24 startup candles (so we have 1 day of additional data). The assumption also is that more data exists at the start.

Freqtrade will load data from 20220131-20220401 (an additional 24 candles) - and run analysis on this (calculating indicators - which may take up to 24 candles to actually produce values (e.g. an EMA24).

After the analysis - freqtrade will remove this additional 24 canldles from the start - leaving only the desired timerange (20220201-20220401).
Now it goes on and checks for entry/exit signals. Assuming your strategy produces a buy signal on the very first candle (2022-02-01 00:00) - so we enter the trade at 2022-02-01 01:00.
We'll now call `custom_stoploss()` (or any other callback, really) - and you call `get_analyzed_dataframe()` - which will give you data from the start of your timerange. 
In this case, that'll be 2-3 candles. 

---

Now you can assume "I need a lookback of 200 candles for my stoploss logic" (that'd bring us to roughly 2022-01-21) - but that's data that was never loaded, never analyzed, no indicators were calculated - so that's data freqtrade cannot provide to the strategy.

We must take the assumption that each callback will use any indicator (any column) in the strategy - so assuming the callback uses the EMA24 - but you use `df['ema24'].shift(200)` for your stoploss calculation - as you really care on the EMA from 200 candles ago.

That's data we never calculated, never loaded - so freqtrade cannot provide this data. It'll provide data from the start of the timerange - after the startup indicators are removed (otherwise the strategy would have the responsibility to check "has this already been analyzed, or is it NaN").


>  that's data that was never loaded, never analyzed, no indicators were calculated 

They were (because of `startup_candle_count ` being > 0); e.g. the buy signal is based on indicators that look back for days before `--timerange`; `get_analyzed_dataframe()` drops them if they happen to be before `--timerange` which is counter-intuitive and it defies `get_analyzed_dataframe()` 's purpose in part.

It's also a practical advantage to `custom_stoploss()` to have that data. Currently it acts as if no data exists at all before `--timerange` which is technically wrong because it did use more data because `startup_candle_count` was > 0.


> They were (because of startup_candle_count being > 0); 

Your assumption and/or statement is wrong.
Based on the above sample - 24 additional candles were loaded (1 day). 
0 additional candle had ema24 calculated correctly - so EMA would come out as NaN (or whatever else the instable startup provides for the indicator used).

You can therefore not rely to that data in the strategy.

On top of that - i assumed to go back 200 candles (startup candle count was 24) - so that data (2022-01-20) has never been loaded.

> It's also a practical advantage to custom_stoploss() to have that data.

That's correct - but what's the limit (so what would you expect when supplying `--timerange 20220201-`) - should we load data for another month? 2 months? maybe years? 

we have no idea what the logic in the callback looks like - or what it's expecting - so it could go back 1 day - or 1 month - but that'd mean we always MUST load all available data - as you could want to go back all the way (which wont' work in live anyway)


I understand your 2 arguments but the balanced solution is to return more data up to a limit (user-set or not) before `--timerange`, because if you claim you can't know how far back (before `--timerange`) you have to go: then how can anyone rely on `get_analyzed_dataframe()` returning _any_ data before the last candle?

It effectively makes the entire function of `get_analyzed_dataframe()` bugged for anything beyond the last candle (in backtesting).
> I understand your 2 arguments but the balanced solution is to return more data up to a limit (

The limit exists and is defined in the left border of `--timerange`.
to ensure buy signals starting from that time - you can set `startup_candle_count` - to ensure you only get "stable" signals (siganls calculated after an indicator's unstable period).

nobody forces a strategy to do that though - leaving `startup_candle_count=0`  works just as well - obviously you'll have to do "is my indicator now stable" checks yourself (probably by checking for `NaN` values).

We'll surely not load a random additional junk of data that will be analyzed - but then thrown away as "uups, still didn't need it".


your strategy must be able to handle non-available data (for example after a downtime you won't get a guaranteed dataframe length).
If your strategy can't handle such a situation, it'll fail in that event just as it will do in backtesting.
Point is all strategies are effectively bugged if they use `get_analyzed_dataframe()` for more than the last candle if they look for signals that can happen before the last candle based on the data that it returns (even if they do a `len(df)` check to ignore a few dates: it's still an inaccurate backtest).

Though I was meaning to ask you if you can just detect how far back `populate_indicators()` used data in order to produce indicators (which is likely what `get_analyzed_dataframe()` needs for a "lookback length") **but then I realized**,

why don't you just return up to `startup_candle_count` number of candles in the past (before `--timerange`) since that's already defined and that's almost certainly up to what anyone needs both practically and intuitively?
> why don't you just return up to startup_candle_count number of candles in the past (before --timerange) since that's already defined and that's almost certainly up to what anyone needs both practically and intuitively?

That's the point i'm trying to make above - you can't rely on this data - even if we were providing this as part of `get_analyzed_dataframe()` - as that part of the dataframe is NOT analyzed properly.

in fake/conceptual code:

``` python
df = dp.get_analyzed_dataframe(pair)

if df.iloc[-5]['ema24'] > 20:
   return newStopLoss
```


The above code will (at least from a conceptual perspective) work perfectly fine if we were providing the analized dataframe as "timerange - startup_candle_count".

It'll work - UNLESS you get a buy on the first candle - since ema24 will be NaN - so you'll get a "can't compare NaN" error. The dataframe is not properly analyzed - but the content will be empty - so what exactly would we gain with this?
Because in practice I may have a larger `startup_candle_count` than the length of any indicator's lookback, so I may know for certain that an indicator from the first candle of `--timerange` will be valid if I look back only up to a certain number of candles in the past.

Currently: it's certain that nothing is reliable before the last candle so you either have to choose to never rely on candles beyond that last one or: take the "accuracy hit" of using an `if len(df) > #` for the df returned to make sure you'll have a df row available.
letme repeat what i said above - you can't rely on the dataframe lenght in live either.
if an exchange was down for 500 candles - you'll only get the remainder of what that exchange provides (500 for binance).
You can say "we get enough to avoid that" - but you won't know how long an outage takes - (and okx only provides 300 candles by default ...) 

so having a `if len(df) > #` in is always good / necessary - also because you'll  get a completely empty dataframe at bot startup (before any candles are analyzed).
Sure, but `if len(df) > #` will be most likely `True` on live (given no outages) when currently on backtesting `if len(df) > #` will be always `False` on the first few candles of the `--timerange`.

The `len(df)` check could always be there and still try to go as far back as `startup_candle_count` so that backtesting is closer to live.
i don't think you fully understood what i mean.

Even if `len(df) > #` passes for the first candle - it'll have to be replaced with `~df['someindicator'].isnan()` (or the equivalent for a single row) - as you have no guarantee the data is filled with what you expect (a number, most likely).

the assumption that `startup_candle_count` is way bigger than what i need is NOT an assumption freqtrade will base any code on - nor anything we'll recommend in the documentation.

I understand your reasoning, which leaves the problem of non-last-candle data from `get_analyzed_dataframe()` not being guaranteed. It leads to either never using them (and staying only on using the last candle from it) or getting an accuracy hit with a `len(df)` check (and I'm closing this because we've already exhausted it I believe).
unfortunately, backtesting will always have tradeoffs - otherwise we'd all be rich already. 

i can see your point, too - at least to some degree, but i don't think we have many options which don't cause other, potentially worse problems.
we can't guarantee "you get the historic data you need" - because we simply don't know how much you're going to need.
we could apply startup-candles twice - and then provide "half of that" as additional `get_analyzed_dataframe()` - but in more cases than not, that will lead to backtesting loading quite some more data than necessary, which is then never needed/used.

simply providing "whatever we initially loaded before cutoff" is however the worst of all approaches - as that'll no longer give you ANY ensurance on the content of the dataframe (other than ohlcv columns) - as all indicators have a startup time of some sorts (either via NaN - or via incorrect values).
Let's leave this one open for now so i don't forget it (closed issues = forgotten issues).

Will have to have a good thought on how this could be approached (and it's consequences to different scenarios) - it's not a bug per se (it's currently intended) - but if behavior can be improved - that's always welcome.


Resolving patch:
diff --git a/freqtrade/data/dataprovider.py b/freqtrade/data/dataprovider.py
--- a/freqtrade/data/dataprovider.py
+++ b/freqtrade/data/dataprovider.py
@@ -17,7 +17,7 @@
 from freqtrade.data.history import load_pair_history
 from freqtrade.enums import CandleType, RPCMessageType, RunMode
 from freqtrade.exceptions import ExchangeError, OperationalException
-from freqtrade.exchange import Exchange, timeframe_to_seconds
+from freqtrade.exchange import Exchange, timeframe_to_prev_date, timeframe_to_seconds
 from freqtrade.exchange.types import OrderBook
 from freqtrade.misc import append_candles_to_dataframe
 from freqtrade.rpc import RPCManager
@@ -46,6 +46,8 @@ def __init__(
         self.__rpc = rpc
         self.__cached_pairs: Dict[PairWithTimeframe, Tuple[DataFrame, datetime]] = {}
         self.__slice_index: Optional[int] = None
+        self.__slice_date: Optional[datetime] = None
+
         self.__cached_pairs_backtesting: Dict[PairWithTimeframe, DataFrame] = {}
         self.__producer_pairs_df: Dict[str,
                                        Dict[PairWithTimeframe, Tuple[DataFrame, datetime]]] = {}
@@ -64,10 +66,19 @@ def __init__(
     def _set_dataframe_max_index(self, limit_index: int):
         """
         Limit analyzed dataframe to max specified index.
+        Only relevant in backtesting.
         :param limit_index: dataframe index.
         """
         self.__slice_index = limit_index
 
+    def _set_dataframe_max_date(self, limit_date: datetime):
+        """
+        Limit infomrative dataframe to max specified index.
+        Only relevant in backtesting.
+        :param limit_date: "current date"
+        """
+        self.__slice_date = limit_date
+
     def _set_cached_df(
         self,
         pair: str,
@@ -356,6 +367,11 @@ def get_pair_dataframe(
             # Get historical OHLCV data (cached on disk).
             timeframe = timeframe or self._config['timeframe']
             data = self.historic_ohlcv(pair=pair, timeframe=timeframe, candle_type=candle_type)
+            # Cut date to timeframe-specific date.
+            # This is necessary to prevent lookahead bias in callbacks through informative pairs.
+            if self.__slice_date:
+                cutoff_date = timeframe_to_prev_date(timeframe, self.__slice_date)
+                data = data.loc[data['date'] < cutoff_date]
         if len(data) == 0:
             logger.warning(f"No data found for ({pair}, {timeframe}, {candle_type}).")
         return data
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -369,13 +369,14 @@ def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tupl
                 # Cleanup from prior runs
                 pair_data.drop(HEADERS[5:] + ['buy', 'sell'], axis=1, errors='ignore')
             df_analyzed = self.strategy.ft_advise_signals(pair_data, {'pair': pair})
-            # Trim startup period from analyzed dataframe
-            df_analyzed = processed[pair] = pair_data = trim_dataframe(
-                df_analyzed, self.timerange, startup_candles=self.required_startup)
             # Update dataprovider cache
             self.dataprovider._set_cached_df(
                 pair, self.timeframe, df_analyzed, self.config['candle_type_def'])
 
+            # Trim startup period from analyzed dataframe
+            df_analyzed = processed[pair] = pair_data = trim_dataframe(
+                df_analyzed, self.timerange, startup_candles=self.required_startup)
+
             # Create a copy of the dataframe before shifting, that way the entry signal/tag
             # remains on the correct candle for callbacks.
             df_analyzed = df_analyzed.copy()
@@ -1196,7 +1197,8 @@ def backtest(self, processed: Dict,
 
                 row_index += 1
                 indexes[pair] = row_index
-                self.dataprovider._set_dataframe_max_index(row_index)
+                self.dataprovider._set_dataframe_max_index(self.required_startup + row_index)
+                self.dataprovider._set_dataframe_max_date(current_time)
                 current_detail_time: datetime = row[DATE_IDX].to_pydatetime()
                 trade_dir: Optional[LongShort] = self.check_for_trade_entry(row)
 
@@ -1229,12 +1231,14 @@ def backtest(self, processed: Dict,
                     is_first = True
                     current_time_det = current_time
                     for det_row in detail_data[HEADERS].values.tolist():
+                        self.dataprovider._set_dataframe_max_date(current_time_det)
                         open_trade_count_start = self.backtest_loop(
                             det_row, pair, current_time_det, end_date,
                             open_trade_count_start, trade_dir, is_first)
                         current_time_det += timedelta(minutes=self.timeframe_detail_min)
                         is_first = False
                 else:
+                    self.dataprovider._set_dataframe_max_date(current_time)
                     open_trade_count_start = self.backtest_loop(
                         row, pair, current_time, end_date,
                         open_trade_count_start, trade_dir)


Test patch:
diff --git a/tests/data/test_dataprovider.py b/tests/data/test_dataprovider.py
--- a/tests/data/test_dataprovider.py
+++ b/tests/data/test_dataprovider.py
@@ -129,9 +129,14 @@ def test_get_pair_dataframe(mocker, default_conf, ohlcv_history, candle_type):
     default_conf["runmode"] = RunMode.BACKTEST
     dp = DataProvider(default_conf, exchange)
     assert dp.runmode == RunMode.BACKTEST
-    assert isinstance(dp.get_pair_dataframe(
-        "UNITTEST/BTC", timeframe, candle_type=candle_type), DataFrame)
-    # assert dp.get_pair_dataframe("NONESENSE/AAA", timeframe).empty
+    df = dp.get_pair_dataframe("UNITTEST/BTC", timeframe, candle_type=candle_type)
+    assert isinstance(df, DataFrame)
+    assert len(df) == 3  # ohlcv_history mock has just 3 rows
+
+    dp._set_dataframe_max_date(ohlcv_history.iloc[-1]['date'])
+    df = dp.get_pair_dataframe("UNITTEST/BTC", timeframe, candle_type=candle_type)
+    assert isinstance(df, DataFrame)
+    assert len(df) == 2  # ohlcv_history is limited to 2 rows now
 
 
 def test_available_pairs(mocker, default_conf, ohlcv_history):
diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py
--- a/tests/optimize/test_backtesting.py
+++ b/tests/optimize/test_backtesting.py
@@ -20,7 +20,7 @@
 from freqtrade.data.history import get_timerange
 from freqtrade.enums import CandleType, ExitType, RunMode
 from freqtrade.exceptions import DependencyException, OperationalException
-from freqtrade.exchange.exchange import timeframe_to_next_date
+from freqtrade.exchange import timeframe_to_next_date, timeframe_to_prev_date
 from freqtrade.optimize.backtest_caching import get_backtest_metadata_filename, get_strategy_run_id
 from freqtrade.optimize.backtesting import Backtesting
 from freqtrade.persistence import LocalTrade, Trade
@@ -1135,6 +1135,12 @@ def tmp_confirm_entry(pair, current_time, **kwargs):
         assert candle_date == current_time
         # These asserts don't properly raise as they are nested,
         # therefore we increment count and assert for that.
+        df = dp.get_pair_dataframe(pair, backtesting.strategy.timeframe)
+        prior_time = timeframe_to_prev_date(backtesting.strategy.timeframe,
+                                            candle_date - timedelta(seconds=1))
+        assert prior_time == df.iloc[-1].squeeze()['date']
+        assert df.iloc[-1].squeeze()['date'] < current_time
+
         count += 1
 
     backtesting.strategy.confirm_trade_entry = tmp_confirm_entry
@@ -1353,11 +1359,11 @@ def _trend_alternate_hold(dataframe=None, metadata=None):
 
     # Cached data correctly removed amounts
     offset = 1 if tres == 0 else 0
-    removed_candles = len(data[pair]) - offset - backtesting.strategy.startup_candle_count
+    removed_candles = len(data[pair]) - offset
     assert len(backtesting.dataprovider.get_analyzed_dataframe(pair, '5m')[0]) == removed_candles
     assert len(
         backtesting.dataprovider.get_analyzed_dataframe('NXT/BTC', '5m')[0]
-    ) == len(data['NXT/BTC']) - 1 - backtesting.strategy.startup_candle_count
+    ) == len(data['NXT/BTC']) - 1
 
     backtesting.strategy.max_open_trades = 1
     backtesting.config.update({'max_open_trades': 1})