Repository: freqtrade/freqtrade

Issue:
Lag between trigger and order execution for strategy callbacks
## Describe your environment

  * Operating system: MacOS 12.0.1
  * Python Version: n/a
  * CCXT version: n/a
  * Freqtrade Version: freqtrade docker-6c4b2614
  
## Your question

I've been experimenting with the newly released position adjustment feature (using `adjust_trade_position`) and noticed the entry price (i.e. `current_rate`) is delayed by a couple of bars compared to the actual trigger.

For instance, if the last candle is retrieved as:
```
dataframe, _ = self.dp.get_analyzed_dataframe(trade.pair, self.timeframe)
candle = dataframe.iloc[-1].squeeze()
```

by returning a valid stake size, the actual order will be placed at the open price of the second bar after the trigger. 

<table>
<tr>
	<td>candle #
	<td>open rate
	<td>trigger
	<td>order rate
	<td>triggered by
<tr>
	<td>1
	<td>100
	<td>buy
	<td>
	<td>
<tr>
	<td>2
	<td>110
	<td>
	<td>110
	<td>candle # 1
<tr>
	<td>3
	<td>120
	<td>
	<td>
	<td>
<tr>
	<td>3
	<td>130
	<td>adjust_trade
	<td>
	<td>
<tr>
	<td>4
	<td>140
	<td>
	<td>
	<td>
<tr>
	<td>5
	<td>150
	<td>
	<td>150
	<td>candle # 3
<tr>
</table>

I think the ideal logic would be to execute the order at the open price right after the trigger bar has been spotted (which is the current behaviour for buy signals triggered using the `populate_buy_trend` method).

I can see why this is happening (e.g. the callbacks are called once per bar and the result will be used only in the following bar) and that a quick fix could be to just use the `timeframe-detail` argument. However, I think the best way to handle this would be to use the "previous open_rate" as `current_rate` since that should approximate better the live execution. Another issue is that the timeframe-detail argument is not supported in hyperopt.
Lag between trigger and order execution for strategy callbacks
## Describe your environment

  * Operating system: MacOS 12.0.1
  * Python Version: n/a
  * CCXT version: n/a
  * Freqtrade Version: freqtrade docker-6c4b2614
  
## Your question

I've been experimenting with the newly released position adjustment feature (using `adjust_trade_position`) and noticed the entry price (i.e. `current_rate`) is delayed by a couple of bars compared to the actual trigger.

For instance, if the last candle is retrieved as:
```
dataframe, _ = self.dp.get_analyzed_dataframe(trade.pair, self.timeframe)
candle = dataframe.iloc[-1].squeeze()
```

by returning a valid stake size, the actual order will be placed at the open price of the second bar after the trigger. 

<table>
<tr>
	<td>candle #
	<td>open rate
	<td>trigger
	<td>order rate
	<td>triggered by
<tr>
	<td>1
	<td>100
	<td>buy
	<td>
	<td>
<tr>
	<td>2
	<td>110
	<td>
	<td>110
	<td>candle # 1
<tr>
	<td>3
	<td>120
	<td>
	<td>
	<td>
<tr>
	<td>3
	<td>130
	<td>adjust_trade
	<td>
	<td>
<tr>
	<td>4
	<td>140
	<td>
	<td>
	<td>
<tr>
	<td>5
	<td>150
	<td>
	<td>150
	<td>candle # 3
<tr>
</table>

I think the ideal logic would be to execute the order at the open price right after the trigger bar has been spotted (which is the current behaviour for buy signals triggered using the `populate_buy_trend` method).

I can see why this is happening (e.g. the callbacks are called once per bar and the result will be used only in the following bar) and that a quick fix could be to just use the `timeframe-detail` argument. However, I think the best way to handle this would be to use the "previous open_rate" as `current_rate` since that should approximate better the live execution. Another issue is that the timeframe-detail argument is not supported in hyperopt.


Discussion:
are you talking about backtesting? 
please make sure to always mention this - it is unclear from the post above, and only a suspicion based on the very last paragraph.

I'm not sure where you see the delay between adjust_trade and the order entering - but i suspect it's caused by something in your strategy, as backtesting itself will create the buy order as part of the candle that returned a valid response from `adjust_trade_position`. 
The price that's provided as "current rate" / "current profit" will be the open of the 130$ candle (actually as open-price with every one of these candles, as the callback will be called for every candle).

Now alignment between backtesting and live might be difficult here. In live, this callback is called ~every 5 seconds - while in backtesting, it's at maximum called once per timeframe detail.

This means, un-careful code can enter 10 orders within the initial candle, something you can't replicate with backtesting.

I think I might have found the issue. Is it possible the columns `buy` and `buy_tag` get shifted by one within the dataframe returned by `self.dp.get_analyzed_dataframe(trade.pair, self.timeframe)`? 

I'm triggering the position adjustment referencing the `buy_tag` column but I noticed the values are shifted when I tried to print the last candle (`dataframe.iloc[-1].squeeze()`) and compare it with the df created at `populate_sell_trend`

edit: yep I found it [here](https://github.com/freqtrade/freqtrade/blob/6319c104fe1c51e0343c08def9f4511cc537b377/freqtrade/optimize/backtesting.py#L270)

But I don't think this is necessary since the buy signal will be associated to the previous candle anyway using `dataframe.iloc[-1].squeeze()`
The shift is necessary for alignment with live.
a candle from 10:00-11:00 in live will have the buy-signal acted on earliest at 11:00:01 (once the candle is complete)
To have the same behavior in backtesting, we're shifting the buy-signal by one candle.

This is not for the user, but for backtesting internals to have it aligned with live behavior.
I understand that the buy trigger should be executed only on the following candle, but this means buy and sell signals from the previous candle are not accessible within the strategy callbacks. It's a bit misleading to call the method `get_analyzed_dataframe` when you're not really getting the analyzed dataframe itself (it has been tweaked shifting some columns).

Wouldn't it be better to reference the previous candle dynamically in the internals instead of replacing the dataframe columns?

edit: maybe a quick fix could be to shift the columns after having called `_set_cached_df` instead of before? I think this shouldn't break any backtesting functionality.
if your strategy relies on getting the entering candle, you might have a problem anyway.
in live, candles will "age out" eventually (say after 1000 candles) - so after that, you'll not get that candle anymore - and need some fallback code for this.

the problem itself will however start earlier (say after around 900 candles, assuming you use SMA100 to calculate the signals) - as the indicator has 100 candles startup time - only after this it'll be available reliably (exact behavior on will differ by indicator).

for the shifted buy signal, maybe @rokups can have a thought, too? i know he's been thinking about the buy signal shift for a while now.
In my strategy, the callback just needs to reference the previous candle (not the candle when the trade started), so I don't think I'd face the sort of issues you're talking about here.

The overall logic is centralized so the triggers get populated upfront and then the adjust_trade_position checks whether the trigger happened in the last candle and buys if needed. i.e. the trigger logic lives outside of the callback

I'd actually prefer if I could execute trade position adjustments just like I do in `populate_buy_trend` (i.e. with a simple column value) but at the moment it doesn't seem to be possible.
> for the shifted buy signal, maybe @rokups can have a thought, too? i know he's been thinking about the buy signal shift for a while now.

Yes, for that i have been doing `df['_buy'] = df['buy']` in `populate_buy_trend` and using `_buy` column in other places as a quick workaround. IMO a proper solution would be not altering `buy` column. We can still do shifting for convenience like now, but it should be done with another column that gets dropped later after backtesting converts dataframe into lists.
@rokups what do you think of the idea of setting the cache before shifting the columns [here](https://github.com/freqtrade/freqtrade/blob/6319c104fe1c51e0343c08def9f4511cc537b377/freqtrade/optimize/backtesting.py#L275) 
Dataframe does not get copied so it wont make any difference if columns are shifted before or after
You'd have to either copy the frame (performance ...?) 
or shift it, convert to lists - and then shift it back (again - performance??).

Personally, I'd prefer the first option - IF that won't lead to 2 dataframes being "stuck" in memory.
@rokups would you mind to memory-profile this one below?

it seems to not have a negative memory impact in my tests ... (don't look at naming) - and i think fixes this problem.


``` python
    def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]:
        """
        Helper function to convert a processed dataframes into lists for performance reasons.

        Used by backtest() - so keep this optimized for performance.

        :param processed: a processed dictionary with format {pair, data}, which gets cleared to
        optimize memory usage!
        """
        # Every change to this headers list must evaluate further usages of the resulting tuple
        # and eventually change the constants for indexes at the top
        headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high', 'buy_tag', 'exit_tag']
        data: Dict = {}
        self.progress.init_step(BacktestState.CONVERT, len(processed))

        # Create dict with data
        for pair in processed.keys():
            pair_data = processed[pair]
            self.check_abort()
            self.progress.increment()
            if not pair_data.empty:
                pair_data.loc[:, 'buy'] = 0  # cleanup if buy_signal is exist
                pair_data.loc[:, 'sell'] = 0  # cleanup if sell_signal is exist
                pair_data.loc[:, 'buy_tag'] = None  # cleanup if buy_tag is exist
                pair_data.loc[:, 'exit_tag'] = None  # cleanup if exit_tag is exist

            df_analyzed = self.strategy.advise_sell(
                self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair}).copy()
            # Trim startup period from analyzed dataframe
            df_analyzed = processed[pair] = pair_data = trim_dataframe(
                df_analyzed, self.timerange, startup_candles=self.required_startup)

            # Update dataprovider cache
            self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
            df_22 = df_analyzed.copy()
            # To avoid using data from future, we use buy/sell signals shifted
            # from the previous candle
            df_22.loc[:, 'buy'] = df_22.loc[:, 'buy'].shift(1)
            df_22.loc[:, 'sell'] = df_22.loc[:, 'sell'].shift(1)
            df_22.loc[:, 'buy_tag'] = df_22.loc[:, 'buy_tag'].shift(1)
            df_22.loc[:, 'exit_tag'] = df_22.loc[:, 'exit_tag'].shift(1)

            df_22 = df_22.drop(df_22.head(1).index)

            # Convert from Pandas to list for performance reasons
            # (Looping Pandas is slow.)
            data[pair] = df_22[headers].values.tolist()
        return data

```
<details>
<summary>Original max 96.3GB RAM</summary>

```
Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   298  10526.2 MiB  10526.2 MiB           1       @profile
   299                                             def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]:
   300                                                 """
   301                                                 Helper function to convert a processed dataframes into lists for performance reasons.
   302                                         
   303                                                 Used by backtest() - so keep this optimized for performance.
   304                                         
   305                                                 :param processed: a processed dictionary with format {pair, data}, which gets cleared to
   306                                                 optimize memory usage!
   307                                                 """
   308                                                 # Every change to this headers list must evaluate further usages of the resulting tuple
   309                                                 # and eventually change the constants for indexes at the top
   310  10526.2 MiB      0.0 MiB           1           headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high', 'buy_tag', 'exit_tag']
   311  10526.2 MiB      0.0 MiB           1           data: Dict = {}
   312  10526.2 MiB      0.0 MiB           1           self.progress.init_step(BacktestState.CONVERT, len(processed))
   313                                         
   314                                                 # Create dict with data
   315  14777.6 MiB   -467.4 MiB          46           for pair in processed.keys():
   316  14653.6 MiB   -467.4 MiB          45               pair_data = processed[pair]
   317  14653.6 MiB   -467.4 MiB          45               self.check_abort()
   318  14653.6 MiB   -467.4 MiB          45               self.progress.increment()
   319  14653.6 MiB   -467.4 MiB          45               if not pair_data.empty:
   320  14653.6 MiB   -467.4 MiB          45                   pair_data.loc[:, 'buy'] = 0  # cleanup if buy_signal is exist
   321  14653.6 MiB   -467.4 MiB          45                   pair_data.loc[:, 'sell'] = 0  # cleanup if sell_signal is exist
   322  14653.6 MiB   -467.4 MiB          45                   pair_data.loc[:, 'buy_tag'] = None  # cleanup if buy_tag is exist
   323  14653.6 MiB   -467.4 MiB          45                   pair_data.loc[:, 'exit_tag'] = None  # cleanup if exit_tag is exist
   324                                         
   325  14663.8 MiB   -717.1 MiB          90               df_analyzed = self.strategy.advise_sell(
   326  14660.3 MiB   -227.1 MiB          45                   self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair}).copy()
   327                                                     # Trim startup period from analyzed dataframe
   328  14663.8 MiB -10863.3 MiB          90               df_analyzed = processed[pair] = pair_data = trim_dataframe(
   329  14663.8 MiB   -132.5 MiB          45                   df_analyzed, self.timerange, startup_candles=self.required_startup)
   330                                                     # To avoid using data from future, we use buy/sell signals shifted
   331                                                     # from the previous candle
   332  14410.7 MiB  -9553.0 MiB          45               df_analyzed.loc[:, 'buy'] = df_analyzed.loc[:, 'buy'].shift(1)
   333  14410.7 MiB      0.0 MiB          45               df_analyzed.loc[:, 'sell'] = df_analyzed.loc[:, 'sell'].shift(1)
   334  14410.7 MiB      0.0 MiB          45               df_analyzed.loc[:, 'buy_tag'] = df_analyzed.loc[:, 'buy_tag'].shift(1)
   335  14410.7 MiB      0.0 MiB          45               df_analyzed.loc[:, 'exit_tag'] = df_analyzed.loc[:, 'exit_tag'].shift(1)
   336                                         
   337                                                     # Update dataprovider cache
   338  14410.7 MiB      0.0 MiB          45               self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
   339                                         
   340  14673.4 MiB   9774.8 MiB          45               df_analyzed = df_analyzed.drop(df_analyzed.head(1).index)
   341                                         
   342                                                     # Convert from Pandas to list for performance reasons
   343                                                     # (Looping Pandas is slow.)
   344  14777.6 MiB   3307.4 MiB          45               data[pair] = df_analyzed[headers].values.tolist()
   345  14777.6 MiB      0.0 MiB           1           return data
```

</details>

<details>
<summary>Modified max 95.8GB RAM</summary>

```
Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   248   9855.1 MiB   9855.1 MiB           1       @profile
   249                                             def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]:
   250                                                 """
   251                                                 Helper function to convert a processed dataframes into lists for performance reasons.
   252                                         
   253                                                 Used by backtest() - so keep this optimized for performance.
   254                                         
   255                                                 :param processed: a processed dictionary with format {pair, data}, which gets cleared to
   256                                                 optimize memory usage!
   257                                                 """
   258                                                 # Every change to this headers list must evaluate further usages of the resulting tuple
   259                                                 # and eventually change the constants for indexes at the top
   260   9855.1 MiB      0.0 MiB           1           headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high', 'buy_tag', 'exit_tag']
   261   9855.1 MiB      0.0 MiB           1           data: Dict = {}
   262   9855.1 MiB      0.0 MiB           1           self.progress.init_step(BacktestState.CONVERT, len(processed))
   263                                         
   264                                                 # Create dict with data
   265  14771.6 MiB   -486.4 MiB          46           for pair in processed.keys():
   266  14635.8 MiB   -486.4 MiB          45               pair_data = processed[pair]
   267  14635.8 MiB   -486.4 MiB          45               self.check_abort()
   268  14635.8 MiB   -486.4 MiB          45               self.progress.increment()
   269  14635.8 MiB   -486.4 MiB          45               if not pair_data.empty:
   270  14635.8 MiB   -486.2 MiB          45                   pair_data.loc[:, 'buy'] = 0  # cleanup if buy_signal is exist
   271  14635.8 MiB   -486.4 MiB          45                   pair_data.loc[:, 'sell'] = 0  # cleanup if sell_signal is exist
   272  14635.8 MiB   -485.9 MiB          45                   pair_data.loc[:, 'buy_tag'] = None  # cleanup if buy_tag is exist
   273  14635.8 MiB   -485.9 MiB          45                   pair_data.loc[:, 'exit_tag'] = None  # cleanup if exit_tag is exist
   274                                         
   275  14905.8 MiB   8741.9 MiB          90               df_analyzed = self.strategy.advise_sell(
   276  14642.5 MiB  -4933.1 MiB          45                   self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair}).copy()
   277                                                     # Trim startup period from analyzed dataframe
   278  14905.8 MiB  -9797.9 MiB          90               df_analyzed = processed[pair] = pair_data = trim_dataframe(
   279  14905.8 MiB   -188.7 MiB          45                   df_analyzed, self.timerange, startup_candles=self.required_startup)
   280                                         
   281                                                     # Update dataprovider cache
   282  14652.7 MiB  -9609.2 MiB          45               self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
   283  14649.4 MiB   -309.8 MiB          45               df_22 = df_analyzed.copy()
   284                                                     # To avoid using data from future, we use buy/sell signals shifted
   285                                                     # from the previous candle
   286  14649.4 MiB    -93.1 MiB          45               df_22.loc[:, 'buy'] = df_22.loc[:, 'buy'].shift(1)
   287  14649.4 MiB    -93.1 MiB          45               df_22.loc[:, 'sell'] = df_22.loc[:, 'sell'].shift(1)
   288  14649.4 MiB    -93.1 MiB          45               df_22.loc[:, 'buy_tag'] = df_22.loc[:, 'buy_tag'].shift(1)
   289  14649.4 MiB    -93.1 MiB          45               df_22.loc[:, 'exit_tag'] = df_22.loc[:, 'exit_tag'].shift(1)
   290                                         
   291  14652.7 MiB     30.4 MiB          45               df_22 = df_22.drop(df_22.head(1).index)
   292                                         
   293                                                     # Convert from Pandas to list for performance reasons
   294                                                     # (Looping Pandas is slow.)
   295  14771.6 MiB   4033.1 MiB          45               data[pair] = df_22[headers].values.tolist()
   296  14771.6 MiB      0.0 MiB           1           return data
```

</details>
@rokups are you telling me that copying once more consumes LESS memory??
Difference is so insignificant i would call it a margin of error. Copying isnt that big of a deal, we only really pay price of time taking to allocate memory, since we discard that memory soon after. There is that point in time that we have two copies of same dataframe in memory, but it is only true for a single pair at that point of time so it doesnt really have a negative impact even on systems with low amount of RAM.
are you talking about backtesting? 
please make sure to always mention this - it is unclear from the post above, and only a suspicion based on the very last paragraph.

I'm not sure where you see the delay between adjust_trade and the order entering - but i suspect it's caused by something in your strategy, as backtesting itself will create the buy order as part of the candle that returned a valid response from `adjust_trade_position`. 
The price that's provided as "current rate" / "current profit" will be the open of the 130$ candle (actually as open-price with every one of these candles, as the callback will be called for every candle).

Now alignment between backtesting and live might be difficult here. In live, this callback is called ~every 5 seconds - while in backtesting, it's at maximum called once per timeframe detail.

This means, un-careful code can enter 10 orders within the initial candle, something you can't replicate with backtesting.

I think I might have found the issue. Is it possible the columns `buy` and `buy_tag` get shifted by one within the dataframe returned by `self.dp.get_analyzed_dataframe(trade.pair, self.timeframe)`? 

I'm triggering the position adjustment referencing the `buy_tag` column but I noticed the values are shifted when I tried to print the last candle (`dataframe.iloc[-1].squeeze()`) and compare it with the df created at `populate_sell_trend`

edit: yep I found it [here](https://github.com/freqtrade/freqtrade/blob/6319c104fe1c51e0343c08def9f4511cc537b377/freqtrade/optimize/backtesting.py#L270)

But I don't think this is necessary since the buy signal will be associated to the previous candle anyway using `dataframe.iloc[-1].squeeze()`
The shift is necessary for alignment with live.
a candle from 10:00-11:00 in live will have the buy-signal acted on earliest at 11:00:01 (once the candle is complete)
To have the same behavior in backtesting, we're shifting the buy-signal by one candle.

This is not for the user, but for backtesting internals to have it aligned with live behavior.
I understand that the buy trigger should be executed only on the following candle, but this means buy and sell signals from the previous candle are not accessible within the strategy callbacks. It's a bit misleading to call the method `get_analyzed_dataframe` when you're not really getting the analyzed dataframe itself (it has been tweaked shifting some columns).

Wouldn't it be better to reference the previous candle dynamically in the internals instead of replacing the dataframe columns?

edit: maybe a quick fix could be to shift the columns after having called `_set_cached_df` instead of before? I think this shouldn't break any backtesting functionality.
if your strategy relies on getting the entering candle, you might have a problem anyway.
in live, candles will "age out" eventually (say after 1000 candles) - so after that, you'll not get that candle anymore - and need some fallback code for this.

the problem itself will however start earlier (say after around 900 candles, assuming you use SMA100 to calculate the signals) - as the indicator has 100 candles startup time - only after this it'll be available reliably (exact behavior on will differ by indicator).

for the shifted buy signal, maybe @rokups can have a thought, too? i know he's been thinking about the buy signal shift for a while now.
In my strategy, the callback just needs to reference the previous candle (not the candle when the trade started), so I don't think I'd face the sort of issues you're talking about here.

The overall logic is centralized so the triggers get populated upfront and then the adjust_trade_position checks whether the trigger happened in the last candle and buys if needed. i.e. the trigger logic lives outside of the callback

I'd actually prefer if I could execute trade position adjustments just like I do in `populate_buy_trend` (i.e. with a simple column value) but at the moment it doesn't seem to be possible.
> for the shifted buy signal, maybe @rokups can have a thought, too? i know he's been thinking about the buy signal shift for a while now.

Yes, for that i have been doing `df['_buy'] = df['buy']` in `populate_buy_trend` and using `_buy` column in other places as a quick workaround. IMO a proper solution would be not altering `buy` column. We can still do shifting for convenience like now, but it should be done with another column that gets dropped later after backtesting converts dataframe into lists.
@rokups what do you think of the idea of setting the cache before shifting the columns [here](https://github.com/freqtrade/freqtrade/blob/6319c104fe1c51e0343c08def9f4511cc537b377/freqtrade/optimize/backtesting.py#L275) 
Dataframe does not get copied so it wont make any difference if columns are shifted before or after
You'd have to either copy the frame (performance ...?) 
or shift it, convert to lists - and then shift it back (again - performance??).

Personally, I'd prefer the first option - IF that won't lead to 2 dataframes being "stuck" in memory.
@rokups would you mind to memory-profile this one below?

it seems to not have a negative memory impact in my tests ... (don't look at naming) - and i think fixes this problem.


``` python
    def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]:
        """
        Helper function to convert a processed dataframes into lists for performance reasons.

        Used by backtest() - so keep this optimized for performance.

        :param processed: a processed dictionary with format {pair, data}, which gets cleared to
        optimize memory usage!
        """
        # Every change to this headers list must evaluate further usages of the resulting tuple
        # and eventually change the constants for indexes at the top
        headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high', 'buy_tag', 'exit_tag']
        data: Dict = {}
        self.progress.init_step(BacktestState.CONVERT, len(processed))

        # Create dict with data
        for pair in processed.keys():
            pair_data = processed[pair]
            self.check_abort()
            self.progress.increment()
            if not pair_data.empty:
                pair_data.loc[:, 'buy'] = 0  # cleanup if buy_signal is exist
                pair_data.loc[:, 'sell'] = 0  # cleanup if sell_signal is exist
                pair_data.loc[:, 'buy_tag'] = None  # cleanup if buy_tag is exist
                pair_data.loc[:, 'exit_tag'] = None  # cleanup if exit_tag is exist

            df_analyzed = self.strategy.advise_sell(
                self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair}).copy()
            # Trim startup period from analyzed dataframe
            df_analyzed = processed[pair] = pair_data = trim_dataframe(
                df_analyzed, self.timerange, startup_candles=self.required_startup)

            # Update dataprovider cache
            self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
            df_22 = df_analyzed.copy()
            # To avoid using data from future, we use buy/sell signals shifted
            # from the previous candle
            df_22.loc[:, 'buy'] = df_22.loc[:, 'buy'].shift(1)
            df_22.loc[:, 'sell'] = df_22.loc[:, 'sell'].shift(1)
            df_22.loc[:, 'buy_tag'] = df_22.loc[:, 'buy_tag'].shift(1)
            df_22.loc[:, 'exit_tag'] = df_22.loc[:, 'exit_tag'].shift(1)

            df_22 = df_22.drop(df_22.head(1).index)

            # Convert from Pandas to list for performance reasons
            # (Looping Pandas is slow.)
            data[pair] = df_22[headers].values.tolist()
        return data

```
<details>
<summary>Original max 96.3GB RAM</summary>

```
Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   298  10526.2 MiB  10526.2 MiB           1       @profile
   299                                             def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]:
   300                                                 """
   301                                                 Helper function to convert a processed dataframes into lists for performance reasons.
   302                                         
   303                                                 Used by backtest() - so keep this optimized for performance.
   304                                         
   305                                                 :param processed: a processed dictionary with format {pair, data}, which gets cleared to
   306                                                 optimize memory usage!
   307                                                 """
   308                                                 # Every change to this headers list must evaluate further usages of the resulting tuple
   309                                                 # and eventually change the constants for indexes at the top
   310  10526.2 MiB      0.0 MiB           1           headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high', 'buy_tag', 'exit_tag']
   311  10526.2 MiB      0.0 MiB           1           data: Dict = {}
   312  10526.2 MiB      0.0 MiB           1           self.progress.init_step(BacktestState.CONVERT, len(processed))
   313                                         
   314                                                 # Create dict with data
   315  14777.6 MiB   -467.4 MiB          46           for pair in processed.keys():
   316  14653.6 MiB   -467.4 MiB          45               pair_data = processed[pair]
   317  14653.6 MiB   -467.4 MiB          45               self.check_abort()
   318  14653.6 MiB   -467.4 MiB          45               self.progress.increment()
   319  14653.6 MiB   -467.4 MiB          45               if not pair_data.empty:
   320  14653.6 MiB   -467.4 MiB          45                   pair_data.loc[:, 'buy'] = 0  # cleanup if buy_signal is exist
   321  14653.6 MiB   -467.4 MiB          45                   pair_data.loc[:, 'sell'] = 0  # cleanup if sell_signal is exist
   322  14653.6 MiB   -467.4 MiB          45                   pair_data.loc[:, 'buy_tag'] = None  # cleanup if buy_tag is exist
   323  14653.6 MiB   -467.4 MiB          45                   pair_data.loc[:, 'exit_tag'] = None  # cleanup if exit_tag is exist
   324                                         
   325  14663.8 MiB   -717.1 MiB          90               df_analyzed = self.strategy.advise_sell(
   326  14660.3 MiB   -227.1 MiB          45                   self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair}).copy()
   327                                                     # Trim startup period from analyzed dataframe
   328  14663.8 MiB -10863.3 MiB          90               df_analyzed = processed[pair] = pair_data = trim_dataframe(
   329  14663.8 MiB   -132.5 MiB          45                   df_analyzed, self.timerange, startup_candles=self.required_startup)
   330                                                     # To avoid using data from future, we use buy/sell signals shifted
   331                                                     # from the previous candle
   332  14410.7 MiB  -9553.0 MiB          45               df_analyzed.loc[:, 'buy'] = df_analyzed.loc[:, 'buy'].shift(1)
   333  14410.7 MiB      0.0 MiB          45               df_analyzed.loc[:, 'sell'] = df_analyzed.loc[:, 'sell'].shift(1)
   334  14410.7 MiB      0.0 MiB          45               df_analyzed.loc[:, 'buy_tag'] = df_analyzed.loc[:, 'buy_tag'].shift(1)
   335  14410.7 MiB      0.0 MiB          45               df_analyzed.loc[:, 'exit_tag'] = df_analyzed.loc[:, 'exit_tag'].shift(1)
   336                                         
   337                                                     # Update dataprovider cache
   338  14410.7 MiB      0.0 MiB          45               self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
   339                                         
   340  14673.4 MiB   9774.8 MiB          45               df_analyzed = df_analyzed.drop(df_analyzed.head(1).index)
   341                                         
   342                                                     # Convert from Pandas to list for performance reasons
   343                                                     # (Looping Pandas is slow.)
   344  14777.6 MiB   3307.4 MiB          45               data[pair] = df_analyzed[headers].values.tolist()
   345  14777.6 MiB      0.0 MiB           1           return data
```

</details>

<details>
<summary>Modified max 95.8GB RAM</summary>

```
Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   248   9855.1 MiB   9855.1 MiB           1       @profile
   249                                             def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tuple]:
   250                                                 """
   251                                                 Helper function to convert a processed dataframes into lists for performance reasons.
   252                                         
   253                                                 Used by backtest() - so keep this optimized for performance.
   254                                         
   255                                                 :param processed: a processed dictionary with format {pair, data}, which gets cleared to
   256                                                 optimize memory usage!
   257                                                 """
   258                                                 # Every change to this headers list must evaluate further usages of the resulting tuple
   259                                                 # and eventually change the constants for indexes at the top
   260   9855.1 MiB      0.0 MiB           1           headers = ['date', 'buy', 'open', 'close', 'sell', 'low', 'high', 'buy_tag', 'exit_tag']
   261   9855.1 MiB      0.0 MiB           1           data: Dict = {}
   262   9855.1 MiB      0.0 MiB           1           self.progress.init_step(BacktestState.CONVERT, len(processed))
   263                                         
   264                                                 # Create dict with data
   265  14771.6 MiB   -486.4 MiB          46           for pair in processed.keys():
   266  14635.8 MiB   -486.4 MiB          45               pair_data = processed[pair]
   267  14635.8 MiB   -486.4 MiB          45               self.check_abort()
   268  14635.8 MiB   -486.4 MiB          45               self.progress.increment()
   269  14635.8 MiB   -486.4 MiB          45               if not pair_data.empty:
   270  14635.8 MiB   -486.2 MiB          45                   pair_data.loc[:, 'buy'] = 0  # cleanup if buy_signal is exist
   271  14635.8 MiB   -486.4 MiB          45                   pair_data.loc[:, 'sell'] = 0  # cleanup if sell_signal is exist
   272  14635.8 MiB   -485.9 MiB          45                   pair_data.loc[:, 'buy_tag'] = None  # cleanup if buy_tag is exist
   273  14635.8 MiB   -485.9 MiB          45                   pair_data.loc[:, 'exit_tag'] = None  # cleanup if exit_tag is exist
   274                                         
   275  14905.8 MiB   8741.9 MiB          90               df_analyzed = self.strategy.advise_sell(
   276  14642.5 MiB  -4933.1 MiB          45                   self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair}).copy()
   277                                                     # Trim startup period from analyzed dataframe
   278  14905.8 MiB  -9797.9 MiB          90               df_analyzed = processed[pair] = pair_data = trim_dataframe(
   279  14905.8 MiB   -188.7 MiB          45                   df_analyzed, self.timerange, startup_candles=self.required_startup)
   280                                         
   281                                                     # Update dataprovider cache
   282  14652.7 MiB  -9609.2 MiB          45               self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
   283  14649.4 MiB   -309.8 MiB          45               df_22 = df_analyzed.copy()
   284                                                     # To avoid using data from future, we use buy/sell signals shifted
   285                                                     # from the previous candle
   286  14649.4 MiB    -93.1 MiB          45               df_22.loc[:, 'buy'] = df_22.loc[:, 'buy'].shift(1)
   287  14649.4 MiB    -93.1 MiB          45               df_22.loc[:, 'sell'] = df_22.loc[:, 'sell'].shift(1)
   288  14649.4 MiB    -93.1 MiB          45               df_22.loc[:, 'buy_tag'] = df_22.loc[:, 'buy_tag'].shift(1)
   289  14649.4 MiB    -93.1 MiB          45               df_22.loc[:, 'exit_tag'] = df_22.loc[:, 'exit_tag'].shift(1)
   290                                         
   291  14652.7 MiB     30.4 MiB          45               df_22 = df_22.drop(df_22.head(1).index)
   292                                         
   293                                                     # Convert from Pandas to list for performance reasons
   294                                                     # (Looping Pandas is slow.)
   295  14771.6 MiB   4033.1 MiB          45               data[pair] = df_22[headers].values.tolist()
   296  14771.6 MiB      0.0 MiB           1           return data
```

</details>
@rokups are you telling me that copying once more consumes LESS memory??
Difference is so insignificant i would call it a margin of error. Copying isnt that big of a deal, we only really pay price of time taking to allocate memory, since we discard that memory soon after. There is that point in time that we have two copies of same dataframe in memory, but it is only true for a single pair at that point of time so it doesnt really have a negative impact even on systems with low amount of RAM.

Resolving patch:
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -275,6 +275,13 @@ def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tupl
             # Trim startup period from analyzed dataframe
             df_analyzed = processed[pair] = pair_data = trim_dataframe(
                 df_analyzed, self.timerange, startup_candles=self.required_startup)
+            # Update dataprovider cache
+            self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
+
+            # Create a copy of the dataframe before shifting, that way the buy signal/tag
+            # remains on the correct candle for callbacks.
+            df_analyzed = df_analyzed.copy()
+
             # To avoid using data from future, we use buy/sell signals shifted
             # from the previous candle
             df_analyzed.loc[:, 'buy'] = df_analyzed.loc[:, 'buy'].shift(1)
@@ -282,9 +289,6 @@ def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tupl
             df_analyzed.loc[:, 'buy_tag'] = df_analyzed.loc[:, 'buy_tag'].shift(1)
             df_analyzed.loc[:, 'exit_tag'] = df_analyzed.loc[:, 'exit_tag'].shift(1)
 
-            # Update dataprovider cache
-            self.dataprovider._set_cached_df(pair, self.timeframe, df_analyzed)
-
             df_analyzed = df_analyzed.drop(df_analyzed.head(1).index)
 
             # Convert from Pandas to list for performance reasons


Test patch:
diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py
--- a/tests/optimize/test_backtesting.py
+++ b/tests/optimize/test_backtesting.py
@@ -21,6 +21,7 @@
 from freqtrade.data.history import get_timerange
 from freqtrade.enums import RunMode, SellType
 from freqtrade.exceptions import DependencyException, OperationalException
+from freqtrade.exchange.exchange import timeframe_to_next_date
 from freqtrade.misc import get_strategy_run_id
 from freqtrade.optimize.backtesting import Backtesting
 from freqtrade.persistence import LocalTrade
@@ -650,6 +651,7 @@ def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
                              timerange=timerange)
     processed = backtesting.strategy.advise_all_indicators(data)
     min_date, max_date = get_timerange(processed)
+
     result = backtesting.backtest(
         processed=deepcopy(processed),
         start_date=min_date,
@@ -741,6 +743,46 @@ def test_processed(default_conf, mocker, testdatadir) -> None:
         assert col in cols
 
 
+def test_backtest_dataprovider_analyzed_df(default_conf, fee, mocker, testdatadir) -> None:
+    default_conf['use_sell_signal'] = False
+    mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
+    mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001)
+    patch_exchange(mocker)
+    backtesting = Backtesting(default_conf)
+    backtesting._set_strategy(backtesting.strategylist[0])
+    timerange = TimeRange('date', None, 1517227800, 0)
+    data = history.load_data(datadir=testdatadir, timeframe='5m', pairs=['UNITTEST/BTC'],
+                             timerange=timerange)
+    processed = backtesting.strategy.advise_all_indicators(data)
+    min_date, max_date = get_timerange(processed)
+
+    global count
+    count = 0
+
+    def tmp_confirm_entry(pair, current_time, **kwargs):
+        dp = backtesting.strategy.dp
+        df, _ = dp.get_analyzed_dataframe(pair, backtesting.strategy.timeframe)
+        current_candle = df.iloc[-1].squeeze()
+        assert current_candle['buy'] == 1
+
+        candle_date = timeframe_to_next_date(backtesting.strategy.timeframe, current_candle['date'])
+        assert candle_date == current_time
+        # These asserts don't properly raise as they are nested,
+        # therefore we increment count and assert for that.
+        global count
+        count = count + 1
+
+    backtesting.strategy.confirm_trade_entry = tmp_confirm_entry
+    backtesting.backtest(
+        processed=deepcopy(processed),
+        start_date=min_date,
+        end_date=max_date,
+        max_open_trades=10,
+        position_stacking=False,
+    )
+    assert count == 5
+
+
 def test_backtest_pricecontours_protections(default_conf, fee, mocker, testdatadir) -> None:
     # While this test IS a copy of test_backtest_pricecontours, it's needed to ensure
     # results do not carry-over to the next run, which is not given by using parametrize.