Repository: freqtrade/freqtrade

Issue:
backtesting with timeframe-detail fills orders on "long" timeframe
<!-- 
Have you searched for similar issues before posting it?

If you have discovered a bug in the bot, please [search the issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue). 
If it hasn't been reported, please create a new issue.

Please do not use bug reports to request new features.
-->

## Describe your environment

  * Operating system: All
  * Python Version: 3.10
  * CCXT version: 1.93.90
  * Freqtrade Version: 2022.9

## Describe the problem:

Seems there is incorrect handling of the custom_stoploss in backtesting. When timeframe detail used, the stoploss call does not match the expected price changes within the timeframe.
Example:
- timeframe=1h, timeframe-detail=1m
- enter trade and stop loss transactions will happen at a single 1h candle (let's say, candle starts at 0:00)
- enter trade transaction is in the middle of the candle (e.g. 0:45)
- custom_stoploss seems to be activated before actual enter trade, so seems this function does not respect the timeframe-detail parameter (assume stop loss condition was satisfied only between 0:30-0:35)
- observed trade executed at the price from 0:45 and stop loss price taken from 0:30

### Steps to reproduce:

  1. request a transaction that will be executed in the middle of the candle
  2. define a custom-stoploss,
  3. run a backtesting with timeframe-detail
  4. incorrect stop loss transaction may be observed when stop loss conditions satisfied at the candle before the moment when transaction should be executed
  
### Observed Results:

  * What happened?
 Observed the stop loss call that seems to be relevant to the data which is valid before the transaction execution.
  * What did you expect to happen?
 Stop loss should be executed obviously only after transaction execution.





Discussion:
Are you please able to provide us with a minimal reproduction?
basically a simple strategy that shows this problem, as well as the command used  (so we know the timerange, ...) to highlight the problem.

> request a transaction that will be executed in the middle of the candle

i'm currently not sure how this should be possible - entries are only checked against the long timeframe (all signals are on the long timeframe only, so using the detail-timeframe would accomplish nothing other than slowing down backtesting).

As such, an entry will either fill at open - or at the next "long" candle (if price is used outside of the initial candle).



Hi @xmatthias,

OK So you are saying that entries are checked only at the "long" candle open? What if I have trade at price that is reached in the middle of the "long" candle? Then the trade will be placed at the open of the next one? And using detail-timeframe is not able to fix this behavior?
I'm not sure what the purpose of this would be if i'm honest.

It'd be slower - and you'd get "by minute" fills (fill time is not kept in backtesting though) - so your trade will still be at 08:00.

if you're trading on 1h candles (your main timeframe is 1h) - then your trade duration will not be <1h.
That'd be guessing at best - as you'd never have the information to know if the next 1h candle is a green or a red one - not without sub-hour data.


So if the trade fills at 08:00, or at 08:45 - it'll make little difference.
it'll either hit stoploss within this candle, or it doesn't (backtesting will make it at 08:00 - as price was between high and low).

So assuming you have an entry at 08:00 - with a price between high/low of the 08:00-09:00 candle - it'd fill at 08:00 - and exit processing (in detail timeframe) will commence at 08:01 (actually at 08:00 - but for clarity's sake ...).

---

It might be an improvement to change behavior - but i'm not sure its worth the additional complexity - and potentially the additional performance penalty this would imply.

it might even make gaming backtesting easier - but i'm not sure it'd really improve accuracy (alignment with dry-runs) by a relevant margin.
Hi @xmatthias ,

You can easilly imagine situation when at a single 1 hour candle you enter the trade and the stoploss is reached. My expectation was that it can be handled using the timeframe-detail option. That is expected to provide higher resolution in trades. If that's not the case - what is the advantage of using timeframe-detail? I understand that under current implementation assumptions we must change the "long" interval to minutes and adjust the strategy to it to ensure the resolution requirement is satisfied.

I don't see how that would change behavior, really (not if you think about it carefully).

Assuming you place a long order on a 1h candle with a range of 90-110 - with a price of 100.

if you fill the order at :00 - or at :30 won't really change the outcome (not for the initial candle).
the fill price will still be 100 - and the low point has to be after the :30 point (otherwise it would have filled earlier - the moment price is ==100.

therefore stoploss calcs will work just fine (we know the open price - and we know the candle's low point is reached after the trade fills.

if the stoploss is within the candle's low point - it'll trigger either way (and will fill at the stop price).

As said above - it can be a future enhancement (not something i'll have time to look at anytime soon) - but it's not diminishing the value of timeframe-detail - as it will not make a real-world difference.
It is making a significant order when using custom_entry_price.
Imagine this situation
Candle 1h opens on 110, falls down to 100, then falls back to 90
9:00 - 110
9:30 - 100
9:59 - 90

You used custom_entry_price with value 100 and position is long.

You are using following custom_stop_loss 
```
    stop_loss = 0.05
    use_custom_stoploss = False
    def custom_stoploss(self, pair: str, trade: Trade, current_time: datetime, current_rate: float,
                        current_profit: float, **kwargs) -> float:
        if current_profit > 0.05
            return 0.05  # trailing stop
        return 1  # default SL
```

The custom stop loss is executed from date time 9:00
So the first execution is at current_rate 110 so the current_profit is 0.10 the trailing stop in custom_stop_loss is activated.
The price falls down to 105 the trade is finished with profit 5%

In reality the order is filled at 100, and the price keeps falling down until it hits default stop loss 5%
So the trade ends on 95 with 5% loss.

I also checked the sitution with default trailing_stop_loss and there is a similar bug.

Everything is OK, if we are not using custom_entry_price and the order is filled ~ at the open of the candle.



Resolving patch:
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -688,10 +688,11 @@ def _exit_trade(self, trade: LocalTrade, sell_row: Tuple,
         trade.orders.append(order)
         return trade
 
-    def _get_exit_trade_entry(self, trade: LocalTrade, row: Tuple) -> Optional[LocalTrade]:
+    def _get_exit_trade_entry(
+            self, trade: LocalTrade, row: Tuple, is_first: bool) -> Optional[LocalTrade]:
         exit_candle_time: datetime = row[DATE_IDX].to_pydatetime()
 
-        if self.trading_mode == TradingMode.FUTURES:
+        if is_first and self.trading_mode == TradingMode.FUTURES:
             trade.funding_fees = self.exchange.calculate_funding_fees(
                 self.futures_data[trade.pair],
                 amount=trade.amount,
@@ -700,32 +701,7 @@ def _get_exit_trade_entry(self, trade: LocalTrade, row: Tuple) -> Optional[Local
                 close_date=exit_candle_time,
             )
 
-        if self.timeframe_detail and trade.pair in self.detail_data:
-            exit_candle_end = exit_candle_time + timedelta(minutes=self.timeframe_min)
-
-            detail_data = self.detail_data[trade.pair]
-            detail_data = detail_data.loc[
-                (detail_data['date'] >= exit_candle_time) &
-                (detail_data['date'] < exit_candle_end)
-            ].copy()
-            if len(detail_data) == 0:
-                # Fall back to "regular" data if no detail data was found for this candle
-                return self._get_exit_trade_entry_for_candle(trade, row)
-            detail_data.loc[:, 'enter_long'] = row[LONG_IDX]
-            detail_data.loc[:, 'exit_long'] = row[ELONG_IDX]
-            detail_data.loc[:, 'enter_short'] = row[SHORT_IDX]
-            detail_data.loc[:, 'exit_short'] = row[ESHORT_IDX]
-            detail_data.loc[:, 'enter_tag'] = row[ENTER_TAG_IDX]
-            detail_data.loc[:, 'exit_tag'] = row[EXIT_TAG_IDX]
-            for det_row in detail_data[HEADERS].values.tolist():
-                res = self._get_exit_trade_entry_for_candle(trade, det_row)
-                if res:
-                    return res
-
-            return None
-
-        else:
-            return self._get_exit_trade_entry_for_candle(trade, row)
+        return self._get_exit_trade_entry_for_candle(trade, row)
 
     def get_valid_price_and_stake(
         self, pair: str, row: Tuple, propose_rate: float, stake_amount: float,
@@ -1070,7 +1046,7 @@ def validate_row(
 
     def backtest_loop(
             self, row: Tuple, pair: str, current_time: datetime, end_date: datetime,
-            max_open_trades: int, open_trade_count_start: int) -> int:
+            max_open_trades: int, open_trade_count_start: int, is_first: bool = True) -> int:
         """
         NOTE: This method is used by Hyperopt at each iteration. Please keep it optimized.
 
@@ -1088,9 +1064,11 @@ def backtest_loop(
         # without positionstacking, we can only have one open trade per pair.
         # max_open_trades must be respected
         # don't open on the last row
+        # We only open trades on the main candle, not on detail candles
         trade_dir = self.check_for_trade_entry(row)
         if (
             (self._position_stacking or len(LocalTrade.bt_trades_open_pp[pair]) == 0)
+            and is_first
             and self.trade_slot_available(max_open_trades, open_trade_count_start)
             and current_time != end_date
             and trade_dir is not None
@@ -1116,7 +1094,7 @@ def backtest_loop(
 
                 # 4. Create exit orders (if any)
             if not trade.open_order_id:
-                self._get_exit_trade_entry(trade, row)  # Place exit order if necessary
+                self._get_exit_trade_entry(trade, row, is_first)  # Place exit order if necessary
 
                 # 5. Process exit orders.
             order = trade.select_order(trade.exit_side, is_open=True)
@@ -1167,7 +1145,6 @@ def backtest(self, processed: Dict,
 
         self.progress.init_step(BacktestState.BACKTEST, int(
             (end_date - start_date) / timedelta(minutes=self.timeframe_min)))
-
         # Loop timerange and get candle for each pair at that point in time
         while current_time <= end_date:
             open_trade_count_start = LocalTrade.bt_open_open_trade_count
@@ -1181,9 +1158,37 @@ def backtest(self, processed: Dict,
                 row_index += 1
                 indexes[pair] = row_index
                 self.dataprovider._set_dataframe_max_index(row_index)
-
-                open_trade_count_start = self.backtest_loop(
-                    row, pair, current_time, end_date, max_open_trades, open_trade_count_start)
+                current_detail_time: datetime = row[DATE_IDX].to_pydatetime()
+                if self.timeframe_detail and pair in self.detail_data:
+                    exit_candle_end = current_detail_time + timedelta(minutes=self.timeframe_min)
+
+                    detail_data = self.detail_data[pair]
+                    detail_data = detail_data.loc[
+                        (detail_data['date'] >= current_detail_time) &
+                        (detail_data['date'] < exit_candle_end)
+                    ].copy()
+                    if len(detail_data) == 0:
+                        # Fall back to "regular" data if no detail data was found for this candle
+                        open_trade_count_start = self.backtest_loop(
+                            row, pair, current_time, end_date, max_open_trades,
+                            open_trade_count_start)
+                    detail_data.loc[:, 'enter_long'] = row[LONG_IDX]
+                    detail_data.loc[:, 'exit_long'] = row[ELONG_IDX]
+                    detail_data.loc[:, 'enter_short'] = row[SHORT_IDX]
+                    detail_data.loc[:, 'exit_short'] = row[ESHORT_IDX]
+                    detail_data.loc[:, 'enter_tag'] = row[ENTER_TAG_IDX]
+                    detail_data.loc[:, 'exit_tag'] = row[EXIT_TAG_IDX]
+                    is_first = True
+                    current_time_det = current_time
+                    for det_row in detail_data[HEADERS].values.tolist():
+                        open_trade_count_start = self.backtest_loop(
+                            det_row, pair, current_time_det, end_date, max_open_trades,
+                            open_trade_count_start, is_first)
+                        current_time_det += timedelta(minutes=self.timeframe_detail_min)
+                        is_first = False
+                else:
+                    open_trade_count_start = self.backtest_loop(
+                        row, pair, current_time, end_date, max_open_trades, open_trade_count_start)
 
             # Move time one configured time_interval ahead.
             self.progress.increment()


Test patch:
diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py
--- a/tests/optimize/test_backtesting.py
+++ b/tests/optimize/test_backtesting.py
@@ -663,30 +663,9 @@ def test_backtest__get_sell_trade_entry(default_conf, fee, mocker) -> None:
         '',  # Exit Signal Name
 
     ]
-    row_detail = pd.DataFrame(
-        [
-            [
-                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=0, tzinfo=timezone.utc),
-                200, 200.1, 197, 199, 1, 0, 0, 0, '', '', '',
-            ], [
-                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=1, tzinfo=timezone.utc),
-                199, 199.7, 199, 199.5, 0, 0, 0, 0, '', '', '',
-            ], [
-                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=2, tzinfo=timezone.utc),
-                199.5, 200.8, 199, 200.9, 0, 0, 0, 0, '', '', '',
-            ], [
-                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=3, tzinfo=timezone.utc),
-                200.5, 210.5, 193, 210.5, 0, 0, 0, 0, '', '', '',  # ROI sell (?)
-            ], [
-                pd.Timestamp(year=2020, month=1, day=1, hour=5, minute=4, tzinfo=timezone.utc),
-                200, 200.1, 193, 199, 0, 0, 0, 0, '', '', '',
-            ],
-        ], columns=['date', 'open', 'high', 'low', 'close', 'enter_long', 'exit_long',
-                    'enter_short', 'exit_short', 'long_tag', 'short_tag', 'exit_tag']
-    )
 
     # No data available.
-    res = backtesting._get_exit_trade_entry(trade, row_sell)
+    res = backtesting._get_exit_trade_entry(trade, row_sell, True)
     assert res is not None
     assert res.exit_reason == ExitType.ROI.value
     assert res.close_date_utc == datetime(2020, 1, 1, 5, 0, tzinfo=timezone.utc)
@@ -699,20 +678,9 @@ def test_backtest__get_sell_trade_entry(default_conf, fee, mocker) -> None:
         [], columns=['date', 'open', 'high', 'low', 'close', 'enter_long', 'exit_long',
                      'enter_short', 'exit_short', 'long_tag', 'short_tag', 'exit_tag'])
 
-    res = backtesting._get_exit_trade_entry(trade, row)
+    res = backtesting._get_exit_trade_entry(trade, row, True)
     assert res is None
 
-    # Assign backtest-detail data
-    backtesting.detail_data[pair] = row_detail
-
-    res = backtesting._get_exit_trade_entry(trade, row_sell)
-    assert res is not None
-    assert res.exit_reason == ExitType.ROI.value
-    # Sell at minute 3 (not available above!)
-    assert res.close_date_utc == datetime(2020, 1, 1, 5, 3, tzinfo=timezone.utc)
-    sell_order = res.select_order('sell', True)
-    assert sell_order is not None
-
 
 def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
     default_conf['use_exit_signal'] = False
@@ -787,17 +755,98 @@ def test_backtest_one(default_conf, fee, mocker, testdatadir) -> None:
     for _, t in results.iterrows():
         assert len(t['orders']) == 2
         ln = data_pair.loc[data_pair["date"] == t["open_date"]]
-        # Check open trade rate alignes to open rate
+        # Check open trade rate aligns to open rate
         assert not ln.empty
         assert round(ln.iloc[0]["open"], 6) == round(t["open_rate"], 6)
-        # check close trade rate alignes to close rate or is between high and low
+        # check close trade rate aligns to close rate or is between high and low
         ln1 = data_pair.loc[data_pair["date"] == t["close_date"]]
-        assert not ln1.empty
         assert (round(ln1.iloc[0]["open"], 6) == round(t["close_rate"], 6) or
                 round(ln1.iloc[0]["low"], 6) < round(
                 t["close_rate"], 6) < round(ln1.iloc[0]["high"], 6))
 
 
+@pytest.mark.parametrize('use_detail', [True, False])
+def test_backtest_one_detail(default_conf_usdt, fee, mocker, testdatadir, use_detail) -> None:
+    default_conf_usdt['use_exit_signal'] = False
+    mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
+    mocker.patch("freqtrade.exchange.Exchange.get_min_pair_stake_amount", return_value=0.00001)
+    mocker.patch("freqtrade.exchange.Exchange.get_max_pair_stake_amount", return_value=float('inf'))
+    if use_detail:
+        default_conf_usdt['timeframe_detail'] = '1m'
+    patch_exchange(mocker)
+
+    def advise_entry(df, *args, **kwargs):
+        # Mock function to force several entries
+        df.loc[(df['rsi'] < 40), 'enter_long'] = 1
+        return df
+
+    def custom_entry_price(proposed_rate, **kwargs):
+        return proposed_rate * 0.997
+
+    backtesting = Backtesting(default_conf_usdt)
+    backtesting._set_strategy(backtesting.strategylist[0])
+    backtesting.strategy.populate_entry_trend = advise_entry
+    backtesting.strategy.custom_entry_price = custom_entry_price
+    pair = 'XRP/ETH'
+    # Pick a timerange adapted to the pair we use to test
+    timerange = TimeRange.parse_timerange('20191010-20191013')
+    data = history.load_data(datadir=testdatadir, timeframe='5m', pairs=['XRP/ETH'],
+                             timerange=timerange)
+    if use_detail:
+        data_1m = history.load_data(datadir=testdatadir, timeframe='1m', pairs=['XRP/ETH'],
+                                    timerange=timerange)
+        backtesting.detail_data = data_1m
+    processed = backtesting.strategy.advise_all_indicators(data)
+    min_date, max_date = get_timerange(processed)
+
+    result = backtesting.backtest(
+        processed=deepcopy(processed),
+        start_date=min_date,
+        end_date=max_date,
+        max_open_trades=10,
+    )
+    results = result['results']
+    assert not results.empty
+    # Timeout settings from default_conf = entry: 10, exit: 30
+    assert len(results) == (2 if use_detail else 3)
+
+    assert 'orders' in results.columns
+    data_pair = processed[pair]
+
+    data_1m_pair = data_1m[pair] if use_detail else pd.DataFrame()
+    late_entry = 0
+    for _, t in results.iterrows():
+        assert len(t['orders']) == 2
+
+        entryo = t['orders'][0]
+        entry_ts = datetime.fromtimestamp(entryo['order_filled_timestamp'] // 1000, tz=timezone.utc)
+        if entry_ts > t['open_date']:
+            late_entry += 1
+
+        # Get "entry fill" candle
+        ln = (data_1m_pair.loc[data_1m_pair["date"] == entry_ts]
+              if use_detail else data_pair.loc[data_pair["date"] == entry_ts])
+        # Check open trade rate aligns to open rate
+        assert not ln.empty
+
+        # assert round(ln.iloc[0]["open"], 6) == round(t["open_rate"], 6)
+        assert round(ln.iloc[0]["low"], 6) <= round(
+                t["open_rate"], 6) <= round(ln.iloc[0]["high"], 6)
+        # check close trade rate aligns to close rate or is between high and low
+        ln1 = data_pair.loc[data_pair["date"] == t["close_date"]]
+        if use_detail:
+            ln1_1m = data_1m_pair.loc[data_1m_pair["date"] == t["close_date"]]
+            assert not ln1.empty or not ln1_1m.empty
+        else:
+            assert not ln1.empty
+        ln2 = ln1_1m if ln1.empty else ln1
+
+        assert (round(ln2.iloc[0]["low"], 6) <= round(
+                t["close_rate"], 6) <= round(ln2.iloc[0]["high"], 6))
+
+    assert late_entry > 0
+
+
 def test_backtest_timedout_entry_orders(default_conf, fee, mocker, testdatadir) -> None:
     # This strategy intentionally places unfillable orders.
     default_conf['strategy'] = 'StrategyTestV3CustomEntryPrice'