Repository: freqtrade/freqtrade

Issue:
Difference backtest live results (two consecutive buys)
## Step 1: Have you search for this issue before posting it?

Yes

  ## Step 3: Describe the problem:

I compare backtest and live differences (5 minute ticker same pair):

If there is a buy signal at 21:00 and the stop loss in reach before 21:05 and the buy signal still exist on the next candle (at 21:05) then the bot will buy 2 times in live  (that's expected behavior).

But in backtest that is not the case he make only 1 trade at 21:00.

![freqtrade_backtest](https://user-images.githubusercontent.com/2643583/60197629-07574700-9840-11e9-843b-60eb52a19d96.png)
![freq_trade_live](https://user-images.githubusercontent.com/2643583/60197630-07574700-9840-11e9-906c-ca6f6abc3d02.png)

Have a nice day.




Discussion:
yes - but i doubt there is much we can do about this, since we won't know the exact close-time during backtesting, so we'll use close (actually next candle open i think).

This blocks this pair for for that duration (of two candles, the 21:00 and 21:05 candles in this case) - which will result in the backtest not buying again at 21:05.

There is actually another thing in this backtest which could be interpreted as misalignment (that's the real problem i think), which is that your strategy buys at 21:00:05, and sells at 21:00:18 (seconds later) - but backtest shows a duration of 5m (buy at 21:00, sell at 21:05).

I guess the asumption during backtesting is that you always hold at least one candle, since the internals of the candle are a black-box during testing (we don't know what happens during that time, other than high/low and close).

I'll need to think about this since i'm not sure what would need to change to facilitate what you are suggesting, and if it's even possible at all.





@xmatthias oki the idea was just to show the difference with a limit use case : about the solution i don't know (maybe just explicit backtest rules/usecase in documentation  : for example if i have roi and stop loss possible on the same candle it takes the worst or the better scenario ?)

Many tks.

I hope one day i can help better in this project than just open issues :-)
  
Testing (and especially reporting!) is a big part of contributing - everyone has his own ideas - which shows new issues, so please keep going :+1: 

I hope you're happy to help me test this (you seem to have a strategy / case where this is happening).

If the below fix solves the issue i'll try to come up with a testcase covering this (but for now, i think it's easier/quicker to run the above backtest again if you don't mind

please move this line (Line 359 in backtesting.py) to line 385 (careful with intentation, it should not be part of the if/else statement, but come after (see below))

https://github.com/freqtrade/freqtrade/blob/086d690df76e85bb1205a5037e73a9eae96a7fdd/freqtrade/optimize/backtesting.py#L359

https://github.com/freqtrade/freqtrade/blob/086d690df76e85bb1205a5037e73a9eae96a7fdd/freqtrade/optimize/backtesting.py#L379-L387


``` python
                if trade_entry:
                    lock_pair_until[pair] = trade_entry.close_time
                    trades.append(trade_entry)
                else:
                    # Set lock_pair_until to end of testing period if trade could not be closed
                    lock_pair_until[pair] = end_date.datetime

                indexes[pair] += 1
            # Move time one configured time_interval ahead.
            tmp += timedelta(minutes=self.ticker_interval_mins)
```
At a first glance I don't think this is the correct way to fix to the problem. If we move `indexes[pair] += 1` down to line 385 and a pair has `row.buy == 0 or row.sell == 1` condition true it will be stuck at the same index all the time.

From what I understand the problem is in the line (with `process_only_new_candles=1`): https://github.com/freqtrade/freqtrade/blob/175fc8591e1d6619f891b4136c7adae106272125/freqtrade/freqtradebot.py#L290 because we return and the loop over `whitelist` was not completed. This causes all the following pairs to not have set the value for `_last_candle_seen_per_pair`.

Same as #1850, #1904 
the line you think the error lies in is not called in backtesting (nothing from that file is) ...

It's not related to #1850, #1904 in my opinion - unless you think your "live" behavior is incorrect, and backtesting would be correct (which i don't think - you buy and sell intracandle (no matter if it's Roi or stoploss) ....)

Your buy then happens one candle later (21:05) - so the 2 bugs you mention don't apply in this case, unless you would expect the bot to hold off buying for 2 or 3 candles (which it clearly doesn't).

For the test:
You're correct, the row.buy == 0 or row.sell == 1 would fault (move the `+1` in there too - before the continue statement) ...
To me, the problem in your case is that that we only evaluate sell-conditions after the buy-candle in backtesting - so you buy at 21:00, the first candle that is sent to anything checking for sells is the one starting at 21:05. This should be solved by this (i did not test this myself, as i don't have your strategy, causing this buy/sell behaviour and creating one from scratch will take a while).
>It's not related to #1850, #1904 in my opinion

Well yes I read too fast the issue and I thought that gaugau3000 was referring to the same candle instead of the next. My bad

EDITED: **the comment below is not related to this issue, but 1904**

Said this, I think there is a problem only in dry run/live (1850, 1904). In backtest we don't know what happens intra-candle.

I try to explain my point of view:

With `process_only_new_candles=1` I expect that **all** buy signals are evaluated only when a new candle come and I don't want a delayed buy. Example:

timeframe: 2h max_open_trades=1, new candle at 12:00 and there are 2 buy signals: ETH/BTC and LTC/BTC. Few seconds and ETH is bought. At 12:45 SL is reached and the bot sells. After few seconds it buys LTC, but this buy is 45 min late (and in my view this shouldn't happen). This is also true with max_open_trades>1.

With `process_only_new_candles=0` I think we have multiple options:

- let the bot re-buy the same pair multiple times during a candle - like now
- exclude from whitelist only pairs already bought in this candle 
- exclude all pairs to avoid delayed buys
let's be careful here - this issue has **nothing** to do with `process_only_new_candles`, so i would rather not mix this in here since it creates nothing than confusion.

The issue, as you reported it, is as follows:
bot buys at 21:00, sells intracandle of 21:00 in live/dry, can (and did) buy again at 21:05.
bot buys at 21:00, sells at 21:05 in backtesting. This prevents the bot from buying again at 21:05 - and is therefore misaligned.

The problem is because during backtesting, roi/stoploss is not evaluated for the 21:00 candle, but only for the 21:05 candle (always assuming candle-open times). This is because the index is moved +1 before passing the data to the sell-evaluation function. This was my (half-hearted, since i didn't think it through) attempt to have you test this for me.

I still stand by that, so i still think that this would fix it (including the fix to my solution you identified).
At the moment I don't have a situation like this, so I can't test. I'll check in the future if it should happen. Maybe gaugau3000 can test.

Proposal: instead of adding `indexes[pair] += 1` three times after every `continue` change
```
trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]:],
                                                         trade_count_lock, args)
to

trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]-1:],
                                                         trade_count_lock, args)
```
___

I have updated my previous comment to avoid confusion and commented on the correct issue.

yes, that should also work (and has the same effect).
@xmatthias @gianlup 
Many tks i will test next week and come back to you.
Hi,

By moving (with the last develop branch from 357 to 384)

> indexes[pair] += 1 

I don't have any trades in the backtest.

and 

> trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]:],
>                                                          trade_count_lock, args)
> to
> 
> trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]-1:],
>                                                          trade_count_lock, args)

work.

Have a nive day.

Resolving patch:
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -373,7 +373,9 @@ def backtest(self, args: Dict) -> DataFrame:
                         continue
                     trade_count_lock[row.date] = trade_count_lock.get(row.date, 0) + 1
 
-                trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]:],
+                # since indexes has been incremented before, we need to go one step back to
+                # also check the buying candle for sell conditions.
+                trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]-1:],
                                                          trade_count_lock, stake_amount,
                                                          max_open_trades)
 


Test patch:
diff --git a/freqtrade/tests/optimize/test_backtest_detail.py b/freqtrade/tests/optimize/test_backtest_detail.py
--- a/freqtrade/tests/optimize/test_backtest_detail.py
+++ b/freqtrade/tests/optimize/test_backtest_detail.py
@@ -14,9 +14,8 @@
                                       _get_frame_time_from_offset,
                                       tests_ticker_interval)
 
-# Test 0 Sell signal sell
+# Test 0: Sell with signal sell in candle 3
 # Test with Stop-loss at 1%
-# TC0: Sell signal in candle 3
 tc0 = BTContainer(data=[
     # D  O     H     L     C     V    B  S
     [0, 5000, 5025, 4975, 4987, 6172, 1, 0],
@@ -29,9 +28,8 @@
     trades=[BTrade(sell_reason=SellType.SELL_SIGNAL, open_tick=1, close_tick=4)]
 )
 
-# Test 1 Minus 8% Close
+# Test 1: Stop-Loss Triggered 1% loss
 # Test with Stop-loss at 1%
-# TC1: Stop-Loss Triggered 1% loss
 tc1 = BTContainer(data=[
     # D  O     H     L     C     V    B  S
     [0, 5000, 5025, 4975, 4987, 6172, 1, 0],
@@ -45,9 +43,8 @@
 )
 
 
-# Test 2 Minus 4% Low, minus 1% close
+# Test 2: Minus 4% Low, minus 1% close
 # Test with Stop-Loss at 3%
-# TC2: Stop-Loss Triggered 3% Loss
 tc2 = BTContainer(data=[
     # D  O     H     L     C     V    B  S
     [0, 5000, 5025, 4975, 4987, 6172, 1, 0],
@@ -61,12 +58,12 @@
 )
 
 
-# Test 3 Candle drops 4%, Recovers 1%.
-#               Entry Criteria Met
-# 	            Candle drops 20%
-# Test with Stop-Loss at 2%
-# TC3: Trade-A: Stop-Loss Triggered 2% Loss
-#          Trade-B: Stop-Loss Triggered 2% Loss
+# Test 3: Multiple trades.
+#         Candle drops 4%, Recovers 1%.
+#         Entry Criteria Met
+#         Candle drops 20%
+#  Trade-A: Stop-Loss Triggered 2% Loss
+#           Trade-B: Stop-Loss Triggered 2% Loss
 tc3 = BTContainer(data=[
     # D  O     H     L     C     V    B  S
     [0, 5000, 5025, 4975, 4987, 6172, 1, 0],
@@ -81,10 +78,10 @@
             BTrade(sell_reason=SellType.STOP_LOSS, open_tick=4, close_tick=5)]
 )
 
-# Test 4 Minus 3% / recovery +15%
+# Test 4: Minus 3% / recovery +15%
 # Candle Data for test 3 – Candle drops 3% Closed 15% up
 # Test with Stop-loss at 2% ROI 6%
-# TC4: Stop-Loss Triggered 2% Loss
+# Stop-Loss Triggered 2% Loss
 tc4 = BTContainer(data=[
     # D  O     H     L     C     V    B  S
     [0, 5000, 5025, 4975, 4987, 6172, 1, 0],
@@ -97,9 +94,8 @@
     trades=[BTrade(sell_reason=SellType.STOP_LOSS, open_tick=1, close_tick=2)]
 )
 
-# Test 5 / Drops 0.5% Closes +20%
-# Set stop-loss at 1% ROI 3%
-# TC5: ROI triggers 3% Gain
+# Test 5: Drops 0.5% Closes +20%, ROI triggers 3% Gain
+# stop-loss: 1%, ROI: 3%
 tc5 = BTContainer(data=[
     # D  O     H     L     C     V    B  S
     [0, 5000, 5025, 4980, 4987, 6172, 1, 0],
@@ -112,9 +108,8 @@
     trades=[BTrade(sell_reason=SellType.ROI, open_tick=1, close_tick=3)]
 )
 
-# Test 6 / Drops 3% / Recovers 6% Positive / Closes 1% positve
-# Set stop-loss at 2% ROI at 5%
-# TC6: Stop-Loss triggers 2% Loss
+# Test 6: Drops 3% / Recovers 6% Positive / Closes 1% positve, Stop-Loss triggers 2% Loss
+# stop-loss: 2% ROI: 5%
 tc6 = BTContainer(data=[
     # D  O     H     L     C     V    B  S
     [0, 5000, 5025, 4975, 4987, 6172, 1, 0],
@@ -127,9 +122,8 @@
     trades=[BTrade(sell_reason=SellType.STOP_LOSS, open_tick=1, close_tick=2)]
 )
 
-# Test 7 - 6% Positive / 1% Negative / Close 1% Positve
-# Set stop-loss at 2% ROI at 3%
-# TC7: ROI Triggers 3% Gain
+# Test 7: 6% Positive / 1% Negative / Close 1% Positve, ROI Triggers 3% Gain
+# stop-loss: 2% ROI: 3%
 tc7 = BTContainer(data=[
     # D  O     H     L     C     V    B  S
     [0, 5000, 5025, 4975, 4987, 6172, 1, 0],
@@ -143,9 +137,8 @@
 )
 
 
-# Test 8 - trailing_stop should raise so candle 3 causes a stoploss.
-# Set stop-loss at 10%, ROI at 10% (should not apply)
-# TC8: Trailing stoploss - stoploss should be adjusted candle 2
+# Test 8: trailing_stop should raise so candle 3 causes a stoploss.
+# stop-loss: 10%, ROI: 10% (should not apply), stoploss adjusted in candle 2
 tc8 = BTContainer(data=[
     # D   O     H     L    C     V    B  S
     [0, 5000, 5050, 4950, 5000, 6172, 1, 0],
@@ -158,10 +151,8 @@
 )
 
 
-# Test 9 - trailing_stop should raise - high and low in same candle.
-# Candle Data for test 9
-# Set stop-loss at 10%, ROI at 10% (should not apply)
-# TC9: Trailing stoploss - stoploss should be adjusted candle 3
+# Test 9: trailing_stop should raise - high and low in same candle.
+# stop-loss: 10%, ROI: 10% (should not apply), stoploss adjusted in candle 3
 tc9 = BTContainer(data=[
     # D   O     H     L     C    V    B  S
     [0, 5000, 5050, 4950, 5000, 6172, 1, 0],
@@ -173,10 +164,9 @@
     trades=[BTrade(sell_reason=SellType.TRAILING_STOP_LOSS, open_tick=1, close_tick=3)]
 )
 
-# Test 10 - trailing_stop should raise so candle 3 causes a stoploss
+# Test 10: trailing_stop should raise so candle 3 causes a stoploss
 # without applying trailing_stop_positive since stoploss_offset is at 10%.
-# Set stop-loss at 10%, ROI at 10% (should not apply)
-# TC10: Trailing stoploss - stoploss should be adjusted candle 2
+# stop-loss: 10%, ROI: 10% (should not apply), stoploss adjusted candle 2
 tc10 = BTContainer(data=[
     # D   O     H     L     C    V    B  S
     [0, 5000, 5050, 4950, 5000, 6172, 1, 0],
@@ -190,10 +180,9 @@
     trades=[BTrade(sell_reason=SellType.STOP_LOSS, open_tick=1, close_tick=4)]
 )
 
-# Test 11 - trailing_stop should raise so candle 3 causes a stoploss
+# Test 11: trailing_stop should raise so candle 3 causes a stoploss
 # applying a positive trailing stop of 3% since stop_positive_offset is reached.
-# Set stop-loss at 10%, ROI at 10% (should not apply)
-# TC11: Trailing stoploss - stoploss should be adjusted candle 2,
+# stop-loss: 10%, ROI: 10% (should not apply), stoploss adjusted candle 2
 tc11 = BTContainer(data=[
     # D   O     H     L     C    V    B  S
     [0, 5000, 5050, 4950, 5000, 6172, 1, 0],
@@ -207,10 +196,9 @@
     trades=[BTrade(sell_reason=SellType.TRAILING_STOP_LOSS, open_tick=1, close_tick=3)]
 )
 
-# Test 12 - trailing_stop should raise in candle 2 and cause a stoploss in the same candle
+# Test 12: trailing_stop should raise in candle 2 and cause a stoploss in the same candle
 # applying a positive trailing stop of 3% since stop_positive_offset is reached.
-# Set stop-loss at 10%, ROI at 10% (should not apply)
-# TC12: Trailing stoploss - stoploss should be adjusted candle 2,
+# stop-loss: 10%, ROI: 10% (should not apply), stoploss adjusted candle 2
 tc12 = BTContainer(data=[
     # D   O     H     L     C    V    B  S
     [0, 5000, 5050, 4950, 5000, 6172, 1, 0],
@@ -224,6 +212,47 @@
     trades=[BTrade(sell_reason=SellType.TRAILING_STOP_LOSS, open_tick=1, close_tick=2)]
 )
 
+# Test 13: Buy and sell ROI on same candle
+# stop-loss: 10% (should not apply), ROI: 1%
+tc13 = BTContainer(data=[
+    # D   O     H     L     C    V    B  S
+    [0, 5000, 5050, 4950, 5000, 6172, 1, 0],
+    [1, 5000, 5100, 4950, 5100, 6172, 0, 0],
+    [2, 5100, 5251, 4850, 5100, 6172, 0, 0],
+    [3, 4850, 5050, 4850, 4750, 6172, 0, 0],
+    [4, 4750, 4950, 4850, 4750, 6172, 0, 0]],
+    stop_loss=-0.10, roi=0.01, profit_perc=0.01,
+    trades=[BTrade(sell_reason=SellType.ROI, open_tick=1, close_tick=1)]
+)
+
+# Test 14 - Buy and Stoploss on same candle
+# stop-loss: 5%, ROI: 10% (should not apply)
+tc14 = BTContainer(data=[
+    # D   O     H     L     C    V    B  S
+    [0, 5000, 5050, 4950, 5000, 6172, 1, 0],
+    [1, 5000, 5100, 4600, 5100, 6172, 0, 0],
+    [2, 5100, 5251, 4850, 5100, 6172, 0, 0],
+    [3, 4850, 5050, 4850, 4750, 6172, 0, 0],
+    [4, 4750, 4950, 4350, 4750, 6172, 0, 0]],
+    stop_loss=-0.05, roi=0.10, profit_perc=-0.05,
+    trades=[BTrade(sell_reason=SellType.STOP_LOSS, open_tick=1, close_tick=1)]
+)
+
+
+# Test 15 - Buy and ROI on same candle, followed by buy and Stoploss on next candle
+# stop-loss: 5%, ROI: 10% (should not apply)
+tc15 = BTContainer(data=[
+    # D   O     H     L     C    V    B  S
+    [0, 5000, 5050, 4950, 5000, 6172, 1, 0],
+    [1, 5000, 5100, 4900, 5100, 6172, 1, 0],
+    [2, 5100, 5251, 4650, 5100, 6172, 0, 0],
+    [3, 4850, 5050, 4850, 4750, 6172, 0, 0],
+    [4, 4750, 4950, 4350, 4750, 6172, 0, 0]],
+    stop_loss=-0.05, roi=0.01, profit_perc=-0.04,
+    trades=[BTrade(sell_reason=SellType.ROI, open_tick=1, close_tick=1),
+            BTrade(sell_reason=SellType.STOP_LOSS, open_tick=2, close_tick=2)]
+)
+
 TESTS = [
     tc0,
     tc1,
@@ -238,6 +267,9 @@
     tc10,
     tc11,
     tc12,
+    tc13,
+    tc14,
+    tc15,
 ]
 
 
diff --git a/freqtrade/tests/optimize/test_backtesting.py b/freqtrade/tests/optimize/test_backtesting.py
--- a/freqtrade/tests/optimize/test_backtesting.py
+++ b/freqtrade/tests/optimize/test_backtesting.py
@@ -618,8 +618,9 @@ def test_processed(default_conf, mocker) -> None:
 
 
 def test_backtest_pricecontours(default_conf, fee, mocker) -> None:
+    # TODO: Evaluate usefullness of this, the patterns and buy-signls are unrealistic
     mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
-    tests = [['raise', 19], ['lower', 0], ['sine', 18]]
+    tests = [['raise', 19], ['lower', 0], ['sine', 35]]
     # We need to enable sell-signal - otherwise it sells on ROI!!
     default_conf['experimental'] = {"use_sell_signal": True}