Repository: freqtrade/freqtrade

Issue:
Stoploss on exchange correctly executed but not cleared in Freqtrade
## Describe your environment

  * Operating system: Ubuntu 20.04
  * Python Version: 3.8.10
  * CCXT version: 1.71.73
  * Freqtrade Version: develop-06387478b

## Describe the problem:
If bot is configured to use market stoploss on exchange (FTX), the stop market order get correctly executed on exchange but Freqtrade still have the trade open not being able to sell it.

*Explain the problem you have encountered*
Running a live bot on FTX exchange, configured with market stoploss.
When the trade was opened, the stoploss market order has been correctly set on exchange.
When the trade got stopped, the order on exchange has been correctly and safely executed but Freqtrade still had the trade open and the slot busy with it.

### Steps to reproduce:
Strategy is not using `custom_stoploss` nor `confirm_trade_exit`
```
stoploss = -0.03
```
Configuration used:
```
    "order_types": {
        "buy": "limit",
        "sell": "market",
        "emergencysell": "market",
        "forcesell": "market",
        "stoploss": "market",
        "stoploss_on_exchange": true,
        "stoploss_on_exchange_interval": 60
    },
```
  
### Observed Results:

  * What happened?  
  Trade correctly closed in stoploss at the exchange
  * What did you expect to happen?  
  Freqtrade selling the trade on database and clearing the slot

### Relevant code exceptions or logs
```
2022-02-11 09:18:00,325 - freqtrade.exchange.ftx - INFO - stoploss order added for SOL/USD. stop price: 102.8325.
```
  ```
2022-02-11 19:28:43,057 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:29:43,094 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:30:24,608 - VolumePairList - INFO - Searching 500 pairs: ['BTC/USD', 'ETH/USD', 'SOL/USD', 'AVAX/USD', 'BNB/USD', 'XRP/USD', 'FTT/USD', 'MATIC/USD', 'SLP/USD', 'LTC/USD', 'LOOKS/USD', 'LINK/USD', 'FTM/USD', 'SUSHI/USD', 'DOGE/USD', 'RAY/USD', 'SHIB/USD', 'DOT/USD', 'SAND/USD', 'CRO/USD', 'BCH/USD', 'SRM/USD', 'MANA/USD', 'CRV/USD', 'AXS/USD', 'RUNE/USD', 'SPELL/USD', 'DYDX/USD', 'WBTC/USD', 'ATLAS/USD', 'UNI/USD', 'OMG/USD', 'BIT/USD', 'KNC/USD', 'TONCOIN/USD', 'GALA/USD', 'AAVE/USD', 'STEP/USD', 'RSR/USD', 'POLIS/USD', 'TRX/USD', 'YFI/USD', 'SXP/USD', 'SKL/USD', 'PERP/USD', 'CLV/USD', 'BOBA/USD', 'HNT/USD', 'RNDR/USD', 'UST/USD', 'KSHIB/USD']
2022-02-11 19:30:41,639 - freqtrade.exchange.common - WARNING - fetch_ticker() returned exception: "Could not load ticker due to ExchangeNotAvailable. Message: ftx GET https://ftx.com/api/markets/SOL/USD 500 Internal Server Error <html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
<script defer src="https://static.cloudflareinsights.com/beacon.min.js/v652eace1692a40cfa3763df669d7439c1639079717194" integrity="sha512-Gi7xpJR8tSkrpF7aordPZQlW2DLtzUlZcumS8dMQjwDHEnw9I7ZLyiOj/6tZStRBGtGgN6ceN6cMH8z7etPGlw==" data-cf-beacon='{"rayId":"6dbfa65c48d66946","token":"7a17a4f473fb4f15a21c9ed7ec04e38d","version":"2021.12.0","si":100}' crossorigin="anonymous"></script>
</body>
</html>". Retrying still for 4 times.
2022-02-11 19:30:44,529 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:31:44,642 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:32:44,681 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:33:44,720 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:34:44,760 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:35:46,078 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:36:46,118 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:37:46,158 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:38:46,197 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:39:46,235 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:40:46,821 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:41:46,859 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:42:46,899 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:43:46,937 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:44:25,703 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 5 times.
2022-02-11 19:44:26,362 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 4 times.
2022-02-11 19:44:27,006 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 3 times.
2022-02-11 19:44:27,637 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 2 times.
2022-02-11 19:44:28,264 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 1 times.
2022-02-11 19:44:28,866 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Giving up.
2022-02-11 19:44:28,866 - freqtrade.freqtradebot - WARNING - Unable to sell trade SOL/USD: Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}
  ```

Screenshot from telegram while the trade was already closed in exchange at 3% stoploss. Freqtrade is reporting the trade still open and the slot still in use.

![immagine](https://user-images.githubusercontent.com/20667441/153664976-138b2c3a-31a1-4334-a61d-179074ba51e9.png)

Screenshot of correct order execution at the exchange
![immagine](https://user-images.githubusercontent.com/20667441/153666525-fdce9336-d6cd-4987-be8f-b26d2c6c6bdf.png)




Discussion:
> `2022-02-11 19:44:25,703 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 5 times.
`

The exchange returns a 404 - which means it doesn't find the order - so freqtrade has to assume the order has "disappeared".

binance had a similar problem a while ago - where the order would show up again in the API a few minutes later during "busy" times - but we can't "wait" for that to happen, really - as that would either mean that the trade-slot is blocked until then - or that we'd really wait for that.

```
2022-02-11 19:30:41,639 - freqtrade.exchange.common - WARNING - fetch_ticker() returned exception: "Could not load ticker due to ExchangeNotAvailable. Message: ftx GET https://ftx.com/api/markets/SOL/USD 500 Internal Server Error <html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
<script defer src="https://static.cloudflareinsights.com/beacon.min.js/v652eace1692a40cfa3763df669d7439c1639079717194" integrity="sha512-Gi7xpJR8tSkrpF7aordPZQlW2DLtzUlZcumS8dMQjwDHEnw9I7ZLyiOj/6tZStRBGtGgN6ceN6cMH8z7etPGlw==" data-cf-beacon='{"rayId":"6dbfa65c48d66946","token":"7a17a4f473fb4f15a21c9ed7ec04e38d","version":"2021.12.0","si":100}' crossorigin="anonymous"></script>
</body>
</html>". Retrying still for 4 times.
```

I fear a bit that it's been a (temporary) exchange problem. A few minutes before the stop error cloudflare threw a 500 error (a sign of backend problems).

that "404" error is happening on a request that's not order related (it's the conditionals endpoint for this pair).
if it'd be a general problem - it'd happen all the time (for every SOL/USD order at the very least).
As that was showing up only once (it's actually called once per iteration) - it's not a generic problem.

there's also no dynamic in the call apart from the pair.
if you'd still have the trade, you could get the affected order via `show-trades`.

Freqtrade does have code which should refind the order (the order must be in the database if freqtrade opened it) - so `/forcesell` would recheck all orders trying to find one that did fill - but from a quick look, it seems like that won't work for stop orders (which i'm now obviously looking to fix).



Resolving patch:
diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py
--- a/freqtrade/freqtradebot.py
+++ b/freqtrade/freqtradebot.py
@@ -298,28 +298,6 @@ def update_closed_trades_without_assigned_fees(self):
                     self.update_trade_state(trade, order.order_id, send_msg=False)
 
     def handle_insufficient_funds(self, trade: Trade):
-        """
-        Determine if we ever opened a sell order for this trade.
-        If not, try update buy fees - otherwise "refind" the open order we obviously lost.
-        """
-        sell_order = trade.select_order('sell', None)
-        if sell_order:
-            self.refind_lost_order(trade)
-        else:
-            self.reupdate_enter_order_fees(trade)
-
-    def reupdate_enter_order_fees(self, trade: Trade):
-        """
-        Get buy order from database, and try to reupdate.
-        Handles trades where the initial fee-update did not work.
-        """
-        logger.info(f"Trying to reupdate buy fees for {trade}")
-        order = trade.select_order('buy', False)
-        if order:
-            logger.info(f"Updating buy-fee on trade {trade} for order {order.order_id}.")
-            self.update_trade_state(trade, order.order_id, send_msg=False)
-
-    def refind_lost_order(self, trade):
         """
         Try refinding a lost trade.
         Only used when InsufficientFunds appears on sell orders (stoploss or sell).
@@ -332,9 +310,6 @@ def refind_lost_order(self, trade):
             if not order.ft_is_open:
                 logger.debug(f"Order {order} is no longer open.")
                 continue
-            if order.ft_order_side == 'buy':
-                # Skip buy side - this is handled by reupdate_buy_order_fees
-                continue
             try:
                 fo = self.exchange.fetch_order_or_stoploss_order(order.order_id, order.ft_pair,
                                                                  order.ft_order_side == 'stoploss')
@@ -346,6 +321,9 @@ def refind_lost_order(self, trade):
                     if fo and fo['status'] == 'open':
                         # Assume this as the open order
                         trade.open_order_id = order.order_id
+                elif order.ft_order_side == 'buy':
+                    if fo and fo['status'] == 'open':
+                        trade.open_order_id = order.order_id
                 if fo:
                     logger.info(f"Found {order} for trade {trade}.")
                     self.update_trade_state(trade, order.order_id, fo,


Test patch:
diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@@ -4110,15 +4110,17 @@ def test_reupdate_enter_order_fees(mocker, default_conf_usdt, fee, caplog):
     freqtrade = get_patched_freqtradebot(mocker, default_conf_usdt)
     mock_uts = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.update_trade_state')
 
+    mocker.patch('freqtrade.exchange.Exchange.fetch_order_or_stoploss_order',
+                 return_value={'status': 'open'})
     create_mock_trades(fee)
     trades = Trade.get_trades().all()
 
-    freqtrade.reupdate_enter_order_fees(trades[0])
-    assert log_has_re(r"Trying to reupdate buy fees for .*", caplog)
+    freqtrade.handle_insufficient_funds(trades[3])
+    # assert log_has_re(r"Trying to reupdate buy fees for .*", caplog)
     assert mock_uts.call_count == 1
-    assert mock_uts.call_args_list[0][0][0] == trades[0]
-    assert mock_uts.call_args_list[0][0][1] == mock_order_1()['id']
-    assert log_has_re(r"Updating buy-fee on trade .* for order .*\.", caplog)
+    assert mock_uts.call_args_list[0][0][0] == trades[3]
+    assert mock_uts.call_args_list[0][0][1] == mock_order_4()['id']
+    assert log_has_re(r"Trying to refind lost order for .*", caplog)
     mock_uts.reset_mock()
     caplog.clear()
 
@@ -4136,52 +4138,13 @@ def test_reupdate_enter_order_fees(mocker, default_conf_usdt, fee, caplog):
     )
     Trade.query.session.add(trade)
 
-    freqtrade.reupdate_enter_order_fees(trade)
-    assert log_has_re(r"Trying to reupdate buy fees for .*", caplog)
+    freqtrade.handle_insufficient_funds(trade)
+    # assert log_has_re(r"Trying to reupdate buy fees for .*", caplog)
     assert mock_uts.call_count == 0
-    assert not log_has_re(r"Updating buy-fee on trade .* for order .*\.", caplog)
 
 
 @pytest.mark.usefixtures("init_persistence")
-def test_handle_insufficient_funds(mocker, default_conf_usdt, fee):
-    freqtrade = get_patched_freqtradebot(mocker, default_conf_usdt)
-    mock_rlo = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.refind_lost_order')
-    mock_bof = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.reupdate_enter_order_fees')
-    create_mock_trades(fee)
-    trades = Trade.get_trades().all()
-
-    # Trade 0 has only a open buy order, no closed order
-    freqtrade.handle_insufficient_funds(trades[0])
-    assert mock_rlo.call_count == 0
-    assert mock_bof.call_count == 1
-
-    mock_rlo.reset_mock()
-    mock_bof.reset_mock()
-
-    # Trade 1 has closed buy and sell orders
-    freqtrade.handle_insufficient_funds(trades[1])
-    assert mock_rlo.call_count == 1
-    assert mock_bof.call_count == 0
-
-    mock_rlo.reset_mock()
-    mock_bof.reset_mock()
-
-    # Trade 2 has closed buy and sell orders
-    freqtrade.handle_insufficient_funds(trades[2])
-    assert mock_rlo.call_count == 1
-    assert mock_bof.call_count == 0
-
-    mock_rlo.reset_mock()
-    mock_bof.reset_mock()
-
-    # Trade 3 has an opne buy order
-    freqtrade.handle_insufficient_funds(trades[3])
-    assert mock_rlo.call_count == 0
-    assert mock_bof.call_count == 1
-
-
-@pytest.mark.usefixtures("init_persistence")
-def test_refind_lost_order(mocker, default_conf_usdt, fee, caplog):
+def test_handle_insufficient_funds(mocker, default_conf_usdt, fee, caplog):
     caplog.set_level(logging.DEBUG)
     freqtrade = get_patched_freqtradebot(mocker, default_conf_usdt)
     mock_uts = mocker.patch('freqtrade.freqtradebot.FreqtradeBot.update_trade_state')
@@ -4204,7 +4167,7 @@ def reset_open_orders(trade):
     assert trade.open_order_id is None
     assert trade.stoploss_order_id is None
 
-    freqtrade.refind_lost_order(trade)
+    freqtrade.handle_insufficient_funds(trade)
     order = mock_order_1()
     assert log_has_re(r"Order Order(.*order_id=" + order['id'] + ".*) is no longer open.", caplog)
     assert mock_fo.call_count == 0
@@ -4222,13 +4185,13 @@ def reset_open_orders(trade):
     assert trade.open_order_id is None
     assert trade.stoploss_order_id is None
 
-    freqtrade.refind_lost_order(trade)
+    freqtrade.handle_insufficient_funds(trade)
     order = mock_order_4()
     assert log_has_re(r"Trying to refind Order\(.*", caplog)
-    assert mock_fo.call_count == 0
-    assert mock_uts.call_count == 0
-    # No change to orderid - as update_trade_state is mocked
-    assert trade.open_order_id is None
+    assert mock_fo.call_count == 1
+    assert mock_uts.call_count == 1
+    # Found open buy order
+    assert trade.open_order_id is not None
     assert trade.stoploss_order_id is None
 
     caplog.clear()
@@ -4240,11 +4203,11 @@ def reset_open_orders(trade):
     assert trade.open_order_id is None
     assert trade.stoploss_order_id is None
 
-    freqtrade.refind_lost_order(trade)
+    freqtrade.handle_insufficient_funds(trade)
     order = mock_order_5_stoploss()
     assert log_has_re(r"Trying to refind Order\(.*", caplog)
     assert mock_fo.call_count == 1
-    assert mock_uts.call_count == 1
+    assert mock_uts.call_count == 2
     # stoploss_order_id is "refound" and added to the trade
     assert trade.open_order_id is None
     assert trade.stoploss_order_id is not None
@@ -4259,7 +4222,7 @@ def reset_open_orders(trade):
     assert trade.open_order_id is None
     assert trade.stoploss_order_id is None
 
-    freqtrade.refind_lost_order(trade)
+    freqtrade.handle_insufficient_funds(trade)
     order = mock_order_6_sell()
     assert log_has_re(r"Trying to refind Order\(.*", caplog)
     assert mock_fo.call_count == 1
@@ -4275,7 +4238,7 @@ def reset_open_orders(trade):
                            side_effect=ExchangeError())
     order = mock_order_5_stoploss()
 
-    freqtrade.refind_lost_order(trades[4])
+    freqtrade.handle_insufficient_funds(trades[4])
     assert log_has(f"Error updating {order['id']}.", caplog)