Repository: freqtrade/freqtrade

Issue:
Stoploss on exchange correctly executed but not cleared in Freqtrade
## Describe your environment

  * Operating system: Ubuntu 20.04
  * Python Version: 3.8.10
  * CCXT version: 1.71.73
  * Freqtrade Version: develop-06387478b

## Describe the problem:
If bot is configured to use market stoploss on exchange (FTX), the stop market order get correctly executed on exchange but Freqtrade still have the trade open not being able to sell it.

*Explain the problem you have encountered*
Running a live bot on FTX exchange, configured with market stoploss.
When the trade was opened, the stoploss market order has been correctly set on exchange.
When the trade got stopped, the order on exchange has been correctly and safely executed but Freqtrade still had the trade open and the slot busy with it.

### Steps to reproduce:
Strategy is not using `custom_stoploss` nor `confirm_trade_exit`
```
stoploss = -0.03
```
Configuration used:
```
    "order_types": {
        "buy": "limit",
        "sell": "market",
        "emergencysell": "market",
        "forcesell": "market",
        "stoploss": "market",
        "stoploss_on_exchange": true,
        "stoploss_on_exchange_interval": 60
    },
```
  
### Observed Results:

  * What happened?  
  Trade correctly closed in stoploss at the exchange
  * What did you expect to happen?  
  Freqtrade selling the trade on database and clearing the slot

### Relevant code exceptions or logs
```
2022-02-11 09:18:00,325 - freqtrade.exchange.ftx - INFO - stoploss order added for SOL/USD. stop price: 102.8325.
```
  ```
2022-02-11 19:28:43,057 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:29:43,094 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:30:24,608 - VolumePairList - INFO - Searching 500 pairs: ['BTC/USD', 'ETH/USD', 'SOL/USD', 'AVAX/USD', 'BNB/USD', 'XRP/USD', 'FTT/USD', 'MATIC/USD', 'SLP/USD', 'LTC/USD', 'LOOKS/USD', 'LINK/USD', 'FTM/USD', 'SUSHI/USD', 'DOGE/USD', 'RAY/USD', 'SHIB/USD', 'DOT/USD', 'SAND/USD', 'CRO/USD', 'BCH/USD', 'SRM/USD', 'MANA/USD', 'CRV/USD', 'AXS/USD', 'RUNE/USD', 'SPELL/USD', 'DYDX/USD', 'WBTC/USD', 'ATLAS/USD', 'UNI/USD', 'OMG/USD', 'BIT/USD', 'KNC/USD', 'TONCOIN/USD', 'GALA/USD', 'AAVE/USD', 'STEP/USD', 'RSR/USD', 'POLIS/USD', 'TRX/USD', 'YFI/USD', 'SXP/USD', 'SKL/USD', 'PERP/USD', 'CLV/USD', 'BOBA/USD', 'HNT/USD', 'RNDR/USD', 'UST/USD', 'KSHIB/USD']
2022-02-11 19:30:41,639 - freqtrade.exchange.common - WARNING - fetch_ticker() returned exception: "Could not load ticker due to ExchangeNotAvailable. Message: ftx GET https://ftx.com/api/markets/SOL/USD 500 Internal Server Error <html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
<script defer src="https://static.cloudflareinsights.com/beacon.min.js/v652eace1692a40cfa3763df669d7439c1639079717194" integrity="sha512-Gi7xpJR8tSkrpF7aordPZQlW2DLtzUlZcumS8dMQjwDHEnw9I7ZLyiOj/6tZStRBGtGgN6ceN6cMH8z7etPGlw==" data-cf-beacon='{"rayId":"6dbfa65c48d66946","token":"7a17a4f473fb4f15a21c9ed7ec04e38d","version":"2021.12.0","si":100}' crossorigin="anonymous"></script>
</body>
</html>". Retrying still for 4 times.
2022-02-11 19:30:44,529 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:31:44,642 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:32:44,681 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:33:44,720 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:34:44,760 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:35:46,078 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:36:46,118 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:37:46,158 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:38:46,197 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:39:46,235 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:40:46,821 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:41:46,859 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:42:46,899 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:43:46,937 - freqtrade.worker - INFO - Bot heartbeat. PID=28934, version='develop-06387478b, strategy_version: 0.3.6', state='RUNNING'
2022-02-11 19:44:25,703 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 5 times.
2022-02-11 19:44:26,362 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 4 times.
2022-02-11 19:44:27,006 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 3 times.
2022-02-11 19:44:27,637 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 2 times.
2022-02-11 19:44:28,264 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 1 times.
2022-02-11 19:44:28,866 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Giving up.
2022-02-11 19:44:28,866 - freqtrade.freqtradebot - WARNING - Unable to sell trade SOL/USD: Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}
  ```

Screenshot from telegram while the trade was already closed in exchange at 3% stoploss. Freqtrade is reporting the trade still open and the slot still in use.

![immagine](https://user-images.githubusercontent.com/20667441/153664976-138b2c3a-31a1-4334-a61d-179074ba51e9.png)

Screenshot of correct order execution at the exchange
![immagine](https://user-images.githubusercontent.com/20667441/153666525-fdce9336-d6cd-4987-be8f-b26d2c6c6bdf.png)




Discussion:
> `2022-02-11 19:44:25,703 - freqtrade.exchange.common - WARNING - fetch_stoploss_order() returned exception: "Could not get order due to BadRequest. Message: ftx {"error":"404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.","success":false}". Retrying still for 5 times.
`

The exchange returns a 404 - which means it doesn't find the order - so freqtrade has to assume the order has "disappeared".

binance had a similar problem a while ago - where the order would show up again in the API a few minutes later during "busy" times - but we can't "wait" for that to happen, really - as that would either mean that the trade-slot is blocked until then - or that we'd really wait for that.

```
2022-02-11 19:30:41,639 - freqtrade.exchange.common - WARNING - fetch_ticker() returned exception: "Could not load ticker due to ExchangeNotAvailable. Message: ftx GET https://ftx.com/api/markets/SOL/USD 500 Internal Server Error <html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
<script defer src="https://static.cloudflareinsights.com/beacon.min.js/v652eace1692a40cfa3763df669d7439c1639079717194" integrity="sha512-Gi7xpJR8tSkrpF7aordPZQlW2DLtzUlZcumS8dMQjwDHEnw9I7ZLyiOj/6tZStRBGtGgN6ceN6cMH8z7etPGlw==" data-cf-beacon='{"rayId":"6dbfa65c48d66946","token":"7a17a4f473fb4f15a21c9ed7ec04e38d","version":"2021.12.0","si":100}' crossorigin="anonymous"></script>
</body>
</html>". Retrying still for 4 times.
```

I fear a bit that it's been a (temporary) exchange problem. A few minutes before the stop error cloudflare threw a 500 error (a sign of backend problems).

that "404" error is happening on a request that's not order related (it's the conditionals endpoint for this pair).
if it'd be a general problem - it'd happen all the time (for every SOL/USD order at the very least).
As that was showing up only once (it's actually called once per iteration) - it's not a generic problem.

there's also no dynamic in the call apart from the pair.
if you'd still have the trade, you could get the affected order via `show-trades`.

Freqtrade does have code which should refind the order (the order must be in the database if freqtrade opened it) - so `/forcesell` would recheck all orders trying to find one that did fill - but from a quick look, it seems like that won't work for stop orders (which i'm now obviously looking to fix).


#6396 will not directly fix the problem (we can't really fix the exchange not working properly) - however it'll trigger fixing the problematic trade with a `/forcesell <id>` - as it'll determine "not enough funds" - which triggers the "Refind order" code.



Resolving patch:
diff --git a/freqtrade/exchange/ftx.py b/freqtrade/exchange/ftx.py
--- a/freqtrade/exchange/ftx.py
+++ b/freqtrade/exchange/ftx.py
@@ -106,15 +106,18 @@ def fetch_stoploss_order(self, order_id: str, pair: str) -> Dict:
                 if order[0].get('status') == 'closed':
                     # Trigger order was triggered ...
                     real_order_id = order[0].get('info', {}).get('orderId')
+                    # OrderId may be None for stoploss-market orders
+                    # But contains "average" in these cases.
+                    if real_order_id:
+                        order1 = self._api.fetch_order(real_order_id, pair)
+                        self._log_exchange_response('fetch_stoploss_order1', order1)
+                        # Fake type to stop - as this was really a stop order.
+                        order1['id_stop'] = order1['id']
+                        order1['id'] = order_id
+                        order1['type'] = 'stop'
+                        order1['status_stop'] = 'triggered'
+                        return order1
 
-                    order1 = self._api.fetch_order(real_order_id, pair)
-                    self._log_exchange_response('fetch_stoploss_order1', order1)
-                    # Fake type to stop - as this was really a stop order.
-                    order1['id_stop'] = order1['id']
-                    order1['id'] = order_id
-                    order1['type'] = 'stop'
-                    order1['status_stop'] = 'triggered'
-                    return order1
                 return order[0]
             else:
                 raise InvalidOrderException(f"Could not get stoploss order for id {order_id}")


Test patch:
diff --git a/tests/exchange/test_ftx.py b/tests/exchange/test_ftx.py
--- a/tests/exchange/test_ftx.py
+++ b/tests/exchange/test_ftx.py
@@ -125,7 +125,7 @@ def test_stoploss_adjust_ftx(mocker, default_conf):
     assert not exchange.stoploss_adjust(1501, order)
 
 
-def test_fetch_stoploss_order(default_conf, mocker, limit_sell_order):
+def test_fetch_stoploss_order_ftx(default_conf, mocker, limit_sell_order):
     default_conf['dry_run'] = True
     order = MagicMock()
     order.myid = 123
@@ -147,9 +147,15 @@ def test_fetch_stoploss_order(default_conf, mocker, limit_sell_order):
     with pytest.raises(InvalidOrderException, match=r"Could not get stoploss order for id X"):
         exchange.fetch_stoploss_order('X', 'TKN/BTC')['status']
 
-    api_mock.fetch_orders = MagicMock(return_value=[{'id': 'X', 'status': 'closed'}])
+    # stoploss Limit order
+    api_mock.fetch_orders = MagicMock(return_value=[
+        {'id': 'X', 'status': 'closed',
+         'info': {
+             'orderId': 'mocked_limit_sell',
+         }}])
     api_mock.fetch_order = MagicMock(return_value=limit_sell_order)
 
+    # No orderId field - no call to fetch_order
     resp = exchange.fetch_stoploss_order('X', 'TKN/BTC')
     assert resp
     assert api_mock.fetch_order.call_count == 1
@@ -158,6 +164,17 @@ def test_fetch_stoploss_order(default_conf, mocker, limit_sell_order):
     assert resp['type'] == 'stop'
     assert resp['status_stop'] == 'triggered'
 
+    # Stoploss market order
+    # Contains no new Order, but "average" instead
+    order = {'id': 'X', 'status': 'closed', 'info': {'orderId': None}, 'average': 0.254}
+    api_mock.fetch_orders = MagicMock(return_value=[order])
+    api_mock.fetch_order.reset_mock()
+    resp = exchange.fetch_stoploss_order('X', 'TKN/BTC')
+    assert resp
+    # fetch_order not called (no regular order ID)
+    assert api_mock.fetch_order.call_count == 0
+    assert order == order
+
     with pytest.raises(InvalidOrderException):
         api_mock.fetch_orders = MagicMock(side_effect=ccxt.InvalidOrder("Order not found"))
         exchange = get_patched_exchange(mocker, default_conf, api_mock, id='ftx')