Repository: freqtrade/freqtrade

Issue:
Error on start Bittrex
  * Operating system: Windows10 Ubunutu WSL
  * Python Version: latest
  * CCXT version: latest
  * Freqtrade latest and all others before
  
## Describe the problem:

Tons of error at start

### Steps to reproduce:

  1. Starting/restarting manually
  
### Observed Results:

    Each time I restart freqtrade, it try to search for orders history on bittrex, it happen since a long time but now I've a lot of trades and testing 5 times for each take a long time  
  Lucky me I've just ~30 trades, this take 30s before all  is working well

### Relevant code exceptions or logs

  ```
2020-12-23 09:17:59,604 - freqtrade.exchange.common - WARNING - fetch_order() returned exception: "Could not get order due to ExchangeNotAvailable. Message: bittrex GET https://api.bittrex.com/v3/orders/f93ad21b-2d4c-426f-86ad-ad566cc78446 404 Not Found {"code":"NOT_FOUND"}"
2020-12-23 09:17:59,608 - freqtrade.exchange.common - WARNING - Giving up retrying: fetch_order()
2020-12-23 09:17:59,608 - freqtrade.freqtradebot - WARNING - Error updating Order f93ad21b-2d4c-426f-86ad-ad566cc78446 due to Could not get order due to ExchangeNotAvailable. Message: bittrex GET https://api.bittrex.com/v3/orders/f93ad21b-2d4c-426f-86ad-ad566cc78446 404 Not Found {"code":"NOT_FOUND"}
2020-12-23 09:18:01,116 - freqtrade.exchange.common - WARNING - fetch_order() returned exception: "Could not get order due to ExchangeNotAvailable. Message: bittrex GET https://api.bittrex.com/v3/orders/d16132ae-3841-45fd-bd20-583560e95db8 404 Not Found {"code":"NOT_FOUND"}"
 etc...
  ```



Discussion:
well it looks like your database contains some orders (like `f93ad21b-2d4c-426f-86ad-ad566cc78446`) which don't exist on the exchange.

this can have multiple reasons - for example:
* you switched exchange - so the orders are from another exchange like binance
* you've changed account (so the orders are not for the account the API keys are for)
* your api-keys somehow restrict access to these orders
...

It's been a long time i used bittrex - but maybe your exchange keys only allow "looking back" for a certain amount of time (i know kraken has such a feature, not sure about bittrex)?

best also check within the database if this happens for each order or just for a few - probably older ones (orders table).
Also check the status of these orders in the order table (are they filled / closed? cancelled? ...)

Based on the [Bittrex API documentation](https://bittrex.github.io/api/v3#operation--orders--orderId--get), the query seems to be correct, so it might be a bug in the bittrex API, where it's not returning such orders.

Maybe also check if this is happening for new orders as well (If the bot buys, does the next startup include the order for this trade)?
If yes - then we'll need the logs from the time when this order was placed (and filled).

No changes on account/exchange/api-keys
f93ad21b-2d4c-426f-86ad-ad566cc78446 for example is marked open in the sqlite database
So I presume the others are marked open too (I've 41 trades marked open on 155)

Why not automaticaly marked them closed if they are not found in the exchange ?
As the error message is wrong, because exchange is know as available and not : 
````WARNING - fetch_order() returned exception: "Could not get order due to ExchangeNotAvailable.````

The 41 false open orders are making 5 calls on each every 1s because the market is considered as unreachable, wich is wrong
why would we automatically assume them as closed? 
they could just as well be deleted or still be open on the exchange.

If it's 41 out of 155 - are these the most recent ones, or older ones?

If you check one of the ID's on the exchange, what's the status of that order there?
Also, what's the column "ft_order_side" for these orders? are they all identical or different?

If it's older ones, best close them in the database manually and monitor if it happens again (i assume it won't ... but i can also not explain why these few would not be found anymore).

> WARNING - fetch_order() returned exception: "Could not get order due to ExchangeNotAvailable.

feel free to report this to CCXT, we're getting ExchangeNotAvailable from there.
Ok, Thank you.
Wich column I've to modify in orders table as workaround? ft_is_open or status ?

Ok I've set all ft_is_open to 0, and it start normaly
Please let me know if this happens with new orders again - i think it's just a temporary issue - but it's difficult to tell without further input
Happen again, ~10 trades marked 1 in ft_is_open in database.
I know they all closed

Manually set ft_is_open to 0 resolve the restart
Please logs from when this order was closed, otherwise there's no point in leaving this open.
Is this can be related to the fact I stop freqtrade with Ctrl-C in ubuntu console ?
Even if closing things seems to happen normaly..

Is there a debug log more verbose ?
Thank you
no, the only way to stop freqtrade if it's running in a console is via ctrl+c.

in general, you could run the bot with `-v` - which enables debug logging. please note that this mode is quite verbose, and for this particular problem, will not provide any additional insights.

The logs i'm looking for (logs from around the time when the order in question was closed) will NOT require debug logging (which will even clutter the information), as changes to orders are logged at the default level to provide all necessary information.
2 orders with the same id have `ft_is_open` to 1

```
id	ft_trade_id	ft_order_side	ft_pair	ft_is_open	order_id	status	symbol	order_type	side	price	amount	filled	remaining	cost	order_date	order_filled_date	order_update_date
274	88	sell	USD/EUR	0	a6662c00-debd-4537-bda1-d643461a5b43	closed	USD/EUR	limit	sell	0.8099	15.56430421	15.56430421	0	12.60552997	2021-01-10 09:12:31.610000	2021-01-10 09:16:30.991283	2021-01-10 09:16:30.991349
273	88	sell	USD/EUR	1	4c0b31a9-e501-4318-9e63-0ca8798d7076	open	USD/EUR	limit	sell	0.8128	15.56430421	0	15.56430421	0	2021-01-10 08:27:24.090000		2021-01-10 09:12:24.725714
272	88	sell	USD/EUR	1	4ab355d5-2b03-4edc-9a69-b7f6929ff4ff	open	USD/EUR	limit	sell	0.813	15.56430421	0	15.56430421	0	2021-01-10 07:42:14.200000		2021-01-10 08:27:15.712748
271	88	buy	USD/EUR	0	64228bb7-f66d-4d86-b177-695287118936	closed	USD/EUR	limit	buy	0.8038	15.56430421	15.56430421	0	12.51058772	2021-01-10 04:10:10.080000	2021-01-10 04:12:24.574861	2021-01-10 04:12:24.574894```
```
```
2021-01-10 08:42:11,815 - freqtrade.freqtradebot - INFO - Executing Sell for USD/EUR. Reason: roi
2021-01-10 08:42:13,351 - freqtrade.wallets - INFO - Wallets synced.
2021-01-10 08:42:14,944 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': sell, 'trade_id': 88, 'exchange': 'Bittrex', 'pair': 'USD/EUR', 'gain': 'profit', 'limit': 0.813, 'order_type': 'limit', 'amount': 15.56430421, 'open_rate': 0.8037999997431302, 'current_rate': 0.813, 'profit_amount': 0.09286287, 'profit_ratio': 0.00740793, 'sell_reason': 'roi', 'open_date': datetime.datetime(2021, 1, 10, 4, 10, 10, 687216), 'close_date': datetime.datetime(2021, 1, 10, 7, 42, 14, 944125), 'stake_currency': 'EUR', 'fiat_currency': 'USD'}
2021-01-10 08:42:14,944 - freqtrade.rpc.fiat_convert - WARNING - unsupported crypto-symbol eur - returning 0.0
2021-01-10 08:42:16,307 - freqtrade.wallets - INFO - Wallets synced.
```

```
2021-01-10 09:27:17,199 - freqtrade.freqtradebot - INFO - Sell order cancelled due to timeout for Trade(id=88, pair=USD/EUR, amount=15.56430421, open_rate=0.80380000, open_since=2021-01-10 04:10:10).
2021-01-10 09:27:18,693 - freqtrade.wallets - INFO - Wallets synced.
2021-01-10 09:27:20,275 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': sell_cancel, 'trade_id': 88, 'exchange': 'Bittrex', 'pair': 'USD/EUR', 'gain': 'loss', 'limit': None, 'order_type': 'limit', 'amount': 15.56430421, 'open_rate': 0.8037999997431302, 'current_rate': 0.8128, 'profit_amount': -12.5356089, 'profit_ratio': -1.0, 'sell_reason': 'roi', 'open_date': datetime.datetime(2021, 1, 10, 4, 10, 10, 687216), 'close_date': None, 'stake_currency': 'EUR', 'fiat_currency': 'USD', 'reason': 'cancelled due to timeout'}
2021-01-10 09:27:21,714 - freqtrade.freqtradebot - INFO - Executing Sell for USD/EUR. Reason: roi
2021-01-10 09:27:23,203 - freqtrade.wallets - INFO - Wallets synced.
2021-01-10 09:27:24,739 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': sell, 'trade_id': 88, 'exchange': 'Bittrex', 'pair': 'USD/EUR', 'gain': 'profit', 'limit': 0.8128, 'order_type': 'limit', 'amount': 15.56430421, 'open_rate': 0.8037999997431302, 'current_rate': 0.8128, 'profit_amount': 0.08975623, 'profit_ratio': 0.0071601, 'sell_reason': 'roi', 'open_date': datetime.datetime(2021, 1, 10, 4, 10, 10, 687216), 'close_date': datetime.datetime(2021, 1, 10, 8, 27, 24, 739959), 'stake_currency': 'EUR', 'fiat_currency': 'USD'}
2021-01-10 09:27:26,212 - freqtrade.wallets - INFO - Wallets synced.

```
what does the UI say about these orders?
What do you mean by 'UI' ?
Bittrex webpage / UI.

exchanges will usually show you historic orders (at least binance, ftx and kraken do this) - and i seem to remember that bittrex did do this in the past too, but i didn't do any trading with bittrex in the last 2.5 years due to their bad volume (after you go down the list of top pairs by 5 or 10).
Nothing wrong : 

```
Closed Date:2021/01/10 10:16:28
Opened Date:2021/01/10 10:12:31
Market:EUR-USD
Side:SELL
Type:Limit
Price:€0.80990000
Quantity Filled:15.56430421 USD
Total Quantity:15.56430421 USD
Actual Rate:€0.80989999
Commission:€0.02521105
Total:€12.58031892

Closed Date:2021/01/10 05:12:22
Opened Date:2021/01/10 05:10:10
Market:EUR-USD
Side:BUY
Type:Limit
Price:€0.80380000
Quantity Filled:15.56430421 USD
Total Quantity:15.56430421 USD
Actual Rate:€0.80379999
Commission:€0.02502117
Total:€12.53560889
```
These are the filled orders (first and last in your above list).

What about the 2 orders you posted above "in the middle" - namely the following two.

While i don't know for sure for bittrex, you should be able to see them under "orders" (or something similar).

```
273	88	sell	USD/EUR	1	4c0b31a9-e501-4318-9e63-0ca8798d7076	open	USD/EUR	limit	sell	0.8128	15.56430421	0	15.56430421	0	2021-01-10 08:27:24.090000		2021-01-10 09:12:24.725714
272	88	sell	USD/EUR	1	4ab355d5-2b03-4edc-9a69-b7f6929ff4ff	open	USD/EUR	limit	sell	0.813	15.56430421	0	15.56430421	0	2021-01-10 07:42:14.200000		2021-01-10 08:27:15.712748

```
As you see before, it's the same trade and doesnt appear in bittrex trade history 
they just appear in freqtrade log
the first  was canceled due to time out
the second is the resell 
As mentioned above, bittrex should have a "orders" log - which will list all orders (open, closed, cancelled).

Without this information from bittrex, there is nothing i can do for this case as it's unclear what happened to these orders on the exchange.
Bittrex havent cancelled orders logs
Otherwise i’ll tell you.

Seems freqtrade doesnt clear an sql field when trades are cancelled
But up to you, I dont think that was an ccxt error
(even if they dont give the good error code)

De : Matthias
Envoyé le :lundi 11 janvier 2021 09:08
À : freqtrade/freqtrade
Cc : Priveyes; State change
Objet :Re: [freqtrade/freqtrade] Error on start Bittrex (#4128)

As mentioned above, bittrex should have a "orders" log - which will list all orders (open, closed, cancelled).
Without this information from bittrex, there is nothing i can do for this case as it's unclear what happened to these orders on the exchange.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or unsubscribe.


> Seems freqtrade doesnt clear an sql field when trades are cancelled

Freqtrade is not supposed to clear or close this order without confirmation from the exchange. 

It's an Order log - keeping track of all orders the bot created.
They are closed only when we have confirmation of a order change (like close or cancel) - as cancelling orders can go wrong.

If bittrex does simply "delete" canceled orders and "forgets" they ever existed, then this is a problem, but one you should take up with bittrex support.
in my opinion, their API has to return all orders that you created, either with the status Open (not executed), closed / filled (executed), partially filled - and also cancelled.

---

Considering how many problems we've had with bittrex lately (80% of all exchange-specific issues have been with bittrex in the last few months), i'll have to think about this very carefully - but the most likely course of action will be to remove bittrex from the list of officially supported exchanges - as it's taking too much time to maintain their unreliable API quirks, and their volume  does not really justify the time put into it.


The car, or the wheel ?
Something is wrong with this case, FQ delete the order but keep a track of that and loose his mind
I agree, this is difficult (I remove all of that manualy, because i’ve only 2-6 trades by days)
I dont want to look closer on the code, I hate python.
But love you work.


De : Matthias
Envoyé le :lundi 11 janvier 2021 11:18
À : freqtrade/freqtrade
Cc : Priveyes; State change
Objet :Re: [freqtrade/freqtrade] Error on start Bittrex (#4128)

Seems freqtrade doesnt clear an sql field when trades are cancelled
Freqtrade is not supposed to clear or close this order without confirmation from the exchange.
It's an Order log - keeping track of all orders the bot created.
They are closed only when we have confirmation of a order change (like close or cancel) - as cancelling orders can go wrong.
If bittrex does simply "delete" canceled orders and "forgets" they ever existed, then this is a problem, but one you should take up with bittrex support.
in my opinion, their API has to return all orders that you created, either with the status Open (not executed), closed / filled (executed), partially filled - and also cancelled.

Considering how many problems we've had with bittrex lately, i'll have to think about this - but the most likely course of action will be to remove bittrex from the list of officially supported exchanges - as it's taking too much time to maintain their unreliable API quirks.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or unsubscribe.


I'll do it by myself
> Something is wrong with this case, FQ delete the order but keep a track of that and loose his mind

Your **assumption** is that freqtrade removes the order from the exchange. 
As said above - the exchange does not CONFIRM that the deletion was successful (the order could fill right in this moment).
Simply assuming that it's successful will cause other problems, as freqtrade can then end up with a filled order ft thinks was cancelled - but it really wasn't.

If you find out what the correct behavior for bittrex should be and you're able to fix it, best open open a pull request fixing this for everyone else too.
Might also help identify problems in your solution - assuming from the above that you are not used to code in python.


As said, i no longer use Bittrex - so supporting it is difficult as i have no way to test this behavour with orders at the moment (no money there, and honestly, i also have no interest in having money there, and no time to dig deep into this topic either).



Ok. 
This is not a problem of bittrex, or ccxt.
I can reproduce it each time, and on each 'cancel' then 'sell/buy' .
This concern a problem of a simple variable wih isnt updated somewhere.
I can search for it, but you better do it really faster than me, as long as you want it.

Please run the following code with your api keys (don't post them here) and show me the output.

``` python
import ccxt
ct = ccxt.bittrex({'apiKey':<key>, 'secret': '<secret>' })
# This is the order-id you posted above
ct.fetch_order('4c0b31a9-e501-4318-9e63-0ca8798d7076', 'USD/EUR')
```

If bittrex does not return anything, then this is an issue - as said above, we can't blindly trust the exchange to cancel an order - the order might've been filled already.

in that case, best open an issue with bittrex requesting that they also report canceled orders via their GET orders endpoint.


Maybe they provide the needed info in the "cancel_order" call - but for that, i'd need the logs from the time when the order was cancelled (basically this should be a few seconds before the next order was placed).


Waiting from an answer of bittrex, btw, they doesnt keep history of canceled orders.
So I'm pretty sure that if the fecth_order id doesnt exist, ie error 404 by ccxt, then this id was canceled, nothing else.
Comming back to you with further answer.
Interrestingly, this doesn't seem to be fully correct - as binance seems to keep historic cancelled orders as well.

I've just tried to create an order (off-price, so it doesn't fill as i wanted to test cancelling).
No matter if i cancel the order manually (on exchange) or via the API, the order is still available via `fetch_order()` endpoint.

So to me, it seems like they keep the order history in their API (maybe not in the UI) - at least for a certain amount of time.

Even after 12+ hours of posting (and cancelling) the order, the order is still "fetchable" via the API.


----

What was the result of the "plain ccxt" test i've posted above for you? 

if you're getting `403 Client Error: Forbidden for url:` - then your API permissions are most likely wrong.
For bittrex, apparently the key needs both READ INFO, as well as TRADE to successfully fetch historic orders.


Hello, just to confirm a bad news from bittrex:
So, using ccxt with it can never give good results with the way of canceled trades are handled.

Pat Brewer (Bittrex Global) 
Feb 4, 2021, 2:36 GMT+1 
Hello, 

You are correct. We do not record cancelled orders. Partial fills will record. Please let us know if you have any questions! 
Best Regards,
Pat Brewer @ Bittrex Global

well but it's not ccxt's fault either, it's the exchange's fault.

i guess we can make the "order catchup" optional - which will mean fees will eventually be wrong if it's disabled.
I agree for exchange fault (if it’s really a fault).
But up to you. I think I’m not alone in bittrex.

De : Matthias
Envoyé le :jeudi 4 février 2021 11:54
À : freqtrade/freqtrade
Cc : Priveyes; State change
Objet :Re: [freqtrade/freqtrade] Error on start Bittrex (#4128)

well but it's not ccxt's fault either, it's the exchange's fault.
i guess we can make the "order catchup" optional - which will mean fees will eventually be wrong if it's disabled.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or unsubscribe.



Resolving patch:
diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py
--- a/freqtrade/freqtradebot.py
+++ b/freqtrade/freqtradebot.py
@@ -246,7 +246,7 @@ def update_open_orders(self):
         Updates open orders based on order list kept in the database.
         Mainly updates the state of orders - but may also close trades
         """
-        if self.config['dry_run']:
+        if self.config['dry_run'] or self.config['exchange'].get('skip_open_order_update', False):
             # Updating open orders in dry-run does not make sense and will fail.
             return
 
@@ -1070,7 +1070,9 @@ def handle_cancel_sell(self, trade: Trade, order: Dict, reason: str) -> str:
             if not self.exchange.check_order_canceled_empty(order):
                 try:
                     # if trade is not partially completed, just delete the order
-                    self.exchange.cancel_order(trade.open_order_id, trade.pair)
+                    co = self.exchange.cancel_order_with_result(trade.open_order_id, trade.pair,
+                                                                trade.amount)
+                    trade.update_order(co)
                 except InvalidOrderException:
                     logger.exception(f"Could not cancel sell order {trade.open_order_id}")
                     return 'error cancelling order'
@@ -1078,6 +1080,7 @@ def handle_cancel_sell(self, trade: Trade, order: Dict, reason: str) -> str:
             else:
                 reason = constants.CANCEL_REASON['CANCELLED_ON_EXCHANGE']
                 logger.info('Sell order %s for %s.', reason, trade)
+                trade.update_order(order)
 
             trade.close_rate = None
             trade.close_rate_requested = None


Test patch:
diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@@ -2609,7 +2609,7 @@ def test_handle_cancel_sell_cancel_exception(mocker, default_conf) -> None:
     patch_RPCManager(mocker)
     patch_exchange(mocker)
     mocker.patch(
-        'freqtrade.exchange.Exchange.cancel_order', side_effect=InvalidOrderException())
+        'freqtrade.exchange.Exchange.cancel_order_with_result', side_effect=InvalidOrderException())
 
     freqtrade = FreqtradeBot(default_conf)