Repository: freqtrade/freqtrade

Issue:
Discrepency between Freqtrade and Exchange balance after multiple Rebuys and Sells
## Describe your environment

  * Operating system: Ubuntu
  * Python Version: 3.10.6
  * CCXT version: 3.1.44
  * Freqtrade Version: docker-2023.6.dev-859f7ff3
  
I am getting discrepancies Between the FT balance and the Exchange balance after multiple rebuys or position adjustments on a token.
This is happening on OKX and Gateio.
I will show the example on okx as this is currently happening.
I have an idea that this is something to do with Fee calculations but not sure.
 
![Screenshot from 2023-06-28 10-27-36](https://github.com/freqtrade/freqtrade/assets/6954079/a900b04b-fd27-45a6-b8b9-852165c302ec)
![Screenshot from 2023-06-28 10-27-12](https://github.com/freqtrade/freqtrade/assets/6954079/853778c8-9cf8-48d1-8be6-c355c09f1f01)

This ultimately ends in the bot not being able to sell as there is less of the token on the exchange than on the bot.
I can supply the trades on the exchange and Freqtrade if required,But i am pretty sure they match,token amounts etc,So it can only be the fees where the problem lies?


Discussion:
Orders on FT
```
 (#1) 2023-06-28 00:17:10 (BST)buy for 0.02748 | 725.365957
(#2) 2023-06-28 05:12:14 (BST)sell for 0.025498822524571 | 708.652452
(#3) 2023-06-28 06:17:12 (BST)buy for 0.02517 | 197.984271
(#4) 2023-06-28 06:27:15 (BST)buy for 0.02518 | 197.905644
(#5) 2023-06-28 06:41:50 (BST)buy for 0.02523 | 197.513441
(#6) 2023-06-28 07:37:03 (BST)buy for 0.0245 | 203.398535
(#7) 2023-06-28 07:56:42 (BST)buy for 0.02458 | 202.736538
```
Orders on the exchange
![Screenshot from 2023-06-28 10-33-58](https://github.com/freqtrade/freqtrade/assets/6954079/cd46301f-9985-432d-86ed-11dcc7d6f3cb)

It has been mentioned on the NFI Discord that this could be something to do with Maker/Taker Fees?

There is literally 0 information in this issue to even start to look at.

We'll need
- logs for the affected trade (especially if there are warnings)
- screenshots of the actual, individual orders on the exchange for that trade
- output of the below command

Please provide the output of the following command for the affected trade:
`freqtrade show-trades --db-url sqlite:///<trades.sqlite> --print-json --trade-ids <tradeid>` 
Please use a "details block" to keep the issue itself kindof readable (see below for details on how to do that). 

<details>
<summary>Expand to see how to do details blocks</summary>

```
<details>
<summary>Expand full </summary>
(wrap in tripple ticks `)
Trades output ...
</details>
```
</details>



Screenshots of trade details
![Screenshot from 2023-06-28 19-50-25](https://github.com/freqtrade/freqtrade/assets/6954079/670bd600-0d28-4bf9-922f-a139614979f6)
![Screenshot from 2023-06-28 19-50-56](https://github.com/freqtrade/freqtrade/assets/6954079/be437d6c-0af2-4426-8f6d-3e26ae053fa0)
![Screenshot from 2023-06-28 19-51-22](https://github.com/freqtrade/freqtrade/assets/6954079/62d2fa45-155f-4fc3-bfba-e6f938fb0eaa)
![Screenshot from 2023-06-28 19-51-51](https://github.com/freqtrade/freqtrade/assets/6954079/b80af396-836d-42cc-be49-3c4948163ad1)
![Screenshot from 2023-06-28 19-52-47](https://github.com/freqtrade/freqtrade/assets/6954079/07069cd9-46e5-43ad-869b-a50c9873e018)
![Screenshot from 2023-06-28 19-53-11](https://github.com/freqtrade/freqtrade/assets/6954079/0b77dd57-9126-47f4-b8db-ccb488e04eee)
![Screenshot from 2023-06-28 19-53-32](https://github.com/freqtrade/freqtrade/assets/6954079/3f8ef6c1-72dc-452d-9f36-43521579c106)
Output that you require is not as easy as i am using docker-compose.yaml but i will try

Logs during the affected time
[Log Exert.txt](https://github.com/freqtrade/freqtrade/files/11899169/Log.Exert.txt)

With docker compose, just prefix the command with `docker compose exec freqtrade ..` ...
It seems it still managed to exit ok by falling back to the wallet amount
2023-06-28 21:12:19 - freqtrade.freqtradebot - INFO - Exit for BTM/USDT detected. Reason: custom_exit
2023-06-28 21:12:19 - freqtrade.wallets - INFO - Wallets synced.
2023-06-28 21:12:19 - freqtrade.freqtradebot - INFO - BTM/USDT - Falling back to wallet-amount 607.3282532324 -> 608.627056.
2023-06-28 21:12:20 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': exit, 'trade_id': 39, 'exchange': 'Okx', 'pair': 'BTM/USDT', 'leverage': 1.0, 'direction': 'Long', 'gain': 'profit', 'limit': 0.02689, 'order_rate': 0.02689, 'order_type': 'market', 'amount': 607.3282532324, 'open_rate': 0.024600012520402816, 'close_rate': 0.02689, 'current_rate': 0.02689, 'profit_amount': 1.4665316800000001, 'profit_ratio': 0.08901242, 'buy_tag': '44 ', 'enter_tag': '44 ', 'sell_reason': 'exit_quick_u_9 ( 44 )', 'exit_reason': 'exit_quick_u_9 ( 44 )', 'open_date': datetime.datetime(2023, 6, 27, 23, 17, 10, 713708, tzinfo=datetime.timezone.utc), 'close_date': datetime.datetime(2023, 6, 28, 21, 12, 20, 23639, tzinfo=datetime.timezone.utc), 'stake_amount': 14.97223320720883, 'stake_currency': 'USDT', 'base_currency': 'BTM', 'fiat_currency': 'USD', 'sub_trade': False, 'cumulative_profit': 0.13275078}
I'll still need the output from `freqtrade show-trades --db-url sqlite:///<trades.sqlite> --print-json --trade-ids <tradeid>` to look at this further (logs suggest that the reduction in amount was detected and applied correctly).

as mentioned above, with docker-compose, you can use `docker compose exec freqtrade show-trades ...` to execute this.
Hi
I ran the following in the freqtrade folder:-
sudo docker-compose exec freqtrade show-trades --db-url sqlite:///freqtrade/user_data/tradesv3_okx_usdt.sqlite --print-json --trade-ids 39
But can not see any output anywhere?
Also no errors

i think i missed one freqtrade - so it should be `docker compose exec freqtrade freqtrade show-trades` (the first is the name, the 2nd the command to execute).

if i run the above command, i get 
```
$ docker-compose exec freqtrade show-trades
OCI runtime exec failed: exec failed: unable to start container process: exec: "show-trades": executable file not found in $PATH: unknown

```
This any good?

<details>
<summary>Expand full </summary>

```
[
    {
        "trade_id": 39,
        "pair": "BTM/USDT",
        "base_currency": "BTM",
        "quote_currency": "USDT",
        "is_open": false,
        "exchange": "okx",
        "amount": 1.298803,
        "amount_requested": 725.36595737,
        "stake_amount": 0.03195058,
        "max_stake_amount": 54.79995034,
        "strategy": "NostalgiaForInfinityX3",
        "enter_tag": "44 ",
        "timeframe": 5,
        "fee_open": 0.0008,
        "fee_open_cost": 2.5560463531569284e-05,
        "fee_open_currency": "BTM",
        "fee_close": 0.0008,
        "fee_close_cost": 0.01806980310515,
        "fee_close_currency": "USDT",
        "open_date": "2023-06-27 23:17:10",
        "open_timestamp": 1687907830713,
        "open_rate": 0.024600012520402895,
        "open_rate_requested": 0.02369,
        "open_trade_value": 0.03197613,
        "close_date": "2023-06-28 21:12:24",
        "close_timestamp": 1687986744123,
        "realized_profit": 1.50457623,
        "realized_profit_ratio": 0.09174718,
        "close_rate": 0.0269,
        "close_rate_requested": 0.02689,
        "close_profit": 0.09174718,
        "close_profit_pct": 9.17,
        "close_profit_abs": 1.37182545,
        "trade_duration_s": 78913,
        "trade_duration": 1315,
        "profit_ratio": 0.09174718,
        "profit_pct": 9.17,
        "profit_abs": 1.37182545,
        "exit_reason": "exit_quick_u_9 ( 44 )",
        "exit_order_status": "closed",
        "stop_loss_abs": 0.00028,
        "stop_loss_ratio": -0.99,
        "stop_loss_pct": -99.0,
        "stoploss_order_id": null,
        "stoploss_last_update": null,
        "stoploss_last_update_timestamp": null,
        "initial_stop_loss_abs": 0.00028,
        "initial_stop_loss_ratio": -0.99,
        "initial_stop_loss_pct": -99.0,
        "min_rate": 0.02359,
        "max_rate": 0.02748,
        "leverage": 1.0,
        "interest_rate": 0.0,
        "liquidation_price": null,
        "is_short": false,
        "trading_mode": "spot",
        "funding_fees": 0.0,
        "open_order_id": null,
        "amount_precision": 1e-06,
        "price_precision": 1e-05,
        "precision_mode": 4,
        "orders": [
            {
                "amount": 725.365957,
                "safe_price": 0.02748,
                "ft_order_side": "buy",
                "order_filled_timestamp": 1687907831459,
                "ft_is_entry": true,
                "pair": "BTM/USDT",
                "order_id": "594063730913497092",
                "status": "closed",
                "average": 0.02748,
                "cost": 19.93305649836,
                "filled": 725.365957,
                "is_open": false,
                "order_date": "2023-06-27 23:17:10",
                "order_timestamp": 1687907830620,
                "order_filled_date": "2023-06-27 23:17:11",
                "order_type": "limit",
                "price": 0.02748,
                "remaining": 0.0
            },
            {
                "amount": 708.652452,
                "safe_price": 0.0254988225245709,
                "ft_order_side": "sell",
                "order_filled_timestamp": 1687925538638,
                "ft_is_entry": false,
                "pair": "BTM/USDT",
                "order_id": "594137986020077568",
                "status": "closed",
                "average": 0.02549882,
                "cost": 18.069803105149997,
                "filled": 708.652452,
                "is_open": false,
                "order_date": "2023-06-28 04:12:14",
                "order_timestamp": 1687925534417,
                "order_filled_date": "2023-06-28 04:12:18",
                "order_type": "market",
                "price": 0.0254988225245709,
                "remaining": 0.0
            },
            {
                "amount": 197.984271,
                "safe_price": 0.02517,
                "ft_order_side": "buy",
                "order_filled_timestamp": 1687929433286,
                "ft_is_entry": true,
                "pair": "BTM/USDT",
                "order_id": "594154334691454979",
                "status": "closed",
                "average": 0.02517,
                "cost": 4.98326410107,
                "filled": 197.984271,
                "is_open": false,
                "order_date": "2023-06-28 05:17:12",
                "order_timestamp": 1687929432244,
                "order_filled_date": "2023-06-28 05:17:13",
                "order_type": "limit",
                "price": 0.02517,
                "remaining": 0.0
            },
            {
                "amount": 197.905644,
                "safe_price": 0.02518,
                "ft_order_side": "buy",
                "order_filled_timestamp": 1687930040472,
                "ft_is_entry": true,
                "pair": "BTM/USDT",
                "order_id": "594156866956357638",
                "status": "closed",
                "average": 0.02518,
                "cost": 4.98326411592,
                "filled": 197.905644,
                "is_open": false,
                "order_date": "2023-06-28 05:27:15",
                "order_timestamp": 1687930035983,
                "order_filled_date": "2023-06-28 05:27:20",
                "order_type": "limit",
                "price": 0.02518,
                "remaining": 0.0
            },
            {
                "amount": 197.513441,
                "safe_price": 0.02523,
                "ft_order_side": "buy",
                "order_filled_timestamp": 1687930934911,
                "ft_is_entry": true,
                "pair": "BTM/USDT",
                "order_id": "594160536787808259",
                "status": "closed",
                "average": 0.02523,
                "cost": 4.98326411643,
                "filled": 197.513441,
                "is_open": false,
                "order_date": "2023-06-28 05:41:50",
                "order_timestamp": 1687930910939,
                "order_filled_date": "2023-06-28 05:42:14",
                "order_type": "limit",
                "price": 0.02523,
                "remaining": 0.0
            },
            {
                "amount": 203.398535,
                "safe_price": 0.0245,
                "ft_order_side": "buy",
                "order_filled_timestamp": 1687934224981,
                "ft_is_entry": true,
                "pair": "BTM/USDT",
                "order_id": "594174432735064075",
                "status": "closed",
                "average": 0.0245,
                "cost": 4.9832641075,
                "filled": 203.398535,
                "is_open": false,
                "order_date": "2023-06-28 06:37:03",
                "order_timestamp": 1687934223991,
                "order_filled_date": "2023-06-28 06:37:04",
                "order_type": "limit",
                "price": 0.0245,
                "remaining": 0.0
            },
            {
                "amount": 202.736538,
                "safe_price": 0.02458,
                "ft_order_side": "buy",
                "order_filled_timestamp": 1687935412094,
                "ft_is_entry": true,
                "pair": "BTM/USDT",
                "order_id": "594179376573771780",
                "status": "closed",
                "average": 0.02458,
                "cost": 4.98326410404,
                "filled": 202.736538,
                "is_open": false,
                "order_date": "2023-06-28 06:56:42",
                "order_timestamp": 1687935402694,
                "order_filled_date": "2023-06-28 06:56:52",
                "order_type": "limit",
                "price": 0.02458,
                "remaining": 0.0
            },
            {
                "amount": 209.998488,
                "safe_price": 0.02373,
                "ft_order_side": "buy",
                "order_filled_timestamp": 1687983170968,
                "ft_is_entry": true,
                "pair": "BTM/USDT",
                "order_id": "594379538256556038",
                "status": "closed",
                "average": 0.02373,
                "cost": 4.98326412024,
                "filled": 209.998488,
                "is_open": false,
                "order_date": "2023-06-28 20:12:04",
                "order_timestamp": 1687983124958,
                "order_filled_date": "2023-06-28 20:12:50",
                "order_type": "limit",
                "price": 0.02373,
                "remaining": 0.0
            },
            {
                "amount": 210.353065,
                "safe_price": 0.02369,
                "ft_order_side": "buy",
                "order_filled_timestamp": 1687984593867,
                "ft_is_entry": true,
                "pair": "BTM/USDT",
                "order_id": "594384504509624324",
                "status": "closed",
                "average": 0.02369,
                "cost": 4.98326410985,
                "filled": 210.353065,
                "is_open": false,
                "order_date": "2023-06-28 20:31:49",
                "order_timestamp": 1687984309005,
                "order_filled_date": "2023-06-28 20:36:33",
                "order_type": "limit",
                "price": 0.02369,
                "remaining": 0.0
            },
            {
                "amount": 210.353065,
                "safe_price": 0.0261954342150421,
                "ft_order_side": "sell",
                "order_filled_timestamp": 1687986727005,
                "ft_is_entry": false,
                "pair": "BTM/USDT",
                "order_id": "594394569861951494",
                "status": "closed",
                "average": 0.02619543,
                "cost": 5.510289876139975,
                "filled": 210.353065,
                "is_open": false,
                "order_date": "2023-06-28 21:11:48",
                "order_timestamp": 1687986708772,
                "order_filled_date": "2023-06-28 21:12:07",
                "order_type": "market",
                "price": 0.0261954342150421,
                "remaining": 0.0
            },
            {
                "amount": 209.604346,
                "safe_price": 0.02664,
                "ft_order_side": "sell",
                "order_filled_timestamp": 1687986729135,
                "ft_is_entry": false,
                "pair": "BTM/USDT",
                "order_id": "594394652145807360",
                "status": "closed",
                "average": 0.02664,
                "cost": 5.58385977744,
                "filled": 209.604346,
                "is_open": false,
                "order_date": "2023-06-28 21:12:08",
                "order_timestamp": 1687986728390,
                "order_filled_date": "2023-06-28 21:12:09",
                "order_type": "market",
                "price": 0.02664,
                "remaining": 0.0
            },
            {
                "amount": 202.736538,
                "safe_price": 0.026610184122706,
                "ft_order_side": "sell",
                "order_filled_timestamp": 1687986734126,
                "ft_is_entry": false,
                "pair": "BTM/USDT",
                "order_id": "594394660354060288",
                "status": "closed",
                "average": 0.02661018,
                "cost": 5.3948566045799815,
                "filled": 202.736538,
                "is_open": false,
                "order_date": "2023-06-28 21:12:10",
                "order_timestamp": 1687986730347,
                "order_filled_date": "2023-06-28 21:12:14",
                "order_type": "market",
                "price": 0.026610184122706,
                "remaining": 0.0
            },
            {
                "amount": 204.70188,
                "safe_price": 0.0267,
                "ft_order_side": "sell",
                "order_filled_timestamp": 1687986739132,
                "ft_is_entry": false,
                "pair": "BTM/USDT",
                "order_id": "594394681438826499",
                "status": "closed",
                "average": 0.0267,
                "cost": 5.465540196,
                "filled": 204.70188,
                "is_open": false,
                "order_date": "2023-06-28 21:12:15",
                "order_timestamp": 1687986735374,
                "order_filled_date": "2023-06-28 21:12:19",
                "order_type": "market",
                "price": 0.0267,
                "remaining": 0.0
            },
            {
                "amount": 607.328253,
                "safe_price": 0.0269,
                "ft_order_side": "sell",
                "order_filled_timestamp": 1687986744119,
                "ft_is_entry": false,
                "pair": "BTM/USDT",
                "order_id": "594394700543881216",
                "status": "closed",
                "average": 0.0269,
                "cost": 16.3371300057,
                "filled": 607.328253,
                "is_open": false,
                "order_date": "2023-06-28 21:12:19",
                "order_timestamp": 1687986739929,
                "order_filled_date": "2023-06-28 21:12:24",
                "order_type": "market",
                "price": 0.0269,
                "remaining": 0.0
            }
        ]
    }
]
```
</details>
Or maybe a better format?

[tradejson.txt](https://github.com/freqtrade/freqtrade/files/11990549/tradejson.txt)

the format was ok :+1: 
but that doesn't mean i have immediately time to look at it :laughing: sorry
That's quite an interesting case - as the logic **per se** seems to work (just not for all orders).
Assuming we only look at the first 7 orders (that's the one we have screenshots for, and i think is around the time you reported this).

the sum of all amounts is 1016.251934 (that's what the bot TRIED to buy).
The exchange, you show 1014.7088  - which is roughly correct (so the trade clearly matches), but is not exact.
The UI is reporting 1015.671332 - which is something in the middle.

Based on your screenshots, i've added the individual fee amounts (They're not shown by the output currently), and recalculated the trade (up to order 7).

Assuming the correct fees - i get 1014.70881051 - which is close enough to the exchange to be a rounding / display error (it's just cut off on the exchange).

The difference between the two values seems to be 0.962532 - which corresponds to the sum of all fees except for the first order.
The log shows that it detected the fees, but ignored them as "eating into dust" (which would imply that you had more BTM on exchange than what's actually shown - which i however believe is not true based on the screenshots shown.

---

I think i have a rough idea how it comes to that - but it's not immediately trivial to recreate this exact behavior.
will continue looking at this though - as it's clearly not how we'd like freqtrade to behave.
Thank you for taking the time to investigate this.
If i can supply anymore data etc to help,please let me know.
Or if the issue happens again i will report it


Resolving patch:
diff --git a/freqtrade/freqtradebot.py b/freqtrade/freqtradebot.py
--- a/freqtrade/freqtradebot.py
+++ b/freqtrade/freqtradebot.py
@@ -1939,6 +1939,7 @@ def apply_fee_conditional(self, trade: Trade, trade_base_currency: str,
         """
         Applies the fee to amount (either from Order or from Trades).
         Can eat into dust if more than the required asset is available.
+        In case of trade adjustment orders, trade.amount will not have been adjusted yet.
         Can't happen in Futures mode - where Fees are always in settlement currency,
         never in base currency.
         """
@@ -1948,6 +1949,10 @@ def apply_fee_conditional(self, trade: Trade, trade_base_currency: str,
             # check against remaining amount!
             amount_ = trade.amount - amount
 
+        if trade.nr_of_successful_entries >= 1 and order_obj.ft_order_side == trade.entry_side:
+            # In case of rebuy's, trade.amount doesn't contain the amount of the last entry.
+            amount_ = trade.amount + amount
+
         if fee_abs != 0 and self.wallets.get_free(trade_base_currency) >= amount_:
             # Eat into dust if we own more than base currency
             logger.info(f"Fee amount for {trade} was in base currency - "
@@ -1977,7 +1982,11 @@ def get_real_amount(self, trade: Trade, order: Dict, order_obj: Order) -> Option
         # Init variables
         order_amount = safe_value_fallback(order, 'filled', 'amount')
         # Only run for closed orders
-        if trade.fee_updated(order.get('side', '')) or order['status'] == 'open':
+        if (
+            trade.fee_updated(order.get('side', ''))
+            or order['status'] == 'open'
+            or order_obj.ft_fee_base
+        ):
             return None
 
         trade_base_currency = self.exchange.get_pair_base_currency(trade.pair)


Test patch:
diff --git a/tests/test_freqtradebot.py b/tests/test_freqtradebot.py
--- a/tests/test_freqtradebot.py
+++ b/tests/test_freqtradebot.py
@@ -5023,7 +5023,7 @@ def test_get_real_amount_in_point(default_conf_usdt, buy_order_fee, fee, mocker,
     (8.0, 0.1, 8.0, None),
     (8.0, 0.1, 7.9, 0.1),
 ])
-def test_apply_fee_conditional(default_conf_usdt, fee, mocker,
+def test_apply_fee_conditional(default_conf_usdt, fee, mocker, caplog,
                                amount, fee_abs, wallet, amount_exp):
     walletmock = mocker.patch('freqtrade.wallets.Wallets.update')
     mocker.patch('freqtrade.wallets.Wallets.get_free', return_value=wallet)
@@ -5048,6 +5048,60 @@ def test_apply_fee_conditional(default_conf_usdt, fee, mocker,
     # Amount is kept as is
     assert freqtrade.apply_fee_conditional(trade, 'LTC', amount, fee_abs, order) == amount_exp
     assert walletmock.call_count == 1
+    if fee_abs != 0 and amount_exp is None:
+        assert log_has_re(r"Fee amount.*Eating.*dust\.", caplog)
+
+
+@pytest.mark.parametrize('amount,fee_abs,wallet,amount_exp', [
+    (8.0, 0.0, 16, None),
+    (8.0, 0.0, 0, None),
+    (8.0, 0.1, 8, 0.1),
+    (8.0, 0.1, 20, None),
+    (8.0, 0.1, 16.0, None),
+    (8.0, 0.1, 7.9, 0.1),
+    (8.0, 0.1, 12, 0.1),
+    (8.0, 0.1, 15.9, 0.1),
+])
+def test_apply_fee_conditional_multibuy(default_conf_usdt, fee, mocker, caplog,
+                                        amount, fee_abs, wallet, amount_exp):
+    walletmock = mocker.patch('freqtrade.wallets.Wallets.update')
+    mocker.patch('freqtrade.wallets.Wallets.get_free', return_value=wallet)
+    trade = Trade(
+        pair='LTC/ETH',
+        amount=amount,
+        exchange='binance',
+        open_rate=0.245441,
+        fee_open=fee.return_value,
+        fee_close=fee.return_value,
+        open_order_id="123456"
+    )
+    # One closed order
+    order = Order(
+        ft_order_side='buy',
+        order_id='10',
+        ft_pair=trade.pair,
+        ft_is_open=False,
+        filled=amount,
+        status="closed"
+    )
+    trade.orders.append(order)
+    # Add additional order - this should NOT eat into dust unless the wallet was bigger already.
+    order1 = Order(
+        ft_order_side='buy',
+        order_id='100',
+        ft_pair=trade.pair,
+        ft_is_open=True,
+    )
+    trade.orders.append(order1)
+
+    freqtrade = get_patched_freqtradebot(mocker, default_conf_usdt)
+
+    walletmock.reset_mock()
+    # The new trade amount will be 2x amount - fee / wallet will have to be adapted to this.
+    assert freqtrade.apply_fee_conditional(trade, 'LTC', amount, fee_abs, order1) == amount_exp
+    assert walletmock.call_count == 1
+    if fee_abs != 0 and amount_exp is None:
+        assert log_has_re(r"Fee amount.*Eating.*dust\.", caplog)
 
 
 @pytest.mark.parametrize("delta, is_high_delta", [