Repository: hummingbot/hummingbot
Pull request: Fix / balance limit to account for orders filled
**Before submitting this PR, please make sure**:
- [x] Your code builds clean without any errors or warnings
- [x] You are using approved title ("feat/", "fix/", "docs/", "refactor/", etc)


**Optional**:
- Related Github issue: #1443 Fixes #2145
- Related Clubhouse Story:


**A description of the changes proposed in the pull request**:
As per this comment https://github.com/CoinAlpha/hummingbot/issues/1443#issuecomment-670929423

Problem statement:
[Feature] Allow limiting the amount of assets HB can access in an exchange/wallet
**Feature Suggestion**
There should be an ability to limit the amount of assets HB can access/use in an exchange or wallet. The asset symbol and the limit size in asset units should be configurable.

**Impact**
It can be a good security measure to avoid unexpected and undesirable losses. Along with that, it is just convenient for a user who doesn't wish to remove the 'untouchable' amounts from the exchange/wallet before starting HB.

**Additional context**
None

[Feature] Dict json format for paper_trade_account_balance and min_quote_order_amount
**Feature Suggestion**
Currently `paper_trade_account_balance` and` min_quote_order_amount` store their data in a json list format, which, I think, is harder for users to manually update (than the dictionary format one). And from the data structure prospective, they should be dictionary (as you shouldn't have the same entry for the same token).  `balance_asset_limit` (implemented by @CrimsonJacket) uses the dictionary format and looks cleaner imo.

Example
```
paper_trade_account_balance:
- - USDT
  - 3000
- - ONE
  - 1000
- - BTC
  - 1
- - ETH
  - 10
balance_asset_limit:
  radar_relay:
  kraken:
    BTC: 0.2
  binance:
    LINK: 99
    BTC: 0.2
```

The above `paper_trade_account_balance` will then be
```
paper_trade_account_balance:
  USDT: 3000
  ONE: 1000
  BTC: 1
  ETH: 10
```
**Impact**
Changing the format is quite a simple task, but you might need to handle migration for users of the old format to the new one.

Hints from issue discussion:
The closes function that I could think of is Hummingbot feature called [kill-switch](https://docs.hummingbot.io/utilities/kill-switch/). Automatically stops the bot when it reaches a certain performance threshold, which can be either positive or negative.

But this could be a good new feature.
I like this feature. Should be useful for users running bots on many different trading pairs with shared assets.

Commit message: (fix) no update on avaliable balance, when get account for order filled balance

Patch to review:
diff --git a/hummingbot/market/bamboo_relay/bamboo_relay_market.pyx b/hummingbot/market/bamboo_relay/bamboo_relay_market.pyx
index e16e812b0ef..bb0587af251 100755
--- a/hummingbot/market/bamboo_relay/bamboo_relay_market.pyx
+++ b/hummingbot/market/bamboo_relay/bamboo_relay_market.pyx
@@ -487,7 +487,6 @@ cdef class BambooRelayMarket(MarketBase):
                 self._account_available_balances = self._account_balances.copy()
 
             self._last_update_available_balance_timestamp = current_timestamp
-        self.apply_balance_restriction()
 
     async def list_market(self) -> Dict[str, Any]:
         url = f"{self._api_endpoint}{self._api_prefix}/markets?perPage=1000&include=base"
diff --git a/hummingbot/market/binance/binance_market.pyx b/hummingbot/market/binance/binance_market.pyx
index ad24973cb39..51b8d3133c2 100755
--- a/hummingbot/market/binance/binance_market.pyx
+++ b/hummingbot/market/binance/binance_market.pyx
@@ -298,8 +298,6 @@ cdef class BinanceMarket(MarketBase):
             del self._account_available_balances[asset_name]
             del self._account_balances[asset_name]
 
-        self.apply_balance_restriction()
-
     async def _update_trade_fees(self):
         cdef:
             double current_timestamp = self._current_timestamp
@@ -702,7 +700,6 @@ cdef class BinanceMarket(MarketBase):
                     for asset_name in asset_names_to_remove:
                         del self._account_available_balances[asset_name]
                         del self._account_balances[asset_name]
-                    self.apply_balance_restriction()
 
             except asyncio.CancelledError:
                 raise
diff --git a/hummingbot/market/bittrex/bittrex_market.pyx b/hummingbot/market/bittrex/bittrex_market.pyx
index 1e8f1ff67de..3dd9a5c097b 100644
--- a/hummingbot/market/bittrex/bittrex_market.pyx
+++ b/hummingbot/market/bittrex/bittrex_market.pyx
@@ -226,8 +226,6 @@ cdef class BittrexMarket(MarketBase):
             del self._account_available_balances[asset_name]
             del self._account_balances[asset_name]
 
-        self.apply_balance_restriction()
-
     def _format_trading_rules(self, market_dict: Dict[str, Any]) -> List[TradingRule]:
         cdef:
             list retval = []
@@ -484,7 +482,6 @@ cdef class BittrexMarket(MarketBase):
                     available_balance = Decimal(balance_delta["a"])
                     self._account_available_balances[asset_name] = available_balance
                     self._account_balances[asset_name] = total_balance
-                    self.apply_balance_restriction()
                 elif event_type == "uO":  # Updates track order status
                     order = content["o"]
                     order_status = content["TY"]
diff --git a/hummingbot/market/eterbase/eterbase_market.pyx b/hummingbot/market/eterbase/eterbase_market.pyx
index c0e3420e89c..d7f69c2de32 100644
--- a/hummingbot/market/eterbase/eterbase_market.pyx
+++ b/hummingbot/market/eterbase/eterbase_market.pyx
@@ -375,8 +375,6 @@ cdef class EterbaseMarket(MarketBase):
             del self._account_available_balances[asset_name]
             del self._account_balances[asset_name]
 
-        self.apply_balance_restriction()
-
     async def _update_trading_rules(self):
         """
         Pulls the API for trading rules (min / max order size, etc)
diff --git a/hummingbot/market/huobi/huobi_market.pyx b/hummingbot/market/huobi/huobi_market.pyx
index 36e6546d452..77ad9b58c58 100644
--- a/hummingbot/market/huobi/huobi_market.pyx
+++ b/hummingbot/market/huobi/huobi_market.pyx
@@ -364,8 +364,6 @@ cdef class HuobiMarket(MarketBase):
             self._account_balances.clear()
             self._account_balances = new_balances
 
-        self.apply_balance_restriction()
-
     cdef object c_get_fee(self,
                           str base_currency,
                           str quote_currency,
diff --git a/hummingbot/market/kraken/kraken_market.pyx b/hummingbot/market/kraken/kraken_market.pyx
index 71aa9dd74af..b32909d13cf 100755
--- a/hummingbot/market/kraken/kraken_market.pyx
+++ b/hummingbot/market/kraken/kraken_market.pyx
@@ -326,8 +326,6 @@ cdef class KrakenMarket(MarketBase):
             del self._account_available_balances[asset_name]
             del self._account_balances[asset_name]
 
-        self.apply_balance_restriction()
-
     cdef object c_get_fee(self,
                           str base_currency,
                           str quote_currency,
diff --git a/hummingbot/market/kucoin/kucoin_market.pyx b/hummingbot/market/kucoin/kucoin_market.pyx
index abe4517f408..5f8d0f0ecf6 100644
--- a/hummingbot/market/kucoin/kucoin_market.pyx
+++ b/hummingbot/market/kucoin/kucoin_market.pyx
@@ -297,7 +297,6 @@ cdef class KucoinMarket(MarketBase):
                     total_balance = execution_data["total"]
                     self._account_balances.update({currency, total_balance})
                     self._account_available_balances.update({currency, available_balance})
-                    self.apply_balance_restriction()
                 else:
                     continue
 
@@ -454,8 +453,6 @@ cdef class KucoinMarket(MarketBase):
             self._account_balances.clear()
             self._account_balances = new_balances
 
-        self.apply_balance_restriction()
-
     cdef object c_get_fee(self,
                           str base_currency,
                           str quote_currency,
diff --git a/hummingbot/market/liquid/liquid_market.pyx b/hummingbot/market/liquid/liquid_market.pyx
index 64bd42edc73..eba8d8399d4 100644
--- a/hummingbot/market/liquid/liquid_market.pyx
+++ b/hummingbot/market/liquid/liquid_market.pyx
@@ -419,8 +419,6 @@ cdef class LiquidMarket(MarketBase):
             del self._account_available_balances[asset_name]
             del self._account_balances[asset_name]
 
-        self.apply_balance_restriction()
-
     async def _update_trading_rules(self):
         """
         Pulls the API for trading rules (min / max order size, etc)
diff --git a/hummingbot/market/market_base.pyx b/hummingbot/market/market_base.pyx
index 7b7d01ef587..b1bbe7ef2e3 100755
--- a/hummingbot/market/market_base.pyx
+++ b/hummingbot/market/market_base.pyx
@@ -31,6 +31,7 @@ from hummingbot.core.network_iterator import NetworkIterator
 from hummingbot.core.data_type.order_book import OrderBook
 from hummingbot.market.in_flight_order_base import InFlightOrderBase
 from .deposit_info import DepositInfo
+from hummingbot.core.event.events import OrderFilledEvent
 
 NaN = float("nan")
 s_decimal_NaN = Decimal("nan")
@@ -75,6 +76,8 @@ cdef class MarketBase(NetworkIterator):
     def in_flight_asset_balances(in_flight_orders: Dict[str, InFlightOrderBase]) -> Dict[str, Decimal]:
         """
         Calculates the individual asset balances used in in_flight_orders
+        For BUY order, this is the quote asset balance locked in the order
+        For SELL order, this is the base asset balance locked in the order
         """
         asset_balances = {}
         for order in [o for o in in_flight_orders.values() if not (o.is_done or o.is_failure or o.is_cancelled)]:
@@ -91,6 +94,31 @@ cdef class MarketBase(NetworkIterator):
                 asset_balances[order.base_asset] += outstanding_value
         return asset_balances
 
+    def order_filled_balances(self):
+        """
+        Calculates the individual asset balances as a result of order being filled
+        For BUY filled order, the quote balance goes down while the base balance goes up, and for SELL order, it's the
+        opposite. This does not account for fee.
+        """
+        order_filled_events = list(filter(lambda e: isinstance(e, OrderFilledEvent), self.event_logs))
+        balances = {}
+        for event in order_filled_events:
+            hb_trading_pair = self.convert_from_exchange_trading_pair(event.trading_pair)
+            base, quote = hb_trading_pair.split("-")[0], hb_trading_pair.split("-")[1]
+            if event.trade_type is TradeType.BUY:
+                quote_value = Decimal("-1") * event.price * event.amount
+                base_value = event.amount
+            else:
+                quote_value = event.price * event.amount
+                base_value = Decimal("-1") * event.amount
+            if base not in balances:
+                balances[base] = s_decimal_0
+            if quote not in balances:
+                balances[quote] = s_decimal_0
+            balances[base] += base_value
+            balances[quote] += quote_value
+        return balances
+
     @staticmethod
     def convert_from_exchange_trading_pair(exchange_trading_pair: str) -> Optional[str]:
         return exchange_trading_pair
@@ -150,24 +178,6 @@ cdef class MarketBase(NetworkIterator):
         exchange_limits = all_ex_limit.get(market, {})
         return exchange_limits if exchange_limits is not None else {}
 
-    def apply_balance_restriction(self):
-        """
-        Updates self._account_balances and self._account_available_balances.
-        To be called after every REST API fetch or WebSocket API update.
-        """
-        exchange_limits = self.get_exchange_limit_config(self.name)
-
-        for asset_name, available_balance in self._account_available_balances.items():
-            if asset_name.upper() in exchange_limits:
-                asset_limit = Decimal(str(exchange_limits[asset_name.upper()]))
-
-                active_asset_balance = self.in_flight_asset_balances(self.in_flight_orders).get(asset_name.upper(),
-                                                                                                s_decimal_0)
-                current_asset_available_balance = self._account_available_balances.get(asset_name.upper(), s_decimal_0)
-                new_asset_available_balance = min(current_asset_available_balance,
-                                                  max(asset_limit - active_asset_balance, s_decimal_0))
-                self._account_available_balances[asset_name] = new_asset_available_balance
-
     async def get_active_exchange_markets(self) -> pd.DataFrame:
         """
         :return: data frame with trading_pair as index, and at least the following columns --
@@ -217,12 +227,34 @@ cdef class MarketBase(NetworkIterator):
         """
         return self._account_balances.get(currency, s_decimal_0)
 
+    def available_balance_limit_applied(self, currency: str, limit: Decimal) -> Decimal:
+        """
+        Apply budget limit on an available balance, the limit is calculated as followings:
+        - Minus balance used in outstanding orders (in flight orders), if the budget is 1 ETH and the bot has already
+          used 0.5 ETH to put a maker buy order, the budget is now 0.5
+        - Plus balance accredited from filled orders (since the bot started), if the budget is 1 ETH and the bot has
+          bought LINK (for 0.5 ETH), the ETH budget is now 0.5. However if later on the bot has sold LINK (for 0.5 ETH)
+          the budget is now 1 ETH
+        """
+        in_flight_balance = self.in_flight_asset_balances(self.in_flight_orders).get(currency.upper(), s_decimal_0)
+        limit -= in_flight_balance
+        filled_balance = self.order_filled_balances().get(currency.upper(), s_decimal_0)
+        limit += filled_balance
+        asset_limit = max(limit, s_decimal_0)
+        available_balance = self._account_available_balances.get(currency.upper(), s_decimal_0)
+        return min(available_balance, asset_limit)
+
     cdef object c_get_available_balance(self, str currency):
         """
+        If there is a budget limit set on the balance
         :returns: Balance available for trading for a specific asset
-        (balances used to place open orders are not available for trading)
         """
-        return self._account_available_balances.get(currency, s_decimal_0)
+        exchange_limits = self.get_exchange_limit_config(self.name)
+        if currency.upper() in exchange_limits:
+            asset_limit = Decimal(str(exchange_limits[currency.upper()]))
+            return self.available_balance_limit_applied(currency, asset_limit)
+        else:
+            return self._account_available_balances.get(currency, s_decimal_0)
 
     cdef str c_withdraw(self, str address, str currency, object amount):
         raise NotImplementedError
diff --git a/hummingbot/market/radar_relay/radar_relay_market.pyx b/hummingbot/market/radar_relay/radar_relay_market.pyx
index 1e947ac0a72..bc984012734 100755
--- a/hummingbot/market/radar_relay/radar_relay_market.pyx
+++ b/hummingbot/market/radar_relay/radar_relay_market.pyx
@@ -314,7 +314,6 @@ cdef class RadarRelayMarket(MarketBase):
                 self._account_available_balances = self._account_balances.copy()
 
             self._last_update_available_balance_timestamp = current_timestamp
-        self.apply_balance_restriction()
 
     async def list_market(self) -> Dict[str, Any]:
         url = f"{RADAR_RELAY_REST_ENDPOINT}/markets?include=base"


Code review comments:
diff_hunk: @@ -217,12 +227,34 @@ cdef class MarketBase(NetworkIterator):
         """
         return self._account_balances.get(currency, s_decimal_0)
 
+    def available_balance_limit_applied(self, currency: str, limit: Decimal) -> Decimal:
+        """
+        Apply budget limit on an available balance, the limit is calculated as followings:
+        - Minus balance used in outstanding orders (in flight orders), if the budget is 1 ETH and the bot has already
+          used 0.5 ETH to put a maker buy order, the budget is now 0.5
+        - Plus balance accredited from filled orders (since the bot started), if the budget is 1 ETH and the bot has
+          bought LINK (for 0.5 ETH), the ETH budget is now 0.5. However if later on the bot has sold LINK (for 0.5 ETH)
+          the budget is now 1 ETH
+        """
+        in_flight_balance = self.in_flight_asset_balances(self.in_flight_orders).get(currency.upper(), s_decimal_0)
+        limit -= in_flight_balance
+        filled_balance = self.order_filled_balances().get(currency.upper(), s_decimal_0)
+        limit += filled_balance
+        asset_limit = max(limit, s_decimal_0)
+        available_balance = self._account_available_balances.get(currency.upper(), s_decimal_0)
path: hummingbot/market/market_base.pyx
text: @user1:
I don't think this should be `self._account_available_balances.get(currency.upper(), s_decimal_0)` here since this is an exchange function and some use lower case names? It caused issues for me but changing to `self._account_available_balances.get(currency, s_decimal_0)` fixed it for me.

@author:
You're right. Fixed now.

@user1:
> You're right. Fixed now.

👍