Repository: rotki/rotki

Issue:
Binance API: Some assets not reflected in v1.8.3
## Defi staked assets not showing up in v1.8.3

When using the Rotki portfolio tracker with the Binance API, Rotki does not display my full holdings. Specifically, what is missing are BNB tokens held under 'DeFi staking' on https://www.binance.com/en/my/wallet/account/saving. I hold no other assets under DeFi Staking, so not sure if it's all of them or just BNB. Other balances appear to be accurate, both in the main wallet and locked under 'Earn'. Problem was present in previous builds as well.

## Logs

No error messages, no logs.


### System Description


Operating system: Windows 10 Home, 18363
Rotki version: v.1.8.3

Binance API: Some assets not reflected in v1.8.3
## Defi staked assets not showing up in v1.8.3

When using the Rotki portfolio tracker with the Binance API, Rotki does not display my full holdings. Specifically, what is missing are BNB tokens held under 'DeFi staking' on https://www.binance.com/en/my/wallet/account/saving. I hold no other assets under DeFi Staking, so not sure if it's all of them or just BNB. Other balances appear to be accurate, both in the main wallet and locked under 'Earn'. Problem was present in previous builds as well.

## Logs

No error messages, no logs.


### System Description


Operating system: Windows 10 Home, 18363
Rotki version: v.1.8.3



Discussion:
Hey @Rhamni1988 sorry for not responding to this issue sooner. It fell through the cracks.

I suppose you refer to https://www.binance.com/en/defi-staking?

By looking at their API docs I guess this is it: https://binance-docs.github.io/apidocs/spot/en/#get-flexible-product-position-user_data. Should not be hard to add.

Will schedule this in one of the next releases.

>I suppose you refer to https://www.binance.com/en/defi-staking?

That's the one!

Sounds great.

Other than that, there are a few other assets on Binance that Rotki doesn't recognize, as it gives me error messages for each when I start the client. These are BETH, SCRT, JEX and BCX. Of these, BETH is the important one, as it's Beacon ETH/ETH 2.0, the token you get when you lock and stake ETH on Binance.

Anyway. Loved the changes for 1.9.0, with the tracking of performance over time.
All right. Thanks a lot for letting us know. Will have a look. `BETH` is `ETH2` for Rotki so this should be solved by simply mapping it properly.

For more real-time chat you can also join our discord server: https://discord.gg/aGCxHG7 
Hey @Rhamni1988 sorry for not responding to this issue sooner. It fell through the cracks.

I suppose you refer to https://www.binance.com/en/defi-staking?

By looking at their API docs I guess this is it: https://binance-docs.github.io/apidocs/spot/en/#get-flexible-product-position-user_data. Should not be hard to add.

Will schedule this in one of the next releases.

>I suppose you refer to https://www.binance.com/en/defi-staking?

That's the one!

Sounds great.

Other than that, there are a few other assets on Binance that Rotki doesn't recognize, as it gives me error messages for each when I start the client. These are BETH, SCRT, JEX and BCX. Of these, BETH is the important one, as it's Beacon ETH/ETH 2.0, the token you get when you lock and stake ETH on Binance.

Anyway. Loved the changes for 1.9.0, with the tracking of performance over time.
All right. Thanks a lot for letting us know. Will have a look. `BETH` is `ETH2` for Rotki so this should be solved by simply mapping it properly.

For more real-time chat you can also join our discord server: https://discord.gg/aGCxHG7 

Resolving patch:
diff --git a/rotkehlchen/assets/asset.py b/rotkehlchen/assets/asset.py
--- a/rotkehlchen/assets/asset.py
+++ b/rotkehlchen/assets/asset.py
@@ -144,6 +144,8 @@
     'YOYOW': 'YOYO',
     # Solana is SOL-2 in rotki
     'SOL-2': 'SOL',
+    # BETH is the eth staked in beacon chain
+    'BETH': 'ETH2',
 }
 
 
diff --git a/rotkehlchen/assets/converters.py b/rotkehlchen/assets/converters.py
--- a/rotkehlchen/assets/converters.py
+++ b/rotkehlchen/assets/converters.py
@@ -503,6 +503,11 @@ def asset_from_bittrex(bittrex_name: str) -> Asset:
 
 
 def asset_from_binance(binance_name: str) -> Asset:
+    """May raise:
+    - DeserializationError
+    - UnsupportedAsset
+    - UnknownAsset
+    """
     if not isinstance(binance_name, str):
         raise DeserializationError(f'Got non-string type {type(binance_name)} for binance asset')
 
diff --git a/rotkehlchen/exchanges/binance.py b/rotkehlchen/exchanges/binance.py
--- a/rotkehlchen/exchanges/binance.py
+++ b/rotkehlchen/exchanges/binance.py
@@ -68,6 +68,9 @@
 
 SAPI_ENDPOINTS = (
     'futures/loan/wallet',
+    'lending/daily/token/position',
+    'lending/daily/product/list',
+    'lending/union/account',
 )
 
 RETRY_AFTER_LIMIT = 60
@@ -397,6 +400,59 @@ def _query_spot_balances(self, balances: Dict) -> Dict:
 
         return balances
 
+    def _query_lending_balances(self, balances: Dict) -> Dict:
+        data = self.api_query_dict('lending/union/account')
+        positions = data.get('positionAmountVos', None)
+        if positions is None:
+            raise RemoteError(
+                f'Could not find key positionAmountVos in lending account data '
+                f'{data} returned by binance',
+            )
+
+        for entry in positions:
+            try:
+                asset = asset_from_binance(entry['asset'])
+                amount = FVal(entry['amount'])
+            except UnsupportedAsset as e:
+                self.msg_aggregator.add_warning(
+                    f'Found unsupported binance asset {e.asset_name}. '
+                    f' Ignoring its lending balance query.',
+                )
+                continue
+            except UnknownAsset as e:
+                self.msg_aggregator.add_warning(
+                    f'Found unknown binance asset {e.asset_name}. '
+                    f' Ignoring its lending balance query.',
+                )
+                continue
+            except (DeserializationError, KeyError) as e:
+                msg = str(e)
+                if isinstance(e, KeyError):
+                    msg = f'Missing key entry for {msg}.'
+                self.msg_aggregator.add_error(
+                    f'Error at deserializing binance asset. {msg}.'
+                    f' Ignoring its lending balance query.',
+                )
+                continue
+
+            try:
+                usd_price = Inquirer().find_usd_price(asset)
+            except RemoteError as e:
+                self.msg_aggregator.add_error(
+                    f'Error processing binance balance entry due to inability to '
+                    f'query USD price: {str(e)}. Skipping balance entry',
+                )
+                continue
+
+            balance = Balance(amount=amount, usd_value=amount * usd_price)
+            if asset not in balances:
+                balances[asset] = balance.to_dict()
+            else:
+                balances[asset]['amount'] += balance.amount
+                balances[asset]['usd_value'] += balance.usd_value
+
+        return balances
+
     def _query_futures_balances(self, balances: Dict) -> Dict:
         futures_response = self.api_query_dict('futures/loan/wallet')
         try:
@@ -455,12 +511,13 @@ def query_balances(self) -> Tuple[Optional[Dict], str]:
         try:
             returned_balances = self._query_spot_balances(returned_balances)
             returned_balances = self._query_futures_balances(returned_balances)
+            returned_balances = self._query_lending_balances(returned_balances)
         except RemoteError as e:
             msg = (
                 'Binance account API request failed. Could not reach binance due '
                 'to {}'.format(e)
             )
-            log.error(msg)
+            self.msg_aggregator.add_error(msg)
             return None, msg
 
         log.debug(


Test patch:
diff --git a/rotkehlchen/tests/api/test_async.py b/rotkehlchen/tests/api/test_async.py
--- a/rotkehlchen/tests/api/test_async.py
+++ b/rotkehlchen/tests/api/test_async.py
@@ -11,11 +11,7 @@
     assert_ok_async_response,
     assert_proper_response,
 )
-from rotkehlchen.tests.utils.exchanges import (
-    BINANCE_BALANCES_RESPONSE,
-    BINANCE_FUTURES_WALLET_RESPONSE,
-)
-from rotkehlchen.tests.utils.mock import MockResponse
+from rotkehlchen.tests.utils.exchanges import mock_binance_balance_response
 
 
 @pytest.mark.parametrize('added_exchanges', [('binance', 'poloniex')])
@@ -33,13 +29,7 @@ def test_query_async_tasks(rotkehlchen_api_server_with_exchanges):
     server = rotkehlchen_api_server_with_exchanges
     binance = server.rest_api.rotkehlchen.exchange_manager.connected_exchanges['binance']
 
-    def mock_binance_asset_return(url):  # pylint: disable=unused-argument
-        if 'futures' in url:
-            return MockResponse(200, BINANCE_FUTURES_WALLET_RESPONSE)
-        # else
-        return MockResponse(200, BINANCE_BALANCES_RESPONSE)
-
-    binance_patch = patch.object(binance.session, 'get', side_effect=mock_binance_asset_return)
+    binance_patch = patch.object(binance.session, 'get', side_effect=mock_binance_balance_response)
 
     # Check querying the async taks resource when no async task is scheduled
     response = requests.get(api_url_for(server, "asynctasksresource"))
diff --git a/rotkehlchen/tests/api/test_balances.py b/rotkehlchen/tests/api/test_balances.py
--- a/rotkehlchen/tests/api/test_balances.py
+++ b/rotkehlchen/tests/api/test_balances.py
@@ -175,6 +175,8 @@ def test_query_all_balances(
         else:
             outcome = assert_proper_response_with_result(response)
 
+    errors = rotki.msg_aggregator.consume_errors()
+    assert len(errors) == 0
     assert_all_balances(
         result=outcome,
         db=rotki.data.db,
@@ -276,7 +278,7 @@ def test_query_all_balances_ignore_cache(
         )
         for fn in function_call_counters:
             if fn._mock_wraps == original_binance_query_dict:
-                assert fn.call_count == 2
+                assert fn.call_count == 3
             else:
                 assert fn.call_count == 1
         full_query_etherscan_count = etherscan_mock.call_count
@@ -298,7 +300,7 @@ def test_query_all_balances_ignore_cache(
         msg = 'call count should stay the same since cache should have been used'
         for fn in function_call_counters:
             if fn._mock_wraps == original_binance_query_dict:
-                assert fn.call_count == 2, msg
+                assert fn.call_count == 3, msg
             else:
                 assert fn.call_count == 1, msg
         msg = 'etherscan call_count should have remained the same due to no token detection '
@@ -321,7 +323,7 @@ def test_query_all_balances_ignore_cache(
         msg = 'call count should increase since cache should have been ignored'
         for fn in function_call_counters:
             if fn._mock_wraps == original_binance_query_dict:
-                assert fn.call_count == 4, msg
+                assert fn.call_count == 6, msg
             else:
                 assert fn.call_count == 2, msg
         msg = 'etherscan call count should have doubled after forced token detection'
@@ -504,28 +506,33 @@ def test_multiple_balance_queries_not_concurrent(
         outcome_all = wait_for_async_task_with_result(
             rotkehlchen_api_server_with_exchanges,
             task_id_all,
+            timeout=ASYNC_TASK_WAIT_TIMEOUT * 2,
         )
         outcome_one_exchange = wait_for_async_task(
             rotkehlchen_api_server_with_exchanges,
             task_id_one_exchange,
+            timeout=ASYNC_TASK_WAIT_TIMEOUT * 2,
         )
         if separate_blockchain_calls:
             outcome_eth = wait_for_async_task_with_result(
                 rotkehlchen_api_server_with_exchanges,
                 task_id_blockchain_eth,
+                timeout=ASYNC_TASK_WAIT_TIMEOUT * 2,
             )
             outcome_btc = wait_for_async_task_with_result(
                 rotkehlchen_api_server_with_exchanges,
                 task_id_blockchain_btc,
+                timeout=ASYNC_TASK_WAIT_TIMEOUT * 2,
             )
         else:
             outcome_blockchain = wait_for_async_task_with_result(
                 rotkehlchen_api_server_with_exchanges,
                 task_id_blockchain,
+                timeout=ASYNC_TASK_WAIT_TIMEOUT * 2,
             )
         assert eth.call_count == 1, 'eth balance query should only fire once'
         assert btc.call_count == 1, 'btc balance query should only happen once'
-        assert bn.call_count == 2, 'binance balance query should do 2 calls'
+        assert bn.call_count == 3, 'binance balance query should do 2 calls'
 
     assert_all_balances(
         result=outcome_all,
diff --git a/rotkehlchen/tests/api/test_exchanges.py b/rotkehlchen/tests/api/test_exchanges.py
--- a/rotkehlchen/tests/api/test_exchanges.py
+++ b/rotkehlchen/tests/api/test_exchanges.py
@@ -507,7 +507,7 @@ def test_exchange_query_balances_ignore_cache(rotkehlchen_api_server_with_exchan
         json_data = response.json()
         assert json_data['message'] == ''
         assert_binance_balances_result(json_data['result'])
-        assert bn.call_count == 2
+        assert bn.call_count == 3
         # Do the query again. Cache should be used.
         binance_patch = patch_binance_balances_query(binance)
         response = requests.get(api_url_for(
@@ -519,7 +519,7 @@ def test_exchange_query_balances_ignore_cache(rotkehlchen_api_server_with_exchan
         json_data = response.json()
         assert json_data['message'] == ''
         assert_binance_balances_result(json_data['result'])
-        assert bn.call_count == 2, 'call count should not have changed. Cache must have been used'
+        assert bn.call_count == 3, 'call count should not have changed. Cache must have been used'
         # Finally do the query and request ignoring of the cache
         binance_patch = patch_binance_balances_query(binance)
         response = requests.get(api_url_for(
@@ -531,7 +531,7 @@ def test_exchange_query_balances_ignore_cache(rotkehlchen_api_server_with_exchan
         json_data = response.json()
         assert json_data['message'] == ''
         assert_binance_balances_result(json_data['result'])
-        assert bn.call_count == 4, 'call count should have changed. Cache should have been ignored'
+        assert bn.call_count == 6, 'call count should have changed. Cache should have been ignored'
 
 
 @pytest.mark.parametrize('added_exchanges', [('binance', 'poloniex')])
diff --git a/rotkehlchen/tests/exchanges/test_binance.py b/rotkehlchen/tests/exchanges/test_binance.py
--- a/rotkehlchen/tests/exchanges/test_binance.py
+++ b/rotkehlchen/tests/exchanges/test_binance.py
@@ -24,9 +24,8 @@
 from rotkehlchen.fval import FVal
 from rotkehlchen.tests.utils.constants import A_BNB, A_BUSD, A_RDN, A_USDT, A_XMR
 from rotkehlchen.tests.utils.exchanges import (
-    BINANCE_BALANCES_RESPONSE,
-    BINANCE_FUTURES_WALLET_RESPONSE,
     BINANCE_MYTRADES_RESPONSE,
+    mock_binance_balance_response,
 )
 from rotkehlchen.tests.utils.factories import make_api_key, make_api_secret
 from rotkehlchen.tests.utils.mock import MockResponse
@@ -195,47 +194,18 @@ def test_binance_assets_are_known(
     assert asset_from_binance('LDUSDT') == Asset('USDT')
 
 
-def test_binance_query_balances_unknown_asset(function_scope_binance):
-    """Test that if a binance balance query returns unknown asset no exception
-    is raised and a warning is generated. Same for unsupported asset."""
-    binance = function_scope_binance
-
-    def mock_unknown_asset_return(url):  # pylint: disable=unused-argument
-        return MockResponse(200, BINANCE_BALANCES_RESPONSE)
-
-    with patch.object(binance.session, 'get', side_effect=mock_unknown_asset_return):
-        # Test that after querying the assets only ETH and BTC are there
-        balances, msg = binance.query_balances()
-
-    assert msg == ''
-    assert len(balances) == 2
-    assert balances[A_BTC]['amount'] == FVal('4723846.89208129')
-    assert balances[A_ETH]['amount'] == FVal('4763368.68006011')
-
-    warnings = binance.msg_aggregator.consume_warnings()
-    assert len(warnings) == 2
-    assert 'unknown binance asset IDONTEXIST' in warnings[0]
-    assert 'unsupported binance asset ETF' in warnings[1]
-
-
 def test_binance_query_balances_include_features(function_scope_binance):
     """Test that querying binance balances includes the futures wallet"""
     binance = function_scope_binance
-
-    def mock_binance_response(url):  # pylint: disable=unused-argument
-        if 'futures' in url:
-            return MockResponse(200, BINANCE_FUTURES_WALLET_RESPONSE)
-        # else
-        return MockResponse(200, BINANCE_BALANCES_RESPONSE)
-
-    with patch.object(binance.session, 'get', side_effect=mock_binance_response):
+    with patch.object(binance.session, 'get', side_effect=mock_binance_balance_response):
         balances, msg = binance.query_balances()
 
     assert msg == ''
-    assert len(balances) == 3
+    assert len(balances) == 4
     assert balances[A_BTC]['amount'] == FVal('4723846.89208129')
     assert balances[A_ETH]['amount'] == FVal('4763368.68006011')
-    assert balances[A_BUSD]['amount'] == FVal('5.82211108')
+    assert balances[A_BUSD]['amount'] == FVal('7.49283144')
+    assert balances[A_USDT]['amount'] == FVal('75.46')
 
     warnings = binance.msg_aggregator.consume_warnings()
     assert len(warnings) == 2
diff --git a/rotkehlchen/tests/utils/exchanges.py b/rotkehlchen/tests/utils/exchanges.py
--- a/rotkehlchen/tests/utils/exchanges.py
+++ b/rotkehlchen/tests/utils/exchanges.py
@@ -170,6 +170,29 @@
     ]
 }"""
 
+BINANCE_LENDING_WALLET_RESPONSE = """{
+    "positionAmountVos": [
+        {
+            "amount": "75.46000000",
+            "amountInBTC": "0.01044819",
+            "amountInUSDT": "75.46000000",
+            "asset": "USDT"
+        },
+        {
+            "amount": "1.67072036",
+            "amountInBTC": "0.00023163",
+            "amountInUSDT": "1.67289230",
+            "asset": "BUSD"
+        }
+    ],
+    "totalAmountInBTC": "0.01067982",
+    "totalAmountInUSDT": "77.13289230",
+    "totalFixedAmountInBTC": "0.00000000",
+    "totalFixedAmountInUSDT": "0.00000000",
+    "totalFlexibleInBTC": "0.01067982",
+    "totalFlexibleInUSDT": "77.13289230"
+}"""
+
 BINANCE_MYTRADES_RESPONSE = """
 [
     {
@@ -202,9 +225,24 @@ def assert_poloniex_balances_result(balances: Dict[str, Any]) -> None:
     assert balances['ETH']['usd_value'] is not None
 
 
+def mock_binance_balance_response(url):
+    if 'futures' in url:
+        return MockResponse(200, BINANCE_FUTURES_WALLET_RESPONSE)
+    if 'lending' in url:
+        return MockResponse(200, BINANCE_LENDING_WALLET_RESPONSE)
+    # else
+    return MockResponse(200, BINANCE_BALANCES_RESPONSE)
+
+
 def patch_binance_balances_query(binance: 'Binance'):
     def mock_binance_asset_return(url, *args):  # pylint: disable=unused-argument
-        return MockResponse(200, BINANCE_BALANCES_RESPONSE)
+        if 'futures' in url:
+            response = '{"crossCollaterals":[]}'
+        elif 'lending' in url:
+            response = '{"positionAmountVos":[]}'
+        else:
+            response = BINANCE_BALANCES_RESPONSE
+        return MockResponse(200, response)
 
     binance_patch = patch.object(binance.session, 'get', side_effect=mock_binance_asset_return)
     return binance_patch