Repository: rotki/rotki

Issue:
BItstamp "Failed to deserialize a Bitstamp asset movement" for Songbird (SGB) token.
## Problem Definition

Hello, while syncing my Bitstamp account I do get an *Failed to deserialize a Bitstamp asset movement. Check logs for details. Ignoring it.* notification. This is for some Songbird tokens I got Airdropped. I had a look at your code and believe that adding `sgb` to `BITSTAMP_ASSET_MOVEMENT_SYMBOLS`  in `/rotkehlchen/exchanges/bitstamp.py` (around line #95) should probably fix this. 

```python
# from https://www.bitstamp.net/api/#user-transactions
BITSTAMP_ASSET_MOVEMENT_SYMBOLS = (
    'usd',
    'eur',
    'btc',
    'xrp',
    'gbp',
    'ltc',
    'eth',
    'bch',
    'xlm',
    'pax',
    'link',
    'omg',
    'usdc',
)
```

I'm happy to submit a pull request but noticed that you link on line #94 to a list of [assets](https://www.bitstamp.net/api/#user-transactions) from which the list below is probably derived. However, the list is much longer than in you code. I wonder if it would be worthwhile to add all listed an missing tokens or just sgb? 

Thanks!

## Logs

```
[03/07/2022 14:24:36 CEST] DEBUG rotkehlchen.tasks.manager Greenlet-1: At task scheduling. Current greenlets: 1 Max greenlets: 2. Will schedule.
[03/07/2022 14:24:36 CEST] DEBUG rotkehlchen.tasks.manager Greenlet-1: Scheduling task to Query history of bitstamp exchange
[03/07/2022 14:24:36 CEST] DEBUG rotkehlchen.exchanges.exchange Greenlet-3: Querying trade history for bitstamp exchange
[03/07/2022 14:24:36 CEST] DEBUG rotkehlchen.exchanges.exchange Greenlet-3: Querying online trade history for bitstamp between 1656850460 and 1656851076
[03/07/2022 14:24:36 CEST] DEBUG rotkehlchen.exchanges.bitstamp Greenlet-3: Bitstamp API request request_url=https://www.bitstamp.net/api/v2/user_transactions/, options={'limit': 1000, 'since_id': 215294571, 'sort': 'asc', 'offset': 0}

[03/07/2022 14:24:37 CEST] DEBUG rotkehlchen.exchanges.bitstamp Greenlet-3: Attempting to deserialize bitstamp asset movement: {'fee': '0.00000000', 'btc_usd': '0.00', 'datetime': '2021-12-08 10:01:12.340061', 'usd': 0.0, 'btc': 0.0, 'sgb': '3.74697800', 'type': '0', 'id': 6888588014, 'eur': 0.0}
[03/07/2022 14:24:37 CEST] ERROR rotkehlchen.exchanges.bitstamp Greenlet-3: Error processing a Bitstamp asset movement. raw_result={'fee': '0.00000000', 'btc_usd': '0.00', 'datetime': '2021-12-08 10:01:12.340061', 'usd': 0.0, 'btc': 0.0, 'sgb': '3.74697800', 'type': '0', 'id': 6888588014, 'eur': 0.0}, error=Could not deserialize Bitstamp asset movement from user transaction. Unexpected asset amount combination found in: {'fee': '0.00000000', 'btc_usd': '0.00', 'datetime': '2021-12-08 10:01:12.340061', 'usd': 0.0, 'btc': 0.0, 'sgb': '3.74697800', 'type': '0', 'id': 6888588014, 'eur': 0.0}.
[03/07/2022 14:24:37 CEST] ERROR rotkehlchen.user_messages Greenlet-3: Failed to deserialize a Bitstamp asset movement. Check logs for details. Ignoring it.
```


### System Description
Operating system: MacOs
Rotki version: 1.24.1




Discussion:
Hello @novazembla ! Yes, that list should include all the values listed on the endpoint documentation. If you could create a PR for that it would be great. We have a discord server if you need help in the process https://discord.gg/aGCxHG7
@novazembla As pointed by lefteris in discord if the asset doesn't properly resolve we could have an unexpected error. The only one that is not there as per what I checked is `ETH2R`, do you know what is this asset? There is a token but seems to be inactive and very old.
@yabirgb thanks for the quick reply. Are you sure that we're looking at the same "list"?. I've been poking into `/rotkehlchen/exchanges/bitstamp.py` and found that the `_deserialize_asset_movement` function is using the following loop to go through a fixed list of assets. 

```python
      ...
      for symbol in BITSTAMP_ASSET_MOVEMENT_SYMBOLS:
            amount = deserialize_asset_amount(raw_movement.get(symbol, '0'))
            if amount != ZERO:
                fee_asset = asset_from_bitstamp(symbol)
                break

    ...
```
[L556 - 560](https://github.com/rotki/rotki/blob/develop/rotkehlchen/exchanges/bitstamp.py#L96-L110)

The constant is defined further up `BITSTAMP_ASSET_MOVEMENT_SYMBOLS` [L96-110](https://github.com/rotki/rotki/blob/develop/rotkehlchen/exchanges/bitstamp.py#L96-L110).

I skimmed through the other "exchanges" and couldn't find any similar implementation and might be due to the Bitstamp's API requirements/structure. 

I'm happy to update that list if it is actually time well spent. 




  
Hello @novazembla sorry for the late response. So as you saw what this code is doing is iterating over the keys of the json response and detect if the balance is not zero. Also as you said the approach here is unique but is due to the structure of the bitstamp json response. What we could do here is change the approach to avoid having to manually maintain the structure of this module and iterate over all the keys trying to detect if it is a known asset or not.

So my idea here is that you create a set of the known fields of the response that we know they are not assets and have something like

```python
amount = ZERO # (to raise DeserializationError if we doin't find it)
for json_key, value in raw_movement.items():
    if json_key in KNOWN_NON_ASSET_KEYS:
       continue
    # try to get the amount and it doesn't raise deserialization error and is not ZERO check if the asset is known
    try:
        candicate_fee_asset = asset_from_bitstamp(json_key)
    except UnknownAsset:
        # just skipping I believe is fine
   # change the fee_asset and break the loop
```

What do you think about this approach? I believe will improve the experience here and be better than mantaining that list. Also please could you test that the approach works for you and make some tests with faked data for this?

If you need any help in the process let me know and I'll answer asap. This days I was in ethbarcelona and couldn't manage to answer earlier
Also @novazembla can you check the response with your key? We don't know if the response contains all the entries or just the one used. If it is all it's a bit messy. if you can also share that info would be great. 
@yabirgb no worries about slow replies. We're all busy people :)

I've got only a few transactions entries on Bitstamp, so "even" my API access is probably not too useful for testing purposes. I just happened to stumble of the Songbird issue. 

The transaction record that triggers the error is the following. 

```json
{
  "fee": "0.00000000",
  "btc_usd": "0.00",
  "datetime": "2022-01-08 23:25:12.366061",
  "usd": 0.0,
  "btc": 0.0,
  "sgb": "3.45697800",
  "type": "0",
  "id": 222687105,
  "eur": 0.0
}
```

as you can see they just dropped the `"sgb": "3.45697800",` in. So to answer your question. They just drop the used token in. 

As they add an exchange rate pair like the `btc_usd` (of which Bitstamp says that it might vary depending on the transaction) it might not be enough to add a `KNOWN_NON_ASSET_KEYS` as suggested. A simple solution might be to also skip any keys that contain an `_`. I do not know if token symbols could contain one. A look in your `all_assests.json` did not bring up one. So this might be good enough to skip that key. 

Bitstamp does unfortunately not offer an API endpoint that could be queried for all their active tokens/currencies. But I wonder if it would not be good to use the Rotki asset database as source for all possible keys. I know that the DB has quite a few records but not "that" many hence I dare to suggest the following. 

```python
 for asset_data in GlobalDBHandler().get_all_asset_data(mapping=False):
            amount = deserialize_asset_amount(raw_movement.get(asset_data.symbol, '0'))
            if amount != ZERO:
                fee_asset = asset_from_bitstamp(asset_data.symbol)
                break
```

What do you think?
> 

using all the asset in the globaldb is not a good idea, two main reasons:
1. This iterates over a list with many assets that are not in the exchange and are in the database making it slower since happens for every event
2. The symbol might not be the same, that is why we have mappings for most of the exchanges for some special cases.

So about iterating the keys those keys will just raise `DeserializationError` so you can just catch it. the reason to make the check is with that set was to avoid one extra function call + error raise. The `all_assets.json` is not used anymore and exists there just for compatibility with older versions

Resolving patch:
diff --git a/rotkehlchen/exchanges/bitstamp.py b/rotkehlchen/exchanges/bitstamp.py
--- a/rotkehlchen/exchanges/bitstamp.py
+++ b/rotkehlchen/exchanges/bitstamp.py
@@ -91,23 +91,12 @@
 USER_TRANSACTION_TRADE_TYPE = {2}
 # Asset movement type int: 0 - deposit, 1 - withdrawal
 USER_TRANSACTION_ASSET_MOVEMENT_TYPE = {0, 1}
-
-# from https://www.bitstamp.net/api/#user-transactions
-BITSTAMP_ASSET_MOVEMENT_SYMBOLS = (
-    'usd',
-    'eur',
-    'btc',
-    'xrp',
-    'gbp',
-    'ltc',
-    'eth',
-    'bch',
-    'xlm',
-    'pax',
-    'link',
-    'omg',
-    'usdc',
-)
+KNOWN_NON_ASSET_KEYS_FOR_MOVEMENTS = {
+    'datetime',
+    'id',
+    'type',
+    'fee',
+}
 
 
 class TradePairData(NamedTuple):
@@ -551,12 +540,21 @@ def _deserialize_asset_movement(
             raise AssertionError(f'Unexpected Bitstamp asset movement case: {type_}.')
 
         timestamp = deserialize_timestamp_from_bitstamp_date(raw_movement['datetime'])
-        amount: FVal
+        amount: FVal = ZERO
         fee_asset: Asset
-        for symbol in BITSTAMP_ASSET_MOVEMENT_SYMBOLS:
-            amount = deserialize_asset_amount(raw_movement.get(symbol, '0'))
+        for raw_movement_key, value in raw_movement.items():
+            if raw_movement_key in KNOWN_NON_ASSET_KEYS_FOR_MOVEMENTS:
+                continue
+            try:
+                candidate_fee_asset = asset_from_bitstamp(raw_movement_key)
+            except (UnknownAsset, DeserializationError):
+                continue
+            try:
+                amount = deserialize_asset_amount(value)
+            except DeserializationError:
+                continue
             if amount != ZERO:
-                fee_asset = asset_from_bitstamp(symbol)
+                fee_asset = candidate_fee_asset
                 break
 
         if amount == ZERO:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -38,6 +38,7 @@
             'data/uniswapv2_lp_tokens.meta',
             'data/global.db',
             'data/curve_pools.json',
+            'data/nodes.json',
             'chain/ethereum/modules/dxdaomesa/data/contracts.json',
         ],
     },


Test patch: