Repository: rotki/rotki

Issue:
Importing IOTA trades from bitcoin.de throw errors
## Problem Definition

On importing exchange data from bitcoin.de an error is thrown:
"Query trades of bitcoinde exchange died with exception: Unknown asset IOT provided.. Check the logs for more details"
This is probably the same issue as in https://github.com/rotki/rotki/issues/299 -> IOT needs to be mapped to IOTA

## Logs

The log is empty.


### System Description


Operating system: 
Rotki version: 1.12.2



Discussion:
Hey @franzkugelmann thanks a lot for the bug report!

 Sounds like a mapping issue. Will get it fixed and include it in the next release.

> The log is empty.

Sidenote for that: We probably need to include it in the issue template but the log is always empty by default. You would need to activate the debug log mode as per the [docs](https://rotki.readthedocs.io/en/latest/contribute.html#id2) in order to get anything in the logs.

The reason is simple. The logs are unencrypted and when we had them on by default many of our users complained that sensitive data were kept unencrypted and in plain text in their system. So they are now off by default.

Here probably logs are not needed though, as it does indeed sound like a mapping issue.

-----

re: testing

I don't have any bitcoin.de account with trades so can't do a real integration test it so you would be the one to verify the fix works at the end. I can also provide you with a nightly once fixed so you can try it before the release if you would like. Best way to communicate would be via our discord: https://discord.gg/aGCxHG7 but can also use this issue.


----

@coinyon another note here. For other exchanges we have mapping tests to make sure such errors never hit our users and that there is a properly mapped asset for each exchange asset. Such as [here](https://github.com/rotki/rotki/blob/9aebabfe58c7f0385f6d5fcf7123e68ce7644adb/rotkehlchen/tests/exchanges/test_binance.py#L163-L191) for binance.

Can we make such a test for bitcoin.de too? I can't find an api call that can accomplish that when looking at their [API docs](https://www.bitcoin.de/de/api/tapi/v4/docu) but perhaps that's due to my non-native German knowledge.

But what's strange is that [here](https://www.bitcoin.de/en/api/tapi/v4/docu#handelspaarliste_c2f) in the docs they have a list of pairs and there `IOTA` appears as `IOTA` and not `IOT`.
@LefterisJP Yea a test like that is included here: https://github.com/rotki/rotki/blob/9aebabfe58c7f0385f6d5fcf7123e68ce7644adb/rotkehlchen/tests/exchanges/test_bitcoinde.py#L80

I could find any API endpoint either so I hardcoded the pairs. IOTA support is still missing.
Ah ok, I only added the EUR pairs from the docs. Will add the BTC pairs as well.
> Ah ok, I only added the EUR pairs from the docs. Will add the BTC pairs as well.

Thanks :+1: 

> I could find any API endpoint either so I hardcoded the pairs. IOTA support is still missing.

What is missing for it? Just the mapping? That should be trivial to add it it's just (bitcoin.de)IOT -> IOTA (rotki)

Resolving patch:
diff --git a/rotkehlchen/exchanges/bitcoinde.py b/rotkehlchen/exchanges/bitcoinde.py
--- a/rotkehlchen/exchanges/bitcoinde.py
+++ b/rotkehlchen/exchanges/bitcoinde.py
@@ -10,7 +10,7 @@
 from typing_extensions import Literal
 
 from rotkehlchen.assets.asset import Asset
-from rotkehlchen.errors import RemoteError
+from rotkehlchen.errors import DeserializationError, RemoteError, UnknownAsset
 from rotkehlchen.exchanges.data_structures import (
     AssetMovement,
     Location,
@@ -41,7 +41,8 @@
 # This corresponds to md5('') and is used in signature generation
 MD5_EMPTY_STR = 'd41d8cd98f00b204e9800998ecf8427e'
 
-# Pairs can be found in Basic API doc: https://www.bitcoin.de/de/api/marketplace
+# Pairs can be found in Basic API doc:
+# https://www.bitcoin.de/en/api/tapi/v4/docu#handelspaarliste_c2f
 BITCOINDE_TRADING_PAIRS = (
     'btceur',
     'bcheur',
@@ -49,6 +50,10 @@
     'etheur',
     'bsveur',
     'ltceur',
+    'iotabtc',
+    'dashbtc',
+    'gntbtc',
+    'ltcbtc',
 )
 
 
@@ -57,8 +62,14 @@ def bitcoinde_asset(asset: str) -> Asset:
 
 
 def bitcoinde_pair_to_world(pair: str) -> Tuple[Asset, Asset]:
-    tx_asset = bitcoinde_asset(pair[:3])
-    native_asset = bitcoinde_asset(pair[3:])
+    if len(pair) == 6:
+        tx_asset = bitcoinde_asset(pair[:3])
+        native_asset = bitcoinde_asset(pair[3:])
+    elif len(pair) in (7, 8):
+        tx_asset = bitcoinde_asset(pair[:4])
+        native_asset = bitcoinde_asset(pair[4:])
+    else:
+        raise DeserializationError(f'Could not parse pair: {pair}')
     return tx_asset, native_asset
 
 
@@ -277,7 +288,28 @@ def query_online_trade_history(
                 continue
             if timestamp < start_ts or timestamp > end_ts:
                 continue
-            trades.append(trade_from_bitcoinde(tx))
+            try:
+                trades.append(trade_from_bitcoinde(tx))
+            except UnknownAsset as e:
+                self.msg_aggregator.add_warning(
+                    f'Found bitcoin.de trade with unknown asset '
+                    f'{e.asset_name}. Ignoring it.',
+                )
+                continue
+            except (DeserializationError, KeyError) as e:
+                msg = str(e)
+                if isinstance(e, KeyError):
+                    msg = f'Missing key entry for {msg}.'
+                self.msg_aggregator.add_error(
+                    'Error processing a Bitcoin.de trade. Check logs '
+                    'for details. Ignoring it.',
+                )
+                log.error(
+                    'Error processing a Bitcoin.de trade',
+                    trade=tx,
+                    error=msg,
+                )
+                continue
 
         return trades
 


Test patch:
diff --git a/rotkehlchen/tests/exchanges/test_bitcoinde.py b/rotkehlchen/tests/exchanges/test_bitcoinde.py
--- a/rotkehlchen/tests/exchanges/test_bitcoinde.py
+++ b/rotkehlchen/tests/exchanges/test_bitcoinde.py
@@ -1,7 +1,10 @@
 from unittest.mock import patch
 
+import pytest
+
 from rotkehlchen.assets.asset import Asset
 from rotkehlchen.constants.assets import A_BTC, A_ETH, A_EUR
+from rotkehlchen.errors import DeserializationError, UnknownAsset
 from rotkehlchen.exchanges.bitcoinde import (
     BITCOINDE_TRADING_PAIRS,
     Bitcoinde,
@@ -80,3 +83,11 @@ def mock_api_return(url, **kwargs):  # pylint: disable=unused-argument
 def test_bitcoinde_trading_pairs():
     for pair in BITCOINDE_TRADING_PAIRS:
         _ = bitcoinde_pair_to_world(pair)
+
+
+def test_bitcoinde_invalid_trading_pair():
+    with pytest.raises(UnknownAsset):
+        _ = bitcoinde_pair_to_world('000btc')
+
+    with pytest.raises(DeserializationError):
+        _ = bitcoinde_pair_to_world('invalidpair')