Repository: rotki/rotki

Issue:
Missing key entry for 'limit' for Bittrex trades
## Problem Definition

I'm currently using rotki from source branch: develop. I've added my bittrex API key and I'm doing a tax calculation for the year 2018. Sometimes, I now get the following error message from the UI:

```
Error processing a bittrex trade. Check logs for details. Ignoring it.
```

When I look into the logs, I get 

```
[06/08/2020 15:23:54 CEST] rotkehlchen.exchanges.bittrex: Error processing a bittrex trade trade={ trade object stuff}, error=Missing key entry for 'limit'
```

```
uname -a
> Darwin kazoo 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64
[ -d .git ] && git rev-parse HEAD
> dcea1fb3c13bcca64ca85a10bebf329d9ebf7c04
```



Discussion:
Hmmm thanks a lot for this report and for using the development branch.

Can you tell me what does the trade object show? Just give fake numbers. I need to know the format. I have few trades in bittrex and it works fine for me.
```
[06/08/2020 15:23:54 CEST] rotkehlchen.exchanges.bittrex: Error processing a bittrex trade trade={'id': 'abc', 'marketSymbol': 'A-BTC', 'direction': 'SELL', 'type': 'MARKET', 'quantity': FVal(420.0), 'timeInForce': 'IMMEDIATE_OR_CANCEL', 'fillQuantity': FVal(420.0), 'commission': FVal(0.00000689), 'proceeds': FVal(0.00010221), 'status': 'CLOSED', 'createdAt': '2020-04-27T12:00:00.00', 'updatedAt': '2020-04-27T12:00:00.00', 'closedAt': '2020-04-27T12:00:00.00'}, error=Missing key entry for 'limit'.
[06/08/2020 15:23:54 CEST] rotkehlchen.user_messages: Error processing a bittrex trade. Check logs for details. Ignoring it.
[06/08/2020 15:23:54 CEST] rotkehlchen.exchanges.bittrex: Error processing a bittrex trade trade={'id': 'l0000l', 'marketSymbol': 'A-BTC', 'direction': 'SELL', 'type': 'MARKET', 'quantity': FVal(1337.0), 'timeInForce': 'IMMEDIATE_OR_CANCEL', 'fillQuantity': FVal(1337.0), 'commission': FVal(0.0000213123), 'proceeds': FVal(0.0123123), 'status': 'CLOSED', 'createdAt': '2020-04-27T14:00:00.0Z', 'updatedAt': '2020-04-27T14:00:00.0Z', 'closedAt': '2020-04-27T14:00:00.0Z'}, error=Missing key entry for 'limit'.
[06/08/2020 15:23:54 CEST] rotkehlchen.user_messages: Error processing a bittrex trade. Check logs for details. Ignoring it.
```

I've changed the values to random stuff.
Oh I see. All right, now that makes sense. I can fix it fast for you. And you saved us from a bug in v1.6.2. You are awesome!

See, the trades I had in bittrex were all limit trades. The problematic trades here are market trades. The bittrex API docs are not specifying it well, but after looking deeper `limit` is only in limit orders and for market orders there is no way to get the price. Have to calculate it from the other fields.

Let me make a PR to fix this. Since you play in develop you will get the fix soon.

Resolving patch:
diff --git a/rotkehlchen/exchanges/bittrex.py b/rotkehlchen/exchanges/bittrex.py
--- a/rotkehlchen/exchanges/bittrex.py
+++ b/rotkehlchen/exchanges/bittrex.py
@@ -46,6 +46,7 @@
     AssetMovementCategory,
     Fee,
     Location,
+    Price,
     Timestamp,
     TradePair,
 )
@@ -103,14 +104,25 @@ def trade_from_bittrex(bittrex_trade: Dict[str, Any]) -> Trade:
     """Turn a bittrex trade returned from bittrex trade history to our common trade
     history format
 
+    As we saw in https://github.com/rotki/rotki/issues/1281 it's quite possible
+    that some keys don't exist in a trade. The required fields are here:
+    https://bittrex.github.io/api/v3#definition-Order
+
     Throws:
         - UnknownAsset/UnsupportedAsset due to bittrex_pair_to_world()
         - DeserializationError due to unexpected format of dict entries
         - KeyError due to dict entries missing an expected entry
     """
-    amount = deserialize_asset_amount(bittrex_trade['quantity'])
-    timestamp = deserialize_timestamp_from_date(bittrex_trade['closedAt'], 'iso8601', 'bittrex')
-    rate = deserialize_price(bittrex_trade['limit'])
+    amount = deserialize_asset_amount(bittrex_trade['fillQuantity'])
+    timestamp = deserialize_timestamp_from_date(
+        date=bittrex_trade['closedAt'],  # we only check closed orders
+        formatstr='iso8601',
+        location='bittrex',
+    )
+    if 'limit' in bittrex_trade:
+        rate = deserialize_price(bittrex_trade['limit'])
+    else:
+        rate = Price(FVal(bittrex_trade['proceeds']) / FVal(bittrex_trade['fillQuantity']))
     order_type = deserialize_trade_type(bittrex_trade['direction'])
     fee = deserialize_fee(bittrex_trade['commission'])
     pair = bittrex_pair_to_world(bittrex_trade['marketSymbol'])
@@ -431,17 +443,21 @@ def _deserialize_asset_movement(self, raw_data: Dict[str, Any]) -> Optional[Asse
         Can log error/warning and return None if something went wrong at deserialization
         """
         try:
+            if raw_data['status'] != 'COMPLETED':
+                # Don't mind failed/in progress asset movements
+                return None
+
             if 'source' in raw_data:
                 category = AssetMovementCategory.DEPOSIT
                 fee = Fee(ZERO)
             else:
                 category = AssetMovementCategory.WITHDRAWAL
-                fee = deserialize_fee(raw_data['txCost'])
+                fee = deserialize_fee(raw_data.get('txCost', 0))
 
             timestamp = deserialize_timestamp_from_date(
-                raw_data['completedAt'],
-                'iso8601',
-                'bittrex',
+                date=raw_data['completedAt'],  # we only check completed orders
+                formatstr='iso8601',
+                location='bittrex',
             )
             asset = asset_from_bittrex(raw_data['currencySymbol'])
             return AssetMovement(
@@ -452,7 +468,7 @@ def _deserialize_asset_movement(self, raw_data: Dict[str, Any]) -> Optional[Asse
                 amount=deserialize_asset_amount(raw_data['quantity']),
                 fee_asset=asset,
                 fee=fee,
-                link=str(raw_data['txId']),
+                link=str(raw_data.get('txId', '')),
             )
         except UnknownAsset as e:
             self.msg_aggregator.add_warning(


Test patch:
diff --git a/rotkehlchen/tests/exchanges/test_bittrex.py b/rotkehlchen/tests/exchanges/test_bittrex.py
--- a/rotkehlchen/tests/exchanges/test_bittrex.py
+++ b/rotkehlchen/tests/exchanges/test_bittrex.py
@@ -92,18 +92,31 @@ def mock_unknown_asset_return(method, url, json):  # pylint: disable=unused-argu
     assert 'unsupported bittrex asset PTON' in warnings[1]
 
 
-BITTREX_ORDER_HISTORY_RESPONSE = """
-[
+BITTREX_LIMIT_TRADE = """
     {
       "id": "fd97d393-e9b9-4dd1-9dbf-f288fc72a185",
       "marketSymbol": "BTC-LTC",
       "direction": "BUY",
       "type": "LIMIT",
-      "quantity": 667.03644955,
+      "fillQuantity": 667.03644955,
       "limit": 0.0000295,
+      "proceeds": 0.0196775752617,
       "commission": 0.00004921,
       "closedAt": "2014-02-13T00:00:00.00Z"
-    }]"""
+    }"""
+
+BITTREX_MARKET_TRADE = """
+    {
+      "id": "ad97d393-19b9-6dd1-9dbf-f288fc72a185",
+      "marketSymbol": "BTC-ETH",
+      "direction": "SELL",
+      "type": "MARKET",
+      "fillQuantity": 2,
+      "proceeds": 10,
+      "commission": 0.00001,
+      "closedAt": "2014-02-13T00:00:00.00Z"
+    }"""
+BITTREX_ORDER_HISTORY_RESPONSE = f'[{BITTREX_LIMIT_TRADE}, {BITTREX_MARKET_TRADE}]'
 
 
 def test_bittrex_query_trade_history(bittrex):
@@ -116,7 +129,7 @@ def mock_order_history(url, method, json):  # pylint: disable=unused-argument
     with patch.object(bittrex.session, 'request', side_effect=mock_order_history):
         trades = bittrex.query_trade_history(start_ts=0, end_ts=1564301134)
 
-    expected_trade = Trade(
+    expected_trades = [Trade(
         timestamp=1392249600,
         location=Location.BITTREX,
         pair='LTC_BTC',
@@ -126,10 +139,18 @@ def mock_order_history(url, method, json):  # pylint: disable=unused-argument
         fee=FVal('0.00004921'),
         fee_currency=A_BTC,
         link='fd97d393-e9b9-4dd1-9dbf-f288fc72a185',
-    )
-
-    assert len(trades) == 1
-    assert trades[0] == expected_trade
+    ), Trade(
+        timestamp=1392249600,
+        location=Location.BITTREX,
+        pair='ETH_BTC',
+        trade_type=TradeType.SELL,
+        amount=FVal('2'),
+        rate=FVal('5'),
+        fee=FVal('0.00001'),
+        fee_currency=A_BTC,
+        link='ad97d393-19b9-6dd1-9dbf-f288fc72a185',
+    )]
+    assert expected_trades == trades
 
 
 def test_bittrex_query_trade_history_unexpected_data(bittrex):
@@ -167,45 +188,46 @@ def query_bittrex_and_test(
         if error_str_test:
             assert error_str_test in errors[0]
 
-    input_str = BITTREX_ORDER_HISTORY_RESPONSE.replace(
-        '"quantity": 667.03644955',
-        '"quantity": "fdfdsf"',
+    history = f'[{BITTREX_LIMIT_TRADE}]'
+    input_str = history.replace(
+        '"fillQuantity": 667.03644955',
+        '"fillQuantity": "fdfdsf"',
     )
     query_bittrex_and_test(input_str, expected_warnings_num=0, expected_errors_num=1)
 
-    input_str = BITTREX_ORDER_HISTORY_RESPONSE.replace(
+    input_str = history.replace(
         '"closedAt": "2014-02-13T00:00:00.00Z"',
         '"closedAt": null',
     )
     query_bittrex_and_test(input_str, expected_warnings_num=0, expected_errors_num=1)
 
-    input_str = BITTREX_ORDER_HISTORY_RESPONSE.replace(
+    input_str = history.replace(
         '"limit": 0.0000295',
         '"limit": "sdad"',
     )
     query_bittrex_and_test(input_str, expected_warnings_num=0, expected_errors_num=1)
 
-    input_str = BITTREX_ORDER_HISTORY_RESPONSE.replace(
+    input_str = history.replace(
         '"direction": "BUY"',
         '"direction": "dsadsd"',
     )
     query_bittrex_and_test(input_str, expected_warnings_num=0, expected_errors_num=1)
 
-    input_str = BITTREX_ORDER_HISTORY_RESPONSE.replace(
+    input_str = history.replace(
         '"commission": 0.00004921',
         '"commission": "dasdsad"',
     )
     query_bittrex_and_test(input_str, expected_warnings_num=0, expected_errors_num=1)
 
     # Check that for non-string pairs we give a graceful error
-    input_str = BITTREX_ORDER_HISTORY_RESPONSE.replace(
+    input_str = history.replace(
         '"marketSymbol": "BTC-LTC"',
         '"marketSymbol": 4324234',
     )
     query_bittrex_and_test(input_str, expected_warnings_num=0, expected_errors_num=1)
 
     # Check that for unsupported assets in the pair are caught
-    input_str = BITTREX_ORDER_HISTORY_RESPONSE.replace(
+    input_str = history.replace(
         '"marketSymbol": "BTC-LTC"',
         '"marketSymbol": "BTC-PTON"',
     )
@@ -217,7 +239,7 @@ def query_bittrex_and_test(
     )
 
     # Check that unprocessable pair is caught
-    input_str = BITTREX_ORDER_HISTORY_RESPONSE.replace(
+    input_str = history.replace(
         '"marketSymbol": "BTC-LTC"',
         '"marketSymbol": "SSSS"',
     )
@@ -233,6 +255,7 @@ def query_bittrex_and_test(
 [
     {
       "id": 1,
+      "status": "COMPLETED",
       "quantity": 2.12345678,
       "currencySymbol": "BTC",
       "confirmations": 2,
@@ -241,7 +264,8 @@ def query_bittrex_and_test(
       "cryptoAddress": "15VyEAT4uf7ycrNWZVb1eGMzrs21BH95Va",
       "source": "foo"
     }, {
-      "Id": 2,
+      "id": 2,
+      "status": "COMPLETED",
       "quantity": 50.81,
       "currencySymbol": "ETH",
       "confirmations": 5,
@@ -257,6 +281,7 @@ def query_bittrex_and_test(
 [
     {
       "id": "b52c7a5c-90c6-4c6e-835c-e16df12708b1",
+      "status": "COMPLETED",
       "currencySymbol": "BTC",
       "quantity": 17,
       "cryptoAddress": "1DeaaFBdbB5nrHj87x3NHS4onvw1GPNyAu",
@@ -265,6 +290,7 @@ def query_bittrex_and_test(
       "txId": "b4a575c2a71c7e56d02ab8e26bb1ef0a2f6cf2094f6ca2116476a569c1e84f6e"
     }, {
       "id": "b52c7a5c-90c6-4c6e-835c-e16df12708b1",
+      "status": "COMPLETED",
       "currencySymbol": "ETH",
       "quantity": 55,
       "cryptoAddress": "0x717E2De923A6377Fbd7e3c937491f71ad370e9A8",
@@ -465,6 +491,7 @@ def check_permutations_of_input_invalid_data(deposits, withdrawals):
     empty_response = '[]'
     input_withdrawals = """[{
       "id": "b52c7a5c-90c6-4c6e-835c-e16df12708b1",
+      "status": "COMPLETED",
       "currencySymbol": "BTC",
       "quantity": 17,
       "cryptoAddress": "1DeaaFBdbB5nrHj87x3NHS4onvw1GPNyAu",
@@ -479,6 +506,7 @@ def check_permutations_of_input_invalid_data(deposits, withdrawals):
 
     input_deposits = """[{
       "id": 1,
+      "status": "COMPLETED",
       "quantity": 17,
       "currencySymbol": "BTC",
       "confirmations": 2,
diff --git a/rotkehlchen/tests/utils/history.py b/rotkehlchen/tests/utils/history.py
--- a/rotkehlchen/tests/utils/history.py
+++ b/rotkehlchen/tests/utils/history.py
@@ -318,7 +318,8 @@ def mock_bittrex_api_queries(url, method, json):  # pylint: disable=unused-argum
       "marketSymbol": "BTC-LTC",
       "closedAt": "2017-05-01T15:00:00.00Z",
       "direction": "BUY",
-      "quantity": 667.03644955,
+      "type": "LIMIT",
+      "fillQuantity": 667.03644955,
       "limit": 0.0000295,
       "commission": 0.00004921
     }, {
@@ -326,7 +327,8 @@ def mock_bittrex_api_queries(url, method, json):  # pylint: disable=unused-argum
       "marketSymbol": "ETH-LTC",
       "closedAt": "2017-05-02T15:00:00.00Z",
       "direction": "SELL",
-      "quantity": 667.03644955,
+      "type": "LIMIT",
+      "fillQuantity": 667.03644955,
       "commission": 0.00004921,
       "limit": 0.0000295
     }, {
@@ -334,7 +336,8 @@ def mock_bittrex_api_queries(url, method, json):  # pylint: disable=unused-argum
       "marketSymbol": "PTON-ETH",
       "closedAt": "2017-05-02T15:00:00.00Z",
       "direction": "SELL",
-      "quantity": 667.03644955,
+      "type": "LIMIT",
+      "fillQuantity": 667.03644955,
       "commission": 0.00004921,
       "limit": 0.0000295
     }, {
@@ -342,7 +345,8 @@ def mock_bittrex_api_queries(url, method, json):  # pylint: disable=unused-argum
       "marketSymbol": "ETH-IDONTEXIST",
       "closedAt": "2017-05-02T15:00:00.00Z",
       "direction": "SELL",
-      "quantity": 667.03644955,
+      "type": "LIMIT",
+      "fillQuantity": 667.03644955,
       "commission": 0.00004921,
       "limit": 0.0000295
     }, {
@@ -350,7 +354,8 @@ def mock_bittrex_api_queries(url, method, json):  # pylint: disable=unused-argum
       "marketSymbol": "%$#%$#%#$%",
       "closedAt": "2017-05-02T15:00:00.00Z",
       "direction": "BUY",
-      "quantity": 667.03644955,
+      "type": "LIMIT",
+      "fillQuantity": 667.03644955,
       "commission": 0.00004921,
       "limit": 0.0000295
 }]