Repository: rotki/rotki

Issue:
Fix crypto.com data import
## Problem Definition

Running the current import script for crypto.com leads to import errors.

This ticket is to track fixing those errors.

## Logs

Unknown entrype type:

- invest_deposit
- airdrop_locked
- invest_withdrawal
- airdrop_to_exchange_transfer
- interest_swap_debited
- interest_swap_credited
- mco_stake_reward




Discussion:
Just a sanity check. This is from crypto.com mobile app? Because the import we support at the moment is only from their mobile app.
> Just a sanity check. This is from crypto.com mobile app? Because the import we support at the moment is only from their mobile app.

Yes it's the mobile app (not exchange).
Hey @CPatchane any idea what's wrong here? I don't have crypto.com mobile app so I don't have any data to test with.
Yep, I just don't have those types so no way I could handle them :P
I didn't find the full list of entry types used by crypto.com in their CSV, I just sent a message to the support to see if they want to give some help/docs on that 🤞 
I can provide test data over email if that helps?
@a7e8, is it possible to get those data anonymised? Just for the lines you said here. You can keep the rows order but fake the amount or/and the time or/and the currencies as you prefer (we will need a fake data anyway for the test in rotki).
Order is the same. Just the rows with errors listed above. `mco_stake_reward` has two rows because the currency is different.

```
Timestamp (UTC),Transaction Description,Currency,Amount,To Currency,To Amount,Native Currency,Native Amount,Native Amount (in USD),Transaction Kind
2021-01-01 22:00:00,CRO Stake Rewards,CRO,1,,,USD,1,1,mco_stake_reward
2021-01-01 22:00:00,MCO Earnings/Rewards Swap,CRO,1,,,USD,1,1,interest_swap_credited
2021-01-01 22:00:00,MCO Earnings/Rewards Swap,MCO,-0.1,,,USD,-1,-1,interest_swap_debited
2021-01-01 22:00:00,MCO Stake Rewards,MCO,0.5,,,USD,1,1,mco_stake_reward
2021-01-01 22:00:00,CRO Airdrop to Exchange,CRO,1,,,USD,-1,-1,airdrop_to_exchange_transfer
2021-01-01 22:00:00,Invest Withdrawal,BTC,0.01005,,,USD,100,100,invest_withdrawal
2021-01-01 22:00:00,CRO Airdrop,CRO,1,,,USD,1,1,airdrop_locked
2021-01-01 22:00:00,Invest Deposit,BTC,-0.01,,,USD,-100,-100,invest_deposit
```
Hey @yabirgb. Since you recently touched the data imports I am thinking if you could have a look at this.


Resolving patch:
diff --git a/rotkehlchen/data/importer.py b/rotkehlchen/data/importer.py
--- a/rotkehlchen/data/importer.py
+++ b/rotkehlchen/data/importer.py
@@ -1,3 +1,4 @@
+from collections import defaultdict
 import csv
 import functools
 import logging
@@ -21,7 +22,7 @@
     deserialize_fee,
     deserialize_timestamp_from_date,
 )
-from rotkehlchen.typing import AssetAmount, Fee, Location, Price, TradeType
+from rotkehlchen.typing import AssetAmount, Fee, Location, Price, TradeType, Timestamp
 
 log = logging.getLogger(__name__)
 
@@ -297,7 +298,54 @@ def _consume_cryptocom_entry(self, csv_row: Dict[str, Any]) -> None:
                 link='',
             )
             self.db.add_asset_movements([asset_movement])
-
+        elif row_type in ('airdrop_to_exchange_transfer', 'mco_stake_reward'):
+            asset = symbol_to_asset_or_token(csv_row['Currency'])
+            amount = deserialize_asset_amount(csv_row['Amount'])
+            action = LedgerAction(
+                identifier=0,  # whatever is not used at insertion
+                timestamp=timestamp,
+                action_type=LedgerActionType.INCOME,
+                location=Location.CRYPTOCOM,
+                amount=amount,
+                asset=asset,
+                rate=None,
+                rate_asset=None,
+                link=None,
+                notes=None,
+            )
+            self.db_ledger.add_ledger_action(action)
+        elif row_type == 'invest_deposit':
+            asset = symbol_to_asset_or_token(csv_row['Currency'])
+            amount = deserialize_asset_amount(csv_row['Amount'])
+            asset_movement = AssetMovement(
+                location=Location.CRYPTOCOM,
+                category=AssetMovementCategory.DEPOSIT,
+                address=None,
+                transaction_id=None,
+                timestamp=timestamp,
+                asset=asset,
+                amount=amount,
+                fee=fee,
+                fee_asset=fee_currency,
+                link='',
+            )
+            self.db.add_asset_movements([asset_movement])
+        elif row_type == 'invest_withdrawal':
+            asset = symbol_to_asset_or_token(csv_row['Currency'])
+            amount = deserialize_asset_amount(csv_row['Amount'])
+            asset_movement = AssetMovement(
+                location=Location.CRYPTOCOM,
+                category=AssetMovementCategory.WITHDRAWAL,
+                address=None,
+                transaction_id=None,
+                timestamp=timestamp,
+                asset=asset,
+                amount=amount,
+                fee=fee,
+                fee_asset=fee_currency,
+                link='',
+            )
+            self.db.add_asset_movements([asset_movement])
         elif row_type in (
             'crypto_earn_program_created',
             'crypto_earn_program_withdrawn',
@@ -321,6 +369,10 @@ def _consume_cryptocom_entry(self, csv_row: Dict[str, Any]) -> None:
             'dynamic_coin_swap_credited',
             'dust_conversion_debited',
             'dust_conversion_credited',
+            'interest_swap_credited',
+            'interest_swap_debited',
+            # The user has received an aidrop but can't claim it yet
+            'airdrop_locked',
         ):
             # those types are ignored because it doesn't affect the wallet balance
             # or are not handled here
@@ -344,6 +396,8 @@ def _import_cryptocom_associated_entries(self, data: Any, tx_kind: str) -> None:
         - KeyError if a row contains unexpected data entries
         """
         multiple_rows: Dict[Any, Dict[str, Any]] = {}
+        investments_deposits: Dict[str, List[Any]] = defaultdict(list)
+        investments_withdrawals: Dict[str, List[Any]] = defaultdict(list)
         debited_row = None
         credited_row = None
         for row in data:
@@ -368,6 +422,12 @@ def _import_cryptocom_associated_entries(self, data: Any, tx_kind: str) -> None:
                 if timestamp not in multiple_rows:
                     multiple_rows[timestamp] = {}
                 multiple_rows[timestamp]['credited'] = row
+            elif row['Transaction Kind'] == f'{tx_kind}_deposit':
+                asset = row['Currency']
+                investments_deposits[asset].append(row)
+            elif row['Transaction Kind'] == f'{tx_kind}_withdrawal':
+                asset = row['Currency']
+                investments_withdrawals[asset].append(row)
 
         for timestamp in multiple_rows:
             # When we convert multiple assets dust to CRO
@@ -380,7 +440,7 @@ def _import_cryptocom_associated_entries(self, data: Any, tx_kind: str) -> None:
                     acc +
                     deserialize_asset_amount(row['Native Amount (in USD)']),
                 debited_rows,
-                FVal('0'),
+                ZERO,
             )
 
             # If the value of the transaction is too small (< 0,01$),
@@ -430,6 +490,69 @@ def _import_cryptocom_associated_entries(self, data: Any, tx_kind: str) -> None:
                     )
                     self.db.add_trades([trade])
 
+        # Compute investments profit
+        if len(investments_withdrawals) != 0:
+            for asset in investments_withdrawals:
+                asset_object = symbol_to_asset_or_token(asset)
+                if asset not in investments_deposits:
+                    log.error(
+                        f'Investment withdrawal without deposit at crypto.com. Ignoring '
+                        f'staking info for asset {asset_object}',
+                    )
+                    continue
+                # Sort by date in ascending order
+                withdrawals_rows = sorted(
+                    investments_withdrawals[asset],
+                    key=lambda x: deserialize_timestamp_from_date(
+                        date=x['Timestamp (UTC)'],
+                        formatstr='%Y-%m-%d %H:%M:%S',
+                        location='cryptocom',
+                    ),
+                )
+                investments_rows = sorted(
+                    investments_deposits[asset],
+                    key=lambda x: deserialize_timestamp_from_date(
+                        date=x['Timestamp (UTC)'],
+                        formatstr='%Y-%m-%d %H:%M:%S',
+                        location='cryptocom',
+                    ),
+                )
+                last_date = Timestamp(0)
+                for withdrawal in withdrawals_rows:
+                    withdrawal_date = deserialize_timestamp_from_date(
+                        date=withdrawal['Timestamp (UTC)'],
+                        formatstr='%Y-%m-%d %H:%M:%S',
+                        location='cryptocom',
+                    )
+                    amount_deposited = ZERO
+                    for deposit in investments_rows:
+                        deposit_date = deserialize_timestamp_from_date(
+                            date=deposit['Timestamp (UTC)'],
+                            formatstr='%Y-%m-%d %H:%M:%S',
+                            location='cryptocom',
+                        )
+                        if last_date < deposit_date <= withdrawal_date:
+                            # Amount is negative
+                            amount_deposited += deserialize_asset_amount(deposit['Amount'])
+                    amount_withdrawal = deserialize_asset_amount(withdrawal['Amount'])
+                    # Compute profit
+                    profit = amount_withdrawal + amount_deposited
+                    if profit >= ZERO:
+                        last_date = withdrawal_date
+                        action = LedgerAction(
+                            identifier=0,  # whatever is not used at insertion
+                            timestamp=withdrawal_date,
+                            action_type=LedgerActionType.INCOME,
+                            location=Location.CRYPTOCOM,
+                            amount=AssetAmount(profit),
+                            asset=asset_object,
+                            rate=None,
+                            rate_asset=None,
+                            link=None,
+                            notes=f'Stake profit for asset {asset}',
+                        )
+                        self.db_ledger.add_ledger_action(action)
+
     def import_cryptocom_csv(self, filepath: Path) -> Tuple[bool, str]:
         with open(filepath, 'r', encoding='utf-8-sig') as csvfile:
             data = csv.DictReader(csvfile)
@@ -445,6 +568,14 @@ def import_cryptocom_csv(self, filepath: Path) -> Tuple[bool, str]:
                 self._import_cryptocom_associated_entries(data, 'dust_conversion')
                 csvfile.seek(0)
                 next(data)
+
+                self._import_cryptocom_associated_entries(data, 'interest_swap')
+                csvfile.seek(0)
+                next(data)
+
+                self._import_cryptocom_associated_entries(data, 'invest')
+                csvfile.seek(0)
+                next(data)
             except KeyError as e:
                 return False, f'Crypto.com csv missing entry for {str(e)}'
             except UnknownAsset as e:


Test patch:
diff --git a/rotkehlchen/tests/api/test_data_import.py b/rotkehlchen/tests/api/test_data_import.py
--- a/rotkehlchen/tests/api/test_data_import.py
+++ b/rotkehlchen/tests/api/test_data_import.py
@@ -15,6 +15,7 @@
 from rotkehlchen.tests.utils.dataimport import (
     assert_cointracking_import_results,
     assert_cryptocom_import_results,
+    assert_cryptocom_special_events_import_results,
     assert_blockfi_transactions_import_results,
     assert_blockfi_trades_import_results,
     assert_nexo_results,
@@ -63,10 +64,10 @@ def test_data_import_cointracking(rotkehlchen_api_server, file_upload):
 def test_data_import_cryptocom(rotkehlchen_api_server):
     """Test that the data import endpoint works successfully for cryptocom"""
     rotki = rotkehlchen_api_server.rest_api.rotkehlchen
-    dir_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
-    filepath = os.path.join(dir_path, 'data', 'cryptocom_trades_list.csv')
+    dir_path = Path(__file__).resolve().parent.parent
+    filepath = dir_path / 'data' / 'cryptocom_trades_list.csv'
 
-    json_data = {'source': 'cryptocom', 'file': filepath}
+    json_data = {'source': 'cryptocom', 'file': str(filepath)}
     response = requests.put(
         api_url_for(
             rotkehlchen_api_server,
@@ -79,14 +80,34 @@ def test_data_import_cryptocom(rotkehlchen_api_server):
     assert_cryptocom_import_results(rotki)
 
 
+@pytest.mark.parametrize('number_of_eth_accounts', [0])
+def test_data_import_cryptocom_special_types(rotkehlchen_api_server):
+    """Test that the data import endpoint works successfully for cryptocom"""
+    rotki = rotkehlchen_api_server.rest_api.rotkehlchen
+    dir_path = Path(__file__).resolve().parent.parent
+    filepath = dir_path / 'data' / 'cryptocom_special_events.csv'
+
+    json_data = {'source': 'cryptocom', 'file': str(filepath)}
+    response = requests.put(
+        api_url_for(
+            rotkehlchen_api_server,
+            'dataimportresource',
+        ), json=json_data,
+    )
+    result = assert_proper_response_with_result(response)
+    assert result is True
+    # And also assert data was imported succesfully
+    assert_cryptocom_special_events_import_results(rotki)
+
+
 @pytest.mark.parametrize('number_of_eth_accounts', [0])
 def test_data_import_blockfi_transactions(rotkehlchen_api_server):
     """Test that the data import endpoint works successfully for blockfi transactions"""
     rotki = rotkehlchen_api_server.rest_api.rotkehlchen
-    dir_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
-    filepath = os.path.join(dir_path, 'data', 'blockfi-transactions.csv')
+    dir_path = Path(__file__).resolve().parent.parent
+    filepath = dir_path / 'data' / 'blockfi-transactions.csv'
 
-    json_data = {'source': 'blockfi-transactions', 'file': filepath}
+    json_data = {'source': 'blockfi-transactions', 'file': str(filepath)}
     response = requests.put(
         api_url_for(
             rotkehlchen_api_server,
@@ -103,10 +124,10 @@ def test_data_import_blockfi_transactions(rotkehlchen_api_server):
 def test_data_import_blockfi_trades(rotkehlchen_api_server):
     """Test that the data import endpoint works successfully for blockfi trades"""
     rotki = rotkehlchen_api_server.rest_api.rotkehlchen
-    dir_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
-    filepath = os.path.join(dir_path, 'data', 'blockfi-trades.csv')
+    dir_path = Path(__file__).resolve().parent.parent
+    filepath = dir_path / 'data' / 'blockfi-trades.csv'
 
-    json_data = {'source': 'blockfi-trades', 'file': filepath}
+    json_data = {'source': 'blockfi-trades', 'file': str(filepath)}
     response = requests.put(
         api_url_for(
             rotkehlchen_api_server,
@@ -123,10 +144,10 @@ def test_data_import_blockfi_trades(rotkehlchen_api_server):
 def test_data_import_nexo(rotkehlchen_api_server):
     """Test that the data import endpoint works successfully for nexo"""
     rotki = rotkehlchen_api_server.rest_api.rotkehlchen
-    dir_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
-    filepath = os.path.join(dir_path, 'data', 'nexo.csv')
+    dir_path = Path(__file__).resolve().parent.parent
+    filepath = dir_path / 'data' / 'nexo.csv'
 
-    json_data = {'source': 'nexo', 'file': filepath}
+    json_data = {'source': 'nexo', 'file': str(filepath)}
     response = requests.put(
         api_url_for(
             rotkehlchen_api_server,
@@ -144,7 +165,7 @@ def test_data_import_nexo(rotkehlchen_api_server):
 @pytest.mark.parametrize('use_clean_caching_directory', [True])
 def test_data_import_wrong_extension(rotkehlchen_api_server, file_upload):
     """Test that uploading a file without the proper extension fails"""
-    dir_path = Path(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
+    dir_path = Path(__file__).resolve().parent.parent
     filepath = dir_path / 'data' / 'cointracking_trades_list.csv'
 
     # Let's also try to upload a file without the csv prefix
@@ -180,8 +201,8 @@ def test_data_import_wrong_extension(rotkehlchen_api_server, file_upload):
 @pytest.mark.parametrize('number_of_eth_accounts', [0])
 def test_data_import_errors(rotkehlchen_api_server, tmpdir_factory):
     """Test that errors in the data import endpoint are handled correctly"""
-    dir_path = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
-    filepath = os.path.join(dir_path, 'data', 'cointracking_trades_list.csv')
+    dir_path = Path(__file__).resolve().parent.parent
+    filepath = dir_path / 'data' / 'cointracking_trades_list.csv'
 
     # Test that if filepath is missing, an error is returned
     json_data = {'source': 'cointracking.info'}
@@ -198,7 +219,7 @@ def test_data_import_errors(rotkehlchen_api_server, tmpdir_factory):
     )
 
     # Test that if source is missing, an error is returned
-    json_data = {'filepath': filepath}
+    json_data = {'filepath': str(filepath)}
     response = requests.put(
         api_url_for(
             rotkehlchen_api_server,
@@ -212,7 +233,7 @@ def test_data_import_errors(rotkehlchen_api_server, tmpdir_factory):
     )
 
     # Test that if source is an invalid type an error is returned
-    json_data = {'source': 55, 'filepath': filepath}
+    json_data = {'source': 55, 'filepath': str(filepath)}
     response = requests.put(
         api_url_for(
             rotkehlchen_api_server,
@@ -226,7 +247,7 @@ def test_data_import_errors(rotkehlchen_api_server, tmpdir_factory):
     )
 
     # Test that if source is invalid an error is returned
-    json_data = {'source': 'somewhere', 'file': filepath}
+    json_data = {'source': 'somewhere', 'file': str(filepath)}
     response = requests.put(
         api_url_for(
             rotkehlchen_api_server,
diff --git a/rotkehlchen/tests/data/cryptocom_special_events.csv b/rotkehlchen/tests/data/cryptocom_special_events.csv
new file mode 100644
--- /dev/null
+++ b/rotkehlchen/tests/data/cryptocom_special_events.csv
@@ -0,0 +1,14 @@
+Timestamp (UTC),Transaction Description,Currency,Amount,To Currency,To Amount,Native Currency,Native Amount,Native Amount (in USD),Transaction Kind
+2021-01-05 22:00:00,CRO Stake Rewards,CRO,1,,,USD,1,1,mco_stake_reward
+2021-01-05 22:00:00,MCO Earnings/Rewards Swap,CRO,1,,,USD,1,1,interest_swap_credited
+2021-01-05 22:00:00,MCO Earnings/Rewards Swap,MCO,-0.1,,,USD,-1,-1,interest_swap_debited
+2021-01-05 22:00:00,MCO Stake Rewards,MCO,0.5,,,USD,1,1,mco_stake_reward
+2021-01-05 22:00:00,CRO Airdrop to Exchange,CRO,1,,,USD,-1,-1,airdrop_to_exchange_transfer
+2021-01-04 22:00:00,Invest Withdrawal,BTC,1.02005,,,USD,100,100,invest_withdrawal
+2021-01-03 23:00:00,Invest Deposit,BTC,-1.0,,,USD,-100,-100,invest_deposit
+2021-01-02 22:00:00,Invest Withdrawal,BTC,0.02005,,,USD,100,100,invest_withdrawal
+2021-01-01 22:00:00,CRO Airdrop,CRO,1,,,USD,1,1,airdrop_locked
+2021-01-01 23:00:00,Invest Deposit,BTC,-0.01,,,USD,-100,-100,invest_deposit
+2021-01-01 22:00:00,Invest Deposit,BTC,-0.01,,,USD,-100,-100,invest_deposit
+2021-01-01 21:01:00,Invest Withdrawal,ETC,0.24,,,USD,100,100,invest_withdrawal
+2021-01-01 21:00:00,Invest Deposit,ETC,-1.0,,,USD,-100,-100,invest_deposit
\ No newline at end of file
diff --git a/rotkehlchen/tests/utils/dataimport.py b/rotkehlchen/tests/utils/dataimport.py
--- a/rotkehlchen/tests/utils/dataimport.py
+++ b/rotkehlchen/tests/utils/dataimport.py
@@ -289,6 +289,90 @@ def get_trade_note(desc: str):
     assert expected_movements == asset_movements
 
 
+def assert_cryptocom_special_events_import_results(rotki: Rotkehlchen):
+    """A utility function to help assert on correctness of importing data from crypto.com"""
+    trades = rotki.data.db.get_trades()
+    ledger_db = DBLedgerActions(rotki.data.db, rotki.msg_aggregator)
+    ledger_actions = ledger_db.get_ledger_actions(None, None, None)
+    warnings = rotki.msg_aggregator.consume_warnings()
+    errors = rotki.msg_aggregator.consume_errors()
+    assert len(errors) == 0
+    assert len(warnings) == 0
+
+    expected_actions = [LedgerAction(
+        identifier=5,
+        timestamp=Timestamp(1609884000),
+        action_type=LedgerActionType.INCOME,
+        location=Location.CRYPTOCOM,
+        amount=AssetAmount(FVal('1')),
+        asset=symbol_to_asset_or_token('CRO'),
+        rate=None,
+        rate_asset=None,
+        link=None,
+        notes=None,
+    ), LedgerAction(
+        identifier=4,
+        timestamp=Timestamp(1609884000),
+        action_type=LedgerActionType.INCOME,
+        location=Location.CRYPTOCOM,
+        amount=AssetAmount(FVal('0.5')),
+        asset=symbol_to_asset_or_token('MCO'),
+        rate=None,
+        rate_asset=None,
+        link=None,
+        notes=None,
+    ), LedgerAction(
+        identifier=3,
+        timestamp=Timestamp(1609884000),
+        action_type=LedgerActionType.INCOME,
+        location=Location.CRYPTOCOM,
+        amount=AssetAmount(FVal('1')),
+        asset=symbol_to_asset_or_token('CRO'),
+        rate=None,
+        rate_asset=None,
+        link=None,
+        notes=None,
+    ), LedgerAction(
+        identifier=2,
+        timestamp=Timestamp(1609797600),
+        action_type=LedgerActionType.INCOME,
+        location=Location.CRYPTOCOM,
+        amount=AssetAmount(FVal('0.02005')),
+        asset=A_BTC,
+        rate=None,
+        rate_asset=None,
+        link=None,
+        notes='Stake profit for asset BTC',
+    ), LedgerAction(
+        identifier=1,
+        timestamp=Timestamp(1609624800),
+        action_type=LedgerActionType.INCOME,
+        location=Location.CRYPTOCOM,
+        amount=AssetAmount(FVal('0.00005')),
+        asset=A_BTC,
+        rate=None,
+        rate_asset=None,
+        link=None,
+        notes='Stake profit for asset BTC',
+    )]
+    assert list(reversed(expected_actions)) == ledger_actions
+
+    expected_trades = [Trade(
+        timestamp=Timestamp(1609884000),
+        location=Location.CRYPTOCOM,
+        base_asset=symbol_to_asset_or_token('CRO'),
+        quote_asset=symbol_to_asset_or_token('MCO'),
+        trade_type=TradeType.BUY,
+        amount=AssetAmount(FVal('1')),
+        rate=Price(FVal('10')),
+        fee=Fee(ZERO),
+        fee_currency=A_USD,
+        link='',
+        notes='MCO Earnings/Rewards Swap\nSource: crypto.com (CSV import)',
+    )]
+    assert trades == expected_trades
+
+
 def assert_blockfi_transactions_import_results(rotki: Rotkehlchen):
     """A utility function to help assert on correctness of importing data from blockfi"""
     ledger_db = DBLedgerActions(rotki.data.db, rotki.msg_aggregator)