Repository: rotki/rotki

Issue:
CDC Import of CSV
"Invalid CSV format, missing required field: 'debited'"


I can't change anything to get debited field



Operating system: Win 10 64
Rotki version: 1.11.0



Discussion:
Is this for the Crypto.com import? 

This is only for the mobile app's data that @CPatchane implemented for his own use case and added it to the app in case others can benefit.

@CPatchane that is correct right? Or did @Blackswan122 find a bug? Or did crypto.com just change their csv format?
Yep, it only concerns the application export file, not the exchange.
I did an import yesterday and I don't have any issue 🤔 
@Blackswan122 Do you have dust conversions or the MCO/CRO swap in your transactions? It seems related to those two transaction type.
The error here is not very helpful 🙈 
I just saved my CSV file and imported it. I do have dust conversiosions and MCO/CRO swap everyone will have as it was forced on everyone. 
@Blackswan122 the question is where did you save the CSV file from? If this is from the mobile app then it should work. If it is from the website then it will not work as the website is not supported.
True. Is it possible to get those rows (only dust conversions and MCO/CRO swap ones) without amounts to help debugging 🙏 ? You can join the rotki discord if you prefer to share direct to me in private.
@Blackswan122 so which service's CSV file did you try to export? Would like to know here to see what needs to be done with this issue.
I just tried it again.....downloaded from Mobile app the CSV file. Sent it per email...loaded file from downloads and got the error 
"Invalid CSV format, missing required field 'debited'

There's nothing for me to chose in CDC app except download CSV
@Blackswan122 thank you. So it is indeed from the mobile app then I understand. So @CPatchane there is a bug in the importer or they changed their format.

@Blackswan122  Can you maybe DM me in discord or twitter how the csv file looks like. This is why dislike supporting CSV imports. The format can change at any time.
Timestamp (UTC)	Transaction Description	Currency	Amount	To Currency	To Amount	Native Currency	Native Amount	Native Amount (in USD)	Transaction Kind

Here are the current headers of CSV file
@CPatchane any idea why he would get different CSV format than you? He is also using the mobile app.
Headers from @Blackswan122 are good and are the same than the one I have and the one from the tests.
I get the issue I think, do you have at a moment two `dust_conversion_debited` lines followed by only one `dust_conversion_credited`? They should have the exact same timestamp.
@LefterisJP CDC seems to gather dust conversions in one credited line, the script about double entries is working by matching 1:1 for credited and debited lines. I have to add that case then.
@CPatchane I see. When you can handle this we can close the issue

Resolving patch:
diff --git a/rotkehlchen/data/importer.py b/rotkehlchen/data/importer.py
--- a/rotkehlchen/data/importer.py
+++ b/rotkehlchen/data/importer.py
@@ -1,8 +1,10 @@
 import csv
+import functools
 from itertools import count
 from pathlib import Path
 from typing import Any, Dict, List, Tuple
 
+from rotkehlchen.fval import FVal
 from rotkehlchen.assets.asset import Asset
 from rotkehlchen.constants.assets import A_USD
 from rotkehlchen.constants.misc import ZERO
@@ -310,7 +312,7 @@ def _consume_cryptocom_entry(self, csv_row: Dict[str, Any]) -> None:
             # supercharger actions
             'supercharger_deposit',
             'supercharger_withdrawal',
-            # already handled using _import_cryptocom_double_entries
+            # already handled using _import_cryptocom_associated_entries
             'dynamic_coin_swap_debited',
             'dynamic_coin_swap_credited',
             'dust_conversion_debited',
@@ -325,67 +327,100 @@ def _consume_cryptocom_entry(self, csv_row: Dict[str, Any]) -> None:
                 f'cryptocom data import. Ignoring entry',
             )
 
-    def _import_cryptocom_double_entries(self, data: Any, double_type: str) -> None:
-        """Look for events that have double entries and handle them as trades.
+    def _import_cryptocom_associated_entries(self, data: Any, tx_kind: str) -> None:
+        """Look for events that have associated entries and handle them as trades.
 
         This method looks for `*_debited` and `*_credited` entries using the
         same timestamp to handle them as one trade.
 
-        Known double_type: 'dynamic_coin_swap' or 'dust_conversion'
+        Known kind: 'dynamic_coin_swap' or 'dust_conversion'
         """
-        double_rows: Dict[Any, Dict[str, Any]] = {}
+        multiple_rows: Dict[Any, Dict[str, Any]] = {}
         debited_row = None
         credited_row = None
         for row in data:
-            if row['Transaction Kind'] == f'{double_type}_debited':
+            if row['Transaction Kind'] == f'{tx_kind}_debited':
                 timestamp = deserialize_timestamp_from_date(
                     date=row['Timestamp (UTC)'],
                     formatstr='%Y-%m-%d %H:%M:%S',
                     location='crypto.com',
                 )
-                if timestamp not in double_rows:
-                    double_rows[timestamp] = {}
-                double_rows[timestamp]['debited'] = row
-            elif row['Transaction Kind'] == f'{double_type}_credited':
+                if timestamp not in multiple_rows:
+                    multiple_rows[timestamp] = {}
+                if 'debited' not in multiple_rows[timestamp]:
+                    multiple_rows[timestamp]['debited'] = []
+                multiple_rows[timestamp]['debited'].append(row)
+            elif row['Transaction Kind'] == f'{tx_kind}_credited':
+                # They only is one credited row
                 timestamp = deserialize_timestamp_from_date(
                     date=row['Timestamp (UTC)'],
                     formatstr='%Y-%m-%d %H:%M:%S',
                     location='crypto.com',
                 )
-                if timestamp not in double_rows:
-                    double_rows[timestamp] = {}
-                double_rows[timestamp]['credited'] = row
-
-        for timestamp in double_rows:
-            credited_row = double_rows[timestamp]['credited']
-            debited_row = double_rows[timestamp]['debited']
-            if credited_row is not None and debited_row is not None:
-                description = credited_row['Transaction Description']
-                notes = f'{description}\nSource: crypto.com (CSV import)'
-                # No fees here
-                fee = Fee(ZERO)
-                fee_currency = A_USD
-
-                base_asset = Asset(credited_row['Currency'])
-                quote_asset = Asset(debited_row['Currency'])
-                pair = TradePair(f'{base_asset.identifier}_{quote_asset.identifier}')
-                base_amount_bought = deserialize_asset_amount(credited_row['Amount'])
-                quote_amount_sold = deserialize_asset_amount(debited_row['Amount'])
-                rate = Price(abs(base_amount_bought / quote_amount_sold))
-
-                trade = Trade(
-                    timestamp=timestamp,
-                    location=Location.CRYPTOCOM,
-                    pair=pair,
-                    trade_type=TradeType.BUY,
-                    amount=base_amount_bought,
-                    rate=rate,
-                    fee=fee,
-                    fee_currency=fee_currency,
-                    link='',
-                    notes=notes,
-                )
-                self.db.add_trades([trade])
+                if timestamp not in multiple_rows:
+                    multiple_rows[timestamp] = {}
+                multiple_rows[timestamp]['credited'] = row
+
+        for timestamp in multiple_rows:
+            # When we convert multiple assets dust to CRO
+            # in one time, it will create multiple debited rows with
+            # the same timestamp
+            debited_rows = multiple_rows[timestamp]['debited']
+            credited_row = multiple_rows[timestamp]['credited']
+            total_debited_usd = functools.reduce(
+                lambda acc, row:
+                    acc +
+                    deserialize_asset_amount(row['Native Amount (in USD)']),
+                debited_rows,
+                FVal('0'),
+            )
+
+            # If the value of the transaction is too small (< 0,01$),
+            # crypto.com will display 0 as native amount
+            # if we have multiple debited rows, we can't import them
+            # since we can't compute their dedicated rates, so we skip them
+            if len(debited_rows) > 1 and total_debited_usd == 0:
+                return
+
+            if credited_row is not None and len(debited_rows) != 0:
+                for debited_row in debited_rows:
+                    description = credited_row['Transaction Description']
+                    notes = f'{description}\nSource: crypto.com (CSV import)'
+                    # No fees here
+                    fee = Fee(ZERO)
+                    fee_currency = A_USD
+
+                    base_asset = Asset(credited_row['Currency'])
+                    quote_asset = Asset(debited_row['Currency'])
+                    pair = TradePair(f'{base_asset.identifier}_{quote_asset.identifier}')
+                    part_of_total = (
+                        FVal(1)
+                        if len(debited_rows) == 1
+                        else deserialize_asset_amount(
+                            debited_row["Native Amount (in USD)"],
+                        ) / total_debited_usd
+                    )
+                    quote_amount_sold = deserialize_asset_amount(
+                        debited_row['Amount'],
+                    ) * part_of_total
+                    base_amount_bought = deserialize_asset_amount(
+                        credited_row['Amount'],
+                    ) * part_of_total
+                    rate = Price(abs(base_amount_bought / quote_amount_sold))
+
+                    trade = Trade(
+                        timestamp=timestamp,
+                        location=Location.CRYPTOCOM,
+                        pair=pair,
+                        trade_type=TradeType.BUY,
+                        amount=AssetAmount(base_amount_bought),
+                        rate=rate,
+                        fee=fee,
+                        fee_currency=fee_currency,
+                        link='',
+                        notes=notes,
+                    )
+                    self.db.add_trades([trade])
 
     def import_cryptocom_csv(self, filepath: Path) -> Tuple[bool, str]:
         with open(filepath, 'r', encoding='utf-8-sig') as csvfile:
@@ -393,13 +428,13 @@ def import_cryptocom_csv(self, filepath: Path) -> Tuple[bool, str]:
             try:
                 #  Notice: Crypto.com csv export gathers all swapping entries (`lockup_swap_*`,
                 # `crypto_wallet_swap_*`, ...) into one entry named `dynamic_coin_swap_*`.
-                self._import_cryptocom_double_entries(data, 'dynamic_coin_swap')
+                self._import_cryptocom_associated_entries(data, 'dynamic_coin_swap')
                 # reset the iterator
                 csvfile.seek(0)
                 # pass the header since seek(0) make the first row to be the header
                 next(data)
 
-                self._import_cryptocom_double_entries(data, 'dust_conversion')
+                self._import_cryptocom_associated_entries(data, 'dust_conversion')
                 csvfile.seek(0)
                 next(data)
             except KeyError as e:


Test patch:
diff --git a/rotkehlchen/tests/data/cryptocom_trades_list.csv b/rotkehlchen/tests/data/cryptocom_trades_list.csv
--- a/rotkehlchen/tests/data/cryptocom_trades_list.csv
+++ b/rotkehlchen/tests/data/cryptocom_trades_list.csv
@@ -1,4 +1,7 @@
 ﻿Timestamp (UTC),Transaction Description,Currency,Amount,To Currency,To Amount,Native Currency,Native Amount,Native Amount (in USD),Transaction Kind
+2020-12-15 09:25:14,Convert Dust,CRO,193.02779901061902311,,,EUR,10.59,12.8794521,dust_conversion_credited
+2020-12-15 09:25:14,Convert Dust,UNI,-1.3387887,,,EUR,-5.84,-7.1025496,dust_conversion_debited
+2020-12-15 09:25:14,Convert Dust,DOT,-0.6295092,,,EUR,-4.8,-5.837712,dust_conversion_debited
 2020-12-01 14:39:25,Convert Dust,CRO,0.007231228760408149,,,EUR,0.0,0,dust_conversion_credited
 2020-12-01 14:39:25,Convert Dust,DAI,-0.00050680380674961,,,EUR,0.0,0,dust_conversion_debited
 2020-10-12 19:10:36,Transfer: Exchange -> App wallet,ETH,0.033,,,EUR,10.75,12.7743325,exchange_to_crypto_transfer
diff --git a/rotkehlchen/tests/utils/dataimport.py b/rotkehlchen/tests/utils/dataimport.py
--- a/rotkehlchen/tests/utils/dataimport.py
+++ b/rotkehlchen/tests/utils/dataimport.py
@@ -220,6 +220,28 @@ def get_trade_note(desc: str):
         fee_currency=A_USD,
         link='',
         notes=get_trade_note('Convert Dust'),
+    ), Trade(
+        timestamp=Timestamp(1608024314),
+        location=Location.CRYPTOCOM,
+        pair=TradePair('CRO_UNI'),
+        trade_type=TradeType.BUY,
+        amount=AssetAmount(FVal('105.9475889306405164438345865')),
+        rate=Price(FVal('144.1809293808791657040427665')),
+        fee=Fee(ZERO),
+        fee_currency=A_USD,
+        link='',
+        notes=get_trade_note('Convert Dust'),
+    ), Trade(
+        timestamp=Timestamp(1608024314),
+        location=Location.CRYPTOCOM,
+        pair=TradePair('CRO_DOT'),
+        trade_type=TradeType.BUY,
+        amount=AssetAmount(FVal('87.08021007997850666616541352')),
+        rate=Price(FVal('306.6322128582378511862892551')),
+        fee=Fee(ZERO),
+        fee_currency=A_USD,
+        link='',
+        notes=get_trade_note('Convert Dust'),
     )]
     assert expected_trades == trades