Repository: rotki/rotki

Issue:
Crypto CSV from CDC fails to import
Hi,
I tried to import import the CDC file ...Crypto.. it failed.

The other two Fiat and Cards worked fine.

![CDC Rotki 3](https://user-images.githubusercontent.com/59722401/120929279-bc5f3300-c6e8-11eb-91b0-1c0d073064c1.PNG)

![CDC Rotki 2](https://user-images.githubusercontent.com/59722401/120929157-37741980-c6e8-11eb-9c2a-87c80ca7715c.PNG)
![CDC Rotki](https://user-images.githubusercontent.com/59722401/120929158-380cb000-c6e8-11eb-97a2-f3fddbfeaa5e.PNG)

I tried to send Excel CSV but it says this file not supported.

App Version: 1.17.2
Frontend Version: 1.17.2
Platform: win32 x64 10.0.19042
Electron: 12.0.7




Operating system: 
Rotki version:



Discussion:
In order:
- First warning might be because you have an empty line in the CSV files
- About `viban_*` I don't see any code in our side to handle this kind of transactions so we need to add support for it. It would be necessary that you provided us with faked data for this type of rows.

About that error I tried with the CSV in the screenshot and is not the cause. Could you also give me fake data to check what could it be?
I tried to give you real data but the file wouldn’t send. What do you mean via fake data? 

I tried to send a real CSV with the current headers as they are and only two rows of real data the rest I deleted. 

I don’t even know what viban_* is 



> On 6. Jun 2021, at 17:37, Yábir Benchakhtir ***@***.***> wrote:
> 
> ﻿
> In order:
> 
> First warning might be because you have an empty line in the CSV files
> About viban_* I don't see any code in our side to handle this kind of transactions so we need to add support for it. It would be necessary that you provided us with faked data for this type of rows.
> About that error I tried with the CSV in the screenshot and is not the cause. Could you also give me fake data to check what could it be?
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Sorry, with fake data I meant a CSV file that didn't contain information about you (values) but is similar to what you see.
In this case headers are not the problem but some row, that is why you see `missing entry for 'debited'`.

With the `viban_` I was referring to the warnings that you see in the UI. They mention `viban_deposit`, `viban_purchase`, `viban_card_top_up`. Those are actions that we don't have any information about as we haven't seen them before.
But if it won’t accept a CSV formatted file how am I supposed to send it? Fake or real… it keeps saying file format not accepted. 

> On 6. Jun 2021, at 18:35, Yábir Benchakhtir ***@***.***> wrote:
> 
> ﻿
> Sorry, with fake data I meant a CSV file that didn't contain information about you (values) but is similar to what you see.
> In this case headers are not the problem but some row, that is why you see missing entry for 'debited'.
> 
> With the viban_ I was referring to the warnings that you see in the UI. They mention viban_deposit, viban_purchase, viban_card_top_up. Those are actions that we don't have any information about as we haven't seen them before.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Sorry @Blackswan122 I believe I didn't explain well. I meant to say that if you could send either to me in discord or here the csv file that is failing so I can check why this error is showing with random numbers so no information about you is revealed. 
@Blackswan122 CSV formats are experimental because crypto.com, nexo and all other CSV services changes them every other day. If you want us to fix this you will need to send us an example CSV file. You are also in discord, you can DM yabir.

@yabirgb This is why I keep saying that CSV support is hard. There is no spec. They just change it all the time. We will need to implement a generic CSV import so users can fix these stuff on their own: https://github.com/rotki/rotki/issues/2270
Any updates here? With absence of more information I am going to close the issue.
Shame on me. Not for the moment. Now that I have more time I'll take a look at it ASAP. We fixed an issue related to nexo already but there are still one file failing
Yes but do you have the CSV that lead to the failure? Without it you can't debug this.

Resolving patch:
diff --git a/rotkehlchen/data/importer.py b/rotkehlchen/data/importer.py
--- a/rotkehlchen/data/importer.py
+++ b/rotkehlchen/data/importer.py
@@ -240,6 +240,7 @@ def _consume_cryptocom_entry(self, csv_row: Dict[str, Any]) -> None:
             'referral_card_cashback',
             'card_cashback_reverted',
             'reimbursement',
+            'viban_purchase',
         ):
             # variable mapping to raw data
             currency = csv_row['Currency']
@@ -281,8 +282,13 @@ def _consume_cryptocom_entry(self, csv_row: Dict[str, Any]) -> None:
             )
             self.db.add_trades([trade])
 
-        elif row_type in ('crypto_withdrawal', 'crypto_deposit'):
-            if row_type == 'crypto_withdrawal':
+        elif row_type in (
+            'crypto_withdrawal',
+            'crypto_deposit',
+            'viban_deposit',
+            'viban_card_top_up',
+        ):
+            if row_type in ('crypto_withdrawal', 'viban_deposit', 'viban_card_top_up'):
                 category = AssetMovementCategory.WITHDRAWAL
                 amount = deserialize_asset_amount_force_positive(csv_row['Amount'])
             else:
@@ -406,18 +412,44 @@ def _import_cryptocom_associated_entries(self, data: Any, tx_kind: str) -> None:
         investments_withdrawals: Dict[str, List[Any]] = defaultdict(list)
         debited_row = None
         credited_row = None
+        expects_debited = False
+        credited_timestamp = None
         for row in data:
+            # If we don't have the corresponding debited entry ignore them
+            # and warn the user
+            if (
+                expects_debited is True and
+                row['Transaction Kind'] != f'{tx_kind}_debited'
+            ):
+                self.db.msg_aggregator.add_warning(
+                    f'Error during cryptocom CSV import consumption. Found {tx_kind}_credited '
+                    f'but no amount debited afterwards at date {row["Timestamp (UTC)"]}',
+                )
+                # Pop the last credited event as it's invalid. We always assume to be at least
+                # one debited event and one credited event. If we don't find the debited event
+                # we have to remove the credit at the right timestamp or our logic will break.
+                # We notify the user about this issue so (s)he can take actions.
+                multiple_rows.pop(credited_timestamp, None)
+                # reset expects_debited value
+                expects_debited = False
             if row['Transaction Kind'] == f'{tx_kind}_debited':
                 timestamp = deserialize_timestamp_from_date(
                     date=row['Timestamp (UTC)'],
                     formatstr='%Y-%m-%d %H:%M:%S',
                     location='cryptocom',
                 )
+                if expects_debited is False and timestamp != credited_timestamp:
+                    self.db.msg_aggregator.add_warning(
+                        f'Error during cryptocom CSV import consumption. Found {tx_kind}_debited'
+                        f' but no amount credited before at date {row["Timestamp (UTC)"]}',
+                    )
+                    continue
                 if timestamp not in multiple_rows:
                     multiple_rows[timestamp] = {}
                 if 'debited' not in multiple_rows[timestamp]:
                     multiple_rows[timestamp]['debited'] = []
                 multiple_rows[timestamp]['debited'].append(row)
+                expects_debited = False
             elif row['Transaction Kind'] == f'{tx_kind}_credited':
                 # They only is one credited row
                 timestamp = deserialize_timestamp_from_date(
@@ -427,6 +459,8 @@ def _import_cryptocom_associated_entries(self, data: Any, tx_kind: str) -> None:
                 )
                 if timestamp not in multiple_rows:
                     multiple_rows[timestamp] = {}
+                expects_debited = True
+                credited_timestamp = timestamp
                 multiple_rows[timestamp]['credited'] = row
             elif row['Transaction Kind'] == f'{tx_kind}_deposit':
                 asset = row['Currency']
@@ -439,8 +473,14 @@ def _import_cryptocom_associated_entries(self, data: Any, tx_kind: str) -> None:
             # When we convert multiple assets dust to CRO
             # in one time, it will create multiple debited rows with
             # the same timestamp
-            debited_rows = multiple_rows[timestamp]['debited']
-            credited_row = multiple_rows[timestamp]['credited']
+            try:
+                debited_rows = multiple_rows[timestamp]['debited']
+                credited_row = multiple_rows[timestamp]['credited']
+            except KeyError as e:
+                self.db.msg_aggregator.add_warning(
+                    f'Failed to get {str(e)} event at timestamp {timestamp}.',
+                )
+                continue
             total_debited_usd = functools.reduce(
                 lambda acc, row:
                     acc +


Test patch: