Repository: rotki/rotki

Issue:
Bug Import CSV cointracking
Dear,
I have exported my cointracking CSV and tried to import it.
For every line I have : 
`Error during cointracking CSV import deserialization. Error was Failed to deserialize a timestamp from a null entry in cointracking.info. Ignoring entry`

I have used v1.4.2 (and also tried on v1.4.1).
Thank you
Valentin


Discussion:
[rotkehlchen.log](https://github.com/rotki/rotki/files/4627768/rotkehlchen.log)

Exported CSV : 
```
"Type","Buy","Cur.","Sell","Cur.","Fee","Cur.","Exchange","Group","Comment","Date"
"Trade","3300.00000000","CRO","197.01000000","EUR","","","Crypto.com","","EUR -> CRO","2020-05-14 11:46:20"
"Deposit","197.01000000","EUR","","","","","Crypto.com","","EUR Deposit (Via SEPA)","2020-05-14 11:44:20"
"Trade","4784.80000000","CRO","286.57000000","EUR","","","Crypto.com","","EUR -> CRO","2020-05-14 10:54:20"
"Trade","0.00120200","BTC","80.00000000","CRO","","","Crypto.com","","Syndicate 50% BTC","2020-05-14 10:44:20"
"Trade","2000.00000000","CRO","117.54000000","EUR","","","Crypto.com","","EUR->CRO","2020-05-11 21:58:34"
"Deposit","117.54000000","EUR","","","","","Crypto.com","","EUR Deposit (Via SEPA)","2020-05-11 21:06:27"
"Trade","197.17000000","EUR","0.02300000","BTC","","","Crypto.com","","BTC -> EUR","2020-05-06 23:32:11"
"Trade","89.40000000","EUR","18.51000000","MCO","","","Crypto.com","","MCO -> EUR","2020-05-02 13:51:11"
"Withdrawal","","","0.57000000","EUR","","","Crypto.com","","EUR Withdrawal (via SEPA)","2020-03-29 11:20:38"
"Trade","0.01310000","BTC","80.30000000","EUR","","","Crypto.com","","EUR -> BTC","2020-03-25 12:15:34"
"Trade","1.00000000","ETH","119.13000000","EUR","","","Crypto.com","","EUR -> ETH","2020-03-19 13:22:25"
"Deposit","200.00000000","EUR","","","","","Crypto.com","","EUR Deposit (Via SEPA)","2020-03-19 08:10:02"
"Trade","0.05000000","BTC","259.96000000","EUR","","","Crypto.com","","Buy BTC","2020-03-13 15:44:01"
"Income","18.51347874","MCO","","","","","Crypto.com","","Sign-up Bonus Unlocked","2020-03-13 15:23:01"
"Trade","50.00000000","MCO","128.54000000","EUR","","","Crypto.com","","Buy MCO","2020-03-13 15:22:01"
"Deposit","128.54000000","EUR","","","4.50000000","EUR","","","","2020-03-13 14:21:25"
"Deposit","259.96000000","EUR","","","9.10000000","EUR","","","","2020-03-13 14:21:25"
```
Thank you for the issue @vchatela. We are going to take a look.

Cointracking CSV import is experimental since very few people use it, the format of the data changes often and they are missing important data from trades

Also from looking at the CSV you provided I see Crypto.com was used as an exchange which we don't support yet. But we definitely could consider it as "other" source and still add the trades.
Hi @LefterisJP 
I did the import from Crypto.com to Cointracking and now I wanted to move it to Rotki.
I will import them manually I done have a lot of transactions :)
Thank you !
If it can help I have made this one worked : 
```
"","Type","Buy","Cur.","Sell","Cur.","Fee","Cur.","Exchange","Date"
"","Trade","3300.00000000","CRO","197.01000000","EUR","no exchange","","","14.05.2020 11:46"
```
Which was previsouly : 
```
"Type","Buy","Cur.","Sell","Cur.","Fee","Cur.","Exchange","Group","Comment","Date"
"Trade","3300.00000000","CRO","197.01000000","EUR","","","Crypto.com","","EUR -> CRO","2020-05-14 11:46:20"
```

So removed 'Comment' and 'Group' and added no exchange because "" not matched/accepted. 
Then date changed from - to . and display to EN mode.
All right so I see what's wrong. They changed the import format again. The timestamp key used to be `Date` and now it's `Trade Date`.

We will fix it and include the fix in the next release, but yeah as I mentioned this feature is really experimental since they just change their data format all the time.

Also we don't support crypto.com yet, but yeah using "no exchange" should work :)

Thanks for the report. Your test data help a lot to support the latest version of their export.
Thank you very much for your help and future changes.
Very nice job ! 👍 

Resolving patch:
diff --git a/rotkehlchen/data/importer.py b/rotkehlchen/data/importer.py
--- a/rotkehlchen/data/importer.py
+++ b/rotkehlchen/data/importer.py
@@ -1,6 +1,7 @@
 import csv
+from itertools import count
 from pathlib import Path
-from typing import List
+from typing import Any, Dict, List
 
 from rotkehlchen.assets.asset import Asset
 from rotkehlchen.constants.assets import A_USD
@@ -11,11 +12,18 @@
 from rotkehlchen.serialization.deserialize import (
     deserialize_asset_amount,
     deserialize_asset_movement_category,
+    deserialize_fee,
     deserialize_timestamp_from_date,
 )
 from rotkehlchen.typing import AssetAmount, Fee, Location, Price, TradePair, TradeType
 
 
+def remap_header(fieldnames: List[str]) -> List[str]:
+    cur_count = count(1)
+    mapping = {1: 'Buy', 2: 'Sell', 3: 'Fee'}
+    return [f'Cur.{mapping[next(cur_count)]}' if f.startswith('Cur.') else f for f in fieldnames]
+
+
 class UnsupportedCointrackingEntry(Exception):
     """Thrown for Cointracking CSV export entries we can't support to import"""
 
@@ -65,7 +73,7 @@ class DataImporter():
     def __init__(self, db: DBHandler) -> None:
         self.db = db
 
-    def _consume_cointracking_entry(self, csv_row: List[str]) -> None:
+    def _consume_cointracking_entry(self, csv_row: Dict[str, Any]) -> None:
         """Consumes a cointracking entry row from the CSV and adds it into the database
         Can raise:
             - DeserializationError if something is wrong with the format of the expected values
@@ -73,28 +81,34 @@ def _consume_cointracking_entry(self, csv_row: List[str]) -> None:
             - IndexError if the CSV file is corrupt
             - UnknownAsset if one of the assets founds in the entry are not supported
         """
-        row_type = csv_row[1]  # Type
+        row_type = csv_row['Type']
         timestamp = deserialize_timestamp_from_date(
-            date=csv_row[9],
-            formatstr='%d.%m.%Y %H:%M',
+            date=csv_row['Date'],
+            formatstr='%d.%m.%Y %H:%M:%S',
             location='cointracking.info',
         )
-        notes = csv_row[8]
-        location = exchange_row_to_location(csv_row[6])
-
-        if row_type == 'Gift/Tip' or row_type == 'Trade':
-            base_asset = Asset(csv_row[3])
-            quote_asset = None if csv_row[5] == '' else Asset(csv_row[5])
-            if not quote_asset and row_type != 'Gift/Tip':
+        notes = csv_row['Comment']
+        location = exchange_row_to_location(csv_row['Exchange'])
+
+        fee = Fee(ZERO)
+        fee_currency = A_USD  # whatever (used only if there is no fee)
+        if csv_row['Fee'] != '':
+            fee = deserialize_fee(csv_row['Fee'])
+            fee_currency = Asset(csv_row['Cur.Fee'])
+
+        if row_type in ('Gift/Tip', 'Trade', 'Income'):
+            base_asset = Asset(csv_row['Cur.Buy'])
+            quote_asset = None if csv_row['Cur.Sell'] == '' else Asset(csv_row['Cur.Sell'])
+            if not quote_asset and row_type not in ('Gift/Tip', 'Income'):
                 raise DeserializationError('Got a trade entry with an empty quote asset')
 
             if quote_asset is None:
                 # Really makes no difference as this is just a gift and the amount is zero
                 quote_asset = A_USD
             pair = TradePair(f'{base_asset.identifier}_{quote_asset.identifier}')
-            base_amount_bought = deserialize_asset_amount(csv_row[2])
-            if csv_row[4] != '-':
-                quote_amount_sold = deserialize_asset_amount(csv_row[4])
+            base_amount_bought = deserialize_asset_amount(csv_row['Buy'])
+            if csv_row['Sell'] != '-':
+                quote_amount_sold = deserialize_asset_amount(csv_row['Sell'])
             else:
                 quote_amount_sold = AssetAmount(ZERO)
             rate = Price(quote_amount_sold / base_amount_bought)
@@ -106,8 +120,8 @@ def _consume_cointracking_entry(self, csv_row: List[str]) -> None:
                 trade_type=TradeType.BUY,  # It's always a buy during cointracking import
                 amount=base_amount_bought,
                 rate=rate,
-                fee=Fee(ZERO),  # There are no fees when import from cointracking
-                fee_currency=base_asset,
+                fee=fee,
+                fee_currency=fee_currency,
                 link='',
                 notes=notes,
             )
@@ -115,11 +129,11 @@ def _consume_cointracking_entry(self, csv_row: List[str]) -> None:
         elif row_type == 'Deposit' or row_type == 'Withdrawal':
             category = deserialize_asset_movement_category(row_type.lower())
             if category == AssetMovementCategory.DEPOSIT:
-                amount = deserialize_asset_amount(csv_row[2])
-                asset = Asset(csv_row[3])
+                amount = deserialize_asset_amount(csv_row['Buy'])
+                asset = Asset(csv_row['Cur.Buy'])
             else:
-                amount = deserialize_asset_amount(csv_row[4])
-                asset = Asset(csv_row[5])
+                amount = deserialize_asset_amount(csv_row['Sell'])
+                asset = Asset(csv_row['Cur.Sell'])
 
             asset_movement = AssetMovement(
                 location=location,
@@ -127,8 +141,8 @@ def _consume_cointracking_entry(self, csv_row: List[str]) -> None:
                 timestamp=timestamp,
                 asset=asset,
                 amount=amount,
-                fee_asset=asset,
-                fee=Fee(ZERO),
+                fee=fee,
+                fee_asset=fee_currency,
                 link='',
             )
             self.db.add_asset_movements([asset_movement])
@@ -139,12 +153,12 @@ def _consume_cointracking_entry(self, csv_row: List[str]) -> None:
             )
 
     def import_cointracking_csv(self, filepath: Path) -> None:
-        with open(filepath, 'r') as csvfile:
+        with open(filepath, 'r', encoding='utf-8-sig') as csvfile:
             data = csv.reader(csvfile, delimiter=',', quotechar='"')
-            next(data)  # skip header row
+            header = remap_header(next(data))
             for row in data:
                 try:
-                    self._consume_cointracking_entry(row)
+                    self._consume_cointracking_entry(dict(zip(header, row)))
                 except UnknownAsset as e:
                     self.db.msg_aggregator.add_warning(
                         f'During cointracking CSV import found action with unknown '
@@ -153,8 +167,8 @@ def import_cointracking_csv(self, filepath: Path) -> None:
                     continue
                 except IndexError:
                     self.db.msg_aggregator.add_warning(
-                        f'During cointracking CSV import found entry with '
-                        f'unexpected number of columns',
+                        'During cointracking CSV import found entry with '
+                        'unexpected number of columns',
                     )
                     continue
                 except DeserializationError as e:


Test patch:
diff --git a/rotkehlchen/tests/data/cointracking_trades_list.csv b/rotkehlchen/tests/data/cointracking_trades_list.csv
--- a/rotkehlchen/tests/data/cointracking_trades_list.csv
+++ b/rotkehlchen/tests/data/cointracking_trades_list.csv
@@ -1,8 +1,9 @@
-﻿"","Type","Buy","Cur.","Sell","Cur.","Exchange","Trade Group","Comment","Trade Date"
-"","Withdrawal","-","","0.00136700","ETH","ETH Transaction","","","20.09.2019 13:00"
-"","Deposit","0.02000000","ETH","-","","ETH Transaction","","","20.09.2019 02:01"
-"","Gift/Tip","0.00100000","BTC","-","","no exchange","what is trade group?","Just a small gift from someone","02.09.2019 10:00"
-"","Deposit","0.33676476","BTC","-","","BTC Transaction","","","02.09.2019 09:29"
-"","Deposit","5.00000000","XMR","-","","Poloniex","","","15.08.2019 05:57"
-"","Trade","0.05772716","ETH","10.99000000","EUR","Coinbase","","","24.08.2019 23:01"
-"","Withdrawal","-","","0.05770427","ETH","Coinbase","","","25.08.2019 09:42"
\ No newline at end of file
+﻿"Type","Buy","Cur.","Sell","Cur.","Fee","Cur.","Exchange","Group","Comment","Date"
+"Withdrawal","-","","0.00136700","ETH","0.0001","ETH","ETH Transaction","","","20.09.2019 13:00:00"
+"Deposit","0.02000000","ETH","-","","","","ETH Transaction","","","20.09.2019 02:01:10"
+"Gift/Tip","0.00100000","BTC","-","","","","no exchange","what is trade group?","Just a small gift from someone","02.09.2019 10:00:10"
+"Income","2","ETH","-","","","","no exchange","","Sign up bonus","03.09.2019 10:00:05"
+"Deposit","0.33676476","BTC","-","","","","BTC Transaction","","","02.09.2019 09:29:00"
+"Deposit","5.00000000","XMR","-","","","","Poloniex","","","15.08.2019 05:57:04"
+"Trade","0.05772716","ETH","10.99000000","EUR","0.02","EUR","Coinbase","","","24.08.2019 23:01:59"
+"Withdrawal","-","","0.05770427","ETH","0.0001","ETH","Coinbase","","","25.08.2019 09:42:35"
\ No newline at end of file
diff --git a/rotkehlchen/tests/utils/dataimport.py b/rotkehlchen/tests/utils/dataimport.py
--- a/rotkehlchen/tests/utils/dataimport.py
+++ b/rotkehlchen/tests/utils/dataimport.py
@@ -1,4 +1,4 @@
-from rotkehlchen.constants.assets import A_BTC, A_ETH
+from rotkehlchen.constants.assets import A_ETH, A_EUR, A_USD
 from rotkehlchen.constants.misc import ZERO
 from rotkehlchen.exchanges.data_structures import AssetMovement, Trade
 from rotkehlchen.fval import FVal
@@ -26,47 +26,58 @@ def assert_cointracking_import_results(rotki: Rotkehlchen):
     assert len(warnings) == 3
 
     expected_trades = [Trade(
-        timestamp=Timestamp(1566687660),
+        timestamp=Timestamp(1566687719),
         location=Location.COINBASE,
         pair=TradePair('ETH_EUR'),
         trade_type=TradeType.BUY,
         amount=AssetAmount(FVal('0.05772716')),
         rate=Price(FVal('190.3783245183029963712055123')),
-        fee=Fee(ZERO),
-        fee_currency=A_ETH,
+        fee=Fee(FVal("0.02")),
+        fee_currency=A_EUR,
         link='',
         notes='',
     ), Trade(
-        timestamp=Timestamp(1567418400),
+        timestamp=Timestamp(1567418410),
         location=Location.EXTERNAL,
         pair=TradePair('BTC_USD'),
         trade_type=TradeType.BUY,
         amount=AssetAmount(FVal('0.00100000')),
         rate=Price(ZERO),
         fee=Fee(ZERO),
-        fee_currency=A_BTC,
+        fee_currency=A_USD,
         link='',
         notes='Just a small gift from someone',
+    ), Trade(
+        timestamp=Timestamp(1567504805),
+        location=Location.EXTERNAL,
+        pair=TradePair('ETH_USD'),
+        trade_type=TradeType.BUY,
+        amount=AssetAmount(FVal('2')),
+        rate=Price(ZERO),
+        fee=Fee(ZERO),
+        fee_currency=A_USD,
+        link='',
+        notes='Sign up bonus',
     )]
     assert expected_trades == trades
 
     expected_movements = [AssetMovement(
         location=Location.POLONIEX,
         category=AssetMovementCategory.DEPOSIT,
-        timestamp=Timestamp(1565848620),
+        timestamp=Timestamp(1565848624),
         asset=A_XMR,
         amount=AssetAmount(FVal('5')),
-        fee_asset=A_XMR,
+        fee_asset=A_USD,
         fee=Fee(ZERO),
         link='',
     ), AssetMovement(
         location=Location.COINBASE,
         category=AssetMovementCategory.WITHDRAWAL,
-        timestamp=Timestamp(1566726120),
+        timestamp=Timestamp(1566726155),
         asset=A_ETH,
         amount=AssetAmount(FVal('0.05770427')),
         fee_asset=A_ETH,
-        fee=Fee(ZERO),
+        fee=Fee(FVal("0.0001")),
         link='',
     )]
     assert expected_movements == asset_movements