Repository: rotki/rotki

Issue:
Failed at database upgrade from version 31 to 32
## Problem Definition

Login fails with following message:
`Failed at database upgrade from version 31 to 32: UNIQUE constraint failed: history_events_copy.event_identifier, history_events_copy.sequence_index`

## Logs

Which logs do you need, if you need any?

### System Description

Operating system: Debian Buster (using the AppImage)
Rotki version: 1.24.0



Discussion:
Hey hello @HydraGene. This is quite weird. I am not sure why this would happen but we can investigate if you can provide us with the database and the password. You can do so via a discord DM: https://discord.com/invite/aGCxHG7
(have DMs open there is a captcha challenge)

## What we need

rotki would have taken a copy of the database right before attempting the DB upgrade. So we need this. You can also revert to this and use the previous version until we figure out what's wrong.

Since you are in Linux the database is:

`~/.local/share/rotki/data/XXX` where `XXX` is your username.

The current database is `rotkehlchen.db`. Right before a DB upgrade a backup is taken so it should look like `1653690238_rotkehlchen_db_v31.backup` where the initial number is timestamp of the backup and vXX is the version of the DB at the time.

Please send me that file and the password in order to be able to debug this. Logs will not help with such an error.

## How to keep using rotki in the meantime

You can copy that backup file and rename `rotkehlchen.db`, replacing the database with the backup eventually. And then open rotki again with 1.23.4 until we figure out what's wrong.

For the devs: By looking at the code it has to be a special case of data that the function `_update_history_entries_from_kraken`
does not properly take care of. https://github.com/rotki/rotki/blob/6b85c5c53b8eb0af20c4cdb780fb3fec2c65f68f/rotkehlchen/db/upgrades/v31_v32.py#L134 

I tried to just add a duplicate sequence index and event identifier in the DB upgrade kraken data and the sequence index was changed and upgrade worked. So the only logical explanation is that there must be a case not covered by the upgrade.

Resolving patch:
diff --git a/rotkehlchen/db/upgrades/v31_v32.py b/rotkehlchen/db/upgrades/v31_v32.py
--- a/rotkehlchen/db/upgrades/v31_v32.py
+++ b/rotkehlchen/db/upgrades/v31_v32.py
@@ -142,22 +142,20 @@ def _update_history_entries_from_kraken(cursor: 'Cursor') -> None:
     DELETE FROM history_events where location="B" AND asset="KFEE" AND
      type="trade" AND subtype=NULL;
     """)
+
+    cursor.execute('SELECT event_identifier, sequence_index from history_events')
+    eventid_to_indices: Dict[str, Set[int]] = defaultdict(set)
+    for event_identifier, sequence_index in cursor:
+        eventid_to_indices[event_identifier].add(sequence_index)
+
     cursor.execute("""
     SELECT e.event_identifier, e.sequence_index, e.identifier from history_events e JOIN (SELECT event_identifier,
     sequence_index, COUNT(*) as cnt FROM history_events GROUP BY event_identifier, sequence_index)
     other ON e.event_identifier = other.event_identifier and e.sequence_index=other.sequence_index
     WHERE other.cnt > 1;
     """)  # noqa: E501
-
     update_tuples = []
-    eventid_to_indices: Dict[str, Set[int]] = defaultdict(set)
     for event_identifier, sequence_index, identifier in cursor:
-        last_indices = eventid_to_indices.get(event_identifier)
-        if last_indices is None:
-            # Let the first one be the same as it was in the database
-            eventid_to_indices[event_identifier].add(sequence_index)
-            continue
-
         new_index = sequence_index + 1
         while new_index in eventid_to_indices[event_identifier]:
             new_index += 1


Test patch:
diff --git a/rotkehlchen/tests/data/v31_rotkehlchen.db b/rotkehlchen/tests/data/v31_rotkehlchen.db
Binary files a/rotkehlchen/tests/data/v31_rotkehlchen.db and b/rotkehlchen/tests/data/v31_rotkehlchen.db differ
diff --git a/rotkehlchen/tests/db/test_db_upgrades.py b/rotkehlchen/tests/db/test_db_upgrades.py
--- a/rotkehlchen/tests/db/test_db_upgrades.py
+++ b/rotkehlchen/tests/db/test_db_upgrades.py
@@ -2428,7 +2428,7 @@ def test_upgrade_db_31_to_32(user_data_dir):  # pylint: disable=unused-argument
     cursor = db_v31.conn.cursor()
     result = cursor.execute('SELECT rowid from history_events')
     old_ids = {row[0] for row in result}
-    assert len(old_ids) == 15
+    assert len(old_ids) == 19
     cursor.execute(
         'SELECT subtype from history_events',
     )
@@ -2521,7 +2521,7 @@ def test_upgrade_db_31_to_32(user_data_dir):  # pylint: disable=unused-argument
     subtypes = {row[0] for row in cursor}
     assert subtypes == {'deposit asset', 'receive wrapped', 'reward', 'fee', None, 'remove asset'}
     result = cursor.execute('SELECT identifier FROM history_events ORDER BY identifier')
-    assert [x[0] for x in result] == list(range(1, 16)), 'identifier column should be added'
+    assert [x[0] for x in result] == list(range(1, 20)), 'identifier column should be added'
     # check used query range got delete and rest are intact
     result = cursor.execute('SELECT * from used_query_ranges').fetchall()
     assert result == [