Repository: rotki/rotki

Issue:
Upgrading to 1.26.0: FOREIGN key error during DB upgrade under very specific circumstances
## Problem Definition

A single user could not upgrade from version 1.25.3 to 1.26.0. They were getting Foreign key error during their DB upgrade.

Since we already had more than 200 users upgrade in the last day alone it was obvious this was some edge case.

The user was super kind to provide logs so we noticed it was inside 
`_update_history_event_assets_identifiers_to_caip_format`
https://github.com/rotki/rotki/blob/ced7952b652b2fe05f4447a775a3962ba8c06abc/rotkehlchen/db/upgrades/v34_v35.py#L215

What's more they provided access to the DB, so reproducing was easy and with a debugger we figured out the problem.

When recreating the history events table the FOREIGN key relation to the assets table is enforced during this upgrade. This relation was not existant in the last version. What is not checked during the upgrade and before creating the relation though is if each single history event's asset does match the one in the assets table.

And this is what bit us here. The user had an asset in a history event, possibly detected automatically through the transaction decoders, which was not in the assets table. There can be many reasons for this discrepancy ... user moved something, deleted some assets.

But it was not protected against so it's possible to happen and it happened with one user. So we need to handle it.

## Task

Handle it and protect against it.

I think the best way to handle it is to actually add the asset to the user DB assets table. Even if missing from global DB it's fine as the user will just get a warning whenever this history event is supposed to be loaded and they can simply use the warning and add the asset to the global DB themselves.

The alternative approach is to:
1. do nothing -> **bad** for some users like this guy they won't be able to upgrade to 1.26
2. if asset is missing skip the entire event -> less bad, but still bad as we got incomplete history                                           


Discussion:

Resolving patch:
diff --git a/rotkehlchen/db/upgrades/v34_v35.py b/rotkehlchen/db/upgrades/v34_v35.py
--- a/rotkehlchen/db/upgrades/v34_v35.py
+++ b/rotkehlchen/db/upgrades/v34_v35.py
@@ -2,6 +2,8 @@
 import logging
 from typing import TYPE_CHECKING, List, Literal, Sequence, Tuple, Type
 
+from pysqlcipher3 import dbapi2 as sqlcipher
+
 from rotkehlchen.assets.spam_assets import update_spam_assets
 from rotkehlchen.constants.resolver import (
     ETHEREUM_DIRECTIVE,
@@ -249,7 +251,8 @@ def _update_history_event_assets_identifiers_to_caip_format(cursor: 'DBCursor')
     FOREIGN KEY(asset) REFERENCES assets(identifier) ON UPDATE CASCADE,
     UNIQUE(event_identifier, sequence_index)
     );""")
-    cursor.executemany(
+
+    insertion_query = (
         'INSERT INTO history_events( '
         'identifier, '
         'event_identifier, '
@@ -264,9 +267,23 @@ def _update_history_event_assets_identifiers_to_caip_format(cursor: 'DBCursor')
         'type, '
         'subtype, '
         'counterparty, '
-        'extra_data) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)',
-        new_entries,
+        'extra_data) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)'
     )
+    try:
+        cursor.executemany(insertion_query, new_entries)
+    except sqlcipher.IntegrityError:  # pylint: disable=no-member
+        # handle https://github.com/rotki/rotki/issues/5052
+        for entry in new_entries:
+            try:
+                cursor.execute(insertion_query, entry)
+            except sqlcipher.IntegrityError as e:  # pylint: disable=no-member
+                if 'UNIQUE constraint failed: history_events.identifier' in str(e):
+                    continue  # already input by executemany
+
+                # otherwise can only be foreign key error. insert the missing asset that lead to it
+                cursor.execute('INSERT OR IGNORE INTO assets(identifier) VALUES(?)', (entry[6],))
+                cursor.execute(insertion_query, entry)
+
     log.debug('Exit _update_history_event_assets_identifiers_to_caip_format')
 
 


Test patch:
diff --git a/rotkehlchen/tests/data/v34_rotkehlchen.db b/rotkehlchen/tests/data/v34_rotkehlchen.db
Binary files a/rotkehlchen/tests/data/v34_rotkehlchen.db and b/rotkehlchen/tests/data/v34_rotkehlchen.db differ
diff --git a/rotkehlchen/tests/db/test_db_upgrades.py b/rotkehlchen/tests/db/test_db_upgrades.py
--- a/rotkehlchen/tests/db/test_db_upgrades.py
+++ b/rotkehlchen/tests/db/test_db_upgrades.py
@@ -818,13 +818,22 @@ def test_upgrade_db_34_to_35(user_data_dir):  # pylint: disable=unused-argument
         assert tokens['tokens'] == ['_ceth_0x0bc529c00C6401aEF6D220BE8C6Ea1667F6Ad93e']
         # check that history events contain a transaction with an old style token identifier
         old_tx_assets_ids = cursor.execute('SELECT DISTINCT asset FROM history_events;').fetchall()  # noqa: E501
+        asset_missing_from_assets_table = '_ceth_0xb3d608c31ACa7a1c7D6DAcec5978E5493181b67A'
         assert old_tx_assets_ids == [
             ('ETH',),
             ('AVAX',),
             ('BTC',),
             ('_ceth_0x429881672B9AE42b8EbA0E26cD9C73711b891Ca5',),
+            (asset_missing_from_assets_table,),
         ]
 
+        # check that _ceth_0xb3d608c31ACa7a1c7D6DAcec5978E5493181b67A is not in assets table
+        # essentially reproduce: https://github.com/rotki/rotki/issues/5052
+        asset_in_table = cursor.execute(
+            'SELECT COUNT(*) FROM assets WHERE identifier=?', (asset_missing_from_assets_table,),
+        ).fetchone()[0]
+        assert asset_in_table == 0, 'asset should be in history event but not in assets table'
+
         # Check that oracles exist in the test db
         oracles_before_upgrade = cursor.execute(
             'SELECT value FROM settings WHERE name="current_price_oracles"',
@@ -944,6 +953,7 @@ def try_insert_mapping(cur):
             assert new_ignored_assets_id in new_ignored_assets_ids
         assert len(new_ignored_assets_ids) >= 135
 
+        fixed_asset_id = 'eip155:1/erc20:0xb3d608c31ACa7a1c7D6DAcec5978E5493181b67A'
         # check that history events contain a transaction with new style token identifier
         new_tx_assets_ids = cursor.execute('SELECT DISTINCT asset FROM history_events;').fetchall()  # noqa: E501
         assert new_tx_assets_ids == [
@@ -951,7 +961,12 @@ def try_insert_mapping(cur):
             ('AVAX',),
             ('BTC',),
             ('eip155:1/erc20:0x429881672B9AE42b8EbA0E26cD9C73711b891Ca5',),
+            (fixed_asset_id,),
         ]
+        asset_in_table = cursor.execute(
+            'SELECT COUNT(*) FROM assets WHERE identifier=?', (fixed_asset_id,),
+        ).fetchone()[0]
+        assert asset_in_table == 1, 'asset should now be in both history events and asset table'
 
         # Check that token_list for accounts has been correctly upgraded
         cursor.execute('SELECT value from accounts_details WHERE account="0x45E6CA515E840A4e9E02A3062F99216951825eB2" AND blockchain="eth" AND key="tokens"')  # noqa: E501