Repository: rotki/rotki

Issue:
User ended up with a corrupt token entry in the DB after conflict during upgrade
## Problem Definition

A user contacted us via email and shared some screenshots which gave funny errors when trying to find/insert a token which should have been added via the v1 assets upgrade. The user said they had it already locally but opted to solve the confict by choosing remote.

I got their global.db and it seems that something is wrong.

```
sqlite> SELECT * from assets WHERE identifier="_ceth_0xf1f955016EcbCd7321c7266BccFB96c68ea5E49b";
_ceth_0xf1f955016EcbCd7321c7266BccFB96c68ea5E49b|C|Rally|RLY|||rally-2||0xf1f955016EcbCd7321c7266BccFB96c68ea5E49b

sqlite> SELECT * from ethereum_tokens where address="0xf1f955016EcbCd7321c7266BccFB96c68ea5E49b";
sqlite> 
```
So the asset has an entry in the `assets` table and not in the `ethereum_tokens` table. 

## Task

Figure out how this could have happened and if it's something on our side fix.
User ended up with a corrupt token entry in the DB after conflict during upgrade
## Problem Definition

A user contacted us via email and shared some screenshots which gave funny errors when trying to find/insert a token which should have been added via the v1 assets upgrade. The user said they had it already locally but opted to solve the confict by choosing remote.

I got their global.db and it seems that something is wrong.

```
sqlite> SELECT * from assets WHERE identifier="_ceth_0xf1f955016EcbCd7321c7266BccFB96c68ea5E49b";
_ceth_0xf1f955016EcbCd7321c7266BccFB96c68ea5E49b|C|Rally|RLY|||rally-2||0xf1f955016EcbCd7321c7266BccFB96c68ea5E49b

sqlite> SELECT * from ethereum_tokens where address="0xf1f955016EcbCd7321c7266BccFB96c68ea5E49b";
sqlite> 
```
So the asset has an entry in the `assets` table and not in the `ethereum_tokens` table. 

## Task

Figure out how this could have happened and if it's something on our side fix.


Discussion:
Locally I've tried:

- Exactly the same info local and remote -> DB is ok
- Same address different symbol -> DB is ok
- Tried insert -> delete -> update -> DB is ok
- Also tried from 1.15.X in case something special happened for one version
- Adding account with this asset and try update -> DB is OK
- Adding account with this asset -> delete asset -> update -> Db is OK

From reading the code I can't see a path where an insert of the asset will happen but not for the ethereum token. Need to think more ways of breaking it. 
I don't know how to reproduce this. If we get some more reports of users who got this or if we manage to do it ourselves let's reopen. But closing it as not reproducible.
Well taking it back. Happened again. We can't brush this off under the rug. We need to figure it out.
The second user who saw this said they had those assets manually inserted before. Then those assets are also part of the new upgrade. Which causes the conflict. And at the conflict they chose to keep remote. And this happened.
I reproduced this. I know what's wrong. It's going to take some thinking to fix since it's a bit specific AND some of our assumptions about how sqlite works seem to have been wrong.
Locally I've tried:

- Exactly the same info local and remote -> DB is ok
- Same address different symbol -> DB is ok
- Tried insert -> delete -> update -> DB is ok
- Also tried from 1.15.X in case something special happened for one version
- Adding account with this asset and try update -> DB is OK
- Adding account with this asset -> delete asset -> update -> Db is OK

From reading the code I can't see a path where an insert of the asset will happen but not for the ethereum token. Need to think more ways of breaking it. 
I don't know how to reproduce this. If we get some more reports of users who got this or if we manage to do it ourselves let's reopen. But closing it as not reproducible.
Well taking it back. Happened again. We can't brush this off under the rug. We need to figure it out.
The second user who saw this said they had those assets manually inserted before. Then those assets are also part of the new upgrade. Which causes the conflict. And at the conflict they chose to keep remote. And this happened.
I reproduced this. I know what's wrong. It's going to take some thinking to fix since it's a bit specific AND some of our assumptions about how sqlite works seem to have been wrong.

Resolving patch:
diff --git a/rotkehlchen/globaldb/handler.py b/rotkehlchen/globaldb/handler.py
--- a/rotkehlchen/globaldb/handler.py
+++ b/rotkehlchen/globaldb/handler.py
@@ -51,7 +51,7 @@ def _initialize_temp_db_directory() -> TemporaryDirectory:
     return tempdir
 
 
-def _initialize_globaldb(dbpath: Path) -> sqlite3.Connection:
+def initialize_globaldb(dbpath: Path) -> sqlite3.Connection:
     connection = sqlite3.connect(dbpath)
     connection.executescript(DB_SCRIPT_CREATE_TABLES)
     cursor = connection.cursor()
@@ -81,7 +81,7 @@ def _initialize_global_db_directory(data_dir: Path) -> sqlite3.Connection:
         root_dir = Path(__file__).resolve().parent.parent
         builtin_data_dir = root_dir / 'data'
         shutil.copyfile(builtin_data_dir / 'global.db', global_dir / 'global.db')
-    return _initialize_globaldb(dbname)
+    return initialize_globaldb(dbname)
 
 
 class GlobalDBHandler():
@@ -123,7 +123,7 @@ def __new__(
             tempdir = _initialize_temp_db_directory()
             GlobalDBHandler.__instance._temp_db_directory = tempdir
             dbname = Path(tempdir.name) / 'global.db'
-            GlobalDBHandler.__instance._conn = _initialize_globaldb(dbname)
+            GlobalDBHandler.__instance._conn = initialize_globaldb(dbname)
         else:  # probably tests
             GlobalDBHandler.__instance._data_directory = data_dir
             GlobalDBHandler.__instance._conn = _initialize_global_db_directory(data_dir)
diff --git a/rotkehlchen/globaldb/updates.py b/rotkehlchen/globaldb/updates.py
--- a/rotkehlchen/globaldb/updates.py
+++ b/rotkehlchen/globaldb/updates.py
@@ -3,6 +3,8 @@
 import re
 import sqlite3
 import sys
+from pathlib import Path
+from tempfile import TemporaryDirectory
 from typing import Any, Dict, List, NamedTuple, Optional, Tuple, Union
 
 import requests
@@ -16,7 +18,7 @@
 from rotkehlchen.typing import ChecksumEthAddress, Timestamp
 from rotkehlchen.user_messages import MessagesAggregator
 
-from .handler import GlobalDBHandler
+from .handler import GlobalDBHandler, initialize_globaldb
 
 log = logging.getLogger(__name__)
 
@@ -35,12 +37,36 @@ def executeall(cursor: sqlite3.Cursor, statements: str) -> None:
         cursor.execute(statement)
 
 
+def _replace_assets_from_db(
+        connection: sqlite3.Connection,
+        sourcedb_path: Path,
+) -> None:
+    cursor = connection.cursor()
+    cursor.executescript(f"""
+    ATTACH DATABASE "{sourcedb_path}" AS other_db;
+    PRAGMA foreign_keys = OFF;
+    DELETE FROM assets;
+    DELETE FROM ethereum_tokens;
+    DELETE FROM underlying_tokens_list;
+    DELETE FROM common_asset_details;
+    INSERT INTO assets SELECT * FROM other_db.assets;
+    INSERT INTO ethereum_tokens SELECT * FROM other_db.ethereum_tokens;
+    INSERT INTO underlying_tokens_list SELECT * FROM other_db.underlying_tokens_list;
+    INSERT INTO common_asset_details SELECT * FROM other_db.common_asset_details;
+    UPDATE settings SET value=(
+        SELECT value FROM other_db.settings WHERE name="{ASSETS_VERSION_KEY}"
+    ) WHERE name="{ASSETS_VERSION_KEY}";
+    PRAGMA foreign_keys = ON;
+    DETACH DATABASE "other_db";
+    """)
+
+
 def _force_remote(cursor: sqlite3.Cursor, local_asset: Asset, full_insert: str) -> None:
     """Force the remote entry into the database by deleting old one and doing the full insert.
 
     May raise an sqlite3 error if something fails.
     """
-    cursor.execute('PRAGMA foreign_keys=off;')
+    cursor.executescript('BEGIN TRANSACTION; PRAGMA foreign_keys=off; COMMIT;')
     # Delete the old entry
     if local_asset.asset_type == AssetType.ETHEREUM_TOKEN:
         token = EthereumToken.from_asset(local_asset)
@@ -57,10 +83,10 @@ def _force_remote(cursor: sqlite3.Cursor, local_asset: Asset, full_insert: str)
         'DELETE FROM assets WHERE identifier=?;',
         (local_asset.identifier,),
     )
+    cursor.executescript('BEGIN TRANSACTION; PRAGMA foreign_keys=on; COMMIT;')
     # Insert new entry. Since identifiers are the same, no foreign key constrains should break
     executeall(cursor, full_insert)
     AssetResolver().clean_memory_cache(local_asset.identifier.lower())
-    cursor.execute('PRAGMA foreign_keys=on;')
 
 
 class ParsedAssetData(NamedTuple):
@@ -316,7 +342,10 @@ def _apply_single_version_update(
                 self.conflicts.append((local_data, remote_asset_data))
 
         # at the very end update the current version in the DB
-        GlobalDBHandler().add_setting_value(ASSETS_VERSION_KEY, version, commit=False)
+        cursor.execute(
+            'INSERT OR REPLACE INTO settings(name, value) VALUES(?, ?)',
+            (ASSETS_VERSION_KEY, str(version)),
+        )
 
     def perform_update(
             self,
@@ -337,12 +366,47 @@ def perform_update(
         self.conflicts = []  # reset the stored conflicts
         infojson = self._get_remote_info_json()
         local_schema_version = GlobalDBHandler().get_schema_version()
+        data_directory = GlobalDBHandler()._data_directory
+        assert data_directory is not None, 'data directory should be initialized at this point'
+        global_db_path = data_directory / 'global_data' / 'global.db'
+        with TemporaryDirectory() as tmpdirname:
+            tempdbpath = Path(tmpdirname) / 'temp.db'
+            connection = initialize_globaldb(tempdbpath)
+            _replace_assets_from_db(connection, global_db_path)
+            self._perform_update(
+                connection=connection,
+                conflicts=conflicts,
+                local_schema_version=local_schema_version,
+                infojson=infojson,
+                up_to_version=up_to_version,
+            )
+            if len(self.conflicts) != 0:
+                # close the temporary DB and return the conflicts
+                connection.close()
+                return [
+                    {'identifier': x[0].identifier, 'local': x[0].serialize(), 'remote': x[1].serialize()}  # noqa: E501
+                    for x in self.conflicts
+                ]
+
+            # otherwise we are sure the DB will work without conflicts so let's
+            # now move the data to the actual global DB
+            connection.close()
+            connection = GlobalDBHandler()._conn
+            _replace_assets_from_db(connection, tempdbpath)
+            return None
+
+    def _perform_update(
+            self,
+            connection: sqlite3.Connection,
+            conflicts: Optional[Dict[Asset, Literal['remote', 'local']]],
+            local_schema_version: int,
+            infojson: Dict[str, Any],
+            up_to_version: Optional[int],
+    ) -> None:
         version = self.local_assets_version + 1
-        connection = GlobalDBHandler()._conn
-        cursor = connection.cursor()
         target_version = min(up_to_version, self.last_remote_checked_version) if up_to_version else self.last_remote_checked_version   # type: ignore # noqa: E501
         # type ignore since due to check_for_updates we know last_remote_checked_version exists
-
+        cursor = connection.cursor()
         while version <= target_version:
             try:
                 min_schema_version = infojson['updates'][str(version)]['min_schema_version']
@@ -355,7 +419,10 @@ def perform_update(
                         f'You will have to follow an alternative method to '
                         f'obtain the assets of this update.',
                     )
-                    GlobalDBHandler().add_setting_value(ASSETS_VERSION_KEY, version, commit=False)
+                    cursor.execute(
+                        'INSERT OR REPLACE INTO settings(name, value) VALUES(?, ?)',
+                        (ASSETS_VERSION_KEY, str(version)),
+                    )
                     version += 1
                     continue
             except KeyError as e:
@@ -390,11 +457,7 @@ def perform_update(
 
         if self.conflicts == []:
             connection.commit()
-            return None
+            return
 
         # In this case we have conflicts. Everything should also be rolled back
         connection.rollback()
-        return [
-            {'identifier': x[0].identifier, 'local': x[0].serialize(), 'remote': x[1].serialize()}
-            for x in self.conflicts
-        ]


Test patch:
diff --git a/rotkehlchen/tests/api/test_assets_updates.py b/rotkehlchen/tests/api/test_assets_updates.py
--- a/rotkehlchen/tests/api/test_assets_updates.py
+++ b/rotkehlchen/tests/api/test_assets_updates.py
@@ -10,6 +10,7 @@
 
 from rotkehlchen.assets.asset import Asset, EthereumToken
 from rotkehlchen.assets.typing import AssetType
+from rotkehlchen.chain.ethereum.typing import CustomEthereumToken
 from rotkehlchen.constants.resolver import strethaddress_to_identifier
 from rotkehlchen.errors import UnknownAsset
 from rotkehlchen.globaldb.handler import GLOBAL_DB_VERSION
@@ -22,6 +23,7 @@
 )
 from rotkehlchen.tests.utils.constants import A_GLM
 from rotkehlchen.tests.utils.mock import MockResponse
+from rotkehlchen.typing import ChecksumEthAddress
 
 
 def mock_asset_updates(original_requests_get, latest: int, updates: Dict[str, Any], sql_actions: Dict[str, str]):  # noqa: E501
@@ -188,8 +190,27 @@ def test_update_conflicts(rotkehlchen_api_server, globaldb):
 INSERT INTO ethereum_tokens(address, decimals, protocol) VALUES("0x6B175474E89094C44Da98b954EedeAC495271d0F", 8, "maker");INSERT INTO assets(identifier,type, name, symbol,started, swapped_for, coingecko, cryptocompare, details_reference) VALUES("_ceth_0x6B175474E89094C44Da98b954EedeAC495271d0F", "C", "New Multi Collateral DAI", "NDAI", 1573672677, NULL, "dai", NULL, "0x6B175474E89094C44Da98b954EedeAC495271d0F");
 *
 INSERT INTO assets(identifier,type,name,symbol,started, swapped_for, coingecko, cryptocompare, details_reference) VALUES("DASH", "B","Dash","DASH",1337, NULL, "dash-coingecko", NULL, "DASH");INSERT INTO common_asset_details(asset_id, forked) VALUES("DASH", "BTC");
+*
+INSERT INTO ethereum_tokens(address, decimals, protocol) VALUES("0x1B175474E89094C44Da98b954EedeAC495271d0F", 18, NULL); INSERT INTO assets(identifier,type, name, symbol,started, swapped_for, coingecko, cryptocompare, details_reference) VALUES("_ceth_0x1B175474E89094C44Da98b954EedeAC495271d0F", "C", "Conflicting token", "CTK", 1573672677, NULL, "ctk", NULL, "0x1B175474E89094C44Da98b954EedeAC495271d0F");
 *
     """  # noqa: E501
+    globaldb.add_asset(  # add a conflicting token
+        asset_id='_ceth_0x1B175474E89094C44Da98b954EedeAC495271d0F',
+        asset_type=AssetType.ETHEREUM_TOKEN,
+        data=CustomEthereumToken(
+            address=ChecksumEthAddress('0x1B175474E89094C44Da98b954EedeAC495271d0F'),
+            decimals=12,
+            name='Conflicting token',
+            symbol='CTK',
+            started=None,
+            swapped_for=None,
+            coingecko='ctk',
+            cryptocompare=None,
+            protocol=None,
+            underlying_tokens=None,
+        ),
+    )
+    globaldb.add_user_owned_assets([Asset('_ceth_0x1B175474E89094C44Da98b954EedeAC495271d0F')])
     update_patch = mock_asset_updates(
         original_requests_get=requests.get,
         latest=999999991,
@@ -255,7 +276,7 @@ def test_update_conflicts(rotkehlchen_api_server, globaldb):
         warnings = rotki.msg_aggregator.consume_warnings()
         assert len(errors) == 0, f'Found errors: {errors}'
         assert len(warnings) == 0, f'Found warnings: {warnings}'
-        # See that we get two conflicts
+        # See that we get 3 conflicts
         expected_result = [{
             'identifier': '_ceth_0x6B175474E89094C44Da98b954EedeAC495271d0F',
             'local': {
@@ -312,11 +333,39 @@ def test_update_conflicts(rotkehlchen_api_server, globaldb):
                 'coingecko': 'dash-coingecko',
                 'protocol': None,
             },
+        }, {
+            'identifier': '_ceth_0x1B175474E89094C44Da98b954EedeAC495271d0F',
+            'local': {
+                'asset_type': 'ethereum token',
+                'coingecko': 'ctk',
+                'cryptocompare': None,
+                'decimals': 12,
+                'ethereum_address': '0x1B175474E89094C44Da98b954EedeAC495271d0F',
+                'forked': None,
+                'name': 'Conflicting token',
+                'protocol': None,
+                'started': None,
+                'swapped_for': None,
+                'symbol': 'CTK',
+            },
+            'remote': {
+                'asset_type': 'ethereum token',
+                'coingecko': 'ctk',
+                'cryptocompare': None,
+                'decimals': 18,
+                'ethereum_address': '0x1b175474E89094C44DA98B954EeDEAC495271d0f',
+                'forked': None,
+                'name': 'Conflicting token',
+                'protocol': None,
+                'started': 1573672677,
+                'swapped_for': None,
+                'symbol': 'CTK',
+            },
         }]
         assert result == expected_result
 
         # now try the update again but specify the conflicts resolution
-        conflicts = {'_ceth_0x6B175474E89094C44Da98b954EedeAC495271d0F': 'remote', 'DASH': 'local'}
+        conflicts = {'_ceth_0x6B175474E89094C44Da98b954EedeAC495271d0F': 'remote', 'DASH': 'local', '_ceth_0x1B175474E89094C44Da98b954EedeAC495271d0F': 'remote'}  # noqa: E501
         response = requests.post(
             api_url_for(
                 rotkehlchen_api_server,
@@ -339,6 +388,7 @@ def test_update_conflicts(rotkehlchen_api_server, globaldb):
                 status_code=HTTPStatus.OK,
             )
 
+        cursor = globaldb._conn.cursor()
         # check conflicts were solved as per the given choices and new asset also added
         assert result is True
         assert globaldb.get_setting_value(ASSETS_VERSION_KEY, None) == 999999991
@@ -359,6 +409,9 @@ def test_update_conflicts(rotkehlchen_api_server, globaldb):
         assert dai.ethereum_address == '0x6B175474E89094C44Da98b954EedeAC495271d0F'
         assert dai.decimals == 8
         assert dai.protocol == 'maker'
+        # make sure data is in both tables
+        assert cursor.execute('SELECT COUNT(*) from ethereum_tokens WHERE address="0x6B175474E89094C44Da98b954EedeAC495271d0F";').fetchone()[0] == 1  # noqa: E501
+        assert cursor.execute('SELECT COUNT(*) from assets WHERE identifier="_ceth_0x6B175474E89094C44Da98b954EedeAC495271d0F";').fetchone()[0] == 1  # noqa: E501
 
         dash = Asset('DASH')
         assert dash.identifier == 'DASH'
@@ -370,6 +423,8 @@ def test_update_conflicts(rotkehlchen_api_server, globaldb):
         assert dash.swapped_for is None
         assert dash.coingecko == 'dash'
         assert dash.cryptocompare is None
+        assert cursor.execute('SELECT COUNT(*) from common_asset_details WHERE asset_id="DASH";').fetchone()[0] == 1  # noqa: E501
+        assert cursor.execute('SELECT COUNT(*) from assets WHERE identifier="DASH";').fetchone()[0] == 1  # noqa: E501
 
         new_asset = Asset('121-ada-FADS-as')
         assert new_asset.identifier == '121-ada-FADS-as'
@@ -381,13 +436,32 @@ def test_update_conflicts(rotkehlchen_api_server, globaldb):
         assert new_asset.swapped_for is None
         assert new_asset.coingecko == ''
         assert new_asset.cryptocompare == ''
+        assert cursor.execute('SELECT COUNT(*) from common_asset_details WHERE asset_id="121-ada-FADS-as";').fetchone()[0] == 1  # noqa: E501
+        assert cursor.execute('SELECT COUNT(*) from assets WHERE identifier="121-ada-FADS-as";').fetchone()[0] == 1  # noqa: E501
+
+        ctk = EthereumToken('0x1B175474E89094C44Da98b954EedeAC495271d0F')
+        assert ctk.name == 'Conflicting token'
+        assert ctk.symbol == 'CTK'
+        assert ctk.asset_type == AssetType.ETHEREUM_TOKEN
+        assert ctk.started == 1573672677
+        assert ctk.forked is None
+        assert ctk.swapped_for is None
+        assert ctk.coingecko == 'ctk'
+        assert ctk.cryptocompare is None
+        assert ctk.ethereum_address == '0x1B175474E89094C44Da98b954EedeAC495271d0F'
+        assert ctk.decimals == 18
+        assert ctk.protocol is None
+        assert cursor.execute('SELECT COUNT(*) from ethereum_tokens WHERE address="0x1B175474E89094C44Da98b954EedeAC495271d0F";').fetchone()[0] == 1  # noqa: E501
+        assert cursor.execute('SELECT COUNT(*) from assets WHERE identifier="_ceth_0x1B175474E89094C44Da98b954EedeAC495271d0F";').fetchone()[0] == 1  # noqa: E501
 
 
 @pytest.mark.parametrize('use_clean_caching_directory', [True])
 def test_foreignkey_conflict(rotkehlchen_api_server, globaldb):
     """Test that when a conflict that's not solvable happens the entry is ignored
 
-    One such case is when the update of an asset would violate a foreign key constraint"""
+    One such case is when the update of an asset would violate a foreign key constraint.
+    So we try to update the swapped_for to a non existing asset and make sure it's skipped.
+    """
     async_query = random.choice([False, True])
     rotki = rotkehlchen_api_server.rest_api.rotkehlchen
     update_1 = """INSERT INTO assets(identifier,type,name,symbol,started, swapped_for, coingecko, cryptocompare, details_reference) VALUES("121-ada-FADS-as", "F","A name","SYMBOL",NULL, NULL,"", "", "121-ada-FADS-as");INSERT INTO common_asset_details(asset_id, forked) VALUES("121-ada-FADS-as", "BTC");
diff --git a/rotkehlchen/tests/fixtures/rotkehlchen.py b/rotkehlchen/tests/fixtures/rotkehlchen.py
--- a/rotkehlchen/tests/fixtures/rotkehlchen.py
+++ b/rotkehlchen/tests/fixtures/rotkehlchen.py
@@ -129,8 +129,11 @@ def mock_get_settings() -> DBSettings:
         '_connect_ksm_manager_on_startup',
         return_value=bool(blockchain_accounts.ksm),
     )
-
-    with settings_patch, eth_rpcconnect_patch, ksm_rpcconnect_patch, ksm_connect_on_startup_patch:
+    # patch the constants to make sure that the periodic query for icons
+    # does not run during tests
+    size_patch = patch('rotkehlchen.rotkehlchen.ICONS_BATCH_SIZE', new=0)
+    sleep_patch = patch('rotkehlchen.rotkehlchen.ICONS_QUERY_SLEEP', new=999999)
+    with settings_patch, eth_rpcconnect_patch, ksm_rpcconnect_patch, ksm_connect_on_startup_patch, size_patch, sleep_patch:  # noqa: E501
         rotki.unlock_user(
             user=username,
             password=db_password,
@@ -186,12 +189,7 @@ def fixture_uninitialized_rotkehlchen(cli_args, inquirer, asset_resolver, global
 
     Adding the AssetResolver as a requirement so that the first initialization happens here
     """
-    # patch the constants to make sure that the periodic query for icons
-    # does not run during tests
-    size_patch = patch('rotkehlchen.rotkehlchen.ICONS_BATCH_SIZE', new=0)
-    sleep_patch = patch('rotkehlchen.rotkehlchen.ICONS_QUERY_SLEEP', new=999999)
-    with size_patch, sleep_patch:
-        rotki = Rotkehlchen(cli_args)
+    rotki = Rotkehlchen(cli_args)
     return rotki