Repository: rotki/rotki

Issue:
SubstrateManager update type registry during instantiation
Currently SubstrateManager is not [updating the type registry](https://github.com/polkascan/py-scale-codec#using-the-type-registry-updater-in-your-application). We must get up-to-date the codecs of the chains.

At some point they will be [available via the metadata endpoint.](https://github.com/paritytech/substrate/issues/7971) and there won't be any need to update them as per described in this issue.

Why is important?

> It is essential to keep these type registries up to date, there is for example an upcoming change for Kusama where the Address and LookupSource types will be changed (MultiAddress support) and this will break all signed extrinsics if not updated..


Discussion:
When is that update going to happen? We need to know to know when to schedule this fix.
It seems not until March according to the issue comments
Still need it soon then. This is a month. Needs to be in one of the next 3 releases, assuming we do 1 tomorrow and one every 2 weeks.

When you say update the type registry it essentially means to update the json? We need to see how these json are stored also then and what happens with pyinstaller and if this happens in runtime then where would the app store them.
> Still need it soon then. This is a month. Needs to be in one of the next 3 releases, assuming we do 1 tomorrow and one every 2 weeks.
> 
> When you say update the type registry it essentially means to update the json? We need to see how these json are stored also then and what happens with pyinstaller and if this happens in runtime then where would the app store them.

Until the codecs don't come via the metadata endpoint, we need to add [this call](https://github.com/polkascan/py-scale-codec#using-the-type-registry-updater-in-your-application) when the server starts. According to Arjan, it retrieves the latest type registry presets from Github and replaces the local ones on disk. The advantage is that you don't have to rely on the latest update of both Polkascan libraries. Agree with your doubts re pyinstaller and storage.
[Good news](https://github.com/polkascan/py-substrate-interface#keeping-type-registry-presets-up-to-date)

I've tested the new version of py-substrate-interface (v.0.11.16) and its new argument `use_remote_preset` and it works. Although it has required to amend some exception messages.

Pending to address why now some tests in `test_susbtrate_utils.py` fail. It seems that `is_valid_substrate_address` returns True for a substrate public address (which does not have the ss58 2 format).  `ss58_decode` could be misbehaving.

Resolving patch:
diff --git a/rotkehlchen/chain/substrate/manager.py b/rotkehlchen/chain/substrate/manager.py
--- a/rotkehlchen/chain/substrate/manager.py
+++ b/rotkehlchen/chain/substrate/manager.py
@@ -378,12 +378,6 @@ def _get_last_block(self, node_interface: SubstrateInterface) -> BlockNumber:
         except (
             requests.exceptions.RequestException,
             SubstrateRequestException,
-            # TODO: remove TypeError once py-susbtrate-interface `get_block_number`
-            # handles a None response. Keep ValueError just in case `get_chain_head`
-            # returns None.
-            # https://github.com/polkascan/py-substrate-interface/issues/68
-            TypeError,
-            ValueError,
             WebSocketException,
         ) as e:
             message = (
@@ -417,6 +411,7 @@ def _get_node_interface(self, endpoint: str) -> SubstrateInterface:
             node_interface = SubstrateInterface(
                 url=endpoint,
                 type_registry_preset=si_attributes.type_registry_preset,
+                use_remote_preset=True,
             )
         except (requests.exceptions.RequestException, WebSocketException) as e:
             message = (
diff --git a/rotkehlchen/chain/substrate/utils.py b/rotkehlchen/chain/substrate/utils.py
--- a/rotkehlchen/chain/substrate/utils.py
+++ b/rotkehlchen/chain/substrate/utils.py
@@ -29,12 +29,25 @@ def is_valid_substrate_address(
 ) -> bool:
     """Validates a ss58 encoded substrate address for the given chain.
 
+    TODO: this function relies on py-scale-codec `ss58_decode()` for validating
+    that a str is a valid substrate address for the given chain. The recent
+    changes introduced on the py-scale-codec library have altered the behavior
+    of this function by validating positively any str starting with `0x`. An
+    issue has been opened regarding this matter (link below). Once py-scale-codec
+    implements its own address validation method this function may be no longer
+    needed. Issue:
+    https://github.com/polkascan/py-scale-codec/issues/27
+
     Polkascan ss58 utils documentation:
     https://github.com/polkascan/py-substrate-interface/blob/master/substrateinterface/utils/ss58.py  # noqa: E501
 
     External Address Format (SS58) documentation:
     https://github.com/paritytech/substrate/wiki/External-Address-Format-(SS58)
     """
+    if value.startswith('0x'):
+        # TODO: temporary patch for py-scale-codec/issue/27
+        return False
+
     if chain == SubstrateChain.KUSAMA:
         valid_ss58_format = 2
     else:
@@ -47,6 +60,7 @@ def is_valid_substrate_address(
         )
     except ValueError:
         return False
+
     return True
 
 
diff --git a/tools/pyinstaller_hooks/hook-scalecodec.py b/tools/pyinstaller_hooks/hook-scalecodec.py
deleted file mode 100644
--- a/tools/pyinstaller_hooks/hook-scalecodec.py
+++ /dev/null
@@ -1,5 +0,0 @@
-from PyInstaller.utils.hooks import collect_data_files
-
-# scalecodec has a lot of json files from which it loads the types
-# we need to make sure pyinstaller bundles them in the created binary
-datas = collect_data_files('scalecodec')


Test patch:
diff --git a/rotkehlchen/tests/unit/test_substrate_manager.py b/rotkehlchen/tests/unit/test_substrate_manager.py
--- a/rotkehlchen/tests/unit/test_substrate_manager.py
+++ b/rotkehlchen/tests/unit/test_substrate_manager.py
@@ -49,7 +49,7 @@ def test_get_account_balance_invalid_account(kusama_manager):
     )
     account_balance_errors = [error for error in errors if error.startswith(user_error_msg)]
     for error in account_balance_errors:
-        assert 'Invalid Address type' in error
+        assert 'Invalid SS58 format' in error
 
     assert 'Kusama request failed after trying the following nodes' in str(e.value)
 
@@ -73,7 +73,7 @@ def test_get_accounts_balance_invalid_account(kusama_manager):
     )
     account_balance_errors = [error for error in errors if error.startswith(user_error_msg)]
     for error in account_balance_errors:
-        assert 'Invalid Address type' in error
+        assert 'Invalid SS58 format' in error
 
     assert 'Kusama request failed after trying the following nodes' in str(e.value)