Repository: rotki/rotki

Issue:
Error at querying blockchain balances
## Problem Definition

When I start and login it starts syncing and after a while a get the following error in logs.

![image](https://user-images.githubusercontent.com/1188574/100088162-04ef8700-2e50-11eb-94ae-49ab88637585.png)


## Logs

Error at querying blockchain balances: The backend query task died unexpectedly: {'code': -32005, 'message': 'query returned more than 10000 results'}

### System Description


Operating system: OSX
Rotki version: 1.9

I use Infura


Discussion:
Hey @froid1911 thanks for the issue. Do you have a premium account? And would it be possible for you to maybe run this with debug logs, recreate it and give us the logs. The way to run with debug logs on is detailed here: https://rotki.readthedocs.io/en/latest/contribute.html#id1

Also if you are not in our discord please join, and DM me the logs or even just the address for which you see it so I can debug and fix this: https://discord.gg/aGCxHG7
The same error was just seen in Github actions, in a test using infura.

Github actions run: https://github.com/rotki/rotki/pull/1786/checks?check_run_id=1447281738
Raw logs: https://pipelines.actions.githubusercontent.com/pzx0zrXmeq6QIlZyZm9oxyZTZyxPul0UQTasPXHu9fMGJXpoxp/_apis/pipelines/1/runs/1861/signedlogcontent/30?urlExpires=2020-11-24T14%3A31%3A05.8888583Z&urlSigningMethod=HMACV1&urlSignature=cZKdf4uQ6Dmn9m9D5uZMbF%2FyErsMj%2Fn%2FHZ89gAISIwI%3D

Relevant part of the exception:

```
2020-11-24T11:32:08.2589227Z DEBUG    rotkehlchen.chain.ethereum.manager:manager.py:650 Querying node for contract event contract_address=0x00000000219ab540356cBB839Cbe05303d7705Fa, event_name=DepositEvent, argument_filters={}, from_block=11302985, to_block=11320717
2020-11-24T11:32:08.2591126Z DEBUG    web3.RequestManager:manager.py:127 Making request. Method: eth_getLogs
2020-11-24T11:32:08.2592712Z DEBUG    web3.providers.HTTPProvider:rpc.py:83 Making request HTTP. URI: https://mainnet.infura.io/v3/66302b8fb9874614905a3cbe903a0dbb, Method: eth_getLogs
2020-11-24T11:32:08.2595943Z DEBUG    web3.providers.HTTPProvider:rpc.py:93 Getting response HTTP. URI: https://mainnet.infura.io/v3/66302b8fb9874614905a3cbe903a0dbb, Method: eth_getLogs, Response: {'jsonrpc': '2.0', 'id': 19, 'error': {'code': -32005, 'message': 'query returned more than 10000 results'}}
2020-11-24T11:32:08.2597970Z ERROR    rotkehlchen.api.server:app.py:1891 Exception on /api/1/blockchains/ETH/modules/yearn/vaults/history [GET]
2020-11-24T11:32:08.2599122Z Traceback (most recent call last):
2020-11-24T11:32:08.2600720Z   File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
2020-11-24T11:32:08.2601741Z     rv = self.dispatch_request()
2020-11-24T11:32:08.2603112Z   File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
2020-11-24T11:32:08.2604188Z     return self.view_functions[rule.endpoint](**req.view_args)
2020-11-24T11:32:08.2605676Z   File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/flask_restful/__init__.py", line 458, in wrapper
2020-11-24T11:32:08.2606598Z     resp = resource(*args, **kwargs)
2020-11-24T11:32:08.2607868Z   File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/flask/views.py", line 89, in view
2020-11-24T11:32:08.2608830Z     return self.dispatch_request(*args, **kwargs)
2020-11-24T11:32:08.2610537Z   File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
2020-11-24T11:32:08.2611456Z     resp = meth(*args, **kwargs)
2020-11-24T11:32:08.2612735Z   File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/webargs/core.py", line 366, in wrapper
2020-11-24T11:32:08.2613619Z     return func(*args, **kwargs)
2020-11-24T11:32:08.2614507Z   File "/home/runner/work/rotki/rotki/rotkehlchen/api/v1/resources.py", line 983, in get
2020-11-24T11:32:08.2615276Z     to_timestamp=to_timestamp,
2020-11-24T11:32:08.2616168Z   File "/home/runner/work/rotki/rotki/rotkehlchen/api/rest.py", line 166, in wrapper
2020-11-24T11:32:08.2616963Z     return f(wrappingobj, *args, **kwargs)
2020-11-24T11:32:08.2617960Z   File "/home/runner/work/rotki/rotki/rotkehlchen/api/rest.py", line 1759, in get_yearn_vaults_history
2020-11-24T11:32:08.2618772Z     to_timestamp=to_timestamp,
2020-11-24T11:32:08.2619712Z   File "/home/runner/work/rotki/rotki/rotkehlchen/api/rest.py", line 1604, in _api_query_for_eth_module
2020-11-24T11:32:08.2620425Z     **kwargs,
2020-11-24T11:32:08.2621318Z   File "/home/runner/work/rotki/rotki/rotkehlchen/api/rest.py", line 1562, in _eth_module_query
2020-11-24T11:32:08.2622259Z     blockchain=SupportedBlockchain.ETHEREUM,
2020-11-24T11:32:08.2623290Z   File "/home/runner/work/rotki/rotki/rotkehlchen/utils/interfaces.py", line 111, in wrapper
2020-11-24T11:32:08.2624089Z     result = f(wrappingobj, *args, **kwargs)
2020-11-24T11:32:08.2625073Z   File "/home/runner/work/rotki/rotki/rotkehlchen/utils/interfaces.py", line 70, in wrapper
2020-11-24T11:32:08.2625865Z     result = f(wrappingobj, *args, **kwargs)
2020-11-24T11:32:08.2626777Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/manager.py", line 380, in query_balances
2020-11-24T11:32:08.2627957Z     self.query_ethereum_balances(force_token_detection=force_token_detection)
2020-11-24T11:32:08.2629038Z   File "/home/runner/work/rotki/rotki/rotkehlchen/utils/interfaces.py", line 111, in wrapper
2020-11-24T11:32:08.2630279Z     result = f(wrappingobj, *args, **kwargs)
2020-11-24T11:32:08.2631356Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/manager.py", line 838, in query_ethereum_balances
2020-11-24T11:32:08.2632211Z     self._add_protocol_balances()
2020-11-24T11:32:08.2633103Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/manager.py", line 874, in _add_protocol_balances
2020-11-24T11:32:08.2633959Z     self.get_staked_eth2_balances()
2020-11-24T11:32:08.2634879Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/manager.py", line 976, in get_staked_eth2_balances
2020-11-24T11:32:08.2635721Z     database=self.database,
2020-11-24T11:32:08.2636643Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/ethereum/eth2.py", line 202, in get_eth2_staked_amount
2020-11-24T11:32:08.2637402Z     to_ts=to_ts,
2020-11-24T11:32:08.2638279Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/ethereum/eth2.py", line 114, in _get_eth2_staked_amount_onchain
2020-11-24T11:32:08.2639326Z     to_block='latest',
2020-11-24T11:32:08.2640169Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/ethereum/contracts.py", line 52, in get_logs
2020-11-24T11:32:08.2640974Z     call_order=call_order,
2020-11-24T11:32:08.2641799Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/ethereum/manager.py", line 605, in get_logs
2020-11-24T11:32:08.2642702Z     to_block=to_block,
2020-11-24T11:32:08.2643515Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/ethereum/manager.py", line 275, in query
2020-11-24T11:32:08.2644308Z     result = method(web3, **kwargs)
2020-11-24T11:32:08.2645189Z   File "/home/runner/work/rotki/rotki/rotkehlchen/chain/ethereum/manager.py", line 654, in _get_logs
2020-11-24T11:32:08.2646184Z     new_events_web3 = cast(List[Dict[str, Any]], web3.eth.getLogs(filter_args))
2020-11-24T11:32:08.2647552Z   File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/web3/eth.py", line 503, in getLogs
2020-11-24T11:32:08.2648450Z     RPC.eth_getLogs, [filter_params],
2020-11-24T11:32:08.2649931Z   File "/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/web3/manager.py", line 153, in request_blocking
2020-11-24T11:32:08.2650954Z     raise ValueError(response["error"])
2020-11-24T11:32:08.2651925Z ValueError: {'code': -32005, 'message': 'query returned more than 10000 results'}
2020-11-24T11:32:08.2652778Z travis_fold:end:py-8199.log.1
```

The failed test seems to be

tests/api/test_yearn_vaults.py::test_query_yearn_vault_history

Seems like it's an infura issue with the way `eth_getLogs()` works from here: https://infura.io/docs/ethereum/json-rpc/eth-getLogs

Resolving patch:
diff --git a/rotkehlchen/chain/ethereum/manager.py b/rotkehlchen/chain/ethereum/manager.py
--- a/rotkehlchen/chain/ethereum/manager.py
+++ b/rotkehlchen/chain/ethereum/manager.py
@@ -1,3 +1,4 @@
+import json
 import logging
 import random
 from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union, cast
@@ -17,6 +18,7 @@
 from web3._utils.filters import construct_event_filter_params
 from web3.datastructures import MutableAttributeDict
 from web3.middleware.exception_retry_request import http_retry_request_middleware
+from web3.types import FilterParams
 
 from rotkehlchen.chain.ethereum.transactions import EthTransactions
 from rotkehlchen.constants.ethereum import ETH_SCAN
@@ -65,6 +67,72 @@ def _is_synchronized(current_block: int, latest_block: int) -> Tuple[bool, str]:
     return True, message
 
 
+WEB3_LOGQUERY_BLOCK_RANGE = 250000
+
+
+def _query_web3_get_logs(
+        web3: Web3,
+        filter_args: FilterParams,
+        from_block: int,
+        to_block: Union[int, Literal['latest']],
+        contract_address: ChecksumEthAddress,
+        event_name: str,
+        argument_filters: Dict[str, Any],
+) -> List[Dict[str, Any]]:
+    until_block = web3.eth.blockNumber if to_block == 'latest' else to_block
+    events: List[Dict[str, Any]] = []
+    start_block = from_block
+    block_range = WEB3_LOGQUERY_BLOCK_RANGE
+    while start_block <= until_block:
+        filter_args['fromBlock'] = start_block
+        end_block = min(start_block + block_range, until_block)
+        filter_args['toBlock'] = end_block
+        log.debug(
+            'Querying web3 node for contract event',
+            contract_address=contract_address,
+            event_name=event_name,
+            argument_filters=argument_filters,
+            from_block=filter_args['fromBlock'],
+            to_block=filter_args['toBlock'],
+        )
+        # As seen in https://github.com/rotki/rotki/issues/1787, the json RPC, if it
+        # is infura can throw an error here which we can only parse by catching the  exception
+        try:
+            new_events_web3 = cast(List[Dict[str, Any]], web3.eth.getLogs(filter_args))
+        except ValueError as e:
+            try:
+                decoded_error = json.loads(str(e).replace("'", '"'))
+            except json.JSONDecodeError:
+                # reraise the value error if the error is not json
+                raise e from None
+
+            msg = decoded_error.get('message', '')
+            # errors from: https://infura.io/docs/ethereum/json-rpc/eth-getLogs
+            if msg in ('query returned more than 10000 results', 'query timeout exceeded'):
+                # repeat the query with smaller block range
+                block_range = int(block_range / 2)
+                continue
+            else:
+                # well we tried .. reraise the Value error
+                raise e
+
+        # Turn all HexBytes into hex strings
+        for e_idx, event in enumerate(new_events_web3):
+            new_events_web3[e_idx]['blockHash'] = event['blockHash'].hex()
+            new_topics = []
+            for topic in (event['topics']):
+                new_topics.append(topic.hex())
+            new_events_web3[e_idx]['topics'] = new_topics
+            new_events_web3[e_idx]['transactionHash'] = event['transactionHash'].hex()
+
+        start_block = end_block + 1
+        events.extend(new_events_web3)
+        # end of the loop, end of 1 query. Reset the block range to max
+        block_range = WEB3_LOGQUERY_BLOCK_RANGE
+
+    return events
+
+
 OPEN_NODES = (
     NodeName.MYCRYPTO,
     NodeName.BLOCKSCOUT,
@@ -636,33 +704,15 @@ def _get_logs(
         events: List[Dict[str, Any]] = []
         start_block = from_block
         if web3 is not None:
-            until_block = web3.eth.blockNumber if to_block == 'latest' else to_block
-            while start_block <= until_block:
-                filter_args['fromBlock'] = start_block
-                end_block = min(start_block + 250000, until_block)
-                filter_args['toBlock'] = end_block
-                log.debug(
-                    'Querying node for contract event',
-                    contract_address=contract_address,
-                    event_name=event_name,
-                    argument_filters=argument_filters,
-                    from_block=filter_args['fromBlock'],
-                    to_block=filter_args['toBlock'],
-                )
-                # WTF: for some reason the first time we get in here the loop resets
-                # to the start without querying eth_getLogs and ends up with double logging
-                new_events_web3 = cast(List[Dict[str, Any]], web3.eth.getLogs(filter_args))
-                # Turn all HexBytes into hex strings
-                for e_idx, event in enumerate(new_events_web3):
-                    new_events_web3[e_idx]['blockHash'] = event['blockHash'].hex()
-                    new_topics = []
-                    for topic in (event['topics']):
-                        new_topics.append(topic.hex())
-                    new_events_web3[e_idx]['topics'] = new_topics
-                    new_events_web3[e_idx]['transactionHash'] = event['transactionHash'].hex()
-
-                start_block = end_block + 1
-                events.extend(new_events_web3)
+            events = _query_web3_get_logs(
+                web3=web3,
+                filter_args=filter_args,
+                from_block=from_block,
+                to_block=to_block,
+                contract_address=contract_address,
+                event_name=event_name,
+                argument_filters=argument_filters,
+            )
         else:  # etherscan
             until_block = (
                 self.etherscan.get_latest_block_number() if to_block == 'latest' else to_block


Test patch: