Repository: rotki/rotki

Issue:
Inputting an address for an ethereum RPC node can throw 500 if an http exception is raised
## Problem Definition

A user tried to input a quicknode URL as the ethereum rpc endpoint for Rotki. In one case it worked and in the other it did not with Rotki throwing a 500 error.

Relevant logs:

```
25/04/2020 13:11:04 EDT -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-04-25 13:11:04] "OPTIONS /api/1/settings HTTP/1.1" 200 319 0.001073
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: web3_clientVersion
25/04/2020 13:11:04 EDT -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"warnings": [], "errors": []}, "message": ""}, status_code=200
25/04/2020 13:11:04 EDT -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-04-25 13:11:04] "GET /api/1/messages/ HTTP/1.1" 200 245 0.000892
25/04/2020 13:11:04 EDT -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"last_balance_save": 1587833627, "eth_node_connection": false, "history_process_start_ts": -1, "history_process_current_ts": -1}, "message": ""}, status_code=200
25/04/2020 13:11:04 EDT -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-04-25 13:11:04] "GET /api/1/periodic/ HTTP/1.1" 200 345 0.000973
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Getting response HTTP. URI: https://<edited>, Method: web3_clientVersion, Response: {'jsonrpc': '2.0', 'result': 'Parity-Ethereum//v2.5.13-stable-253ff3f-20191231/x86_64-linux-gnu/rustc1.40.0', 'id': 0}
25/04/2020 13:11:04 EDT -- DEBUG:web3.RequestManager:Making request. Method: eth_chainId
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_chainId
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Getting response HTTP. URI: https://<edited>, Method: eth_chainId, Response: {'jsonrpc': '2.0', 'result': '0x1', 'id': 1}
25/04/2020 13:11:04 EDT -- DEBUG:web3.RequestManager:Making request. Method: eth_syncing
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:05 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:05 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:05 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:05 EDT -- ERROR:rotkehlchen.api.server:Exception on /api/1/settings [PUT]
Traceback (most recent call last):
  File "flask/app.py", line 1949, in full_dispatch_request
  File "flask/app.py", line 1935, in dispatch_request
  File "flask_restful/__init__.py", line 458, in wrapper
  File "flask/views.py", line 89, in view
  File "flask_restful/__init__.py", line 573, in dispatch_request
  File "webargs/core.py", line 366, in wrapper
  File "rotkehlchen/api/v1/resources.py", line 175, in put
  File "rotkehlchen/api/rest.py", line 109, in wrapper
  File "rotkehlchen/api/rest.py", line 247, in set_settings
  File "rotkehlchen/rotkehlchen.py", line 601, in set_settings
  File "rotkehlchen/chain/manager.py", line 182, in set_eth_rpc_endpoint
  File "rotkehlchen/chain/ethereum/manager.py", line 161, in set_rpc_endpoint
  File "rotkehlchen/chain/ethereum/manager.py", line 103, in attempt_connect
  File "web3/eth.py", line 78, in syncing
  File "web3/manager.py", line 109, in request_blocking
  File "web3/manager.py", line 96, in _make_request
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/gas_price_strategy.py", line 18, in middleware
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/attrdict.py", line 18, in middleware
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/normalize_errors.py", line 9, in middleware
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/exception_retry_request.py", line 83, in middleware
  File "web3/providers/rpc.py", line 68, in make_request
  File "web3/_utils/request.py", line 27, in make_post_request
  File "requests/models.py", line 940, in raise_for_status
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://<edited>
```

It's due to Rotki not handling http requests to an own node like other http requests and not catching failures properly.

## Task

Handle http errors and if it's a 429 retry again after backing off for a few seconds.
Inputting an address for an ethereum RPC node can throw 500 if an http exception is raised
## Problem Definition

A user tried to input a quicknode URL as the ethereum rpc endpoint for Rotki. In one case it worked and in the other it did not with Rotki throwing a 500 error.

Relevant logs:

```
25/04/2020 13:11:04 EDT -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-04-25 13:11:04] "OPTIONS /api/1/settings HTTP/1.1" 200 319 0.001073
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: web3_clientVersion
25/04/2020 13:11:04 EDT -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"warnings": [], "errors": []}, "message": ""}, status_code=200
25/04/2020 13:11:04 EDT -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-04-25 13:11:04] "GET /api/1/messages/ HTTP/1.1" 200 245 0.000892
25/04/2020 13:11:04 EDT -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"last_balance_save": 1587833627, "eth_node_connection": false, "history_process_start_ts": -1, "history_process_current_ts": -1}, "message": ""}, status_code=200
25/04/2020 13:11:04 EDT -- INFO:rotkehlchen.api.server.pywsgi:127.0.0.1 - - [2020-04-25 13:11:04] "GET /api/1/periodic/ HTTP/1.1" 200 345 0.000973
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Getting response HTTP. URI: https://<edited>, Method: web3_clientVersion, Response: {'jsonrpc': '2.0', 'result': 'Parity-Ethereum//v2.5.13-stable-253ff3f-20191231/x86_64-linux-gnu/rustc1.40.0', 'id': 0}
25/04/2020 13:11:04 EDT -- DEBUG:web3.RequestManager:Making request. Method: eth_chainId
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_chainId
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Getting response HTTP. URI: https://<edited>, Method: eth_chainId, Response: {'jsonrpc': '2.0', 'result': '0x1', 'id': 1}
25/04/2020 13:11:04 EDT -- DEBUG:web3.RequestManager:Making request. Method: eth_syncing
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:04 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:05 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:05 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:05 EDT -- DEBUG:web3.providers.HTTPProvider:Making request HTTP. URI: https://<edited>, Method: eth_syncing
25/04/2020 13:11:05 EDT -- ERROR:rotkehlchen.api.server:Exception on /api/1/settings [PUT]
Traceback (most recent call last):
  File "flask/app.py", line 1949, in full_dispatch_request
  File "flask/app.py", line 1935, in dispatch_request
  File "flask_restful/__init__.py", line 458, in wrapper
  File "flask/views.py", line 89, in view
  File "flask_restful/__init__.py", line 573, in dispatch_request
  File "webargs/core.py", line 366, in wrapper
  File "rotkehlchen/api/v1/resources.py", line 175, in put
  File "rotkehlchen/api/rest.py", line 109, in wrapper
  File "rotkehlchen/api/rest.py", line 247, in set_settings
  File "rotkehlchen/rotkehlchen.py", line 601, in set_settings
  File "rotkehlchen/chain/manager.py", line 182, in set_eth_rpc_endpoint
  File "rotkehlchen/chain/ethereum/manager.py", line 161, in set_rpc_endpoint
  File "rotkehlchen/chain/ethereum/manager.py", line 103, in attempt_connect
  File "web3/eth.py", line 78, in syncing
  File "web3/manager.py", line 109, in request_blocking
  File "web3/manager.py", line 96, in _make_request
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/gas_price_strategy.py", line 18, in middleware
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/attrdict.py", line 18, in middleware
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/normalize_errors.py", line 9, in middleware
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/exception_retry_request.py", line 83, in middleware
  File "web3/providers/rpc.py", line 68, in make_request
  File "web3/_utils/request.py", line 27, in make_post_request
  File "requests/models.py", line 940, in raise_for_status
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://<edited>
```

It's due to Rotki not handling http requests to an own node like other http requests and not catching failures properly.

## Task

Handle http errors and if it's a 429 retry again after backing off for a few seconds.


Discussion:

Resolving patch:
diff --git a/rotkehlchen/chain/ethereum/manager.py b/rotkehlchen/chain/ethereum/manager.py
--- a/rotkehlchen/chain/ethereum/manager.py
+++ b/rotkehlchen/chain/ethereum/manager.py
@@ -9,6 +9,7 @@
 from web3._utils.abi import get_abi_output_types
 from web3._utils.contracts import find_matching_event_abi
 from web3._utils.filters import construct_event_filter_params
+from web3.middleware.exception_retry_request import http_retry_request_middleware
 
 from rotkehlchen.assets.asset import EthereumToken
 from rotkehlchen.constants.ethereum import ETH_SCAN_ABI, ETH_SCAN_ADDRESS
@@ -75,6 +76,7 @@ def attempt_connect(
                 request_kwargs={'timeout': self.eth_rpc_timeout},
             )
             self.web3 = Web3(provider)
+            self.web3.middleware_onion.inject(http_retry_request_middleware, layer=0)
         except requests.exceptions.ConnectionError:
             log.warning('Could not connect to an ethereum node. Will use etherscan only')
             self.connected = False


Test patch:
diff --git a/rotkehlchen/tests/utils/mock.py b/rotkehlchen/tests/utils/mock.py
--- a/rotkehlchen/tests/utils/mock.py
+++ b/rotkehlchen/tests/utils/mock.py
@@ -31,10 +31,17 @@ def getBlock(  # noqa: N802 pylint: disable=no-self-use
         return {'hash': HexBytes(genesis)}
 
 
+class MockMiddlewareOnion():
+
+    def inject(self, middleware, layer) -> None:
+        pass
+
+
 class MockWeb3():
 
     def __init__(self, _providers=None, _middlewares=None):
         self.eth = MockEth(0)
+        self.middleware_onion = MockMiddlewareOnion()
 
     def isConnected(self) -> bool:  # noqa: N802 pylint: disable=no-self-use
         return True