Repository: rotki/rotki

Issue:
With slow connection and many tokens/addresses balance query may throw exception at etherscan call
## Problem Definition

This has been noticed a few times for slow connections and if many tokens and addresses are involved. At some point the etherscan balance query may fail with the following stack trace and crash the application:

```
Traceback (most recent call last):
  File "/home/lefteris/.virtualenvs/rotkehlchen/bin/zerorpc", line 11, in <module>
    sys.exit(main())
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/zerorpc/cli.py", line 310, in main
    return run_client(args)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/zerorpc/cli.py", line 269, in run_client
    results = client(args.command, *call_args)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/zerorpc/core.py", line 270, in __call__
    return self._process_response(request_event, bufchan, timeout)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/zerorpc/core.py", line 238, in _process_response
    reply_event, self._handle_remote_error)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/zerorpc/patterns.py", line 45, in process_answer
    raise exception
zerorpc.exceptions.RemoteError: Traceback (most recent call last):
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/gevent/_socketcommon.py", line 208, in getaddrinfo
    return get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/gevent/resolver/thread.py", line 65, in getaddrinfo
    return self.pool.apply(_socket.getaddrinfo, args, kwargs)
File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/gevent/pool.py", line 159, in apply                                                                                                                                                           [0/172]
    return self.spawn(func, *args, **kwds).get()
  File "src/gevent/event.py", line 381, in gevent._event.AsyncResult.get
  File "src/gevent/event.py", line 409, in gevent._event.AsyncResult.get
  File "src/gevent/event.py", line 399, in gevent._event.AsyncResult.get
  File "src/gevent/event.py", line 379, in gevent._event.AsyncResult._raise_exception
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/gevent/_compat.py", line 47, in reraise
    raise value.with_traceback(tb)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/gevent/threadpool.py", line 281, in _worker
    value = func(*args, **kwargs)
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a9b736470>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.etherscan.io', port=443): Max retries exceeded with url: /api?module=account&action=tokenbalance&contractaddress=0xXX&address=0xXX (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a9b736470>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/zerorpc/core.py", line 153, in _async_task
    functor.pattern.process_call(self._context, bufchan, event, functor)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/zerorpc/patterns.py", line 30, in process_call
    result = functor(*req_event.args)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/zerorpc/decorators.py", line 44, in __call__
    return self._functor(*args, **kargs)
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/server.py", line 225, in query_balances
    result = self.rotkehlchen.query_balances(save_data)
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/rotkehlchen.py", line 429, in query_balances
    result, error_or_empty = self.blockchain.query_balances()
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/utils.py", line 92, in wrapper
    result = f(wrappingobj, *args)
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/blockchain.py", line 83, in query_balances
    self.query_ethereum_balances()
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/blockchain.py", line 354, in query_ethereum_balances
    self.query_ethereum_tokens(self.owned_eth_tokens, eth_balances)
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/blockchain.py", line 319, in query_ethereum_tokens
    self.accounts.eth,
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/ethchain.py", line 191, in get_multitoken_balance
    account,
  File "/home/lefteris/w/rotkehlchen/rotkehlchen/utils.py", line 201, in request_get
    response = requests.get(uri)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/lefteris/.virtualenvs/rotkehlchen/lib/python3.7/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.etherscan.io', port=443): Max retries exceeded with url: /api?module=account&action=tokenbalance&contractaddress=0xXX&address=0xXX (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f4a9b736470>: Failed to establish a new connection: [Errno -2] Name or service not known'))
```


Discussion:

Resolving patch:
diff --git a/rotkehlchen/utils.py b/rotkehlchen/utils.py
--- a/rotkehlchen/utils.py
+++ b/rotkehlchen/utils.py
@@ -198,7 +198,16 @@ def retry_calls(
 
 
 def request_get(uri, timeout=ALL_REMOTES_TIMEOUT):
-    response = requests.get(uri)
+    # TODO make this a bit more smart. Perhaps conditional on the type of request.
+    # Not all requests would need repeated attempts
+    response = retry_calls(
+        5,
+        '',
+        uri,
+        requests.get,
+        uri,
+    )
+
     if response.status_code != 200:
         raise RemoteError('Get {} returned status code {}'.format(uri, response.status_code))
 


Test patch: