Repository: zalando/patroni

Issue:
Question: DCS Read timeout error (3.333 seconds)
Hello team,
I am using patroni with consul. The setup was working fine. Recently got random master down alerts.
Patroni connection to consul gets timed out at 3.33 seconds and it shutsdown master and brings it back up in few seconds.

```
Patroni-   ttl: 30,  loop_wait: 10,  retry_timeout: 10 
consul raft_multiplier : 5 
```

```

WARNING: Retry got exception: HTTPConnectionPool(host='master_ip', port=8500): Max retries exceeded with url: /v1/kv/service/pgcluster/?recurse=1 (Caused by ReadTimeoutError("HTTPConnectionPool(host='master_ip', port=8500): 
Read timed out. (read timeout=3.3333333333333335)",))
ERROR: get_cluster
patroni.utils.RetryFailedError: 'Exceeded retry deadline'
ERROR: Error communicating with DCS
INFO: closed patroni connection to the postgresql cluster
INFO: demoted self because DCS is not accessible and i was a leader
WARNING: Loop time exceeded, rescheduling immediately.
INFO: promoted self to leader by acquiring session lock
server promoting
INFO: cleared rewind state after becoming the leader

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 423, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='master_ip', port=8500): Read timed out. (read timeout=3.3333333333333335)
```

I have below questions

1. How is the read timeout being calculated ?
2. Which parameter can we change to increase this.
3. Is this consul related?
4. Any best practices to resolve this ?




Discussion:
The total timeout equals 10 seconds (retry_timeout), which is devided by three connection attempts. So in fact, most likely demote happened after 10 seconds.

When using consul as DCS you should always configure Patroni to connect to the local consul agent.
Besides that it is highly recommended to disable serfHeath check. The final configuration should look like:
```yaml
consul:
  host: '127.0.0.1:8500'
  checks: []
```
Thanks for quick reply. Will apply this configuration.

Resolving patch:
diff --git a/patroni/dcs/consul.py b/patroni/dcs/consul.py
--- a/patroni/dcs/consul.py
+++ b/patroni/dcs/consul.py
@@ -217,7 +217,7 @@ def __init__(self, config):
         self.set_retry_timeout(config['retry_timeout'])
         self.set_ttl(config.get('ttl') or 30)
         self._last_session_refresh = 0
-        self.__session_checks = config.get('checks')
+        self.__session_checks = config.get('checks', [])
         self._register_service = config.get('register_service', False)
         if self._register_service:
             self._service_name = service_name_from_scope_name(self._scope)


Test patch: