Repository: zalando/patroni
Pull request #1997 (2021-07-04 13:17:37)

Fix typos discovered by codespell
This pull request was created by [`codespell --ignore-words-list="datas" -w .`](https://github.com/codespell-project/codespell)


Patch:
diff --git a/README.rst b/README.rst
index b26cfafe..e40abb76 100644
--- a/README.rst
+++ b/README.rst
@@ -119,7 +119,7 @@ For example, the command in order to install Patroni together with dependencies
 
     pip install patroni[etcd,aws]
 
-Note that external tools to call in the replica creation or custom bootstap scripts (i.e. WAL-E) should be installed independently of Patroni.
+Note that external tools to call in the replica creation or custom bootstrap scripts (i.e. WAL-E) should be installed independently of Patroni.
 
 =======================
 Running and Configuring
diff --git a/docs/ENVIRONMENT.rst b/docs/ENVIRONMENT.rst
index 618497aa..7c837a6b 100644
--- a/docs/ENVIRONMENT.rst
+++ b/docs/ENVIRONMENT.rst
@@ -166,6 +166,8 @@ REST API
 -  **PATRONI\_RESTAPI\_CAFILE**: Specifies the file with the CA_BUNDLE with certificates of trusted CAs to use while verifying client certs.
 -  **PATRONI\_RESTAPI\_CIPHERS**: (optional) Specifies the permitted cipher suites (e.g. "ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:!SSLv1:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1")
 -  **PATRONI\_RESTAPI\_VERIFY\_CLIENT**: ``none`` (default), ``optional`` or ``required``. When ``none`` REST API will not check client certificates. When ``required`` client certificates are required for all REST API calls. When ``optional`` client certificates are required for all unsafe REST API endpoints. When ``required`` is used, then client authentication succeeds, if the certificate signature verification succeeds. For ``optional`` the client cert will only be checked for ``PUT``, ``POST``, ``PATCH``, and ``DELETE`` requests.
+-  **PATRONI\_RESTAPI\_ALLOWLIST**: (optional): Specifies the set of hosts that are allowed to call unsafe REST API endpoints. The single element could be a host name, an IP address or a network address using CIDR notation. By default ``allow all`` is used. In case if ``allowlist`` or ``allowlist_include_members`` are set, anything that is not included is rejected.
+-  **PATRONI\_RESTAPI\_ALLOWLIST\_INCLUDE\_MEMBERS**: (optional): If set to ``true`` it allows accessing unsafe REST API endpoints from other cluster members registered in DCS (IP address or hostname is taken from the members ``api_url``). Be careful, it might happen that OS will use a different IP for outgoing connections.
 -  **PATRONI\_RESTAPI\_HTTP\_EXTRA\_HEADERS**: (optional) HTTP headers let the REST API server pass additional information with an HTTP response.
 -  **PATRONI\_RESTAPI\_HTTPS\_EXTRA\_HEADERS**: (optional) HTTPS headers let the REST API server pass additional information with an HTTP response when TLS is enabled. This will also pass additional information set in ``http_extra_headers``.
 
diff --git a/docs/README.rst b/docs/README.rst
index f0aab0fd..ee8f0c20 100644
--- a/docs/README.rst
+++ b/docs/README.rst
@@ -166,7 +166,7 @@ When connecting from an application, always use a non-superuser. Patroni require
 
 Testing Your HA Solution
 --------------------------------------
-Testing an HA solution is a time consuming process, with many variables. This is particularly true considering a cross-platform application. You need a trained system administrator or a consultant to do this work. It is not something we can cover in depth in the documentaiton.
+Testing an HA solution is a time consuming process, with many variables. This is particularly true considering a cross-platform application. You need a trained system administrator or a consultant to do this work. It is not something we can cover in depth in the documentation.
 
 That said, here are some pieces of your infrastructure you should be sure to test:
 
diff --git a/docs/SETTINGS.rst b/docs/SETTINGS.rst
index 664a62c8..89400b03 100644
--- a/docs/SETTINGS.rst
+++ b/docs/SETTINGS.rst
@@ -228,7 +228,7 @@ Raft
 
   - Q: Where to get the ``syncobj_admin`` utility?
 
-    A: It is installed together with ``pysyncobj`` module (python RAFT implementation), which is Patroni dependancy.
+    A: It is installed together with ``pysyncobj`` module (python RAFT implementation), which is Patroni dependency.
 
   - Q: it is possible to run Patroni node without adding in to the consensus?
 
@@ -327,6 +327,8 @@ REST API
         -  **cafile**: (optional): Specifies the file with the CA_BUNDLE with certificates of trusted CAs to use while verifying client certs.
         -  **ciphers**: (optional): Specifies the permitted cipher suites (e.g. "ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:!SSLv1:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1")
         -  **verify\_client**: (optional): ``none`` (default), ``optional`` or ``required``. When ``none`` REST API will not check client certificates. When ``required`` client certificates are required for all REST API calls. When ``optional`` client certificates are required for all unsafe REST API endpoints. When ``required`` is used, then client authentication succeeds, if the certificate signature verification succeeds.  For ``optional`` the client cert will only be checked for ``PUT``, ``POST``, ``PATCH``, and ``DELETE`` requests.
+        -  **allowlist**: (optional): Specifies the set of hosts that are allowed to call unsafe REST API endpoints. The single element could be a host name, an IP address or a network address using CIDR notation. By default ``allow all`` is used. In case if ``allowlist`` or ``allowlist_include_members`` are set, anything that is not included is rejected.
+        -  **allowlist\_include\_members**: (optional): If set to ``true`` it allows accessing unsafe REST API endpoints from other cluster members registered in DCS (IP address or hostname is taken from the members ``api_url``). Be careful, it might happen that OS will use a different IP for outgoing connections.
         -  **http\_extra\_headers**: (optional): HTTP headers let the REST API server pass additional information with an HTTP response.
         -  **https\_extra\_headers**: (optional): HTTPS headers let the REST API server pass additional information with an HTTP response when TLS is enabled. This will also pass additional information set in ``http_extra_headers``.
 
diff --git a/docs/releases.rst b/docs/releases.rst
index 6d484c4f..ea312ed0 100644
--- a/docs/releases.rst
+++ b/docs/releases.rst
@@ -682,7 +682,7 @@ Version 1.6.1
 
 - Some improvements in logging infrastructure (Alexander Kukushkin)
 
-  Previously threre was a possibility to loose the last few log lines on shutdown because the logging thread was a ``daemon`` thread.
+  Previously there was a possibility to loose the last few log lines on shutdown because the logging thread was a ``daemon`` thread.
 
 - Use ``spawn`` multiprocessing start method on python 3.4+ (Maciej Kowalczyk)
 
@@ -732,7 +732,7 @@ Version 1.6.1
 
   If the method is executed from the REST API thread, it requires a separate cursor object to be created.
 
-- Fix the problem of not promoting the sync standby that had a name contaning upper case letters (Alexander Kukushkin)
+- Fix the problem of not promoting the sync standby that had a name containing upper case letters (Alexander Kukushkin)
 
   We converted the name to the lower case because Postgres was doing the same while comparing the ``application_name`` with the value in ``synchronous_standby_names``.
 
@@ -1008,7 +1008,7 @@ Compatibility and bugfix release.
 
 - Fix broken compatibility with postgres 9.3 (Alexander)
 
-  When opening a replication connection we should specify replication=1, beacuse 9.3 does not understand replication='database'
+  When opening a replication connection we should specify replication=1, because 9.3 does not understand replication='database'
 
 - Make sure we refresh Consul session at least once per HA loop and improve handling of consul sessions exceptions (Alexander)
 
@@ -1093,7 +1093,7 @@ This version enables Patroni HA cluster to operate in a standby mode, introduces
 
 - Immediately reserve the WAL position upon creation of the replication slot (Alexander Kukushkin)
 
-  Starting from 9.6, `pg_create_physical_replication_slot` function provides an additional boolean parameter `immediately_reserve`. When it is set to `false`, which is also the default, the slot doesn't reserve the WAL position until it receives the first client connection, potentially losing some segments required by the client in a time window between the slot creation and the intiial client connection.
+  Starting from 9.6, `pg_create_physical_replication_slot` function provides an additional boolean parameter `immediately_reserve`. When it is set to `false`, which is also the default, the slot doesn't reserve the WAL position until it receives the first client connection, potentially losing some segments required by the client in a time window between the slot creation and the initial client connection.
 
 - Fix bug in strict synchronous replication (Alexander Kukushkin)
 
@@ -1333,7 +1333,7 @@ This version adds support for using Kubernetes as a DCS, allowing to run Patroni
 
 **Upgrade notice**
 
-Installing Patroni via pip will no longer bring in dependencies for (such as libraries for Etcd, Zookeper, Consul or Kubernetes, or support for AWS). In order to enable them one need to list them in pip install command explicitely, for instance `pip install patroni[etcd,kubernetes]`.
+Installing Patroni via pip will no longer bring in dependencies for (such as libraries for Etcd, Zookeper, Consul or Kubernetes, or support for AWS). In order to enable them one need to list them in pip install command explicitly, for instance `pip install patroni[etcd,kubernetes]`.
 
 **Kubernetes support**
 
@@ -1352,7 +1352,7 @@ In addition to using Endpoints, Patroni supports ConfigMaps. You can find more i
 
 - Remove leader key on shutdown only when we have the lock (Ants)
 
-  Unconditional removal was generating unnecessary and missleading exceptions.
+  Unconditional removal was generating unnecessary and misleading exceptions.
 
 **Improvements in patronictl**
 
@@ -1383,7 +1383,7 @@ In addition to using Endpoints, Patroni supports ConfigMaps. You can find more i
 
 - Alter the behavior of ``patronictl failover`` (Alexander)
 
-  It will work even if there is no leader, but in that case you will have to explicitely specify a node which should become the new leader.
+  It will work even if there is no leader, but in that case you will have to explicitly specify a node which should become the new leader.
 
 **Expose information about timeline and history**
 
@@ -1399,7 +1399,7 @@ In addition to using Endpoints, Patroni supports ConfigMaps. You can find more i
 
 - Add new /sync and /async endpoints (Alexander, Oleksii Kliukin)
 
- Those endpoints (also accessible as /synchronous and /asynchronous) return 200 only for synchronous and asynchornous replicas correspondingly (exclusing those marked as `noloadbalance`).
+ Those endpoints (also accessible as /synchronous and /asynchronous) return 200 only for synchronous and asynchronous replicas correspondingly (exclusing those marked as `noloadbalance`).
 
 **Allow multiple hosts for Etcd**
 
@@ -1487,7 +1487,7 @@ Version 1.3.4
 
 - Pass the consul token as a header (Andrew Colin Kissa)
 
-  Headers are now the prefered way to pass the token to the consul `API <https://www.consul.io/api/index.html#authentication>`__.
+  Headers are now the preferred way to pass the token to the consul `API <https://www.consul.io/api/index.html#authentication>`__.
 
 
 - Advanced configuration for Consul (Alexander Kukushkin)
@@ -1805,7 +1805,7 @@ In addition, patronictl supports new ``pause`` and ``resume`` commands to toggle
   Originally, ping_timeout and connect_timeout values were calculated from the negotiated session timeout. Patroni loop_wait was not taken into account. As
   a result, a single retry could take more time than the session timeout, forcing Patroni to release the lock and demote.
 
-  This change set ping and connect timeout to half of the value of loop_wait, speeding up detection of connection issues and  leaving enough time to retry the connection attempt before loosing the lock.
+  This change set ping and connect timeout to half of the value of loop_wait, speeding up detection of connection issues and  leaving enough time to retry the connection attempt before losing the lock.
 
 - Update Etcd topology only after original request succeed (Alexander)
 
@@ -1883,7 +1883,7 @@ When upgrading from v0.90 or below, always upgrade all replicas before the maste
 
   See the :ref:`dynamic configuration <dynamic_configuration>`  for the details on which parameters can be changed and the order of processing difference configuration sources.
 
-  The configuration file format *has changed* since the v0.90. Patroni is still compatible with the old configuration files, but in order to take advantage of the bootstrap parameters one needs to change it. Users are encourage to update them by referring to the :ref:`dynamic configuraton documentation page <dynamic_configuration>`.
+  The configuration file format *has changed* since the v0.90. Patroni is still compatible with the old configuration files, but in order to take advantage of the bootstrap parameters one needs to change it. Users are encourage to update them by referring to the :ref:`dynamic configuration documentation page <dynamic_configuration>`.
 
 **More flexible configuration***
 
diff --git a/docs/rest_api.rst b/docs/rest_api.rst
index 05d579c9..79b12461 100644
--- a/docs/rest_api.rst
+++ b/docs/rest_api.rst
@@ -57,7 +57,7 @@ For all health check ``GET`` requests Patroni returns a JSON document with the s
 
 - ``GET /liveness``: always returns HTTP status code **200** what only indicates that Patroni is running. Could be used for ``livenessProbe``.
 
-- ``GET /readiness``: returns HTTP status code **200** when the Patroni node is running as the leader or when PostgreSQL is up and running. The endpoint could be used for ``readinessProbe`` when it is not possible to use Kubenetes endpoints for leader elections (OpenShift).
+- ``GET /readiness``: returns HTTP status code **200** when the Patroni node is running as the leader or when PostgreSQL is up and running. The endpoint could be used for ``readinessProbe`` when it is not possible to use Kubernetes endpoints for leader elections (OpenShift).
 
 Both, ``readiness`` and ``liveness`` endpoints are very light-weight and not executing any SQL. Probes should be configured in such a way that they start failing about time when the leader key is expiring. With the default value of ``ttl``, which is ``30s`` example probes would look like:
 
diff --git a/features/environment.py b/features/environment.py
index 130a5bf5..cfa16f04 100644
--- a/features/environment.py
+++ b/features/environment.py
@@ -869,7 +869,7 @@ class WatchdogMonitor(object):
         return triggered
 
 
-# actions to execute on start/stop of the tests and before running invidual features
+# actions to execute on start/stop of the tests and before running individual features
 def before_all(context):
     os.environ.update({'PATRONI_RESTAPI_USERNAME': 'username', 'PATRONI_RESTAPI_PASSWORD': 'password'})
     context.ci = any(a in os.environ for a in ('TRAVIS_BUILD_NUMBER', 'BUILD_NUMBER', 'GITHUB_ACTIONS'))
diff --git a/patroni/api.py b/patroni/api.py
index 2a34c242..e5ebb8ef 100644
--- a/patroni/api.py
+++ b/patroni/api.py
@@ -12,6 +12,7 @@ import six
 import socket
 import sys
 
+from ipaddress import ip_address, ip_network as _ip_network
 from six.moves.BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
 from six.moves.socketserver import ThreadingMixIn
 from six.moves.urllib_parse import urlparse, parse_qs
@@ -25,6 +26,10 @@ from .utils import deep_compare, enable_keepalive, parse_bool, patch_config, Ret
 logger = logging.getLogger(__name__)
 
 
+def ip_network(value):
+    return _ip_network(value.decode('utf-8') if six.PY2 else value, False)
+
+
 class RestApiHandler(BaseHTTPRequestHandler):
 
     def _write_status_code_only(self, status_code):
@@ -47,17 +52,17 @@ class RestApiHandler(BaseHTTPRequestHandler):
     def _write_json_response(self, status_code, response):
         self._write_response(status_code, json.dumps(response, default=str), content_type='application/json')
 
-    def check_auth(func):
-        """Decorator function to check authorization header or client certificates
+    def check_access(func):
+        """Decorator function to check the source ip, authorization header. or client certificates
 
         Usage example:
-        @check_auth
+        @check_access
         def do_PUT_foo():
             pass
         """
 
         def wrapper(self, *args, **kwargs):
-            if self.server.check_auth(self):
+            if self.server.check_access(self):
                 return func(self, *args, **kwargs)
 
         return wrapper
@@ -299,7 +304,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
             logger.exception('Bad request')
         self.send_error(400)
 
-    @check_auth
+    @check_access
     def do_PATCH_config(self):
         request = self._read_json_content()
         if request:
@@ -314,7 +319,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
             self.server.patroni.ha.wakeup()
             self._write_json_response(200, data)
 
-    @check_auth
+    @check_access
     def do_PUT_config(self):
         request = self._read_json_content()
         if request:
@@ -325,7 +330,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
                     return self.send_error(502)
             self._write_json_response(200, request)
 
-    @check_auth
+    @check_access
     def do_POST_reload(self):
         self.server.patroni.sighup_handler()
         self._write_response(202, 'reload scheduled')
@@ -351,7 +356,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
             status_code = 422
         return (status_code, error, scheduled_at)
 
-    @check_auth
+    @check_access
     def do_POST_restart(self):
         status_code = 500
         data = 'restart failed'
@@ -412,7 +417,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
                     status_code = 409
         self._write_response(status_code, data)
 
-    @check_auth
+    @check_access
     def do_DELETE_restart(self):
         if self.server.patroni.ha.delete_future_restart():
             data = "scheduled restart deleted"
@@ -422,7 +427,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
             code = 404
         self._write_response(code, data)
 
-    @check_auth
+    @check_access
     def do_DELETE_switchover(self):
         failover = self.server.patroni.dcs.get_cluster().failover
         if failover and failover.scheduled_at:
@@ -436,7 +441,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
             code = 404
         self._write_response(code, data)
 
-    @check_auth
+    @check_access
     def do_POST_reinitialize(self):
         request = self._read_json_content(body_is_optional=True)
 
@@ -468,7 +473,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
                 if not cluster.failover:
                     return 503, action.title() + ' failed'
             except Exception as e:
-                logger.debug('Exception occured during polling %s result: %s', action, e)
+                logger.debug('Exception occurred during polling %s result: %s', action, e)
         return 503, action.title() + ' status unknown'
 
     def is_failover_possible(self, cluster, leader, candidate, action):
@@ -493,7 +498,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
                 return None
         return action + ' is not possible: no good candidates have been found'
 
-    @check_auth
+    @check_access
     def do_POST_failover(self, action='failover'):
         request = self._read_json_content()
         (status_code, data) = (400, '')
@@ -641,7 +646,6 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
         self.patroni = patroni
         self.__listen = None
         self.__ssl_options = None
-        self.http_extra_headers = {}
         self.reload_config(config)
         self.daemon = True
         self.__ssl_serial_number = None
@@ -675,7 +679,35 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
             if not auth_header.startswith('Basic ') or not self.check_basic_auth_key(auth_header[6:]):
                 return 'not authenticated'
 
-    def check_auth(self, rh):
+    @staticmethod
+    def __resolve_ips(host, port):
+        try:
+            for _, _, _, _, sa in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM, socket.IPPROTO_TCP):
+                yield ip_network(sa[0])
+        except Exception as e:
+            logger.error('Failed to resolve %s: %r', host, e)
+
+    def __members_ips(self):
+        cluster = self.patroni.dcs.cluster
+        if self.__allowlist_include_members and cluster:
+            for member in cluster.members:
+                if member.api_url:
+                    try:
+                        r = urlparse(member.api_url)
+                        host = r.hostname
+                        port = r.port or (443 if r.scheme == 'https' else 80)
+                        for ip in self.__resolve_ips(host, port):
+                            yield ip
+                    except Exception as e:
+                        logger.debug('Failed to parse url %s: %r', member.api_url, e)
+
+    def check_access(self, rh):
+        if self.__allowlist or self.__allowlist_include_members:
+            incoming_ip = rh.client_address[0]
+            incoming_ip = ip_address(incoming_ip.decode('utf-8') if six.PY2 else incoming_ip)
+            if not any(incoming_ip in net for net in self.__allowlist + tuple(self.__members_ips())):
+                return rh._write_response(403, 'Access is denied')
+
         if not hasattr(rh.request, 'getpeercert') or not rh.request.getpeercert():  # valid client cert isn't present
             if self.__protocol == 'https' and self.__ssl_options.get('verify_client') in ('required', 'optional'):
                 return rh._write_response(403, 'client certificate required')
@@ -799,10 +831,25 @@ class RestApiServer(ThreadingMixIn, HTTPServer, Thread):
                 self.__ssl_serial_number = on_disk_cert_serial_number
                 return True
 
+    def _build_allowlist(self, value):
+        if isinstance(value, list):
+            for v in value:
+                if '/' in v:  # netmask
+                    try:
+                        yield ip_network(v)
+                    except Exception as e:
+                        logger.error('Invalid value "%s" in the allowlist: %r', v, e)
+                else:  # ip or hostname, try to resolve it
+                    for ip in self.__resolve_ips(v, 8080):
+                        yield ip
+
     def reload_config(self, config):
         if 'listen' not in config:  # changing config in runtime
             raise ValueError('Can not find "restapi.listen" config')
 
+        self.__allowlist = tuple(self._build_allowlist(config.get('allowlist')))
+        self.__allowlist_include_members = config.get('allowlist_include_members')
+
         ssl_options = {n: config[n] for n in ('certfile', 'keyfile', 'keyfile_password',
                                               'cafile', 'ciphers') if n in config}
 
diff --git a/patroni/config.py b/patroni/config.py
index 6c6c73b0..1308cc49 100644
--- a/patroni/config.py
+++ b/patroni/config.py
@@ -232,7 +232,7 @@ class Config(object):
                 for name, value in (value or {}).items():
                     if name in self.__DEFAULT_CONFIG['standby_cluster']:
                         config['standby_cluster'][name] = deepcopy(value)
-            elif name in config:  # only variables present in __DEFAULT_CONFIG allowed to be overriden from DCS
+            elif name in config:  # only variables present in __DEFAULT_CONFIG allowed to be overridden from DCS
                 if name in ('synchronous_mode', 'synchronous_mode_strict'):
                     config[name] = value
                 else:
@@ -268,11 +268,42 @@ class Config(object):
 
         _set_section_values('restapi', ['listen', 'connect_address', 'certfile', 'keyfile', 'keyfile_password',
                                         'cafile', 'ciphers', 'verify_client', 'http_extra_headers',
-                                        'https_extra_headers'])
+                                        'https_extra_headers', 'allowlist', 'allowlist_include_members'])
         _set_section_values('ctl', ['insecure', 'cacert', 'certfile', 'keyfile'])
         _set_section_values('postgresql', ['listen', 'connect_address', 'config_dir', 'data_dir', 'pgpass', 'bin_dir'])
         _set_section_values('log', ['level', 'traceback_level', 'format', 'dateformat', 'max_queue_size',
                                     'dir', 'file_size', 'file_num', 'loggers'])
+        _set_section_values('raft', ['data_dir', 'self_addr', 'partner_addrs', 'password', 'bind_addr'])
+
+        for first, second in (('restapi', 'allowlist_include_members'), ('ctl', 'insecure')):
+            value = ret.get(first, {}).pop(second, None)
+            if value:
+                value = parse_bool(value)
+                if value is not None:
+                    ret[first][second] = value
+
+        for second in ('max_queue_size', 'file_size', 'file_num'):
+            value = ret.get('log', {}).pop(second, None)
+            if value:
+                value = parse_int(value)
+                if value is not None:
+                    ret['log'][second] = value
+
+        def _parse_list(value):
+            if not (value.strip().startswith('-') or '[' in value):
+                value = '[{0}]'.format(value)
+            try:
+                return yaml.safe_load(value)
+            except Exception:
+                logger.exception('Exception when parsing list %s', value)
+                return None
+
+        for first, second in (('raft', 'partner_addrs'), ('restapi', 'allowlist')):
+            value = ret.get(first, {}).pop(second, None)
+            if value:
+                value = _parse_list(value)
+                if value:
+                    ret[first][second] = value
 
         def _parse_dict(value):
             if not value.strip().startswith('{'):
@@ -283,11 +314,13 @@ class Config(object):
                 logger.exception('Exception when parsing dict %s', value)
                 return None
 
-        value = ret.get('log', {}).pop('loggers', None)
-        if value:
-            value = _parse_dict(value)
-            if value:
-                ret['log']['loggers'] = value
+        for first, params in (('restapi', ('http_extra_headers', 'https_extra_headers')), ('log', ('loggers',))):
+            for second in params:
+                value = ret.get(first, {}).pop(second, None)
+                if value:
+                    value = _parse_dict(value)
+                    if value:
+                        ret[first][second] = value
 
         def _get_auth(name, params=None):
             ret = {}
@@ -310,19 +343,6 @@ class Config(object):
         if authentication:
             ret['postgresql']['authentication'] = authentication
 
-        def _parse_list(value):
-            if not (value.strip().startswith('-') or '[' in value):
-                value = '[{0}]'.format(value)
-            try:
-                return yaml.safe_load(value)
-            except Exception:
-                logger.exception('Exception when parsing list %s', value)
-                return None
-
-        _set_section_values('raft', ['data_dir', 'self_addr', 'partner_addrs', 'password', 'bind_addr'])
-        if 'raft' in ret and 'partner_addrs' in ret['raft']:
-            ret['raft']['partner_addrs'] = _parse_list(ret['raft']['partner_addrs'])
-
         for param in list(os.environ.keys()):
             if param.startswith(PATRONI_ENV_PREFIX):
                 # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
@@ -339,7 +359,7 @@ class Config(object):
                         value = value and _parse_list(value)
                     elif suffix == 'LABELS':
                         value = _parse_dict(value)
-                    elif suffix in ('USE_PROXIES', 'REGISTER_SERVICE', 'USE_ENDPOINTS', 'BYPASS_API_SERVICE'):
+                    elif suffix in ('USE_PROXIES', 'REGISTER_SERVICE', 'USE_ENDPOINTS', 'BYPASS_API_SERVICE', 'VERIFY'):
                         value = parse_bool(value)
                     if value:
                         ret[name.lower()][suffix.lower()] = value
diff --git a/patroni/dcs/__init__.py b/patroni/dcs/__init__.py
index 9e0e225c..46e3bd74 100644
--- a/patroni/dcs/__init__.py
+++ b/patroni/dcs/__init__.py
@@ -634,7 +634,7 @@ class AbstractDCS(object):
     _HISTORY = 'history'
     _MEMBERS = 'members/'
     _OPTIME = 'optime'
-    _STATUS = 'status'  # JSON, containts "leader_lsn" and confirmed_flush_lsn of logical "slots" on the leader
+    _STATUS = 'status'  # JSON, contains "leader_lsn" and confirmed_flush_lsn of logical "slots" on the leader
     _LEADER_OPTIME = _OPTIME + '/' + _LEADER  # legacy
     _SYNC = 'sync'
 
diff --git a/patroni/dcs/consul.py b/patroni/dcs/consul.py
index 6c395252..e2a0f749 100644
--- a/patroni/dcs/consul.py
+++ b/patroni/dcs/consul.py
@@ -111,7 +111,7 @@ class HTTPClient(object):
                 # According to the documentation a small random amount of additional wait time is added to the
                 # supplied maximum wait time to spread out the wake up time of any concurrent requests. This adds
                 # up to wait / 16 additional time to the maximum duration. Since our goal is actually getting a
-                # response rather read timeout we will add to the timeout a sligtly bigger value.
+                # response rather read timeout we will add to the timeout a slightly bigger value.
                 kwargs['timeout'] = timeout + max(timeout/15.0, 1)
             else:
                 kwargs['timeout'] = self._read_timeout
diff --git a/patroni/dcs/etcd3.py b/patroni/dcs/etcd3.py
index 4abb998b..1e738100 100644
--- a/patroni/dcs/etcd3.py
+++ b/patroni/dcs/etcd3.py
@@ -375,6 +375,7 @@ class KVCache(Thread):
         self._config_key = base64_encode(dcs.config_path)
         self._leader_key = base64_encode(dcs.leader_path)
         self._optime_key = base64_encode(dcs.leader_optime_path)
+        self._status_key = base64_encode(dcs.status_path)
         self._name = base64_encode(dcs._name)
         self._is_ready = False
         self._response = None
@@ -421,14 +422,15 @@ class KVCache(Thread):
             new_value = kv.get('value')
 
             value_changed = old_value != new_value and \
-                (key == self._leader_key or key == self._optime_key and new_value is not None or
+                (key == self._leader_key or key in (self._optime_key, self._status_key) and new_value is not None or
                  key == self._config_key and old_value is not None and new_value is not None)
 
             if value_changed:
                 logger.debug('%s changed from %s to %s', key, old_value, new_value)
 
-            # We also want to wake up HA loop on replicas if leader optime was updated
-            if value_changed and (key != self._optime_key or self.get(self._leader_key) != self._name):
+            # We also want to wake up HA loop on replicas if leader optime (or status key) was updated
+            if value_changed and (key not in (self._optime_key, self._status_key) or
+                                  (self.get(self._leader_key) or {}).get('value') != self._name):
                 self._dcs.event.set()
 
     def _process_message(self, message):
diff --git a/patroni/dcs/raft.py b/patroni/dcs/raft.py
index 76c65efb..fed6d5df 100644
--- a/patroni/dcs/raft.py
+++ b/patroni/dcs/raft.py
@@ -5,8 +5,10 @@ import threading
 import time
 
 from pysyncobj import SyncObj, SyncObjConf, replicated, FAIL_REASON
+from pysyncobj.dns_resolver import globalDnsResolver
+from pysyncobj.node import TCPNode
 from pysyncobj.transport import TCPTransport, CONNECTION_STATE
-from pysyncobj.utility import TcpUtility, UtilityException
+from pysyncobj.utility import TcpUtility
 
 from . import AbstractDCS, ClusterConfig, Cluster, Failover, Leader, Member, SyncState, TimelineHistory
 from ..utils import validate_directory
@@ -20,6 +22,20 @@ class _TCPTransport(TCPTransport):
         super(_TCPTransport, self).__init__(syncObj, selfNode, otherNodes)
         self.setOnUtilityMessageCallback('members', syncObj.getMembers)
 
+    def _connectIfNecessarySingle(self, node):
+        try:
+            return super(_TCPTransport, self)._connectIfNecessarySingle(node)
+        except Exception as e:
+            logger.debug('Connection to %s failed: %r', node, e)
+            return False
+
+
+def resolve_host(self):
+    return globalDnsResolver().resolve(self.host)
+
+
+setattr(TCPNode, 'ip', property(resolve_host))
+
 
 class SyncObjUtility(object):
 
@@ -30,7 +46,7 @@ class SyncObjUtility(object):
     def executeCommand(self, command):
         try:
             return self._utility.executeCommand(self.__node, command)
-        except UtilityException:
+        except Exception:
             return None
 
     def getMembers(self):
@@ -99,7 +115,8 @@ class KVStoreTTL(DynMemberSyncObj):
         file_template = file_template.replace(':', '_') if os.name == 'nt' else file_template
         file_template = os.path.join(raft_data_dir, file_template)
         conf = SyncObjConf(password=config.get('password'), autoTick=False, appendEntriesUseBatch=False,
-                           bindAddress=config.get('bind_addr'), commandsWaitLeader=config.get('commandsWaitLeader'),
+                           bindAddress=config.get('bind_addr'), dnsFailCacheTime=(config.get('loop_wait') or 10),
+                           dnsCacheTime=(config.get('ttl') or 30), commandsWaitLeader=config.get('commandsWaitLeader'),
                            fullDumpFile=(file_template + '.dump' if self_addr else None),
                            journalFile=(file_template + '.journal' if self_addr else None),
                            onReady=on_ready, dynamicMembershipChange=True)
@@ -264,7 +281,8 @@ class Raft(AbstractDCS):
         leader = (self._sync_obj.get(self.leader_path) or {}).get('value')
         if key == value['created'] == value['updated'] and \
                 (key.startswith(self.members_path) or key == self.leader_path and leader != self._name) or \
-                key == self.leader_optime_path and leader != self._name or key in (self.config_path, self.sync_path):
+                key in (self.leader_optime_path, self.status_path) and leader != self._name or \
+                key in (self.config_path, self.sync_path):
             self.event.set()
 
     def _on_delete(self, key):
@@ -281,6 +299,10 @@ class Raft(AbstractDCS):
     def set_retry_timeout(self, retry_timeout):
         self._sync_obj.set_retry_timeout(retry_timeout)
 
+    def reload_config(self, config):
+        super(Raft, self).reload_config(config)
+        globalDnsResolver().setTimeouts(self.ttl, self.loop_wait)
+
     @staticmethod
     def member(key, value):
         return Member.from_node(value['index'], os.path.basename(key), None, value['value'])
diff --git a/patroni/ha.py b/patroni/ha.py
index ee1c78e8..52745110 100644
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -84,7 +84,7 @@ class Ha(object):
         self._disable_sync = 0
 
         # We need following property to avoid shutdown of postgres when join of Patroni to the postgres
-        # already running as replica was aborted due to cluster not beeing initialized in DCS.
+        # already running as replica was aborted due to cluster not being initialized in DCS.
         self._join_aborted = False
 
         # used only in backoff after failing a pre_promote script
@@ -1219,9 +1219,6 @@ class Ha(object):
                 self._delete_leader()
                 return 'removed leader key after trying and failing to start postgres'
             return 'failed to start postgres'
-        self._crash_recovery_executed = False
-        if self._rewind.executed and not self._rewind.failed:
-            self._rewind.reset_state()
         return None
 
     def cancel_initialization(self):
@@ -1345,12 +1342,24 @@ class Ha(object):
             if self.state_handler.bootstrapping:
                 return self.post_bootstrap()
 
-            if self.recovering and not self._rewind.is_needed:
+            if self.recovering:
                 self.recovering = False
-                # Check if we tried to recover and failed
-                msg = self.post_recover()
-                if msg is not None:
-                    return msg
+
+                if not self._rewind.is_needed:
+                    # Check if we tried to recover from postgres crash and failed
+                    msg = self.post_recover()
+                    if msg is not None:
+                        return msg
+
+                # Reset some states after postgres successfully started up
+                self._crash_recovery_executed = False
+                if self._rewind.executed and not self._rewind.failed:
+                    self._rewind.reset_state()
+
+                # The Raft cluster without a quorum takes a bit of time to stabilize.
+                # Therefore we want to postpone the leader race if we just started up.
+                if self.cluster.is_unlocked() and self.dcs.__class__.__name__ == 'Raft':
+                    return 'started as a secondary'
 
             # is data directory empty?
             if self.state_handler.data_directory_empty():
diff --git a/patroni/postgresql/__init__.py b/patroni/postgresql/__init__.py
index 13055dbe..9fd1b660 100644
--- a/patroni/postgresql/__init__.py
+++ b/patroni/postgresql/__init__.py
@@ -1,20 +1,22 @@
 import logging
 import os
 import psycopg2
+import re
 import shlex
 import shutil
+import six
 import subprocess
 import time
 
 from contextlib import contextmanager
 from copy import deepcopy
-from dateutil import tz
 from datetime import datetime
+from dateutil import tz
 from psutil import TimeoutExpired
 from threading import current_thread, Lock
 
-from .callback_executor import CallbackExecutor
 from .bootstrap import Bootstrap
+from .callback_executor import CallbackExecutor
 from .cancellable import CancellableSubprocess
 from .config import ConfigHandler, mtime
 from .connection import Connection, get_connection_cursor
@@ -386,16 +388,41 @@ class Postgresql(object):
         except (TypeError, ValueError):
             logger.exception('Failed to parse timeline from pg_controldata output')
 
+    def parse_wal_record(self, timeline, lsn):
+        out, err = self.waldump(timeline, lsn, 1)
+        if out and not err:
+            match = re.match(r'^rmgr:\s+(.+?)\s+len \(rec/tot\):\s+\d+/\s+\d+, tx:\s+\d+, '
+                             r'lsn: ([0-9A-Fa-f]+/[0-9A-Fa-f]+), prev ([0-9A-Fa-f]+/[0-9A-Fa-f]+), '
+                             r'.*?desc: (.+)', out.decode('utf-8'))
+            if match:
+                return match.groups()
+        return None, None, None, None
+
     def latest_checkpoint_location(self):
-        """Returns checkpoint location for the cleanly shut down primary"""
+        """Returns checkpoint location for the cleanly shut down primary.
+           But, if we know that the checkpoint was written to the new WAL
+           due to the archive_mode=on, we will return the LSN of prev wal record (SWITCH)."""
 
         data = self.controldata()
-        lsn = data.get('Latest checkpoint location')
-        if data.get('Database cluster state') == 'shut down' and lsn:
+        timeline = data.get("Latest checkpoint's TimeLineID")
+        lsn = checkpoint_lsn = data.get('Latest checkpoint location')
+        if data.get('Database cluster state') == 'shut down' and lsn and timeline:
             try:
-                return str(parse_lsn(lsn))
-            except (IndexError, ValueError) as e:
-                logger.error('Exception when parsing lsn %s: %r', lsn, e)
+                checkpoint_lsn = parse_lsn(checkpoint_lsn)
+                rm_name, lsn, prev, desc = self.parse_wal_record(timeline, lsn)
+                desc = desc.strip().lower()
+                if rm_name == 'XLOG' and parse_lsn(lsn) == checkpoint_lsn and prev and\
+                        desc.startswith('checkpoint') and desc.endswith('shutdown'):
+                    _, lsn, _, desc = self.parse_wal_record(timeline, prev)
+                    prev = parse_lsn(prev)
+                    # If the cluster is shutdown with archive_mode=on, WAL is switched before writing the checkpoint.
+                    # In this case we want to take the LSN of previous record (switch) as the last known WAL location.
+                    if parse_lsn(lsn) == prev and desc.strip() in ('xlog switch', 'SWITCH'):
+                        return str(prev)
+            except Exception as e:
+                logger.error('Exception when parsing WAL pg_%sdump output: %r', self.wal_name, e)
+            if isinstance(checkpoint_lsn, six.integer_types):
+                return str(checkpoint_lsn)
 
     def is_running(self):
         """Returns PostmasterProcess if one is running on the data directory or None. If most recently seen process
@@ -767,6 +794,20 @@ class Postgresql(object):
                 logger.exception("Error when calling pg_controldata")
         return {}
 
+    def waldump(self, timeline, lsn, limit):
+        cmd = self.pgcommand('pg_{0}dump'.format(self.wal_name))
+        env = os.environ.copy()
+        env.update(LANG='C', LC_ALL='C', PGDATA=self._data_dir)
+        try:
+            waldump = subprocess.Popen([cmd, '-t', str(timeline), '-s', str(lsn), '-n', str(limit)],
+                                       stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
+            out, err = waldump.communicate()
+            waldump.wait()
+            return out, err
+        except Exception as e:
+            logger.error('Failed to execute `%s -t %s -s %s -n %s`: %r', cmd, timeline, lsn, limit, e)
+            return None, None
+
     @contextmanager
     def get_replication_connection_cursor(self, host='localhost', port=5432, **kwargs):
         conn_kwargs = self.config.replication.copy()
@@ -1011,7 +1052,7 @@ class Postgresql(object):
 
         Current synchronous standby is always preferred, unless it has disconnected or does not want to be a
         synchronous standby any longer.
-        Parameter sync_node_maxlag(maximum_lag_on_syncnode) would help swapping unhealthy sync replica incase
+        Parameter sync_node_maxlag(maximum_lag_on_syncnode) would help swapping unhealthy sync replica in case
         if it stops responding (or hung). Please set the value high enough so it won't unncessarily swap sync
         standbys during high loads. Any less or equal of 0 value keep the behavior backward compatible and
         will not swap. Please note that it will not also swap sync standbys in case where all replicas are hung.
diff --git a/patroni/postgresql/bootstrap.py b/patroni/postgresql/bootstrap.py
index 2c328718..c52be97d 100644
--- a/patroni/postgresql/bootstrap.py
+++ b/patroni/postgresql/bootstrap.py
@@ -53,7 +53,7 @@ class Bootstrap(object):
                     error_handler('Error when parsing {0} option {1}: value should be string value'
                                   ' or a single key-value pair'.format(tool, opt))
         else:
-            error_handler('{0} options must be list ot dict'.format(tool))
+            error_handler('{0} options must be list or dict'.format(tool))
         return user_options
 
     def _initdb(self, config):
diff --git a/patroni/postgresql/cancellable.py b/patroni/postgresql/cancellable.py
index 8e46fbfb..09057e8e 100644
--- a/patroni/postgresql/cancellable.py
+++ b/patroni/postgresql/cancellable.py
@@ -36,7 +36,7 @@ class CancellableExecutor(object):
         with self._lock:
             if self._process is not None and self._process.is_running() and not self._process_children:
                 try:
-                    self._process.suspend()  # Suspend the process before getting list of childrens
+                    self._process.suspend()  # Suspend the process before getting list of children
                 except psutil.Error as e:
                     logger.info('Failed to suspend the process: %s', e.msg)
 
diff --git a/patroni/postgresql/config.py b/patroni/postgresql/config.py
index c61a7695..8cae0dfb 100644
--- a/patroni/postgresql/config.py
+++ b/patroni/postgresql/config.py
@@ -353,7 +353,7 @@ class ConfigHandler(object):
 
     def save_configuration_files(self, check_custom_bootstrap=False):
         """
-            copy postgresql.conf to postgresql.conf.backup to be able to retrive configuration files
+            copy postgresql.conf to postgresql.conf.backup to be able to retrieve configuration files
             - originally stored as symlinks, those are normally skipped by pg_basebackup
             - in case of WAL-E basebackup (see http://comments.gmane.org/gmane.comp.db.postgresql.wal-e/239)
         """
@@ -403,7 +403,7 @@ class ConfigHandler(object):
                     f.write_param(name, value)
             # when we are doing custom bootstrap we assume that we don't know superuser password
             # and in order to be able to change it, we are opening trust access from a certain address
-            # therefore we need to make sure that hba_file is not overriden
+            # therefore we need to make sure that hba_file is not overridden
             # after changing superuser password we will "revert" all these "changes"
             if self._postgresql.bootstrap.running_custom_bootstrap or 'hba_file' not in self._server_parameters:
                 f.write_param('hba_file', self._pg_hba_conf)
@@ -884,7 +884,7 @@ class ConfigHandler(object):
         unix_local_address = {'port': port}
         unix_socket_directories = self._server_parameters.get('unix_socket_directories')
         if unix_socket_directories is not None:
-            # fallback to tcp if unix_socket_directories is set, but there are no sutable values
+            # fallback to tcp if unix_socket_directories is set, but there are no suitable values
             unix_local_address['host'] = self._get_unix_local_address(unix_socket_directories) or tcp_local_address
 
         tcp_local_address = {'host': tcp_local_address, 'port': port}
diff --git a/patroni/postgresql/misc.py b/patroni/postgresql/misc.py
index 1d93d3de..3a6faa1c 100644
--- a/patroni/postgresql/misc.py
+++ b/patroni/postgresql/misc.py
@@ -37,7 +37,7 @@ def postgres_version_to_int(pg_version):
         raise PostgresException('Invalid PostgreSQL version format: X.Y or X.Y.Z is accepted: {0}'.format(pg_version))
 
     if len(components) == 2:
-        # new style verion numbers, i.e. 10.1 becomes 100001
+        # new style version numbers, i.e. 10.1 becomes 100001
         components.insert(1, 0)
 
     return int(''.join('{0:02d}'.format(c) for c in components))
diff --git a/patroni/postgresql/rewind.py b/patroni/postgresql/rewind.py
index 4f40f189..506e822f 100644
--- a/patroni/postgresql/rewind.py
+++ b/patroni/postgresql/rewind.py
@@ -68,23 +68,14 @@ class Rewind(object):
     def _get_checkpoint_end(self, timeline, lsn):
         """The checkpoint record size in WAL depends on postgres major version and platform (memory alignment).
         Hence, the only reliable way to figure out where it ends, read the record from file with the help of pg_waldump
-        and parse the output. We are trying to read two records, and expect that it wil fail to read the second one:
+        and parse the output. We are trying to read two records, and expect that it will fail to read the second one:
         `pg_waldump: fatal: error in WAL record at 0/182E220: invalid record length at 0/182E298: wanted 24, got 0`
         The error message contains information about LSN of the next record, which is exactly where checkpoint ends."""
 
-        cmd = self._postgresql.pgcommand('pg_{0}dump'.format(self._postgresql.wal_name))
         lsn8 = format_lsn(lsn, True)
         lsn = format_lsn(lsn)
-        env = os.environ.copy()
-        env.update(LANG='C', LC_ALL='C', PGDATA=self._postgresql.data_dir)
-        try:
-            waldump = subprocess.Popen([cmd, '-t', str(timeline), '-s', lsn, '-n', '2'],
-                                       stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
-            out, err = waldump.communicate()
-            waldump.wait()
-        except Exception as e:
-            logger.error('Failed to execute `%s -t %s -s %s -n 2`: %r', cmd, timeline, lsn, e)
-        else:
+        out, err = self._postgresql.waldump(timeline, lsn, 2)
+        if out is not None and err is not None:
             out = out.decode('utf-8').rstrip().split('\n')
             err = err.decode('utf-8').rstrip().split('\n')
             pattern = 'error in WAL record at {0}: invalid record length at '.format(lsn)
@@ -97,7 +88,7 @@ class Rewind(object):
                         return parse_lsn(err[0][i:j])
                     except Exception as e:
                         logger.error('Failed to parse lsn %s: %r', err[0][i:j], e)
-            logger.error('Failed to parse `%s -t %s -s %s -n 2` output', cmd, timeline, lsn)
+            logger.error('Failed to parse pg_%sdump output', self._postgresql.wal_name)
             logger.error(' stdout=%s', '\n'.join(out))
             logger.error(' stderr=%s', '\n'.join(err))
 
diff --git a/patroni/scripts/wale_restore.py b/patroni/scripts/wale_restore.py
index b4c70991..055fc204 100755
--- a/patroni/scripts/wale_restore.py
+++ b/patroni/scripts/wale_restore.py
@@ -250,7 +250,7 @@ class WALERestore(object):
                 diff_in_bytes = 0
             break
 
-        # if the size of the accumulated WAL segments is more than a certan percentage of the backup size
+        # if the size of the accumulated WAL segments is more than a certain percentage of the backup size
         # or exceeds the pre-determined size - pg_basebackup is chosen instead.
         is_size_thresh_ok = diff_in_bytes < int(threshold_megabytes) * 1048576
         threshold_pct_bytes = backup_size * threshold_percent / 100.0
@@ -308,7 +308,7 @@ class WALERestore(object):
             try:
                 os.mkdir(path)
             except OSError:
-                logger.exception("coud not create missing %s directory path", dirname)
+                logger.exception("could not create missing %s directory path", dirname)
                 return False
         return True
 
diff --git a/setup.py b/setup.py
index 7302858a..8cc5940a 100644
--- a/setup.py
+++ b/setup.py
@@ -24,7 +24,7 @@ KEYWORDS = 'etcd governor patroni postgresql postgres ha haproxy confd' +\
 
 EXTRAS_REQUIRE = {'aws': ['boto'], 'etcd': ['python-etcd'], 'etcd3': ['python-etcd'],
                   'consul': ['python-consul'], 'exhibitor': ['kazoo'], 'zookeeper': ['kazoo'],
-                  'kubernetes': ['ipaddress'], 'raft': ['pysyncobj', 'cryptography']}
+                  'kubernetes': [], 'raft': ['pysyncobj', 'cryptography']}
 COVERAGE_XML = True
 COVERAGE_HTML = False
 
@@ -175,13 +175,12 @@ def setup_package(version):
         for e, deps in EXTRAS_REQUIRE.items():
             for i, v in enumerate(deps):
                 if r.startswith(v):
-                    if e != 'kubernetes' or sys.version_info < (3, 0, 0):
-                        deps[i] = r
-                    else:
-                        deps = []
+                    deps[i] = r
                     EXTRAS_REQUIRE[e] = deps
                     extra = True
                     break
+            if extra:
+                break
         if not extra:
             install_requires.append(r)
 


Test patch:
diff --git a/tests/test_api.py b/tests/test_api.py
index 4b4d8ebf..34a6224e 100644
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -538,7 +538,9 @@ class TestRestApiServer(unittest.TestCase):
     @patch.object(BaseHTTPServer.HTTPServer, '__init__', Mock())
     def setUp(self):
         self.srv = MockRestApiServer(Mock(), '', {'listen': '*:8008', 'certfile': 'a', 'verify_client': 'required',
-                                                  'ciphers': '!SSLv1:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1'})
+                                                  'ciphers': '!SSLv1:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1',
+                                                  'allowlist': ['127.0.0.1', '::1/128', '::1/zxc'],
+                                                  'allowlist_include_members': True})
 
     @patch.object(BaseHTTPServer.HTTPServer, '__init__', Mock())
     def test_reload_config(self):
@@ -549,10 +551,17 @@ class TestRestApiServer(unittest.TestCase):
         with patch.object(socket.socket, 'setsockopt', Mock(side_effect=socket.error)):
             self.srv.reload_config({'listen': ':8008'})
 
-    def test_check_auth(self):
+    @patch.object(MockPatroni, 'dcs')
+    def test_check_access(self, mock_dcs):
+        mock_dcs.cluster = get_cluster_initialized_without_leader()
+        mock_dcs.cluster.members[1].data['api_url'] = 'http://127.0.0.1z:8011/patroni'
+        mock_dcs.cluster.members.append(Member(0, 'bad-api-url', 30, {'api_url': 123}))
         mock_rh = Mock()
+        mock_rh.client_address = ('127.0.0.2',)
+        self.assertIsNot(self.srv.check_access(mock_rh), True)
+        mock_rh.client_address = ('127.0.0.1',)
         mock_rh.request.getpeercert.return_value = None
-        self.assertIsNot(self.srv.check_auth(mock_rh), True)
+        self.assertIsNot(self.srv.check_access(mock_rh), True)
 
     def test_handle_error(self):
         try:
diff --git a/tests/test_config.py b/tests/test_config.py
index b656da26..dddae174 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -30,12 +30,14 @@ class TestConfig(unittest.TestCase):
             'PATRONI_SCOPE': 'batman2',
             'PATRONI_LOGLEVEL': 'ERROR',
             'PATRONI_LOG_LOGGERS': 'patroni.postmaster: WARNING, urllib3: DEBUG',
+            'PATRONI_LOG_FILE_NUM': '5',
             'PATRONI_RESTAPI_USERNAME': 'username',
             'PATRONI_RESTAPI_PASSWORD': 'password',
             'PATRONI_RESTAPI_LISTEN': '0.0.0.0:8008',
             'PATRONI_RESTAPI_CONNECT_ADDRESS': '127.0.0.1:8008',
             'PATRONI_RESTAPI_CERTFILE': '/certfile',
             'PATRONI_RESTAPI_KEYFILE': '/keyfile',
+            'PATRONI_RESTAPI_ALLOWLIST_INCLUDE_MEMBERS': 'on',
             'PATRONI_POSTGRESQL_LISTEN': '0.0.0.0:5432',
             'PATRONI_POSTGRESQL_CONNECT_ADDRESS': '127.0.0.1:5432',
             'PATRONI_POSTGRESQL_DATA_DIR': 'data/postgres0',
diff --git a/tests/test_ha.py b/tests/test_ha.py
index 68a9a4ea..9f1f0402 100644
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -157,7 +157,8 @@ def run_async(self, func, args=()):
 @patch.object(Postgresql, 'controldata', Mock(return_value={
     'Database system identifier': SYSID,
     'Database cluster state': 'shut down',
-    'Latest checkpoint location': '0/12345678'}))
+    'Latest checkpoint location': '0/12345678',
+    "Latest checkpoint's TimeLineID": '2'}))
 @patch.object(SlotsHandler, 'load_replication_slots', Mock(side_effect=Exception))
 @patch.object(ConfigHandler, 'append_pg_hba', Mock())
 @patch.object(ConfigHandler, 'write_pgpass', Mock(return_value={}))
@@ -250,6 +251,15 @@ class TestHa(PostgresInit):
         self.assertEqual(self.ha.run_cycle(), 'starting as a secondary')
         self.assertEqual(self.ha.run_cycle(), 'failed to start postgres')
 
+    def test_recover_raft(self):
+        self.p.controldata = lambda: {'Database cluster state': 'in recovery', 'Database system identifier': SYSID}
+        self.p.is_running = false
+        self.p.follow = true
+        self.assertEqual(self.ha.run_cycle(), 'starting as a secondary')
+        self.p.is_running = true
+        self.ha.dcs.__class__.__name__ = 'Raft'
+        self.assertEqual(self.ha.run_cycle(), 'started as a secondary')
+
     def test_recover_former_master(self):
         self.p.follow = false
         self.p.is_running = false
@@ -278,7 +288,17 @@ class TestHa(PostgresInit):
     def test_recover_with_rewind(self):
         self.p.is_running = false
         self.ha.cluster = get_cluster_initialized_with_leader()
-        self.assertEqual(self.ha.run_cycle(), 'running pg_rewind from leader')
+        self.ha.cluster.leader.member.data.update(version='2.0.2', role='master')
+        self.ha._rewind.pg_rewind = true
+        self.ha._rewind.check_leader_is_not_in_recovery = true
+        with patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True)):
+            self.assertEqual(self.ha.run_cycle(), 'running pg_rewind from leader')
+        with patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=False)):
+            self.p.follow = true
+            self.assertEqual(self.ha.run_cycle(), 'starting as a secondary')
+            self.p.is_running = true
+            self.ha.follow = Mock(return_value='fake')
+            self.assertEqual(self.ha.run_cycle(), 'fake')
 
     @patch.object(Rewind, 'rewind_or_reinitialize_needed_and_possible', Mock(return_value=True))
     @patch.object(Bootstrap, 'create_replica', Mock(return_value=1))
@@ -681,7 +701,7 @@ class TestHa(PostgresInit):
 
     def test_evaluate_scheduled_restart(self):
         self.p.postmaster_start_time = Mock(return_value=str(postmaster_start_time))
-        # restart already in progres
+        # restart already in progress
         with patch('patroni.async_executor.AsyncExecutor.busy', PropertyMock(return_value=True)):
             self.assertIsNone(self.ha.evaluate_scheduled_restart())
         # restart while the postmaster has been already restarted, fails
diff --git a/tests/test_postgresql.py b/tests/test_postgresql.py
index 54b5c674..96f0b5bc 100644
--- a/tests/test_postgresql.py
+++ b/tests/test_postgresql.py
@@ -322,10 +322,37 @@ class TestPostgresql(BaseTestPostgresql):
         with patch.object(Postgresql, '_query', Mock(side_effect=RetryFailedError(''))):
             self.assertRaises(PostgresConnectionException, self.p.is_leader)
 
-    @patch.object(Postgresql, 'controldata',
-                  Mock(return_value={'Database cluster state': 'shut down', 'Latest checkpoint location': 'X/678'}))
-    def test_latest_checkpoint_location(self):
-        self.assertIsNone(self.p.latest_checkpoint_location())
+    @patch.object(Postgresql, 'controldata', Mock(return_value={'Database cluster state': 'shut down',
+                                                                'Latest checkpoint location': '0/1ADBC18',
+                                                                "Latest checkpoint's TimeLineID": '1'}))
+    @patch('subprocess.Popen')
+    def test_latest_checkpoint_location(self, mock_popen):
+        mock_popen.return_value.communicate.return_value = (None, None)
+        self.assertEqual(self.p.latest_checkpoint_location(), '28163096')
+        # 9.3 and 9.4 format
+        mock_popen.return_value.communicate.side_effect = [
+            (b'rmgr: XLOG        len (rec/tot):     72/   104, tx:          0, lsn: 0/01ADBC18, prev 0/01ADBBB8, ' +
+             b'bkp: 0000, desc: checkpoint: redo 0/1ADBC18; tli 1; prev tli 1; fpw true; xid 0/727; oid 16386; multi' +
+             b' 1; offset 0; oldest xid 715 in DB 1; oldest multi 1 in DB 1; oldest running xid 0; shutdown', None),
+            (b'rmgr: Transaction len (rec/tot):     64/    96, tx:        726, lsn: 0/01ADBBB8, prev 0/01ADBB70, ' +
+             b'bkp: 0000, desc: commit: 2021-02-26 11:19:37.900918 CET; inval msgs: catcache 11 catcache 10', None)]
+        self.assertEqual(self.p.latest_checkpoint_location(), '28163096')
+        mock_popen.return_value.communicate.side_effect = [
+            (b'rmgr: XLOG        len (rec/tot):     72/   104, tx:          0, lsn: 0/01ADBC18, prev 0/01ADBBB8, ' +
+             b'bkp: 0000, desc: checkpoint: redo 0/1ADBC18; tli 1; prev tli 1; fpw true; xid 0/727; oid 16386; multi' +
+             b' 1; offset 0; oldest xid 715 in DB 1; oldest multi 1 in DB 1; oldest running xid 0; shutdown', None),
+            (b'rmgr: XLOG        len (rec/tot):      0/    32, tx:          0, lsn: 0/01ADBBB8, prev 0/01ADBBA0, ' +
+             b'bkp: 0000, desc: xlog switch ', None)]
+        self.assertEqual(self.p.latest_checkpoint_location(), '28163000')
+        # 9.5+ format
+        mock_popen.return_value.communicate.side_effect = [
+            (b'rmgr: XLOG        len (rec/tot):    114/   114, tx:          0, lsn: 0/01ADBC18, prev 0/018260F8, ' +
+             b'desc: CHECKPOINT_SHUTDOWN redo 0/1825ED8; tli 1; prev tli 1; fpw true; xid 0:494; oid 16387; multi 1' +
+             b'; offset 0; oldest xid 479 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0;' +
+             b' oldest running xid 0; shutdown', None),
+            (b'rmgr: XLOG        len (rec/tot):     24/    24, tx:          0, lsn: 0/018260F8, prev 0/01826080, ' +
+             b'desc: SWITCH ', None)]
+        self.assertEqual(self.p.latest_checkpoint_location(), '25321720')
 
     def test_reload(self):
         self.assertTrue(self.p.reload())
diff --git a/tests/test_raft.py b/tests/test_raft.py
index 716295da..a80a668d 100644
--- a/tests/test_raft.py
+++ b/tests/test_raft.py
@@ -4,7 +4,7 @@ import tempfile
 import time
 
 from mock import Mock, PropertyMock, patch
-from patroni.dcs.raft import DynMemberSyncObj, KVStoreTTL, Raft, SyncObjUtility
+from patroni.dcs.raft import DynMemberSyncObj, KVStoreTTL, Raft, SyncObjUtility, TCPTransport, _TCPTransport
 from pysyncobj import SyncObjConf, FAIL_REASON
 
 
@@ -23,6 +23,16 @@ def remove_files(prefix):
                     time.sleep(1.0)
 
 
+class TestTCPTransport(unittest.TestCase):
+
+    @patch.object(TCPTransport, '__init__', Mock())
+    @patch.object(TCPTransport, 'setOnUtilityMessageCallback', Mock())
+    @patch.object(TCPTransport, '_connectIfNecessarySingle', Mock(side_effect=Exception))
+    def test__connectIfNecessarySingle(self):
+        t = _TCPTransport(Mock(), None, [])
+        self.assertFalse(t._connectIfNecessarySingle(None))
+
+
 @patch('pysyncobj.tcp_server.TcpServer.bind', Mock())
 class TestDynMemberSyncObj(unittest.TestCase):
 
@@ -114,8 +124,7 @@ class TestRaft(unittest.TestCase):
     def test_raft(self):
         raft = Raft({'ttl': 30, 'scope': 'test', 'name': 'pg', 'self_addr': '127.0.0.1:1234',
                      'retry_timeout': 10, 'data_dir': self._TMP})
-        raft.set_retry_timeout(20)
-        raft.set_ttl(60)
+        raft.reload_config({'retry_timeout': 20, 'ttl': 60, 'loop_wait': 10})
         self.assertTrue(raft._sync_obj.set(raft.members_path + 'legacy', '{"version":"2.0.0"}'))
         self.assertTrue(raft.touch_member(''))
         self.assertTrue(raft.initialize())