Repository: zalando/patroni
Pull request #2360 (2022-07-14 09:39:20)

#2021 add HEAD support - minimal
Could it be as simple as this?

jocutajar: Ok, added a unit test and adjusted the white space. Thanks for the swift review @CyberDem0n !
jocutajar: Oh dear, does it fail on the trailing white space or something else? It's not quite clear to me.
CyberDem0n: > Oh dear, does it fail on the trailing white space or something else?

Yes: https://github.com/zalando/patroni/runs/7339310338?check_suite_focus=true#step:5:9
```
running flake8
tests/test_api.py:289:28: W291 trailing whitespace
tests/test_api.py:290:83: W291 trailing whitespace
```
jocutajar: There's also the warning on crypto - not related to this PR I suppose:

> /opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/pysyncobj/encryptor.py:3
>   /opt/hostedtoolcache/Python/2.7.18/x64/lib/python2.7/site-packages/pysyncobj/encryptor.py:3: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
>     import cryptography

I'll fix the white space one more time! :D
CyberDem0n: > warning on crypto

yes, it is unrelated.


Please also update docs: https://github.com/zalando/patroni/blob/master/docs/rest_api.rst#health-check-endpoints
jocutajar: > Please also update docs

Done, also did the haproxy.tmpl
jocutajar: Oh, what's the process now? Shall I do something?
hughcapet: 👍 

Patch:
diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml
new file mode 100644
index 00000000..3ba13e0c
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/config.yml
@@ -0,0 +1,1 @@
+blank_issues_enabled: false
diff --git a/docs/pause.rst b/docs/pause.rst
index a33b04b4..285ebac1 100644
--- a/docs/pause.rst
+++ b/docs/pause.rst
@@ -27,6 +27,8 @@ When Patroni runs in a paused mode, it does not change the state of PostgreSQL,
 
 - When Postgres is stopped, Patroni does not try to start it. When Patroni is stopped, it does not try to stop the Postgres instance it is managing.
 
+- Patroni will not try to remove replication slots that don't represent the other cluster member or are not listed in the configuration of the permanent slots.
+
 User guide
 ----------
 
diff --git a/docs/replica_bootstrap.rst b/docs/replica_bootstrap.rst
index 1524f7f2..729e5d47 100644
--- a/docs/replica_bootstrap.rst
+++ b/docs/replica_bootstrap.rst
@@ -187,6 +187,13 @@ Standby leader holds and updates a leader lock in DCS. If the leader lock
 expires, cascade replicas will perform an election to choose another leader
 from the standbys.
 
+There is no further relationship between the standby cluster and the primary
+cluster it replicates from, in particular, they must not share the same DCS
+scope if they use the same DCS. They do not know anything else from each other
+apart from replication information. Also, the standby cluster is not being
+displayed in ``patronictl list`` or ``patronictl topology`` output on the
+primary cluster.
+
 For the sake of flexibility, you can specify methods of creating a replica and
 recovery WAL records when a cluster is in the "standby mode" by providing
 `create_replica_methods` key in `standby_cluster` section. It is distinct from
diff --git a/docs/rest_api.rst b/docs/rest_api.rst
index ad1ac75c..5a2879d3 100644
--- a/docs/rest_api.rst
+++ b/docs/rest_api.rst
@@ -7,7 +7,7 @@ Patroni has a rich REST API, which is used by Patroni itself during the leader r
 
 Health check endpoints
 ----------------------
-For all health check ``GET`` requests Patroni returns a JSON document with the status of the node, along with the HTTP status code. If you don't want or don't need the JSON document, you might consider using the ``OPTIONS`` method instead of ``GET``.
+For all health check ``GET`` requests Patroni returns a JSON document with the status of the node, along with the HTTP status code. If you don't want or don't need the JSON document, you might consider using the ``HEAD`` or ``OPTIONS`` method instead of ``GET``.
 
 - The following requests to Patroni REST API will return HTTP status code **200** only when the Patroni node is running as the primary with leader lock:
 
diff --git a/extras/confd/templates/haproxy.tmpl b/extras/confd/templates/haproxy.tmpl
index 752986f3..31f88194 100644
--- a/extras/confd/templates/haproxy.tmpl
+++ b/extras/confd/templates/haproxy.tmpl
@@ -18,14 +18,14 @@ listen stats
 
 listen master
 	bind *:5000
-	option httpchk OPTIONS /master
+	option httpchk HEAD /master
 	http-check expect status 200
 	default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
 {{range gets "/members/*"}}	server {{base .Key}} {{$data := json .Value}}{{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 100 check port {{index (split (index (split $data.api_url "/") 2) ":") 1}}
 {{end}}
 listen replicas
 	bind *:5001
-	option httpchk OPTIONS /replica
+	option httpchk HEAD /replica
 	http-check expect status 200
 	default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
 {{range gets "/members/*"}}	server {{base .Key}} {{$data := json .Value}}{{base (replace (index (split $data.conn_url "/") 2) "@" "/" -1)}} maxconn 100 check port {{index (split (index (split $data.api_url "/") 2) ":") 1}}
diff --git a/extras/startup-scripts/patroni.service b/extras/startup-scripts/patroni.service
index b2289f75..7c156492 100644
--- a/extras/startup-scripts/patroni.service
+++ b/extras/startup-scripts/patroni.service
@@ -14,7 +14,7 @@ Group=postgres
 # Read in configuration file if it exists, otherwise proceed
 EnvironmentFile=-/etc/patroni_env.conf
 
-# the default is the user's home directory, and if you want to change it, you must provide an absolute path.
+# The default is the user's home directory, and if you want to change it, you must provide an absolute path.
 # WorkingDirectory=/home/sameuser
 
 # Where to send early-startup messages from the server
@@ -32,14 +32,14 @@ ExecStart=/bin/patroni /etc/patroni.yml
 # Send HUP to reload from patroni.yml
 ExecReload=/bin/kill -s HUP $MAINPID
 
-# only kill the patroni process, not it's children, so it will gracefully stop postgres
+# Only kill the patroni process, not it's children, so it will gracefully stop postgres
 KillMode=process
 
 # Give a reasonable amount of time for the server to start up/shut down
 TimeoutSec=30
 
-# Do not restart the service if it crashes, we want to manually inspect database on failure
-Restart=no
+# Restart the service if it crashed
+Restart=on-failure
 
 [Install]
 WantedBy=multi-user.target
diff --git a/features/basic_replication.feature b/features/basic_replication.feature
index 3aaeada6..2c01a92f 100644
--- a/features/basic_replication.feature
+++ b/features/basic_replication.feature
@@ -5,7 +5,7 @@ Feature: basic replication
     Given I start postgres0
     Then postgres0 is a leader after 10 seconds
     And there is a non empty initialize key in DCS after 15 seconds
-    When I issue a PATCH request to http://127.0.0.1:8008/config with {"ttl": 20, "loop_wait": 2, "synchronous_mode": true}
+    When I issue a PATCH request to http://127.0.0.1:8008/config with {"ttl": 20, "synchronous_mode": true}
     Then I receive a response code 200
     When I start postgres1
     And I configure and start postgres2 with a tag replicatefrom postgres0
diff --git a/features/environment.py b/features/environment.py
index 3cb1c533..eb307452 100644
--- a/features/environment.py
+++ b/features/environment.py
@@ -181,7 +181,7 @@ class PatroniController(AbstractController):
         config['postgresql']['data_dir'] = self._data_dir
         config['postgresql']['basebackup'] = [{'checkpoint': 'fast'}]
         config['postgresql']['use_unix_socket'] = os.name != 'nt'  # windows doesn't yet support unix-domain sockets
-        config['postgresql']['use_unix_socket_repl'] = os.name != 'nt'  # windows doesn't yet support unix-domain sockets
+        config['postgresql']['use_unix_socket_repl'] = os.name != 'nt'
         config['postgresql']['pgpass'] = os.path.join(tempfile.gettempdir(), 'pgpass_' + name)
         config['postgresql']['parameters'].update({
             'logging_collector': 'on', 'log_destination': 'csvlog', 'log_directory': self._output_dir,
@@ -197,7 +197,7 @@ class PatroniController(AbstractController):
             self.recursive_update(config, custom_config)
 
         self.recursive_update(config, {
-            'bootstrap': {'dcs': {'postgresql': {'parameters': {'wal_keep_segments': 100}}}}})
+            'bootstrap': {'dcs': {'loop_wait': 2, 'postgresql': {'parameters': {'wal_keep_segments': 100}}}}})
         if config['postgresql'].get('callbacks', {}).get('on_role_change'):
             config['postgresql']['callbacks']['on_role_change'] += ' ' + str(self.__PORT)
 
@@ -626,7 +626,8 @@ class RaftController(AbstractDcsController):
             self.start()
 
         ready_event = threading.Event()
-        self._raft = KVStoreTTL(ready_event.set, None, None, partner_addrs=[self.CONTROLLER_ADDR], password=self.PASSWORD)
+        self._raft = KVStoreTTL(ready_event.set, None, None,
+                                partner_addrs=[self.CONTROLLER_ADDR], password=self.PASSWORD)
         self._raft.startAutoTick()
         ready_event.wait()
 
diff --git a/features/ignored_slots.feature b/features/ignored_slots.feature
index cb6dbf79..abaaef74 100644
--- a/features/ignored_slots.feature
+++ b/features/ignored_slots.feature
@@ -3,7 +3,7 @@ Feature: ignored slots
     Given I start postgres1
     Then postgres1 is a leader after 10 seconds
     And there is a non empty initialize key in DCS after 15 seconds
-    When I issue a PATCH request to http://127.0.0.1:8009/config with {"loop_wait": 2, "ignore_slots": [{"name": "unmanaged_slot_0", "database": "postgres", "plugin": "test_decoding", "type": "logical"}, {"name": "unmanaged_slot_1", "database": "postgres", "plugin": "test_decoding"}, {"name": "unmanaged_slot_2", "database": "postgres"}, {"name": "unmanaged_slot_3"}], "postgresql": {"parameters": {"wal_level": "logical"}}}
+    When I issue a PATCH request to http://127.0.0.1:8009/config with {"ignore_slots": [{"name": "unmanaged_slot_0", "database": "postgres", "plugin": "test_decoding", "type": "logical"}, {"name": "unmanaged_slot_1", "database": "postgres", "plugin": "test_decoding"}, {"name": "unmanaged_slot_2", "database": "postgres"}, {"name": "unmanaged_slot_3"}], "postgresql": {"parameters": {"wal_level": "logical"}}}
     Then I receive a response code 200
     And Response on GET http://127.0.0.1:8009/config contains ignore_slots after 10 seconds
     # Make sure the wal_level has been changed.
diff --git a/features/patroni_api.feature b/features/patroni_api.feature
index 5078ea18..ade92200 100644
--- a/features/patroni_api.feature
+++ b/features/patroni_api.feature
@@ -35,13 +35,13 @@ Scenario: check local configuration reload
 	Then I receive a response code 202
 
 Scenario: check dynamic configuration change via DCS
-	Given I run patronictl.py edit-config -s 'ttl=10' -s 'loop_wait=2' -p 'max_connections=101' --force batman
+	Given I run patronictl.py edit-config -s 'ttl=10' -p 'max_connections=101' --force batman
 	Then I receive a response returncode 0
-	And I receive a response output "+loop_wait: 2"
+	And I receive a response output "+ttl: 10"
 	And Response on GET http://127.0.0.1:8008/patroni contains pending_restart after 11 seconds
 	When I issue a GET request to http://127.0.0.1:8008/config
 	Then I receive a response code 200
-	And I receive a response loop_wait 2
+	And I receive a response ttl 10
 	When I issue a GET request to http://127.0.0.1:8008/patroni
 	Then I receive a response code 200
 	And I receive a response tags {'new_tag': 'new_value'}
diff --git a/features/standby_cluster.feature b/features/standby_cluster.feature
index 82a800b0..5333b5d6 100644
--- a/features/standby_cluster.feature
+++ b/features/standby_cluster.feature
@@ -3,7 +3,7 @@ Feature: standby cluster
     Given I start postgres1
     Then postgres1 is a leader after 10 seconds
     And there is a non empty initialize key in DCS after 15 seconds
-    When I issue a PATCH request to http://127.0.0.1:8009/config with {"loop_wait": 2, "slots": {"pm_1": {"type": "physical"}}, "postgresql": {"parameters": {"wal_level": "logical"}}}
+    When I issue a PATCH request to http://127.0.0.1:8009/config with {"slots": {"pm_1": {"type": "physical"}}, "postgresql": {"parameters": {"wal_level": "logical"}}}
     Then I receive a response code 200
     And Response on GET http://127.0.0.1:8009/config contains slots after 10 seconds
     And I sleep for 3 seconds
diff --git a/patroni/api.py b/patroni/api.py
index 3b7b4192..963be433 100644
--- a/patroni/api.py
+++ b/patroni/api.py
@@ -38,6 +38,7 @@ class RestApiHandler(BaseHTTPRequestHandler):
         self.log_request(status_code)
 
     def _write_response(self, status_code, body, content_type='text/html', headers=None):
+        # TODO: try-catch ConnectionResetError: [Errno 104] Connection reset by peer and log it in DEBUG level
         self.send_response(status_code)
         headers = headers or {}
         if content_type:
@@ -185,6 +186,9 @@ class RestApiHandler(BaseHTTPRequestHandler):
     def do_OPTIONS(self):
         self.do_GET(write_status_code_only=True)
 
+    def do_HEAD(self):
+        self.do_GET(write_status_code_only=True)
+
     def do_GET_liveness(self):
         self._write_status_code_only(200)
 
diff --git a/patroni/ha.py b/patroni/ha.py
index 9f84613f..45cf538d 100644
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -1407,7 +1407,14 @@ class Ha(object):
                     return 'started as a secondary'
 
             # is data directory empty?
-            if self.state_handler.data_directory_empty():
+            try:
+                data_directory_is_empty = self.state_handler.data_directory_empty()
+                data_directory_is_accessible = True
+            except OSError as e:
+                data_directory_is_accessible = False
+                data_directory_error = e
+
+            if not data_directory_is_accessible or data_directory_is_empty:
                 self.state_handler.set_role('uninitialized')
                 self.state_handler.stop('immediate', stop_timeout=self.patroni.config['retry_timeout'])
                 # In case datadir went away while we were master.
@@ -1416,8 +1423,11 @@ class Ha(object):
                 # is this instance the leader?
                 if self.has_lock():
                     self.release_leader_key_voluntarily()
-                    return 'released leader key voluntarily as data dir empty and currently leader'
+                    return 'released leader key voluntarily as data dir {0} and currently leader'.format(
+                                'empty' if data_directory_is_accessible else 'not accessible')
 
+                if not data_directory_is_accessible:
+                    return 'data directory is not accessible: {0}'.format(data_directory_error)
                 if self.is_paused():
                     return 'running with empty data directory'
                 return self.bootstrap()  # new node
@@ -1483,7 +1493,9 @@ class Ha(object):
                 # asynchronous processes are running (should be always the case for the master)
                 if not self._async_executor.busy and not self.state_handler.is_starting():
                     create_slots = self.state_handler.slots_handler.sync_replication_slots(self.cluster,
-                                                                                           self.patroni.nofailover)
+                                                                                           self.patroni.nofailover,
+                                                                                           self.patroni.replicatefrom,
+                                                                                           self.is_paused())
                     if not self.state_handler.cb_called:
                         if not self.state_handler.is_leader():
                             self._rewind.trigger_check_diverged_lsn()
diff --git a/patroni/postgresql/__init__.py b/patroni/postgresql/__init__.py
index a5b3d592..3e2e1d38 100644
--- a/patroni/postgresql/__init__.py
+++ b/patroni/postgresql/__init__.py
@@ -659,7 +659,7 @@ class Postgresql(object):
         if on_safepoint:
             # Wait for our connection to terminate so we can be sure that no new connections are being initiated
             self._wait_for_connection_close(postmaster)
-            postmaster.wait_for_user_backends_to_close()
+            postmaster.wait_for_user_backends_to_close(stop_timeout)
             on_safepoint()
 
         if on_shutdown and mode in ('fast', 'smart'):
diff --git a/patroni/postgresql/postmaster.py b/patroni/postgresql/postmaster.py
index f7b39622..e2a88856 100644
--- a/patroni/postgresql/postmaster.py
+++ b/patroni/postgresql/postmaster.py
@@ -171,8 +171,8 @@ class PostmasterProcess(psutil.Process):
         else:
             return not self.is_running()
 
-    def wait_for_user_backends_to_close(self):
-        # These regexps are cross checked against versions PostgreSQL 9.1 .. 11
+    def wait_for_user_backends_to_close(self, stop_timeout):
+        # These regexps are cross checked against versions PostgreSQL 9.1 .. 15
         aux_proc_re = re.compile("(?:postgres:)( .*:)? (?:(?:archiver|startup|autovacuum launcher|autovacuum worker|"
                                  "checkpointer|logger|stats collector|wal receiver|wal writer|writer)(?: process  )?|"
                                  "walreceiver|wal sender process|walsender|walwriter|background writer|"
@@ -184,19 +184,23 @@ class PostmasterProcess(psutil.Process):
             return logger.debug('Failed to get list of postmaster children')
 
         user_backends = []
-        user_backends_cmdlines = []
+        user_backends_cmdlines = {}
         for child in children:
             try:
                 cmdline = child.cmdline()
                 if cmdline and not aux_proc_re.match(cmdline[0]):
                     user_backends.append(child)
-                    user_backends_cmdlines.append(cmdline[0])
+                    user_backends_cmdlines[child.pid] = cmdline[0]
             except psutil.NoSuchProcess:
                 pass
         if user_backends:
-            logger.debug('Waiting for user backends %s to close', ', '.join(user_backends_cmdlines))
-            psutil.wait_procs(user_backends)
-        logger.debug("Backends closed")
+            logger.debug('Waiting for user backends %s to close', ', '.join(user_backends_cmdlines.values()))
+            gone, live = psutil.wait_procs(user_backends, stop_timeout)
+            if stop_timeout and live:
+                live = [user_backends_cmdlines[b.pid] for b in live]
+                logger.warning('Backends still alive after %s: %s', stop_timeout, ', '.join(live))
+            else:
+                logger.debug("Backends closed")
 
     @staticmethod
     def start(pgcommand, data_dir, conf, options):
diff --git a/patroni/postgresql/rewind.py b/patroni/postgresql/rewind.py
index 481f0b7a..595105ba 100644
--- a/patroni/postgresql/rewind.py
+++ b/patroni/postgresql/rewind.py
@@ -1,6 +1,8 @@
 import logging
 import os
+import re
 import shlex
+import shutil
 import six
 import subprocess
 
@@ -277,28 +279,37 @@ class Rewind(object):
     def checkpoint_after_promote(self):
         return self._state == REWIND_STATUS.CHECKPOINT
 
-    def _fetch_missing_wal(self, restore_command, wal_filename):
+    def _buid_archiver_command(self, command, wal_filename):
+        """Replace placeholders in the given archiver command's template.
+        Applicable for archive_command and restore_command.
+        Can also be used for archive_cleanup_command and recovery_end_command,
+        however %r value is always set to 000000010000000000000001."""
         cmd = ''
-        length = len(restore_command)
+        length = len(command)
         i = 0
         while i < length:
-            if restore_command[i] == '%' and i + 1 < length:
+            if command[i] == '%' and i + 1 < length:
                 i += 1
-                if restore_command[i] == 'p':
+                if command[i] == 'p':
                     cmd += os.path.join(self._postgresql.wal_dir, wal_filename)
-                elif restore_command[i] == 'f':
+                elif command[i] == 'f':
                     cmd += wal_filename
-                elif restore_command[i] == 'r':
+                elif command[i] == 'r':
                     cmd += '000000010000000000000001'
-                elif restore_command[i] == '%':
+                elif command[i] == '%':
                     cmd += '%'
                 else:
                     cmd += '%'
                     i -= 1
             else:
-                cmd += restore_command[i]
+                cmd += command[i]
             i += 1
 
+        return cmd
+
+    def _fetch_missing_wal(self, restore_command, wal_filename):
+        cmd = self._buid_archiver_command(restore_command, wal_filename)
+
         logger.info('Trying to fetch the missing wal: %s', cmd)
         return self._postgresql.cancellable.call(shlex.split(cmd)) == 0
 
@@ -315,6 +326,42 @@ class Rewind(object):
                     if waldir.endswith('/pg_' + self._postgresql.wal_name) and len(wal_filename) == 24:
                         return wal_filename
 
+    def _archive_ready_wals(self):
+        """Try to archive WALs that have .ready files just in case
+        archive_mode was not set to 'always' before promote, while
+        after it the WALs were recycled on the promoted replica.
+        With this we prevent the entire loss of such WALs and the
+        consequent old leader's start failure."""
+        archive_mode = self._postgresql.get_guc_value('archive_mode')
+        archive_cmd = self._postgresql.get_guc_value('archive_command')
+        if archive_mode not in ('on', 'always') or not archive_cmd:
+            return
+
+        walseg_regex = re.compile(r'^[0-9A-F]{24}(\.partial){0,1}\.ready$')
+        status_dir = os.path.join(self._postgresql.wal_dir, 'archive_status')
+        try:
+            wals_to_archive = [f[:-6] for f in os.listdir(status_dir) if walseg_regex.match(f)]
+        except OSError as e:
+            return logger.error('Unable to list %s: %r', status_dir, e)
+
+        # skip fsync, as postgres --single or pg_rewind will anyway run it
+        for wal in sorted(wals_to_archive):
+            old_name = os.path.join(status_dir, wal + '.ready')
+            # wal file might have alredy been archived
+            if os.path.isfile(old_name) and os.path.isfile(os.path.join(self._postgresql.wal_dir, wal)):
+                cmd = self._buid_archiver_command(archive_cmd, wal)
+                # it is the author of archive_command, who is responsible
+                # for not overriding the WALs already present in archive
+                logger.info('Trying to archive %s: %s', wal, cmd)
+                if self._postgresql.cancellable.call(shlex.split(cmd)) == 0:
+                    new_name = os.path.join(status_dir, wal + '.done')
+                    try:
+                        shutil.move(old_name, new_name)
+                    except Exception as e:
+                        logger.error('Unable to rename %s to %s: %r', old_name, new_name, e)
+                else:
+                    logger.info('Failed to archive WAL segment %s', wal)
+
     def pg_rewind(self, r):
         # prepare pg_rewind connection
         env = self._postgresql.config.write_pgpass(r)
@@ -367,6 +414,8 @@ class Rewind(object):
         if self._postgresql.is_running() and not self._postgresql.stop(checkpoint=False):
             return logger.warning('Can not run pg_rewind because postgres is still running')
 
+        self._archive_ready_wals()
+
         # prepare pg_rewind connection
         r = self._conn_kwargs(leader, self._postgresql.config.rewind_credentials)
 
@@ -465,6 +514,7 @@ class Rewind(object):
             logger.exception('Unable to list %s', status_dir)
 
     def ensure_clean_shutdown(self):
+        self._archive_ready_wals()
         self.cleanup_archive_status()
 
         # Start in a single user mode and stop to produce a clean shutdown
diff --git a/patroni/postgresql/slots.py b/patroni/postgresql/slots.py
index 32cbc9b8..0d5051f7 100644
--- a/patroni/postgresql/slots.py
+++ b/patroni/postgresql/slots.py
@@ -112,10 +112,10 @@ class SlotsHandler(object):
         # In normal situation rowcount should be 1, otherwise either slot doesn't exists or it is still active
         return cursor.rowcount == 1
 
-    def _drop_incorrect_slots(self, cluster, slots):
+    def _drop_incorrect_slots(self, cluster, slots, paused):
         # drop old replication slots which are not presented in desired slots
         for name in set(self._replication_slots) - set(slots):
-            if not self.ignore_replication_slot(cluster, name) and not self.drop_replication_slot(name):
+            if not paused and not self.ignore_replication_slot(cluster, name) and not self.drop_replication_slot(name):
                 logger.error("Failed to drop replication slot '%s'", name)
                 self._schedule_load_slots = True
 
@@ -206,7 +206,7 @@ class SlotsHandler(object):
                         self._schedule_load_slots = True
         return create_slots
 
-    def sync_replication_slots(self, cluster, nofailover, replicatefrom=None):
+    def sync_replication_slots(self, cluster, nofailover, replicatefrom=None, paused=False):
         ret = None
         if self._postgresql.major_version >= 90400 and cluster.config:
             try:
@@ -215,7 +215,7 @@ class SlotsHandler(object):
                 slots = cluster.get_replication_slots(self._postgresql.name, self._postgresql.role,
                                                       nofailover, self._postgresql.major_version, True)
 
-                self._drop_incorrect_slots(cluster, slots)
+                self._drop_incorrect_slots(cluster, slots, paused)
 
                 self._ensure_physical_slots(slots)
 
diff --git a/requirements.dev.txt b/requirements.dev.txt
index 35726b50..36fd575c 100644
--- a/requirements.dev.txt
+++ b/requirements.dev.txt
@@ -1,7 +1,7 @@
 psycopg2-binary
 behave
 coverage
-flake8
+flake8>=3.0.0
 mock
 pytest-cov
 pytest
diff --git a/setup.py b/setup.py
index 786c1a26..d5129854 100644
--- a/setup.py
+++ b/setup.py
@@ -93,12 +93,10 @@ class Flake8(_Command):
         return [package for package in self.package_files()] + ['tests', 'setup.py']
 
     def run(self):
-        from flake8.main import application
+        from flake8.main.cli import main
 
         logging.getLogger().setLevel(logging.ERROR)
-        flake8 = application.Application()
-        flake8.run(self.targets())
-        flake8.exit()
+        main(self.targets())
 
 
 class PyTest(_Command):


Test patch:
diff --git a/tests/test_api.py b/tests/test_api.py
index 01e5dc89..681814b0 100644
--- a/tests/test_api.py
+++ b/tests/test_api.py
@@ -286,6 +286,9 @@ class TestRestApiHandler(unittest.TestCase):
     def test_do_OPTIONS(self):
         self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'OPTIONS / HTTP/1.0'))
 
+    def test_do_HEAD(self):
+        self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'HEAD / HTTP/1.0'))
+
     def test_do_GET_liveness(self):
         self.assertIsNotNone(MockRestApiServer(RestApiHandler, 'GET /liveness HTTP/1.0'))
 
diff --git a/tests/test_ha.py b/tests/test_ha.py
index 5ed4503d..1d6e2904 100644
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -1133,17 +1133,19 @@ class TestHa(PostgresInit):
         self.ha.shutdown()
 
     @patch('time.sleep', Mock())
-    def test_leader_with_empty_directory(self):
+    def test_leader_with_not_accessible_data_directory(self):
         self.ha.cluster = get_cluster_initialized_with_leader()
         self.ha.has_lock = true
-        self.p.data_directory_empty = true
-        self.assertEqual(self.ha.run_cycle(), 'released leader key voluntarily as data dir empty and currently leader')
+        self.p.data_directory_empty = Mock(side_effect=OSError(5, "Input/output error: '{}'".format(self.p.data_dir)))
+        self.assertEqual(self.ha.run_cycle(),
+                         'released leader key voluntarily as data dir not accessible and currently leader')
         self.assertEqual(self.p.role, 'uninitialized')
 
         # as has_lock is mocked out, we need to fake the leader key release
         self.ha.has_lock = false
-        # will not say bootstrap from leader as replica can't self elect
-        self.assertEqual(self.ha.run_cycle(), "trying to bootstrap from replica 'other'")
+        # will not say bootstrap because data directory is not accessible
+        self.assertEqual(self.ha.run_cycle(),
+                         "data directory is not accessible: [Errno 5] Input/output error: '{}'".format(self.p.data_dir))
 
     @patch('patroni.postgresql.mtime', Mock(return_value=1588316884))
     @patch.object(builtins, 'open', mock_open(read_data=('1\t0/40159C0\tno recovery target specified\n\n'
diff --git a/tests/test_postmaster.py b/tests/test_postmaster.py
index 07f09de2..61a45049 100644
--- a/tests/test_postmaster.py
+++ b/tests/test_postmaster.py
@@ -133,14 +133,20 @@ class TestPostmasterProcess(unittest.TestCase):
         c2.cmdline = Mock(return_value=["postgres: postgres postgres [local] idle"])
         c3 = Mock()
         c3.cmdline = Mock(side_effect=psutil.NoSuchProcess(123))
+        mock_wait.return_value = ([], [c2])
         with patch('psutil.Process.children', Mock(return_value=[c1, c2, c3])):
             proc = PostmasterProcess(123)
-            self.assertIsNone(proc.wait_for_user_backends_to_close())
-            mock_wait.assert_called_with([c2])
+            self.assertIsNone(proc.wait_for_user_backends_to_close(1))
+            mock_wait.assert_called_with([c2], 1)
+
+        mock_wait.return_value = ([c2], [])
+        with patch('psutil.Process.children', Mock(return_value=[c1, c2, c3])):
+            proc = PostmasterProcess(123)
+            proc.wait_for_user_backends_to_close(1)
 
         with patch('psutil.Process.children', Mock(side_effect=psutil.NoSuchProcess(123))):
             proc = PostmasterProcess(123)
-            self.assertIsNone(proc.wait_for_user_backends_to_close())
+            self.assertIsNone(proc.wait_for_user_backends_to_close(None))
 
     @patch('subprocess.Popen')
     @patch('os.setsid', Mock(), create=True)
diff --git a/tests/test_rewind.py b/tests/test_rewind.py
index 2d510b33..b9398205 100644
--- a/tests/test_rewind.py
+++ b/tests/test_rewind.py
@@ -218,6 +218,62 @@ class TestRewind(BaseTestPostgresql):
         self.r.cleanup_archive_status()
         self.r.cleanup_archive_status()
 
+    @patch('os.path.isfile', Mock(return_value=True))
+    @patch('shutil.move', Mock(side_effect=OSError))
+    @patch('patroni.postgresql.rewind.logger.info')
+    def test_archive_ready_wals(self, mock_logger_info):
+        with patch('os.listdir', Mock(side_effect=OSError)), \
+              patch.object(Postgresql, 'get_guc_value', Mock(side_effect=['on', 'command %f'])):
+            self.r._archive_ready_wals()
+            mock_logger_info.assert_not_called()
+
+        # each assert_not_called() calls get_guc_value('archive_mode') + get_guc_value('archive_command')
+        get_guc_value_res = [
+            '', 'command %f',
+            'on', '',
+        ]
+        with patch.object(Postgresql, 'get_guc_value', Mock(side_effect=get_guc_value_res)):
+            for _ in range(len(get_guc_value_res)//2):
+                self.r._archive_ready_wals()
+                mock_logger_info.assert_not_called()
+
+        with patch('os.listdir', Mock(return_value=['000000000000000000000000.ready'])):
+            # successful archive_command call
+            with patch.object(CancellableSubprocess, 'call',  Mock(return_value=0)):
+                get_guc_value_res = [
+                    'on', 'command %f',
+                    'always', 'command %f',
+                ]
+                with patch.object(Postgresql, 'get_guc_value', Mock(side_effect=get_guc_value_res)):
+                    for _ in range(len(get_guc_value_res)//2):
+                        self.r._archive_ready_wals()
+                        mock_logger_info.assert_called_once()
+                        self.assertEqual(('Trying to archive %s: %s',
+                                          '000000000000000000000000', 'command 000000000000000000000000'),
+                                         mock_logger_info.call_args[0])
+                        mock_logger_info.reset_mock()
+
+            # failed archive_command call
+            with patch.object(CancellableSubprocess, 'call', Mock(return_value=1)):
+                with patch.object(Postgresql, 'get_guc_value', Mock(side_effect=['on', 'command %f'])):
+                    self.r._archive_ready_wals()
+                    self.assertEqual(('Trying to archive %s: %s',
+                                      '000000000000000000000000', 'command 000000000000000000000000'),
+                                     mock_logger_info.call_args_list[0][0])
+                    self.assertEqual(('Failed to archive WAL segment %s', '000000000000000000000000'),
+                                     mock_logger_info.call_args_list[1][0])
+                    mock_logger_info.reset_mock()
+
+        wal_files_to_skip = [
+            '000000000000000000000000.done',
+            '000000000000000000000001.partial.done',
+            '002.ready',
+            'U00000000000000000000001.ready',
+        ]
+        with patch('os.listdir', Mock(return_value=wal_files_to_skip)):
+            self.r._archive_ready_wals()
+            mock_logger_info.assert_not_called()
+
     @patch('os.unlink', Mock())
     @patch('os.listdir', Mock(return_value=[]))
     @patch('os.path.isfile', Mock(return_value=True))
diff --git a/tests/test_slots.py b/tests/test_slots.py
index 454fab8d..2301ae5f 100644
--- a/tests/test_slots.py
+++ b/tests/test_slots.py
@@ -42,8 +42,10 @@ class TestSlotsHandler(BaseTestPostgresql):
         self.p.set_role('standby_leader')
         self.s.sync_replication_slots(cluster, False)
         self.p.set_role('replica')
-        with patch.object(Postgresql, 'is_leader', Mock(return_value=False)):
-            self.s.sync_replication_slots(cluster, False)
+        with patch.object(Postgresql, 'is_leader', Mock(return_value=False)),\
+                patch.object(SlotsHandler, 'drop_replication_slot') as mock_drop:
+            self.s.sync_replication_slots(cluster, False, paused=True)
+            mock_drop.assert_not_called()
         self.p.set_role('master')
         with mock.patch('patroni.postgresql.Postgresql.role', new_callable=PropertyMock(return_value='replica')):
             self.s.sync_replication_slots(cluster, False)
diff --git a/tests/test_watchdog.py b/tests/test_watchdog.py
index e70213c5..c326c1f3 100644
--- a/tests/test_watchdog.py
+++ b/tests/test_watchdog.py
@@ -35,7 +35,7 @@ def mock_ioctl(fd, op, arg=None, mutate_flag=False):
     sys.stderr.write("Ioctl %d %d %r\n" % (fd, op, arg))
     if op == linuxwd.WDIOC_GETSUPPORT:
         sys.stderr.write("Get support\n")
-        assert(mutate_flag is True)
+        assert (mutate_flag is True)
         arg.options = sum(map(linuxwd.WDIOF.get, ['SETTIMEOUT', 'KEEPALIVEPING']))
         arg.identity = (ctypes.c_ubyte*32)(*map(ord, 'Mock Watchdog'))
     elif op == linuxwd.WDIOC_GETTIMEOUT: