Repository: zalando/patroni

Issue:
Permanent replication slot not progressing
```
patronictl version
patronictl version 2.1.2
```
We have a number of patroni clusters which maintain multiple replication slots. Cluster has three nodes:

```
+ Cluster: cluster-1 (7072422856714349561) -------+---------+---------+----+-----------+------------------+
| Member                           | Host         | Role    | State   | TL | Lag in MB | Tags             |
+----------------------------------+--------------+---------+---------+----+-----------+------------------+
| node-2                           | 10.1.0.3     | Replica | running |  2 |         0 | nofailover: true |
|                                  |              |         |         |    |           | nosync: true     |
+----------------------------------+--------------+---------+---------+----+-----------+------------------+
| node-0                           | 10.1.0.1     | Leader  | running |  2 |           |                  |
+----------------------------------+--------------+---------+---------+----+-----------+------------------+
| node-1                           | 10.1.0.2     | Replica | running |  2 |         0 |                  |
+----------------------------------+--------------+---------+---------+----+-----------+------------------+
```

One of the standbys randomly stopped progressing at least one replication slot. You could see it attempting to do so in:

```
  pid  |                                       left                                       |   datname   | usename  | application_name |       age       | wait_event
-------+----------------------------------------------------------------------------------+-------------+----------+------------------+-----------------+------------
 17804 | SELECT pg_catalog.pg_replication_slot_advance('bad_slot2', '199/65915E78') | db5 | pgadmin | Patroni          | 00:19:38.426721 | WALRead
```

This leads to wal files not being removed on a secondary and slowly filling the disk. 

Replication slots stats:

```
       slot_name       |  plugin  | slot_type | datoid  |     database      | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn  | confirmed_flush_lsn | wal_status | safe_wal_size | two_phase
-----------------------+----------+-----------+---------+-------------------+-----------+--------+------------+------+--------------+--------------+---------------------+------------+---------------+-----------
 bad_slot1                | wal2json | logical   |   32273 | db1        | f         | f      |            |      |      6457848 | 15C/EA222C90 | 199/65B5FBE8        | reserved   |               | f
 good_slot1               | wal2json | logical   | 4624065 | db2               | f         | f      |            |      |      6529903 | 18F/D1D5BB0  | 199/65A9B820        | reserved   |               | f
 good_slot2              | wal2json | logical   | 5265720 | db3          | f         | f      |            |      |      6529903 | 18F/D1D5BB0  | 199/6578D8D0        | reserved   |               | f
 good_slot3            | wal2json | logical   | 6664819 | db4            | f         | f      |            |      |      6529903 | 18F/D1D5BB0  | 199/657A7928        | reserved   |               | f
 bad_slot2              | wal2json | logical   | 8515756 | db5       | f         | t      |      17804 |      |      6529654 | 18F/CFFFB08  | 18F/D4988C0         | reserved   |               | f
```

Logs:

```
Mar 11 00:26:34 node-1 patroni: 2022-03-11 00:26:34,128 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:26:49 node-1 patroni: 2022-03-11 00:26:49,528 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:27:07 node-1 patroni: 2022-03-11 00:27:07,713 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:27:07 node-1 patroni: 2022-03-11 00:27:07,714 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 00:27:42 node-1 patroni: 2022-03-11 00:27:42,944 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:27:42 node-1 patroni: 2022-03-11 00:27:42,946 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 00:28:46 node-1 patroni: 2022-03-11 00:28:46,002 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:28:46 node-1 patroni: 2022-03-11 00:28:46,004 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 00:30:50 node-1 patroni: 2022-03-11 00:30:50,153 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:30:50 node-1 patroni: 2022-03-11 00:30:50,154 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 00:35:20 node-1 patroni: 2022-03-11 00:35:20,798 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:35:20 node-1 patroni: 2022-03-11 00:35:20,799 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 00:42:35 node-1 patroni: 2022-03-11 00:42:35,619 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:42:35 node-1 patroni: 2022-03-11 00:42:35,620 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 00:54:12 node-1 patroni: 2022-03-11 00:54:12,748 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 00:54:12 node-1 patroni: 2022-03-11 00:54:12,749 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 01:14:27 node-1 patroni: 2022-03-11 01:14:27,781 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 01:14:27 node-1 patroni: 2022-03-11 01:14:27,782 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 01:49:01 node-1 patroni: 2022-03-11 01:49:01,098 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 01:49:01 node-1 patroni: 2022-03-11 01:49:01,100 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 03:23:20 node-1 patroni: 2022-03-11 03:23:20,014 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 03:23:20 node-1 patroni: 2022-03-11 03:23:20,016 WARNING: Loop time exceeded, rescheduling immediately.
Mar 11 15:59:16 node-1 patroni: 2022-03-11 15:59:16,008 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
Mar 11 15:59:16 node-1 patroni: 2022-03-11 15:59:16,010 WARNING: Loop time exceeded, rescheduling immediately.
Mar 12 01:49:10 node-1 patroni: 2022-03-11 15:59:16,014 INFO: Lock owner: node-0; I am node-1
Mar 12 01:49:10 node-1 patroni: 2022-03-12 01:49:10,266 ERROR: Failed to advance logical replication slot 'bad_slot2': AdminShutdown('terminating connection due to administrator command\nSSL connection has been closed unexpectedly\n',)
```

The last message is me terminating the backend from postgres, which did not alleviate the issue.

During the issue, the node was not reported in `patronictl list` at all, though patroni service was up and running. The command returned the master and the `no_failover: true` replica, which never has any slots on it (that seems like a feature):

```
+ Cluster: cluster-1 (7072422856714349561) -------+---------+---------+----+-----------+------------------+
| Member                           | Host         | Role    | State   | TL | Lag in MB | Tags             |
+----------------------------------+--------------+---------+---------+----+-----------+------------------+
| node-2                           | 10.1.0.3     | Replica | running |  2 |         0 | nofailover: true |
|                                  |              |         |         |    |           | nosync: true     |
+----------------------------------+--------------+---------+---------+----+-----------+------------------+
| node-0                           | 10.1.0.1     | Leader  | running |  2 |           |                  |
+----------------------------------+--------------+---------+---------+----+-----------+------------------+
```

All the slots are configured as follows:
```yaml
slots:
  bad_slot2:
    database: db5
    plugin: wal2json
    type: logical
 ...
```

We recently upgraded patroni version on our clusters. We were aware that the persistent slot feature was changed in the last update, so we tested it prior to the upgrade and it worked just fine. However, within the last two weeks this happened twice and I'm wondering if there's something that could prevent patroni from progressing the slot.

The only solution that I found effective was to completely stop patroni and postgres and drop all the files from `$PGDATA/pg_replslot/` folder and restart patroni.


Discussion:
What is in postgres logs?
It is also not clear whether it is completely stale or slowly moving forward.

> What is in postgres logs?

Postgres logs don't really show anything. There's no errors, no messages other than the fact that the `pg_replication_slot_advance` statement has been started. This is the extract from when I terminated the session running the advance function:

```
2022-03-12 01:49:10.265 UTC [17804] (user=pgadmin) (db=db5) (rhost=::1) (app=Patroni) [vxid:25/23577 txid:0] [SELECT] FATAL:  terminating connection due to administrator command
2022-03-12 01:49:10.265 UTC [17804] (user=pgadmin) (db=db5) (rhost=::1) (app=Patroni) [vxid:25/23577 txid:0] [SELECT] STATEMENT:  SELECT pg_catalog.pg_replication_slot_advance('slot_db5', '199/65915E78')
2022-03-12 01:49:10.265 UTC [17804] (user=pgadmin) (db=db5) (rhost=::1) (app=Patroni) [vxid: txid:0] [SELECT] LOG:  disconnection: session time: 0:21:19.803 user=pgadmin database=db5 host=::1 port=40284
2022-03-12 01:49:10.268 UTC [26458] (user=[unknown]) (db=[unknown]) (rhost=::1) (app=[unknown]) [vxid: txid:0] [] LOG:  connection received: host=::1 port=48062
2022-03-12 01:49:10.277 UTC [26458] (user=pgadmin) (db=db2) (rhost=::1) (app=[unknown]) [vxid:25/23578 txid:0] [authentication] LOG:  connection authenticated: identity="pgadmin" method=md5 (/pgsql/cluster/data/pg_hba.conf:14)
2022-03-12 01:49:10.420 UTC [26458] (user=pgadmin) (db=db2) (rhost=::1) (app=Patroni) [vxid:25/23579 txid:0] [SELECT] LOG:  starting logical decoding for slot "slot_db2"
2022-03-12 01:49:10.420 UTC [26458] (user=pgadmin) (db=db2) (rhost=::1) (app=Patroni) [vxid:25/23579 txid:0] [SELECT] DETAIL:  Streaming transactions committing after 18F/D824C38, reading WAL from 15C/EA222C90.
2022-03-12 01:49:10.420 UTC [26458] (user=pgadmin) (db=db2) (rhost=::1) (app=Patroni) [vxid:25/23579 txid:0] [SELECT] STATEMENT:  SELECT pg_catalog.pg_replication_slot_advance('slot_db2', '199/657A7928')
2022-03-12 01:49:10.558 UTC [26458] (user=pgadmin) (db=db2) (rhost=::1) (app=Patroni) [vxid:25/23579 txid:0] [SELECT] LOG:  logical decoding found consistent point at 15C/EA222C90
2022-03-12 01:49:10.558 UTC [26458] (user=pgadmin) (db=db2) (rhost=::1) (app=Patroni) [vxid:25/23579 txid:0] [SELECT] DETAIL:  Logical decoding will begin using saved snapshot.
2022-03-12 01:49:10.558 UTC [26458] (user=pgadmin) (db=db2) (rhost=::1) (app=Patroni) [vxid:25/23579 txid:0] [SELECT] STATEMENT:  SELECT pg_catalog.pg_replication_slot_advance('slot_db2', '199/657A7928')
```

> It is also not clear whether it is completely stale or slowly moving forward.

As far as my terminal log goes, the slot was stuck at the same LSN and was not progressing.
> no messages other than the fact that the pg_replication_slot_advance statement has been started

Well, pg_replication_slot_advance() doesn't run instantly, it has to actually read and process WAL records.
Your pg_stat_activity query actually proves it, because `wait_event` is `WALRead`.
According to the pg_replication_slots view, the current lsn of this slot is `18F/CFFFB08` and it is being advanced to the `199/65915E78`. That requires to process 44GB of WAL files and of course it takes time.
It does take time, but I'd imagine it is not supposed to take more than 24 hours. Seeing how the slot wasn't progressing, I'm inclined to believe that it was stuck for whatever reason.

This is the count of wal files from our monitoring:

![image](https://user-images.githubusercontent.com/30303784/158441598-6af5e867-e9e9-47bd-8d4f-b56e8fab387d.png)

Replication slot on the replica is only advanced when it is moving forward on the primary.  Are you sure that a client was connected to the slot and more important it was periodically updating the flush lsn?
Yes, we monitor the replication slot lag on a primary. If the LSN wasn't updated on primary, removing the slot on the secondary would not impact the slot lag and it would still be stuck, even after re-creating. To the same extent, the number of  wal files on the leader would also increase at the similar rate, while in reality this happened:

![image](https://user-images.githubusercontent.com/30303784/158482564-788de485-f3cd-4ce4-9506-4d393b34d9a0.png)

The wal files on primary grew together with the wal files on secondary, but then got properly removed. The secondary got stuck in a weird state. What could have impacted this: at midnight, a logical backup kicks off, which enables the hot_standby_feedback for the replica to prevent backup termination. But I'm not sure how a replication slot progress could be affected by this.
> which enables the hot_standby_feedback for the replica

Hmm, the `hot_standby_feedback` is already enabled on replicas that are either supposed to be failover targets or doing cascading replication for failover targets.
Without the `hot_standby_feedback` logical slots are not guaranteed to work because the primary might have cleaned some dead tuples in pg_catalog that are required for logical decoding to work.

In any case, according to Patroni logs it was correctly calling `pg_replication_slot_advance()`. Hence, you have to examine postgres logs from the moment when lag started accumulating.
Small update: we have experienced this problem once more since this issue was opened. We decided to enable `hot_standby_feedback` everywhere and see if it alleviates the issue. I'll make another update in a couple of months.
Update: Enabling `hot_standby_feedback` did not help, and after having a few more issues like the one described we decided to stop using the permanent slots feature. It would be cool if we could get back to the old slot recreation logic instead of the current logic.

As for the symptoms, whatever patroni is doing during the slot progression, it seems that this is a synchronous operation, and the whole the main failover loop/healthchecks are not running while the slot advancement is in progress. This might explain why the replica is removed from the cluster node list and why we're seeing a lot of messages like these in patroni logs:

```
May 11 16:26:43 node-1 patroni: 2022-05-11 16:26:43,731 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
May 11 16:26:43 node-1 patroni: 2022-05-11 16:26:43,733 WARNING: Loop time exceeded, rescheduling immediately.
May 11 16:27:12 node-1 patroni: 2022-05-11 16:27:12,842 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
May 11 16:27:12 node-1 patroni: 2022-05-11 16:27:12,844 WARNING: Loop time exceeded, rescheduling immediately.
May 11 16:28:13 node-1 patroni: 2022-05-11 16:28:13,370 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
May 11 16:28:13 node-1 patroni: 2022-05-11 16:28:13,371 WARNING: Loop time exceeded, rescheduling immediately.
May 11 16:30:45 node-1 patroni: 2022-05-11 16:30:45,542 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
May 11 16:30:45 node-1 patroni: 2022-05-11 16:30:45,544 WARNING: Loop time exceeded, rescheduling immediately.
May 11 16:36:10 node-1 patroni: 2022-05-11 16:36:10,325 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
May 11 16:36:10 node-1 patroni: 2022-05-11 16:36:10,327 WARNING: Loop time exceeded, rescheduling immediately.
May 11 16:43:48 node-1 patroni: 2022-05-11 16:43:48,736 INFO: no action. I am (node-1), a secondary, and following a leader (node-0)
May 11 16:43:48 node-1 patroni: 2022-05-11 16:43:48,737 WARNING: Loop time exceeded, rescheduling immediately.
```
> Update: Enabling hot_standby_feedback did not help

Well, according to logs that you post here slot is certainly moving forward. In the opposite case Patroni would log an error or an exception.

> t would be cool if we could get back to the old slot recreation logic instead of the current logic.

It would be not. In case if you really want it is possible to mimic the old behavior by using `ignore_slots` and `callbacks`.

> whatever patroni is doing during the slot progression, it seems that this is a synchronous operation, and the whole the main failover loop/healthchecks are not running while the slot advancement is in progress. This might explain why the replica is removed from the cluster node list and why we're seeing a lot of messages like these in patroni logs:

It is indeed calling `pg_advance_replication_slot()` from the main thread. We certainly can improve it by starting a dedicated thread for that. But, what I find really bad is that the replica can't keep up with the pace of the primary in your case. I.e., if the logical slot in the primary moved forward X bytes in 10 seconds it shouldn't be taking much longer to do the same on the replica or the replica isn't running on a hardware that is comparable with the primary.

Resolving patch:
diff --git a/patroni/postgresql/slots.py b/patroni/postgresql/slots.py
--- a/patroni/postgresql/slots.py
+++ b/patroni/postgresql/slots.py
@@ -5,6 +5,7 @@
 
 from collections import defaultdict
 from contextlib import contextmanager
+from threading import Condition, Thread
 
 from .connection import get_connection_cursor
 from .misc import format_lsn
@@ -31,10 +32,94 @@ def fsync_dir(path):
             os.close(fd)
 
 
+class SlotsAdvanceThread(Thread):
+
+    def __init__(self, slots_handler):
+        super(SlotsAdvanceThread, self).__init__()
+        self.daemon = True
+        self._slots_handler = slots_handler
+
+        # _copy_slots and _failed are used to asynchronously give some feedback to the main thread
+        self._copy_slots = []
+        self._failed = False
+
+        self._scheduled = defaultdict(dict)  # {'dbname1': {'slot1': 100, 'slot2': 100}, 'dbname2': {'slot3': 100}}
+        self._condition = Condition()  # protect self._scheduled from concurrent access and to wakeup the run() method
+
+        self.start()
+
+    def sync_slot(self, cur, database, slot, lsn):
+        failed = copy = False
+        try:
+            cur.execute("SELECT pg_catalog.pg_replication_slot_advance(%s, %s)", (slot, format_lsn(lsn)))
+        except Exception as e:
+            logger.error("Failed to advance logical replication slot '%s': %r", slot, e)
+            failed = True
+            copy = isinstance(e, OperationalError) and e.diag.sqlstate == '58P01'  # WAL file is gone
+        with self._condition:
+            if self._scheduled and failed:
+                if copy and slot not in self._copy_slots:
+                    self._copy_slots.append(slot)
+                self._failed = True
+
+            new_lsn = self._scheduled.get(database, {}).get(slot, 0)
+            # remove slot from the self._scheduled structure only if it wasn't changed
+            if new_lsn == lsn and database in self._scheduled:
+                self._scheduled[database].pop(slot)
+                if not self._scheduled[database]:
+                    self._scheduled.pop(database)
+
+    def sync_slots_in_database(self, database, slots):
+        with self._slots_handler.get_local_connection_cursor(dbname=database, options='-c statement_timeout=0') as cur:
+            for slot in slots:
+                with self._condition:
+                    lsn = self._scheduled.get(database, {}).get(slot, 0)
+                if lsn:
+                    self.sync_slot(cur, database, slot, lsn)
+
+    def sync_slots(self):
+        with self._condition:
+            databases = list(self._scheduled.keys())
+        for database in databases:
+            with self._condition:
+                slots = list(self._scheduled.get(database, {}).keys())
+            if slots:
+                try:
+                    self.sync_slots_in_database(database, slots)
+                except Exception as e:
+                    logger.error('Failed to advance replication slots in database %s: %r', database, e)
+
+    def run(self):
+        while True:
+            with self._condition:
+                if not self._scheduled:
+                    self._condition.wait()
+
+            self.sync_slots()
+
+    def schedule(self, advance_slots):
+        with self._condition:
+            for database, values in advance_slots.items():
+                self._scheduled[database].update(values)
+            ret = (self._failed, self._copy_slots)
+            self._copy_slots = []
+            self._failed = False
+            self._condition.notify()
+
+        return ret
+
+    def on_promote(self):
+        with self._condition:
+            self._scheduled.clear()
+            self._failed = False
+            self._copy_slots = []
+
+
 class SlotsHandler(object):
 
     def __init__(self, postgresql):
         self._postgresql = postgresql
+        self._advance = None
         self._replication_slots = {}  # already existing replication slots
         self._unready_logical_slots = {}
         self.schedule()
@@ -143,7 +228,7 @@ def _ensure_physical_slots(self, slots):
                 self._schedule_load_slots = True
 
     @contextmanager
-    def _get_local_connection_cursor(self, **kwargs):
+    def get_local_connection_cursor(self, **kwargs):
         conn_kwargs = self._postgresql.config.local_connect_kwargs
         conn_kwargs.update(kwargs)
         with get_connection_cursor(**conn_kwargs) as cur:
@@ -162,7 +247,7 @@ def _ensure_logical_slots_primary(self, slots):
 
         # Create new logical slots
         for database, values in logical_slots.items():
-            with self._get_local_connection_cursor(dbname=database) as cur:
+            with self.get_local_connection_cursor(dbname=database) as cur:
                 for name, value in values.items():
                     try:
                         cur.execute("SELECT pg_catalog.pg_create_logical_replication_slot(%s, %s)" +
@@ -175,6 +260,11 @@ def _ensure_logical_slots_primary(self, slots):
                         slots.pop(name)
                     self._schedule_load_slots = True
 
+    def schedule_advance_slots(self, slots):
+        if not self._advance:
+            self._advance = SlotsAdvanceThread(self)
+        return self._advance.schedule(slots)
+
     def _ensure_logical_slots_replica(self, cluster, slots):
         advance_slots = defaultdict(dict)  # Group logical slots to be advanced by database name
         create_slots = []  # And collect logical slots to be created on the replica
@@ -186,25 +276,16 @@ def _ensure_logical_slots_replica(self, cluster, slots):
                     if name in cluster.slots:
                         try:  # Skip slots that doesn't need to be advanced
                             if value['confirmed_flush_lsn'] < int(cluster.slots[name]):
-                                advance_slots[value['database']][name] = value
+                                advance_slots[value['database']][name] = int(cluster.slots[name])
                         except Exception as e:
                             logger.error('Failed to parse "%s": %r', cluster.slots[name], e)
                 elif name in cluster.slots:  # We want to copy only slots with feedback in a DCS
                     create_slots.append(name)
 
-        # Advance logical slots
-        for database, values in advance_slots.items():
-            with self._get_local_connection_cursor(dbname=database, options='-c statement_timeout=0') as cur:
-                for name, value in values.items():
-                    try:
-                        cur.execute("SELECT pg_catalog.pg_replication_slot_advance(%s, %s)",
-                                    (name, format_lsn(int(cluster.slots[name]))))
-                    except Exception as e:
-                        logger.error("Failed to advance logical replication slot '%s': %r", name, e)
-                        if isinstance(e, OperationalError) and e.diag.sqlstate == '58P01':  # WAL file is gone
-                            create_slots.append(name)
-                        self._schedule_load_slots = True
-        return create_slots
+        error, copy_slots = self.schedule_advance_slots(advance_slots)
+        if error:
+            self._schedule_load_slots = True
+        return create_slots + copy_slots
 
     def sync_replication_slots(self, cluster, nofailover, replicatefrom=None, paused=False):
         ret = None
@@ -331,6 +412,9 @@ def schedule(self, value=None):
         self._schedule_load_slots = self._force_readiness_check = value
 
     def on_promote(self):
+        if self._advance:
+            self._advance.on_promote()
+
         if self._unready_logical_slots:
             logger.warning('Logical replication slots that might be unsafe to use after promote: %s',
                            set(self._unready_logical_slots))


Test patch:
diff --git a/tests/test_slots.py b/tests/test_slots.py
--- a/tests/test_slots.py
+++ b/tests/test_slots.py
@@ -4,17 +4,19 @@
 
 
 from mock import Mock, PropertyMock, patch
+from threading import Thread
 
 from patroni import psycopg
 from patroni.dcs import Cluster, ClusterConfig, Member
 from patroni.postgresql import Postgresql
-from patroni.postgresql.slots import SlotsHandler, fsync_dir
+from patroni.postgresql.slots import SlotsAdvanceThread, SlotsHandler, fsync_dir
 
 from . import BaseTestPostgresql, psycopg_connect, MockCursor
 
 
 @patch('subprocess.call', Mock(return_value=0))
 @patch('patroni.psycopg.connect', psycopg_connect)
+@patch.object(Thread, 'start', Mock())
 @patch.object(Postgresql, 'is_running', Mock(return_value=True))
 class TestSlotsHandler(BaseTestPostgresql):
 
@@ -91,6 +93,7 @@ def test__ensure_logical_slots_replica(self):
             self.assertEqual(self.s.sync_replication_slots(self.cluster, False), [])
         self.s._schedule_load_slots = False
         with patch.object(MockCursor, 'execute', Mock(side_effect=psycopg.OperationalError)),\
+                patch.object(SlotsAdvanceThread, 'schedule', Mock(return_value=(True, ['ls']))),\
                 patch.object(psycopg.OperationalError, 'diag') as mock_diag:
             type(mock_diag).sqlstate = PropertyMock(return_value='58P01')
             self.assertEqual(self.s.sync_replication_slots(self.cluster, False), ['ls'])
@@ -123,6 +126,7 @@ def test_check_logical_slots_readiness(self):
     @patch.object(Postgresql, 'start', Mock(return_value=True))
     @patch.object(Postgresql, 'is_leader', Mock(return_value=False))
     def test_on_promote(self):
+        self.s.schedule_advance_slots({'foo': {'bar': 100}})
         self.s.copy_logical_slots(self.cluster, ['ls'])
         self.s.on_promote()
 
@@ -132,3 +136,18 @@ def test_on_promote(self):
     @patch('os.fsync', Mock(side_effect=OSError))
     def test_fsync_dir(self):
         self.assertRaises(OSError, fsync_dir, 'foo')
+
+    def test_slots_advance_thread(self):
+        with patch.object(MockCursor, 'execute', Mock(side_effect=psycopg.OperationalError)),\
+                patch.object(psycopg.OperationalError, 'diag') as mock_diag:
+            type(mock_diag).sqlstate = PropertyMock(return_value='58P01')
+            self.s.schedule_advance_slots({'foo': {'bar': 100}})
+            self.s._advance.sync_slots()
+
+        with patch.object(SlotsAdvanceThread, 'sync_slots', Mock(side_effect=Exception)):
+            self.s._advance._condition.wait = Mock()
+            self.assertRaises(Exception, self.s._advance.run)
+
+        with patch.object(SlotsHandler, 'get_local_connection_cursor', Mock(side_effect=Exception)):
+            self.s.schedule_advance_slots({'foo': {'bar': 100}})
+            self.s._advance.sync_slots()