Repository: zalando/patroni

Issue:
Should active unrecognized slots throw an error each HA cycle?
**Describe the bug**
If you create an (external) slot and use it (e.g. with `pg_recvlogical`), Patroni runs fine for now.

Once you put the cluster in pause mode, restart Patroni on the leader, and unpause the cluster, the error `ERROR: Failed to drop replication slot 'slot_recvlogical` is logged for each HA cycle.

First off, it's a bit unclear why this doesn't happen immediately, is that a bug/consistency issue in itself?

Also, this is an active slot that probably has some purpose; I understand that from Patroni's logic this slot isn't recognized, but logging this at `ERROR` level each HA cycle makes it difficult to maintain a strict "no errors in log" policy.

On the other hand, if you add those slots to `ignore_slots`, Patroni won't drop them once they are inactive (after a switchover or a demote) and they hang around with all the problems that makes.

So I think it might be better to log this once at `ERROR` (or `WARNING`?) level and then only at `DEBUG` at most? Maybe I am missing something which makes it important to have this logged as error each time?

One alternative would be to register those slots properly (and let Patroni do logical slot failover), but if those slots are manually maintained externally that might not be an option.

**To Reproduce**

1. on follower (or elsewhere), run `pg_recvlogical -h pg3  -S slot_recvlogical -f archive_logical -U $REPL_USER -d postgres --create-slot --start`
2. Verify that no errors are logged waiting for one HA cycle
3. put patroni in maintenance mode with `patronictl pause`
4. restart patroni on leader
5. put patroni back in production mode with `patronictl resume`
6. now I see the `ERROR`
7. Add `ignore_slots` via `patronictl edit-config`:
```
ignore_slots:
- database: postgres
  name: pg1_recvlogical
  type: logical
```
8. Error is gone
9. Do a switchover with `patronictl switchover`
10. Verify that slot is still around and inactive via `SELECT * FROM pg_replication_slots;` on old leader

**Environment**
- Patroni version: 2.1.5
- PostgreSQL version: n/a
- DCS (and its version): n/a


Discussion:
If the slot is active there is not danger that it will result in pg_wal growth. The error/warning for active slots isn't reported just not to spam logs and to avoid some other potential problems: https://github.com/zalando/patroni/pull/266

The other thing is that Patroni isn't reading `pg_replication_slots` every cycle (unless there are permanent logical slots defined). Therefore it is possible that the newly created slot will not be noticed until pause/resume or until the new node joins the cluster.

Ignore slots is a different story. I don't know if it should be taken into account on replicas.
Right, I see that it checks for `active`, but I think #819 then changed the logic so it returns false if not exactly one row is returned (which is the case of an active slot now), and then the error gets logged (which is what I reported above):

```
+    def drop_replication_slot(self, name):
+        cursor = self._query(('SELECT pg_drop_replication_slot(%s) WHERE EXISTS (SELECT 1 ' +
+                              'FROM pg_replication_slots WHERE slot_name = %s AND NOT active)'), name, name)
+        # In normal situation rowcount should be 1, otherwise either slot doesn't exists or it is still active
+        return cursor.rowcount == 1
+
```
> https://github.com/zalando/patroni/pull/819 then changed the logic so it returns false if not exactly one row is returned (which is the case of an active slot now), and then the error gets logged

Yeah. Forgot about it... Logging it only once could be an option, but it will require having yet another object keeping a state :(
How about doing a check first whether the slot is active, and only do the rowcount check afterwards if the first query does return some rows?
How would it help to show the error only once?
Well it won't if that is what we want. Personally, I'd be fine with demoting this to Warning (or Info or even Debug) if the slot is active
> I'd be fine with demoting this to Warning (or Info or even Debug) if the slot is active

Yeah, probably DEBUG should be good enough.

Resolving patch:
diff --git a/patroni/postgresql/slots.py b/patroni/postgresql/slots.py
--- a/patroni/postgresql/slots.py
+++ b/patroni/postgresql/slots.py
@@ -200,10 +200,18 @@ def drop_replication_slot(self, name):
     def _drop_incorrect_slots(self, cluster, slots, paused):
         # drop old replication slots which are not presented in desired slots
         for name in set(self._replication_slots) - set(slots):
-            if not paused and not self.ignore_replication_slot(cluster, name) and not self.drop_replication_slot(name):
-                logger.error("Failed to drop replication slot '%s'", name)
-                self._schedule_load_slots = True
-
+            if not paused and not self.ignore_replication_slot(cluster, name):
+                if not self.drop_replication_slot(name):
+                    self._schedule_load_slots = True
+                    # Check if slot is still active, this is not considered an error
+                    cursor = self._query(('SELECT 1 FROM pg_catalog.pg_replication_slots WHERE ' +
+                                          'slot_name = %s AND active'), name)
+                    if (cursor.rowcount == 1):
+                        logger.debug("Unable to drop unknown replication slot '%s', slot is still active", name)
+                    else:
+                        logger.error("Failed to drop replication slot '%s'", name)
+                else:
+                    logger.info("Dropped unknown replication slot '%s'", name)
         for name, value in slots.items():
             if name in self._replication_slots and not compare_slots(value, self._replication_slots[name]):
                 logger.info("Trying to drop replication slot '%s' because value is changing from %s to %s",


Test patch: