Repository: patroni/patroni

Issue:
Patroni still not sync new data from DCS after the reconnection of the whole AZ
### What happened?

### Summary
We operate multiple PostgreSQL clusters with Patroni, deployed across three availability zones (AZs), all using the same etcd cluster as the DCS. During regular failure tests involving the disconnection of one AZ (simulating a network partition), we encountered an issue where Patroni nodes in the previously disconnected AZ do not refresh their local DCS cache after reconnection. As a result, they retain outdated information about the cluster leader and do not synchronize the current state from the DCS.

### Environment
Multiple PostgreSQL clusters running in the same three AZs
Patroni and etcd used for high availability
One etcd node and one PostgreSQL node in each AZ (3x PG, 3x etcd)
All clusters share the same etcd backend

### Observed Behavior
After reconnection, some Patroni nodes in the previously disconnected AZ retain outdated information about the cluster leader.
These nodes do not refresh their DCS cache with live data from etcd.
As a result, they may attempt to operate with stale state, which could lead to split-brain or incorrect cluster behavior.
In our observations, Patroni did not automatically drop or refresh the local DCS cache after reconnecting to etcd.
Previous problem in issue https://github.com/patroni/patroni/issues/3359

### How can we reproduce it (as minimally and precisely as possible)?

1. Disconnect one AZ from the network (simulate AZ failure).
- The disconnected AZ contains one etcd node and one PostgreSQL node.
- The PostgreSQL leader is located in the AZ being disconnected.
2. Wait until a new leader is elected in one of the remaining AZs.
3. Reconnect the AZ to the network.

### What did you expect to happen?

After reconnection, Patroni should discard the outdated local DCS cache and reload the current state from the DCS to avoid stale data and ensure proper cluster state synchronization.

### Patroni/PostgreSQL/DCS version

-   Patroni version: 4.0.5 (with patches 3318, 3338, 3341, 3373)
-   PostgreSQL version: 15,16,17
-   DCS (and its version): etcd-server 3.5.21


### Patroni configuration file

```yaml
---
bootstrap:
  dcs:
    loop_wait: 10
    maximum_lag_on_failover: 0
    postgresql:
      parameters:
        archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn
          '%p'
        archive_mode: on
        max_connections: '100'
        max_locks_per_transaction: '64'
        max_prepared_transactions: 0
        max_replication_slots: '5'
        max_wal_senders: '20'
        max_worker_processes: 8
        ssl_cert_file: CERT
        ssl_key_file: KEY
        synchronous_commit: on
        track_commit_timestamp: off
        wal_keep_segments: '100'
        wal_keep_size: '1600'
        wal_level: hot_standby
        wal_log_hints: on
        wal_sender_timeout: 5min
      recovery_conf:
        recovery_target_timeline: latest
        restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
          %f '%p' || true
      use_pg_rewind: true
    retry_timeout: 10
    ttl: 30
  method: initdb
  pgbackrest:
    command: pgbackrest --stanza=CLUSTER restore --log-level-console=warn
    keep_existing_recovery_conf: true
    no_params: true
    recovery_conf:
      recovery_target_timeline: latest
      restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
        %f '%p'
  initdb:
    - encoding: UTF8
    - data-checksums
  pg_hba:
    - host replication replicator 127.0.0.1/32 scram-sha-256
    - host replication replicator ::1/128 scram-sha-256
    - host replication replicator PG1.AZ1 scram-sha-256
    - host replication replicator PG1.AZ2 scram-sha-256
    - host replication replicator PG1.AZ3 scram-sha-256
    - host all all 0.0.0.0/0 scram-sha-256
    - host all all ::0/0 scram-sha-256
  pg_ident:
    - some postgres postgres
  post_bootstrap: /tmp/post_bootstrap.sh
etcd3:
  cacert: ETCDCA
  cert: ETCDCERT
  hosts:
    - ETCD1.AZ1:2379
    - ETCD1.AZ2:2379
    - ETCD1.AZ3:2379
  key: ETCDKEY
  protocol: https
log:
  level: INFO
name: PG.AZ1
namespace: /CLUSTER/
postgresql:
  authentication:
    replication:
      password: PASS
      username: replicator
    superuser:
      password: PASS
      username: postgres
  basebackup:
    checkpoint: fast
    max-rate: 400M
  bin_dir: /usr/lib/postgresql/17/bin/
  callbacks:
    on_role_change: /usr/bin/flock --timeout 7200 /var/lock/LOCK.lock
      /usr/bin/pgbackrest_backup --stanza=CLUSTER --type=diff --patroni-port=8008
      --delay 10
  config_dir: /etc/postgresql/17/main
  connect_address: IP:5432
  create_replica_methods:
    - pgbackrest
    - basebackup
  data_dir: /var/lib/postgresql/17/main
  listen: '*:5432'
  parameters:
    archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn
      '%p'
    archive_mode: on
    max_connections: '100'
    max_locks_per_transaction: '64'
    max_prepared_transactions: 0
    max_replication_slots: '5'
    max_wal_senders: '20'
    max_worker_processes: 8
    ssl_cert_file: CERT
    ssl_key_file: KEY
    synchronous_commit: on
    track_commit_timestamp: off
    wal_keep_segments: '100'
    wal_keep_size: '1600'
    wal_level: hot_standby
    wal_log_hints: on
    wal_sender_timeout: 5min
  pg_hba:
    - host replication replicator 127.0.0.1/32 scram-sha-256
    - host replication replicator ::1/128 scram-sha-256
    - host replication replicator PG1.AZ1 scram-sha-256
    - host replication replicator PG1.AZ2 scram-sha-256
    - host replication replicator PG1.AZ3 scram-sha-256
    - host all all 0.0.0.0/0 scram-sha-256
    - host all all ::0/0 scram-sha-256
  pg_ident:
    - some postgres postgres
  pgbackrest:
    command: pgbackrest --stanza CLUSTER restore --log-level-console=warn --type=none
      --delta
    keep_data: true
    no_params: true
  pgpass: /var/lib/postgresql/.pgpass
  recovery_conf:
    recovery_target_timeline: latest
    restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
      %f '%p' || true
  remove_data_directory_on_diverged_timelines: false
  remove_data_directory_on_rewind_failure: false
  use_pg_rewind: true
  use_slots: true
  use_unix_socket: true
restapi:
  authentication:
    password: PASS
    username: CLUSTER
  certfile:
  connect_address: IP:8008
  keyfile:
  listen: :8008
scope: main17
tags:
  clonefrom: false
  nofailover: false
  noloadbalance: false
  nosync: false
  replicatefrom: ''
```

### patronictl show-config

```yaml
check_timeline: true
failsafe_mode: true
loop_wait: 10
maximum_lag_on_failover: 0
postgresql:
  parameters:
    archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn '%p'
    archive_mode: 'on'
    max_connections: '100'
    max_locks_per_transaction: '64'
    max_prepared_transactions: 0
    max_replication_slots: '5'
    max_wal_senders: '20'
    max_worker_processes: 8
    ssl_cert_file: CERT
    ssl_key_file: KEY
    synchronous_commit: 'on'
    track_commit_timestamp: 'off'
    wal_keep_segments: '100'
    wal_keep_size: '1600'
    wal_level: hot_standby
    wal_log_hints: 'on'
    wal_sender_timeout: 5min
  recovery_conf:
    recovery_target_timeline: latest
    restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn %f '%p' || true
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
synchronous_mode: true
synchronous_mode_strict: false
synchronous_node_count: 2
ttl: 30
```

### Patroni log files

```shell
In comment
```

### PostgreSQL log files

```shell
No logs needed
```

### Have you tried to use GitHub issue search?

- [x] Yes

### Anything else we need to know?

Patroni logs are [here](https://drive.marek-jan.cz/public.php/dav/files/8GKa7ccken5CKqk/patroni_logs_2025_07_01.tar.gz)


Discussion:
> Patroni version: 4.0.5 (with and without https://github.com/patroni/patroni/pull/3373)

Latest version is 4.0.6.
From attached logs I can't understand anything because all offsets are off (even if I apply two PRs).
That is, I don't really know what you run.

Besides that, it is very suspicious that there is not a single `Connected to Etcd node with term %d. Old known term %d. Switching to another node.` message in attached logs.
We used debian package 4.0.5 with applied patch [3318](https://github.com/patroni/patroni/pull/3318), [3338](https://github.com/patroni/patroni/pull/3338), [3341](https://github.com/patroni/patroni/pull/3341), [3373](https://github.com/patroni/patroni/pull/3373).

Yes, this is suspicious. We were expecting it and didn't find it in the log.
I would suggest you to run tests with DEBUG logs enabled and the following patch.
```diff
diff --git a/patroni/dcs/etcd.py b/patroni/dcs/etcd.py
index 3cd84ba4..e9ef34e6 100644
--- a/patroni/dcs/etcd.py
+++ b/patroni/dcs/etcd.py
@@ -285,6 +285,7 @@ class AbstractEtcdClientWithFailover(abc.ABC, etcd.Client, StaleEtcdNodeGuard):
                 if some_request_failed:
                     self.set_base_uri(base_uri)
                     self._refresh_machines_cache()
+                logger.debug('%s %s%s (%s) => %s', method, base_uri, path, kwargs, response.data)
                 return self._handle_server_response(response)
             except (HTTPError, HTTPException, socket.error, socket.timeout, StaleEtcdNode) as e:
                 self.http.clear()
```
We have prepared a new version including the patch. The new disconnection test on our side is scheduled for July 29.
We will contact you after the test has been completed.
Hello everyone,

I think our problem is related to this issue, so I post my message here instead of a new issue.
We have a 5 node etcd cluster for currently 68 Patroni nodes (34 clusters).
Today the etcd leader changed due to a slow sync duration.
66 of the Patroni nodes handled this very well, but 2 nodes weren't able to reconnect.
On these nodes it is still possible to run ```patronictl``` commands like ```patronictl -c /etc/patroni/patroni.yaml list```, so a general communication is still possible (in the ```list``` output only the partner node is listed).
After a restart of Patroni the connection was established and everything worked again, but since there were primaries a failover happened what we try to avoid.
Failafe mode is active in our setup.

Patroni: 4.0.6
PostgreSQL: 16 on these nodes, but also 14 and 15 in general
etcdctl version: 3.4.30
API version: 3.4

Patroni log of one of the nodes
```
[...]
[2025-07-22 05:25:14,683][INFO] no action. I am (it-dads-pg-a4.domain.de), the leader with the lock
[2025-07-22 05:25:24,596][WARNING] Connected to Etcd node with term 2318. Old known term 2319. Switching to another node.
[2025-07-22 05:25:24,596][DEBUG] Error on socket.shutdown: TypeError('WrappedSocket.shutdown() takes 1 positional argument but 2 were given')
[2025-07-22 05:25:25,279][DEBUG] Received message: {'result': {'header': {'cluster_id': '100169269788787505', 'member_id': '7796990930729603261', 'revision': '66886141', 'raft_term': '2319'}, 'events': [{'kv': {'key': 'L3Nwd2Vpdi9wb3N0Z3Jlcy9tZW1iZXJzL2l0LWRhZHMtcGctbzQuZGVzeS5kZQ==', 'create_revision': '61042745', 'mod_revision': '66886141', 'version': '40153', 'value': 'eyJjb25uX3VybCI6InBvc3RncmVzOi8vaXQtZGFkcy1wZy1vNC5kZXN5LmRlOjU0MzIvcG9zdGdyZXMiLCJhcGlfdXJsIjoiaHR0cDovL2l0LWRhZHMtcGctbzQuZGVzeS5kZTo4MDA4L3BhdHJvbmkiLCJzdGF0ZSI6InJ1bm5pbmciLCJyb2xlIjoicmVwbGljYSIsInZlcnNpb24iOiI0LjAuNiIsInhsb2dfbG9jYXRpb24iOjE4NTg2MzAzOTE5MiwicmVwbGljYXRpb25fc3RhdGUiOiJzdHJlYW1pbmciLCJ0aW1lbGluZSI6Nn0=', 'lease': '1206285270182687511'}}]}}
[2025-07-22 05:25:34,596][ERROR] get_cluster
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/patroni/dcs/etcd3.py", line 840, in _load_cluster
    cluster = loader(path)
              ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/patroni/dcs/etcd3.py", line 816, in _postgresql_cluster_loader
    for node in self._client.get_cluster(path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/patroni/dcs/etcd3.py", line 638, in get_cluster
    self._wait_cache(self.read_timeout)
  File "/usr/local/lib/python3.12/dist-packages/patroni/dcs/etcd3.py", line 632, in _wait_cache
    raise RetryFailedError('Exceeded retry deadline')
patroni.utils.RetryFailedError: Exceeded retry deadline
[2025-07-22 05:25:34,602][ERROR] Error communicating with DCS
[2025-07-22 05:25:34,605][DEBUG] Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=0, status=None)
[2025-07-22 05:25:34,605][DEBUG] Starting new HTTP connection (2): it-dads-pg-o4.domain.de:8008
[2025-07-22 05:25:34,672][DEBUG] http://it-dads-pg-o4.domain.de:8008 "POST /failsafe HTTP/1.1" 200 None
[2025-07-22 05:25:34,672][INFO] Got response from it-dads-pg-o4.domain.de http://it-dads-pg-o4.domain.de:8008/failsafe: Accepted
[2025-07-22 05:25:34,674][INFO] continue to run as a leader because failsafe mode is enabled and all members are accessible
[2025-07-22 05:25:34,675][WARNING] Loop time exceeded, rescheduling immediately.
[2025-07-22 05:25:35,286][DEBUG] Received message: {'result': {'header': {'cluster_id': '100169269788787505', 'member_id': '7796990930729603261', 'revision': '66886212', 'raft_term': '2319'}, 'events': [{'kv': {'key': 'L3Nwd2Vpdi9wb3N0Z3Jlcy9tZW1iZXJzL2l0LWRhZHMtcGctbzQuZGVzeS5kZQ==', 'create_revision': '61042745', 'mod_revision': '66886212', 'version': '40154', 'value': 'eyJjb25uX3VybCI6InBvc3RncmVzOi8vaXQtZGFkcy1wZy1vNC5kZXN5LmRlOjU0MzIvcG9zdGdyZXMiLCJhcGlfdXJsIjoiaHR0cDovL2l0LWRhZHMtcGctbzQuZGVzeS5kZTo4MDA4L3BhdHJvbmkiLCJzdGF0ZSI6InJ1bm5pbmciLCJyb2xlIjoicmVwbGljYSIsInZlcnNpb24iOiI0LjAuNiIsInhsb2dfbG9jYXRpb24iOjE4NTg2MzAzOTI0OCwicmVwbGljYXRpb25fc3RhdGUiOiJzdHJlYW1pbmciLCJ0aW1lbGluZSI6Nn0=', 'lease': '1206285270182687511'}}]}}
[2025-07-22 05:25:44,675][ERROR] Exceeded retry deadline
[2025-07-22 05:25:44,676][ERROR] Error communicating with DCS
[2025-07-22 05:25:44,678][DEBUG] Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=0, status=None)
[2025-07-22 05:25:44,678][DEBUG] Resetting dropped connection: it-dads-pg-o4.domain.de
[2025-07-22 05:25:44,680][DEBUG] http://it-dads-pg-o4.domain.de:8008 "POST /failsafe HTTP/1.1" 200 None
[2025-07-22 05:25:44,680][INFO] Got response from it-dads-pg-o4.domain.de http://it-dads-pg-o4.domain.de:8008/failsafe: Accepted
[2025-07-22 05:25:44,681][INFO] continue to run as a leader because failsafe mode is enabled and all members are accessible
[2025-07-22 05:25:44,683][WARNING] Loop time exceeded, rescheduling immediately.
[2025-07-22 05:25:44,754][DEBUG] Received message: {'result': {'header': {'cluster_id': '100169269788787505', 'member_id': '7796990930729603261', 'revision': '66886265', 'raft_term': '2319'}, 'events': [{'type': 'DELETE', 'kv': {'key': 'L3Nwd2Vpdi9wb3N0Z3Jlcy9sZWFkZXI=', 'mod_revision': '66886265'}}, {'type': 'DELETE', 'kv': {'key': 'L3Nwd2Vpdi9wb3N0Z3Jlcy9tZW1iZXJzL2l0LWRhZHMtcGctYTQuZGVzeS5kZQ==', 'mod_revision': '66886265'}}]}}
[2025-07-22 05:25:44,754][DEBUG] L3Nwd2Vpdi9wb3N0Z3Jlcy9sZWFkZXI= changed from aXQtZGFkcy1wZy1hNC5kZXN5LmRl to None
[2025-07-22 05:25:44,852][DEBUG] Received message: {'result': {'header': {'cluster_id': '100169269788787505', 'member_id': '7796990930729603261', 'revision': '66886268', 'raft_term': '2319'}, 'events': [{'kv': {'key': 'L3Nwd2Vpdi9wb3N0Z3Jlcy9tZW1iZXJzL2l0LWRhZHMtcGctbzQuZGVzeS5kZQ==', 'create_revision': '61042745', 'mod_revision': '66886268', 'version': '40155', 'value': 'eyJjb25uX3VybCI6InBvc3RncmVzOi8vaXQtZGFkcy1wZy1vNC5kZXN5LmRlOjU0MzIvcG9zdGdyZXMiLCJhcGlfdXJsIjoiaHR0cDovL2l0LWRhZHMtcGctbzQuZGVzeS5kZTo4MDA4L3BhdHJvbmkiLCJzdGF0ZSI6InJ1bm5pbmciLCJyb2xlIjoicmVwbGljYSIsInZlcnNpb24iOiI0LjAuNiIsInhsb2dfbG9jYXRpb24iOjE4NTg2MzEyNjUwNCwicmVwbGljYXRpb25fc3RhdGUiOiJzdHJlYW1pbmciLCJ0aW1lbGluZSI6Nn0=', 'lease': '1206285270182687511'}}]}}
[2025-07-22 05:25:54,683][ERROR] Exceeded retry deadline
[2025-07-22 05:25:54,684][ERROR] Error communicating with DCS
[2025-07-22 05:25:54,685][DEBUG] Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=0, status=None)
[2025-07-22 05:25:54,685][DEBUG] Resetting dropped connection: it-dads-pg-o4.domain.de
[2025-07-22 05:25:54,688][DEBUG] http://it-dads-pg-o4.domain.de:8008 "POST /failsafe HTTP/1.1" 200 None
[2025-07-22 05:25:54,688][INFO] Got response from it-dads-pg-o4.domain.de http://it-dads-pg-o4.domain.de:8008/failsafe: Accepted
[2025-07-22 05:25:54,689][INFO] continue to run as a leader because failsafe mode is enabled and all members are accessible
[2025-07-22 05:25:54,690][WARNING] Loop time exceeded, rescheduling immediately.
[2025-07-22 05:25:54,848][DEBUG] Received message: {'result': {'header': {'cluster_id': '100169269788787505', 'member_id': '7796990930729603261', 'revision': '66886325', 'raft_term': '2319'}, 'events': [{'kv': {'key': 'L3Nwd2Vpdi9wb3N0Z3Jlcy9tZW1iZXJzL2l0LWRhZHMtcGctbzQuZGVzeS5kZQ==', 'create_revision': '61042745', 'mod_revision': '66886325', 'version': '40156', 'value': 'eyJjb25uX3VybCI6InBvc3RncmVzOi8vaXQtZGFkcy1wZy1vNC5kZXN5LmRlOjU0MzIvcG9zdGdyZXMiLCJhcGlfdXJsIjoiaHR0cDovL2l0LWRhZHMtcGctbzQuZGVzeS5kZTo4MDA4L3BhdHJvbmkiLCJzdGF0ZSI6InJ1bm5pbmciLCJyb2xlIjoicmVwbGljYSIsInZlcnNpb24iOiI0LjAuNiIsInhsb2dfbG9jYXRpb24iOjE4NTg2MzEyNjU2MCwicmVwbGljYXRpb25fc3RhdGUiOiJzdHJlYW1pbmciLCJ0aW1lbGluZSI6Nn0=', 'lease': '1206285270182687511'}}]}}
[2025-07-22 05:26:04,690][ERROR] Exceeded retry deadline
[2025-07-22 05:26:04,691][ERROR] Error communicating with DCS
[2025-07-22 05:26:04,693][DEBUG] Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=0, status=None)
[2025-07-22 05:26:04,693][DEBUG] Resetting dropped connection: it-dads-pg-o4.domain.de
[2025-07-22 05:26:04,695][DEBUG] http://it-dads-pg-o4.domain.de:8008 "POST /failsafe HTTP/1.1" 200 None
[2025-07-22 05:26:04,695][INFO] Got response from it-dads-pg-o4.domain.de http://it-dads-pg-o4.domain.de:8008/failsafe: Accepted
[2025-07-22 05:26:04,696][INFO] continue to run as a leader because failsafe mode is enabled and all members are accessible
[2025-07-22 05:26:04,698][WARNING] Loop time exceeded, rescheduling immediately.
[...]
```

This will go on until the Patroni restart.

Patroni configuration
```
scope: postgres
namespace: /spweiv/
name: it-dads-pg-a4.domain.de

log:
  # https://docs.python.org/3.6/library/logging.html#levels
  level: DEBUG
  dir: /var/log/patroni
  file_size: 104857600
  format: '[%(asctime)s][%(levelname)s] %(message)s'

restapi:
    listen: "*:8008"
    connect_address: it-dads-pg-a4.domain.de:8008
    authentication:
        username: REDACTED
        password: REDACTED
etcd3:
    protocol: https
    hosts:
    - it-dads-pg-etcd-1.domain.de:2379
    - it-dads-pg-etcd-2.domain.de:2379
    - it-dads-pg-etcd-3.domain.de:2379
    - it-dads-pg-etcd-4.domain.de:2379
    - it-dads-pg-etcd-5.domain.de:2379
    cert: /etc/patroni/server.pem
    key: /etc/patroni/server.key
    cacert: /etc/ssl/certs/ca-certificates.crt
    username: REDACTED
    password: REDACTED

bootstrap:
    dcs:
        maximum_lag_on_failover: 1048576
        failsafe_mode: true

        postgresql:
            parameters:
                max_connections: 105
            recovery_conf:
                recovery_target_timeline: latest
                restore_command: /usr/bin/pgbackrest --stanza=spweiv archive-get %f %p
    initdb:
    - encoding: UTF8
    - locale: C.UTF-8

    post_bootstrap: /var/lib/postgresql/post_bootstrap.sh spweiv

postgresql:
    listen: it-dads-pg-a4.domain.de:5432
    connect_address: it-dads-pg-a4.domain.de:5432
    data_dir: /data/patroni/postgresql
    bin_dir: /usr/lib/postgresql/16/bin
    pgpass: /var/lib/postgresql/init_pgpass
    pg_hba:
      REDACTED
    authentication:
        replication:
            username: REDACTED
            password: REDACTED
        superuser:
            username: REDACTED
            password: REDACTED
    parameters:
        # logging settings
        logging_collector: true
        log_directory: /var/log/patroni
        log_destination: "jsonlog"
        log_filename: postgresql
        log_min_messages: NOTICE
        log_min_error_statement: WARNING
        log_min_duration_statement: 600000
        log_checkpoints: true
        log_temp_files: 10MB
        client_min_messages: NOTICE

        # Performance settings
        effective_cache_size: 3GB
        hash_mem_multiplier: 2
        maintenance_work_mem: 256MB
        min_dynamic_shared_memory: 0
        wal_buffers: 16MB
        work_mem: 6MB
        effective_io_concurrency: 200
        shared_buffers: 1GB
        tcp_keepalives_count: 10
        tcp_keepalives_idle: 180
        tcp_keepalives_interval: 60
        random_page_cost: 1.5
        maintenance_io_concurrency: 200
        default_toast_compression: lz4
        wal_compression: false

        # archive settings
        archive_mode: on
        archive_timeout: 3600
        archive_command: "pgbackrest --stanza=spweiv archive-push %p"

        # General settings
        shared_preload_libraries: pgaudit,pg_stat_statements
        pgaudit.log: ROLE,DDL
        track_io_timing: true

        # SSL settings
        ssl: true
        ssl_cert_file: /etc/patroni/server.pem
        ssl_key_file: /etc/patroni/server.key
        ssl_ca_file: /etc/ssl/certs/ca-certificates.crt
    create_replica_methods:
        - pgbackrest
        - basebackup
    pgbackrest:
        command: /usr/bin/pgbackrest --stanza=spweiv --delta restore
        keep_data: True
        no_params: True
    basebackup:
        max-rate: 100M

tags:
    nofailover: false
    noloadbalance: false
    clonefrom: false
    nosync: false
```

patronictl show-config
```
failsafe_mode: true
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    max_connections: 105
  recovery_conf:
    recovery_target_timeline: latest
    restore_command: /usr/bin/pgbackrest --stanza=spweiv archive-get %f %p
```

If you think this is a seperate issue please let me know.

Best regards,
Patrick

Hi @CyberDem0n,  
We had to postpone the AZ disconnection test to August 12th. We will announce the results afterwards.  

Thank you for your help in identifying and resolving the issue.

Hi @CyberDem0n,  
we have completed the test.
Here are patroni [logs](https://drive.marek-jan.cz/public.php/dav/files/8GKa7ccken5CKqk/patroni_logs_2025_08_12.tar.gz)

AZ2 was disconnected from 10:00:00 to 11:00:00.
@marekj11 thanks!
It was very useful.
I think now I understand what is going on.
Etcd leader wasn't running in AZ2 and therefore raft_term didn't increase.
However, in Patroni logs I found the following:
> 2025-08-12 11:00:27,022 DEBUG: Received message: {'result': {'header': {'cluster_id': '10669693111759494504', 'member_id': '17043355627402701013', 'raft_term': '125728'}, **'canceled': True, 'compact_revision': '3974519060'**}}

It means that this watch is no longer valid and we need to close connection.
Strange that Etcd isn't closing it on its side, but anyway, now I know how to handle it.



Patch:
diff --git a/patroni/dcs/etcd3.py b/patroni/dcs/etcd3.py
index f2cf6f848..0eab956b8 100644
--- a/patroni/dcs/etcd3.py
+++ b/patroni/dcs/etcd3.py
@@ -21,7 +21,7 @@
 from ..collections import EMPTY_DICT
 from ..exceptions import DCSError, PatroniException
 from ..postgresql.mpp import AbstractMPP
-from ..utils import deep_compare, enable_keepalive, iter_response_objects, RetryFailedError, USER_AGENT
+from ..utils import deep_compare, enable_keepalive, iter_response_objects, parse_bool, RetryFailedError, USER_AGENT
 from . import catch_return_false_exception, Cluster, ClusterConfig, \
     Failover, Leader, Member, Status, SyncState, TimelineHistory
 from .etcd import AbstractEtcd, AbstractEtcdClientWithFailover, catch_etcd_errors, \
@@ -66,6 +66,10 @@ class Etcd3Exception(etcd.EtcdException):
     pass
 
 
+class Etcd3WatchCanceled(Etcd3Exception):
+    pass
+
+
 class Etcd3ClientError(Etcd3Exception):
 
     def __init__(self, code: Optional[int] = None, error: Optional[str] = None, status: Optional[int] = None) -> None:
@@ -356,7 +360,6 @@ def handle_auth_errors(self: 'Etcd3Client', func: Callable[..., Any], *args: Any
                 exc = e
             self._reauthenticate = True
             if retry:
-                logger.error('retry = %s', retry)
                 retry.ensure_deadline(0.5, exc)
             elif reauthenticated:
                 raise exc
@@ -508,6 +511,8 @@ def _process_message(self, message: Dict[str, Any]) -> None:
         if 'error' in message:
             raise _raise_for_data(message)
         result = message.get('result', EMPTY_DICT)
+        if parse_bool(result.get('canceled')):
+            raise Etcd3WatchCanceled('Watch canceled')
         header = result.get('header', EMPTY_DICT)
         self._check_cluster_raft_term(header.get('cluster_id'), header.get('raft_term'))
         events: List[Dict[str, Any]] = result.get('events', [])
@@ -555,8 +560,10 @@ def _build_cache(self) -> None:
 
         try:
             self._do_watch(result['header']['revision'])
+        except Etcd3WatchCanceled:
+            logger.info('Watch request canceled')
         except Exception as e:
-            # Following exceptions are expected on Windows because the /watch request  is done with `read_timeout`
+            # Following exceptions are expected on Windows because the /watch request is done with `read_timeout`
             if not (os.name == 'nt' and isinstance(e, (ReadTimeoutError, ProtocolError))):
                 logger.error('watchprefix failed: %r', e)
         finally:


Test patch:
diff --git a/tests/test_etcd3.py b/tests/test_etcd3.py
index ae52b7420..0edbd0286 100644
--- a/tests/test_etcd3.py
+++ b/tests/test_etcd3.py
@@ -96,8 +96,12 @@ class TestKVCache(BaseTestEtcd3):
 
     @patch.object(urllib3.PoolManager, 'urlopen', mock_urlopen)
     @patch.object(Etcd3Client, 'watchprefix', Mock(return_value=urllib3.response.HTTPResponse()))
+    @patch.object(urllib3.response.HTTPResponse, 'read_chunked',
+                  Mock(return_value=[b'{"result":{"canceled":true}}']))
     def test__build_cache(self):
-        self.kv_cache._build_cache()
+        with patch('patroni.dcs.etcd3.logger') as mock_logger:
+            self.kv_cache._build_cache()
+            mock_logger.info.assert_called_once_with('Watch request canceled')
 
     def test__do_watch(self):
         self.client.watchprefix = Mock(return_value=False)