Repository: patroni/patroni

Issue:
Patroni not sync new data from DCS after the reconnection of the whole AZ
## What happened?

### Summary
We operate multiple PostgreSQL clusters with Patroni, deployed across three availability zones (AZs), all using the same etcd cluster as the DCS. During regular failure tests involving the disconnection of one AZ (simulating a network partition), we encountered an issue where Patroni nodes in the previously disconnected AZ do not refresh their local DCS cache after reconnection. As a result, they retain outdated information about the cluster leader and do not synchronize the current state from the DCS.

### Environment
- Multiple PostgreSQL clusters running in the same three AZs
- Patroni and etcd used for high availability
- One etcd node and one PostgreSQL node in each AZ (3x PG, 3x etcd)
- All clusters share the same etcd backend

### Observed Behavior
- After reconnection, some Patroni nodes in the previously disconnected AZ retain outdated information about the cluster leader.
- These nodes do not refresh their DCS cache with live data from etcd.
- As a result, they may attempt to operate with stale state, which could lead to split-brain or incorrect cluster behavior.
- In our observations, Patroni did not automatically drop or refresh the local DCS cache after reconnecting to etcd.

Previous problem in issue #3314 

### How can we reproduce it (as minimally and precisely as possible)?

1. Disconnect one AZ from the network (simulate AZ failure).
  - The disconnected AZ contains one etcd node and one PostgreSQL node.
  - The PostgreSQL leader is located in the AZ being disconnected.
2. Wait until a new leader is elected in one of the remaining AZs.
3. Reconnect the AZ to the network.

### What did you expect to happen?

After reconnection, Patroni should discard the outdated local DCS cache and reload the current state from the DCS to avoid stale data and ensure proper cluster state synchronization.

### Patroni/PostgreSQL/DCS version

-   Patroni version: Patroni version: 4.0.5 (with and without [patch](https://github.com/patroni/patroni/pull/3318))
-   PostgreSQL version: 15,16,17
-   DCS (and its version): etcd 3.4.35
-   PG and ETCD nodes on Debian Bookworm

### Patroni configuration file

```yaml
---
bootstrap:
  dcs:
    loop_wait: 10
    maximum_lag_on_failover: 0
    postgresql:
      parameters:
        archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn
          '%p'
        archive_mode: on
        max_connections: '100'
        max_locks_per_transaction: '64'
        max_prepared_transactions: 0
        max_replication_slots: '5'
        max_wal_senders: '20'
        max_worker_processes: 8
        ssl_cert_file: CERT
        ssl_key_file: KEY
        synchronous_commit: on
        track_commit_timestamp: off
        wal_keep_segments: '100'
        wal_keep_size: '1600'
        wal_level: hot_standby
        wal_log_hints: on
        wal_sender_timeout: 5min
      recovery_conf:
        recovery_target_timeline: latest
        restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
          %f '%p' || true
      use_pg_rewind: true
    retry_timeout: 10
    ttl: 30
  method: initdb
  pgbackrest:
    command: pgbackrest --stanza=CLUSTER restore --log-level-console=warn
    keep_existing_recovery_conf: true
    no_params: true
    recovery_conf:
      recovery_target_timeline: latest
      restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
        %f '%p'
  initdb:
    - encoding: UTF8
    - data-checksums
  pg_hba:
    - host replication replicator 127.0.0.1/32 scram-sha-256
    - host replication replicator ::1/128 scram-sha-256
    - host replication replicator PG1.AZ1 scram-sha-256
    - host replication replicator PG1.AZ2 scram-sha-256
    - host replication replicator PG1.AZ3 scram-sha-256
    - host all all 0.0.0.0/0 scram-sha-256
    - host all all ::0/0 scram-sha-256
  pg_ident:
    - some postgres postgres
  post_bootstrap: /tmp/post_bootstrap.sh
etcd3:
  cacert: ETCDCA
  cert: ETCDCERT
  hosts:
    - ETCD1.AZ1:2379
    - ETCD1.AZ2:2379
    - ETCD1.AZ3:2379
  key: ETCDKEY
  protocol: https
log:
  level: INFO
name: PG.AZ1
namespace: /CLUSTER/
postgresql:
  authentication:
    replication:
      password: PASS
      username: replicator
    superuser:
      password: PASS
      username: postgres
  basebackup:
    checkpoint: fast
    max-rate: 400M
  bin_dir: /usr/lib/postgresql/17/bin/
  callbacks:
    on_role_change: /usr/bin/flock --timeout 7200 /var/lock/LOCK.lock
      /usr/bin/pgbackrest_backup --stanza=CLUSTER --type=diff --patroni-port=8008
      --delay 10
  config_dir: /etc/postgresql/17/main
  connect_address: IP:5432
  create_replica_methods:
    - pgbackrest
    - basebackup
  data_dir: /var/lib/postgresql/17/main
  listen: '*:5432'
  parameters:
    archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn
      '%p'
    archive_mode: on
    max_connections: '100'
    max_locks_per_transaction: '64'
    max_prepared_transactions: 0
    max_replication_slots: '5'
    max_wal_senders: '20'
    max_worker_processes: 8
    ssl_cert_file: CERT
    ssl_key_file: KEY
    synchronous_commit: on
    track_commit_timestamp: off
    wal_keep_segments: '100'
    wal_keep_size: '1600'
    wal_level: hot_standby
    wal_log_hints: on
    wal_sender_timeout: 5min
  pg_hba:
    - host replication replicator 127.0.0.1/32 scram-sha-256
    - host replication replicator ::1/128 scram-sha-256
    - host replication replicator PG1.AZ1 scram-sha-256
    - host replication replicator PG1.AZ2 scram-sha-256
    - host replication replicator PG1.AZ3 scram-sha-256
    - host all all 0.0.0.0/0 scram-sha-256
    - host all all ::0/0 scram-sha-256
  pg_ident:
    - some postgres postgres
  pgbackrest:
    command: pgbackrest --stanza CLUSTER restore --log-level-console=warn --type=none
      --delta
    keep_data: true
    no_params: true
  pgpass: /var/lib/postgresql/.pgpass
  recovery_conf:
    recovery_target_timeline: latest
    restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
      %f '%p' || true
  remove_data_directory_on_diverged_timelines: false
  remove_data_directory_on_rewind_failure: false
  use_pg_rewind: true
  use_slots: true
  use_unix_socket: true
restapi:
  authentication:
    password: PASS
    username: CLUSTER
  certfile:
  connect_address: IP:8008
  keyfile:
  listen: :8008
scope: main17
tags:
  clonefrom: false
  nofailover: false
  noloadbalance: false
  nosync: false
  replicatefrom: ''
```

### patronictl show-config

```yaml
check_timeline: true
failsafe_mode: true
loop_wait: 10
maximum_lag_on_failover: 0
postgresql:
  parameters:
    archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn '%p'
    archive_mode: 'on'
    max_connections: '100'
    max_locks_per_transaction: '64'
    max_prepared_transactions: 0
    max_replication_slots: '5'
    max_wal_senders: '20'
    max_worker_processes: 8
    ssl_cert_file: CERT
    ssl_key_file: KEY
    synchronous_commit: 'on'
    track_commit_timestamp: 'off'
    wal_keep_segments: '100'
    wal_keep_size: '1600'
    wal_level: hot_standby
    wal_log_hints: 'on'
    wal_sender_timeout: 5min
  recovery_conf:
    recovery_target_timeline: latest
    restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn %f '%p' || true
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
synchronous_mode: true
synchronous_mode_strict: false
synchronous_node_count: 2
ttl: 30
```

### Patroni log files

#### PG with patch:

```shell
2025-05-06T10:55:02.493437+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:02,492 ERROR: Failed to get list of machines from https://ETCD.AZ2:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f14faa18f90>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:05.832378+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:05,831 ERROR: Failed to get list of machines from https://ETCD.AZ3:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ3', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f14fae93910>, 'Connection to ETCD.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:05.841514+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:05,840 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-05-06T10:55:05.842328+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:05,841 ERROR: Error communicating with DCS
2025-05-06T10:55:05.843147+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:05,842 INFO: DCS is not accessible
2025-05-06T10:55:05.844237+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:05,844 WARNING: Loop time exceeded, rescheduling immediately.
2025-05-06T10:55:09.181861+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:09,180 ERROR: Failed to get list of machines from https://ETCD.AZ2:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f14fae92290>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:12.520444+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:12,519 ERROR: Failed to get list of machines from https://ETCD.AZ3:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ3', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f14fb0aab10>, 'Connection to ETCD.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:15.915309+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:12,575 INFO: Lock owner: PG_PATCHED.AZ1; I am PG_PATCHED.AZ1
2025-05-06T10:55:15.915632+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:15,914 ERROR: Request to server https://ETCD.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD.AZ1', port=2379): Read timed out. (read timeout=3.327138942976793)")
2025-05-06T10:55:15.915682+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:15,914 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:15.915714+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:15,914 INFO: Retrying on https://ETCD.AZ2:2379
2025-05-06T10:55:19.254080+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:19,253 ERROR: Request to server https://ETCD.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f14faa8b090>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:19.254831+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:19,253 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:19.254889+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:19,253 INFO: Retrying on https://ETCD.AZ3:2379
2025-05-06T10:55:22.593026+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:22,591 ERROR: Request to server https://ETCD.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f14faec1150>, 'Connection to ETCD.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:22.593142+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:22,592 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:24.272688+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:24,271 ERROR: Error communicating with DCS
2025-05-06T10:55:24.273678+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:24,273 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-05-06T10:55:24.274272+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:24,274 INFO: DCS is not accessible
2025-05-06T10:55:24.275338+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:24,275 WARNING: Loop time exceeded, rescheduling immediately.
2025-05-06T10:55:26.915969+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:24,328 INFO: Lock owner: PG_PATCHED.AZ1; I am PG_PATCHED.AZ1
2025-05-06T10:55:26.916179+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:26,914 ERROR: failed to update leader lock
2025-05-06T10:55:26.966158+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:26,965 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:55:34.275839+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:34,274 WARNING: I am the leader but not owner of the lease
2025-05-06T10:55:34.379801+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:34,276 INFO: Lock owner: PG_PATCHED.AZ1; I am PG_PATCHED.AZ1
2025-05-06T10:55:34.380195+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:34,378 ERROR: failed to update leader lock
2025-05-06T10:55:34.380260+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:34,379 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:55:44.275659+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:44,274 WARNING: I am the leader but not owner of the lease
2025-05-06T10:55:44.375996+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:44,275 INFO: Lock owner: PG_PATCHED.AZ1; I am PG_PATCHED.AZ1
2025-05-06T10:55:44.376326+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:44,375 ERROR: failed to update leader lock
2025-05-06T10:55:44.376377+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:44,375 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:55:54.275412+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:54,274 WARNING: I am the leader but not owner of the lease
2025-05-06T10:55:54.377128+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:54,275 INFO: Lock owner: PG_PATCHED.AZ1; I am PG_PATCHED.AZ1
2025-05-06T10:55:54.377462+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:54,376 ERROR: failed to update leader lock
2025-05-06T10:55:54.377552+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:54,376 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:56:04.275457+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:56:04,274 WARNING: I am the leader but not owner of the lease
2025-05-06T10:56:04.375591+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:56:04,275 INFO: Lock owner: PG_PATCHED.AZ1; I am PG_PATCHED.AZ1
2025-05-06T10:56:04.375758+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:56:04,374 ERROR: failed to update leader lock
2025-05-06T10:56:04.375818+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:56:04,375 INFO: not promoting because failed to update leader lock in DCS

2025-05-06T10:55:07.704683+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:55:07,704 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:55:17.752406+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:55:17,752 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:55:27.725710+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:55:27,725 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:55:37.796220+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:55:37,795 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:55:47.706326+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:55:47,705 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:55:57.748525+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:55:57,748 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:56:07.704844+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:56:07,704 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:56:17.752478+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:56:17,752 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:56:27.704851+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:56:27,704 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:56:37.748789+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:56:37,748 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:56:47.705258+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:56:47,704 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)
2025-05-06T10:56:57.748361+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:56:57,748 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)

2025-05-06T10:55:07.700914+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:55:07,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:55:17.701203+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:55:17,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:55:27.768557+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:55:27,768 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:55:37.749611+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:55:37,749 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:55:47.702843+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:55:47,702 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:55:57.701687+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:55:57,701 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:56:07.701257+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:56:07,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:56:17.701769+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:56:17,701 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:56:27.700949+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:56:27,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:56:37.701720+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:56:37,701 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:56:47.701547+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:56:47,701 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:56:57.701185+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:56:57,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:57:07.701053+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:57:07,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:57:17.701392+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:57:17,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:57:27.701409+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:57:27,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:57:37.701754+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:57:37,701 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:57:47.701271+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:57:47,700 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
2025-05-06T10:57:57.701988+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:57:57,701 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
```

#### PG without patch:

```shell
2025-05-06T10:55:03.176936+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:03,175 ERROR: Failed to get list of machines from https://ETCD.AZ2:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9c5c806a10>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:06.516683+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:06,515 ERROR: Failed to get list of machines from https://ETCD.AZ3:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ3', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9c5c808ed0>, 'Connection to ETCD.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:06.526744+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:06,526 ERROR: Error communicating with DCS
2025-05-06T10:55:06.527978+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:06,527 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-05-06T10:55:06.528651+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:06,528 INFO: DCS is not accessible
2025-05-06T10:55:06.529545+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:06,529 WARNING: Loop time exceeded, rescheduling immediately.
2025-05-06T10:55:09.868856+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:09,867 ERROR: Failed to get list of machines from https://ETCD.AZ2:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9c5b977650>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:13.207549+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:13,206 ERROR: Failed to get list of machines from https://ETCD.AZ3:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ3', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9c5c805d90>, 'Connection to ETCD.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:16.605428+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:13,264 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:55:16.605567+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:16,604 ERROR: Request to server https://ETCD.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD.AZ1', port=2379): Read timed out. (read timeout=3.3277685773403696)")
2025-05-06T10:55:16.605604+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:16,604 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:16.605638+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:16,604 INFO: Retrying on https://ETCD.AZ2:2379
2025-05-06T10:55:19.945269+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:19,943 ERROR: Request to server https://ETCD.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9c5c61f290>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:19.945577+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:19,943 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:19.945647+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:19,943 INFO: Retrying on https://ETCD.AZ3:2379
2025-05-06T10:55:23.284019+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:23,282 ERROR: Request to server https://ETCD.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f9c5c61e9d0>, 'Connection to ETCD.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:23.284161+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:23,282 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:24.965375+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:24,964 ERROR: Error communicating with DCS
2025-05-06T10:55:24.966018+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:24,965 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-05-06T10:55:24.966769+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:24,966 INFO: DCS is not accessible
2025-05-06T10:55:24.968084+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:24,967 WARNING: Loop time exceeded, rescheduling immediately.
2025-05-06T10:55:27.115549+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:25,025 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:55:27.115999+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:27,114 ERROR: failed to update leader lock
2025-05-06T10:55:27.177333+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:27,177 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:55:34.968555+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:34,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:55:35.068907+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:34,968 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:55:35.069075+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:35,067 ERROR: failed to update leader lock
2025-05-06T10:55:35.069142+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:35,068 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:55:44.968224+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:44,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:55:45.068678+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:44,967 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:55:45.068881+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:45,067 ERROR: failed to update leader lock
2025-05-06T10:55:45.068948+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:45,068 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:55:54.968536+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:54,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:55:55.069818+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:54,968 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:55:55.069962+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:55,068 ERROR: failed to update leader lock
2025-05-06T10:55:55.070010+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:55:55,069 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:56:04.968492+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:04,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:56:05.068817+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:04,968 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:56:05.069009+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:05,067 ERROR: failed to update leader lock
2025-05-06T10:56:05.069068+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:05,068 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:56:14.968235+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:14,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:56:15.069567+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:14,967 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:56:15.069747+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:15,068 ERROR: failed to update leader lock
2025-05-06T10:56:15.069801+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:15,069 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:56:24.968214+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:24,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:56:25.068042+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:24,967 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:56:25.068160+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:25,067 ERROR: failed to update leader lock
2025-05-06T10:56:25.068203+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:25,067 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:56:34.967983+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:34,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:56:35.064691+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:34,967 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:56:35.064981+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:35,064 ERROR: failed to update leader lock
2025-05-06T10:56:35.065022+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:35,064 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:56:44.968072+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:44,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:56:45.487163+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:44,968 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:56:45.487504+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:45,486 ERROR: failed to update leader lock
2025-05-06T10:56:45.487564+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:45,486 INFO: not promoting because failed to update leader lock in DCS
2025-05-06T10:56:54.968555+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:54,967 WARNING: I am the leader but not owner of the lease
2025-05-06T10:56:55.024879+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:54,968 INFO: Lock owner: PG.AZ1; I am PG.AZ1
2025-05-06T10:56:55.024987+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:55,024 ERROR: failed to update leader lock
2025-05-06T10:56:55.025024+02:00 PG.AZ1 patroni[828]: 2025-05-06 10:56:55,024 INFO: not promoting because failed to update leader lock in DCS

2025-05-06T10:55:00.615644+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:55:00,615 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:55:10.620725+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:55:10,620 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:55:20.616095+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:55:20,615 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:55:30.667491+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:55:30,667 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:55:40.620069+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:55:40,619 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:55:50.619966+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:55:50,619 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:56:00.616569+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:56:00,615 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:56:10.620521+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:56:10,619 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:56:20.620591+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:56:20,620 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:56:30.616242+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:56:30,615 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:56:40.615515+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:56:40,615 INFO: no action. I am (PG.AZ2), the leader with the lock
2025-05-06T10:56:50.620560+02:00 PG.AZ2 patroni[1125001]: 2025-05-06 10:56:50,619 INFO: no action. I am (PG.AZ2), the leader with the lock

2025-05-06T10:54:50.665898+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:54:50,665 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:55:00.622000+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:55:00,621 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:55:10.665914+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:55:10,665 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:55:20.622053+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:55:20,621 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:55:30.677662+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:55:30,676 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:55:40.621723+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:55:40,620 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:55:50.666192+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:55:50,665 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:56:00.623108+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:56:00,622 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:56:10.670394+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:56:10,669 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:56:20.622232+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:56:20,621 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:56:30.666126+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:56:30,665 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:56:40.621319+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:56:40,620 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
2025-05-06T10:56:50.666098+02:00 PG.AZ3 patroni[2190215]: 2025-05-06 10:56:50,665 INFO: no action. I am (PG.AZ3), a secondary, and following a leader (PG.AZ2)
```

#### PG with patch and logged patch message:

```shell
2025-05-06T10:55:01.219188+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:01,218 ERROR: Request to server https://ETCD.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f18842e1a50>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:01.219470+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:01,218 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:01.219509+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:01,218 INFO: Retrying on https://ETCD.AZ3:2379
2025-05-06T10:55:04.558410+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:04,557 ERROR: Request to server https://ETCD.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f18842f6710>, 'Connection to ETCD.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:04.558528+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:04,557 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:07.898168+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:07,897 ERROR: Failed to get list of machines from https://ETCD.AZ2:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f18842cb1d0>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:07.907919+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:07,905 ERROR: Error communicating with DCS
2025-05-06T10:55:07.908316+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:07,906 INFO: DCS is not accessible
2025-05-06T10:55:07.908625+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:07,906 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-05-06T10:55:07.908665+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:07,907 WARNING: Loop time exceeded, rescheduling immediately.
2025-05-06T10:55:11.247724+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:11,246 ERROR: Failed to get list of machines from https://ETCD.AZ2:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f18842b1ed0>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:14.639446+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:11,299 INFO: Lock owner: PG_PATCHED2.AZ1; I am PG_PATCHED2.AZ1
2025-05-06T10:55:14.639921+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:14,638 ERROR: Request to server https://ETCD.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD.AZ1', port=2379): Read timed out. (read timeout=3.3277005621542535)")
2025-05-06T10:55:14.640001+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:14,638 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:14.640051+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:14,638 INFO: Retrying on https://ETCD.AZ3:2379
2025-05-06T10:55:17.978283+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:17,977 ERROR: Request to server https://ETCD.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1880526e90>, 'Connection to ETCD.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:17.978562+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:17,977 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:17.978628+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:17,977 INFO: Retrying on https://ETCD.AZ2:2379
2025-05-06T10:55:21.317068+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:21,315 ERROR: Request to server https://ETCD.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1882e02390>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:21.317427+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:21,315 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:24.656717+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:24,655 ERROR: Failed to get list of machines from https://ETCD.AZ2:2379/v3: MaxRetryError("HTTPSConnectionPool(host='ETCD.AZ2', port=2379): Max retries exceeded with url: /v3/cluster/member/list (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f18842c4b10>, 'Connection to ETCD.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-05-06T10:55:24.666636+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:24,665 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-05-06T10:55:24.667038+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:24,666 ERROR: Error communicating with DCS
2025-05-06T10:55:24.667863+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:24,667 INFO: DCS is not accessible
2025-05-06T10:55:24.669032+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:24,668 WARNING: Loop time exceeded, rescheduling immediately.
2025-05-06T10:55:26.403801+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:26,402 WARNING: Connected to Etcd node with term 123449. Old known term 125717. Switchnig to another node.
2025-05-06T10:55:26.404270+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:26,403 ERROR: Request to server https://ETCD.AZ1:2379 failed: StaleEtcdNode()
2025-05-06T10:55:26.404413+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:26,403 INFO: Reconnection allowed, looking for another server.
2025-05-06T10:55:26.404477+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:26,403 INFO: Retrying on https://ETCD.AZ2:2379
2025-05-06T10:55:28.091562+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:28,090 INFO: Selected new etcd server https://ETCD.AZ2:2379
2025-05-06T10:55:29.965579+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:29,945 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:55:29.965714+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:29,964 INFO: Local timeline=17 lsn=D/30000A0
2025-05-06T10:55:30.079456+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:30,079 INFO: primary_timeline=18
2025-05-06T10:55:30.081933+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:30,081 INFO: primary: history=14#011A/910000A0#011no recovery target specified
2025-05-06T10:55:30.082022+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 15#011C/7E22E2C0#011no recovery target specified
2025-05-06T10:55:30.082055+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 16#011D/A0#011no recovery target specified
2025-05-06T10:55:30.082081+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 17#011D/2081BA0#011no recovery target specified
2025-05-06T10:55:30.141992+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:30,141 INFO: running pg_rewind from PG_PATCHED2.AZ3
2025-05-06T10:55:30.241669+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:30,241 INFO: Trying to archive 000000110000000D00000002: pgbackrest --stanza=mapy-vectmap-dev archive-push --log-level-console=warn '/var/lib/postgresql/17/main/pg_wal/000000110000000D00000002'
2025-05-06T10:55:34.721043+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:34,669 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:55:34.721174+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:34,720 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:55:36.948321+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:36,947 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:55:36.948476+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:36,947 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:55:46.995974+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:46,947 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:55:46.996226+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:46,994 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:55:46.996273+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:46,994 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:55:46.996314+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:46,995 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:55:53.428367+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:53,426 INFO: running pg_rewind from dbname=postgres user=postgres host=10.130.12.240 port=5432 target_session_attrs=read-write
2025-05-06T10:55:56.948182+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:56,946 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:55:56.948423+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:55:56,947 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:56:06.997101+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:06,947 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:56:06.997247+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:06,995 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:56:06.997343+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:06,995 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:56:06.997394+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:06,996 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:56:16.948592+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:16,947 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:56:16.948722+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:16,947 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:56:27.000005+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:26,948 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
2025-05-06T10:56:27.000541+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:26,998 INFO: running pg_rewind from PG_PATCHED2.AZ3 in progress
2025-05-06T10:56:29.550405+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:29,549 INFO: pg_rewind exit code=0
2025-05-06T10:56:29.551049+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:29,549 INFO:  stdout=
2025-05-06T10:56:29.551116+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:29,549 INFO:  stderr=pg_rewind: servers diverged at WAL location D/2081BA0 on timeline 17
2025-05-06T10:56:29.551159+02:00 PG_PATCHED2.AZ1 patroni[4076236]: pg_rewind: rewinding from last common checkpoint at D/2071FE8 on timeline 17
2025-05-06T10:56:29.551195+02:00 PG_PATCHED2.AZ1 patroni[4076236]: pg_rewind: Done!
2025-05-06T10:56:29.551513+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:29,551 WARNING: Postgresql is not running.
2025-05-06T10:56:29.554283+02:00 PG_PATCHED2.AZ1 patroni[4076236]: 2025-05-06 10:56:29,551 INFO: Lock owner: PG_PATCHED2.AZ3; I am PG_PATCHED2.AZ1
```

### PostgreSQL log files

```shell
No logs needed
```

### Have you tried to use GitHub issue search?

- [x] Yes

### Anything else we need to know?

_No response_


Discussion:
What am I supposed to understand from provided logs without knowing initial state of every node?
@CyberDem0n 
The leader was in AZ1 before AZ1 was disconnected. A failover to another AZ was correctly triggered when AZ1 was disconnected.

At 10:55:20 AZ1 was reconnected.

After that, some nodes in AZ1 kept the old cached DCS cluster information because the new leader in another AZ was holding the lock and updating the cluster information in the DCS.

```shell
2025-05-06T10:55:26.915969+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:24,328 INFO: Lock owner: PG_PATCHED.AZ1; I am PG_PATCHED.AZ1
2025-05-06T10:55:26.916179+02:00 PG_PATCHED.AZ1 patroni[4060657]: 2025-05-06 10:55:26,914 ERROR: failed to update leader lock

2025-05-06T10:55:27.725710+02:00 PG_PATCHED.AZ2 patroni[1161727]: 2025-05-06 10:55:27,725 INFO: no action. I am (PG_PATCHED.AZ2), a secondary, and following a leader (PG_PATCHED.AZ3)

2025-05-06T10:55:27.768557+02:00 PG_PATCHED.AZ3 patroni[2842232]: 2025-05-06 10:55:27,768 INFO: no action. I am (PG_PATCHED.AZ3), the leader with the lock
```
It doesn't change the fact that without having logs from the moment _before_ problem started it is not possible to understand much.
The log before the issue contains the same information as the 20 seconds before AZ reconnected. There is no additional log message. Unfortunately, we don't have the full logs anymore. The next test is on June 3rd and we will send longer logs from the process.
Hi,
here are the Patroni logs from today's AZ disconnection test of AZ3.

[logs](https://drive.marek-jan.cz/public.php/dav/files/8GKa7ccken5CKqk/PG.tar.gz)

etcd-server version 3.5.21



Patch:
diff --git a/patroni/dcs/etcd.py b/patroni/dcs/etcd.py
index 1d8911d9e..3cd84ba47 100644
--- a/patroni/dcs/etcd.py
+++ b/patroni/dcs/etcd.py
@@ -101,40 +101,28 @@ def _do_resolve(host: str, port: int) -> List[_AddrInfo]:
             return []
 
 
-class AbstractEtcdClientWithFailover(abc.ABC, etcd.Client):
+class StaleEtcdNodeGuard(object):
 
-    ERROR_CLS: Type[Exception]
+    def __init__(self) -> None:
+        self._reset_cluster_raft_term()
 
-    def __init__(self, config: Dict[str, Any], dns_resolver: DnsCachingResolver, cache_ttl: int = 300) -> None:
+    def _reset_cluster_raft_term(self) -> None:
         self._cluster_id = None
         self._raft_term = 0
-        self._dns_resolver = dns_resolver
-        self.set_machines_cache_ttl(cache_ttl)
-        self._machines_cache_updated = 0
-        kwargs = {p: config.get(p) for p in ('host', 'port', 'protocol', 'use_proxies', 'version_prefix',
-                                             'username', 'password', 'cert', 'ca_cert') if config.get(p)}
-        super(AbstractEtcdClientWithFailover, self).__init__(read_timeout=config['retry_timeout'], **kwargs)
-        # For some reason python3-etcd on debian and ubuntu are not based on the latest version
-        # Workaround for the case when https://github.com/jplana/python-etcd/pull/196 is not applied
-        self.http.connection_pool_kw.pop('ssl_version', None)
-        self._config = config
-        self._load_machines_cache()
-        self._allow_reconnect = True
-        # allow passing retry argument to api_execute in params
-        self._comparison_conditions.add('retry')
-        self._read_options.add('retry')
-        self._del_conditions.add('retry')
 
     def _check_cluster_raft_term(self, cluster_id: Optional[str], value: Union[None, str, int]) -> None:
         """Check that observed Raft Term in Etcd cluster is increasing.
 
-        If we observe that the new value is smaller than the previously known one, it could be an
-        indicator that we connected to a stale node and should switch to some other node.
-        However, we need to reset the memorized value when we notice that Cluster ID changed.
+        :param cluster_id: last observed Etcd Cluster ID
+        :param raft_term: last observed Raft Term
+
+        :raises:
+            :exc::`StaleEtcdNode` if last observed *raft_term* is smaller than previously known *raft_term*.
         """
         if not (cluster_id and value):
             return
 
+        # We need to reset the memorized value when we notice that Cluster ID changed.
         if self._cluster_id and self._cluster_id != cluster_id:
             logger.warning('Etcd Cluster ID changed from %s to %s', self._cluster_id, cluster_id)
             self._raft_term = 0
@@ -151,6 +139,30 @@ def _check_cluster_raft_term(self, cluster_id: Optional[str], value: Union[None,
             raise StaleEtcdNode
         self._raft_term = raft_term
 
+
+class AbstractEtcdClientWithFailover(abc.ABC, etcd.Client, StaleEtcdNodeGuard):
+
+    ERROR_CLS: Type[Exception]
+
+    def __init__(self, config: Dict[str, Any], dns_resolver: DnsCachingResolver, cache_ttl: int = 300) -> None:
+        StaleEtcdNodeGuard.__init__(self)
+        self._dns_resolver = dns_resolver
+        self.set_machines_cache_ttl(cache_ttl)
+        self._machines_cache_updated = 0
+        kwargs = {p: config.get(p) for p in ('host', 'port', 'protocol', 'use_proxies', 'version_prefix',
+                                             'username', 'password', 'cert', 'ca_cert') if config.get(p)}
+        super(AbstractEtcdClientWithFailover, self).__init__(read_timeout=config['retry_timeout'], **kwargs)
+        # For some reason python3-etcd on debian and ubuntu are not based on the latest version
+        # Workaround for the case when https://github.com/jplana/python-etcd/pull/196 is not applied
+        self.http.connection_pool_kw.pop('ssl_version', None)
+        self._config = config
+        self._load_machines_cache()
+        self._allow_reconnect = True
+        # allow passing retry argument to api_execute in params
+        self._comparison_conditions.add('retry')
+        self._read_options.add('retry')
+        self._del_conditions.add('retry')
+
     def _calculate_timeouts(self, etcd_nodes: int, timeout: Optional[float] = None) -> Tuple[int, float, int]:
         """Calculate a request timeout and number of retries per single etcd node.
         In case if the timeout per node is too small (less than one second) we will reduce the number of nodes.
diff --git a/patroni/dcs/etcd3.py b/patroni/dcs/etcd3.py
index e18a86fed..f2cf6f848 100644
--- a/patroni/dcs/etcd3.py
+++ b/patroni/dcs/etcd3.py
@@ -24,7 +24,8 @@
 from ..utils import deep_compare, enable_keepalive, iter_response_objects, RetryFailedError, USER_AGENT
 from . import catch_return_false_exception, Cluster, ClusterConfig, \
     Failover, Leader, Member, Status, SyncState, TimelineHistory
-from .etcd import AbstractEtcd, AbstractEtcdClientWithFailover, catch_etcd_errors, DnsCachingResolver, Retry
+from .etcd import AbstractEtcd, AbstractEtcdClientWithFailover, catch_etcd_errors, \
+    DnsCachingResolver, Retry, StaleEtcdNode, StaleEtcdNodeGuard
 
 logger = logging.getLogger(__name__)
 
@@ -432,10 +433,11 @@ def watchprefix(self, key: str, start_revision: Optional[str] = None,
         return self.watchrange(key, prefix_range_end(key), start_revision, filters, read_timeout)
 
 
-class KVCache(Thread):
+class KVCache(StaleEtcdNodeGuard, Thread):
 
     def __init__(self, dcs: 'Etcd3', client: 'PatroniEtcd3Client') -> None:
-        super(KVCache, self).__init__()
+        Thread.__init__(self)
+        StaleEtcdNodeGuard.__init__(self)
         self.daemon = True
         self._dcs = dcs
         self._client = client
@@ -505,7 +507,10 @@ def _process_message(self, message: Dict[str, Any]) -> None:
         logger.debug('Received message: %s', message)
         if 'error' in message:
             raise _raise_for_data(message)
-        events: List[Dict[str, Any]] = message.get('result', {}).get('events', [])
+        result = message.get('result', EMPTY_DICT)
+        header = result.get('header', EMPTY_DICT)
+        self._check_cluster_raft_term(header.get('cluster_id'), header.get('raft_term'))
+        events: List[Dict[str, Any]] = result.get('events', [])
         for event in events:
             self._process_event(event)
 
@@ -539,8 +544,11 @@ def _do_watch(self, revision: str) -> None:
 
     def _build_cache(self) -> None:
         result = self._dcs.retry(self._client.prefix, self._dcs.cluster_prefix)
+        header = result.get('header', EMPTY_DICT)
         with self._object_cache_lock:
+            self._reset_cluster_raft_term()
             self._object_cache = {node['key']: node for node in result.get('kvs', [])}
+            self._check_cluster_raft_term(header.get('cluster_id'), header.get('raft_term'))
         with self.condition:
             self._is_ready = True
             self.condition.notify()
@@ -586,6 +594,12 @@ def kill_stream(self) -> None:
 
     def is_ready(self) -> bool:
         """Must be called only when holding the lock on `condition`"""
+        if self._is_ready:
+            try:
+                self._client._check_cluster_raft_term(self._cluster_id, self._raft_term)
+            except StaleEtcdNode:
+                self._is_ready = False
+                self.kill_stream()
         return self._is_ready
 
 


Test patch:
diff --git a/tests/test_etcd3.py b/tests/test_etcd3.py
index a201c3bed..ae52b7420 100644
--- a/tests/test_etcd3.py
+++ b/tests/test_etcd3.py
@@ -51,7 +51,7 @@ def mock_urlopen(self, method, url, **kwargs):
     elif url.endswith('/watch'):
         key = base64_encode('/patroni/test/config')
         ret.read_chunked = Mock(return_value=[json.dumps({
-            'result': {'events': [
+            'result': {'header': {'cluster_id': '1', 'raft_term': 1}, 'events': [
                 {'kv': {'key': key, 'value': base64_encode('bar'), 'mod_revision': '2'}},
                 {'kv': {'key': key, 'value': base64_encode('buzz'), 'mod_revision': '3'}},
                 {'type': 'DELETE', 'kv': {'key': key, 'mod_revision': '4'}},
@@ -120,6 +120,14 @@ def test_kill_stream(self):
             type(mock_conn).sock = PropertyMock(side_effect=Exception)
             self.kv_cache.kill_stream()
 
+    @patch.object(urllib3.PoolManager, 'urlopen', mock_urlopen)
+    def test_is_ready(self):
+        self.kv_cache._build_cache()
+        with self.kv_cache.condition:
+            self.kv_cache._is_ready = True
+            self.client._raft_term = 2
+            self.assertFalse(self.kv_cache.is_ready())
+
 
 class TestPatroniEtcd3Client(BaseTestEtcd3):