Repository: patroni/patroni

Issue:
The new leader (from the failover) is demoted after the reconnection of the whole AZ
### What happened?

We operate PG clusters with Patroni across 3 AZ. We regularly test the shutdown of one AZ (network disconnection). During these shutdowns, we started experiencing an issue where Patroni leader gets demoted upon the reconnection of the AZ (the old leader has a lock but no lease).
We are currently running approximately 200 PG clusters against a single etcd cluster (one node in every AZ). The problem does not occur on all clusters whose leader is located in the disconnected AZ at the beginning of the outage.

So far, everything suggests that outdated information about the cluster leader is being retained in the DCS cache.

### How can we reproduce it (as minimally and precisely as possible)?

- Disconnect AZ with PG cluster node and ETCD cluster node (2 nodes of both running in another 2 AZ) - PG leader in this AZ
- Reconnect AZ
- Some of the clusters were demoted - approx 1 to 10 from the total of 200

### What did you expect to happen?

New leader in one of the remaining active AZ will not be demoted after connecting the disconnected AZ.
DCS cache is updated with retry and no exception is thrown

### Patroni/PostgreSQL/DCS version

-   Patroni version: 4.0.4 (first seen on 3.3.0 with active failsafe mode)
-   PostgreSQL versions: 15,16,17
-   DCS (and its version): etcd 3.4.23-4+b4
-   PG and ETCD nodes on Debian Bookworm


### Patroni configuration file

```yaml
---
bootstrap:
  dcs:
    loop_wait: 10
    maximum_lag_on_failover: 0
    postgresql:
      parameters:
        archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn
          '%p'
        archive_mode: on
        max_connections: '100'
        max_locks_per_transaction: '64'
        max_prepared_transactions: 0
        max_replication_slots: '5'
        max_wal_senders: '20'
        max_worker_processes: 8
        ssl_cert_file: CERT
        ssl_key_file: KEY
        synchronous_commit: on
        track_commit_timestamp: off
        wal_keep_segments: '100'
        wal_keep_size: '1600'
        wal_level: hot_standby
        wal_log_hints: on
        wal_sender_timeout: 5min
      recovery_conf:
        recovery_target_timeline: latest
        restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
          %f '%p' || true
      use_pg_rewind: true
    retry_timeout: 10
    ttl: 30
  method: initdb
  pgbackrest:
    command: pgbackrest --stanza=CLUSTER restore --log-level-console=warn
    keep_existing_recovery_conf: true
    no_params: true
    recovery_conf:
      recovery_target_timeline: latest
      restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
        %f '%p'
  initdb:
    - encoding: UTF8
    - data-checksums
  pg_hba:
    - host replication replicator 127.0.0.1/32 scram-sha-256
    - host replication replicator ::1/128 scram-sha-256
    - host replication replicator PG1.AZ1 scram-sha-256
    - host replication replicator PG1.AZ2 scram-sha-256
    - host replication replicator PG1.AZ3 scram-sha-256
    - host all all 0.0.0.0/0 scram-sha-256
    - host all all ::0/0 scram-sha-256
  pg_ident:
    - some postgres postgres
  post_bootstrap: /tmp/post_bootstrap.sh
etcd3:
  cacert: ETCDCA
  cert: ETCDCERT
  hosts:
    - ETCD1.AZ1:2379
    - ETCD1.AZ2:2379
    - ETCD1.AZ3:2379
  key: ETCDKEY
  protocol: https
log:
  level: INFO
name: PG.AZ1
namespace: /CLUSTER/
postgresql:
  authentication:
    replication:
      password: PASS
      username: replicator
    superuser:
      password: PASS
      username: postgres
  basebackup:
    checkpoint: fast
    max-rate: 400M
  bin_dir: /usr/lib/postgresql/17/bin/
  callbacks:
    on_role_change: /usr/bin/flock --timeout 7200 /var/lock/LOCK.lock
      /usr/bin/pgbackrest_backup --stanza=CLUSTER --type=diff --patroni-port=8008
      --delay 10
  config_dir: /etc/postgresql/17/main
  connect_address: IP:5432
  create_replica_methods:
    - pgbackrest
    - basebackup
  data_dir: /var/lib/postgresql/17/main
  listen: '*:5432'
  parameters:
    archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn
      '%p'
    archive_mode: on
    max_connections: '100'
    max_locks_per_transaction: '64'
    max_prepared_transactions: 0
    max_replication_slots: '5'
    max_wal_senders: '20'
    max_worker_processes: 8
    ssl_cert_file: CERT
    ssl_key_file: KEY
    synchronous_commit: on
    track_commit_timestamp: off
    wal_keep_segments: '100'
    wal_keep_size: '1600'
    wal_level: hot_standby
    wal_log_hints: on
    wal_sender_timeout: 5min
  pg_hba:
    - host replication replicator 127.0.0.1/32 scram-sha-256
    - host replication replicator ::1/128 scram-sha-256
    - host replication replicator PG1.AZ1 scram-sha-256
    - host replication replicator PG1.AZ2 scram-sha-256
    - host replication replicator PG1.AZ3 scram-sha-256
    - host all all 0.0.0.0/0 scram-sha-256
    - host all all ::0/0 scram-sha-256
  pg_ident:
    - some postgres postgres
  pgbackrest:
    command: pgbackrest --stanza CLUSTER restore --log-level-console=warn --type=none
      --delta
    keep_data: true
    no_params: true
  pgpass: /var/lib/postgresql/.pgpass
  recovery_conf:
    recovery_target_timeline: latest
    restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn
      %f '%p' || true
  remove_data_directory_on_diverged_timelines: false
  remove_data_directory_on_rewind_failure: false
  use_pg_rewind: true
  use_slots: true
  use_unix_socket: true
restapi:
  authentication:
    password: PASS
    username: CLUSTER
  certfile:
  connect_address: IP:8008
  keyfile:
  listen: :8008
scope: main17
tags:
  clonefrom: false
  nofailover: false
  noloadbalance: false
  nosync: false
  replicatefrom: ''
```

### patronictl show-config

```yaml
check_timeline: true
failsafe_mode: true
loop_wait: 10
maximum_lag_on_failover: 0
postgresql:
  parameters:
    archive_command: pgbackrest --stanza=CLUSTER archive-push --log-level-console=warn '%p'
    archive_mode: 'on'
    max_connections: '100'
    max_locks_per_transaction: '64'
    max_prepared_transactions: 0
    max_replication_slots: '5'
    max_wal_senders: '20'
    max_worker_processes: 8
    ssl_cert_file: CERT
    ssl_key_file: KEY
    synchronous_commit: 'on'
    track_commit_timestamp: 'off'
    wal_keep_segments: '100'
    wal_keep_size: '1600'
    wal_level: hot_standby
    wal_log_hints: 'on'
    wal_sender_timeout: 5min
  recovery_conf:
    recovery_target_timeline: latest
    restore_command: pgbackrest --stanza=CLUSTER archive-get --log-level-console=warn %f '%p' || true
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
synchronous_mode: true
synchronous_mode_strict: false
synchronous_node_count: 2
ttl: 30
```

### Patroni log files

```shell
2025-03-11T23:54:03.373599+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:00,033 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:54:03.373907+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:03,372 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.3280809121982506)")
2025-03-11T23:54:03.373994+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:03,372 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:03.374024+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:03,372 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:54:06.713000+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:06,711 ERROR: Request to server https://ETCD1.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824171a50>, 'Connection to ETCD1.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:06.713283+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:06,712 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:06.713356+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:06,712 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:54:10.052256+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:10,050 ERROR: Request to server https://ETCD1.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f181c7a6410>, 'Connection to ETCD1.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:10.052658+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:10,051 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:10.061511+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:10,060 ERROR: Error communicating with DCS
2025-03-11T23:54:10.062968+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:10,062 INFO: DCS is not accessible
2025-03-11T23:54:10.063041+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:10,062 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:54:10.063763+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:10,063 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:54:13.493869+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:10,153 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:54:13.494122+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:13,492 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.3264659143363438)")
2025-03-11T23:54:13.494177+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:13,492 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:13.494260+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:13,492 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:54:16.833378+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:16,832 ERROR: Request to server https://ETCD1.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824142bd0>, 'Connection to ETCD1.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:16.833553+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:16,832 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:16.833618+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:16,832 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:54:20.172993+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:20,171 ERROR: Request to server https://ETCD1.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824155150>, 'Connection to ETCD1.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:20.173359+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:20,171 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:20.183073+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:20,182 ERROR: Error communicating with DCS
2025-03-11T23:54:20.184415+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:20,183 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:54:20.184901+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:20,184 INFO: DCS is not accessible
2025-03-11T23:54:20.186394+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:20,186 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:54:23.618799+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:20,277 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:54:23.619269+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:23,617 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.3269806793735675)")
2025-03-11T23:54:23.619346+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:23,617 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:23.619395+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:23,617 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:54:26.957790+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:26,956 ERROR: Request to server https://ETCD1.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f18241715d0>, 'Connection to ETCD1.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:26.958106+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:26,956 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:26.958171+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:26,956 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:54:30.296974+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:30,295 ERROR: Request to server https://ETCD1.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824172590>, 'Connection to ETCD1.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:30.297223+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:30,295 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:30.308102+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:30,306 ERROR: Error communicating with DCS
2025-03-11T23:54:30.308307+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:30,307 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:54:30.309005+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:30,308 INFO: DCS is not accessible
2025-03-11T23:54:30.309855+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:30,309 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:54:33.736822+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:30,396 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:54:33.737226+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:33,735 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.3280347705197832)")
2025-03-11T23:54:33.737304+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:33,735 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:33.737357+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:33,735 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:54:37.075892+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:37,074 ERROR: Request to server https://ETCD1.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824143f10>, 'Connection to ETCD1.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:37.076092+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:37,074 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:37.076155+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:37,074 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:54:40.415290+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:40,413 ERROR: Request to server https://ETCD1.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824143190>, 'Connection to ETCD1.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:40.415510+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:40,413 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:40.425960+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:40,425 ERROR: Error communicating with DCS
2025-03-11T23:54:40.427966+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:40,426 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:54:40.428218+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:40,427 INFO: DCS is not accessible
2025-03-11T23:54:40.429144+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:40,428 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:54:43.866642+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:40,525 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:54:43.866870+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:43,865 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.3272797864240906)")
2025-03-11T23:54:43.866959+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:43,865 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:43.867045+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:43,865 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:54:47.205403+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:47,204 ERROR: Request to server https://ETCD1.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f182416b0d0>, 'Connection to ETCD1.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:47.205689+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:47,204 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:47.205741+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:47,204 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:54:50.544386+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:50,543 ERROR: Request to server https://ETCD1.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f182416a090>, 'Connection to ETCD1.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:50.544560+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:50,543 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:50.554483+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:50,554 ERROR: Error communicating with DCS
2025-03-11T23:54:50.556396+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:50,554 INFO: DCS is not accessible
2025-03-11T23:54:50.556552+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:50,555 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:54:50.556627+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:50,556 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:54:53.986190+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:50,646 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:54:53.986722+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:53,985 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.3272768951331577)")
2025-03-11T23:54:53.986780+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:53,985 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:53.986825+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:53,985 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:54:57.325405+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:57,324 ERROR: Request to server https://ETCD1.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824e087d0>, 'Connection to ETCD1.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:54:57.325533+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:57,324 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:54:57.325584+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:54:57,324 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:55:00.665512+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:00,663 ERROR: Request to server https://ETCD1.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f182416a990>, 'Connection to ETCD1.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:55:00.665746+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:00,663 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:00.676583+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:00,675 ERROR: Error communicating with DCS
2025-03-11T23:55:00.676851+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:00,675 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:55:00.677804+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:00,677 INFO: DCS is not accessible
2025-03-11T23:55:00.678898+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:00,678 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:55:04.106010+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:00,765 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:55:04.106134+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:04,104 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.3270530552448085)")
2025-03-11T23:55:04.106168+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:04,104 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:04.106220+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:04,104 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:55:07.445287+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:07,444 ERROR: Request to server https://ETCD1.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f18241439d0>, 'Connection to ETCD1.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:55:07.445524+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:07,444 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:07.445570+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:07,444 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:55:10.784674+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:10,783 ERROR: Request to server https://ETCD1.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f182413a890>, 'Connection to ETCD1.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:55:10.784830+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:10,783 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:10.795645+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:10,794 ERROR: Error communicating with DCS
2025-03-11T23:55:10.795910+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:10,794 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:55:10.796566+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:10,796 INFO: DCS is not accessible
2025-03-11T23:55:10.797875+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:10,797 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:55:14.229658+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:10,889 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:55:14.229909+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:14,228 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.3268492383261523)")
2025-03-11T23:55:14.229963+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:14,228 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:14.229990+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:14,228 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:55:17.568618+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:17,567 ERROR: Request to server https://ETCD1.AZ3:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824fca110>, 'Connection to ETCD1.AZ3 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:55:17.568755+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:17,567 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:17.568786+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:17,567 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:55:20.908061+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:20,906 ERROR: Request to server https://ETCD1.AZ2:2379 failed: MaxRetryError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Max retries exceeded with url: /v3/lease/keepalive (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1824170a10>, 'Connection to ETCD1.AZ2 timed out. (connect timeout=1.6666666666666667)'))")
2025-03-11T23:55:20.909503+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:20,906 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:20.917945+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:20,917 ERROR: Error communicating with DCS
2025-03-11T23:55:20.918568+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:20,918 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:55:20.919340+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:20,919 INFO: DCS is not accessible
2025-03-11T23:55:20.920547+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:20,920 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:55:24.349543+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:21,009 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:55:24.349647+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:24,348 ERROR: Request to server https://ETCD1.AZ1:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ1', port=2379): Read timed out. (read timeout=3.327339498170962)")
2025-03-11T23:55:24.349687+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:24,348 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:24.349714+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:24,348 INFO: Retrying on https://ETCD1.AZ2:2379
2025-03-11T23:55:27.686884+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:27,685 ERROR: Request to server https://ETCD1.AZ2:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ2', port=2379): Read timed out. (read timeout=1.6491017433193824)")
2025-03-11T23:55:27.687336+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:27,685 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:27.687377+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:27,685 INFO: Retrying on https://ETCD1.AZ3:2379
2025-03-11T23:55:31.025845+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:31,024 ERROR: Request to server https://ETCD1.AZ3:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Read timed out. (read timeout=3.3244329524071268)")
2025-03-11T23:55:31.025985+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:31,024 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:31.035737+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:31,034 ERROR: Error communicating with DCS
2025-03-11T23:55:31.037179+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:31,035 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:55:31.037377+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:31,037 INFO: DCS is not accessible
2025-03-11T23:55:31.038516+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:31,038 WARNING: Loop time exceeded, rescheduling immediately.
2025-03-11T23:55:33.492091+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:31,124 INFO: Lock owner: PG1.AZ1; I am PG1.AZ1
2025-03-11T23:55:33.492476+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:33,490 ERROR: failed to update leader lock
2025-03-11T23:55:33.541633+01:00 PG1.AZ1 patroni[832]: 2025-03-11 23:55:33,541 INFO: not promoting because failed to update leader lock in DCS



2025-03-11T23:54:07.276395+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:54:07,275 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:54:17.232645+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:54:17,231 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:54:27.277020+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:54:27,275 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:54:37.232547+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:54:37,231 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:54:47.276904+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:54:47,276 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:54:57.232406+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:54:57,231 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:55:07.280839+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:55:07,279 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:55:17.232660+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:55:17,231 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:55:29.548798+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:55:29,548 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:55:30.552201+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:55:30,551 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:55:41.122249+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:55:41,121 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:55:51.016982+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:55:51,016 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:56:01.060620+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:56:01,059 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:56:11.016104+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:56:11,015 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)
2025-03-11T23:56:21.065143+01:00 PG1.AZ2 patroni[823]: 2025-03-11 23:56:21,064 INFO: no action. I am (PG1.AZ2), a secondary, and following a leader (PG1.AZ3)



2025-03-11T23:54:07.233916+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:54:07,233 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:54:17.233427+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:54:17,232 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:54:27.233481+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:54:27,232 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:54:37.234072+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:54:37,233 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:54:47.234578+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:54:47,233 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:54:57.233800+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:54:57,233 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:55:07.237408+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:07,236 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:55:17.233385+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:17,232 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:55:30.433019+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:27,091 INFO: Lock owner: PG1.AZ3; I am PG1.AZ3
2025-03-11T23:55:30.434392+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:30,431 ERROR: Request to server https://ETCD1.AZ3:2379 failed: ReadTimeoutError("HTTPSConnectionPool(host='ETCD1.AZ3', port=2379): Read timed out. (read timeout=3.333142254346361)")
2025-03-11T23:55:30.434464+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:30,431 INFO: Reconnection allowed, looking for another server.
2025-03-11T23:55:30.434536+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:30,431 INFO: Retrying on https://ETCD1.AZ1:2379
2025-03-11T23:55:30.454039+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:30,453 INFO: Selected new etcd server https://ETCD1.AZ1:2379
2025-03-11T23:55:30.455742+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:30,454 ERROR: watchprefix failed: ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
2025-03-11T23:55:33.489657+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:33,488 INFO: no action. I am (PG1.AZ3), the leader with the lock
2025-03-11T23:55:37.094656+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:37,092 INFO: Lock owner: PG1.AZ1; I am PG1.AZ3
2025-03-11T23:55:37.097494+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:37,093 INFO: Demoting self (immediate-nolock)
2025-03-11T23:55:38.992618+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:38,991 INFO: demoting self because I do not have the lock and I was a leader
2025-03-11T23:55:38.993899+01:00 PG1.AZ3 patroni[831]: 2025-03-11 23:55:38,993 INFO: closed patroni connections to postgres
```

### PostgreSQL log files

```shell
No logs needed
```

### Have you tried to use GitHub issue search?

- [x] Yes

### Anything else we need to know?

_No response_


Discussion:
There is not enough Patroni logs to understand anything.
Please provide logs at least starting from `23:54:00`.

And, please also check logs on all Etcd nodes.
@marekj11 


### ETCD log files

```shell
2025-03-11T23:54:00.325287+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:00 INFO: 38c2171bd506eaa is starting a new election at term 120788
2025-03-11T23:54:00.325380+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:00 INFO: 38c2171bd506eaa became candidate at term 120789
2025-03-11T23:54:00.325421+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:00 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120789
2025-03-11T23:54:00.325459+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:00 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120789
2025-03-11T23:54:00.325506+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:00 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120789
2025-03-11T23:54:03.624837+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:03 INFO: 38c2171bd506eaa is starting a new election at term 120789
2025-03-11T23:54:03.624999+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:03 INFO: 38c2171bd506eaa became candidate at term 120790
2025-03-11T23:54:03.625034+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:03 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120790
2025-03-11T23:54:03.625067+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:03 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120790
2025-03-11T23:54:03.625093+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:03 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120790
2025-03-11T23:54:03.859097+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:03.859447+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:03.986717+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:03.990585+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:06.624970+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:06 INFO: 38c2171bd506eaa is starting a new election at term 120790
2025-03-11T23:54:06.625066+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:06 INFO: 38c2171bd506eaa became candidate at term 120791
2025-03-11T23:54:06.625102+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:06 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120791
2025-03-11T23:54:06.625129+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:06 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120791
2025-03-11T23:54:06.625219+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:06 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120791
2025-03-11T23:54:08.860170+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:08.860356+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:08.987316+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:08.990696+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:09.924492+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:09 INFO: 38c2171bd506eaa is starting a new election at term 120791
2025-03-11T23:54:09.924626+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:09 INFO: 38c2171bd506eaa became candidate at term 120792
2025-03-11T23:54:09.924672+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:09 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120792
2025-03-11T23:54:09.924699+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:09 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120792
2025-03-11T23:54:09.924728+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:09 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120792
2025-03-11T23:54:09.960973+01:00 ETCD1.AZ1 etcd[1075425]: rejected connection from "[2a02:598:64:8a00::4800:39b]:58714" (error "tls: client didn't provide a certificate", ServerName "ETCD1.AZ1")
2025-03-11T23:54:11.725312+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:11 INFO: 38c2171bd506eaa is starting a new election at term 120792
2025-03-11T23:54:11.725480+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:11 INFO: 38c2171bd506eaa became candidate at term 120793
2025-03-11T23:54:11.725509+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:11 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120793
2025-03-11T23:54:11.725575+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:11 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120793
2025-03-11T23:54:11.725616+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:11 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120793
2025-03-11T23:54:13.524560+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:13 INFO: 38c2171bd506eaa is starting a new election at term 120793
2025-03-11T23:54:13.524653+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:13 INFO: 38c2171bd506eaa became candidate at term 120794
2025-03-11T23:54:13.524682+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:13 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120794
2025-03-11T23:54:13.524724+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:13 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120794
2025-03-11T23:54:13.524775+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:13 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120794
2025-03-11T23:54:13.861261+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:13.861395+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:13.988283+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:13.991573+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:15.924555+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:15 INFO: 38c2171bd506eaa is starting a new election at term 120794
2025-03-11T23:54:15.924654+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:15 INFO: 38c2171bd506eaa became candidate at term 120795
2025-03-11T23:54:15.924698+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:15 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120795
2025-03-11T23:54:15.924728+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:15 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120795
2025-03-11T23:54:15.924758+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:15 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120795
2025-03-11T23:54:17.725611+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:17 INFO: 38c2171bd506eaa is starting a new election at term 120795
2025-03-11T23:54:17.725716+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:17 INFO: 38c2171bd506eaa became candidate at term 120796
2025-03-11T23:54:17.725750+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:17 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120796
2025-03-11T23:54:17.725786+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:17 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120796
2025-03-11T23:54:17.725844+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:17 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120796
2025-03-11T23:54:18.862437+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:18.862559+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:18.988441+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:18.992649+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:19.825094+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:19 INFO: 38c2171bd506eaa is starting a new election at term 120796
2025-03-11T23:54:19.825300+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:19 INFO: 38c2171bd506eaa became candidate at term 120797
2025-03-11T23:54:19.825362+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:19 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120797
2025-03-11T23:54:19.825424+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:19 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120797
2025-03-11T23:54:19.825496+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:19 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120797
2025-03-11T23:54:21.925375+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:21 INFO: 38c2171bd506eaa is starting a new election at term 120797
2025-03-11T23:54:21.925465+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:21 INFO: 38c2171bd506eaa became candidate at term 120798
2025-03-11T23:54:21.925498+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:21 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120798
2025-03-11T23:54:21.925525+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:21 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120798
2025-03-11T23:54:21.925556+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:21 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120798
2025-03-11T23:54:23.863396+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:23.863530+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:23.989546+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:23.992929+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:24.024642+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:24 INFO: 38c2171bd506eaa is starting a new election at term 120798
2025-03-11T23:54:24.024838+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:24 INFO: 38c2171bd506eaa became candidate at term 120799
2025-03-11T23:54:24.024947+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:24 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120799
2025-03-11T23:54:24.025033+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:24 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120799
2025-03-11T23:54:24.025071+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:24 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120799
2025-03-11T23:54:26.425509+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:26 INFO: 38c2171bd506eaa is starting a new election at term 120799
2025-03-11T23:54:26.425708+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:26 INFO: 38c2171bd506eaa became candidate at term 120800
2025-03-11T23:54:26.425742+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:26 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120800
2025-03-11T23:54:26.425789+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:26 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120800
2025-03-11T23:54:26.425839+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:26 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120800
2025-03-11T23:54:28.863698+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:28.863879+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:28.989553+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:28.993899+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:29.124743+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:29 INFO: 38c2171bd506eaa is starting a new election at term 120800
2025-03-11T23:54:29.124835+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:29 INFO: 38c2171bd506eaa became candidate at term 120801
2025-03-11T23:54:29.124871+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:29 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120801
2025-03-11T23:54:29.124901+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:29 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120801
2025-03-11T23:54:29.124926+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:29 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120801
2025-03-11T23:54:31.824567+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:31 INFO: 38c2171bd506eaa is starting a new election at term 120801
2025-03-11T23:54:31.824717+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:31 INFO: 38c2171bd506eaa became candidate at term 120802
2025-03-11T23:54:31.824750+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:31 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120802
2025-03-11T23:54:31.824780+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:31 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120802
2025-03-11T23:54:31.824827+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:31 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120802
2025-03-11T23:54:33.864136+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:33.864635+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:33.990273+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:33.994725+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:34.224659+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:34 INFO: 38c2171bd506eaa is starting a new election at term 120802
2025-03-11T23:54:34.224755+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:34 INFO: 38c2171bd506eaa became candidate at term 120803
2025-03-11T23:54:34.224788+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:34 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120803
2025-03-11T23:54:34.224818+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:34 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120803
2025-03-11T23:54:34.224847+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:34 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120803
2025-03-11T23:54:36.325532+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:36 INFO: 38c2171bd506eaa is starting a new election at term 120803
2025-03-11T23:54:36.325628+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:36 INFO: 38c2171bd506eaa became candidate at term 120804
2025-03-11T23:54:36.325662+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:36 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120804
2025-03-11T23:54:36.325700+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:36 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120804
2025-03-11T23:54:36.325867+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:36 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120804
2025-03-11T23:54:38.071321+01:00 ETCD1.AZ1 etcd[1075425]: rejected connection from "[2a02:598:64:8a00::4800:39b]:48106" (error "tls: client didn't provide a certificate", ServerName "ETCD1.AZ1")
2025-03-11T23:54:38.864980+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:38.865138+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:38.991416+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:38.995812+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:39.024770+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:39 INFO: 38c2171bd506eaa is starting a new election at term 120804
2025-03-11T23:54:39.024882+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:39 INFO: 38c2171bd506eaa became candidate at term 120805
2025-03-11T23:54:39.024921+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:39 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120805
2025-03-11T23:54:39.024951+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:39 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120805
2025-03-11T23:54:39.024994+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:39 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120805
2025-03-11T23:54:42.325351+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:42 INFO: 38c2171bd506eaa is starting a new election at term 120805
2025-03-11T23:54:42.325476+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:42 INFO: 38c2171bd506eaa became candidate at term 120806
2025-03-11T23:54:42.325525+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:42 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120806
2025-03-11T23:54:42.325571+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:42 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120806
2025-03-11T23:54:42.325615+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:42 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120806
2025-03-11T23:54:43.865894+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:43.866100+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:43.991888+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:43.996246+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:45.624816+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:45 INFO: 38c2171bd506eaa is starting a new election at term 120806
2025-03-11T23:54:45.624924+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:45 INFO: 38c2171bd506eaa became candidate at term 120807
2025-03-11T23:54:45.624960+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:45 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120807
2025-03-11T23:54:45.624990+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:45 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120807
2025-03-11T23:54:45.625023+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:45 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120807
2025-03-11T23:54:48.866664+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:48.866793+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:48.924767+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:48 INFO: 38c2171bd506eaa is starting a new election at term 120807
2025-03-11T23:54:48.924858+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:48 INFO: 38c2171bd506eaa became candidate at term 120808
2025-03-11T23:54:48.924900+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:48 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120808
2025-03-11T23:54:48.924932+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:48 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120808
2025-03-11T23:54:48.924962+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:48 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120808
2025-03-11T23:54:48.992734+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:48.996490+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:51.325277+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:51 INFO: 38c2171bd506eaa is starting a new election at term 120808
2025-03-11T23:54:51.325388+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:51 INFO: 38c2171bd506eaa became candidate at term 120809
2025-03-11T23:54:51.325420+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:51 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120809
2025-03-11T23:54:51.325465+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:51 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120809
2025-03-11T23:54:51.325500+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:51 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120809
2025-03-11T23:54:53.125487+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:53 INFO: 38c2171bd506eaa is starting a new election at term 120809
2025-03-11T23:54:53.125595+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:53 INFO: 38c2171bd506eaa became candidate at term 120810
2025-03-11T23:54:53.125635+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:53 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120810
2025-03-11T23:54:53.125670+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:53 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120810
2025-03-11T23:54:53.125696+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:53 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120810
2025-03-11T23:54:53.867657+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:53.867800+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:53.993293+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:53.997348+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:56.124491+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:56 INFO: 38c2171bd506eaa is starting a new election at term 120810
2025-03-11T23:54:56.124688+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:56 INFO: 38c2171bd506eaa became candidate at term 120811
2025-03-11T23:54:56.124772+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:56 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120811
2025-03-11T23:54:56.124810+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:56 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120811
2025-03-11T23:54:56.124863+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:56 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120811
2025-03-11T23:54:58.525570+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:58 INFO: 38c2171bd506eaa is starting a new election at term 120811
2025-03-11T23:54:58.525681+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:58 INFO: 38c2171bd506eaa became candidate at term 120812
2025-03-11T23:54:58.525725+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:58 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120812
2025-03-11T23:54:58.525773+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:58 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120812
2025-03-11T23:54:58.525824+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:54:58 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120812
2025-03-11T23:54:58.867999+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:58.868150+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:54:58.994297+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:54:58.997698+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:01.225102+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:01 INFO: 38c2171bd506eaa is starting a new election at term 120812
2025-03-11T23:55:01.225283+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:01 INFO: 38c2171bd506eaa became candidate at term 120813
2025-03-11T23:55:01.225318+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:01 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120813
2025-03-11T23:55:01.225348+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:01 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120813
2025-03-11T23:55:01.225375+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:01 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120813
2025-03-11T23:55:03.868700+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:03.869036+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:03.924288+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:03 INFO: 38c2171bd506eaa is starting a new election at term 120813
2025-03-11T23:55:03.924391+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:03 INFO: 38c2171bd506eaa became candidate at term 120814
2025-03-11T23:55:03.924424+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:03 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120814
2025-03-11T23:55:03.924475+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:03 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120814
2025-03-11T23:55:03.924508+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:03 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120814
2025-03-11T23:55:03.994301+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:03.998324+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:06.324551+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:06 INFO: 38c2171bd506eaa is starting a new election at term 120814
2025-03-11T23:55:06.324648+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:06 INFO: 38c2171bd506eaa became candidate at term 120815
2025-03-11T23:55:06.324678+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:06 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120815
2025-03-11T23:55:06.324715+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:06 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120815
2025-03-11T23:55:06.324783+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:06 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120815
2025-03-11T23:55:08.124989+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:08 INFO: 38c2171bd506eaa is starting a new election at term 120815
2025-03-11T23:55:08.125168+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:08 INFO: 38c2171bd506eaa became candidate at term 120816
2025-03-11T23:55:08.125225+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:08 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120816
2025-03-11T23:55:08.125270+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:08 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120816
2025-03-11T23:55:08.125321+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:08 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120816
2025-03-11T23:55:08.868962+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:08.869123+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:08.995248+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:08.999009+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:09.959286+01:00 ETCD1.AZ1 etcd[1075425]: rejected connection from "[2a02:598:64:8a00::4800:39b]:57792" (error "tls: client didn't provide a certificate", ServerName "ETCD1.AZ1")
2025-03-11T23:55:11.425160+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:11 INFO: 38c2171bd506eaa is starting a new election at term 120816
2025-03-11T23:55:11.425249+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:11 INFO: 38c2171bd506eaa became candidate at term 120817
2025-03-11T23:55:11.425280+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:11 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120817
2025-03-11T23:55:11.425307+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:11 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120817
2025-03-11T23:55:11.425341+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:11 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120817
2025-03-11T23:55:13.869451+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:13.869657+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:13.995365+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:13.999760+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:14.124632+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:14 INFO: 38c2171bd506eaa is starting a new election at term 120817
2025-03-11T23:55:14.124729+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:14 INFO: 38c2171bd506eaa became candidate at term 120818
2025-03-11T23:55:14.124768+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:14 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120818
2025-03-11T23:55:14.124795+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:14 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120818
2025-03-11T23:55:14.124826+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:14 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120818
2025-03-11T23:55:16.824628+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:16 INFO: 38c2171bd506eaa is starting a new election at term 120818
2025-03-11T23:55:16.824729+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:16 INFO: 38c2171bd506eaa became candidate at term 120819
2025-03-11T23:55:16.824766+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:16 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120819
2025-03-11T23:55:16.824801+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:16 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120819
2025-03-11T23:55:16.824834+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:16 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120819
2025-03-11T23:55:18.870294+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:18.870491+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:18.996375+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:19.000690+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:20.124729+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:20 INFO: 38c2171bd506eaa is starting a new election at term 120819
2025-03-11T23:55:20.124832+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:20 INFO: 38c2171bd506eaa became candidate at term 120820
2025-03-11T23:55:20.124865+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:20 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120820
2025-03-11T23:55:20.124891+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:20 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120820
2025-03-11T23:55:20.124930+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:20 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120820
2025-03-11T23:55:22.524727+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:22 INFO: 38c2171bd506eaa is starting a new election at term 120820
2025-03-11T23:55:22.524915+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:22 INFO: 38c2171bd506eaa became candidate at term 120821
2025-03-11T23:55:22.524957+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:22 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120821
2025-03-11T23:55:22.524987+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:22 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120821
2025-03-11T23:55:22.525016+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:22 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120821
2025-03-11T23:55:23.021834+01:00 ETCD1.AZ1 etcd[1075425]: peer a38a9b279530ed1 became active
2025-03-11T23:55:23.021975+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer a38a9b279530ed1 (stream Message writer)
2025-03-11T23:55:23.022751+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer a38a9b279530ed1 (stream MsgApp v2 writer)
2025-03-11T23:55:23.114579+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer a38a9b279530ed1 (stream MsgApp v2 reader)
2025-03-11T23:55:23.437372+01:00 ETCD1.AZ1 etcd[1075425]: peer 48ad1cafe078d6d5 became active
2025-03-11T23:55:23.437526+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer 48ad1cafe078d6d5 (stream Message reader)
2025-03-11T23:55:23.870190+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:23.870326+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:23.996545+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer 48ad1cafe078d6d5 could not connect: dial tcp [ETCD1.AZ2]:2380: i/o timeout
2025-03-11T23:55:24.001090+01:00 ETCD1.AZ1 etcd[1075425]: health check for peer a38a9b279530ed1 could not connect: dial tcp [ETCD1.AZ3]:2380: i/o timeout
2025-03-11T23:55:24.351589+01:00 ETCD1.AZ1 etcd[1075425]: failed to dial 48ad1cafe078d6d5 on stream MsgApp v2 (dial tcp [ETCD1.AZ2]:2380: i/o timeout)
2025-03-11T23:55:24.351727+01:00 ETCD1.AZ1 etcd[1075425]: peer 48ad1cafe078d6d5 became inactive (message send to peer failed)
2025-03-11T23:55:24.590738+01:00 ETCD1.AZ1 etcd[1075425]: failed to dial a38a9b279530ed1 on stream Message (dial tcp [ETCD1.AZ3]:2380: i/o timeout)
2025-03-11T23:55:24.590871+01:00 ETCD1.AZ1 etcd[1075425]: peer a38a9b279530ed1 became inactive (message send to peer failed)
2025-03-11T23:55:24.670492+01:00 ETCD1.AZ1 etcd[1075425]: peer 48ad1cafe078d6d5 became active
2025-03-11T23:55:24.670620+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer 48ad1cafe078d6d5 (stream MsgApp v2 reader)
2025-03-11T23:55:24.736838+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer 48ad1cafe078d6d5 (stream MsgApp v2 writer)
2025-03-11T23:55:24.851489+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer 48ad1cafe078d6d5 (stream Message writer)
2025-03-11T23:55:24.913082+01:00 ETCD1.AZ1 etcd[1075425]: peer a38a9b279530ed1 became active
2025-03-11T23:55:24.913194+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer a38a9b279530ed1 (stream Message reader)
2025-03-11T23:55:25.824880+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa is starting a new election at term 120821
2025-03-11T23:55:25.825086+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa became candidate at term 120822
2025-03-11T23:55:25.825175+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa received MsgVoteResp from 38c2171bd506eaa at term 120822
2025-03-11T23:55:25.825271+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to a38a9b279530ed1 at term 120822
2025-03-11T23:55:25.825346+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444] sent MsgVote request to 48ad1cafe078d6d5 at term 120822
2025-03-11T23:55:25.831785+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa received MsgVoteResp rejection from 48ad1cafe078d6d5 at term 120822
2025-03-11T23:55:25.831874+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa has received 1 MsgVoteResp votes and 1 vote rejections
2025-03-11T23:55:25.831916+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa received MsgVoteResp rejection from a38a9b279530ed1 at term 120822
2025-03-11T23:55:25.831954+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa has received 1 MsgVoteResp votes and 2 vote rejections
2025-03-11T23:55:25.831995+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:25 INFO: 38c2171bd506eaa became follower at term 120822
2025-03-11T23:55:28.349017+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:28 INFO: 38c2171bd506eaa [term: 120822] received a MsgVote message with higher term from a38a9b279530ed1 [term: 120823]
2025-03-11T23:55:28.349180+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:28 INFO: 38c2171bd506eaa became follower at term 120823
2025-03-11T23:55:28.349216+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:28 INFO: 38c2171bd506eaa [logterm: 119522, index: 5322828444, vote: 0] cast MsgVote for a38a9b279530ed1 [logterm: 119522, index: 5323014247] at term 120823
2025-03-11T23:55:28.350382+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:28 INFO: raft.node: 38c2171bd506eaa elected leader a38a9b279530ed1 at term 120823
2025-03-11T23:55:29.269783+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.358562+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.441002+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.465697+01:00 ETCD1.AZ1 etcd[1075425]: receiving database snapshot [index: 5323014273, from: a38a9b279530ed1, raft message size: 735 kB]
2025-03-11T23:55:29.578460+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.641372+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.670635+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.727094+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.868635+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.881624+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.884946+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:29.990443+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:29 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:30.363974+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:30 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:30.471933+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:30 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:30.493797+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:30 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:30.518234+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:30 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:30.527019+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:30 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:30.540289+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:30 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.472419+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.726051+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.728716+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.745043+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.748559+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.781852+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.782231+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.785878+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.796615+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.828073+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.840938+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.843274+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.865117+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.896850+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.903753+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.910237+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.921784+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:31.921904+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:31 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.073419+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.073517+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.083406+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.101929+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.102813+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.167339+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.181514+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.190786+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.232732+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.246133+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.275983+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.285337+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.335349+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.338083+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.338232+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.340915+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.344735+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.351693+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.361962+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.391258+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.415820+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.466643+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.519068+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.586546+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.587864+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.598571+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.613787+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.621370+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.629320+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.633687+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.680694+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.698891+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.714094+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.735916+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.737643+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.744647+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.770101+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.781189+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.833836+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.850145+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.874424+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.884826+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.893566+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.899225+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.907737+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.920469+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.949097+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.957418+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:32.964281+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:32 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.062726+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.074979+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.088743+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.093275+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.119344+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.160101+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.178801+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.179587+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.185653+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.186110+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.214086+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.250182+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.305656+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.329511+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.345813+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.365243+01:00 ETCD1.AZ1 etcd[1075425]: saved database snapshot to disk [total bytes: 20623360]
2025-03-11T23:55:33.365373+01:00 ETCD1.AZ1 etcd[1075425]: successfully received and saved database snapshot [index: 5323014273, from: a38a9b279530ed1, raft message size: 735 kB, db size: 21 MB, took: 5.000034066s]
2025-03-11T23:55:33.365420+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:33 INFO: log [committed=5322828444, applied=5322828444, unstable.offset=5322828445, len(unstable.Entries)=0] starts to restore snapshot [index: 5323014273, term: 120823]
2025-03-11T23:55:33.365457+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:33 INFO: 38c2171bd506eaa switched to configuration voters=(255616051244461738 736525123079835345 5236873483412625109)
2025-03-11T23:55:33.365502+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:33 INFO: 38c2171bd506eaa [commit: 5323014273, lastindex: 5323014273, lastterm: 120823] restored snapshot [index: 5323014273, term: 120823]
2025-03-11T23:55:33.365529+01:00 ETCD1.AZ1 etcd[1075425]: raft2025/03/11 23:55:33 INFO: 38c2171bd506eaa [commit: 5323014273] restored snapshot [index: 5323014273, term: 120823]
2025-03-11T23:55:33.365796+01:00 ETCD1.AZ1 etcd[1075425]: applying snapshot at index 5322774018...
2025-03-11T23:55:33.368658+01:00 ETCD1.AZ1 etcd[1075425]: raft applied incoming snapshot at index 5323014273
2025-03-11T23:55:33.375335+01:00 ETCD1.AZ1 etcd[1075425]: recovering lessor...
2025-03-11T23:55:33.375800+01:00 ETCD1.AZ1 etcd[1075425]: finished recovering lessor
2025-03-11T23:55:33.375856+01:00 ETCD1.AZ1 etcd[1075425]: restoring mvcc store...
2025-03-11T23:55:33.375949+01:00 ETCD1.AZ1 etcd[1075425]: restore compact to 2912700256
2025-03-11T23:55:33.383817+01:00 ETCD1.AZ1 etcd[1075425]: finished restoring mvcc store
2025-03-11T23:55:33.383930+01:00 ETCD1.AZ1 etcd[1075425]: recovering alarms...
2025-03-11T23:55:33.383985+01:00 ETCD1.AZ1 etcd[1075425]: closing old backend...
2025-03-11T23:55:33.385969+01:00 ETCD1.AZ1 etcd[1075425]: finished recovering alarms
2025-03-11T23:55:33.386055+01:00 ETCD1.AZ1 etcd[1075425]: recovering auth store...
2025-03-11T23:55:33.386593+01:00 ETCD1.AZ1 etcd[1075425]: finished recovering auth store
2025-03-11T23:55:33.386669+01:00 ETCD1.AZ1 etcd[1075425]: recovering store v2...
2025-03-11T23:55:33.388369+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.390519+01:00 ETCD1.AZ1 etcd[1075425]: finished closing old backend
2025-03-11T23:55:33.396723+01:00 ETCD1.AZ1 etcd[1075425]: finished recovering store v2
2025-03-11T23:55:33.396810+01:00 ETCD1.AZ1 etcd[1075425]: recovering cluster configuration...
2025-03-11T23:55:33.396860+01:00 ETCD1.AZ1 etcd[1075425]: added member 48ad1cafe078d6d5 [https://ETCD1.AZ2:2380] to cluster 94125d5accdb5d68 from store
2025-03-11T23:55:33.396913+01:00 ETCD1.AZ1 etcd[1075425]: added member a38a9b279530ed1 [https://ETCD1.AZ3:2380] to cluster 94125d5accdb5d68 from store
2025-03-11T23:55:33.396963+01:00 ETCD1.AZ1 etcd[1075425]: added member 38c2171bd506eaa [https://ETCD1.AZ1:2380] to cluster 94125d5accdb5d68 from store
2025-03-11T23:55:33.397013+01:00 ETCD1.AZ1 etcd[1075425]: set the cluster version to 3.4 from store
2025-03-11T23:55:33.397059+01:00 ETCD1.AZ1 etcd[1075425]: finished recovering cluster configuration
2025-03-11T23:55:33.397111+01:00 ETCD1.AZ1 etcd[1075425]: removing old peers from network...
2025-03-11T23:55:33.397163+01:00 ETCD1.AZ1 etcd[1075425]: stopping peer a38a9b279530ed1...
2025-03-11T23:55:33.397736+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.398514+01:00 ETCD1.AZ1 etcd[1075425]: closed the TCP streaming connection with peer a38a9b279530ed1 (stream MsgApp v2 writer)
2025-03-11T23:55:33.398578+01:00 ETCD1.AZ1 etcd[1075425]: stopped streaming with peer a38a9b279530ed1 (writer)
2025-03-11T23:55:33.400076+01:00 ETCD1.AZ1 etcd[1075425]: closed the TCP streaming connection with peer a38a9b279530ed1 (stream Message writer)
2025-03-11T23:55:33.400132+01:00 ETCD1.AZ1 etcd[1075425]: stopped streaming with peer a38a9b279530ed1 (writer)
2025-03-11T23:55:33.400173+01:00 ETCD1.AZ1 etcd[1075425]: stopped HTTP pipelining with peer a38a9b279530ed1
2025-03-11T23:55:33.400321+01:00 ETCD1.AZ1 etcd[1075425]: lost the TCP streaming connection with peer a38a9b279530ed1 (stream MsgApp v2 reader)
2025-03-11T23:55:33.400373+01:00 ETCD1.AZ1 etcd[1075425]: failed to read a38a9b279530ed1 on stream MsgApp v2 (context canceled)
2025-03-11T23:55:33.400429+01:00 ETCD1.AZ1 etcd[1075425]: peer a38a9b279530ed1 became inactive (message send to peer failed)
2025-03-11T23:55:33.400474+01:00 ETCD1.AZ1 etcd[1075425]: stopped streaming with peer a38a9b279530ed1 (stream MsgApp v2 reader)
2025-03-11T23:55:33.400626+01:00 ETCD1.AZ1 etcd[1075425]: lost the TCP streaming connection with peer a38a9b279530ed1 (stream Message reader)
2025-03-11T23:55:33.400676+01:00 ETCD1.AZ1 etcd[1075425]: stopped streaming with peer a38a9b279530ed1 (stream Message reader)
2025-03-11T23:55:33.400721+01:00 ETCD1.AZ1 etcd[1075425]: stopped peer a38a9b279530ed1
2025-03-11T23:55:33.400769+01:00 ETCD1.AZ1 etcd[1075425]: removed peer a38a9b279530ed1
2025-03-11T23:55:33.400817+01:00 ETCD1.AZ1 etcd[1075425]: stopping peer 48ad1cafe078d6d5...
2025-03-11T23:55:33.402652+01:00 ETCD1.AZ1 etcd[1075425]: closed the TCP streaming connection with peer 48ad1cafe078d6d5 (stream MsgApp v2 writer)
2025-03-11T23:55:33.402737+01:00 ETCD1.AZ1 etcd[1075425]: stopped streaming with peer 48ad1cafe078d6d5 (writer)
2025-03-11T23:55:33.404185+01:00 ETCD1.AZ1 etcd[1075425]: closed the TCP streaming connection with peer 48ad1cafe078d6d5 (stream Message writer)
2025-03-11T23:55:33.404238+01:00 ETCD1.AZ1 etcd[1075425]: stopped streaming with peer 48ad1cafe078d6d5 (writer)
2025-03-11T23:55:33.404283+01:00 ETCD1.AZ1 etcd[1075425]: stopped HTTP pipelining with peer 48ad1cafe078d6d5
2025-03-11T23:55:33.404346+01:00 ETCD1.AZ1 etcd[1075425]: lost the TCP streaming connection with peer 48ad1cafe078d6d5 (stream MsgApp v2 reader)
2025-03-11T23:55:33.404399+01:00 ETCD1.AZ1 etcd[1075425]: failed to read 48ad1cafe078d6d5 on stream MsgApp v2 (context canceled)
2025-03-11T23:55:33.404446+01:00 ETCD1.AZ1 etcd[1075425]: peer 48ad1cafe078d6d5 became inactive (message send to peer failed)
2025-03-11T23:55:33.404484+01:00 ETCD1.AZ1 etcd[1075425]: stopped streaming with peer 48ad1cafe078d6d5 (stream MsgApp v2 reader)
2025-03-11T23:55:33.404533+01:00 ETCD1.AZ1 etcd[1075425]: lost the TCP streaming connection with peer 48ad1cafe078d6d5 (stream Message reader)
2025-03-11T23:55:33.404580+01:00 ETCD1.AZ1 etcd[1075425]: stopped streaming with peer 48ad1cafe078d6d5 (stream Message reader)
2025-03-11T23:55:33.404627+01:00 ETCD1.AZ1 etcd[1075425]: stopped peer 48ad1cafe078d6d5
2025-03-11T23:55:33.404670+01:00 ETCD1.AZ1 etcd[1075425]: removed peer 48ad1cafe078d6d5
2025-03-11T23:55:33.404713+01:00 ETCD1.AZ1 etcd[1075425]: finished removing old peers from network
2025-03-11T23:55:33.404756+01:00 ETCD1.AZ1 etcd[1075425]: adding peers from new cluster configuration into network...
2025-03-11T23:55:33.404890+01:00 ETCD1.AZ1 etcd[1075425]: starting peer a38a9b279530ed1...
2025-03-11T23:55:33.404937+01:00 ETCD1.AZ1 etcd[1075425]: started HTTP pipelining with peer a38a9b279530ed1
2025-03-11T23:55:33.405285+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.406380+01:00 ETCD1.AZ1 etcd[1075425]: started streaming with peer a38a9b279530ed1 (writer)
2025-03-11T23:55:33.407841+01:00 ETCD1.AZ1 etcd[1075425]: started streaming with peer a38a9b279530ed1 (writer)
2025-03-11T23:55:33.409729+01:00 ETCD1.AZ1 etcd[1075425]: started peer a38a9b279530ed1
2025-03-11T23:55:33.409814+01:00 ETCD1.AZ1 etcd[1075425]: started streaming with peer a38a9b279530ed1 (stream MsgApp v2 reader)
2025-03-11T23:55:33.409864+01:00 ETCD1.AZ1 etcd[1075425]: added peer a38a9b279530ed1
2025-03-11T23:55:33.409908+01:00 ETCD1.AZ1 etcd[1075425]: starting peer 48ad1cafe078d6d5...
2025-03-11T23:55:33.409956+01:00 ETCD1.AZ1 etcd[1075425]: started HTTP pipelining with peer 48ad1cafe078d6d5
2025-03-11T23:55:33.410032+01:00 ETCD1.AZ1 etcd[1075425]: started streaming with peer a38a9b279530ed1 (stream Message reader)
2025-03-11T23:55:33.411179+01:00 ETCD1.AZ1 etcd[1075425]: started streaming with peer 48ad1cafe078d6d5 (writer)
2025-03-11T23:55:33.412360+01:00 ETCD1.AZ1 etcd[1075425]: started streaming with peer 48ad1cafe078d6d5 (writer)
2025-03-11T23:55:33.415331+01:00 ETCD1.AZ1 etcd[1075425]: started peer 48ad1cafe078d6d5
2025-03-11T23:55:33.415434+01:00 ETCD1.AZ1 etcd[1075425]: started streaming with peer 48ad1cafe078d6d5 (stream MsgApp v2 reader)
2025-03-11T23:55:33.415503+01:00 ETCD1.AZ1 etcd[1075425]: started streaming with peer 48ad1cafe078d6d5 (stream Message reader)
2025-03-11T23:55:33.415597+01:00 ETCD1.AZ1 etcd[1075425]: added peer 48ad1cafe078d6d5
2025-03-11T23:55:33.416236+01:00 ETCD1.AZ1 etcd[1075425]: finished adding peers from new cluster configuration into network...
2025-03-11T23:55:33.416308+01:00 ETCD1.AZ1 etcd[1075425]: finished applying incoming snapshot at index 5323014273
2025-03-11T23:55:33.416359+01:00 ETCD1.AZ1 etcd[1075425]: failed to find member 48ad1cafe078d6d5 in cluster 94125d5accdb5d68
2025-03-11T23:55:33.416721+01:00 ETCD1.AZ1 etcd[1075425]: peer a38a9b279530ed1 became active
2025-03-11T23:55:33.416803+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer a38a9b279530ed1 (stream MsgApp v2 writer)
2025-03-11T23:55:33.416888+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer a38a9b279530ed1 (stream Message writer)
2025-03-11T23:55:33.417353+01:00 ETCD1.AZ1 etcd[1075425]: peer 48ad1cafe078d6d5 became active
2025-03-11T23:55:33.417433+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer 48ad1cafe078d6d5 (stream MsgApp v2 writer)
2025-03-11T23:55:33.418058+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer a38a9b279530ed1 (stream Message reader)
2025-03-11T23:55:33.418242+01:00 ETCD1.AZ1 etcd[1075425]: WARNING: 2025/03/11 23:55:33 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:33.422297+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer a38a9b279530ed1 (stream MsgApp v2 reader)
2025-03-11T23:55:33.425566+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer 48ad1cafe078d6d5 (stream MsgApp v2 reader)
2025-03-11T23:55:33.426368+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer 48ad1cafe078d6d5 (stream Message reader)
2025-03-11T23:55:33.508866+01:00 ETCD1.AZ1 etcd[1075425]: established a TCP streaming connection with peer 48ad1cafe078d6d5 (stream Message writer)




2025-03-11T23:54:00.316454+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:00.322771+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:00.322865+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:04.480189+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:05.316774+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:07.125935+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:07.126146+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:09.480193+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:10.317223+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:13.929346+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:13.929475+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:14.480621+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:15.318116+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:19.481080+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:20.318340+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:20.733262+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:20.733364+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:24.481538+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:25.318647+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:27.542046+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:27.542161+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:28.286188+01:00 ETCD1.AZ2 etcd[1017234]: Starting auto-compaction at revision 2912700256 (retention: 3 revisions)
2025-03-11T23:54:28.288394+01:00 ETCD1.AZ2 etcd[1017234]: store.index: compact 2912700256
2025-03-11T23:54:28.288500+01:00 ETCD1.AZ2 etcd[1017234]: Finished auto-compaction at revision 2912700256
2025-03-11T23:54:28.505210+01:00 ETCD1.AZ2 etcd[1017234]: finished scheduled compaction at 2912700256 (took 215.613241ms)
2025-03-11T23:54:29.481814+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:30.319353+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:34.345447+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:34.345587+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:34.481641+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:35.320335+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:39.482031+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:40.321058+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:41.147076+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:41.147235+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:44.482768+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:45.322138+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:47.948890+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:47.949101+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:49.482850+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:50.322986+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:54.483338+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:54.756722+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:54.756830+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:54:55.323232+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:59.483545+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:00.324031+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:01.559381+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:01.559530+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:04.483834+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:05.325083+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:08.361318+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:08.361452+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:09.483958+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:10.326229+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:14.484653+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:15.164236+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:15.164367+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:15.327087+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:19.485542+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:20.327527+01:00 ETCD1.AZ2 etcd[1017234]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:21.971849+01:00 ETCD1.AZ2 etcd[1017234]: failed to reach the peerURL(https://ETCD1.AZ1:2380) of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:21.971965+01:00 ETCD1.AZ2 etcd[1017234]: cannot get the version of member 38c2171bd506eaa (Get "https://ETCD1.AZ1:2380/version": dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:23.434965+01:00 ETCD1.AZ2 etcd[1017234]: peer 38c2171bd506eaa became active
2025-03-11T23:55:23.435117+01:00 ETCD1.AZ2 etcd[1017234]: established a TCP streaming connection with peer 38c2171bd506eaa (stream Message writer)
2025-03-11T23:55:23.850552+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:23 INFO: 48ad1cafe078d6d5 [logterm: 119522, index: 5323014247, vote: 48ad1cafe078d6d5] ignored MsgVote from 38c2171bd506eaa [logterm: 119522, index: 5322828444] at term 119522: lease is not expired (remaining ticks: 4)
2025-03-11T23:55:23.854495+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:23 INFO: 48ad1cafe078d6d5 [term: 119522] received a MsgAppResp message with higher term from 38c2171bd506eaa [term: 120821]
2025-03-11T23:55:23.854569+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:23 INFO: 48ad1cafe078d6d5 became follower at term 120821
2025-03-11T23:55:23.854600+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:23 INFO: raft.node: 48ad1cafe078d6d5 lost leader 48ad1cafe078d6d5 at term 120821
2025-03-11T23:55:23.854621+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:23 INFO: 48ad1cafe078d6d5 [term: 120821] ignored a MsgHeartbeatResp message with lower term from a38a9b279530ed1 [term: 119522]
2025-03-11T23:55:24.417263+01:00 ETCD1.AZ2 etcd[1017234]: failed to dial 38c2171bd506eaa on stream MsgApp v2 (dial tcp [ETCD1.AZ1]:2380: i/o timeout)
2025-03-11T23:55:24.417395+01:00 ETCD1.AZ2 etcd[1017234]: peer 38c2171bd506eaa became inactive (message send to peer failed)
2025-03-11T23:55:24.666018+01:00 ETCD1.AZ2 etcd[1017234]: peer 38c2171bd506eaa became active
2025-03-11T23:55:24.666209+01:00 ETCD1.AZ2 etcd[1017234]: established a TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 writer)
2025-03-11T23:55:24.735373+01:00 ETCD1.AZ2 etcd[1017234]: established a TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 reader)
2025-03-11T23:55:24.851388+01:00 ETCD1.AZ2 etcd[1017234]: established a TCP streaming connection with peer 38c2171bd506eaa (stream Message reader)
2025-03-11T23:55:25.825200+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:25 INFO: 48ad1cafe078d6d5 [term: 120821] received a MsgVote message with higher term from 38c2171bd506eaa [term: 120822]
2025-03-11T23:55:25.825293+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:25 INFO: 48ad1cafe078d6d5 became follower at term 120822
2025-03-11T23:55:25.825357+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:25 INFO: 48ad1cafe078d6d5 [logterm: 119522, index: 5323014247, vote: 0] rejected MsgVote from 38c2171bd506eaa [logterm: 119522, index: 5322828444] at term 120822
2025-03-11T23:55:25.929759+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:25 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.017795+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.238101+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.270210+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.529079+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.650531+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.818062+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.910259+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.152274+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.179187+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.186687+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.187214+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.200530+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.490568+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.536682+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.846389+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:28.137525+01:00 ETCD1.AZ2 etcd[1017234]: WARNING: 2025/03/11 23:55:28 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:28.343682+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:28 INFO: 48ad1cafe078d6d5 [term: 120822] received a MsgVote message with higher term from a38a9b279530ed1 [term: 120823]
2025-03-11T23:55:28.343768+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:28 INFO: 48ad1cafe078d6d5 became follower at term 120823
2025-03-11T23:55:28.343792+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:28 INFO: 48ad1cafe078d6d5 [logterm: 119522, index: 5323014247, vote: 0] cast MsgVote for a38a9b279530ed1 [logterm: 119522, index: 5323014247] at term 120823
2025-03-11T23:55:28.345359+01:00 ETCD1.AZ2 etcd[1017234]: raft2025/03/11 23:55:28 INFO: raft.node: 48ad1cafe078d6d5 elected leader a38a9b279530ed1 at term 120823
2025-03-11T23:55:33.400084+01:00 ETCD1.AZ2 etcd[1017234]: lost the TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 reader)
2025-03-11T23:55:33.401764+01:00 ETCD1.AZ2 etcd[1017234]: lost the TCP streaming connection with peer 38c2171bd506eaa (stream Message reader)
2025-03-11T23:55:33.415198+01:00 ETCD1.AZ2 etcd[1017234]: established a TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 reader)
2025-03-11T23:55:33.415309+01:00 ETCD1.AZ2 etcd[1017234]: failed to dial 38c2171bd506eaa on stream Message (peer 38c2171bd506eaa failed to find local node 48ad1cafe078d6d5)
2025-03-11T23:55:33.415365+01:00 ETCD1.AZ2 etcd[1017234]: peer 38c2171bd506eaa became inactive (message send to peer failed)
2025-03-11T23:55:33.423315+01:00 ETCD1.AZ2 etcd[1017234]: peer 38c2171bd506eaa became active
2025-03-11T23:55:33.423377+01:00 ETCD1.AZ2 etcd[1017234]: closed an existing TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 writer)
2025-03-11T23:55:33.423427+01:00 ETCD1.AZ2 etcd[1017234]: established a TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 writer)
2025-03-11T23:55:33.423473+01:00 ETCD1.AZ2 etcd[1017234]: closed an existing TCP streaming connection with peer 38c2171bd506eaa (stream Message writer)
2025-03-11T23:55:33.423517+01:00 ETCD1.AZ2 etcd[1017234]: established a TCP streaming connection with peer 38c2171bd506eaa (stream Message writer)
2025-03-11T23:55:33.507643+01:00 ETCD1.AZ2 etcd[1017234]: established a TCP streaming connection with peer 38c2171bd506eaa (stream Message reader)



2025-03-11T23:54:03.642710+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:03.991823+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:08.642973+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:08.991984+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:13.643894+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:13.993092+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:18.644185+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:18.993632+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:23.645008+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:23.994610+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:28.289054+01:00 ETCD1.AZ3 etcd[1025201]: store.index: compact 2912700256
2025-03-11T23:54:28.512614+01:00 ETCD1.AZ3 etcd[1025201]: finished scheduled compaction at 2912700256 (took 221.308601ms)
2025-03-11T23:54:28.645164+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:28.995401+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:33.646203+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:33.995950+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:38.646739+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:38.996831+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:43.647388+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:43.997835+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:48.648100+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:48.998318+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:49.279321+01:00 ETCD1.AZ3 etcd[1025201]: start to snapshot (applied: 5323012330, lastsnap: 5322912329)
2025-03-11T23:54:49.289621+01:00 ETCD1.AZ3 etcd[1025201]: saved snapshot at index 5323012330
2025-03-11T23:54:49.290134+01:00 ETCD1.AZ3 etcd[1025201]: compacted raft log at 5323007330
2025-03-11T23:54:53.648356+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:53.999289+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:58.649600+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:54:58.999760+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:03.650181+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:04.000869+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:08.508474+01:00 ETCD1.AZ3 etcd[1025201]: purged file /var/lib/etcd/member/snap/000000000001d2de-000000013d3f17c5.snap successfully
2025-03-11T23:55:08.650656+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:09.001226+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:13.652001+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:14.002020+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:18.652409+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:19.002434+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:23.020940+01:00 ETCD1.AZ3 etcd[1025201]: peer 38c2171bd506eaa became active
2025-03-11T23:55:23.021221+01:00 ETCD1.AZ3 etcd[1025201]: established a TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 reader)
2025-03-11T23:55:23.021282+01:00 ETCD1.AZ3 etcd[1025201]: established a TCP streaming connection with peer 38c2171bd506eaa (stream Message reader)
2025-03-11T23:55:23.112831+01:00 ETCD1.AZ3 etcd[1025201]: established a TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 writer)
2025-03-11T23:55:23.652353+01:00 ETCD1.AZ3 etcd[1025201]: health check for peer 38c2171bd506eaa could not connect: dial tcp [ETCD1.AZ1]:2380: i/o timeout
2025-03-11T23:55:23.852470+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:23 INFO: a38a9b279530ed1 [logterm: 119522, index: 5323014247, vote: 48ad1cafe078d6d5] ignored MsgVote from 38c2171bd506eaa [logterm: 119522, index: 5322828444] at term 119522: lease is not expired (remaining ticks: 5)
2025-03-11T23:55:24.907381+01:00 ETCD1.AZ3 etcd[1025201]: established a TCP streaming connection with peer 38c2171bd506eaa (stream Message writer)
2025-03-11T23:55:25.824974+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:25 INFO: a38a9b279530ed1 [term: 119522] received a MsgVote message with higher term from 38c2171bd506eaa [term: 120822]
2025-03-11T23:55:25.825132+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:25 INFO: a38a9b279530ed1 became follower at term 120822
2025-03-11T23:55:25.825192+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:25 INFO: a38a9b279530ed1 [logterm: 119522, index: 5323014247, vote: 0] rejected MsgVote from 38c2171bd506eaa [logterm: 119522, index: 5322828444] at term 120822
2025-03-11T23:55:25.825246+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:25 INFO: raft.node: a38a9b279530ed1 lost leader 48ad1cafe078d6d5 at term 120822
2025-03-11T23:55:26.102523+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.301910+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.330494+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.389857+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.543067+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.548039+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.681717+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:26.899882+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:26 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.023671+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.132601+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.194218+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:27.795127+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:27 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:28.133863+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:28 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:28.272315+01:00 ETCD1.AZ3 etcd[1025201]: WARNING: 2025/03/11 23:55:28 [core] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2025-03-11T23:55:28.341969+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: a38a9b279530ed1 is starting a new election at term 120822
2025-03-11T23:55:28.342103+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: a38a9b279530ed1 became candidate at term 120823
2025-03-11T23:55:28.342150+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: a38a9b279530ed1 received MsgVoteResp from a38a9b279530ed1 at term 120823
2025-03-11T23:55:28.342183+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: a38a9b279530ed1 [logterm: 119522, index: 5323014247] sent MsgVote request to 38c2171bd506eaa at term 120823
2025-03-11T23:55:28.342213+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: a38a9b279530ed1 [logterm: 119522, index: 5323014247] sent MsgVote request to 48ad1cafe078d6d5 at term 120823
2025-03-11T23:55:28.344344+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: a38a9b279530ed1 received MsgVoteResp from 48ad1cafe078d6d5 at term 120823
2025-03-11T23:55:28.344414+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: a38a9b279530ed1 has received 2 MsgVoteResp votes and 0 vote rejections
2025-03-11T23:55:28.344457+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: a38a9b279530ed1 became leader at term 120823
2025-03-11T23:55:28.344487+01:00 ETCD1.AZ3 etcd[1025201]: raft2025/03/11 23:55:28 INFO: raft.node: a38a9b279530ed1 elected leader a38a9b279530ed1 at term 120823
2025-03-11T23:55:28.359326+01:00 ETCD1.AZ3 etcd[1025201]: start to send database snapshot [index: 5323014273, to 38c2171bd506eaa, size 21 MB]...
2025-03-11T23:55:33.317376+01:00 ETCD1.AZ3 etcd[1025201]: wrote database snapshot out [total bytes: 20623360]
2025-03-11T23:55:33.364514+01:00 ETCD1.AZ3 etcd[1025201]: database snapshot [index: 5323014273, to: 38c2171bd506eaa] sent out successfully
2025-03-11T23:55:33.396114+01:00 ETCD1.AZ3 etcd[1025201]: lost the TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 reader)
2025-03-11T23:55:33.397588+01:00 ETCD1.AZ3 etcd[1025201]: lost the TCP streaming connection with peer 38c2171bd506eaa (stream Message reader)
2025-03-11T23:55:33.414556+01:00 ETCD1.AZ3 etcd[1025201]: established a TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 reader)
2025-03-11T23:55:33.415700+01:00 ETCD1.AZ3 etcd[1025201]: established a TCP streaming connection with peer 38c2171bd506eaa (stream Message reader)
2025-03-11T23:55:33.415877+01:00 ETCD1.AZ3 etcd[1025201]: failed to write 38c2171bd506eaa on stream MsgApp v2 (write tcp ETCD1.AZ3:2380->ETCD1.AZ1:38150: write: broken pipe)
2025-03-11T23:55:33.415939+01:00 ETCD1.AZ3 etcd[1025201]: peer 38c2171bd506eaa became inactive (message send to peer failed)
2025-03-11T23:55:33.416576+01:00 ETCD1.AZ3 etcd[1025201]: peer 38c2171bd506eaa became active
2025-03-11T23:55:33.416650+01:00 ETCD1.AZ3 etcd[1025201]: closed an existing TCP streaming connection with peer 38c2171bd506eaa (stream Message writer)
2025-03-11T23:55:33.416700+01:00 ETCD1.AZ3 etcd[1025201]: established a TCP streaming connection with peer 38c2171bd506eaa (stream Message writer)
2025-03-11T23:55:33.417469+01:00 ETCD1.AZ3 etcd[1025201]: lost the TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 writer)
2025-03-11T23:55:33.418735+01:00 ETCD1.AZ3 etcd[1025201]: established a TCP streaming connection with peer 38c2171bd506eaa (stream MsgApp v2 writer)
```
I added the logs from patroni and etcd.

ETCD doesn't show anything suspicious.



Patch:
diff --git a/patroni/dcs/etcd.py b/patroni/dcs/etcd.py
index a5b55e7e3..1d8911d9e 100644
--- a/patroni/dcs/etcd.py
+++ b/patroni/dcs/etcd.py
@@ -41,6 +41,10 @@ class EtcdRaftInternal(etcd.EtcdException):
     """Raft Internal Error"""
 
 
+class StaleEtcdNode(Exception):
+    """Node is stale (raft term is older than previous known)."""
+
+
 class EtcdError(DCSError):
     pass
 
@@ -102,6 +106,8 @@ class AbstractEtcdClientWithFailover(abc.ABC, etcd.Client):
     ERROR_CLS: Type[Exception]
 
     def __init__(self, config: Dict[str, Any], dns_resolver: DnsCachingResolver, cache_ttl: int = 300) -> None:
+        self._cluster_id = None
+        self._raft_term = 0
         self._dns_resolver = dns_resolver
         self.set_machines_cache_ttl(cache_ttl)
         self._machines_cache_updated = 0
@@ -119,6 +125,32 @@ def __init__(self, config: Dict[str, Any], dns_resolver: DnsCachingResolver, cac
         self._read_options.add('retry')
         self._del_conditions.add('retry')
 
+    def _check_cluster_raft_term(self, cluster_id: Optional[str], value: Union[None, str, int]) -> None:
+        """Check that observed Raft Term in Etcd cluster is increasing.
+
+        If we observe that the new value is smaller than the previously known one, it could be an
+        indicator that we connected to a stale node and should switch to some other node.
+        However, we need to reset the memorized value when we notice that Cluster ID changed.
+        """
+        if not (cluster_id and value):
+            return
+
+        if self._cluster_id and self._cluster_id != cluster_id:
+            logger.warning('Etcd Cluster ID changed from %s to %s', self._cluster_id, cluster_id)
+            self._raft_term = 0
+        self._cluster_id = cluster_id
+
+        try:
+            raft_term = int(value)
+        except Exception:
+            return
+
+        if raft_term < self._raft_term:
+            logger.warning('Connected to Etcd node with term %d. Old known term %d. Switching to another node.',
+                           raft_term, self._raft_term)
+            raise StaleEtcdNode
+        self._raft_term = raft_term
+
     def _calculate_timeouts(self, etcd_nodes: int, timeout: Optional[float] = None) -> Tuple[int, float, int]:
         """Calculate a request timeout and number of retries per single etcd node.
         In case if the timeout per node is too small (less than one second) we will reduce the number of nodes.
@@ -227,7 +259,7 @@ def set_read_timeout(self, timeout: float) -> None:
     def _do_http_request(self, retry: Optional[Retry], machines_cache: List[str],
                          request_executor: Callable[..., urllib3.response.HTTPResponse],
                          method: str, path: str, fields: Optional[Dict[str, Any]] = None,
-                         **kwargs: Any) -> urllib3.response.HTTPResponse:
+                         **kwargs: Any) -> Any:
         is_watch_request = isinstance(fields, dict) and fields.get('wait') == 'true'
         if fields is not None:
             kwargs['fields'] = fields
@@ -241,8 +273,8 @@ def _do_http_request(self, retry: Optional[Retry], machines_cache: List[str],
                 if some_request_failed:
                     self.set_base_uri(base_uri)
                     self._refresh_machines_cache()
-                return response
-            except (HTTPError, HTTPException, socket.error, socket.timeout) as e:
+                return self._handle_server_response(response)
+            except (HTTPError, HTTPException, socket.error, socket.timeout, StaleEtcdNode) as e:
                 self.http.clear()
                 if not retry:
                     if len(machines_cache) == 1:
@@ -285,8 +317,7 @@ def api_execute(self, path: str, method: str, params: Optional[Dict[str, Any]] =
 
         while True:
             try:
-                response = self._do_http_request(retry, machines_cache, request_executor, method, path, **kwargs)
-                return self._handle_server_response(response)
+                return self._do_http_request(retry, machines_cache, request_executor, method, path, **kwargs)
             except etcd.EtcdWatchTimedOut:
                 raise
             except etcd.EtcdConnectionFailed as ex:
@@ -463,6 +494,10 @@ def __del__(self) -> None:
     def _prepare_get_members(self, etcd_nodes: int) -> Dict[str, Any]:
         return self._prepare_common_parameters(etcd_nodes)
 
+    def _handle_server_response(self, response: urllib3.response.HTTPResponse) -> Any:
+        self._check_cluster_raft_term(response.headers.get('x-etcd-cluster-id'), response.headers.get('x-raft-term'))
+        return super(EtcdClient, self)._handle_server_response(response)
+
     def _get_members(self, base_uri: str, **kwargs: Any) -> List[str]:
         response = self.http.request(self._MGET, base_uri + self.version_prefix + '/machines', **kwargs)
         data = self._handle_server_response(response).data.decode('utf-8')
diff --git a/patroni/dcs/etcd3.py b/patroni/dcs/etcd3.py
index 4688cc210..9f790fa92 100644
--- a/patroni/dcs/etcd3.py
+++ b/patroni/dcs/etcd3.py
@@ -18,6 +18,7 @@
 
 from urllib3.exceptions import ProtocolError, ReadTimeoutError
 
+from ..collections import EMPTY_DICT
 from ..exceptions import DCSError, PatroniException
 from ..postgresql.mpp import AbstractMPP
 from ..utils import deep_compare, enable_keepalive, iter_response_objects, RetryFailedError, USER_AGENT
@@ -240,6 +241,10 @@ def _handle_server_response(self, response: urllib3.response.HTTPResponse) -> Di
         try:
             data = data.decode('utf-8')
             ret: Dict[str, Any] = json.loads(data)
+
+            header = ret.get('header', EMPTY_DICT)
+            self._check_cluster_raft_term(header.get('cluster_id'), header.get('raft_term'))
+
             if response.status < 400:
                 return ret
         except (TypeError, ValueError, UnicodeError) as e:


Test patch:
diff --git a/tests/test_etcd.py b/tests/test_etcd.py
index eb88b6867..d897ec8e9 100644
--- a/tests/test_etcd.py
+++ b/tests/test_etcd.py
@@ -117,6 +117,12 @@ def http_request(method, url, **kwargs):
         ret.content = 'http://localhost:2379,http://localhost:4001'
     elif url == 'http://localhost:4001/v2/machines':
         ret.content = ''
+    elif url == 'http://localhost:4001/term/':
+        ret.headers['x-etcd-cluster-id'] = 'a'
+        ret.headers['x-raft-term'] = '1'
+    elif url == 'http://localhost:2379/term/':
+        ret.headers['x-etcd-cluster-id'] = 'b'
+        ret.headers['x-raft-term'] = 'x'
     elif url != 'http://localhost:2379/':
         raise socket.error
     return ret
@@ -165,6 +171,23 @@ def test_machines(self):
         except Exception:
             self.assertIsNone(machines)
 
+    def test__check_cluster_raft_term(self):
+        self.client._raft_term = 2
+        self.client._base_uri = 'http://localhost:4001/term'
+        self.client._machines_cache = [self.client._base_uri, 'http://localhost:2379/term']
+        rtry = Retry(deadline=10, max_delay=1, max_tries=-1, retry_exceptions=(etcd.EtcdLeaderElectionInProgress,))
+        with patch('patroni.dcs.etcd.logger.warning') as mock_logger:
+            rtry(self.client.api_execute, '/', 'POST', timeout=0, params={'retry': rtry})
+            self.assertEqual(mock_logger.call_args_list[0][0],
+                             ('Connected to Etcd node with term %d. Old known term %d. Switching to another node.',
+                              1, 2))
+            self.assertEqual(mock_logger.call_args_list[1][0], ('Etcd Cluster ID changed from %s to %s', 'a', 'b'))
+        self.client._base_uri = self.client._machines_cache[0]
+        with patch('patroni.dcs.etcd.logger.warning') as mock_logger:
+            rtry(self.client.api_execute, '/', 'POST', timeout=0, params={'retry': rtry})
+            self.assertEqual(mock_logger.call_args[0], ('Etcd Cluster ID changed from %s to %s', 'b', 'a'))
+
+    @patch('time.sleep', Mock())
     @patch.object(EtcdClient, '_get_machines_list',
                   Mock(return_value=['http://localhost:4001', 'http://localhost:2379']))
     def test_api_execute(self):