Repository: patroni/patroni

Issue:
Error on first switchover with message: Invalid auth token
### What happened?

We encounter a problem with the following setup:
2 node postgresql 16 cluster running on Ubuntu 22.04 with patroni 3.2.2
3 node etcd cluster docker deployment (initially 3.5.9, upgraded to 3.5.12 today)
All nodes (db cluster + etcd cluster) are on the same network

Postgresql cluster runs fine, but a switchover initially fails. When we run the switchover a second time, it works fine.
After the successful switchover, we see the same behavior on the new primary node when we try to switchover to the new replica node.

We do not see this behavior on our postgresql 12 clusters with the same patroni and etcd versions.

In the configuration files below, I replaced node names, paths, IPs, etc.

### How can we reproduce it (as minimally and precisely as possible)?

setup etcd cluster (enable authentication, add role patroni with password)
setup postgresql 16 cluster
run patronictl switchover


### What did you expect to happen?

See successful switchover the first time you run the switchover command.

### Patroni/PostgreSQL/DCS version

-   Patroni version: 3.2.2
-   PostgreSQL version: 16.2-1.pgdg22.04+1
-   DCS (and its version): etcd 3.5.12 (did an update from 3.5.9 yesterday to see, if this helps)
-   python3-etcd (added after initial post): 0.4.5-2 

### Patroni configuration file

```yaml
scope: "16-psdb12"
namespace: "/patroni/"
name: psdb1

restapi:
  listen: xxx.xxx.xxx.xxx:8443
  connect_address: xxx.xxx.xxx.xxx:8443
  authentication:
    username: patroni
    password: ""
  certfile: /pathxxx/xxxx.pem
  keyfile: /pathxxx/xxxx.key
  ciphers: ECDHE+AESGCM+AES256:ECDHE+CHACHA20:ECDHE+AESGCM+AES128:!SHA:!AESCCM:!SSLv1:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1

bootstrap:
  method: pg_createcluster
  pg_createcluster:
    command: /usr/share/patroni/pg_createcluster_patroni
  post_init: /etc/patroni/post_init.sh
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    #synchronous_mode: false
    #synchronous_mode_strict: 
    postgresql:
      use_pg_rewind: true
      remove_data_directory_on_rewind_failure: true
      remove_data_directory_on_diverged_timelines: false
      parameters:
        unix_socket_directories: '/var/run/postgresql/'
        logging_collector: 'on'
        log_filename: 'postgresql-%Y-%m-%d_%H%M%S.log'
        shared_buffers: 7864320kB
        effective_cache_size: 11796480kB
        maintenance_work_mem: 983040kB
        wal_buffers: 16384kB
        work_mem: 122880kB
        autovacuum_work_mem: 16384kB
        huge_pages: 'on'
        max_parallel_maintenance_workers: 2
        max_worker_processes: 4
        max_parallel_workers_per_gather: 2
        max_parallel_workers: 4
        max_prepared_transactions: 16
        max_connections: 16
        log_directory: '/pathxxx/16/xxxxx/pg_log'
        effective_io_concurrency: '1'
        random_page_cost: '4'
        cpu_tuple_cost: '0.03'
        archive_mode: 'on'
        archive_command: 'rsync -a %p xxx@xxx.xxx.xxx.xxx:/xxx/xxx/xxxx/xxxx/xxxxxx/%f'
        jit: 'on'
        synchronous_commit: 'local'
        synchronous_standby_names: '*'
        wal_keep_size: 5000MB
        max_wal_senders: 10
        max_replication_slots: 10
        ssl: 'on'
        ssl_cert_file: '/etc/patroni/psdb102.petafuel.intern.pem'
        ssl_ciphers: 'ECDHE+AESGCM:ECDHE+CHACHA20:ECDHE+AES256:ECDHE+AES128:!SHA:!AESCCM'
        ssl_ecdh_curve: 'secp384r1'
        ssl_key_file: '/etc/patroni/psdb102.petafuel.intern.key'
        ssl_min_protocol_version: tlsv1.2
        ssl_prefer_server_ciphers: 'true'
      pg_hba:
      - local  all  postgres  ident
      - local  all  all       scram-sha-256
      - hostssl  replication  replication  0.0.0.0/0  scram-sha-256
      - hostssl  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
      - hostssl  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
      - host  replication  replication  0.0.0.0/0  scram-sha-256
      - host  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
      - host  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
      - host  all  all  0.0.0.0/0  scram-sha-256
    slots:
      standby_cluster:
        type: physical

postgresql:
  basebackup:
  - waldir: /var/lib/pg/16/psdb12/pg_wal
  use_pg_rewind: true
  remove_data_directory_on_rewind_failure: true
  remove_data_directory_on_diverged_timelines: false

  listen: "0.0.0.0"
  connect_address: xxx.xxx.xxx.xxx:5432
  use_unix_socket: true
  data_dir: /var/lib/postgresql/16/psdb12
  bin_dir: /usr/lib/postgresql/16/bin
  config_dir: /etc/postgresql/16/psdb12
  pgpass: /var/lib/postgresql/16-psdb12.pgpass
  authentication:
    replication:
      username: replication
      password: ''
      sslrootcert: '/etc/ssl/certs/ca-certificates.crt'
      sslmode: verify-full
    superuser:
      username: postgres
      password: ''
    rewind:
      username: rewind
      password: ''
      sslrootcert: '/etc/ssl/certs/ca-certificates.crt'
      sslmode: verify-full
  parameters:
    unix_socket_directories: '/var/run/postgresql/'
    logging_collector: 'on'
    log_filename: 'postgresql-%Y-%m-%d_%H%M%S.log'
    shared_buffers: 7864320kB
    effective_cache_size: 11796480kB
    maintenance_work_mem: 983040kB
    wal_buffers: 16384kB
    work_mem: 122880kB
    autovacuum_work_mem: 16384kB
    huge_pages: 'on'
    max_parallel_maintenance_workers: 2
    max_worker_processes: 4
    max_parallel_workers_per_gather: 2
    max_parallel_workers: 4
    max_prepared_transactions: 16
    max_connections: 16
    log_directory: '/pathxxx/16/xxxxx/pg_log'
    archive_mode: 'on'
    archive_command: 'rsync -a %p xxx@xxx.xxx.xxx.xxx:/xxx/xxx/xxxx/xxxx/xxxxxx/%f'
    jit: 'on'
    synchronous_commit: 'local'
    synchronous_standby_names: '*'
    ssl: 'on'
    ssl_cert_file: '/pathxxx/xxxx.pem'
    ssl_ciphers: 'ECDHE+AESGCM:ECDHE+CHACHA20:ECDHE+AES256:ECDHE+AES128:!SHA:!AESCCM'
    ssl_ecdh_curve: 'secp384r1'
    ssl_key_file: '/pathxxx/xxxx.key'
    ssl_min_protocol_version: tlsv1.2
    ssl_prefer_server_ciphers: 'true'
  pg_hba:
  - local  all  postgres  ident
  - local  all  all       scram-sha-256
  - hostssl  replication  replication  0.0.0.0/0  scram-sha-256
  - hostssl  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
  - hostssl  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
  - host  replication  replication  0.0.0.0/0  scram-sha-256
  - host  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
  - host  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
  - host  all  all  0.0.0.0/0  scram-sha-256

watchdog:
  mode: off

tags:
  nofailover: false
  noloadbalance: false
  clonefrom: true
  nosync: false
```


### patronictl show-config

```yaml
loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    archive_command: rsync -a %p xxx@xxx.xxx.xxx.xxx:/xxx/xxx/xxxx/xxxx/xxxxxx/%f
    archive_mode: 'on'
    autovacuum_work_mem: 16384kB
    cpu_tuple_cost: '0.03'
    effective_cache_size: 11796480kB
    effective_io_concurrency: '1'
    huge_pages: 'on'
    jit: 'on'
    log_directory: /pathxxx/16/xxxxx/pg_log
    log_filename: postgresql-%Y-%m-%d_%H%M%S.log
    logging_collector: 'on'
    maintenance_work_mem: 983040kB
    max_connections: 16
    max_parallel_maintenance_workers: 2
    max_parallel_workers: 4
    max_parallel_workers_per_gather: 2
    max_prepared_transactions: 16
    max_replication_slots: 10
    max_wal_senders: 10
    max_worker_processes: 4
    random_page_cost: '4'
    shared_buffers: 7864320kB
    ssl: 'on'
    ssl_cert_file: /pathxxx/xxxx.pem
    ssl_ciphers: ECDHE+AESGCM:ECDHE+CHACHA20:ECDHE+AES256:ECDHE+AES128:!SHA:!AESCCM
    ssl_ecdh_curve: secp384r1
    ssl_key_file: /pathxxx/xxxx.key
    ssl_min_protocol_version: tlsv1.2
    ssl_prefer_server_ciphers: 'true'
    synchronous_commit: local
    synchronous_standby_names: '*'
    unix_socket_directories: /var/run/postgresql/
    wal_buffers: 16384kB
    wal_keep_size: 5000MB
    work_mem: 122880kB
  pg_hba:
  - local  all  postgres  ident
  - local  all  all       scram-sha-256
  - hostssl  replication  replication  0.0.0.0/0  scram-sha-256
  - hostssl  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
  - hostssl  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
  - host  replication  replication  0.0.0.0/0  scram-sha-256
  - host  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
  - host  postgres  healthcheck  xxx.xxx.xxx.xxx/32  trust
  - host  all  all  0.0.0.0/0  scram-sha-256
  remove_data_directory_on_diverged_timelines: false
  remove_data_directory_on_rewind_failure: true
  use_pg_rewind: true
retry_timeout: 10
slots:
  standby_cluster:
    type: physical
ttl: 30
```


### Patroni log files

```shell
first switchover:
Feb 20 10:25:00 psdb1 patroni[5475]: 2024-02-20 10:25:00,795 INFO: no action. I am (psdb1), the leader with the lock
Feb 20 10:25:01 psdb1 CRON[175523]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Feb 20 10:25:10 psdb1 patroni[5475]: 2024-02-20 10:25:10,790 INFO: no action. I am (psdb1), the leader with the lock
Feb 20 10:25:17 psdb1 patroni[5475]: 2024-02-20 10:25:17,709 INFO: received switchover request with leader=psdb1 candidate=psdb2 scheduled_at=None
Feb 20 10:25:17 psdb1 patroni[5475]: 2024-02-20 10:25:17,806 INFO: Got response from psdb2 https://xxx.xxx.xxx.xxx:8443/patroni: {"state": "running", "postmaster_start_time": "2024-02-20 09:55:42.06
2019+01:00", "role": "replica", "server_version": 160002, "xlog": {"received_location": 167772600, "replayed_location": 167772600, "replayed_timestamp": null, "paused": false}, "timeline": 6, "replication
_state": "streaming", "dcs_last_seen": 1708421111, "tags": {"clonefrom": true}, "database_system_identifier": "7336171685209230919", "patroni": {"version": "3.2.2", "scope": "16-psdb12", "name": "psdb2"}}
Feb 20 10:25:17 psdb1 patroni[5475]: 2024-02-20 10:25:17,852 ERROR: Invalid auth token: xngTQztELWWOPFCy.255340
Feb 20 10:25:17 psdb1 patroni[5475]: 2024-02-20 10:25:17,853 ERROR:
Feb 20 10:25:17 psdb1 patroni[5475]: Traceback (most recent call last):
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd.py", line 511, in handle_etcd_exceptions
Feb 20 10:25:17 psdb1 patroni[5475]:     retval = func(self, *args, **kwargs)
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd3.py", line 880, in set_failover_value
Feb 20 10:25:17 psdb1 patroni[5475]:     return bool(self._client.put(self.failover_path, value, mod_revision=version))
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd3.py", line 208, in wrapper
Feb 20 10:25:17 psdb1 patroni[5475]:     return self.handle_auth_errors(func, *args, **kwargs)
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd3.py", line 352, in handle_auth_errors
Feb 20 10:25:17 psdb1 patroni[5475]:     raise exc
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd3.py", line 334, in handle_auth_errors
Feb 20 10:25:17 psdb1 patroni[5475]:     return func(self, *args, retry=retry, **kwargs)
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd3.py", line 392, in put
Feb 20 10:25:17 psdb1 patroni[5475]:     return self.call_rpc('/kv/put', fields, retry)
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd3.py", line 634, in call_rpc
Feb 20 10:25:17 psdb1 patroni[5475]:     ret = super(PatroniEtcd3Client, self).call_rpc(method, fields, retry)
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd3.py", line 294, in call_rpc
Feb 20 10:25:17 psdb1 patroni[5475]:     return self.api_execute(self.version_prefix + method, self._MPOST, fields)
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd.py", line 283, in api_execute
Feb 20 10:25:17 psdb1 patroni[5475]:     return self._handle_server_response(response)
Feb 20 10:25:17 psdb1 patroni[5475]:   File "/usr/lib/python3/dist-packages/patroni/dcs/etcd3.py", line 252, in _handle_server_response
Feb 20 10:25:17 psdb1 patroni[5475]:     raise _raise_for_data(ret or data, response.status)
Feb 20 10:25:17 psdb1 patroni[5475]: patroni.dcs.etcd3.InvalidAuthToken: <InvalidAuthToken error: 'etcdserver: invalid auth token', code: 16>
Feb 20 10:25:20 psdb1 patroni[5475]: 2024-02-20 10:25:20,757 INFO: no action. I am (psdb1), the leader with the lock

second switchover:
Feb 20 10:32:40 psdb1 patroni[5475]: 2024-02-20 10:32:40,794 INFO: no action. I am (psdb1), the leader with the lock
Feb 20 10:32:49 psdb1 patroni[5475]: 2024-02-20 10:32:49,193 INFO: received switchover request with leader=psdb1 candidate=psdb2 scheduled_at=None
Feb 20 10:32:49 psdb1 patroni[5475]: 2024-02-20 10:32:49,302 INFO: Got response from psdb2 https://xxx.xxx.xxx.xxx:8443/patroni: {"state": "running", "postmaster_start_time": "2024-02-20 09:55:42.062019+01:00", "role": "replica", "server_version": 160002, "xlog": {"received_location": 167772600, "replayed_location": 167772600, "replayed_timestamp": null, "paused": false}, "timeline": 6, "replication_state": "streaming", "dcs_last_seen": 1708421561, "tags": {"clonefrom": true}, "database_system_identifier": "7336171685209230919", "patroni": {"version": "3.2.2", "scope": "16-psdb12", "name": "psdb2"}}
Feb 20 10:32:49 psdb1 patroni[5475]: 2024-02-20 10:32:49,303 INFO: Trying to authenticate on Etcd...
Feb 20 10:32:49 psdb1 patroni[5475]: 2024-02-20 10:32:49,570 INFO: Got response from psdb2 https://xxx.xxx.xxx.xxx:8443/patroni: {"state": "running", "postmaster_start_time": "2024-02-20 09:55:42.062019+01:00", "role": "replica", "server_version": 160002, "xlog": {"received_location": 167772600, "replayed_location": 167772600, "replayed_timestamp": null, "paused": false}, "timeline": 6, "replication_state": "streaming", "dcs_last_seen": 1708421561, "tags": {"clonefrom": true}, "database_system_identifier": "7336171685209230919", "patroni": {"version": "3.2.2", "scope": "16-psdb12", "name": "psdb2"}}
Feb 20 10:32:49 psdb1 patroni[5475]: 2024-02-20 10:32:49,415 INFO: Lock owner: psdb1; I am psdb1
Feb 20 10:32:49 psdb1 patroni[5475]: 2024-02-20 10:32:49,619 INFO: switchover: demoting myself
Feb 20 10:32:49 psdb1 patroni[5475]: 2024-02-20 10:32:49,619 INFO: Demoting self (graceful)
Feb 20 10:32:50 psdb1 patroni[5475]: 2024-02-20 10:32:50,810 INFO: Leader key released
Feb 20 10:32:50 psdb1 patroni[5475]: 2024-02-20 10:32:50,811 INFO: Lock owner: None; I am psdb1
Feb 20 10:32:50 psdb1 patroni[5475]: 2024-02-20 10:32:50,811 INFO: not healthy enough for leader race
Feb 20 10:32:50 psdb1 patroni[5475]: 2024-02-20 10:32:50,811 INFO: switchover: demote in progress
Feb 20 10:32:51 psdb1 patroni[5475]: 2024-02-20 10:32:51,022 INFO: Lock owner: psdb2; I am psdb1
Feb 20 10:32:51 psdb1 patroni[5475]: 2024-02-20 10:32:51,022 INFO: switchover: demote in progress
Feb 20 10:32:52 psdb1 patroni[5475]: 2024-02-20 10:32:52,214 INFO: Lock owner: psdb2; I am psdb1
Feb 20 10:32:52 psdb1 patroni[5475]: 2024-02-20 10:32:52,215 INFO: switchover: demote in progress


Feb 20 10:33:02 psdb1 patroni[5475]: 2024-02-20 10:33:02,315 INFO: no action. I am (psdb1), a secondary, and following a leader (psdb2)
```


### PostgreSQL log files

```shell
Please see patroni Log Files. No relevant information in postgresql log.
```


### Have you tried to use GitHub issue search?

- [X] Yes

### Anything else we need to know?

We use environment files for patroni-etcd and secrets (replaced sensitive data with xxx):
etcd-environment:
```
PATRONI_ETCD3_HOSTS=xxx.xxx.xxx.xxx:2379,xxx.xxx.xxx.xxx:2379,xxx.xxx.xxx.xxx:2379
PATRONI_ETCD3_PROTOCOL=https
PATRONI_ETCD3_USERNAME=patroni
PATRONI_RESTAPI_USERNAME=patroni
PATRONI_postgres_exporter_OPTIONS='createrole'

```
secrets-environment:
```
PATRONI_postgres_exporter_PASSWORD=xxx
PATRONI_ETCD3_PASSWORD=xxxx
PATRONI_REPLICATION_PASSWORD=xxxxx
PATRONI_RESTAPI_PASSWORD=xxxxxx
PATRONI_REWIND_PASSWORD=xxxxxxx
PATRONI_SUPERUSER_PASSWORD=xxxxxxxx
```


Discussion:

Patch:
diff --git a/patroni/dcs/consul.py b/patroni/dcs/consul.py
index bbfd24536..c6747d060 100644
--- a/patroni/dcs/consul.py
+++ b/patroni/dcs/consul.py
@@ -577,14 +577,17 @@ def _do_attempt_to_acquire_leader(self, retry: Retry) -> bool:
         try:
             return retry(self._client.kv.put, self.leader_path, self._name, acquire=self._session)
         except InvalidSession:
-            logger.error('Our session disappeared from Consul. Will try to get a new one and retry attempt')
             self._session = None
-            retry.ensure_deadline(0)
+
+            if not retry.ensure_deadline(0):
+                logger.error('Our session disappeared from Consul. Deadline exceeded, giving up')
+                return False
+
+            logger.error('Our session disappeared from Consul. Will try to get a new one and retry attempt')
 
             retry(self._do_refresh_session)
 
             retry.ensure_deadline(1, ConsulError('_do_attempt_to_acquire_leader timeout'))
-
             return retry(self._client.kv.put, self.leader_path, self._name, acquire=self._session)
 
     @catch_return_false_exception
diff --git a/patroni/dcs/etcd3.py b/patroni/dcs/etcd3.py
index 946f9318e..4a840c7f3 100644
--- a/patroni/dcs/etcd3.py
+++ b/patroni/dcs/etcd3.py
@@ -198,12 +198,6 @@ def build_range_request(key: str, range_end: Union[bytes, str, None] = None) ->
     return fields
 
 
-class ReAuthenticateMode(IntEnum):
-    NOT_REQUIRED = 0
-    REQUIRED = 1
-    WITHOUT_WATCHER_RESTART = 2
-
-
 def _handle_auth_errors(func: Callable[..., Any]) -> Any:
     def wrapper(self: 'Etcd3Client', *args: Any, **kwargs: Any) -> Any:
         return self.handle_auth_errors(func, *args, **kwargs)
@@ -215,7 +209,7 @@ class Etcd3Client(AbstractEtcdClientWithFailover):
     ERROR_CLS = Etcd3Error
 
     def __init__(self, config: Dict[str, Any], dns_resolver: DnsCachingResolver, cache_ttl: int = 300) -> None:
-        self._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
+        self._reauthenticate = False
         self._token = None
         self._cluster_version: Tuple[int, ...] = tuple()
         super(Etcd3Client, self).__init__({**config, 'version_prefix': '/v3beta'}, dns_resolver, cache_ttl)
@@ -294,7 +288,7 @@ def call_rpc(self, method: str, fields: Dict[str, Any], retry: Optional[Retry] =
         fields['retry'] = retry
         return self.api_execute(self.version_prefix + method, self._MPOST, fields)
 
-    def authenticate(self, *, restart_watcher: bool = True, retry: Optional[Retry] = None) -> bool:
+    def authenticate(self, *, retry: Optional[Retry] = None) -> bool:
         if self._use_proxies and not self._cluster_version:
             kwargs = self._prepare_common_parameters(1)
             self._ensure_version_prefix(self._base_uri, **kwargs)
@@ -316,20 +310,18 @@ def authenticate(self, *, restart_watcher: bool = True, retry: Optional[Retry] =
 
     def handle_auth_errors(self: 'Etcd3Client', func: Callable[..., Any], *args: Any,
                            retry: Optional[Retry] = None, **kwargs: Any) -> Any:
+        reauthenticated = False
         exc = None
         while True:
-            if self._reauthenticate_reason:
+            if self._reauthenticate:
                 if self.username and self.password:
-                    self.authenticate(
-                        restart_watcher=self._reauthenticate_reason != ReAuthenticateMode.WITHOUT_WATCHER_RESTART,
-                        retry=retry)
-                    self._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
-                    if retry:
-                        retry.ensure_deadline(0)
+                    self.authenticate(retry=retry)
+                    self._reauthenticate = False
                 else:
                     msg = 'Username or password not set, authentication is not possible'
                     logger.fatal(msg)
                     raise exc or Etcd3Exception(msg)
+                reauthenticated = True
 
             try:
                 return func(self, *args, retry=retry, **kwargs)
@@ -347,11 +339,12 @@ def handle_auth_errors(self: 'Etcd3Client', func: Callable[..., Any], *args: Any
             except AuthOldRevision as e:
                 logger.error('Auth token is for old revision of auth store')
                 exc = e
-            self._reauthenticate_reason = ReAuthenticateMode.WITHOUT_WATCHER_RESTART \
-                if isinstance(exc, AuthOldRevision) else ReAuthenticateMode.REQUIRED
-            if not retry:
+            self._reauthenticate = True
+            if retry:
+                logger.error('retry = %s', retry)
+                retry.ensure_deadline(0.5, exc)
+            elif reauthenticated:
                 raise exc
-            retry.ensure_deadline(0.5, exc)
 
     @_handle_auth_errors
     def range(self, key: str, range_end: Union[bytes, str, None] = None, serializable: bool = True,
@@ -603,12 +596,6 @@ def set_base_uri(self, value: str) -> None:
         super(PatroniEtcd3Client, self).set_base_uri(value)
         self._restart_watcher()
 
-    def authenticate(self, *, restart_watcher: bool = True, retry: Optional[Retry] = None) -> bool:
-        ret = super(PatroniEtcd3Client, self).authenticate(restart_watcher=restart_watcher, retry=retry)
-        if ret and restart_watcher:
-            self._restart_watcher()
-        return ret
-
     def _wait_cache(self, timeout: float) -> None:
         stop_time = time.time() + timeout
         while self._kv_cache and not self._kv_cache.is_ready():
@@ -866,14 +853,16 @@ def _retry(*args: Any, **kwargs: Any) -> Any:
         try:
             return _retry(self._client.put, self.leader_path, self._name, self._lease, create_revision='0')
         except LeaseNotFound:
-            logger.error('Our lease disappeared from Etcd. Will try to get a new one and retry attempt')
             self._lease = None
-            retry.ensure_deadline(0)
+            if not retry.ensure_deadline(0):
+                logger.error('Our lease disappeared from Etcd. Deadline exceeded, giving up')
+                return False
+
+            logger.error('Our lease disappeared from Etcd. Will try to get a new one and retry attempt')
 
             _retry(self._do_refresh_lease)
 
             retry.ensure_deadline(1, Etcd3Error('_do_attempt_to_acquire_leader timeout'))
-
             return _retry(self._client.put, self.leader_path, self._name, self._lease, create_revision='0')
 
     @catch_return_false_exception
diff --git a/patroni/utils.py b/patroni/utils.py
index dc0a7c95e..c40e1248e 100644
--- a/patroni/utils.py
+++ b/patroni/utils.py
@@ -716,7 +716,7 @@ def stoptime(self) -> float:
         return self._cur_stoptime or 0
 
     def ensure_deadline(self, timeout: float, raise_ex: Optional[Exception] = None) -> bool:
-        """Calculates, sets, and checks the remaining deadline time.
+        """Calculates and checks the remaining deadline time.
 
         :param timeout: if the *deadline* is smaller than the provided *timeout* value raise *raise_ex* exception.
         :param raise_ex: the exception object that will be raised if the *deadline* is smaller than provided *timeout*.
@@ -727,8 +727,7 @@ def ensure_deadline(self, timeout: float, raise_ex: Optional[Exception] = None)
         :raises:
             :class:`Exception`: *raise_ex* if calculated deadline is smaller than provided *timeout*.
         """
-        self.deadline = self.stoptime - time.time()
-        if self.deadline < timeout:
+        if self.stoptime - time.time() < timeout:
             if raise_ex:
                 raise raise_ex
             return False


Test patch:
diff --git a/tests/test_consul.py b/tests/test_consul.py
index 494d11266..87fb47045 100644
--- a/tests/test_consul.py
+++ b/tests/test_consul.py
@@ -160,10 +160,8 @@ def test_take_leader(self):
         self.c.set_ttl(20)
         self.c._do_refresh_session = Mock()
         self.assertFalse(self.c.take_leader())
-        with patch('time.time', Mock(side_effect=[0, 100])):
-            self.assertRaises(ConsulError, self.c.take_leader)
-        with patch('time.time', Mock(side_effect=[0, 0, 0, 0, 0, 0, 100])):
-            self.assertRaises(ConsulError, self.c.take_leader)
+        with patch('time.time', Mock(side_effect=[0, 0, 0, 100, 100])):
+            self.assertFalse(self.c.take_leader())
 
     @patch.object(consul.Consul.KV, 'put', Mock(return_value=True))
     def test_set_failover_value(self):
diff --git a/tests/test_etcd3.py b/tests/test_etcd3.py
index fcfd4e4b6..3e1bd9e83 100644
--- a/tests/test_etcd3.py
+++ b/tests/test_etcd3.py
@@ -7,7 +7,7 @@
 from patroni.dcs import get_dcs
 from patroni.dcs.etcd import DnsCachingResolver
 from patroni.dcs.etcd3 import PatroniEtcd3Client, Cluster, Etcd3, Etcd3Client, \
-    Etcd3Error, Etcd3ClientError, ReAuthenticateMode, RetryFailedError, InvalidAuthToken, Unavailable, \
+    Etcd3Error, Etcd3ClientError, RetryFailedError, InvalidAuthToken, Unavailable, \
     Unknown, UnsupportedEtcdVersion, UserEmpty, AuthFailed, AuthOldRevision, base64_encode
 from patroni.postgresql.mpp import get_mpp
 from threading import Thread
@@ -166,12 +166,14 @@ def test__handle_auth_errors(self, mock_urlopen):
             retry = self.etcd3._retry.copy()
             with patch('time.time', Mock(side_effect=[0, 10, 20, 30, 40])):
                 self.assertRaises(InvalidAuthToken, retry, self.client.deleteprefix, 'foo', retry=retry)
+            with patch('time.time', Mock(side_effect=[0, 10])):
+                self.assertRaises(InvalidAuthToken, self.client.deleteprefix, 'foo')
             self.client.username = None
-            self.client._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
+            self.client._reauthenticate = False
             retry = self.etcd3._retry.copy()
             self.assertRaises(InvalidAuthToken, retry, self.client.deleteprefix, 'foo', retry=retry)
             mock_urlopen.return_value.content = '{"code":3,"error":"etcdserver: revision of auth store is old"}'
-            self.client._reauthenticate_reason = ReAuthenticateMode.NOT_REQUIRED
+            self.client._reauthenticate = False
             self.assertRaises(AuthOldRevision, retry, self.client.deleteprefix, 'foo', retry=retry)
 
     def test__handle_server_response(self):
@@ -271,8 +273,8 @@ def test_take_leader(self):
 
     def test_attempt_to_acquire_leader(self):
         self.assertFalse(self.etcd3.attempt_to_acquire_leader())
-        with patch('time.time', Mock(side_effect=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 100, 200])):
-            self.assertRaises(Etcd3Error, self.etcd3.attempt_to_acquire_leader)
+        with patch('time.time', Mock(side_effect=[0, 0, 0, 0, 0, 100, 200])):
+            self.assertFalse(self.etcd3.attempt_to_acquire_leader())
         with patch('time.time', Mock(side_effect=[0, 100, 200, 300, 400])):
             self.assertRaises(Etcd3Error, self.etcd3.attempt_to_acquire_leader)
         with patch.object(PatroniEtcd3Client, 'put', Mock(return_value=False)):