Repository: patroni/patroni

Issue #2331: Patroni failed to grant lease when system recover from OOM
Created: 2022-06-12T05:27:14Z
Describe the bug
A three node patroni cluster, using etcdv3 (3.5.4) as dcs.
When a follower node OOM (it's not patroni's problem, QA's doing some fault injection testing). After it came back, patroni reporting error in loop:
2022-06-09 09:08:07,376 ERROR: refresh_lease
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd3.py", line 617, in refresh_lease
    return self.retry(self._do_refresh_lease)
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd.py", line 447, in retry
    return retry(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/patroni/utils.py", line 334, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd3.py", line 610, in _do_refresh_lease
    self._lease = self._client.lease_grant(self._ttl, retry)
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd3.py", line 319, in lease_grant
    return self.call_rpc('/lease/grant', {'TTL': ttl}, retry)['ID']
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd3.py", line 558, in call_rpc
    ret = super(PatroniEtcd3Client, self).call_rpc(method, fields, retry)
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd3.py", line 262, in call_rpc
    return self.api_execute(self.version_prefix + method, self._MPOST, fields)
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd.py", line 257, in api_execute
    return self._handle_server_response(response)
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd3.py", line 220, in _handle_server_response
    _raise_for_data(data, response.status)
  File "/usr/local/lib/python3.9/dist-packages/patroni/dcs/etcd3.py", line 136, in _raise_for_data
    raise err(code, error, status_code)
patroni.dcs.etcd3.InvalidAuthToken: <InvalidAuthToken error: 'etcdserver: invalid auth token', code: 16>
2022-06-09 09:08:07,376 INFO: no action. I am (patroni-10.150.60.42), a secondary, and following a leader (patroni-10.150.60.41)

patroni didn't handle auth error for lease_grant & lease_keepalive call https://github.com/zalando/patroni/blob/fb06af9adbf1267dbc0da72c2d8c3eaf802561a6/patroni/dcs/etcd3.py#L318
In etcdv3, lease_grant & lease_keepalive needn't auth token(even auth is enabled), I think the  Etcd3Client should skip pass authorization header to etcd for those calls(or just add the @ _handle_auth_errors)
BTW, etcd server's token validation behavior is wired:

for grant, if you pass a valid token or not pass token header, both okay, but if you pass a invalid token, it reports invalid auth token.
for keepalive, it didn't check the token header at all, even you pass a wrong token

I tested with https://github.com/kragniz/python-etcd3 ( modify its source code to pass a wrong token), it's using grpc, I think it's same for http json api.
In normal case, touch_member call will:

refresh_lease
put(....)

since put will refresh token when token expired, so refresh_lease won't come to this problem in normal case.
It only happens when system stuck for some minutes after the put call, then token and lease expired expired, in next loop, when try to refresh_lease, will do grant first(with the wrong token), so will get the invalid auth token error Infinitely.
Expected behavior
When follower node came back from OOM, it shoulld join cluster successfully.
Environment

Patroni version: 2.1.3
PostgreSQL version:  14.2
DCS (and its version):  etcd 3.5.4 (using etcdv3)

Patroni configuration file
Please copy&paste your Patroni configuration file here

patronictl show-config
ignore_slots: []
loop_wait: 10
master_start_timeout: 300
master_stop_timeout: -1
max_timelines_history: 100
maximum_lag_on_failover: 1048576
maximum_lag_on_syncnode: -1
postgresql:
  parameters:
    idle_in_transaction_session_timeout: 300s
    log_directory: /opt/sddc/patroni/pgsql_logs
    log_filename: postgresql-%d.log
    log_rotation_age: 24h
    log_rotation_size: 0
    log_truncate_on_rotation: true
    logging_collector: true
    max_connections: 1000
    max_replication_slots: 50
    max_wal_senders: 64
    pg_stat_statements.max: 5000
    pg_stat_statements.track: all
    shared_preload_libraries: pg_stat_statements
    wal_level: logical
  use_pg_rewind: true
  use_slots: false
retry_timeout: 10
slots: {}
synchronous_mode: false
synchronous_mode_strict: false
ttl: 30

Duplicate issue #2457: Problem with auth in etcd3
Created: 2022-11-17T07:54:06Z
Describe the bug
Problem with auth in etcd3
Hello.
I have a number of patroni clusters which using an external shared etcd3 (3 nodes HA-cluster)
Simple authentication is configured on the etcd.
Today 4 Patroni Clusters reported  the same issue. I found that some nodes get an error when interacting with etcd
2022-11-17 10:00:54,681 ERROR: refresh_lease
Traceback (most recent call last):
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd3.py", line 617, in refresh_lease
return self.retry(self._do_refresh_lease)
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd.py", line 447, in retry
return retry(*args, **kwargs)
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/utils.py", line 334, in call
return func(*args, **kwargs)
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd3.py", line 610, in _do_refresh_lease
self._lease = self._client.lease_grant(self._ttl, retry)
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd3.py", line 319, in lease_grant
return self.call_rpc('/lease/grant', {'TTL': ttl}, retry)['ID']
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd3.py", line 558, in call_rpc
ret = super(PatroniEtcd3Client, self).call_rpc(method, fields, retry)
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd3.py", line 262, in call_rpc
return self.api_execute(self.version_prefix + method, self._MPOST, fields)
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd.py", line 257, in api_execute
return self._handle_server_response(response)
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd3.py", line 220, in _handle_server_response
_raise_for_data(data, response.status)
File "/var/lib/pgsql/patroni_venv/lib64/python3.6/site-packages/patroni/dcs/etcd3.py", line 136, in _raise_for_data
raise err(code, error, status_code)
patroni.dcs.etcd3.InvalidAuthToken: <InvalidAuthToken error: 'etcdserver: invalid auth token', code: 16>
2022-11-17 10:00:54,682 INFO: Lock owner: postgres1; I am postgres2
....
and then repeating this error
...
This error happened on 4 patroni clusters at the same time. I examined the etcd logs, and decided that the problem was due to performance problems with one of the etcd nodes. (nodes made reelect, but all finished well, cluster was health)
But even after everything returned to normal on the etcd, some nodes of the patroni clusters continued to receive an error.
It helped restarting the patroni on each problem node, or restarting the node etcd with which the patroni was trying to contact.
Any thoughts why this could happen? I suspect simple authentication is the problem.
patronictl show-config
check_timeline: false
loop_wait: 10
master_start_timeout: 300
maximum_lag_on_failover: 1048576
postgresql:
parameters:
max_connections: 300
max_locks_per_transaction: 64
max_prepared_transactions: 0
max_replication_slots: 10
max_wal_senders: 10
max_worker_processes: 8
track_commit_timestamp: false
wal_keep_size: 6GB
wal_level: replica
wal_log_hints: true
pg_hba:

use_pg_rewind: true
use_slots: true
retry_timeout: 10
synchronous_mode: true
synchronous_mode_strict: false
ttl: 30
patroni.yml
etcd3:
hosts: etcd-cluster-1:2379,etcd-cluster-2:2379,etcd-cluster-3:2379
protocol: http
username: patroni_etcd
password: password

Closing comment (2022-11-18T07:12:57Z): I think it is a duplicate of #2331
It would have been much easier to verify if you followed the issue template