Repository: patroni/patroni

Issue #2879: Error setting the parameter `wal_keep_size` with unit in Patroni 3.1.1
Created: 2023-09-22T19:54:39Z
What happened?
Error setting the PostgreSQL parameter wal_keep_size with unit, this is related to the PR #2861
Not sure if other parameters with units are affected.
How can we reproduce it (as minimally and precisely as possible)?
.
What did you expect to happen?
The parameter is valid but the conversion to int is not correct as is not interpreting the unit.
Patroni/PostgreSQL/DCS version

Patroni version: 3.1.1
PostgreSQL version:  15.2
DCS (and its version): kubernetes

Patroni configuration file
.
patronictl show-config
.
Patroni log files
2023-09-22 15:25:51,367 ERROR: Exception when setting dynamic_configuration
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/patroni/config.py", line 312, in set_dynamic_configuration
    self.__effective_configuration = self._build_effective_configuration(configuration,
  File "/usr/lib/python3.9/site-packages/patroni/config.py", line 537, in _build_effective_configuration
    config = self._safe_copy_dynamic_configuration(dynamic_configuration)
  File "/usr/lib/python3.9/site-packages/patroni/config.py", line 359, in _safe_copy_dynamic_configuration
    config['postgresql'][name].update(self._process_postgresql_parameters(value))
  File "/usr/lib/python3.9/site-packages/patroni/config.py", line 345, in _process_postgresql_parameters
    pg_params[name] = int(value) if isinstance(validator, IntValidator) else value
ValueError: invalid literal for int() with base 10: '1536MB'
PostgreSQL log files
.
Have you tried to use GitHub issue search?

 Yes

Anything else we need to know?
No response

Duplicate issue #2882: Dynamic config parameters are gone in patroni 3.1.1
Created: 2023-09-23T16:53:35Z
What happened?
Everything is OK with Patroni 3.0.4 and 3.1.0. Cluster bootstrap broken with 3.1.1.
The postgres cluster can be bootstrapped with patroni 3.1.1, but the dynamic config is empty.
Replace the 3.1.1 with 3.0.4 or 3.1.0 in the same environment, the problem is gone.
Here is the detail:
The cluster is bootstrapped, but the dynamic config section is empty:

There's only {} when patronictl show-config or edit-config
There's no corresponding dynamic config in bootstraped PostgreSQL instance. postgresql.conf
There's no dynamic config in DCS etcd either

The patronictl list command works which proves the DCS is working well.
$ pg list pg-el7test       # the 'pg' is an alias of patronictl
+ Cluster: pg-el7test (7282014022210771169) -------+----+-----------+------------------+
| Member       | Host        | Role    | State     | TL | Lag in MB | Tags             |
+--------------+-------------+---------+-----------+----+-----------+------------------+
| pg-el7test-1 | 10.10.10.77 | Leader  | running   |  1 |           | clonefrom: true  |
+--------------+-------------+---------+-----------+----+-----------+------------------+
| pg-el7test-2 | 10.10.10.78 | Replica | streaming |  1 |         0 | clonefrom: true  |
+--------------+-------------+---------+-----------+----+-----------+------------------+
| pg-el7test-3 | 10.10.10.79 | Replica | streaming |  1 |         0 | clonefrom: true  |
+--------------+-------------+---------+-----------+----+-----------+------------------+
But the show-config of this cluster will print empty json object, so does edit-config
$ pg show-config pg-el7test
{}

And the postgresql.conf generated/managed by patroni is nearly empty, seems like the dynamic config is not applied.
There's only key parameters directly managed by patroni in postgresql.conf:
# It will be overwritten by Patroni!
include 'postgresql.base.conf'

cluster_name = 'pg-el7test'
hot_standby = 'on'
listen_addresses = '0.0.0.0'
max_connections = '100'
max_locks_per_transaction = '64'
max_prepared_transactions = '0'
max_replication_slots = '10'
max_wal_senders = '10'
max_worker_processes = '8'
port = '5432'
track_commit_timestamp = 'off'
wal_keep_size = '128MB'
wal_level = 'replica'
wal_log_hints = 'on'
hba_file = '/pg/data/pg_hba.conf'
ident_file = '/pg/data/pg_ident.conf'
And there's no corresponding config in DCS etcd :
get etcd metadata
$  e get /pg --prefix
/pg/pg-el7/config
{}
/pg/pg-el7/initialize
7282014011448899717
/pg/pg-el7/leader
pg-el7-1
/pg/pg-el7/members/pg-el7-1
{"conn_url":"postgres://10.10.10.7:5432/postgres","api_url":"http://10.10.10.7:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"15","spec":"4C.8G.125G","conf":"tiny.yml"},"xlog_location":29867432,"timeline":1}
/pg/pg-el7/status
{"optime":29867432}
/pg/pg-el7test/config
{}
/pg/pg-el7test/initialize
7282014022210771169
/pg/pg-el7test/leader
pg-el7test-1
/pg/pg-el7test/members/pg-el7test-1
{"conn_url":"postgres://10.10.10.77:5432/postgres","api_url":"http://10.10.10.77:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"14","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":70284904,"timeline":1}
/pg/pg-el7test/members/pg-el7test-2
{"conn_url":"postgres://10.10.10.78:5432/postgres","api_url":"http://10.10.10.78:8008/patroni","state":"running","role":"replica","version":"3.1.1","tags":{"clonefrom":true,"version":"14","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":70284904,"replication_state":"streaming","timeline":1}
/pg/pg-el7test/members/pg-el7test-3
{"conn_url":"postgres://10.10.10.79:5432/postgres","api_url":"http://10.10.10.79:8008/patroni","state":"running","role":"replica","version":"3.1.1","tags":{"clonefrom":true,"version":"14","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":70284904,"replication_state":"streaming","timeline":1}
/pg/pg-el7test/status
{"optime":70284904}
/pg/pg-el7v12/config
{}
/pg/pg-el7v12/initialize
7282014022445232509
/pg/pg-el7v12/leader
pg-el7v12-1
/pg/pg-el7v12/members/pg-el7v12-1
{"conn_url":"postgres://10.10.10.72:5432/postgres","api_url":"http://10.10.10.72:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"12","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":31311040,"timeline":1}
/pg/pg-el7v12/status
{"optime":31311040}
/pg/pg-el7v13/config
{}
/pg/pg-el7v13/initialize
7282014021746618735
/pg/pg-el7v13/leader
pg-el7v13-1
/pg/pg-el7v13/members/pg-el7v13-1
{"conn_url":"postgres://10.10.10.73:5432/postgres","api_url":"http://10.10.10.73:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"13","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":31213864,"timeline":1}
/pg/pg-el7v13/status
{"optime":31213864}
/pg/pg-el7v14/config
{}
/pg/pg-el7v14/initialize
7282014021806876781
/pg/pg-el7v14/leader
pg-el7v14-1
/pg/pg-el7v14/members/pg-el7v14-1
{"conn_url":"postgres://10.10.10.74:5432/postgres","api_url":"http://10.10.10.74:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"14","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":33994192,"timeline":1}
/pg/pg-el7v14/status
{"optime":33994192}
/pg/pg-el7v15/config
{}
/pg/pg-el7v15/initialize
7282014021964335187
/pg/pg-el7v15/leader
pg-el7v15-1
/pg/pg-el7v15/members/pg-el7v15-1
{"conn_url":"postgres://10.10.10.75:5432/postgres","api_url":"http://10.10.10.75:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"15","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":58585096,"timeline":1}
/pg/pg-el7v15/members/pg-el7v15-2
{"conn_url":"postgres://10.10.10.76:5432/postgres","api_url":"http://10.10.10.76:8008/patroni","state":"running","role":"replica","version":"3.1.1","tags":{"clonefrom":true,"version":"15","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":58585096,"replication_state":"streaming","timeline":1}
/pg/pg-el7v15/status
{"optime":58585096}
/pg/pg-el8/config
{}
/pg/pg-el8/initialize
7282013999476332216
/pg/pg-el8/leader
pg-el8-1
/pg/pg-el8/members/pg-el8-1
{"conn_url":"postgres://10.10.10.8:5432/postgres","api_url":"http://10.10.10.8:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"15","spec":"4C.8G.125G","conf":"tiny.yml"},"xlog_location":29603448,"timeline":1}
/pg/pg-el8/status
{"optime":29603448}
/pg/pg-el8test/config
{"ttl":30,"loop_wait":10,"retry_timeout":10,"primary_start_timeout":10,"maximum_lag_on_failover":1048576,"maximum_lag_on_syncnode":-1,"primary_stop_timeout":30,"synchronous_mode":true,"synchronous_mode_strict":false,"failsafe_mode":true,"pg_version":15,"pg_cluster":"pg-el8test","pg_shard":"pg-el8test","pg_group":0,"postgresql":{"use_slots":true,"use_pg_rewind":true,"remove_data_directory_on_rewind_failure":true,"parameters":{"max_connections":200,"superuser_reserved_connections":10,"max_locks_per_transaction":200,"max_prepared_transactions":0,"track_commit_timestamp":true,"wal_level":"logical","wal_log_hints":true,"wal_keep_size":"8GB","max_worker_processes":8,"max_wal_senders":24,"max_replication_slots":16,"password_encryption":"scram-sha-256","ssl":true,"ssl_cert_file":"/pg/cert/server.crt","ssl_key_file":"/pg/cert/server.key","ssl_ca_file":"/etc/pki/ca.crt","shared_buffers":"933MB","maintenance_work_mem":"234MB","work_mem":"64MB","max_parallel_workers":8,"max_parallel_maintenance_workers":2,"max_parallel_workers_per_gather":0,"hash_mem_multiplier":8.0,"huge_pages":"try","temp_file_limit":"4GB","vacuum_cost_delay":"20ms","vacuum_cost_limit":2000,"bgwriter_delay":"10ms","bgwriter_lru_maxpages":800,"bgwriter_lru_multiplier":5.0,"min_wal_size":"4GB","max_wal_size":"16GB","max_slot_wal_keep_size":"24GB","wal_buffers":"16MB","wal_writer_delay":"10ms","wal_writer_flush_after":0,"commit_delay":20,"commit_siblings":10,"checkpoint_timeout":"15min","checkpoint_completion_target":0.8,"archive_mode":true,"archive_timeout":300,"archive_command":"pgbackrest --stanza=pg-el8test archive-push %p","vacuum_defer_cleanup_age":500000,"max_standby_archive_delay":"10min","max_standby_streaming_delay":"3min","wal_receiver_status_interval":"1s","hot_standby_feedback":true,"wal_receiver_timeout":"60s","max_logical_replication_workers":8,"max_sync_workers_per_subscription":6,"random_page_cost":1.1,"effective_io_concurrency":1000,"effective_cache_size":"2799MB","default_statistics_target":400,"log_destination":"csvlog","logging_collector":true,"log_directory":"/pg/log/postgres","log_filename":"postgresql-%Y-%m-%d.log","log_checkpoints":true,"log_lock_waits":true,"log_replication_commands":true,"log_statement":"ddl","log_min_duration_statement":100,"log_connections":true,"log_disconnections":true,"track_io_timing":true,"track_functions":"all","track_activity_query_size":32768,"log_autovacuum_min_duration":"1s","autovacuum_max_workers":3,"autovacuum_naptime":"1min","autovacuum_vacuum_scale_factor":0.08,"autovacuum_analyze_scale_factor":0.04,"autovacuum_vacuum_cost_delay":-1,"autovacuum_vacuum_cost_limit":-1,"autovacuum_freeze_max_age":1000000000,"deadlock_timeout":"50ms","idle_in_transaction_session_timeout":"1min","shared_preload_libraries":"pg_stat_statements, auto_explain","auto_explain.log_min_duration":"1s","auto_explain.log_analyze":true,"auto_explain.log_verbose":true,"auto_explain.log_timing":true,"auto_explain.log_nested_statements":true,"pg_stat_statements.max":10000,"pg_stat_statements.track":"all","pg_stat_statements.track_utility":true,"pg_stat_statements.track_planning":true,"timescaledb.telemetry_level":"off","timescaledb.max_background_workers":8,"citus.node_conninfo":"sslmode=prefer"}}}
/pg/pg-el8test/failsafe
{"pg-el8test-1":"http://10.10.10.87:8008/patroni","pg-el8test-3":"http://10.10.10.89:8008/patroni","pg-el8test-2":"http://10.10.10.88:8008/patroni"}
/pg/pg-el8test/initialize
7282058498504229834
/pg/pg-el8test/leader
pg-el8test-1
/pg/pg-el8test/members/pg-el8test-1
{"conn_url":"postgres://10.10.10.87:5432/postgres","api_url":"http://10.10.10.87:8008/patroni","state":"running","role":"master","version":"3.0.4","tags":{"clonefrom":true,"version":"15","spec":"2C.4G.70G","conf":"crit.yml"},"xlog_location":117440512,"timeline":1}
/pg/pg-el8test/members/pg-el8test-2
{"conn_url":"postgres://10.10.10.88:5432/postgres","api_url":"http://10.10.10.88:8008/patroni","state":"running","role":"replica","version":"3.0.4","tags":{"clonefrom":true,"version":"15","spec":"2C.4G.125G","conf":"crit.yml"},"xlog_location":117440512,"replication_state":"streaming","timeline":1}
/pg/pg-el8test/members/pg-el8test-3
{"conn_url":"postgres://10.10.10.89:5432/postgres","api_url":"http://10.10.10.89:8008/patroni","state":"running","role":"replica","version":"3.0.4","tags":{"clonefrom":true,"version":"15","spec":"2C.4G.125G","conf":"crit.yml"},"xlog_location":117440512,"replication_state":"streaming","timeline":1}
/pg/pg-el8test/status
{"optime":117440512}
/pg/pg-el8test/sync
{"leader":"pg-el8test-1","sync_standby":"pg-el8test-3"}
/pg/pg-el8v12/config
{}
/pg/pg-el8v12/initialize
7282014005542497153
/pg/pg-el8v12/leader
pg-el8v12-1
/pg/pg-el8v12/members/pg-el8v12-1
{"conn_url":"postgres://10.10.10.82:5432/postgres","api_url":"http://10.10.10.82:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"12","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":30944200,"timeline":1}
/pg/pg-el8v12/status
{"optime":30944200}
/pg/pg-el8v13/config
{}
/pg/pg-el8v13/initialize
7282014005267606384
/pg/pg-el8v13/leader
pg-el8v13-1
/pg/pg-el8v13/members/pg-el8v13-1
{"conn_url":"postgres://10.10.10.83:5432/postgres","api_url":"http://10.10.10.83:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"13","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":30916544,"timeline":1}
/pg/pg-el8v13/status
{"optime":30916544}
/pg/pg-el8v14/config
{}
/pg/pg-el8v14/initialize
7282014011430714647
/pg/pg-el8v14/leader
pg-el8v14-1
/pg/pg-el8v14/members/pg-el8v14-1
{"conn_url":"postgres://10.10.10.84:5432/postgres","api_url":"http://10.10.10.84:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"14","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":33667024,"timeline":1}
/pg/pg-el8v14/status
{"optime":33667024}
/pg/pg-el8v15/config
{}
/pg/pg-el8v15/initialize
7282014007042079626
/pg/pg-el8v15/leader
pg-el8v15-1
/pg/pg-el8v15/members/pg-el8v15-1
{"conn_url":"postgres://10.10.10.85:5432/postgres","api_url":"http://10.10.10.85:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"15","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":35104568,"timeline":1}
/pg/pg-el8v15/status
{"optime":35104568}
/pg/pg-el8v16/config
{}
/pg/pg-el8v16/initialize
7282014006952783122
/pg/pg-el8v16/leader
pg-el8v16-1
/pg/pg-el8v16/members/pg-el8v16-1
{"conn_url":"postgres://10.10.10.86:5432/postgres","api_url":"http://10.10.10.86:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"16","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":34829032,"timeline":1}
/pg/pg-el8v16/status
{"optime":34829032}
/pg/pg-el9/config
{}
/pg/pg-el9/initialize
7282013999568516326
/pg/pg-el9/leader
pg-el9-1
/pg/pg-el9/members/pg-el9-1
{"conn_url":"postgres://10.10.10.9:5432/postgres","api_url":"http://10.10.10.9:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"15","spec":"4C.8G.125G","conf":"tiny.yml"},"xlog_location":29580792,"timeline":1}
/pg/pg-el9/status
{"optime":29580792}
/pg/pg-el9test/config
{}
/pg/pg-el9test/initialize
7282014004649065188
/pg/pg-el9test/leader
pg-el9test-1
/pg/pg-el9test/members/pg-el9test-1
{"conn_url":"postgres://10.10.10.97:5432/postgres","api_url":"http://10.10.10.97:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"16","spec":"2C.4G.125G","conf":"olap.yml"},"xlog_location":75275472,"timeline":1}
/pg/pg-el9test/members/pg-el9test-2
{"conn_url":"postgres://10.10.10.98:5432/postgres","api_url":"http://10.10.10.98:8008/patroni","state":"running","role":"replica","version":"3.1.1","tags":{"clonefrom":true,"version":"16","spec":"2C.4G.125G","conf":"olap.yml"},"xlog_location":75275472,"replication_state":"streaming","timeline":1}
/pg/pg-el9test/members/pg-el9test-3
{"conn_url":"postgres://10.10.10.99:5432/postgres","api_url":"http://10.10.10.99:8008/patroni","state":"running","role":"replica","version":"3.1.1","tags":{"clonefrom":true,"replicatefrom":"10.10.10.98","version":"16","spec":"2C.4G.125G","conf":"olap.yml"},"xlog_location":75275472,"replication_state":"streaming","timeline":1}
/pg/pg-el9test/status
{"optime":75275472}
/pg/pg-el9v12/config
{}
/pg/pg-el9v12/initialize
7282014003194810416
/pg/pg-el9v12/leader
pg-el9v12-1
/pg/pg-el9v12/members/pg-el9v12-1
{"conn_url":"postgres://10.10.10.92:5432/postgres","api_url":"http://10.10.10.92:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"12","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":30979304,"timeline":1}
/pg/pg-el9v12/status
{"optime":30979304}
/pg/pg-el9v13/config
{}
/pg/pg-el9v13/initialize
7282014003877654094
/pg/pg-el9v13/leader
pg-el9v13-1
/pg/pg-el9v13/members/pg-el9v13-1
{"conn_url":"postgres://10.10.10.93:5432/postgres","api_url":"http://10.10.10.93:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"13","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":30901840,"timeline":1}
/pg/pg-el9v13/status
{"optime":30901840}
/pg/pg-el9v14/config
{}
/pg/pg-el9v14/initialize
7282014003451246963
/pg/pg-el9v14/leader
pg-el9v14-1
/pg/pg-el9v14/members/pg-el9v14-1
{"conn_url":"postgres://10.10.10.94:5432/postgres","api_url":"http://10.10.10.94:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"14","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":33702352,"timeline":1}
/pg/pg-el9v14/status
{"optime":33702352}
/pg/pg-el9v15/config
{}
/pg/pg-el9v15/initialize
7282014003279932353
/pg/pg-el9v15/leader
pg-el9v15-1
/pg/pg-el9v15/members/pg-el9v15-1
{"conn_url":"postgres://10.10.10.95:5432/postgres","api_url":"http://10.10.10.95:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"15","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":35091048,"timeline":1}
/pg/pg-el9v15/status
{"optime":35091048}
/pg/pg-el9v16/config
{}
/pg/pg-el9v16/initialize
7282014003369228394
/pg/pg-el9v16/leader
pg-el9v16-1
/pg/pg-el9v16/members/pg-el9v16-1
{"conn_url":"postgres://10.10.10.96:5432/postgres","api_url":"http://10.10.10.96:8008/patroni","state":"running","role":"master","version":"3.1.1","tags":{"clonefrom":true,"version":"16","spec":"2C.4G.125G","conf":"tiny.yml"},"xlog_location":34864632,"timeline":1}
/pg/pg-el9v16/status
{"optime":34864632}

The above exception pg-el8test is manully fixed by uninstall patroni 3.1.1 (PGDG RPM) and reinstall patroni 3.0.4 . Everything works well again
How can we reproduce it (as minimally and precisely as possible)?
The Pigsty v2.4.0 is using patroni v3.1.0. you can install a four-node cluster and pg edit-config pg-test
The Pigsty v2.4.1 (beta) is using the v3.1.1, and the same procedure will bootstrap the postgres cluster with empty dynamic section
It is found by routine CI tasks, You can use the 4-node demo to reproduce this.
What did you expect to happen?
When perform patronictl edit-config, It should work as expected.
Patroni/PostgreSQL/DCS version

Patroni version:  3.1.1
PostgreSQL version: 12 ~ 16, all version
DCS (and its version): etcd 3.5.9

And I'm using the 1PGDG official RPM packages on CentOS 7.9/Rocky 8.7/Rocky 9.1.
Install patroni 3.1.1 from pip still have the same problem.
Patroni configuration file
This is an auto-generated config file, the template is tiny.yml
#!/usr/bin/env patroni
#==============================================================#
# File      :   patroni.yml (tiny)
# Desc      :   Tiny Database Cluster Template (1-3C)
# Time      :   2023-09-23 21:32
# Host      :   pg-el7test-1 @ 10.10.10.77:8008
# Path      :   /pg/bin/patroni.yml -> /pg/conf/pg-el7test-1.yml
# Note      :   ANSIBLE MANAGED, DO NOT CHANGE!
# Docs      :   https://patroni.readthedocs.io/en/latest/SETTINGS.html
# Author    :   Ruohang Feng (rh@vonng.com)
# License   :   AGPLv3
#==============================================================#

#==============================================================#
# Summary for patroni TINY config template (1-3 cpu cores)
#==============================================================#
# 10.10.10.77:5432 | [primary] CLUSTER LEADER @ pg-el7test-1
# Node: 2C4G , node_tune = oltp
# Memory: 3790MB , Disk : 125GB SSD @ /data
# ETCD: 10.10.10.7:2379 10.10.10.8:2379 10.10.10.9:2379
#--------------------------------------------------------------#
# pgconf, maxconn, workmem   = tiny.yml , 100 , 16MB
# buffer, cache, maintain    = 948MB, 2842MB,  237MB
# prepared xact, locks/xact  = 0 , 200
# wal min,keep,max,slot,temp = 7GB, 14GB, 28GB, 42GB, 7GB
# proc, worker, mtw, gather  = 8, 8, 2, 0
# libraries = timescaledb, pg_stat_statements, auto_explain
#==============================================================#

---
#--------------------------------------------------------------#
# identity
#--------------------------------------------------------------#
# namespace, instance, cluster/shard scope
namespace: /pg/
name: pg-el7test-1
scope: pg-el7test   # common pgsql cluster


#--------------------------------------------------------------#
# log
#--------------------------------------------------------------#
log:
  level: INFO                           #  NOTEST|DEBUG|INFO|WARNING|ERROR|CRITICAL
  dir: /pg/log/patroni            #  patroni log dir
  file_size: 33554432                   #  32MB log triggers log rotation
  file_num: 20                          #  keep at most 30x32MB = 1GB log
  dateformat: '%Y-%m-%d %H:%M:%S %z'    #  IMPORTANT: discard milli timestamp
  format: '%(asctime)s %(levelname)s: %(message)s'

#--------------------------------------------------------------#
# etcd
#--------------------------------------------------------------#
etcd3:
  hosts: '10.10.10.7:2379,10.10.10.8:2379,10.10.10.9:2379'
  protocol: https
  cacert: /etc/pki/ca.crt
  cert:   /pg/cert/server.crt
  key:    /pg/cert/server.key

#--------------------------------------------------------------#
# api
#--------------------------------------------------------------#
# how to expose patroni service
restapi:
  listen: 10.10.10.77:8008
  connect_address: 10.10.10.77:8008
  # unsafe api can only been accessed from meta nodes with auth
  authentication:
    username: 'postgres'
    password: 'Patroni.API'
  allowlist: [ 10.10.10.7, 10.10.10.77, 10.10.10.78, 10.10.10.79, 10.10.10.8, 10.10.10.9 ]

#--------------------------------------------------------------#
# ctl
#--------------------------------------------------------------#
ctl:
  insecure: true

#--------------------------------------------------------------#
# tags
#--------------------------------------------------------------#
tags:
  nofailover: false
  clonefrom: true
  noloadbalance: false
  nosync: false
  version:  '14'
  spec: '2C.4G.125G'
  conf: 'tiny.yml'

#--------------------------------------------------------------#
# watchdog
#--------------------------------------------------------------#
# available mode: off|automatic|required
watchdog:
  mode: False
  device: /dev/watchdog
  #safety_margin: 25   # -1 to use ttl/2

#--------------------------------------------------------------#
# bootstrap
#--------------------------------------------------------------#
bootstrap:

  #------------------------------------------------------------#
  # bootstrap method
  #------------------------------------------------------------#
  method: initdb
  # add custom bootstrap method here

  # default bootstrap method: initdb
  initdb:
    - encoding: UTF8
    - locale: C
    - lc-collate: C
    - lc-ctype: en_US.UTF8

  #------------------------------------------------------------#
  # bootstrap users
  #------------------------------------------------------------#
  # additional users which need to be created after initializing new cluster
  # replication user and monitor user are required
  users:
    replicator:
      password: 'DBUser.Replicator'
    dbuser_monitor:
      password: 'DBUser.Monitor'

  # bootstrap hba, allow local and intranet password access & replication
  # will be overwritten later
  pg_hba:
    - local   all             postgres                                ident
    - local   all             all                                     scram-sha-256
    - host    all             all            0.0.0.0/0                scram-sha-256
    - local   replication     postgres                                ident
    - local   replication     all                                     scram-sha-256
    - host    replication     all            0.0.0.0/0                scram-sha-256


  #------------------------------------------------------------#
  # customization
  #------------------------------------------------------------#
  # post_init: /pg/bin/pg-init

  #------------------------------------------------------------#
  # bootstrap config
  #------------------------------------------------------------#
  # this section will be written to //pg/pg-el7test/config
  # if will NOT take any effect after cluster bootstrap
  dcs:


    #----------------------------------------------------------#
    # important parameters
    #----------------------------------------------------------#
    # constraint: ttl >: loop_wait + retry_timeout * 2

    # the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30
    # config this according to your network condition to avoid false-positive failover
    ttl: 30

    # the number of seconds the loop will sleep. Default value: 10
    # this is patroni check loop interval
    loop_wait: 10

    # timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10
    retry_timeout: 10

    # the amount of time a primary is allowed to recover from failures before failover is triggered (in seconds)
    # Max RTO: 2 loop wait + primary_start_timeout
    primary_start_timeout: 10

    # import: candidate will not be promoted if replication lag is higher than this
    # maximum RPO: 1MB
    maximum_lag_on_failover: 1048576

    # the maximum bytes a synchronous follower may lag before it is considered as an unhealthy candidate and swapped by healthy asynchronous follower
    maximum_lag_on_syncnode: -1

    # The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled
    primary_stop_timeout: 30

    # turns on synchronous replication mode. In this mode a replica will be chosen as synchronous and only the latest leader and synchronous replica are able to participate in leader election
    # if crit template is used or RPO set to 0,1 , sync mode will be enabled
    synchronous_mode: false

    # prevents disabling synchronous replication if no synchronous replicas are available, blocking all client writes to the primary
    synchronous_mode_strict: false
    # synchronous_node_count: 1  # edit-config only, when using quorum commit

    # Postgres may continue to run as a primary if it can access all known members of the cluster via Patroni REST API.
    failsafe_mode: true

    # postgres cluster metadata
    pg_version: 14
    pg_cluster: pg-el7test
    pg_shard:   pg-el7test
    pg_group:   0

    #----------------------------------------------------------#
    # postgres parameters
    #----------------------------------------------------------#
    postgresql:
      use_slots: true
      use_pg_rewind: true
      remove_data_directory_on_rewind_failure: true

      parameters:
        #------------------------------------------------------#
        # IMPORTANT PARAMETERS
        #------------------------------------------------------#
        max_connections: 100
        superuser_reserved_connections: 10
        max_locks_per_transaction: 200
        max_prepared_transactions: 0
        track_commit_timestamp: on                  # enabled xact timestamp
        wal_level: logical                          # force logical level
        wal_log_hints: on                           # wal log hints to support rewind
        wal_keep_size: 14GB                        # 1/10 disk size
        max_worker_processes: 8
        max_wal_senders: 10                         # default 10
        max_replication_slots: 10                   # default 10

        #------------------------------------------------------#
        # AUTHENTICATION
        #------------------------------------------------------#
        password_encryption: scram-sha-256
        ssl: on                                 # enable server ssl
        ssl_cert_file: '/pg/cert/server.crt'    # public cert path
        ssl_key_file: '/pg/cert/server.key'     # private key path
        ssl_ca_file: '/etc/pki/ca.crt'          # ca file path

        #------------------------------------------------------#
        # RESOURCE USAGE (except WAL)
        #------------------------------------------------------#
        # memory: shared_buffers and maintenance_work_mem will be dynamically set
        shared_buffers: 948MB
        maintenance_work_mem: 237MB
        work_mem: 16MB
        max_parallel_workers: 8
        max_parallel_maintenance_workers: 2
        max_parallel_workers_per_gather: 0
        hash_mem_multiplier: 8.0                # use 8x work mem in hash-based operations
        huge_pages: try                         # try huge pages
        temp_file_limit: 7GB                  # 1/20 of disk size
        vacuum_cost_delay: 20ms                 # wait 20ms per 2000 cost
        vacuum_cost_limit: 2000                 # 2000 cost each round
        bgwriter_delay: 10ms                    # check dirty page every 10ms
        bgwriter_lru_maxpages: 800              # 100 -> 800
        bgwriter_lru_multiplier: 5.0            # 2.0 -> 5.0  more cushion buffer

        #------------------------------------------------------#
        # WAL
        #------------------------------------------------------#
        min_wal_size: 7GB                      # 1/20 disk size
        max_wal_size: 28GB                     # 2/10 disk size
        max_slot_wal_keep_size: 42GB           # 3/10 disk size at most!
        wal_buffers: 16MB                       # max to 16MB
        wal_writer_delay: 20ms                  # wait period
        wal_writer_flush_after: 1MB             # max allowed data loss
        commit_delay: 20                        # 200ms -> 20ms, increase speed
        commit_siblings: 10                     # 5 -> 10
        checkpoint_timeout: 15min               # checkpoint 5min -> 15min
        checkpoint_completion_target: 0.80      # 0.5 -> 0.8
        archive_mode: on
        archive_timeout: 300                    # 5 min timeout for archiving
        archive_command: 'pgbackrest --stanza=pg-el7test archive-push %p'

        #------------------------------------------------------#
        # REPLICATION
        #------------------------------------------------------#
        # synchronous_standby_names: ''
        max_standby_archive_delay: 10min        # max delay before canceling queries when reading WAL from archive;
        max_standby_streaming_delay: 3min       # max delay before canceling queries when reading streaming WAL;
        wal_receiver_status_interval: 1s        # send replies at least this often
        hot_standby_feedback: on                # send info from standby to prevent query conflicts
        wal_receiver_timeout: 60s               # time that receiver waits for
        max_logical_replication_workers: 8      # default 4, 6 sync worker + 1~2 apply worker
        max_sync_workers_per_subscription: 6    # default 2, 6 sync worker

        #------------------------------------------------------#
        # QUERY TUNING
        #------------------------------------------------------#
        # enable_partitionwise_join: on
        random_page_cost: 1.1                   # 4 for HDD, 1.1 for SSD
        effective_io_concurrency: 1000          # 1000 for SSD
        effective_cache_size: 2842MB           # max mem - shared buffer
        default_statistics_target: 200          # stat bucket 100 -> 200

        #------------------------------------------------------#
        # REPORTING AND LOGGING
        #------------------------------------------------------#
        log_destination: csvlog                 # use standard csv log
        logging_collector: on                   # enable csvlog
        log_directory: /pg/log/postgres         # postgres log dir
        # log_filename: 'postgresql-%a.log'     # weekly auto-recycle
        log_filename: 'postgresql-%Y-%m-%d.log' # YYYY-MM-DD full log retention
        log_checkpoints: on                     # log checkpoint info
        log_lock_waits: on                      # log lock wait info
        log_replication_commands: on            # log replication info
        log_statement: ddl                      # log ddl change
        log_min_duration_statement: 100         # log slow query (>100ms)

        #------------------------------------------------------#
        # STATISTICS
        #------------------------------------------------------#
        track_io_timing: on                     # collect io statistics
        track_functions: all                    # track all functions (none|pl|all)
        track_activity_query_size: 8192         # max query length in pg_stat_activity

        #------------------------------------------------------#
        # AUTOVACUUM
        #------------------------------------------------------#
        log_autovacuum_min_duration: 1s         # log autovacuum activity take more than 1s
        autovacuum_max_workers: 1               # default autovacuum worker 3 -> 1
        autovacuum_naptime: 1min                # default autovacuum naptime 1min
        # autovacuum_vacuum_scale_factor: 0.16    # fraction of table size before vacuum   20% -> 8%
        # autovacuum_analyze_scale_factor: 0.04   # fraction of table size before analyze  10% -> 4%
        autovacuum_vacuum_cost_delay: -1        # default vacuum cost delay: same as vacuum_cost_delay
        autovacuum_vacuum_cost_limit: -1        # default vacuum cost limit: same as vacuum_cost_limit
        autovacuum_freeze_max_age: 1000000000   # age > 1 billion triggers force vacuum

        #------------------------------------------------------#
        # CLIENT
        #------------------------------------------------------#
        deadlock_timeout: 50ms                      # 50ms for deadlock
        idle_in_transaction_session_timeout: 10min  # 10min timeout for idle in transaction

        #------------------------------------------------------#
        # CUSTOMIZED OPTIONS
        #------------------------------------------------------#
        # extensions
        shared_preload_libraries: 'timescaledb, pg_stat_statements, auto_explain'

        # auto_explain
        auto_explain.log_min_duration: 1s       # auto explain query slower than 1s
        auto_explain.log_analyze: true          # explain analyze
        auto_explain.log_verbose: true          # explain verbose
        auto_explain.log_timing: true           # explain timing
        auto_explain.log_nested_statements: true

        # pg_stat_statements
        pg_stat_statements.max: 5000            # default 5000 queries
        pg_stat_statements.track: all           # track all statements (all|top|none)
        pg_stat_statements.track_utility: off   # do not track query other than CRUD
        pg_stat_statements.track_planning: off  # do not track planning metrics

        # timescaledb
        timescaledb.telemetry_level: 'off'      # disable timescale telemetry
        timescaledb.max_background_workers: 8

        # citus
        citus.node_conninfo: 'sslmode=prefer'   # prefer ssl instead of require


#--------------------------------------------------------------#
# postgres
#--------------------------------------------------------------#
postgresql:

  #------------------------------------------------------------#
  # how to connect to postgres
  #------------------------------------------------------------#
  bin_dir: /usr/pgsql/bin
  data_dir: /pg/data
  config_dir: /pg/data
  pgpass: /var/lib/pgsql/.pgpass
  listen: 0.0.0.0:5432
  connect_address: 10.10.10.77:5432
  use_unix_socket: true # default: /var/run/postgresql, /tmp

  #------------------------------------------------------------#
  # who to connect to postgres
  #------------------------------------------------------------#
  authentication:
    superuser:
      username: postgres
    replication:
      username: replicator
      password: 'DBUser.Replicator'
    rewind:
      username: replicator
      password: 'DBUser.Replicator'

  #------------------------------------------------------------#
  # how to react to database operations
  #------------------------------------------------------------#
  # event callback script log: /pg/log/patroni/callback.log
  callbacks:
    on_start: /pg/bin/pg-failover-callback
    on_stop: /pg/bin/pg-failover-callback
    on_reload: /pg/bin/pg-failover-callback
    on_restart: /pg/bin/pg-failover-callback
    on_role_change: /pg/bin/pg-failover-callback

  # rewind policy: data checksum should be enabled before using rewind
  use_pg_rewind: true
  remove_data_directory_on_rewind_failure: true
  remove_data_directory_on_diverged_timelines: false

  #------------------------------------------------------------#
  # how to create replica
  #------------------------------------------------------------#
  # create replica method: default pg_basebackup
  create_replica_methods:
    - basebackup
  basebackup:
    - max-rate: '1000M'
    - checkpoint: fast
    - verbose
    - progress
  pgbackrest:
    command: /usr/bin/pgbackrest --stanza=pg-el7test --delta restore
    keep_data: true
    no_params: true
    no_leader: true
    no_leader: true

  #------------------------------------------------------------#
  # ad hoc parameters (overwrite with default)
  #------------------------------------------------------------#
  # parameters:

  #------------------------------------------------------------#
  # host based authentication, overwrite default pg_hba.conf
  #------------------------------------------------------------#
  # pg_hba:
  #   - local  all          postgres             ident
  #   - local  all          all                  scram-sha-256
  #   - host   all          all       0.0.0.0/0  scram-sha-256
  #   - local  replication  postgres             ident
  #   - local  replication  all                  scram-sha-256
  #   - host   replication  all       0.0.0.0/0  scram-sha-256

...
patronictl show-config
These are two 3-node clusters on el7/el8


[09-24 00:45:41] vagrant@build-el7:~
$ pg show-config pg-el7test
{}



```bash
[09-24 00:45:41] vagrant@build-el7:~
$ pg show-config pg-el8test
{}
And when I remove patroni 3.1.1 and replace it with 3.0.4 on the el8 cluster, it can print the patroni dynamic config
...
failsafe_mode: true
loop_wait: 10
maximum_lag_on_failover: 1048576
maximum_lag_on_syncnode: -1
pg_cluster: pg-el8test
pg_group: 0
pg_shard: pg-el8test
pg_version: 15
postgresql:
  parameters:
    archive_command: pgbackrest --stanza=pg-el8test archive-push %p
    archive_mode: true
    archive_timeout: 300
    auto_explain.log_analyze: true



### Patroni log files


nothing special. Leader Lock info only



### PostgreSQL log files

```shell
nothing special.

But there's no `dynamic.json` file either.





### Have you tried to use GitHub issue search?

- [X] Yes



### Anything else we need to know?

Closing comment (2023-09-25T06:44:37Z): Duplicate of #2879