Repository: zalando/patroni
Pull request #2874 (2023-09-20 13:15:07)

Add a documentation page for `patronictl`
This PR introduces a documentation page for `patronictl` application.

We adopted a top-down approach when writing this document. We start by describing the outer most parts, and then keep writing new sections that specialize the knowledge.

We basically added a section called `patronictl` to the left menu. Inside that section we created a page with this structure:

- `patronictl`: describes what it is
    - `Configuraiton`: how to configure `patronictl`
    - `Usage`: how to use the CLI. Inside this section, there are subsections for each of the subcommands exposed by `patronictl`, and each of them are described using the following subsubsections:
        - `Synopsis`: syntax of the command and its positional and optional arguments
        - `Description`: a description of what the command does
        - `Parameters`: a detailed description of the arguments and how to use them
        - `Examples`: one or more examples of execution of the command

References: PAT-200.

hughcapet: one thought straight away - should we add references to the new page for the existing occurrences of patronictl in the other parts of the docs?
barthisrael: > one thought straight away - should we add references to the new page for the existing occurrences of patronictl in the other parts of the docs?

Hmm, good point, I guess so.
coveralls: ## Pull Request Test Coverage Report for [Build 6249125129](https://coveralls.io/builds/62781635)

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage remained the same at **99.862%**

---



|  Totals | [![Coverage Status](https://coveralls.io/builds/62781635/badge)](https://coveralls.io/builds/62781635) |
| :-- | --: |
| Change from base [Build 6248911064](https://coveralls.io/builds/62781118): |  0.0% |
| Covered Lines: | 13070 |
| Relevant Lines: | 13088 |

---
##### 💛  - [Coveralls](https://coveralls.io)


Patch:
diff --git a/docs/ENVIRONMENT.rst b/docs/ENVIRONMENT.rst
index e3762aca..f859649d 100644
--- a/docs/ENVIRONMENT.rst
+++ b/docs/ENVIRONMENT.rst
@@ -209,10 +209,10 @@ REST API
 CTL
 ---
 -  **PATRONICTL\_CONFIG\_FILE**: (optional) location of the configuration file.
--  **PATRONI\_CTL\_USERNAME**: (optional) Basic-auth username for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "username" parameter.
--  **PATRONI\_CTL\_PASSWORD**: (optional) Basic-auth password for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "password" parameter.
+-  **PATRONI\_CTL\_USERNAME**: (optional) Basic-auth username for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "username" parameter.
+-  **PATRONI\_CTL\_PASSWORD**: (optional) Basic-auth password for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "password" parameter.
 -  **PATRONI\_CTL\_INSECURE**: (optional) Allow connections to REST API without verifying SSL certs.
--  **PATRONI\_CTL\_CACERT**: (optional) Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided patronictl will use the value provided for REST API "cafile" parameter.
+-  **PATRONI\_CTL\_CACERT**: (optional) Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided :ref:`patronictl` will use the value provided for REST API "cafile" parameter.
 -  **PATRONI\_CTL\_CERTFILE**: (optional) Specifies the file with the client certificate in the PEM format.
 -  **PATRONI\_CTL\_KEYFILE**: (optional) Specifies the file with the client secret key in the PEM format.
 -  **PATRONI\_CTL\_KEYFILE\_PASSWORD**: (optional) Specifies a password for decrypting the client keyfile.
diff --git a/docs/citus.rst b/docs/citus.rst
index 030cf93e..084931df 100644
--- a/docs/citus.rst
+++ b/docs/citus.rst
@@ -57,7 +57,7 @@ clusters that are just logically groupped together using the
 PostgreSQL. Therefore in most cases it is not possible to manage them as a
 single entity.
 
-It results in two major differences in ``patronictl`` behaviour when
+It results in two major differences in :ref:`patronictl` behaviour when
 ``patroni.yaml`` has the ``citus`` section comparing with the usual:
 
 1. The ``list`` and the ``topology`` by default output all members of the Citus
@@ -65,12 +65,12 @@ It results in two major differences in ``patronictl`` behaviour when
    which Citus group they belong to.
 2. For all ``patronictl`` commands the new option is introduced, named
    ``--group``. For some commands the default value for the group might be
-   taken from the ``patroni.yaml``. For example, ``patronictl pause`` will
+   taken from the ``patroni.yaml``. For example, :ref:`patronictl_pause` will
    enable the maintenance mode by default for the ``group`` that is set in the
-   ``citus`` section, but for example for ``patronictl  switchover`` or
-   ``patronictl remove`` the group must be explicitly specified.
+   ``citus`` section, but for example for :ref:`patronictl_switchover` or
+   :ref:`patronictl_remove` the group must be explicitly specified.
 
-An example of ``patronictl list`` output for the Citus cluster::
+An example of :ref:`patronictl_list` output for the Citus cluster::
 
     postgres@coord1:~$ patronictl list demo
     + Citus cluster: demo ----------+--------------+---------+----+-----------+
@@ -115,7 +115,7 @@ the coordinator for the shards hosted on a worker node. The switchover then
 happens while the traffic is kept on the coordinator, and resumes as soon as a
 new primary worker node is ready to accept read-write queries.
 
-An example of ``patronictl switchover`` on the worker cluster::
+An example of :ref:`patronictl_switchover` on the worker cluster::
 
     postgres@coord1:~$ patronictl switchover demo
     + Citus cluster: demo ----------+--------------+---------+----+-----------+
@@ -343,7 +343,7 @@ Citus upgrades and PostgreSQL major upgrades
 
 First, please read about upgrading Citus version in the `documentation`__.
 There is one minor change in the process. When executing upgrade, you have to
-use ``patronictl restart`` instead of ``systemctl restart`` to restart
+use :ref:`patronictl_restart` instead of ``systemctl restart`` to restart
 PostgreSQL.
 
 __ https://docs.citusdata.com/en/latest/admin_guide/upgrading_citus.html
diff --git a/docs/contributing_guidelines.rst b/docs/contributing_guidelines.rst
index 05d98987..bfa1f4c5 100644
--- a/docs/contributing_guidelines.rst
+++ b/docs/contributing_guidelines.rst
@@ -3,21 +3,25 @@
 Contributing guidelines
 =======================
 
-Wanna contribute to Patroni? Yay - here is how!
-
 Chatting
 --------
 
-Just want to chat with other Patroni users?  Looking for interactive troubleshooting help? Join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://pgtreats.info/slack-invite>`__.
+If you have a question, looking for an interactive troubleshooting help or want to chat with other Patroni users, join us on channel `#patroni <https://postgresteam.slack.com/archives/C9XPYG92A>`__ in the `PostgreSQL Slack <https://pgtreats.info/slack-invite>`__.
+
+Reporting bugs
+--------------
+
+Before reporting a bug please make sure to **reproduce it with the latest Patroni version**!
+Also please double check if the issue already exists in our `Issues Tracker <https://github.com/zalando/patroni/issues>`__.
 
 Running tests
 -------------
 
 Requirements for running behave tests:
 
-1. PostgreSQL packages need to be installed.
-2. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`.
-3. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`.
+#. PostgreSQL packages including `contrib <https://www.postgresql.org/docs/current/contrib.html>`__ modules need to be installed.
+#. PostgreSQL binaries must be available in your `PATH`. You may need to add them to the path with something like `PATH=/usr/lib/postgresql/11/bin:$PATH python -m behave`.
+#. If you'd like to test with external DCSs (e.g., Etcd, Consul, and Zookeeper) you'll need the packages installed and respective services running and accepting unencrypted/unprotected connections on localhost and default port. In the case of Etcd or Consul, the behave test suite could start them up if binaries are available in the `PATH`.
 
 Install dependencies:
 
@@ -163,19 +167,12 @@ the watchdog behave feature test scenario with all versions of Postgres.
 
 Of course you can combine the two.
 
-Reporting issues
-----------------
-
-If you have a question about patroni or have a problem using it, please read the :ref:`README <readme>` before filing an issue.
-Also double check with the current issues on our `Issues Tracker <https://github.com/zalando/patroni/issues>`__.
-
 Contributing a pull request
 ---------------------------
 
-1) Submit a comment to the relevant issue or create a new issue describing your proposed change.
-2) Do a fork, develop and test your code changes.
-3) Include documentation
-4) Submit a pull request.
+#. Fork the repository, develop and test your code changes.
+#. Reflect changes in the user documentation.
+#. Submit a pull request with a clear description of the changes objective. Link an existing issue if necessary.
 
 You'll get feedback about your pull request as soon as possible.
 
diff --git a/docs/dcs_failsafe_mode.rst b/docs/dcs_failsafe_mode.rst
index e4eb6061..e6ce363e 100644
--- a/docs/dcs_failsafe_mode.rst
+++ b/docs/dcs_failsafe_mode.rst
@@ -60,4 +60,4 @@ F.A.Q.
 
 - How to enable the Failsafe Mode?
 
-  Before enabling the ``failsafe_mode`` please make sure that Patroni version on all members is up-to-date. After that, you can use either the ``PATCH /config`` :ref:`REST API <rest_api>` or ``patronictl edit-config -s failsafe_mode=true``
+  Before enabling the ``failsafe_mode`` please make sure that Patroni version on all members is up-to-date. After that, you can use either the ``PATCH /config`` :ref:`REST API <rest_api>` or :ref:`patronictl edit-config -s failsafe_mode=true <patronictl_edit_config_parameters>`
diff --git a/docs/dynamic_configuration.rst b/docs/dynamic_configuration.rst
index f5bb4ee5..285bcab3 100644
--- a/docs/dynamic_configuration.rst
+++ b/docs/dynamic_configuration.rst
@@ -6,7 +6,7 @@ Dynamic Configuration Settings
 
 Dynamic configuration is stored in the DCS (Distributed Configuration Store) and applied on all cluster nodes.
 
-In order to change the dynamic configuration you can use either ``patronictl edit-config`` tool or Patroni :ref:`REST API <rest_api>`.
+In order to change the dynamic configuration you can use either :ref:`patronictl_edit_config` tool or Patroni :ref:`REST API <rest_api>`.
 
 -  **loop\_wait**: the number of seconds the loop will sleep. Default value: 10
 -  **ttl**: the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30
@@ -46,7 +46,7 @@ In order to change the dynamic configuration you can use either ``patronictl edi
    -  **archive\_cleanup\_command**: cleanup command for standby leader
    -  **recovery\_min\_apply\_delay**: how long to wait before actually apply WAL records on a standby leader
 
--  **slots**: define permanent replication slots. These slots will be preserved during switchover/failover. Permanent slots that don't exist will be created by Patroni. The physical slots are maintained only in the current primary. The logical slots are copied from the primary to a standby with restart, and after that their position advanced every **loop_wait** seconds (if necessary). Copying logical slot files performed via ``libpq`` connection and using either rewind or superuser credentials (see **postgresql.authentication** section). There is always a chance that the logical slot position on the replica is a bit behind the former primary, therefore application should be prepared that some messages could be received the second time after the failover. The easiest way of doing so - tracking ``confirmed_flush_lsn``. Enabling permanent logical replication slots requires **postgresql.use_slots** to be set and will also automatically enable the ``hot_standby_feedback``. Since the failover of logical replication slots is unsafe on PostgreSQL 9.6 and older and PostgreSQL version 10 is missing some important functions, the feature only works with PostgreSQL 11+.
+-  **slots**: define permanent replication slots. These slots will be preserved during switchover/failover. Permanent slots that don't exist will be created by Patroni. With PostgreSQL 11 onwards permanent physical slots are created on all nodes and their position is advanced every **loop_wait** seconds. For PostgreSQL versions older than 11 permanent physical replication slots are maintained only on the current primary. The logical slots are copied from the primary to a standby with restart, and after that their position advanced every **loop_wait** seconds (if necessary). Copying logical slot files performed via ``libpq`` connection and using either rewind or superuser credentials (see **postgresql.authentication** section). There is always a chance that the logical slot position on the replica is a bit behind the former primary, therefore application should be prepared that some messages could be received the second time after the failover. The easiest way of doing so - tracking ``confirmed_flush_lsn``. Enabling permanent replication slots requires **postgresql.use_slots** to be set to ``true``. If there are permanent logical replication slots defined Patroni will automatically enable the ``hot_standby_feedback``. Since the failover of logical replication slots is unsafe on PostgreSQL 9.6 and older and PostgreSQL version 10 is missing some important functions, the feature only works with PostgreSQL 11+.
 
    -  **my\_slot\_name**: the name of the permanent replication slot. If the permanent slot name matches with the name of the current leader it will not be created. Please note that Patroni does not make checks for permanent slot names added to this configuration matching those that Patroni creates automatically for members. If those names are added, Patroni will ensure that any slots that were created are not removed even if the member becomes unresponsive, situation which would normally result in the slot's removal by Patroni. Although this can be useful in some situations, such as when importing existing members to a new Patroni cluster (see :ref:`Convert a Standalone to a Patroni Cluster <existing_data>` for details), caution should be exercised by the operator that these clashes in names are not persisted in the DCS due to its effect on normal functioning of Patroni.
 
@@ -80,4 +80,17 @@ Note: **slots** is a hashmap while **ignore_slots** is an array. For example:
             plugin: test_decoding
           - name: ignored_physical_slot_name
             type: physical
-          ...
\ No newline at end of file
+          ...
+
+Note: if cluster topology is static (fixed number of nodes that never change their names) you can configure permanent physical replication slots with names corresponding to names of nodes to avoid recycling of WAL files while replica is temporary down:
+
+.. code:: YAML
+
+        slots:
+          node_name1:
+            type: physical
+          node_name2:
+            type: physical
+          node_name3:
+            type: physical
+          ...
diff --git a/docs/existing_data.rst b/docs/existing_data.rst
index cb07bfa9..442b0b1c 100644
--- a/docs/existing_data.rst
+++ b/docs/existing_data.rst
@@ -42,12 +42,12 @@ You can find below an overview of steps for converting an existing Postgres clus
 
    #. Start Patroni using the ``patroni`` systemd service unit. It automatically detects that Postgres is already running and starts monitoring the instance.
 
-#. Hand over Postgres "start up procedure" to Patroni. In order to do that you need to restart the cluster members through ``patronictl restart cluster-name member-name`` command. For minimal downtime you might want to split this step into:
+#. Hand over Postgres "start up procedure" to Patroni. In order to do that you need to restart the cluster members through :ref:`patronictl restart cluster-name member-name <patronictl_restart_parameters>` command. For minimal downtime you might want to split this step into:
 
    #. Immediate restart of the standby nodes.
    #. Scheduled restart of the primary node within a maintenance window.
 
-#. If you configured permanent slots in step ``1.2.``, then you should remove them from ``slots`` configuration through ``patronictl edit-config cluster-name member-name`` command once the ``restart_lsn`` of the slots created by Patroni is able to catch up with the ``restart_lsn`` of the original slots for the corresponding members. By removing the slots from ``slots`` configuration you will allow Patroni to drop the original slots from your cluster once they are not needed anymore. You can find below an example query to check the ``restart_lsn`` of a couple slots, so you can compare them:
+#. If you configured permanent slots in step ``1.2.``, then you should remove them from ``slots`` configuration through :ref:`patronictl edit-config cluster-name member-name <patronictl_edit_config_parameters>` command once the ``restart_lsn`` of the slots created by Patroni is able to catch up with the ``restart_lsn`` of the original slots for the corresponding members. By removing the slots from ``slots`` configuration you will allow Patroni to drop the original slots from your cluster once they are not needed anymore. You can find below an example query to check the ``restart_lsn`` of a couple slots, so you can compare them:
 
    .. code-block:: sql
 
@@ -73,7 +73,7 @@ The only possible way to do a major upgrade currently is:
 #. Stop Patroni
 #. Upgrade PostgreSQL binaries and perform `pg_upgrade <https://www.postgresql.org/docs/current/pgupgrade.html>`_ on the primary node
 #. Update patroni.yml
-#. Remove the initialize key from DCS or wipe complete cluster state from DCS. The second one could be achieved by running ``patronictl remove <cluster-name>``. It is necessary because pg_upgrade runs initdb which actually creates a new database with a new PostgreSQL system identifier.
+#. Remove the initialize key from DCS or wipe complete cluster state from DCS. The second one could be achieved by running :ref:`patronictl remove cluster-name <patronictl_remove_parameters>` . It is necessary because pg_upgrade runs initdb which actually creates a new database with a new PostgreSQL system identifier.
 #. If you wiped the cluster state in the previous step, you may wish to copy patroni.dynamic.json from old data dir to the new one.  It will help you to retain some PostgreSQL parameters you had set before.
 #. Start Patroni on the primary node.
 #. Upgrade PostgreSQL binaries, update patroni.yml and wipe the data_dir on standby nodes.
diff --git a/docs/index.rst b/docs/index.rst
index c8f94aaf..f5a4f2d4 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -25,6 +25,7 @@ Currently supported PostgreSQL versions: 9.3 to 16.
    installation
    patroni_configuration
    rest_api
+   patronictl
    replica_bootstrap
    replication_modes
    watchdog
diff --git a/docs/patroni_configuration.rst b/docs/patroni_configuration.rst
index ef5ca020..3978e48d 100644
--- a/docs/patroni_configuration.rst
+++ b/docs/patroni_configuration.rst
@@ -15,7 +15,7 @@ There are 3 types of Patroni configuration:
 
 - Global :ref:`dynamic configuration <dynamic_configuration>`.
 	These options are stored in the DCS (Distributed Configuration Store) and applied on all cluster nodes.
-	Dynamic configuration can be set at any time using ``patronictl edit-config`` tool or Patroni :ref:`REST API <rest_api>`.
+	Dynamic configuration can be set at any time using :ref:`patronictl_edit_config` tool or Patroni :ref:`REST API <rest_api>`.
 	If the options changed are not part of the startup configuration, they are applied asynchronously (upon the next wake up cycle)
 	to every node, which gets subsequently reloaded.
 	If the node requires a restart to apply the configuration (for `PostgreSQL parameters <https://www.postgresql.org/docs/current/view-pg-settings.html>`__ with context postmaster, if their values
@@ -24,7 +24,7 @@ There are 3 types of Patroni configuration:
 
 - Local :ref:`configuration file <yaml_configuration>` (patroni.yml).
 	These options are defined in the configuration file and take precedence over dynamic configuration.
-	``patroni.yml`` can be changed and reloaded at runtime (without restart of Patroni) by sending SIGHUP to the Patroni process, performing ``POST /reload`` REST-API request or executing ``patronictl reload``. Local configuration can be either a single YAML file or a directory. When it is a directory, all YAML files in that directory are loaded one by one in sorted order. In case a key is defined in multiple files, the occurrence in the last file takes precedence.
+	``patroni.yml`` can be changed and reloaded at runtime (without restart of Patroni) by sending SIGHUP to the Patroni process, performing ``POST /reload`` REST-API request or executing :ref:`patronictl_reload`. Local configuration can be either a single YAML file or a directory. When it is a directory, all YAML files in that directory are loaded one by one in sorted order. In case a key is defined in multiple files, the occurrence in the last file takes precedence.
 
 - :ref:`Environment configuration <environment>`.
 	It is possible to set/override some of the "Local" configuration parameters with environment variables.
@@ -105,10 +105,10 @@ Changing these parameters require a PostgreSQL restart to take effect, and their
 
 As explained before, Patroni restrict changing their values through :ref:`dynamic configuration <dynamic_configuration>`, which usually consists of:
 
-1. Applying changes through ``patronictl edit-config`` (or via REST API ``/config`` endpoint)
-2. Restarting nodes through ``patronictl restart`` (or via REST API ``/restart`` endpoint)
+1. Applying changes through :ref:`patronictl_edit_config` (or via REST API ``/config`` endpoint)
+2. Restarting nodes through :ref:`patronictl_restart` (or via REST API ``/restart`` endpoint)
 
-**Note:** please keep in mind that you should perform a restart of the PostgreSQL nodes through ``patronictl restart`` command, or via REST API ``/restart`` endpoint. An attempt to restart PostgreSQL by restarting the Patroni daemon, e.g. by executing ``systemctl restart patroni``, can cause a failover to occur in the cluster, if you are restarting the primary node.
+**Note:** please keep in mind that you should perform a restart of the PostgreSQL nodes through :ref:`patronictl_restart` command, or via REST API ``/restart`` endpoint. An attempt to restart PostgreSQL by restarting the Patroni daemon, e.g. by executing ``systemctl restart patroni``, can cause a failover to occur in the cluster, if you are restarting the primary node.
 
 However, as those settings manage shared memory, some extra care should be taken when restarting the nodes:
 
diff --git a/docs/patronictl.rst b/docs/patronictl.rst
new file mode 100644
index 00000000..8b8cfb7e
--- /dev/null
+++ b/docs/patronictl.rst
@@ -0,0 +1,1975 @@
+.. _patronictl:
+
+patronictl
+==========
+
+Patroni has a command-line interface named ``patronictl``, which is used basically to interact with Patroni's REST API and with the DCS. It is intended to make it easier to perform operations in the cluster, and can easily be used by humans or scripts.
+
+.. _patronictl_configuration:
+
+Configuration
+-------------
+
+``patronictl`` uses 3 sections of the configuration:
+
+- **ctl**: how to authenticate against the Patroni REST API, and how to validate the server identity. Refer to :ref:`ctl settings <patronictl_settings>` for more details;
+- **restapi**: how to authenticate against the Patroni REST API, and how to validate the server identity. Only used if ``ctl`` configuration is not enough. ``patronictl`` is mainly interested in ``restapi.authentication`` section (in case ``ctl.authentication`` is missing) and ``restapi.cafile`` setting (in case ``ctl.cacert`` is missing). Refer to :ref:`REST API settings <restapi_settings>` for more details;
+- DCS (e.g. **etcd**): how to contact and authenticate against the DCS used by Patroni.
+
+Those configuration options can come either from environment variables or from a configuration file. Look for the above sections in :ref:`Environment Configuration Settings <environment>` or :ref:`YAML Configuration Settings <yaml_configuration>` to understand how you can set the options for them through environment variables or through a configuration file.
+
+If you opt for using environment variables, it's a straight forward approach. Patronictl will read the environment variables and use their values.
+
+If you opt for using a configuration file, you have different ways to inform ``patronictl`` about the file to be used. By default ``patronictl`` will attempt to load a configuration file named ``patronictl.yaml``, which is expected to be found under either of these paths, according to your system:
+
+- Mac OS X: ``~/Library/Application Support/patroni``
+- Mac OS X (POSIX): ``~/.patroni``
+- Unix: ``~/.config/patroni``
+- Unix (POSIX): ``~/.patroni``
+- Windows (roaming): ``C:\Users\<user>\AppData\Roaming\patroni``
+- Windows (not roaming): ``C:\Users\<user>\AppData\Local\patroni``
+
+You can override that behavior either by:
+
+- Setting the environment variable ``PATRONICTL_CONFIG_FILE`` with the path to a custom configuration file;
+- Using the ``-c`` / ``--config-file`` command-line argument of ``patronictl`` with the path to a custom configuration file.
+
+.. note::
+    If you are running ``patronictl`` in the same host as ``patroni`` daemon is running, you may just use the same configuration file if it contains all the configuration sections required by ``patronictl``.
+
+.. _patronictl_usage:
+
+Usage
+-----
+
+``patronictl`` exposes several handy operations. This section is intended to describe each of them.
+
+Before jumping into each of the sub-commands of ``patronictl``, be aware that ``patronictl`` itself has the following command-line arguments:
+
+``-c`` / ``--config-file``
+    As explained before, used to provide a path to a configuration file for ``patronictl``.
+
+``-d`` / ``--dcs-url`` / ``--dcs``
+    Provide a connection string to the DCS used by Patroni.
+
+    This argument can be used either to override the DCS settings from the ``patronictl`` configuration, or to define it if it's missing in the configuration.
+
+    The value should be in the format ``DCS://HOST:PORT``, e.g. ``etcd3://localhost:2379`` to connect to etcd v3 running on ``localhost``.
+
+``-k`` / ``--insecure``
+    Flag to bypass validation of REST API server SSL certificate.
+
+This is the synopsis for running a command from the ``patronictl``:
+
+.. code:: text
+
+    patronictl [ { -c | --config-file } CONFIG_FILE ]
+      [ { -d | --dcs-url | --dcs } DCS_URL ] 
+      [ { -k | --insecure } ]
+      SUBCOMMAND
+
+.. note::
+
+    This is the syntax for the synopsis:
+
+    - Options between square brackets are optional;
+    - Options between curly brackets represent a "choose one of set" operation;
+    - Options with ``[, ... ]`` can be specified multiple times;
+    - Things written in uppercase represent a literal that should be given a value to.
+
+    We will use this same syntax when describing ``patronictl`` sub-commands in the following sub-sections.
+    Also, when describing sub-commands in the following sub-sections, the commands' synposis should be seen as a replacement for the ``SUBCOMMAND`` in the above synopsis.
+
+In the following sub-sections you can find a description of each command implemented by ``patronictl``. For sake of example, we will use the configuration files present in the GitHub repository of Patroni (files ``postgres0.yml``, ``postgres1.yml`` and ``postgres2.yml``).
+
+.. _patronictl_dsn:
+
+patronictl dsn
+^^^^^^^^^^^^^^
+
+.. _patronictl_dsn_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    dsn
+      [ CLUSTER_NAME ]
+      [ { { -r | --role } { leader | primary | standby-leader | replica | standby | any } | { -m | --member } MEMBER_NAME } ]
+      [ --group CITUS_GROUP ]
+
+.. _patronictl_dsn_description:
+
+Description
+"""""""""""
+
+``patronictl dsn`` gets the connection string for one member of the Patroni cluster.
+
+If multiple members match the parameters of this command, one of them will be chosen, prioritizing the primary node.
+
+.. _patronictl_dsn_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``-r`` / ``--role``
+    Choose a member that has the given role.
+
+    Role can be one of:
+
+    - ``leader``: the leader of either a regular Patroni cluster or a standby Patroni cluster; or
+    - ``primary``: the leader of a regular Patroni cluster; or
+    - ``standby-leader``: the leader of a standby Patroni cluster; or
+    - ``replica``: a replica of a Patroni cluster; or
+    - ``standby``: same as ``replica``; or
+    - ``any``: any role. Same as omitting this parameter; or
+
+``-m`` / ``--member``
+    Choose a member of the cluster with the given name.
+
+    ``MEMBER_NAME`` is the name of the member.
+
+``--group``
+    Choose a member that is part of the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+.. _patronictl_dsn_examples:
+
+Examples
+""""""""
+
+Get DSN of the primary node:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml dsn batman -r primary
+    host=127.0.0.1 port=5432
+
+Get DSN of the node named ``postgresql1``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml dsn batman --member postgresql1
+    host=127.0.0.1 port=5433
+
+.. _patronictl_edit_config:
+
+patronictl edit-config
+^^^^^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_edit_config_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    edit-config
+      [ CLUSTER_NAME ]
+      [ --group CITUS_GROUP ]
+      [ { -q | --quiet } ]
+      [ { -s | --set } CONFIG="VALUE" [, ... ] ]
+      [ { -p | --pg } PG_CONFIG="PG_VALUE" [, ... ] ]
+      [ { --apply | --replace } CONFIG_FILE ]
+      [ --force ]
+
+.. _patronictl_edit_config_description:
+
+Description
+"""""""""""
+
+``patronictl edit-config`` changes the dynamic configuration of the cluster and updates the DCS with that.
+
+.. note::
+    When invoked through a TTY the command attempts to show a diff of the dynamic configuration through a pager. By default, it attempts to use either ``less`` or ``more``. If you want a different pager, set the ``PAGER`` environment variable with the desired one.
+
+.. _patronictl_edit_config_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Change dynamic configuration of the given Citus group.
+    
+    If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``-q`` / ``--quiet``
+    Flag to skip showing the configuration diff.
+
+``-s`` / ``--set``
+    Set a given dynamic configuration option with a given value.
+
+    ``CONFIG`` is the name of the dynamic configuration path in the YAML tree, with levels joined by ``.`` .
+
+    ``VALUE`` is the value for ``CONFIG``. If it is ``null``, then ``CONFIG`` will be removed from the dynamic configuration.
+
+``-p`` / ``--pg``
+    Set a given dynamic Postgres configuration option with the given value.
+
+    It is essentially a shorthand for ``--s`` / ``--set`` with ``CONFIG`` prepended with ``postgresql.parameters.``.
+
+    ``PG_CONFIG`` is the name of the Postgres configuration to be set.
+
+    ``PG_VALUE`` is the value for ``PG_CONFIG``. If it is ``nulll``, then ``PG_CONFIG`` will be removed from the dynamic configuration.
+
+``--apply``
+    Apply dynamic configuration from the given file.
+
+    It is similar to specifying multiple ``-s`` / ``--set`` options, one for each configuration from ``CONFIG_FILE``.
+
+    ``CONFIG_FILE`` is the path to a file containing the dynamic configuration to be applied, in YAML format. Use ``-`` if you want to read from ``stdin``.
+
+``--replace``
+    Replace the dynamic configuration in the DCS with the dynamic configuration specified in the given file.
+
+    ``CONFIG_FILE`` is the path to a file containing the new dynamic configuration to take effect, in YAML format. Use ``-`` if you want to read from ``stdin``.
+
+``--force``
+    Flag to skip confirmation prompts when changing the dynamic configuration.
+
+    Useful for scripts.
+
+.. _patronictl_edit_config_examples:
+
+Examples
+""""""""
+
+Change ``max_connections`` Postgres GUC:
+
+.. code:: diff
+
+    patronictl -c postgres0.yml edit-config batman --pg max_connections="150" --force
+    ---
+    +++
+    @@ -1,6 +1,8 @@
+    loop_wait: 10
+    maximum_lag_on_failover: 1048576
+    postgresql:
+    +  parameters:
+    +    max_connections: 150
+      pg_hba:
+      - host replication replicator 127.0.0.1/32 md5
+      - host all all 0.0.0.0/0 md5
+
+    Configuration changed
+
+Change ``loop_wait`` and ``ttl`` settings:
+
+.. code:: diff
+
+    patronictl -c postgres0.yml edit-config batman --set loop_wait="15" --set ttl="45" --force
+    ---
+    +++
+    @@ -1,4 +1,4 @@
+    -loop_wait: 10
+    +loop_wait: 15
+    maximum_lag_on_failover: 1048576
+    postgresql:
+      pg_hba:
+    @@ -6,4 +6,4 @@
+      - host all all 0.0.0.0/0 md5
+      use_pg_rewind: true
+    retry_timeout: 10
+    -ttl: 30
+    +ttl: 45
+
+    Configuration changed
+
+Remove ``maximum_lag_on_failover`` setting from dynamic configuration:
+
+.. code:: diff
+
+    patronictl -c postgres0.yml edit-config batman --set maximum_lag_on_failover="null" --force
+    ---
+    +++
+    @@ -1,5 +1,4 @@
+    loop_wait: 10
+    -maximum_lag_on_failover: 1048576
+    postgresql:
+      pg_hba:
+      - host replication replicator 127.0.0.1/32 md5
+
+    Configuration changed
+
+.. _patronictl_failover:
+
+patronictl failover
+^^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_failover_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    failover
+      [ CLUSTER_NAME ]
+      [ --group CITUS_GROUP ]
+      [ { --leader | --primary } LEADER_NAME ]
+      --candidate CANDIDATE_NAME
+      [ --force ]
+
+.. _patronictl_failover_description:
+
+Description
+"""""""""""
+
+``patronictl failover`` performs a manual failover in the cluster.
+
+It is designed to be used when the cluster is not healthy, e.g.:
+
+- There is no leader; or
+- There is no synchronous standby available in a synchronous cluster.
+
+It also allows to fail over to an asynchronous node if synchronous mode is enabled.
+
+.. note::
+    Nothing prevents you from running ``patronictl failover`` in a healthy cluster. However, we recommend using ``patronictl switchover`` in those cases.
+
+.. warning::
+    Triggering a failover can cause data loss depending on how up-to-date the promoted replica is in comparison to the primary.
+
+.. _patronictl_failover_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Perform a failover in the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``--leader`` / ``--primary``
+    Indicate who is the expected leader at failover time.
+
+    If given, a switchover is performed instead of a failover.
+
+    ``LEADER_NAME`` should match the name of the current leader in the cluster.
+
+    .. warning::
+        This argument is deprecated and will be removed in a future release.
+
+``--candidate``
+    The node to be promoted on failover.
+
+    ``CANDIDATE_NAME`` is the name of the node to be promoted.
+
+``--force``
+    Flag to skip confirmation prompts when performing the failover.
+
+    Useful for scripts.
+
+.. _patronictl_failover_examples:
+
+Examples
+""""""""
+
+Fail over to node ``postgresql2``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml failover batman --candidate postgresql2 --force
+    Current cluster topology
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  3 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  3 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  3 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    2023-09-12 11:52:27.50978 Successfully failed over to "postgresql2"
+    + Cluster: batman (7277694203142172922) -+---------+----+-----------+
+    | Member      | Host           | Role    | State   | TL | Lag in MB |
+    +-------------+----------------+---------+---------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Replica | stopped |    |   unknown |
+    | postgresql1 | 127.0.0.1:5433 | Replica | running |  3 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Leader  | running |  3 |           |
+    +-------------+----------------+---------+---------+----+-----------+
+
+.. _patronictl_flush:
+
+patronictl flush
+^^^^^^^^^^^^^^^^
+
+.. _patronictl_flush_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    flush
+      CLUSTER_NAME
+      [ MEMBER_NAME [, ... ] ]
+      { restart | switchover }
+      [ --group CITUS_GROUP ]
+      [ { -r | --role } { leader | primary | standby-leader | replica | standby | any } ]
+      [ --force ]
+
+.. _patronictl_flush_description:
+
+Description
+"""""""""""
+
+``patronictl flush`` discards scheduled events, if any.
+
+.. _patronictl_flush_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+``MEMBER_NAME``
+    Discard scheduled events for the given Patroni member(s).
+
+    Multiple members can be specified. If no members are specified, all of them are considered.
+
+    .. note::
+        Only used if discarding scheduled restart events.
+
+``restart``
+    Discard scheduled restart events.
+
+``switchover``
+    Discard scheduled switchover event.
+
+``--group``
+    Discard scheduled events from the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``-r`` / ``--role``
+    Discard scheduled events for members that have the given role.
+
+    Role can be one of:
+
+    - ``leader``: the leader of either a regular Patroni cluster or a standby Patroni cluster; or
+    - ``primary``: the leader of a regular Patroni cluster; or
+    - ``standby-leader``: the leader of a standby Patroni cluster; or
+    - ``replica``: a replica of a Patroni cluster; or
+    - ``standby``: same as ``replica``; or
+    - ``any``: any role. Same as omitting this parameter.
+
+    .. note::
+        Only used if discarding scheduled restart events.
+
+``--force``
+    Flag to skip confirmation prompts when performing the flush.
+
+    Useful for scripts.
+
+.. _patronictl_flush_examples:
+
+Examples
+""""""""
+
+Discard a scheduled switchover event:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml flush batman switchover --force
+    Success: scheduled switchover deleted
+
+Discard scheduled restart of all standby nodes:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml flush batman restart -r replica --force
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+---------------------------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB | Scheduled restart         |
+    +-------------+----------------+---------+-----------+----+-----------+---------------------------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |           | 2023-09-12T17:17:00+00:00 |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |         0 | 2023-09-12T17:17:00+00:00 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |         0 | 2023-09-12T17:17:00+00:00 |
+    +-------------+----------------+---------+-----------+----+-----------+---------------------------+
+    Success: flush scheduled restart for member postgresql1
+    Success: flush scheduled restart for member postgresql2
+
+Discard scheduled restart of nodes ``postgresql0`` and ``postgresql1``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml flush batman postgresql0 postgresql1 restart --force
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+---------------------------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB | Scheduled restart         |
+    +-------------+----------------+---------+-----------+----+-----------+---------------------------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |           | 2023-09-12T17:17:00+00:00 |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |         0 | 2023-09-12T17:17:00+00:00 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |         0 | 2023-09-12T17:17:00+00:00 |
+    +-------------+----------------+---------+-----------+----+-----------+---------------------------+
+    Success: flush scheduled restart for member postgresql0
+    Success: flush scheduled restart for member postgresql1
+
+.. _patronictl_history:
+
+patronictl history
+^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_history_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    history
+      [ CLUSTER_NAME ]
+      [ --group CITUS_GROUP ]
+      [ { -f | --format } { pretty | tsv | json | yaml } ]
+
+.. _patronictl_history_description:
+
+Description
+"""""""""""
+
+``patronictl history`` shows a history of failover and switchover events from the cluster, if any.
+
+The following information is included in the output:
+
+``TL``
+    Postgres timeline at which the event occurred.
+
+``LSN``
+    Postgres LSN at which the event occurred.
+
+``Reason``
+    Reason fetched from the Postgres ``.history`` file.
+
+``Timestamp``
+    Time when the event occurred.
+
+``New Leader``
+    Patroni member that has been promoted during the event.
+
+.. _patronictl_history_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Show history of events from the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+    
+    If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
+
+``-f`` / ``--format``
+    How to format the list of events in the output.
+
+    Format can be one of:
+
+    - ``pretty``: prints history as a pretty table; or
+    - ``tsv``: prints history as tabular information, with columns delimited by ``\t``; or
+    - ``json``: prints history in JSON format; or
+    - ``yaml``: prints history in YAML format.
+
+    The default is ``pretty``.
+
+``--force``
+    Flag to skip confirmation prompts when performing the flush.
+
+    Useful for scripts.
+
+.. _patronictl_history_examples:
+
+Examples
+""""""""
+
+Show the history of events:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml history batman
+    +----+----------+------------------------------+----------------------------------+-------------+
+    | TL |      LSN | Reason                       | Timestamp                        | New Leader  |
+    +----+----------+------------------------------+----------------------------------+-------------+
+    |  1 | 24392648 | no recovery target specified | 2023-09-11T22:11:27.125527+00:00 | postgresql0 |
+    |  2 | 50331864 | no recovery target specified | 2023-09-12T11:34:03.148097+00:00 | postgresql0 |
+    |  3 | 83886704 | no recovery target specified | 2023-09-12T11:52:26.948134+00:00 | postgresql2 |
+    |  4 | 83887280 | no recovery target specified | 2023-09-12T11:53:09.620136+00:00 | postgresql0 |
+    +----+----------+------------------------------+----------------------------------+-------------+
+
+Show the history of events in YAML format:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml history batman -f yaml
+    - LSN: 24392648
+      New Leader: postgresql0
+      Reason: no recovery target specified
+      TL: 1
+      Timestamp: '2023-09-11T22:11:27.125527+00:00'
+    - LSN: 50331864
+      New Leader: postgresql0
+      Reason: no recovery target specified
+      TL: 2
+      Timestamp: '2023-09-12T11:34:03.148097+00:00'
+    - LSN: 83886704
+      New Leader: postgresql2
+      Reason: no recovery target specified
+      TL: 3
+      Timestamp: '2023-09-12T11:52:26.948134+00:00'
+    - LSN: 83887280
+      New Leader: postgresql0
+      Reason: no recovery target specified
+      TL: 4
+      Timestamp: '2023-09-12T11:53:09.620136+00:00'
+
+.. _patronictl_list:
+
+patronictl list
+^^^^^^^^^^^^^^^
+
+.. _patronictl_list_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    list
+      [ CLUSTER_NAME [, ... ] ]
+      [ --group CITUS_GROUP ]
+      [ { -e | --extended } ]
+      [ { -t | --timestamp } ]
+      [ { -f | --format } { pretty | tsv | json | yaml } ]
+      [ { -W | { -w | --watch } TIME } ]
+
+.. _patronictl_list_description:
+
+Description
+"""""""""""
+
+``patronictl list`` shows information about Patroni cluster and its members.
+
+The following information is included in the output:
+
+``Cluster``
+    Name of the Patroni cluster.
+
+``Member``
+    Name of the Patroni member.
+
+``Host``
+    Host where the member is located.
+
+``Role``
+    Current role of the member.
+
+    Can be one among:
+
+    * ``Leader``: the current leader of a regular Patroni cluster; or
+    * ``Standby Leader``: the current leader of a Patroni standby cluster; or
+    * ``Sync Standby``: a synchronous standby of a Patroni cluster with synchronous mode enabled; or
+    * ``Replica``: a regular standby of a Patroni cluster.
+
+``State``
+    Current state of Postgres in the Patroni member.
+
+    Some examples among the possible states:
+
+    * ``running``: if Postgres is currently up and running;
+    * ``streaming``: if a replica and Postgres is currently streaming WALs from the primary node;
+    * ``in archive recovery``: if a replica and Postgres is currently fetching WALs from the archive;
+    * ``stopped``: if Postgres had been shut down;
+    * ``crashed``: if Postgres has crashed.
+
+``TL``
+    Current Postgres timeline in the Patroni member.
+
+``Lag in MB``
+    Amount worth of replication lag in megabytes between the Patroni member and its upstream.
+
+Besides that, the following information may be included in the output:
+
+``System identifier``
+    Postgres system identifier.
+
+    .. note::
+        Shown in the table header.
+
+        Only shown if output format is ``pretty``.
+
+``Group``
+    Citus group ID.
+
+    .. note::
+        Shown in the table header.
+
+        Only shown if a Citus cluster.
+
+``Pending restart``
+    ``*`` indicates that the node needs a restart for some Postgres configuration to take effect. An empty value indicates the node does not require a restart.
+
+    .. note::
+        Shown as a member attribute.
+
+        Shown if:
+
+        - Printing in ``pretty`` or ``tsv`` format and with extended output enabled; or
+        - If node requires a restart.
+
+``Scheduled restart``
+    Timestamp at which a restart has been scheduled for the Postgres instance managed by the Patroni member. An empty value indicates there is no scheduled restart for the member.
+
+    .. note::
+        Shown as a member attribute.
+
+        Shown if:
+
+        - Printing in ``pretty`` or ``tsv`` format and with extended output enabled; or
+        - If node has a scheduled restart.
+
+``Tags``
+    Contains tags set for the Patroni member. An empty value indicates that either no tags have been configured, or that they have been configured with default values.
+
+    .. note::
+        Shown as a member attribute.
+
+        Shown if:
+
+        - Printing in ``pretty`` or ``tsv`` format and with extended output enabled; or
+        - If node has any custom tags, or any default tags with non-default values.
+
+``Scheduled switchover``
+    Timestamp at which a switchover has been scheduled for the Patroni cluster, if any.
+
+    .. note::
+        Shown in the table footer.
+
+        Only shown if there is a scheduled switchover, and output format is ``pretty``.
+
+``Maintenance mode``
+
+    If the cluster monitoring is currently paused.
+
+    .. note::
+        Shown in the table footer.
+
+        Only shown if the cluster is paused, and output format is ``pretty``.
+
+.. _patronictl_list_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Show information about members from the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``-e`` / ``--extended``
+    Show extended information.
+
+    Force showing ``Pending restart``, ``Scheduled restart`` and ``Tags`` attributes, even if their value is empty.
+
+    .. note::
+        Only applies to ``pretty`` and ``tsv`` output formats.
+
+``-t`` / ``--timestamp``
+    Print timestamp before printing information about the cluster and its members.
+
+``-f`` / ``--format``
+    How to format the list of events in the output.
+
+    Format can be one of:
+
+    - ``pretty``: prints history as a pretty table; or
+    - ``tsv``: prints history as tabular information, with columns delimited by ``\t``; or
+    - ``json``: prints history in JSON format; or
+    - ``yaml``: prints history in YAML format.
+
+    The default is ``pretty``.
+
+``-W``
+    Automatically refresh information every 2 seconds.
+
+``-w`` / ``--watch``
+    Automatically refresh information at the specified interval.
+
+    ``TIME`` is the interval between refreshes, in seconds.
+
+.. _patronictl_list_examples:
+
+Examples
+""""""""
+
+Show information about the cluster in pretty format:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml list batman
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+
+Show information about the cluster in pretty format with extended columns:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml list batman -e
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+-----------------+-------------------+------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB | Pending restart | Scheduled restart | Tags |
+    +-------------+----------------+---------+-----------+----+-----------+-----------------+-------------------+------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |           |                 |                   |      |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |         0 |                 |                   |      |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |         0 |                 |                   |      |
+    +-------------+----------------+---------+-----------+----+-----------+-----------------+-------------------+------+
+
+Show information about the cluster in YAML format, with timestamp of execution:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml list batman -f yaml -t
+    2023-09-12 13:30:48
+    - Cluster: batman
+      Host: 127.0.0.1:5432
+      Member: postgresql0
+      Role: Leader
+      State: running
+      TL: 5
+    - Cluster: batman
+      Host: 127.0.0.1:5433
+      Lag in MB: 0
+      Member: postgresql1
+      Role: Replica
+      State: streaming
+      TL: 5
+    - Cluster: batman
+      Host: 127.0.0.1:5434
+      Lag in MB: 0
+      Member: postgresql2
+      Role: Replica
+      State: streaming
+      TL: 5
+
+.. _patronictl_pause:
+
+patronictl pause
+^^^^^^^^^^^^^^^^
+
+.. _patronictl_pause_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    pause
+      [ CLUSTER_NAME ]
+      [ --group CITUS_GROUP ]
+      [ --wait ]
+
+.. _patronictl_pause_description:
+
+Description
+"""""""""""
+
+``patronictl pause`` temporarily puts the Patroni cluster in maintenance mode and disables automatic failover.
+
+.. _patronictl_pause_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Pause the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+    
+    If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
+
+``--wait``
+    Wait until all Patroni members are paused before returning control to the caller.
+
+.. _patronictl_pause_examples:
+
+Examples
+""""""""
+
+Put the cluster in maintenance mode, and wait until all nodes have been paused:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml pause batman --wait
+    'pause' request sent, waiting until it is recognized by all nodes
+    Success: cluster management is paused
+
+.. _patronictl_query:
+
+patronictl query
+^^^^^^^^^^^^^^^^
+
+.. _patronictl_query_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    query
+      [ CLUSTER_NAME ]
+      [ --group CITUS_GROUP ]
+      [ { { -r | --role } { leader | primary | standby-leader | replica | standby | any } | { -m | --member } MEMBER_NAME } ]
+      [ { -d | --dbname } DBNAME ]
+      [ { -U | --username } USERNAME ]
+      [ --password ]
+      [ --format { pretty | tsv | json | yaml } ]
+      [ { { -f | --file } FILE_NAME | { -c | --command } SQL_COMMAND } ]
+      [ --delimiter ]
+      [ { -W | { -w | --watch } TIME } ]
+
+.. _patronictl_query_description:
+
+Description
+"""""""""""
+
+``patronictl query`` executes a SQL command or script against a member of the Patroni cluster.
+
+.. _patronictl_query_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Query the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``-r`` / ``--role``
+    Choose a member that has the given role.
+
+    Role can be one of:
+
+    - ``leader``: the leader of either a regular Patroni cluster or a standby Patroni cluster; or
+    - ``primary``: the leader of a regular Patroni cluster; or
+    - ``standby-leader``: the leader of a standby Patroni cluster; or
+    - ``replica``: a replica of a Patroni cluster; or
+    - ``standby``: same as ``replica``; or
+    - ``any``: any role. Same as omitting this parameter.
+
+``-m`` / ``--member``
+    Choose a member that has the given name.
+
+    ``MEMBER_NAME`` is the name of the member to be picked.
+
+``-d`` / ``--dbname``
+    Database to connect and run the query.
+
+    ``DBNAME`` is the name of the database. If not given, defaults to ``USERNAME``.
+
+``-U`` / ``--username``
+    User to connect to the database.
+
+    ``USERNAME`` name of the user. If not given, defaults to the operating system user running ``patronictl query``.
+
+``--password``
+    Prompt for the password of the connecting user.
+
+    As Patroni uses ``libpq``, alternatively you can create a ``~/.pgpass`` file or set the ``PGPASSWORD`` environment variable.
+
+``--format``
+    How to format the output of the query.
+
+    Format can be one of:
+
+    - ``pretty``: prints query output as a pretty table; or
+    - ``tsv``: prints query output as tabular information, with columns delimited by ``\t``; or
+    - ``json``: prints query output in JSON format; or
+    - ``yaml``: prints query output in YAML format.
+
+    The default is ``tsv``.
+
+``-f`` / ``--file``
+    Use a file as source of commands to run queries.
+
+    ``FILE_NAME`` is the path to the source file.
+
+``-c`` / ``--command``
+    Run the given SQL command in the query.
+
+    ``SQL_COMMAND`` is the SQL command to be executed.
+
+``--delimiter``
+    The delimiter when printing information in ``tsv`` format, or ``\t`` if omitted.
+
+``-W``
+    Automatically re-run the query every 2 seconds.
+
+``-w`` / ``--watch``
+    Automatically re-run the query at the specified interval.
+
+    ``TIME`` is the interval between re-runs, in seconds.
+
+.. _patronictl_query_examples:
+
+Examples
+""""""""
+
+Run a SQL command as ``postgres`` user, and ask for its password:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml query batman -U postgres --password -c "SELECT now()"
+    Password:
+    now
+    2023-09-12 18:10:53.228084+00:00
+
+Run a SQL command as ``postgres`` user, and take password from ``libpq`` environment variable:
+
+.. code:: bash
+
+    $ PGPASSWORD=zalando patronictl -c postgres0.yml query batman -U postgres -c "SELECT now()"
+    now
+    2023-09-12 18:11:37.639500+00:00
+
+Run a SQL command and print in ``pretty`` format every 2 seconds:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml query batman -c "SELECT now()" --format pretty -W
+    +----------------------------------+
+    | now                              |
+    +----------------------------------+
+    | 2023-09-12 18:12:16.716235+00:00 |
+    +----------------------------------+
+    +----------------------------------+
+    | now                              |
+    +----------------------------------+
+    | 2023-09-12 18:12:18.732645+00:00 |
+    +----------------------------------+
+    +----------------------------------+
+    | now                              |
+    +----------------------------------+
+    | 2023-09-12 18:12:20.750573+00:00 |
+    +----------------------------------+
+
+Run a SQL command on database ``test`` and print the output in YAML format:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml query batman -d test -c "SELECT now() AS column_1, 'test' AS column_2" --format yaml
+    - column_1: 2023-09-12 18:14:22.052060+00:00
+      column_2: test
+
+Run a SQL command on member ``postgresql2``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml query batman -m postgresql2 -c "SHOW port"
+    port
+    5434
+
+Run a SQL command on any of the standbys:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml query batman -r replica -c "SHOW port"
+    port
+    5433
+
+.. _patronictl_reinit:
+
+patronictl reinit
+^^^^^^^^^^^^^^^^^
+
+.. _patronictl_reinit_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    reinit
+      CLUSTER_NAME
+      [ MEMBER_NAME [, ... ] ]
+      [ --group CITUS_GROUP ]
+      [ --wait ]
+      [ --force ]
+
+.. _patronictl_reinit_description:
+
+Description
+"""""""""""
+
+``patronictl reinit`` rebuilds a Postgres standby instance managed by a replica member of the Patroni cluster.
+
+.. _patronictl_reinit_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+``MEMBER_NAME``
+    Name of the replica member for which the Postgres instance will be rebuilt.
+
+    Multiple replica members can be specified. If no members are specified, the command does nothing.
+
+``--group``
+    Rebuild a replica member of the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``--wait``
+    Wait until the reinitialization of the Postgres standby node(s) is finished.
+
+``--force``
+    Flag to skip confirmation prompts when rebuilding Postgres standby instances.
+
+    Useful for scripts.
+
+.. _patronictl_reinit_examples:
+
+Examples
+""""""""
+
+Request a rebuild of all replica members of the Patroni cluster and immediately return control to the caller:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml reinit batman postgresql1 postgresql2 --force
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    Success: reinitialize for member postgresql1
+    Success: reinitialize for member postgresql2
+
+Request a rebuild of ``postgresql2`` and wait for it to complete:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml reinit batman postgresql2 --wait --force
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    Success: reinitialize for member postgresql2
+    Waiting for reinitialize to complete on: postgresql2
+    Reinitialize is completed on: postgresql2
+
+.. _patronictl_reload:
+
+patronictl reload
+^^^^^^^^^^^^^^^^^
+
+.. _patronictl_reload_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    reload
+      CLUSTER_NAME
+      [ MEMBER_NAME [, ... ] ]
+      [ --group CITUS_GROUP ]
+      [ { -r | --role } { leader | primary | standby-leader | replica | standby | any } ]
+      [ --force ]
+
+.. _patronictl_reload_description:
+
+Description
+"""""""""""
+
+``patronictl reload`` requests a reload of local configuration for one or more Patroni members.
+
+It also triggers ``pg_ctl reload`` on the managed Postgres instance, even if nothing has changed.
+
+.. _patronictl_reload_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+``MEMBER_NAME``
+    Request a reload of local configuration for the given Patroni member(s).
+
+    Multiple members can be specified. If no members are specified, all of them are considered.
+
+``--group``
+    Request a reload of members of the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``-r`` / ``--role``
+    Select members that have the given role.
+
+    Role can be one of:
+
+    - ``leader``: the leader of either a regular Patroni cluster or a standby Patroni cluster; or
+    - ``primary``: the leader of a regular Patroni cluster; or
+    - ``standby-leader``: the leader of a standby Patroni cluster; or
+    - ``replica``: a replica of a Patroni cluster; or
+    - ``standby``: same as ``replica``; or
+    - ``any``: any role. Same as omitting this parameter.
+
+``--force``
+    Flag to skip confirmation prompts when requesting a reload of the local configuration.
+
+    Useful for scripts.
+
+.. _patronictl_reload_examples:
+
+Examples
+""""""""
+
+Request a reload of the local configuration of all members of the Patroni cluster:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml reload batman --force
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    Reload request received for member postgresql0 and will be processed within 10 seconds
+    Reload request received for member postgresql1 and will be processed within 10 seconds
+    Reload request received for member postgresql2 and will be processed within 10 seconds
+
+.. _patronictl_remove:
+
+patronictl remove
+^^^^^^^^^^^^^^^^^
+
+.. _patronictl_remove_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    remove
+      CLUSTER_NAME
+      [ --group CITUS_GROUP ]
+      [ { -f | --format } { pretty | tsv | json | yaml } ]
+
+.. _patronictl_remove_description:
+
+Description
+"""""""""""
+
+``patronictl remove`` removes information of the cluster from the DCS.
+
+It is an interactive action.
+
+.. warning::
+    This operation will destroy the information of the Patroni cluster from the DCS.
+
+.. _patronictl_remove_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+``--group``
+    Remove information about the Patroni cluster related with the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``-f`` / ``--format``
+    How to format the list of members in the output when prompting for confirmation.
+
+    Format can be one of:
+
+    - ``pretty``: prints members as a pretty table; or
+    - ``tsv``: prints members as tabular information, with columns delimited by ``\t``; or
+    - ``json``: prints members in JSON format; or
+    - ``yaml``: prints members in YAML format.
+
+    The default is ``pretty``.
+
+.. _patronictl_remove_examples:
+
+Examples
+""""""""
+
+Remove information about Patroni cluster ``batman`` from the DCS:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml remove batman
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  5 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  5 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  5 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    Please confirm the cluster name to remove: batman
+    You are about to remove all information in DCS for batman, please type: "Yes I am aware": Yes I am aware
+    This cluster currently is healthy. Please specify the leader name to continue: postgresql0
+
+.. _patronictl_restart:
+
+patronictl restart
+^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_restart_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    restart
+      CLUSTER_NAME
+      [ MEMBER_NAME [, ...] ]
+      [ --group CITUS_GROUP ]
+      [ { -r | --role } { leader | primary | standby-leader | replica | standby | any } ]
+      [ --any ]
+      [ --pg-version PG_VERSION ]
+      [ --pending ]
+      [ --timeout TIMEOUT ]
+      [ --scheduled TIMESTAMP ]
+      [ --force ]
+
+.. _patronictl_restart_description:
+
+Description
+"""""""""""
+
+``patronictl restart`` requests a restart of the Postgres instance managed by a member of the Patroni cluster.
+
+The restart can be performed immediately or scheduled for later.
+
+.. _patronictl_restart_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+``--group``
+    Restart the Patroni cluster related with the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``-r`` / ``--role``
+    Choose members that have the given role.
+
+    Role can be one of:
+
+    - ``leader``: the leader of either a regular Patroni cluster or a standby Patroni cluster; or
+    - ``primary``: the leader of a regular Patroni cluster; or
+    - ``standby-leader``: the leader of a standby Patroni cluster; or
+    - ``replica``: a replica of a Patroni cluster; or
+    - ``standby``: same as ``replica``; or
+    - ``any``: any role. Same as omitting this parameter.
+
+``--any``
+    Restart a single random node among the ones which match the given filters.
+
+``--pg-version``
+    Select only members which version of the managed Postgres instance is older than the given version.
+
+    ``PG_VERSION`` is the Postgres version to be compared.
+
+``--pending``
+    Select only members which are flagged as ``Pending restart``.
+
+``timeout``
+    Abort the restart if it takes more than the specified timeout, and fail over to a replica if the issue is on the primary.
+
+    ``TIMEOUT`` is the amount of seconds to wait before aborting the restart.
+
+``--scheduled``
+    Schedule a restart to occur at the given timestamp.
+
+    ``TIMESTAMP`` is the timestamp when the restart should occur. Specify it in unambiguous format, preferrably with time zone. You can also use the literal ``now`` for the restart to be executed immediately.
+
+``--force``
+    Flag to skip confirmation prompts when requesting the restart operations.
+
+    Useful for scripts.
+
+.. _patronictl_restart_examples:
+
+Examples
+""""""""
+
+Restart all members of the cluster immediately:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml restart batman --force
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  6 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  6 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  6 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    Success: restart on member postgresql0
+    Success: restart on member postgresql1
+    Success: restart on member postgresql2
+
+Restart a random member of the cluster immediately:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml restart batman --any --force
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  6 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  6 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  6 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    Success: restart on member postgresql1
+
+Schedule a restart to occur at ``2023-09-13T18:00-03:00``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml restart batman --scheduled 2023-09-13T18:00-03:00 --force
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  6 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  6 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  6 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    Success: restart scheduled on member postgresql0
+    Success: restart scheduled on member postgresql1
+    Success: restart scheduled on member postgresql2
+
+.. _patronictl_resume:
+
+patronictl resume
+^^^^^^^^^^^^^^^^^
+
+.. _patronictl_resume_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    resume
+      [ CLUSTER_NAME ]
+      [ --group CITUS_GROUP ]
+      [ --wait ]
+
+.. _patronictl_resume_description:
+
+Description
+"""""""""""
+
+``patronictl resume`` takes the Patroni cluster out of maintenance mode and re-enables automatic failover.
+
+.. _patronictl_resume_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Resume the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+    
+    If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
+
+``--wait``
+    Wait until all Patroni members are unpaused before returning control to the caller.
+
+.. _patronictl_resume_examples:
+
+Examples
+""""""""
+
+Put the cluster out of maintenance mode:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml resume batman --wait
+    'resume' request sent, waiting until it is recognized by all nodes
+    Success: cluster management is resumed
+
+.. _patronictl_show_config:
+
+patronictl show-config
+^^^^^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_show_config_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    show-config
+      [ CLUSTER_NAME ]
+      [ --group CITUS_GROUP ]
+
+.. _patronictl_show_config_description:
+
+Description
+"""""""""""
+
+``patronictl show-config`` shows the dynamic configuration of the cluster that is stored in the DCS.
+
+.. _patronictl_show_config_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Show dynamic configuration of the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+    
+    If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
+
+.. _patronictl_show_config_examples:
+
+Examples
+""""""""
+
+Show dynamic configuration of cluster ``batman``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml show-config batman
+    loop_wait: 10
+    postgresql:
+      parameters:
+        max_connections: 250
+      pg_hba:
+      - host replication replicator 127.0.0.1/32 md5
+      - host all all 0.0.0.0/0 md5
+      use_pg_rewind: true
+    retry_timeout: 10
+    ttl: 30
+
+.. _patronictl_switchover:
+
+patronictl switchover
+^^^^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_switchover_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    switchover
+      [ CLUSTER_NAME ]
+      [ --group CITUS_GROUP ]
+      [ { --leader | --primary } LEADER_NAME ]
+      --candidate CANDIDATE_NAME
+      [ --force ]
+
+.. _patronictl_switchover_description:
+
+Description
+"""""""""""
+
+``patronictl switchover`` performs a switchover in the cluster.
+
+It is designed to be used when the cluster is healthy, e.g.:
+
+- There is a leader;
+- There are synchronous standbys available in a synchronous cluster.
+
+.. note::
+    If your cluster is unhealthy you might be interested in ``patronictl failover`` instead.
+
+.. _patronictl_switchover_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Perform a switchover in the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``--leader`` / ``--primary``
+    Indicate who is the leader to be demoted at switchover time.
+
+    ``LEADER_NAME`` should match the name of the current leader in the cluster.
+
+``--candidate``
+    The node to be promoted on switchover, and take the primary role.
+
+    ``CANDIDATE_NAME`` is the name of the node to be promoted.
+
+``--scheduled``
+    Schedule a switchover to occur at the given timestamp.
+
+    ``TIMESTAMP`` is the timestamp when the switchover should occur. Specify it in unambiguous format, preferrably with time zone. You can also use the literal ``now`` for the switchover to be executed immediately.
+
+``--force``
+    Flag to skip confirmation prompts when performing the switchover.
+
+    Useful for scripts.
+
+.. _patronictl_switchover_examples:
+
+Examples
+""""""""
+
+Switch over with node ``postgresql2``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml switchover batman --leader postgresql0 --candidate postgresql2 --force
+    Current cluster topology
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  6 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  6 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  6 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    2023-09-13 14:15:23.07497 Successfully switched over to "postgresql2"
+    + Cluster: batman (7277694203142172922) -+---------+----+-----------+
+    | Member      | Host           | Role    | State   | TL | Lag in MB |
+    +-------------+----------------+---------+---------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Replica | stopped |    |   unknown |
+    | postgresql1 | 127.0.0.1:5433 | Replica | running |  6 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Leader  | running |  6 |           |
+    +-------------+----------------+---------+---------+----+-----------+
+
+Schedule a switchover between ``postgresql0`` and ``postgresql2`` to occur at ``2023-09-13T18:00:00-03:00``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml switchover batman --leader postgresql0 --candidate postgresql2 --scheduled 2023-09-13T18:00-03:00 --force
+    Current cluster topology
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  8 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  8 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  8 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    2023-09-13 14:18:11.20661 Switchover scheduled
+    + Cluster: batman (7277694203142172922) -+-----------+----+-----------+
+    | Member      | Host           | Role    | State     | TL | Lag in MB |
+    +-------------+----------------+---------+-----------+----+-----------+
+    | postgresql0 | 127.0.0.1:5432 | Leader  | running   |  8 |           |
+    | postgresql1 | 127.0.0.1:5433 | Replica | streaming |  8 |         0 |
+    | postgresql2 | 127.0.0.1:5434 | Replica | streaming |  8 |         0 |
+    +-------------+----------------+---------+-----------+----+-----------+
+    Switchover scheduled at: 2023-09-13T18:00:00-03:00
+                        from: postgresql0
+                        to: postgresql2
+
+.. _patronictl_topology:
+
+patronictl topology
+^^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_topology_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    topology
+      [ CLUSTER_NAME [, ... ] ]
+      [ --group CITUS_GROUP ]
+      [ { -W | { -w | --watch } TIME } ]
+
+.. _patronictl_topology_description:
+
+Description
+"""""""""""
+
+``patronictl topology`` shows information about the Patroni cluster and its members with a tree view approach.
+
+The following information is included in the output:
+
+``Cluster``
+    Name of the Patroni cluster.
+
+    .. note::
+        Shown in the table header.
+
+``System identifier``
+    Postgres system identifier.
+
+    .. note::
+        Shown in the table header.
+
+``Member``
+    Name of the Patroni member.
+
+    .. note::
+        Information in this column is shown as a tree view of members in terms of replication connections.
+
+``Host``
+    Host where the member is located.
+
+``Role``
+    Current role of the member.
+
+    Can be one among:
+
+    * ``Leader``: the current leader of a regular Patroni cluster; or
+    * ``Standby Leader``: the current leader of a Patroni standby cluster; or
+    * ``Sync Standby``: a synchronous standby of a Patroni cluster with synchronous mode enabled; or
+    * ``Replica``: a regular standby of a Patroni cluster.
+
+``State``
+    Current state of Postgres in the Patroni member.
+
+    Some examples among the possible states:
+
+    * ``running``: if Postgres is currently up and running;
+    * ``streaming``: if a replica and Postgres is currently streaming WALs from the primary node;
+    * ``in archive recovery``: if a replica and Postgres is currently fetching WALs from the archive;
+    * ``stopped``: if Postgres had been shut down;
+    * ``crashed``: if Postgres has crashed.
+
+``TL``
+    Current Postgres timeline in the Patroni member.
+
+``Lag in MB``
+    Amount worth of replication lag in megabytes between the Patroni member and its upstream.
+
+Besides that, the following information may be included in the output:
+
+``Group``
+    Citus group ID.
+
+    .. note::
+        Shown in the table header.
+
+        Only shown if a Citus cluster.
+
+``Pending restart``
+    ``*`` indicates the node needs a restart for some Postgres configuration to take effect. An empty value indicates the node does not require a restart.
+
+    .. note::
+        Shown as a member attribute.
+
+        Shown if node requires a restart.
+
+``Scheduled restart``
+    Timestamp at which a restart has been scheduled for the Postgres instance managed by the Patroni member. An empty value indicates there is no scheduled restart for the member.
+
+    .. note::
+        Shown as a member attribute.
+
+        Shown if node has a scheduled restart.
+
+``Tags``
+    Contains tags set for the Patroni member. An empty value indicates that either no tags have been configured, or that they have been configured with default values.
+
+    .. note::
+        Shown as a member attribute.
+
+        Shown if node has any custom tags, or any default tags with non-default values.
+
+``Scheduled switchover``
+    Timestamp at which a switchover has been scheduled for the Patroni cluster, if any.
+
+    .. note::
+        Shown in the table footer.
+
+        Only shown if there is a scheduled switchover.
+
+``Maintenance mode``
+
+    If the cluster monitoring is currently paused.
+
+    .. note::
+        Shown in the table footer.
+
+        Only shown if the cluster is paused.
+
+.. _patronictl_topology_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--group``
+    Show information about members from the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+``-W``
+    Automatically refresh information every 2 seconds.
+
+``-w`` / ``--watch``
+    Automatically refresh information at the specified interval.
+
+    ``TIME`` is the interval between refreshes, in seconds.
+
+.. _patronictl_topology_examples:
+
+Examples
+""""""""
+
+Show topology of the cluster ``batman`` -- ``postgresql1`` and ``postgresql2`` are replicating from ``postgresql0``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml topology batman
+    + Cluster: batman (7277694203142172922) ---+-----------+----+-----------+
+    | Member        | Host           | Role    | State     | TL | Lag in MB |
+    +---------------+----------------+---------+-----------+----+-----------+
+    | postgresql0   | 127.0.0.1:5432 | Leader  | running   |  8 |           |
+    | + postgresql1 | 127.0.0.1:5433 | Replica | streaming |  8 |         0 |
+    | + postgresql2 | 127.0.0.1:5434 | Replica | streaming |  8 |         0 |
+    +---------------+----------------+---------+-----------+----+-----------+
+
+.. _patronictl_version:
+
+patronictl version
+^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_version_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    version
+      [ CLUSTER_NAME [, ... ] ]
+      [ MEMBER_NAME [, ... ] ]
+      [ --group CITUS_GROUP ]
+
+.. _patronictl_version_description:
+
+Description
+"""""""""""
+
+``patronictl version`` gets the version of ``patronictl`` application. Besides that it may also include version information about Patroni clusters and their members.
+
+.. _patronictl_version_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+``MEMBER_NAME``
+    Name of the member of the Patroni cluster.
+
+``--group``
+    Consider a Patroni cluster with the given Citus group.
+
+    ``CITUS_GROUP`` is the ID of the Citus group.
+
+.. _patronictl_version_examples:
+
+Examples
+""""""""
+
+Get version of ``patronictl`` only:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml version
+    patronictl version 3.1.0
+
+Get version of ``patronictl`` and of all members of cluster ``batman``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml version batman
+    patronictl version 3.1.0
+
+    postgresql0: Patroni 3.1.0 PostgreSQL 15.2
+    postgresql1: Patroni 3.1.0 PostgreSQL 15.2
+    postgresql2: Patroni 3.1.0 PostgreSQL 15.2
+
+Get version of ``patronictl`` and of members ``postgresql1`` and ``postgresql2`` of cluster ``batman``:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml version batman postgresql1 postgresql2
+    patronictl version 3.1.0
+
+    postgresql1: Patroni 3.1.0 PostgreSQL 15.2
+    postgresql2: Patroni 3.1.0 PostgreSQL 15.2
diff --git a/docs/pause.rst b/docs/pause.rst
index 60a01a4c..9a74a2e0 100644
--- a/docs/pause.rst
+++ b/docs/pause.rst
@@ -32,6 +32,6 @@ When Patroni runs in a paused mode, it does not change the state of PostgreSQL,
 User guide
 ----------
 
-``patronictl`` supports ``pause`` and ``resume`` commands.
+``patronictl`` supports :ref:`pause <patronictl_pause>` and :ref:`resume <patronictl_resume>` commands.
 
 One can also issue a ``PATCH`` request to the ``{namespace}/{cluster}/config`` key with ``{"pause": true/false/null}``
diff --git a/docs/releases.rst b/docs/releases.rst
index 5d4af2cc..34c44047 100644
--- a/docs/releases.rst
+++ b/docs/releases.rst
@@ -3,6 +3,28 @@
 Release notes
 =============
 
+Version 3.1.2
+-------------
+
+**Bugfixes**
+
+- Fixed bug with ``wal_keep_size`` checks (Alexander Kukushkin)
+
+  The ``wal_keep_size`` is a GUC that normally has a unit and Patroni was failing to cast its value to ``int``. As a result the value of ``bootstrap.dcs`` was not written to the ``/config`` key afterwards.
+
+- Detect and resolve inconsistencies between ``/sync`` key and ``synchronous_standby_names`` (Alexander Kukushkin)
+
+  Normally, Patroni updates ``/sync`` and ``synchronous_standby_names`` in a very specific order, but in case of a bug or when someone manually reset ``synchronous_standby_names``, Patroni was getting into an inconsistent state. As a result it was possible that the failover happens to an asynchronous node.
+
+- Read GUC's values when joining running Postgres (Alexander Kukushkin)
+
+  When restarted in ``pause``, Patroni was discarding the ``synchronous_standby_names`` GUC from the ``postgresql.conf``. To solve it and avoid similar issues, Patroni will read GUC's value if it is joining an already running Postgres.
+
+- Silenced annoying warnings when checking for node uniqueness (Alexander Kukushkin)
+
+  ``WARNING`` messages are produced by ``urllib3`` if Patroni is quickly restarted.
+
+
 Version 3.1.1
 -------------
 
diff --git a/docs/replica_bootstrap.rst b/docs/replica_bootstrap.rst
index 42a7d3b0..5ae53103 100644
--- a/docs/replica_bootstrap.rst
+++ b/docs/replica_bootstrap.rst
@@ -191,7 +191,7 @@ There is no further relationship between the standby cluster and the primary
 cluster it replicates from, in particular, they must not share the same DCS
 scope if they use the same DCS. They do not know anything else from each other
 apart from replication information. Also, the standby cluster is not being
-displayed in ``patronictl list`` or ``patronictl topology`` output on the
+displayed in :ref:`patronictl_list` or :ref:`patronictl_topology` output on the
 primary cluster.
 
 For the sake of flexibility, you can specify methods of creating a replica and
diff --git a/docs/rest_api.rst b/docs/rest_api.rst
index 52d417ec..00272100 100644
--- a/docs/rest_api.rst
+++ b/docs/rest_api.rst
@@ -3,7 +3,7 @@
 Patroni REST API
 ================
 
-Patroni has a rich REST API, which is used by Patroni itself during the leader race, by the ``patronictl`` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. Below you will find the list of Patroni REST API endpoints.
+Patroni has a rich REST API, which is used by Patroni itself during the leader race, by the :ref:`patronictl` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring. Below you will find the list of Patroni REST API endpoints.
 
 Health check endpoints
 ----------------------
@@ -619,9 +619,9 @@ In the JSON body of the ``POST`` request you must specify the ``candidate`` fiel
 	:ref:`Be very careful <failover_healthcheck>` when using this endpoint, as this can cause data loss in certain situations. In most cases, :ref:`the switchover endpoint <switchover_api>` satisfies the administrator's needs. 
 
 
-``POST /switchover`` and ``POST /failover`` endpoints are used by ``patronictl switchover`` and ``patronictl failover``, respectively.
+``POST /switchover`` and ``POST /failover`` endpoints are used by :ref:`patronictl_switchover` and :ref:`patronictl_failover`, respectively.
 
-``DELETE /switchover`` is used by ``patronictl flush <cluster-name> switchover``.
+``DELETE /switchover`` is used by :ref:`patronictl flush cluster-name switchover <patronictl_flush_parameters>`.
 
 .. list-table:: Failover/Switchover comparison
    :widths: 25 25 25
@@ -680,15 +680,15 @@ Restart endpoint
 
 - ``DELETE /restart``: delete the scheduled restart
 
-``POST /restart`` and ``DELETE /restart`` endpoints are used by ``patronictl restart`` and ``patronictl flush <cluster-name> restart`` respectively.
+``POST /restart`` and ``DELETE /restart`` endpoints are used by :ref:`patronictl_restart` and :ref:`patronictl flush cluster-name restart <patronictl_flush_parameters>` respectively.
 
 
 Reload endpoint
 ---------------
 
-The ``POST /reload`` call will order Patroni to re-read and apply the configuration file. This is the equivalent of sending the ``SIGHUP`` signal to the Patroni process. In case you changed some of the Postgres parameters which require a restart (like **shared_buffers**), you still have to explicitly do the restart of Postgres by either calling the ``POST /restart`` endpoint or with the help of ``patronictl restart``.
+The ``POST /reload`` call will order Patroni to re-read and apply the configuration file. This is the equivalent of sending the ``SIGHUP`` signal to the Patroni process. In case you changed some of the Postgres parameters which require a restart (like **shared_buffers**), you still have to explicitly do the restart of Postgres by either calling the ``POST /restart`` endpoint or with the help of :ref:`patronictl_restart`.
 
-The reload endpoint is used by ``patronictl reload``.
+The reload endpoint is used by :ref:`patronictl_reload`.
 
 
 Reinitialize endpoint
@@ -698,4 +698,4 @@ Reinitialize endpoint
 
 The call might fail if Patroni is in a loop trying to recover (restart) a failed Postgres. In order to overcome this problem one can specify ``{"force":true}`` in the request body.
 
-The reinitialize endpoint is used by ``patronictl reinit``.
+The reinitialize endpoint is used by :ref:`patronictl_reinit`.
diff --git a/docs/security.rst b/docs/security.rst
index cddefe0c..24af168e 100644
--- a/docs/security.rst
+++ b/docs/security.rst
@@ -9,7 +9,7 @@ A Patroni cluster has two interfaces to be protected from unauthorized access: t
 Protecting DCS
 ==============
 
-Patroni and patronictl both store and retrieve data to/from the DCS.
+Patroni and :ref:`patronictl` both store and retrieve data to/from the DCS.
 
 Despite DCS doesn't contain any sensitive information, it allows changing some of Patroni/Postgres configuration. Therefore the very first thing that should be protected is DCS itself.
 
@@ -22,7 +22,7 @@ Protecting the REST API
 
 Protecting the REST API is a more complicated task.
 
-The Patroni REST API is used by Patroni itself during the leader race, by the ``patronictl`` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring.
+The Patroni REST API is used by Patroni itself during the leader race, by the :ref:`patronictl` tool in order to perform failovers/switchovers/reinitialize/restarts/reloads, by HAProxy or any other kind of load balancer to perform HTTP health checks, and of course could also be used for monitoring.
 
 From the point of view of security, REST API contains safe (``GET`` requests, only retrieve information) and unsafe (``PUT``, ``POST``, ``PATCH`` and ``DELETE`` requests, change the state of nodes) endpoints.
 
@@ -32,6 +32,6 @@ When TLS for the REST API is enabled and a PKI is established, mutual authentica
 
 The ``restapi`` section parameters enable TLS client authentication to the server. Depending on the value of the ``verify_client`` parameter, the API server requires a successful client certificate verification for both safe and unsafe API calls (``verify_client: required``), or only for unsafe API calls (``verify_client: optional``), or for no API calls (``verify_client: none``).
 
-The ``ctl`` section parameters enable TLS server authentication to the client (the ``patronictl`` tool which uses the same config as patroni). Set ``insecure: true`` to disable the server certificate verification by the client. See :ref:`settings <patronictl_settings>` for a detailed description of the TLS client parameters.
+The ``ctl`` section parameters enable TLS server authentication to the client (the :ref:`patronictl` tool which uses the same config as patroni). Set ``insecure: true`` to disable the server certificate verification by the client. See :ref:`settings <patronictl_settings>` for a detailed description of the TLS client parameters.
 
 Protecting the PostgreSQL database proper from unauthorized access is beyond the scope of this document and is covered in https://www.postgresql.org/docs/current/client-authentication.html
diff --git a/docs/yaml_configuration.rst b/docs/yaml_configuration.rst
index d9283192..4c2ed7c0 100644
--- a/docs/yaml_configuration.rst
+++ b/docs/yaml_configuration.rst
@@ -34,7 +34,7 @@ Bootstrap configuration
 .. note::
     Once Patroni has initialized the cluster for the first time and settings have been stored in the DCS, all future
     changes to the ``bootstrap.dcs`` section of the YAML configuration will not take any effect! If you want to change
-    them please use either ``patronictl edit-config`` or the Patroni :ref:`REST API <rest_api>`.
+    them please use either :ref:`patronictl_edit_config` or the Patroni :ref:`REST API <rest_api>`.
 
 -  **bootstrap**:
 
@@ -366,10 +366,10 @@ CTL
 
    -  **authentication**:
 
-      -  **username**: Basic-auth username for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "username" parameter.
-      -  **password**: Basic-auth password for accessing protected REST API endpoints. If not provided patronictl will use the value provided for REST API "password" parameter.
+      -  **username**: Basic-auth username for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "username" parameter.
+      -  **password**: Basic-auth password for accessing protected REST API endpoints. If not provided :ref:`patronictl` will use the value provided for REST API "password" parameter.
    -  **insecure**: Allow connections to REST API without verifying SSL certs.
-   -  **cacert**: Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided patronictl will use the value provided for REST API "cafile" parameter.
+   -  **cacert**: Specifies the file with the CA_BUNDLE file or directory with certificates of trusted CAs to use while verifying REST API SSL certs. If not provided :ref:`patronictl` will use the value provided for REST API "cafile" parameter.
    -  **certfile**: Specifies the file with the client certificate in the PEM format.
    -  **keyfile**: Specifies the file with the client secret key in the PEM format.
    -  **keyfile\_password**: Specifies a password for decrypting the client keyfile.
@@ -397,4 +397,4 @@ In addition to these predefined tags, you can also add your own ones:
 -  **key3**: ``1.4``
 -  **key4**: ``"RandomString"``
 
-Tags are visible in the :ref:`REST API <rest_api>` and ``patronictl list`` You can also check for an instance health using these tags. If the tag isn't defined for an instance, or if the respective value doesn't match the querying value, it will return HTTP Status Code 503.
+Tags are visible in the :ref:`REST API <rest_api>` and :ref:`patronictl_list` You can also check for an instance health using these tags. If the tag isn't defined for an instance, or if the respective value doesn't match the querying value, it will return HTTP Status Code 503.
diff --git a/features/dcs_failsafe_mode.feature b/features/dcs_failsafe_mode.feature
index 4a34b754..8345d8ea 100644
--- a/features/dcs_failsafe_mode.feature
+++ b/features/dcs_failsafe_mode.feature
@@ -4,8 +4,8 @@ Feature: dcs failsafe mode
   Scenario: check failsafe mode can be successfully enabled
     Given I start postgres0
     And postgres0 is a leader after 10 seconds
-    And I sleep for 3 seconds
-    When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 5, "failsafe_mode": true}
+    Then "config" key in DCS has ttl=30 after 10 seconds
+    When I issue a PATCH request to http://127.0.0.1:8008/config with {"loop_wait": 2, "ttl": 20, "retry_timeout": 3, "failsafe_mode": true}
     Then I receive a response code 200
     And Response on GET http://127.0.0.1:8008/failsafe contains postgres0 after 10 seconds
     When I issue a GET request to http://127.0.0.1:8008/failsafe
@@ -28,7 +28,6 @@ Feature: dcs failsafe mode
     When I do a backup of postgres0
     And I shut down postgres0
     When I start postgres1 in a cluster batman from backup with no_leader
-    And I sleep for 2 seconds
     Then postgres1 role is the replica after 12 seconds
 
   Scenario: check leader and replica are both in /failsafe key after leader is back
@@ -59,12 +58,12 @@ Feature: dcs failsafe mode
     Given DCS is down
     And I kill postgres1
     And I kill postmaster on postgres1
-    And I sleep for 2 seconds
     Then postgres0 role is the replica after 12 seconds
 
   @dcs-failsafe
   Scenario: check known replica is promoted when leader is down and DCS is up
-    Given I shut down postgres0
+    Given I kill postgres0
+    And I shut down postmaster on postgres0
     And DCS is up
     When I start postgres1
     Then "members/postgres1" key in DCS has state=running after 10 seconds
diff --git a/features/environment.py b/features/environment.py
index e3c21252..79db5d16 100644
--- a/features/environment.py
+++ b/features/environment.py
@@ -162,9 +162,10 @@ class PatroniController(AbstractController):
 
     def stop(self, kill=False, timeout=15, postgres=False):
         if postgres:
-            return subprocess.call(['pg_ctl', '-D', self._data_dir, 'stop', '-mi', '-w'])
+            mode = 'i' if kill else 'f'
+            return subprocess.call(['pg_ctl', '-D', self._data_dir, 'stop', '-m' + mode, '-w'])
         super(PatroniController, self).stop(kill, timeout)
-        if isinstance(self._context.dcs_ctl, KubernetesController):
+        if isinstance(self._context.dcs_ctl, KubernetesController) and not kill:
             self._context.dcs_ctl.delete_pod(self._name[8:])
         if self.watchdog:
             self.watchdog.stop()
diff --git a/features/ignored_slots.feature b/features/ignored_slots.feature
index 08a6dda5..e0c53ea3 100644
--- a/features/ignored_slots.feature
+++ b/features/ignored_slots.feature
@@ -25,10 +25,10 @@ Feature: ignored slots
     # but Patroni can actually end up dropping them almost immediately, so it's helpful
     # to verify they exist before we begin testing whether they persist through failover
     # cycles.
-    Then postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin
+    Then postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
 
     When I start postgres0
     Then "members/postgres0" key in DCS has role=replica after 10 seconds
@@ -46,16 +46,16 @@ Feature: ignored slots
     And "members/postgres1" key in DCS has role=replica after 10 seconds
     # give Patroni time to sync replication slots
     And I sleep for 2 seconds
-    And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin
+    And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
     And postgres1 does not have a logical replication slot named dummy_slot
 
     # 3. After a failover the server (now a primary) still has the slot.
     When I shut down postgres0
     Then "members/postgres1" key in DCS has role=master after 10 seconds
-    And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin
-    And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin
+    And postgres1 has a logical replication slot named unmanaged_slot_0 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_1 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_2 with the test_decoding plugin after 2 seconds
+    And postgres1 has a logical replication slot named unmanaged_slot_3 with the test_decoding plugin after 2 seconds
diff --git a/features/patroni_api.feature b/features/patroni_api.feature
index 624d3271..ff06f3c9 100644
--- a/features/patroni_api.feature
+++ b/features/patroni_api.feature
@@ -68,6 +68,7 @@ Scenario: check API requests for the primary-replica pair in the pause mode
 	When I kill postmaster on postgres1
 	And I issue a GET request to http://127.0.0.1:8009/replica
 	Then I receive a response code 503
+	And "members/postgres1" key in DCS has state=stopped after 10 seconds
 	When I run patronictl.py restart batman postgres1 --force
 	Then I receive a response returncode 0
 	Then replication works from postgres0 to postgres1 after 20 seconds
@@ -76,7 +77,7 @@ Scenario: check API requests for the primary-replica pair in the pause mode
 	Then I receive a response code 200
 	And I receive a response state running
 	And I receive a response role replica
-	When I run patronictl.py reinit batman postgres1 --force
+	When I run patronictl.py reinit batman postgres1 --force --wait
 	Then I receive a response returncode 0
 	And I receive a response output "Success: reinitialize for member postgres1"
 	And postgres1 role is the secondary after 30 seconds
diff --git a/features/permanent_slots.feature b/features/permanent_slots.feature
new file mode 100644
index 00000000..656e6ade
--- /dev/null
+++ b/features/permanent_slots.feature
@@ -0,0 +1,34 @@
+Feature: permanent slots
+  Scenario: check that physical permanent slots are created
+    Given I start postgres0
+    Then postgres0 is a leader after 10 seconds
+    And there is a non empty initialize key in DCS after 15 seconds
+    When I issue a PATCH request to http://127.0.0.1:8008/config with {"slots": {"test_physical": {"type": "physical"}}, "postgresql": {"parameters": {"wal_level": "logical"}}}
+    Then I receive a response code 200
+    And Response on GET http://127.0.0.1:8008/config contains slots after 10 seconds
+    Then postgres0 has a physical replication slot named test_physical after 10 seconds
+    And I start postgres1
+
+  @slot-advance
+  Scenario: check that logical permanent slots are created
+    Given I run patronictl.py restart batman postgres0 --force
+    And I issue a PATCH request to http://127.0.0.1:8008/config with {"slots": {"test_logical": {"type": "logical", "database": "postgres", "plugin": "test_decoding"}}}
+    Then postgres0 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
+
+  @slot-advance
+  Scenario: check that permanent slots are created on the replica
+    Given postgres1 has a logical replication slot named test_logical with the test_decoding plugin after 10 seconds
+    Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
+    And postgres1 has a physical replication slot named test_physical after 2 seconds
+
+  @slot-advance
+  Scenario: check that permanent slots are advanced on the replica
+    Given I add the table replicate_me to postgres0
+    And I get all changes from physical slot test_physical on postgres0
+    When I get all changes from logical slot test_logical on postgres0
+    Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
+    And Physical slot test_physical is in sync between postgres0 and postgres1 after 10 seconds
+
+  Scenario: check permanent physical replication slot after failover
+    Given I shut down postgres0
+    Then postgres1 has a physical replication slot named test_physical after 10 seconds
diff --git a/features/standby_cluster.feature b/features/standby_cluster.feature
index 850c7970..7ff07679 100644
--- a/features/standby_cluster.feature
+++ b/features/standby_cluster.feature
@@ -22,9 +22,6 @@ Feature: standby cluster
   Scenario: check permanent logical slots are synced to the replica
     Given I run patronictl.py restart batman postgres1 --force
     Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
-    When I add the table replicate_me to postgres1
-    And I get all changes from logical slot test_logical on postgres1
-    Then Logical slot test_logical is in sync between postgres0 and postgres1 after 10 seconds
 
   Scenario: Detach exiting node from the cluster
     When I shut down postgres1
diff --git a/features/steps/basic_replication.py b/features/steps/basic_replication.py
index 6718cb01..5977eb61 100644
--- a/features/steps/basic_replication.py
+++ b/features/steps/basic_replication.py
@@ -35,11 +35,16 @@ def kill_patroni(context, name):
     return context.pctl.stop(name, kill=True)
 
 
-@step('I kill postmaster on {name:w}')
+@step('I shut down postmaster on {name:w}')
 def stop_postgres(context, name):
     return context.pctl.stop(name, postgres=True)
 
 
+@step('I kill postmaster on {name:w}')
+def kill_postgres(context, name):
+    return context.pctl.stop(name, kill=True, postgres=True)
+
+
 @step('I add the table {table_name:w} to {pg_name:w}')
 def add_table(context, table_name, pg_name):
     # parse the configuration file and get the port
diff --git a/features/steps/cascading_replication.py b/features/steps/cascading_replication.py
index 9783fbae..c6b43f31 100644
--- a/features/steps/cascading_replication.py
+++ b/features/steps/cascading_replication.py
@@ -28,7 +28,7 @@ def check_member(context, name, key, value, time_limit):
     while time.time() < max_time:
         try:
             response = json.loads(context.dcs_ctl.query(name))
-            dcs_value = response.get(key)
+            dcs_value = str(response.get(key))
             if dcs_value == value:
                 return
         except Exception:
diff --git a/features/steps/slots.py b/features/steps/slots.py
index 8a742b1c..f4a3cfa5 100644
--- a/features/steps/slots.py
+++ b/features/steps/slots.py
@@ -15,17 +15,25 @@ def create_logical_replication_slot(context, slot_name, pg_name, plugin):
         assert False, "Error creating slot {0} on {1} with plugin {2}".format(slot_name, pg_name, plugin)
 
 
-@then('{pg_name:w} has a logical replication slot named {slot_name} with the {plugin:w} plugin')
-def has_logical_replication_slot(context, pg_name, slot_name, plugin):
-    try:
-        row = context.pctl.query(pg_name, ("SELECT slot_type, plugin FROM pg_replication_slots"
-                                           " WHERE slot_name = '{0}'").format(slot_name)).fetchone()
-        assert row, "Couldn't find replication slot named {0}".format(slot_name)
-        assert row[0] == "logical", "Found replication slot named {0} but wasn't a logical slot".format(slot_name)
-        assert row[1] == plugin, ("Found replication slot named {0} but was using plugin "
-                                  "{1} rather than {2}").format(slot_name, row[1], plugin)
-    except pg.Error:
-        assert False, "Error looking for slot {0} on {1} with plugin {2}".format(slot_name, pg_name, plugin)
+@step('{pg_name:w} has a logical replication slot named {slot_name}'
+      ' with the {plugin:w} plugin after {time_limit:d} seconds')
+@then('{pg_name:w} has a logical replication slot named {slot_name}'
+      ' with the {plugin:w} plugin after {time_limit:d} seconds')
+def has_logical_replication_slot(context, pg_name, slot_name, plugin, time_limit):
+    time_limit *= context.timeout_multiplier
+    max_time = time.time() + int(time_limit)
+    while time.time() < max_time:
+        try:
+            row = context.pctl.query(pg_name, ("SELECT slot_type, plugin FROM pg_replication_slots"
+                                               f" WHERE slot_name = '{slot_name}'")).fetchone()
+            if row:
+                assert row[0] == "logical", f"Replication slot {slot_name} isn't a logical but {row[0]}"
+                assert row[1] == plugin, f"Replication slot {slot_name} using plugin {row[1]} rather than {plugin}"
+                return
+        except Exception:
+            pass
+        time.sleep(1)
+    assert False, f"Error looking for slot {slot_name} on {pg_name} with plugin {plugin}"
 
 
 @then('{pg_name:w} does not have a logical replication slot named {slot_name}')
@@ -38,13 +46,14 @@ def does_not_have_logical_replication_slot(context, pg_name, slot_name):
         assert False, "Error looking for slot {0} on {1}".format(slot_name, pg_name)
 
 
-@step('Logical slot {slot_name:w} is in sync between {pg_name1:w} and {pg_name2:w} after {time_limit:d} seconds')
-def logical_slots_in_sync(context, slot_name, pg_name1, pg_name2, time_limit):
+@step('{slot_type:w} slot {slot_name:w} is in sync between {pg_name1:w} and {pg_name2:w} after {time_limit:d} seconds')
+def slots_in_sync(context, slot_type, slot_name, pg_name1, pg_name2, time_limit):
     time_limit *= context.timeout_multiplier
     max_time = time.time() + int(time_limit)
+    column = 'confirmed_flush_lsn' if slot_type.lower() == 'logical' else 'restart_lsn'
+    query = f"SELECT {column} FROM pg_replication_slots WHERE slot_name = '{slot_name}'"
     while time.time() < max_time:
         try:
-            query = "SELECT confirmed_flush_lsn FROM pg_replication_slots WHERE slot_name = '{0}'".format(slot_name)
             slot1 = context.pctl.query(pg_name1, query).fetchone()
             slot2 = context.pctl.query(pg_name2, query).fetchone()
             if slot1[0] == slot2[0]:
@@ -52,9 +61,31 @@ def logical_slots_in_sync(context, slot_name, pg_name1, pg_name2, time_limit):
         except Exception:
             pass
         time.sleep(1)
-    assert False, "Logical slot {0} is not in sync between {1} and {2}".format(slot_name, pg_name1, pg_name2)
+    assert False, \
+        f"{slot_type} slot {slot_name} is not in sync between {pg_name1} and {pg_name2} after {time_limit} seconds"
 
 
 @step('I get all changes from logical slot {slot_name:w} on {pg_name:w}')
 def logical_slot_get_changes(context, slot_name, pg_name):
     context.pctl.query(pg_name, "SELECT * FROM pg_logical_slot_get_changes('{0}', NULL, NULL)".format(slot_name))
+
+
+@step('I get all changes from physical slot {slot_name:w} on {pg_name:w}')
+def physical_slot_get_changes(context, slot_name, pg_name):
+    context.pctl.query(pg_name, f"SELECT * FROM pg_replication_slot_advance('{slot_name}', pg_current_wal_lsn())")
+
+
+@step('{pg_name:w} has a physical replication slot named {slot_name} after {time_limit:d} seconds')
+def has_physical_replication_slot(context, pg_name, slot_name, time_limit):
+    time_limit *= context.timeout_multiplier
+    max_time = time.time() + int(time_limit)
+    query = f"SELECT * FROM pg_catalog.pg_replication_slots WHERE slot_type = 'physical' AND slot_name = '{slot_name}'"
+    while time.time() < max_time:
+        try:
+            row = context.pctl.query(pg_name, query).fetchone()
+            if row:
+                return
+        except Exception:
+            pass
+        time.sleep(1)
+    assert False, f"Physical slot {slot_name} doesn't exist after {time_limit} seconds"
diff --git a/patroni/__main__.py b/patroni/__main__.py
index 2b318a67..7d56172b 100644
--- a/patroni/__main__.py
+++ b/patroni/__main__.py
@@ -106,6 +106,8 @@ class Patroni(AbstractPatroniDaemon, Tags):
 
     def ensure_unique_name(self) -> None:
         """A helper method to prevent splitbrain from operator naming error."""
+        from urllib.parse import urlparse
+        from urllib3.connection import HTTPConnection
         from patroni.dcs import Member
 
         cluster = self.dcs.get_cluster()
@@ -115,9 +117,12 @@ class Patroni(AbstractPatroniDaemon, Tags):
         if not isinstance(member, Member):
             return
         try:
-            _ = self.request(member, endpoint="/liveness", timeout=3)
-            logger.fatal("Can't start; there is already a node named '%s' running", self.config['name'])
-            sys.exit(1)
+            parts = urlparse(member.api_url)
+            if isinstance(parts.hostname, str):
+                connection = HTTPConnection(parts.hostname, port=parts.port or 80, timeout=3)
+                connection.connect()
+                logger.fatal("Can't start; there is already a node named '%s' running", self.config['name'])
+                sys.exit(1)
         except Exception:
             return
 
diff --git a/patroni/config.py b/patroni/config.py
index 1650934d..fee2147a 100644
--- a/patroni/config.py
+++ b/patroni/config.py
@@ -490,7 +490,8 @@ class Config(object):
             elif not is_local:
                 validator = ConfigHandler.CMDLINE_OPTIONS[name][1]
                 if validator(value):
-                    pg_params[name] = int(value) if isinstance(validator, IntValidator) else value
+                    int_val = parse_int(value) if isinstance(validator, IntValidator) else None
+                    pg_params[name] = int_val if isinstance(int_val, int) else value
                 else:
                     logger.warning("postgresql parameter %s=%s failed validation, defaulting to %s",
                                    name, value, ConfigHandler.CMDLINE_OPTIONS[name][0])
diff --git a/patroni/config_generator.py b/patroni/config_generator.py
index 956e18d2..4bcb2819 100644
--- a/patroni/config_generator.py
+++ b/patroni/config_generator.py
@@ -38,7 +38,7 @@ _AUTH_ALLOWED_PARAMETERS_MAPPING = {
     'gssencmode': 'PGGSSENCMODE',
     'channel_binding': 'PGCHANNELBINDING'
 }
-_NO_VALUE_MSG = '#FIXME'
+NO_VALUE_MSG = '#FIXME'
 
 
 def get_address() -> Tuple[str, str]:
@@ -50,7 +50,7 @@ def get_address() -> Tuple[str, str]:
     :returns: tuple consisting of the hostname returned by :func:`~socket.gethostname`
         and the first element in the sorted list of the addresses returned by :func:`~socket.getaddrinfo`.
         Sorting guarantees it will prefer IPv4.
-        If an exception occured, hostname and ip values are equal to :data:`~patroni.config_generator._NO_VALUE_MSG`.
+        If an exception occured, hostname and ip values are equal to :data:`~patroni.config_generator.NO_VALUE_MSG`.
     """
     hostname = None
     try:
@@ -59,7 +59,7 @@ def get_address() -> Tuple[str, str]:
                                 key=lambda x: x[0])[0][4][0]
     except Exception as err:
         logging.warning('Failed to obtain address: %r', err)
-        return _NO_VALUE_MSG, _NO_VALUE_MSG
+        return NO_VALUE_MSG, NO_VALUE_MSG
 
 
 class AbstractConfigGenerator(abc.ABC):
@@ -88,24 +88,24 @@ class AbstractConfigGenerator(abc.ABC):
         """Generate a template config for further extension (e.g. in the inherited classes).
 
         :returns: dictionary with the values gathered from Patroni env, hopefully defined hostname and ip address
-                  (otherwise set to :data:`~patroni.config_generator._NO_VALUE_MSG`), and some sane defaults.
+                  (otherwise set to :data:`~patroni.config_generator.NO_VALUE_MSG`), and some sane defaults.
         """
         template_config: Dict[str, Any] = {
-            'scope': _NO_VALUE_MSG,
+            'scope': NO_VALUE_MSG,
             'name': cls._HOSTNAME,
             'postgresql': {
-                'data_dir': _NO_VALUE_MSG,
-                'connect_address': _NO_VALUE_MSG + ':5432',
-                'listen': _NO_VALUE_MSG + ':5432',
+                'data_dir': NO_VALUE_MSG,
+                'connect_address': NO_VALUE_MSG + ':5432',
+                'listen': NO_VALUE_MSG + ':5432',
                 'bin_dir': '',
                 'authentication': {
                     'superuser': {
                         'username': 'postgres',
-                        'password': _NO_VALUE_MSG
+                        'password': NO_VALUE_MSG
                     },
                     'replication': {
                         'username': 'replicator',
-                        'password': _NO_VALUE_MSG
+                        'password': NO_VALUE_MSG
                     }
                 }
             },
@@ -185,7 +185,7 @@ class SampleConfigGenerator(AbstractConfigGenerator):
         self.config['bootstrap']['dcs']['postgresql']['use_pg_rewind'] = True
         if self.pg_major >= 110000:
             self.config['postgresql']['authentication'].setdefault(
-                'rewind', {'username': 'rewind_user'}).setdefault('password', _NO_VALUE_MSG)
+                'rewind', {'username': 'rewind_user'}).setdefault('password', NO_VALUE_MSG)
 
 
 class RunningClusterConfigGenerator(AbstractConfigGenerator):
@@ -287,7 +287,7 @@ class RunningClusterConfigGenerator(AbstractConfigGenerator):
 
         :param cur: connection cursor to use.
         """
-        cur.execute("SELECT name, current_setting(name) FROM pg_settings "
+        cur.execute("SELECT name, pg_catalog.current_setting(name) FROM pg_catalog.pg_settings "
                     "WHERE context <> 'internal' "
                     "AND source IN ('configuration file', 'command line', 'environment variable') "
                     "AND category <> 'Write-Ahead Log / Recovery Target' "
@@ -335,7 +335,7 @@ class RunningClusterConfigGenerator(AbstractConfigGenerator):
             getpass('Please enter the user password:')
         self.config['postgresql']['authentication'] = {
             'superuser': su_params,
-            'replication': {'username': _NO_VALUE_MSG, 'password': _NO_VALUE_MSG}
+            'replication': {'username': NO_VALUE_MSG, 'password': NO_VALUE_MSG}
         }
 
     def _set_conf_files(self) -> None:
diff --git a/patroni/dcs/__init__.py b/patroni/dcs/__init__.py
index 87d3c9c1..08112651 100644
--- a/patroni/dcs/__init__.py
+++ b/patroni/dcs/__init__.py
@@ -28,6 +28,7 @@ from ..tags import Tags
 if TYPE_CHECKING:  # pragma: no cover
     from ..config import Config
 
+SLOT_ADVANCE_AVAILABLE_VERSION = 110000
 CITUS_COORDINATOR_GROUP_ID = 0
 citus_group_re = re.compile('^(0|[1-9][0-9]*)$')
 slot_name_re = re.compile('^[a-z0-9_]{1,63}$')
@@ -350,6 +351,11 @@ class Member(Tags, NamedTuple('Member',
                 logger.debug('Failed to parse Patroni version %s', version)
         return None
 
+    @property
+    def lsn(self) -> Optional[int]:
+        """Current LSN (receive/flush/replay)."""
+        return self.data.get('xlog_location')
+
 
 class RemoteMember(Member):
     """Represents a remote member (typically a primary) for a standby cluster.
@@ -964,24 +970,42 @@ class Cluster(NamedTuple('Cluster',
         candidates = [m for m in self.members if m.clonefrom and m.is_running and m.name not in exclude]
         return candidates[randint(0, len(candidates) - 1)] if candidates else self.leader
 
+    @staticmethod
+    def is_physical_slot(value: Union[Any, Dict[str, Any]]) -> bool:
+        """Check whether provided configuration is for permanent physical replication slot.
+
+        :returns: ``True`` if this is a physical replication slot, otherwise ``False``.
+        """
+        return not value or isinstance(value, dict) and value.get('type', 'physical') == 'physical'
+
     @property
     def __permanent_slots(self) -> Dict[str, Union[Dict[str, Any], Any]]:
         """Dictionary of permanent replication slots with their known LSN."""
-        ret = deepcopy(self.config.permanent_slots if self.config else {})
-        # If primary reported flush LSN for permanent slots we want to enrich our structure with it
-        for name, lsn in (self.slots or {}).items():
-            if name in ret:
-                if not ret[name]:
-                    ret[name] = {}
-                if isinstance(ret[name], dict):
-                    ret[name]['lsn'] = lsn
+        leader = self.leader and self.leader.member
+        leader_name = slot_name_from_member_name(leader.name) if leader and leader.lsn else None
+
+        slots = self.slots or {}
+        ret: Dict[str, Union[Dict[str, Any], Any]] = deepcopy(self.config.permanent_slots if self.config else {})
+
+        for name, value in list(ret.items()):
+            if not value:
+                value = ret[name] = {}
+            if isinstance(value, dict):
+                if name in slots:
+                    # If primary reported flush LSN for permanent slots we want to enrich our structure with it
+                    value['lsn'] = slots[name]
+                elif self.is_physical_slot(value) and name == leader_name and leader and leader.lsn:
+                    # there is no slot on the leader for itself, use `lsn` from the member key.
+                    value['lsn'] = leader.lsn
+                else:
+                    # Don't let anyone set 'lsn' in the global configuration :)
+                    value.pop('lsn', None)
         return ret
 
     @property
     def __permanent_physical_slots(self) -> Dict[str, Any]:
         """Dictionary of permanent ``physical`` replication slots."""
-        return {name: value for name, value in self.__permanent_slots.items()
-                if not value or isinstance(value, dict) and value.get('type', 'physical') == 'physical'}
+        return {name: value for name, value in self.__permanent_slots.items() if self.is_physical_slot(value)}
 
     @property
     def __permanent_logical_slots(self) -> Dict[str, Any]:
@@ -1013,7 +1037,7 @@ class Cluster(NamedTuple('Cluster',
         :returns: final dictionary of slot names, after merging with permanent slots and performing sanity checks.
         """
         slots: Dict[str, Dict[str, str]] = self._get_members_slots(my_name, role)
-        permanent_slots: Dict[str, Any] = self._get_permanent_slots(is_standby_cluster, role, nofailover)
+        permanent_slots: Dict[str, Any] = self._get_permanent_slots(is_standby_cluster, role, nofailover, major_version)
 
         disabled_permanent_logical_slots: List[str] = self._merge_permanent_slots(
             slots, permanent_slots, my_name, major_version)
@@ -1061,7 +1085,7 @@ class Cluster(NamedTuple('Cluster',
                     continue
 
                 if value['type'] == 'logical' and value.get('database') and value.get('plugin'):
-                    if major_version < 110000:
+                    if major_version < SLOT_ADVANCE_AVAILABLE_VERSION:
                         disabled_permanent_logical_slots.append(name)
                     elif name in slots:
                         logger.error("Permanent logical replication slot {'%s': %s} is conflicting with"
@@ -1073,7 +1097,8 @@ class Cluster(NamedTuple('Cluster',
             logger.error("Bad value for slot '%s' in permanent_slots: %s", name, permanent_slots[name])
         return disabled_permanent_logical_slots
 
-    def _get_permanent_slots(self, is_standby_cluster: bool, role: str, nofailover: bool) -> Dict[str, Any]:
+    def _get_permanent_slots(self, is_standby_cluster: bool, role: str,
+                             nofailover: bool, major_version: int) -> Dict[str, Any]:
         """Get configured permanent replication slots.
 
         .. note::
@@ -1089,6 +1114,7 @@ class Cluster(NamedTuple('Cluster',
                                    the outside because we want to protect from the ``/config`` key removal.
         :param role: role of this node -- ``primary``, ``standby_leader`` or ``replica``.
         :param nofailover: ``True`` if this node is tagged to not be a failover candidate.
+        :param major_version: postgresql major version.
 
         :returns: dictionary of permanent slot names mapped to attributes.
         """
@@ -1096,9 +1122,11 @@ class Cluster(NamedTuple('Cluster',
             return {}
 
         if is_standby_cluster:
-            return self.__permanent_physical_slots if role == 'standby_leader' else {}
+            return self.__permanent_physical_slots \
+                if major_version >= SLOT_ADVANCE_AVAILABLE_VERSION or role == 'standby_leader' else {}
 
-        return self.__permanent_slots if role in ('master', 'primary') else self.__permanent_logical_slots
+        return self.__permanent_slots if major_version >= SLOT_ADVANCE_AVAILABLE_VERSION\
+            or role in ('master', 'primary') else self.__permanent_logical_slots
 
     def _get_members_slots(self, my_name: str, role: str) -> Dict[str, Dict[str, str]]:
         """Get physical replication slots configuration for members that sourcing from this node.
@@ -1143,21 +1171,33 @@ class Cluster(NamedTuple('Cluster',
                                    for k, v in slot_conflicts.items() if len(v) > 1))
         return slots
 
-    def has_permanent_logical_slots(self, my_name: str, nofailover: bool, major_version: int = 110000) -> bool:
+    def has_permanent_slots(self, my_name: str, nofailover: bool = False) -> bool:
+        """Check if the given member node has permanent replication slots configured.
+
+        :param my_name: name of the member node to check.
+        :param nofailover: ``True`` if this node is tagged to not be a failover candidate.
+
+        :returns: ``True`` if there are permanent replication slots configured, otherwise ``False``.
+        """
+        members_slots: Dict[str, Dict[str, str]] = self._get_members_slots(my_name, 'replica')
+        permanent_slots: Dict[str, Any] = self._get_permanent_slots(nofailover, 'replica', False,
+                                                                    SLOT_ADVANCE_AVAILABLE_VERSION)
+        slots = deepcopy(members_slots)
+        self._merge_permanent_slots(slots, permanent_slots, my_name, SLOT_ADVANCE_AVAILABLE_VERSION)
+        return len(slots) > len(members_slots)
+
+    def _has_permanent_logical_slots(self, my_name: str, nofailover: bool) -> bool:
         """Check if the given member node has permanent ``logical`` replication slots configured.
 
         :param my_name: name of the member node to check.
         :param nofailover: ``True`` if this node is tagged to not be a failover candidate.
-        :param major_version: the PostgreSQL major version number.
 
-        :returns: ``False`` if PostgreSQL is < 11, ``True`` if any detected replications slots are ``logical``.
+        :returns: ``True`` if any detected replications slots are ``logical``, otherwise ``False``.
         """
-        if major_version < 110000:
-            return False
-        slots = self.get_replication_slots(my_name, 'replica', nofailover, major_version).values()
+        slots = self.get_replication_slots(my_name, 'replica', nofailover, SLOT_ADVANCE_AVAILABLE_VERSION).values()
         return any(v for v in slots if v.get("type") == "logical")
 
-    def should_enforce_hot_standby_feedback(self, my_name: str, nofailover: bool, major_version: int) -> bool:
+    def should_enforce_hot_standby_feedback(self, my_name: str, nofailover: bool) -> bool:
         """Determine whether ``hot_standby_feedback`` should be enabled for the given member.
 
         The ``hot_standby_feedback`` must be enabled if the current replica has ``logical`` slots,
@@ -1165,20 +1205,16 @@ class Cluster(NamedTuple('Cluster',
 
         :param my_name: name of the member node to check.
         :param nofailover: ``True`` if this node is tagged to not be a failover candidate.
-        :param major_version: PostgreSQL major version number.
 
-        :returns: ``True`` if this node or any member replicating from this node has permanent logical slots.
-                 ``False`` if PostgreSQL major version is < 11.
+        :returns: ``True`` if this node or any member replicating from this node has
+                  permanent logical slots, otherwise ``False``.
         """
-        if major_version < 110000:
-            return False
-
-        if self.has_permanent_logical_slots(my_name, nofailover, major_version):
+        if self._has_permanent_logical_slots(my_name, nofailover):
             return True
 
         if self.use_slots:
             members = [m for m in self.members if m.replicatefrom == my_name and m.name != self.leader_name]
-            return any(self.should_enforce_hot_standby_feedback(m.name, m.nofailover, major_version) for m in members)
+            return any(self.should_enforce_hot_standby_feedback(m.name, m.nofailover) for m in members)
         return False
 
     def get_my_slot_name_on_primary(self, my_name: str, replicatefrom: Optional[str]) -> str:
diff --git a/patroni/dcs/zookeeper.py b/patroni/dcs/zookeeper.py
index e3cc1d69..863d7b40 100644
--- a/patroni/dcs/zookeeper.py
+++ b/patroni/dcs/zookeeper.py
@@ -301,7 +301,7 @@ class ZooKeeper(AbstractDCS):
                 raise ZooKeeperError('ZooKeeper in not responding properly')
         # The /status ZNode was updated or doesn't exist
         elif self._fetch_status and not self._fetch_cluster or not cluster.last_lsn \
-                or cluster.has_permanent_logical_slots(self._name, False) and not cluster.slots:
+                or cluster.has_permanent_slots(self._name) and not cluster.slots:
             # If current node is the leader just clear the event without fetching anything (we are updating the /status)
             if cluster.leader and cluster.leader.name == self._name:
                 self.event.clear()
@@ -379,21 +379,28 @@ class ZooKeeper(AbstractDCS):
         cluster = self.cluster
         member = cluster and cluster.get_member(self._name, fallback_to_leader=False)
         member_data = self.__last_member_data or member and member.data
-        #  We want to notify leader if some important fields in the member key changed by removing ZNode
-        if member and (self._client.client_id is not None and member.session != self._client.client_id[0]
-                       or not (member_data and deep_compare(member_data.get('tags', {}), data.get('tags', {}))
-                               and (member_data.get('state') == data.get('state')
-                                    or 'running' not in (member_data.get('state'), data.get('state')))
-                               and member_data.get('version') == data.get('version')
-                               and member_data.get('checkpoint_after_promote')
-                               == data.get('checkpoint_after_promote'))):
-            try:
-                self._client.delete_async(self.member_path).get(timeout=1)
-            except NoNodeError:
-                pass
-            except Exception:
-                return False
-            member = None
+        if member and member_data:
+            is_leader = data.get('role') in ('master', 'primary', 'standby_leader')
+            checkpoint_after_promote_changed = member_data.get('checkpoint_after_promote') \
+                != data.get('checkpoint_after_promote')
+            state_running_changed = member_data.get('state') != data.get('state') \
+                and 'running' in (member_data.get('state'), data.get('state'))
+            tags_changed = not deep_compare(member_data.get('tags', {}), data.get('tags', {}))
+
+            # We want delete the member ZNode if:
+            # - our session doesn't match with session id on our member key; or
+            # - we want to notify leader if some important fields in the member key changed; or
+            # - if we are the leader and want to notify replicas about checkpoint_after_promote;
+            if self._client.client_id is not None and member.session != self._client.client_id[0] \
+                    or is_leader and checkpoint_after_promote_changed \
+                    or not is_leader and (state_running_changed or tags_changed):
+                try:
+                    self._client.delete_async(self.member_path).get(timeout=1)
+                except NoNodeError:
+                    pass
+                except Exception:
+                    return False
+                member = None
 
         encoded_data = json.dumps(data, separators=(',', ':')).encode('utf-8')
         if member and member_data:
diff --git a/patroni/ha.py b/patroni/ha.py
index d5d6e16b..3b936e4b 100644
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -691,6 +691,14 @@ class Ha(object):
             current = CaseInsensitiveSet(sync.members)
             picked, allow_promote = self.state_handler.sync_handler.current_state(self.cluster)
 
+            if picked == current and current != allow_promote:
+                logger.warning('Inconsistent state between synchronous_standby_names = %s and /sync = %s key '
+                               'detected, updating synchronous replication key...', list(allow_promote), list(current))
+                sync = self.dcs.write_sync_state(self.state_handler.name, allow_promote, version=sync.version)
+                if not sync:
+                    return logger.warning("Updating sync state failed")
+                current = CaseInsensitiveSet(sync.members)
+
             if picked != current:
                 # update synchronous standby list in dcs temporarily to point to common nodes in current and picked
                 sync_common = current & allow_promote
diff --git a/patroni/postgresql/__init__.py b/patroni/postgresql/__init__.py
index 9bd53dac..cab33e8f 100644
--- a/patroni/postgresql/__init__.py
+++ b/patroni/postgresql/__init__.py
@@ -27,7 +27,7 @@ from .sync import SyncHandler
 from .. import psycopg
 from ..async_executor import CriticalTask
 from ..collections import CaseInsensitiveSet
-from ..dcs import Cluster, Leader, Member
+from ..dcs import Cluster, Leader, Member, SLOT_ADVANCE_AVAILABLE_VERSION
 from ..exceptions import PostgresConnectionException
 from ..utils import Retry, RetryFailedError, polling_loop, data_directory_is_empty, parse_int
 
@@ -112,24 +112,35 @@ class Postgresql(object):
         self._state_entry_timestamp = 0
 
         self._cluster_info_state = {}
-        self._has_permanent_logical_slots = True
+        self._has_permanent_slots = True
         self._enforce_hot_standby_feedback = False
         self._cached_replica_timeline = None
 
         # Last known running process
         self._postmaster_proc = None
 
-        if self.is_running():  # we are "joining" already running postgres
-            self.set_state('running')
+        if self.is_running():
+            # If we found postmaster process we need to figure out whether postgres is accepting connections
+            self.set_state('starting')
+            self.check_startup_state_changed()
+
+        if self.state == 'running':  # we are "joining" already running postgres
+            # we know that PostgreSQL is accepting connections and can read some GUC's from pg_settings
+            self.config.load_current_server_parameters()
+
             self.set_role('master' if self.is_primary() else 'replica')
-            # postpone writing postgresql.conf for 12+ because recovery parameters are not yet known
-            if self.major_version < 120000 or self.is_primary():
-                self.config.write_postgresql_conf()
+
             hba_saved = self.config.replace_pg_hba()
             ident_saved = self.config.replace_pg_ident()
-            if hba_saved or ident_saved:
+
+            if self.major_version < 120000 or self.role in ('master', 'primary'):
+                # If PostgreSQL is running as a primary or we run PostgreSQL that is older than 12 we can
+                # call reload_config() once again (the first call happened in the ConfigHandler constructor),
+                # so that it can figure out if config files should be updated and pg_ctl reload executed.
+                self.config.reload_config(config, sighup=bool(hba_saved or ident_saved))
+            elif hba_saved or ident_saved:
                 self.reload()
-        elif self.role in ('master', 'primary'):
+        elif not self.is_running() and self.role in ('master', 'primary'):
             self.set_role('demoted')
 
     @property
@@ -174,6 +185,11 @@ class Postgresql(object):
         """:returns: `True` if Postgres version supports more than one synchronous node."""
         return self._major_version >= 90600
 
+    @property
+    def can_advance_slots(self) -> bool:
+        """``True`` if :attr:``major_version`` is greater than 110000."""
+        return self.major_version >= SLOT_ADVANCE_AVAILABLE_VERSION
+
     @property
     def cluster_info_query(self) -> str:
         """Returns the monitoring query with a fixed number of fields.
@@ -208,8 +224,9 @@ class Postgresql(object):
             extra = ("pg_catalog.current_setting('restore_command')" if self._major_version >= 120000 else "NULL") +\
                 ", " + ("(SELECT pg_catalog.json_agg(s.*) FROM (SELECT slot_name, slot_type as type, datoid::bigint, "
                         "plugin, catalog_xmin, pg_catalog.pg_wal_lsn_diff(confirmed_flush_lsn, '0/0')::bigint"
-                        " AS confirmed_flush_lsn FROM pg_catalog.pg_get_replication_slots()) AS s)"
-                        if self._has_permanent_logical_slots and self._major_version >= 110000 else "NULL") + extra
+                        " AS confirmed_flush_lsn, pg_catalog.pg_wal_lsn_diff(restart_lsn, '0/0')::bigint"
+                        " AS restart_lsn FROM pg_catalog.pg_get_replication_slots()) AS s)"
+                        if self._has_permanent_slots and self.can_advance_slots else "NULL") + extra
             extra = (", CASE WHEN latest_end_lsn IS NULL THEN NULL ELSE received_tli END,"
                      " slot_name, conninfo, status, {0} FROM pg_catalog.pg_stat_get_wal_receiver()").format(extra)
             if self.role == 'standby_leader':
@@ -434,18 +451,15 @@ class Postgresql(object):
             return
 
         if self._global_config.is_standby_cluster:
+            self._has_permanent_slots = False
             # Standby cluster can't have logical replication slots, and we don't need to enforce hot_standby_feedback
-            self._has_permanent_logical_slots = False
             self.set_enforce_hot_standby_feedback(False)
         elif cluster and cluster.config and cluster.config.modify_version:
-            self._has_permanent_logical_slots =\
-                cluster.has_permanent_logical_slots(self.name, nofailover, self.major_version)
-
+            self._has_permanent_slots = cluster.has_permanent_slots(self.name, nofailover)
             # We want to enable hot_standby_feedback if the replica is supposed
             # to have a logical slot or in case if it is the cascading replica.
             self.set_enforce_hot_standby_feedback(
-                self._has_permanent_logical_slots
-                or cluster.should_enforce_hot_standby_feedback(self.name, nofailover, self.major_version))
+                self.can_advance_slots and cluster.should_enforce_hot_standby_feedback(self.name, nofailover))
 
     def _cluster_info_state_get(self, name: str) -> Optional[Any]:
         if not self._cluster_info_state:
@@ -456,7 +470,7 @@ class Postgresql(object):
                                                'received_tli', 'slot_name', 'conninfo', 'receiver_state',
                                                'restore_command', 'slots', 'synchronous_commit',
                                                'synchronous_standby_names', 'pg_stat_replication'], result))
-                if self._has_permanent_logical_slots:
+                if self._has_permanent_slots and self.can_advance_slots:
                     cluster_info_state['slots'] =\
                         self.slots_handler.process_permanent_slots(cluster_info_state['slots'])
                 self._cluster_info_state = cluster_info_state
diff --git a/patroni/postgresql/config.py b/patroni/postgresql/config.py
index a918a2cf..b15f7f23 100644
--- a/patroni/postgresql/config.py
+++ b/patroni/postgresql/config.py
@@ -244,9 +244,10 @@ class ConfigWriter(object):
             self._fd.write(line)
             self._fd.write('\n')
 
-    def writelines(self, lines: List[str]) -> None:
+    def writelines(self, lines: List[Optional[str]]) -> None:
         for line in lines:
-            self.writeline(line)
+            if isinstance(line, str):
+                self.writeline(line)
 
     @staticmethod
     def escape(value: Any) -> str:  # Escape (by doubling) any single quotes or backslashes in given string
@@ -326,14 +327,22 @@ class ConfigHandler(object):
                                         .format(self._pgpass))
         self._passfile = None
         self._passfile_mtime = None
-        self._synchronous_standby_names = None
         self._postmaster_ctime = None
         self._current_recovery_params: Optional[CaseInsensitiveDict] = None
         self._config = {}
         self._recovery_params = CaseInsensitiveDict()
-        self._server_parameters: CaseInsensitiveDict
+        self._server_parameters: CaseInsensitiveDict = CaseInsensitiveDict()
         self.reload_config(config)
 
+    def load_current_server_parameters(self) -> None:
+        """Read GUC's values from ``pg_settings`` when Patroni is joining the the postgres that is already running."""
+        exclude = [name.lower() for name, value in self.CMDLINE_OPTIONS.items() if value[1] == _false_validator] \
+            + [name.lower() for name in self._RECOVERY_PARAMETERS]
+        self._server_parameters = CaseInsensitiveDict({r[0]: r[1] for r in self._postgresql.query(
+            "SELECT name, pg_catalog.current_setting(name) FROM pg_catalog.pg_settings"
+            " WHERE (source IN ('command line', 'environment variable') OR sourcefile = %s)"
+            " AND pg_catalog.lower(name) != ALL(%s)", self._postgresql_conf, exclude)})
+
     def setup_server_parameters(self) -> None:
         self._server_parameters = self.get_server_parameters(self._config)
         self._adjust_recovery_parameters()
@@ -922,14 +931,15 @@ class ConfigHandler(object):
         listen_addresses, port = split_host_port(config['listen'], 5432)
         parameters.update(cluster_name=self._postgresql.scope, listen_addresses=listen_addresses, port=str(port))
         if not self._postgresql.global_config or self._postgresql.global_config.is_synchronous_mode:
-            if self._synchronous_standby_names is None:
+            synchronous_standby_names = self._server_parameters.get('synchronous_standby_names')
+            if synchronous_standby_names is None:
                 if self._postgresql.global_config and self._postgresql.global_config.is_synchronous_mode_strict\
                         and self._postgresql.role in ('master', 'primary', 'promoted'):
                     parameters['synchronous_standby_names'] = '*'
                 else:
                     parameters.pop('synchronous_standby_names', None)
             else:
-                parameters['synchronous_standby_names'] = self._synchronous_standby_names
+                parameters['synchronous_standby_names'] = synchronous_standby_names
 
         # Handle hot_standby <-> replica rename
         if parameters.get('wal_level') == ('hot_standby' if self._postgresql.major_version >= 90600 else 'replica'):
@@ -1150,12 +1160,11 @@ class ConfigHandler(object):
     def set_synchronous_standby_names(self, value: Optional[str]) -> Optional[bool]:
         """Updates synchronous_standby_names and reloads if necessary.
         :returns: True if value was updated."""
-        if value != self._synchronous_standby_names:
+        if value != self._server_parameters.get('synchronous_standby_names'):
             if value is None:
                 self._server_parameters.pop('synchronous_standby_names', None)
             else:
                 self._server_parameters['synchronous_standby_names'] = value
-            self._synchronous_standby_names = value
             if self._postgresql.state == 'running':
                 self.write_postgresql_conf()
                 self._postgresql.reload()
diff --git a/patroni/postgresql/slots.py b/patroni/postgresql/slots.py
index f6090a3b..29bbb130 100644
--- a/patroni/postgresql/slots.py
+++ b/patroni/postgresql/slots.py
@@ -16,6 +16,7 @@ from .misc import format_lsn, fsync_dir
 from ..dcs import Cluster, Leader
 from ..file_perm import pg_perm
 from ..psycopg import OperationalError
+from ..utils import parse_int
 
 if TYPE_CHECKING:  # pragma: no cover
     from psycopg import Cursor
@@ -231,15 +232,16 @@ class SlotsHandler:
         ret: Dict[str, int] = {}
 
         slots_dict: Dict[str, Dict[str, Any]] = {slot['slot_name']: slot for slot in slots or []}
-        if slots_dict:
-            for name, value in slots_dict.items():
-                if name in self._replication_slots:
-                    if compare_slots(value, self._replication_slots[name], 'datoid'):
-                        if value['type'] == 'logical':
-                            ret[name] = value['confirmed_flush_lsn']
-                            self._copy_items(value, self._replication_slots[name])
+        for name, value in slots_dict.items():
+            if name in self._replication_slots:
+                if compare_slots(value, self._replication_slots[name], 'datoid'):
+                    if value['type'] == 'logical':
+                        ret[name] = value['confirmed_flush_lsn']
+                        self._copy_items(value, self._replication_slots[name])
                     else:
-                        self._schedule_load_slots = True
+                        self._replication_slots[name]['restart_lsn'] = ret[name] = value['restart_lsn']
+                else:
+                    self._schedule_load_slots = True
 
         # It could happen that the slot was deleted in the background, we want to detect this case
         if any(name not in slots_dict for name in self._replication_slots.keys()):
@@ -260,16 +262,19 @@ class SlotsHandler:
         """
         if self._postgresql.major_version >= 90400 and self._schedule_load_slots:
             replication_slots: Dict[str, Dict[str, Any]] = {}
-            extra = ", catalog_xmin, pg_catalog.pg_wal_lsn_diff(confirmed_flush_lsn, '0/0')::bigint" \
+            pg_wal_lsn_diff = f"pg_catalog.pg_{self._postgresql.wal_name}_{self._postgresql.lsn_name}_diff"
+            extra = f", catalog_xmin, {pg_wal_lsn_diff}(confirmed_flush_lsn, '0/0')::bigint" \
                 if self._postgresql.major_version >= 100000 else ""
             skip_temp_slots = ' WHERE NOT temporary' if self._postgresql.major_version >= 100000 else ''
-            for r in self._query('SELECT slot_name, slot_type, plugin, database, datoid'
-                                 f'{extra} FROM pg_catalog.pg_replication_slots{skip_temp_slots}'):
+            for r in self._query(f"SELECT slot_name, slot_type, {pg_wal_lsn_diff}(restart_lsn, '0/0')::bigint, plugin,"
+                                 f" database, datoid{extra} FROM pg_catalog.pg_replication_slots{skip_temp_slots}"):
                 value = {'type': r[1]}
                 if r[1] == 'logical':
-                    value.update(plugin=r[2], database=r[3], datoid=r[4])
+                    value.update(plugin=r[3], database=r[4], datoid=r[5])
                     if self._postgresql.major_version >= 100000:
-                        value.update(catalog_xmin=r[5], confirmed_flush_lsn=r[6])
+                        value.update(catalog_xmin=r[6], confirmed_flush_lsn=r[7])
+                else:
+                    value['restart_lsn'] = r[2]
                 replication_slots[r[0]] = value
             self._replication_slots = replication_slots
             self._schedule_load_slots = False
@@ -353,7 +358,7 @@ class SlotsHandler:
                     self._schedule_load_slots = True
 
     def _ensure_physical_slots(self, slots: Dict[str, Any]) -> None:
-        """Create any missing physical replication *slots*.
+        """Create or advance physical replication *slots*.
 
         Any failures are logged and do not interrupt creation of all *slots*.
 
@@ -362,7 +367,9 @@ class SlotsHandler:
         """
         immediately_reserve = ', true' if self._postgresql.major_version >= 90600 else ''
         for name, value in slots.items():
-            if name not in self._replication_slots and value['type'] == 'physical':
+            if value['type'] != 'physical':
+                continue
+            if name not in self._replication_slots:
                 try:
                     self._query(f"SELECT pg_catalog.pg_create_physical_replication_slot(%s{immediately_reserve})"
                                 f" WHERE NOT EXISTS (SELECT 1 FROM pg_catalog.pg_replication_slots"
@@ -371,6 +378,16 @@ class SlotsHandler:
                 except Exception:
                     logger.exception("Failed to create physical replication slot '%s'", name)
                 self._schedule_load_slots = True
+            elif not self._postgresql.is_primary() and self._postgresql.can_advance_slots \
+                    and self._replication_slots[name]['type'] == 'physical':
+                value['restart_lsn'] = self._replication_slots[name]['restart_lsn']
+                lsn = parse_int(value.get('lsn'))
+                if lsn and lsn > value['restart_lsn']:  # The slot has feedback in DCS and needs to be advanced
+                    try:
+                        lsn = format_lsn(lsn)
+                        self._query("SELECT pg_catalog.pg_replication_slot_advance(%s, %s)", name, lsn)
+                    except Exception as exc:
+                        logger.error("Error while advancing replication slot %s to position '%s': %r", name, lsn, exc)
 
     @contextmanager
     def get_local_connection_cursor(self, **kwargs: Any) -> Iterator[Union['cursor', 'Cursor[Any]']]:
@@ -484,10 +501,11 @@ class SlotsHandler:
                                replicatefrom: Optional[str] = None, paused: bool = False) -> List[str]:
         """During the HA loop read, check and alter replication slots found in the cluster.
 
-        Read physical and logical slots found on the primary, then compare to those configured in the DCS.
+        Read physical and logical slots from ``pg_replication_slots``, then compare to those configured in the DCS.
         Drop any slots that do not match those required by configuration and are not configured as permanent.
-        Create any missing physical slots. If we are the leader then logical slots too, otherwise if logical slots
-        are known and active create them on replica nodes.
+        Create any missing physical slots, or advance their position according to feedback stored in DCS.
+        If we are the primary then create logical slots, otherwise if logical slots are known and active create
+        them on replica nodes by copying slot files from the primary.
 
         :param cluster: object containing stateful information for the cluster.
         :param nofailover: ``True`` if this node has been tagged to not be a failover candidate.
diff --git a/patroni/version.py b/patroni/version.py
index ff98d3c0..b2ab33ce 100644
--- a/patroni/version.py
+++ b/patroni/version.py
@@ -2,4 +2,4 @@
 
 :var __version__: the current Patroni version.
 """
-__version__ = '3.1.1'
+__version__ = '3.1.2'


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
index 68961af7..32172926 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -104,18 +104,20 @@ class MockCursor(object):
         elif sql.startswith('SELECT slot_name, slot_type, datname, plugin, catalog_xmin'):
             self.results = [('ls', 'logical', 'a', 'b', 100, 500, b'123456')]
         elif sql.startswith('SELECT slot_name'):
-            self.results = [('blabla', 'physical'), ('foobar', 'physical'), ('ls', 'logical', 'b', 'a', 5, 100, 500)]
+            self.results = [('blabla', 'physical', 12345),
+                            ('foobar', 'physical', 12345),
+                            ('ls', 'logical', 499, 'b', 'a', 5, 100, 500)]
         elif sql.startswith('WITH slots AS (SELECT slot_name, active'):
             self.results = [(False, True)] if self.rowcount == 1 else []
         elif sql.startswith('SELECT CASE WHEN pg_catalog.pg_is_in_recovery()'):
             self.results = [(1, 2, 1, 0, False, 1, 1, None, None, 'streaming', '',
-                             [{"slot_name": "ls", "confirmed_flush_lsn": 12345}],
+                             [{"slot_name": "ls", "confirmed_flush_lsn": 12345, "restart_lsn": 12344}],
                              'on', 'n1', None)]
         elif sql.startswith('SELECT pg_catalog.pg_is_in_recovery()'):
             self.results = [(False, 2)]
         elif sql.startswith('SELECT pg_catalog.pg_postmaster_start_time'):
             self.results = [(datetime.datetime.now(tzutc),)]
-        elif sql.startswith('SELECT name, current_setting(name) FROM pg_settings'):
+        elif sql.startswith('SELECT name, pg_catalog.current_setting(name) FROM pg_catalog.pg_settings'):
             self.results = [('data_directory', 'data'),
                             ('hba_file', os.path.join('data', 'pg_hba.conf')),
                             ('ident_file', os.path.join('data', 'pg_ident.conf')),
@@ -135,6 +137,11 @@ class MockCursor(object):
                             ('wal_block_size', '8192', None, 'integer', 'internal'),
                             ('shared_buffers', '16384', '8kB', 'integer', 'postmaster'),
                             ('wal_buffers', '-1', '8kB', 'integer', 'postmaster'),
+                            ('max_connections', '100', None, 'integer', 'postmaster'),
+                            ('max_prepared_transactions', '0', None, 'integer', 'postmaster'),
+                            ('max_worker_processes', '8', None, 'integer', 'postmaster'),
+                            ('max_locks_per_transaction', '64', None, 'integer', 'postmaster'),
+                            ('max_wal_senders', '5', None, 'integer', 'postmaster'),
                             ('search_path', 'public', None, 'string', 'user'),
                             ('port', '5433', None, 'integer', 'postmaster'),
                             ('listen_addresses', '*', None, 'string', 'postmaster'),
@@ -246,14 +253,15 @@ class PostgresInit(unittest.TestCase):
 
 class BaseTestPostgresql(PostgresInit):
 
+    @patch('time.sleep', Mock())
     def setUp(self):
         super(BaseTestPostgresql, self).setUp()
 
         if not os.path.exists(self.p.data_dir):
             os.makedirs(self.p.data_dir)
 
-        self.leadermem = Member(0, 'leader', 28, {
-            'state': 'running', 'conn_url': 'postgres://replicator:rep-pass@127.0.0.1:5435/postgres'})
+        self.leadermem = Member(0, 'leader', 28, {'xlog_location': 100, 'state': 'running',
+                                                  'conn_url': 'postgres://replicator:rep-pass@127.0.0.1:5435/postgres'})
         self.leader = Leader(-1, 28, self.leadermem)
         self.other = Member(0, 'test-1', 28, {'conn_url': 'postgres://replicator:rep-pass@127.0.0.1:5433/postgres',
                                               'state': 'running', 'tags': {'replicatefrom': 'leader'}})
diff --git a/tests/test_config.py b/tests/test_config.py
index bd8d0a90..f0a780bc 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -155,6 +155,7 @@ class TestConfig(unittest.TestCase):
         expected_params = {
             'f.oo': 'bar',  # not in ConfigHandler.CMDLINE_OPTIONS
             'max_connections': 100,  # IntValidator
+            'wal_keep_size': '128MB',  # IntValidator
             'wal_level': 'hot_standby',  # EnumValidator
         }
         input_params = deepcopy(expected_params)
diff --git a/tests/test_config_generator.py b/tests/test_config_generator.py
index 49799c01..e702f957 100644
--- a/tests/test_config_generator.py
+++ b/tests/test_config_generator.py
@@ -1,6 +1,5 @@
 import os
 import psutil
-import socket
 import unittest
 
 from . import MockConnect, MockCursor, MockConnectionInfo
@@ -9,28 +8,25 @@ from mock import MagicMock, Mock, PropertyMock, mock_open, patch
 
 from patroni.__main__ import main as _main
 from patroni.config import Config
-from patroni.config_generator import AbstractConfigGenerator, get_address
-
+from patroni.config_generator import AbstractConfigGenerator, get_address, NO_VALUE_MSG
 from patroni.utils import patch_config
 
 from . import psycopg_connect
 
+HOSTNAME = 'test_hostname'
+IP = '1.9.8.4'
+
 
 @patch('patroni.psycopg.connect', psycopg_connect)
-@patch('socket.getaddrinfo', Mock(return_value=[(0, 0, 0, 0, ('1.9.8.4', 1984))]))
 @patch('builtins.open', MagicMock())
 @patch('subprocess.check_output', Mock(return_value=b"postgres (PostgreSQL) 16.2"))
 @patch('psutil.Process.exe', Mock(return_value='/bin/dir/from/running/postgres'))
 @patch('psutil.Process.__init__', Mock(return_value=None))
+@patch.object(AbstractConfigGenerator, '_HOSTNAME', HOSTNAME)
+@patch.object(AbstractConfigGenerator, '_IP', IP)
 class TestGenerateConfig(unittest.TestCase):
 
-    no_value_msg = '#FIXME'
-    _HOSTNAME = socket.gethostname()
-    _IP = sorted(socket.getaddrinfo(_HOSTNAME, 0, socket.AF_UNSPEC, socket.SOCK_STREAM, 0), key=lambda x: x[0])[0][4][0]
-
     def setUp(self):
-        self.maxDiff = None
-
         os.environ['PATRONI_SCOPE'] = 'scope_from_env'
         os.environ['PATRONI_POSTGRESQL_BIN_DIR'] = '/bin/from/env'
         os.environ['PATRONI_SUPERUSER_USERNAME'] = 'su_user_from_env'
@@ -54,14 +50,14 @@ class TestGenerateConfig(unittest.TestCase):
 
         self.config = {
             'scope': self.environ['PATRONI_SCOPE'],
-            'name': self._HOSTNAME,
+            'name': HOSTNAME,
             'bootstrap': {
                 'dcs': dynamic_config
             },
             'postgresql': {
-                'connect_address': self.no_value_msg + ':5432',
-                'data_dir': self.no_value_msg,
-                'listen': self.no_value_msg + ':5432',
+                'connect_address': NO_VALUE_MSG + ':5432',
+                'data_dir': NO_VALUE_MSG,
+                'listen': NO_VALUE_MSG + ':5432',
                 'pg_hba': ['host all all all md5',
                            f'host replication {self.environ["PATRONI_REPLICATION_USERNAME"]} all md5'],
                 'authentication': {'superuser': {'username': self.environ['PATRONI_SUPERUSER_USERNAME'],
@@ -99,7 +95,7 @@ class TestGenerateConfig(unittest.TestCase):
                 }
             },
             'postgresql': {
-                'connect_address': f'{self._IP}:bar',
+                'connect_address': f'{IP}:bar',
                 'listen': '6.6.6.6:1984',
                 'data_dir': 'data',
                 'bin_dir': '/bin/dir/from/running',
@@ -118,8 +114,8 @@ class TestGenerateConfig(unittest.TestCase):
                         'sslmode': 'prefer'
                     },
                     'replication': {
-                        'username': self.no_value_msg,
-                        'password': self.no_value_msg
+                        'username': NO_VALUE_MSG,
+                        'password': NO_VALUE_MSG
                     },
                     'rewind': None
                 },
@@ -179,7 +175,7 @@ class TestGenerateConfig(unittest.TestCase):
                 'authentication': {
                     'rewind': {
                         'username': self.environ['PATRONI_REWIND_USERNAME'],
-                        'password': self.no_value_msg}
+                        'password': NO_VALUE_MSG}
                 },
             }
         }
@@ -230,7 +226,7 @@ class TestGenerateConfig(unittest.TestCase):
                 }
             },
             'postgresql': {
-                'connect_address': f'{self._IP}:1984',
+                'connect_address': f'{IP}:1984',
                 'authentication': {
                     'superuser': {
                         'username': self.environ['PGUSER'],
@@ -330,5 +326,5 @@ class TestGenerateConfig(unittest.TestCase):
     def test_get_address(self):
         with patch('socket.getaddrinfo', Mock(side_effect=Exception)), \
              patch('logging.warning') as mock_warning:
-            self.assertEqual(get_address(), (self.no_value_msg, self.no_value_msg))
+            self.assertEqual(get_address(), (NO_VALUE_MSG, NO_VALUE_MSG))
             self.assertIn('Failed to obtain address: %r', mock_warning.call_args_list[0][0])
diff --git a/tests/test_ha.py b/tests/test_ha.py
index 9b286f4f..ea8fa7c0 100644
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -1478,6 +1478,24 @@ class TestHa(PostgresInit):
             self.ha.run_cycle()
             self.assertEqual(mock_logger.call_args[0][0], 'Updating sync state failed')
 
+    @patch.object(Cluster, 'is_unlocked', Mock(return_value=False))
+    def test_inconsistent_synchronous_state(self):
+        self.ha.is_synchronous_mode = true
+        self.ha.has_lock = true
+        self.p.name = 'leader'
+        self.ha.cluster = get_cluster_initialized_without_leader(sync=('leader', 'a'))
+        self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet('a'), CaseInsensitiveSet()))
+        self.ha.dcs.write_sync_state = Mock(return_value=SyncState.empty())
+        mock_set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock()
+        with patch('patroni.ha.logger.warning') as mock_logger:
+            self.ha.run_cycle()
+            mock_set_sync.assert_called_once()
+            self.assertTrue(mock_logger.call_args_list[0][0][0].startswith('Inconsistent state between '))
+        self.ha.dcs.write_sync_state = Mock(return_value=None)
+        with patch('patroni.ha.logger.warning') as mock_logger:
+            self.ha.run_cycle()
+            self.assertEqual(mock_logger.call_args[0][0], 'Updating sync state failed')
+
     def test_effective_tags(self):
         self.ha._disable_sync = True
         self.assertEqual(self.ha.get_effective_tags(), {'foo': 'bar', 'nosync': True})
diff --git a/tests/test_patroni.py b/tests/test_patroni.py
index 0385731c..df59677d 100644
--- a/tests/test_patroni.py
+++ b/tests/test_patroni.py
@@ -40,7 +40,7 @@ class MockFrozenImporter(object):
 @patch('time.sleep', Mock())
 @patch('subprocess.call', Mock(return_value=0))
 @patch('patroni.psycopg.connect', psycopg_connect)
-@patch('urllib3.PoolManager.request', Mock(side_effect=Exception))
+@patch('urllib3.connection.HTTPConnection.connect', Mock(side_effect=Exception))
 @patch.object(ConfigHandler, 'append_pg_hba', Mock())
 @patch.object(ConfigHandler, 'write_postgresql_conf', Mock())
 @patch.object(ConfigHandler, 'write_recovery_conf', Mock())
@@ -64,7 +64,7 @@ class TestPatroni(unittest.TestCase):
             self.assertRaises(SystemExit, _main)
 
     @patch('pkgutil.iter_importers', Mock(return_value=[MockFrozenImporter()]))
-    @patch('urllib3.PoolManager.request', Mock(side_effect=Exception))
+    @patch('urllib3.connection.HTTPConnection.connect', Mock(side_effect=Exception))
     @patch('sys.frozen', Mock(return_value=True), create=True)
     @patch.object(HTTPServer, '__init__', Mock())
     @patch.object(etcd.Client, 'read', etcd_read)
@@ -108,6 +108,7 @@ class TestPatroni(unittest.TestCase):
     @patch('os.getpid')
     @patch('multiprocessing.Process')
     @patch('patroni.__main__.patroni_main', Mock())
+    @patch('sys.argv', ['patroni.py', 'postgres0.yml'])
     def test_patroni_main(self, mock_process, mock_getpid):
         mock_getpid.return_value = 2
         _main()
@@ -233,8 +234,8 @@ class TestPatroni(unittest.TestCase):
         )
         with patch('patroni.dcs.AbstractDCS.get_cluster', Mock(return_value=bad_cluster)):
             # If the api of the running node cannot be reached, this implies unique name
-            with patch.object(self.p, 'request', Mock(side_effect=ConnectionError)):
+            with patch('urllib3.connection.HTTPConnection.connect', Mock(side_effect=ConnectionError)):
                 self.assertIsNone(self.p.ensure_unique_name())
             # Only if the api of the running node is reachable do we throw an error
-            with patch.object(self.p, 'request', Mock()):
+            with patch('urllib3.connection.HTTPConnection.connect', Mock()):
                 self.assertRaises(SystemExit, self.p.ensure_unique_name)
diff --git a/tests/test_postgresql.py b/tests/test_postgresql.py
index e2fad6ba..6ae69e85 100644
--- a/tests/test_postgresql.py
+++ b/tests/test_postgresql.py
@@ -721,6 +721,7 @@ class TestPostgresql(BaseTestPostgresql):
         self.assertEqual(self.p.get_primary_timeline(), 1)
 
     @patch.object(Postgresql, 'get_postgres_role_from_data_directory', Mock(return_value='replica'))
+    @patch.object(Postgresql, 'is_running', Mock(return_value=False))
     @patch.object(Bootstrap, 'running_custom_bootstrap', PropertyMock(return_value=True))
     @patch.object(Postgresql, 'controldata', Mock(return_value={'max_connections setting': '200',
                                                                 'max_worker_processes setting': '20',
@@ -964,6 +965,7 @@ class TestPostgresql2(BaseTestPostgresql):
     @patch('patroni.postgresql.CallbackExecutor', Mock())
     @patch.object(Postgresql, 'get_major_version', Mock(return_value=140000))
     @patch.object(Postgresql, 'is_running', Mock(return_value=True))
+    @patch.object(Postgresql, 'is_primary', Mock(return_value=False))
     def setUp(self):
         super(TestPostgresql2, self).setUp()
 
diff --git a/tests/test_slots.py b/tests/test_slots.py
index 884f76b1..d1b8d458 100644
--- a/tests/test_slots.py
+++ b/tests/test_slots.py
@@ -53,6 +53,7 @@ class TestSlotsHandler(BaseTestPostgresql):
         self.p.set_role('replica')
         with patch.object(Postgresql, 'is_primary', Mock(return_value=False)), \
                 patch.object(SlotsHandler, 'drop_replication_slot') as mock_drop:
+            config.data['slots'].pop('ls')
             self.s.sync_replication_slots(cluster, False, paused=True)
             mock_drop.assert_not_called()
         self.p.set_role('primary')
@@ -69,6 +70,8 @@ class TestSlotsHandler(BaseTestPostgresql):
             self.assertTrue("test.3" in ca, "non matching {0}".format(ca))
             with patch.object(Postgresql, 'major_version', PropertyMock(return_value=90618)):
                 self.s.sync_replication_slots(cluster, False)
+                self.p.set_role('replica')
+                self.s.sync_replication_slots(cluster, False)
 
     def test_cascading_replica_sync_replication_slots(self):
         """Test sync with a cascading replica so physical slots are present on a replica."""
@@ -82,12 +85,12 @@ class TestSlotsHandler(BaseTestPostgresql):
         self.p.set_role('replica')
         with patch.object(Postgresql, '_query') as mock_query, \
                 patch.object(Postgresql, 'is_primary', Mock(return_value=False)):
-            mock_query.return_value = [('ls', 'logical', 'b', 'a', 5, 12345, 105)]
+            mock_query.return_value = [('ls', 'logical', 104, 'b', 'a', 5, 12345, 105)]
             ret = self.s.sync_replication_slots(cluster, False)
         self.assertEqual(ret, [])
 
     def test_process_permanent_slots(self):
-        config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'}},
+        config = ClusterConfig(1, {'slots': {'ls': {'database': 'a', 'plugin': 'b'}, 'blabla': {'type': 'physical'}},
                                    'ignore_slots': [{'name': 'blabla'}]}, 1)
         cluster = Cluster(True, config, self.leader, Status.empty(), [self.me, self.other, self.leadermem],
                           None, SyncState.empty(), None, None)
@@ -98,8 +101,10 @@ class TestSlotsHandler(BaseTestPostgresql):
             mock_query.return_value = [(
                 1, 0, 0, 0, 0, 0, 0, 0, 0, None, None,
                 [{"slot_name": "ls", "type": "logical", "datoid": 5, "plugin": "b",
-                  "confirmed_flush_lsn": 12345, "catalog_xmin": 105}])]
-            self.assertEqual(self.p.slots(), {'ls': 12345})
+                  "confirmed_flush_lsn": 12345, "catalog_xmin": 105, "restart_lsn": 12344},
+                 {"slot_name": "blabla", "type": "physical", "datoid": None, "plugin": None,
+                  "confirmed_flush_lsn": None, "catalog_xmin": 105, "restart_lsn": 12344}])]
+            self.assertEqual(self.p.slots(), {'ls': 12345, 'blabla': 12344})
 
             self.p.reset_cluster_info_state(None)
             mock_query.return_value = [(
@@ -114,8 +119,8 @@ class TestSlotsHandler(BaseTestPostgresql):
         self.cluster.slots['ls'] = 12346
         with patch.object(SlotsHandler, 'check_logical_slots_readiness', Mock(return_value=False)):
             self.assertEqual(self.s.sync_replication_slots(self.cluster, False), [])
-        self.s._schedule_load_slots = False
-        with patch.object(MockCursor, 'execute', Mock(side_effect=psycopg.OperationalError)), \
+        with patch.object(SlotsHandler, '_query', Mock(return_value=[('ls', 'logical', 499, 'b', 'a', 5, 100, 500)])), \
+                patch.object(MockCursor, 'execute', Mock(side_effect=psycopg.OperationalError)), \
                 patch.object(SlotsAdvanceThread, 'schedule', Mock(return_value=(True, ['ls']))), \
                 patch.object(psycopg.OperationalError, 'diag') as mock_diag:
             type(mock_diag).sqlstate = PropertyMock(return_value='58P01')
@@ -177,3 +182,18 @@ class TestSlotsHandler(BaseTestPostgresql):
         with patch.object(SlotsHandler, 'get_local_connection_cursor', Mock(side_effect=Exception)):
             self.s.schedule_advance_slots({'foo': {'bar': 100}})
             self.s._advance.sync_slots()
+
+    @patch.object(Postgresql, 'is_primary', Mock(return_value=False))
+    def test_advance_physical_slots(self):
+        config = ClusterConfig(1, {'slots': {'blabla': {'type': 'physical'}, 'leader': None}}, 1)
+        cluster = Cluster(True, config, self.leader, Status(0, {'blabla': 12346}),
+                          [self.me, self.other, self.leadermem], None, SyncState.empty(), None, None)
+        self.s.sync_replication_slots(cluster, False)
+        with patch.object(SlotsHandler, '_query', Mock(side_effect=[[('blabla', 'physical', 12345, None, None, None,
+                                                                      None, None)], Exception])) as mock_query, \
+                patch('patroni.postgresql.slots.logger.error') as mock_error:
+            self.s.sync_replication_slots(cluster, False)
+            self.assertEqual(mock_query.call_args[0],
+                             ("SELECT pg_catalog.pg_replication_slot_advance(%s, %s)", "blabla", '0/303A'))
+            self.assertEqual(mock_error.call_args[0][0],
+                             "Error while advancing replication slot %s to position '%s': %r")
diff --git a/tests/test_zookeeper.py b/tests/test_zookeeper.py
index 45aeac10..184fda05 100644
--- a/tests/test_zookeeper.py
+++ b/tests/test_zookeeper.py
@@ -276,6 +276,7 @@ class TestZooKeeper(unittest.TestCase):
         self.assertTrue(self.zk.delete_cluster())
 
     def test_watch(self):
+        self.zk.event.wait = Mock()
         self.zk.watch(None, 0)
         self.zk.event.is_set = Mock(return_value=True)
         self.zk._fetch_status = False