Repository: patroni/patroni

Issue:
Patroni overwrite synchronous_standby_names on primary in async mode
### What happened?

We run a patroni cluster of 4 hosts and we have synchronous_mode off, but we have turned on postgresqls synchronous_commits and set synchronous_standby_names. [The documentation](https://patroni.readthedocs.io/en/latest/replication_modes.html#postgresql-synchronous-replication) tells this is possible and what we want. But patroni sets synchronous_standby_names to the configured value on all hosts except on the primary. On this host patroni sets it to the empty value, thus deactivating synchronous_commits.[](url)

### How can we reproduce it (as minimally and precisely as possible)?

Set up a patroni cluster of 3+ nodes, set `synchronous_commit: "on"` and `synchronous_standby_names: "*"` to enable Postgres’s [synchronous replication](http://www.postgresql.org/docs/current/static/warm-standby.html#SYNCHRONOUS-REPLICATION) but leave synchronous_mode off.

### What did you expect to happen?

Patroni to set synchronous_standby_names to  the configured value on all hosts and not interfere with it.

### Patroni/PostgreSQL/DCS version

-   Patroni version: 3.3.0
-   PostgreSQL version: 15.0
-   DCS (and its version): etcd 


### Patroni configuration file

```yaml
scope: xxx
namespace: /db_xxx/
name: xxx
restapi:
    listen: 0:8008
    connect_address: 10.0.1.100:8008

etcd3:
    host: localhost:2379

bootstrap:
    dcs:
        ttl: 30
        loop_wait: 10
        retry_timeout: 10
        maximum_lag_on_failover: 1048576
        postgresql:
          use_pg_rewind: true
          parameters:
            wal_keep_size: 8GB
            max_connections: 500
            shared_buffers: 700MB
            maintenance_work_mem: 512MB
            work_mem: 4MB
            effective_cache_size: 8GB
            log_lock_waits: on

    initdb:
    - encoding: UTF8
    - data-checksums


postgresql:
    listen: 0:5432
    connect_address: 10.0.1.100:5432
    data_dir: /var/lib/patroni/data
    bin_dir: /usr/lib/postgresql/15/bin
    pgpass: /tmp/pgpass
    authentication:
        replication:
            username: replicator
            password: "xxx"
        superuser:
            username: postgres
            password: "xxx"
        rewind:
            username: rewind_user
            password: "xxx"
    use_pg_rewind: true
    parameters:
        unix_socket_directories: '.'
synchronous_commit: "on"
        synchronous_standby_names: '*'

        wal_keep_size: 8GB
        max_connections: 500
        log_lock_waits: on
        shared_buffers: 700MB
        maintenance_work_mem: 512MB
        work_mem: 4MB
        effective_cache_size: 8GB
        shared_preload_libraries: 'pg_stat_statements,auto_explain'
        pg_stat_statements.max: 10000
        pg_stat_statements.track: all
        auto_explain.log_min_duration: "1s"
        tcp_keepalives_idle: 60s
        tcp_keepalives_interval: 5s
        tcp_keepalives_count: 3
        client_connection_check_interval: 5s
        autovacuum_analyze_scale_factor: 0.05
        autovacuum_vacuum_scale_factor: 0.1
        autovacuum_vacuum_insert_scale_factor: 0.1
        log_destination: 'jsonlog'
        logging_collector : on
        log_directory: '/var/log/postgresql'
        

    pg_hba:
    - local   all          all                trust
    - host    all          all  127.0.0.1/32  trust
    - host    all          all  ::1/128       trust
    - local   replication  all                trust
    - host    replication  all  127.0.0.1/32  trust
    - host    replication  all  ::1/128       trust
    - host replication replicator 10.0.1.100/0 md5
    - host replication replicator 10.0.1.101/0 md5
    - host replication replicator 10.0.1.102/0 md5
    - host replication replicator 127.0.0.1/32 md5
    - host replication replicator ::1/128 md5
    - host all all 0.0.0.0/0 md5

tags:
    nofailover: false
    noloadbalance: false
    clonefrom: false
    nosync: false
```


### patronictl show-config

```yaml
loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    effective_cache_size: 8GB
    log_lock_waits: true
    maintenance_work_mem: 512MB
    max_connections: 500
    shared_buffers: 700MB
    wal_keep_size: 8GB
    work_mem: 4MB
  use_pg_rewind: true
retry_timeout: 10
synchronous_mode: true
ttl: 30
```


### Patroni log files

```shell
-
```


### PostgreSQL log files

```shell
-
```


### Have you tried to use GitHub issue search?

- [X] Yes

### Anything else we need to know?

Discussed it in the Slack Channel:
https://postgresteam.slack.com/archives/C9XPYG92A/p1719398978518599


Discussion:

Patch:
diff --git a/patroni/ha.py b/patroni/ha.py
index 59b5b6dd1..8bbcaf660 100644
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -709,6 +709,10 @@ def process_sync_replication(self) -> None:
         then that node must have synchronous_standby set to that value. Or more simple, first set in postgresql.conf
         and then in DCS. When removing, first remove in DCS, then in postgresql.conf. This is so we only consider
         promoting standbys that were guaranteed to be replicating synchronously.
+
+        .. note::
+            If ``synchronous_mode`` is disabled, we fall back to using the value configured by the user for
+            ``synchronous_standby_names``, if any.
         """
         if self.is_synchronous_mode():
             sync = self.cluster.sync
@@ -760,9 +764,14 @@ def process_sync_replication(self) -> None:
                         return logger.info("Synchronous replication key updated by someone else")
                     logger.info("Synchronous standby status assigned to %s", list(allow_promote))
         else:
+            # If synchronous_mode was turned off, we need to update synchronous_standby_names in Postgres
             if not self.cluster.sync.is_empty and self.dcs.delete_sync_state(version=self.cluster.sync.version):
                 logger.info("Disabled synchronous replication")
-            self.state_handler.sync_handler.set_synchronous_standby_names(CaseInsensitiveSet())
+                self.state_handler.sync_handler.set_synchronous_standby_names(CaseInsensitiveSet())
+
+            # As synchronous_mode is off, check if the user configured Postgres synchronous replication instead
+            ssn = self.state_handler.config.synchronous_standby_names
+            self.state_handler.config.set_synchronous_standby_names(ssn)
 
     def is_sync_standby(self, cluster: Cluster) -> bool:
         """:returns: `True` if the current node is a synchronous standby."""
diff --git a/patroni/postgresql/config.py b/patroni/postgresql/config.py
index 77d31cb60..f7678045d 100644
--- a/patroni/postgresql/config.py
+++ b/patroni/postgresql/config.py
@@ -1344,3 +1344,12 @@ def get(self, key: str, default: Optional[Any] = None) -> Optional[Any]:
 
     def restore_command(self) -> Optional[str]:
         return (self.get('recovery_conf') or EMPTY_DICT).get('restore_command')
+
+    @property
+    def synchronous_standby_names(self) -> Optional[str]:
+        """Get ``synchronous_standby_names`` value configured by the user.
+
+        :returns: value of ``synchronous_standby_names`` in the Patroni configuration,
+            if any, otherwise ``None``.
+        """
+        return (self.get('parameters') or EMPTY_DICT).get('synchronous_standby_names')


Test patch:
diff --git a/tests/test_ha.py b/tests/test_ha.py
index 5f6069d6b..6ec28775e 100644
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -1306,6 +1306,7 @@ def test_demote_immediate(self, follow):
     def test_process_sync_replication(self):
         self.ha.has_lock = true
         mock_set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock()
+        mock_cfg_set_sync = self.p.config.set_synchronous_standby_names = Mock()
         self.p.name = 'leader'
 
         # Test sync key removed when sync mode disabled
@@ -1314,16 +1315,20 @@ def test_process_sync_replication(self):
             self.ha.run_cycle()
             mock_delete_sync.assert_called_once()
             mock_set_sync.assert_called_once_with(CaseInsensitiveSet())
+            mock_cfg_set_sync.assert_called_once()
 
         mock_set_sync.reset_mock()
+        mock_cfg_set_sync.reset_mock()
         # Test sync key not touched when not there
         self.ha.cluster = get_cluster_initialized_with_leader()
         with patch.object(self.ha.dcs, 'delete_sync_state') as mock_delete_sync:
             self.ha.run_cycle()
             mock_delete_sync.assert_not_called()
-            mock_set_sync.assert_called_once_with(CaseInsensitiveSet())
+            mock_set_sync.assert_not_called()
+            mock_cfg_set_sync.assert_called_once()
 
         mock_set_sync.reset_mock()
+        mock_cfg_set_sync.reset_mock()
 
         self.ha.is_synchronous_mode = true
 
@@ -1335,12 +1340,14 @@ def test_process_sync_replication(self):
         mock_set_sync.assert_not_called()
 
         mock_set_sync.reset_mock()
+        mock_cfg_set_sync.reset_mock()
 
         # Test sync standby is replaced when switching standbys
         self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet(['other2']), CaseInsensitiveSet()))
         self.ha.dcs.write_sync_state = Mock(return_value=SyncState.empty())
         self.ha.run_cycle()
         mock_set_sync.assert_called_once_with(CaseInsensitiveSet(['other2']))
+        mock_cfg_set_sync.assert_not_called()
 
         # Test sync standby is replaced when new standby is joined
         self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet(['other2', 'other3']),
@@ -1349,14 +1356,18 @@ def test_process_sync_replication(self):
         self.ha.run_cycle()
         self.assertEqual(mock_set_sync.call_args_list[0][0], (CaseInsensitiveSet(['other2']),))
         self.assertEqual(mock_set_sync.call_args_list[1][0], (CaseInsensitiveSet(['other2', 'other3']),))
+        mock_cfg_set_sync.assert_not_called()
 
         mock_set_sync.reset_mock()
+        mock_cfg_set_sync.reset_mock()
         # Test sync standby is not disabled when updating dcs fails
         self.ha.dcs.write_sync_state = Mock(return_value=None)
         self.ha.run_cycle()
         mock_set_sync.assert_not_called()
+        mock_cfg_set_sync.assert_not_called()
 
         mock_set_sync.reset_mock()
+        mock_cfg_set_sync.reset_mock()
         # Test changing sync standby
         self.ha.dcs.write_sync_state = Mock(return_value=SyncState.empty())
         self.ha.dcs.get_cluster = Mock(return_value=get_cluster_initialized_with_leader(sync=('leader', 'other')))
@@ -1384,10 +1395,23 @@ def test_process_sync_replication(self):
 
         # Test sync set to '*' when synchronous_mode_strict is enabled
         mock_set_sync.reset_mock()
+        mock_cfg_set_sync.reset_mock()
         self.p.sync_handler.current_state = Mock(return_value=(CaseInsensitiveSet(), CaseInsensitiveSet()))
         self.ha.cluster.config.data['synchronous_mode_strict'] = True
         self.ha.run_cycle()
         mock_set_sync.assert_called_once_with(CaseInsensitiveSet('*'))
+        mock_cfg_set_sync.assert_not_called()
+
+        # Test the value configured by the user for synchronous_standby_names is used when synchronous mode is disabled
+        self.ha.is_synchronous_mode = false
+
+        mock_set_sync.reset_mock()
+        mock_cfg_set_sync.reset_mock()
+        ssn_mock = PropertyMock(return_value="SOME_SSN")
+        with patch('patroni.postgresql.config.ConfigHandler.synchronous_standby_names', ssn_mock):
+            self.ha.run_cycle()
+            mock_set_sync.assert_not_called()
+            mock_cfg_set_sync.assert_called_once_with("SOME_SSN")
 
     def test_sync_replication_become_primary(self):
         self.ha.is_synchronous_mode = true