Repository: patroni/patroni

Issue:
Incorrect host in .pgpass file for post_init script when using unix_socket_directories
### What happened?

When `postgresql.parameters.unix_socket_directories` is set and `postgresql.use_unix_socket` is true, patroni creates `.pgpass` file using socket path as `host`, while the [documentation](https://www.postgresql.org/docs/current/libpq-pgpass.html) says that we should set host to `hostname` in .pgpass file when connecting through UNIX socket. This causes the post_init script to terminate with an error due to an authentication error, which in turn leads to a cluster bootstrap failure.

Example of .pgpass file created by patroni:
```
/var/run/postgresql:5432:*:postgres:password
```

### How can we reproduce it (as minimally and precisely as possible)?

Patroni configuration:
```yaml
bootstrap:
  ...
  post_bootstrap: /etc/patroni/post_bootstrap.sh

postgresql:
  authentication:
    replication:
      password: password
      username: replicator
    superuser:
      password: password
      username: postgres
  ...
  parameters:
    ...
    unix_socket_directories: /var/run/postgresql
  pgpass: /tmp/.pgpass_patroni
  use_unix_socket: true
```

Content of `/etc/patroni/post_bootstrap.sh`:
```bash
#!/bin/bash

echo "\c template1
CREATE EXTENSION pg_stat_statements;" \
| psql -d "$1"
```

### What did you expect to happen?

Patroni should create .pgpass file with host set to `localhost`:
```
localhost:5432:*:postgres:password
```

### Patroni/PostgreSQL/DCS version

-   Patroni version: 3.3.2 (package `patroni-3.3.2-1PGDG.rhel9.noarch.rpm` from [postgresql-common yum repository](https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-9-x86_64/))
-   PostgreSQL version: 12.20
-   DCS (and its version): `etcd Version: 3.5.15 Git SHA: 9a5533382 Go Version: go1.21.12 Go OS/Arch: linux/amd64`
-   Python: 3.9.18


### Patroni configuration file

```yaml
# Properties like host names, IPs, passwords and scope are redacted

scope: example
name: example-db-02
namespace: /service/

log:
  level: INFO
  loggers:
    patroni.postmaster: INFO
    urllib3: INFO
  max_queue_size: 1000
  traceback_level: ERROR

restapi:
  listen: 192.168.0.11:8008
  connect_address: 192.168.0.11:8008

etcd3:
  hosts: ['192.168.0.10:2379', '192.168.0.11:2379', '192.168.0.12:2379']

bootstrap:
  dcs:
    loop_wait: 10
    master_start_timeout: 300
    maximum_lag_on_failover: 1048576
    maximum_lag_on_syncnode: 1048576
    postgresql:
      parameters:
        max_connections: 400
      use_pg_rewind: false
      use_slots: true
    retry_timeout: 10
    ttl: 30
  initdb:
  - locale: C.utf8
  - auth: scram-sha-256
  method: initdb
  post_bootstrap: /etc/patroni/post_bootstrap.sh

postgresql:
  authentication:
    replication:
      password: password
      username: replicator
    superuser:
      password: password
      username: postgres
  bin_dir: /usr/pgsql-12/bin
  connect_address: 192.168.0.11:5432
  create_replica_methods:
  - basebackup
  data_dir: /pgdata/12/data
  listen: 192.168.0.11:5432
  parameters:
    password_encryption: scram-sha-256
    shared_preload_libraries: pg_stat_statements, pgaudit, pg_stat_kcache
    stats_temp_directory: /pgstats
    unix_socket_directories: /var/run/postgresql
  pg_hba:
  - local all postgres scram-sha-256
  - local postgres postgres_exporter scram-sha-256
  - host postgres postgres 192.168.0.10/32 scram-sha-256
  - host replication replicator 192.168.0.10/32 scram-sha-256
  - host postgres postgres 192.168.0.11/32 scram-sha-256
  - host replication replicator 192.168.0.11/32 scram-sha-256
  pgpass: /tmp/.pgpass_patroni
  remove_data_directory_on_diverged_timelines: false
  remove_data_directory_on_rewind_failure: false
  use_unix_socket: true

tags:
  nofailover: false
  noloadbalance: false
  clonefrom: false
  nosync: false
```


### patronictl show-config

```yaml
loop_wait: 10
master_start_timeout: 300
maximum_lag_on_failover: 1048576
maximum_lag_on_syncnode: 1048576
postgresql:
  parameters:
    max_connections: 400
  use_pg_rewind: false
  use_slots: true
retry_timeout: 10
ttl: 30
```


### Patroni log files

```shell
2024-08-22 19:01:50,836 INFO: Selected new etcd server http://192.168.0.11:2379
2024-08-22 19:01:50,842 INFO: No PostgreSQL configuration items changed, nothing to reload.
2024-08-22 19:01:50,887 INFO: Lock owner: None; I am example-db-02
2024-08-22 19:01:50,976 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /pgdata/12/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Europe/Moscow
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
   /usr/pgsql-12/bin/pg_ctl -D /pgdata/12/data -l logfile start
2024-08-22 19:01:51.894 MSK [127846] LOG:  pgaudit extension initialized
2024-08-22 19:01:51.895 MSK [127846] LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2024-08-22 19:01:51.895 MSK [127846] LOG:  pg_stat_kcache.linux_hz is set to 1000000
2024-08-22 19:01:51.895 MSK [127846] LOG:  starting PostgreSQL 12.20 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3), 64-bit
2024-08-22 19:01:51.895 MSK [127846] LOG:  listening on IPv4 address "192.168.0.11", port 5432
2024-08-22 19:01:51.896 MSK [127846] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-08-22 19:01:51,908 INFO: postmaster pid=127846
2024-08-22 19:01:51.941 MSK [127846] LOG:  redirecting log output to logging collector process
2024-08-22 19:01:51.941 MSK [127846] HINT:  Future log output will appear in directory "/pgerrorlogs".
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - accepting connections
2024-08-22 19:01:51,961 INFO: establishing a new patroni heartbeat connection to postgres
2024-08-22 19:01:52,013 INFO: running post_bootstrap
Password for user postgres:
psql: error: FATAL:  password authentication failed for user "postgres"
2024-08-22 19:01:52,038 ERROR: post_init script /etc/patroni/post_bootstrap.sh returned non-zero code 2
2024-08-22 19:01:52,038 INFO: removing initialize key after failed attempt to bootstrap the cluster
2024-08-22 19:01:52,109 INFO: renaming data directory to /pgdata/12/data.failed
Traceback (most recent call last):
 File "/usr/bin/patroni", line 33, in <module>
   sys.exit(load_entry_point('patroni==3.3.2', 'console_scripts', 'patroni')())
 File "/usr/lib/python3.9/site-packages/patroni/__main__.py", line 344, in main
   return patroni_main(args.configfile)
 File "/usr/lib/python3.9/site-packages/patroni/__main__.py", line 232, in patroni_main
   abstract_main(Patroni, configfile)
 File "/usr/lib/python3.9/site-packages/patroni/daemon.py", line 174, in abstract_main
   controller.run()
 File "/usr/lib/python3.9/site-packages/patroni/__main__.py", line 192, in run
   super(Patroni, self).run()
 File "/usr/lib/python3.9/site-packages/patroni/daemon.py", line 143, in run
   self._run_cycle()
 File "/usr/lib/python3.9/site-packages/patroni/__main__.py", line 201, in _run_cycle
   logger.info(self.ha.run_cycle())
 File "/usr/lib/python3.9/site-packages/patroni/ha.py", line 1989, in run_cycle
   info = self._run_cycle()
 File "/usr/lib/python3.9/site-packages/patroni/ha.py", line 1806, in _run_cycle
   return self.post_bootstrap()
 File "/usr/lib/python3.9/site-packages/patroni/ha.py", line 1690, in post_bootstrap
   self.cancel_initialization()
 File "/usr/lib/python3.9/site-packages/patroni/ha.py", line 1683, in cancel_initialization
   raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: Failed to bootstrap cluster
```


### PostgreSQL log files

```shell
2024-08-22 19:01:51.943 MSK [127850] FATAL:  the database system is starting up
2024-08-22 19:01:51.947 MSK [127846] LOG:  database system is ready to accept connections
2024-08-22 19:01:52.014 MSK [127860] LOG:  AUDIT: SESSION,1,1,FUNCTION,DO,,,"DO $$
        BEGIN
            SET local synchronous_commit = 'local';
            PERFORM * FROM pg_catalog.pg_authid WHERE rolname = 'postgres';
            IF FOUND THEN
                ALTER ROLE ""postgres"" WITH SUPERUSER LOGIN PASSWORD 'password';
            ELSE
                CREATE ROLE ""postgres"" WITH SUPERUSER LOGIN PASSWORD 'password';
            END IF;
        END;$$",<none>
2024-08-22 19:01:52.019 MSK [127860] LOG:  AUDIT: SESSION,1,2,ROLE,ALTER ROLE,,,"ALTER ROLE ""postgres"" WITH SUPERUSER LOGIN PASSWORD <REDACTED>",<none>
2024-08-22 19:01:52.037 MSK [127867] FATAL:  password authentication failed for user "postgres"
2024-08-22 19:01:52.037 MSK [127867] DETAIL:  Connection matched pg_hba.conf line 3: "local all postgres scram-sha-256"
2024-08-22 19:01:52.082 MSK [127846] LOG:  received immediate shutdown request
2024-08-22 19:01:52.083 MSK [127860] WARNING:  terminating connection because of crash of another server process
2024-08-22 19:01:52.083 MSK [127860] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2024-08-22 19:01:52.083 MSK [127860] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2024-08-22 19:01:52.084 MSK [127855] WARNING:  terminating connection because of crash of another server process
2024-08-22 19:01:52.084 MSK [127855] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2024-08-22 19:01:52.084 MSK [127855] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2024-08-22 19:01:52.084 MSK [127846] LOG:  archiver process (PID 127856) exited with exit code 2
2024-08-22 19:01:52.095 MSK [127846] LOG:  database system is shut down
```


### Have you tried to use GitHub issue search?

- [X] Yes

### Anything else we need to know?

I think this problem is related to `call_post_bootstrap()` in `bootstrap.py`:
https://github.com/patroni/patroni/blob/805bdcfe50f238af6f60d358c4bdf84631395442/patroni/postgresql/bootstrap.py#L218-L223

The write_pgpass function is called with a statically specified hostname + connection parameters for psycopg, which in turn are set in `config.py`:
https://github.com/patroni/patroni/blob/805bdcfe50f238af6f60d358c4bdf84631395442/patroni/postgresql/config.py#L1071-L1084

Here `unix_local_address` OR `tcp_local_address` is selected as the connection host, so when the resulted `self._postgresql.connection_pool.conn_kwargs` is passed to `write_pgpass` in `bootstrap.py`, in fact its arguments looks like this:

```python
env = self._postgresql.config.write_pgpass({'host': 'localhost', 'host': '/var/run/postgresql', 
        'port': '5432', 'password': 'password', 'dbname': 'postgres', 
        'fallback_application_name': 'Patroni', 'connect_timeout': 3,
        'options': '-c statement_timeout=2000', 'user': 'postgres'})
```

So the first `host` parameter is being overriden by the second.

I was able to work around this by changing the order of arguments passed to `write_pgpass()` like this:
```python
env = self._postgresql.config.write_pgpass({**r, 'host': 'localhost'})
```

But it will not work if UNIX sockets are not used and the server is not listening on `localhost` or `0.0.0.0`, so it might be better to add logic to `write_pgpass()` like this:
```python
def write_pgpass(self, record: Dict[str, Any]) -> Dict[str, str]:
    if self._config.get('use_unix_socket'):
        record['host'] = 'localhost'
    content = self._pgpass_content(record)
    ...
```

And remove statically defined `host` parameter from `write_pgpass()` call in `bootstrap.py`.


Discussion:
@Tereorik thanks for report.
Could you please verify if the following patch helps:
```diff
diff --git a/patroni/postgresql/config.py b/patroni/postgresql/config.py
index bcab56a0..9d1710c0 100644
--- a/patroni/postgresql/config.py
+++ b/patroni/postgresql/config.py
@@ -914,7 +914,7 @@ class ConfigHandler(object):
             hosts = map(escape, filter(None, map(str.strip,
                         (record.get('host', '') or '*').split(','))))  # pyright: ignore [reportUnknownArgumentType]
             record = {n: escape(record.get(n) or '*') for n in ('port', 'user', 'password')}
-            return '\n'.join('{host}:{port}:*:{user}:{password}'.format(**record, host=host) for host in hosts)
+            return '\n'.join('{host}:{port}:*:{user}:{password}'.format(**record, host=escape(host)) for host in hosts)
 
     def write_pgpass(self, record: Dict[str, Any]) -> Dict[str, str]:
         """Maybe creates :attr:`_passfile` based on connection parameters.
```
Oh... scratch my previous message. The `escape(host)` will not do anything useful and will not help.

>  The host name localhost is also searched for when the connection is a Unix-domain socket connection and the host parameter matches libpq's default socket directory path.

We are dealing with a weird corner-case.
The `localhost` will be search when we connect using `libpq`:
1. without specifying the `host` (libpq will use `default socket directory path` in this case)
2. to the `host` that matches `default socket directory path`.

The second case is less obvious and you are facing exactly this one.

For all remaining cases the pgpassfile may perfectly contain directory name in the field of host (e.g. /tmp or similar).

Unfortunately we can't get the value of `default socket directory path` from libpq.
But, what probably can do - if we detect that `host` starts with '/' character then add yet another line to pgpass with `localhost`.



Patch:
diff --git a/patroni/postgresql/config.py b/patroni/postgresql/config.py
index bcab56a0a..18f5c0552 100644
--- a/patroni/postgresql/config.py
+++ b/patroni/postgresql/config.py
@@ -911,10 +911,12 @@ def escape(value: Any) -> str:
                 return re.sub(r'([:\\])', r'\\\1', str(value))
 
             # 'host' could be several comma-separated hostnames, in this case we need to write on pgpass line per host
-            hosts = map(escape, filter(None, map(str.strip,
-                        (record.get('host', '') or '*').split(','))))  # pyright: ignore [reportUnknownArgumentType]
+            hosts = [escape(host) for host in filter(None, map(str.strip,
+                     (record.get('host', '') or '*').split(',')))]  # pyright: ignore [reportUnknownArgumentType]
+            if any(host.startswith('/') for host in hosts) and 'localhost' not in hosts:
+                hosts.append('localhost')
             record = {n: escape(record.get(n) or '*') for n in ('port', 'user', 'password')}
-            return '\n'.join('{host}:{port}:*:{user}:{password}'.format(**record, host=host) for host in hosts)
+            return ''.join('{host}:{port}:*:{user}:{password}\n'.format(**record, host=host) for host in hosts)
 
     def write_pgpass(self, record: Dict[str, Any]) -> Dict[str, str]:
         """Maybe creates :attr:`_passfile` based on connection parameters.


Test patch:
diff --git a/tests/test_postgresql.py b/tests/test_postgresql.py
index 76b51aeb4..383c3a3a5 100644
--- a/tests/test_postgresql.py
+++ b/tests/test_postgresql.py
@@ -262,6 +262,10 @@ def test_write_pgpass(self):
         self.p.config.write_pgpass({'host': 'localhost', 'port': '5432', 'user': 'foo'})
         self.p.config.write_pgpass({'host': 'localhost', 'port': '5432', 'user': 'foo', 'password': 'bar'})
 
+    def test__pgpass_content(self):
+        pgpass = self.p.config._pgpass_content({'host': '/tmp', 'port': '5432', 'user': 'foo', 'password': 'bar'})
+        self.assertEqual(pgpass, "/tmp:5432:*:foo:bar\nlocalhost:5432:*:foo:bar\n")
+
     def test_checkpoint(self):
         with patch.object(MockCursor, 'fetchone', Mock(return_value=(True, ))):
             self.assertEqual(self.p.checkpoint({'user': 'postgres'}), 'is_in_recovery=true')