Repository: zalando/patroni

Issue:
FileExistsError When Creating Tablespace and/or WAL symlinks Following Failed Cluster Bootstrap
**Describe the bug**

When Patroni attempts to rename any tablespace or WAL directories and then recreate symlinks for those directories following a failed cluster bootstrap attempt (specifically by calling the `move_data_directory` function), a `FileExistsError` is thrown when the symlinks are recreated (log output is displayed below).  Because of this, the PGDATA directory is not renamed, nor are certain tablespace and/or WAL directories, and any tablespace and/or WAL symlinks are not successfully recreated.

Looking at the calls to `os.symlink` in function `move_data_directory`, it appears as though the arguments being provided are out of order.  Therefore, the following:

https://github.com/zalando/patroni/blob/1530ed0b9c4a7f9e98b85497c7116d2ca8d44759/patroni/postgresql/__init__.py#L918
https://github.com/zalando/patroni/blob/1530ed0b9c4a7f9e98b85497c7116d2ca8d44759/patroni/postgresql/__init__.py#L926

...should really be:

```python
os.symlink(new_name, source)
```

Specifically, being that `new_name` represents the directory the symlink is being created for, and `source` represents the name of the symlink being created, this aligns with the [Python documentation for os.symlink](https://docs.python.org/3/library/os.html#os.symlink):

> Create a symbolic link pointing to src named dst.

**To Reproduce**

1) Create a custom pgBackRest bootstrap method per the Patroni configuration file snippet shown below that will restore from a backup of from database that utilizes tablespaces.  For this example, an invalid recovery option is provided to force any error after the restore is completed, and therefore trigger a call to `move_data_directory` to demonstrate the issue:

>```bash
>pgbackrest restore --recovery-option=invalid=option
>```

2) Initialize a cluster using the `pgbackrest` bootstrap method.
3) View the `FileExistsError` error in the Patroni logs when an attempt (log output shown below).

**Expected behavior**

All tablespace and/or WAL volumes should be renamed, and any symlinks for those directories should then be successfully recreated.

**Screenshots**

N/A

**Environment**

- Patroni version: v2.0.1
- PostgreSQL version: v12.5
- DCS (and its version): Kubernetes v1.19.5

**Patroni configuration file**

This is the custom bootstrap method used for steps provided to reproduce above:
```yaml
bootstrap:
  method: pgbackrest
    pgbackrest:
      command: 'pgbackrest restore --recovery-option=invalid=option'
      keep_existing_recovery_conf: true
      no_params: true
```

As described above, the `recovery-option` is simply set to force a failure when starting the database following the pgBackRest restore.

**patronictl show-config**

N/A

**Have you checked Patroni logs?**

This is the error that is displayed when Patroni attempts to recreate a tablespace symlink following a failed init attempt:

```bash
2020-12-05 19:29:38.389 GMT [270] LOG:  unrecognized configuration parameter "invalid" in file "/pgdata/mycluster1/postgresql.auto.conf" line 7
2020-12-05 19:29:38.389 GMT [270] FATAL:  configuration file "/pgdata/mycluster1/postgresql.auto.conf" contains errors
2020-12-05 19:29:39,148 INFO: Lock owner: None; I am mycluster1-bootstrap-kjczx
2020-12-05 19:29:39,148 INFO: not healthy enough for leader race
2020-12-05 19:29:39,206 INFO: bootstrap in progress
2020-12-05 19:29:39,388 ERROR: postmaster is not running
2020-12-05 19:29:39,389 INFO: removing initialize key after failed attempt to bootstrap the cluster
2020-12-05 19:29:39,437 INFO: renaming user defined tablespace directory and updating symlink: /tablespaces/ts1/ts1
2020-12-05 19:29:39,444 ERROR: Could not rename data directory /pgdata/mycluster1
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/patroni/postgresql/__init__.py", line 926, in move_data_directory
    os.symlink(source, new_name)
FileExistsError: [Errno 17] File exists: '/pgdata/mycluster1/pg_tblspc/16414' -> '/tablespaces/ts1/ts1_2020-12-05-19-29-39'
Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 170, in main
    return patroni_main()
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 138, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 100, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 108, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 59, in run
    self._run_cycle()
  File "/usr/local/lib/python3.6/site-packages/patroni/__init__.py", line 111, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1452, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1346, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1242, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1235, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
```

This same error will also be shown if an attempt is made to rename a WAL directory, and then recreate it's associated symlink.

**Have you checked PostgreSQL logs?**

N/A

**Have you tried to use GitHub issue search?**

Yes - unable to find the same or a similar issue.

**Additional context**

N/A



Discussion:

Resolving patch:
diff --git a/patroni/postgresql/__init__.py b/patroni/postgresql/__init__.py
--- a/patroni/postgresql/__init__.py
+++ b/patroni/postgresql/__init__.py
@@ -915,7 +915,7 @@ def move_data_directory(self):
                     new_name = '{0}_{1}'.format(pg_wal_realpath, postfix)
                     os.rename(pg_wal_realpath, new_name)
                     os.unlink(source)
-                    os.symlink(source, new_name)
+                    os.symlink(new_name, source)
 
                 # Move user defined tablespace directory
                 for (source, pg_tsp_rpath) in self.pg_tblspc_realpaths().items():
@@ -923,7 +923,7 @@ def move_data_directory(self):
                     new_name = '{0}_{1}'.format(pg_tsp_rpath, postfix)
                     os.rename(pg_tsp_rpath, new_name)
                     os.unlink(source)
-                    os.symlink(source, new_name)
+                    os.symlink(new_name, source)
 
                 new_name = '{0}_{1}'.format(self._data_dir, postfix)
                 logger.info('renaming data directory to %s', new_name)


Test patch: