Repository: zalando/patroni

Issue:
rejoin former master fails
I'm not entirely sure if this is expected behaviour or not. If I stop a master (systemctl stop patroni) a slave takes over. This works fine. If I then want to rejoin the master as slave using just systemctl start patroni it fails: 

> < 2016-03-31 09:13:38.119 CEST >ERROR:  requested starting point 0/26000000 on timeline 5 is not in this server's history
> < 2016-03-31 09:13:38.119 CEST >DETAIL:  This server's history forked from timeline 5 at 0/250001A8.

I can fix it by manually running pg_rewind on the former master:
`sudo -u postgres pg_rewind -D /var/lib/pgsql/9.5/data --source-server='host=currentmaster password=pwforpostgres' 
`
After running that command the former master can join again as slave and everything appears to work. Looking at the code I see that there is quite a bit of code there for running pg_rewind already so I am wondering if I have hit a bug or if this is how it is expected to work and I should always manually intervene before rejoining a former master even if it was shut down manually (not crashed.)



Discussion:
You have to configure the username and the password for pg_rewind first (i.e. https://github.com/zalando/patroni/blob/master/postgres0.yml). Create a pg_rewind section in the patroni configuration file and specify there the superuser username and the password, i.e.
pg_rewind:
    username: postgres
    password: pwforpostgres

I assume you have set either the wal_log_hints or enabled the checksum, because this is a prerequisite to running pg_rewind. 

Hmm, usually pg_rewind is not necessary if patroni (postgres) was shut down cleanly.
How it works:
1. `systemctl stop patroni` sends (hopefully) `SIGTERM` to patroni
2. patroni handles this signal and shuts down postgres with: `pg_ctl stop -m fast`
3. after clean shutdown of postgres patroni removes `leader` lock from etcd (or zookeeper)
4. one of replicas acquires leader lock and promote.

What can go wrong?
It could happen that patroni was not able to shut down postgres during 10 seconds (https://github.com/zalando/patroni/blob/master/extras/startup-scripts/patroni.service#L21) and was killed with `SIGKILL`. In this case replica has a chance to promote earlier then former master was shut down.

In order to get full picture and answer all questions I need to get:
1. config yaml file from the former master
2. patroni logs from the former master (starting from the moment when you run  `systemctl stop patroni`)
3. postgres logs from the former master (starting from the moment when you run  `systemctl stop patroni`)

And also please check, that after execution of `systemctl stop patroni` there is no postgres running on this host any more.

@alexeyklyukin Yes wal_log_hints is on and I have tried with both data-checksums of and on. The pg_rewind settings are also there (assuming that is supposed to be the postgres  pasword for the database, not for the OS user.) 

@CyberDem0n Sorry for the delay, found the time to get all the requested info:
[logfiles.txt](https://github.com/zalando/patroni/files/204181/logfiles.txt)

[patroni.yaml.txt](https://github.com/zalando/patroni/files/204183/patroni.yaml.txt)

ps.: I can reproduce this every time (you can actually see from the logfile the slave is just starting for the first time and creating the new replication slot.) 

**< 2016-04-05 11:11:11.130 CEST >LOG:  received smart shutdown request**
< 2016-04-05 11:11:11.130 CEST >LOG:  autovacuum launcher shutting down
< 2016-04-05 11:11:11.130 CEST >FATAL:  terminating connection due to administrator command
< 2016-04-05 11:11:11.130 CEST >FATAL:  terminating connection due to administrator command
< 2016-04-05 11:11:11.137 CEST >LOG:  shutting down
< 2016-04-05 11:11:11.147 CEST >FATAL:  the database system is shutting down
**< 2016-04-05 11:11:11.232 CEST >LOG:  received fast shutdown request**

Somehow postgres received two shutdown requests. But only one of then actually was initiated by patroni, the last one (fast shutdown).
Replica stopped streaming right after the first shutdown request (smart) and didn't got latest changes. Actually it should be solved by pg_rewind, but somehow it was not called...

Now we need to figure out who and why sends smart shutdown request. I guess it was sent by `systemctl`

```
KillMode=

    Specifies how processes of this unit shall be killed. One of control-group, process, mixed, none.

    If set to control-group, all remaining processes in the control group of this unit will be killed on unit stop (for services: after the stop command is executed, as configured with ExecStop=). If set to process, only the main process itself is killed. If set to mixed, the SIGTERM signal (see below) is sent to the main process while the subsequent SIGKILL signal (see below) is sent to all remaining processes of the unit's control group. If set to none, no process is killed. In this case, only the stop command will be executed on unit stop, but no process be killed otherwise. Processes remaining alive after stop are left in their control group and the control group continues to exist after stop unless it is empty.

    Processes will first be terminated via SIGTERM (unless the signal to send is changed via KillSignal=). Optionally, this is immediately followed by a SIGHUP (if enabled with SendSIGHUP=). If then, after a delay (configured via the TimeoutStopSec= option), processes still remain, the termination request is repeated with the SIGKILL signal (unless this is disabled via the SendSIGKILL= option). See kill(2) for more information.

    Defaults to control-group.
```

At least in our example we didn't changed KillMode, so probably it defaults to `control-group`.
If you are using not the latest version of patroni (which doesn't include https://github.com/zalando/patroni/commit/7b524bc5571f2c49d7375d2cbd4eb3c7d303dca8), then it could explain such behaviour.

In any case please try to update your service file and set `KillMode=process`, hopefully it should help.

Thanks, this does indeed solve the problem! 

For me it is still not clear why patroni was not calling pg_rewind...

True, it is still reproducible by sending the kill signals systemd is sending manually. 
So first ps -ef | grep postgres (assuming patroni and postgres run as postgres user) and then a single kill command with a list of all the pid's. 


Resolving patch:
diff --git a/patroni/ha.py b/patroni/ha.py
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -106,12 +106,6 @@ def bootstrap(self):
             return 'waiting for leader to bootstrap'
 
     def recover(self):
-        # try to see if we are the former master that crashed. If so - we likely need to run pg_rewind
-        # in order to join the former standby being promoted.
-        if self.state_handler.role == 'master':
-            pg_controldata = self.state_handler.controldata()
-            if pg_controldata and pg_controldata.get('Database cluster state', '') == 'in production':  # crashed master
-                self.state_handler.require_rewind()
         self.recovering = True
         return self.follow("starting as readonly because i had the session lock", "starting as a secondary", True, True)
 
diff --git a/patroni/postgresql.py b/patroni/postgresql.py
--- a/patroni/postgresql.py
+++ b/patroni/postgresql.py
@@ -73,14 +73,13 @@ def __init__(self, config):
 
         self._connection = None
         self._cursor_holder = None
-        self._need_rewind = False
         self._sysid = None
         self.replication_slots = []  # list of already existing replication slots
         self.retry = Retry(max_tries=-1, deadline=5, max_delay=1, retry_exceptions=PostgresConnectionException)
 
         self._state = 'stopped'
         self._state_lock = Lock()
-        self._role = 'replica'
+        self._role = self.get_postgres_role_from_data_directory()
         self._role_lock = Lock()
 
         if self.is_running():
@@ -115,9 +114,6 @@ def sysid(self):
             self._sysid = data.get('Database system identifier', "")
         return self._sysid
 
-    def require_rewind(self):
-        self._need_rewind = True
-
     def get_local_address(self):
         listen_addresses = self.listen_addresses.split(',')
         local_address = listen_addresses[0].strip()  # take first address from listen_addresses
@@ -128,6 +124,9 @@ def get_local_address(self):
                 break
         return local_address + ':' + self.port
 
+    def get_postgres_role_from_data_directory(self):
+        return 'replica' if os.path.exists(self.recovery_conf) else 'master'
+
     @property
     def _connect_kwargs(self):
         r = parseurl('postgres://{0}/postgres'.format(self.local_address))
@@ -351,7 +350,7 @@ def start(self, block_callbacks=False):
             logger.error('Cannot start PostgreSQL because one is already running.')
             return True
 
-        self.set_role('replica' if os.path.exists(self.recovery_conf) else 'master')
+        self.set_role(self.get_postgres_role_from_data_directory())
         if os.path.exists(self.postmaster_pid):
             os.remove(self.postmaster_pid)
             logger.info('Removed %s', self.postmaster_pid)
@@ -564,13 +563,12 @@ def cleanup_archive_status(self):
     def follow(self, leader, recovery=False):
         if self.check_recovery_conf(leader) and not recovery:
             return True
-
         change_role = self.role == 'master'
-        self._need_rewind = (self._need_rewind or change_role) and self.can_rewind
-        if self._need_rewind:
+        need_rewind = change_role and self.can_rewind
+        if need_rewind:
             logger.info("set the rewind flag after demote")
         self.write_recovery_conf(leader)
-        if leader and self._need_rewind:  # we have a leader and need to rewind
+        if leader and need_rewind:  # we have a leader and need to rewind
             if self.is_running():
                 self.stop()
             # at present, pg_rewind only runs when the cluster is shut down cleanly
@@ -594,7 +592,6 @@ def follow(self, leader, recovery=False):
                 logger.error("unable to rewind the former master")
                 self.remove_data_directory()
                 ret = True
-            self._need_rewind = False
         else:  # do not rewind until the leader becomes available
             ret = self.restart()
         if change_role and ret:
@@ -630,7 +627,6 @@ def promote(self):
         if ret:
             self.set_role('master')
             logger.info("cleared rewind flag after becoming the leader")
-            self._need_rewind = False
             self.call_nowait(ACTION_ON_ROLE_CHANGE)
         return ret
 


Test patch:
diff --git a/tests/test_ha.py b/tests/test_ha.py
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -91,6 +91,7 @@ def setUp(self):
                                  'data_dir': 'data/postgresql0', 'superuser': {}, 'admin': {},
                                  'replication': {'username': '', 'password': '', 'network': ''}})
             self.p.set_state('running')
+            self.p.set_role('replica')
             self.p.check_replication_lag = true
             self.p.can_create_replica_without_replication_connection = MagicMock(return_value=False)
             self.e = Etcd('foo', {'ttl': 30, 'host': 'ok:2379', 'scope': 'test'})
diff --git a/tests/test_postgresql.py b/tests/test_postgresql.py
--- a/tests/test_postgresql.py
+++ b/tests/test_postgresql.py
@@ -257,12 +257,10 @@ def test_follow(self, mock_pg_rewind):
         self.p.follow(Leader(-1, 28, self.other))
         self.p.rewind = mock_pg_rewind
         self.p.follow(self.leader)
-        self.p.require_rewind()
         with mock.patch('os.path.islink', MagicMock(return_value=True)):
             with mock.patch('patroni.postgresql.Postgresql.can_rewind', new_callable=PropertyMock(return_value=True)):
                 with mock.patch('os.unlink', MagicMock(return_value=True)):
                     self.p.follow(self.leader, recovery=True)
-        self.p.require_rewind()
         with mock.patch('patroni.postgresql.Postgresql.can_rewind', new_callable=PropertyMock(return_value=True)):
             self.p.rewind.return_value = True
             self.p.follow(self.leader, recovery=True)