Repository: zalando/patroni

Issue:
Patroni does not perform failover if master lost data directory
We are migrating from 1.2 to 1.3.4 and one of our tests fails - we removed data directory and are expecting failover to replica and master reinitialization. But with new version master detects empty dir, releases key and .. wait for leader to bootstrap. Meanwhile replica is considering master as "master" and wait for master too. 
From master: 

> 2017-10-11 11:35:27,431 ERROR: get_postgresql_status
> Traceback (most recent call last):
>   File "/patroni/patroni/api.py", line 405, in get_postgresql_status
>     retry=retry)[0]
>   File "/patroni/patroni/api.py", line 381, in query
>     return retry(self.server.query, sql, *params)
>   File "/patroni/patroni/utils.py", line 259, in __call__
>     return func(*args, **kwargs)
>   File "/patroni/patroni/api.py", line 452, in query
>     raise e
> OperationalError: could not open file "base/12404/2600": No such file or directory
> 
> 2017-10-11 11:35:27,433 INFO: released leader key voluntarily as data dir empty and currently leader
> ...
> 2017-10-11 11:35:37,416 INFO: Lock owner: None; I am node1
> 2017-10-11 11:35:37,420 INFO: waiting for leader to bootstrap

From slave:

> 2017-10-11 14:20:50,223 INFO: Got response from node1 http://10.129.1.63:8008/patroni: {"state": "unknown", "database_system_identifier": "6475561736777527351", "role": "master", "patroni": {"scope": "common", "version": "1.3.4"}}
> 2017-10-11 14:20:50,262 WARNING: Master (node1) is still alive
> 2017-10-11 14:20:50,267 INFO: following a different leader because i am not the healthiest node

I suppose that replica cannot promote itself because broken master declares wrong status. Broken master declares wrong status because it released key but never update `postgresql.role` value.


Discussion:
Good catch, will fix it tomorrow.

Resolving patch:
diff --git a/patroni/ha.py b/patroni/ha.py
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -1014,6 +1014,7 @@ def _run_cycle(self):
 
             # is data directory empty?
             if self.state_handler.data_directory_empty():
+                self.state_handler.set_role('uninitialized')
                 # In case datadir went away while we were master. TODO: check for this and try to stop postgresql.
                 self.watchdog.disable()
 
diff --git a/patroni/version.py b/patroni/version.py
--- a/patroni/version.py
+++ b/patroni/version.py
@@ -1 +1 @@
-__version__ = '1.3.4'
+__version__ = '1.3.5'


Test patch:
diff --git a/tests/test_ha.py b/tests/test_ha.py
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -862,6 +862,7 @@ def test_leader_with_empty_directory(self):
         self.ha.has_lock = true
         self.p.data_directory_empty = true
         self.assertEquals(self.ha.run_cycle(), 'released leader key voluntarily as data dir empty and currently leader')
+        self.assertEquals(self.p.role, 'uninitialized')
 
         # as has_lock is mocked out, we need to fake the leader key release
         self.ha.has_lock = false
diff --git a/tests/test_postgresql.py b/tests/test_postgresql.py
--- a/tests/test_postgresql.py
+++ b/tests/test_postgresql.py
@@ -905,6 +905,7 @@ def test_single_user_mode(self, subprocess_popen_mock):
         self.assertEquals(self.p.single_user_mode(command="CHECKPOINT"), 0)
         subprocess_popen_mock.return_value = None
         self.assertEquals(self.p.single_user_mode(), 1)
+        self.assertEquals(self.p.single_user_mode(options={'archive_mode': 'on'}), 1)
 
     @patch('os.listdir', Mock(side_effect=[OSError, ['a', 'b']]))
     @patch('os.unlink', Mock(side_effect=OSError))