Repository: zalando/patroni

Issue:
No election after loss of leader
I am trying to get patroni setup in Kubernetes (well minikube right now) by [adapting](https://github.com/abkfenris/atomicdb) @jberkus [atomicbd](https://github.com/jberkus/atomicdb/tree/master/patroni_petset).

When I kill the leader pod it will restart fast enough that it reacquires the leader key so their is no failover. It then will try to bootstrap from itself.

```sh
2017-03-24 13:11:11,066 INFO: Selected new etcd server http://172.17.0.6:2379
2017-03-24 13:11:11,066 INFO: Starting new HTTP connection (1): 172.17.0.6
2017-03-24 13:11:11,087 INFO: trying to bootstrap from leader 'patroni_0'
2017-03-24 13:11:11,089 INFO: Lock owner: patroni_0; I am patroni_0
2017-03-24 13:11:11,096 INFO: updated leader lock during bootstrap from leader 'patroni_0'
pg_basebackup: could not connect to server: could not translate host name "patroni-0.patroni" to address: Name or service not known
2017-03-24 13:11:11,129 ERROR: Error when fetching backup: pg_basebackup exited with code=1
2017-03-24 13:11:11,130 WARNING: Trying again in 5 seconds
2017-03-24 13:11:16,090 INFO: Lock owner: patroni_0; I am patroni_0
2017-03-24 13:11:16,097 INFO: updated leader lock during bootstrap from leader 'patroni_0'
pg_basebackup: could not connect to server: could not connect to server: Connection refused
	Is the server running on host "patroni-0.patroni" (172.17.0.7) and accepting
	TCP/IP connections on port 5432?
2017-03-24 13:11:16,146 ERROR: Error when fetching backup: pg_basebackup exited with code=1
2017-03-24 13:11:16,146 ERROR: failed to bootstrap from leader 'patroni_0'
```

The other two pods don't realize that the leader is trying to bootstrap and try to keep following it rather than having an election.

```sh
[patroni-2] 2017-03-24 13:23:56,100 INFO: Lock owner: patroni_0; I am patroni_2
[patroni-2] 2017-03-24 13:23:56,100 INFO: does not have lock
[patroni-1] 2017-03-24 13:23:56,097 INFO: Lock owner: patroni_0; I am patroni_1
[patroni-1] 2017-03-24 13:23:56,098 INFO: does not have lock
[patroni-1] 2017-03-24 13:23:56,108 INFO: no action.  i am a secondary and i am following a leader
[patroni-2] 2017-03-24 13:23:56,110 INFO: no action.  i am a secondary and i am following a leader
[patroni-2] FATAL:  could not connect to the primary server: could not connect to server: Connection refused
[patroni-2] Is the server running on host "patroni-0.patroni" (172.17.0.7) and accepting
[patroni-2] TCP/IP connections on port 5432?
[patroni-2]
[patroni-1] FATAL:  could not connect to the primary server: could not connect to server: Connection refused
[patroni-1] Is the server running on host "patroni-0.patroni" (172.17.0.7) and accepting
[patroni-1] TCP/IP connections on port 5432?
```

As far as I can tell, there is a method ([ha.release_leader_key_voluntarily](https://github.com/zalando/patroni/blob/8fda95780451029cee6b9dc3fdcf9f71ab4480a6/patroni/ha.py#L474)) for the leader to give up the key but it doesn't look like there is a check if a newly created instance is the leader and to release the key.


Discussion:
Yeah, the lock should be released if the master doesn't have data.  I'll look into it.

This hasn't come up previously because in a multi-machine configuration the original leader node never comes back that fast unless it's mounting the same storage (in which case it's still the master).  However, it should Do The Right Thing regardless.

Resolving patch:
diff --git a/patroni/ha.py b/patroni/ha.py
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -886,6 +886,11 @@ def _run_cycle(self):
 
             # is data directory empty?
             if self.state_handler.data_directory_empty():
+                # is this instance the leader?
+                if self.has_lock():
+                    self.release_leader_key_voluntarily()
+                    return 'released leader key voluntarily as data dir empty and currently leader'
+
                 return self.bootstrap()  # new node
             # "bootstrap", but data directory is not empty
             elif not self.sysid_valid(self.cluster.initialize) and self.cluster.is_unlocked() and not self.is_paused():


Test patch:
diff --git a/tests/test_ha.py b/tests/test_ha.py
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -771,3 +771,14 @@ def test_watch(self):
 
     def test_wakup(self):
         self.ha.wakeup()
+
+    def test_leader_with_empty_directory(self):
+        self.ha.cluster = get_cluster_initialized_with_leader()
+        self.ha.has_lock = true
+        self.p.data_directory_empty = true
+        self.assertEquals(self.ha.run_cycle(), 'released leader key voluntarily as data dir empty and currently leader')
+
+        # as has_lock is mocked out, we need to fake the leader key release
+        self.ha.has_lock = false
+        # will not say bootstrap from leader as replica can't self elect
+        self.assertEquals(self.ha.run_cycle(), "trying to bootstrap from replica 'other'")