Repository: zalando/patroni

Issue:
Feature request: make value of replication role labels configurable
**Describe the feature**
Currently, on K8s the value of the labels that identify the replication role of each patroni/postgresql Pod is hardcoded to be "master" for the leader and "replica" for the followers.

I would like to make those values configurable.

That allows us to stop using the non-inclusive "master" term and replace it with something like "leader" or primary.

I would also like to opt out from using "replica". The reason is that I'm running patroni/spilo in a context where "replica" is already widely used to describe something different than postgresql's replication replicas. So currently there's a name overloading that causes confusion.

If there's consensus and someone can work on it it'd be great. I might also do this, but I'm not a python expert (at all) and have never touched patroni's codebase so I'd need guidance.


Discussion:
Getting rif from non-inclusive terms is in our radar.
The thing is that the problem is much deeper than just labels.
There are plenty of internal checks on the "master" and we can't rename it without doing an intermediate release (compatible between old and new terms).

Even with labels it is much harder than you probably anticipate. If Patroni is managing the endpoint on it's own - everything will be fine. If Patroni is using configmaps and there is a Service with label selector, then they could not be migrated to the new scheme without a downtime, because set-based requirements aren't supported for Services: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements
@CyberDem0n thank you for your reply. I now have a better understanding of the problem and its caveats.

Strictly speaking my change request is not for Patroni to replace the non-inclusive terms. It's just to make the values of the replication role label for both master and replica pods configurable. This could be implemented in a backward-compatible way with a change that:

1. adds two env variables/config parameters that determine the value of the replication role label for leader and followers, respecitvely (e.g. `PATRONI_KUBERNETES_LEADER_LABEL_VALUE` and `PATRONI_KUBERNETES_FOLLOWER_LABEL_VALUE`
2. keeps the defaults for the two aforementioned env vars/config parameters to the legacy values (`master` and `replica`); this is what ensures backward compatibility
3. makes patroni's code use the value of the aforementioned config parameters rather than the hardcoded current values.

For my use case this would be already enough.

The change I described could be the intermediate non-breaking release you mentioned. Its release notes could advertise that in a few releases the defaults would be changed to the inclusive terms, so users could get ready.

> If Patroni is using configmaps and there is a Service with label selector, then they could not be migrated to the new scheme without a downtime, because set-based requirements aren't supported for Services

A couple of random thoughts on this (not sure whether they make sense):
1. No matter how this change is done, it would involve updating patroni's version. It would mean that all the pg pods would be restarted (likely 1 at a time if they run a statefulset). Given this, isn't there some downtime anyway, from when the old master goes down to when a new one is elected? I suspect this downtime would be comparable in duration to the one caused by updating the primary pod label and the service selector non-atomically (granted, this is an unproven claim though).
2. I'm thinking about a fairly convoluted way to update both the pod label and the service selector to minimize downtime:
a. t0: add to the leader pod a temporary master label with a different key while also keeping the old one (i.e. the pod would have both labels at this point in time, `role: master`and `tmp-role: leader`
b. t1: update the service selector to select the leader only via `tmp-role: leader`
c. t2: update the leader pod by updating the usual master label's value to the new inclusive value (after this step the leader has labels `role: leader` and `tmp-role: leader`
d. t3: update the service selector to select only on `role: leader`
e. t4: update the leader pod by removing the temporary label `tmp-role`.

It might take too long but there's also the option to wait for K8s to add set-based selectors to services, a K8s enhancement proposal should be filed at some point: https://github.com/kubernetes/kubernetes/issues/48528#issuecomment-1256458873 

> isn't there some downtime anyway, from when the old master goes down to when a new one is elected?

I wouldn't call a switchover a "real" downtime. Typically it takes just a couple of seconds. If someone screwed up with the label selector it is a toally different story. Applications will not be able to connect for a much longer time, until someone fixes the service.

> I'm thinking about a fairly convoluted way to update both the pod label and the service selector to minimize downtime

Yes, I was thinking exactly about the same process.

> It might take too long but there's also the option to wait for K8s to add set-based selectors to services, a K8s enhancement proposal should be filed at some point: https://github.com/kubernetes/kubernetes/issues/48528#issuecomment-1256458873

Taking into account that the issue was open more than 5 years ago...

> Strictly speaking my change request is not for Patroni to replace the non-inclusive terms. It's just to make the values of the replication role label for both master and replica pods configurable. This could be implemented in a backward-compatible way with a change that:

Yes, it will make life easier for people not wanting to go through the complex-multi-step migration process. Instead they could just opt in using old terms.

> I might also do this, but I'm not a python expert (at all) and have never touched patroni's codebase so I'd need guidance.

It shoudn't be hard. First, there is a short doc describing how to setup dev environment and run test: https://patroni.readthedocs.io/en/latest/CONTRIBUTING.html
After that you'll have to touch only two files: [config.py](https://github.com/zalando/patroni/blob/e5d750e9b8f7fc6a4172932056eddeb3fc892e7f/patroni/config.py#L351-L357) and [kubernetes.py](https://github.com/zalando/patroni/blob/master/patroni/dcs/kubernetes.py)
I will begin implementing the proposed change 2 weeks from now (unless you want to go ahead before that).
Really sorry for the long delay and silence, I've been busier than anticipated with other work. I'm starting on this now.
@matteoolivi is there any progress?
Maybe you need some help?
I had started to look into this, then got pulled into a different project and completely forgot to provide updates on this.

Really really sorry for the wast of time and lack of professionalism.

Unfortunately I won't have time to do this for a long time - whoever can take over this should do that.
@CyberDem0n I can help with this, It just so happens that my colleague also asked for this feature this week
Dear Patroni Team. Polite ping: any progress here/ETA?

Resolving patch:
diff --git a/patroni/config.py b/patroni/config.py
--- a/patroni/config.py
+++ b/patroni/config.py
@@ -475,7 +475,8 @@ def _get_auth(name: str, params: Optional[Collection[str]] = None) -> Dict[str,
                               'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'SERVICE_CHECK_TLS_SERVER_NAME',
                               'SERVICE_TAGS', 'NAMESPACE', 'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL',
                               'POD_IP', 'PORTS', 'LABELS', 'BYPASS_API_SERVICE', 'RETRIABLE_HTTP_CODES', 'KEY_PASSWORD',
-                              'USE_SSL', 'SET_ACLS', 'GROUP', 'DATABASE') and name:
+                              'USE_SSL', 'SET_ACLS', 'GROUP', 'DATABASE', 'LEADER_LABEL_VALUE', 'FOLLOWER_LABEL_VALUE',
+                              'STANDBY_LEADER_LABEL_VALUE', 'TMP_ROLE_LABEL') and name:
                     value = os.environ.pop(param)
                     if name == 'CITUS':
                         if suffix == 'GROUP':
diff --git a/patroni/dcs/kubernetes.py b/patroni/dcs/kubernetes.py
--- a/patroni/dcs/kubernetes.py
+++ b/patroni/dcs/kubernetes.py
@@ -752,6 +752,10 @@ def __init__(self, config: Dict[str, Any]) -> None:
         self._label_selector = ','.join('{0}={1}'.format(k, v) for k, v in self._labels.items())
         self._namespace = config.get('namespace') or 'default'
         self._role_label = config.get('role_label', 'role')
+        self._leader_label_value = config.get('leader_label_value', 'master')
+        self._follower_label_value = config.get('follower_label_value', 'replica')
+        self._standby_leader_label_value = config.get('standby_leader_label_value', 'standby-leader')
+        self._tmp_role_label = config.get('tmp_role_label')
         self._ca_certs = os.environ.get('PATRONI_KUBERNETES_CACERT', config.get('cacert')) or SERVICE_CERT_FILENAME
         super(Kubernetes, self).__init__({**config, 'namespace': ''})
         if self._citus_group:
@@ -1263,19 +1267,30 @@ def set_config_value(self, value: str, version: Optional[str] = None) -> bool:
     def touch_member(self, data: Dict[str, Any]) -> bool:
         cluster = self.cluster
         if cluster and cluster.leader and cluster.leader.name == self._name:
-            role = 'master'
+            role = self._leader_label_value
+            tmp_role = 'master'
         elif data['state'] == 'running' and data['role'] not in ('master', 'primary'):
-            role = data['role']
+            role = {
+                'replica': self._follower_label_value,
+                'standby-leader': self._standby_leader_label_value,
+            }.get(data['role'], data['role'])
+            tmp_role = data['role']
         else:
             role = None
+            tmp_role = None
+
+        role_labels = {self._role_label: role}
+        if self._tmp_role_label:
+            role_labels[self._tmp_role_label] = tmp_role
 
         member = cluster and cluster.get_member(self._name, fallback_to_leader=False)
         pod_labels = member and member.data.pop('pod_labels', None)
         ret = member and pod_labels is not None\
-            and pod_labels.get(self._role_label) == role and deep_compare(data, member.data)
+            and all(pod_labels.get(k) == v for k, v in role_labels.items())\
+            and deep_compare(data, member.data)
 
         if not ret:
-            metadata = {'namespace': self._namespace, 'name': self._name, 'labels': {self._role_label: role},
+            metadata = {'namespace': self._namespace, 'name': self._name, 'labels': role_labels,
                         'annotations': {'status': json.dumps(data, separators=(',', ':'))}}
             body = k8s_client.V1Pod(metadata=k8s_client.V1ObjectMeta(**metadata))
             ret = self._api.patch_namespaced_pod(self._name, self._namespace, body)
diff --git a/patroni/validator.py b/patroni/validator.py
--- a/patroni/validator.py
+++ b/patroni/validator.py
@@ -995,6 +995,10 @@ def validate_watchdog_mode(value: Any) -> None:
             Optional("namespace"): str,
             Optional("scope_label"): str,
             Optional("role_label"): str,
+            Optional("leader_label_value"): str,
+            Optional("follower_label_value"): str,
+            Optional("standby_leader_label_value"): str,
+            Optional("tmp_role_label"): str,
             Optional("use_endpoints"): bool,
             Optional("pod_ip"): Or(is_ipv4_address, is_ipv6_address),
             Optional("ports"): [{"name": str, "port": int}],


Test patch:
diff --git a/tests/test_kubernetes.py b/tests/test_kubernetes.py
--- a/tests/test_kubernetes.py
+++ b/tests/test_kubernetes.py
@@ -298,6 +298,28 @@ def test_touch_member(self, mock_patch_namespaced_pod):
         self.k.touch_member({'state': 'running', 'role': 'replica'})
         self.k.touch_member({'state': 'stopped', 'role': 'primary'})
 
+        self.k._role_label = 'isMaster'
+        self.k._leader_label_value = 'true'
+        self.k._follower_label_value = 'false'
+        self.k._standby_leader_label_value = 'false'
+        self.k._tmp_role_label = 'tmp_role'
+
+        self.k.touch_member({'state': 'running', 'role': 'replica'})
+        mock_patch_namespaced_pod.assert_called()
+        self.assertEqual(mock_patch_namespaced_pod.call_args.args[2].metadata.labels['isMaster'], 'false')
+        self.assertEqual(mock_patch_namespaced_pod.call_args.args[2].metadata.labels['tmp_role'], 'replica')
+
+        self.k.touch_member({'state': 'running', 'role': 'standby-leader'})
+        mock_patch_namespaced_pod.assert_called()
+        self.assertEqual(mock_patch_namespaced_pod.call_args.args[2].metadata.labels['isMaster'], 'false')
+        self.assertEqual(mock_patch_namespaced_pod.call_args.args[2].metadata.labels['tmp_role'], 'standby-leader')
+
+        self.k._name = 'p-0'
+        self.k.touch_member({'role': 'primary'})
+        mock_patch_namespaced_pod.assert_called()
+        self.assertEqual(mock_patch_namespaced_pod.call_args.args[2].metadata.labels['isMaster'], 'true')
+        self.assertEqual(mock_patch_namespaced_pod.call_args.args[2].metadata.labels['tmp_role'], 'master')
+
     def test_initialize(self):
         self.k.initialize()