Repository: zalando/patroni

Issue:
Unexpected error from Kubernetes API
I recently observed the following log. I didn't notice any negative effect, but the exception and stack trace in the log indicates that there's some check missing somewhere in the Patroni implementation. I don't know exactly how to reproduce, but it was during a rolling upgrade triggered by the Zalando postgres-operator due to some change in the PostgreSQL cluster manifest.

Patroni version was 2.1.3 or 2.1.2, not sure anymore.

```
nextcloud-clone2-db-0 postgres 2022-05-02 12:20:46,555 INFO: Lock owner: nextcloud-clone2-db-0; I am nextcloud-clone2-db-0
nextcloud-clone2-db-0 postgres 2022-05-02 12:20:46,763 INFO: manual failover: demoting myself
nextcloud-clone2-db-0 postgres 2022-05-02 12:20:46,765 INFO: Demoting self (graceful)
nextcloud-clone2-db-0 postgres 2022-05-02 12:20:47,935 INFO: Got response from nextcloud-clone2-db-1 http://10.100.34.110:8008/patroni: {"state": "running", "postmaster_start_time": "2022-05-02 12:19:50.980308+00:00", "role": "replica", "server_version": 120010, "xlog": {"received_location": 4731175072, "replayed_location": 4731175072, "replayed_timestamp": null, "paused": false}, "timeline": 7, "dcs_last_seen": 1651494046, "database_system_identifier": "7053754064532697170", "patroni": {"version": "2.1.3", "scope": "nextcloud-clone2-db"}}
nextcloud-clone2-db-0 postgres 2022-05-02 12:20:48,076 ERROR: Unexpected error from Kubernetes API
nextcloud-clone2-db-0 postgres Traceback (most recent call last):
nextcloud-clone2-db-0 postgres   File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper
nextcloud-clone2-db-0 postgres     return func(*args, **kwargs)
nextcloud-clone2-db-0 postgres   File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 877, in patch_or_create
nextcloud-clone2-db-0 postgres     return self._patch_or_create(name, annotations, resource_version, patch, retry, ips)
nextcloud-clone2-db-0 postgres   File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 868, in _patch_or_create
nextcloud-clone2-db-0 postgres     ret = retry(func, self._namespace, body) if retry else func(self._namespace, body)
nextcloud-clone2-db-0 postgres   File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 468, in wrapper
nextcloud-clone2-db-0 postgres     return getattr(self._core_v1_api, func)(*args, **kwargs)
nextcloud-clone2-db-0 postgres   File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 404, in wrapper
nextcloud-clone2-db-0 postgres     return self._api_client.call_api(method, path, headers, body, **kwargs)
nextcloud-clone2-db-0 postgres   File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 373, in call_api
nextcloud-clone2-db-0 postgres     return self._handle_server_response(response, _preload_content)
nextcloud-clone2-db-0 postgres   File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 203, in _handle_server_response
nextcloud-clone2-db-0 postgres     raise k8s_client.rest.ApiException(http_resp=response)
nextcloud-clone2-db-0 postgres patroni.dcs.kubernetes.K8sClient.rest.ApiException: (422)
nextcloud-clone2-db-0 postgres Reason: Unprocessable Entity
nextcloud-clone2-db-0 postgres HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 02 May 2022 12:20:48 GMT', 'Content-Length': '764'})
nextcloud-clone2-db-0 postgres HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":" \\"\\" is invalid: patch: Invalid value: \\"map[metadata:map[annotations:map[leader:\\u003cnil\\u003e optime:4714398200] labels:map[application:spilo cluster-name:nextcloud-clone2-db] name:nextcloud-clone2-db namespace:nextcloud resourceVersion:766376796] subsets:[]]\\": cannot convert int64 to string","reason":"Invalid","details":{"causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \\"map[metadata:map[annotations:map[leader:\\u003cnil\\u003e optime:4714398200] labels:map[application:spilo cluster-name:nextcloud-clone2-db] name:nextcloud-clone2-db namespace:nextcloud resourceVersion:766376796] subsets:[]]\\": cannot convert int64 to string","field":"patch"}]},"code":422}\n'
nextcloud-clone2-db-0 postgres 
nextcloud-clone2-db-0 postgres 2022-05-02 12:20:48,136 INFO: Leader key released
nextcloud-clone2-db-0 postgres 2022-05-02 12:20:55,007 INFO: closed patroni connection to the postgresql cluster
```



Discussion:
If it is not possible to reproduce it reliably there isn't much we can do about it.
Judging from `optime:4714398200` and `cannot convert int64 to string` in the log, could it possible that Patroni tried to set the annotation `optime: 4714398200` (on the Endpoints resource) with an integer value instead of string?

Resolving patch:
diff --git a/patroni/dcs/kubernetes.py b/patroni/dcs/kubernetes.py
--- a/patroni/dcs/kubernetes.py
+++ b/patroni/dcs/kubernetes.py
@@ -1060,7 +1060,7 @@ def delete_leader(self, last_lsn=None):
         if kind and (kind.metadata.annotations or {}).get(self._LEADER) == self._name:
             annotations = {self._LEADER: None}
             if last_lsn:
-                annotations[self._OPTIME] = last_lsn
+                annotations[self._OPTIME] = str(last_lsn)
             self.patch_or_create(self.leader_path, annotations, kind.metadata.resource_version, True, False, [])
             self.reset_cluster()
 


Test patch: