Repository: zalando/patroni

Issue:
Hot certificate change (reload patroni?)
Hello,
I'm securing the rest api with TLS+auth for my lovely patroni 2.0.2 (Debian 10 with python 3.7.3).

Is there a way to reload patroni in order to switch the certificate? I tried:
```
systemctl reload patroni
patronictl -c /etc/patroni/ reload cluster_name node_name
patronictl -c /etc/patroni/ restart cluster_name node_name
``` 
with no effect the certificate being used. I'm checking the cert with
```
openssl s_client -connect node_name.domain.com:8008 </dev/null 2>/dev/null | openssl x509 -noout -text | grep -E 'Subject:|DNS:'
```

The only thing that made it change is a violent
```
systemctl restart patroni
```
which makes the cluster to switch.

Is there another way to reload the certificate? I may try to provide a PR for this if it's not supported yet.

Thanks!


Discussion:
Hi @michael-todorovic,

no, it is not supported yet.

`patronictl restart` - in fact restarts only postgres and doesn't affect Patroni at all.
`patronictl reload` - forces Patroni to re-read config file. Some of the configuration settings are applied on the fly. For example, you can change `listen`, `connect_address`, `authentication`, and so on.

Actually, even if you change the `certfile`, `keyfile`, or something similar, it will apply it on the fly (after reload).

What is not done yet - Patroni doesn't check that the content of these files has been changed. Although, probably we can even make it simpler. It would be enough to change the condition https://github.com/zalando/patroni/blob/master/patroni/api.py#L766 to:
```python
-        if self.__listen != config['listen'] or self.__ssl_options != ssl_options:
+        if self.__listen != config['listen'] or ssl_options:
```

Feel free to open a PR.
Ok, I was only changing file content. If I change the filename, `patroni reload` works just fine.

I'll check the file content change and submit a PR

Thanks for your quick answer!

Resolving patch:
diff --git a/patroni/__init__.py b/patroni/__init__.py
--- a/patroni/__init__.py
+++ b/patroni/__init__.py
@@ -74,6 +74,7 @@ def reload_config(self, sighup=False, local=False):
             if local:
                 self.tags = self.get_tags()
                 self.request.reload_config(self.config)
+            if local or self.api.reload_local_certificate():
                 self.api.reload_config(self.config['restapi'])
             self.watchdog.reload_config(self.config)
             self.postgresql.reload_config(self.config['postgresql'], sighup)
diff --git a/patroni/api.py b/patroni/api.py
--- a/patroni/api.py
+++ b/patroni/api.py
@@ -616,6 +616,8 @@ def __init__(self, patroni, config):
         self.http_extra_headers = {}
         self.reload_config(config)
         self.daemon = True
+        self.__ssl_serial_number = None
+        self._received_new_cert = False
 
     def query(self, sql, *params):
         cursor = None
@@ -701,6 +703,7 @@ def __initialize(self, listen, ssl_options):
 
         self.__listen = listen
         self.__ssl_options = ssl_options
+        self._received_new_cert = False  # reset to False after reload_config()
 
         self.__httpserver_init(host, port)
         Thread.__init__(self, target=self.serve_forever)
@@ -723,6 +726,7 @@ def __initialize(self, listen, ssl_options):
                     ctx.verify_mode = modes[verify_client]
                 else:
                     logger.error('Bad value in the "restapi.verify_client": %s', verify_client)
+            self.__ssl_serial_number = self.get_certificate_serial_number()
             self.socket = ctx.wrap_socket(self.socket, server_side=True)
         if reloading_config:
             self.start()
@@ -750,6 +754,23 @@ def shutdown_request(self, request):
             _, request = request  # SSLSocket
         return super(RestApiServer, self).shutdown_request(request)
 
+    def get_certificate_serial_number(self):
+        if self.__ssl_options.get('certfile'):
+            import ssl
+            try:
+                crt = ssl._ssl._test_decode_cert(self.__ssl_options['certfile'])
+                return crt.get('serialNumber')
+            except ssl.SSLError as e:
+                logger.error('Failed to get serial number from certificate %s: %r', self.__ssl_options['certfile'], e)
+
+    def reload_local_certificate(self):
+        if self.__protocol == 'https':
+            on_disk_cert_serial_number = self.get_certificate_serial_number()
+            if on_disk_cert_serial_number != self.__ssl_serial_number:
+                self._received_new_cert = True
+                self.__ssl_serial_number = on_disk_cert_serial_number
+                return True
+
     def reload_config(self, config):
         if 'listen' not in config:  # changing config in runtime
             raise ValueError('Can not find "restapi.listen" config')
@@ -763,7 +784,7 @@ def reload_config(self, config):
         if isinstance(config.get('verify_client'), six.string_types):
             ssl_options['verify_client'] = config['verify_client'].lower()
 
-        if self.__listen != config['listen'] or self.__ssl_options != ssl_options:
+        if self.__listen != config['listen'] or self.__ssl_options != ssl_options or self._received_new_cert:
             self.__initialize(config['listen'], ssl_options)
 
         self.__auth_key = base64.b64encode(config['auth'].encode('utf-8')) if 'auth' in config else None


Test patch: