Repository: zalando/patroni

Issue:
Issue when using etcd and HTTPS
PostgreSQL version : 10.6
Patroni version : 1.5.4
etcd version : 3.2.22

I have an issue using Patroni with etcd as DCS and HTTPS to reach etcd. Patroni is configured with both environment for some dynamic things and configuration file. 

Here is the environment:

```sh
/# env | grep PATRONI_ETCD
PATRONI_ETCD_CERT=/etc/ssl/etcd_db_cluster_mgmt/etcd_clients.crt
PATRONI_ETCD_CACERT=/etc/ssl/etcd_db_cluster_mgmt/ca.crt
PATRONI_ETCD_KEY=/etc/ssl/etcd_db_cluster_mgmt/etcd_clients_private.key
PATRONI_ETCD_PROTOCOL=https
PATRONI_ETCD_HOSTS='10.1.0.103:2381','10.2.0.105:2381','10.1.0.104:2381'
PATRONI_ETCD_USERNAME=XXX
PATRONI_ETCD_PASSWORD=YYY
```

Here is the configuration file part about DCS:

```yml
bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    postgresql:
      use_pg_rewind: true
```

When I start Patroni, I have the following in my logs:

```
2019-01-24 09:52:52.176991641 +0100 CET [postgresql-2] 2019-01-24 08:52:52,175 INFO: Failed to import patroni.dcs.consul
2019-01-24 09:52:52.217419769 +0100 CET [postgresql-2] 2019-01-24 08:52:52,216 INFO: Selected new etcd server http://10.1.0.104:2381
2019-01-24 09:52:52.438946548 +0100 CET [postgresql-2] 2019-01-24 08:52:52,438 WARNING: etcd response did not contain a cluster ID
2019-01-24 09:52:52.442648840 +0100 CET [postgresql-2] 2019-01-24 08:52:52,438 ERROR: get_cluster
2019-01-24 09:52:52.442674079 +0100 CET [postgresql-2] Traceback (most recent call last):
2019-01-24 09:52:52.442690076 +0100 CET [postgresql-2] File "/usr/local/lib/python2.7/dist-packages/patroni/dcs/etcd.py", line 450, in _load_cluster
2019-01-24 09:52:52.442692989 +0100 CET [postgresql-2] result = self.retry(self._client.read, self.client_path(''), recursive=True)
2019-01-24 09:52:52.442757389 +0100 CET [postgresql-2] File "/usr/local/lib/python2.7/dist-packages/patroni/dcs/etcd.py", line 325, in retry
2019-01-24 09:52:52.442760518 +0100 CET [postgresql-2] return self._retry.copy()(*args, **kwargs)
2019-01-24 09:52:52.443041781 +0100 CET [postgresql-2] File "/usr/local/lib/python2.7/dist-packages/patroni/utils.py", line 254, in __call__
2019-01-24 09:52:52.443043718 +0100 CET [postgresql-2] return func(*args, **kwargs)
2019-01-24 09:52:52.443045878 +0100 CET [postgresql-2] return self._result_from_response(response)
2019-01-24 09:52:52.443044670 +0100 CET [postgresql-2] File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 598, in read
2019-01-24 09:52:52.443046840 +0100 CET [postgresql-2] File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 812, in _result_from_response
2019-01-24 09:52:52.443048048 +0100 CET [postgresql-2] 'Server response was not valid JSON: %r' % e)
2019-01-24 09:52:52.443048962 +0100 CET [postgresql-2] EtcdException: Server response was not valid JSON: ValueError('No JSON object could be decoded',)
2019-01-24 09:52:52.443505732 +0100 CET [postgresql-2] 2019-01-24 08:52:52,443 WARNING: Can not get cluster from dcs
[...]
2019-01-24 09:52:57.655428574 +0100 CET [postgresql-2] 2019-01-24 08:52:57,654 WARNING: etcd response did not contain a cluster ID
2019-01-24 09:52:57.655615414 +0100 CET [postgresql-2] 2019-01-24 08:52:57,654 ERROR: Server response was not valid JSON: ValueError('No JSON object could be decoded',)
2019-01-24 09:52:57.655617894 +0100 CET [postgresql-2] 2019-01-24 08:52:57,655 WARNING: Can not get cluster from dcs
[...]
2019-01-24 09:53:02.871336218 +0100 CET [postgresql-2] 2019-01-24 08:53:02,870 WARNING: etcd response did not contain a cluster ID
2019-01-24 09:53:02.871997748 +0100 CET [postgresql-2] 2019-01-24 08:53:02,871 ERROR: Server response was not valid JSON: ValueError('No JSON object could be decoded',)
2019-01-24 09:53:02.872999967 +0100 CET [postgresql-2] 2019-01-24 08:53:02,872 WARNING: Can not get cluster from dcs
[...]
```

The last bloc of three lines show up in a loop. 

We can see the log line:

```
2019-01-24 09:52:52.217419769 +0100 CET [postgresql-2] 2019-01-24 08:52:52,216 INFO: Selected new etcd server http://10.1.0.104:2381
```

Display `http` whereas it should be `https`. I assume that the following errors are just due to this but I might be wrong. Am I doing something wrong here? I tried to infer something from the source code but not sure. I don't see why this line (https://github.com/zalando/patroni/blob/v1.5.4/patroni/dcs/etcd.py#L365) would not return `https` as `PATRONI_ETCD_PROTOCOL` is defined.




Discussion:
1. Do you really run etcd on the port 2381? It doesn't look like the standard client port, which is 2379.
2. Specifying of protocol via ENV is not [supported](https://github.com/zalando/patroni/blob/master/docs/ENVIRONMENT.rst#etcd), but it could be easily fixed by adding 'PROTOCOL' [here](https://github.com/zalando/patroni/blob/master/patroni/config.py#L291)
1. Yes it is the correct port. We have another etcd on our infrastructure for other purpose so we changed the port for this one.
2. Oh OK. I thought that any configuration of the YML file have an equivalent in environment configuration hence I thought about a missing piece in the documentation. I will open a PR to add the possibility to define the protocol if it's OK for you?

Resolving patch:
diff --git a/patroni/config.py b/patroni/config.py
--- a/patroni/config.py
+++ b/patroni/config.py
@@ -288,9 +288,10 @@ def _parse_list(value):
                 name, suffix = (param[8:].split('_', 1) + [''])[:2]
                 if name and suffix:
                     # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
-                    if suffix in ('HOST', 'HOSTS', 'PORT', 'SRV', 'URL', 'PROXY', 'CACERT', 'CERT', 'KEY', 'VERIFY',
-                                  'TOKEN', 'CHECKS', 'DC', 'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'NAMESPACE',
-                                  'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP', 'PORTS', 'LABELS'):
+                    if suffix in ('HOST', 'HOSTS', 'PORT', 'PROTOCOL', 'SRV', 'URL', 'PROXY', 'CACERT', 'CERT',
+                                  'KEY', 'USERNAME', 'PASSWORD', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'REGISTER_SERVICE',
+                                  'SERVICE_CHECK_INTERVAL', 'NAMESPACE', 'CONTEXT', 'USE_ENDPOINTS', 'SCOPE_LABEL',
+                                  'ROLE_LABEL', 'POD_IP', 'PORTS', 'LABELS'):
                         value = os.environ.pop(param)
                         if suffix == 'PORT':
                             value = value and parse_int(value)


Test patch: