Repository: zalando/patroni (python)

Issue:
Add ETCD srv_name config parametter
Please consider adding ETCD SRV name config parameter to Patroni. 

As per ETCD documentation:

> The -discovery-srv-name flag additionally configures a suffix to the SRV name that is queried during discovery. Use this flag to differentiate between multiple etcd clusters under the same domain. For example, if discovery-srv=example.com and -discovery-srv-name=foo are set, the following DNS SRV queries are made:
> 
>     _etcd-server-ssl-foo._tcp.example.com
>     _etcd-server-foo._tcp.example.com

we use SRV name suffixes to differentiate ETCD clusters in our private domain (around 40k users) and there is no simple way how to point Patroni to a specific ETCD cluster using SRV. 



Pull request description:
Support for ETCD SRV name suffix
Add support for ETCD SRV name suffix as per description in ETCD dosc:

> The -discovery-srv-name flag additionally configures a suffix to the SRV name that is queried during discovery. Use this flag to differentiate between multiple etcd clusters under the same domain. For example, if discovery-srv=example.com and -discovery-srv-name=foo are set, the following DNS SRV queries are made:
> 
> _etcd-server-ssl-foo._tcp.example.com
> _etcd-server-foo._tcp.example.com

All test passes, but not been tested on the live ETCD system yet... Please, take a look and send feedback. 

Resolves #2028

Patch:
diff --git a/docs/ENVIRONMENT.rst b/docs/ENVIRONMENT.rst
index 7c837a6b..43a35921 100644
--- a/docs/ENVIRONMENT.rst
+++ b/docs/ENVIRONMENT.rst
@@ -59,7 +59,8 @@ Etcd
 -  **PATRONI\_ETCD\_USE\_PROXIES**: If this parameter is set to true, Patroni will consider **hosts** as a list of proxies and will not perform a topology discovery of etcd cluster but stick to a fixed list of **hosts**.
 -  **PATRONI\_ETCD\_PROTOCOL**: http or https, if not specified http is used. If the **url** or **proxy** is specified - will take protocol from them.
 -  **PATRONI\_ETCD\_HOST**: the host:port for the etcd endpoint.
--  **PATRONI\_ETCD\_SRV**: Domain to search the SRV record(s) for cluster autodiscovery.
+-  **PATRONI\_ETCD\_SRV**: Domain to search the SRV record(s) for cluster autodiscovery. Patroni will try to query these SRV service names for specified domain (in that order until first success): ``_etcd-client-ssl``, ``_etcd-client``, ``_etcd-ssl``, ``_etcd``, ``_etcd-server-ssl``, ``_etcd-server``. If SRV records for ``_etcd-server-ssl`` or ``_etcd-server`` are retrieved then ETCD peer protocol is used do query ETCD for available members. Otherwise hosts from SRV records will be used.
+-  **PATRONI\_ETCD\_SRV\_SUFFIX**: Configures a suffix to the SRV name that is queried during discovery. Use this flag to differentiate between multiple etcd clusters under the same domain. Works only with conjunction with **PATRONI\_ETCD\_SRV**. For example, if ``PATRONI_ETCD_SRV_SUFFIX=foo`` and ``PATRONI_ETCD_SRV=example.org`` are set, the following DNS SRV query is made:``_etcd-client-ssl-foo._tcp.example.com`` (and so on for every possible ETCD SRV service name).
 -  **PATRONI\_ETCD\_USERNAME**: username for etcd authentication.
 -  **PATRONI\_ETCD\_PASSWORD**: password for etcd authentication.
 -  **PATRONI\_ETCD\_CACERT**: The ca certificate. If present it will enable validation.
diff --git a/docs/SETTINGS.rst b/docs/SETTINGS.rst
index 111854aa..77f41c22 100644
--- a/docs/SETTINGS.rst
+++ b/docs/SETTINGS.rst
@@ -156,7 +156,8 @@ Most of the parameters are optional, but you have to specify one of the **host**
 -  **use\_proxies**: If this parameter is set to true, Patroni will consider **hosts** as a list of proxies and will not perform a topology discovery of etcd cluster.
 -  **url**: url for the etcd.
 -  **proxy**: proxy url for the etcd. If you are connecting to the etcd using proxy, use this parameter instead of **url**.
--  **srv**: Domain to search the SRV record(s) for cluster autodiscovery.
+-  **srv**: Domain to search the SRV record(s) for cluster autodiscovery. Patroni will try to query these SRV service names for specified domain (in that order until first success): ``_etcd-client-ssl``, ``_etcd-client``, ``_etcd-ssl``, ``_etcd``, ``_etcd-server-ssl``, ``_etcd-server``. If SRV records for ``_etcd-server-ssl`` or ``_etcd-server`` are retrieved then ETCD peer protocol is used do query ETCD for available members. Otherwise hosts from SRV records will be used.
+-  **srv\_suffix**: Configures a suffix to the SRV name that is queried during discovery. Use this flag to differentiate between multiple etcd clusters under the same domain. Works only with conjunction with **srv**. For example, if ``srv_suffix: foo`` and ``srv: example.org`` are set, the following DNS SRV query is made:``_etcd-client-ssl-foo._tcp.example.com`` (and so on for every possible ETCD SRV service name).
 -  **protocol**: (optional) http or https, if not specified http is used. If the **url** or **proxy** is specified - will take protocol from them.
 -  **username**: (optional) username for etcd authentication.
 -  **password**: (optional) password for etcd authentication.
diff --git a/patroni/config.py b/patroni/config.py
index 1308cc49..d007fa6c 100644
--- a/patroni/config.py
+++ b/patroni/config.py
@@ -347,7 +347,7 @@ class Config(object):
             if param.startswith(PATRONI_ENV_PREFIX):
                 # PATRONI_(ETCD|CONSUL|ZOOKEEPER|EXHIBITOR|...)_(HOSTS?|PORT|..)
                 name, suffix = (param[8:].split('_', 1) + [''])[:2]
-                if suffix in ('HOST', 'HOSTS', 'PORT', 'USE_PROXIES', 'PROTOCOL', 'SRV', 'URL', 'PROXY',
+                if suffix in ('HOST', 'HOSTS', 'PORT', 'USE_PROXIES', 'PROTOCOL', 'SRV', 'SRV_SUFFIX', 'URL', 'PROXY',
                               'CACERT', 'CERT', 'KEY', 'VERIFY', 'TOKEN', 'CHECKS', 'DC', 'CONSISTENCY',
                               'REGISTER_SERVICE', 'SERVICE_CHECK_INTERVAL', 'NAMESPACE', 'CONTEXT',
                               'USE_ENDPOINTS', 'SCOPE_LABEL', 'ROLE_LABEL', 'POD_IP', 'PORTS', 'LABELS',
diff --git a/patroni/dcs/etcd.py b/patroni/dcs/etcd.py
index 198dede2..aef0d2b2 100644
--- a/patroni/dcs/etcd.py
+++ b/patroni/dcs/etcd.py
@@ -283,13 +283,14 @@ class AbstractEtcdClientWithFailover(etcd.Client):
         except DNSException:
             return []
 
-    def _get_machines_cache_from_srv(self, srv):
+    def _get_machines_cache_from_srv(self, srv, srv_suffix=None):
         """Fetch list of etcd-cluster member by resolving _etcd-server._tcp. SRV record.
         This record should contain list of host and peer ports which could be used to run
         'GET http://{host}:{port}/members' request (peer protocol)"""
 
         ret = []
         for r in ['-client-ssl', '-client', '-ssl', '', '-server-ssl', '-server']:
+            r = r.format('{0}-{1}', r, srv_suffix) if srv_suffix else r
             protocol = 'https' if '-ssl' in r else 'http'
             endpoint = '/members' if '-server' in r else ''
             for host, port in self.get_srv_record('_etcd{0}._tcp.{1}'.format(r, srv)):
@@ -326,7 +327,7 @@ class AbstractEtcdClientWithFailover(etcd.Client):
 
         machines_cache = []
         if 'srv' in self._config:
-            machines_cache = self._get_machines_cache_from_srv(self._config['srv'])
+            machines_cache = self._get_machines_cache_from_srv(self._config['srv'], self._config.get('srv_suffix'))
 
         if not machines_cache and 'hosts' in self._config:
             machines_cache = list(self._config['hosts'])
diff --git a/patroni/postgresql/__init__.py b/patroni/postgresql/__init__.py
index 9fd1b660..3e5ccee3 100644
--- a/patroni/postgresql/__init__.py
+++ b/patroni/postgresql/__init__.py
@@ -1077,7 +1077,7 @@ class Postgresql(object):
         for app_name, sync_state, replica_lsn in self.query(
                 "SELECT pg_catalog.lower(application_name), sync_state, pg_{2}_{1}_diff({0}_{1}, '0/0')::bigint"
                 " FROM pg_catalog.pg_stat_replication"
-                " WHERE state = 'streaming'"
+                " WHERE state = 'streaming' AND {0}_{1} IS NOT NULL"
                 " ORDER BY sync_state DESC, {0}_{1} DESC".format(sort_col, self.lsn_name, self.wal_name)):
             member = members.get(app_name)
             if member and not member.tags.get('nosync', False):
diff --git a/patroni/validator.py b/patroni/validator.py
index e7aed603..6d2a060c 100644
--- a/patroni/validator.py
+++ b/patroni/validator.py
@@ -302,10 +302,11 @@ validate_host_port_listen.expected_type = string_types
 validate_host_port_listen_multiple_hosts.expected_type = string_types
 validate_data_dir.expected_type = string_types
 validate_etcd = {
-    Or("host", "hosts", "srv", "url", "proxy"): Case({
+    Or("host", "hosts", "srv", "srv_suffix", "url", "proxy"): Case({
         "host": validate_host_port,
         "hosts": Or(comma_separated_host_port, [validate_host_port]),
         "srv": str,
+        "srv_suffix": str,
         "url": str,
         "proxy": str})
 }


Test patch:
diff --git a/tests/test_etcd.py b/tests/test_etcd.py
index b96082ae..c631bd90 100644
--- a/tests/test_etcd.py
+++ b/tests/test_etcd.py
@@ -87,7 +87,8 @@ def dns_query(name, _):
         raise DNSException()
     srv = Mock()
     srv.port = 2380
-    srv.target.to_text.return_value = 'localhost' if name == '_etcd-server._tcp.foobar' else '127.0.0.1'
+    srv.target.to_text.return_value = \
+        'localhost' if name in ['_etcd-server._tcp.foobar', '_etcd-server-baz._tcp.foobar'] else '127.0.0.1'
     return [srv]
 
 
@@ -183,6 +184,7 @@ class TestClient(unittest.TestCase):
 
     def test__get_machines_cache_from_srv(self):
         self.client._get_machines_cache_from_srv('foobar')
+        self.client._get_machines_cache_from_srv('foobar', 'baz')
         self.client.get_srv_record = Mock(return_value=[('localhost', 2380)])
         self.client._get_machines_cache_from_srv('blabla')