Repository: patroni/patroni

Issue #1199: `patronictl list` shows wrong information when etcd cluster is unhealthy
Created: 2019-09-30T15:30:44Z
Hi !
Thanks for this great software !
Here's a problem I encountered recently with a minimal setup :

3 etcd nodes
2 PostgreSQL nodes

Now imagine a situation where 2 etcd nodes are down. Patroni will demote the primary PostgreSQL node. However the command patronictl list  displays the previous state of the PostgreSQL nodes. The old primary is still presented as the "Leader" but patroni has put it in standby mode.
Maybe I missed something in my configuration. Let me know if this is not the expected behaviour of patronictl.
Anyway, this may be confusing for end users. At least when the etcd cluster is unhealthy, patronictl should output a warning saying that "the DCS cluster has a problem" and "the information displayed may be incorrect".
Thanks again for your work

Duplicate issue #1379: Patroni Reports a leader when DCS quorum is lost
Created: 2020-01-24T21:16:44Z
Patroni 1.6.3
DCS is etcd
I shut down 2 of the 3 etcd members to break quorum.
Patroni itself behaves correctly shutting down the leader, however the API (and patronictl) is still reporting that it has a leader).
For example:
curl -s http://127.0.0.1:8008/cluster | jq .
{
  "members": [
    {
      "timeline": 1,
      "role": "replica",
      "host": "1.2.3.4",
      "port": 5432,
      "name": "pg-m-0",
      "state": "running",
      "api_url": "http://1.2.3.4:8008/patroni",
      "lag": 0
    },
    {
      "timeline": 1,
      "role": "leader",
      "host": "5.6.7.8",
      "port": 5432,
      "name": "pg-m-1",
      "state": "running",
      "api_url": "http://5.6.7.8:8008/patroni"
    }
  ]
}

Logs from the former leader at the time I broke quorum:
2020-01-24 20:45:59,631 INFO: Lock owner: m-1; I am m-1
2020-01-24 20:45:59,644 INFO: no action.  i am the leader with the lock
2020-01-24 20:46:09,626 ERROR: Request to server http://etcd-m-1:2379 failed: MaxRetryError("HTTPConnectionPool(host='etcd-m-1', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74fff28cf8>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2020-01-24 20:46:09,626 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:09,626 INFO: Selected new etcd server http://etcd-m-0:2379
2020-01-24 20:46:09,629 ERROR: Request to server http://etcd-m-0:2379 failed: MaxRetryError("HTTPConnectionPool(host='etcd-m-0', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74ff2d2978>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2020-01-24 20:46:09,629 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:09,629 INFO: Selected new etcd server http://etcd-m-2:2379
2020-01-24 20:46:09,639 INFO: Lock owner: m-1; I am m-1
2020-01-24 20:46:12,977 ERROR: Request to server http://etcd-m-2:2379 failed: MaxRetryError('HTTPConnectionPool(host=\'etcd-m-2\', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by ReadTimeoutError("HTTPConnectionPool(host=\'etcd-m-2\', port=2379): Read timed out. (read timeout=3.3333333333333335)",))',)
2020-01-24 20:46:12,977 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:12,977 INFO: Selected new etcd server http://etcd-m-1:2379
2020-01-24 20:46:12,978 ERROR: Request to server http://etcd-m-1:2379 failed: MaxRetryError("HTTPConnectionPool(host='etcd-m-1', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74ff2f7748>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2020-01-24 20:46:12,978 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:12,978 INFO: Selected new etcd server http://etcd-m-0:2379
2020-01-24 20:46:12,981 ERROR: Request to server http://etcd-m-0:2379 failed: MaxRetryError("HTTPConnectionPool(host='etcd-m-0', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74ff2f7240>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2020-01-24 20:46:12,981 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:12,981 ERROR: Machines cache is empty, no machines to try.
2020-01-24 20:46:15,427 ERROR: Request to server http://etcd-m-2:2379 failed: MaxRetryError('HTTPConnectionPool(host=\'etcd-m-2\', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by ReadTimeoutError("HTTPConnectionPool(host=\'etcd-m-2\', port=2379): Read timed out. (read timeout=2.109329620997111)",))',)
2020-01-24 20:46:15,427 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:15,427 INFO: Selected new etcd server http://etcd-m-1:2379
2020-01-24 20:46:15,429 ERROR: Request to server http://etcd-m-1:2379 failed: MaxRetryError("HTTPConnectionPool(host='etcd-m-1', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74ff2f74e0>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2020-01-24 20:46:15,429 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:15,429 INFO: Selected new etcd server http://etcd-m-0:2379
2020-01-24 20:46:15,432 ERROR: Request to server http://etcd-m-0:2379 failed: MaxRetryError("HTTPConnectionPool(host='etcd-m-0', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74ff2f70b8>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2020-01-24 20:46:15,432 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:15,432 ERROR: Machines cache is empty, no machines to try.
2020-01-24 20:46:17,067 ERROR: Request to server http://etcd-m-2:2379 failed: MaxRetryError('HTTPConnectionPool(host=\'etcd-m-2\', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by ReadTimeoutError("HTTPConnectionPool(host=\'etcd-m-2\', port=2379): Read timed out. (read timeout=1.289056698481242)",))',)
2020-01-24 20:46:17,067 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:17,067 INFO: Selected new etcd server http://etcd-m-1:2379
2020-01-24 20:46:17,069 ERROR: Request to server http://etcd-m-1:2379 failed: MaxRetryError("HTTPConnectionPool(host='etcd-m-1', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74ff2c22e8>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2020-01-24 20:46:17,069 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:17,069 INFO: Selected new etcd server http://etcd-m-0:2379
2020-01-24 20:46:17,077 ERROR: Request to server http://etcd-m-0:2379 failed: MaxRetryError("HTTPConnectionPool(host='etcd-m-0', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f74ff2d2518>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2020-01-24 20:46:17,077 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:17,077 ERROR: Machines cache is empty, no machines to try.
2020-01-24 20:46:19,646 ERROR: Request to server http://etcd-m-2:2379 failed: MaxRetryError('HTTPConnectionPool(host=\'etcd-m-2\', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by ReadTimeoutError("HTTPConnectionPool(host=\'etcd-m-2\', port=2379): Read timed out. (read timeout=1.7925755977630615)",))',)
2020-01-24 20:46:19,646 INFO: Reconnection allowed, looking for another server.
2020-01-24 20:46:19,646 ERROR: Machines cache is empty, no machines to try.
2020-01-24 20:46:19,646 ERROR: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.5/http/client.py", line 1198, in getresponse
    response.begin()
  File "/usr/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.5/socket.py", line 576, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 423, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='etcd-m-2', port=2379): Read timed out. (read timeout=1.7925755977630615)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 193, in _do_http_request
    response = request_executor(method, url, fields=fields, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/request.py", line 171, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/poolmanager.py", line 330, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.5/dist-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='etcd-m-2', port=2379): Max retries exceeded with url: /v2/keys/patroni/postgresql/leader (Caused by ReadTimeoutError("HTTPConnectionPool(host='etcd-m-2', port=2379): Read timed out. (read timeout=1.7925755977630615)",))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/etcd/client.py", line 828, in _next_server
    mach = self._machines_cache.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 391, in wrapper
    retval = func(self, *args, **kwargs) is not None
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 590, in _update_leader
    return self.retry(self._client.write, self.leader_path, self._name, prevValue=self._name, ttl=self._ttl)
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 375, in retry
    return retry(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/patroni/utils.py", line 328, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/etcd/client.py", line 500, in write
    response = self.api_execute(path, method, params=params)
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 243, in api_execute
    response = self._do_http_request(request_executor, method, self._base_uri + path, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/patroni/dcs/etcd.py", line 207, in _do_http_request
    self._base_uri = self._next_server(cause=e)
  File "/usr/local/lib/python3.5/dist-packages/etcd/client.py", line 832, in _next_server
    cause=cause)
etcd.EtcdConnectionFailed: No more machines in the cluster
2020-01-24 20:46:19,648 ERROR: failed to update leader lock

Closing comment (2020-01-25T07:33:34Z): This is a duplicate of #1199
It shows the stale leader because we are not using quorum reads.