Change:
Currently, etcd livenessProbe is defined here: This means that it proves /health endpoint 3 times (default failureThreshold value) and if all of them fails, it restarts etcd server. /health endpoint returns "unhealthy" if one of the conditions is met (): there is an active Alarm (there are two kinds of alarm: NOSPACE and CORRUPT) there is no raft leader the latency of a QGET request doesn't exceed 1s The problem is that in most of those cases, etcd's restart isn't a right behavior: if there is NOSPACE alarm, the restart will not free that space if there is no raft leader, it means that it cannot talk to other etcd members, so restart will not fix them if the request latency 1s, the etcd cluster is overloaded which is bad, but after restart we will get similar amount of requests (if not more), so restart isn't a right action e.g. in we see that event-etcd gets restarted by kubelet when it's overloaded. If we do not restart etcd at that point, we will likely simply finish work to be done and heal ourselves. I think we should change livenessProbe to detect cases where restart will actually help. /cc
I think that's not necessary true, that may potentially help sometimes. But I agree that's probably pretty rare. /sig api-machinery
Do we have an analysis of why the 3 healthchecks in row failed in Was it timeout, timeout, timeout or something else? Do you say that if etcd is overloaded then restarting it will help? Wouldn't it be even more overloaded and run into restart-loop because it will never become /healthy in time again?
Sorry, typo. I tried to say that in none of those cases restart will help (including 1s latency), but in fact I wrote something else. I changed initial post.
should fix etcd such that it does not declares as unhealthy as a result of overload (while its still making a progress). Ideally the not-healthchecking traffic should be throttled / load-shed, such that healthcheck stays responsive. this is implemented, I would consider giving higher timeout (e.g. 30sec), such that HC request have higher chance to reach beginning of the processing queue and get responded. is a usual disjoint between 2 purpuses of healthchecking: a) autohealing -by restart -where we need to be very conservative about timeouts, as cost is high. b) measuring SLO & alerting -where we want to measure customer's happiness, and we cannot consider request waiting for 15s+ seconds as healthy/happy state. The lower tolerance more represents customers needs. In practical application we should optimize etcd healthcheck for a), while b) should be based on prometheous metrics.
This is basically the difference between "livez" and "readyz". "livez" is whether the app is running (for autohealing) "readyz" is whether it's useful from customer point of view (e.g. for SLO) In the ideal world, those should be two separate handlers.
In addition to doubling the timeout we can also consider halving the probing period (from default 10s to 5s) and doubling failure thresholds (from 3 failed probes in row to 6). It would decrease the probability of etcd being restarted in case of temporarily overloaded master.
There is no option to keep using "/health" endpoint and changing timeout to something higher, as this is hardcoded in etcd: Also I think we should stop restarting etcd on NOSPACE or CORRUPT alarms. This way we will stop restarting on alarms and have ability to control timeout (and setting it to some higher value). Once etcd starts exposing some better endpoint that can be used for liveness probes, we will switch to that. WDYT?
/cc /triage accepted

Discussion:
local etcd_apiserver_creds="${ETCD_APISERVER_CREDS:-}" local etcd_extra_args="${ETCD_EXTRA_ARGS:-}" local suffix="$1" <del> local etcd_livenessprobe_port="$2" </del> <ins> local etcd_listen_metrics_port="$2" local etcdctl_certs="" </ins> if [[ -n "${INITIAL_ETCD_CLUSTER_STATE:-}" ]]; then cluster_state="${INITIAL_ETCD_CLUSTER_STATE}"

# mTLS should only be enabled for etcd server but not etcd-events. if $1 suffix is empty, it's etcd server. if [[ -z "${suffix}" && -n "${ETCD_APISERVER_CA_KEY:-}" && -n "${ETCD_APISERVER_CA_CERT:-}" && -n "${ETCD_APISERVER_SERVER_KEY:-}" && -n "${ETCD_APISERVER_SERVER_CERT:-}" && -n "${ETCD_APISERVER_CLIENT_KEY:-}" && -n "${ETCD_APISERVER_CLIENT_CERT:-}" ]]; then etcd_apiserver_creds=" --client-cert-auth --trusted-ca-file ${ETCD_APISERVER_CA_CERT_PATH} --cert-file ${ETCD_APISERVER_SERVER_CERT_PATH} --key-file ${ETCD_APISERVER_SERVER_KEY_PATH} " <ins> etcdctl_certs="--cacert ${ETCD_APISERVER_CA_CERT_PATH} --cert ${ETCD_APISERVER_CLIENT_CERT_PATH} --key ${ETCD_APISERVER_CLIENT_KEY_PATH}" </ins> etcd_apiserver_protocol="https" <del> etcd_livenessprobe_port="2382" etcd_extra_args+=" --listen-metrics-urls=http://${ETCD_LISTEN_CLIENT_IP:-127.0.0.1}:${etcd_livenessprobe_port} " </del> <ins> etcd_listen_metrics_port="2382" etcd_extra_args+=" --listen-metrics-urls=http://${ETCD_LISTEN_CLIENT_IP:-127.0.0.1}:${etcd_listen_metrics_port} " </ins> fi if [[ -n "${ETCD_PROGRESS_NOTIFY_INTERVAL:-}" ]]; then

sed -i -e "s@{{ *etcd_protocol *}}@$etcd_protocol@g" "${temp_file}" sed -i -e "s@{{ *etcd_apiserver_protocol *}}@$etcd_apiserver_protocol@g" "${temp_file}" sed -i -e "s@{{ *etcd_creds *}}@$etcd_creds@g" "${temp_file}" <ins> sed -i -e "s@{{ *etcdctl_certs *}}@$etcdctl_certs@g" "${temp_file}" </ins> sed -i -e "s@{{ *etcd_apiserver_creds *}}@$etcd_apiserver_creds@g" "${temp_file}" sed -i -e "s@{{ *etcd_extra_args *}}@$etcd_extra_args@g" "${temp_file}" <del> sed -i -e "s@{{ *etcd_livenessprobe_port *}}@$etcd_livenessprobe_port@g" "${temp_file}" </del> if [[ -n "${ETCD_VERSION:-}" ]]; then sed -i -e "s@{{ *pillar.get('etcd_version', '(.*)') *}}@${ETCD_VERSION}@g" "${temp_file}" else

} ], "livenessProbe": { <del> "httpGet": { "host": "127.0.0.1", "port": {{ etcd_livenessprobe_port }}, "path": "/health" </del> <ins> "exec": { "command": [ "/bin/sh", "-c", "set -x; exec /usr/local/bin/etcdctl --endpoints=127.0.0.1:{{ port }} {{ etcdctl_certs }} --command-timeout=15s endpoint health" ] </ins> }, "initialDelaySeconds": {{ liveness_probe_initial_delay }}, <del> "timeoutSeconds": 15 </del> <ins> "timeoutSeconds": 15, "periodSeconds": 5, "failureThreshold": 5 </ins> }, "ports": [ { "name": "serverport",

COPY --from=builder /sh /bin/ EXPOSE 2379 2380 4001 7001 <ins> # etcdctl is used by etcd.manifest for livenessProbe. </ins> COPY etcd* etcdctl* /usr/local/bin/ COPY cp* /bin/ COPY migrate-if-needed.sh migrate /usr/local/bin/