Project: Kuma (Orchestration & Management / Service Mesh)
File: HealthChecks-L4.md

at the moment, default configuration for dataplanes doesn't include health checks without health checks (or, alternatively, without health statuses delivered via `EDS`), Envoy cannot exclude a non-responsive upstream endpoint from the list of valid targets for load balancing there are several ways how Envoy can be configured to obtain health statuses for upstream endpoints Envoy can do "active" health checks for TCP it means that Envoy will periodically try establishing a connection Envoy can do "passive" health checks for TCP it means that Envoy will be remembering statuses of recent connection attempts Envoy can receive health statuses from the Control Plane as part of EDS API support `Envoy -> upstream` "health checks" support `Envoy -> local app` "health checks" forward results of "health checks" to the Control Plane aggregate results of "health checks" in the Control Plane include "health statuses" into EDS configuration sent to Envoy ```yaml type: HealthCheck name: rule-name mesh: default sources: match: service: '' # values other than '' can be set here, but will it be used in practice? destinations: match: service: backend conf: activeChecks: # configuration for a TCP check that only verifies whether a connection can be established timeout: 1s interval: 5s # how often health check requests should be made unhealthyThreshold: 3 healthyThreshold: 2 passiveChecks: unhealthyThreshold: 3 penaltyInterval:

# for how long endpoint should be considered unhealthy ``` In order to generate `Cluster`s for a given `Dataplane`: List all `HealthCheck`s in the same `mesh` as `Dataplane` For each outbound interface of the `Dataplane` Go through `HealthCheck`s and select those where `destinations` selector matches `service` tag of that outbound* interface `sources` selector matches tags on one of the inbound* interfaces of that `Dataplane` Order matched `rule`s and keep only 1 "best match" If there is no "best match" `rule` Generate a `Cluster` without "active" and "passive" health checks If there is a "best match" `rule` Generate a `Cluster` with "active" and "passive" health checks according to that rule TODO: TODO: TODO: TODO: Envoy already has a couple features we could build on top of, namely for "active" health checks for "passive" health checks for re-using Envoy as a worker node capable of doing health checks (completely unrelated to regular proxying) event log for "active" health checks: at the moment, ONLY file output is supported see an example in the end configured individually for each `Cluster` event log for "passive" health checks: at the moment, ONLY file output is supported see an example in the end configured globally for all `Cluster`s as part of `bootstrap` config Health xDS: as part of this API, Envoy receives assignments and reports results back to the Control Plane over gRPC see an example in the end is not considered production ready yet (developed in summer 2018 by an intern) does not support mTLS Conclusions: we can already use `Health xDS` for `Envoy -> local app` health checks changes to the Envoy will be necessary to use `Health xDS` for `Envoy -> upstream` "health checks" (add support for mTLS) changes to the Envoy will be necessary to send event logs to the Control Plane (instead of logging to a file) TODO: ```json { "healthcheckertype": "TCP", "host": { "socket_address": { "protocol": "TCP", "address": "127.0.0.1", "resolver_name": "", "ipv4_compat": false, "port_value": 8081 // endpoint #1 } }, "clustername": "serviceresponder", "addhealthyevent": { // health check PASSED "first_check": true }, "timestamp": "2019-11-02T11:04:39.858Z" } { "healthcheckertype": "TCP", "host": { "socket_address": { "protocol": "TCP", "address": "127.0.0.1", "resolver_name": "", "ipv4_compat": false, "port_value": 8081 // endpoint #1 } }, "clustername": "serviceresponder", "ejectunhealthyevent": { // health check FAILED "failure_type": "NETWORK" }, "timestamp": "2019-11-02T11:18:39.898Z" } { "healthcheckertype": "TCP", "host": { "socket_address": { "protocol": "TCP", "address": "127.0.0.1", "resolver_name": "", "ipv4_compat": false, "port_value": 18081 // endpoint #2 } }, "clustername": "serviceresponder", "healthcheckfailure_event": { // health check FAILED "failure_type": "NETWORK", "first_check": true }, "timestamp": "2019-11-02T11:04:39.859Z" } { "healthcheckertype": "TCP", "host": { "socket_address": { "protocol": "TCP", "address": "127.0.0.1", "resolver_name": "", "ipv4_compat": false, "port_value": 18081 // endpoint #2 } }, "clustername": "serviceresponder", "addhealthyevent": { // health check PASSED "first_check": false }, "timestamp": "2019-11-02T11:18:39.901Z" } ``` ```json { "type": "CONSECUTIVEGATEWAYFAILURE", "clustername": "serviceresponder", "upstream_url": "127.0.0.1:18081", "action": "EJECT", "num_ejections": 0, "enforced": false, "ejectconsecutiveevent": {}, "timestamp": "2019-11-02T11:41:25.814Z" } { "type": "CONSECUTIVE_5XX", "clustername": "serviceresponder", "upstream_url": "127.0.0.1:18081", "action": "EJECT", "num_ejections": 1, "enforced": true, "ejectconsecutiveevent": {}, "timestamp": "2019-11-02T11:41:25.814Z" } { "type": "CONSECUTIVE_5XX", "clustername": "serviceresponder", "upstream_url": "127.0.0.1:18081", "action": "UNEJECT", "num_ejections": 1, "enforced": false, "timestamp": "2019-11-02T11:41:58.713Z", "secssincelast_action": "32" } ``` ``` clusterHealthChecks: clusterName: 127.0.0.1:8081 healthChecks: alwaysLogHealthCheckFailures: true eventLogPath: /tmp/127.0.0.1_8081.log healthyEdgeInterval: 3s healthyThreshold: 2 interval: 3s noTrafficInterval: 3s tcpHealthCheck: {} timeout: 1s unhealthyEdgeInterval: 3s unhealthyInterval: 3s unhealthyThreshold: 3 localityEndpoints: endpoints: address: socketAddress: address: 127.0.0.1 portValue: 8081 locality: region: eu subZone: demo zone: eu-west-1a interval: 2s ``` ``` endpointHealthResponse: endpointsHealth: endpoint: address: socketAddress: address: 127.0.0.1 portValue: 8081 healthStatus: UNHEALTHY ... endpointHealthResponse: endpointsHealth: endpoint: address: socketAddress: address: 127.0.0.1 portValue: 8081 healthStatus: HEALTHY ``` Notice that Envoy's report consists only of 1 field - "health status", one of: HEALTHY UNHEALTHY DRAINING TIMEOUT DEGRADED