Implementing Observability in a Service Mesh
By: Jude Dsouza
Tags: Consul, Site Reliability, Service Mesh, Envoy Proxy, Observability

Implementing Observability

Envoy is just awesome! It has a comprehensive set of configuration options and has tons of metrics to share. It is also the component that can give you rich access logs and HTTP Request Tracing. As discussed above, Consul is the control plane through which you can configure Envoy with these options. In this section I will zoom into the implementation details of how this can be done.

Implementing Observability in a Consul Connect Service Mesh using Prometheus, AWS CloudWatch and AWS X-Ray

HTTP Layer 7 Metrics

In this setup I’ve used Prometheus to scrape metrics from all the sidecar proxies. By setting the envoy_prometheus_bind_addr field in the proxy.config service definition, Envoy can be configured easily to listen for scrape requests on port 9102.

Further, Prometheus can discover the proxies automatically using its Consul service discovery mechanism as show in the example config below:

Prometheus config that discovers Envoy Proxies via Consul SD

These metrics can then be visualized in Grafana.

Screenshot for Overview of Basic Service Health Layer7 Metrics

Access Logs

Access Logs can also be configured in the Envoy proxy using the Escape-Hatch Override option. This option allows you to define a custom listener configuration that enables access logging by setting the envoy_public_listener_json field as part of the proxy.config definition. You can also define it as part of the global proxy-defaults configuration entry as shown below:

Consul Connect `proxy-defaults.hcl` config for HTTP Tracing and Access Logs in Envoy

With this config (skipping over the tracing related information which will be covered in the next section), access logs will be written directly to /dev/stdout and automatically collected by AWS CloudWatch Logs driver and pushed into AWS CloudWatch Logs. You should then see logs similar to the following and of-course you can add more fields to it depending on your requirements:

{

"method": "GET",

"request_id": "420e2950-50be-43c3-8d41-9b8e9ac9937b",

"bytes_sent": "188",

"origin_path": "/",

"authority": "dashboard.ingress",

"x_forward_for": "10.140.21.77",

"protocol": "HTTP/1.1",

"upstream_service_time": "10",

"duration": "10",

"downstream_remote_addr_without_port": "10.140.21.77",

"user_agent": "curl/7.61.1",

"upstream": "127.0.0.1:8080",

"response_code": "200",

"bytes_recv": "0",

"response_flags": "-",

"start_time": "2020-09-03T11:47:01.662Z"

}

The request_id field is important here as with this field you can correlate specific traces of request for example in tools like AWS X-Ray and even in Application Logs.

HTTP Request Tracing

An instance of an AWS X-Ray HTTP Trace b/w the Dashboard and Counter service communication

Finally, HTTP request tracing can be configured in Envoy by setting the envoy_tracing_json and envoy_extra_static_clusters_json fields of the proxy.config service definition. The former defines the tracing provider, in my case it’s AWS X-Ray, and the later defines the corresponding static cluster configuration that contains AWS X-Ray’s endpoint so that HTTP request tracing data can be sent to it.

In order to preserve end-to-end tracing between all services in the service mesh, the services themselves require code that extracts tracing data from the request headers on every inbound connection and forward that on every outbound connection. Additionally, tracing data, in particular the trace_id and request_id must also be included in every log message for traceability and better debugging. With AWS X-Ray, this is possible using their language specific SDK. In the case of Python, you can achieve this using the aws-xray-sdk. Similarly, there is an SDK for Go, Node.js, Java, Ruby and .NET.

Putting it all together

For a single service running in AWS ECS, at least 3 sidecar containers are needed for it to be fully integrated into the Consul Service Mesh, these are the Consul Agent, Envoy Proxy and AWS X-Ray daemon. The Dockerfile for each of these can be found in the linked Github repo above. An example list of ECS Container Definitions may look like the one below:

AWS ECS Container Definitions for a Service to adapt to the Consul Service Mesh

For simplicity, in this setup, these containers are configured to run on AWS ECS Fargate with the AWS VPC Task Networking enabled so that all containers get access to the same localhost and can communicate with each other over that interface.