Shadow Traffic with Envoy
By: Pankaj Takawale
Tags: Istio, Distributed Systems, Shadow Traffic, Grpc, Envoy

How did we capture the request and response payloads?

The Envoy Http Filters are allowed to augment the request and response headers.

We needed a Http Filter to capture the request and response payloads and add them as additional headers to the existing request and response headers.

The Lua Http Filter envoy.lua allows us to embed the Lua code into the Envoy Proxy which would run during request and response flow in the Envoy’s process address space.

In the Lua code, we just copy the request/response body and add these payloads as additional request/response headers. It’s just a memory copy, so not much additional latency was introduced due to this step. However, it could increase the memory footprint. So, we decided to add a size check on the payloads to limit the size of the request/response payloads to be less than 8K each. All of our requests went through this criteria, although we ended up dropping a few large size response payloads from sinking into our Data Lake. This was acceptable since response body was not much of a concern for us.

However, there are two unwanted side effects of adding these additional request/response headers.

First, the additional request headers would be passed to the upstream micro services. Second, additional response headers would be passed to the clients.

How to suppress the additional payload headers beyond Envoy ecosystem?

The Envoy provides a solution to this — we can just prefix the header names by “:” and these headers are treated specially. They are suppressed so that they will not reach the upstream micro services or the clients. Notice the special header names “:req-body” and “:resp-body”.

However, these headers stay within Envoy and also get passed to the gRPC access log service as expected.

Here is the sample snippet on how to capture the request/response payloads and add them as the headers.

Http gRPC Access Log Configuration

Here is the sample Envoy configuration snippet to send the selective request/response headers to the gRPC cluster.

How to anonymize the sensitive PII (Personally Identifiable Information)?

On receiving the request and response payloads in addition to other headers, the Traffic Sink Service would simply write the events to the Kafka Sink. The anonymize service would consume from this fire hose and anonymize the PII such as password hash, access/auth tokens and others before sinking the records into the Data Lake.

Use Cases

Having this infrastructure in place opens up interesting opportunities to solve many use cases.

Replay Traffic in Shadow Production Environment

We could have another service to consume the ongoing traffic from Kafka Sink to replay it in the shadow production environment. This would also allow us to run different versions of the algorithms or ML models in the shadow environment. We can run the live nDGC evaluation against these algorithms before rolling them out into the production environment.

Load Test in Performance environment

Usually organizations maintain the test suits for running the load tests. These test suits may not be as close to the production traffic pattern. Often the test suit goes stale as well. Having all the production requests saved in the Data Lake along with the event time, would allow us to replay traffic with appropriate rate limiter tuning to perform load testing in the performance environment.

Since we are hashing out the auth/access tokens, running the tests using these captured requests would work well for the read APIs like Search and Browse. However, it would still be possible to replace the tokens smartly with the test user’s tokens in test environment to replicate close to production traffic pattern.

Investigate Requests Failure

The Data Lake would have all the requests detail, response meta-data including response status code. It’s easy to pull out the failed requests and replay them in the Dev environment to reproduce the issue seamlessly.

Improvise Bot Detection Algorithms

Detecting the bot traffic is a cat and mouse game. The bad bots often find the gaps and make their way through. With the ability to replay traffic pattern as it occurred in the production environment allows us to improvise the bot detection algorithms easily.

Offload Async Operations

It’s possible to offload the async operations from the main thread of the micro services to this infrastructure. The operations like Recently viewed items, Add to favorite etc. could be served asynchronously.