A Short Note on Cascading Failures in MicroServices Architecture
By: Mourya Venkat
Tags: Cascading Failure, Resilience, Scalability, Circuit Breaker, Microservices

If there is one big transition that took place in the past decade or so in terms of application development style and deployment patterns then it has to be migration from Monolithic application development to Microservices.

The latter had provided us with various advantages by splitting the bigger application into various smaller chunks with each chunk being called a microservice.

Each chunk can be written in a different programming language and have its own database that suits the scale at which the chunk operates.

Each chunk can scale independently based on the amount of processing it does.

Build time and application startup time will be drastically reduced which allows us to have a lesser downtime in case any of the microservice crashes due to an outage in resources or any unhandled exceptions.

More importantly, if something goes wrong in one service, it will not impact the overall funnel.

There are many such advantages that the newer microservices architecture brings to us but there is one thing and one major thing that this approach is failing to provide us which is TRUST.

When I say trust, it's a trust put by one service on all the dependent services and the dependent services on their respective depending services. The reasons for not being able to trust the other services being

The communication happens through a network which might be unreliable.

The core functionality of the other services is abstracted out in such a way that the caller won’t be sure how big of processing the other service does in order to serve the response required by the caller.

The caller was restricted in a boundary wherein if something goes wrong outside his boundary, he/she won’t be able to figure it out.

There are many such trust issues between the caller and receiver in terms of communication but in the end, when a client requests our application, as per the promises delivered by microservices architecture one failure in any of the leaf services shouldn’t impact the overall funnel. The process of one service failure affecting the overall funnel is termed as Cascading Failure.

So without bragging much about the theoretical details, let’s get some practical knowledge. In this article, I will detail one such Cascading failure that we experienced a few weeks ago that took the entire goibibo.com/hotels for a toss.

For any sort of customer-facing products, the time in which the system responds to the user request is the key measure for user satisfaction. Keeping this in mind, each service determines its own SLA(Service Level Agreement) after benchmarking their service against the production-grade resources.

There are many key things in SLA agreement out of which response time is one. Response time is further divided into 3 parts.

95 percentile response time(50 ms)

97 percentile response time(200 ms)

99 percentile response time(500 ms)

This says that out of 100 requests that my service receives, 95 of them get served in 50 ms and 97 of them get served under 200 ms and 99 of them get served under 500 ms. In most of the cases, the 99 percentile response time is conceded as the response time SLA of the service.

But here is the catch.

In order to make sure that 99 out of 100 requests are served properly, we are setting a larger wait time at the client end. The client assumes that 95 requests are served under 50 ms and only 4 requests take more time because of various reasons like request queuing, context switching, etc, and based on that it provides its SLA to the other upstream services.

Assume that the receiver service made some database changes or took a defective release and the 95% response time of the receiver service is now exceeding 500ms.

For the case of simplicity, assume that I am running 1 instance of each of these microservices and each on 1 core CPU. Now I get a request and make parallel calls to each of these 3 leaf services and once all of them return a response, I’ll process the data from all 3 and return the aggregated response back to the mobile front end to render the data.

The total time taken by the aggregator to get the response from all dependent services is max resp time (S1, S2, S3) + no of context switches * time taken b/w each context switch.

As soon as the aggregator gets a call from the Gateway, the work to serve the request is given to the single CPU core. Now as soon as the aggregator reached a point where it called these 3 services and wait for the response, the CPU switches the context to serve the next waiting request and the same happens with all the upcoming requests.

If the requests are completed in a short span of time, the wait queue will be very lean and so does the load on CPU but as soon as the response time increases for any of the dependent services, the number of half-boiled requests waiting for completion will start queuing up and with increased queuing it not only affects the requests that we already got but also the requests which are new as these will be placed at end of the queue waiting for CPU to serve them.

This means that the shortcomings of service 1 SLA are impacting all the upstream services which are directly or indirectly dependent. But if we look at the advantages that I’ve pointed out at the start of this article(More importantly, if something goes wrong in one service, it will not impact the overall funnel), this condition is breached. So, how do we handle these sort of cascading failures?

This is where Circuit Breakers comes into the picture. Stay tuned for my next article on Circuit Breakers wherein I will brief you about the functionality of these circuit breakers and how to set up one.