Load Balancers — In-depth guide. In this post we will explore :
By: Aman Yadav
Tags: High Availability, Distributed Systems, Load Balancer

In this post we will explore :

What is a load balancer?

Infra levels where we can use load balancer

Benefits of load balancers

Types of load balancers

Algorithms used in load balancers

A Load Balancer is used to distribute incoming requests from clients to cluster of servers. Here we can have a lots of clients (thousands/millions) which are trying to use a service which is being served by few(hundreds) servers.

Here all the client requests will first land on load balancer and then then load balancer will distribute those request to different workers(servers) so that they can do the requested task. Load balancer hold the client connection till the worker is working on the task but it not blocked on that only, It also accept other client request and delegate that to other workers based on selected request distribution algorithm. Once the worker has completed the task then it returns to the load balancer and load balancer return back to the client and connection is released.

Load balancers continuously check for the active workers using health/ping API’s. If some host is not returning proper response to these hits then it is marked as unhealthy and thus removed from the active group and no more requests are sent to it. Once this worker is healthy or new worker is added to the pool then load balancer will start sending requests to these workers.

Load balancer delegating request to multiple worker instance

A load balancer is not required for distributing http request only. Below are the scenarios where we will need a load balancer.

User on Internet who want some information from some service like Amazon

One service requesting some resource from other service(Inter service communication). Let’s say Delivery service is requesting user information from User service in a e-commerce setup. In this scenario all the services will be behind a load balancer so that load can be distributed.

Application server asking for data from load balanced database. Let’s say we have a highly available MySQL database with 1 master and 3 slaves. So in this setup instead of making connection to each of the slave database we will be using a load balancer which will distribute read request within these slaves.

Various type of load balancer in single diagram

Don’t be confused by the above diagram. I have included 3 levels of load balancers here.

External load balancer which is between client on internet and Delivery service. Internal load balancer which is between two internal service. So that we can scale both of the services independently. Database load balancer which is between application and database instances to balance reads from MySQL master and various slaves.

Benefits of Load Balancers

There are a ton of benefits of using load balancers. We will try to explain a few of them below.

Load Distribution : Primary function of a load balancer is to distribute the load to different worker nodes. Here we have to choose proper algorithm for load balancing so that our worker nodes are utilised properly. Reduced Downtime : Since our worker nodes are behind the load balancer so we can replace faulty workers and spin up new one and external world didn’t even know about it. The load balancer automatically stop sending traffic to faulty worker. This happens using a ping/health API exposed by the worker. Load balancer frequently hits this API for all worker nodes to know about the current state and stop sending traffic to un-health workers High Availability : If we have a load balancer then if some of the workers are down still then service will up as load balancer will distribute the request to the health workers. This way load balancer provides high availability to the system. Auto Scaling : Since our worker nodes are running behind load balancer, we can increase and decrease the number of workers. This can be configured on basis of some parameter like response time i.e if response time of some service is increase by defined threshold then spin up one more instance to handle requests. SSL Offloading : SSL offloading is a process of decrypting incoming https traffic and encrypt server responses. This process is a bit expensive as need to be done for all requests. Load balancer can be configured to offload ssl at load balancer level and then forward http traffic to worker nodes. User Authentication : Load balancers can be configured to authenticate users by integrating user authentication service. This service is used to authenticate each incoming request and some token in headers is used to authenticate that user. If authentication is successful then that request is allowed to enter in system and a header with userId is added for downstream to identify that user. Security : We are exposing IP address of the load balancer to the world and IP of the worker nodes will be internal. Security of our infrastructure is maintained as we are not exposing any of our internal system info to the world.

Different types of Load balancers

So far we have learned what are load balancers and where can we have these load balancers. In this section we will look into different types of load balancers.

Hardware Load Balancers : A hardware load balancer is a hardware device with a specialised operating system that distributes incoming requests to worker nodes. To ensure optimal performance, the hardware load balancer distributes traffic according to customised rules so that worker nodes are not overwhelmed. Traditionally, hardware load balancers and worker nodes are deployed in on-premises data centre and the number of load balancers depends on the expected amount of peak traffic. Load balancers are usually deployed in pairs in case one fails. Software Load Balancers : A software load balancer is implemented using software layer. This can run in virtual machine/cloud/pod.Software load balancer are cheap as compared to hardware load balancer. These load balancer consume a lot of system resources as they are doing a lot of computations and IO operations, so please reserve proper resources in infra while deploying these. HAProxy is quite popular software load balancer.

Algorithms used for Load Balancer

Load balancing algorithms are formulas to determine which worker will serve the current client request. The algorithms can be very simple, like round robin, or they can be hybrid which take multiple factors into consideration.

Random : Randomly choose the workers from the pool and forward request to process. This algorithm do not take into consideration any of current state of the worker. Round Robin : Here the requests are forwarded in round robin fashion. This do not consider any attributes of the server for forwarding request. IP Hashing : Hash of IP is created and based on that hash worker is chosen for current request. This technique will forward all the requests from same client to same worker each time as it will have same IP. This algorithm might cause skewing and few of the worker can be overloaded. Least Connection : In this algorithm the request is forwarded to the worker having least active connections. This algorithm is used exhaustively in production as it evenly distributes the load to all worker nodes and load balancer is aware about current states of the worker. Least Response : In this algorithm load balancer take note of the response time from each instance and gives priority to forwarding traffic to instance which is taking least time to respond.

What about single point of failure?

Since single instance of load balancer is running, it can be single point of failure and if load balancer is down then whole service is down. So to tackle this problem always provision one passive load balancer which can take place of this active load balancer if it fails.

Hope this article have cleared a lots of doubt regarding load balancer. Please applaud and share this article with others.