Concepts of Load Balancing
By: Akash Srivastava
Tags: AWS, Coding, Software Architecture, Programming, Software Engineering

For a company witnessing a rapid expansion, handling the load is very important. The load balancing techniques that are used across various organizations generally remains the same. If you are using AWS, you can use an Elastic Load Balancer(ELB), which will help you to distribute traffic across multiple instances. In this article, we will try to understand how load balancing works internally. To understand the concept, we will deploy our service on a single instance and will keep on increasing the load in the machine.

We are having a service A that is interacting directly with the internet and serving the entire load. Now if the load on this machine increases i.e. the number of requests increases, CPU or memory utilization increases and further this machine will not be able to handle the load. At that moment it will become necessary for us to scale our service.

There are 2 ways of scaling the services -

1. Horizontal Scaling — Increasing the number of instances of service A to handle the load

2. Vertical Scaling — Increasing the resource allocated to the machine ie number of CPUs, RAM, disk, etc. Vertical scaling can’t be done beyond a certain limit.

L7 Load balancer in front of our service

In order to horizontally scale our service, we will put our service behind an L7 load balancer in front of our service. L7 stands for Layer 7 or the Application Layer. This load balancer is able to read the content of the request and will be able to redirect it to instances. Generally, Nginx or HAProxy are used to implement an L7 load balancer. Examples of load balancing using Nginx are as follows:

L7 Load balancing based on request type on Nginx: In this example, we can redirect traffic based on the request. We can also redirect the request to 1 machine or a group of machines. Sample Nginx config is as follows:

upstream backend {

least_conn;

server 10.128.1.4;

server 10.128.1.5;

}

server { location /admin/ {

proxy_set_header Host $host;

proxy_pass <http://10.128.1.2>;

} location /category/ {

proxy_set_header Host $host;

proxy_pass <http://10.128.1.1>;

} location / {

proxy_set_header Host $host;

proxy_pass <http://backend>;

} }

L7 Load balancing based on health check on Nginx: We can also use Nginx to balance the load among the healthy/unhealthy instances. A sample config is as follows:

server {

location / {

proxy_pass <http://backend>;

health_check port=8080;

}

} http {

upstream backend {

zone backend 64k;

server backend1.example.com;

server backend2.example.com;

server backend3.example.com;

server backend4.example.com;

}

}

If the health check fails, Nginx will not redirect traffic to that node.

Increasing the load further

Increasing the load further, our L7 load balancer will not be able to handle the traffic. In that scenario we gonna need multiple L7 load balancers balanced by an L4 Load balancer. L4 stands for Layer 4 or Transport Layer Load balancer. When the Layer 4 Load Balancer receives a request and makes the load balancing decisions, it also performs Network address translation (NAT) on the request packet, changing the record destination IP address from its own to that of the content server it has chosen in the internal network. L4 Load balancers make their routing decisions based on the address information extracted from the first few packets in the TCP stream and don’t inspect the packet content.

Layer 4 load balancing is generally implemented using HAProxy. On this layer, we balance the traffic based on the type of network-level protocol they are using i.e. TCP/UDP.

Increasing the load further

On further increase in load, we can place a Layer 3 Load balancer in front of 2 L4 Load balancers. L3 stands for Layer 3 or Network Layer. Layer 3 Load balancers use ECMP which generally uses Equal Cost Multipath Routing (ECMP). These are designed to avoid the problem of sending all packets from any particular network flow down a single deterministic path while balancing multiple flows over multiple paths.

Further increasing the load

DNS is the system that translates names to IP addresses. DNS service can return multiple IP addresses for a particular request. This is a common load- balancing technique. DNS provides a few common techniques for load balancing -

GeoDNS — GeoDNS is such a system that provides strategically located name servers on different continents and a process of distributing the traffic based on the locations of the queries Round Robin DNS — This is the simplest load balancing techniques. The DNS server responds with the IP of the servers in a Robin Robin fashion.

The above-mentioned techniques can be combined in different manners as per your requirement. Few alternates styles are as follows —

DNS to L7 ELB

DNS to L7 LB

DNS to L4 ELB

DNS to L4 ELB

Summary

When your service is getting exhausted due to an increase in load, you can spin another instance of your service and can balance the load among the services using an L7 load balancer. You can use combinations of L3, L4, and L7 load balancers to balance the traffic on your system. What combination you have to use depends mostly on the type of your system. You can try multiple combinations and measure the throughput of your system and can proceed with the one having maximum throughput and minimum latency.

References: