Building Pubsub for 50M concurrent socket connections
By: Manas Chaudhari
Tags: Mqtt, Scalability, Load Testing, Pub Sub, Emq

A single instance of EMQX performed great. So, we decided to go ahead with it.

There was one caveat though. After the load test, EMQX used to get stuck. In this state, it accepted new connections & subscriptions, but it didn’t route the messages to these new subscriptions. In case there was a subscription done before the load test stopped, that client received those messages. This indicated that maybe there is some heavy computation happening during disconnections, which prevents routing for any new subscriptions. Thus, we had to restart EMQX before every load test. We eventually learned Erlang and fixed this bug. However, the details of this are beyond the scope of this post. More info here.

2. Cluster Tests

After achieving 250k connections on a single machine, our next goal was to add more machines and achieve proportional connections, without any performance degradation.

In clustered mode, messages can be published to any of the nodes, and EMQX will perform the routing to ensure that it gets delivered to subscribers on other nodes.

Infrastructure

We setup EMQX on 5 c5.4xlarge machines and joined them to form a cluster. We spawned 50 m4.xlarge MZBench worker nodes.

Results

The latency increased linearly as we added more nodes. This makes sense, as hops needed to deliver the message increase with every node.

2.1 Separating Publishers and Subscribers

Say there are 5 nodes, and there are 250k subscribers on each of the nodes. Now, if a message is published to node 1, it needs to forward it to the 250k subscribers + 4 other nodes. Only after sending a message to the other node, can the other node forward it to its own subscribers. So, this internal node delay gets added to the latency of every subscriber on nodes 2–5.

In order to not have inter-node latency, we dedicated one node only for handling publishes. This way, it needs to forward the messages only to 4 other nodes, which it can do rapidly.

By separating publisher and subscriber, we achieved constant latency irrespective of the number of nodes. Our load test passed for 2 Million SUBS 🚀.

However, this worked well only till 8 nodes. EMQX internally uses Mnesia distributed database, which is part of the Erlang OTP framework. However, as this book suggests, Mnesia is designed for the low number of nodes (< 10). This could be a reason for this inefficiency.

Owing to this limitation, we decided to move to a multi-cluster architecture, where each cluster of not more than 8 nodes was horizontally scalable.

3. Load Balancers

In our load tests, we were explicitly connecting to the IPs of each broker. Now, we needed a load balancer so that clients have a single endpoint to connect to.

Our initial tests using classic ELBs in their default configurations were not successful. At even a low scale of ~500k, connections got dropped randomly. We saw a spike in SurgeQueueLength and SpilloverCount ELB metrics, indicating the problem at ELB level.

We had to get ELBs pre-warmed to a very high capacity. With pre-warmed ELBs, we could reach 4M connections on an ELB. Beyond 4M, the only way was to have multiple ELBs.

We also explored various other options:

AWS NLB

Software load balancing: HAProxy/Nginx

Exposing EMQ nodes publicly and performing client-side load balancing

It turned out that load-balancing persistent TCP connections is not so simple! We even discovered some bugs in AWS NLB. Ultimately, we decided to go ahead with multiple classic ELBs.

4. Scaling beyond 2M

As we couldn’t add more nodes into the cluster, we either had to set up multiple clusters OR replace the clustering mechanism with our own routing.

4.1 Multiple Clusters

We explored having multiple clusters and joining them via EMQX bridges. By creating a bridge, an emqx node will blindly forward all messages to the destination emqx node.

So, each cluster contained 1 publish node and 8 subscribe nodes. Additionally, we deployed a master publish node, which forwards all messages to the publish nodes from each cluster using bridges.

Infra: Aiming for 10 Million, we deployed 5 EMQX clusters with 8 c5.4xlarge subscribe nodes each. All the publish nodes and master publish nodes were also of type c5.4xlarge, although this can be optimised.

Results: We ran our tests, by adding a cluster one by one. However, we found that the number of supported clients did not increase linearly as we added more clusters. This was surprising as clusters are supposed to be independent. After digging deeper into how Erlang Distribution works, we found that we needed to make the master publisher node hidden to keep the clusters independent.

With this, we crossed the 10 Million connections mark 👏

However, our deployment of multi-clustered emqx had become very rigid. As the master publisher needs to be aware of all the child publishers in order to create bridges, it was difficult to dynamically add/remove a cluster. Also, as there were two parameters: the number of clusters and the number of nodes in a cluster, it wasn’t trivial to scale of the system. The following section describes how we solved this problem.

4.2 Reverse Bridge

As we had a single publisher and lots of subscribers, it was unnecessary for subscribers to know about each other (at least for broadcast). So, we added a Golang service that ran on every subscribe node. It subscribed to all the messages (wildcard topic) from the publish node and forwarded them to the emqx node on localhost. We call this service “Reverse Bridge”, as this does exactly the opposite work of emqx bridge (pull vs push).

With the reverse bridge, we just had N subscribe nodes and a single publish node. And by putting the subscribe nodes inside an Autoscaling Group, we could easily scale up and scale down the infra based on traffic. Additionally, the ASG ensured that any unhealthy node got replaced, giving us resiliency.

And of course, our 10M test passed with the reverse bridge approach too.

Broadcast Achieved!

At this point, the publisher was the only central component, a single instance of which could easily support thousands of subscribers. Each subscriber could support 250k clients. With 200 subscribe nodes, the system can support 50M connections and more!

Of course, this architecture was built for broadcast, and our next problem was unicast. Stay tuned for how we are cracking the unicast use-case!

If you want to hear us talk about the topic in this blog, check out our talk from RootConf 2020.

We’re hiring — if you want to participate in similar adventures, please check out tech.hotstar.com for job listings!