Why did we choose NATS
By: Yaniv Ben-Hemo
Tags: Big Data Analytics, Apache Kafka, Rabbitmq, Big Data, Nats Streaming

Our story and usage of NATS.IO.

We are “Strech”, The Data Streaming Platform the world loves.

To understand our choice, we need to start by understanding our needs, define KPIs, and then compare the possible solutions and their deployment options, protection abilities, scaling, support, community, and not less important — the people behind the projects.

Let’s start.

“Strech” is completely cloud-native. One of our strengths is our ability to deploy over any Kubernetes cluster, anywhere, on any cloud or edge.

Our SaaS platform wraps our proprietary, asynchronous engine.

Async is the game’s name in our use case. Why? Basically, in any big-data transportation and analytics use case, we need a data store or a persistent data structure to support our data while in transit between different components.

At its core, NATS is about publishing and listening for messages. NATS is a very small infrastructure compared to Kafka. Kafka is more matured compared to NATS and performs very well with massive data streams. NATS Server has a subset of Kafka’s features as it is focused on a narrower set of use cases. NATS is designed for scenarios where high performance and low latency are critical but losing some data is acceptable (We solved this one).

Last but not least, NATS is easy to operate — deploy over Kubernetes, monitor by Grafana, encrypt traffic, and much more.

We use NATS, specifically a feature called STAN (replaced by JetStream), which is NATS’s older streaming service for moving bits of data between the different engine services our platform offers to our users -

*Bear in mind that the following might not be needed while using JetStream*

What was required to take under consideration and self-developed which doesn’t exist out-of-the-box in STAN -

NATS Massage size

NATS has a message size limitation enforced by the server and communicated to the client during connection setup. Currently, the limit is 1MB. For us or any other solution/use case such as ours, this means that the data receiver service should be able to split the data into small fragments no bigger than 1MB.

Data Routing

When fragments of data travel between different STAN channels/microservices, a particular mechanism requires orchestrating and navigating the fragments from one stage to another with dynamic path management, meaning that a specific message should pass through actions 1,3,5. Still, other messages, which belongs to different source/pipeline should pass through 2,3,5,6.

Data loss protection

When working in a cloud-native environment, stateless microservices, and the ability to scale in and out, there are some challenges. Imagine the following scenario: Some stage which resides in a pod processing ingesting fragments of data for some reason fail during processing a fragment. Because it’s a pod and managed by Kubernetes, it will probably restart, but what happens to that lost fragment? Probably lost…

We built our atomic system using a cache database to preserve the while-in-process fragments were restored from failure.

NATS’ new streaming service — JetStream.

JetStream was created to solve the problems identified with streaming technology today — complexity, fragility, and a lack of scalability. Some technologies address these better than others, but no current streaming technology is genuinely multi-tenant, horizontally scalable, and supports multiple deployment models. No technology we are aware of can scale from edge to cloud under the same security context while having complete deployment observability for operations.

credit: NATS docs

Besides great technology and architecture — NATS lead by software heroes with a great community and roadmap. It is essential as the technical aspects itself.

I hope our use case helped you make a better decision, and as NATS support us, we would love to support others with their journey with NATS.

team@strech.io