Streaming Events through RabbitMQ
By: Workable Tech Blog
Tags: Rabbitmq, Nodejs, Asynchronous, Software Development, Microservices

Streams in NodeJS

A stream is an abstract interface for working with streaming data in Node.js. The stream module provides an API for implementing the stream interface. (Node.js docs)

The stream module is one of the biggest powers of Node.js. Streams together with async io enable a server to act while still receiving data, minimizing memory and cpu usage.

Especially when working with big chunks of data, you can work on them in small chunks (e.g. as you read them from the file system) and don’t have to load big parts in memory.

There are many build in streams:

Http request

Http response

Process.stdout

However, working with streams is difficult. It requires a lot of boilerplate code and a deeper understanding of its concepts.

A native way to perform an http get request:

Example taken from: 5 Ways to Make HTTP Requests in Node.js — Twilio

And a way to serve http requests:

Example taken from: Stream | Node.js v15.11.0 Documentation

Directly handling streams is difficult and that’s the reason we have many frameworks and libraries doing the boilerplate and handling errors for us.

When custom logic needs to be implemented though, streams are a powerful module to work with.

Type of streams

Streams can be readable, writable, or both. All streams are instances of EventEmitter. (Node,js docs)

Streams can be any of the following 4:

Readable

Writable

Duplex

Transform

We will describe in more detail Readable and Writable streams.

Readable Stream

Readable streams are an abstraction for a source from which data is consumed. (Node,js docs)

In simple words a Readable stream provides an interface to “read” data from a place. It could be the file system, network or anything else.

A Readable Stream can be either in flowing mode or in paused mode. While in flowing mode the stream will continue to read data from its source. To go from paused to flowing mode one of the following should occur:

a ‘data’ event handler is added

stream.resume() method is called

stream.pipe() is called

The opposite can be done with:

stream.pause()

stream.unpipe()

The Readable stream API evolved across multiple Node.js versions and provides multiple methods of consuming stream data. In general, developers should choose one of the methods of consuming data and should never use multiple methods to consume data from a single stream. (Node,js docs)

Writable Stream

Writable streams are an abstraction for a destination to which data is written. (Node,js docs)

In simple words a Writable stream provides an interface to “write” data to a place. It could be the file system, network or anything else.

Pipe

readable.pipe(destination[, options])

destination can be Writable , Duplex , Transform

can be , , Returns destination

Pipe is a convenient method to direct a readable stream into a writable stream.

eg:

fs.createReadStream(file).pipe(duplex).pipe(fs.createWriteStream(otherFile))

Figure 1: Pipe Readable/Writable

For more information about streams, the events and methods they support, the ways to implement custom ones, the official node docs is an excellent resource: https://nodejs.org/api/stream.html

RabbitMQ and RPCs

RabbitMQ is the most widely deployed open source message broker. (RabbitMQ official page)

RabbitMQ is an open source smart Message Broker. It is smart because it knows what messages to send to each consumer. It is responsible for routing messages to queues.

We use RabbitMQ to make our microservices communication asynchronous. Sometimes a microservice needs a response from another service. Instead of using HTTP we utilize RPCs over RabbitMQ a lot.

To better visualize how this works let’s imagine a service Calculator that knows how to do mathematical calculations and a service Billing that uses the Calculator service to do math.

Calculator is consuming messages from a queue named add among others.

Billing service wants to calculate the sum of 1+2. So it sends a message to add queue with the extra header reply-to: billing_queue* the name of a queue that only this process of the service is consuming messages. This is important because if we have many instances of the Billing service we would like the process that sends to the add queue to be the process that receives the response.

Now the Calculator service consumes the message from the add queue and sends the results to the queue mentioned in reply-to header: billing_queue. Billing service was waiting for the response so it will read it and continue its processing etc.

Figure 2: Calculator RPC example

If we had many instances of the Billing service this would work the same way.

Figure 3: Calculator RPC example in scale

Billing has 3 instances. Each one has an exclusive queue. Each one can use the add RPC from the Calculator service.

Each message that Billing service sends to the add queue also has a correlation_id header that is used to correctly map responses to the corresponding requests. This allows us to send multiple concurrent requests to the Calculator service from the same process without the responses being mixed.

* actually the name of the queue is a random string so that it is unique, usually a server named queue.

Examples

Let’s see now how the above could be implemented in Node.js with code.

We are using rabbit-queue, a module that handles the RabbitMQ integration. It is a high level module that provides an abstraction layer above amqplib. It also has convenient classes to easily implement retries on failures and add messages to dlqs after the retry limit is reached.

Disclaimer: The author of this article is the main contributor of rabbit-queue. The module is production battle tested as we have 10+ microservices using it for various use cases.

Calculator and Billing Service Examples

As we see in the examples above, calling an RPC is as simple as calling an async local function. That’s why it is so powerful. It makes the communication between our microservices seamless and easy.

The Problem

A couple of years ago we wanted to scale our Campaigns product. So we started designing an internal campaigns management tool for this. Until then, our Campaigns manager would use an internal tool to calculate audiences and do many other steps and configurations manually.

Our internal tool was producing some big files that our Campaigns manager needed to download. To be more specific the internal tool was calling another microservice (Orchestrator from now on) to do this job. Orchestrator would produce a big file and ping the internal tool when it had finished with the s3 url of the file.

Figure 4: Communication between Internal tool and Orchestrator through RabbitMQ

The process was problematic because the Campaigns manager had no idea of the progress of this long running task that usually took hours. The processing that the internal tool had to do was added on top of the time to produce the whole file, upload it to s3 and then download it. To make matters worse the memory of the internal tool was reaching its limits when the whole file was downloaded as we had a solution that processed all the data in memory.

How we solved it

When redesigning this process to scale, we knew we could do better. We were developing a new microservice to handle the campaign management called Campaigns. We knew Orchestrator was producing data at smaller chunks, we just had to find a good way to communicate them back to our Campaigns microservice.

The desired architecture would look like this:

Figure 5: Ideal communication between Campaigns and Orchestrator

We were using RabbitMQ rpcs for the communication between the two services. We just had to figure out how we would implement such a solution:

There was no easy way for Campaigns to paginate somehow to the Orchestrator . Eg to send rpcs requesting more and more data. Only the Orchestrator had the ability to do its internal pagination.

to paginate somehow to the . Eg to send rpcs requesting more and more data. Only the had the ability to do its internal pagination. There was a way for Campaigns to initiate the request to the Orchestrator, and Orchestrator to send data in chunks to a queue that Campaigns would listen to. This however would require the process to be stateless and fully asynchronous. Different campaigns processes should be able to handle different chunks of data. However this solution wasn’t very elegant, we needed something better.

Photo by Jason Strull on Unsplash

And then it crossed our minds. We needed to transform a flow that reads the whole file into reading it in small chunks at a time. If this was not between microservices but in one service the obvious solution would be to use streams (in Nodejs).

What if we could extend the implementation of our RPCs to offer a similar interface with that of streams? Was this even possible?

It was a good thing that we maintain the Node.js rabbitMQ driver we use: rabbit-queue. We could experiment with that and find out!

Actually, the first happy path implementation was less than 2 days of effort!

The interface was very elegant too. Let’s see:

Reading a file in a single process:

Pretty nice? Right?

What is happening behind the scenes? Let’s again take our example with the Orchestrator and Campaigns. Let’s assume that a Campaigns process is requesting from the Orchestrator using a jobId to calculate the file needed and send it over a few lines at a time.

Figure 6: An example with 25 lines returned

Pretty straightforward right?

This way we managed to send all Orchestrator responses to the same Campaigns process.

But are we done yet? Not quite…

Error handling

The happy path didn’t cover most error handling cases.

The interface is creating a local readStream. So we had to transfer errors that happened to the remote readStream to the local one.

Figure 7: Errors happening in the middle of a stream are transferred to the consumer

Backpressure

What would happen if the producer of the stream(Orchestrator) was creating data faster than the consumer(Campaigns) could consume? The consumer would start piling up memory. Until it would run out of memory.

If you have implemented your custom stream you probably are already familiar with backpressure. In essence you should:

- Never .push() if you are not asked. - Never call .write() after it returns false but wait for ‘drain’ instead.

from Node,js docs

Moving back to our example:

The idea here was to make the orchestrator wait for the Campaigns to acknowledge it has processed a message before continuing to the next one. Stream implementation has a default buffer of 16 messages (in case the stream is in objectMode). So by default the consumer will acknowledge the first 16 messages even if your program hasn’t processed any.

To implement this acknowledgement:

Like before, Campaigns sends a message to the queue of the Orchestrator with a replyTo header and a correlation id (RPC).

But now the Orchestrator instead of replying to replyTo queue the chunks of data it has, it actually initiates an RPC call sending a message to the replyTo queue and expecting back a reply. Inception right?

Figure 8: Orchestrator will wait for Campaigns acknowledgement before sending new messages

What if consumer is down while reading

Now that we added a backpressure, the Orchestrator implementing the RPC had a problem. What if the Campaigns instance was killed? The Orchestrator would wait indefinitely for an acknowledgement. If a new Campaigns process was started it would be unaware of this. The acknowledgement was expected by the Campaigns instance that was killed as the RPC was sent to its own exclusive queue.

Restarting Orchestrator would not resolve this issue too as the initial RPC message would remain in the queue and another process of it would try to process it, waiting acknowledgements from a reply-to queue that no longer existed. There are 2 possible solutions to this one:

Know if the replyTo queue exists and stop the process if it does not Add a timeout when waiting for the acknowledgement

Although the 1 seems preferable we implemented the 2nd one as it was easier and already supported by rabbit-queue.

Figure 9: Last Campaigns ack is delayed and Orchestrator times out

What if the consumer has finished its job?

At last at some point we wanted to give the consumer the option to stop the service from sending more data in an elegant way. Eg. without skipping an acknowledgement and relying on the timeout being reached. This was easy to support when using the backpressure feature because Orchestrator was already waiting for acknowledgements. We could send in the last acknowledgement that the Campaigns needs to stop consuming messages. Note that due to buffering Campaigns might have in memory several more chunks when we send the stop acknowledgement.

Figure 10: Campaigns can ask from Orchestrator to stop

A full example with backpressure and timeout and stop:

The backpressure, timeout options are passed as rabbitmq headers.

The option to stop a stream is only available if backpressure is true.

You can see more examples.

Next steps

Although this implementation covers all our current use cases the feature is far from complete. There are improvements that can be made. Also more options could be added. Eg.: