Message Queuing with RabbitMq
By: Krishani Indrachapa
Tags: Spring Boot, Rabbitmq

Message Queuing with RabbitMq

A message queue is a buffer which keeps messages for asynchronous processing. Producer applications send the message to the queue and consumer applications retrieve them and process them. These messages are stored in the queue until consumer retrieves them.

message queue

Why message queues are used?

Message queues enable asynchronous communication between applications.

Create resiliency, even though a consumer application goes down messages will be kept in the queue until consumer application gets back up and running.

Facilitates decoupling in micro services.

Makes producer and consumer applications autonomous. Ex: producer application can be written in Java while consumer application can be written in Node.

Improves scalability.

Use cases

Let’s assume we have an online ordering application. Our application should be highly available and ready to receive new requests instead of being blocked by the processing of previous order requests. Here we can place a queue between the web service and the order processing service. Then the two services will be decoupled and web service will not be blocked by order processing. Even if the number of requests increase this system can continue to receive requests since queue persists order requests and they can be processed asynchronously. If the system needs to be scaled up we only have to increase the number of consumers.

Can be used as an event bus in micro services.

RabbitMq

RabbitMq is the most widely used open source message broker. Therefore let’s focus on how to publish messages and consume from a rabbitMq message queue.

Publish messages to a queue

When it comes to publishing messages there’s an important concept as Exchange . Exchange receives messages from producers and routes those messages to the relevant queue.

You can think of an Exchange as a postman, who delivers mails from mail box to the correct address, in this scenario queues.

Exchange decides how this routing happens. For an Example whether to push messages to a specific queue or many queues. This is decided by the exchange type and the binding of the queue to Exchange. A binding is a relationship between an Exchange and a queue. Furthermore, we are specifying that the queue is interested in messages from this exchange. Binding takes a parameter as routing key to bind. There can be multiple queues bound to an exchange as explained in the below diagram.

queue binding

There are several exchange types as direct, topic, headers and fanout. Only direct, topic and fanout exchange types are discussed here.

Fanout exchange type

Fanout exchange broadcasts all the messages it receives to all the queues it knows, also known as dummy broadcasting.

Direct exchange type

A message goes to the queue whose routing key exactly matches the routing key of the message.

direct exchange src[https://www.rabbitmq.com/tutorials/tutorial-four-python.html]

According to above diagram we can see that Exchange X is of direct Exchange type and there are two queues bound to it. Q1 is bound to X with routing key orange and Q2 has two bindings with black and green routing keys.

Let’s assume a message with routing key orange is published to X . Then this message will be routed to Q1 . If a message with routing key black or green is published to X , then this will be routed to Q2 . Other messages will be discarded.

Topic exchange type

Topic exchange is similar to direct exchange but more flexible due to matching with wild cards and patterns.

And also routing key of messages and the queue should be a list of words delimited by dots. Ex: my.route.key

There are two special characters which can be used when configuring routing key.

* can substitute for exactly one word.

can substitute for exactly one word. # can substitute for zero or more words.

topic exchange src[https://www.rabbitmq.com/tutorials/tutorial-five-python.html]

Here X is of topic exchange type. Q1 is bound with routing key *.orange.* . Q2 is bound with routing key *.*.rabbit and lazy.# .

Let’s assume I’m publishing stupid.orange.rabbit to X . then this message will be routed to both Q1 and Q2 . Then let’s assume I’m sending message lazy.orange.elephant to X , again this is routed to both queues. My next message is lazy.yellow.rabbit , now this will only be routed to Q2 . Any message which doesn’t match with these routing key patterns will be discarded.

Implementation

We’ll implement an online order application similar to what I explained in use cases above. Users can place orders via our API and these order requests will be published to a queue to be consumed and processed asynchronously.

We’ll be using spring boot and Java 11 for the implementation.

Install rabbitMq locally

I’m using a docker image to run rabbitmq locally (https://www.rabbitmq.com/download.html). Once the container is up and running access the rabbitMq management console and set a username and a password.

Spring boot application

Given below is the folder structure that I used.

folder structure

Given below is the pom file with the relevant dependencies.

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-parent</artifactId>

<version>2.1.4.RELEASE</version>

<relativePath/> <!-- lookup parent from repository -->

</parent>



<groupId>com.demo.queue</groupId>

<artifactId>spring-boot-rabbit-mq-example</artifactId>

<version>1.0-SNAPSHOT</version>



<properties>

<java.version>11</java.version>

</properties>



<dependencies>

<dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-amqp</artifactId>

</dependency>

<dependency>

<groupId>com.fasterxml.jackson.core</groupId>

<artifactId>jackson-annotations</artifactId>

<version>2.9.10</version>

</dependency>

<dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-web</artifactId>

</dependency>

<dependency>

<groupId>org.springframework</groupId>

<artifactId>spring-test</artifactId>

<version>5.2.6.RELEASE</version>

</dependency>

</dependencies>

<build>

<plugins>

<plugin>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-maven-plugin</artifactId>

</plugin>

</plugins>

</build>

</project>

Create configs

First I create a connectionFactory in RabbitMqConfig.java file, which contains the configuration for making a TCP connection with rabbitMq server. Ex: username, password, rabbitMq host and port, ssl Protocols, virtual hosts etc.

private CachingConnectionFactory createConnectionFactory() {

com.rabbitmq.client.ConnectionFactory connectionFactory = new ConnectionFactory();

connectionFactory.setUsername(rabbitMQUsername);

connectionFactory.setPassword(rabbitMQPassword);

connectionFactory.setHost(rabbitMQHost);

connectionFactory.setPort(Integer.valueOf(rabbitMQPort));

return new CachingConnectionFactory(connectionFactory);

}

Then I declare the queue, exchange type and binding with the routing key as below.

@Bean

public Queue queue() {

return new Queue(rabbitQueueName);

}



@Bean

public TopicExchange exchange() {

return new TopicExchange(rabbitExchange);

}



@Bean

public Binding binding(Queue queue, TopicExchange exchange) {

return BindingBuilder.bind(queue).to(exchange).with(rabbitRoutingKey);

}

Then I define a message converter since we are dealing with objects here, then I define the template bean to publish messages to queue. This template takes the connection factory we created early and the message converter as arguments.

@Bean

public MessageConverter messageConverter() {

return new Jackson2JsonMessageConverter();

}



@Bean

public RabbitTemplate rabbitTemplate() {

RabbitTemplate rabbitTemplate = new RabbitTemplate(createConnectionFactory());

rabbitTemplate.setMessageConverter(messageConverter());

return rabbitTemplate;

}

Now I create the file Publisher.java which publish the given object to the queue using rabbit template defined above.

package com.demo.queue.queue;



import com.demo.queue.config.RabbitMqConfig;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.stereotype.Component;

import org.springframework.amqp.rabbit.core.RabbitTemplate;



@Component

public class Publisher {

@Autowired

private RabbitTemplate rabbitTemplate;



@Autowired

private RabbitMqConfig rabbitMqConfig;



public void publishMessaages(Object message){

rabbitTemplate.convertAndSend(rabbitMqConfig.getRabbitExchange(), rabbitMqConfig.getRabbitRoutingKey(), message);

}

}

Since our application is related to orders I create an Order object as below.

package com.demo.queue.beans;



public class Order {

private String accountId;

private double price;

private OrderStatus orderStatus;



public double getPrice() {

return price;

}



public OrderStatus getOrderStatus() {

return orderStatus;

}



public Order setOrderStatus(OrderStatus orderStatus) {

this.orderStatus = orderStatus;

return this;

}



public Order setPrice(double price) {

this.price = price;

return this;



}



public String getAccountId() {

return accountId;

}



public Order setAccountId(String accountId) {

this.accountId = accountId;

return this;

}



@Override

public String toString() {

return "Order{" +

"accountId='" + accountId + '\'' +

", price=" + price +

", orderStatus=" + orderStatus +

'}';

}

}

Given below is the OrderStatus enum which can be PENDING , SUCCESS or REJECTED .

package com.demo.queue.beans;

import com.fasterxml.jackson.annotation.JsonProperty;



public enum OrderStatus {

@JsonProperty("pending") PENDING("pending"),

@JsonProperty("success") SUCCESS("success"),

@JsonProperty("rejected") REJECTED("rejected");



private final String name;

OrderStatus(String value) { name = value;}



@Override

public String toString() { return this.name; }

}

Now I’m moving on to defining the controller, here post endpoint will publish order requests to the queue and send a success response to the user with the Order status set to PENDING , Therefore will not be blocked by order processing.

package com.demo.queue.controller;

import com.demo.queue.beans.Order;

import com.demo.queue.beans.OrderStatus;

import com.demo.queue.queue.Publisher;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.web.bind.annotation.RestController;

import org.springframework.web.bind.annotation.RequestMapping;

import org.springframework.web.bind.annotation.PostMapping;

import org.springframework.web.bind.annotation.RequestBody;





@RestController

@RequestMapping("/order")

public class OrderController {

@Autowired

Publisher queuePublisher;



@PostMapping("")

public Order placeOrder(@RequestBody Order order) {

queuePublisher.publishMessaages(order);

return order.setOrderStatus(OrderStatus.PENDING);

}

}

Given below are the properties used.

rabbitmq.host=localhost

rabbitmq.username=guest

rabbitmq.password=guest

rabbitmq.port=5672

rabbitmq.queue.name=myQueue

rabbitmq.exchange=myQueueExchange

rabbitmq.routing.key=myQueue.route.key

server.port=9000

Next I’ll add the main class as below.

package com.demo.queue;



import org.springframework.boot.SpringApplication;

import org.springframework.boot.autoconfigure.SpringBootApplication;



@SpringBootApplication

public class ExampleApplication {

public static void main(String[] args) {

SpringApplication.run(ExampleApplication.class, args);

}

}

Now let’s run the application. and send a post request to the order endpoint and you’ll get the order object with the orderStatus set to PENDING as the response.

request and response

Now if you go to the rabbitMq management console and navigate to queues and select myQueue and select get Messages you’d see the message we sent is there. Next we’ll create a consumer class to consume messages from the queue.

Consume from a queue

First let’s create a listener class. We need this class to create listener configs later. I name this class as QueueConsumer since this is responsible for consuming messages and processing them according to our business logic.

package com.demo.queue.queue;



import com.rabbitmq.client.Channel;

import org.slf4j.LoggerFactory;

import org.slf4j.Logger;

import org.springframework.amqp.core.Message;

import org.springframework.amqp.rabbit.listener.api.ChannelAwareMessageListener;

import org.springframework.stereotype.Component;



@Component

public class QueueConsumer implements ChannelAwareMessageListener {

private static final Logger LOGGER = LoggerFactory.getLogger(QueueConsumer.class);



@Override

public void onMessage(Message message, Channel channel) {

if (message == null || channel == null) {

LOGGER.error("MQ -- The received message is null or channel is null {} {}", message, channel);

return;

}

LOGGER.info("Messages received from queue {}", message);

}

}

As you can see, I used ChannelAwareMessageListner interface since it’s aware of the Channel on which the message received, instead you can implement MessageListener interface as well.

Here inside onMessage method you can write the processing logic for the received messages. In this example I only logged the message.

Now let’s create the message listener bean in RabbitMqConfig.java file.

@Bean

public CachingConnectionFactory listenerConnectionFactory() {

return createConnectionFactory();

}



private CachingConnectionFactory createConnectionFactory() {

com.rabbitmq.client.ConnectionFactory connectionFactory = new ConnectionFactory();

connectionFactory.setUsername(rabbitMQUsername);

connectionFactory.setPassword(rabbitMQPassword);

connectionFactory.setHost(rabbitMQHost);

connectionFactory.setPort(Integer.valueOf(rabbitMQPort));

return new CachingConnectionFactory(connectionFactory);

}



@Bean

public SimpleMessageListenerContainer messageListenerContainer() {

SimpleMessageListenerContainer container = null;

container = new SimpleMessageListenerContainer();

container.setConnectionFactory(listenerConnectionFactory());

container.setQueueNames(rabbitQueueName);

container.setMessageListener(messageQueueListener);

return container;

}

We are passing an instance of the QueueConsumer class to the message listener config bean. Now the final RabbitMqConfig.java file will look like below.

package com.demo.queue.config;



import com.demo.queue.queue.QueueConsumer;

import com.rabbitmq.client.ConnectionFactory;

import org.springframework.amqp.core.Queue;

import org.springframework.amqp.core.TopicExchange;

import org.springframework.amqp.core.Binding;

import org.springframework.amqp.core.BindingBuilder;

import org.springframework.amqp.rabbit.connection.CachingConnectionFactory;

import org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.context.annotation.Bean;

import org.springframework.beans.factory.annotation.Value;

import org.springframework.context.annotation.Configuration;

import org.springframework.amqp.support.converter.MessageConverter;

import org.springframework.amqp.support.converter.Jackson2JsonMessageConverter;

import org.springframework.amqp.rabbit.core.RabbitTemplate;



@Configuration

public class RabbitMqConfig {



@Autowired

private QueueConsumer messageQueueListener;



@Value("${rabbitmq.username}")

private String rabbitMQUsername;



@Value("${rabbitmq.password}")

private String rabbitMQPassword;



@Value("${rabbitmq.port}")

private String rabbitMQPort;



@Value("${rabbitmq.host}")

private String rabbitMQHost;



@Value("${rabbitmq.queue.name}")

private String rabbitQueueName;



@Value("${rabbitmq.exchange}")

private String rabbitExchange;



@Value("${rabbitmq.routing.key}")

private String rabbitRoutingKey;



public String getRabbitRoutingKey() {

return rabbitRoutingKey;

}



public String getRabbitExchange() {

return rabbitExchange;

}





@Bean

public CachingConnectionFactory listenerConnectionFactory() {

return createConnectionFactory();

}



private CachingConnectionFactory createConnectionFactory() {

com.rabbitmq.client.ConnectionFactory connectionFactory = new ConnectionFactory();

connectionFactory.setUsername(rabbitMQUsername);

connectionFactory.setPassword(rabbitMQPassword);

connectionFactory.setHost(rabbitMQHost);

connectionFactory.setPort(Integer.valueOf(rabbitMQPort));

return new CachingConnectionFactory(connectionFactory);

}



@Bean

public SimpleMessageListenerContainer messageListenerContainer() {

SimpleMessageListenerContainer container = null;

container = new SimpleMessageListenerContainer();

container.setConnectionFactory(listenerConnectionFactory());

container.setQueueNames(rabbitQueueName);

container.setMessageListener(messageQueueListener);

return container;

}



@Bean

public Queue queue() {

return new Queue(rabbitQueueName);

}



@Bean

public TopicExchange exchange() {

return new TopicExchange(rabbitExchange);

}



@Bean

public Binding binding(Queue queue, TopicExchange exchange) {

return BindingBuilder.bind(queue).to(exchange).with(rabbitRoutingKey);

}



@Bean

public MessageConverter messageConverter() {

return new Jackson2JsonMessageConverter();

}



@Bean

public RabbitTemplate rabbitTemplate() {

RabbitTemplate rabbitTemplate = new RabbitTemplate(createConnectionFactory());

rabbitTemplate.setMessageConverter(messageConverter());

return rabbitTemplate;

}

}

Now if you re run the application and again send a post request to our order endpoint you could see that our application consumes messages from the queue and logs like below.

2020–12–15 21:07:05.981 INFO 4942 — — [enerContainer-1] com.demo.queue.queue.QueueConsumer : Messages received from queue (Body:’{“accountId”:”1234",”price”:80.0,”orderStatus”:null}’ MessageProperties [headers={__TypeId__=com.demo.queue.beans.Order}, contentType=application/json, contentEncoding=UTF-8, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=myQueueExchange, receivedRoutingKey=myQueue.route.key, deliveryTag=1, consumerTag=amq.ctag-dPRyYKJmXcSLjitt3EAsig, consumerQueue=myQueue])

Conclusion

Message queuing plays a vital role in asynchronous communication. We discussed about the advantages and use cases of message queuing. RabbitMq is the most widely used open source message broker, and we looked at how publishing and consuming can be implemented with RabbitMq.