Pulsar Advantages Over Kafka
By: Javier Ramos
Tags: Kafka, Microservices, AWS, Big Data, Pulsar

Introduction

Recently, I’ve been looking at Pulsar and how it compares to Kafka. A quick search will show you that there is a current “war” between the two most famous open source messaging systems.

As a Kafka user, I do struggle with some of the issues with Kafka and I’m very exited about Pulsar. So finally, I managed to have some time to play around with it and I did quite a lot of research. In this article, I will focus on the Pulsar advantages and give you some reasons why you should consider it over Kafka. But let’s be clear, in terms of production usage, support, community, documentation, etc; Kafka clearly surpasses Pulsar, and I would only consider Pulsar if most of the advantages discussed in this article hold true to your use case. Let’s begin!

Kafka in a Nutshell

Kafka is the king of messaging systems. Created by LinkedIn in 2011, it has spread widely thanks to the support of Confluent who has released to the open source community many new features and add-ons such as Schema Registry for schema evolution, Kafka Connect for easy streaming from other data sources such as databases to Kafka, Kafka Streams for distributed stream processing, and most recently KSQL for performing SQL-like querying over Kafka topics and much more. It has also many connectors to many system, check Confluent Platform for more details.

Kafka is fast, easy to setup, extremely popular and can be used for a wide range or use cases. While Apache Kafka has always been friendly from a developer’s point of view, it has been something of a mixed bag operationally. So, lets’ review some the pain points of Kafka.

Kafka pain points

Scaling Kafka is tricky, this is due to the coupled architecture where brokers also store data. Spinning off another broker means it has to replicate the topic partitions and replicas, which is time consuming.

this is due to the coupled architecture where brokers also store data. Spinning off another broker means it has to replicate the topic partitions and replicas, which is time consuming. No native multi-tenancy with complete isolation of tenants .

Storage can become quite expensive, and although you can store data for a long period of time, it is rarely used because of the cost implications.

It is possible to lose messages in case replicas are out of sync.

You must plan and calculate number of brokers, topics, partitions and replicas ahead of time (that fits planned future usage growth) to avoid scaling problems, this is extremely difficult.

Working with offsets could be complicated if you just need a messaging system.

Cluster re-balancing can impact the performance of connected producers and consumers.

MirrorMaker Geo replication mechanism is problematic. Companies such Uber have created their own solution to overcome these issues.

As you can see, most of the problems are related to the operational aspects. Although, it is relative easy to setup up, Kafka is difficult to manage and tune. Also, it is not quite as flexible and resilient as it could be.

Pulsar in a Nutshell

Pulsar was created by Yahoo in 2013 and donated to the Apache foundation in 2016. Pulsar is now an Apache top level project. Yahoo, Verizon, Twitter among other companies use it in production to process millions of messages. It has many features and it is very flexible. It claims to be faster than Kafka and hence cheaper to run. It aims to solve most of the pain points of Kafka making it easier to scale.

Pulsar is very flexible; it can act as a distributed log like Kafka or a pure messaging system like RabbitMQ. It has multiple types of subscriptions, several delivery guarantees, retention policies and several ways to deal with schema evolution. It also has a big list of features…

Multi-tenancy is built in, different teams can use the same cluster and be isolated. This solves many administration headaches. It supports isolation, authentication, authorization and quotas.

is built in, different teams can use the same cluster and be isolated. This solves many administration headaches. It supports isolation, authentication, authorization and quotas. Multi tier architecture : Pulsar stores all the topic data in a specialized data layer powered by Apache BookKeeper as data ledgers. Separation of storage and messaging solves many issues with scaling, rebalancing and maintaining the cluster. It also improves reliability and makes almost impossible to loose data. Also, when reading the data, you can connect directly to Bookeeper without affecting the real time ingestion. For example, you can use Presto to execute SQL queries on your topics, similar to KSQL but with the peace of mind that this will not affect real time data processing.

: Pulsar stores all the topic data in a specialized data layer powered by as data ledgers. Separation of storage and messaging solves many issues with scaling, rebalancing and maintaining the cluster. It also improves reliability and makes almost impossible to loose data. Also, when reading the data, you can connect directly to without affecting the real time ingestion. For example, you can use to execute SQL queries on your topics, similar to KSQL but with the peace of mind that this will not affect real time data processing. Virtual topics . Because of the n-tier architecture, there is no limitation on the number of topics, topics and its storage are decoupled. You can also create non persistent topics.

. Because of the n-tier architecture, there is no limitation on the number of topics, topics and its storage are decoupled. You can also create non persistent topics. N-tier storage . One problem of Kafka, is that storage can become expensive. So it is rarely used to store “cold” data and messages are often deleted.Apache Pulsar, through Tiered Storage, can automatically move older data to Amazon S3, or any other deep storage system; and still present a transparent view back to the client; the client can read from the start of time just as if all of the messages were present in the log.

. One problem of Kafka, is that storage can become expensive. So it is rarely used to store “cold” data and messages are often deleted.Apache Pulsar, through Tiered Storage, can automatically move older data to Amazon S3, or any other deep storage system; and still present a transparent view back to the client; the client can read from the start of time just as if all of the messages were present in the log. Pulsar Functions . Easy to deploy, lightweight compute process, developer-friendly APIs, no need to run your own stream processing engine like Kafka.

Easy to deploy, lightweight compute process, developer-friendly APIs, no need to run your own stream processing engine like Kafka. Security : It has a built in proxy, multi tenant security, plug-able authentication and much more.

: It has a built in proxy, multi tenant security, plug-able authentication and much more. Fast Re-balancing . Partitions are split into segments that are easy to re balance.

. Partitions are split into segments that are easy to re balance. Server side de-duplication and dead lettering . No need to do this in the client, also de duplication can be done during compaction.

. No need to do this in the client, also de duplication can be done during compaction. Built in Schema registry . Supports multiple strategies and it is very easy to use.

. Supports multiple strategies and it is very easy to use. Geo replication and built in discovery. It is very easy to replicate your cluster to multiple regions.

It is very easy to replicate your cluster to multiple regions. Integrated load balancer and Prometheus metrics .

. Multiple integrations : Kafka, RabbitMQ and much more.

: Kafka, RabbitMQ and much more. Support for many programming languages such GoLang, Java, Scala, Node, Python…

Clients do not need to be aware of shards and data partition, this is done transparently on the server side.

List of features: https://pulsar.apache.org/

As you can see, Pulsar is has lots of interesting features.

Pulsar Hands-on

It is quite easy to get started with Pulsar. Make sure you have the JDK installed!

Download Pulsar and unzip it:

$ wget https://archive.apache.org/dist/pulsar/pulsar-2.6.1/apache-pulsar-2.6.1-bin.tar.gz

2. Download the connectors(optional):

$ wget https://archive.apache.org/dist/pulsar/pulsar-2.6.1/connectors/{connector}-2.6.1.nar

3. After you download the nar file, copy the file to the connectors directory in the pulsar directory

4. Start Pulsar!

$ bin/pulsar standalone

Pulsar provides a CLI tool called pulsar-client that we can use to interact with the cluster.

To produce a message:

$ bin/pulsar-client produce my-topic --messages "hello-pulsar"

To read a message:

$ bin/pulsar-client consume my-topic -s "first-subscription"

Akka Streams Example

As a client example, lets use Pulsar4s with Akka!

First we need to create a Source to consume the stream of data, all that is required is a function that will create a consumer on demand and the message id to seek:

val topic = Topic("persistent://standalone/mytopic")

val consumerFn = () => client.consumer(ConsumerConfig(topic, subscription))

Then, we pass consumerFn function to create the source:

import com.sksamuel.pulsar4s.akka.streams._

val pulsarSource = source(consumerFn, Some(MessageId.earliest))

The materialized value of the Akka source is an instance of Control , an object which provides a method 'close' which can be used to stop consuming messages. Now, we can process the data with Akka Streams as usual.

To create a sink:

val topic = Topic("persistent://standalone/mytopic")

val producerFn = () => client.producer(ProducerConfig(topic)) import com.sksamuel.pulsar4s.akka.streams._

val pulsarSink = sink(producerFn)

Full Example taken from Pulsar4s:

Pulsar Function Example

Pulsar functions process messages from one or more topics, transform it and output the result to a different topic:

You can choose between two interfaces to write your functions:

Language-native interface : No Pulsar-specific libraries or special dependencies required. You cannot access the context. Only Java and Python are supported.

: No Pulsar-specific libraries or special dependencies required. You cannot access the context. Only and are supported. Pulsar Function SDK: Available for Java/Python/Go and provide more functionality including access to the context object.

Using language native interface is very easy, you can just write a simple functions that transform the message:

def process(input):

return "{}!".format(input)

This simple function written in Python just adds an exclamation point to all incoming strings and publishes the resulting string to a topic.

To use the SDK you need to import the dependencies, for example in Go we would write:

package main



import (

"context"

"fmt"



"github.com/apache/pulsar/pulsar-function-go/pf"

)



func HandleRequest(ctx context.Context, in []byte) error{

fmt.Println(string(in) + "!")

return nil

}



func main() {

pf.Start(HandleRequest)

}

To publish the Serverless function and deploy it to the cluster, we use pulsar-admin CLI, in case of Python we would use:

$ bin/pulsar-admin functions create \

--py ~/router.py \

--classname router.RoutingFunction \

--tenant public \

--namespace default \

--name route-fruit-veg \

--inputs persistent://public/default/basket-items

A great feature of Pulsar Functions is that you can set the delivery guarantee when you publish the function:

$ bin/pulsar-admin functions create \

--name my-effectively-once-function \

--processing-guarantees EFFECTIVELY_ONCE

You have the following options:

Pulsar functions run on your Pulsar cluster so you don’t have to manage their deployments as opposed to Kafka Streams applications.

Pulsar Advantages

Let’s review the main advantages over Kafka:

More features : Pulsar Functions, multi tenancy, schema registry, n-tier storage, multiple consumption modes and persistence modes, etc.

: Pulsar Functions, multi tenancy, schema registry, n-tier storage, multiple consumption modes and persistence modes, etc. More flexibility : 3 types of subscriptions (exclusive, shared and failover), You can listen to multiple topics on one subscription. Durability options: non persistent(fast), persistent, compacted(only last key per message). You can choose delivery guarantee, it has server side de duplication and dead lettering. Many retention policies and TTL.

: 3 types of subscriptions (exclusive, shared and failover), You can listen to multiple topics on one subscription. Durability options: non persistent(fast), persistent, compacted(only last key per message). You can choose delivery guarantee, it has server side de duplication and dead lettering. Many retention policies and TTL. No need to define your scaling needs ahead of time.

Supports Queuing and Streaming . So it can act like RabbitMQ or Kafka.

. So it can act like RabbitMQ or Kafka. It scales better because storage is separated from the brokers. Re balancing is faster and more reliable.

because storage is separated from the brokers. Re balancing is faster and more reliable. Easier to operate : Thanks to the de coupling and the n-tier storage. Also the admin REST API is great.

: Thanks to the de coupling and the n-tier storage. Also the admin REST API is great. SQL integration with Presto that queries directly the storage without affecting the brokers.

that queries directly the storage without affecting the brokers. Cheaper storage thanks to the n-tier automated storage options.

thanks to the n-tier automated storage options. Faster : Many benchmarks has show better performance for a wide range of scenarios. Pulsar claims to have lower latency and better scaling capabilities. However, this is being challenged by Confluent, so take this with grain of salt and do your own benchmarks.

: Many benchmarks has show better performance for a wide range of scenarios. Pulsar claims to have lower latency and better scaling capabilities. However, this is being challenged by Confluent, so take this with grain of salt and do your own benchmarks. Pulsar Functions bring Serverless computing to your messaging platform. No need to manage the deployments.

bring Serverless computing to your messaging platform. No need to manage the deployments. Integrated Schema registry supporting easy schema evolution

supporting easy schema evolution Integrated load balancer, and Prometheus metrics.

Geo replication works better and it easier to setup. Pulsar also has built in discover-ability.

works better and it easier to setup. Pulsar also has built in discover-ability. No limit on the number of topics you can create.

Compatible with Kafka, easy to integrate.

Pulsar Cons

Pulsar is not perfect, Kafka is popular for a reason, it does one thing and it does it well. Pulsar tries to tackle too many fields and fails to exceed on any of them. Let’s summarize some of the problems with Pulsar:

Popularity : Pulsar is not as popular. It lacks support, documentation and real world usage. This is a major problem for big organizations.

: Pulsar is not as popular. It lacks support, documentation and real world usage. This is a major problem for big organizations. It required more components because of the n-tier architecture: Bookkeeper .

No proper support for streaming applications within the platform. Pulsar Functions are not the same as Kafka Streams, they are much simpler and are not meant for real time stream processing. You cannot do stateful processing.

Less plugins and clients compared to Kafka. Also, less people available with Pulsar skills, it needs to be learned in house.

It has less support in the cloud. Confluent has a managed cloud offering.

Confluent, has put a comparison of between Pulsar and Kafka where you can go more into details. This blog also answers some of the questions regarding Kafka vs Pulsar, but be aware they may biased.

Pulsar Use Cases

Pulsar can be used for a wide range of use case:

Pub/sub queue messaging

Distributed Log

Event Sourcing ledge for permanent event storage

Microservices

SQL Analytics

Serverless functions

When you should consider Pulsar

You need both queues like RabbitMQ and stream processing like Kafka.

You need easy geo-replication.

Multi tenancy is a must have and you want to secure the access for each of your teams.

You need to persist all your messages for a long time and you don’t want to off load them to another storage.

Performance is critical for you and your benchmarks have shown that Pulsar provides lower latency and higher throughput.

You run on-prem and you don’t have experience setting up Kafka but you have Hadoop experience.

Note that if you are in the cloud, consider cloud based solutions. Cloud providers have different services that cover some of the use cases. For example, for queue messaging cloud providers provide many services like Google pub/sub. For a distributed log, you have Confluent cloud or AWS Kinesis. Cloud providers also provide really good security. The advantage of Pulsar is that provides many features in a single platform. Some teams may use it as a messaging system for microservices while other as a distributed log for data processing.

Conclusion

I’m a big fan of Kafka that’s why I’m so interested in Pulsar. Competition is good, it drives innovation.

Kafka is a mature, resilient, and battle-tested product used all over the world with great success. I cannot image any company without it. However, I do see Kafka a victim of its own success, the huge growth has slowed down feature development, since they need to support so many big players. Important features like removing ZooKeeper dependency are taking too long. This has created room for tools such Pulsar to thrive; fixing some of the issues with Kafka and adding many more features.

However, Pulsar is still quite immature and I would be careful before introducing into production. Perform analysis, do benchmarks, research and write proof of concepts before incorporating Pulsar into your organization. Start small, do a proof of concept before migrating from Kafka and measure the impact before deciding on performing a full migration.