Event-Driven API with Apicurio Registry, Spring Boot, Apache Avro and Kafka
Tags: API, Spring Boot, Kafka, Avro, Schemaregistry

As Michael Amundsen mentioned (thanks for allowing citing it) in his Moving from RESTful to EVENTful:

While RESTful systems focused on resources, EVENTful solutions focus on actions. And, more importantly, EVENTful solutions rely on asynchronous interactions. This opens up many more possibilities for building responsive, real-time solutions. That means an EVENTful design offers services the ability to request and respond in real time and to do it in a way that provides more flexibility in the way the data is shared, displayed, and mixed across devices and platforms. Mike Amundsen

While I’m still waiting for my print copy of his awesome Design and Build Great Web APIs: Robust, Reliable, and Resilient book I would like to describe my technical way to EVENTful with Apicurio Registry

Disclaimer: if you’re with Confluent, no worries! It works the same way with Confluent Schema Registry I would like to have independent opinion based on Open Source product though. And Apicurio Registry looks like a good match.

Update: you can use Confluent Registry as well and vice-versa. Both tools have their commercial version with first class support (see GA from RedHat or Confluent) or you can simply use it as it is with some minor limitations:

“The only limitation of CCL is that you cannot sell the Schema Registry as a service (say AWS Kafka SR). However, you can build a commercial application and use SR under the hood (without any royalty or something else)” Kai Waehner

If you still don’t have Docker installed, get it now. We need docker-compose to setup infrastructure:

version: "3"

services:

postgres:

image: postgres

environment:

POSTGRES_USER: apicurio-registry

POSTGRES_PASSWORD: password

registry:

image: apicurio/apicurio-registry-jpa

ports:

- 8081:8080

environment:

QUARKUS_DATASOURCE_URL: 'jdbc:postgresql://postgres/apicurio-registry'

QUARKUS_DATASOURCE_USERNAME: apicurio-registry

QUARKUS_DATASOURCE_PASSWORD: password

zookeeper:

image: 'bitnami/zookeeper:latest'

ports:

- '2181:2181'

environment:

- ALLOW_ANONYMOUS_LOGIN=yes

kafka:

image: 'bitnami/kafka:latest'

ports:

- '9092:9092'

environment:

- KAFKA_BROKER_ID=1

- KAFKA_LISTENERS=PLAINTEXT://:9092

- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092

- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181

- ALLOW_PLAINTEXT_LISTENER=yes

depends_on:

- zookeeper

Save this as a docker-compose.yml or download it here. Then execute

docker-compose up

Check that Apicurio Registry is up and running by opening http://localhost:8081/api

We’re using apicurio/apicurio-registry-jpa for simplicity in our example, but you can choose between:

Now it’s time to create sample Spring-Boot project:

Or you can simply download it from GitHub: tillias/spring-boot-kafka-apicurio.

Navigate to /src/main/resources/avro/schema and create sample Avro Schema:

{

"name": "Event",

"namespace": "com.github.tillias.spbka.schema.avro",

"type": "record",

"doc": "Avro Schema for Event",

"fields" : [ {

"name" : "name",

"type" : "string"

}, {

"name" : "description",

"type" : "string"

}, {

"name" : "createdOn",

"type" : "int",

"logicalType": "timestamp-millis"

}

]

}

With this schema in place we invoke code generation using avro-maven-plugin. Open pom.xml and add plugin into build section:

<build>

<plugins>

<plugin>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-maven-plugin</artifactId>

</plugin>

<plugin>

<groupId>org.apache.avro</groupId>

<artifactId>avro-maven-plugin</artifactId>

<version>${avro.version}</version>

<executions>

<execution>

<phase>generate-sources</phase>

<goals>

<goal>schema</goal>

</goals>

<configuration>

<sourceDirectory>${project.basedir}/src/main/resources/avro/schema</sourceDirectory>

<includes>

<include>**/*.avsc</include>

</includes>

<outputDirectory>${project.build.directory}/generated-sources/avro/schema</outputDirectory>

</configuration>

</execution>

</executions>

</plugin>

</plugins>

</build>

This will automatically compile your schemas and generate sources under target\generated-sources\avro\schema\com\github\tillias\spbka\schema\avro:

Schema is ready, now it’s time to integrate Apicurio Registry. Add following dependencies into your pom.xml:

<properties>

<avro.version>1.10.0</avro.version>

<apicurio.version>1.3.1.Final</apicurio.version>

</properties> <dependencies>

<dependency>

<groupId>org.apache.avro</groupId>

<artifactId>avro</artifactId>

<version>${avro.version}</version>

</dependency>

<dependency>

<groupId>io.apicurio</groupId>

<artifactId>apicurio-registry-utils-serde</artifactId>

<version>${apicurio.version}</version>

</dependency>

<dependency>

<groupId>org.jboss.resteasy</groupId>

<artifactId>resteasy-client</artifactId>

<version>4.5.6.Final</version>

</dependency>

<dependency>

<groupId>org.projectlombok</groupId>

<artifactId>lombok</artifactId>

<optional>true</optional>

</dependency>

We need resteasy-client until https://github.com/Apicurio/apicurio-registry/issues/702 will be fixed. According to Eric Wittmann (it seems he is project lead):

For now we’ll have to document the issue (users might need the RE rest client dependency as well, depending on their application). The old client interface that uses the JAX-RS classes has been deprecated — so the next major release should clean the last of this up.

Now let’s add a glue between Spring Boot, Apicurio Registry and Kafka using some spring boot magic. Add following into your application.yaml

First of all we’re telling that Kafka listener is available on 9092. Then we define value-deserializer and value-serializer provided by Apicurio.

spring.json.trusted.packages for consumer is mandatory, otherwise our generated sources for Event won’t be handled properly.

use-specific-avro-reader will allow us using generated sources directly in consumer instead of GenericData.Record. More about it later in this article.

producer artifact-id property can be either

ArtifactIdStrategy to return an artifact ID

to return an artifact ID GlobalIdStrategy to return a global ID

Please refer to the documentation (section 9.4. “Strategies to lookup a schema”) for more details. I’m using here io.apicurio.registry.utils.serde.strategy.TopicIdStrategy for simplicity.

Let’s push our schema to Apicurio Registry:

curl --location --request POST 'http://localhost:8081/api/artifacts' \

--header 'Content-Type: application/json; artifactType=AVRO' \

--header 'X-Registry-ArtifactId: events-value' \

--data-raw '{

"name": "Event",

"namespace": "com.github.tillias.spbka.schema.avro",

"type": "record",

"doc": "Avro Schema for Event",

"fields" : [ {

"name" : "name",

"type" : "string"

}, {

"name" : "description",

"type" : "string"

}, {

"name" : "createdOn",

"type" : "int",

"logicalType": "timestamp-millis"

}

]

}'

Notice following header:

X-Registry-ArtifactId: events-value

This is exactly where TopicIdStrategy plays important role:

The default strategy is TopicIdStrategy , which looks for Service Registry artifacts with the same name as the Kafka topic receiving messages.

I used curl for simplicity, and as Hugo Guerrero mentioned you can:

…also use the maven plugin to register your schema instead of the cURL POST call or event better, you can configure the serdes to automatically register it if it’s not found in the registry already Hugo Guerrero

More about it in “Strategies to return an artifact ID” section.

Verify that schema is successfully created by opening http://localhost:8081/ui/artifacts/events-value/versions/latest

We’re finally ready writing our first producer:

public class Topic {

public static final String NAME = "events";

} @Service

@Slf4j

@RequiredArgsConstructor

public class Producer { private final KafkaTemplate<String, Event> kafkaTemplate; public void send(Event payload) {

log.info("Producer sending message {} to topic {}", payload, Topic.NAME);

this.kafkaTemplate.send(Topic.NAME, payload);

}

}

And consumer:

@Service

@Slf4j

@RequiredArgsConstructor

public class Consumer { @KafkaListener(topics = Topic.NAME, groupId = "${spring.kafka.consumer.group-id}")

public void consume(Event message) {

log.info("Consumer consumed message {} from topic {}", message, Topic.NAME);

}

}

I’m using couple of lombok annotations here: @Slf4j and @RequiredArgsConstructor.

Finally let’s add resource to trigger producer using HTTP POST:

@RestController

@RequiredArgsConstructor

@Slf4j

@RequestMapping(value = "/kafka")

public class Resource { private final Producer producer; @PostMapping(value = "/publish")

public void publish(@RequestBody Event event) {

log.info("REST Controller has received entity: {}", event);

this.producer.send(event);

}

}

We’re ready to start our spring-boot application and verify that everything is working fine:

curl --location --request POST 'http://localhost:8080/kafka/publish' \

--header 'Content-Type: application/json' \

--data-raw '{

"name": "some name",

"description": "some description",

"createdOn": 1603214435

}'

You will see something like this in your console:

020-10-20 19:20:44.354 INFO 29016 --- [ctor-http-nio-3] com.github.tillias.spbka.Producer : Producer sending message {"name": "some name", "description": "some description", "createdOn": 1603214435} to topic events

2020-10-20 19:20:44.393 INFO 29016 --- [ntainer#0-0-C-1] com.github.tillias.spbka.Consumer : Consumer consumed message {"name": "some name", "description": "some description", "createdOn": 1603214435} from topic events

Now you can evolve you async API by enabling Validity Rules, Compatibility Rules or built-in versioning:

Backward- , Full-, Forward-compatibility, automatic validation… So many good stuff out of the box!

In real-world scenarios you can trigger Apicurio Registry via REST API instead of manual upload: https://www.apicur.io/registry/docs/apicurio-registry/assets-attachments/registry-rest-api.htm

I hope you enjoyed the stuff. Feel free to ping me on LinkedIn or twitter.

You can find all code from this article available on GitHub: tillias/spring-boot-kafka-apicurio.

This is copy of my original article located here: https://tillias.wordpress.com/2020/10/20/event-driven-api-with-apicurio-registry-spring-boot-apache-avro-and-kafka/