Solution Architects Guide to Event-Driven Integration
By: Martin Arndt
Tags: Cloud Native Application, Event Driven Architecture, Solution Architect, Microservices, Integration

Photo by Crystal Kwok on Unsplash

As a solution architect, you need to keep an eye on technology trends and decide when is the right time for your organisation to get involved, and I would like to draw your attention to event-driven integration. This is an approach to integration that developers and architects are starting to recognise, the recent State of API Technology¹ report concluded that developers believe event-driven integration is the most important category in the integration strategy of their business.

Cloud Elements The State of Integration Report 2020

Event-driven integration is an evolution away from the traditional hub and spoke integration patterns like an ESB, which is becoming an anti-pattern² for agile cloud-native integration. The agile cloud-native applications require faster change than could typically be delivered using the centralised monolithic integration teams with their ESB from traditional vendors. Also, the cloud-native applications and distributed data silos have a challenge of real-time data integration where a change in one data system needs to be simultaneously reflected in multiple other systems. This is generally hard and expensive to achieve with synchronous REST-based API integration patterns, as development teams have to maintain that state (REpresentational State Transfer) in their system APIs³ as these systems are based on request and response so they have to poll for change or a specific processes set up to look for change.

I have also observed that COVID-19 is driving digital transformations⁴ and fuelling demand for real-time application integration and data movement, giving systems the ability to react to a state change in real-time. An event-driven integration pattern removes the need for clients to ask repeatedly if a given system has new data available (changed state), instead, it is built on an Publisher-Subscribe model⁵ where the client subscribes to events and the publisher will produce the event (data) and doesn’t care how many subscribers will consume this event. Event-driven integration architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

The event represents something that happened within a system, a user action or even an action caused by existing request/response interfaces. I have included a made-up example below of a COVID-19 test result which describes the test result, which includes the who, what, when and where of the event. For event-driven integration, there is no need to ask a given system for more information or to enrich the data any further, known as Event-Carried State Transfer⁶. The associated data will then be delivered to other systems that are subscribing to the event and multiple outcomes are possible in real-time.

Example Event based on a COVID-19 Test result

I will continue on with the COVID-19 testing result example, as the system that generates the testing event must publish to an event broker, and consumers subscribe to a topic and then different outcomes are possible. It can be consumed by a Contact Tracing System to initiate contact tracing; a data warehouse for important data analysis; the person who had the test can be notified of their test result; local health units can be alerted that there is an outbreak in their area and all this in real-time. This is extremely powerful and shows what is possible with event-driven integration, and these consuming applications can be small agile components that can be developed quickly to adapt and react to change which is critical in COVID-19 contact tracing for instance.

COVID-19 Event Broker example architecture

As with all cloud-native distributed systems like this sprawl and lack of visibility can become an issue as publishers and subscribers proliferate, this was a problem that API integration had at first as well and why API Management came into existence. This is being addressed in the event-based architecture world with AsynchAPI⁷ and CloudEvents⁸ which are open source-based initiatives to create industry standards for defining events and Asynchronous APIs. It is harder than RESTful API Management as that has one protocol of HTTP, compared to many options in the event-based world. This is a good initiative and backed by some large players in this market and I feel that open source communities need this to be successful.

If you want to get started with event-based Integration there are many options to get you started, and it can be confusing knowing which one is right for you. I will try to give some guidance but I do not have a vendor agenda and will give my impartial advice based on my experience using the different solutions. If your business is a digital native and cloud-based on one cloud provider then it is easy as they all have their event brokers that work on their cloud and integrate well with there various PaaS offerings, making it easy to create event-driven applications and even event-driven Serverless functions. I do not want to rank or rate the Cloud event broker but will simply list Azure Event Grid, AWS Event Bridge and Google Cloud Pub/Sub; they all have their strengths and weaknesses but are a great way to get started with Event-based Integration.

However, if you have a multi-cloud strategy or have on-premise data centres then you will want an event broker that spans all of these and you then need to have a look at Confluent Kafka and Solace PubSub+, which are robust event brokers that can handle large volumes of data but are actually very different animals under the hood. The differences come from their origins as Kafka⁹ was created by LinkedIn and is a log-based broker based on handling large volumes of LinkedIn posts with eventual consistency. Whereas Solace Pubsub+ broker¹⁰ has origins in investment banking where it had to handle large volumes of financial transactions with guaranteed delivery, it can handle impressive throughput and can buffer data but doesn’t store it all like Kafka.

There is another option and that is to go at it alone and go open source with no support, just do it yourself with a team of skilled engineers. Then you have a few options like Apache Kafka, NATS, Apache Pulsar, Solace Pubsub+ and if you really want some choices then have a look at CNCF streaming and messaging¹¹ category and there are more than a few options in there to get you started.

In summary, I believe that this approach is another useful tool in the solution architect’s tool belt, so that you don’t always have to use the hammer (ESB traditional Hub and Spoke approach). Event-driven integration should coexist with API Driven Integration and can enable faster and more agile integration with the distributed data silos present within the business. In traditional system integrations, we have to handle the complexities and limitations of the systems of record and that is not always a quick process. Whereas if we can identify business events like Purchase Orders, Test Results, Bookings, Login Events, Customer Interactions, etc then we can start to look at changing the business processes associated with them and make it easier for others to do business with us and not be limited by the complexity of our systems of record.