Go with the flow: the power of event-driven microservices
By: Virginia Backaitis
Tags: Data Flow Diagram, Event Driven, Orchestration, Messaging, Microservices

Go with the flow: the power of event-driven microservices

Guest post by Jonathan Schabowsky, Field CTO, Solace

In almost all major industries, there is a fast-growing interest in microservices to allow rapid, regular and reliable delivery of complex applications. But microservices alone are not enough. To maximize the potential of microservices, they need to be underpinned by event-driven architecture to take data from being simply ‘at rest’ to being fully ‘in motion’ to increase application performance.

This article provides a valuable checklist and explains how only through event-driven microservices will businesses realize the full promise of their business applications.

In a recent O’Reilly study of software engineers and technical professionals, more than three-fifths (61%) of respondents have been using microservices for a year or more. The same report found finance and banking lead the way in using microservices — but many other industries are following suit including retail/eCommerce, telecoms, manufacturing, transportation & logistics and more.

Microservices architecture is definitely on the rise and beginning to be part of a number of current transformation projects, breaking down traditionally monolithic applications into self-contained, independently deployed services that are identified using domain-driven design.

Why not all microservices are created equal

The promise of microservices is enhanced agility, scalability, resiliency, and speed to deploy and maintain components. Unfortunately, not all microservice architectures unlock these values. The brittleness of distributed processing, ecosystem integration challenges, and a lack of service harmony represent three major barriers.

The real-world implementation of microservices is, as always, more complicated than first thought. This is in large part because of the fallacies of distributed computing — a set of false assumptions that programmers and architects make when they enter the world of distributed applications.

1. The Service Decomposition Paradox — the fallacies of distributed computing

The list of distributed computing fallacies was originally addressed in 1994 by L. Peter Deutsch and others at Sun Microsystems, and still holds true today. In particular there are five of the key fallacies/assumptions that hold special importance to microservices implementation: the network is reliable, latency is zero, the network is secure, transport cost is zero and the network is homogeneous.

The rule is that the smaller you make each microservice, the larger your service count, and the more the fallacies of distributed computing impact stability and user experience/system performance. This makes it mission critical to establish an architecture and implementation that minimizes latency while handling the realities of network and service outages.

New eventing/messaging tools required

The further challenge is that microservices, of course, require connectivity and data to perform their roles and provide business value. However, data acquisition/communication has been largely ignored, so much so that the tooling has severely lagged behind. We can see this with API management or gateway products that only support synchronous, request/reply exchange patterns. This only goes to exacerbate the challenges of distributed computing — they don’t have the ability to integrate with or acquire data from legacy systems.

At the same time, eventing/messaging tools have also been stuck in the antiquated, non-agile world, being incompatible with many of the guiding principles of microservices such as DevOps and self-service.

But it is eventing/messaging that best deals with the idiosyncrasies of distributed computing and is the key to unlocking the potential of microservices architecture.

2. The Integration Conundrum — the challenges of enabling data flow between microservices in a legacy-based environment

As services become smaller and their purpose more singular, the potential for reusability increases, but that’s contingent on the ability for services to collaborate. Acquiring data for greenfield systems is easy, but microservices almost always come as a side effect of digital transformation, modernization or the need to build new capabilities at a more rapid pace. So developers are almost always dealing with an ecosystem of legacy systems — some will be modernized, others will remain as is for the foreseeable future.

Here’s the disconnect — most existing systems live on premises, while microservices live in private and public clouds. The ability for data to transit the often unstable and unpredictable world of wide area networks (WANs) is tricky and time consuming. Then we need to factor for the emergence of and specialization caused by IoT, mobile devices and big data. The volume and variety of these systems results in a very large upfront risk to microservices initiatives.

Check out these disconnects — How many do you have?

When you add it all up, there are mismatches everywhere you look: updates to legacy systems are slow, but microservices need to be fast and agile. Legacy systems use old communication mediums, but microservices use modern open protocols and APIs. Legacy systems are nearly always on premise and at best use virtualization, but microservices rely on clouds and IaaS abstraction. IoT systems use highly specialized protocols, but most microservices APIs and frameworks do not natively support them. Finally, mobile devices may use REST but also require asynchronous communication, but most API gateways only support synchronous RESTful interactions.

Factoring all of the above, the case becomes clear — organizations need an event-driven architecture to link all these legacy systems vs. microservices mismatches.

3. Point-to-point Microservice Orchestration and tightly coupled systems that hamper agility and scalability

As I described earlier, the smaller the service, the less value it discretely offers the end user — value comes from orchestration. Historically, orchestration was handled by a central component like BPEL engines or ESBs, or, these days, API gateways.

Orchestration is a good description of what is required. Let me explain. Composers create scores containing sheets of music which will be played by musicians with differing instruments. Each score and its musician are like a microservice. In a complex symphony with a hundred musicians playing a wide range of instruments — like any enterprise with complex applications — far more orchestration is required.

Microservices don’t miss a beat

So an example of microservices orchestration is when a microservice performs a series of steps within the code. The input or output of a microservice is a data event which has domain significance. But what is key is that since the microservice is merely producing an event, it does not have knowledge of if or when it will be processed. Other services must register their interest in the event or set of events and react accordingly. This all needs microservices orchestration.

The future of microservices is event-driven

Event-driven maximizes agility as it liberates data from being ‘at rest’ (for example, stuck in a database behind an API) to being ‘in motion’ — consumable in real-time as events happen throughout the business. Choosing the right eventing or messaging platform is one of the most critical steps in the path to realizing the vast benefits of microservices.

Microservices per se are not enough. Gartner has identified this need as far back as 2018 when it spotlighted ‘event-driven’ as one of its top strategic technology trends, citing “Digital business is event-driven, so organizations need to invest in event-centric design practices and technologies to exploit digital business moments.”

Event-driven platform support needed to maximize microservice success

Take the example of a banking organization implementing a marketing campaign for a new bank-offered credit card. The use of event-driven architecture makes it easy to consume account creation events and market the bank’s credit card to the new customer based off the account metadata — such as opening balance, address, job and more.

In the same way other businesses can, by combining event-driven architecture and microservices, developers can build distributed, highly scalable, available, fault-tolerant, and extensible systems. These systems can consume, process, aggregate, or correlate extremely large amounts of events or information in real-time.

Modern events platforms should be able to be deployed in every cloud and platform as a service and easily span WANs. They should also support DevOps automation and provide an almost completely self-service experience for developers. Businesses need to be able to go with the flow in a rapidly-evolving world. They can only do this if their business applications enable them to.

That’s the essential role of event-driven microservices.