How Events Will Fail You
By: Christopher Galtenberg
Tags: Event Sourcing, Event Architecture, Event Driven Architecture, Kafka

tl;dr — if you have to depend on events, scan the catalog and 1–2–3 below

Events have become a fixture of modern software solutions, whether it’s the no-longer-trendy Event Sourcing model or the also-blockchain-like but typically client-focused CRDT, discrete units of computing state (whether termed “events” or not) offer a beguiling manner of keeping state of systems.

For those needing a quick primer, see the always reliable Martin Fowler starting here https://martinfowler.com/articles/201701-event-driven.html (don’t forget to come back).

When tasked with building what would become the graph representation of our new generation of cloud at IBM, one material we had in abundance was events, flying around from all corners. Control plane, supporting units, orchestration services — almost all systems published manifold, bite-sized data updates to multiple streams. It was assumed that given a wide enough butterfly net and a long enough replay log, we could assemble a coherent view of the current state of the world. There were enough success stories out in the field around services like Kafka and ElasticSearch for high-volume event processing and indexing.

Could we build a view of our cloud using just events? Most engineers presumed so, awed by the sheer quantity of available data.

But something just didn’t add up — many somethings, in fact. It started with the “moment zero” or T0 problem: given only events, on startup we certainly don’t know the state of the world — we have to find that somewhere / somehow. And if we already have to solve for gathering the state of the world, (and could theoretically lean on that whenever we need), events immediately place themselves at a disadvantage, a lesser tier of importance.

So we would at least leverage events for real-time updates — probably the ideal purpose. Yet over time, we’d get reports that our data was not in sync with the source systems-of-record. This is an absolutely intolerable state for our systems to be in, especially when we provide the views of objects for support, operations, SREs, those who can’t reach lower-level systems. What were we doing wrong? We had a correct T0 state and processed all events. And it wasn’t sufficient.

Next step, we’d use the sync processes that built our T0 view of the world on a repeated, 30-minute basis. If we were ever out-of-sync, we wouldn’t be for long. Yet we’d still hit multiple cases where the event system sent objects that didn’t match the source systems-of-record, e.g. startup events for objects that ultimately failed to provision, or events signaling an object’s pending deletion that were never finalized on the source system-of-record. Ultimately we had to deprioritize events (certain decisions based on events) to resolve this.

The trouble with events didn’t stop there. One of the systems we were required to incorporate into our view of the world was itself populated by events. It had an even more sophisticated model than ours — it wouldn’t just rely on event data; upon receiving an event it would contact the originating, authoritative system and ask for the current state of the object that fired the event. Even better, right? Frequent source system synchronization at the ideal granular level. And yet… we found it was repeatedly out-of-sync with the source system. Wildly. Frequently. This system spawned its own events to us, and we learned to distrust what it had to say.

As our system (which had deprioritized events) got more reliable, we became the primary reporter of out-of-sync issues for this other event-based system. We tracked down several root causes ourselves, and built up documentation to speed investigations in the future.

We’ll never be able to live without events and maintain our stance as a real-time view of multiple systems-of-record — we simply can’t synchronize every minute with every database. So we have to continually be on the lookout for failure patterns we’ve seen in the past and continue to encounter.

Below is our catalog of situations discovered while processing events or leaning upon event-oriented systems — a list of how events will ultimately fail your system, if you don’t monitor and guard against these conditions.

At the Source — Where Events Originate

Exceptions in Code: Events are typically programmed to be sent after a non-trivial change in the source system — a change which often involves communicating to other services. This logic is often multi-step, and can result in one or more exception paths in code. When an exception fires, it often leaves the function before an event would have been sent.

Events Not Sent for all Changes: Events are programmed to be sent after certain conditions are met — creates, deletes, major lifecycle changes — not necessarily when any update happens on the authoritative system. This results in some fields being correct and some incorrect on event-processing systems.

Events Not Prioritized by Developers and Managers: Occasionally we discovered that expected events didn’t arrive for a single, painful reason — they weren’t even programmed in. When getting services stood up is priority one, priority twos sometimes never make it up the backlog. Or event logic is given short shrift in developer care and testing, with missed cases that are obvious on retrospective code review.

At the Source Pipe — Where Events are Handed to the Event Bus

Event Bus Unreachable: Due to partitions in a cluster or general instability of the microservice that provides the entry point to the event bus, there can be failures in events leaving the source system. These failures can be intermittent or long-running (intermittent failures can be more insidious and hard to detect).

In the Pipe — the Event Bus Itself

Handoff to Central Bus: Similar partitions or instability, or resource exhaustion due to event volume or elsewhere in the cluster, can cause events to be lost in the pipe.

Central Bus Service Unavailable: Outages of critical infrastructure, even those with multi-zone and failover capabilities, can happen periodically, resulting in total blackout and loss of event data. For regional & zonal network and DNS outages, we’ve had hours of gaps.

At the Destination — When Receiving Events

Mismatched Expectations: Channel and queue names and authorization secrets can be changed without notice; the poorer the service ownership (sometimes a factor of org size), the likelier you won’t get the memo. Similar mismatches can occur with data payloads, as type names and schemas can make it seem as though old sources of data have vanished and new ones have appeared. Depending on your ingestion policy, you may or may not be able to deal with schema changes.

Overflow: Depending on the volume of data on the event bus and the efficiency of your processing logic, you may find that you cannot process events at the rate they arrive. There may be backpressure mechanisms, but depending on your configuration and code, overflow or other general error conditions can affect uptime of your processing service and the processing of event data.

Any of these (along with more typical mistakes like neglecting proper time order) can result in missed and misprocessed events, causing your view of state to fall out-of-sync with the reality of source systems-of-record, resulting in painful user reports and troubleshooting sessions.

So what solutions are recommended?

1) Leverage existing monitoring of systems along the event bus: i.e., don’t just monitor your own systems. If outages are discovered along the bus that produces the events you rely on, make sure you are notified. Then build resilience for outages that occur more than once.

2) Treat event outages like database outages: Related to point (1), think about why systems that connect to a database, rather than rely on event-processing, simply seem more reliable (and are more reliable). After all, isn’t running a database much like event-processing, receiving data and commands in serial, storing time-ordered logs which can be replayed? The reason databases are more reliable is because database outages are fatal — they cause broader software outages. If a database can’t process records, it probably can’t serve queries, and all software comes to a halt. Forms that try to send data break. Users know their data probably wasn’t received, and will have to try again later. SREs are paged, root causes are triaged and repaired, and service is restored. Does this happen for event systems? Are outages along the event bus treated as severely as database outages? If not, then it’s guaranteed that when purported less-severe outages occur (i.e., the catalog above), you will simply be without vital data for that period of time, with no hope of discovering what you missed.

3) Re-sync outside of events, frequently: the way to survive outages that will happen is the same solution that provides your T0 view of the world: have the ability to resync with source systems-of-record. Yes it’s another major piece of architecture that needs to perform reliably, but if you are investing in software, invest in the accuracy of your data. Build re-sync mechanisms that can crawl entire APIs and even whole databases, while not affecting performance of those source systems (lest you find your vital access revoked).

Events as a silver bullet frequently contain a higher percentage of lead—and they fly around by the millions. Armor up your software, your team, and your service ownership practices.