Event Data at Scale
By: Paul C
Tags: Database, Kafka, Data Architecture, Data, Event Data

Event Data at Scale

Creating data infrastructure for modern organisations

Photo by Mike Benna on Unsplash

As the data requirements placed on companies increases year-on-year there has to be a solid infrastructure and process in place to make sure that not only is data safe and secure, but also is captured in a meaningful way allowing it to become useful to a business in various different ways.

While this applies to nearly all modern companies, SaaS companies such as Beamery have to be very astute in the way that data is captured, stored, and used. Event data is the life-blood of any company wishing to be around for the foreseeable future as it allows them to make informed decisions about the business and what customers are doing. In this article we will discuss the key issues that need to be tackled to make sure that event data is captured and used in the correct way.

We will delve into the specifics of the technologies that Beamery settled on, and why, at the end of this article, but first we need to cover some important topics.

So, what are the main things that need to be considered when storing event data at scale? There are three main areas that should be considered and will lead to the correct selection of technologies used.

Security

Scalability

Usability

Event Data

Before we dive headlong into each of these topics it’s worth taking a step back and defining the kind of data that we are trying to capture. In simple terms event data can be broken down and defined as a piece of information about something very specific that happened at an exact point in time.

Examples of an event is a user of a web application clicking on a button, sending an email, or altering some data. Each of these events are very different in and of themselves, but when broken down they are all essentially the same thing, some time-series data that needs to be captured, analysed, and made sense of.

There are many different ways to capture events, and we will discuss those in following articles, but for now we will look at a more generalised view of what is important from a data perspective.

Security

Security in general is a big topic, especially data security with governmental rules as laid out in the General Data Protection Regulation (GDPR) directives that came into force on the 25th May 2018. GDPR puts a heavy emphasis on data security and not following through on a robust data security policy can result in large fines that some businesses might not be able to recover from.

When we talk about data security as a business we need to compartmentalise things. This is encapsulated by international standards such as ISO 27001 which asks a business who is responsible for specific data within an organisation.

In general the main considerations are about securing access to data in multiple ways. This encompasses ensuring that only the correct systems and people have physical access to hardware or cloud systems storing the data, making sure that data is encrypted when it is not in transit (encryption at rest), or making sure that data is only ever transmitted over secure links (TLS, SSH etc.).

These are but a few considerations and there is not enough room within this post to go into fine detail, suffice to say data security is something that should be taken very seriously and researched fully. You have been warned!

Scalability

When we talk about scale in the technology industry it will normally be around making sure an API or some other user components can meet the demands made of them in a consistent and predictable way. Scaling data systems is just as important and it is not always immediately obvious what should be scaled and how.

We all know that to be able to service requests from a database we can scale that database horizontally by adding more nodes, increasing shards etc. But what of the systems that connect to large data stores? These will also need to scale just as much as the primary databases, without this processing data for business intelligence reports will become time consuming and lag behind events as they arrive. This essentially means that intelligence data on which business decisions are based will not be current, and worst case not relevant any longer. Being able to react to data changes quickly is extremely important.

Scaling considerations for connected systems will normally be focused on processing system, things such as message brokers, cold storage systems, data lakes etc. With these systems scaled to be able to cope with an increased event data rate it then becomes possible to process data in near real-time with the correct technology choices.

Usability

The concept of usability is generally referenced in relation to user facing interfaces in software applications or web browsers, but in this instance we are talking about how useable a piece of event data is. We must be careful in the definition of a schema for events data. A schema must be not only future-proof, but also contain fixed data points that make each event useful to derive information from when processed along with other events that have been collected.

All event data is time-series, that is we will have a number of data points associated with a specific point in time. There are a number of ways we can define a useful schema, but to do so we must know what we are trying to produce at the end of the event ingestion.

Things to consider are:

Are the proper identifiers attached to each event?

Do events need transforming or enriching to be more meaningful (location data, user data, etc.)?

What format should transformed event data be in for ease of processing (JSON, Avro, something else…)?

Can engineers easily interact with event ingestion systems?

Do engineers have the skillset now or will training be needed?

While these questions are not exhaustive they provide a guide that will result in a good schema and graceful systems design that can be used to generate useful intelligence data later on. This is of paramount importance.

Technology Selection

There are many different technologies that live within the data processing arena, from older well known systems based on Hadoop clusters to newer systems like Flink, ksqlDB, and others. The right selection of technology to use when processing your event data can depend on many different things, though probably the most important question you need to ask is; “what do you want to achieve?”.

Here are a few options that will likely need to be considered before any technology selection takes place.

How much data do we need to process now?

How much data will we need to process in one or two years?

Will we need to scale consistently over time or account for sudden jumps in event data volume?

Do we need a system that can operate at near real-time or are we happier working with a system that does batch processing at specific times?

Can the existing engineers in the organisation access the technology easily or are new skills required?

Do we need to hire or train staff?

Assuming that all questions have been answered in a satisfactory way it’s time to start actually selecting the technologies that are going to be used in your data processing pipeline. Generally there are five or six distinct areas to an event processing system, these are:

Producers — as the name gives away, these deal with event production. Event producers are probably the most varied group of components as they can encompass many different things. For example; a database generating a change stream, users clicking on web page buttons, an email being sent etc. The point is a producer in some form will generate events.

— as the name gives away, these deal with event production. Event producers are probably the most varied group of components as they can encompass many different things. For example; a database generating a change stream, users clicking on web page buttons, an email being sent etc. The point is a producer in some form will generate events. Collectors — Closely related to producers, a collector will take generated events and move them into some form of transient data store. In effect they act as a proxy to provide a common way to move events into the data pipeline. Producers and collectors are quite often embedded in the same component. How these are split up — or not — will depend very much on the events being captured.

— Closely related to producers, a collector will take generated events and move them into some form of transient data store. In effect they act as a proxy to provide a common way to move events into the data pipeline. Producers and collectors are quite often embedded in the same component. How these are split up — or not — will depend very much on the events being captured. Transient Storage — more often than not, the transient event storage will be some kind of message broker. This component allows data events to move as messages in a predictable way from the producers/collectors for processing. There are many different message brokers, or queue systems, on the market with each having their own way of handling data. The selection here will depend very much on the requirements of the pipeline being designed.

— more often than not, the transient event storage will be some kind of message broker. This component allows data events to move as messages in a predictable way from the producers/collectors for processing. There are many different message brokers, or queue systems, on the market with each having their own way of handling data. The selection here will depend very much on the requirements of the pipeline being designed. Consumers — consumers have the task of retrieving each new message — event — from the transient store and processing the data in some way. How the data is transformed will be dependent on what the event pipeline is designed to do. However, generally we find that events will be transformed into a common format for final storage. Transformation can also include enrichment which is where additional information can be added to the event to give it greater context. There can be many different types of consumers from small custom workers, to larger and more complex stream processing platforms.

— consumers have the task of retrieving each new message — event — from the transient store and processing the data in some way. How the data is transformed will be dependent on what the event pipeline is designed to do. However, generally we find that events will be transformed into a common format for final storage. Transformation can also include enrichment which is where additional information can be added to the event to give it greater context. There can be many different types of consumers from small custom workers, to larger and more complex stream processing platforms. Data Lake (optional) — before data is processed into the final schema to be stored and queried it might be desirable to maintain a data lake. That is, a storage area of the raw, unprocessed, events. This is useful to have as over time business needs change and different types of transforms might need to be applied to the raw data to gain different insights. A data lake could be a database or files in cold storage (cloud provider buckets and objects).

— before data is processed into the final schema to be stored and queried it might be desirable to maintain a data lake. That is, a storage area of the raw, unprocessed, events. This is useful to have as over time business needs change and different types of transforms might need to be applied to the raw data to gain different insights. A data lake could be a database or files in cold storage (cloud provider buckets and objects). Data Warehouse — the data warehouse is the final resting place for processed — and also if desired raw — events. That is not to say each event ends its journey here, but in general terms this is where events can be queried to produce useful business insights. As with other components in the pipeline there are many forms of storage, but generally we could say this is some kind of database.

The Beamery Event Data Pipeline

So, after all of this discussion what did Beamery actually settle on for an events pipeline? The first thing we did was to answer all of our questions over data volumes, ease of use, schema design etc. This narrowed the field down greatly to allow us to go more in-depth on a smaller set of technologies. For completeness here is a quick recap on some of the questions posed and our answers to them.