GAVRO — Managed Big Data Schema Evolution | by Gary Strange | Towards Data Science

Managing schema changes has always proved troublesome for architects and software engineers. Building a big-data platform is no different and managing schema evolution is still a challenge that needs solving. NoSQL, Hadoop and the schema-on-read mantra have gone some way towards alleviating the trappings of strict schema enforcement. However, integration developers, analysts and data scientists are still hindered by the amount of data wrangling they need to perform when extracting accurate insights from big-data.
Successful business’ grow and evolve at pace accelerating and amplifying the volatility of known data schemas. Datasets are not static and constantly evolving, so knowing what business-fact data represents in the current and historical periods of the business is crucial to making confident information insights.
Through this article and accompanying GitHub repo, I’ll demonstrate how you can manage schema evolution in a big-data platform using Microsoft Azure technologies.
This is an area that tends to be overlooked in practice until you run into your first production issues. Without thinking through data management and schema evolution carefully, people often pay a much higher cost later on.
Confluent.io (29th April 2020), Schema Evolution and Compatibility. https://docs.confluent.io/current/schema-registry/avro.html#schema-evolution-and-compatibility
As a writer, it's difficult to decide how to tell your story. Do I jump straight into the technical solution to satisfy the engineers looking for succinct examples or do I start with the why’s and motivations? So I’ll leave it up to the reader. If you want to jump straight into the technical example head to the GitHub repo. If you want the finer details, read on...
Serializing data to be stored in files to be analysed later, is fairly straight forward if consumers understand the schema that was used to write the data and the schema never changes.
But what happens if the schema evolves over time? It becomes a little more complicated.
When the write-schema evolves due to a new business requirement, consumers (readers) must understand when the new schema was introduced and the definition of the new schema to successfully de-serialize the data. Failure to comprehend the schema-change event will impact data processing pipelines and services will error as they fail to de-serialize the data.
There are a few solutions to this problem... (this is by no means an exhaustive list).
Release Co-ordination
The writer and the reader coordinate their backlogs and software releases. This may work well when the writer and the reader applications are developed and maintained by the same engineering team. However, it’s often the case that the writer and reader are working to different objectives and priorities across the enterprise. Temporally coupling independent team backlogs through strict interface dependencies is to be avoided as it inhibits agility and delivery velocity.
Schema Version Management
Versioning write-schemas enables forward and backwards compatibility management. Providing forward and backward compatibility de-couples backlogs and priorities, allowing engineering teams independent progression of their goals.
Versioning is generally discussed in the context of two distinct sub-topics.
Major — A major version change typically breaks interfaces and contracts between systems. A major schema change would typically inhibit readers from reading the data written by the new schema version. Forward and backward compatibility is difficult or impossible.
Minor — A minor version change is typically considered to be a low impact change. Forward and backward compatibility is often possible. Readers typically continue to operate as they previously did, successfully de-serialising data without progressing to the newest version of the schema.
Wouldn’t it be nice to build a data ingestion architecture that had some resilience to change? More specifically, resilience to schema evolution.
Below is the Azure architecture I’ll use to describe how schema evolution can be managed successfully.
Kafka’s Schema Registry provides a great example of managing schema evolution over streaming architecture. Azure Event Hubs, Microsoft’s Kafka like product, doesn’t currently have a schema registry feature. Events published to Event Hubs are serialised into a binary blob nested in the body of Event Hubs Avro schema (Fig.1). We will get into the details shortly, but essentially the published event data is schema-less, any down-stream readers need to de-serialise the binary blob by asserting a schema at read time. There are some clever-work-arounds1 that utilise Confluent’s schema-registry alongside Event Hubs. I will build on these suggestions and provide an alternative approach to schema evolution resilience. In my previous story (Evolving into a Big-Data Driven Business in the Azure Cloud: Data Ingestion), I described a Data Lake ingestion architecture that utilises Event Hubs and Event Hub Capture to form a batch layer for big data analytics. I’ll use this architecture as reference for handling schema evolution.
Figure 1.
{"type":"record",    "name":"EventData",    "namespace":"Microsoft.ServiceBus.Messaging",    "fields":[                 {"name":"SequenceNumber","type":"long"},                 {"name":"Offset","type":"string"},                 {"name":"EnqueuedTimeUtc","type":"string"},                 {"name":"SystemProperties","type":{"type":"map","values":["long","double","string","bytes"]}},                 {"name":"Properties","type":{"type":"map","values":["long","double","string","bytes"]}},                 {"name":"Body","type":["null","bytes"]}             ]}
Event Hubs
How many? How many Event Hubs should I have? Or to put it another way, should I have one big pipe for all my data or many smaller pipes for each message type? The same question has been asked regarding Kafka topics and there is no definitive answer2. One thing is highly probably, different use cases will favour different approaches. If your concern is just to get messages from A to B or you’re integrating with architecture outside of your control, messages might flow through one Event Hub, one big pipe. If some of your data is highly sensitive and you only want certain subscribers to read and process that data or you may need specific partition strategies which would lead to the adoption of many event hubs within a namespace, many smaller pipes.
Big Pipe
If an Event Hub contains many message types with varying schemas how would we identify and deserialize the various messages correctly?
Small Pipe
When an Event Hub contains just one message type and that message type evolves over time how do consumers deserialize the new versions of the message?
At first glance, these issues may seem to be unrelated. However, they are manifestations of the same core problem. How to manage the de-serialisation of data.
All messages on Event Hubs are anonymous blobs of binary. One option would be for consumers to infer the schema. However, this approach is non-deterministic and based on sampling, so the inferred schema can only be an approximation. Another approach might be to assert the schema on consumption. However, this means that engineering teams consuming messages are temporarily coupled to the evolution of the schema, even for minor changes. Event Hub Capture offers us an opportunity to break the temporal coupling and allow consumers to consume data from t0** at their own pace. However, if a consumer wants to read and make use of all the AVRO files, produced by the Event Hub Capture process, they will also need to know which write schemas were used to write the binary messages over the period that the events were captured. This could be many months or even years of data. As a consumer, I would need to know the schema evolution time-line or I will struggle to make use of the data.
** well at least from the begging of the Event Hub capture configuration.
Early impressions of Event Hub Capture might lead you to believe that AVRO was being used to help address the concerns detailed above. However, after reading the AVRO specification it would seem that only minor version changes are possible. So breaking changes cannot be managed and AVRO files with multiple message types would be impossible.
Event Hubs allow us to add additional metadata when we publish messages. This metadata is the key to managing schema evolution.
There are two possible options.
1) The write-schema is stored with each message in the Event Hub client properties dictionary. This would severely inflate the storage costs.
2) A message type identifier is stored in the Event Hub client properties dictionary. The identifier is then used to lookup the schema from a central store.
With both of these solutions, the schema is always directly or indirectly stored with the data. The files produced by Event Hub Capture will always have a means of identifying the write schema. Moreover, each file can contain x number of message types and y number of message versions.
Let’s take a look at an Azure Function that publishes messages to Event Hub using the client SDK.
The schema identifier is always stored alongside the data (line 17).
In the example above, the function uses a timer trigger to execute new instances of the function every 5 seconds. The function trigger is irrelevant, and it could easily be a CosmosDB Change Feed Processing binding or any other bindings that produce data to be processed. Moreover, using a function app is also irrelevant, what matters is what you publish to the Event Hub. The function app lends itself to a succinct example.
It's important to note the schema version of the message is being persisted alongside the message by adding a reference to eventData.Properties. This metadata attribution is critical when it comes to reading the data at a later date. When events are published to Event Hub the schema identifier is always stored alongside the data.
I configure Event Hub Capture to produce a new AVRO file every minute or every 500mb, whichever comes first. So, we now have the schema identifier and data captured in neatly partitioned AVRO files, but how do we process it in our big data pipelines. In my previous story, I covered the subject of maintaining a schema repository to capture a truthful account of all the enterprise's schemas. This repo is used to create an artefact that will be consumed in the data processing pipeline. The artefact is a simple key-value store connecting versioned schema identifiers with the write schema used. For the purpose of this document, I’ll use a simple Databrick Python notebook to process the AVRO data.
I won’t go into a full description of the complete notebook but focus on the most important cells (the complete notebook is in the GitHub repo).
The first is the reading of the Event Hub Data Capture AVRO. Spark’s AVRO dataframeReader is used to read AVRO files from storage and de-serialise them into a data-frame. We can allow Spark to infer the schema at this point as we know it to be non-volatile (i.e. the Azure Event Hub schema). The properties attribute holds the information about the schema version that was used to write the data in the binary field ‘Body’. A simple projection is run over the data to process a refined data-frame with three columns. The schema version is extracted from the properties object (the stored value from the serialised properties dictionary is stored in the child attribute member2). The ‘Body’ attribute is cast to a string as we want to use spark’s JSON de-serialiser on it later in the notebook.
from pyspark.sql.functions import colrawAvroDf = spark.read.format("avro").load("wasbs://" + containerName + "@" + storageAccName + ".blob.core.windows.net/gavroehnamespace/gavroeh/*/2020/*/*/*/*/*.avro")avroDf = rawAvroDf.select(col("Properties.SchemaVersion.member2").alias('SchemaVersion'), col("Body").cast("string"))display(avroDf)
The second is the schema lookup object. For the purpose of simplifying the example, I’m manually creating some schemas that will be used to deserialise the AVRO data. However, in practice, these schema’s will be generated from a schema repository and be stored as runtime artefacts. Note to self, need to write this up as a follow-up article.
The schemas, stored in a one-dimensional array, represent an entity that has evolved. In this theoretical example, the business has grown and started trading overseas in new currencies. Transactions now need currency identifiers, so a new attribute ‘Currency’ was added to the sales-order data schema. As readers, we need to be able to de-serialise the new data successfully.
from pyspark.sql.types import StructType, StructField, LongType, StringType, ArrayType, DoubleTypesalesOrderV1 =StructType([StructField('OrderId',StringType(),True),StructField('OrderAmount',DoubleType(),True),StructField('OrderCreatedUTC',StringType(),True)])salesOrderV2 =StructType([StructField('OrderId',StringType(),True),StructField('OrderAmount',DoubleType(),True),StructField('Currency',StringType(),False),StructField('OrderCreatedUTC',StringType(),True)])salesOrderSchemaDictionary = { "v1.0":salesOrderV1, "v2.0":salesOrderV2 }salesOrderSchemaDictionary
The third cell I’d like to focus on is the cell that actually reads and de-serialises the data. The Event Hub Data Capture output that was read into a data-frame previously is used to determine a distinct list of schema versions present in the data. For each schema version, a new temporary SparkSQL table will be created to access the de-serialised data. The original AVRO data-frame is filtered on each iteration of the ‘for’ loop, grouping records by distinct schema-version to produce subsets of data. Each subset is then de-serialised using the corresponding schema in the salesOrderSchemaDictionary. A number of new temporary tables will be created and the output of this cell will display a list of created objects.
from pyspark.sql.functions import concat, lit, regexp_replacedistinctSchemaVersions = avroDf.select('SchemaVersion').distinct()objectToCreate = distinctSchemaVersions.withColumn('TableName', concat(lit('SalesOrder'),regexp_replace(col('SchemaVersion'), '[.]', '_'))).collect()display(objectToCreate)for record in objectToCreate:schemaVersion = record.SchemaVersionjsonRdd = avroDf.filter(col("SchemaVersion") == schemaVersion).select(avroDf.Body)objectJson = jsonRdd.rdd.map(lambda x: x[0])dataExtract = spark.read.schema(salesOrderSchemaDictionary[schemaVersion]).json(objectJson)dataExtract.registerTempTable(record.TableName)
Finally, SparkSQL can be used to explore the successful deserialised data in the temporary tables.
%sqlselect * from SalesOrderv1_0%sqlselect * from SalesOrderv2_0
I should have started by clearing up what GAVRO is. Sorry to disappoint, but it’s not some new Apache incubator project that you wasn’t aware of. It’s an endearing name that my colleagues gave to the method I described in this article. I believe it’s a combination of my first initial and AVRO, at first I found their nickname for the method to be a product of the team's camaraderie, but then it stuck.
I don’t believe in designing and prescribing methods that are completely exact and should be unconditionally applied to every enterprise because every enterprise is different. So if you take anything away from reading this then I hope it’s the motivation to think about the connotations of badly managed schema evolution within your big data pipe-lines. We hear time and time again about the struggles organisation’s have with extracting information and actionable insight from big-data and how expensive data-scientists are wasting 80% of their time wrestling with data preparation. Schema management is a weapon when applied properly, that can be used to accelerate data understanding and reduce time to insight. So take the time to invest in it and you will reap healthy returns.
[1] Volkan Civelek, Schema validation with Event Hubs (1st April 2019), https://azure.microsoft.com/en-gb/blog/schema-validation-with-event-hubs/
[2] Martin Kleppman, Should you put several event types in the same Kafka topic?(18th Jan 2018), https://martin.kleppmann.com/2018/01/18/event-types-in-kafka-topic.html
Yahoo’s Apache Pulsar System: https://pulsar.apache.org/docs/en/schema-evolution-compatibility/
Confluent.io’s Schema-Registry: https://docs.confluent.io/current/schema-registry/index.html
Jay Kreps, The Log: What every software engineer should know about real-time data’s unifying abstraction (16th Dec 2013), https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying