Merging different schemas in Apache Spark
By: Thiago Cordon
Tags: Data, Schema, Spark, Data Engineering

The image above is showing the differences in each partition. As we can see, columns and structs were added, datatypes changed and columns were removed.

The desired result is a schema containing a merge of these changes without losing any column or struct even it doesn’t exist anymore.

Attempt 1: Reading all files at once

What happens if we try to read all these files at once with spark.read.parquet() ?

data_path = "/home/jovyan/work/data/raw/test_data_parquet"

df = spark.read.parquet(data_path)

df.printSchema()

It’s possible to read all files but as we can see above, only the schema of the first partition was considered. As all partitions have these columns, the read function can read all files but it will not merge the differences as we want.

As we can see below, all rows in each partition can be read because they share the same fields mentioned above but we are interested in merge all fields so, this solution doesn’t work.

from pyspark.sql.functions import col

df.groupBy(col("date")).count().sort(col("date")).show()

Attempt 2: Reading all files at once using mergeSchema option

Apache Spark has a feature to merge schemas on read. This feature is an option when you are reading your files, as shown below:

data_path = "/home/jovyan/work/data/raw/test_data_parquet"

df = spark.read.option("mergeSchema", "true").parquet(data_path)

Unfortunately, this option cannot handle our different schemas. It could not merge the schema of the partition 2020-04-01 because the postal_code has incompatible data types as shown in the following error message (simplified to be legible):

org.apache.spark.SparkException: Failed merging schema of file file:/home/jovyan/work/data/raw/test_data_parquet/date=2020-04-01/part-00000-796d0c3c-69c0-44c5-a4fa-635195e8d6a9.c000.snappy.parquet

Caused by: org.apache.spark.SparkException: Failed to merge fields 'address' and 'address'. Failed to merge fields 'postal_code' and 'postal_code'. Failed to merge incompatible data types string and int

Attempt 3: Reading all files at once forcing a schema on read

Let’s see what happens when we force the desired schema when reading the parquet files.

from pyspark.sql.types import StructType

import json schema_json = '{"fields":[{"metadata":{},"name":"address","nullable":true,"type":{"fields":[{"metadata":{},"name":"address","nullable":true,"type":"string"},{"metadata":{},"name":"address_details","nullable":true,"type":{"fields":[{"metadata":{},"name":"number","nullable":true,"type":"string"},{"metadata":{},"name":"street","nullable":true,"type":{"fields":[{"metadata":{},"name":"lat","nullable":true,"type":"string"},{"metadata":{},"name":"latitude","nullable":true,"type":"string"},{"metadata":{},"name":"long","nullable":true,"type":"string"},{"metadata":{},"name":"longitude","nullable":true,"type":"string"},{"metadata":{},"name":"street_name","nullable":true,"type":"string"}],"type":"struct"}}],"type":"struct"}},{"metadata":{},"name":"city","nullable":true,"type":"string"},{"metadata":{},"name":"country","nullable":true,"type":"string"},{"metadata":{},"name":"country_code","nullable":true,"type":"string"},{"metadata":{},"name":"postal_code","nullable":true,"type":"string"},{"metadata":{},"name":"state","nullable":true,"type":"string"}],"type":"struct"}},{"metadata":{},"name":"age","nullable":true,"type":"string"},{"metadata":{},"name":"date","nullable":true,"type":"string"},{"metadata":{},"name":"first_name","nullable":true,"type":"string"},{"metadata":{},"name":"identifier","nullable":true,"type":"string"},{"metadata":{},"name":"last_name","nullable":true,"type":"string"},{"metadata":{},"name":"occupation","nullable":true,"type":"string"},{"metadata":{},"name":"title","nullable":true,"type":"string"},{"metadata":{},"name":"title_name","nullable":true,"type":"string"}],"type":"struct"}' schema = StructType.fromJson(json.loads(schema_json)) data_path = "/home/jovyan/work/data/raw/test_data_parquet"

df = spark.read.schema(schema).option("mergeSchema", "true").parquet(data_path) df.show()

When we ask the data frame to return a sample of the lines ( df.show() ), we get the following error indicating that it could not read the partition 2020-04-01 (error message simplified to be legible).

org.apache.spark.sql.execution.QueryExecutionException: Encounter error while reading parquet files. One possible cause: Parquet column cannot be converted in the corresponding files

Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file file:/home/jovyan/work/data/raw/test_data_parquet/date=2020-04-01/part-00000-796d0c3c-69c0-44c5-a4fa-635195e8d6a9.c000.snappy.parquet

This partition has significant changes in the address struct and it can be the reason why Spark could not read it properly.

Attempt 4: Reading each partition at a time and union the dataframes

What if we read each partition at a time and make a union of the dataframes? Let’s see what happens.

import os

from pyspark.sql.functions import lit data_path = "/home/jovyan/work/data/raw/test_data_parquet"

idx = 0 for dir in [d for d in os.listdir(data_path) if d.find("=") != -1]: df_temp = spark.read.parquet(data_path + "/" + dir).withColumn(dir.split("=")[0], lit(dir.split("=")[1])) if idx == 0:

df = df_temp

else:

df = df.union(df_temp) idx = idx + 1

We cannot union dataframes with different schemas, so it doesn’t work.

org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the same number of columns, but the first table has 7 columns and the second table has 8 columns

Final solution: Custom function

After trying to merge the schemas using the methods described above, I ended up building a custom function that do the following:

Reads each partition path and get the schema of that partition

Converts the columns to String to assure that the data types will be compatible between schemas avoiding errors faced in attempt 2

Converts the dataframe to a JSON RDD before union the partitions. JSON RDD allows the union even when the structures are different, avoiding the error encountered in attempt 4.

After reading the partitions with this custom function, we have a dataframe with the desired schema, containing columns and structures of all partitions.

df.printSchema()

We can also easily identify which columns were not used by doing a count of null rows by partition and column.