Notes about saving data with Spark 3.0 | by David Vrba | Towards Data Science

Apache Spark is a computational engine frequently used in a big data environment for data processing but it doesn’t provide storage so in a typical scenario the output of the data processing has to be stored in an external storage system. Spark SQL provides a couple of connectors for datasources such as file format (CSV, JSON, text, Parquet, ORC) or JDBC. Since 2.4 there is also support for Apache Avro and since 3.0 also read support for binary files. In addition, there are several libraries that allow connecting to other datasources, for example, MongoDB, ElasticSearch, or DynamoDB to name some.
In this article, we will go over some possibilities that we have when saving data to a file format such as Apache Parquet, because the combination of Spark with Parquet provides a very nice experience, especially for analytical queries. We will first explain why Parquet is such a popular format for storing data in the Hadoop ecosystem and then we will describe different ways how to save the data with Spark while focussing especially on features that are missing in the official documentation or are difficult to find. One of these non-trivial features that we will look at is how to save the data in a pre-sorted state so it can be leveraged for data skipping with appropriate filters.
The provided code snippets are using the Python API and are checked against Spark 3.0 and 2.4.
Apache Parquet is an open-source file format originally developed at Twitter and Cloudera. It is a self-describing format, which means that it contains metadata with information about the schema (among other things). It is using a hybrid model for storing the data on the disk and by hybrid, we mean that it is not strictly row-oriented (such as JSON or CSV) neither strictly columnar but instead the data is first divided vertically into so-called row-groups and then each row-group is stored in a columnar representation. These row-groups allow for splitting the dataset into multiple files (each file can have one or more row-groups) so we don’t end up with one huge file which is important especially in the big data environment.
The columnar representation of each row-group is also a very important feature because it allows for a so-called column pruning. In analytical queries, we are not usually interested in all columns of the dataset but rather we select a few columns and do some aggregation on it. This column pruning lets us skip all other columns and scan only those selected in the query so it makes the reading much more efficient as compared to row-oriented formats where all columns have to be scanned. Another advantage of columnar formats is compression — each value in the column has the same data type and the values may even get repeated which can be used to store the data more efficiently using various encoding and compression techniques (that Parquet supports under the hood).
Parquet files support data skipping on different levels, namely on partition level and row-group level. So a dataset can be partitioned by some (usually low-cardinality) key and this means that for each distinct value of this field there is going to be a subfolder in the root directory that will contain all records with that particular value of the key (each partition can still be divided into multiple files). The use case for the partitioning is to reduce the volume of the data when reading because when the partition key is used in a filter, Spark will apply partition pruning and it will skip all partitions that are not relevant for the query (this is supported also for other file formats such as JSON or CSV).
The row-group level data skipping is based on parquet metadata because each parquet file has a footer that contains metadata about each row-group and this metadata contains statistical information such as min and max value for each column in the row-group. When reading the parquet file, Spark will first read the footer and use these statistics to check whether a given row-group can potentially contain relevant data for the query. This will be useful especially if the parquet file is sorted by the column that we use for filtering. Because, if the file is not sorted, then small and large values can be scattered across all row-groups and thus each row-group will have to be scanned because it can potentially contain some rows satisfied by the filter. The sort here is crucial and as we will see later on, it is not trivial to save the data in the sorted state.
One of the options for saving the output of computation in Spark to a file format is using the save method
(   df.write  .mode('overwrite') # or append  .partitionBy(col_name) # this is optional  .format('parquet') # this is optional, parquet is default  .option('path', output_path)  .save())
As you can see it allows you to specify partition columns if you want the data to be partitioned in the file system where you save it. The default format is parquet so if you don’t specify it, it will be assumed.
The data analyst who will be using the data will probably more appreciate if you save the data with the saveAsTable method because it will allow him/her to access the data using
df = spark.table(table_name)
The saveAsTable function allows also for using bucketing where each bucket can be also (optionally) sorted:
(   df.write  .mode('overwrite') # or append  .partitionBy(col_name) # this is optional  .bucketBy(n, col_name) # n is number of buckets  .sortBy(col_name)  .format('parquet') # this is optional, parquet is default  .option('path', output_path)  .saveAsTable(table_name))
We will not dive into bucketing here, but in general, it is a technique on how to pre-shuffle the data and save it in this state so it can be leveraged by follow-up queries to avoid shuffle. This is going to work if the metastore is properly set up with Spark because the information about bucketing will be saved here. This approach has also other benefits since the metastore can keep also other information about your dataset such as statistics when you call the ANALYZE TABLE command later on. The sortBy can be used only after bucketBy, because what will be sorted are the created buckets. Both modes overwrite and append will work here even if the table doesn’t exist yet because the function will simply create it.
Yet another possibility of how to save data is using the insertInto function. Unlike in the previous case, the table has to exist first, because if it doesn’t, you will get an error: AnalysisException: Table not found. The syntax for using this function is rather simple
(  df.write  .insertInto(table_name))
but there are some cautions that are good to be aware of:
You don’t have to specify the output path. Spark will find that information in the metastore. If you specify the path anyway, it will be ignored.
The default mode is append, so it will simply add your data to the existing table.
The schema of your DataFrame must match the schema of the table. If the order of the columns in your DataFrame is different than the order in the table, Spark will throw an exception if the data types are different and can't be safely cast. But if there is not a mismatch in data types, Spark will insert the data to the table in this wrong order. So this is quite dangerous and it is good to make sure that the schema is the same for example as follows:
(  df.select(spark.table(table_name).columns)  .write  .insertInto(table_name))
if your table is partitioned or bucketed, you don’t specify that (you would get an error) because Spark will pick that information from the metastore.
The function insertInto has one big advantage over saveAsTable because it allows for a so-called dynamic overwrite. This feature lets you overwrite a specific partition in a partitioned table. For example, if your table is partitioned by year and you want to update only one year, then with saveAsTable you would have to overwrite the entire table, but with insertInto, you can overwrite only this single partition so it will be a much cheaper operation especially if there are lots of big partitions. To use this feature you have to set a corresponding configuration setting:
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")(  df # having data only for specific partitions  .write  .insertInto(table_name, overwrite=True))
Here again, it is good to be careful, because if the partitionOverwriteMode is set to static (which is the default value) it would overwrite the entire table and so all other partitions would become empty. The dynamic value makes sure that Spark will overwrite only partitions that we have data for in our DataFrame.
As we mentioned above, sometimes it is desirable to save the data in the sorted state according to some column so it could be used for data skipping in analytical queries that use this column as a filter. In Spark SQL there are three functions when it comes to sorting. There is orderBy (or equivalently sort), sortWithinPartitions, and sortBy. Let’s see what is the difference between them and how they can be used:
orderBy — it is a DataFrame transformation that will invoke a global sort. This will first run a separate job that will sample the data to check the distribution of values in the sorting column. This distribution is then used to create boundaries for partitions and the dataset will be shuffled to create these partitions. It is quite an expensive operation and if you save the data into a partitioned table it may create lots of small files if the data distribution on the cluster is very different from the final distribution of the table in the file system (for example if you sort by date and your table is partitioned by country, each task in the final stage of the Spark job may potentially carry data for each country, so each task will write a new file into each file system partition).sortWithinPartitions — it is also a DataFrame transformation and unlike in the previous case Spark will not try to achieve a global sort but instead, it will sort each partition separately. So here you can distribute the data on the Spark cluster as you require for the final layout using the repartition() function (this will also create a shuffle) and then call sortWithinPartitions to have each partition sorted. The advantage of this approach over the previous one is that you can avoid having lots of small files and still have the output sorted.sortBy — this function can be called on a DataFrameWritter (after calling df.write) if we also call bucketBy and use saveAsTable method for saving. It is going to make sure that each bucket is sorted (one bucket can be either one file or more files depending on the distribution of data on the Spark cluster at the moment of writing).
orderBy — it is a DataFrame transformation that will invoke a global sort. This will first run a separate job that will sample the data to check the distribution of values in the sorting column. This distribution is then used to create boundaries for partitions and the dataset will be shuffled to create these partitions. It is quite an expensive operation and if you save the data into a partitioned table it may create lots of small files if the data distribution on the cluster is very different from the final distribution of the table in the file system (for example if you sort by date and your table is partitioned by country, each task in the final stage of the Spark job may potentially carry data for each country, so each task will write a new file into each file system partition).
sortWithinPartitions — it is also a DataFrame transformation and unlike in the previous case Spark will not try to achieve a global sort but instead, it will sort each partition separately. So here you can distribute the data on the Spark cluster as you require for the final layout using the repartition() function (this will also create a shuffle) and then call sortWithinPartitions to have each partition sorted. The advantage of this approach over the previous one is that you can avoid having lots of small files and still have the output sorted.
sortBy — this function can be called on a DataFrameWritter (after calling df.write) if we also call bucketBy and use saveAsTable method for saving. It is going to make sure that each bucket is sorted (one bucket can be either one file or more files depending on the distribution of data on the Spark cluster at the moment of writing).
Using the first two functions has one big catch that you would not find in documentation (written in October 2020). Let’s see it on a simple example. For the sake of simplicity we want to partition the data by year and have each partition sorted by user_id, and for saving we use saveAsTable(). If you just call
(  df.repartition('year')  .sortWithinPartitions('user_id')  .write  .mode('overwrite')  .partitionBy('year')  .option('path', output_path)  .saveAsTable(table_name))
It is not going to work! It will not raise any error and it will save the data but the order will not be preserved! The point is that when writing data to a file format, Spark requires this ordering:
(partitionColumns + bucketIdExpression + sortColumns)
here partitionColumns are columns by which we partition the data to the file system, bucketingIdExpression is derived from the bucketing column (not relevant in our query because we are not using bucketing here) and sortColumns are columns used in sortBy with bucketing (again not relevant for our query). If the data is sorted by these columns (and possibly some more), the requirement will be satisfied and Spark will preserve the order. If however, this requirement is not satisfied, Spark will forget the previous order and will sort the data again according to this requirement while writing the data. In our example, the required ordering is (year) which is the partition column and we don’t have any bucketing here. This requirement is however not satisfied, because the actual ordering is (user_id), which is the column by which we sorted the data and this is the reason why Spark will not preserve our order and will sort the data again by the year column.
required_orering = (year)actual_ordering = (user_id) ... this doesn't satisfy the requirement
To achieve our goal and save the data (each partition) sorted by user_id we can instead do this:
(  df.repartition('year')  .sortWithinPartitions('year', 'user_id')  .write  .mode('overwrite')  .partitionBy('year')  .option('path', output_path)  .saveAsTable(table_name))
Notice the difference now, we sort explicitly each partition by year and user_id, but since the data will be partitioned by year the sort by this column doesn’t really matter. What matters is that now the Spark’s requirement will be satisfied, so Spark will preserve our order and save the data sorted by year and user_id and since we partition by year, it basically means that each partition will be sorted by user_id, which is exactly what we want.
required_orering = (year)actual_ordering = (year, user_id) ... this satisfies the requirement
Saving the data partitioned where each partition is sorted, will allow for efficient analytical queries with filters that will skip data on two levels — partition level and row-group level. Imagine a query like this:
(  spark.table(table_name)  .filter(col('year') == 2020)  .filter(col('user_id') == 1)  .collect())
This will first use the partition filter to prune the partitions and inside this single partition 2020 it will check the metadata from the parquet footers for each row-group. Based on the statistics in the metadata Spark will pick the row-groups with min≤1 and max≥1 and only these row-groups will be scanned, so this will speed-up the query especially if there are lots of row-groups that can be skipped.
In this article, we described three different ways how to save data to a file format (specifically Apache Parquet) using Spark and we also explained why Parquet is such a popular data format especially when it comes to analytical queries. We also pointed out some Spark features that are not so obvious such as the behavior of the insertInto function or how to save data in the sorted state.