Predicate vs Projection Pushdown in Spark 3 | by Pınar Ersoy | Towards Data Science

Working with big data comes with the challenges of high amounts of processing costs and spending a lot of time scaling them. It is required to be aware of the adequate big data filtering techniques in order to get benefit from them. This article is essential for data scientists and data engineers looking to use the newest enhancements of Apache Spark in the sub-area of filtering especially on nested structured data.
With Spark 2.x, files with a maximum 2-level nested structure with .json and .parquet extensions could be read.
Example:
filter(col(‘library.books’).isNotNull())
With Spark 3, it is now possible to read files with both parquet and .snappy parquet extensions in 2+ level nested structure without any need for schema flattening operations.
Example:
filter(col(‘library.books.title’).isNotNull())
While creating a spark session, the following configurations shall be enabled to use pushdown features of the Spark 3. The setting values ​​linked to Pushdown Filtering activities are activated by default.
"spark.sql.parquet.filterPushdown", "true""spark.hadoop.parquet.filter.stats.enabled", "true""spark.sql.optimizer.nestedSchemaPruning.enabled","true""spark.sql.optimizer.dynamicPartitionPruning.enabled", "true"
There are two kinds of pushdown filtering techniques as Predicate Pushdown and Projection Pushdown in Spark with the following differing features as described in the following sections of the article.
Predicate Pushdown points to the where or filter clause which affects the number of rows returned. It basically relates to which rows will be filtered, not which columns. For this reason, while applying the filter on a nested column as ‘library.books’ to merely return records with the values that are not null, the predicate pushdown function will make parquet read blocks that contain no null values for the specified column.
The partition elimination technique allows optimizing performance when reading folders from the corresponding file system so that the desired files in the specified partition can be read. It will address shifting the filtering of data as close to the source as possible to prevent keeping unnecessary data into memory with the aim of reducing disk I/O.
Below, it can be observed that the filter of partition of the action of push down, which is ‘library.books.title’ = ‘THE HOST’ filter is pushed down into the parquet file scan. This operation enables the minimization of the files and scanning of data.
data.filter(col('library.books.title') == 'THE HOST').explain()
For a more detailed output including Parsed Logical Plan, Analyzed Logical Plan, Optimized Logical Plan, Physical Plan, the ‘extended’ parameter can be added in the explain() function as follows.
data.filter(col('library.books.title') == 'THE HOST').explain('extended')
It can also reduce the amount of data passed back to the Spark Engine for the average operation in the aggregation function on the ‘price’ column.
data.filter(col('library.books.title') == 'THE HOST').groupBy('publisher').agg(avg('price')).explain()
Parquet formatted files keep some different statistical metrics for each column including the minimum and maximum of their values. Predicate Pushdown helps to skip irrelevant data and deals with the required data.
Projection Pushdown stands for the selected column(s) with the select clause which affects the number of columns returned. It stores data in columns, so when your projection limits the query to specified columns, specifically those columns will be returned.
data.select('library.books.title','library.books.author').explain()
This means that the scanning of the ‘library.books.title’, ‘library.books.author’columns meaning that the scanning will occur in the file system/database before sending back the data to the Spark engine.
For both the Projection and Predicate Pushdown, there are some crucial points to highlight. Pushdown Filtering works on partitioned columns which are calculated by the nature of parquet formatted files. To be able to get the most benefit from them, the partition columns shall be carrying smaller-sized values with adequate matching data to scatter the correct files in the directories.
Prevent too many small-sized files causes to make scans less efficient with excessive parallelism. Also, preventing accepting too few big-sized files may damage parallelism.
The Projection Pushdown feature allows the minimization of data transfer between the file system/database and the Spark engine by eliminating unnecessary fields from the table scanning process. It is primarily useful when a dataset contains too many columns.
On the other hand, the Predicate Pushdown boosts performance by scaling down the amount of data passed between the file system/database and the Spark engine when filtering data.
Projection Pushdown is distinguished by column-based and Predicate Pushdown by row-based filtering.
Questions and comments are highly appreciated!
Spark Release 3.0.0Pushdown of disjunctive predicatesGeneralize Nested Column PruningParquet predicate pushdown for nested fields
Spark Release 3.0.0
Pushdown of disjunctive predicates
Generalize Nested Column Pruning
Parquet predicate pushdown for nested fields