Statistics in Spark SQL explained | by David Vrba | Towards Data Science

Spark SQL optimizer uses two types of optimizations: rule-based and cost-based. The former relies on heuristic rules while the latter can use some statistical properties of the data. In this article, we will explain how these statistics are used in Spark under the hood and we will see in which situations they are useful and how to take advantage of them.
Most of the optimizations that Spark does are based on some heuristic rules that do not take into account the properties of the data that are being processed. For example, the PredicatePushDown rule is based on a heuristic rule which assumes that it is better to first reduce the data by filtering and then apply some computation on it. There are however some situations in which Spark can also use some statistical information about the data in order to come up with yet a better plan and this is often referred to as the cost-based optimization or CBO. In this article, we will explore it more in detail.
To see the statistics of a table we first need to compute them by running a SQL statement (notice that all the SQL statements can be executed in Spark using the sql() function spark.sql(sql_statement_as_string)):
ANALYZE TABLE table_name COMPUTE STATISTICS
After this, the table level statistics are computed and saved in metastore and we can see them by calling
DESCRIBE EXTENDED table_name
This is going to show us some properties of the table including the table-level statistics. There are two metrics available, namely the rowCount and sizeInBytes:
Apart from these table-level statistics there are also column-level statistics and to compute and see them we can call:
ANALYZE TABLE table_name COMPUTE STATISTICS FOR COLUMNS col_nameDESCRIBE EXTENDED table_name column_name
This is going to show us a table like this (the column used in this example is user_id):
So as you can see, there are various metrics for the column, such as min/max values, number of null values, number of distinct values (which is approximated), and some others.
Since Spark 3.0 there is one more option on how to display the stats and see it not only for the table but for the actual query that we want to run. This can be done by using the new mode argument of the explain function:
spark.table(table_name).explain(mode="cost")
This is going to show us two query plans, namely the physical plan and also the optimized logical plan which will now contain also the information about the statistics as you can see in the image below:
The point is that now you can see the statistics for each operator in the plan, so you can see what are the estimates of the stats after various transformations. The stats are first computed by the Relation operator which is a so-called leaf node and each leaf node is responsible to compute the stats somehow, and then they are propagated through the plan according to some rules.
In the next, we will see how the leaf node computes the stats and how the propagation works.
There are three ways of how the leaf node can compute the statistics. The first (and best) way is that the stats are taken from the metastore. The second option is that Spark will use InMemoryFileIndex which calls Hadoop API under the hood to gather the size of each file in the datasource and sum it up to get the total sizeInBytes (in this option only this one metric would be computed). And finally, the last way is that Spark would use a default value for the sizeInBytes metric which is given by the spark.sql.defaultSizeInBytes configuration setting and the default of this setting is 8 EiB, so basically Spark will overestimate the size for the Relation as much as possible (and again only this one metric will be used). Which of these three options takes place can be described by this diagram:
This diagram is a tree where each node is a condition and if the condition is true, we go in the direction of T (True) otherwise we go in the direction of F (False). The leaves represent the actual way how the stats will be computed, so for example InMemoryFI means that only sizeInBytes will be computed by calling the Hadoop API. On the other hand, Stats from M means that the statistics will be taken from the metastore, however, notice that on the left side of the tree all the stats will be taken from the metastore but on the right side only one metric sizeInBytes will be taken. The leaf CatalogFileIndex represents the last case where the default value 8 EiB for the sizeInBytes will be used.
In the diagram we have altogether four conditions, the first determines how the data is accessed: if we read the data as a table df=spark.table(table_name) then we go to the left, otherwise, we go to the right. The next condition is whether the cost-based optimizer (CBO) is turned On or Off. This is given by the configuration setting spark.sql.cbo.enabled and the default value is False (still in Spark 3.0). The third condition asks if the stats were computed in metastore by running the analyze table command (ATC) and finally the last condition is whether the table is partitioned or not.
The best scenario occurs if we read the data as a table, if the cost-based optimizer is On, and if we first run ATC, in that case, all the statistics will be taken from the metastore (except for the sizeInBytes which is computed from the rowCount as described below). On the other hand, the worst situation with the worst estimates happens when we also access the data as a table, but if the ATC didn’t run and moreover the table is partitioned, in this case, the default sizeInBytes is taken from the configuration property and the estimates will be very imprecise. Notice that for the worst-case scenario it is irrelevant whether the CBO is On or Off. Also notice that if the table is not partitioned, Spark will actually compute at least the sizeInBytes using the Hadoop API so the table partitioning has a direct impact on how the stats are computed in the leaf node of the query plan.
Once the statistics are computed in the leaf node, they can be propagated to other nodes in the query plan. There are two ways of how this propagation can happen. The first way (let’s call it the Old way) is rather basic and only one metric, namely sizeInBytes, is propagated and here the rules by which the metric is adjusted on various operators are very basic. For instance, the Filter operator doesn’t adjust the value at all as you can see in this example:
(  spark.table(table_name)  .filter(col("user_id") < 0)).explain(mode="cost")
In this query, we filter for all records where the user_id is negative, and there are actually no such records, but Spark doesn’t have this information available because that would require to use also column-level statistics and they are not used in this Old way. So as you can see from the query plan, there is only sizeInBytes propagated and it stays the same on both operators. In other words, we can say that Spark thinks that after applying the Filter, the size of the data doesn’t change:
The second way of stats propagation (let’s call it the New way) is more mature, it is available since Spark 2.2 and it requires having the CBO turned ON. It also requires to have the stats computed in metastore with ATC. Here all the stats are propagated and if we provide also the column level metrics, Spark can compute the selectivity for the Filter operator and compute a better estimate:
As you can see, the stats on the Filter operator changed, the rowCount is now zero and the sizeInBytes is 1B which is the minimum value. From the column level stats of the user_id column, Spark was able to see that there are no records with the negative value of user_id and is able to reflect that in the query plan.
In this New way, to compute the sizeInBytes, Spark first computes the size of a single row based on the data types information for each column and then multiplies by the rowCount to get the final sizeInBytes. If the rowCount is zero, the sizeInBytes is set to 1 to avoid division by zero in some other stats computation. This is also how the sizeInBytes is adjusted on the Project operator (Spark knows what columns will be projected, so again it computes the size of a single row first).
At this moment we have an idea of how the stats are computed and propagated through the plan, let’s now finally see how they are also used during the query planning to achieve a more optimal plan. There are two places where these statistics are used. The first one is the JoinSelection strategy where Spark decides which algorithm will be used for joining two DataFrames (see my other article where I describe this more in detail). The basic logic is that if one of the DataFrames is smaller than a certain threshold, Spark will use BroadcastHashJoin (BHJ), because it is a very efficient algorithm if the broadcasted DataFrame is very small. The threshold is given by spark.sql.autoBroadcastJoinThreshold configuration setting which default value is 10 MB. So here having a good estimate for the size of the DataFrame can help to choose a more optimal algorithm for the join.
The second place where stats are used is also related to joins and namely, it is the joinReorder rule. Using this rule Spark can find the most optimal order in which DataFrames will be joined (if you join more than two DataFrames). This rule is by default turned off and if you want to use it you have to enable it by this configuration setting:
spark.conf.set("spark.sql.cbo.joinReorder.enabled", True)
because the default value is False. The maximum number of DataFrames for which this can be used is controlled by this property:
spark.conf.set("spark.sql.cbo.joinReorder.dp.threshold", n)
where the default value of n is 12.
Well, we have seen that if a table is partitioned and we don’t run ATC, Spark will overestimate the size by using the default value (which is very large — 8 EiB). So in a situation where we join lots of tables and some of these tables are partitioned and rather small, so they could be good candidates for BHJ, in that case, it makes sense to run for them ATC. Also, we have to remember that if a table is appended or overwritten, the stats will be deleted, so we have to run ATC again. In some environments, having the stats in metastore up to date might be complicated. A partial solution to this problem provides adaptive query execution — a new feature in Spark 3.0.
In Spark 3.0 a new feature Adaptive Query Execution (AQE) was released and it uses statistics in an even more enhanced way. If the AQE is enabled (by default it is not), the statistics are recomputed after each stage is executed during runtime. This leads to more precise stats estimates and a better ability to decide whether BHJ can be used or not. AQE itself is quite a big topic and we can cover it in a separate article.
In this article, we have seen how statistical information about data is used by Spark in order to achieve a more optimal execution plan. We have seen that there are three ways how the stats are computed in the first node of the query plan (in a so-called leaf node) and then how this information can be propagated through the plan. We pointed out a special caveat related to partitioned tables which are small and may be good candidates for broadcasting in joins. If we don’t run the analyze table command for them, Spark will overestimate the size of these tables and the broadcast join will be hardly achieved. We also showed that having better propagation of the statistics through the plan requires having the cost-based optimizer enabled and column statistics computed especially for those columns that are used in filtering conditions.