How to Debug Queries by Just Using Spark UI
Tags: Technology, Spark, Programming, Spark Ui

How to Debug Queries by Just Using Spark UI

You already have the thing you need to debug a query

Spark is the most widely used big data computation engine, capable of running jobs on petabytes of data. Spark provides a suite of web user interfaces (UIs) that you can use to monitor the status and resource consumption of your Spark cluster. Most of the issues that we encounter while running a job can be debugged by heading to the spark UI.

spark2-shell --queue=P0 --num-executors 20 Spark context Web UI available at http://<hostname>:<port>

Spark context available as 'sc'

Spark session available as 'spark'

In this document, I will try to showcase how to debug a spark job just by using the Spark UI. I will run a few Spark jobs and show how the Spark UI reflects the run of the job. I will also add some tips and tricks along the way

This is how a Spark UI looks like

We will start with the SQL tab, which encompasses a lot of info to do an initial review. If using RDDs, you may not see the SQL tab in some cases.

Here is a query I ran for reference

spark.sql("select id, count(1) from table1 group by id”).show(10, false)

Now, let us correlate this with the Physical plan of the query. The physical plan is below the SQL DAG when you expand the details tab. We should read the plan from bottom to top

== Physical Plan == CollectLimit 11 +- *(2) HashAggregate(keys=[id#1], functions=[count(1)], output=[id#1, count(1)#79]) +- Exchange hashpartitioning(id#1, 200) +- *(1) HashAggregate(keys=[id#1], functions=[partial_count(1)], output=[id#1, count#83L]) +- *(1) FileScan parquet [id#1] Batched: true, Format: Parquet, Location: InMemoryFileIndex[hdfs://<node>:<port><location>, PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:string>

Here is how to read the plan:

Scan the parquet file. Note the Pushed filters. I will demonstrate what this means later Create a Hash Aggregate with the keys. Please note the partial_count. This means that the aggregate — count is partial since the aggregation has been done in each individual task and not shuffled to get the entire value set. Now, the generated data is aggregated based on the key, id in this case Now the entire count is computed. The resulting output is shared

With this being clear, let us look at the PushedFilters data. Spark is highly optimized for predicates and any filters applied is pushed to the source. To show this, let us look at a different version of this query

spark.sql("select id, count(1) from table1 where status = 'false' group by id”).show(10, false)

This is the plan for the above

+- *(2) HashAggregate(keys=[id#1], functions=[count(1)], output=[id#1, count(1)#224]) +- Exchange hashpartitioning(id#1, 200) +- *(1) HashAggregate(keys=[id#1], functions=[partial_count(1)], output=[id#1, count#228L]) +- *(1) Project [id#1] +- *(1) Filter (isnotnull(status#3) && (status#3 = false)) +- *(1) FileScan parquet [id#1,status#3] Batched: true, Format: Parquet, Location: InMemoryFileIndex[hdfs://mr25p01if-ingx03010101.mr.if.apple.com:50001/home/hadoop/work/news/20200..., PartitionFilters: [], PushedFilters: [IsNotNull(status), EqualTo(status,false)], ReadSchema: struct<id:string,status:string>

Note the changes from the previous plan.

We see something different in pushed filters — Not null check and ewuality check. The column on which we are applying the filter is pushed to the source. i.e While reading the data, these rows are ignored. The output of this is projected to the next steps.

When applying filters, can we reduce the total number of data read (or files read) as well?

Yes, we can. In both the able examples shown above, the total number of data read is ~ 23.8M. To reduce this, we can use the magical parquet files. Parquet has a row group, which has statistics that can be used to ignore a few row groups/files. This results in these files not being read at all. Please refer to my medium doc on how to do this —

Executor Tab

This tab gives us an idea of the number of executors your spark session is currently active with.

spark2-shell — queue=P0 — driver-memory 20g — executor-memory 20g — num-executors 40

I requested 40 executors to set up the session, however on startup, you can see it provided me with 10 active executors. This could be due to hosts not up or Spark not needing so many executors. This also can cause a delay in scheduling jobs, since you only have 10 executors and you need 40, so parallelism will be impacted.

Environment Tab

The environment tab has the details of all the config parameters that the spark session is currently using.

See how the params provided by me earlier are reflected here. This is useful to just be sure that the config you provided is accepted here.

Storage Tab

This shows information about one of the most debated features of Spark — Caching. There are many documents available online whether to cache or not. Luckily, this document is not about when to cache, etc. It's more about what happens when we cache

But before that, let us rewind and spend a few minutes on some basics on caching.

There are two ways to cache a Dataframe:

→ df.persist

It takes multiple properties to cache the dataset.

→ df.cache

This internally calls the ‘persist’ method. Refer source code

def cache(): this.type = persist() /** * Persist this Dataset with the given storage level. * @param newLevel One of: `MEMORY_ONLY`, `MEMORY_AND_DISK`, `MEMORY_ONLY_SER`,`MEMORY_AND_DISK_SER`, `DISK_ONLY`, `MEMORY_ONLY_2`,`MEMORY_AND_DISK_2`, etc. * @group basic * @since 1.6.0 */

DISK_ONLY: Persist data on disk only in a serialized format.

MEMORY_ONLY: Persist data in memory only in deserialized format.

MEMORY_AND_DISK: Persist data in memory and if enough memory is not available evicted blocks will be stored on disk.

MEMORY_ONLY_SER: This level of Spark store the RDD as a serialized Java object (one-byte array per partition). It is more space-efficient as compared to deserialized objects. But it increases the overhead on CPU

MEMORY_AND_DISK_SER: Similar to MEMORY_ONLY_SER, but write to disk when data does not fit in memory

In our example, let us use df.cache and see what happens

a.cache()

— > Nothing is visible on the Storage tab. As you know, this is because of Spark’s lazy evaluation

a.groupBy(“id”).count().show(10,false)

We see some data cache. The size in memory is 5.2 GB, while my file size is 2 GB … hmmm …what happened here

hadoop dfs -dus <dirName>

2,134,751,429 6,404,254,287 <dirName>

This is because the data in memory is Deserialized and uncompressed. This results in a higher size in memory, compared to disk.

So, whenever you want to take a decision on whether to cache or not, keep this in mind.

I saw some online docs on whether to cache or not. Good idea to refer those:

Next, we will dwell into Jobs and Stages tab, The cause for lot of problems can be debugged through these tabs.

spark.sql("select is_new_user,count(1) from table1 group by is_new_user").show(10,false)

For the above query, I see 3 jobs being submitted. But 2 of those are skipped. Usually, it means that data has been fetched from the cache and there was no need to re-execute the given stage. Also, the spark runs a lot of dummy jobs to evaluate the data. The skipped jobs could also be due to this.

Let us, deep-dive, into the job that was not skipped. This is the DAG visualization for the job

We can clearly see that this job has two stages, divided by the Shuffle/exchange operation. Stages mean that data was written to disk to be consumed by the next process

Let us, deep-dive, into the stages tab.

The first thing you always want to check is the Summary metrics for tasks. You can click “show additional metrics” for additional facts. This reveals a lot of necessary parameters by the min, median, and max. In an ideal world, min should be close to max.

Here are a few things I noted:

→ duration: In our example, the min and max duration is 0.4 seconds and 4 seconds respectively. These could be due to multiple reasons and we will try to debug them through the below points.

→ Task deserialization time:

In our example, there is some time taken in task deserialization for the other tasks. One of the primary reasons was GC processes running in the executors. For me, other processes were running which had some data cached, causing GC. GC processes are given the highest priority and will stop all running processes to cater to the GC process. If you do not see your process consuming a lot of memory, talking to admin / OPS can be the first step to resolve such an issue.

→ Scheduler delay: There is a max scheduler delay of 0.4 seconds. This means that one of the tasks had to wait an extra 0.4 seconds to be submitted. Whether this is large or small depends on your use case.

→ The input size is very distributed. This is a very good thing since all the tasks are reading the same amount of data. This is one of the most primary things when looking for a bad/skewed query. This can be visualized in the “shuffle read” column in the “Summary metrics for tasks” section. The simplest logic for solving such problems is adding salt to the group which can parallelize the data, then finally aggregating the data without the salt. This principle can be applied in many forms to optimize data skewness issue.

The other thing you want to take note of is the locality level.

* PROCESS_LOCAL → This task will be run within the same process as the source data

* NODE_LOCAL → This task will be run on the same machine as the source data

* RACK_LOCAL → This task will be run in the same rack as the source data

* NO_PREF (Shows up as ANY) → This task cannot be run on the same process as the source data or it doesn’t matter

Let us assume we are consuming data from a Cassandra node in a 3 node Spark Cluster. The Cassandra is running on machine X of Spark nodes X,Y, and Z. All of the data will be tagged as NODE_LOCAL to Node X. This means once every core on X is occupied we will be left with tasks whose preferred location is X but we only have execution space on Y and Z. Spark only has two options, wait for cores to become available on X or downgrade the locality level of the task and try to find a space for it and take whatever penalty there is for running non-local.

The spark.locality.wait parameter describes how long to wait before downgrading tasks that could potentially be run a higher locality level to a lower level. This parameter is basically our estimate of how much time waiting for the locality is worth. The default is 3 seconds which means that in our Cassandra example, once our co-located node X is saturated with tasks, our other machines Y and Z will stand idle for 3 seconds before tasks which could have been NODE_LOCAL are downgraded to ANY* and run.

Linked the code reference for this.

I hope this document helps us as a guide to using Spark UI to debug your spark performance issues. There are a lot of additional features in Spark 3, which is also worth going through.

Also, will be a good idea to refer to the Spark UI document page — here

You can also reach out to me on Linkedin

Want to know how Apache Druid indexes the data for super fast querying? Check out below