Sparklens on Qubole
By: Aayushi Das
Tags: Qubole, Optimization, How To, Github

An optimized code saves a lot of time and cost. But, optimizing a spark code is challenging and time taking, without understanding where the task is facing bottlenecks and what is the optimized run time a user can aspire to achieve. Such bottlenecks can be caused by mainly 3 reasons -

The driver compute time is large

There is a task skew

There are a lack of tasks, implying parallelism issues

Based on the situation, two optimization goals can be achieved — we can either work on reducing the runtime of the application, or we can make it optimal by making it work better with fewer resources. There has been a need for a solution that helps in identifying such bottlenecks which can help us in achieving the desired optimization goal.

How do we do it?

Sparklens helps in identifying such bottlenecks that a Spark application is facing. The Sparklens report gives us an idea of the next steps that can be taken up for optimization.

What is Sparklens?

Sparklens is an offering by Qubole. It is a profiling and performance prediction tool for Spark with a built-in Spark Scheduler simulator. Its primary goal is to make it easy to understand the scalability limits of Spark applications. It helps in understanding how efficiently a given Spark application uses the compute resources provided. Maybe your application will run faster with more executors, and maybe it won’t. SparkLens can answer this question by looking at a single run of your application. It helps in approaching spark application tuning as a well-defined method/process instead of something that one can learn by trial and error, thus saving both developer and compute time, as well as costs. Sparklens helps in identifying bottlenecks and provides us with critical path time.

Sparklens usage

Sparklens is very easy to use. There is no prerequisite and we don’t have to install anything specifically. We just have to pass a fixed set of commands along with the spark-submit shell, as mentioned below and right after the run, the Sparklens report gets generated.

— packages Qubole:sparklens:0.3.1-s_2.11

— conf spark.extraListeners=com.qubole.sparklens.QuboleJobListener

The major components that the Sparklens report talks about are -

Driver vs executor wall clock time — The total Spark application wall clock time can be divided into time spent on driver and time spent on executors. When a Spark application spends too much time on the driver, it wastes the executors’ compute time.

An estimate of ideal application runtime — Ideal application time is computed by assuming ideal partitioning (tasks == cores and no skew) of data in all stages of the application.

Critical path time — It is the minimum time that the application will take, even if we give it infinite executors. The difference between the critical path time and application runtime is the real elasticity of the Spark application. If the difference is large, then more executors are needed.

Compute hours — It tells us how much of the compute resources given to an application are not used at all.

Skewness and lack of tasks — Executors can also waste compute time because of lack of tasks or skew.

The beauty of the Sparklens report is that, if a person is looking at it for the very first time, then he/she will not be confused with the terminologies used, as it provides detailed definitions of the major parameters mentioned in the report.

Conclusion

The bottlenecks of a Spark job can be identified through the report, be it from the resource, data, or script side. The Sparklens report identifies the minimum time that could be achieved and with a fixed target time. We can understand how far we are from an optimized code. A critical path solution may not always be achievable, but we can try to reach as close as possible. Based on the findings of Sparklens, a detailed action and implementation plan can be prepared. Sparklens implementation can be carried out across different pipelines, as Sparklens is easy to use — certain fixed sets of commands can be passed along with the spark-submit shell. Sparklens reports will be generated with similar metrics each time, thus making it easier to understand.

References