Basics of Apache Spark Configuration Settings
By: Halil Ertan
Tags: Yarn, Hadoop, Pyspark, Apache Spark, Configuration Management

Spark Architecture — In a simple fashion

Before continuing further, I will mention Spark architecture and terminology in brief. Spark uses a master/slave architecture with a central coordinator called Driver and a set of executable workflows called Executors that are located at various nodes in the cluster.

Resource Manager is the decision-maker unit about the allocation of resources between all applications in the cluster, and it is a part of Cluster Manager. Cluster Manager is a process that controls, governs, and reserves computing resources in the form of containers on the cluster. There are lots of cluster manager options for Spark applications, one of them is Hadoop YARN.

When a Spark application launches, Resource Manager starts Application Master(AM) and allocates one container for it. AM coordinates the execution of all tasks within its application. AM can be considered as a non-executor container with the special capability of requesting containers from YARN, takes up resources of its own.

Once AM launches, it asks for containers and resource requests of containers from Resource Manager. After successfully receiving the containers, AM launches them. Inside the containers, it runs the user application code via Driver. After completion of the application, AM releases the resources back to Resource Manager. If AM crashes or becomes unavailable, Resource Manager can create another container and restart AM on it.

Driver is placed inside AM in cluster mode and responsible for converting a user application to smaller execution units called tasks and then schedules them to run on executors. These tasks are executed on the worker nodes and then return results to the Driver. Driver also informs AM of the executor’s needs for the application. It runs in its own JVM. Executors are the processes(computing units) at the worker’s nodes, whose job is to complete the assigned tasks. One executor container is just one JVM.

Spark Context is the main entry point into Spark functionality. Spark Context also tracks executors in real-time by sending regular heartbeat messages. Spark Context is created by Driver for each Spark application when it is first submitted by the user. It exists throughout the lifetime of the Spark application. Spark Context stops working after the Spark application is finished.

There are two different running modes available for Spark jobs — client mode and cluster mode. The difference basically depends on where Driver is running. While it runs in the client process in client mode, it runs in the AM in cluster mode. In production, cluster mode makes sense, the client can go away after initializing the application.

YARN Dependent Parameters

One of the leading cluster management frameworks for Spark is YARN. In YARN terminology, executors and AM run inside containers. A container is an allocation of memory and CPU simply. When a Spark application is submitted through YARN in the cluster mode, the resources will be allocated in the form of containers by the Resource Manager.

In the yarn-site.xml file, adjusting the following parameters is a good starting point if Spark is used together with YARN as a cluster management framework. In the case of running Spark applications with non-spark applications in the same cluster, firstly setting these YARN parameters correctly is very important. In a way, these parameters will define the borders of your Spark applications within the cluster. The followings are the basic YARN parameters to be set correctly.

yarn.nodemanager.resource.memory-mb

yarn.scheduler.maximum-allocation-mb

yarn.scheduler.minimum-allocation-mb

yarn.nodemanager.resource.cpu-vcores

yarn.scheduler.maximum-allocation-vcores

yarn.scheduler.minimum-allocation-vcores

yarn.nodemanager.resource.memory-mb simply refers to the amount of physical memory that can be allocated for containers in a single node. It must be something lower than the total RAM value of the node considering the OS daemons and other running processes in the node. yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb parameters state minimum and maximum memory allocation values respectively a single container can get.

Similar reasoning is valid for CPU allocation of containers. yarn.nodemanager.resource.cpu-vcores determines total available vcores allocated for all containers in a single node. And each container gets vcores within the values of yarn.scheduler.minimum-allocation-vcores and yarn.scheduler.maximum-allocation-vcores parameters as the lower and upper limit.

Memory Management in Spark

Memory utilization is a bit more tricky compared to CPU utilization in Spark. Before deep dive into the configuration tuning, it would be useful to look at what is going on under the hood in memory management. A good summarization for memory management in Spark is depicted in the following diagram.