Apache Spark: Out Of Memory Issue?
By: Aditi Sinha
Tags: Hadoop, Big Data, Data Analytics, Apache Spark, Data Engineering

Driver is a Java process where the main() method of our Java/Scala/Python program runs. It executes the code and creates a SparkSession/ SparkContext which is responsible to create Data Frame, Dataset, RDD to execute SQL, perform Transformation & Action, etc.

Executors are launched at the start of a Spark Application with the help of Cluster Manager. These can be dynamically launched and removed by the Driver as and when required. It runs an individual task and returns the result to the Driver. It can also persist data in the worker nodes for re-usability.

Each of these requires memory to perform all operations and if it exceeds the allocated memory, an OutOfMemory error is raised.

OutOfMemory at the Driver Level

OutOfMemory error can occur here due to incorrect usage of Spark. The driver in the Spark architecture is only supposed to be an orchestrator and is therefore provided less memory than the executors. You should always be aware of what operations or tasks are loaded to your driver. Few unconscious operations which we might have performed could also be the cause of error.

Collect()

Example:

val data = df.collect()

Collect() operation will collect results from all the Executors and send it to your Driver. The Driver will try to merge it into a single object but there is a possibility that the result becomes too big to fit into the driver’s memory.

Collect() in spark

We can solve this problem with two approaches: either use spark.driver.maxResultSize or repartition. Setting a proper limit using spark.driver.maxResultSize can protect the driver from OutOfMemory errors and repartitioning before saving the result to your output file can help too.

df.repartition(1).write.csv(“/output/file/path”)

Broadcast Join

There could be another scenario where you may be working with Spark SQL queries and there could be multiple tables being broadcasted. When performing a BroadcastJoin Operation,the table is first materialized at the driver side and then broadcasted to the executors. In this case there arise two possibilities to resolve this issue: either increase the driver memory or reduce the value for spark.sql.autoBroadcastJoinThreshold.

OutOfMemory at the Executor Level

Total executor memory = total RAM per instance / number of executors per instance

There are a few common reasons also that would cause this failure:

Inefficient queries High concurrency Incorrect configuration

Let’s understand these in detail.

Inefficient Queries

Example: Selecting all the columns from a Parquet/ORC table.

Explanation: Each column needs some in-memory column batch state. The overhead will directly increase with the number of columns being selected.

The Catalyst optimizer in Spark tries as much as possible to optimize the queries but it can’t help you with scenarios like this when the query itself is inefficiently written.

Solution: Try to reduce the load of executors by filtering as much data as possible, use partition pruning(partition columns) if possible, it will largely decrease the movement of data. But there could be another issue which can arise in case of big partitions. You can resolve it by setting the partition size: increase the value of spark.sql.shuffle.partitions.

2. High Concurrency

Apache Spark Parallel Processing.

Incase of an inappropriate number of spark cores for our executors, we will have to process too many partitions.All these will be running in parallel and will have it’s own memory overhead therefore, they would be needing the executor memory and can probably cause OutOfMemory errors.

To fix this, we can configure spark.default.parallelism and spark.executor.cores and based on your requirement you can decide the numbers.

3. Incorrect Configuration

Each Spark Application will have a different requirement of memory.

There is a possibility that the application fails due to YARN memory overhead issue(if Spark is running on YARN). Therefore, based on each requirement, the configuration has to be done properly so that output does not spill on disk. Configuring memory using spark.yarn.executor.memoryOverhead will help you resolve this.

e.g.

--conf “spark.executor.memory=12g”

--conf “spark.yarn.executor.memoryOverhead=2048”

or, --executor-memory=12g

Conclusion

By understanding the error in detail, the spark developers can get the idea of setting configurations properly required for their use case and application. Analyzing the error and its probable causes will help in optimizing the performance of operations or queries to be run in the application framework.

We hope this blog post will help you make better decisions while configuring properties for your spark application.

Thank you for reading this till the end. Hope you enjoyed it!

Explore more blogs at http://blog.clairvoyantsoft.com

References