Spark SQL in YARN: OutOfMemoryError Problem description
I'm trying to query data from ORC files which are stored in HDFS with Spark SQL. When I am working with small data volumes (2-5 Gb), I don't face any issues. However, in case I'm trying to process over 400 Gb I have an error on my executors.
What is the root cause of an issue - driver code or cluster configuration?
21/08/09 14:28:35 ERROR executor.Executor: Exception in task 70.0 in stage 1.0 (TID 90) java.lang.OutOfMemoryError: Java heap space
at org.apache.orc.impl.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:565)
at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readFileData(RecordReaderUtils.java:285)
at org.apache.orc.impl.RecordReaderImpl.readAllDataStreams(RecordReaderImpl.java:1147)
at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1103)
at org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1256)
at org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1291)
at org.apache.orc.impl.RecordReaderImpl.<init>(RecordReaderImpl.java:286)
at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:669)
at org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.initialize(OrcColumnarBatchReader.java:130)
at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.$anonfun$buildReaderWithPartitionValues$1(OrcFileFormat.scala:216)
at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$Lambda$1041/844883412.apply(Unknown Source)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:116)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:169)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
at org.apache.spark.sql.execution.FileSourceScanExec$$anon$1.hasNext(DataSourceScanExec.scala:503)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.columnartorow_nextBatch_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
at org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer$$anon$1.hasNext(InMemoryRelation.scala:118)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1423)
at org.apache.spark.storage.BlockManager$$Lambda$585/222823930.apply(Unknown Source)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1350)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1414)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1237)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)

Code of my Spark driver and environment configuration is given below.
Code
I use the following code:
SparkSession spark = SparkSession.builder().getOrCreate();

spark.read()
     .option(MERGE_SCHEMA_OPTION, true)
     .orc("/path/to/folder/with/orc/group/by/partition")
     .persist(StorageLevel.MEMORY_AND_DISK()))
     .createOrReplaceTempView("viewName");

spark.sql("SELECT * FROM viewName WHERE KEY IN ('2021-08-01', '2021-08-02') AND time >= '2021-08-01T00:00:00.000Z' AND time <= '2021-08-02T23:59:59.000Z' AND code > 400")
      .write()
      .orc(properties.getResultLocation();

In my solution I used all operators like WHERE and KEY IN in SQL string. As query structure depends on many parameters, it is the easiest way to build a query.
Data description
Data is presented as *.ORC files with the following schema:




Field
Type




message
string


code
int


rate
double


time
timestamp




All files with data are stored in partition folders named after time value field:
.
├── /path/to/folder/with/orc/group/by/partition
│   └──key=2021-08-01
│       ├── part_1.orc
│       ├── part_2.orc
│       └── part_...
│   └── key=2021-08-02
│       ├── part_1.orc
│       ├── part_2.orc
│       └── part_...
│   └── key=...
│       ├── ...
│       └── ...

For testing purposes I used ~400 GB of data (1.2 TB disk space consumed).
Environment configuration
I'm running my application in YARN cluster, which contains 3 Node Managers with the following configuration:




RAM
Cores




16 Gb
8




I submit my application with properties:




Property
Value




spark.executors.cores
5


spark.executor.memory
14 Gb


spark.yarn.executor.memory.overhead
2 Gb


spark.driver.memory
14 Gb


spark.driver.cores
5


spark.executor.instances
2


spark.default.parallelism
20


spark.driver.extraJavaOptions
-XX:+UseG1GC -XX:UseCompressedOops


spark.executor.extraJavaOptions
-XX:+UseG1GC -XX:UseCompressedOops




Spark version - 3.1.2
YARN version - 3.0.0-cdh6.1.1

with compression your 400GB of orc file can be 1.6TB or more of data. Also the parallelism is 20 so each executor is likely to get ~ 1/20 of the data (depending on how much of your query can be pushed down to prevent reading data into memory) - it might be likely that the amount of data that an executor with 14GB needs to read a chunk that is larger than its working memory
Try to increase parallelism and/or the memory in the executors (you may want to add executors as well).