Bug: Memory limit exception on high usage of direct buffer pool
Status: Open | Priority: Major | Resolution: 
Created: 10/Jul/20 23:53
Affects Version/s: 2.4.4

We have a spark application that uses jdbc to read a table from sql server and write as parquet file in s3. It simply reads data from jdbc and write as parquet file.

We are using numPartitions = 200. So, driver creates 200 executors and creates 200 parquet files each with each files size ranging from 500 to 700 MB.

Since maximum file size is 700MB, we assigned more than enough executor memory of 10GB.

Out of 200 tasks, ~ 1-3 tasks fail at first attempt with below error and get succedeed in retry. I tried running the job multiple times, every time 1-3 tasks get failed at first attempt and then get succeeded in 2nd or 3rd attempt and ultimately job get succeeded. Very rarely, even 3 rd attempt also fails resulting in job failure.
{noformat}
ExecutorLostFailure (executor 13 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 12.1 GB of 11.9 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling yarn.nodemanager.vmem-check-enabled because of YARN-4714.
{noformat}
I used [JVM Profile|https://github.com/uber-common/jvm-profiler] to see what is happening

First run
{noformat}
ConsoleOutputReporter - ProcessInfo: {"jvmInputArguments":"","role":"executor","jvmClassPath":"","epochMillis":1594423898037,"cmdline":"/usr/lib/jvm/java-openjdk/bin/java -server -Xmx10240m -Dlog4j.configuration=log4j_custom.properties -Droot.logging.level=INFO -Dapp.logging.level=DEBUG -javaagent:jvm-profiler-1.0.0.jar -Djava.io.tmpdir=/mnt/yarn/usercache/hadoop/appcache/masked/masked/tmp -Dspark.driver.port=40795 -Dspark.history.ui.port=18080 -Dspark.ui.port=0 -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/masked/masked -XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@masked:40795 --executor-id 10 --hostname masked.us-west-2.compute.internal --cores 1 --app-id masked --user-class-path file:/mnt/yarn/usercache/hadoop/appcache/masked/masked/__app__.jar --user-class-path file:/mnt/yarn/usercache/hadoop/appcache/masked/masked/sqljdbc42.jar ","appId":"masked","name":"31286@masked","host":"masked","processUuid":"9a6b4f57-d5df-4dfe-87bd-3205392d2a6c","agentVersion":"1.0.0","appClass":null,"xmxBytes":10737418240,"appJar":null}

ConsoleOutputReporter - CpuAndMemory: {"heapMemoryMax":9.54466304E9,"role":"executor","nonHeapMemoryTotalUsed":1.06792856E8,"bufferPools":[{"totalCapacity":9069172456,"name":"direct","count":280,"memoryUsed":9069172457},{"totalCapacity":0,"name":"mapped","count":0,"memoryUsed":0}],"heapMemoryTotalUsed":2.5333354E9,"vmRSS":13090037760,"epochMillis":1594424049590,"nonHeapMemoryCommitted":1.1071488E8,"heapMemoryCommitted":4.758437888E9,"memoryPools":[{"peakUsageMax":251658240,"usageMax":251658240,"peakUsageUsed":24027264,"name":"Code Cache","peakUsageCommitted":24248320,"usageUsed":21922944,"type":"Non-heap memory","usageCommitted":24248320},{"peakUsageMax":-1,"usageMax":-1,"peakUsageUsed":75585800,"name":"Metaspace","peakUsageCommitted":76898304,"usageUsed":75585800,"type":"Non-heap memory","usageCommitted":76898304},{"peakUsageMax":1073741824,"usageMax":1073741824,"peakUsageUsed":9284112,"name":"Compressed Class Space","peakUsageCommitted":9568256,"usageUsed":9284112,"type":"Non-heap memory","usageCommitted":9568256},{"peakUsageMax":3468165120,"usageMax":3044016128,"peakUsageUsed":3054501888,"name":"PS Eden Space","peakUsageCommitted":3054501888,"usageUsed":2171684568,"type":"Heap memory","usageCommitted":3036151808},{"peakUsageMax":280494080,"usageMax":260571136,"peakUsageUsed":227304480,"name":"PS Survivor Space","peakUsageCommitted":280494080,"usageUsed":189281504,"type":"Heap memory","usageCommitted":260571136},{"peakUsageMax":7158628352,"usageMax":7158628352,"peakUsageUsed":172369328,"name":"PS Old Gen","peakUsageCommitted":1461714944,"usageUsed":172369328,"type":"Heap memory","usageCommitted":1461714944}],"processCpuLoad":0.025501861918418012,"systemCpuLoad":0.3168226139213957,"processCpuTime":208310000000,"vmHWM":13090111488,"appId":"masked","vmPeak":22408749056,"name":"31286@masked","host":"masked","processUuid":"9a6b4f57-d5df-4dfe-87bd-3205392d2a6c","nonHeapMemoryMax":-1.0,"vmSize":22408744960,"gc":[{"collectionTime":799,"name":"PS Scavenge","collectionCount":25},{"collectionTime":489,"name":"PS MarkSweep","collectionCount":3}]}
{noformat}

Important thing note from above is direct buffer pool is using 9 GB
{noformat}
"bufferPools":[{"totalCapacity":9069172456,"name":"direct","count":280,"memoryUsed":9069172457}{noformat}

Second attempt


{noformat}
ConsoleOutputReporter - CpuAndMemory: \{"heapMemoryMax":9.54466304E9,"role":"executor","nonHeapMemoryTotalUsed":1.08760784E8,"bufferPools":[{"totalCapacity":186613945,"name":"direct","count":9,"memoryUsed":186613946},\{"totalCapacity":0,"name":"mapped","count":0,"memoryUsed":0}],"heapMemoryTotalUsed":4.7919392E8,"vmRSS":4663619584,"epochMillis":1594424315035,"nonHeapMemoryCommitted":1.1300864E8,"heapMemoryCommitted":4.180148224E9,"memoryPools":[\{"peakUsageMax":251658240,"usageMax":251658240,"peakUsageUsed":25152448,"name":"Code Cache","peakUsageCommitted":26279936,"usageUsed":23439488,"type":"Non-heap memory","usageCommitted":26279936},\{"peakUsageMax":-1,"usageMax":-1,"peakUsageUsed":76023336,"name":"Metaspace","peakUsageCommitted":77160448,"usageUsed":76023336,"type":"Non-heap memory","usageCommitted":77160448},\{"peakUsageMax":1073741824,"usageMax":1073741824,"peakUsageUsed":9297960,"name":"Compressed Class Space","peakUsageCommitted":9568256,"usageUsed":9297960,"type":"Non-heap memory","usageCommitted":9568256},\{"peakUsageMax":3490185216,"usageMax":2050490368,"peakUsageUsed":3131047936,"name":"PS Eden Space","peakUsageCommitted":3131047936,"usageUsed":40669520,"type":"Heap memory","usageCommitted":1742209024},\{"peakUsageMax":1192755200,"usageMax":755499008,"peakUsageUsed":955224704,"name":"PS Survivor Space","peakUsageCommitted":1192755200,"usageUsed":624832,"type":"Heap memory","usageCommitted":755499008},\{"peakUsageMax":7158628352,"usageMax":7158628352,"peakUsageUsed":933385336,"name":"PS Old Gen","peakUsageCommitted":1682440192,"usageUsed":437899568,"type":"Heap memory","usageCommitted":1682440192}],"processCpuLoad":7.326443423830174E-5,"systemCpuLoad":0.06332881529782401,"processCpuTime":454170000000,"vmHWM":13918568448,"appId":"application_1591825945062_6066","vmPeak":22872002560,"name":"31285@ip-100-106-124-90","host":"ip-100-106-124-90","processUuid":"9d00d0c1-2be2-406e-981e-31d55cd2fcac","nonHeapMemoryMax":-1.0,"vmSize":13525504000,"gc":[\{"collectionTime":2966,"name":"PS Scavenge","collectionCount":68},\{"collectionTime":1316,"name":"PS MarkSweep","collectionCount":5}]}
{noformat}

Direct buffer pool usage is now only 186MB. It is exactly same task, but interstingly in second attempt usage reduced from 9GB to only 186MB.

{noformat}
"bufferPools":[{"totalCapacity":186613945,"name":"direct","count":9,"memoryUsed":186613946}
{noformat}

I think by default Direct buffer pool  memory allocated in jvm is same as executor memory. So, I passed below config to limit it to 512 MB.


{noformat}
--conf spark.executor.extraJavaOptions="-XX:MaxDirectMemorySize=512m"
{noformat}


After i set this config, there is no more task failures. Even i reduced executor memory from 10GB to 4GB and it still worked fine without any issue.  I think we can increase MaxDirectMemorySize value to even higher but didn't try that.

We would like to know whether we can control the usage of direct buffer pool in spark without explictly setting its value, so that we can avoid yarn from killing executor for execeeding physical memory limit