Question: How do I set Spark parameters in Amazon EMR?

I want to configure Apache Spark parameters in Amazon EMR.

Short descriptionThere are two methods for configuring Spark applications:Use command line arguments such as the spark-submit command to avoid hardcoding values.Configure the values in the spark-defaults.conf file to makes the changes permanent.

ResolutionConfigure Spark parameters using spark-submitThe Spark shell and spark-submit command supports two ways to load configurations dynamically:Use command line options, such as --num-executors.Use the flag --conf.Note: Run spark-submit--help to show the complete options list.The spark-submit command also reads the configuration options from spark-defaults.conf, In the spark-defaults.conf file, each line consists of a key and a value separated by white space.For more information, see Submitting user applications with spark-submit.For more information on the parameters supported by Spark, see Spark configuration.The following are some of the most common configuration options:--class <main-class> \--master <master-url>--deploy-mode <deploy-mode> --conf <key>=<value> --num-executors <value> \--executor-memory <value>G \--driver-memory <value>G \--executor-cores <number of cores> \--driver-cores <number of cores> \--jars <Comma-separated list of jars> \--packages <Comma-separated list of Maven coordinates> \--py-files < Comma-separated list of .zip, .egg, or .py files to place on the PYTHONPATH for Python apps> \When using spark-submit, the application JAR, and any JARs included with the --jars option are automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. The list is included in the driver and executor class paths and the JARs and files are copied to the working directory for each SparkContext on the executor nodes. Keep in mind that directory expansion doesn't work with --jars.Examplespark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi --conf spark.dynamicAllocation.enabled=false --master yarn --num-executors 4 --driver-memory 4G --executor-memory 4G --executor-cores 1 /usr/lib/spark/examples/jars/spark-examples.jar 10You can pass the memory parameters using the flag --conf as shown in the following example:spark-submit --deploy-mode cluster --class org.apache.spark.examples.SparkPi --conf spark.dynamicAllocation.enabled=false --master yarn >--conf spark.driver.memory=1G --conf spark.executor.memory=1G /usr/lib/spark/examples/jars/spark-examples.jar 10Launch spark-shell and pyspark shell using custom Spark parametersTo launch spark-shell or pyspark shell, run the following commands:spark-shellspark-shell --conf spark.driver.maxResultSize=1G --conf spark.driver.memory=1G --deploy-mode client --conf spark.executor.memory=1G --conf spark.executor.heartbeatInterval=10000000s --conf spark.network.timeout=10000001s --executor-cores 1 --num-executors 5 --packages org.apache.spark:spark-avro_2.12:3.1.2 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'pyspark shellpyspark --conf spark.driver.maxResultSize=1G --conf spark.driver.memory=1G --deploy-mode client --conf spark.executor.memory=1G --conf spark.executor.heartbeatInterval=10000000s --conf spark.network.timeout=10000001s --executor-cores 1 --num-executors 5 --packages org.apache.spark:spark-avro_2.12:3.1.2 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'Configure Spark parameters using spark-defaults.confTo make the configuration changes permanent, append the configuration to the file /etc/spark/conf/spark-defaults.conf. Then, restart the Spark History Server. The following example configures the executor memory and driver memory in spark-defaults.conf. In this example, each line consists of a key and a value separated by white space.Examplespark.executor.memory      9486M spark.driver.memory     9486MThe following example configuration configures the Spark driver and executor memory during cluster launch:{  "Classification": "spark-defaults",  "Properties": {    "spark.executor.memory": "9486M",    "spark.driver.memory": "9486M"    }  }]Related informationBest practices for successfully managing memory for Apache Spark applications on Amazon EMRAdd a Spark stepModify your cluster on the fly with Amazon EMR reconfiguration