Package: scopt
scoptSpark Config Optimizer calculate optimal cpu cores and memory values for Spark executor and driver.InstallingInstall scopt from PyPI via pip.pipinstallscoptUsageBasicfromscoptimportSparkConfOptimizerfromscopt.instancesimportInstanceexecutor_instance=Instance(32,250)num_nodes=10deploy_mode='client'sco=SparkConfOptimizer(executor_instance,num_nodes,deploy_mode)print(sco)# spark.driver.cores: 5# spark.driver.memory: 36g# spark.driver.memoryOverhead: 5g# spark.executor.cores: 5# spark.executor.memory: 36g# spark.executor.memoryOverhead: 5g# spark.executor.instances: 60# spark.default.parallelism: 600# spark.sql.shuffle.partitions: 600Cluster mode is also supported.deploy_mode='cluster'sco=SparkConfOptimizer(executor_instance,num_nodes,deploy_mode)print(sco)# spark.driver.cores: 5# spark.driver.memory: 36g# spark.driver.memoryOverhead: 5g# spark.executor.cores: 5# spark.executor.memory: 36g# spark.executor.memoryOverhead: 5g# spark.executor.instances: 59# spark.default.parallelism: 590# spark.sql.shuffle.partitions: 590Different instance type for driver node is also supported.
Specifying driver instance is enabled onlyclient mode.executor_instance=Instance(32,250)driver_instance=Instance(4,30)deploy_mode='client'sco=SparkConfOptimizer(executor_instance,num_nodes,deploy_mode,driver_instance,)print(sco)# spark.driver.cores: 3# spark.driver.memory: 26g# spark.driver.memoryOverhead: 3g# spark.executor.cores: 5# spark.executor.memory: 36g# spark.executor.memoryOverhead: 5g# spark.executor.instances: 60# spark.default.parallelism: 600# spark.sql.shuffle.partitions: 600Dynamic AllocationFor Spark dynamic allocation mode, you can calculate withdynamic_allocationis setTrue(defaultFalse).Not specify num_nodesWhendynamic_allocationisTrueandnum_nodesisNone, optimizer returns only Spark properties about resources (Not contains about parallelism likespark.default.parallelism).sco=SparkConfOptimizer(executor_instance,deploy_model=deploy_mode,dynamic_allocation=True,)print(sco)# spark.driver.cores: 3# spark.driver.memory: 26g# spark.driver.memoryOverhead: 3g# spark.executor.cores: 5# spark.executor.memory: 36g# spark.executor.memoryOverhead: 5gSpecify num_nodesIfdynamic_allocationsetTrue(defaultFalse) and specifynum_nodes, optimizer returnsspark.default.parallelismandspark.sql.shuffle.partitionsfor when executor nodes reach to num_nodes, but does not returnspark.executor.instances.sco=SparkConfOptimizer(executor_instance,num_nodes,deploy_model=deploy_mode,dynamic_allocation=True,)print(sco)# spark.driver.cores: 3# spark.driver.memory: 26g# spark.driver.memoryOverhead: 3g# spark.executor.cores: 5# spark.executor.memory: 36g# spark.executor.memoryOverhead: 5g# spark.default.parallelism: 600# spark.sql.shuffle.partitions: 600Predefined InstanceYou can use predefinedInstanceclass.
Currently supports AWS EC2 instance type.fromscopt.instances.awsimportAwsInstanceMapmapping=AwsInstanceMap()mapping['r5.4xlarge']# Instance(num_cores=16, memory_size=120)mapping['p3.8xlarge']# Instance(num_cores=4, memory_size=236)Set properties to SparkConfYou can set properties to SparkConf directory viaas_listmethod.frompysparkimportSparkConffromscoptimportSparkConfOptimizerfromscopt.instancesimportInstanceexecutor_instance=Instance(32,250)num_nodes=10deploy_mode='client'sco=SparkConfOptimizer(executor_instance,num_nodes,deploy_mode)conf=SparkConf()print(conf.getAll())# Property has not be set yet.# dict_items([])conf.setAll(sco.as_list())# dict_items([#     ('spark.driver.cores', '5'),#     ('spark.driver.memory', '36g'),#     ('spark.driver.memoryOverhead', '5g'),#     ('spark.executor.cores', '5'),#     ('spark.executor.memory', '36g'),#     ('spark.executor.memoryOverhead', '5g'),#     ('spark.executor.instances', '60'),#     ('spark.default.parallelism', '600'),#     ('spark.sql.shuffle.partitions', '600')# ])ReferenceBest practices for successfully managing memory for Apache Spark applications on Amazon EMR