Bug: RpcOutboxMessage: Ask terminated before connecting successfully
Status: Resolved | Priority: Major | Resolution: Invalid
Created: 03/Oct/22 02:38
Affects Version/s: 3.3.0

h1. {color:#ff0000}*Pyspark submit job stuck and infinitely retry.*{color}

 
h2. *pyspark job running with:*
{code:java}
$ PYSPARK_PYTHON=python SPARK_HOME="/Users/mike/Tools/spark-3.3.0-bin-hadoop3" pyspark --master spark://spark-master:7077                                                                                                                                          [10:20:25]
Python 3.8.13 (default, Mar 28 2022, 06:16:26)
[Clang 12.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
22/10/03 10:23:32 WARN Utils: Your hostname, codecan.local resolves to a loopback address: 127.0.0.1; using 192.168.31.31 instead (on interface en5)
22/10/03 10:23:32 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/10/03 10:23:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.3.0
      /_/Using Python version 3.8.13 (default, Mar 28 2022 06:16:26)
Spark context Web UI available at http://192.168.31.31:4040
Spark context available as 'sc' (master = spark://spark-master:7077, app id = app-20221003022333-0000).
SparkSession available as 'spark'.
>>> from pyspark.sql.functions import col
>>> spark.range(0,5).select(col("id").cast("double")).agg({'id': 'sum'}).show()
22/10/03 10:24:24 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/03 10:24:39 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/03 10:24:54 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/03 10:25:09 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/03 10:25:24 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources {code}
h2. spark-defaults.conf
{code:java}
spark.driver.port 13333
spark.executor.memory 512m
spark.executor.cores 1
spark.executor.instances 2
spark.cores.max 1
spark.shuffle.service.enabled false
spark.dynamicAllocation.enabled false {code}
h2. *stderr log page for app-20221003022333-0000/0*
{code:java}
Spark Executor Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx512M" "-Dspark.driver.port=13333" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.31.31:13333" "--executor-id" "0" "--hostname" "spark-worker-1" "--cores" "1" "--app-id" "app-20221003022333-0000" "--worker-url" "spark://Worker@spark-worker-1:6061"
========================================

Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:424)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:413)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.lookupTimeout
	at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)
	at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:62)
	at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:58)
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
	at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:444)
	at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
	at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
	at scala.collection.immutable.Range.foreach(Range.scala:158)
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:442)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	... 4 more
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:293)
	at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
	... 16 more {code}
 
h2.  stdout log page for app-20221003022333-0000/0

 
{code:java}
22/10/03 02:23:35 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 107@spark-worker-1
22/10/03 02:23:35 INFO SignalUtils: Registering signal handler for TERM
22/10/03 02:23:35 INFO SignalUtils: Registering signal handler for HUP
22/10/03 02:23:35 INFO SignalUtils: Registering signal handler for INT
22/10/03 02:23:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/10/03 02:23:35 INFO SecurityManager: Changing view acls to: spark,mike
22/10/03 02:23:35 INFO SecurityManager: Changing modify acls to: spark,mike
22/10/03 02:23:35 INFO SecurityManager: Changing view acls groups to: 
22/10/03 02:23:35 INFO SecurityManager: Changing modify acls groups to: 
22/10/03 02:23:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark, mike); groups with view permissions: Set(); users  with modify permissions: Set(spark, mike); groups with modify permissions: Set()
22/10/03 02:25:35 ERROR RpcOutboxMessage: Ask terminated before connecting successfully
22/10/03 02:25:35 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to /192.168.31.31:13333 timed out (120000 ms)
22/10/03 02:27:35 ERROR RpcOutboxMessage: Ask terminated before connecting successfully
22/10/03 02:27:35 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to /192.168.31.31:13333 timed out (120000 ms) {code}
 
h2. *docker-compose.yml:*
{code:java}
version: '3'services:
  spark-master:
    image: docker.io/bitnami/spark:3.3
    hostname: spark-master
    environment:
      - SPARK_MODE=master
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_LOCAL_HOSTNAME=spark-master  
    ports:
      - '8080:8080'
      - '7077:7077'
    networks:
      - spark-network
      
      
  spark-worker-1:
    image: docker.io/bitnami/spark:3.3
    hostname: spark-worker-1
    depends_on: 
      - spark-master
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=4g
      - SPARK_WORKER_CORES=8
      - SPARK_WORKER_PORT=6061
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_LOCAL_HOSTNAME=spark-worker-1
    ports:
      - '14040:4040'
      - '18081:8081'
      - '16061:6061'
    networks:
      - spark-network
      
      
  spark-worker-2:
    image: docker.io/bitnami/spark:3.3
    hostname: spark-worker-2
    depends_on: 
      - spark-worker-1
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_MEMORY=4g
      - SPARK_WORKER_CORES=8
      - SPARK_WORKER_PORT=6062
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
      - SPARK_LOCAL_HOSTNAME=spark-worker-2
    ports:
      - '24040:4040'
      - '28081:8081'    
      - '26062:6062'
    networks:
      - spark-networknetworks:
    spark-network:{code}
 
h2. *docker-compose log:*
{code:java}
$ docker-compose up                                                                                                                                                                                                                                                [10:21:46]
dcup='docker-compose up'
dco=docker-compose
[+] Running 3/3
 ⠿ Container spark-spark-master-1    Created                                                                                                                                                                                                                             0.0s
 ⠿ Container spark-spark-worker-1-1  Recreated                                                                                                                                                                                                                           0.1s
 ⠿ Container spark-spark-worker-2-1  Recreated                                                                                                                                                                                                                           0.1s
Attaching to spark-spark-master-1, spark-spark-worker-1-1, spark-spark-worker-2-1
spark-spark-master-1    |  02:23:19.14
spark-spark-master-1    |  02:23:19.14 Welcome to the Bitnami spark container
spark-spark-master-1    |  02:23:19.14 Subscribe to project updates by watching https://github.com/bitnami/containers
spark-spark-master-1    |  02:23:19.14 Submit issues and feature requests at https://github.com/bitnami/containers/issues
spark-spark-master-1    |  02:23:19.14
spark-spark-master-1    |  02:23:19.15 INFO  ==> ** Starting Spark setup **
spark-spark-master-1    | realpath: /bitnami/spark/conf: No such file or directory
spark-spark-master-1    |  02:23:19.17 INFO  ==> Detected mounted configuration file...
spark-spark-master-1    | find: '/docker-entrypoint-initdb.d/': No such file or directory
spark-spark-master-1    |  02:23:19.17 INFO  ==> No custom scripts in /docker-entrypoint-initdb.d
spark-spark-master-1    |  02:23:19.18 INFO  ==> ** Spark setup finished! **
spark-spark-master-1    |
spark-spark-master-1    |  02:23:19.19 INFO  ==> ** Starting Spark in master mode **
spark-spark-master-1    | starting org.apache.spark.deploy.master.Master, logging to /opt/bitnami/spark/logs/spark--org.apache.spark.deploy.master.Master-1-spark-master.out
spark-spark-master-1    | Spark Command: /opt/bitnami/java/bin/java -cp /opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host spark-master --port 7077 --webui-port 8080
spark-spark-master-1    | ========================================
spark-spark-worker-1-1  |  02:23:19.48
spark-spark-worker-1-1  |  02:23:19.48 Welcome to the Bitnami spark container
spark-spark-worker-1-1  |  02:23:19.48 Subscribe to project updates by watching https://github.com/bitnami/containers
spark-spark-worker-1-1  |  02:23:19.48 Submit issues and feature requests at https://github.com/bitnami/containers/issues
spark-spark-worker-1-1  |  02:23:19.48
spark-spark-worker-1-1  |  02:23:19.50 INFO  ==> ** Starting Spark setup **
spark-spark-worker-1-1  | realpath: /bitnami/spark/conf: No such file or directory
spark-spark-worker-1-1  |  02:23:19.53 INFO  ==> Generating Spark configuration file...
spark-spark-worker-1-1  | find: '/docker-entrypoint-initdb.d/': No such file or directory
spark-spark-worker-1-1  |  02:23:19.54 INFO  ==> No custom scripts in /docker-entrypoint-initdb.d
spark-spark-worker-1-1  |  02:23:19.55 INFO  ==> ** Spark setup finished! **
spark-spark-worker-1-1  |
spark-spark-worker-1-1  |  02:23:19.57 INFO  ==> ** Starting Spark in worker mode **
spark-spark-worker-1-1  | starting org.apache.spark.deploy.worker.Worker, logging to /opt/bitnami/spark/logs/spark--org.apache.spark.deploy.worker.Worker-1-spark-worker-1.out
spark-spark-worker-1-1  | Spark Command: /opt/bitnami/java/bin/java -cp /opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 --port 6061 spark://spark-master:7077
spark-spark-worker-1-1  | ========================================
spark-spark-worker-2-1  |  02:23:19.98
spark-spark-worker-2-1  |  02:23:19.98 Welcome to the Bitnami spark container
spark-spark-worker-2-1  |  02:23:19.98 Subscribe to project updates by watching https://github.com/bitnami/containers
spark-spark-worker-2-1  |  02:23:19.98 Submit issues and feature requests at https://github.com/bitnami/containers/issues
spark-spark-worker-2-1  |  02:23:19.99
spark-spark-worker-2-1  |  02:23:20.00 INFO  ==> ** Starting Spark setup **
spark-spark-worker-2-1  | realpath: /bitnami/spark/conf: No such file or directory
spark-spark-worker-2-1  |  02:23:20.03 INFO  ==> Generating Spark configuration file...
spark-spark-worker-2-1  | find: '/docker-entrypoint-initdb.d/': No such file or directory
spark-spark-worker-2-1  |  02:23:20.04 INFO  ==> No custom scripts in /docker-entrypoint-initdb.d
spark-spark-worker-2-1  |  02:23:20.04 INFO  ==> ** Spark setup finished! **
spark-spark-worker-2-1  |
spark-spark-worker-2-1  |  02:23:20.06 INFO  ==> ** Starting Spark in worker mode **
spark-spark-worker-2-1  | starting org.apache.spark.deploy.worker.Worker, logging to /opt/bitnami/spark/logs/spark--org.apache.spark.deploy.worker.Worker-1-spark-worker-2.out
spark-spark-worker-2-1  | Spark Command: /opt/bitnami/java/bin/java -cp /opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 --port 6062 spark://spark-master:7077
spark-spark-worker-2-1  | ========================================
spark-spark-master-1    | Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
spark-spark-master-1    | 22/10/03 02:23:20 INFO Master: Started daemon with process name: 41@spark-master
spark-spark-master-1    | 22/10/03 02:23:20 INFO SignalUtils: Registering signal handler for TERM
spark-spark-master-1    | 22/10/03 02:23:20 INFO SignalUtils: Registering signal handler for HUP
spark-spark-master-1    | 22/10/03 02:23:20 INFO SignalUtils: Registering signal handler for INT
spark-spark-worker-1-1  | Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO Worker: Started daemon with process name: 40@spark-worker-1
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO SignalUtils: Registering signal handler for TERM
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO SignalUtils: Registering signal handler for HUP
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO SignalUtils: Registering signal handler for INT
spark-spark-master-1    | 22/10/03 02:23:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
spark-spark-master-1    | 22/10/03 02:23:21 INFO SecurityManager: Changing view acls to: spark
spark-spark-master-1    | 22/10/03 02:23:21 INFO SecurityManager: Changing modify acls to: spark
spark-spark-master-1    | 22/10/03 02:23:21 INFO SecurityManager: Changing view acls groups to:
spark-spark-master-1    | 22/10/03 02:23:21 INFO SecurityManager: Changing modify acls groups to:
spark-spark-master-1    | 22/10/03 02:23:21 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-worker-2-1  | Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
spark-spark-worker-2-1  | 22/10/03 02:23:21 INFO Worker: Started daemon with process name: 40@spark-worker-2
spark-spark-worker-2-1  | 22/10/03 02:23:21 INFO SignalUtils: Registering signal handler for TERM
spark-spark-worker-2-1  | 22/10/03 02:23:21 INFO SignalUtils: Registering signal handler for HUP
spark-spark-worker-2-1  | 22/10/03 02:23:21 INFO SignalUtils: Registering signal handler for INT
spark-spark-worker-1-1  | 22/10/03 02:23:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:23:21 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-master-1    | 22/10/03 02:23:21 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
spark-spark-master-1    | 22/10/03 02:23:21 INFO Master: Starting Spark master at spark://spark-master:7077
spark-spark-master-1    | 22/10/03 02:23:21 INFO Master: Running Spark version 3.3.0
spark-spark-master-1    | 22/10/03 02:23:21 INFO Utils: Successfully started service 'MasterUI' on port 8080.
spark-spark-master-1    | 22/10/03 02:23:21 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://spark-master:8080
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO Utils: Successfully started service 'sparkWorker' on port 6061.
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO Worker: Worker decommissioning not enabled.
spark-spark-worker-2-1  | 22/10/03 02:23:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-master-1    | 22/10/03 02:23:22 INFO Master: I have been elected leader! New state: ALIVE
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO Worker: Starting Spark worker spark-worker-1:6061 with 8 cores, 4.0 GiB RAM
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO Worker: Running Spark version 3.3.0
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO Worker: Spark home: /opt/bitnami/spark
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO ResourceUtils: ==============================================================
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO ResourceUtils: No custom resources configured for spark.worker.
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO ResourceUtils: ==============================================================
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO Utils: Successfully started service 'sparkWorker' on port 6062.
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO Worker: Worker decommissioning not enabled.
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://spark-worker-1:8081
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO Worker: Connecting to master spark-master:7077...
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO TransportClientFactory: Successfully created connection to spark-master/172.18.0.2:7077 after 27 ms (0 ms spent in bootstraps)
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO Worker: Starting Spark worker spark-worker-2:6062 with 8 cores, 4.0 GiB RAM
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO Worker: Running Spark version 3.3.0
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO Worker: Spark home: /opt/bitnami/spark
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO ResourceUtils: ==============================================================
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO ResourceUtils: No custom resources configured for spark.worker.
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO ResourceUtils: ==============================================================
spark-spark-master-1    | 22/10/03 02:23:22 INFO Master: Registering worker spark-worker-1:6061 with 8 cores, 4.0 GiB RAM
spark-spark-worker-1-1  | 22/10/03 02:23:22 INFO Worker: Successfully registered with master spark://spark-master:7077
spark-spark-worker-2-1  | 22/10/03 02:23:22 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
spark-spark-worker-2-1  | 22/10/03 02:23:23 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://spark-worker-2:8081
spark-spark-worker-2-1  | 22/10/03 02:23:23 INFO Worker: Connecting to master spark-master:7077...
spark-spark-worker-2-1  | 22/10/03 02:23:23 INFO TransportClientFactory: Successfully created connection to spark-master/172.18.0.2:7077 after 29 ms (0 ms spent in bootstraps)
spark-spark-master-1    | 22/10/03 02:23:23 INFO Master: Registering worker spark-worker-2:6062 with 8 cores, 4.0 GiB RAM
spark-spark-worker-2-1  | 22/10/03 02:23:23 INFO Worker: Successfully registered with master spark://spark-master:7077
spark-spark-master-1    | 22/10/03 02:23:33 INFO Master: Registering app PySparkShell
spark-spark-master-1    | 22/10/03 02:23:33 INFO Master: Registered app PySparkShell with ID app-20221003022333-0000
spark-spark-master-1    | 22/10/03 02:23:34 INFO Master: Launching executor app-20221003022333-0000/0 on worker worker-20221003022322-spark-worker-1-6061
spark-spark-worker-1-1  | 22/10/03 02:23:34 INFO Worker: Asked to launch executor app-20221003022333-0000/0 for PySparkShell
spark-spark-worker-1-1  | 22/10/03 02:23:34 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:23:34 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:23:34 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:23:34 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:23:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-worker-1-1  | 22/10/03 02:23:34 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx512M" "-Dspark.driver.port=13333" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.31.31:13333" "--executor-id" "0" "--hostname" "spark-worker-1" "--cores" "1" "--app-id" "app-20221003022333-0000" "--worker-url" "spark://Worker@spark-worker-1:6061"
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO Worker: Executor app-20221003022333-0000/0 finished with state EXITED message Command exited with code 1 exitStatus 1
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 0
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20221003022333-0000, execId=0)
spark-spark-master-1    | 22/10/03 02:27:36 INFO Master: Removing executor app-20221003022333-0000/0 because it is EXITED
spark-spark-master-1    | 22/10/03 02:27:36 INFO Master: Launching executor app-20221003022333-0000/1 on worker worker-20221003022322-spark-worker-1-6061
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO Worker: Asked to launch executor app-20221003022333-0000/1 for PySparkShell
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-worker-1-1  | 22/10/03 02:27:36 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx512M" "-Dspark.driver.port=13333" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.31.31:13333" "--executor-id" "1" "--hostname" "spark-worker-1" "--cores" "1" "--app-id" "app-20221003022333-0000" "--worker-url" "spark://Worker@spark-worker-1:6061"
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO Worker: Executor app-20221003022333-0000/1 finished with state EXITED message Command exited with code 1 exitStatus 1
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 1
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20221003022333-0000, execId=1)
spark-spark-master-1    | 22/10/03 02:31:37 INFO Master: Removing executor app-20221003022333-0000/1 because it is EXITED
spark-spark-master-1    | 22/10/03 02:31:37 INFO Master: Launching executor app-20221003022333-0000/2 on worker worker-20221003022322-spark-worker-1-6061
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO Worker: Asked to launch executor app-20221003022333-0000/2 for PySparkShell
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-worker-1-1  | 22/10/03 02:31:37 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx512M" "-Dspark.driver.port=13333" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.31.31:13333" "--executor-id" "2" "--hostname" "spark-worker-1" "--cores" "1" "--app-id" "app-20221003022333-0000" "--worker-url" "spark://Worker@spark-worker-1:6061"
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO Worker: Executor app-20221003022333-0000/2 finished with state EXITED message Command exited with code 1 exitStatus 1
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 2
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20221003022333-0000, execId=2)
spark-spark-master-1    | 22/10/03 02:35:39 INFO Master: Removing executor app-20221003022333-0000/2 because it is EXITED
spark-spark-master-1    | 22/10/03 02:35:39 INFO Master: Launching executor app-20221003022333-0000/3 on worker worker-20221003022322-spark-worker-1-6061
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO Worker: Asked to launch executor app-20221003022333-0000/3 for PySparkShell
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:35:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-worker-1-1  | 22/10/03 02:35:40 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx512M" "-Dspark.driver.port=13333" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.31.31:13333" "--executor-id" "3" "--hostname" "spark-worker-1" "--cores" "1" "--app-id" "app-20221003022333-0000" "--worker-url" "spark://Worker@spark-worker-1:6061"
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO Worker: Executor app-20221003022333-0000/3 finished with state EXITED message Command exited with code 1 exitStatus 1
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 3
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20221003022333-0000, execId=3)
spark-spark-master-1    | 22/10/03 02:39:42 INFO Master: Removing executor app-20221003022333-0000/3 because it is EXITED
spark-spark-master-1    | 22/10/03 02:39:42 INFO Master: Launching executor app-20221003022333-0000/4 on worker worker-20221003022322-spark-worker-1-6061
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO Worker: Asked to launch executor app-20221003022333-0000/4 for PySparkShell
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-worker-1-1  | 22/10/03 02:39:42 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx512M" "-Dspark.driver.port=13333" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.31.31:13333" "--executor-id" "4" "--hostname" "spark-worker-1" "--cores" "1" "--app-id" "app-20221003022333-0000" "--worker-url" "spark://Worker@spark-worker-1:6061"
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO Worker: Executor app-20221003022333-0000/4 finished with state EXITED message Command exited with code 1 exitStatus 1
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 4
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20221003022333-0000, execId=4)
spark-spark-master-1    | 22/10/03 02:43:44 INFO Master: Removing executor app-20221003022333-0000/4 because it is EXITED
spark-spark-master-1    | 22/10/03 02:43:44 INFO Master: Launching executor app-20221003022333-0000/5 on worker worker-20221003022322-spark-worker-1-6061
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO Worker: Asked to launch executor app-20221003022333-0000/5 for PySparkShell
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-worker-1-1  | 22/10/03 02:43:44 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx512M" "-Dspark.driver.port=13333" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.31.31:13333" "--executor-id" "5" "--hostname" "spark-worker-1" "--cores" "1" "--app-id" "app-20221003022333-0000" "--worker-url" "spark://Worker@spark-worker-1:6061"
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO Worker: Executor app-20221003022333-0000/5 finished with state EXITED message Command exited with code 1 exitStatus 1
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 5
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20221003022333-0000, execId=5)
spark-spark-master-1    | 22/10/03 02:47:47 INFO Master: Removing executor app-20221003022333-0000/5 because it is EXITED
spark-spark-master-1    | 22/10/03 02:47:47 INFO Master: Launching executor app-20221003022333-0000/6 on worker worker-20221003022322-spark-worker-1-6061
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO Worker: Asked to launch executor app-20221003022333-0000/6 for PySparkShell
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO SecurityManager: Changing view acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO SecurityManager: Changing modify acls to: spark
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO SecurityManager: Changing view acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO SecurityManager: Changing modify acls groups to:
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
spark-spark-worker-1-1  | 22/10/03 02:47:47 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx512M" "-Dspark.driver.port=13333" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.31.31:13333" "--executor-id" "6" "--hostname" "spark-worker-1" "--cores" "1" "--app-id" "app-20221003022333-0000" "--worker-url" "spark://Worker@spark-worker-1:6061" {code}
 
h2. *network log:*
{code:java}
$ ps -ef|grep java                                                                                                                                                                                                                                                 [10:47:25]   501 69079 69073   0 10:23AM ttys000    0:23.56 /Users/mike/.sdkman/candidates/java/current/bin/java -cp /Users/mike/Tools/spark-3.3.0-bin-hadoop3/conf/:/Users/mike/Tools/spark-3.3.0-bin-hadoop3/jars/* -Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED org.apache.spark.deploy.SparkSubmit --master spark://spark-master:7077 --name PySparkShell pyspark-shell   501 71014 69669   0 10:48AM ttys002    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox java






$ netstat -anv|grep 69079                                                                                                                                                                                                                                          [10:46:39] tcp4       0      0  192.168.31.31.65367    *.*                    LISTEN      131072 131072  69079      0 0x0100 0x00000006 tcp4       0      0  127.0.0.1.65366        127.0.0.1.7077         ESTABLISHED 386579 146988  69079      0 0x0102 0x00000008 tcp46      0      0  *.4040                 *.*                    LISTEN      131072 131072  69079      0 0x0000 0x00000006 tcp4       0      0  127.0.0.1.65363        127.0.0.1.65365        ESTABLISHED 407961 146988  69079      0 0x0002 0x00000004 tcp4       0      0  192.168.31.31.13333    *.*                    LISTEN      131072 131072  69079      0 0x0100 0x00000006 tcp4       0      0  127.0.0.1.65363        127.0.0.1.65364        ESTABLISHED 402840 146988  69079      0 0x0002 0x00000004 tcp4       0      0  127.0.0.1.65363        *.*                    LISTEN      131072 131072  69079      0 0x0100 0x00000006 kctl       0      0 131072   2048  69079      0     17      6 com.apple.netsrc  {code}