Bug: Increase K8s IT timeout to 3 minutes
Status: Resolved | Priority: Major | Resolution: Fixed
Created: 07/Dec/20 00:33
Affects Version/s: 2.4.7

We are using 3 minutes in master/branch-3.1. This issue only happens at branch-3.0/branch-2.4

- https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36905/console
{code}
- Run PySpark with memory customization *** FAILED ***
  The code passed to eventually never returned normally. Attempted 70 times over 2.018373577433333 minutes. Last failure message: "++ id -u
  + myuid=0
  ++ id -g
  + mygid=0
  + set +e
  ++ getent passwd 0
  + uidentry=root:x:0:0:root:/root:/bin/bash
  + set -e
  + '[' -z root:x:0:0:root:/root:/bin/bash ']'
  + SPARK_K8S_CMD=driver-py
  + case "$SPARK_K8S_CMD" in
  + shift 1
  + SPARK_CLASSPATH=':/opt/spark/jars/*'
  + env
  + sort -t_ -k4 -n
  + sed 's/[^=]*=\(.*\)/\1/g'
  + grep SPARK_JAVA_OPT_
  + readarray -t SPARK_EXECUTOR_JAVA_OPTS
  + '[' -n '' ']'
  + '[' -n /opt/spark/tests/py_container_checks.py ']'
  + PYTHONPATH='/opt/spark/python/lib/pyspark.zip:/opt/spark/python/lib/py4j-*.zip:/opt/spark/tests/py_container_checks.py'
  + PYSPARK_ARGS=
  + '[' -n 209715200 ']'
  + PYSPARK_ARGS=209715200
  + R_ARGS=
  + '[' -n '' ']'
  + '[' 3 == 2 ']'
  + '[' 3 == 3 ']'
  ++ python3 -V
  + pyv3='Python 3.7.3'
  + export PYTHON_VERSION=3.7.3
  + PYTHON_VERSION=3.7.3
  + export PYSPARK_PYTHON=python3
  + PYSPARK_PYTHON=python3
  + export PYSPARK_DRIVER_PYTHON=python3
  + PYSPARK_DRIVER_PYTHON=python3
  + '[' -n '' ']'
  + '[' -z ']'
  + case "$SPARK_K8S_CMD" in
  + CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@" $PYSPARK_PRIMARY $PYSPARK_ARGS)
  + exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=172.17.0.4 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner /opt/spark/tests/worker_memory_check.py 209715200
  20/12/07 00:09:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  20/12/07 00:09:33 INFO SparkContext: Running Spark version 2.4.8-SNAPSHOT
  20/12/07 00:09:33 INFO SparkContext: Submitted application: PyMemoryTest
  20/12/07 00:09:33 INFO SecurityManager: Changing view acls to: root
  20/12/07 00:09:33 INFO SecurityManager: Changing modify acls to: root
  20/12/07 00:09:33 INFO SecurityManager: Changing view acls groups to: 
  20/12/07 00:09:33 INFO SecurityManager: Changing modify acls groups to: 
  20/12/07 00:09:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
  20/12/07 00:09:34 INFO Utils: Successfully started service 'sparkDriver' on port 7078.
  20/12/07 00:09:34 INFO SparkEnv: Registering MapOutputTracker
  20/12/07 00:09:34 INFO SparkEnv: Registering BlockManagerMaster
  20/12/07 00:09:34 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
  20/12/07 00:09:34 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
  20/12/07 00:09:34 INFO DiskBlockManager: Created local directory at /var/data/spark-9950d0f1-8753-441f-97cf-1aa6defd1d0e/blockmgr-9f6bcf4d-ff41-4b27-8312-0fb23bf4ed1b
  20/12/07 00:09:34 INFO MemoryStore: MemoryStore started with capacity 546.3 MB
  20/12/07 00:09:34 INFO SparkEnv: Registering OutputCommitCoordinator
  20/12/07 00:09:34 INFO Utils: Successfully started service 'SparkUI' on port 4040.
  20/12/07 00:09:34 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc:4040
  20/12/07 00:09:34 INFO SparkContext: Added file file:///opt/spark/tests/worker_memory_check.py at spark://spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc:7078/files/worker_memory_check.py with timestamp 1607299774831
  20/12/07 00:09:34 INFO Utils: Copying /opt/spark/tests/worker_memory_check.py to /var/data/spark-9950d0f1-8753-441f-97cf-1aa6defd1d0e/spark-8ae1ff4f-2989-43d3-adfe-f26e8ff71ed2/userFiles-cfe3880e-6803-4809-9c01-6f1f582e4481/worker_memory_check.py
  20/12/07 00:09:34 INFO SparkContext: Added file file:///opt/spark/tests/py_container_checks.py at spark://spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc:7078/files/py_container_checks.py with timestamp 1607299774847
  20/12/07 00:09:34 INFO Utils: Copying /opt/spark/tests/py_container_checks.py to /var/data/spark-9950d0f1-8753-441f-97cf-1aa6defd1d0e/spark-8ae1ff4f-2989-43d3-adfe-f26e8ff71ed2/userFiles-cfe3880e-6803-4809-9c01-6f1f582e4481/py_container_checks.py
  20/12/07 00:09:36 INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes.
  20/12/07 00:09:36 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079.
  20/12/07 00:09:36 INFO NettyBlockTransferService: Server created on spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc:7079
  20/12/07 00:09:36 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  20/12/07 00:09:36 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc, 7079, None)
  20/12/07 00:09:36 INFO BlockManagerMasterEndpoint: Registering block manager spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc:7079 with 546.3 MB RAM, BlockManagerId(driver, spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc, 7079, None)
  20/12/07 00:09:36 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc, 7079, None)
  20/12/07 00:09:36 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc, 7079, None)
  20/12/07 00:10:06 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
  20/12/07 00:10:06 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/opt/spark/work-dir/spark-warehouse').
  20/12/07 00:10:06 INFO SharedState: Warehouse path is 'file:/opt/spark/work-dir/spark-warehouse'.
  20/12/07 00:10:07 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
  20/12/07 00:10:07 INFO SparkContext: Starting job: collect at /opt/spark/tests/worker_memory_check.py:43
  20/12/07 00:10:07 INFO DAGScheduler: Got job 0 (collect at /opt/spark/tests/worker_memory_check.py:43) with 2 output partitions
  20/12/07 00:10:07 INFO DAGScheduler: Final stage: ResultStage 0 (collect at /opt/spark/tests/worker_memory_check.py:43)
  20/12/07 00:10:07 INFO DAGScheduler: Parents of final stage: List()
  20/12/07 00:10:07 INFO DAGScheduler: Missing parents: List()
  20/12/07 00:10:07 INFO DAGScheduler: Submitting ResultStage 0 (PythonRDD[1] at collect at /opt/spark/tests/worker_memory_check.py:43), which has no missing parents
  20/12/07 00:10:07 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.5 KB, free 546.3 MB)
  20/12/07 00:10:07 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.1 KB, free 546.3 MB)
  20/12/07 00:10:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on spark-test-app-1607299769587-driver-svc.7c85102d1c1d4c8fb5a453963ab5535a.svc:7079 (size: 3.1 KB, free: 546.3 MB)
  20/12/07 00:10:07 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1184
  20/12/07 00:10:08 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (PythonRDD[1] at collect at /opt/spark/tests/worker_memory_check.py:43) (first 15 tasks are for partitions Vector(0, 1))
  20/12/07 00:10:08 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
  20/12/07 00:10:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  20/12/07 00:10:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  20/12/07 00:10:53 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  20/12/07 00:11:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  20/12/07 00:11:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  20/12/07 00:11:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
  " did not contain "PySpark Worker Memory Check is: True" The application did not complete.. (KubernetesSuite.scala:249)
{code}