Bug: Forever-running streams get completed with finished status when k8s worker is lost
Status: Open | Priority: Major | Resolution: 
Created: 08/Dec/21 18:20
Affects Version/s: 3.1.1

We have forever-running streaming jobs, defined as:

{{spark.readStream}}
{{.format("avro")}}
{{.option("maxFilesPerTrigger", 10)}}
{{.schema(schema)}}
{{.load(loadPath)}}
{{.as[T]}}
{{.writeStream}}
{{.option("checkpointLocation", checkpointPath)}}

We have several of these running, and at the end of main, I call

{{spark.streams.awaitAnyTermination()}}

To fail the job on any failing jobs.

Now, we are running on a k8s runner with arguments (showing only relevant ones):

{{- /opt/spark/bin/spark-submit}}
{{# From `kubectl cluster-info`}}
{{- --master}}
{{- k8s://https://SOMEADDRESS.gr7.us-east-1.eks.amazonaws.com:443}}
{{- --deploy-mode}}
{{- client}}
{{- --name}}
{{- ourstreamingjob}}
{{# Driver connection via headless service}}
{{- --conf}}
{{- spark.driver.host=spark-driver-service}}
{{- --conf}}
{{- spark.driver.port=31337}}
{{# Move ivy temp dirs under /tmp. See https://stackoverflow.com/a/55921242/760482}}
{{- --conf}}
{{- spark.jars.ivy=/tmp/.ivy}}
{{# JVM settings for ivy and log4j.}}
{{- --conf}}
{{- spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp -Dlog4j.configuration=file:///opt/spark/log4j.properties}}
{{- --conf}}
{{- spark.executor.extraJavaOptions=-Dlog4j.configuration=file:///opt/spark/log4j.properties}}
{{# Spark on k8s settings.}}
{{- --conf}}
{{- spark.executor.instances=10}}
{{- --conf}}
{{- spark.executor.memory=8g}}
{{- --conf}}
{{- spark.executor.memoryOverhead=2g}}
{{- --conf}}
{{{}- spark.kubernetes.container.image=xxx.dkr.ecr.us-east-1.amazonaws.com/{}}}{{{}ourstreamingjob{}}}{{{}:latesthash{}}}
{{- --conf}}
{{- spark.kubernetes.container.image.pullPolicy=Always}}



The container image is built with provided spark:3.1.1-hadoop3.2 image and we add our jar to it.

We have found issues with our monitoring that a couple of these streaming were not running anymore, and found that these streams were deemed "complete" with FINISHED status. I couldn't find any error logs from the streams themselves. We did find that the time when these jobs were completed happen to coincide with the time when one of the k8s workers were lost and a new worker was added to the cluster.

From the console, it says:
{quote}Executor 7
Removed at 2021/12/07 22:26:12
Reason: The executor with ID 7 (registered at 1638858647668 ms) was not found in the cluster at the polling time (1638915971948 ms) which is after the accepted detect delta time (30000 ms) configured by `spark.kubernetes.executor.missingPodDetectDelta`. The executor may have been deleted but the driver missed the deletion event. Marking this executor as failed.
{quote}
I think the correct behavior would be to continue on the streaming from the last known checkpoint. If there is really an error, it should throw and the error should propagate so that eventually the pipeline will terminate.

I've checked out changes since 3.1.1 but couldn't find any fix to this specific issue. Also, is there any workaround you could recommend?

Thanks!