Wait for SparkListenerJobEnd before checking DAGScheduler end state