[SPARK-3090] Avoid not stopping SparkContext with YARN Client mode

Also, it's a better idea to use Hadoop's `ShutdownHookManager`. That allows you to make sure this runs before the HDFS shutdown hook, and avoid ugly exceptions (like "file system closed") showing up. See ApplicationMaster.scala, for example.