Correctly clean up staging directory.

This change also avoids overriding the app's status with "SUCCEEDED"
in cluster mode when the shutdown hook runs, by signaling the AM that
the SparkContext was shut down (see YarnClusterScheduler.scala). That
way the AM can correctly expose its final status to the RM.