In gather script, handle SPARK_WORKER_DIRECTORY if set