[SPY-317] Backports of SPARK-1582, SPARK-1601 and SPARK-1602

https://issues.apache.org/jira/browse/SPARK-1582
https://issues.apache.org/jira/browse/SPARK-1601
https://issues.apache.org/jira/browse/SPARK-1602

SPARK-1582 shouldn't affect anything until we set interruptThread = true, at which point HDFS-1208 may bite us.