apache/spark — pull request #3509 by SaintBacchus (2014-11-28 13:13:22+00:00)
Title: [Shuflle][SPARK-4641]file not found excepton in hash shuffle

It's very hard to replay this problem but actually I have meet this.
Assuming that, some hash key bucket have no elems in it and in the code of HashShuffleWriter.write:
for (elem <- iter) {
val bucketId = dep.partitioner.getPartition(elem._1)
shuffle.writers(bucketId).write(elem)
}
The bucket will never call the 'write' funciton and will never create the file(create file in first writer and file.length never check file is exist or not).
But the shuffle fetch will also fetch the unexist file, and causes the problem.
I have tested the problem described in SPARK-4641.
Sort Shuffle will also throw "snappy (parsing_error(2))" but can not be resolved by this modify.