apache/spark — pull request #3275 by mccheah (2014-11-15 00:15:12+00:00)
Title: [SPARK-4349] Checking if parallel collection partition is serializable 

Before, the DAGScheduler would determine if a task is serializable by doing a dry-run serialization of the first task in an array.

However, with parallel collection partitions, some partitions may be empty, and therefore they would be serializable, even though other non-empty partitions contain unserializable objects.

The solution presented here is a little hacky, to manually serialize one non-empty parallel collection partition if any exist in the task set. It would be ideal to implement a more generic solution, but that would involve extending the partition API. If more cases arise that involve some tasks being serializable and other tasks not being serializable in the same set, we would have no choice but to attempt to serialize all of them proactively, which could be expensive.