apache/spark — pull request #3346 by Forevian (2014-11-18 14:54:38+00:00)
Title: [SPARK-2418] Custom checkpointing with an external function as parameter

https://issues.apache.org/jira/browse/SPARK-2418

If a job consists of many shuffle heavy transformations the current resilience model might be unsatisfactory. In our current use-case we need a persistent checkpoint that we can use to save our RDDs on disk in a custom location and load it back even if the driver dies. (Possible other use cases: store the checkpointed data in various formats: SequenceFile, csv, Parquet file, MySQL etc.)
After talking to Patrick Wendell at the Spark Summit 2014 we concluded that a checkpoint where one can customize the saving and RDD reloading behavior can be a good solution. I am open to further suggestions if you have better ideas about how to make checkpointing more flexible.

***

Note1: I deleted the previous fork as I had messed that version up by some unsuccessful rebasing attempt.
Note2: The contribution is my original work and I license the work to the project under the project's open source license.