mpi-sws-rse/datablox — Issue #39: Fault tolerance (data)
Author: t-saideep
Created: 2012-02-08 08:32:16+00:00
State: open
Labels: Analysis

http://www.cs.berkeley.edu/~matei/papers/2011/tr_spark.pdf

http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf

They talk about a specific model which lets them recompute intermediate results if there is data loss. We can use this as an alternate to saving intermediate data for blocks which can be recomputed. For example File-crawler can resend files if any intermediate node fails.