This snippet actually escaped by cleanup :-) was not supposed to be part of the external commit request, but now that it is out there ...

So the reason why I was investigating this was - what happens to recovery when one or more nodes fail (or a node with multiple containers fails) : and there are joins/cogroups/etc which require all keys from previous iteration to recompute the missing rdd splits.
If I am not wrong, the current approach will cause reset back to initial state even if there is a checkpoint (since the checkpoint was on the node which is now lost).

Particularly for ML applications we are considering spark for, where the computation goes for hours on large cluster with 100's of input TB, this would be non trivial amount of wasted computation.


I hope my understanding is wrong ! We are yet to test these aspects - hence why the BlockManager changes are still empty.