twitter/scalding — pull request #725 by johnynek (2013-12-18 17:18:28+00:00)
Title: Set all the seeds in join algorithms

@dvryaboy noticed that restarts will get seeded differently if those happen. Those may (though, it is not 100% clear to me at the moment) be responsible for the slight differences we have seen on skewJoin on the Hadoop cluster in a few cases. I will look back into this on our cluster and see if I can replicate the error with this fix.

In any case, this is safer.

Last question: should we fix a seed completely and never use the time? I tend to think so, as this would give full reproducibility to the code and ease debugging, but I did not do it yet in this PR. Looking forward to comments (particularly if you can explain a failure mode that would overcount the skew join in the case of mapper restarts: does the estimated key count go wrong? does the random bucketing mess up? does it get bucketed one way but then rebucketed another causing an error?)

may fix #353