uwescience/myria — pull request #619 by dhalperi (2014-09-17 18:41:50+00:00)
Title: Deterministic producer

When we had redundant shuffles (i.e., hashing on the same field as we are already partitioned on), we were not keeping data on the same machine. What was happening is that the order of workers was not deterministic, so the same partition index might be mapped to different workers for different destination fragments. Making this order deterministic has two benefits:

1. performance: we use the in-process link instead of the network and the queuing issues associated with that.
2. correctness: though Raco does not do this yet, it seems possible (I have not proven it) that if raco elided a shuffle because a relation was already "hashed the right way" we might have ended up joining different partitions. No existing plan that could have been run actually suffered from this effect.