twitter/scalding — pull request #229 by echen (2012-11-26 13:59:22+00:00)
Title: Add skew join

I haven't tested this out on a non-trivial dataset yet, and I'll probably want to clean some things up. This is mostly an fyi (and to get feedback if I'm doing anything wrong).

Note: to set the replication factor...
Suppose that key K appears in the sampled left pipe 3 times and in the sampled right pipe 5 times.
Then I simply replicate the rows K appears in 5 times each on the left, and 3 times each on the right. (Or, more generally, I have a replicationFactor parameter, and I replicate 5 * replicationFactor on the left, and 3 * replicationFactor on the right.)
This seems natural to me, but maybe I'm missing something?