freeman-lab/thunder — pull request #4 by JoshRosen (2014-01-17 23:40:37+00:00)
Title: Use reduce() instead of sortByKey() to find boundaries in localcorr

This change uses `reduce()` instead of `sortByKey()` to calculate the boundaries in `localcorr`.

Sorting is an expensive operation in Spark since it executes two jobs, one to compute ranges to achieve a balanced range partitioning, and another to actually shuffle and sort the data.

Reduce will be significantly more efficient, since it's effectively calculating the minimum or maximum value of each partition, then taking the minimum or maximum of those values.

There's still room to improve performance by computing all four boundaries in a single pass, but I decided to wait until I've written a clean "run multiple reductions in parallel" abstraction before implementing that.