hpcc-systems/HPCC-Platform — pull request #6791 by jakesmith (2014-12-23 11:27:58+00:00)
Title: HPCC-12779 Avoid dedupping distributor, sorting on all threads

Distribute uses a pool of threads to send to targets, if it is
also dedupping, the buckets are sorted.
These sorts use a parallel quick sort and (by default) use a thread
per core.

The # of rows being sorted is relatively small (depending on rows
size), so the parallel sort is not hugely beneficial per sort, and
when all senders are competing to sort, it is counterproductive,
overcommitting threads to cores.
In tests using a single threaded sort here (whilst still allowing
the parallel senders to sort in parallel) derived the best results.

Signed-off-by: Jake Smith