hpcc-systems/HPCC-Platform — pull request #6741 by jakesmith (2014-12-08 17:47:58+00:00)
Title: HPCC-12633 Improve contention cost in splitter

In cases where the consumers keep pace and are waiting on the
produce side of the splitter, the cost of waiting was costly.
The change here, introduces a callback mechanism into the shared
write ahead, such that the producer [optionally] only triggers
the readers that there is something to consumer per page flush.
In the memory variety of the shared writer which blocks when it
is far enough ahead, the producer will write (and page) as fast
as possible on a async thread, and block when full.
All of this savs the cosumers, entering crits, waiting on
semaphores and waking up and re-entering crits etc. on a per row
basis when they are consuming quickly.

Add a memory pool of a few chunks to disk write too.
This substantially speeds up splitter that can spill, when the
arms are actually close to each other.

Also change QueueOf::dequeue(element) to return element.
It makes it a little cleaner when disposing of a link counted
item.

Signed-off-by: Jake Smith