hpcc-systems/HPCC-Platform — pull request #3336 by jakesmith (2012-09-13 10:57:31+00:00)
Title: HPCC-3286 Minisort could spill and lose records

Mini sort is supposed to estimate when all [global] records can fit into
memory, then gather, sort, partition on 1 node. Due to a few reasons and in
particular circumstances, it could spill whilst it was gathering the rows
from nodes 2->n onto node 1, and as a consequence lose records from the
final output.

The main bug was that a spilling flag was being ignored and therefore when
OOM it spilled instead of errored.
The OOM itself was probably mainly as a consequence of the memory manager
issue HPCC-3321, but also in part because the estimate for memory usage for
the minisort was too conservative (wasn't taking into account overheads).

Changes:
+ Fix disabled spilling flag conditions.
+ If minisort is used, the total row count is known, so pre-expand the large
row array.
+ Improve abort handling.
+ Improve size estiamte.

Signed-off-by: Jake Smith