hpcc-systems/HPCC-Platform — pull request #6600 by jakesmith (2014-11-03 15:59:31+00:00)
Title: HPCC-11928 Smart Join failing over to full join too early.

When smart join ran out of memory, it attempted to 1st failover
to a local lookup join, however, in doing so it consumed too
much memory too early, causing the local lookup to failover to
a standard join too early.
The reasons it used too much memory too early were:

1) A distributor on the LHS was started asynchronously, meaning
the receiver side started buffering up rows contributing to an
overall shortage of memory.

2) The local row tables from each node that are built up as
part of the 1st phase (std. lookup) are cleared of non-local rows
by the first OOM (failover to local lookup), these partial row
arrays can be quite big and sparse and had were not spillable.
a) Transfer the biggest chunk into the loader for the rest, to
allow as much as possible to free up from these row arrays
b) TBD: realloc the sprase array ptr to smaller required size
(See HPCC-12511)
c) Make them spillable at a lower priority than gathering to
local join stage. If the gather OOM's then we have to failover
to std. join, spilling the row arrays may free up enough memory
for local lookup to fit.

Signed-off-by: Jake Smith