hpcc-systems/HPCC-Platform — pull request #6710 by jakesmith (2014-12-01 15:33:02+00:00)
Title: HPCC-11928 Smart Join failing over to full join too early.

When smart join ran out of memory, it attempted to 1st failover
to a local lookup join, however, in doing so it consumed too
much memory too early, causing the local lookup to failover to
a standard join too early.
Improvements:
1) Avoid starting the LHS distributor too early, meaning

2) The local row tables from each node that are built up as
part of the 1st phase (std. lookup) are cleared of non-local rows
by the first OOM (failover to local lookup), these partial row
arrays can be quite big and sparse and had were not spillable.
a) Transfer the biggest chunk into the loader for the rest, to
allow as much as possible to free up from these row arrays
b) TBD: realloc the sprase array ptr to smaller required size
(See HPCC-12511)
c) Make them spillable at a lower priority than gathering to
local join stage. If the gather OOM's then we have to failover
to std. join, spilling the row arrays may free up enough memory
for local lookup to fit.

3) When the broadcast detects an OOM and clears non-locals, it
must still deal with rows in transit, these can cause another OOM
particularly if they involve a ptr array expansion. Handle by
spilling to a separate single overflow (there won't be that many)

4) Needed a separate final broadcast to check if anyone had
failed over from local lookup to full lookup

5) When failing over to local lookup, transfer the largest slave
array into the loader, to free up it's memory (ptr table) asap

6) Ensure ptr table arrays are destroyed by stream spillers asap,
i.e. when all rows consumed, or spilt and remove MM buffer
callbacks asap also, to avoid pointless callbacks to something
that can no longer spill

7) Rename a few test member variables for clarity,
e.g. isLocalLookup()

8) Add some more comments to explain tactics

9) Take account of local and global already sorted meta data
a)Spotted that marker may have been sorting when didn't need to
b) Avoided rhs rows moving in and out of collector if already
collated.

Signed-off-by: Jake Smith