hpcc-systems/HPCC-Platform — Issue #1161: Fix sort/join OOM condition
Author: jakesmith
Created: 2011-12-19 17:39:02+00:00
State: open

Thor global sort (or join), use row arrays and estimate how much memory is used as they are filled.
When 'full' it spills to disk and re-fills etc.
If there are memory fragmentation issues, then this estimate can be a gross under-estimate of total OS memory used by the allocator. Consequently, memory could fill up and a out of memory condition fired before the row array estimate exceeded it's threshold.
Need to detect low allocator memory when gathering for sort/join.