HPCC-8905 Fix minisort OOM and deadlock

Various problems causing mini sort to miscalculate in-memory
size, run out of memory and potentially deadlock

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>