HPCC-11693 Improove OOM handling when sizing tables

Prevent sizing tables, pushing out other memory (use allocate
with maxSpillCost). Catch a couple of other table sizing that
could OOM and ensure they fail over to local lookup join

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>