HPCC-12309 Catch memory wrap and full hash table

checked_malloc and checked_reallooc should have been using size_t parameters
otherwise allocations of 2^32 were assumed to be size 0.  There was also
no checking on hash table overflow (2^32 elements)

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>