Different approach to the mallocOutOfMemory problem: limit the heap size

Also made the test have a set end to its loop as Brad suggested instead of
letting it run to infinity when things don't work right.