fix unintentionally retained finalizables and improve low mem performance

Objects which are eligable for finalization must be retained until
after their finalize methods are called.  However, the VM must
determine the entire set of such objects before retaining any of them;
otherwise the process of retaining a given object may cause others to
become reachable and thus be considered ineligible for finalization
even though they are only reachable via other finalizable objects.
The end result of this mistake is that only a few of the objects which
are finalizable will be recognized at each GC cycle, so it requires
many such cycles to find them all, and if new objects become
finalizable at a faster rate, the VM will never catch up and
eventually run out of memory.

This patch fixes the above mistake and also includes tuning to
minimize the need for GC in low memory situations.