py: Add m_malloc_fail function to handle memory allocation error.

A malloc/realloc fail now throws MemoryError.
py: This time, real proper overflow checking of small int power.

Previous overflow test was inadequate.
tests: Make rge-sm.py use less memory (integration step increased).

In addition to memory allocation failure, other problem was that 10**17
was overflowing.  Now passes.