Fixed memory management issues

Cleaned up all the memory management, ensuring that we return all the
memory that we are allocating.
Cleaned up a little more memory

And removed 1 attempt.  Should reduce the number of attempts to under
11K when complete