Some memory usage help

Essentially, decreasing the number of extraneous data structures kept around.
![memory_graph](https://f.cloud.github.com/assets/865586/600554/ed7030de-cc69-11e2-9fca-3d2385a4ce08.png)

Seems to help quite a bit as far as decreasing the run time as well.