Attempt to limit how much is ever actually in memory

This is a WIP. The tests are failing locally for me and I cannot determine why.