Fix a crash found during the 10 minute nightly stress test. We were excessively allocating, freeing, and re-allocating.