Several reference counting fixes that were badly affecting memory use