Improve memory handling

Less retaining of objects, allowing the gc to work better