Rework the way buffers get set so that the dependant relationship references are cleared