Reference count buffers and clean them up properly.