[perf] avoid using the BufferDestructor since its slow to construct