Attempt to only store hole frames in the buffer