Don't punish buffers during their correlated reference period