Merge only changes into cached histograms to prevent flickering