Turns out having a histogram buffer was faster after all