Fix a case where the highlight count would remain visible in the sidebar for the selected buffer