Scrollbar skips broken annotations without highlights.

Not sure why some of them don't have highlights, but it's causing all
sorts of things to break, so we just skip them when we are populating the
scrollbar.