Eliminate the backref scan.

It turns out to be unproductive, and has the biggest hit for the
case of processing new data, which is the case about which we
should be most conservative because it has the lowest throughput,
and so the least room to absorb a hit, whereas the case of seen
data allows us to take a little hit since we're more likely to
be able to process data faster than we get it.