Stop using backlink cache for link counts.

This makes sure we get accurate link counts which is better for testing.
It is also faster, at least in development where there aren't too many
links.

For bug 53428

Change-Id: Ibfa7ca618523c209013b419e81125cbb51c45aea