Don't cache get_reverse_refs

The values of refs can change on disk, and this won't be reflected in
the cache. This dictionary must be just generated on the fly.