Cache ref lookups

This lowers the time from 8 seconds to 5.5 seconds.

Much slower than the previous version, but it does a lot more; so...

Signed-off-by: Francis Galiegue <fgaliegue@gmail.com>